Yes! One common problem filthy rich people have is finding other people to "trust". Almost all people who befriend them or is interested with them romantically are most of the time only after their money. The worst thing is that it's not only limited to the one with all the excess money. Their sons/daughter or even relatives will likely have that problem too. It's now even hard to find pepple to trust when you are just a normal person or businessman, just imagine when you have that much excess money.
There is also the issue of "security". If you are filthy rich would you feel comfortable sending your children to a normal school? a school with a pretty lax security? when you know for yourself that a lot of bad people will try to do something to your child in exchange of money?
If you have kids, most likely they will rely too much on your money making them spoiled, lazy and too dependent on you. While it is not directly harmful to you it will affect the people around you. Just a good amount of money would probably be good enough to have.