In all wealth there is risk for sure and you have to decide which ones you prefer, the personal responsibility kind or the third party kind. I prefer taking it on myself since theres more control, even if you lose or destory your hardware wallet you should have back ups, you should have a mutli-sig option and you should have duress wallets set up to protect as much of your wealth as possible.
If we compare bitcoin to any other asset, it's got the most security of anything out there