You nailed it. I've thought this for some time - there's actually no solution right now that is actually "secure" and permanent.
Crypto is precisely about this trade off though, isn't it? Freedom. And Responsibility. The current financial system has all sorts of safe-guards that rely on a network of other systems to establish trust - government ID, account numbers, PINs, tax ID, security questions. They're all just creating multi-factor authentication schemes. Crypto has no real analog to this unless you, the user, choose to bring it on board. The choice and its trade offs are yours to make based on your comfort level.
The brain wallet is the closest thing you're going to get and requires discipline to rehearse the phrase regularly from now until you're tired of HODLing.
Possible solutions may exist in the future though - the whole underpinning of modern cryptography is that algorithms can be public and using a tiny seed of secret information, you can secure the whole thing. It's feasible that someone could create a paper-recovery-phrase algorithm that takes a secret-question style input to generate a key/recovery phrase and the algorithm can be run by hand. If we had that, a person could use secret-question style inputs that they are unlikely to forget (not random, tied to other significant things in their life that perhaps only they know), run the algorithm by hand to regenerate the recovery phrase, which they input into a computer to regenerate the key.
The only analogy in my mind is storing of physical gold. The same questions arise when analyzing the trust and storage question. At some point either you assume all the risk or you must trust someone.
Physical gold or bearer bonds are similar but slightly different (more secureable) - a person only needs to have temporary access to your phrase then they can recreate it outside the storage environment. In many ways, the seed information is far more sensitive than something you must physically transport/exchange/transact.