The theory is that a wallet client should be open-source and the community will vet the code to make sure there are no trick or devilry within. Unfortunately, as The DAO recently exposed, mistakes are often left noticed and exploited by bad attackers.
Unless you have coded the client yourself, you can really never be sure of how safe your private keys are.