It affects balances a tiny bit because the fee isn't taken out of the transaction. Trust is a trickier matter: If I send you tokens but there's no easy-to-find record of it, will you really believe that I sent you tokens?
I see the no-confirmation transactions as a thing people do more of in the future when there are better resources for auditing transactions.