Exactly, you can always restore your funds unsing the backup-phrase. (My ledger uses a 24-word phrase). Jaxx wallets can also be restored using a 12-word backup phrase.
The main difference is the fact, that your mobile phone is often connected to the internet and installing apps may potentially give hackers access to your private keys. Here is a good overview about existing hardware-wallets.