Yeah anything physical they can take from you at gunpoint. It happened many times throughout history.
Now with the internet, dont even need to keep the numbers in your head...upload them to a few different places and just remember your passwords.
As for Cuba, Fidel might not have had the run he did if bitcoin was around.....