Oh that's such a pity! Of course you had no idea about taking your shoes off when you visited a Mosque for the first time - I had no idea either before! So unlucky that you met this angry men :( - And to be fair, I wouldn't want to go inside anymore either after this angry guy appeared to me.
But you should definitely try it another time when you have the chance. People at Mosques are usually super nice and happy that foreigners are curious about their religion or religious places!
Yeah I thought the same - It did look quite rough on the outside. But as you mentioned, that might just be the building material they've used.
Wishing you a great week! :))