Damn! This was a very interesting story to read. I would not say it was a missed opportunity as there was clearly not much you could have done about it but I understand where you are coming from.
I think it is always a key element to be good at the referee part of a game to have played the game before. In your case they would have had a very excellent referee who took his job very seriously. That is really a shame that things worked out like they did. But I am sure you took a lot of lessons from it into other parts of your life and I also find it ironic that you insured yourself (probably even worse) by trying to take it easy :D
Life has twists and turns and you just have to go wherever they take you. You do learn a lot about yourself and about others during the ride. I know I would have done very well as I would have been the first ex forward referee blowing the whistle which is an area most of them fail to rule properly as they do not understand it.