Sort:  

You should see a number next to the flag and a minus instead of plus sign in the list of comments with the username of the person that flagged you.

The votes of the so called whales are worth hundreds of dollars each... :)

Someone must have taken a vote back as I do not see any minus signs. I wonder why... Thanks!

Yes, that is also a possiblity. I have done it myself a couple of times after mistakenly clicking on a vote button without actually willing to vote for something.