Ohh sorry, I was trying to see the voting number but like you said, there's no way to change what has been accidentally clicked. Maybe the devs can do something about this.
EDIT: Yup I see why this user interface is prone to accidental DOWN votes, the word "Votes" is sandwiched between TWO similar symbols so a person can unintentionally click the DOWN vote while he meant to see the number of votes and who voted what.