Ah but if it engaged you enough to comment it must of had some worth, maybe another option could be for people to be able to give a partial vote. There may be problems with this if the person were able to vote more than a whole vote.
As to the downvote, if you hover over the flag icon it will say downvote. And I think downvote is a better term for it as flagging implies it is being brought to someone's attention and they will deal with it. While downvoting merely cancels an upvote, one thing that makes steemit's system offset is the fact that there is a reward to upvoting and none to downvoting.
@dantheman did a series of posts on negative voting
https://steemit.com/steem/@dantheman/negative-voting-and-steem