If there was a reason someone should be able to remove the value of someone elses vote that might make sense.
The argument you'll get here on this is that a negative vote takes from the voting percentage just like a positive vote, but unlike the positive vote the negative vote gets no curation reward. So some people will likely argue they already are paying to cast their negative vote.