Just like with anything, there is good and there is bad. Sometimes a downvote is just used because the person doesn't feel that post deserves such a high payout - for being bad quality, and quality can be subjective sometimes.
Everyone who holds SP owns a portion of the rewards pool, so it's like saying "I'm going to take some of my rewards off that post and return them to my reward pool"
I don't think it should be used as a bullying mechanism though.