The trouble with downvoting is that it attracts revenge downvoting, and I'm not sure the downvoting pool would compensate for that.
Unless you set up a separate account just for downvoting and nothing else, no posts, no comments (in which case there is nothing to take revenge at).
The trick is to live by the age old philosophy of "don't be a dick" and usually things work out :D