The comment about downvotes is interesting. I barely ever used the downvote option myself, and by that, I mean less than 10 times in total in the nearly 3 years I've been here! But, the reasons you made do make sense. It is a tool that allows the community to self police and root out undesirable activity.
I haven't used downvotes either but I believe it's not bad that they exist just to keep people in line and not abuse the reward pool. However there is also the danger in abusing the power of downvotes...