I realized that there is some fear when you start downvoting others. What if they become vengeful, and downvote you each time regardless of the quality of articles you produce?
I think this is one of the primary reasons why people tend to avoid hitting on that downvote button.