There are a lot of heated opinions on this. I've read a lot of people say that you shouldn't downvote unless it is a major violation like spamming or threatening someone, etc. Others look at it as you do, and just a different way of assigning value to posts.
One worry I have is retaliation. If I downvote someone's post, then I'm worried that they will start going out of their way to downvote all of mine. It could turn into a downvote war.
"Can't we all just get along?"
In any system with people there will be conflicts. You have to decide whether you want to let others exploit a platform that has value to you and do nothing to avoid retaliation, or if you are willing to take a stand. People will of course make different decisions.
I didn't and wouldn't say it is just a different way of assigning value, but when people are voting in a parasitic manner by piling on "sure things" without sufficient regard for quality nor consideration of the good of the platform that is also a form of abuse, and downvoting/flagging is exactly the way to control it.
The new reputation system makes it even harder to do what you are suggesting. Let's say someone writes an 'average' post, and a ton of people upvote it because it is trending and they want to get a good curation bonus. Then a person with high SP comes along and decides it is too high because everyone was colluding. It wasn't a bad post or anything - it is just overvalued. If they flag/downvote that person's post to lower the amount it gets paid, that person's reputation is affected too. Most likely the person who wrote the post didn't do anything intentionally wrong or bad to make the excessive upvotes happen. Yet their reputation score is penalized by the person who downvoted/flagged their content.
@timcliff All those upvotes (assuming the voters had positive reputation which is usually the case) increased the reputation. The later downvote reduce the reputation, reversing the effect of the upvotes. As long as the downvotes aren't extreme and overwhelming (which usually only happens in the case of serious abuse), reputation will usually be only slightly affected or even increase a little if the payout is merely reduced and not driven all the way to zero and beyond (which would be a waste of vote power by the downvoters).
@smooth I agree with @timcliff. You are trying to fix a disease by ingesting more of the parasite. Pouring more of the wrong to try to make it right.
Conflicts are a sign of a system that is not designed to be harmonious. I am entirely against your propensity (when given the role to do so) to want to top-down manage discussions and forums. I am ENTP. Seems you might have some J in there?
There will be blowback from judging. Remember Matthew 7. Instead let's figure out a way for every coterie to have their own rankings and preferences. One-size-fits-all are always power vacuums that we must fight over.
I have a good example for you. I came across @acassity spam posting a link to his content in a ton of posts. It was done in a way that was completely irrelevant and off-topic. I downvoted him, and he retaliated by going to my blog and downvoting posts of mine.
When I see 4 meta posts in a row for quick buck I will flag indeed and I care not if someone will pursue me and maybe even destroy my account. If people afraid to express true opinions the system is flowed.
Don't have down voting. You down vote by not voting at all.