Yes, I have a few suggestions; and you have a point about taking away people's right. What I really wanted to say is that people should be more careful with their downvotes and should always try to not use emotion when flagging someone. It shouldn't be done arbitrarily because it damages the users rep.
Here is one thing we need to adjust. When choosing the purpose for flagging, should it be for disagreement with payout, it should not affect the user's reputation. For other flagging options which are justfiable, like plagiarism etc, the effect on rep should come into play. Infact the flagging option should be changed to payment adjustement or we can remove it from the flagging option entirely and have it separate.
That flagging/downvoting could become a multi-step process sounds to me like it has potential!
;-) .