It's not an odd vote here and there, it's not about being angry at a downvote on a post of ours, it is about a pattern of daily downvotes, often hitting particular users time and time again, often hitting users on their one lucky trending post who ordinarily make peanuts. It's not about 'fair', it's about what is good and right, it's about a healthy friendly platform, and this behavior creates a toxic environment that is chasing good people away. Defend it all you like, it doesn't make it any more honorable, and the same applies for small payout posts losing half their rewards to smaller downvotes.
It's the nature of the behavior that's the issue here, particularly downvotes not accompanied by comments explaining why the downvote was given to a post, and although it is not the end of the world, it may be the end of a thriving hive blockchain.
Those of us who are regularly downvoted all the time by a variety of users for voicing politically incorrect views understand the scope of this issue, it's happened since before the hive fork, it's never ending, and it is easy to see that certain users and content is targeted far more than others for the content not level of rewards. Maybe you don't see this because you don't make content these whales and curation trails of downvoters out there disapprove of and police rewards on.