You are wrong about:
He simply scrolls and decides and he doesn’t take into account some of those people got consistent one dollars before that post,
His down voting is easy to see and understand. He is not down voting you personally. He is removing reward value because one or two or three bad apples gave you what he feels is are excessive up votes to earn rewards based on your hard work. It is haejin, who is the problem of your down votes. If you do not want down votes tell haejin to go vote and troll someone else's post for rewards.
One day we may have the down vote system to where a specific up vote can be countered, until that day this is how it works.