No small fish dares to downvote a whale needlessly or without basis. Don't forget whales have the SP at their side.
The only case that High rep people would be affected is if another high rep or high SP downvoted them. In that case, I would like to think that these downvotes are not baseless!
Thank you for sharing and discussing.