Then there will be endless debates and complaints: 1) whether the guidelines are "right", and 2) whether the guidelines are being correctly applied in a particular case.
Just accept that downvotes, like upvotes, are subjective and arbitrary.
I do feel that in the specific case where an organized voting/curation initiative feels the need to publish specific upvoting guidelines, it may, by similar logic, publish downvoting guidelines. But the above applies in general to most voters. Or adopt the neutral/subjective guideline:
"If I believe the payout is too low I will upvote and if I believe it is too high I will downvote"