That's a great question. I'm not an expert, but I will give you my understanding - which should hopefully be pretty close.
When you upvote/downvote, there are two things it affects:
-The payout of the post
-The reputation of the poster
The payout of the post is entirely based off of your SP. If you buy more SP, you will be able to affect the payout more.
The reputation of the poster is entirely based off of your reputation (I think). To affect the person's reputation score, you need a higher reputation.
As far as when you down-vote to the point that it becomes marked as spam / hidden - basically it is when the payout amount is negative. This means that SP is what you want to be able to have this control. If for example you got a ton of flags from users with low SP, and one user with high SP upvoted your post, then your post would not be marked as spam. If 10 people voted it up though, and someone with high SP flagged you - then it could cause it to go negative and be hidden / marked as a spam post.
Hope this helps. I apologize if anything I am saying here is not accurate, but I am pretty sure it is right. I know the community is working on a FAQ/wiki project, so hopefully within the next few months we will have these types of questions answered.