Have you read the FAQs recently? There is finally something there that explains why the value of a post drops so drastically sometimes. Yes, it is votes going to other voters. I guess the answer to your question is that the rewards would go to other people, and the value of their posts would drop less.
Why do the earnings for my post go up or down?
The amount that is shown next to a post is a "Potential Payout". This is an estimated value of how much money the post will make based on the votes that have occurred so far. Depending on various factors, this value can go up or down until the payout window closes:
If a post receives more upvotes, the potential payout of the post can go up.
If a post receives more downvotes, the potential payout of the post can go down.
If other posts receive more upvotes, the potential payout of the post can go down.
If other posts receive more downvotes, the potential payout of the post can go up.
If upvotes are removed from a post, the potential payout of the post can go down.
If downvotes are removed from a post, the potential payout of the post can go up.
If the price of STEEM goes up, the potential payout of all posts can go up.
If the price of STEEM goes down, the potential payout of all posts can go down.