I was honestly just as suprised by that as you were. We investigated the issue and it works like this.
My upvote can take you to +6.x. You had 6 before I voted you down and 7 after I voted you up.
If you start with 6.1 and I downvote you with 6.9 then it will take you to -6 because the way log algorithms work. If I downvote you again it probably wouldn't take you to -7.
Whales can make / break small accounts that have no rep. But they cannot make accounts with rep. I doubt I could lower @stellabelle from 9 to 8 with a single vote.
Reputation is hard to earn, easy to lose.