Not really true. The reputation goes down only if the vote is stronger than the upvotes. If the downvote is only worth the upvotes, and just negates them, it shouldn't affect reputation. But if the downvote is double the upvotes, it will soon start affecting reputation.
If someone's reputation is below 25, they've likely done something dubious, or even actually illegal, like for instance copyright infringement, plagiarizing, harmful hoaxes, spamming, tagspamming, or handlespamming.
You are okay with spam or hoaxes getting "censored", right?