Seems like an interesting idea. Many of the commentators have expressed their concerns about a possible abusive behavior and an attack if there is such a feature and in a way I'll agree with that.
But adding a feature like that, gives us endless possibilities.
What if for every downvote someone gives to a witness, he losses the advantage of the 30 votes each time by 1?
Or downvotes could still be implemented but not having the same weight as the upvote, but only 50%? Or the first downvote to have a 50% weight, the second 45% the third 40% and so on?
Also there could be implemented a maximum limit for downvotes (example 3 or 5)
Or even better a combination of all the above...
I don't really know if what I propose is doable, but the more ideas we have the better result we'll get eventually.