Yes, I am all for systems that facilitate consensus and that are at their core community-driven. With that said, when a single whale account can repeatedly nuke posts to zero, just because they personally disagree with the content or dislike the author or with one or more of the curators, it is not 'consensus'; it is merely the weaponization of stake.
Consensus can also counter those downvotes if they so chose to. In the end, the votes are a community effort. The real problem is most people won't be bothered to, or even care.
That's just going to be human nature at work and a you problem not a me problem issue. You also have the fact a third party won't fully understand what is going on and won't be easy to make an educated decision on which side to support. They will have to further invest time, which many won't want to.