Yes, I know what you mean. My point is slightly different. Perhaps it is okay or even better that voters who disagree simply express their individual views rather than decide to engage in a 'collusive scheme' to amplify their vote power by both agreeing to not vote.
No one does that on other sites such as reddit; some people upvote and others downvote, resulting in a net score. While vote power isn't limited there (I think; afaik reddit's current scoring is opaque), attention is.
In fact, having both opposing votes present contains useful information; reddit has a "controversial" ranking which shows content getting both up- and down-votes. Given a healthier downvote economy here, such a thing might be equally useful. I know I'd certainly be interested to see where the disagreements lie.
Individually, yes it would probably be better to agree to not vote and save vote power, but I don't see how this helps the system as a whole.