A couple of different ways:
- reputation - your tag votes would hold more weight the higher your reputation. Also, all votes after your vote on a tad would increase your reputation
- -1 votes - of you think a tag does not relate to a post you can -1 it. If your reputation is greater than those before you then the tag goes into - territory
So is not th necessarily the number of votes but the number of reputation-weighted votes.
There could be restrictions on voting on tags and adding tags below a certain reputation.