"If you read super carefully, you aren't actually dissagreing."
Yes, I am disagreeing. You say, "only 3 people out of 6 are surely [trying to measure the system impartially]"
I'm saying, no such thing as impartial measurement. All 6 could be genuine. All 6 could be biased. Self-voting doesn't tell you anything certain about impartiality.
Anyway, I suspect much of your concerns will be alleviated when the UI makes it clear that there is no voting, there is only consensus.
Therein lies the ultimate reason of our disagreement, it seems.
You are thinking about impartiality as a binary quantity which is either true or false. From that perspective you are right. Self-voting doesn't imply any impartiality value.
On the contrary, I am thinking about impartiality as a probabilistic quantity that can take any continuum value in the [0; 1] interval. And I'm telling you that self-ranking affects the mean of this quantity.
People get really emotional when their reputation is at stake. There is tons of scientific research backing this. That emotion enters as noise in our system through the self-ranking door.