Self-voting doesn't tell you anything certain about impartiality.
Therein lies the ultimate reason of our disagreement, it seems.
You are thinking about impartiality as a binary quantity which is either true or false. From that perspective you are right. Self-voting doesn't imply any impartiality value.
On the contrary, I am thinking about impartiality as a probabilistic quantity that can take any continuum value in the [0; 1] interval. And I'm telling you that self-ranking affects the mean of this quantity.
People get really emotional when their reputation is at stake. There is tons of scientific research backing this. That emotion enters as noise in our system through the self-ranking door.