"don't try to fix something that ain't broke."
My dad has said this my whole life!
We went to Disney World again after seven years. We rode one of our favorite rides that gets you soaked, but when we got on, they no longer had the protective plastic bin in the middle to put your belongings that can't get wet. When we asked them what happened to it, they basically replied, "somebody ruined it for everyone else."
There is no reason to let a few people that are "abusing" self-votes to ruin it for the rest of us.
Also, inhibiting self-votes would take away from the freedom of this platform.