Yes it is difficult to cut back...there is a lot of quality content on here and the inclination is to upvote everyone. That will only lead to decreasing the voting power.
There are so many advantages to powering up one's SP it is amazing. There is no way to convey this until someone really does it for themselves. Then they realize all the benefits to it.