I have bought votes from you, even sold votes for some time and also planning on delegating some SP when I accumulate enough to afford that. You are offering a great service to the community. If anybody used your service for good causes they will use it again and again. So, a memo from a few noob account won't hurt your reputation. But I can understand your feelings!
This types of people will come more and more as steem platform grows. You can maintain a database of blacklisted people along with the reason(s) for blacklisting. Other services can use that database.