Very sorry to hear about all this, there are some witness who are very effective in taking this up and they'll see to it that it's rectified, witness like @themarkymark, I hope I get the name correct can surely help.
Also, if the percentage they are downvoting you with is close to 0% then you don't really have to worry. I received the same from one callisteemmer sometimes ago.