I have heard some people talking about being able to give your downvote stake to another account or just delegating in the meantime. Nothing stopping communities from creating a downvote service that goes out and cleans up abuse and protects against retaliation. I think this will become part of how many communities operate in the future.
The reason I say it has an element of protection is that a user flagging you would need to have a stake in all the tribes to completely remove all your rewards