This is a big problem with the internet in general. Otherwise normal people sometimes turn into monsters behind the veil of anonymity. One solution would be to have everyone use their real names. This would deter a lot of people but could be misused without some means of confirming identity.
It could also discourage people from talking about certain subjects which they for various reasons do not feel comfortable doing without being anonymous e.g. things which they are ashamed or which they don't want family to know about.
Hard as it is the one thing that really "hurts" trolls is ignoring them. I know there is already a MUTE function - perhaps an IGNORE function like Bitcointalk may prove useful in this sense, particularly if it is apparent to the troll that they are being ignored. As the ignores pile up they might be forced to change their ways.