When I flag, I always reply as to why. That's something that I fully encourage doing, otherwise, you're just being passive and not setting a good example of using the feature. It opens up a dialogue to ask "why?" And if the flagger is wrong, if they don't suck, can remove it. Or someone else can step in and counteract the flag with a proportionate upvote.
I think people are scared of chaos. Personally, I'm not. I think it's crucial for this platform to have a healthy future.