I think a FALSE FLAG should get a reputation ding. Yet there is no way to do that currently, and this system is intentionally designed so ideally there is really no such thing as a moderator.
As to cost...
It counts towards their voting percentage for the day, and they get no curation reward, but that is the only cost.
If they are powerful though the cost to the person they flagged can be quite devastating.
The nuclear deterent also makes sense when the actors are similar in strength. It is not so much a deterent when one can swat the other like a fly. :)
Let's try another analogy. :-)
You are in the Wild West. In the street someone pulls a gun on you and fires. Do you shoot back, or do you attempt to negotiate a ceasefire? There are probably a few answers to that and some, none or all of them might be good in any given situation. How are we to judge? We would need to bring in a judge, jury and executioner. That has overheads. People would need to pay taxes to cover those overheads. Do we want that?
It would be simpler to just say that escalating to a flag at any time has a reputation cost. I'd prefer simpler. It's transparent and has a low overhead. We lose some of the nuances of a full trial I guess.