But flags (downvotes) are a part of ...
Of course they are. I never wrote anywhere I was against the option to flag.
But many flags are not used in that sense. For example for quite a while every single comment of @valued-customer got flagged automatically by a whale. That had nothing to do with discovering value or preventing spam (actually the whale added a spam comment under every flagged comment). And that's just one example among many.
If there was an 'anti-abusive-flag-committee' its members would recognize such kinds of flags very easily and could counter them.