the military mindset is trained to think in terms of a hierarchical structure to battle an enemy, rather than the way terrorists think, that being a more independent (i.e. decentralized), "cell" network
I mentioned this above but the world I came from in the military dealt specifically with what you mention here. As an example, I wrote this up and reposted it -- Cryptocurrencies and Unconventional Warfare
You are correct - the only way to beat an insurgency is to wage a legitimate counter-insurgency which requires similar tactics (read: what big Armies are not suited for). This stuff was my bread and butter.
The evidence of that is the atrocities of war US soldiers have taken part in through the centuries.
Unfortunately, its the nature of the beast when you put humans in that sort of a pressure cooker. Precisely why sane veterans hate fighting.
they are just a small part of a big machine that limits the information individual soldiers receive and thus deprives them of true free will to make good decisions
Take heart.. the military is not full of automatons. When bad things happen, people find out. Careers are ruined and people do go to jail. It's far from perfect, but what is in a highly dynamic environment?