I think there is a risk/danger in any group that grows more than a few and a few hundred where we can either deeply or casually know the members of that group. Beyond those two scales, I think we solely the tool of its leaders.
Also, the bigger it gets, the more power it thinks it has and tries to acquire more. I'm not sure how we could build a society that would build groups of limited scope and purpose without trying to expand. That's been an area of contemplation for me.
Even the government types who say "limited government" still want "just one more bon bon."