One of the most important things I've noticed about official interventions like these is that if you want to keep from getting completely dragged through the mud, you have to play their game to a certain extent.
In this case...document, document, document.
You may need to rethink the plan of unschooling and get the kids enrolled in an online curriculum even if you aren't ready to build one yourself. Again, the goal here is to keep your kids with you under your care and get the State out of your business. An online curriculum is going to be a million times better than the kids completely off somewhere else like a school. If you try to convince officialdom that an unstructured approach to schooling is what you intend to do, they almost always hear "lazy, unprepared, not taking this seriously." At the very least, being more structured will help a great deal in any potential legal situation.
I would be pretty mad about the whole "playgroup" thing. Furious, actually. But that's beside the point. For your sanity and the best outcome of your situation, you're going to need to swallow the ire, present neat copies of what your kids are doing, and show some serious organization. If someone's really interested in making an example out of you, this won't help. But nine times out of ten, the thing that usually brings them down even harder on you will be showing any attitude like annoyance and not presenting them the information their computer-like minds need to see to go somewhere else.
I'm sorry you're going through this! Being questioned on what you're doing to take the best care of your children is infuriating and wrong.