I do not make such an assumption, but have what I consider evidence in the form of statements from others and his own history. His father was deeply entrenched in propaganda operations in VOA, and more. The video linked above, and the Esquire article linked below, discuss his family history of involvement, his own overt record, and his own statements that exemplify his work as an operative of Operation Mockingbird, or whatever they're calling it now. My assessment of this, and more evidence I do not have easily at hand, is that the preponderance of conclusions I can draw from it all indicate a lifetime of service to the CIA. It isn't even what he says that convinces me as much as the things he says are preposterous, or doesn't address, but that he surely cannot avoid knowing, that convince me.
IMG source - Esquire.com - Tucker Carlson embedded in private security contractors in Iraq in 2004
Edit: also, while I know you can't prove a negative, what makes you think he isn't CIA that you assert it with such certainty? That assertion is no less an assumption than you say is made by those claiming he is, and they do have evidence he is. He'd be working for a pretty crappy CIA if there was proof of who their covert agents are, so that's a bar we cannot reasonably set.