Let me start with what you provided me with, last. I'll maybe come back to what else you said, later.
As I perceive this debate, both discussants are "believers". Carrol believes that it has "not yet" been found out how the universe could have come into being, and he cites different models that could potentially be used to eventually reach a conclusion. In doing so, he argues, as all scientists argue, that it is "not yet" known, but may eventually be known. But since science, according to its own discipline, only ever knows exactly what it knows at present, and since the position is accepted that present knowledge will be superseded by future knowledge - i.e. is and should be falsifiable - science cannot dispose of final knowledge.
He could have to ask himself "What if, through scientific work, it were discovered what caused the universe to come into being?" Let us further assume that there is an answer to this question that is universally accepted by the scientific community. But what about the rest of the people? The scientific community would have to make the riddle of the existence of the universe generally comprehensible in order to convince everyone else that it had been solved. Since scientists make up a comparatively small proportion of the human community, they could say "It is so because we have found it out and you must now simply trust us". Well, what then would be the consequence of this discovery? Would people then have a doctrine for their practical life that is different from the given one (such as the Ten Commandments)? Would they have a spiritual awakening? Would humanity all at once have certainty in moral and consequent practical matters of daily life?
This scenario, however, is not very likely. For within the scientific community, different theories prevail that contradict each other. They also overlap and complement each other in parts, but there is no general consensus and certainly no proven hypothesis in this area. If a certain scientific group wanted to work towards establishing an all-unifying theory, they would still be faced with the task of having to explain this to all of humanity. Not only does this seem unlikely, it would not be desirable. Because, in the face of people's differences, such differences would only cease to exist if they were forced to believe in something they do not find believable.
I think the debates among cosmology scientists are generally no different from those among the erstwhile theologians and philosophers from which science itself emerged.
Rather, it seems that the scientific field has scattered into so many fragments that, due to this fact, one can rather speak of a regress, since the countless disciplines are hardly capable of a common - mutually understandable - language any more.
But the problem with statements of "we don't know" or "we don't know yet" is, in my view, very underestimated. It is rejected by people to stand on an "I don't know" or "I don't know yet" position. The way I see it, man WANTS to believe something.
But since people know very well from each other that everyone is fallible, mistakable, deceivable, seducible, etc., you need something that they put outside of humanity. And not only outside humanity but also outside what we call the universe or the observable realm of it.
Just imagine you'd be born into a society which has as a believe-system the "We don't know yet"-doctrine. What are you supposed to do with it? The question then logically must be "But what DO we know so far?"
That already distracts from who is "we" and since "we" know from each other that we are fallible, we need something other, less familiar, even no familiar at all, which is God.
John Hick: a necessary being is an eternal, uncaused, indestructible, and incorruptible being.
A Christian might say: "I have faith in God. But if I wouldn't have had faith, I might find it necessarry to invent God."
Now, this indeed fantastic entity needs to be thought of in order to see that a human being wants something "out of and beyond himself" to even grasp towards what and which he can orientate himself, since orientation in questions of conscience cannot be provided by Joe or Patricia or Mr. Burns or Mrs. Miller during sole contemplation. The act of praying differs from the act of washing the dishes or explaining the function of a machine or the solar system.
I don't start a prayer by saying: "Dear all unifying theory, I need guidance to overcome my insecurities. Please help me to find an answer how I shall decide upon my current conflict." Also, you, being alone with you cannot consult yourself without finding it ridiculous, otherwise you'd start a prayer with "Dear honored me, ..." You need to personify it, yet take it not too literally.
The official nature of "praying" differs from mere self talk. It is more structured, it has a beginning, a middle and an end. While if you just try to think through your problems, you'll be distracted by all kinds of interfering thoughts and you may come out of your thinking time more confused than entering into it. But the act of consulting through prayer first, the afterthoughts might get supported by this very prayer.
When Carrol says that it is about observable space and that the universe itself may not obey the physical laws discovered by humans and is therefore something completely different from what we can recognise, he is not as far away from the concept of God as he might think. It seems that he refuses to personify it (in the way I tried to express). If I understood him correctly, which I cannot say for sure because he (both) often spoke too quickly for me and I am German. Even though my English is quite good.
Now, given that it is stated that "we solved the mystery of life" (and the universe) I would assume that people would lose interest in it. If a mystery is solved, like I said, it loses all its fascination and if one becomes really upset of having been taken away the riddle and be given it a final answer, he might want to lynch you. In the same way, if one is bored he finds that it is a very undesirable state of being. Man indeed does not appreciate to be given all answers, he wants to have an appropriate space to interpret himself, which, obviously, the Christian and other religions provide people with (biblical, religious stories and scripts).
So, even within Christianity, the pastor would not and should not say in church: "We know for sure that God exists and we've got proof." He instead says: "We have faith in Him and we trust Him." The certainty lays in the uncertainty. Not the certainty lays in certainty.
That is why I appreciate very much that you took the riddle into account.
First, I apologize for being so late with this response: I've been quite busy.
I'm not sure if I understand what you're getting at entirely, but I'll give it a go anyway ;-)
If you remember the second X-Files film, you'll know that the subtitle was: "I want to believe." That film (and the series) dealt with the supernatural, aliens, and UFOs. I'll open up by asking you a question: how, in light of all that you say here, which seems to me to boil down to that we want to believe rather than know for certain, is it not equally valid, useful, or gratifying to believe in a superior alien species, maybe even one that made a supercomputer in which they programmed a virtual world, the world we're living in now?
I don't want to believe, but rather can't help but believe. This is how our minds work. I don't want to believe, for example, that a water molecule consists of two hydrogen atoms and one oxygen atom. A lot of thought, experiments, and proofs have been involved for us, as a species, to come to the conclusion that water is indeed H2O. When someone's suspected of having committed a crime, we assume innocence until proven guilty, and when enough evidence is gathered one way or the other, we can't help but convict for innocence or guilt. When my child's face is covered with chocolate and the chocolate jar is also empty, I can't help but believe he emptied the jar, even though he keeps insisting that he didn't.
My confusion in your response arises from the contradiction I sense here: on the one hand, you say that man wants to remain in a state of doubt but, on the other hand, he wants answers. The thing is that it doesn't matter, for in my opinion science serves both these desires best. Life in this universe is an enigma that we want to unravel, and we do, we have done and we'll keep at it. The beauty is that with each little piece we unravel, with every answer, new questions arise; just look at the quest for fundamental particles or the quest for the Holy Grail of science, the "Theory of Everything"; we're still far, far removed from getting there, but the scientific quest has given us a much greater understanding of the world, the universe and our own position within.
We don't make this progress because of theology or faith but despite them. As for your assertion that the first observatories were erected by religious people and institutions, I agree. But that's an easy point to score. It's also said that the first people to want to abolish slavery were religious people, and again I'd have to agree. It's that in those times most, if not all people were religious; there weren't any other people to choose from, so they won that competition by default. On the other hand, the first people to have slaves were religious as well, so there's really no point in debating who came first in this particular discussion, as religion comes first by default. In other words: religion was indeed our first attempt to make sense of it all, our first and outdated attempt because everything we've discovered so far, everything we've explained, doesn't need a God.
I'm sure you're familiar with the expression "God of the Gaps"; that's the sense I get from you, that you're hanging on to that God. The God that fills the gaps in our understanding. When early man saw lightning in the sky, he imagined a superior being casting these devastating bolts from the sky. When he associated the sun with the growth of the fruits, plants, and, consequently, the animals he needed to survive, he began worshipping our local star. And since we're programmed by evolution to see our own kind everywhere (two dots and a line are enough for us to see a face, and because we're a social species we've specialized in recognizing the many different faces in our ever-increasing tribes), we anthropomorphize the lot of them. We no longer need a God to explain lightning or the sun; we've reached a point where only life and the universe are left for a God to fill in the gaps, and that's where apologists like Craig specialize with arguments like the Kalam, the fine-tuning of the (fundamental forces and constant values found in the) universe for intelligent life, the current lack of a satisfactory theory of abiogenesis (a natural process by which life has arisen from non-living matter), and so on.
But I don't mind not knowing; for me, that's where the amazement of life comes from. The "knowledge" we as humans need is much more of an agreement than actual "knowledge". For us to live well together, we need a set of agreements so we can depend on each other, and much of these agreements are given to us naturally, through evolution. Or do you believe that man did not know it was wrong to kill before the Ten Commandments, the Sharia, or modern law? The "Golden Rule", treating others the way you want to be treated by others, is necessitated by our nature as social creatures and is baked into our biology through mirror neurons and a host of neurochemical processes. We can't help but treat others the way we want to be treated by others to an extent, due to our social nature. We only disregard this Golden Rule when someone steps outside of the set of previously mentioned agreements that make us live cooperatively and well together.
I'm going on a tangent again... To wrap it up I'll loop it back to the beginning: I don't want to believe but belief comes naturally to me. All our belief systems come from our ability to understand cause and effect on a conceptual level; it's what separates us from other mammals. And our ability to understand cause and effect enables us to extrapolate the presence into the future we make for ourselves. When we see the wind blowing through, and shaking the trees, and then fruit falls down from the tree, we infer that we can shake the tree ourselves to make the fruit fall down. We could not play the same game with lightning from the sky, but since we base so much of our existence and beliefs about the world on cause and effect, we could not help but imagine a cause for what we saw. That's all God is, in my opinion. On the other hand, I (and no one else) can't prove that there isn't a God. Like Craig likes to say: "absence of evidence isn't evidence of absence". But that's also a rather cheap and easy point to make, for we can't prove a negative; I can't prove a pink teapot isn't orbiting Mars. So, for me, there are only beliefs, but there's a difference between reasonable and unreasonable beliefs, and I can't help but stick to reasonable beliefs.
I hope this response finds you in good health, my friend.