Wow! Indeed they are nowhere near.
I have just tried to ask them my quantum field theory exam question, and ChatGPT (that's the one I tried) failed dramatically. The question was however not hard (compute the cross section of the process 𝜋+ 𝜋- → 𝜋- 𝜋+ in scalar quantum electrodynamics and give the answer in terms of Mandelstam variables). As for the cases above, they seem to have some ingredients right but fail the tricky details (and even some obvious ones). I guess it is however just a matter of time for them to get there.
The most interesting part to me was, how the models diverge.
That's indeed the most surprising part. They should at least have a unique answer for a given question to ensure reproducibility.
I thought in my case, the phrase 'line diagram' tricked the model, but it is not!
This is worse that search right now!
Result of a google search!
Accurate.