This is true for many scientific domains as well. In my opinion, the problems with the teaching of math lies in the way this is presented.
I do agree with this. I have heard cases (small anecdotal sample size) where students claim that they have a hard time following the math class because the teacher "sucks" or "can't teach". It is not easy to get excited about a class where the presented material is not organized, clear or even correct.
Good teachers are indeed rare. For many reasons, people are not very attracted by the job, and only the most motivated ones go through. We have not enough of them :(