Maybe it's because I usually wait for any transaction I am doing to finish before starting the next one.
Yep. I thought I had - but it seems there was some lag on the chain or something. Polygon doesn't seem that reliable to me. Just go even slower - because it is annoying :)