It's good you spoke up. Owning it. People can see you're not as bad as the failed humor might make it look.
It's not about you or even that joke though. It just makes me uncomfortable to think something so innocent as a four can now, somehow, get ugly.
I'm actually having trouble explaining the point. I think I could have done a better job.