I always try not to be invasive when taking pictures of strangers
Same do I. Moreover, invasion usually destroys the moment that was the trigger for photographing. At the same time...
I encountered this negativity only in its soft form in Eastern Europe. Some people feel offended because you "break" their privacy, some people feel they are a target for an action with a sexual context, etc. I believe that's a cult of being offended, people have been taught this sick emotional pattern and found this pleasant. They also know that being bullies towards photographers is safe.
I don't accept this negativity against street photography as a norm. I know that many people who don't like street photography work as cops (which is pitiful in countries where cops are all corrupt), commercial doctors, tax collectors, sellers, taxi drivers, bureaucrats, etc. The list of annoying jobs are endless, and someone with a camera is the least annoying thing you can come across. When I take images I keep it in mind. Street photography isn't harmful in any way, and photographers legally have the right (except a short list of countries).
Sometimes, a camera can make a person feel watched or stuff like that, it's indeed an intrusion. But what about thousands of people sneezing openly, isn't that (viral) intrusion? What about traffic jams and road accidents for the reason people simply want to enjoy their own cars instead of traveling by public transport? What about crowds in public transport in the rush hours? It's because public space means intrusion. If someone doesn't want it, they have their homes to stay.
For situations when Eastern Europeans want to talk to me about how they are upset, I have answers. And this is an advantage not to be a traveler from a faraway country. "Stop interfering with my work", "If this is a problem for you, call the police, or go away now", etc.
Once in Kolkata, India, a local man came to me and asked if I had a permission to shoot (that was on a street). I answered him (without even a slight smile): "I have a permission. God gave it to me". The man got confused and walked away without answering. :D
The important thing is that you keep taking those photographic records and despite the inconveniences that may occur, there is always something positive in all of this.