I remember it myself. When I was a university student, I could not refuse the invitations of my friends, and after I started working, I could not refuse the invitations of my bosses and colleagues. At the workplace where I worked until a few years ago, I got sick because I had to listen to senior employees complaining endlessly.
I wonder what that time was, if it was of any use, and if I could have done something more meaningful for myself with that much time... I think.
I don't think anything is truly wasted because we can always learn from our mistakes. 100 mistakes will only make our motivation to fix things stronger.
The question I would ask is...what would have been the consequence if I hadn't done that, and could I have avoided those consequences while still protecting my own rhythm?
I think a lot of the things we are scared of might occur just because we are so scared of them. Like fear of creating an awkward situation is the best way to ensure it's an awkward situation!