Yes, I have had a similar experience. Early on, flying with my first Mavic Pro, I was down in a valley and flew up high to catch some of the mountain rocks and such. The wind picked up, to about 40 mph, and was blowing the drone away, even though I was flying full speed back (I thought). I was finally able to reduce altitude, and make some headway back in-between gusts. It was spooky as only my 3rd or 4th flight at the time, lol.
This is actually one of the reasons I really like the RCPro style controller. It has a switch right on the front to quickly switch between Cine mode, Normal mode, and Sport mode.
Most of the DJI drones have a lot more power and speed in Sport mode (careful, because sensors are disabled) that really helps them fly in the higher wind. I often use that switch to fly in Sport mode in the wind, or just to get a faster RTH if I'm done filming.
It's so scary when you can not control your drone... I should've put my drone into sports mode while returning the drone. I totally forgot about sports mode!
I will have to look into RCPro controller. Thank you for the advise my friend 😊