Either select the three vertical dots to get a menu...
...or swipe sideways to get the remove button. Either way it is a two step process. 😁
I could be wrong but I think the swipe method can be done from either (left) or (right) side.
Of course you can also clear out your entire watch history...
...by selecting these three vertical dots.