We're quite lucky in that my wife's mother lives around the corner and my daughter is very comfortable with her, so we've probably been able to escape 3-4 times a year. But even then, it's hard to go out knowing that you've left your daughter in the care of someone else. Even when it's the grandmother, you get that feeling that she's in someone else's hands.
But the time away, even for just an hour or two is golden for your mental state, and for your relationship.
Nobody tells you how hard some things like that are before you have kids