It is a special shrine so it's possible you could have been to it. My memory of it from last time was totally gone. I remember going into the shrine building for the ceremony, but I didn't remember the courtyards or anything else.
I've got a special connection to Kamakura so the fact that Minato no Yoritomo had been to this shrine for prayer was really cool.
I get what ya mean about having gone places, but not having a solid memory of them. I think that is normal for us because everything here is new and different and it's total sensory overload. Plus you're young and not so appreciative of everything.