Imagine that after every infection, all the T cells stay in your system. That's just too much after a while and clutters up everything.
Chicken pox actually stay in your system as they're related to herpes. They can flare up again decades later, usually when you're already 50 or 60. Really not cool.
A new infection can usually be stopped by the T cells that remember.