That seems a good way to go, checking their back catalogue as such. I think also, people change and that could cause us to follow or unfollow as well. I've seen a few people I follow write some things that I've not responded well to, but I tend to persist to see if it was a once-off thing before I unfollow.
You'll have to tidy up your follow list and eliminate all those dead ones some rainy day when there's nothing else to do.