I came to a similar conclusion about the folks I originally followed when it came to inactivity or changes in content. I've been through my followers list a number of time since I've started, trying to get a good mix of posts I like and folks who will engage not only on their posts, but on mine. I probably still need to expand the numbers, but I like my list a lot more than I did to begin with, and it's because of the trimming. You're right—there's only so much we can realistically follow, and we don't do anyone any service to just follow without interacting. If only we could pick our followers, too! Not sure how that would work, but there's so many inactive accounts among the followers it's not funny.
That would actually be useful too, since a lot of spam accounts follow everyone. But since they don't interact or vote on anything so I guess it doesn't matter too much in the grand scheme of things.