Cleaning out the following list isn't essential, like you say, if they're not posting what does it matter?
Well, for me I like to keep it clean so that when I follow a person they know I did so for good reason meaning, I don't have a gazillion people in there, just the ones I value. I think that shows respect to those who are actually in there, makes it feel a little exclusive.
We all have our ways I guess, and yours works for you.
Kudos on the 'u' in favourites! ✅