Considering yesterday was my first ever post, I can't comment on followers, but I have on a couple of occasions cleared those I am following, mainly due to them having stopped creating content.
I can understand people getting a little hurt if those you are following don't at least take a look at the content you create once or twice, but I fully understand if they choose not to follow you. Afterall they simply may not be interested in the topics you cover and it's nothing personal. I certainly would not unfollow them based on this.
People new to Steemit tend to fill their followers up with all kinds before they realize what they are doing. I know I did, and then did a huge purge. Dormant accounts as you say is another reason.