.. it could be that the inactive accounts never bothered to vote for witnesses in the first place?
I would guess so, either through lack of knowledge or for a reason.
I too think it's a little early to look into removing 'inactive' account witness votes, perhaps in a few years time...