I don't know the algorithm, but under the basics of UA as a concept having people following you shouldn't reduce your score no matter what. I have reason to suspect that may not be true here, but nothing I'd consider evidence, and the effect doesn't seem to be large.
In any case, as you point out, there's nothing you can do about your followers, so don't worry about it.
Interesting, thank you. I wasn't worried, only curious.