Also, trail followers have their own settings. They can choose whether to follow flags or not.
This is a key point in this exchange. ATSD's followers elected to follow his downvotes as well as his upvotes. Presumably because they trust his judgement.
Another key point is this: there is a built in protection against ATSD abusing his DV following. If he does so, he will lose that trust (and as a result, his following). FWIW, i am seriously skeptical that most of his (or anyones) followers would consider DVing a comment where someone drops the C-bomb on him (or the N-bomb on his mom) an abuse of the DV.
if the follower really didn't know that he was following downvotes, then it seems to be a caveat emptor issue. He could have easily opted out of downvoting.