It's complicated.
Following / being followed people doesn't help much with content discovery. Sure, if you have few people that you follow, there's a good chance that you won't miss content that you like (assuming that you followed them for this very reason, not because you like them, but because you like content they post). The more people you follow the more chaos in your feed.
Don't bother with random unexplained downvotes, statistically those are far less common than random unexplained upvotes ;-) Without significant amount of SP they are not affecting your payout and as long as you have non-zero payout they don't affect your reputation either (not to mention that they would have to have more reputation than you).
@gtg thanks for the brief explanation of the random downvotes, and the positive outlook on my random upvotes instead! Was just confused over who they are and if I am the only one they target. I couldn't agree with you more on more following more chaos! I follow about 250 people on Steemit and I do so because I either like the content they produce or have actually built a friendship with them!😀