It's complicated.
Following / being followed people doesn't help much with content discovery. Sure, if you have few people that you follow, there's a good chance that you won't miss content that you like (assuming that you followed them for this very reason, not because you like them, but because you like content they post). The more people you follow the more chaos in your feed.

Don't bother with random unexplained downvotes, statistically those are far less common than random unexplained upvotes ;-) Without significant amount of SP they are not affecting your payout and as long as you have non-zero payout they don't affect your reputation either (not to mention that they would have to have more reputation than you).