I'm pretty sure that decline still gets hit in reputation if flagged.

I started watching my rep more carefully once it was coming down, and it seems that with decline on there is no change in rep - mine has been on 69.74 for the past two years, but previously it could be taken down rapidly by whales with higher reps doing 100% down votes.

Comments could be different but the whales have never really bothered downvoting my comments (except for a few where I give them a bit of a poke)

Thanks for your upvotes on my comments, they may help my rep, but I'm not sure, it hasn't changed yet.