but you will also slightly reduce their reputation score.

Note: This is true as long as your reputation is higher than theirs. If their rep is higher than yours, it wont affect them. This is one of a number of flaws with our reputation system.