This does open the door to spammers and harassers comments not being hidden as well though

As long as negative rep is still hidden then spammers can be dealt with by bringing their rep down below zero, and even harassers too unless they are getting upvotes some other way to maintain their rep, but even in the latter case the downvotes they are getting in response to their harassment is costing them money.

Rep below zero being hidden was the main motivation for the rep system in the first place. It was considered a waste of effort and vote power to have to downvote every single individual spam comment.