Yeah, the bigger rep you have, the better off because it shows you are a member of the community that brings value.

If you are downvoted or blacklisted, your rep is trashed, not just the number, but you won't get seen in feeds.