If you are here for the "spirit, creativity, and freedom" of Hive, then downvotes affect that not at all. It mainly affects your rewards so I assume that's what you meant instead.