Sorry for the late reply. IMHO, it's not a big deal as your downvoters HP is low.
It doesn't affect your ability to engage with others unless you give it an exaggerated importance. Shit happens... move on and don't let it impact your journey on Hive.