I’ve been giving this some thought after your asking. What is harm? Merriam Webster limits it primarily to physical injury or mental damage, while Cambridge includes ideas and concepts that can be harmed. Collins further includes animals and things. My favourite is Macmillan:

to injure, damage, or have a bad effect on someone or something

But that still leaves us pretty much where we started. 😔

What is injury or damage? From a social contract point of view? I would say anything incapacitating or detrimental caused by others without your consent (excluding those being punished for having harmed). Perhaps that might sum up harm as well.