Your health is more important. I think that person who complained was just trying to find a reason to leave. You might have been inactive due to health, but I don't think it could have affected him that much. Get well soon.