Take as much time to rest and recover, as you said we have nothing if our health is slowly fading. Sometimes listening to and trying to conform to what people want or say can be detrimental. Either way whatever you do people will always talk and have an opinion; so do what is best for your health and body including mind.