The magnet is not attached to the refrigerator. Thus it is resisting gravity every moment. And that is work. Of course, modern science gets around this buy saying there is no movement. But, that is just in one frame of reference.

And then, out comes magnet motors. Things that spin by the power of magnetic force of permanent magnets.