Magnetism is able to interact with ferrous metals. Because such materials are present in trace amounts in many compounds, the record of Earth's magnetic field polarity can be found in fossil materials, such as trees. Trees are very useful in dating events because they grow and produce rings yearly, enabling dendrochronology to attempt to pinpoint when an event affecting tree growth occurred.

"An ancient, well-preserved tree that was alive the last time the Earth's magnetic poles flipped has helped scientists pin down more precise timing of that event, which occurred about 42,000 years ago."
https://www.npr.org/2021/02/18/969063568/ancient-trees-show-when-the-earths-magnetic-field-last-flipped-out

The magnetic pole is notoriously vagabond, and one reason the poles are believed to be in the process of flipping is because that wandering of the poles has increased lately. If merely flipping magnetic polarity caused major physical disruption of the Earth geologically, we would be seeing major geological movements now, that could be shown to be caused by the movement of the magnetic poles.

We see no such geological trauma, despite dramatic movements in the polar axes.

Magnets do not stick to wooden doors, but to the steel skin of the fridge. They stick to the fridge in both polarities. It doesn't matter to the fridge whether the N or S pole of the magnet sticks to them, and moving the magnet around on the fridge does not buckle and move the steel surface. It is not powerful enough to do that.

However, Earth's magnetic field does shield us from energetic particles constantly arriving from space. During the disruption of that magnetic field that necessarily occurs when the magnet flips polarity, much more radiation from space reaches Earth, and this is problematic, like walking around in an X-ray machine.

It's not going to rearrange the Andes, but it might damage chromosomes.

Edit: Tiahuanacu was bombarded with energetic particles, but they were much larger than individual atoms, and produced much larger forces, that were capable of flinging 100 ton and larger stones about like marbles. Comets or nukes are about the only things we know that could have done that.