There is one big difference between people who don't understand math and those who seem to have intuitive grasp on it. People in fist category usually think that math is about numbers, while people in the second one will confidently disagree. Math is about nature in it's most fundamental form. Numbers is just one particular example.
Image from Pixabay
Math is often described as purest form of science, and science for some mysterious reasons is often viewed as opposite of nature. Nothing can be further from truth. We haven't invented gravity, electromagnetism or thermodynamics. It's something that exists whether we care about it or not. It's the part of reality, and the fact that we can explain it's influence on the world around us does not change what it is.
Numbers are the easiest example: 2 + 1 = 3 not because scientists say so, but because it is what it is. Because whenever you have 2 things of something and you add another something to it - you get three somethings. It works the same way for apples, atoms and galaxies, because that's what math is about: the most basic rules of reality, those which don't depend on a context.
That's actually the main difference I see between physics and math. Physics does differentiate between objects. Atom of hydrogen is fundamentally different from atom of oxygen and is treated as such. Even two hydrogen atoms might be different, depending on their states and conditions. Light travels differently through air and through water. Numbers aren't like that. You will not see sentences like this in any math book:
it is recommended to calculate derivative of the functions in the second week of January outside of deep gravity wells to get the most precise results
Even physicists that work in most abstract theoretical fields need to establish "rules of engagements" before getting to work: describe the conditions and rules for the predicted phenomenon. Mathematicians are lucky, they can get directly to business with "there exists x such that... ".
Seems like mathematicians have unfair advantage then. But that's not exactly true. The same core difference, dependence from context, allows physicists to test their theories in the real world. Physics benefited immensely from all the practical experiments throughout the history, from simple ones like double slid experiment to ginormous projects like Large Hadron Collider. In math, the only tool available is human brains. Yes, you can calculate some things on a computer, but computer will only do what it was programmed to do, it can not verify consistency of the theory, for example.
Physicists dream of finding famous "Theory of Everything" - unified rule or set of rules that would explain everything we know about the universe. I would argue that if such thing were to exist it would be as much a mathematical theorem as it would be physical theory. Since such theory by definition works in all possible physical environments, it does not need a context. A rule applicable in any context can be only classified as math. So theory of everything not only unifies multitude of physical laws and principles, it also blurs the line between physics and math.
But physics being context-aware is not a means to an end, we might want to dig deeper and think why physics is like that. I think this stems from the fact that we are discovering physics from top to bottom: reverse-engineer nature's complex behaviour into its constituent parts. With mathematics we go in opposite direction: building complex structures from simple blocks already known to us.
Can it be that they will eventually meet in the middle? What if some advanced branch of mathematics will suddenly explain phenomena we observe in quantum world?..
