I can see why Terence Tao isn't worried about AI replacing mathematicians. Tao's comment about AI always needing to be nudged is well taken. The sense is that it's an assistant that you need to manage, so it cannot really replace you.
https://youtu.be/sTDSO74D8Q?si=T690ZLNdcOwvw_h
Tao makes the point that AI doesn't really "learn" the way humans do. That makes sense - when I'm working on a problem, I'm learning from my mistakes, internalizing them.
AI?
It follows a script and nothing else. At best, it may adapt for a while before getting into its old habits.
Not intelligence, but rather imitation.
The bigger picture, however, comes when it is suggested that AI will be working for mathematicians to get into more uncharted territories. That's exciting - it's almost like having a tool which accelerates the mundane tasks. You tell AI to try out an approach to a problem while you go work on the big picture.
That must be a huge advantage that I can see.
It is not replacement; it's amplification of human capability. That is the true positive: AI makes what we do easier and does not take away the essence of why we do it.
Tao views the future of working with AI as collaboration and not as a kind of competition. I feel that there is something rather profound about teasing out a problem with human intuition and with human creativity that machines really cannot do. AI can be good for heavy lifting, but it can never match the ingenuity of a mathematician stretching the bounds of what is possible.