You guys might like this talk.
I'm typically not a fan of the French. BUT, this guy makes some really good points, and also demonstrates how LLMs can't solve basic IQ-like tests.
summary: LLMs are all the rage right now, everyone is focusing on LLMs, but they aren't the key to intelligence.
LLMs are basically memorization tools. If the LLM doesn't have a template in memory, it cannot spot a pattern, so it cannot give you an answer for something new or novel.
He suggests a "hybrid" system where
LLMs = System 1
"Discrete Program Search" = System 2
The System1/System2 concept being from Thinking, Fast and Slow from Kahneman where System 1 is stuff that's stored in our memory and System 2 being the stuff we actually have to think about.
Still not sure what "discrete program search" is or how it works, will have to look into that later.
Going to play around with this later today: https://arcprize.org/