I’d say the other major difference from brains is that LLMs don’t have a long-term memory/state, and this means that trying to keep it coherent over long tasks is impossible.
I’d argue that this difference, along with no long-term memory pretty much compactly explains why attempts to replace jobs/use LLMs for stuff often fails, and arguably why LLMs can’t be substitutes for humans at jobs, which is how I define AGI:
Anyway, let’s move onto more concrete differences between current LLMs and the human cortex. One such difference is that humans continue to learn over the courses of their lifetimes. I’ve heard this termed as neuroplasticity, which current LLMs lack insofar as their weights are frozen during deployment. You can still in some sense “teach” a deployed LLM new things by feeding information it wasn’t previously aware of into its context window, but this is transient and perhaps somewhat different in terms of what information the model is capable of gleaning, relative to if the model was trained on this information. By contrast, the (synaptic) “weights” in the “hidden layers” (the cortex) of the human brain are able to be edited in deployment; something similar for LLMs would probably help them deal better with problem domains that weren’t present in their training data, thereby giving the LLM + learning system more generalizable capabilities than the LLM alone.
It’s not totally clear to me that long-term memory works by a different mechanism than predictive learning does. Base models have totally memorized certain texts word-for-word, like the Bible or literary classics.
The other thought that comes to mind is like… if that isn’t a reliable enough mechanism, I’m not sure why LLMs couldn’t just write up summaries of their own context windows at various points, and then put those in some database it can search and pull from another time?
I’d say the other major difference from brains is that LLMs don’t have a long-term memory/state, and this means that trying to keep it coherent over long tasks is impossible.
I’d argue that this difference, along with no long-term memory pretty much compactly explains why attempts to replace jobs/use LLMs for stuff often fails, and arguably why LLMs can’t be substitutes for humans at jobs, which is how I define AGI:
It’s not totally clear to me that long-term memory works by a different mechanism than predictive learning does. Base models have totally memorized certain texts word-for-word, like the Bible or literary classics.
The other thought that comes to mind is like… if that isn’t a reliable enough mechanism, I’m not sure why LLMs couldn’t just write up summaries of their own context windows at various points, and then put those in some database it can search and pull from another time?