I found this recent paper (pdf warning) to be very good at covering most of what's been talked about here lately. This article covering the paper is also good if you're looking for a summary or just want a website rather than a pdf.
Basically, because language is so tied up with human thought, we tend to think good at language = good at thought, and bad at thought = bad at language. However, these are fallacies, and they trip a lot of us up when looking at llms, which are good at language but bad at thought. To help address this, they distinguish between two kinds of linguistic competence: formal linguistic competence (the knowledge of rules and statistical regularities of language) and functional linguistic competence (the ability to use language in the real world, which often draws on non-linguistic capacities). Llms do really well with the former, but very poorly with the latter. The reason is something we've already talked about here: llms are only imitating the language processing section of the brain. It does it well, but cannot replicate the function of the sections of the brain required for functional linguistic competence, no matter how large or powerful they get. That's not a knock, just a statement of fact. The language processing part of the brain alone also can't do the things required for functional linguistic competence either. So those looking to llms as a path to AGI should look at llms as merely one module in an artifical brain, and replicating the other sections will be necessary to add the ability to perform the other functions required before they can begin to approach functional linguistic competence.
Very well put, although I'd argue that they imitate only reasonable output, rather than the actual language processing going on in the brain. We still understand little about that, not even whether the general rules of language are innate or not.
Do LLMs appear to have impressive capabilities with language because they are a huge leap forward or is it because language is far less complicated than we thought it was? I don’t know enough there to give my own opinion.
I work with SOTA LLMs every day and, while I am leveraging them to build better products, a lot of what I do is work around their shortcomings.
I’m often talking to uninformed/inexperienced people who use simplistic examples as evidence that LLMs are going to solve all their problems or easily enable incredible new things but it’s rarely hard to break their illusions. Not to discourage them, but to help them formulate real solutions that LLMs are part of instead of thinking the LLM is a miracle solution by itself. Product Managers love the idea that they can just think up new features and not have to work with anyone to build them. It’s when you push them to prove their idea solves the whole problem they claim instead of a toy version, at a reasonable price and SLA, that things start to fall apart fast. That’s fine. Then we start talking about reality and LLMs have a lot to offer.
LLMs are the culmination of nearly 25 years of work and advances in various AI algorithms and the resulting models. (Or even longer - the first chat engine was built in 1967, ELIZA.) Language parsing is not simpler than we thought it was, we've simply made some progress along the path of creating AI models that can use language in semi-useful ways.
The part that shows how far we have to go is the realization that they operate entirely without real world context. They babble words that are in some way related to each other, and which our ability to construct mental structures from mere wisps of data can turn into occasionally useful insights or tools, but the fact is that they are shockingly limited in how they interact with the real world. They fall over at the most inappropriate times, it seems.
The advances that will allow them as models to deal with contexts for their babbling - real, psychological, historical, whatever knowledge corpus you can name - will be exponentially more complex. But their functionality as narrowly scoped tools which can be tuned to all sorts of useful tasks? That will grow at least linearly, which means that's the sort of utility we can expect from them over the next five years or so.
That's my prediction.
Do LLMs appear to have impressive capabilities with language because they are a huge leap forward or is it because language is far less complicated than we thought it was? I don’t know enough there to give my own opinion.
I'd say they only have impressive capabilities with certain aspects of language, and that's mostly due to the sheer amount of the data they've been trained on. Rather than them having done a good job of learning the rules, it's more that we've done an extensive job of telling them what the rules are.
Rough tangent: I am enjoying both this conversation and the larger one about LLMs, because it reminds me a lot of the conversations around self-driving cars.
The bar for these tools seems to be substantially higher than for humans that do these tasks. Some folks are up in arms over accidents involving self-driving cars (as if there aren’t substantially more - in all senses of the word ‘more’ - directly because of people); some folks are vigorously opposed to the idea of LLMs “thinking”* because they are just mimicking communication, or because they lie or omit or are incorrect (as if all of those things can’t be attributed to people and in larger quantities than these machines).
*Note that I mean thinking colloquially, rather than defined in a way that requires biology. (Note also that I fundamentally have a problem with that kind of definition if it doesn’t come with the addendum “…in the precise and specific way that we understand biological organisms think”. But I digress. 
)
I can't remember anyone in this thread objecting to saying ai's can think because they lie, omit information, or are incorrect. That only really comes up when people who've failed to check the AI's output try to weasel put of the blame by saying it's the ai's fault and not theirs. People have used those reasons as examples of why they're not as great as they're advertised to be, but not as a reason they can't be said to think.
Fair - I wasn’t referring to anybody in particular, only that the concerns about both technologies, and specifically the complaints about their efficacy and/or usefulness, seem to be similar.
Sort of. The complaint with self-driving cars is mostly that they need to be better than us at the task they'll be replacing us for. We want self driving cars to be better at driving than we ourselves are at driving. Not "we" as in humans in general, but individually. So I dont just want a self-driving car to be better than the average driver, i want it to be better than me, personally before I'd want it to drive for me. They pretty much already are, but the kinds of accidents they do get into tend to be ones that even bad drivers would be able to easily avoid. If they were making the same kind of mistakes human drivers do (just at a lower rate) it'd be much easier to accept that they're still better and safer than most human drivers. It also doesn't help that the biggest name in self-driving has so badly tainted the term by calling his driver assistance system "autopilot" despite it not actually being an full self driving system.
With llms, the complaint is more that we don't want them to replace us for these tasks because we either enjoy doing them, or our livelihoods depend on us doing them.
Fascinating Aug 15, 2023 interview with Timnit Gebru, AI heavyweight and co-author of the prescient 2021 “Stochastic Parrots” paper. The interview is refreshingly hype free. Long, but worth a read/listen.
Timnit Gebru is not just a pioneering critic of dangerous AI datasets who calls bullsh*t on bad science pushed by the likes of OpenAI, or a tireless champion of racial, gender, and climate justice in computing. She’s also someone who wants to build something different.
Pages