ChatGPT is a blurry jpeg of the web
A really engaging and thoughtful explanation of the way large language models (LLMs) work, using surprisingly apt metaphors.
This analogy makes even more sense when we remember that a common technique used by lossy compression algorithms is interpolation—that is, estimating what’s missing by looking at what’s on either side of the gap. When an image program is displaying a photo and has to reconstruct a pixel that was lost during the compression process, it looks at the nearby pixels and calculates the average. This is what ChatGPT does when it’s prompted to describe, say, losing a sock in the dryer using the style of the Declaration of Independence: it is taking two points in “lexical space” and generating the text that would occupy the location between them. (“When in the Course of human events, it becomes necessary for one to separate his garments from their mates, in order to maintain the cleanliness and order thereof. . . .”) ChatGPT is so good at this form of interpolation that people find it entertaining: they’ve discovered a “blur” tool for paragraphs instead of photos, and are having a blast playing with it.
Nontechnical and a little bit long, but an extremely useful explanation for understanding how LLMs perform the magic they seem to accomplish. Could prompt some great discussions, especially among 11th-12th graders in a unit discussing AI technology.
(by Ted Chiang, author of several science fiction short story collections, including “Stories of Your Life and Others,” which included the story that the film “Arrival” was based on.)