You Could Have Invented Transformers From Memory

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on tech for your team

• Business-only prices and quantity discounts
• Tax-exempt purchasing
• Multiple users, one account, clear invoices

Create a free Business account →Free to register

As an affiliate, we earn on qualifying purchases.Hide

Alex Wang’s essay explains neural network layers as associative memory stores, tracing a path from linear maps to MLPs and self-attention. Under the post’s assumptions, a 64-dimensional key space gives rough capacities of 64 memories for a linear store, about 2,700 for an MLP and more than a million for attention; these are analytical estimates, not measured limits for deployed models.

Alex Wang has published an essay showing how associative memory can provide an intuitive path from a simple linear layer to MLPs and self-attention—the building blocks at the center of transformer models. The post estimates that, with 64-dimensional keys, these memory designs can retrieve roughly 64, 2,700 and more than one million memories, respectively, under its stated framework.

Wang frames a memory as a key-value pair: a key is used to find a stored item, and its paired value is what the system should return. A query retrieves values associated with similar keys. The measure that matters in the essay is interference-to-signal ratio—how strongly other stored items disrupt retrieval of the intended one. Capacity is the number of memories that can be held before that interference reaches the strength of the target signal.

The simplest example is a linear memory store, formed by adding an outer product for each key-value pair. On recall, the output combines the target value with contributions from other memories. For random, normalized keys and values, Wang estimates that interference grows until it matches the signal at about M = dₖ, where M is the number of stored memories and dₖ is the key dimension. In this model, a 64-dimensional key space therefore has a rough capacity of 64 memories.

The essay then moves from the linear store to more capable designs, reporting estimates of about 2,700 memories for an MLP and more than one million for self-attention with 64-dimensional keys. Wang’s headline, “You could have invented transformers from memory,” presents this as a conceptual derivation: start with storage and recall, then ask how a system can reduce interference. The figures are estimates developed in the post, not a benchmark of a particular trained model.

Why Attention Stores More

The account gives readers a way to connect transformer architecture to a basic problem: retrieving the right information when many items have been stored. It shifts attention from the vague idea that a model “knows” facts to a specific mechanism—keys, values, similarity and interference—and explains why recall quality can matter as much as the amount written into a system.

Its capacity comparisons also clarify why the architecture matters. In Wang’s simplified analysis, a plain linear map allows stored memories to interfere substantially, while MLPs and attention improve retrieval. That helps explain the motivation behind more elaborate layers without implying that a deployed language model has a fixed, easily counted inventory of facts. The estimates describe a theoretical setup; real models are trained systems with additional structure and behavior.

From Linear Maps to Attention

Wang begins with a familiar observation: language models can answer questions about a wide range of subjects, which can make their stored knowledge seem limitless. The post uses a personal example—asking an assistant named Astra to recall the rules of the board game Dune: Imperium—but its main focus is not a new product or experiment. It is an explanatory framework for interpreting neural network components as memory stores.

The post also connects memory writing to training. For a linear layer, Wang notes that a gradient-descent update changes its weight matrix by an outer product involving the layer input and an output gradient. In this view, an update resembles adding a key-value memory. The essay cites work by Irie, Csordás and Schmidhuber on interpreting a gradient-trained linear layer as attention over training inputs. The supplied material does not provide a publication date or further bibliographic details.

“The key property that characterizes a “good” memory store is its ability to recall what it stored.”

— Alex Wang, in the essay

Limits of the Capacity Estimates

The supplied excerpt does not show the full derivation for the MLP and attention figures, so their assumptions and calculation details cannot be independently assessed from the material provided here. The post’s 64-dimensional-key estimates should not be read as measured storage limits for current language models, or as a count of facts they can reliably recall in practice.

It is also unclear from the supplied source material when the essay was published. Wang’s opening example describes asking Astra for game rules, but the excerpt does not specify how the assistant generated that answer or whether it used any external information. The core argument is theoretical and explanatory; the available material does not report a new experiment testing the estimates against trained transformers.

The Next Step in the Argument

Wang’s post sets up the next question: how can interference be reduced without simply increasing the key dimension? The essay says it will develop the progression from a linear layer to an MLP and then to self-attention, presenting each as a memory store with improved capacity and recall. The supplied source excerpt ends as that discussion begins, so the details of the later derivations are not available here.

For readers evaluating the numerical comparisons, the relevant next step is the full derivation and its assumptions—not treating the headline figures as universal model specifications. The supplied material does not identify a follow-up experiment, release or other scheduled milestone.

Key Questions

What is the main development in Wang’s essay?

Wang offers an associative-memory framework for understanding linear layers, MLPs and self-attention, focusing on how stored key-value pairs are retrieved and how other memories interfere.

Are the capacity numbers measured results?

No such measurement is reported in the supplied material. The figures—about 64, 2,700 and more than one million memories for 64-dimensional keys—are estimates in the essay’s analytical framework.

What does interference mean here?

When a query seeks one stored value, other memories with similar keys can contribute to the output. Interference is the combined effect of those contributions relative to the target signal.

Does the essay claim that transformers were historically invented this way?

No. The headline is a conceptual framing: Wang argues that a reader could build an intuitive route toward transformer components by reasoning about memory and recall. The supplied material does not make a historical claim about how transformers were developed.

Source: hn