ROME, July 8 — The Transformer architecture, introduced in a 2017 paper titled “Attention Is All You Need,” has quietly become the engine of modern artificial intelligence. The paper’s authors — Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia Polosukhin, all then at Google — proposed a mechanism that changed how machines process language.
The claim was bold: that a self-attention mechanism alone could outperform the dominant recurrent neural networks (RNNs) and long short-term memory (LSTM) networks that had defined sequence modeling for years. The mechanics here are worth unpacking.
Self-attention allows the model to weigh the importance of different words in a sequence. This parallel processing capability is the key departure from RNNs and LSTMs, which process tokens sequentially, one after the other. That sequential bottleneck made them slower due to sequential processing.
The Transformer’s encoder-decoder structure, initially designed for machine translation, proved the point on the WMT 2014 English-to-German translation task. The paper reported a BLEU score of 28.4, outperforming previous models.
But the catch is that this was not just a one-off benchmark win. The architecture’s design proved scalable in ways its predecessors were not. On closer reading of the trajectory, the numbers tell a story of exponential growth.
BERT-Large, released by Google in October 2018, had 340 million parameters. GPT, the first version from OpenAI in June 2018, was smaller. But by the time GPT-3 arrived, the parameter count had jumped to 175 billion.
PaLM, from Google in April 2022, reached 540 billion parameters. The training compute for GPT-3 alone required 3.14e23 FLOPs.
The scaling laws, formalized by Kaplan et al. at OpenAI in 2020, showed that performance improves predictably with more parameters, data, and compute. This is not speculation; it is a documented relationship that has driven the industry’s investment strategy.
Follow the money
The investment community has taken note. The scalability of Transformers means that companies like Google and OpenAI have a clear path to improved performance: throw more compute and data at larger models. This is a rare thing in technology — a predictable scaling curve.
The catch is that the compute costs are enormous. GPT-3’s 3.14e23 FLOPs represent a significant capital outlay.
But the payoff, in terms of model capability, has been sufficient to justify the expense. The architecture’s dominance is not accidental; it is the result of a design that rewards scale. The shift from RNNs and LSTMs to Transformers was not immediate, but it was decisive.
The 2017 paper laid out the architecture, and within a year, both Google and OpenAI had released major models built on it. BERT, released in October 2018, and GPT, released in June 2018, were the first wave.
Since then, the parameter counts have grown by orders of magnitude. The pattern is clear: each generation of models is larger than the last, and each generation shows measurable improvements in performance on standard benchmarks.
What to watch next
The question now is not whether Transformers will continue to scale, but what the limits of that scaling will be. The Kaplan et al. scaling laws suggest continued improvement, but they also imply that the compute and data requirements will grow accordingly. The industry is betting that the returns will hold.
The architecture’s design, with its parallel processing and self-attention mechanism, seems well-suited to exploit whatever hardware and data are available. For now, the Transformer remains the dominant architecture in deep learning, and the trajectory points to even larger models in the pipeline.
The mechanics are established; the question is how far the industry is willing to push them.
