Transformers win at scale, but ResNets and alternatives hold the edge everywhere else

ResNets outperform transformers in small-data regimes. That is not a provocation or a contrarian thesis. Eric Jang, a researcher with direct experience training Go-playing models, states it plainly: in small-data settings, ResNets give more performance per dollar at lower budgets, and he has not found a way to make transformers beat them even for the specific game-tree structure of Go. The finding is less about transformer weakness in the abstract and more about where the architecture’s strengths stop applying.

Ramin Hasani, who works on alternative neural network designs including liquid neural networks, supplies a complementary data point. Scale, he argues, is what allows transformers to pull ahead: at smaller scales, transformers cannot beat other formats. The implication is that the transformer’s apparent dominance is partly an artifact of the data and compute regimes in which it has been most thoroughly tested. The architectures it displaces do not disappear; they reassert themselves when those conditions are not met.

Ali Behrouz offers a structural account of why this happens in sequential settings. Attention carries no inherent awareness of the temporal dependencies among tokens, and positional encoding only partially addresses the gap. Behrouz is direct on this point: even with positional encoding, attention is not a great fit for tasks requiring sequential reasoning. That is an architectural critique, not a benchmark complaint. The problem is not that transformers underperform on sequence tasks because they lack training data; it is that the mechanism itself is not built for the dependency structure those tasks demand.

The physical simulation domain makes the constraint most concrete. Anima Anandkumar, a researcher in scientific machine learning, points out that a high-resolution global weather model might contain roughly 50,000 training samples: a dataset size that would be considered trivially small by language modeling standards. The scale problem compounds when the task moves from training data to inference context. At industrial simulation scales, where each spatial dimension may contain a thousand or more grid points, the required context length reaches hundreds of billions to a trillion tokens. Anandkumar’s assessment of what that means for transformers is unambiguous: all of the world’s compute would not be enough.

Hasani adds a countervailing observation about the alternatives. State space models and similar architectures compute efficiently, but their expressive power consistently falls short of transformers when the task involves long-context language dependencies. The picture that emerges is not a simple hierarchy with one architecture on top. It is a map of tradeoffs: transformers win on expressiveness at scale and struggle on data efficiency and sequential structure; alternatives win on efficiency and structure but lose on expressiveness as context grows.

Jang rounds out the picture with a point about current hardware. Once models grow large enough to saturate the capacity of modern graphics processing units, he argues, the choice between transformer and ResNet matters relatively little. The transformer’s practical edge is most pronounced in the large-scale, data-rich regimes for which the current generation of GPU infrastructure has been purpose-built. Neil Patil, commenting on the compute market more broadly, notes how thoroughly that infrastructure has been shaped around large language model workloads, which creates a feedback loop: the hardware favors transformers, which favors more transformer investment, which makes alternative architectures look weaker than a hardware-neutral comparison would show.

What the evidence describes is a conditional dominance. Transformers are not universally superior; they are superior in the conditions that happen to describe the largest and most visible AI deployments of the past several years. Outside those conditions, in the small-data laboratory, on the Go board, inside a physics simulator running at industrial resolution, the architecture’s advantages thin or disappear entirely. The researchers arriving at this boundary are not critics of transformers as a general matter. They are practitioners who encountered a limit the field’s prevailing benchmarks were not designed to reveal.