Hand-written SIMD assembly beats compiler auto-vectorization by multiples, not margins

Jean-Baptiste Kempf puts the number plainly: compiler auto-vectorization does not come within a few percent of hand-written SIMD assembly. It falls behind by multiples. That claim runs against a common assumption in software development, where the compiler is treated as a competent optimizer and hand-written assembly as an eccentric edge case. The evidence suggests the assumption is wrong by a wide margin.

Kempf points to the dav1d project as an example of how far a team will go when performance genuinely demands it. The project invented its own calling convention, bypassing standard operating system conventions entirely to extract more speed. “I don’t know anyone, I’ve never heard any other project than dav1d doing that,” he says. Creating a custom calling convention is not a routine optimization decision. It signals that the performance gap is large enough to justify changes that would ordinarily be considered too costly to maintain.

Reiner Pope, whose analysis works from hardware gate counts, offers a structural account of why low-level optimization yields such disproportionate returns. With a register file of eight entries, data movement alone costs 24 times p gates, compared to 4 times p gates for the actual multiply-add computation. The arithmetic is not the expensive part. Moving data to where the arithmetic happens is. A compiler that does not account for this at a fine-grained level will produce code that spends the majority of its gate budget on overhead the programmer, working by hand, can reduce.

I don't know anyone, I've never heard any other project than dav1d doing that. Jean-Baptiste Kempf

That structural reality has shaped how serious performance-critical projects are built. Video codecs, cryptographic primitives, signal processing kernels: the domains where hand-written SIMD pays off are also the domains where the cost of being slow is highest and most measurable. The gap Kempf describes is not a theoretical ceiling. It is the difference between competitive and uncompetitive software in production.

Against that backdrop, Richard Socher describes a different approach arriving from the opposite direction. Socher notes that his team’s Auto Research system produced near-best CUDA kernel performance without deep CUDA kernel expertise on staff. In fact, as Socher puts it, “the system just did all of these things.” Socher also cites what he describes as an OpenAI announcement that a self-evolving model working on optimization kernels cut costs by 80 percent on systems called Luna and Terra. That second claim is secondhand, Socher reporting an announcement rather than a result his team measured, and it should be read as such rather than as a confirmed benchmark.

What the evidence establishes, taken together, is that the performance distance between naive compiled code and carefully optimized low-level code is far larger than most engineering teams build into their assumptions. Whether that distance gets closed by a specialist writing assembly or by an automated system finding the same gains, the gap itself is real and the architectural reasons for it do not disappear because the tool changes. Pope’s gate-count arithmetic does not care whether a human or a model wrote the memory access pattern. The hardware charges the same either way.