What is vllm?
vLLM is an open-source library for large language model inference and serving. Discussion around it has centered on its role as a high-scale serving engine and on the engineering limits of running newer model architectures through it.
Release history
- Jul 2026 - Aman Sanger noted that loading DeepSeek before the first inference can mean a wait of about 30 minutes.
- Aug 2026 - Simon Mo said the open-source inference engine now runs on half a million GPUs at any moment.
- Aug 2026 - Mo said that in many cases teams use vLLM as a benchmark to confirm a model runs well on it.
- Sep 2026 - Stefano Ermon said diffusion-based LLMs cannot run on vLLM or similar stacks, requiring a custom-built serving engine.
In the discourse
Attributed discussion of vllm.
Diffusion-based LLMs require an entirely new serving stack. They cannot run on vLLM or SGLang, forcing builders to develop proprietary inference engines from scratch.
“You cannot run this diffusion based LLMs on VLM or SG lang like you have we had to build our own serving engine.”Stefano Ermon · 18 Sep 2026
vLLM is running on half a million GPUs at any given moment, reflecting its scale as open-source AI infrastructure.
“The open- source inference engine now running on half a million GPUs at any moment.”Simon Mo · 6 Aug 2026
vLLM has become a de facto hardware validation benchmark, with chip vendors using it to confirm new silicon runs correctly before launch.
“A lot of cases they use VM as a benchmark to make sure it runs well on them.”Simon Mo · 6 Aug 2026
DeepSeek loaded via SGLang or vLLM incurs roughly a 30-minute cold-start wait before first inference, a practical deployment friction worth tracking.
“If it's just loading deepseeek before the first inference maybe you'll wait about 30 minutes.”Aman Sanger · 29 Jul 2026