When an LLM generates a response, it stores intermediate computations called the KV cache in GPU memory at every step. Traditional serving frameworks pre-allocated memory for each request based on the maximum possible response length, meaning a one-sentence query reserved as much GPU memory as a thousand-word one. The result: over 60% of GPU memory sat idle, limiting concurrency and forcing teams to scale by buying more hardware.
PagedAttention: Borrowing a Decades-Old Idea
vLLM was introduced in 2023 by researchers at UC Berkeley, and its central innovation is called PagedAttention. The name is a direct reference to virtual memory paging, a technique operating systems have used for decades to manage RAM efficiently.
In a traditional OS, physical memory is divided into small, fixed-size pages. Programs do not need to hold all their data in one contiguous block. The OS hands out pages as needed and tracks where everything lives. Programs see a clean, continuous address space; the OS handles the messy reality underneath.
PagedAttention applies this same logic to the KV cache. Instead of reserving one large, contiguous block of GPU memory per request, vLLM breaks the cache into small, fixed-size blocks called pages. These pages are allocated dynamically as the model generates each token. When a request finishes, those pages are immediately freed and made available to another request.
The effect on memory waste is dramatic. Traditional systems waste upward of 60% of their KV cache memory through fragmentation and over-reservation. vLLM brings that figure below 4%.
What This Unlocks in Practice
Memory efficiency is a means to an end. What developers actually care about is throughput (how many requests a system can handle per second) and latency (how fast each individual response arrives.)
Because vLLM wastes so little memory, it can fit far more concurrent requests onto the same GPU. Combined with a technique called continuous batching, where new requests slot into the processing queue the moment a slot opens rather than waiting for an entire batch to finish, vLLM delivers throughput improvements of 2–24 times over naive serving approaches, depending on workload and hardware.
How It Fits Into the Stack
vLLM itself is an open-source Python library. Point it at a supported model (Llama, Mistral, Gemma, Falcon, and many more) and it handles batching, cache management, and serving via an OpenAI-compatible API endpoint, making migration from OpenAI’s API straightforward.