I built an LLM gateway on Pingora and measured it against nginx and OpenResty doing the same work. The CPU was fine. The memory was holding every prompt until the last token, and one of the fixes ended up as a patch for Pingora.
This week I built a gateway for LLM inference: one door in front of a set of OpenAI-compatible backends (vLLM, in my case), running as a Nomad job. It checks API keys, applies per-key limits, queues requests when every backend slot is busy, routes by model, keeps a user on the same replica so the prefix cache stays warm, retries before the first byte and writes one usage line per request.
Before trusting it with real traffic I wanted two numbers: what it costs per request and what it costs per open stream. And I wanted them next to the obvious alternative, because the first question anyone asks about a custom proxy is “why not just nginx?”.
The short answer: per request it’s as cheap as the alternatives, and with big prompts it’s seven times cheaper. Per open stream it was holding 3.5 GB of prompts nobody needed for 10,000 streams, and getting that down to 1.67 GB took three fixes, one of them in Pingora itself.
Why not just nginx
LLM traffic doesn’t look like web traffic:
- Requests live for seconds or minutes. A streamed completion keeps its connection open while the model generates, token by token.
- Bodies are big. A prompt with a document in it is 100 KB before anyone notices.
- Routing needs the body. The
modelfield is inside the JSON, so the proxy has to read it before choosing a backend. - The logic is specific. Queueing by free slots, affinity for the prefix cache and rotating provider credentials don’t exist in a stock reverse proxy.
nginx can forward the traffic, but all of that logic ends up in Lua, which means OpenResty. I also tried Bifrost. Pingora, the Rust framework Cloudflare built to replace nginx at their edge, was the only option that let me write that logic as normal code without fighting the tool. So the real comparison is not “Pingora vs nginx”. It’s “Pingora vs OpenResty doing the same work”, with bare nginx as the floor.
How I measured
Everything ran on one machine (an i7-12700K), with the proxy pinned to a single core and the load generator (oha) and fake backends on other cores. With every core available, the differences blur. These are numbers for comparing proxies, not a capacity plan.
Two fake backends: one answers instantly with a JSON body (everything you measure is the proxy), and one streams like vLLM, a ~330-byte SSE event per token at a fixed pace, with the usage at the end. CPU is the proxy process’s user + system time from /proc/<pid>/stat, divided by requests or chunks. Memory is its RSS.
Three proxies, one worker each, on the same core:
| What it does per request | |
|---|---|
| My gateway (Pingora) | SHA-256 key check, per-key limits, reads the body for model, affinity and prompt size, injects stream_options, strips client headers, retries before the first byte, records TTFT and usage, writes a 32-field usage line. |
| OpenResty | The same work in Lua: resty.sha256, shared-dict counters, cjson for the body, a body filter for TTFT and usage, the same log line. |
| Bare nginx | Just forwards. Exact key match, doesn’t read the body, short log. The floor, not an alternative. |
Per request: a tie, until prompts get big
| CPU per request | req/s (1 core) | p99 | |
|---|---|---|---|
| Small body | |||
| My gateway | 18.4 µs | 48,129 | 1.93 ms |
| OpenResty (same work) | 19.5 µs | 46,689 | 1.53 ms |
| Bare nginx | 5.7 µs | 136,597 | 0.52 ms |
| 100 KB body, streaming | |||
| My gateway | 64.2 µs | 14,457 | 7.28 ms |
| OpenResty (same work) | 457.1 µs | 2,196 | 60.68 ms |
| Bare nginx | 24.8 µs | 35,085 | 3.26 ms |
With small bodies, the gateway and OpenResty cost the same, and bare nginx is three times cheaper because it does none of the work. With 100 KB prompts, OpenResty costs seven times more than the gateway: cjson builds the whole document and serialises it again just to add stream_options. The gateway reads the body without building it and injects the field into the bytes.
On a single request with nothing else going on, the median goes from 7 µs straight to the backend to 24 µs through the gateway: 17 µs added, against 21 µs for OpenResty and 12 µs for bare nginx. Next to a model that takes hundreds of milliseconds to produce its first token, that’s noise.
It didn’t start there. The first version cost 27.9 µs per small request and 158 µs per 100 KB one. The profiler, not intuition, found where it went:
- The usage line was built with
json!and cloned twice: 30 % of a small request. - The affinity key was a SHA-256 of the first message: 55 % of a 100 KB request. It isn’t a security use, so it’s now XXH3.
- Two parses into
serde_json::Value, plus re-serialising the prompt just to measure it. - The SHA-256 of the API key, the thing everyone suspects first, was 0.7 %.
What’s left is mostly Pingora itself and one serde pass over the text (~23 µs per 100 KB), which is needed to know where each message ends.
Open streams: where 3.5 GB went
This is the scenario that matters for LLMs. 2,000 streams open at once, 100 tokens each at 50 tokens/s, 100 KB prompts: 1,000 new requests and 100,000 chunks per second. A stream should take 2 seconds; anything beyond that is the proxy.
| CPU (1 core) | CPU per chunk | Stream p50 (ideal 2 s) | |
|---|---|---|---|
| My gateway | 72 % | 7.4 µs | 2.046 s |
| OpenResty (same work) | 92 % | 11.1 µs | 2.418 s |
| Bare nginx | 35 % | 3.5 µs | 1.995 s |
One core holds it with room to spare, while OpenResty is already stretching every stream by 22 %. Pushing a single gateway thread further:
Around 4,000 concurrent streams per thread is the comfortable ceiling; at 10,000 the thread saturates and every stream takes 45 % longer. Pingora runs with a single worker thread here, so the next step is simply more threads.
CPU was fine. Memory was not: 369 KiB per open stream, or 3.5 GB for 10,000 streams. None of it was a leak. It was memory held for longer than it was useful, in three places.
1. The retry buffer outlived the retry
Pingora keeps the request body in case it has to replay the request on another backend, and a request can only be retried until the client has received its first byte. In a streamed response the headers go out almost immediately, so from then on the prompt is dead weight. It stayed in memory until the last token.
Now it’s released as soon as the response is flowing to the client, and not a moment earlier. Releasing it when the upstream accepts the request left a window in which a failure retried with an empty body. There’s a test that fails if anyone moves it back. Without streaming, the body is kept as before, which is correct: until the headers arrive, a 500 can still ask for a retry.
2. glibc kept what it freed
After freeing a large block, glibc raises its mmap threshold. From then on, prompts land in its arenas, get freed at different times between small live allocations, and the arena can’t shrink: with 400 KB prompts, 1,350 MiB of RSS for 665 MiB actually alive. Setting MALLOC_MMAP_THRESHOLD_=65536 in the job sends big buffers to mmap, which goes back to the kernel when freed.
The two fixes together flatten the curve: per-stream memory stops growing with the prompt. The threshold costs about 60 µs on a large request, which is invisible on a stream that lasts seconds (72 % CPU against 68 % without it).
3. 64 KiB for a 200-byte reply
What was left belonged to Pingora. The request body reader keeps its buffer (up to 64 KiB) until the request ends, and every upstream response is read with a 64 KiB buffer, even one that carries 200 bytes. A 19-line change releases the reader buffer once the body is complete and sizes it to Content-Length.
That’s −25 % per stream from the patch alone (219 → 165 KiB), and small requests got cheaper too (21.1 → 18.7 µs), because that 64 KiB response buffer sat exactly on the mmap threshold and was being mapped on every request. It passes Pingora’s own test suite plus three new tests that fail without it, and I checked the two things it could break: a pipelined request that arrives in the same read survives, and an upstream that sends more than its Content-Length is still detected, so its connection still isn’t reused. I’ve opened cloudflare/pingora#1020 to propose it upstream.
What didn’t help
Measured and discarded, so nobody has to try them again:
- mimalloc: 10 % less CPU on small requests, 14 MiB more RSS at idle. Not worth it here.
- A 68 KiB
mmapthreshold instead of 64: cheaper instant requests, but 226 KiB per stream instead of 165. - Avoiding the copy that injects
stream_options: with the field already in the body, the numbers were the same. - Pingora’s retry buffer copy: with the body read up front it activates later and copies nothing.
Bugs only LLM traffic finds
The benchmarks came at the end of the week. Before that came two days of pointing the gateway at real vLLM backends, and most of what broke is specific to LLMs.
Bodies around 64 KiB. If the proxy reads the body to route, Pingora 0.9 can only forward it from its 64 KiB retry buffer. A 95 KB prompt reached the backend truncated, with the full Content-Length, and hung until the timeout: a 504 after 60 seconds. The fix is the early body buffer in cloudflare/pingora#816, and there’s an integration test with 30 KiB, 1 MiB and 10 MiB bodies.
A silence timeout that re-ran generations. A stall timeout only makes sense when streaming. Without streaming, vLLM is silent until it finishes. A 60-second silence limit turned a 2,500-token answer (96 seconds) into a 504, marked a healthy replica as down and ran the whole generation again. Now: silence only counts in streaming, the total limit is 10 minutes, and a timeout neither marks a backend down nor retries.
An open port isn’t health. vLLM listens on its port while it’s still loading the model. The check has to be GET /health with a 2xx.
The usage that disappeared. Without streaming, the gateway keeps the last 8 KiB of the response to find the usage block. A response over 8 KiB, about 2,000 words, arrived cut at the front, wasn’t valid JSON, and its token count was lost: exactly the number you bill by. Now it reads the value after the last "usage".
Counters that survived their routes. In-flight and queued counters have to be shared with the route that replaces them on a hot reload. Copying the value instead left them stuck: 329 requests “in flight” with the whole cluster stopped.
More than one gateway. With several replicas, affinity uses consistent hashing so every gateway picks the same backend, and load is read from the backend itself (vllm:num_requests_running plus waiting) instead of each gateway’s own counter. With 16 requests sent straight to vLLM, a gateway with none of its own waits in its queue instead of piling on.
What 1,000 req/s actually needs
With 100 KB prompts and streamed answers:
- CPU: about 4,000 streams of 4 seconds fit in one thread. With 10-second answers (10,000 open streams) you need two cores.
- Memory grows with open streams, not with requests per second. Open streams are req/s × duration: ~0.9 GB for 4-second answers, ~2.3 GB for 10-second ones.
- Logs: 560 bytes per request, averaged over 2,993 real lines, is 0.56 MB/s, or 48 GB a day. Budget for it.
For perspective: the inference box behind this gateway does about 2 requests per second under load. The gateway won’t be the bottleneck until there are hundreds of backends behind it. The point of the week wasn’t to make it fast. It was to know exactly what it costs before trusting it, and the one lesson that held every time was the same: profile before you optimise. The obvious suspect was 0.7 %.
Next: that inference box. Getting vLLM to run on an AMD Strix Halo iGPU, and taking its startup from 9 minutes to 97 seconds.