Fast, Cheap LLM Inference on a Single GPU

There's a myth that running your own AI means racks of GPUs and a cloud bill to match. For a huge range of real workloads, a single well-tuned GPU is plenty. The trick is inference optimisation — getting the most tokens per second, and the most value per pound, out of the hardware you have.

The levers that matter

• Quantization — running the model at lower precision (4-bit is often the sweet spot) dramatically cuts VRAM and speeds things up, with little quality loss on most tasks.
• KV-cache tuning — reusing computed attention keys/values so long conversations stay fast.
• Flash-attention — a faster, more memory-efficient attention implementation.
• Batching — serving multiple requests together to lift total throughput.
• Right-sizing — a smaller model that passes your evaluation beats a bigger one that's slow and expensive.

Measure everything: tokens per second, time-to-first-token, VRAM headroom, and cost per request. You can't tune what you don't measure.

Time-to-first-token is the one users feel

Raw throughput matters for cost, but what a user experiences is how quickly the response starts. Streaming the first tokens fast — even while the rest generates — is the difference between "instant" and "laggy". It's often worth optimising for latency over peak throughput in interactive products.

When one GPU isn't enough

Eventually, high concurrency or very large models justify scaling out — but you should hit that ceiling with a tuned single-GPU setup first, so you know what you're actually paying to scale. Plenty of businesses never need to.

This is the day-to-day of LLM serving and inference. If your model is slow or your inference bill is climbing, email b.a@live.co.uk and I'll help you tune it.