An open-source model is usually easy to get running and considerably harder to serve well.
A deployment that answers in a few hundred milliseconds during testing can slow badly once a dozen people send long prompts at the same time, and the utilisation figure on the dashboard will often look healthy throughout.
That gap opens because single-request speed and system throughput are different measurements under different constraints, and closing it is a matter of serving configuration more than model choice.
Throughput and latency pull against each other
Throughput counts how much work the system finishes in a period, usually requests or generated tokens per second. Latency counts how long one user waits, both for the first token and for the full response.
Raising one tends to cost the other, because admitting more requests at once lifts total throughput while lengthening the queue each request sits in before work begins.
Which way to lean comes from the application, since an interactive assistant lives or dies on time to first token and an overnight batch job does not.
Why static batching leaves capacity on the floor
Batching exists because feeding a GPU one request at a time leaves arithmetic units idle between operations.
Static batching struggles with language models because prompt and response lengths vary so much.
Hugging Face’s documentation is blunt about the cost: with static batching “you’d have to pad them all to the same length,” which means a short request that finished early stays locked in the batch while the model keeps generating a long answer for someone else, and the padding itself is compute the GPU spends on nothing.
Continuous batching removes that waiting. The same documentation describes it as maximising “GPU utilization by dynamically rescheduling the batch at every generation step,” with completed requests leaving and new ones joining immediately instead of waiting for the whole batch to finish.
NVIDIA’s TensorRT product family documentation calls the identical technique in-flight batching, which is worth knowing when comparing engines.
Memory is usually the binding constraint
During generation a model keeps a key-value cache of the tokens it has already processed, which saves it recomputing earlier work for every new token.
That cache grows with conversation length and with the number of people being served, and it is where most deployments hit their ceiling.
The scale of the waste is larger than most teams expect. The vLLM project, which introduced paged attention, measured existing serving systems wasting between 60% and 80% of memory to “fragmentation and over-reservation.”
Paging the cache into fixed blocks, instead of reserving a contiguous span for every request, recovers most of that, and vLLM reports up to 3.5 times the throughput of Hugging Face’s Text Generation Inference as a result.
Two configuration details follow from this and are worth setting deliberately:
- Block size determines how many tokens each cache block holds. Hugging Face defaults to 256, which matches its decode kernel, and the value interacts with how much memory gets stranded in partly filled blocks.
- Cache memory budget is set explicitly. Hugging Face exposes it as a fraction of free GPU memory, defaulting to 0.8, which makes the trade between cache headroom and concurrent request count a decision instead of an accident.
Two further techniques matter where traffic patterns allow. Prefix caching reuses cache blocks when requests share an opening segment, so a long system prompt is computed once instead of once per request. CPU offloading copies evicted blocks to a pinned host buffer when the GPU cache fills, which trades bandwidth for the ability to keep more sessions alive.
Quantisation attacks the same ceiling from the other side, storing weights and sometimes the cache at lower precision. NVIDIA’s documentation covers FP8, INT8 and INT4 for this.
Lower precision changes model output, so the only safe way to adopt it is measuring quality on your own evaluation set first.
Where throughput leaks outside the serving engine
A well-tuned serving engine still depends on everything feeding it, and throughput leaks in places that have nothing to do with batch size.
Model load time governs how fast a new replica becomes useful. The minutes spent pulling tens of gigabytes of weights off storage are minutes the new GPU spends earning nothing, which is why the Neurotechnology Cloud platform pairs GPU nodes with local NVMe and a parallel filesystem.
Sharded models add communication to every token. A model split across several GPUs exchanges activations at each layer, so GPUDirect and RDMA earn their place by keeping that traffic off the CPU.
The optimisation curves post works through how that overhead bends the returns from adding hardware.
Orchestration decides whether capacity arrives before the queue does. Kubernetes handles replica counts, and the setting worth attention is how early the autoscaler triggers, since a replica that starts loading once the queue is already deep arrives too late to help.
Neurotechnology Cloud offers both shared and dedicated private AI infrastructure. Shared suits teams wanting flexible access, while dedicated capacity makes throughput easier to predict for inference with steady demand or firm data-governance requirements.
Finding the configuration
No batch size transfers between models, and none transfers between workloads on the same model. The method that works is a sweep: establish a single-request baseline, raise concurrency in steps, and record where throughput stops improving and where your latency percentile breaches its target.
The requests in that sweep have to resemble production. The shape of the prompts and the pattern of their arrival both change the answer, and a test built from short uniform prompts will recommend a configuration that collapses on real traffic.
Measure the percentile named in your service commitment, since a mean latency figure hides exactly the behaviour that generates complaints.
The instrument to distrust here is the utilisation dashboard from the opening.
A concurrency-aware costing study notes that the nvidia-smi figure reports the wall-clock fraction of time a kernel is resident, and that it “can read high” even at one request per second because “a single resident decode kernel keeps the GPU nominally busy while it is arrival-limited at batch size 1.
“A reading of 90% confirms a kernel is running, not that the batch is full, so throughput at a known concurrency is the figure worth acting on.
These levers interact, so tuning one in isolation produces a configuration that looks strong on a single measure and fails on another.
The post on generative AI infrastructure sets out how batching, quantisation, autoscaling and monitoring fit together across a serving stack.
That usually resolves into two configurations instead of one. Interactive traffic runs smaller batches with headroom held back for spikes, while offline processing runs larger batches and longer queues to lower the cost of each completed job.
The wider economics of that capacity are covered on the GPU compute and cloud pillar.
Neurotechnology Cloud runs inference workloads on dedicated EU-resident capacity through its AI Cloud and managed GPU services, with batch and memory configuration measured against the traffic a deployment will see.
If you are tuning an open-source model against fixed latency targets, get in touch.
