The serving layer between a model and the applications that call it is where most of that value is realised, or quietly lost.
A trained model is a function: weights plus a forward pass. Turning it into something an application can call reliably under load is a separate engineering problem with its own failure modes.
It needs to able to decide whether latency budgets are met, whether a version update ships cleanly or introduces silent regressions and whether the model fits an existing workflow or forces every consumer to bend around it.
If the model itself is still in question, our post on when PEFT fine-tuning makes sense covers that decision, and the wider engineering context sits in our guide to AI product development.
Three layers people keep conflating
Most comparison writing on serving is muddled because it collapses three distinct layers into one argument.
The inference engine runs the forward pass, loading weights onto the accelerator and managing whatever intermediate state execution requires. vLLM, SGLang, TensorRT-LLM and Hugging Face’s TGI compete here for language models, with ONNX Runtime and TensorRT covering fixed-shape workloads.
The model server wraps an engine in an HTTP or gRPC surface with health checks, metrics, request handling and version management. NVIDIA’s Dynamo-Triton, formerly Triton Inference Server, sits here with multi-framework support across TensorRT, PyTorch, ONNX and OpenVINO. vLLM ships its own server too, which is why these two layers are frequently collapsed in practice.
The orchestration layer arranges servers without serving anything itself, handling autoscaling, traffic routing, health-based replacement and safe version rollouts. KServe is the Kubernetes-native example, with Seldon Core and Ray Serve as alternatives. NVIDIA Dynamo coordinates work across GPU pools above vLLM, TensorRT-LLM or SGLang rather than replacing them.
Comparing vLLM to KServe is a category error, since one runs models and the other arranges the things that do. Deciding what you need at each layer separately is what stops teams overbuying at one level while leaving a hole at another.
What the engine layer actually buys you
For language models the constraint is memory rather than compute. Every token generated produces attention key and value tensors that stay resident in GPU memory, and older frameworks pre-allocated a contiguous block per request sized to the maximum possible output length.
vLLM’s launch analysis found existing systems wasted 60 to 80% of that memory to fragmentation and over-reservation. PagedAttention maps each sequence’s cache through a block table to non-contiguous physical blocks, borrowing the operating system’s virtual memory model and bringing waste under 4%. More concurrent sequences fit on the same card.
Continuous batching solves the scheduling half. Static batching locks the GPU to a fixed batch until the slowest sequence finishes, leaving completed slots idle, while iteration-level scheduling evicts finished sequences and admits waiting ones after every forward pass.
The published figures reward careful reading, because the multiple quoted depends entirely on the baseline. On LLaMA-7B and LLaMA-13B, vLLM reported 14 to 24x the throughput of raw Hugging Face Transformers, and 2.2 to 2.5x that of TGI, which already used continuous batching.
Where requests asked for three parallel completions, the TGI margin rose to between 3.3 and 3.5x. The headline number in circulation is the largest of these, measured against the weakest comparator, and anyone sizing GPU capacity from it will be disappointed.
The newer architectural shift is disaggregated serving. Prefill, which processes the prompt, is compute-bound. Decode, which generates tokens, is memory-bound. Running both on the same GPUs means one phase constrains the other.
NVIDIA’s Dynamo documentation describes splitting them across worker pools that scale independently, with the KV cache transferred between them over RDMA. For large models spanning multiple nodes, this turns the capacity question from how many GPUs into how many of each kind.
Not every model is a language model
Much of the advice above assumes autoregressive generation, and a great deal of published serving guidance quietly does the same. Vision, speech and classification models break those assumptions: there is no KV cache, output shape is fixed rather than variable, and continuous batching solves a problem these workloads do not have. Classic dynamic batching in a model server is the lever instead, with TensorRT or ONNX Runtime at the engine layer.
Two practical differences follow. Preprocessing frequently costs more than inference, since decoding and normalising an input can consume more wall-clock time than the forward pass, which makes where preprocessing runs a throughput decision rather than a tidiness one. And these systems are usually pipelines rather than single models, so a model server with ensemble support can execute the chain server-side and remove several network round trips compared with orchestrating it from the client.
The interface contract
The most consequential decision above the engine is the shape of the serving API itself.
REST over JSON is universal, needs no client setup and works natively in every browser, which is why it stays the sensible default for anything a partner or external team consumes. gRPC runs over HTTP/2 with a Protobuf contract, giving smaller payloads and cheaper serialisation, multiplexed over long-lived connections.
The honest comparison is more conditional than most write-ups admit. Benchmark work running both across distributed cloud instances found REST competitive on small payloads, with gRPC pulling ahead by roughly 15 to 40% on throughput as client load grew and by around an order of magnitude at large payload sizes. F
igures vary enormously by test setup, so the shape of the curve is more useful than any single multiple: gRPC’s benefit scales with payload size and concurrency, and thins out below them.
For inference, that favours gRPC on internal high-throughput paths and wherever payloads are large enough that JSON encoding becomes a real cost, streamed token-by-token output included.
Most teams settle on gRPC between services they control and a REST facade at the edge for everyone else, with schemas kept deliberately model-agnostic so the runtime underneath can change without rewriting business logic.
What actually breaks in production
Batching more requests lifts utilisation and requests per second, but every request waits for the batch to form. The practical rule is to autoscale on queue depth or tokens per second rather than CPU, because a GPU-bound service can sit at modest CPU utilisation while its queue grows, and KEDA makes queue-depth scaling straightforward on Kubernetes.
Whether a throughput gain is worth its latency cost depends on the product’s SLA, which is why a serving API should expose p95 and p99 rather than a comforting average.
The failures that sink a serving layer from there are rarely modelling problems:
- Cold starts – Scale-to-zero saves money on bursty traffic, but loading a container and its weights takes time, and for a large model those weights run to tens of gigabytes. A cold start can stretch into minutes, which is fine for a batch job and unusable for an interactive one. Pre-pulling weights or holding a warm replica is the usual answer, and latency-tolerant workloads are often better served by an asynchronous job endpoint.
- Unsafe version routing – A promotion without a rollback path turns a bad model into an outage. Canary rollout and fast rollback are baseline.
- Unvalidated input – Inputs the model was never tested against arrive on day one, so the API needs strict schema enforcement at the edge and defined error responses for malformed requests and backend timeouts.
- Missing instrumentation – Without tail latency, error rate, queue depth and accelerator utilisation as first-class signals, you have a serving layer you cannot actually operate.
One further decision causes more incidents than it should: what the endpoint promises under partial failure. A request that silently retries against a half-loaded replica, or returns a stale cached result without saying so, is worse than an honest error.
Well-behaved endpoints are explicit about timeouts and make retries idempotent, so a client backing off does not double-charge the accelerator.
Choosing a stack
None of these components is universally correct, and the decision is mostly about fit.
Teams standardising on Kubernetes get the orchestration layer from KServe, with pluggable engine backends behind one interface.
Where raw throughput on committed GPU capacity is the binding constraint, vLLM or a TensorRT-LLM backend does the heavy lifting, with Dynamo above them once a deployment spans multiple nodes. For mixed estates running several model types side by side, a multi-framework model server matters more than peak throughput on any one of them.
The right stack is the one that matches how the organisation already deploys and rolls back everything else because a serving layer demanding its own separate operational discipline rarely keeps it for long.
Check out our post on the topic of debating whether to build-in-house-or-partner.
Getting from a trained checkpoint to a dependable service is the engineering that turns a model into a product, and it is where most of the effort in an AI build actually sits.
That work is the focus of Neurotechnology Cloud’s applied AI solutions: building the serving and integration layers that let a model hold up under real traffic, on GPU infrastructure sized for the inference load it generates.
