Most organisations meet this role by accident. They buy GPU capacity, hire people who can build models, and then find the cluster spends most of its life doing nothing.
Cast AI’s State of Kubernetes Optimization Report measured resource use across tens of thousands of production clusters on AWS, Azure and Google Cloud, before any optimisation had been applied. Average GPU utilisation came out at 5%, against 8% for CPU and 20% for memory. The best-performing cluster in the dataset, a 136-node H200 inference fleet, sustained 49%. The distance between those two figures is technique rather than hardware.
Our complete guide to AI-ready infrastructure sets out what the stack contains and introduces the discipline that operates it. This post goes further into what the person in that seat is accountable for, how the job differs from the ones sitting next to it, and how organisations acquire the capability.
What the role owns day to day
The technologies involved are well documented. What is less obvious from the outside is that almost every one of them presents the engineer with a trade-off, and the trade-offs are the actual job.
How compute gets carved up and queued: Default Kubernetes hands out a GPU as a whole integer resource, which is the mechanical reason utilisation collapses in clusters nobody has tuned. Partitioning a card into isolated instances fixes that for small inference workloads while capping the memory any single job can address, so a fine-tuning run that fits comfortably today may not fit after the next dataset lands. Sitting above that is the choice of scheduler stack. Slurm’s batch queueing model with fair-share policies suits long training runs and research teams that already know it; Kubernetes suits mixed inference and service workloads and gives a common substrate for everything else; many teams run both, with Ray layered on for distributed training. Reversing that decision once pipelines are written against it is expensive.
Keeping the environment layer reproducible: CUDA drivers, kernel modules, container images, device plugins and GPU operators. The difficulty is not the initial build but the upgrade path: rolling a driver across a fleet without killing multi-day training jobs means draining nodes in sequence and running two driver versions in production until the long jobs finish. Where nobody owns this, model engineers accumulate private workarounds in their own images, and reproducibility disappears without anyone deciding to give it up.
Setting checkpointing policy: Automated checkpointing is a standard orchestration capability. Choosing the interval is a judgment call with money attached, because writing state costs throughput and skipping it costs entire runs when a node drops at hour forty. The right frequency follows from the observed failure rate of the cluster and the cost of writing state at that model size, and it is worth revisiting as either changes.
Diagnosing where throughput is lost: When a run slows down, the utilisation graph rarely identifies the cause on its own. It could be storage bandwidth, a collective communication step stalling across the fabric, a dataloader written on a laptop and never profiled at cluster scale, or a job placed across a spine switch when its processes needed to sit on one interconnect. Our post on generative AI infrastructure covers the layers this work touches.
Attributing cost: Per-pod and per-namespace telemetry, usually through DCGM, turns a single unattributed GPU bill into something a finance team can act on. Without it, capacity gets hoarded, because releasing a reservation nobody can price feels riskier than continuing to pay for it.
The signals that the role is missing
It usually becomes necessary well before anyone writes the job requisition:
- Model engineers are writing their own Dockerfiles and debugging CUDA errors instead of training
- Long training jobs fail on node loss and restart from the beginning
- Queue waits are measured in days while the utilisation dashboard reads single digits
- Capacity is held indefinitely because nobody can say what it costs or which team owns it
Each of these has a fix that sits a layer below where the model team works, which is why they persist until somebody takes responsibility for that layer.
How the role differs from the ones next to it
A machine learning engineer owns the model itself: architecture, training procedure, evaluation, and the quality of what ships. Their measure of success is model performance.
An MLOps engineer owns the model’s lifecycle once it is live, covering retraining triggers, drift detection, versioning and promotion. Our post on MLOps and AIOps covers those practices in detail.
A platform or SRE engineer owns service availability and reliability, generally without exposure to accelerator scheduling, fabric behaviour or distributed training failure modes.
An AI infrastructure engineer owns the substrate the other three depend on. In smaller teams one person covers several of these. The distinction starts to matter at the point where one team’s training run can starve another team’s inference workload, because that is a scheduling and quota problem rather than a modelling problem, and nobody else in the org chart is looking at it.
Why the hire is difficult
ManpowerGroup’s Talent Shortage Survey, covering more than 39,000 employers across 41 countries, found AI capabilities had overtaken every other category to become the hardest skills to find globally, with 72% of employers reporting difficulty filling roles.
The scarcity relevant here is narrower than that headline. A large pool of engineers can run Kubernetes competently. The smaller pool can do that while also reading a PyTorch stack trace, reasoning about collective communication patterns during all-reduce, recognising a fabric problem that presents as a slow dataloader, and holding a defensible opinion on whether a given workload belongs on partitioned accelerators.
DevOps and SRE experience is the usual route in, and it is a starting point rather than a qualification.
What the role is measured on
Worth agreeing the outputs before the search begins, whether the capability ends up hired, developed or contracted:
- Sustained GPU utilisation across the fleet, measured on active compute rather than memory residency, since a model loaded into VRAM is holding a reservation rather than doing work
- Queue wait time from job submission to first batch
- Cost per training run and per million inference requests, attributed to a team
- Mean time to recovery on node failure during long training jobs, which is a checkpointing question
- Environment reproducibility, measured by how often a job fails for reasons unrelated to the model
Acquiring the capability
Hiring suits organisations with large internal AI teams, sustained workloads and strict infrastructure requirements. The trade-off is a long search in the most competitive part of the market, followed by a single person becoming a single point of failure for the platform every model team depends on.
Developing internally works where a platform or DevOps team already exists with capacity to specialise. It is slower, and the learning happens on a production cluster.
Using a managed platform moves the capability rather than the headcount. Neurotechnology Cloud’s AI Cloud and managed GPU services run on a Managed PaaS model, under which every client is allocated support from a dedicated AI engineer who helps deploy and configure custom environments, debug training loops and out-of-memory errors, tune models toward target metrics, and optimise data preparation for GPU utilisation. How that works in practice is set out on the platform page. It is a separate question from whether to build or rent the hardware, which the AI-ready infrastructure guide addresses on its own terms.
GPU capacity starts generating a bill the day it is provisioned and starts generating value only once someone is accountable for keeping it busy. For organisations working out how to close that gap, Neurotechnology Cloud engineers and operates AI factory environments covering cluster design, storage, networking and orchestration, along with the operational layer that keeps them producing.
