Serving models to agents

Serving models to agents: what changes when your API consumer is an LLM

Most model APIs were designed for client applications, written by developers who read the documentation before anything goes live. When the consumer is an AI agent, usually connecting through a Model Context Protocol (MCP) server, a language model reads the interface at the moment it uses it and several design decisions change as a result.

The serving stack underneath stays much the same so the changes sit in the layer where the model meets its caller.

What is different about an agent as an API consumer?

A developer integrating with an API makes most decisions in advance, including which endpoints to call and how to handle errors. An agent makes those decisions at runtime. It reads the description of each available tool and picks one for the task in hand, and what it does next depends on whatever comes back.

The Model Context Protocol has become the common standard for this connection. An MCP server lists the tools it offers, and the MCP tools specification describes them as model-controlled, meaning the language model discovers and calls them itself. Tool names, descriptions and input definitions are passed to the model alongside the user’s request, so they shape its behaviour as directly as any prompt.

How should tools be described for an agent?

Anthropic’s guidance on writing tools for agents describes a tool as “a contract between deterministic systems and non-deterministic agents”, and argues that tools should not be designed the way developers design APIs for each other. In practice, that means:

  • Fewer, broader tools: An agent that has to chain several low-level calls has more chances to get one wrong. Anthropic’s own example replaces several calendar tools with a single scheduling tool.
  • Clear parameter names: A parameter called user_id leaves less room for guesswork than one called user.
  • Defined outputs: A tool can declare the structure of its results, so the client can check what comes back before passing it on.
  • Settings the agent cannot change: A fraud-scoring model’s decision threshold carries a known false-positive rate and often a regulatory rationale, so it belongs in server configuration, out of reach of an agent that has just had a transaction declined.

Why does statelessness matter for an MCP server?

The current MCP specification has removed sessions between client and server. Each request now carries everything the server needs to handle it, so any server behind a load balancer can answer any call.

The trade-off is that anything a tool needs to remember between calls has to be passed back explicitly. The specification suggests the server issues a reference, known as a handle, when the item is created, and the agent includes it in later calls. A handle identifies the item without granting access to it, so the server should check the caller’s permissions each time it is used.

If a response is interrupted, it can no longer be resumed either. The client has to send the request again, so any tool that writes data or starts a job needs protection against running twice, the same safeguard our post on model serving architecture recommends for ordinary API retries.

How should an MCP server handle authentication?

Under the MCP authorisation specification, a protected MCP server works as an OAuth 2.1 resource server, the standard role for a service that accepts access tokens issued by a separate sign-in system. The server must check that every token it receives was issued specifically for it.

The server must also never accept or forward tokens meant for anything else. This prevents what security teams call a confused deputy problem, where a trusted service is tricked into using a caller’s credentials somewhere they were never meant to go. A model server calling another system should use its own credentials. A server holding wider permissions than its user is a related risk, covered in our post on running AI agents on private infrastructure.

Permissions can also shape what the agent sees. The list of tools may vary with the caller’s access, so an agent with read-only access need never see a tool that deletes records. Labels marking a tool as read-only or destructive are hints, and the specification tells clients to treat them as untrusted unless they come from a trusted server, so real enforcement has to happen on the server itself.

What should an error message tell an agent?

MCP separates two kinds of failure. Problems with the request itself, such as calling a tool that does not exist, are reported as protocol errors. Problems while the tool runs, including invalid input, come back as results marked as errors, which clients pass to the model so it can try again.

That makes the wording of error messages part of the design. A generic “invalid input” gives the agent nothing to work with, while a message that names the field and the accepted values usually leads to a correct second attempt. It helps most when the message includes something the model could not have known, and the specification’s own example rejects a departure date in the past and states the current date in the same message.

Error messages should still leave out internal detail, which is why the specification requires servers to sanitise tool output. A stack trace helps your engineers and anyone trying to manipulate the agent.

How should rate limits work for agent traffic?

MCP requires servers to rate limit tool calls, but a simple quota per API key suits agents poorly. One instruction can turn into dozens of tool calls, and a quick metadata lookup can sit on the same server as a GPU-heavy inference job. As our post on AI inference cost at scale shows, cost follows the work done, so limits work better when set per tool and per customer.

The MCP Streamable HTTP rules make this easier, because clients must copy the method and tool name into the request headers. A gateway can then apply limits by tool without reading the request body.

Short-term throttling can use the standard HTTP “too many requests” response, but a business limit such as a daily inference quota is better returned as a tool error that states the limit and when it resets, so the agent can plan around it and stop retrying.

From client APIs to agent-ready interfaces

None of this replaces the serving layer, since the MCP server sits on top of it as a narrow, well-controlled entry point. The US National Security Agency’s AI Security Center reaches a similar conclusion in its guidance on MCP security design: authentication and input validation still apply, but established defences do not adequately address risks such as dynamic tool calls and implicit trust between components.

For teams building AI products, agents are one more consumer of the API and integration layer described in our guide to AI product development.

Neurotechnology Cloud builds and hosts custom models on dedicated EU-resident GPU infrastructure through its applied AI solutions practice, including the serving and access controls that agent traffic depends on.

If you are preparing a model to be called by agents and working out what the interface needs, get in touch.

Share: 

Contact us

Interested in our products, custom solutions, or partnership opportunities? Have questions about our technologies or need more information before purchasing? Fill out the form, and our team will get back to you as soon as possible.