The inference path

The gateway can sit between an agent and its model provider. With MCP_INFERENCE_UPSTREAM_URL set, the process that serves /mcp also serves /v1/messages, /v1/messages/count_tokens, /v1/chat/completions, /v1/responses and /v1/models, forwards each request unchanged to that one upstream under the provider credential, decides it on the policy’s models rules, and journals an intent and an outcome that carry usage and never a prompt or completion byte. This is PR 2 of the runtime security and governance plan (§6), with PR 3 (§7) adding inspection of prompts and completions under the policy’s content rules; flow rules and budgets arrive in PR 4 and are not provided by this release.

Deployment

One process. The routes mount inside the all and gateway roles, never on control, and never without an upstream. There is no inference role or container (decision 1): the content inspection, taint and budget state the later PRs add is per process, so the inference path runs where /mcp runs. With flows or budgets in the bundle the gateway runs as one replica; the Helm chart already refuses roles.<name>.replicas != 1 and deploys with Recreate.
Front end. Route /v1/* to the same container as /mcp through the TLS and WAF front end in deploy/compose/. deploy/inference/ overlays the collector, Tempo, Prometheus and Grafana with the five first questions as dashboards; it changes nothing about the gateway container’s shape. Raise the WAF’s body inspection limit together with MCP_INFERENCE_MAX_REQUEST_BYTES: a prompt with tool results is larger than an MCP call.
Settings. MCP_INFERENCE_UPSTREAM_URL, MCP_INFERENCE_PROTOCOL, MCP_INFERENCE_CREDENTIAL, MCP_INFERENCE_MAX_REQUEST_BYTES, MCP_INFERENCE_STREAM_IDLE_TIMEOUT_SECONDS, MCP_TELEMETRY_OTLP_URL and MCP_TELEMETRY_CHANNEL_CAPACITY, in configuration.md. Startup is fail-closed: an upstream without a credential, a credential without an upstream, a plaintext upstream off loopback, or an unknown protocol refuses to start.
Policy. The bundle is schema 3 with a models section:
```
schema_version: 3
version: 7
models:
  - id: engineering
    subjects: { groups: [engineering] }
    allow:
      providers: [anthropic]
      models: ["claude-*"]
    deny_kinds: [machine]
```
Rules are walked in document order over those whose subjects and providers apply to the request. A rule whose deny_kinds names the token’s verified kind denies and ends the walk; a rule whose allow.models glob matches the body’s model allows; a rule that applies but says neither says nothing. No decision is the default deny, with a reason that says so. A request that names no model (GET /v1/models) is allowed by any rule that admits the principal to the provider. A bundle with no models section denies every inference request: the path is enabled by configuration, but who may reach which model is policy. A native .cedar document carries no model rules and denies too. policy explain --model is not in this release.

Authentication and binding (decision 7)

An inference request binds to the owner the server’s auth mode yields, exactly as /mcp does. With inbound authentication on, that is the bearer token’s owner through the same validator, revocation list, generation and rate limiter, with the mcp:inference scope instead of mcp:tools; a request without a token is 401, with the wrong scope 403. With MCP_AUTH_MODE=off, which the server accepts only on a loopback bind, the owner is Local. There is no third rule.

A request may carry Mcp-Session-Id. If it does, the id must name a live MCP session bound to the same owner, else the request is refused with the same 404 Session not found as /mcp gives a foreign session. A matching id narrows the record (its digest is journaled as inference.session), never the enforcement; without the header the record says session: none. Nothing on this path creates or binds a session.

What is forwarded

The request body, byte for byte. The gateway parses it once to read model (for policy) and refuses a body that is not a JSON document, which the provider would refuse too; it never rewrites the body, so on an OpenAI-compatible stream usage is unknown unless the client itself asked for it with stream_options.include_usage (decision 2).
The content-type, accept, anthropic-version and anthropic-beta request headers. Every other header stays here: the caller’s Authorization, x-api-key and cookie first of all. The provider credential is added as x-api-key (anthropic) or a bearer (openai).
The W3C trace. A valid traceparent is continued upstream with this request as the parent span and kept verbatim in the journal; an invalid one is digested, the request starts its own trace, and the record says trace_linked: false (decisions 8 and 9). A well-formed tracestate rides along opaque.
Back to the client: the upstream status, content-type, request-id, x-request-id, retry-after, cache-control and the provider’s rate-limit headers, and the body as a stream, unbuffered. Nothing is retried after the first byte: a stream that ends early is one failed attempt (upstream_failed), never a replay.

Inspection (runtime security plan §7)

A schema 3 content rule over the prompt or completion subject turns inspection on; without one the path forwards and relays exactly as PR 2 did. The guard sees decoded text, never wire bytes (decision 15), and the forwarded bytes are never the decoded form.

content:
  - id: no-secrets-in-prompts
    subjects: [prompt]
    detect: [secret]
    action: deny            # or label
    binary_blocks: deny     # refuse undecoded base64 blocks (default: allow)
  - id: no-secrets-in-completions
    subjects: [completion]
    detect: [secret]
    action: deny            # or label
    mode: stream            # or buffer
  - id: external-engine
    subjects: [prompt]
    engine: webhook         # MCP_GUARD_WEBHOOK_URL
    action: deny
    on_error: deny

Prompts. The body is parsed once (the same parse that reads model); every string in the document, member names included and whatever field holds it (a tool-schema property name as much as a message), is handed to the detectors. A finding folds into the admission decision: deny answers 403 with the provider-shaped error naming the rule and detector, and nothing is forwarded; label records it and forwards. The content_finding records (subject prompt) are journaled before the intent, under the same fail-closed append. Base64 block payloads (an Anthropic source of type: base64, an OpenAI input_audio or file_data member, a data: URI) are not decoded: they are counted and left out of the text, and a rule with binary_blocks: deny refuses a prompt that carries one. A candidate payload that is not made of the base64 alphabet is text and is inspected, so a data: prefix hides nothing.
Completions, mode: stream (the default) on an event stream: each chunk is framed, its text deltas (text_delta, input_json_delta, thinking_delta, a citation’s cited_text; choices[].delta.content and tool-call arguments; a Responses API *.delta) are decoded and reviewed over a lookback window of MCP_GUARD_STREAM_LOOKBACK_BYTES before they are relayed, so a credential split across deltas or JSON-escaped across them is whole in the window. The built-in detectors rescan the window only when its new text could complete a secret: a streaming automaton over the secret patterns, with their look-around assertions dropped so it can only over-approximate, walks each delta once and says when a match could end in it. Otherwise the window holds nothing the previous review did not see, so the lookback costs memory, not CPU per delta. The inspection bound (MCP_GUARD_MAX_SUBJECT_BYTES) still applies to every window. Only the bytes of events that completed and were reviewed go to the client; the bytes of an event still in progress are held until it completes, whatever the network’s chunking. An event past the framer’s 1 MiB event bound cannot be inspected: each rule’s on_error decides (deny by default, ending the stream before any of it is sent), and its bytes are never held, so a rule that fails open relays it uninspected. On a deny finding the relay stops: the event holding the finding is never sent, the client’s stream ends with a transport error (no synthetic protocol event, decision 6), the upstream connection is dropped, and the outcome is content_terminated with bytes_sent the bytes already gone. A label finding is recorded once per rule, detector and class, the stream continues, and the outcome carries the classes as labels.
Completions, mode: buffer, and every non-streamed response: the whole response is held, bounded by MCP_GUARD_MAX_SUBJECT_BYTES, reviewed once, and released whole or refused with a 403 policy envelope and the outcome content_denied with bytes_sent: 0. A response past the bound was not kept, so it is refused whatever the rules’ on_error says (the finding records subject_too_large); nothing uninspected is ever released from this path. A buffered event stream still reaches the client as text/event-stream, in one piece. Usage is read off the buffer either way.
Provider errors (a non-2xx upstream status) are relayed as PR 2 relayed them; they are the provider’s words, not model output.
External engines. A rule with engine: webhook posts the subject’s decoded strings to MCP_GUARD_WEBHOOK_URL as {subject_kind, bytes_b64, classes, tenant} under the configured bearer and applies the answer under the rule’s own action: a verdict without findings is recorded as an engine finding so a refusal always leaves a record, and an engine’s detector and class names are bounded (64 printable ASCII bytes, at most 32 findings). A timeout (MCP_GUARD_WEBHOOK_TIMEOUT_MS), transport error, non-2xx status or malformed answer follows the rule’s on_error (deny by default). Each call that reached the engine is on the outcome’s egress record as vendor: guard.

What a prompt or completion rule cannot do is refused when the bundle loads, never ignored: redact (the forwarded body and the relayed stream are never rewritten, decision 2), hold on a completion (it exists before it could be held), mode outside a completion rule, binary_blocks outside a prompt rule. A prompt rule may hold (below).

For the optional local classifier service, automatic artifact downloads and GPU/ONNX eligibility checks, see Local model guards. The companion service is shipped as source separately from the Rust binary. The webhook adds x-mcp-guard-complete-subject: true for whole subjects and false for streamed lookback windows. The local service requires true; use buffer mode for its completion rules. Existing webhook JSON is unchanged. The supervised deployment example keeps guard readiness separate from process liveness, bounds worker restarts, and preserves operator on_error during recovery/exhaustion. Failed inspections are never replayed automatically.

Flows, budgets, holds and the kill switch (runtime security plan §8)

Admission on this path reads and writes the same per-process state as /mcp, keyed by the verified owner (decision 7), which is why the two paths run in one process (decision 1).

Flow rules. Before the models decision the owner’s taint classes are read from the store: the environment classes of upstreams the owner has acted on through /mcp, and the classes tool_result or completion rules labelled. A flows rule whose when.owner_has the owner satisfies and whose deny.subjects names prompt denies the request (403, the envelope naming the flow rule), unless the principal is in an unless.groups group. A completion a rule labels marks the owner in turn, so what a model said can keep the owner from prompting again. A fresh MCP session, a new bearer for the same subject, a changed run id or a reconnect all inherit the marks; nothing on this path creates or clears them; they expire by MCP_TAINT_TTL_SECONDS.
Budgets. Every budgets rule of the bundle is reserved at admission, after the prompt review and before the intent: the estimate is the request bytes over four plus the body’s max_tokens (max_output_tokens, max_completion_tokens), or the budget’s estimate.completion_tokens when the body sets none; a GET /v1/models reserves its bytes only. Under one lock per meter, spent + reserved + estimate must fit every budget, owner or tenant, all or nothing (decision 17). A budget that does not fit denies, or holds when it says action: hold. The reservation settles when the outcome is known: reported usage (input, output and cache tokens) replaces the estimate and an over-run refuses the next request in the window; an upstream rejection before any output releases it; a client disconnect, idle timeout, content termination or a stream with no usage frame settles at the estimate. An OpenAI-compatible stream: true body without stream_options.include_usage: true cannot report usage and the gateway never adds the option (decision 2): each budget’s unknown_usage says whether that is refused (deny, with the fix in the message), charged at the estimate, or admitted uncharged with usage: unknown on the record. The budget’s live state is per process.
Holds. A prompt rule with action: hold, or an exceeded budget with action: hold, asks the call-approval gate (MCP_CALL_APPROVALS) before the intent. The hold key binds the caller, the provider identity, the route, a digest of the exact body bytes and, for a content hold, the findings digest, so only the same body re-issued after approval runs. A held request answers 403 in the provider’s error shape with the type approval_required, the proposal id and expiry in the message and in the x-mcp-approval-proposal, x-mcp-approval-expires and x-mcp-hold-key headers; a second person approves over the admin API as for a tool call; the re-issue consumes the approval, forwards once, and is charged over the budget’s limit when the hold was a budget’s. A pending re-issue is the same proposal; a different body is a different hold. Without a gate configured, a hold is refused with approval_required.
Kill switch. The same signed revocation list that refuses a subject on /mcp refuses it here, within one poll, and closes the subject’s MCP sessions as before; the inference path has no session list of its own (decision 7). Revocation does not clear the owner’s taint marks: a subject re-admitted after the list is undone is still what it read.

What is recorded

Two journal records per request, joined by the application event id (decision 10), with the same principal, client, decision and upstream_identity fields as a tool call’s records:

Record	When	Carries
`inference_intent`	before the forward, allow, deny or hold; durable before the client hears the decision, else the request is refused (`503`)	provider, route, model, the `models` (or flow, budget or content) decision, `session`, `trace_id`, `trace_linked`, `trace_rejected`, `approval` (the proposal a held request waits on or ran under), `budget.entries` (each budget’s scope, window start, estimate, `unknown_usage` mode and state: `reserved`, `exceeded`, `approved` or `ignored`)
`inference_outcome`	when the stream ends: `completed`, `upstream_rejected` (a 4xx or 5xx relayed), `upstream_failed`, `idle_timeout`, `client_disconnected`, `content_terminated` (a streamed completion cut off by a rule) or `content_denied` (a buffered or non-streamed completion refused whole)	upstream status, `usage` (`known` with the provider’s counts, or `unknown`), `bytes_sent` before termination, `labels` when a completion rule labelled and `marked` for the classes that became the owner’s taint, `budget.entries` with each reservation’s `settlement` (`reported`, `released`, `at_estimate`), what was `charged` and any `over_run`, the egress record naming the upstream origin, the one attempt, and any engine consulted
`content_finding`	one per finding: before the intent for a prompt, after the stream for a completion	rule, subject (`prompt` or `completion`), detector, class, count, action, and the failure cause when the guard could not inspect; never the text

Usage comes from the provider’s own frames: Anthropic’s message_start and message_delta, an OpenAI usage object or a Responses API response.completed event, or the usage of a non-streamed body. An unmeasured completion is unknown, never zero. The activity export gains provider, model, inference_outcome, input_tokens, output_tokens, bytes_sent and trace_id columns, appended after every older column.

Every tool call and inference request also emits one telemetry event (TelemetryEvent, schema 1): correlation (event id, trace and span ids, session and run digests), verified identity, timing, the governance decision and the request’s own fields. With MCP_TELEMETRY_OTLP_URL the events leave as OTLP/JSON spans; without it nothing is buffered. The channel is bounded and drops under pressure, counted. Tool calls do not yet read a trace from the MCP _meta carrier: their spans start a trace of their own and say trace_linked: false.

Limitations in this release

Inspection is probabilistic and reads decoded text only (decision 3): the built-in detectors match credential shapes; base64 blocks are not decoded; a provider error body is not inspected. Prompt and completion bytes transit the process and are never persisted (decision 5), which inference_guard_tests::sensitive_values_never_reach_storage_or_logs checks against a journal file and the process log.
In stream mode the bytes relayed before a finding are gone; the record says how many. A client that needs nothing released on a finding sets mode: buffer and pays the latency.
A prompt or completion rule cannot redact, and a completion rule cannot hold (above).
Budgets count tokens and tool calls; spend_usd is refused at load because this build has no pricing basis. Taint and budget state is per process and in memory: one enforcing replica (decision 1), and a restart starts the windows and the marks afresh. Marks come from the environment class of an upstream and from content-rule labels; field classes on data profiles wait for database dispatch, which admission still gates (CF-48).
models rules are evaluated by the inference path only; tool calls are unaffected by the section.
One upstream per process. Translation between provider protocols is not done (decision 2).
The OpenAI-compatible protocol covers chat completions and the Responses API as far as their usage shapes; other endpoints under /v1/ are not routed.
Telemetry export runs on the HTTP transport only; stdio is a carry-forward item.

Validation record

The exit criterion for PR 2 is a representative Claude Code session on the native Anthropic path through the gateway, with streaming, tool calls and cancellation; PR 3 repeats it with prompt and completion rules in the bundle. Record the client, version, date and limitations here when it is run; until then the path is verified by inference_role_tests, inference_guard_tests, flow_policy_tests and run_budget_tests against fixture providers, not against a live one. PR 4’s exit criterion (one owner reads prod data over /mcp, is refused a vendor write over /mcp and a prompt over /v1/messages from a second session, is revoked, and stays refused on both paths) is flow_policy_tests and revocation_tests::revoked_subject_is_refused_on_both_paths_and_keeps_its_taint in one process; the restart of the control role is not covered by a test.

The optional trained ONNX guard has a separate CPU validation record and reproduction procedure. The opt-in gateway test uses the actual local classifier with a fixture provider; it does not establish live-provider, GPU or production-quality validation.