Incident response runbook

Published from the project repository

This page is generated from the project’s documentation. References to source files, tests and unpublished documents appear as plain text, and test names such as suite::test_name identify the evidence behind a claim. The source repository is not public yet, so that evidence is available on request.

Written 2026-10-28 for WP5 of the AI governance gap-closure plan. This is what an operator does when something goes wrong in traffic that passes through the gateway: a leaked token, an agent that steers toward exfiltration, a vendor credential that turns up in content, a policy that grants too much, or evidence that does not verify. It strings together controls that already exist and are documented elsewhere; each step links to the runbook that explains its control.

The operator roles table in the shared deployment checklist names who does each step. This page uses those role names.

Out of scope: incidents at the vendor (its own audit log, its own credential revocation), at the identity provider, or inside the agent runtime. The gateway’s evidence helps there; it does not replace their procedures. Notifying the model provider, regulators or affected people is the organization’s duty; the governance framework map says which framework clauses ask for it.

Before an incident

An incident is the wrong time to find out a prerequisite is missing.

• Roles are assigned. Each role in the operator roles table has a named person and a deputy, and at least two people hold mcp:admin so two-person approval can proceed while one of them is the subject of the incident.
• The policy key holder can act within minutes. Revocation is a signed generation, so the person holding the policy private key, the paths of the revocation list (MCP_REVOCATION_FILE) and the generation manifest (MCP_AUTHORIZATION_GENERATION_FILE), and the way to publish them to every gateway (a ConfigMap, a shared volume) must be reachable on call.
• Known-good generations are kept. Keep every manifest, document and detached signature you publish (six files per generation). Generations only move forward, so a rollback re-signs a kept policy as a newer generation; the state backup does not cover these files.
• Evidence leaves the host. Journal checkpoints are exported off the gateway host, the audit public key is held where the gateway operator cannot change it, and SIEM forwarding is on (rows “Journal checkpoints leave the host” and “SIEM forwarding” of the shared deployment checklist).
• Signals reach a person. MCP_METRICS=on with the alerts in the usage runbook, SIEM rules over the record kinds below, and the failures and denials dashboard where the inference path is deployed.
• The drill has been run. The demo drill below takes under 15 minutes on a running demo stack.

The sequence

1. Detect: a signal below fires.
2. Triage: scope who, what, when, and whether anything reached a vendor.
3. Contain: stop the subject, the credential or the policy, in that order of urgency for what triage found.
4. Preserve evidence: verify and export the journal for the window.
5. Recover: lift what containment imposed, once the cause is gone.
6. Learn: turn the incident into a rule and a regression case.

Containment comes before evidence when an actor is still active: the journal is append-only, and revoking a subject adds records without changing earlier ones. Preserve first when nothing is still running.

1. Detect

Every signal is metadata. The journal never holds a token, tool arguments, a prompt, a completion or response content (audit reports).

Signal	Where it appears	What it can mean	First step
content_finding on write_argument or prompt (class secret, or a webhook class)	Journal and SIEM; activity export columns content_rule, detector, class, finding_count, content_action; dashboard panel “Spans with content findings”	An agent tried to send a credential-shaped value to a vendor or a model. With content_action: label nothing was stopped	Triage the subject. If the value was real, treat its credential as exposed and rotate it
content_finding on tool_result or completion	Same	Vendor content or a model answer carries a credential-shaped value, or a planted instruction travelled with one	Find the resource from the joined intent (same event_id); the vendor resource owner removes the value and the secret owner rotates it
A tool_call_intent denied by a flows rule	Journal; activity export effect: deny, rule_id the flow rule, reason naming the class and the destination, owner_classes	An owner that read a sensitive class tried to write elsewhere or prompt a model: the exfiltration pattern flows exist to stop	Triage the owner’s run around it; repeated attempts are a containment case
A decision whose reason is denied by budget <id> or budget <id> requires approval	Journal; export budget_ids, budget_states	A runaway agent loop, or abuse of a model or tool allowance	Reject the held proposal if the run is not legitimate; triage the owner
revoked_token_rejected	Journal and SIEM; audit metrics per subject	Something still holds a revoked subject’s or token’s credentials and is using them	Find what is calling: an agent left running, or an attacker. reason: token_id with the owner working normally elsewhere means the leaked token is in someone else’s hands
authorization_generation_rejected (or policy_rejected without a generation); mcp_policy_degraded is 1	Journal; /metrics; health reports degraded	Files on the gateway that no signed generation binds: a broken publish, or an attempt to install an unsigned, unsealed or older policy or list	Treat it as tampering until a failed publish is confirmed; the journal custodian checks who can write those files
authorization_generation_changed with no admin_mutation (source: generation/publish) before it	Journal	Files written on the host: an offline revoke, or a signed generation installed directly, bypassing two-person approval by design	Confirm the policy key holder made it and why; otherwise escalate as a compromised signing key
A rise in deny decisions or egress_decision records for one subject; mcp_tool_calls_total{decision="deny"} rising	Journal; /metrics	Probing: an agent, or a person through it, testing what policy allows	Triage the subject
Repeated call_proposal records from one subject; approval_backlog refusals	Journal; console Approvals	An agent retrying a held write, or flooding approvers	Reject the proposals that are not legitimate
mcp_rate_limited_total rising	/metrics	A loop or abuse past the per-principal rate	Triage which subject the 429s go to (activity export)
mcp_audit_journal_available is 0	/metrics; health	The evidence path is down. Tool calls are refused while it is (fail closed)	Restore journal storage; the outage is itself the incident
mcp_audit_forward_degraded is 1	/metrics	The SIEM copy is behind; the local journal is retained	Restore the receiver before the local journal is the only copy for long
mcp_secrets_degraded is 1	/metrics; operator log names the reference, never the value	The secret source is unreachable or a credential was withdrawn; last good values stay in force	Check the secret source; a withdrawn credential that someone else withdrew may be another team’s containment
mcp-devtools audit verify exits 1	A verification run, scheduled or by hand	A journal edited, truncated or missing checkpoints	Evidence incident: preserve every copy, the off-host checkpoints and the SIEM copy before anything else

2. Triage

Answer four questions from a copy of the journal (nothing here needs the gateway or the network):

• Who: subject, tenant and groups from the validated token; never a client-reported name.
• What: tools, vendors, resources and environment classes, and the classes the owner was marked with (marked, owner_classes).
• When: first and last record in the window.
• Did anything leave: a denied or held intent sent nothing upstream. An allowed call’s outcome lists each egress decision and whether the request was sent; egress_attempted counts sends, including those still in flight.

mcp-devtools audit export activity /path/to/journal-copy \
    --subject "$SUBJECT" --since "$START" --format jsonl --out subject.jsonl
mcp-devtools audit export activity /path/to/journal-copy \
    --since "$START" --kind content_finding --format csv --out findings.csv
mcp-devtools audit report --data-map --tenant "$TENANT" \
    --store sqlite:///var/lib/mcp-devtools/usage.db --since "$START"

The upstream identity on each record (upstream_label, upstream_authority) says whose vendor credential acted: shared means the gateway’s own credential, so the vendor’s log attributes the calls to the gateway and the journal is the attribution. What the calls read or wrote is in the vendor’s own audit log, under that identity and resource; the journal holds neither.

On an all-role process, mcp-devtools admin principals --tenant … --subject … and admin sessions list show the subject’s live sessions. policy explain reproduces any decision from the policy in force (mcp-devtools admin policy explain …) or from a kept file (mcp-devtools policy explain <file> …).

3. Contain

Situation	Action	Role	Takes effect	Journal evidence
A subject’s tokens are compromised, or its agent must stop	mcp-devtools revoke subject <sub> --reason … --file revocations.yaml --generation generation.yaml --key policy-signing.key, then publish the list and the manifest together (revocation runbook)	Revoker (policy key holder)	Within one poll of the files landing, on /mcp and /v1/* alike; the subject’s sessions close and its taint marks stay	authorization_generation_changed, then revoked_token_rejected with reason: subject
One token leaked; its owner keeps working	mcp-devtools revoke token <jti> …	Revoker	Same	revoked_token_rejected with reason: token_id
Identity-provider signing key compromised, or a gateway breach	Rotate the key at the identity provider first, then mcp-devtools revoke all … (revocation runbook)	Identity team, then revoker	Every token issued before the cut-off is refused; every session closes	authorization_generation_changed with not_before; refusals with reason: not_before
A live session must close now, before a revocation lands	mcp-devtools admin sessions remove <id> (all role only). The subject’s next token opens a new session, so follow with a revocation	Any administrator	Immediately, one person by design	admin_mutation with source: sessions/revoke
An artifact holds data that must not be read again	mcp-devtools admin artifacts remove <id>	Any administrator	New reads refused immediately; storage reclaimed after existing reads finish	admin_mutation with source: artifacts/purge
A vendor or model-provider credential is exposed	Revoke or rotate it at the vendor first, then write the new value as a new version in the secret source (secret sources runbook)	Secret owner	At the next refresh (MCP_SECRET_REFRESH_INTERVAL_SECONDS, default 30 s), with no restart; a literal credential in configuration needs a restart	The version on each call’s credential metadata changes
A policy in force grants too much	Re-sign the last known-good policy and seal it as the next generation (mcp-devtools generation seal --manifest … --policy …); submit it with mcp-devtools admin generation publish for two-person approval, or install it directly in an emergency	Policy author and approver; key holder for the emergency install	When applied (approved) or within one poll (direct install). An older generation’s files are refused, never a rollback	admin_proposal, admin_approval, admin_mutation, authorization_generation_changed, admin_application; a direct install shows only the change record
All tool calls must stop	Seal a generation whose policy has rules: []; deleting the policy file is not an emergency deny	Key holder	Within one poll	authorization_generation_changed
Model access must stop	Seal a generation without the models rules that admit it, or revoke the subjects using it	Key holder	Within one poll; refused before any upstream request	inference_intent with effect: deny
Held calls from the suspect are pending	mcp-devtools admin proposals reject <id> --reason …	Any administrator in the tenant	Immediately; the same hold key is refused until the proposal’s TTL passes	call_rejection

A direct install of a signed generation (the revoke commands, or generation seal with the files copied to the gateways) does not wait for a second administrator. That is the emergency path by design; record it in the incident and have a second administrator review the generation afterwards (mcp-devtools admin policy diff).

4. Preserve evidence

Work on copies. The originals stay where they are.

1. Copy MCP_AUDIT_JOURNAL_DIR and the off-host copy of MCP_AUDIT_CHECKPOINT_EXPORT_DIR. A clean stop seals the journal’s tail; on a running gateway, records after the last checkpoint are reported as unsealed rather than verified.

2. Verify the copy against every audit public key it spans and the off-host checkpoints (verifying a copy):

   mcp-devtools audit verify ./journal-copy --public-key "$AUDIT_PUBLIC_KEY" \
       --checkpoints ./checkpoints-offhost --json > verification.json

   Exit 0 is required. Exit 1 makes this an evidence incident too: keep every copy, including the SIEM’s, and record which records the report names.

3. Export the window, all kinds, as JSON Lines and CSV (audit export activity … --since … --until …). Add mcp-devtools audit metrics ./journal-copy --json for each subject’s revocation_window_ms: the time from the generation that revoked it to its first refusal.

4. Keep the generation in force and the one before it: mcp-devtools admin generation read --out-dir ./generation-now and mcp-devtools generation show --manifest … --public-key "$MCP_POLICY_PUBLIC_KEY".

5. Compare the SIEM’s copy of the window with the local export. The SIEM copy is the one a compromised gateway host could not edit.

6. Hash the bundle and store it write-once. It holds no credentials or content, so it can be shared with whoever investigates.

5. Recover

• Lift containment only once its cause is gone: the token expired or was withdrawn at the identity provider, the credential rotated, the policy fixed. Each lift is a new generation (mcp-devtools revoke remove subject …, revoke clear-all); never put an older list back.
• A re-admitted owner keeps its taint. Flow marks survive revocation and its undoing for MCP_TAINT_TTL_SECONDS (default 24 h) or until the process restarts, so flow rules still refuse what they refused before. That is intended.
• Review emergency installs. A generation installed without approval gets a second administrator’s review now, or is replaced through the approval gate.
• Check health is back: mcp-devtools health, and the gauges above at their normal values.

6. Learn

• Draft the rule from what happened. The console’s activity groups, or POST /admin/policy/draft, turn the incident’s grouped requests into a deny or require_approval rule appended to the policy in force (policy drafts from activity). Check it with policy explain against the incident’s calls, then publish it through two-person approval.
• Catch it in band next time. If the pattern was exfiltration, a flows rule; a credential, a content rule with deny or hold; a loop, a budgets rule (policy explain --flow, --content, --budget show what each would do).
• Lock it with a test. Add the sequence as a regression case to tests/adversarial_workflow_tests.rs, with its content under tests/fixtures/adversarial/ (OWASP assessment follow-up 2).
• Update the security documents if the incident showed a control the documents claim and the code does not back.

Automating the response

What a SIEM rule or a SOAR playbook can do today, and what stays with a person.

Step	Automatable today	How	Limit
Detect	Yes	SIEM rules over the forwarded journal’s kind, decision.effect, rule_id and content fields; Prometheus alerts on the gauges and counters above; OTLP spans with governance.decision and governance.findings	Metadata only, by design
Contain in band	Yes, in policy	content rules (deny, hold, redact), flows rules and budgets act on each request without anyone watching	Each decision is per request; nothing reacts to a pattern across requests yet (#200)
Preserve evidence	Yes	audit verify --json (exit code 0 or 1) and audit export activity --format jsonl on a copy; needs read access to the journal and the checkpoints, no gateway privilege
Rotate a vendor credential	Yes	A new version in the secret source; the gateway reads it at the next refresh	Revoking the old credential at the vendor is the vendor’s API, not the gateway’s
Close a session, purge an artifact, reject a proposal	Only with a person’s token	mcp-devtools admin … with --json is scriptable, so a responder can run a prepared script with their own token	/admin/* refuses a principal the identity provider marks as a machine under the default MCP_ADMIN_PRINCIPALS=human-or-unknown
Revoke a subject	No		Revocation is signed with the policy key, and a holder of that key can sign any policy. Do not give it to an automation. A narrow, time-bounded suspension an automation can call is #199

A pattern that works now: the SIEM rule pages the policy key holder with the subject, the evidence export already attached, and the exact revoke subject command to run. Containment then takes as long as it takes that person to run one command, and the decision stays with someone accountable.

Drill on the demo stack

Rehearse containment and evidence on the incident investigation demo before relying on this page. Rehearsed on 2026-10-28 against a running stack: the commands take about two minutes; allow 15 for a first run with this page open. The demo keeps the private policy key inside its container for convenience; production keeps it off the gateway.

From the repository root, with the stack up and the rehearsal complete (./scripts/demo up, then ./scripts/demo rehearse, which leaves the engineer with production access):

demo() { docker compose -f deploy/demo/compose.yaml "$@"; }
mcpd() { demo exec -T mcp /build/debug/mcp-devtools "$@"; }
keys=(--file /state/config/revocations.yaml --generation /state/config/generation.yaml --key /state/config/policy.key)

# 1. Triage: who read loki-prod? The subject column is the engineer's sub.
mcpd audit export activity /state/journal --kind tool_call_intent --vendor grafana --format csv
SUB=<subject from the loki-prod row>

# 2. Contain: revoke the subject. The gateway watches these files directly.
mcpd revoke subject "$SUB" --reason "drill: suspected token compromise" "${keys[@]}"
./scripts/demo before        # fails: 401 invalid_token, "the token has been revoked"

# 3. Roll the policy back to QA-only through two-person approval.
#    The revocation is carried into the new generation.
./scripts/demo restore

# 4. Evidence: a clean stop seals the tail, then verify, export and tamper-check a copy.
./scripts/demo evidence      # prints /state/demo/evidence-<timestamp>
demo cp mcp:/state/demo/evidence-<timestamp> ./incident-evidence

# 5. Recover: restart, lift the revocation, confirm the baseline boundary.
./scripts/demo up
mcpd revoke remove subject "$SUB" "${keys[@]}"
./scripts/demo before        # loki-qa allowed; loki-prod and loki-finance denied
mcpd audit metrics /state/journal --json   # revocation_window_ms per subject

In ./incident-evidence/journal/audit-journal.jsonl the drill leaves:

• the revocation’s authorization_generation_changed with no admin_mutation before it (an offline publish), then revoked_token_rejected with reason: subject for the engineer’s refused request;
• the rollback’s admin_proposal (Alice), admin_approval (Bob, a different subject), admin_mutation, authorization_generation_changed and admin_application;
• verification.json with no problems and tamper-verification.json reporting the removed record.

For the Cedar cousin, use scripts/demo-cedar and add -f deploy/demo/compose.cedar.yaml -p mcp-devtools-demo-cedar to demo.

Tests behind the procedures

The procedures above rely on these behaviors, each locked by a test:

Procedure	Test
Revoking a subject refuses it within a poll and journals the refusal	revocation_tests::a_revoked_subject_is_refused_within_a_poll_and_the_refusal_is_journaled
One revocation covers both paths and keeps the owner’s taint	revocation_tests::revoked_subject_is_refused_on_both_paths_and_keeps_its_taint
revoke all refuses older tokens and closes their sessions	revocation_tests::revoke_all_refuses_tokens_issued_before_the_cutoff_and_closes_their_sessions
A generation’s halves take effect together, and only a newer one	authorization_generation_tests::both_halves_take_effect_together_and_only_under_a_newer_manifest
A revocation published through the admin API is audited and enforced	admin_api_tests::a_published_revocation_is_audited_persistent_and_enforced
A generation publish waits for a second administrator	unified_approvals_tests::required_defers_a_generation_publish_until_a_second_administrator_approves
Self-approval, other tenants and stale tokens are refused	unified_call_approvals_tests::self_approval_other_tenants_rejection_and_stale_tokens_are_refused
Sessions can be listed and closed	admin_api_tests::session_inventory_lists_and_closes_real_rmcp_sessions
Artifact purge withdraws reads and fails closed	admin_api_tests::artifact_purge_uses_real_pin_aware_backend_and_fails_closed
A rotated secret changes the upstream credential without a restart	secret_source_tests::rotation_lock_a_rewritten_file_changes_the_upstream_header_without_a_restart
The verifier catches edits, forgeries, wrong keys and a truncated tail	audit_checkpoint_tests::the_verifier_catches_edits_forgeries_wrong_keys_and_a_truncated_tail
The activity export filters and flattens every record	audit_report_tests::activity_export_flattens_every_record_and_filters
Content findings export without text	audit_report_tests::content_findings_export_without_text
Flow and budget columns reach the export	audit_report_tests::export_includes_inference_and_flow_rows
A draft from observed activity matches only what was observed	activity_groups_tests::a_draft_from_an_observed_group_compiles_and_matches_only_what_was_observed
/admin/* refuses machine principals by default	admin_principal_kind_tests::default_policy_refuses_machines_at_admin_and_nowhere_else

docs_evidence_tests fails CI when any test cited here stops existing. The drill itself runs against Docker and is not a CI gate; rerun it after a change to revoke, generation, audit or the demo stack.