Revocation and key-rotation runbook

Published from the project repository

This page is generated from the project’s documentation. References to source files, tests and unpublished documents appear as plain text, and test names such as suite::test_name identify the evidence behind a claim. The source repository is not public yet, so that evidence is available on request.

Written 2026-09-03 for Phase B (WP B.4) of the enterprise product plan (retired Enterprise repository, docs/enterprise-product-plan.md). Plan §3.6 states the revocation semantics; this is how an operator acts on them, what takes effect when, and what the journal shows afterwards. Revocation is one containment step of the incident response runbook, which covers detection, evidence and recovery around it; the operator roles table names who holds the key it needs.

The controls, and their windows

Event	What to do	Takes effect	Journal evidence
A user leaves, or must lose access now	mcp-devtools revoke subject <sub> --generation …	≤ 500 ms after the list and its sealed generation land on every gateway (one poll), on the next request and at the next decision of a call already in flight, whether or not the token is cached	authorization_generation_changed (with revoked_subjects and both document versions), then a revoked_token_rejected per refused request with the principal, reason: subject, and the generation
One token leaked (pasted, logged) but its owner keeps working	mcp-devtools revoke token <jti> --generation …	same	revoked_token_rejected with reason: token_id
Identity-provider signing key compromised; gateway breach suspected; “everyone re-authenticates now”	rotate the key at the identity provider first, then mcp-devtools revoke all --generation …	same; every token issued before the cut-off is refused, every principal session is closed, the validated-token cache is cleared	authorization_generation_changed with not_before; refusals carry reason: not_before
Group removed at the identity provider (planned offboarding, role change)	nothing — the next token the client obtains lacks the group and policy denies it	when the client mints a new token: ≤ token TTL (set 5–15 min at the identity provider), plus up to 5 min of validated-token cache for the old token’s group claims. A token minted before the change keeps its groups until then. If that is too slow, revoke the subject	the first tool_call_intent with decision.effect: deny after the last allow for that subject
Token expiry	nothing	on the next request after exp (plus configured skew)	none; the challenge says expired
An agent run must stop on both paths (runtime security plan §8 step 4)	mcp-devtools revoke subject <sub> --generation …: the inference routes (/v1/*) consult the same list, so the subject’s prompts are refused within the same poll as its tool calls, and its MCP sessions are closed; the subject’s taint marks stay, so an owner re-admitted later is still what it read	same	revoked_token_rejected for requests on either path; the inference_intent records of a re-admitted owner carry the flow decision its marks produce

Measured in tests/revocation_tests.rs (the printed windows are under 2 s on a loopback gateway with the default 500 ms poll):

• a_revoked_subject_is_refused_within_a_poll_and_the_refusal_is_journaled
• revoke_all_refuses_tokens_issued_before_the_cutoff_and_closes_their_sessions
• group_removal_waits_for_the_token_but_revocation_does_not — the token minted with the group stays allowed after the group is removed at the identity provider until it expires; revoking the subject cuts it off within one poll.
• revoked_subject_is_refused_on_both_paths_and_keeps_its_taint — one list refuses /mcp and /v1/messages alike, and the owner’s taint survives the revocation and its undoing.

The bound is the watcher’s poll interval on each gateway plus however long the signed file takes to reach them (a ConfigMap update propagates in tens of seconds on Kubernetes; a shared volume is immediate). Nothing waits for a cache entry to age out.

The list is half of a generation

The revocation list and the policy are published together, as one signed authorization generation (CF-24): a manifest (MCP_AUTHORIZATION_GENERATION_FILE, generation.yaml below) naming a generation number and the SHA-256 of each document, signed with the policy key. A list on disk that no manifest binds is not in force — however validly signed — so a list edited without sealing a generation revokes nobody, and a gateway says so (health degraded, one authorization_generation_rejected record). Every revoke edit therefore takes --generation: it re-signs the list and seals the next generation in one step, under the generation’s publish lock (generation.yaml.lock), carrying the policy half over from the generation it replaces. It refuses to edit a list that the current generation does not bind, so it never publishes someone else’s unsealed change along with its own.

Generations only move forward: a gateway refuses a manifest whose number is not newer than the one in force. Putting back an older, validly signed list — from before someone was revoked — is a rejected reload, not an un-revocation. Each decision re-checks the list of the generation it is taken under, and its audit record names that generation, the policy version, and the list version (authorization), so the journal says which pair decided every call.

Prerequisites

• MCP_POLICY_PUBLIC_KEY on every gateway, MCP_REVOCATION_FILE pointing at the list, MCP_AUTHORIZATION_GENERATION_FILE at the manifest, and all three files signed (the gateway refuses to start otherwise). A new deployment creates generation 1 once:

  mcp-devtools revoke init --file revocations.yaml --key policy-signing.key
  mcp-devtools policy sign policy.yaml --key policy-signing.key
  mcp-devtools generation init --manifest generation.yaml \
      --policy policy.yaml --revocation revocations.yaml --key policy-signing.key

• The policy signing key (policy-signing.key from mcp-devtools policy keygen) with whoever is allowed to revoke. It is the same key that signs the policy: revocation is an authorization change, and it should need the same authority.

• mcp-devtools on an operator machine (any platform; the CLI reads and writes files, and never needs the gateway’s configuration). Two operators editing at once are serialized on generation.yaml.lock (an advisory lock every revoke edit and generation seal holds from its read to its write), so neither entry is lost and no two edits publish the same generation; revoke init and generation init never replace a file that exists.

Procedures

Revoke a subject

mcp-devtools revoke subject alice@acme.example --reason "offboarded 2026-09-03" \
    --file revocations.yaml --generation generation.yaml --key policy-signing.key
mcp-devtools revoke show --file revocations.yaml --public-key "$MCP_POLICY_PUBLIC_KEY"
mcp-devtools generation show --manifest generation.yaml --revocation revocations.yaml \
    --public-key "$MCP_POLICY_PUBLIC_KEY"

Publish revocations.yaml, generation.yaml, and both .sig files to wherever the gateways read them from — together, in one update (the mcp-devtools-policy ConfigMap in deploy/k8s/config.yaml updates all of its keys at once). A gateway whose control plane has the administrative API can instead take the generation there: mcp-devtools admin generation publish --manifest generation.yaml --revocation revocations.yaml (fetch the files in force first with admin generation read --out-dir). The subject’s sessions are closed and their next request is refused with WWW-Authenticate: … error="invalid_token", error_description="revoked".

Confirm in the journal: an authorization_generation_changed record whose revocation_version moved, then revoked_token_rejected records for that subject.

Revoke a token

Find the jti: it is in the identity provider’s token logs, or — if the token was pasted somewhere — decode its payload. Then:

mcp-devtools revoke token "$JTI" --reason "pasted into JIRA-123" \
    --file revocations.yaml --generation generation.yaml --key policy-signing.key

A token with no jti cannot be named; revoke its subject instead.

Revoke everything (revoke all)

1. If the identity provider’s signing key is the problem, rotate it there first — otherwise the attacker keeps minting tokens the gateway accepts. Gateways refetch the JWKS on an unknown kid (rate-limited to one fetch per 30 s) and evict cached tokens whose key disappeared.
2. mcp-devtools revoke all --reason "IdP key rotated after incident INC-42" \ --file revocations.yaml --generation generation.yaml --key policy-signing.key
3. Publish the list and the manifest, with their signatures. Every gateway clears its validated-token cache and closes every principal session; every token with iat before the cut-off — or with no iat — is refused. A token whose iat is in the future (beyond MCP_OIDC_CLOCK_SKEW_SECONDS / MCP_OKTA_CLOCK_SKEW_SECONDS) is refused at authentication as not_yet_valid and never reaches the cut-off test, so the cut-off cannot be outrun by a mis-dated token. Clients re-authenticate and re-initialise.
4. Once the incident is closed and every legitimate token has aged past the cut-off, the cut-off can stay (it costs nothing) or be dropped with mcp-devtools revoke clear-all.

Undo

mcp-devtools revoke remove subject alice@acme.example --file revocations.yaml --generation generation.yaml --key policy-signing.key
mcp-devtools revoke remove token "$JTI" --file revocations.yaml --generation generation.yaml --key policy-signing.key
mcp-devtools revoke clear-all --file revocations.yaml --generation generation.yaml --key policy-signing.key

Each is a new generation, published like any other. Undoing is never done by putting an older list back: gateways refuse a generation that is not newer. The journal keeps the change history; the files hold only the current generation.

Key rotation

Guided rotation for a systemd user service

The console’s Key rotation page (/console/rotation) shows administrator-only public key information and configured artifact references. Preparation and activation run from the service owner’s terminal: the browser never uploads private keys or executes service commands. IdP keys remain provider-managed.

Prepare one role at a time, using a new output directory for each operation:

mcp-devtools rotation policy \
  --policy /absolute/security/policy.yaml \
  --revocation /absolute/security/revocations.yaml \
  --manifest /absolute/security/generation.yaml \
  --public-key "$MCP_POLICY_PUBLIC_KEY" \
  --out /absolute/operator/rotation-policy

mcp-devtools rotation audit \
  --key /absolute/security/audit-signing.key \
  --out /absolute/operator/rotation-audit

Policy preparation verifies every existing signature and generation binding, compiles the policy and revocations, then re-signs their unchanged bytes and advances the generation. It does not need the old private policy key. Native Cedar policies retain the .cedar extension and require a Cedar-enabled binary. Audit preparation retains the old key and writes both public keys for historical verification. Neither operation modifies active files or clears revocations. A retry with identical inputs reuses the generated key; changed inputs or artifacts require a fresh output directory. Add --compromised when the old key must never be restored. A valid signature from a compromised key is not proof its policy contents are trustworthy: investigate the baseline first.

The output directory is owner-only (0700 on Unix). It contains the replacement private signing.key, signed artifacts for policy rotation, public key files, request.json, a recovery record rotation.json, and redacted transition records events.jsonl. Every file is written to a temporary name and installed whole, so an interrupted preparation resumes. The recovery record binds the environment file by SHA-256 digest only and never copies its contents. Never upload or distribute the entire directory: it holds a private key until you move it. Only the redacted rotation status output is suitable for troubleshooting.

Move the policy key before activation. Activation points the gateway’s policy, revocation and generation settings at files in this directory, so a policy signing.key left here sits beside the documents it controls. Policy activation refuses while signing.key is in the directory: move it off the gateway host (offline media or the operator’s own key store) and bring it back only to sign later policy or revocation changes. Deployment needs only the public signed set, so status, activation and rollback keep working without it; preparation can still resume once its record exists, and refuses to generate a second key if the first was moved before the record was written. An audit rotation’s signing.key is the key the gateway signs with, so it stays. For separate operator/gateway accounts, distribute only the signed policy set and public key; do not grant the gateway access to the operator’s private key. The automatic adapter below targets the same-account systemd user deployment from issue #150. A remote signer, multi-host rollout coordinator, Kubernetes adapter, and browser-triggered activation are not implemented.

Review the summary before activation:

mcp-devtools rotation status --out /absolute/operator/rotation-policy
mcp-devtools rotation activate \
  --out /absolute/operator/rotation-policy \
  --environment-file /absolute/security/service.env \
  --service mcp-devtools.service \
  --health-url http://127.0.0.1:3000/

Activation requires Linux, systemctl --user at /usr/bin/systemctl or /bin/systemctl (never looked up on PATH), one required EnvironmentFile= on the selected loaded unit, and no UnsetEnvironment= overrides. Its affected settings must match the prepared baseline exactly, including absolute paths. This adapter accepts simple, unique KEY=value assignments; quoted, escaped, and multiline environment values are refused. The replacement paths must have no whitespace, quotes or escapes. If changing the unit to select its environment file, run systemctl --user daemon-reload first. Unsupported layouts are refused before stopping the service.

The adapter serializes rotations on the environment file, holds the existing policy publication locks, checks drift, durably records its intent, stops the service, atomically switches the environment file, and starts the service. Only the rotated settings’ values change; every other line, its position and indentation are kept, which is what lets rollback rebuild the original file exactly and check it against the recorded digest. systemctl’s own error text is included in a failure. It checks the new process’s environment through /proc and its direct loopback HTTP health endpoint (default http://127.0.0.1:3000/, no redirects). Use the service’s actual port with --health-url. This checks process startup and health, not client/IdP interoperability or historical audit verification. Connected clients may need to reconnect. Each invocation covers one service; coordinate every deployment sharing a key before retiring that key.

A failed activation leaves recovery_required and a failed rollback leaves rollback_failed; interruption may leave activation_pending or rollback_pending. Correct the cause and resume the same command. Activation refuses rollback_failed, so rerunning it can never re-apply a key whose rollback you started; finish the rollback, or prepare a fresh rotation. There is no automatic rollback, including after a health-check failure. An explicit routine rollback is available only while both signed sets and the environment file remain unchanged:

mcp-devtools rotation rollback \
  --out /absolute/operator/rotation-policy \
  --service mcp-devtools.service \
  --health-url http://127.0.0.1:3000/

Rollback refuses a key marked compromised and never removes either set of keys, artifacts, or journal history. Newer policy/revocation publications prevent rollback so recovery cannot silently discard a denial. Do not edit a pending rotation’s managed files manually; fix deployment problems or prepare a fresh operation against the latest state.

After activation, verify the active generation through the authenticated policy page/API and verify the journal using mcp-devtools audit verify with repeated --public-key arguments for every historical audit key. Keep historical public keys permanently with the evidence. Retire old private keys only after all participating deployments are verified. The local events.jsonl records aid recovery; they are not the gateway’s signed, tamper-evident audit journal.

Identity-provider signing key

Rotate at the identity provider. Gateways pick the new key up on the next token that names its kid, or at the next background refresh (every 10 min). A key that disappears from the JWKS, or is republished under the same kid with different material, evicts the tokens it validated. If the rotation was forced by a compromise, follow it with revoke all (above).

Policy signing key (MCP_POLICY_PUBLIC_KEY)

The gateway accepts exactly one policy signing key, so rotation is a re-sign-and-redeploy:

1. mcp-devtools policy keygen --out policy-signing-new.key — note the printed public key.
2. Sign the current policy and revocation list with the new key, and seal them as a new generation 1 under it: mcp-devtools policy sign policy.yaml --key policy-signing-new.key, mcp-devtools policy sign revocations.yaml --revocation-list --key policy-signing-new.key, then move the old manifest aside and run mcp-devtools generation init --manifest generation.yaml --policy policy.yaml --revocation revocations.yaml --key policy-signing-new.key. (generation seal refuses to advance a manifest the new key did not sign.) A restart accepts whatever signed generation is on disk, so the renumbering is safe across the rollout.
3. Deploy the new MCP_POLICY_PUBLIC_KEY and the new .sig files and manifest in the same rollout. A gateway that restarts with the new key and the old signatures refuses to start (fail closed); one that has the new signatures and the old key ignores the reload and reports degraded until the key follows — neither fails open.
4. Destroy the old private key.

If the old key is suspected compromised, treat any policy or revocation change since the suspected date as untrusted: the journal’s authorization_generation_changed records carry the generation, each document’s bundle hash, and the key_id that signed it.

Audit checkpoint signing key (MCP_AUDIT_SIGNING_KEY, WP B.6)

Generate a new key with mcp-devtools audit keygen, deploy it to the gateways, and keep the old public key: checkpoints already written were signed with it, and mcp-devtools audit verify accepts more than one --public-key so a journal spanning the rotation still verifies end to end.

What the journal shows

Every revocation-list change is an authorization_generation_changed control record carrying the generation, the list’s version label (revocation_version, v<n>+sha256:<16 hex> of the exact file bytes) and the one it replaced, the policy’s likewise, the signing key_id, and the counts; a gateway starts with an authorization_generation_loaded record, and a refused manifest or unsealed document is an authorization_generation_rejected record. Every refused request is a revoked_token_rejected record with the refused principal, the reason (subject, token_id, not_before), the list version, and the generation. Every decision record carries authorization: the generation and both versions it was taken under. mcp-devtools audit export activity (WP B.5) and mcp-devtools audit metrics (WP B.7) read these to report the revocation window per subject.