Secret sources runbook

Published from the project repository

This page is generated from the project’s documentation. References to source files, tests and unpublished documents appear as plain text, and test names such as suite::test_name identify the evidence behind a claim. The source repository is not public yet, so that evidence is available on request.

How to point the gateway’s secret references at each supported provider, and what to set up on the provider’s side (plan §3.8, C.2). The reference syntax and every setting are in configuration.md; this runbook is the provider-side half. Common to every provider:

• References are resolved before the gateway starts serving. Unavailable vendor credentials degrade health and fail affected calls locally; required server settings still refuse startup. Background refresh retains last good values per document and recovers missing credentials when restored. See configuration for the exact scope.
• A tool call never waits on a provider: it reads the snapshot the background refresher fills every MCP_SECRET_REFRESH_INTERVAL_SECONDS (default 30), plus one early refresh on an upstream 401.
• Multi-part credentials (an email and a token, a client id and secret) belong in one document, so they rotate together.
• The audit record and the enterprise result metadata carry source (the scheme) and version (the provider’s version) for every referenced credential; the version changing is the rotation evidence.
• The provider’s own credential (a Vault token, a secret id) can be a literal or a file:// reference — never another provider reference.

Mounted files (file://)

No provider-side setup. A Kubernetes Secret projected as a volume, or a CSI Secrets Store volume backed by Vault, AWS, or Azure (deploy/k8s/secrets-csi.yaml), is re-projected by the kubelet on change; the gateway re-reads it on the refresh interval. This is the path that works on every cloud with no provider-side identity for the gateway; the native adapters below earn their place with what a CSI volume cannot give: non-Kubernetes hosts, a per-read entry in the provider’s own audit log under the gateway’s identity, and no plaintext on a tmpfs.

HashiCorp Vault and OpenBao (vault://)

One adapter serves both; the KV v2 and auth APIs are the same. CI runs the live test against hashicorp/vault:1.21.4 and openbao/openbao:2.6.2.

Setting	Value
Reference	vault://<mount>/<path>#<key> — secret/mcp/grafana#token reads secret/data/mcp/grafana, key token
MCP_VAULT_ADDR	https://vault.example:8200 (http only on loopback)
MCP_VAULT_AUTH	kubernetes on Kubernetes; approle elsewhere; token for development or behind a Vault Agent sidecar
MCP_VAULT_NAMESPACE	Enterprise / HCP only
MCP_VAULT_CACERT	the private CA’s PEM, if any

Container persistence

Vault server -dev stores secrets in memory. Docker volumes and a restart policy do not make dev mode persistent. Use normal server mode with a disk backend, for example single-node integrated storage:

{
  "storage": {"raft": {"path": "/vault/file", "node_id": "vault-1"}},
  "listener": [{"tcp": {"address": "0.0.0.0:8200", "tls_disable": true}}],
  "api_addr": "http://127.0.0.1:8200",
  "cluster_addr": "https://127.0.0.1:8201",
  "disable_mlock": true
}

For a local or SSH-tunneled setup, publish only 127.0.0.1:8200:8200, mount a named volume at /vault/file, mount the nonsecret configuration under /vault/config, and use --restart unless-stopped. The official image loads /vault/config automatically when invoked with server; do not also pass the same file with -config, which duplicates the listener. Use TLS for non-loopback access.

Initialize once, save the unseal shares and initial root token securely, and unseal. Give the MCP server its own read-only token or AppRole rather than the root token. After a restart, data persists but Vault remains sealed until manually unsealed or configured with an external auto-unseal provider. Do not reinitialize or delete the data volume to fix a sealed server.

Before replacing an existing dev container, copy its secrets into the new instance, verify every copied value without logging it, then restart and unseal the new instance and verify again. Keep encrypted Raft snapshots (vault operator raft snapshot save) and a separate secure backup of the unseal material; a volume alone does not protect against disk loss. Preserve the named volume when recreating the container; avoid docker compose down -v.

References: Vault dev mode, seal/unseal, Raft snapshots.

Vault-side setup

1. A KV v2 mount and the secret, one document per credential set:

   vault secrets enable -path=secret -version=2 kv   # dev mode has this already
   vault kv put secret/mcp/grafana token=glsa_…
   vault kv put secret/mcp/atlassian email=svc@acme.example token=ATATT…

   Reference them as GRAFANA_TOKEN=vault://secret/mcp/grafana#token, ATLASSIAN_USER_EMAIL=vault://secret/mcp/atlassian#email, ATLASSIAN_API_TOKEN=vault://secret/mcp/atlassian#token. The two Atlassian halves come from one read and swap together.

2. A read-only policy — the gateway needs read on the data paths and nothing else (renew-self and lookup-self are in the default policy every token carries):

   path "secret/data/mcp/*" { capabilities = ["read"] }

   vault policy write mcp-devtools mcp-devtools.hcl

3. An auth method.

   Kubernetes (MCP_VAULT_AUTH=kubernetes): enable the method, point it at the cluster, and bind a role to the gateway’s service account. The gateway reads the projected token from /var/run/secrets/kubernetes.io/serviceaccount/token at every login.

   vault auth enable kubernetes
   vault write auth/kubernetes/config kubernetes_host="https://$KUBERNETES_SERVICE_HOST:443"
   vault write auth/kubernetes/role/mcp-devtools \
     bound_service_account_names=mcp-devtools \
     bound_service_account_namespaces=mcp-devtools \
     token_policies=mcp-devtools token_ttl=1h token_max_ttl=24h

   Set MCP_VAULT_ROLE=mcp-devtools. deploy/k8s/secrets-vault.yaml is the matching manifest.

   AppRole (MCP_VAULT_AUTH=approle): for VMs and containers outside Kubernetes. Deliver the secret id as a file the gateway reads at login (MCP_VAULT_SECRET_ID=file:///run/secrets/vault-secret-id), so it can be rotated in place.

   vault auth enable approle
   vault write auth/approle/role/mcp-devtools token_policies=mcp-devtools \
     token_ttl=1h token_max_ttl=24h secret_id_ttl=0
   vault read auth/approle/role/mcp-devtools/role-id
   vault write -f auth/approle/role/mcp-devtools/secret-id

   Token (MCP_VAULT_AUTH=token): a dev server’s root token, or the sink file of a Vault Agent sidecar (MCP_VAULT_TOKEN=file:///vault/token, with MCP_VAULT_ADDR=http://127.0.0.1:8100 if the agent proxies).

Import an existing configs.json

scripts/operations/import-configs-vault.py uses Python 3.9+ (standard library only) to import plaintext registered credentials into HashiCorp Vault or OpenBao KV v2. It does not import into Azure Key Vault or perform an Okta login. It reads the process environment, not .env:

export MCP_VAULT_ADDR=http://127.0.0.1:8200
export MCP_VAULT_AUTH=token
export MCP_VAULT_TOKEN=root  # only for the local dev server above

# Preview ~/.mcp/configs.json without network access or writes:
python3 scripts/operations/import-configs-vault.py

# Import and generate ~/.mcp/configs.vault.json:
python3 scripts/operations/import-configs-vault.py --apply

# Import only available sections; names match the input JSON exactly:
python3 scripts/operations/import-configs-vault.py \
  --section jira --section github --prefix mcp/work-test \
  --output /tmp/configs.work-vault.json --apply

Use --config /path/to/config.json for a differently named input. --mount defaults to secret and must already be a KV v2 mount. The import token needs write permission on the destination data paths; the server’s normal read-only token is insufficient. MCP_VAULT_TOKEN=file:///absolute/path (plain token file), MCP_VAULT_NAMESPACE, and MCP_VAULT_CACERT are supported. HTTPS is required except on loopback; HTTP redirects and environment HTTP proxies are not used. AppRole and Kubernetes authentication are not implemented by this import helper; use an import token even if the running server uses another mode.

Each section gets its own document under mcp/import/<section>-<hash>; aliases stay separate. Credential principals such as email and client ID move with their imported secret. URLs, other settings, empty values, keychain sentinels, and existing secret references stay unchanged. Nested NinjaOne credential documents are reported and left unchanged by default. To migrate them, use --include-ninjaone-config: the entire NINJAONE_SERVERS and NINJAONE_DB_ENVIRONMENTS JSON settings (including URLs and other non-secret fields) are stored as JSON strings in Vault. Each local setting becomes a top-level vault:// reference; individual nested fields are not expanded. Existing references are skipped. For an additional import after migrating other vendors, choose a fresh prefix and output filename:

python3 scripts/operations/import-configs-vault.py --include-ninjaone-config \
  --section ninjaone --prefix mcp/import-ninjaone \
  --output ~/.mcp/configs.ninjaone.vault.json --apply

The helper reads only the input file: it does not retrieve keychain values or resolve environment overrides. A file containing only keychain sentinels has nothing to import.

Import using a read-only runtime token

A runtime token with only read capability cannot import secrets; changing the prefix does not fix HTTP 403. Pass --admin-token-file /absolute/path to explicitly use a separate operator token. The helper creates a randomly named temporary ACL policy granting create and update only on the planned KV v2 data paths, issues a non-renewable token with a ten-minute maximum TTL, and uses that token for the import. It then attempts to revoke the token and delete the temporary policy on both success and failure. Cleanup failures are reported; remove a reported leftover policy administratively. A lost response or interrupted process can leave a policy behind; the token’s server-enforced maximum TTL still bounds its lifetime. No token is printed or saved, and the runtime token file is not changed.

For the persistent Vault container on kdb, use the existing SSH tunnel and explicitly pass the locally stored recovery JSON containing root_token:

uv run scripts/operations/import-configs-vault.py \
  --config /tmp/ninjaone-config.json \
  --section ninjaone --include-ninjaone-config \
  --prefix mcp/ninjaone-updated-dev-backup5 \
  --output /tmp/ninjaone-vault-refs.json \
  --admin-recovery-file ~/.config/mcp-devtools/vault-recovery.json --apply

The recovery file is never loaded implicitly; the two admin options are mutually exclusive. Admin mode requires permission to create/delete ACL policies and issue/revoke tokens; a non-root operator must also be authorized to issue the new policy. The script uses the Vault HTTP API, so the CLI and container execution are unnecessary. Preview without --apply neither reads the admin credential nor contacts Vault. Existing output is refused before any administrative changes. The import still uses cas=0: it creates new secrets rather than overwriting existing ones. Review and merge the generated NinjaOne section into the runtime config separately.

API references: ACL policies and token creation/revocation.

Writes use KV v2 check-and-set with cas=0, so existing secrets cannot be overwritten. The output must not already exist and is created with mode 0600. Original configuration is never modified; review the generated file, back up your original, and install the generated file as ~/.mcp/configs.json when ready. It may still contain credentials the helper did not migrate. The server continues to need the MCP_VAULT_* settings and a token with read access; environment and .env overrides still take precedence over the generated references.

An import across several documents is not transactional. On failure, no output config is retained, but earlier Vault writes may remain; the helper reports the created paths. Review those paths and retry with a fresh --prefix. Dev-mode Vault loses its secrets when stopped, so references to it stop working afterwards. Importing credentials does not configure inbound Okta authentication.

Helper validation (no real credentials or running Vault required):

python3 -B -m unittest discover -s scripts/operations -p test_import_configs_vault.py

Export NinjaOne configuration for repeated use

The read-only export helper resolves top-level NinjaOne Vault references from ~/.mcp/configs.json using the existing MCP_VAULT_* token settings (including file:// tokens, namespace and TLS CA). It writes only the NinjaOne section, including resolved credentials, without printing credential values:

python3 scripts/operations/export-ninjaone-vault.py
# Optional alternate input and destination:
python3 scripts/operations/export-ninjaone-vault.py \
  --config ~/.mcp/configs.json --output /tmp/ninjaone-config.json

The default output is ninjaone-config.json in the system temporary directory. Successful runs replace it atomically with an owner-only (0600) file; failed runs preserve the previous export. The input configuration is never modified. Shared Vault documents are read once per run. This helper supports token authentication only and requires Python 3.9+; it does not retrieve keychain values or resolve other secret providers or environment overrides.

python3 -B -m unittest discover -s scripts/operations -p test_export_ninjaone_vault.py

Export all configuration sections

Use the all-sections helper to export the full config, including vendor aliases, with every top-level environments Vault reference resolved:

uv run scripts/operations/export-configs-vault.py
# Optional alternate input and destination:
uv run scripts/operations/export-configs-vault.py \
  --config ~/.mcp/configs.json --output /tmp/configs.export.json

The default output is configs.export.json in the system temporary directory. It contains plaintext credentials and preserves section names, other settings, and NinjaOne server/database JSON objects. The same token authentication, TLS, private atomic replacement, and failure-preservation protections as the NinjaOne-only helper apply. Documents shared across sections are read once per run. This exports only references present in the input config; it does not list all secrets in Vault. Other secret providers and environment overrides are left unchanged. The original config is never modified.

python3 -B -m unittest discover -s scripts/operations -p 'test_export*py'

What the adapter does with the token

It renews a renewable token with renew-self once two thirds of its lease is gone, logs in again when renewal fails or the lease is over, and on a 403 from a read logs in once more and retries once — so a token an operator revokes is replaced on the next refresh, and a policy that stops granting the path shows up as secret_source_unavailable (forbidden) in the operator log, with the reference named and the token never printed.

Rotation

vault kv put a new version. The next refresh (≤ 30 s by default) swaps the values in whole; the audit record’s version becomes the new KV version number. Nothing restarts. For a rollback, vault kv rollback produces a new version too, and is picked up the same way.

Running the live test locally

docker run -d --name vault -p 127.0.0.1:8200:8200 hashicorp/vault:1.21.4 \
  server -dev -dev-root-token-id=root -dev-listen-address=0.0.0.0:8200
MCP_TEST_VAULT_ADDR=http://127.0.0.1:8200 cargo test --test vault_live_tests -- --nocapture

(openbao/openbao:2.6.2 with the same arguments for OpenBao.) The test creates its own policy, AppRole, and secret under secret/mcp-live-test/. scripts/ci/image-vault-smoke.sh <image> runs the published container image against the same server.

Troubleshooting by category

Vendor credential warnings include both failure (the health category) and cause (the sanitized provider detail) at startup and on background refresh. For a packaged systemd service, inspect them with sudo journalctl -u mcp-devtools. A Vault HTTP 503 appears as cause=secret_source_unavailable: vault://<mount>/<path> (sealed). Check Vault’s /v1/sys/health endpoint and unseal it if sealed is true; a standby without forwarding can also return 503. After recovery, the background refresher retries automatically. Tokens and response bodies are never included in this diagnostic.

Category (operator log)	Usual cause
secret_source_unconfigured	A vault:// reference with no MCP_VAULT_ADDR.
cannot configure the secret source vault://: … at startup	MCP_VAULT_AUTH missing or unknown, a method’s credential missing, a plain-http address off loopback, an unreadable CA file.
secret_source_unavailable (login refused)	Wrong role id / secret id, a Kubernetes role not bound to this service account or namespace, or the auth method mounted elsewhere (MCP_VAULT_AUTH_MOUNT).
secret_source_unavailable (forbidden)	The token’s policy does not grant read on <mount>/data/<path>, even after a fresh login.
secret_source_unavailable (sealed)	Vault answers 503: sealed, or a standby without request forwarding. Last good values stay in force.
secret_source_not_found	No secret at the path, or the current version is deleted or destroyed.
secret_document_malformed	The #key is not in the secret, or is not a string.

AWS Secrets Manager (awssm://)

Hand-rolled SigV4 over the shared HTTP client; no AWS SDK in the binary. CI’s cloud-secrets-live workflow (manually triggered) reads a real secret with a GitHub OIDC token exchanged at STS — the web-identity path.

Setting	Value
Reference	awssm://<name-or-arn>[#<key>] — prod/mcp/grafana#token reads the AWSCURRENT version of the secret prod/mcp/grafana and takes token from its JSON
MCP_AWS_AUTH	web-identity on EKS with IRSA (and anywhere an OIDC token is projected); pod-identity on EKS with Pod Identity, or ECS; static for development and CI sessions
MCP_AWS_REGION	the secret’s region (AWS_REGION, which IRSA injects, is accepted)
MCP_AWS_SECRETS_MANAGER_ENDPOINT	only for a VPC or FIPS endpoint

AWS-side setup

1. The secret, one JSON document per credential set so its halves rotate together:

   aws secretsmanager create-secret --name prod/mcp/grafana \
     --secret-string '{"token":"glsa_…"}'
   aws secretsmanager create-secret --name prod/mcp/atlassian \
     --secret-string '{"email":"svc@acme.example","token":"ATATT…"}'

   Reference them as GRAFANA_TOKEN=awssm://prod/mcp/grafana#token, ATLASSIAN_USER_EMAIL=awssm://prod/mcp/atlassian#email, ATLASSIAN_API_TOKEN=awssm://prod/mcp/atlassian#token. A secret that is a plain string (no JSON) is referenced without #key. Binary secrets are refused.

2. A read-only policy — GetSecretValue on the secrets and, when they are encrypted with a customer-managed key, kms:Decrypt on that key:

   {
     "Version": "2012-10-17",
     "Statement": [{
       "Effect": "Allow",
       "Action": "secretsmanager:GetSecretValue",
       "Resource": "arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/mcp/*"
     }]
   }

3. An identity.

   IRSA (MCP_AWS_AUTH=web-identity): a role whose trust policy names the cluster’s OIDC provider and the gateway’s service account; the EKS webhook then injects AWS_ROLE_ARN, AWS_WEB_IDENTITY_TOKEN_FILE, and AWS_REGION into the pod, which the adapter accepts as they are.

   eksctl create iamserviceaccount --cluster prod --namespace mcp-devtools \
     --name mcp-devtools --attach-policy-arn arn:aws:iam::123456789012:policy/mcp-devtools-secrets \
     --approve

   deploy/k8s/secrets-aws.yaml is the matching manifest. The projected token is re-read at every exchange, so its rotation needs nothing from the gateway.

   Pod Identity (MCP_AWS_AUTH=pod-identity): install the EKS Pod Identity agent add-on and create the association; the defaults for MCP_AWS_CREDENTIALS_URI and MCP_AWS_CREDENTIALS_TOKEN_FILE are the agent’s. On ECS and Fargate nothing more is needed: the platform injects AWS_CONTAINER_CREDENTIALS_RELATIVE_URI, a path on the task endpoint 169.254.170.2, and the adapter uses it as it is, without an Authorization token (the task’s network namespace is its identity). deploy/aws/fargate.yaml sets only MCP_AWS_AUTH and the region.

   Static keys (MCP_AWS_AUTH=static): an access key pair — and the session token if it is a temporary one — in MCP_AWS_ACCESS_KEY_ID, MCP_AWS_SECRET_ACCESS_KEY (a file:// reference works), and MCP_AWS_SESSION_TOKEN. A long-lived key in a Kubernetes Secret moves the bootstrap problem rather than removing it; use it for development and for CI, not for a pod. EC2 instance profiles (IMDS) are not a mode in this revision.

   The adapter never falls back from one mode to another: a missing input for the configured mode refuses startup naming it.

What the adapter does with the session

Session credentials are replaced once less than five minutes of their life remain, and once more — with one retry — when a read answers ExpiredTokenException. Only AWSCURRENT is ever read: a rotation function stages the new value as AWSPENDING first, and the gateway must not act with a credential the upstream has not accepted yet.

Rotation

aws secretsmanager put-secret-value (or a rotation function finishing) moves AWSCURRENT to a new VersionId. The next refresh (≤ 30 s by default) swaps the values in whole; the audit record’s version becomes the new VersionId. Nothing restarts.

Running the live test locally

eval "$(aws configure export-credentials --format env)"
MCP_AWS_AUTH=static MCP_AWS_REGION=us-east-1 \
  MCP_TEST_AWS_SECRET_ID=prod/mcp/grafana MCP_TEST_AWS_SECRET_KEY=token \
  cargo test --test aws_live_tests -- --nocapture

Troubleshooting by category

Category (operator log)	Usual cause
secret_source_unconfigured	An awssm:// reference with no MCP_AWS_AUTH.
cannot configure the secret source awssm://: … at startup	MCP_AWS_AUTH missing or unknown, a mode’s input missing, no region, a plain-http endpoint off loopback.
secret_source_unavailable (login refused)	STS refused the web identity token (the role’s trust policy does not name this provider, audience, or subject), or the Pod Identity agent refused the authorization token.
secret_source_unavailable (credentials rejected)	Static keys the service does not know, or a clock skewed by more than fifteen minutes (SigV4 signs the time).
secret_source_unavailable (forbidden)	The policy does not grant GetSecretValue on this secret — or, with a customer-managed KMS key, kms:Decrypt (which surfaces as AccessDenied, not decryption failed).
secret_source_unavailable (decryption failed)	The KMS key is disabled or pending deletion.
secret_source_unavailable (invalid request)	The secret is scheduled for deletion, or has no AWSCURRENT version.
secret_source_unavailable (throttled)	The account’s GetSecretValue rate; the last good values stay in force.
secret_source_not_found	No secret by that name or ARN in this region.
secret_source_malformed (binary secret)	The secret is SecretBinary; store a string.
secret_document_malformed	The #key is not in the JSON, or is not a string.

Azure Key Vault (azkv://)

Two REST calls over the shared HTTP client (a token, then the secret); no Azure SDK in the binary. CI’s cloud-secrets-live workflow (manually triggered) reads a real secret with a GitHub OIDC token presented to Entra as a client assertion — the workload-identity path.

Setting	Value
Reference	azkv://<vault>/<secret-name>[#<key>] — corp-kv/mcp-grafana#token reads the current version of mcp-grafana in https://corp-kv.vault.azure.net and takes token from its JSON
MCP_AZURE_AUTH	workload-identity on AKS (recommended); managed-identity on a VM or node pool; service-principal for development or a host that cannot federate
MCP_AZURE_KEYVAULT_DNS_SUFFIX	only for a sovereign cloud (vault.azure.cn, vault.usgovcloudapi.net), with MCP_AZURE_AUTHORITY_HOST to match

Key Vault secret names allow letters, digits, and hyphens only, so a reference names the secret explicitly: ATLASSIAN_API_TOKEN cannot be a secret name, mcp-atlassian can.

Azure-side setup

1. A vault with RBAC authorization, and the secrets, one JSON document per credential set:

   az keyvault create --name corp-kv --resource-group mcp --enable-rbac-authorization true
   az keyvault secret set --vault-name corp-kv --name mcp-grafana \
     --value '{"token":"glsa_…"}'
   az keyvault secret set --vault-name corp-kv --name mcp-atlassian \
     --value '{"email":"svc@acme.example","token":"ATATT…"}'

   Reference them as GRAFANA_TOKEN=azkv://corp-kv/mcp-grafana#token, ATLASSIAN_USER_EMAIL=azkv://corp-kv/mcp-atlassian#email, ATLASSIAN_API_TOKEN=azkv://corp-kv/mcp-atlassian#token.

2. The role: Key Vault Secrets User at vault scope, for the identity below. Role assignments propagate over minutes; a secret_source_unavailable (forbidden) in the first minutes after assignment is expected, not a bug, and the gateway refuses to start until it clears (or, once running, keeps the last good values).

   az role assignment create --role "Key Vault Secrets User" \
     --assignee-object-id "$IDENTITY_OBJECT_ID" --assignee-principal-type ServicePrincipal \
     --scope "$(az keyvault show --name corp-kv --query id -o tsv)"

3. An identity.

   Workload identity (MCP_AZURE_AUTH=workload-identity): a user-assigned managed identity (or an app registration) with a federated credential naming the cluster’s OIDC issuer and the gateway’s service account; the AKS webhook then injects AZURE_TENANT_ID, AZURE_CLIENT_ID, AZURE_FEDERATED_TOKEN_FILE, and AZURE_AUTHORITY_HOST into the pod, which the adapter accepts as they are. No secret exists anywhere.

   az identity create --name mcp-devtools --resource-group mcp
   az identity federated-credential create --name aks --identity-name mcp-devtools \
     --resource-group mcp --issuer "$AKS_OIDC_ISSUER" \
     --subject system:serviceaccount:mcp-devtools:mcp-devtools \
     --audiences api://AzureADTokenExchange

   deploy/k8s/secrets-azure.yaml is the matching manifest. The projected token is re-read at every exchange.

   Managed identity (MCP_AZURE_AUTH=managed-identity): a VM, VMSS, or AKS node pool with a system-assigned identity, or a user-assigned one selected by MCP_AZURE_CLIENT_ID. Every pod on a node shares the node’s identity, which is why workload identity is preferred on AKS. On Container Apps and App Service, which expose no IMDS, the platform injects IDENTITY_ENDPOINT (a loopback URL) and IDENTITY_HEADER (a per-replica value the endpoint requires as X-IDENTITY-HEADER); the adapter accepts both as they are and uses that endpoint instead of IMDS. The identity is the replica’s own, so the sharing concern above does not apply. deploy/azure/main.bicep sets MCP_AZURE_AUTH and the user-assigned identity’s client id and nothing else.

   Service principal (MCP_AZURE_AUTH=service-principal): tenant, client id, and MCP_AZURE_CLIENT_SECRET (a file:// reference works). The client secret is itself a secret in a Kubernetes Secret: this moves the bootstrap problem rather than removing it.

   The adapter never falls back from one mode to another — it is not DefaultAzureCredential — and a missing input for the configured mode refuses startup naming it.

What the adapter does with the token

An access token is replaced once less than five minutes of its life remain, and once more — with one retry — when a read answers 401. A disabled secret (enabled: false) is reported as secret_source_unavailable (disabled), so an operator can distinguish a switched-off credential from a missing role.

Rotation

az keyvault secret set creates a new version, which becomes the current one. The next refresh (≤ 30 s by default) swaps the values in whole; the audit record’s version becomes the new version id. Nothing restarts.

Running the live test locally

MCP_AZURE_AUTH=service-principal MCP_AZURE_TENANT_ID=… MCP_AZURE_CLIENT_ID=… \
  MCP_AZURE_CLIENT_SECRET=file:///run/user/1000/sp-secret \
  MCP_TEST_AZURE_SECRET=corp-kv/mcp-grafana MCP_TEST_AZURE_SECRET_KEY=token \
  cargo test --test azure_live_tests -- --nocapture

Troubleshooting by category

Category (operator log)	Usual cause
secret_source_unconfigured	An azkv:// reference with no MCP_AZURE_AUTH.
cannot configure the secret source azkv://: … at startup	MCP_AZURE_AUTH missing or unknown, a mode’s input missing, a plain-http authority off loopback.
secret_source_unavailable (login refused)	Entra refused the client assertion (the federated credential’s issuer, subject, or audience does not match) or the client secret; IMDS has no identity for this VM or the client_id is not assigned to it.
secret_source_unavailable (forbidden)	No Key Vault Secrets User assignment yet (propagation takes minutes), or the vault uses access policies rather than RBAC.
secret_source_unavailable (disabled)	The secret’s enabled attribute is off.
secret_source_unavailable (unauthorized)	The token was refused twice in a row: the identity’s tokens are being rejected by the vault (a tenant mismatch, or the vault’s firewall).
secret_source_unavailable (throttled)	The vault’s request limits; the last good values stay in force.
secret_source_not_found	No secret by that name in the vault, or it is soft-deleted.
secret_source_malformed	The reference is not <vault>/<secret-name>, or the name has a character Key Vault does not allow.
secret_document_malformed	The #key is not in the JSON, or is not a string.