Durable Tasks deployment and recovery

Published from the project repository

This page is generated from the project’s documentation. References to source files, tests and unpublished documents appear as plain text, and test names such as suite::test_name identify the evidence behind a claim. The source repository is not public yet, so that evidence is available on request.

Approval-held tool calls return durable approval-status tasks only when the current request explicitly negotiates Tasks. Other clients retain the existing approval response and re-issue workflow. Admitted calls to a fixed set of long-running tools also run in the background under the same negotiation; see background execution. There is no task-creation tool, administrative approval through tasks/update, or automatic execution on approval.

The protocol and exact dependency pins are those in the protocol audit. Modern discovery advertises Tasks when the store opened successfully. Every task operation and creation helper requires protocol 2026-07-28 or later. Modern requests carry explicit metadata, client information and the Tasks capability; rmcp validates lifecycle capabilities before dispatch. The creation helper checks capability before committing any state. Legacy lifecycle requests return method-not-found. Legacy initialize remains unchanged. EMA and authenticated HTTP client-credentials discovery retain their behavior.

Supported deployment

Use one process and one private persistent local directory, provisioned before startup (Unix example: install -d -m 700 /var/lib/mcp-devtools/tasks). Set MCP_TASKS_ENABLED=true and MCP_TASKS_DIRECTORY to its absolute path. These and the bounded tuning settings in configuration are startup-only and read-only in the console. The directory is dedicated to this store: tasks.db, its SQLite sidecars, and tasks.lock. Do not put usage rollups, approval journals, or unrelated databases there.

Direct-call CLI runtimes do not open, lock, or recover the task store, even when Tasks is enabled in their shared configuration. An OS lifetime lock rejects a second serving process. Never delete the lock file to defeat it. SQLite writes remain serialized with zero busy waiting. External database writers, live schema edits, NFS/SMB, multiple replicas, failover based on routing affinity, and storage without reliable sync/locking are unsupported. Unix checks reject group/other permissions, symbolic database paths and multiply linked database files. On Windows provision equivalent private ACLs; Unix mode checks do not establish Windows ACL isolation. Use trusted directory ancestors and keep the directory owned by the service account.

All SQLite calls, startup validation and recovery run on a dedicated thread. The admission queue defaults to 128 commands and refuses overflow immediately. Admitted commands await the actual worker result; queue delay and slow disk operations do not disable the store. Caller cancellation does not stop a queued commit or poison the worker. An uncertain commit never returns a successful handle. A disk operation stuck in the OS can still block the worker and admitted callers; new commands are refused when the bounded queue fills, and an operator may need to terminate the process. The synchronous bootstrap waits for the worker before serving requests.

The database uses WAL and synchronous=FULL, verified on every open. Creation and subsequent transitions acknowledge only committed changes. Schema v2 adds an optional proposal reference and unique owner/proposal index through a transactional v1 migration. Generic v1 records retain their identity and recovery behavior. Unknown/future versions and nonempty unversioned databases are refused. Pristine empty files left by interrupted provisioning resume initialization while holding the ownership lock. Integrity checks and payload decoding run before discovery; a corrupt store is never silently replaced. Uncertain I/O and corruption errors disable the service rather than retrying a potentially committed write. SQLite capacity errors return capacity; reads and later cleanup remain available. External database writers remain unsupported.

Background execution

These tools run detached when the current request negotiates Tasks: splunk_search, wrds_query, ninjaone_db_query_named, artifactory_download, circleci_download, jira_download_attachment, figma_export_images, bb_clone (stdio only, as always), github_wait_workflow_run, gitlab_wait_pipeline, circleci_wait_workflow and teamcity_wait_build. Each is slow enough that holding the request open is the problem, and none is a non-idempotent write whose effect a restart would leave unknown. The explain option on ninjaone_db_query_named is an argument to the same read-only tool, so it runs detached under the same rules; the single-SELECT guard and the statement timeout apply inside the task (background_task_tests::ninjaone_explain_runs_as_a_background_task_and_keeps_the_sql_guard). The _central and _division query tools are not on the list and stay synchronous. Every other tool, and every call without current-request negotiation, stays synchronous.

The call passes the full admission path first: enablement, inbound authentication, policy, approvals, credential selection and the audit intent. A refusal or approval hold is returned exactly as it is for a synchronous call. Only an admitted call commits a generic task, and the tool does not start until that commit succeeds. If the task store is busy, full or unavailable, or all MCP_TASKS_MAX_RUNNING slots are in use, the admitted call runs synchronously and returns its ordinary result. The background run keeps the request’s call scope, so artifacts, cache entries and egress decisions belong to the same principal, and the audit outcome is recorded when the tool finishes.

A completed task carries the tool’s result. A result larger than the 16 KiB record limit is written to the owner-bound artifact store, the same store and retention (STREAMING_ARTIFACT_RETENTION_SECONDS) as downloads, and the task result points to it for artifact_read. If that write fails, the task fails instead of keeping a truncated result. JSON-RPC faults become failed. Artifacts belong to the serving process, so a spilled result is not readable after a restart even while its task receipt is retained.

Owner binding uses the existing issuer/tenant/subject identity (or the local trust boundary). A task identifier is not authorization. Unknown, expired and inaccessible identifiers share an error response. Every HTTP request passes through existing bearer authentication, scope and revocation middleware. Stdio and unauthenticated loopback HTTP use the existing local owner; access to that process and its private directory is the local security boundary.

Creation references are application-generated operation identities. A unique owner/reference association makes duplicate creation return the same retained handle, including when a commit succeeded but its reply was lost. JSON-RPC request IDs are not durable idempotency keys. No credential, token, original tool argument or client update body belongs in a task record. Generic references and owner bindings are hashed at the adapter boundary. A background task’s result is the tool’s own output, so it holds vendor data at rest in the private task directory under the same owner binding and TTL. Approval tasks additionally retain the opaque proposal identifier so the verified approval journal can recover their status. Completion results are bounded to 16 KiB. Approval results contain only that identifier and a fixed status instruction, never reviewer free text, vendor payloads, or credentials.

The initial TTL defaults to one day. Approval-task retention is extended at creation to at least one hour past proposal expiry, even with a short configured task TTL. This keeps proposal expiry observable independently of task expiry. Terminal results are retained until at least one hour after completion, cancellation acknowledgement or restart failure; a later original expiry is preserved. ttlMs always reports expiry relative to creation. Expired tasks are inaccessible immediately and removed in batches of 128 during creation and once per second while the worker runs. There is no early eviction of unexpired results. At the global limit of unexpired rows, creation refuses new work; expired rows awaiting bounded cleanup do not consume admission capacity. Lowering the limit preserves existing records and permits startup, reads and transitions; new creation waits until the live count falls below the limit. This is a finite retention bound, not a promise to retain data forever or a guarantee of admission at capacity.

Successful task polls enforce the advertised interval per task. The interval map contains only authorized retained task IDs and is bounded by the row limit; unknown IDs cannot populate it. The worker queue also bounds updates/cancels. There are no outstanding client-input requests in this phase. Updates check ownership and acknowledge without retaining input or changing approval state.

For generic tasks, cancellation commits an intent and then stops the running call; the task becomes cancelled once the call has stopped and its audit outcome is recorded. A result that completes first is kept. For approval tasks, cancellation checks the exact issuer/tenant/subject owner and commits a dedicated call_cancellation journal event before projecting cancelled. Cancellation releases the hold so an identical re-issue can request fresh approval. Administrator rejection still refuses identical calls until the proposal deadline. Free-text rejection reasons, including task_cancelled, never mean cancellation; older rejection records retain their rejection semantics. Approval, consumption, rejection or expiry that wins the decision race is preserved. Updates acknowledge unknown input keys and never approve.

Waiting for an administrator is working. An unspent approval first observed after its execution deadline completes with approval expired, matching the admin view. Approval observed before that deadline, or already consumed/applied, completes with a historical receipt: the proposal ID, approval recorded, and approvalExpiresAtMs. The text states the execution deadline and that a new approval is required afterward. A retained receipt does not claim the approval is still consumable. Rejection and expiry complete with isError: true; JSON-RPC faults use failed. Every re-issue independently passes authentication, revocation, policy, credential selection, exact-call binding and single consumption.

Ordinary status reads take only the short projection lock. They do not queue behind unrelated admission or decision appends. Observing pending expiry and cancelling still serialize journal mutations. Creation verifies proposal ownership once; cancellation uses the status returned by its journal operation.

Recovery and backup

After process termination, SQLite recovers committed WAL transactions. Generic retained nonterminal tasks, including interrupted background calls, become failed with a JSON-RPC interruption error; work is never replayed. Their audit intent has no outcome, which the journal already defines as “authorized; dispatch not evidenced”. Pending proposal-linked tasks reconcile lazily against the verified approval journal before being completed. Missing or uncertain journal evidence leaves those pending tasks unavailable. Pending expiry is journaled before projection; approved execution deadlines derive from the journaled proposal and approval.

Terminal task results are immutable historical receipts retained under the existing owner checks and TTL. Reading, updating or cancelling a terminal task does not consult the approval gate, so receipts remain readable after approvals are disabled or their journal entry is no longer reconstructed. This deliberately trusts the private task database for retained history; it is not independently verified audit evidence and never authorizes execution. A stale or altered task backup can misstate historical status, but cannot bypass fresh call admission. Restore both stores together when audit consistency matters.

The stores have no shared transaction. A proposal commits first, followed by its owner/task association. Unsupported custom gates and task capacity, availability or projection failures return the existing approval_required tool response, including the durable proposal ID, instead of losing the approval workflow. An ownership/evidence NotFound remains non-disclosing; holds are independently partitioned by issuer as well as the exact-call key. A failed association exposes no handle; retry reuses the proposal and any committed association. A decision commits before its projection; restart and the next authorized pending-task access reconcile a partial commit. Uncertain I/O or corruption still disables the task service until restart; SQLite FULL returns capacity and preserves later reads and cleanup. Clients should use tasks/get when they only want status.

For the supported offline backup procedure:

1. Stop the service and verify every process using this store has exited.
2. Preserve an evidence copy of the whole directory, including any WAL sidecar. A clean shutdown normally checkpoints it; if WAL remains, retain it together with the database. Never copy only the live database or delete a live WAL.
3. Copy the stopped directory to the backup destination with private permissions. Back up the authoritative approval journal separately at the same maintenance boundary. Keep all journal segments needed to recover retained proposals.
4. Restore the complete stopped snapshot into a private directory with no active writer. Start exactly one process. Startup verifies integrity, schema and effective settings before advertising the lifecycle.
5. Verify known retained IDs as their owners. A stale backup can lose handles created after the snapshot; this deployment has no HA or zero-data-loss restore guarantee. Keep damaged originals for investigation.

No online-backup API is exposed and no rusqlite feature or version was changed. If integrity/schema validation fails, stop the service and restore a known-good snapshot. Do not create an empty replacement database under existing client handles. A crash during first provisioning can leave an empty, unversioned file; only that pristine state is initialized automatically. Startup diagnostics distinguish invalid configuration, unsafe paths/permissions, ownership locks, database validation and recovery failures. Actual disk exhaustion during recovery still requires freeing space before startup can safely complete.

Capacity and evidence

The database has a 1,048,576-page ceiling (4 GiB with its required 4 KiB pages), allowing 100,000 maximum-sized 16 KiB results plus row/index headroom, with 256-page automatic checkpoints, a 1 MiB retained WAL target, and periodic truncate checkpoints. These are SQLite bounds/targets, not quotas on every filesystem byte. Results, queue and row limits bound application state. Disk usage includes SQLite indexes, WAL and shared-memory files; reserve headroom.

task_lifecycle_tests::measured_expected_and_tenfold_workloads records synthetic 1,000- and 10,000-task create/poll workloads, p95/p99 latency and peak WAL size. These are representative test workloads, not a production capacity commitment. The implementation completion record records measured results and separate unavailable evidence. Process kills, SQLite page-limit exhaustion and OS write errors do not establish durability through real power loss, dishonest device caches or every storage platform.