WP-02 · Release engineering

Fleet operations: one-command provisioning, numbered releases on a pinned runtime, and nightly backups across tenant servers

Abstract. I describe the control plane that provisions and maintains a fleet of single-tenant agent servers, one per small-business owner. A tenant is created by a one-command, two-phase installer that pins the upstream agent runtime to a fixed commit, injects secrets over standard input, and then converges the host to the head of a declarative release manifest. Releases are ordered, numbered and composed of named steps; each step is guarded by a precondition probe so that every step is safe to run twice. The updater computes the minimal step set from a version marker written on the tenant, defers restarts behind a conversational quiescence gate, and promotes the new version only after a liveness check on the messaging channel. Nightly backups are streamed, verified and atomically published under a 14-archive retention window. I report the release history for the first two days of the current design, the concurrency incident that produced the per-tenant lock, and the open problems.

1. System model

The fleet is a set of tenants. Each tenant is one small cloud server held in an account the owner controls, running one instance of an open source agent runtime plus a thin layer of owner-facing services: a forms store, a reverse proxy for private pages, a settings service and a mail intake. Nothing is shared between tenants at runtime. A single operator host holds the control plane: a tenant registry, the release manifest, step bundles, the updater, the backup job and a read-only status probe. The operator console, a page inside the operator's own CRM, is a thin client over the same scripts.

The tenant registry is deliberately small. Each record carries a slug, the owner's name and business, a trade, a role (test or client), a daily-backup flag and a flag that controls whether the owner is texted about updates. Provider context for tenants created by the self-service path lives in the same record. At the time of writing the registry held four tenants.

Two invariants shape everything below. First, the only state the updater trusts about a tenant is what the tenant itself reports: the control plane keeps logs, but the version of record is a marker file on the tenant. Second, every operation that mutates a tenant holds that tenant's lock for its full duration.

2. Provisioning

Provisioning is a single command that takes a tenant slug. It is destructive by design: it rebuilds the server from a clean operating system image, so it is used both to create a tenant and to return one to a known state.

Identity check. Before anything is rebuilt, the installer asks the cloud provider for the server's public address and compares it with the tenant registry. A mismatch stops the run. This is the cheapest available guard against rebuilding the wrong machine.

Rebuild and reachability. The server is rebuilt with a cloud-init seed that creates an operations account. Because every rebuild produces a new host identity, the control plane re-establishes trust with the rebuilt host before continuing. It then polls SSH every 10 seconds, up to 60 times, and blocks until cloud-init reports completion.

Phase A. Operating system packages, a dedicated unprivileged service account, and the agent runtime installed at a fixed upstream commit. Pinning the commit makes a fresh tenant reproducible: two tenants provisioned a day apart run the same runtime until a release explicitly moves them.

Phase B. Secrets are written over the SSH pipe into a root-only file through standard input, never through command-line arguments, which leak into process tables and audit logs. Configuration, memory, skills, the web layer, the personality file and the messaging gateway follow. Per-tenant parameters are passed as environment for that single invocation.

Convergence. The installer then releases its own lock and hands off to the updater, which takes the same lock, finds no version marker, treats the tenant as version zero and replays the entire release manifest in order. A fresh tenant and a long-lived tenant therefore reach the head of the manifest through the same code path. This is the property that makes the manifest the specification of a tenant, rather than a changelog.

Personalization. Last, a personality template is rendered for the tenant, with the tenant's own forwarding address and page host substituted in. It runs after convergence because several release steps patch the stock personality text and expect to find it unmodified.

A second entry point resumes from convergence for a tenant whose phase B already completed, so a failure late in provisioning does not require a full rebuild.

3. The release manifest

A release is a record with four fields: a version, a title, an ordered list of step names, and two pieces of prose: details for the operator and a one-line note written for the owner. Versions are date based with an optional sequence suffix. Comparison extracts every digit group and compares the resulting integer tuples, so a sequence suffix orders correctly against the bare date.

The manifest covers 21 releases published over two days, 18 of them on 2026-10-22. They reference 25 step entries, which resolve to 22 distinct step handlers; four releases carry two steps and one step is shared by two releases. The titles below are the operator titles, lightly generalized.

Three classes of step appear. Install steps ship a bundle (a directory of scripts and assets) to a private staging directory and run its installer. Patch steps make a narrow edit to the personality or configuration, after first copying the file to a step-specific backup. Upgrade steps move a component to a new version, after first taking a snapshot of the agent's state directory. Trade-specific steps are conditional on the tenant's trade in the registry: a real estate step installs its skills only on real estate tenants, while refreshing shared guide content everywhere.

Every step is required to be idempotent, and most enforce it with a precondition probe rather than by convention. A forms step exits early if the forms service is already installed; a model step exits early if the configured chat model already matches; a runtime upgrade step exits early if the reported runtime version is already at or above the target. The rest are written so that a second run produces the same file as the first.

4. Convergence and the updater

Probe. The updater begins by sending a read-only status script to the tenant over standard input. The script installs nothing and prints one JSON object: the version marker, the runtime version, the configured models and whether the chat model is pinned to specific hosts, the routes registered on the gateway, the state of each service and the state of each messaging platform. It never prints secrets.

Diff. The set of pending releases is every release whose version is greater than the marker. Their steps are flattened in release order and deduplicated, so a step that appears in two pending releases runs once, at its first position.

Apply. Each step receives a fresh status object before it runs, so preconditions see the effects of earlier steps in the same run. Bundles are streamed as a compressed archive into a staging directory created with a random name under a private 0700 parent; the path returned by the tenant is validated against an expected pattern before it is used, and the directory is removed in a finally clause whether the run succeeds or not.

Quiesce. A gateway restart terminates in-flight work, including page builds that take minutes. The updater therefore reads the timestamp of the most recent message from the agent's state database, opened read only, and restarts only after 90 seconds of silence. It polls every 15 seconds, at most 40 times, which bounds the wait at 10 minutes. Restarts were moved to the end of the run, once, rather than after each step, for the same reason.

Restart and liveness. After the restart the updater waits up to 120 seconds for the gateway's local port to accept connections, then up to 120 seconds for the status probe to report both the gateway service active and the messaging channel connected. If the channel does not reconnect the run raises, and the version marker is not written. A tenant is never reported as updated while its owner cannot reach it.

Commit, notify, record. Only then is the marker written. If the tenant's update-text flag is set, the owner receives one message that begins "Queva updated to" the new version and joins every pending release's owner note; if that exceeds 480 characters, only the newest note is kept. The operator can rewrite the text from the console before the run, and the server rejects text that does not start with the required prefix. Client tenants currently update silently; only the test tenant texts. Each run appends one line to a per-tenant history log, success or failure, with the from and to versions, the releases applied and the log file name.

A plan mode runs the probe and diff and prints the result without taking the lock or touching the tenant, so the console can show what an update would do while another job is running.

5. Concurrency control

Every job that mutates a tenant takes an exclusive, non-blocking lock on a per-tenant lock file as its first action and holds it until exit. A second job on the same tenant exits immediately with status 75 and a message that the tenant is busy. The lock lives on a file rather than a process ID, so it survives a restart of the console service, and it is released by the kernel when the process dies, so a crashed job cannot wedge a tenant.

The lock was not always this wide. On 2026-10-22 it lived only inside the updater, the last phase of provisioning. A reinstall started by hand therefore held no lock during its first two phases, the console saw the tenant as idle, and a test of the console's wipe action started a second reinstall, which issued a second rebuild while the first was mid-way through phase A. Both runs were killed and a clean third run completed; about 15 minutes were lost and no data, since the tenant was being wiped anyway. Two rules came out of it: the installer takes the lock on its first line, and destructive actions are never tested against a real target, even to observe a refusal.

6. Backups

Each tenant is backed up nightly by streaming a compressed archive over SSH into the control plane. The archive covers the agent's state directory (settings, memory and the personality), the owner's pages, the forms database and its hooks, the reverse proxy and mail configuration, the gateway's service units, the owner-facing helper programs and their state. Downloaded tools, the runtime's own checkout, caches and logs are excluded, which keeps the archive to the state that cannot be regenerated.

Archives are treated as sensitive and are written with owner-only permissions on the control plane.

The job writes to a dot-prefixed partial file, accepts a tar exit status of 0 or 1 (1 means a file changed while being read, which is expected on a live tenant), rejects anything under 10 KB, lists the archive to prove it is readable, and only then renames it into place. Retention keeps the newest 14 archives and deletes the rest; every success appends a line with time, file and size.

Restores are manual. There is no restore button and no automated restore test yet, which is the largest gap in this design and is listed in section 9.

7. Observability

Every run writes a timestamped log and a small state file that the console polls. The state file is written to a temporary name and moved into place, so the console never reads a partial JSON document. It records the process, start and finish times, the from and to versions, the planned steps, the steps completed so far and the result. Together with the per-tenant history log, this lets the console show a version, a "what's in this update" view and a timeline per tenant without querying the tenant.

The status probe doubles as a configuration audit. Because it reports models, host pinning, routes and service states, drift between what the manifest says and what a tenant runs is visible from the console.

8. Failure analysis

Concurrent mutation. Described in section 5. Root cause: a lock scoped to one phase instead of the job.

Personality changes and live sessions. The runtime's gateway keeps the system prompt a conversation was built with. A release that patched the personality file therefore did not change the behavior of a conversation already open on the tenant: the owner's next message was answered under the old rules. Archiving the session was not enough, since the gateway resumed it; deleting the session and restarting while idle was. Releases that change the personality now carry that step, and the owner is told the conversation starts fresh. A gentler mechanism is listed as open.

Shared state during testing. Test builds run in the default profile on a live tenant share its conversation database with the owner. Tests now run in a scratch profile.

Generation one. The first-generation installer, started on 2026-09-28, wrapped the runtime in many services of its own and was validated by an acceptance run on a fresh server: installed twice, exit 0, 59 checks passed, 0 failed, 4 expected warnings. It was retired on 2026-10-21 in favor of the stock runtime plus this release layer, because most owner-facing problems traced to the wrapper rather than the runtime and every change had to be threaded through it.

9. Open problems

1. Restore automation and restore testing. Backups are verified for readability, not for recoverability.
2. Rollback. Steps are idempotent forward; there is no inverse. Recovery from a bad release is a forward fix or a rebuild.
3. Staged rollout. Releases go to the test tenant first and to clients on an operator click. There is no automated canary or health-gated promotion across tenants.
4. Session-preserving personality updates. Section 8.
5. Drift detection. The probe exposes configuration, but nothing yet compares it to an expected state per version.

10. Claim table

11. Sources

• Fleet tooling: installer, resume entry point, updater, status probe and backup job, October 2026.
• Release manifest, versions 2026.10.22 to 2026.10.23.2.
• Operator console design notes, 2026-10-22.
• Incident notes on concurrent mutation and personality propagation, 2026-10-22.
• Project notes on the first-generation installer, September and October 2026.