Every service announces what it is, which build produced it and that it is still alive. Every host announces how full it is. Both land as retained messages on your own broker, so an operator surface reads current state from one subscription instead of interrogating each box in turn. The same substrate that carries the cost ledger carries the fleet.
The failure this exists to prevent is mundane and expensive: a process is listening, its health check returns 200, and it is running code from three deploys ago. A liveness ping cannot detect that. So identity, version and liveness are three separate facets on three separate topics, with one declared owner each.
Every service subclasses it. Publishes retained identity at startup, then a heartbeat on a 60s interval.
daemon/src/service_base.py
Holds the retained message. A subscriber joining late gets current state immediately, with no request to the service.
Operator surface, tray, or a peer daemon. All read the same message; none of them poll the service.
daemon/docs/standards/SERVICE_IDENTITY.md
| Topic | Retained | What it settles |
|---|---|---|
| kv/service/{name}/identity | yes | Version, role, environment, commit SHA. Owned by the service itself, always. |
| kv/service/{name}/heartbeat | no | Version-liveness. Not retained, because a heartbeat that outlives its process is a lie. |
| kv/service/{name} | yes | Process liveness. Owned by whoever owns the OS process and port — the launcher, or the service when it runs standalone. Never both. |
An MQTT Last-Will clears the retained identity if the process dies ungracefully. Without it, a crashed service keeps advertising itself as present for as long as the broker holds the message — the single most common way a bus-based fleet view goes quietly wrong.
Field set produced by ServiceBase.identity(). Values elided — this is the schema, not a reading.
"service": "…", "version": "…", // {ISO_TIMESTAMP}Z-{git short sha} "role": "…", // validated role tier "status": "…", "environment": "…", // dev|test|prod "commit_sha": "…", "timestamp": 0
daemon/src/build_version.py is the single source: a deploy-time environment variable if set, otherwise the committer date and short SHA. Every build of a given commit reports the same string, so two services claiming the same version are running the same code — which is what makes drift detectable at all.
Version drift is detected against the published identities (daemon/src/version_verifier.py) and a mismatch is published as its own non-retained alert. Automatic remediation on drift exists in the codebase and we do not ask you to rely on it — see the limitations below.
Service identity and liveness answer "is it running". Neither carries a single utilization field, so neither can answer the earlier question. Host metrics ride the machine heartbeat that already exists — no new topic, no new service, and deliberately no collector loop.
daemon/src/host_metrics.py
CPU percent is measured over a real window on the first sample and interval-free afterwards. The naive implementation reports a pinned box as idle on its opening heartbeat, because there is no prior call to measure against.
Retained on kv/{machine}/heartbeat, published every 60s
| Field group | Contents |
|---|---|
| identity | machine, ip, role, tier, environment, version, commit_sha, branch, dirty |
| status | online or degraded, decided by whether the expected core service is answering — not by a count of what happens to be up. |
| ports | The full service slot table, each entry explicitly up or silent, plus ports_engaged / ports_total. |
| host | cpu_pct, cpu_count, mem_pct, mem_total_gb, mem_available_gb, swap_pct, disk_pct, disk_total_gb, disk_free_gb, host_uptime_s — and load_1m on POSIX only, because the Windows value misleads more than it informs. |
Swap is reported separately from RAM on purpose. A box with free memory and exhausted swap is a real condition that reads as healthy on RAM alone.
daemon/src/subsystem_watch.py
One loop shape, reused per subsystem. Metrics are ingested, evaluated against registered threshold rules, and a firing is published to kv/alerts/{service} and to the in-process event bus. Two transports on purpose: a failure in one never blocks the other, and the second is what makes an alert visible on every page rather than only on the alerts page.
The process sweep polls, because the operating system does not announce a new process. The work queue does not poll — it already emits a depth snapshot on every enqueue, claim, reap and retry, so its watcher subscribes to that instead. Same primitive, push or pull depending on whether the subsystem announces itself. A subsystem that already reports its own changes does not get a second poller duplicating the work.
daemon/src/process_sweep.py · retained on kv/ops/process/sweep
Named services that registered themselves are already tracked. This is the other set: a subprocess a test's own service manager spawned and never tore down, which is exactly how a runaway process went undetected here. Rows carry pid, name, memory_mb, cpu_percent, age_s, cmdline, sorted by memory descending, each flagged with whether the registry knew about it.
A PID is killable only if the operating system's own view of its command line references this install and its process name is one the platform actually spawns. Anything else is refused, including the daemon's own process. The command path is an operator intent validated against a config vocabulary, and the ledger entry is written before the kill runs — so a refusal is recorded as durably as a success. This is a cleanup tool for processes we spawned, never a kill-any-PID endpoint.
Production serves a cut, checksummed, versioned build object. The tier pointer that selects it is a tracked copy rather than a symlink, because a link cannot travel through git and the fleet receives a release by pulling. Repointing rewrites that copy: no environment change, no restart of anything but the daemon — and it is a commit, so the change to what production serves is itself in history.
Cuts a release from a known commit. Nothing builds inside the serving path.
Versioned directory plus a build manifest carrying the timestamp and the commit it was built from.
prod/releases/{version}
A tracked copy per tier — prod, test, and a held backup you can swap in.
prod/pointers/<tier>
Explicit override, then tier pointer, then newest release, then the dev build. The precedence is fixed and readable.
daemon/src/mona_serve.py
Reads the served bundle's version, compares it to the running build, and reports the gap on the health surface.
A backup pointer kept alongside the live tiers. Rolling back is swapping it in, not rebuilding a prior commit.
prod/pointers/backup
The staleness check exists because of a specific recurring failure: the daemon serves a frozen bundle, nothing repoints it when git moves, and merged interface changes stay invisible until a human cuts a release — with no signal at all that the served build lagged the running one. That is now a warning in the log and a field on the health surface. It is detection; it never triggers a build.
Service health, host utilization, alerting, process control, releases and the cost ledger are one substrate with one operator surface (Marco), not six products correlated by hand at incident time. That is worth something when your constraint is engineering headcount. It is worth nothing against the constraints below, and we would rather you learn them here.
One further honesty: not every service in the platform has adopted the identity contract yet. A conformance test grades them against the reference implementation, and the gap is visible rather than papered over. Adoption is per service, and the ones that have not adopted it report less.