Health Endpoint

GET /api/health

Returns API status. No authentication required.

Request

curl https://{api-url}/api/health

Response

{
    "status": "ok",
    "environment": "dev",
    "version": "0.1.0"
}

Deep check (?deep=true)

The bare probe deliberately touches no database — it stays cheap and answers even on a cold start. Pass ?deep=true to add a database round-trip:

curl "https://{api-url}/api/health?deep=true"

{
    "status": "ok",
    "environment": "dev",
    "version": "0.1.0",
    "database": "healthy",
    "tables": 24
}

tables is the count of tables in the public schema — a quick confirmation that migrations have applied.

Status Values

status	database	Meaning
ok	(absent)	API is operational (bare probe — no database check ran)
ok	healthy	API and database are operational (?deep=true)
degraded	unhealthy: ...	API is up but the database is unreachable (?deep=true only)

Use Cases

• Monitoring: Point an uptime checker at /api/health
• Router health check: Use as the health check path
• CLI connectivity test: mecca configure test -e dev calls this endpoint
• Debugging: Add ?deep=true to check whether database connectivity is the issue

GET /api/health/summary

Comprehensive platform status used by appmecca health, the Aegis monitor, and the tenant CLI's mecca status. Response-shaped by caller class: platform operators receive the full summary below; any other authenticated caller receives a group-scoped subset; anonymous callers get 401, and a disabled account gets 403.

No environment parameter

The API process is bound to a single environment and resolves it server-side. An environment query parameter is accepted-and-ignored by older clients; it never selects a different environment.

curl -H "Authorization: Bearer am_k1_…" \
  "https://{api-url}/api/health/summary"

Operator shape

{
  "api": { "status": "ok" },
  "database": { "status": "ok", "actual": 12, "ceiling": 100, "pct": 12 },
  "cache": { "status": "ok" },
  "agents": {
    "counts": { "alive": 3, "stale": 0, "dead": 0 },
    "version_consistent": true,
    "versions": ["4.7.3"],
    "agents": [ /* per-host details */ ]
  },
  "applications": {
    "app_count": 12,
    "services": { "healthy": 28, "degraded": 0, "down": 0, "unknown": 0, "total": 28 },
    "uptime_24h": 99.8,
    "uptime_checks_24h": 41280,
    "active_incidents": [],
    "oom_events_24h": 0,
    "read_errors": []
  },
  "pipeline": {
    "pending": 0,
    "in_progress": 1,
    "stuck": 0,
    "active_actions": [ /* … */ ],
    "recent_actions": [ /* … */ ],
    "recent_actions_truncated": false,
    "recent_actions_window_start": "2026-08-29T09:00:00+00:00",
    "read_failed": false
  },
  "routing": { /* per-service route + target-group state */ },
  "configuration": { /* declared-vs-live drift block */ },
  "admission_gates": { /* per-gate pressure when ≥ 70% or refusals in last 5 min */ },
  "admission_gates_read_failed": false,
  "custom_domains": { /* stalled / failed / attachment-drift counts */ },
  "warnings": [ /* criticals sorted first */ ],
  "scope": "admin",
  "timestamp": "2026-07-30T09:00:00Z"
}

active_incidents and oom_events_24h are TRI-STATE (FEATURES #415)

A failed read is null ("not measured"), never [] / 0. Before

415 both were except: pass around a default, so an unreadable

incidents table rendered as a confident "0 incidents" on the operator console, the monitor and both CLIs.

• active_incidents: null → the endpoint still answers 200, and warnings carries one {"level": "warning", "category": "health_summary", "message": "incidents read failed"} row.
• oom_events_24h: null → likewise, "message": "OOM events read failed".
• uptime_24h: null means EITHER "no checks in 24 h" OR "the read failed"; since FEATURES #417 the new uptime_checks_24h tells them apart — 0 = no checks, null = the read failed.
• services (FEATURES #417): a failed read keeps the dict and nulls every VALUE ({"healthy": null, …, "total": null}), so a client doing services.get(...) does not crash on exactly the failure being surfaced.
• read_errors (FEATURES #417) names the reads that failed, from the fixed vocabulary services, uptime, oom, incidents, in that order. An older API does not send it — read it with a default of [].
• The tenant shape's empty-group short-circuit still returns [] / 0 (and uptime_checks_24h: 0, read_errors: []) — no apps is a fact, not a failure.

Since FEATURES #417 services, uptime_24h/uptime_checks_24h and oom_events_24h are computed by the SAME reader functions (services/platform_health_reads.py) that serve GET /api/admin/service-health and GET /api/admin/oom-events, so the summary and those endpoints cannot disagree about what they count.

⚠ Consumers must treat null as unknown: render —, never len(null) and never 0.

recent_actions — what it contains, and what the two sibling keys mean (BUGS #1072)

recent_actions is the last 24 hours of terminal, deploy-lane queue actions, newest first, capped at 500 rows. Both halves of that are derived, not hand-listed:

• terminal is the canonical TERMINAL_STATUSES set — the same one the complementary active_actions query excludes — so the two lists partition the deploy lane between them.
• deploy-lane is the generated deploy_queue.lane column, which excludes the six fast request→response RPCs (EXEC, INSPECT, BACKUP_QUERY, DRIVE_QUERY, GIT_PREFLIGHT, NS_ZONE_SETUP). A new action type is classified deploy-lane by default, so it appears here without a code change.

Before this was derived, the filter was a hand-written 16-status allowlist against a canonical set of 66, so 51 terminal statuses were invisible — every purge, restore, PITR restore, storage repoint, NS setup, rebalance and reconcile, and release_failed, i.e. a deploy that failed.

recent_actions_truncated says the 500-row cap bit, so this list is not the full 24 hours; recent_actions_window_start is the completed_at of the oldest row actually returned (an observation, not a restatement of NOW() - 24h). Without them a busy platform and a quiet one render identically. Both are absent on an API older than 4.29.96 — read them with a default.

Warning rows (FEATURES #417 — additive keys)

Every row still carries level and message (text unchanged). Added:

• source — persisted (a platform_warnings row: it has history) or computed (exists only while the condition does).
• Persisted rows: details, first_seen, last_seen. ⚠ first_seen is the FIRST-EVER firing of that key, not the start of the current episode — every writer UPSERTs with resolved_at = NULL and leaves created_at.
• Incident rows: app_name, service_name; disk and memory rows: instance_id, instance_name. Never parse message for these.
• group_slug on any row carrying app_name — ONLY when exactly one live group owns that canonical name (BUGS #692); otherwise null.
• The config-unreadable row now carries category: "configuration".
• A FAILED platform_warnings read is one {"category": "health_summary", "message": "platform warnings read failed"} row — never an empty list that reads as "nothing persisted is wrong".

⚠ details is now served on the operator summary. It is operator-visible data (the dev monitor reads it with a baked operator key): no writer may put a secret, token or password in platform_warnings.details.

The list is built by assemble_operator_warnings; the operator console's Warnings page renders this array itself rather than re-assembling it.

scope: "admin" is the flag clients use to decide whether to render the operator-only tiles.

Tenant shape

An authenticated non-operator caller gets a strictly narrower document:

{
  "applications": { /* only the caller's group's apps */ },
  "pipeline": { /* only the caller's group's builds + deploys */ },
  "warnings": [],
  "scope": "tenant",
  "timestamp": "2026-07-30T09:00:00Z"
}

• applications aggregates only the caller's group's apps.
• pipeline is likewise group-scoped, and its keys are an explicit allowlist applied after the query — a new operator-only pipeline field is never exposed to a tenant by accident. The fleet-wide aggregates an operator sees are absent by construction.
• The operator-only sections (api, database, cache, agents, routing, configuration, admission_gates, custom_domains, platform warnings) are never present.

The group is resolved from ?group=<slug> or the X-AppMecca-Group header, and only inside the tenant branch — so an anonymous request carrying a group selector gets a plain 401 rather than confirming whether that group exists.

The custom_domains block

Hidden in steady state; reports stalled domains (no progress in over 24 h — usually the customer's two DNS records are missing) and failed ones (terminal issuance/renewal states). A renewal_failed entry means a live certificate is heading for expiry.

GET /api/admin/incidents

Environment-wide incidents for a platform operator (FEATURES #415). This is the read behind the operator console's Overview Incidents tile and its Incidents page, which now agree by construction.

Gate	require_platform_operator at ROUTER level — 401 anonymous, 403 a tenant (including a group OWNER), 200 a platform-admin session or the legacy platform api-key
state	ongoing (default) · resolved · all — anything else is 422
limit	1–500, default 200
offset	≥ 0, default 0
Cache	no-cache

{
  "state": "ongoing",
  "incidents": [
    {
      "id": 42, "app_name": "euan-shop", "branch": "main", "group_slug": "euan",
      "service_name": "web", "started_at": "2026-09-14T08:00:00+00:00",
      "ended_at": null, "duration_seconds": null, "status": "ongoing",
      "check_count": 7, "first_error": "HTTP 502"
    }
  ],
  "count": 1,
  "total": 3
}

count is the page length; total is the full count under state. Both come from the SAME predicate /api/health/summary counts active_incidents through (HealthRepository._INCIDENT_STATE_WHERE), so total for state=ongoing equals len(applications.active_incidents) read at the same moment.

group_slug is the incident's OWN group, or null for a platform-producer incident (those carry no group). The console links a row to /g/{group_slug}/a/{app_name}?branch={branch} only when group_slug is set.

Why not GET /api/status/incidents

That route is anonymous-reachable (the public status page), so its unfiltered branch is pinned to the platform's own status checks (BUGS #684) and can never show a tenant incident. Do not widen it — that re-opens a cross-tenant leak. The console used to read it, which is why its page showed no incidents while the tile counted six.

Operator console reads (FEATURES #417)

Four read-only endpoints behind the operator console's Overview tiles and the pages they link to. Every one:

Gate	require_platform_operator at ROUTER level — 401 anonymous, 403 a tenant (including a group OWNER), 200 a platform-admin session or the legacy platform api-key
Cache	no-cache (listed above the /api/admin/* catch-all — the matcher is first-match)
Failure	200 with read_ok: false and null values, never a 0 or []

⚠ A tile and its page read ONE URL. The Services and Uptime tiles read /api/admin/service-health; the OOM tile reads /api/admin/oom-events?limit=1 — the same endpoints their pages read. Every count in these envelopes is derived from the rows returned beside it.

GET /api/admin/service-health

Both sections of the Services page, from two readers.

{
  "read_at": "2026-09-14T08:00:00+00:00",
  "window_hours": 24,
  "services": {
    "read_ok": true,
    "counts": { "healthy": 2, "degraded": 0, "down": 1, "unknown": 1, "total": 4 },
    "unreadable_configs": [],
    "rows": [
      {
        "app_name": "euan-shop", "service_name": "api", "state": "down",
        "decided_by": "check", "last_checked_at": "2026-09-14T08:00:00+00:00",
        "status_code": 502, "error": "HTTP 502", "check_branch": null,
        "deploy_status": "running", "group_slug": "euan"
      }
    ]
  },
  "uptime": {
    "read_ok": true,
    "aggregate": 95.4,
    "checks": 250,
    "healthy_checks": 239,
    "rows": [
      {
        "app_name": "status", "service_name": "API", "branch": null,
        "checks": 50, "healthy_checks": 49, "platform_status_check": true,
        "group_slug": null
      }
    ]
  }
}

• Services: one row per checkable (app_name, service_name) — an HTTP port and a non-background, non-static type. A latest health check decides (decided_by: "check", with that check's check_branch); otherwise the app's deploy_status (running → healthy, a failure status → down, else unknown).
• ⚠ Branch-blind, and deterministic. A multi-branch app is ONE row: the newest check on any branch, else the deploy status of its FIRST row in ORDER BY app_name, branch, group_id (group_id breaks a BUGS #692 name collision). appmecca health and mecca status count the same way.
• unreadable_configs names apps whose deploy_config did not parse; their services are in neither rows nor counts, and the console says so.
• Uptime: every health_checks row in 24 h grouped per (app_name, service_name, branch). aggregate = round(healthy_checks / checks × 100, 1) — weighted by check, and it includes the platform's own status checks (platform_status_check: true). aggregate: null with checks: 0 means no checks; with checks: null the read failed.

GET /api/admin/oom-events

Param
limit	1–500, default 500

{
  "read_at": "…", "window_hours": 24, "read_ok": true,
  "count": 3, "limit": 1,
  "rows": [
    {
      "id": "7f0c…", "app_name": "euan-shop", "service_name": "web", "branch": "main",
      "instance_id": "i-0aaa", "instance_name": "docker-host-a", "container_id": "c0ffee",
      "memory_limit": "128m", "event_source": "docker_event",
      "killed_at": "2026-09-14T08:00:00+00:00", "group_slug": "euan"
    }
  ]
}

count is the EXACT number of kills in the last 24 h (count(*) OVER ()), whatever limit is — never len(rows). instance_name is null when the agent row has since been reaped.

GET /api/admin/warnings/history

Param
days	1–365, default 30
limit	1–1000, default 200

Persisted warnings (platform_warnings) resolved within days, newest resolution first. Rows: warning_key, level, message, details, first_seen, last_seen, resolved_at. The envelope carries count (the page length), total (every resolution in the window, count(*) OVER ()) and "semantics": "last_resolution_per_key".

⚠ Not an event log. Every writer UPSERTs a key with resolved_at = NULL, so a warning that fires again disappears from this history and reappears among the current warnings. Computed warnings have no history at all. There is no manual resolve: the next scheduled check would re-open the row.

GET /api/admin/environment

{
  "read_at": "…",
  "environment": "us", "region": "us-east-2", "account_id": "121218949814",
  "domain": "us.appmecca.net",
  "tier": null, "tier_source": "unavailable",
  "versions": {
    "api": "5.3.3", "agents_read_ok": true, "agents_running": ["4.29.140"],
    "agents_published": "4.29.140", "version_consistent": true,
    "database_schema": {
      "read_ok": true, "applied_revisions": ["103"],
      "api_chain_head": "103", "active_agent_wheel_head": "103"
    }
  },
  "signup": { "invite_required": true, "selfserve_domains": [], "policy": "invite only" },
  "links_read_ok": true,
  "links": {
    "monitor": "https://monitor.us.appmecca.net",
    "admin_docs": "https://admin-docs.us.appmecca.net",
    "dev_docs": null, "api_docs": "https://api-docs.us.appmecca.net",
    "docs": "https://docs.us.appmecca.net"
  }
}

• An unset Setting is null, never "".
• versions.database_schema (apm-258) is the database's migration state, as a fact for operators — never a rollback verdict. No rollback path reads it; whether a rollback is allowed is decided against the target wheel's own migration chain.
  • applied_revisions — every row of alembic_version, sorted (numerically when every id is all digits, so 1000 follows 103). More than one entry means more than one revision is stamped. [] means nothing is stamped. null with read_ok: false means the read FAILED — a failed read is never [].
  • api_chain_head — the migration head this API process was built with: the head of the appmecca-common chain baked into the RUNNING API IMAGE (the image stages that wheel at image-build time), which is what this image applies when it restarts. It is not what a rollback applies — a rollback deploys an older image, which applies that image's own chain, and this field does not show it. null means unresolvable or more than one head.
  • active_agent_wheel_head — agent_versions.migration_head, the head the published agent wheel recorded. null means not recorded (an older publish) or unreadable.
• ⚠ tier is always null. The API has no tier and cannot read conf/{env}.yaml (BUGS #1183); deriving it from the environment's NAME is wrong for any future country environment.
• signup.policy is describe_signup_policy — the same sentence the API logs at boot.
• links are built with build_service_hostname from the registered apps monitor, appmecca-docs-usage, appmecca-docs-admin, appmecca-docs-developer and appmecca-docs-api — platform-group rows only: a tenant row holding one of those names is never turned into an operator-facing link. An app that is not registered is null, never guessed; links: null (with links_read_ok: false) means the read failed.

Operator pipeline reads (FEATURES #418)

Read-only endpoints behind the operator console's Pipeline and Audit pages. Same gate and cache posture as the P2 reads above: require_platform_operator at router level (401 anonymous, 403 a tenant including a group OWNER), and no-cache rows declared above the /api/admin/* catch-all.

GET /api/admin/actions

Queue actions across every group, newest first.

Param	Default	Meaning
outcome	—	failed, cancelled, in_flight, queued (buckets derived from the canonical status sets)
failure_reason	—	a classified reason; filtered in the API over at most 2000 scanned rows. Implies outcome=failed when outcome is not given (a completed row can carry a stale stamped reason).
phase, action_type, app_name	—	exact match
group	—	a group slug (case-insensitive), matched against the row's resolved group_slug
since_hours	24	1–720
exclude_types	—	comma-separated action types to leave out
limit	101	1–501

{
  "actions": [
    {
      "action_id": "…", "action_type": "DEPLOY", "status": "failed",
      "app_name": "acme-shop", "branch": "main", "phase": "health_check",
      "group_slug": "acme", "group_source": "app", "app_row_present": true,
      "app_name_is_canonical": true,
      "failure_reason": null, "effective_failure_reason": "health_check_startup_refused",
      "reason_source": "derived", "retry_count": 0, "parent_action_id": null,
      "op": null, "created_at": "…", "claimed_at": "…", "completed_at": "…"
    }
  ],
  "count": 1, "limit": 101, "window_hours": 24, "scan_truncated": false
}

• group_slug is null when the canonical app name is held by more than one group (BUGS #692) — never one of them at random. group_source is app (from a live app row) or purge_stamp (a PURGE_APP whose app row is gone, attributed from the group id the delete stamped); null when neither answers.
• A PURGE_APP whose stamped group differs from the group that holds the name today (the name was registered again after the purge) is attributed to nobody: group_slug, group_source and app_row_present are all null.
• app_row_present is false once the row's (app_name, branch) no longer exists — a deleted app or branch — and null whenever group_slug is null because the name is ambiguous: another group's branch is never reported.
• effective_failure_reason is the stamped failure_reason, else a classification of error_message (reason_source: stamped / derived).
• op is the operation name for plumbing actions; no other part of deploy_config is returned.
• scan_truncated: true means the failure_reason filter hit its scan cap — the list is incomplete and says so.

GET /api/admin/failure-summary

Failures in the window grouped by reason and by phase. Takes the same filters as /actions except outcome and limit, and counts exactly the rows /actions?outcome=failed with the same filters returns.

{
  "by_reason": [
    {"reason": "image_pull_error", "n": 1, "reason_source": "stamped", "uncategorised": false},
    {"reason": "unknown", "n": 1, "reason_source": "derived", "uncategorised": true}
  ],
  "by_phase": [{"phase": "health_check", "n": 1}, {"phase": null, "n": 1}],
  "total": 2, "truncated": false, "window_hours": 24
}

reason_source is mixed when a reason was stamped on some rows and derived on others. A phase that is not a simple token (host lifecycle actions store a JSON object there) is counted under phase: null, so every bucket can be reproduced by the /actions filter it links to.

GET /api/admin/actions/{action_id}

One action with everything the detail page needs. action_id may be a prefix of at least 8 hex characters.

Key	Contents
action	the list row shape above, plus error_message, image, commit and admission stamps
ownership	group_slug, group_source, app_row_present, app_name_is_canonical, is_platform_action
timing	created_at, claimed_at, updated_at, swap_committed_at, last_admission_check_at, completed_at
hosts	claimed_by, placement_instance_ids, require_hosts, exclude_hosts
retry	an advisory: applicable, retriable (the classifier's verdict on the reason), can_retry (whether a retry would be accepted — also requires the cap not reached, this row to be the latest failure, and no later attempt), superseded_by ({action_id, action_type, status} of that later attempt, or null), failure_reason, retry_count, max_retries, cap_reached, latest_failed_action_id, hint. The real retry route still decides.
diagnosis / diagnosis_error	the queue-diagnose narrative, or null plus the error when it could not be built
related	up to 20 rows: depends_on, dependent, parent, retry/child, and purge_sibling (the delete's other actions within ±5 minutes)
log_meta	kind (deploy/build), total_bytes, total_lines (a final line without a trailing newline counts)
deploy_warnings	the row's stamped warnings

404 for an unknown id; 422 for a prefix shorter than 8 characters.

GET /api/admin/actions/{action_id}/log

A byte window over the action's stored log (builds.build_log for a build, deploy_queue.deploy_log otherwise; a build still running with no stored log is read live).

Param	Default	Meaning
offset	0	byte offset, 0 to 2147483646
limit_bytes	65536	clamped to 1024–262144, never refused
download	false	true streams the whole log as action-{id}.log, capped at 20 MiB with a …[truncated at 20 MiB] marker

{"kind": "deploy", "live": false, "total_bytes": 48211,
 "offset": 0, "next_offset": 48211, "text": "…", "eof": true}

The window is snapped to UTF-8 character boundaries, so next_offset may be a few bytes short of offset + limit_bytes; page by passing next_offset back. The text is the same stored text the tenant log routes serve — build arguments are redacted when the log is written, and this route transforms nothing.

GET /api/admin/invites

Invitations across every group.

Param	Default	Meaning
state	pending	pending (not accepted and not expired) or all
limit	200	1–500

{"state": "pending", "count": 1, "invites": [
  {"id": "…", "group_slug": "acme", "email": "new@example.com", "role": "MEMBER",
   "created_by_email": "owner@example.com", "created_at": "…",
   "expires_at": "…", "state": "pending"}
]}

The invite token and its hash are never returned.

Summary changes: pipeline.read_failed and admission_gates_read_failed

• pipeline.read_failed is true when the queue could not be read. Its counts and active_actions are then zeros and [] that measure nothing. The key is present on both the operator and tenant shapes. On the operator shape the failure is also a health_summary warning, deploy pipeline read failed.
• admission_gates is null when the gates could not be read, with admission_gates_read_failed: true beside it. It was previously {}, which is indistinguishable from "no gate under pressure".

Summary changes: agents.update_coordination is tri-state

agents.update_coordination (operator shape only) reports the agent rollout. In this block null means the value could not be read. A 0 or false is a real answer, not a failed read. When no rollout is active, held_back_count is 0 because the gate is open, not because rows were counted.

Case	read_failed	What changes
The rollout state was read	false	Every value was read. With no active rollout, held_back_count is 0 because the gate is open; it is not counted.
No rollout has ever been published	false	update_status is "none" and held_back_count is 0. The gate is open, so nothing can be held back.
The rollout state could not be read	true	held_back_count, rollout_active, stuck and stuck_minutes are null. update_status is "unknown". published_version and published_at are absent.
A rollout is active and only the held-back count could not be read	false	held_back_count is null. Every other value is measured.

read_failed is always present. Both failures add a health_summary warning to warnings:

• agent update coordination read failed — rollout state, stuck detection and the held-back count are unknown
• agent rollout in progress — held-back deploy count could not be read

An API from before release 6.2 sends no read_failed key and reports a failed read as update_status: "unknown" with held_back_count: 0. Treat "unknown" as a failed read only when read_failed is absent.

GET /api/docs

Interactive OpenAPI documentation (Swagger UI). No authentication required.

GET /api/openapi.json

Machine-readable OpenAPI specification.