Motir

Repo · motir-core. One PR. Filed as the DEFERRAL card for deploy latency, which the engine-fairness story puts out of scope. Independent of it.

What was observed, 2026-08-28

• 6dafd2ee8 (MOTIR-3418, the Inngest retirement) merged at 07:44:30Z.
• The last Fly release is 182, 08:47:04Z, and it reused the image built at 07:51 ("Container image … already prepared").
• At 11:00Z the running /app/worker/worker.mjs still contained job-lanes ×1, MOTIR_POSTGRES_JOB_IDS ×2 and inngest ×278 — none of which exist anywhere in origin/main. Production was running code the repository no longer has.
• A consequence: system.daily-health-check dead-lettered at 09:04:46Z on a JobLaneDriftError whose message names a second engine that no longer exists.

The two mechanisms, both read from .github/workflows/ci.yml

1. Deploy to Fly sits behind the whole matrix, a second time. needs: [lint, typecheck, build, test, coverage, e2e, e2e-at-scale]. The PR already ran that matrix against the merge commit; the push to main runs it again before anything deploys. The file's own header measures the cost: "a run needs ~35 minutes of clear air on a quiet day and far more under load — run 32233900851 ran 08:42:13 → 10:01:30, 79 minutes."
2. A merge burst leaves only the newest run. cancel-in-progress: ${{ github.event_name == 'pull_request' }} correctly does NOT cancel a running main job — but GitHub holds one PENDING run per group, so rapid merges replace each other in the queue. On 2026-08-28 the main runs at 10:44 and 10:45 were both cancelled; only 10:51 survived. That is the documented, intended behaviour (the survivor carries every commit before it) and it is not a defect — it is why a merge can wait a long time for its deploy without anything looking wrong.

The question this card answers

Should the release re-verify what the merge gate already verified? The industry-standard answer is a merge queue: it tests the exact commit that will become main, and main deploys on green without running the matrix twice. That keeps the gate and removes the duplication, which is a different trade from simply narrowing needs — narrowing trades safety for latency and should be recorded as such if it is chosen.

Weigh and record, do not assume:

• A merge queue — the standard shape; what it costs to adopt here, and whether the at-scale lane can run in it.
• Narrowing needs — deploy on lint + typecheck + build + test, letting the E2E matrix gate the PR rather than the release. State the risk explicitly; this is the shortcut.
• Leaving it — with the staleness window named, and something that reports when production's build is older than main by more than N minutes.

⚠️ A staleness DETECTOR is owed whichever option wins

The morning's real cost was not the delay; it was that nobody could tell. Prod ran a build older than a merged retirement for over three hours and every dashboard was green. Whatever this card decides about latency, it owes a reading that says "the deployed commit is behind main", on the same principle as the queue-depth signal in the sibling story: a deployment cannot be the thing that reports its own staleness.

Acceptance criteria

1. The card records which of the three options is taken, with the reason, and — if a merge queue — what it costs to adopt.
2. The chosen change is implemented in .github/workflows/ci.yml (or the repository's merge-queue settings), and a real merge is observed reaching a Fly release with the elapsed time quoted.
3. A staleness signal exists that compares the deployed build's commit against main's head and is readable outside the deployment.
4. ci.yml's concurrency header is amended to describe the behaviour as it stands after this card, so the next reader is not measuring against a stale narrative.
5. No verification is removed from the MERGE gate — whatever is dropped from the release path is still run before a commit reaches main.

Context refs

• .github/workflows/ci.yml — the concurrency block and its header, and the deploy job's needs / if / concurrency.
• docs/decisions/ — check for an existing record on deploy gating before authoring a new one.
• MOTIR-3692 — main could not deploy for hours on a flaky at-scale leg; the same latency path, a different cause.

---

⚠️ AMENDED ON THE RECORD, 2026-08-30 (MOTIR-3760's own run) — MECHANISM #1 WAS FALSE WHEN IT WAS WRITTEN, and the figures above are stale

Both halves are recorded rather than edited away, because the struck text is what the card was sized and reasoned from.

1. "The PR already ran that matrix against the merge commit; the push to main runs it again." NOT TRUE OF e2e-at-scale, the most expensive lane in the needs list. Its condition is needs.changes.outputs.app == 'true' && (github.event_name == 'push' || contains(github.event.pull_request.labels.*.name, 'e2e-at-scale')), so for an unlabelled pull request — the overwhelming majority — the push-to-main run is the FIRST and ONLY execution of board-at-scale, collab-at-scale, reporting-at-scale and billing-cloud.

This was not drift. git log -S"contains(github.event.pull_request.labels.*.name, 'e2e-at-scale')" -- .github/workflows/ci.yml dates it to 99826661d, 2026-08-19T22:39:19+02:00 (MOTIR-3148) — nine days before this card was written. A second fact points the same way and was also true at filing: protect-main's strict_required_status_checks_policy is false (gh api repos/moooon-B-V/motir-core/rulesets/17227448 → {"checks":["CI complete"],"strict":false}), so a pull request merges while BEHIND main and the push-to-main run is this repository's only verification of the merged SUM.

So narrowing needs does not remove a repetition — it removes a check. The card's own option 2 called that "the shortcut" and asked for the risk to be stated; the risk is larger than the card knew.

2. The 35–79-minute figure is stale. Nine consecutive SUCCESSFUL push-to-main runs, 2026-08-29/30, created → Deploy to Fly complete: 21.2 · 22.1 · 23.0 · 27.1 · 34.0 · 34.1 · 34.4 · 42.4 · 45.2 minutes (runs 33307649462 · 33270573757 · 33285015926 · 33271383815 · 33250609131 · 33282807240 · 33248423068 · 33272081857 · 33285023182). Median ~34, worst 45.2. MOTIR-3902 / 3912 / 3913 / 3928 / 3950 — all merged in the 24 hours before the measurement — are what moved it.

3. And the merge queue is not the latency answer the card assumed. It runs BEFORE the merge, so merge → deployed becomes queue-run + release ≈ 20 + 7 against 14.5 + 6.75 today, and the pull-request lane stays, so every change pays for a second full run. It is a correctness improvement — it is exactly what fixes point 1's second half — and should be re-opened as one, sized against that second run.

Outcome — the option taken is the third, with the detector the card said was owed whichever way it went: the release keeps every gate, and .github/workflows/deploy-freshness.yml reports from OUTSIDE the deployment when the deployed commit is behind main. docs/decisions/ci-minutes-allowance.md §J.5 is the record; PR #2467 is the implementation. Planning bug filed under MOTIR-1465.