Motir

Surfaced while running MOTIR-2946 under notes.html #27 — logged, not absorbed; that card's diff is a service + two repositories + a vitest file and touches no workflow.

The failure

A Playwright E2E leg reports 6h0m — GitHub Actions' workflow timeout, not a slow suite. The leg did not fail, it was killed, and conclusion is cancelled, not failure. Its Run E2E (<shard>) step is skipped: no spec ever executed.

The hang is inside e2e-setup → Install Playwright OS deps, in the apt-get update that playwright install-deps shells out to. The log ends mid-apt and then goes silent for six hours:

23:02:51.1271091Z Get:4 https://archive.ubuntu.com/ubuntu noble-backports InRelease [126 kB]
23:02:51.3024792Z Get:5 https://archive.ubuntu.com/ubuntu noble-security InRelease [126 kB]
05:01:03.5425545Z ##[error]The operation was canceled.

azure.archive.ubuntu.com had been Ign:-ing every index for the preceding 40 seconds, so apt fell back to archive.ubuntu.com and then wedged with no timeout of its own.

Triage note — read the last TIMESTAMP, not the last line. The operation was canceled reads like a cancellation somebody ordered; it is the six-hour gap above it that identifies the wedge. And do NOT take the ::error::playwright install-deps failed after 3 attempts line as evidence — it appears in every leg's log, green ones included, because the step body is echoed at definition time.

Why it is a bug now, not another re-run

.github/actions/e2e-setup/action.yml already wraps both install paths in a 3-attempt retry (lines 121–127 and 132–138):

for attempt in 1 2 3; do
  if pnpm exec playwright install-deps chromium; then exit 0; fi
  echo "::warning::playwright install-deps attempt $attempt failed; retrying in 15s"
  sleep 15
done

The retry can only fire on a non-zero EXIT. It cannot fire on a HANG — and the observed failure mode is a hang, so attempt 1 consumes the entire 6-hour budget and attempts 2 and 3 are unreachable. The resilience the step was written to provide is exactly the resilience it does not have.

Three occurrences in ~6 hours, on two different PRs, each on a diff that touches nothing the leg exercises:

Every other check was green in each case — all Vitest legs, coverage, TypeScript, the sandbox-image matrix, and the other twelve E2E legs running the same e2e-setup.

This is the same class as MOTIR-1679 (the packages.microsoft.com apt-source flake) and MOTIR-1742 (the Docker Hub pull flake): an external dependency in the job's critical path with no in-repo resilience. It meets the same escalation threshold MOTIR-1742 set for itself — a third occurrence is filed, not re-run.

The cost, which is what makes it worth 40 minutes

Each occurrence burns a six-hour runner and blocks the aggregate CI complete for six hours, so the PR cannot be merged until a human notices, reads timestamps, and re-runs. Three occurrences is ~18 runner-hours. It also makes a red check indistinguishable at a glance from a real regression, which is the cost MOTIR-1742 named and the reason that one was fixed rather than tolerated.

Fix

Wrap each attempt in a per-attempt timeout so a hang becomes a failed attempt and the EXISTING retry loop does its job:

for attempt in 1 2 3; do
  if timeout 300 pnpm exec playwright install-deps chromium; then exit 0; fi
  echo "::warning::playwright install-deps attempt $attempt failed or timed out; retrying in 15s"
  sleep 15
done

Five minutes is generous — the step takes ~30–60s on a healthy runner. Apply it to both blocks in e2e-setup (the cache-hit install-deps path AND the cache-miss install --with-deps path; the observed hang was the cache-hit one, but the cache-miss path shells out to the same apt). Consider a job-level timeout-minutes on the E2E legs as defence in depth — a leg that legitimately needs six hours does not exist, and the workflow-level budget is the wrong backstop for one wedged leg.

Acceptance criteria

• Both retry loops in .github/actions/e2e-setup/action.yml bound each attempt with a timeout, so a hung install-deps / install --with-deps fails that attempt instead of consuming the job budget.
• A hang on attempt 1 therefore reaches attempts 2 and 3; only three timed-out attempts fail the step, and the step's ::error:: line is then reached for a real reason.
• The E2E legs carry a timeout-minutes well below the workflow budget, so no single leg can hold CI complete for six hours.
• Verified on the PR's own CI: confirm in a job log that the timed command ran (the timeout wrapper is present in the echoed step body) and the leg completed in its usual 6–9 minutes.
• The flake-log entry for this class is updated with the PR reference.

Context refs

• .github/actions/e2e-setup/action.yml — the two retry blocks, lines ~121–127 (cache miss) and ~132–138 (cache hit).
• .github/workflows/ci.yml — the E2E matrix legs that call it, for the timeout-minutes half.
• Evidence: run 32073646174, job 95529249270 (Playwright E2E (bulk-2), PR #2105) — Run ./.github/actions/e2e-setup 23:01:16 → 05:01:03 cancelled, Run E2E (bulk-2) skipped.