Motir

Type · research (measurement → decision) · Parent · MOTIR-1464 · Repo · motir-core · Follows · MOTIR-3950, which turned the dial 8 → 12 · Do not run before the sample exists — see When this is ready

The question, and why it is open

MOTIR-3950 sized the leg count on the mean leg: per-leg worker time ~1700 s at eight legs ⇒ ~1133 s at twelve ⇒ ~8.3 min a leg. That arithmetic is right and it is incomplete, and the gap is the reason for this card.

Splitting finer takes a maximum over more samples. What sets the lane's contribution to CI's critical path is not the mean leg, it is the SLOWEST one — and the slowest leg is set by runner quality, not by partition size. So more legs lowers the mean and may barely move the worst case, while giving you more draws at a bad runner.

Three twelve-leg runs so far, and two of them show it:

Against eight legs WITH the same packer: 33271194884 ran 9.4–10.8 min, 33263282834 ran 9.5–13.9 min.

So the mean nearly halved (≈10.5 → ≈7 min) and the worst case did not reliably improve. Three runs cannot separate that from luck, which is the whole reason this is a card and not a conclusion.

The packer is NOT under suspicion. On the 17.6-min run, eleven of twelve legs sat at 339–429 s of vitest step — a 1.27x spread — and the outlier was a degraded runner: its migrate deploy, a FIXED workload, took 6 s against a 3 s median (leg 12 took 8 s). That control is how every reading below separates a slow runner from a bad split, and it must be read every time.

When this is ready

Do not run this until at least five completed, successful ci.yml runs exist at twelve legs beyond the three listed above. Check first:

gh run list --workflow ci.yml --limit 40 \
  --json databaseId,headSha,event,status,conclusion,createdAt \
  --jq '.[] | select(.status=="completed" and .conclusion=="success") | "\(.databaseId) \(.createdAt) \(.event)"'

⚠️ Exclude cancelled runs entirely — a cancelled run's job durations measure when somebody stopped it, not what the work cost, and they read as plausible fast legs. ⚠️ Exclude runs where the test lane was skipped (a docs/design-only diff): the changes job gates it, and a skipped matrix job reports a duration of zero, which is an absence wearing a measurement's clothes.

How to measure

The primary metric is the vitest STEP, not the job, because it excludes queue time and per-leg setup — both of which differ between push and pull_request events and would otherwise confound a comparison across the two. Per run, take the MAX step across its legs; across runs, take the MEDIAN of those maxima.

RUN=<id>
gh api "repos/moooon-B-V/motir-core/actions/runs/$RUN/jobs?per_page=100" \
  --jq '.jobs[] | select(.name|startswith("Vitest (")) | "\(.id) \(.name)"' > /tmp/legs.txt
while read -r jid name; do
  gh api "repos/moooon-B-V/motir-core/actions/jobs/$jid" > /tmp/j.json
  jq -r --arg n "$name" 'def d($s): [.steps[]|select(.name|test($s))]
      | if length>0 then ((.[0].completed_at|fromdate)-(.[0].started_at|fromdate)) else -1 end;
    "\($n)|vitest=\(d("Run pnpm vitest"))s|migrate=\(d("migrate deploy"))s"' /tmp/j.json
done < /tmp/legs.txt | sort -t'|' -k2 -rn

For every leg whose vitest step exceeds ~1.5x the run's median, read its migrate deploy: ≈3 s is a healthy runner, 6–8 s is a degraded one. Report degraded legs as such — but do NOT discard them from the metric. A degraded runner is part of the cost of drawing twelve of them; excluding it would answer a different question than the one asked.

The decision rule — COMMITTED BEFORE THE DATA IS READ

This is written down first on purpose. The failure mode for a measurement like this is picking the rule after seeing the numbers, and it would be very easy here: "the outlier was a bad runner" is true and is not an argument for keeping twelve legs.

Let M12 = median across ≥5 twelve-leg runs of that run's slowest vitest step. Let M8 = the same over the eight-leg-with-packer runs (33271194884, 33263282834, plus any others found with leg: ['1'…'8'] in their ci.yml).

• M12 ≤ 0.9 × M8 — twelve legs wins. Keep it. Record both figures on this card and close.
• M12 ≥ 1.1 × M8 — twelve legs is worse. Revert to eight (below).
• Within ±10% — REVERT TO EIGHT. A tie goes to the cheaper configuration: twelve legs costs ~11 extra runner-minutes per run for a wall-clock gain that, by hypothesis, did not materialise. Do not keep the larger number because it is the one already merged.

If the verdict is revert

It is genuinely one edit, and this is the payoff of a thing MOTIR-3950 did on the way past: the guard was de-pinned from a hardcoded 8, so no test needs editing to change the count.

1. tests/helpers/vitestShardPlan.ts — VITEST_LEG_IDS back to eight ids.
2. .github/workflows/ci.yml — the test matrix leg: list, the job name:, and the --shard=<leg>/N denominator.
3. Run pnpm test:guards. ⚠️ A stale denominator does not fail on its own: CostBalancedSequencer declines a shard count it does not recognise and falls back to Vitest's own sha1 partition — correctly, and invisibly — undoing the whole plan while every leg stays green. tests/vitest-shard-plan.test.ts asserts the denominator equals VITEST_LEG_IDS.length; that assertion is the only thing standing between a one-character mistake and a silent regression to the 2.52x split.

Deliverable

A comment on this card carrying: the run ids sampled and their event types, the per-run slowest-step figures, which legs were degraded and their migrate deploy readings, M12 and M8, the branch of the rule that fired, and the action taken. Numbers before conclusion, and say plainly if the sample is still too small or too noisy to fire any branch — "not yet decidable" is a legitimate result and better than a coin-flip dressed as a finding.

Out of scope

• Re-measuring FILE_TEST_SECONDS. It is per-FILE; the leg count repartitions the same numbers. Refresh it when the MEDIAN file cost drifts, never to chase one run's noise (MOTIR-3913's AC-3 finding).
• Anything about runner quality itself. Whether GitHub gives us degraded runners, and how often, is a real question and a different one. This card takes runner variance as a fact of the environment and asks only which leg count survives it better.
• The coverage merge job, the build job, and the E2E lane. MOTIR-3928 already took an E2E leg from 10.9 to 5.4–5.9 min; the two lanes are level again and neither is this card's subject.

Resolution: open.