(motir-core) The WORKER — its own Fly process group, claim with FOR UPDATE SKIP LOCKED, backoff, graceful shutdown
The process that actually runs jobs: claim due rows, execute them, record the outcome, repeat.
Where it runs
Its own Fly process group, declared in fly.toml beside the existing app group — not inside the web process. Job load and request serving then cannot contend for the same event loop, and the two scale independently. The [processes] entry is this card's deliverable; scaling it to a running count is an operator action and belongs to its own manual sibling.
The claim, and why it is the whole card
Claiming is a read-derived write: the loop reads which rows are due, then writes a claim based on what it read. Two workers doing that with a plain read-then-write both see the same row and both run the job. So the claim is:
SELECT … FROM job_queue
WHERE state = 'pending' AND run_at <= now()
ORDER BY run_at
FOR UPDATE SKIP LOCKED
LIMIT n
inside one transaction, with the state write in the same transaction. SKIP LOCKED is what makes a second worker take the NEXT row rather than block on the first.
A serial test cannot see this defect. Two sequential claims never collide; the race needs genuine concurrency against a warm pool, and that is what the acceptance criteria demand.
The rest
- Backoff between empty polls, with
LISTEN/NOTIFY(or the chosen library's equivalent) so a freshly emitted event does not wait out a poll interval. - A lease, so a run claimed by a worker that dies is reclaimable rather than stuck in
runningforever. The lease duration and its renewal are this card's decisions and belong in a comment. - Graceful shutdown — on
SIGTERM, stop claiming, let in-flight runs finish or checkpoint, release claims. A deploy is a routine event and must not orphan work. - Side effects stay outside the claiming transaction. The transaction holds the claim write and nothing else; the job's own handler runs after it commits.
Acceptance criteria
- Two workers running concurrently against a real Postgres never execute the same
job_queuerow twice — asserted by a test that drives genuine concurrency and accepts every legitimate interleaving, not a serial one. - A worker killed mid-run releases or expires its lease, and another worker picks the run up.
SIGTERMdrains rather than aborting: no run is leftrunningwith no live claimant after a shutdown.- The
[processes]worker group is declared infly.toml; the card does NOT scale it. - A newly emitted event is picked up promptly rather than at the next poll boundary.
- Unit and real-Postgres integration tests ship with the change.
Context refs
fly.toml— the[processes]block, andmin_machines_runningas the precedent for how process groups are declared hereprisma/schema.prisma—job_queuefrom the schema cardmotir-core/CLAUDE.md§ concurrency — lock-before-a-contended-update and the warm-pool racelib/services/codeGraphIndexAdmissionService.ts— an existingFOR UPDATEclaim in this codebase, for shape