Skip to content

Fix localQueue running jobs from the same named queue concurrently - #622

Open
anavalo wants to merge 1 commit into
graphile:mainfrom
anavalo:fix-local-queue-named-queues
Open

Fix localQueue running jobs from the same named queue concurrently#622
anavalo wants to merge 1 commit into
graphile:mainfrom
anavalo:fix-local-queue-named-queues

Conversation

@anavalo

@anavalo anavalo commented Aug 6, 2026

Copy link
Copy Markdown

Description

Fixes #621.

With localQueue enabled, jobs from the same named queue could run concurrently, violating the documented serial execution guarantee for named queues. A batch fetch (batchSize > 1) could lock multiple jobs belonging to the same named queue in a single statement: the queue eligibility subquery locks each available queue row once, but every job in that queue then passes the in membership check, so limit batchSize could take several of them. The local queue would hand these out to idle workers concurrently, making per-queue concurrency min(localQueue.size, concurrentJobs) instead of 1. And once the first of those sibling jobs completed, completeJobs would unlock the queue row while the rest were still running, so other pools could lock yet more jobs from the same queue.

This PR makes batch fetches keep only the first job (by priority, run_at) per named queue and discard the rest. The discarded rows never get locked_at/locked_by set and their row locks release when the statement completes, while the queue row remains locked until the one kept job completes — restoring the invariant that a locked queue has exactly one job in flight, which completeJobs, returnJobs, and the batchSize = 1 fetch already rely on. Jobs that aren't in a named queue are unaffected, and the batchSize = 1 query text is byte-for-byte unchanged.

The reproduction script from #621 now reports 1 concurrent job (previously 4), and the 12 jobs execute in insertion order.

Two new tests in __tests__/main.runTaskList.test.ts (the suite previously had no test exercising localQueue):

  • jobs in the same named queue run serially even when the local queue is enabled — fails on main;
  • jobs in different named queues run in parallel when the local queue is enabled — also fails on main (all four jobs start at once there), and guards against over-serialising the batch.

Performance impact

  • localQueue disabled (batchSize = 1): none — the query text is unchanged.
  • Batch fetches: the added distinct on processes at most batchSize already-locked rows, which is negligible.
  • Because the deduplication applies after limit batchSize, a fetch returns fewer than batchSize jobs whenever several of the selected rows share a named queue. The worst case is a backlog whose highest-precedence jobs all belong to one named queue while its tasks complete faster than pollInterval: each poll's fetch then selects that queue's jobs again and discards the duplicates, yielding a single job, so the pool works that queue at roughly one job per pollInterval and other ready jobs wait despite idle workers until the queue's backlog drains below the batch window. The effect is bounded by fetch timing: if the queue's job is still running at fetch time, its queue row is locked, its jobs are excluded, and the batch fills with jobs from other queues and unqueued jobs; a "new job" notification also triggers an immediate fetch with the same effect.
  • Users of localQueue.refetchDelay with a non-zero threshold may see the refetch delay engage more often in workloads dominated by a few named queues.

Security impact

None.

Checklist

  • My code matches the project's code style and yarn lint:fix passes.
  • I've added tests for the new feature, and yarn test passes.
  • I have detailed the new feature in the relevant documentation. Bug fix — the documentation already describes the (serial) behaviour this PR restores.
  • I have added this feature to 'Pending' in the RELEASE_NOTES.md file (if one exists).
  • If this is a breaking change I've explained why. — Not a breaking change: it only removes undocumented concurrency that contradicted the documented behaviour.

With batching enabled (batchSize > 1), a single job fetch could lock
multiple jobs belonging to the same named queue: the queue eligibility
subquery locks each available queue row once, but every job in that
queue then passes the membership check, so `limit batchSize` could take
several of them. The local queue would hand them to workers
concurrently, and per-queue concurrency became
min(localQueue.size, concurrentJobs) instead of 1.

Batch fetches now keep only the first job per named queue and discard
the rest; the discarded rows' locks release at the end of the statement
and the queue row remains locked until the kept job completes, so the
serial execution guarantee holds again. The batchSize = 1 query is
unchanged.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

localQueue breaks the documented serial execution guarantee for named queues

1 participant