Technology · PRACTICAL GUIDE
Python queue.join() hangs? Audit unfinished tasks and worker liveness
Separate an empty container from the unfinished-task ledger, surface dead consumers, count sentinels, and choose graceful or immediate shutdown without faking completion.
queue.join() does not wait for the container to look empty. It waits for a separate unfinished-task counter to return to zero: every put increments it, and every task_done decrements it. A worker can remove the last visible item and then crash, be cancelled, or return before acknowledging it; empty() may be true while join still waits correctly.
Repair the accounting before adding timeouts or clearing internals. Pair each successful get with exactly one task_done in a finally block, monitor worker completion alongside join, and count sentinel items too. On Python 3.13 and later, graceful shutdown preserves the normal join invariant; immediate shutdown can deliberately unblock without completing work, so that choice must be recorded as abandonment rather than success.
1. Prove which counter is stuck
Log a sequence number for every put, get and task_done, plus the item or job identifier, worker name and monotonic time. Reconcile the ledger as puts minus acknowledgements; qsize() and empty() describe the container, not whether retrieved work completed.
Do not mutate queue.queue, unfinished_tasks or private locks in production. A direct clear removes visible items without supplying the acknowledgements that join expects, and private-field fixes can create negative counts or false completion.
2. Make acknowledgement exception-safe and exactly once
After get succeeds, place the processing body inside try/finally and call task_done in finally. Keep the acknowledgement outside code that can return early. If get itself fails or is cancelled before returning an item, do not call task_done for an item you never received.
Exactly once matters. Python raises ValueError when task_done is called more often than items were put. Record business success or failure separately from queue acknowledgement: acknowledging means the worker finished handling that delivery attempt, not that its side effect succeeded.
3. Observe worker liveness as well as join
A finally block fixes exceptions inside its scope, but it cannot help after a process crash, an unhandled BaseException outside the scope, or a consumer task that has already ended. In asyncio, wait for queue.join and the worker tasks together; if a worker finishes first, retrieve and propagate its exception before waiting forever.
For threads, keep explicit worker references and a stop signal, log uncaught exceptions, and fail the coordinator if all consumers die while unfinished work remains. A timeout is a diagnostic boundary, not permission to announce completion.
4. Count sentinels and choose shutdown semantics
A sentinel is still an enqueued item. The worker that receives it must acknowledge it before exiting, and the producer must enqueue the intended number of sentinels only after ordinary work. Too few can strand workers; an unacknowledged sentinel can strand join.
On Python 3.13+, shutdown(False) stops new puts while allowing queued items to drain and join to retain its normal meaning. shutdown(True) drains and adjusts accounting so join may unblock even though work was not performed. Use it only for an explicit abandon policy, and report which items were dropped.
5. Test the repair against failure timelines
Test a normal drain, a handler exception, cancellation after get, a leaked sentinel, a worker that exits before the final item, graceful shutdown with queued work, and immediate shutdown. Assert one acknowledgement per retrieved item and preserve the original exception.
Acceptance requires more than join returning. Prove all accepted jobs have a terminal status, no producer can put after shutdown starts, surviving workers are stopped, and immediate-shutdown runs label unfinished work as abandoned rather than completed.
