Technology · PRACTICAL GUIDE
Python says “BrokenProcessPool”? Identify the dead worker before retrying the batch
The parent exception proves that a worker ended non-cleanly, not why it died. Preserve PID and task markers, separate spawn failures from crashes, and compare cgroup OOM counters before rebuilding the pool.
BrokenProcessPool means at least one ProcessPoolExecutor worker ended non-cleanly. It does not prove an ordinary task exception, an out-of-memory kill, a segmentation fault, or an import mistake. Once a worker vanishes, pending futures can receive the same pool-level exception, so the traceback in the parent may point to the observer rather than the task that killed the process.
The useful response is an evidence join. Preserve the parent task ledger, write worker PID and task markers before risky code, separate failures before the first marker from failures after it, then compare service, crash, and cgroup memory evidence. Recreate the pool only after one cause branch and one suspect input are isolated.
Freeze the parent ledger before retrying
Assign every submission a task ID and stable input key. Record submit time, Future identity, side-effect boundary, and the first result or exception. When the pool breaks, stop bulk resubmission and preserve the complete stderr and service log window.
Mark completed side effects separately from merely submitted work. A pool-level failure can mark many pending futures broken, so “this Future raised BrokenProcessPool” does not prove that its input killed the worker.
Mark worker start and isolate the phase
At task entry, flush a structured marker containing PID, task ID, input key, Python version, start method, and phase. Use an initializer only for small deterministic setup, and log initializer entry and success because an initializer exception breaks the pool.
If no task marker appears, test a real .py file with an importable main module, a main guard where required, top-level callables, and serializable arguments. Do not switch blindly to fork: start-method behavior is versioned and fork can be unsafe in multithreaded programs.
Capture native crash and signal evidence
Enable faulthandler before the suspect native code so fatal Python-runtime signals can write a traceback to a durable stderr destination. Preserve crash reports, service-manager status, container termination reason, and any available nonzero child exit code.
A negative POSIX exit status can represent a signal, but remaining workers may be terminated after the pool is declared broken. Correlate the first dead PID and timestamp instead of treating every later terminated worker as an independent root cause.
Prove or reject the OOM branch
For a Linux cgroup v2 service or container, snapshot memory.events or memory.events.local before the run and immediately after failure. A delta in oom_kill is evidence that an OOM killer killed a process; high, max, or oom counters describe different pressure or limit events and require careful interpretation.
Also record memory.current, memory.peak when available, the cgroup path, worker count, input size, and whether arrays or models are copied per process. Lowering max_workers is a useful experiment, not proof: it can reduce memory amplification without identifying the dead task.
Reduce to one worker and one suspect input
Run the exact callable serially first, then in a one-worker executor from a real file. Add inputs by binary search while keeping task and PID markers. If one input still kills one worker, investigate that native library, allocation, or data boundary.
If the one-worker run succeeds but many workers fail, compare aggregate memory, file descriptors, library thread pools, and start-method behavior. Keep the original pool closed; create a new executor only for the controlled experiment.
Close with a causal recovery test
The closure test has one recorded task, one known worker PID, a normal result or ordinary Future exception, unchanged oom_kill counters, and no fatal crash record. Then repeat at a small concurrency level while verifying each input is processed once.
Only after the branch closes should you adjust worker count, memory limit, start method, native dependency, initializer, or retry policy. Document the proven cause, version, evidence window, and idempotency rule so a future pool break cannot duplicate work.
