Technology · PRACTICAL GUIDE
Python raises BrokenProcessPool? Capture the worker exit boundary before rebuilding the pool
The exception only proves that a worker ended non-cleanly. Preserve task, PID, start method, stderr, exit status and OOM evidence before choosing an import, crash, capacity or deployment fix.
BrokenProcessPool means a ProcessPoolExecutor worker ended non-cleanly; it does not say whether the boundary was an import failure, initializer crash, native fault, signal, OOM kill or external termination. The diagnosis starts before any retry recreates the pool.
Preserve one task-to-worker trail, classify the exit with supported Python and operating-system evidence, make the smallest cause-specific change, then rerun both the original task and a second pool-health task.
1. Freeze one failure before the pool is rebuilt
Record the submitted task ID, a safe fingerprint of its input, submit time, Python version, platform, ProcessPoolExecutor options and multiprocessing.get_start_method(). Preserve the parent exception and its full chained traceback.
Do not start with an automatic retry loop. A replacement pool can work while the original worker PID, exit status, stderr and resource event disappear. Reduce to one failing task, but keep the same saved module, start method, initializer and deployment limit.
2. Separate entry-point failures from process death
First verify that the worker can import the saved __main__ module, that pool creation is guarded, and that the callable, arguments and return value are picklable. Run one trivial top-level function under the same start method. Interactive sessions and lambdas are not equivalent to a saved importable module.
If the trivial task never starts, fix the entry-point, serialization or initializer boundary. If it starts and the real task later breaks the pool, stop treating __main__ advice as a complete diagnosis and preserve the worker exit evidence.
3. Capture the worker exit boundary
Give each submitted task an ID and log worker PID, initializer completion and task start to an inherited stderr or per-worker file that survives pool teardown. Enable faulthandler at interpreter startup or in the initializer, keep its file descriptor open, and collect host-native crash records when available.
Use supported telemetry rather than building a production dependency on private executor attributes. Python process exit semantics provide a classification model: zero is normal, positive values can be explicit exits, and on POSIX a negative value represents termination by signal. Treat the exact availability of worker exit detail as Python-version dependent.
4. Correlate signals, OOM counters and deployment events
On Linux, snapshot the workload cgroup memory.events before the run and after the failure. A rising oom_kill counter is direct evidence that an OOM killer selected a process in the cgroup; memory pressure alone is not the same proof. Also preserve container or service restart and operator-termination events.
Pair the time window with peak memory, concurrency and input size. A fatal-signal faulthandler dump points toward native code; an external termination or deployment event points elsewhere. When evidence is absent, keep the outcome unresolved instead of labeling every abrupt exit an OOM.
5. Fix one proven class and verify both task and pool health
For an import boundary, repair the saved entry point or serialization. For a native crash, update or isolate the extension and reproduce with the smallest input. For an OOM kill, change input shape, concurrency or the measured limit. For an external kill, repair the supervisor or deployment policy.
Rerun the same minimal failing task with the same start method and limits, then run a second ordinary task in the same pool. Accept only when both complete, no new fatal dump or OOM event appears, the pool remains usable, and the evidence ledger names the change.
