Technology · PRACTICAL GUIDE
Python cannot schedule new futures after shutdown? Prove which lifecycle ended first
Separate a closed executor instance, interpreter exit, and asyncio default-pool teardown before retrying, replacing, or resizing anything.
RuntimeError: cannot schedule new futures after shutdown is not a request for a larger pool. It is evidence that a submitter crossed a lifecycle boundary: a particular Executor was closed, the asyncio default executor was shut down, or the interpreter had already begun exiting. The exact suffix and the complete submit stack tell you which owner to investigate.
Preserve that evidence before adding retries. A safe repair assigns one owner to each executor generation, stops new producers first, drains or cancels accepted work deliberately, and calls shutdown once. If the process itself is exiting, keep required control threads non-daemonic and join them before main returns; if asyncio.run owns the loop, let it own the default executor too.
1. Classify the exact shutdown signal
Save the full traceback, not only the last line. Record whether the message says after shutdown or after interpreter shutdown, and whether the call arrived through submit(), map(), a scheduler, a library transfer manager, or asyncio.run_in_executor(). Those paths point to different lifecycle owners.
Log a stable executor generation ID at construction, every accepted submit, shutdown start, shutdown completion, and the first rejected submit. Add process ID, thread or task name, and monotonic time. Private fields can help in a local experiment, but the repair must rely on public ownership and ordering contracts.
2. Fix reuse after an instance shutdown
A context manager closes the executor when the with block exits. If a long-lived command object stores that same executor, the first run can succeed and the second can fail. Either create a fresh executor inside every independent run, or move the with block to the component that owns the full repeated-use lifetime.
Do not catch RuntimeError and silently construct a replacement in every caller. That races multiple generations and can duplicate work. The owner should publish one current generation under a lock or state machine, reject submissions once draining starts, and replace it only after the old generation is accounted for.
3. Fix interpreter-exit and daemon-thread races
The interpreter-shutdown wording means the process lifetime is already ending. Common evidence is a main function that returns while a daemon controller, scheduler, callback, or transfer helper still plans to submit work. An uncaught main-thread exception can create the same ordering problem.
Use an explicit stop signal. Stop external producers, prevent new submissions, wait for the controller thread, settle accepted futures, then close the executor and let main return. If the work must finish cleanly, do not make its only owner a daemon thread. An atexit callback is too late to revive an executor whose interpreter exit handler has already run.
4. Respect asyncio default-executor ownership
run_in_executor(None, ...) uses the event loop's default executor. Once shutdown_default_executor() has run, later default submissions raise RuntimeError. With asyncio.run(), the runner manages generator finalization, the default executor and loop closure; application code should not shut that default pool twice.
For a manually managed loop, stop callbacks and servers that can create executor work, await the tasks that already use the pool, then await shutdown_default_executor() and close the loop. A callback scheduled from another thread must be rejected or drained before this boundary, not allowed to discover shutdown by exception.
5. Prove the repaired lifecycle under restart and exit
Test at least four timelines: two consecutive normal runs, shutdown while work is queued, service restart while a producer is active, and a main-thread failure. Assert that each accepted job has one terminal result, no submit occurs after the owner enters draining, and thread count returns to baseline.
Record whether pending jobs should complete or be cancelled, then test that policy explicitly. A clean run is not enough: the acceptance evidence is ordered timestamps showing producer stop before executor close, plus repeated stress without duplicate side effects, orphaned threads, swallowed exceptions, or late submissions.
