Technology · PRACTICAL GUIDE
zlib loop keeps running at eof? Separate retained input from the next gzip member
Twenty-eight local checks distinguish post-end tail replay, first-member-only output and a truncated second member that still emits every expected plaintext byte.
A decompressor can have eof=True while your gzip input still contains another member. On our installed CPython 3.12.14, it could also retain unconsumed_tail after eof, so feeding that tail into the same completed object repeatedly added suffix bytes to unused_data without producing output.
These are different failures: a loop can keep running after one member ends, or it can stop too soon and omit the next. Our 28 small controls also found a more deceptive boundary: a truncated second member emitted all nine expected plaintext bytes across the two members, yet its eof remained False.
1. Record the wrapper state, not only the output length
Record sys.version, zlib.ZLIB_VERSION, zlib.ZLIB_RUNTIME_VERSION, the selected wbits, and the application rule for one or multiple gzip members. Our actual run used CPython 3.12.14/Linux and zlib 1.3.2 for both build and runtime. The upstream C manual we consulted identifies itself as 1.3.1; that is a reference, not the library version we executed.
For each small diagnostic call, record output length, eof, and lengths of unconsumed_tail and unused_data. Before eof, the retained tail may need another call because max_length limited output. At eof, decide what trailing bytes mean under your format policy instead of blindly feeding both attributes back to the same object.
2. Stop replaying a completed object
Our zlib-wrapped fixture contained 1024 plaintext bytes followed by the five-byte suffix extra. Using 100-byte output calls and continuing only while not d.eof and d.unconsumed_tail produced the exact payload. At completion, both tail attributes contained extra on this installed interpreter. They were not two independent pieces to concatenate.
We then made exactly three additional calls with the retained tail. Each returned zero output; unused_data grew from 5 to20 bytes while unconsumed_tail stayed 5 bytes. We did not run an infinite loop. The upstream issue already supplies the eof guard and is closed with linked fixes. Test your installed runtime; do not describe that report as a new unfixed defect in every current Python release.
3. A gzip member end is a handoff, not whole-input success
The 49-byte gzip fixture joined a25-byte member containing alpha and a24-byte member containing beta. One zlib.decompressobj(31) returned only alpha, reported eof=True, and retained the exact24-byte second member in unused_data. Its unconsumed_tail was empty in this unlimited call.
A new object fed that unused_data returned beta and reached eof with no suffix. gzip.decompress on the tiny complete fixture returned alphabeta directly, consistent with its documented multi-member behavior. These known APIs and the community new-object recipe are not new discoveries. For large or external input, choose a maintained reader with the required policy and limits rather than treating this in-memory comparison as a production ingestion design.
4. Do not use empty output or expected plaintext as the finish test
A valid empty gzip member produced no plaintext but reached eof and left the next member available. A loop that treats an empty output block as the end of all input can therefore skip valid later data. Track member completion and pending input, not only whether the latest output is nonempty.
We removed four trailer bytes from the second member. Processing both objects still emitted alphabeta, all nine expected bytes, but the second object had eof=False. Calling flush() did not establish its missing trailer. When we instead changed the second member’s CRC, the first object still returned alpha; the error appeared only when the second object processed that member.
5. Make the accepted-member policy explicit
A protocol may intentionally allow exactly one complete member and reject all trailing bytes. Another may allow concatenated members and require a fresh decoder for each. Do not silently discard unused_data, assume every suffix is another valid member, or add it to an overlapping unconsumed_tail. Our garbage suffix failed when treated as a new gzip member.
Keep one aggregate output budget across accepted members, plus input, member-count, time and upstream-buffer controls appropriate to the actual reader. max_length limits a call; zero means unrestricted output, and flush(length) specifies an initial buffer size rather than a ceiling. Our controls verified 1024 output bytes with max_length=0 and one byte with max_length=1. They did not measure process memory, CPU limits or a network client.
6. Commit only the result your completion checks establish
For a diagnostic regression test, use tiny known fixtures:one valid member, two valid members, an empty first member, a truncated second trailer, a corrupt second CRC and a nonmember suffix. Verify exact output, each accepted member’s completed state, and the final trailing-data decision. Preserve failures instead of publishing a useful prefix as a complete result.
All28 assertions passed on the stated runtime. The saved report records versions, platform, script SHA and actual state results. This was not a general streaming-reader implementation, a release-fix test, or proof of HTTP completeness, authenticated origin or safe resource use. A correct per-member transition is necessary; the application must still define when the entire accepted input is complete.
