Technology · PRACTICAL GUIDE
pandas CSV IDs changed despite string dtype? Trace the value before and after import
A four-row test separates numeric inference, missing-value rules and damage already written into the CSV. A late string conversion cannot undo them all.
When CSV identifiers change, compare the untouched fields with the values at each processing stage before adding another cast. A string dtype does not automatically disable missing-value recognition, and reading a damaged export as strings cannot recreate digits that an earlier numeric conversion discarded.
We tested four declared identifier values—00123, 9007199254740993, literal NA and an empty field—with Python 3.12.14, pandas 2.2.3 and the C parser. Thirteen checks passed. These are small controlled examples, with NA deliberately defined as text; your application's missing-value rules still need a separate decision.
Start with the untouched fields and a known schema
Keep the original CSV and record its encoding, delimiter, quoting, header and identifier columns. Parse a working copy using the actual dialect. Python's csv.reader normally returns field strings without automatic numeric conversion; avoid QUOTE_NONNUMERIC when building a lexical baseline. Check header positions before mapping fields into dictionaries.
Record exact sample values rather than a formatted display. CSV quotation marks delimit fields; they are not a pandas type declaration. The linked Excel guide explains the same raw-source boundary for spreadsheet imports, while the header guide handles repeated labels separately. A file hash records byte identity, but a local hash alone does not establish the correct business values.
Set the text type and missing-value policy separately
With import pandas as pd, the tested read was pd.read_csv('input.csv', engine='c', dtype={'id': 'string'}, keep_default_na=False). On our known UTF-8 fixture this retained 00123, the exact large identifier, literal NA and the empty string. String dtype alone retained the digits but converted the last two values to missing.
keep_default_na=False affects missing-token interpretation across the read, not just id. It is useful here for inspecting original text, not a declaration that every blank in every column is valid. Define per-column missing tokens and type rules from the producer's schema, then validate them. Do not turn missing values into zero or a literal 'nan' merely to eliminate an error.
Find whether the damage happened before the current reader
Our 63-byte input quoted every field. The default numeric read produced float64 identifiers; casting that column to string afterward yielded 123.0 and 9007199254740992.0. Exporting those inferred values wrote the changed digits into a new CSV and collapsed literal NA and the original empty field into empty fields.
Reading that damaged CSV with the explicit string settings preserved its damaged text. The reader was no longer the first loss point. Both that export and the correctly preserved export happened to be 43 bytes with the same row and column counts. Equal size and shape therefore did not establish value equality in these controls.
Verify a separate round trip against the source values
Write to a new destination and reparse with the same declared schema. Compare the ordered identifier strings and missing-value distinctions with the untouched source; reconcile records and investigate collisions or changed values. The preserved test round trip kept all four values. Its byte representation changed from 63 to 43 bytes because quoting changed, so raw byte equality was not the appropriate semantic exit test.
If the earliest available CSV already contains rounded or shortened identifiers, return to an authoritative producer export or earlier preserved source. Padding zeros or rounding a displayed float cannot identify the lost original. These measurements cover pandas 2.2.3's C engine on four synthetic rows; recheck the actual deployed version and parser before applying the result elsewhere.
