Set a field-size policy for unusually large CSV cells

By Batu · English technical notes

Also published in our Blogger archive.

Quick answer

csv.field_size_limit gets or sets the parser maximum field size for the current process.

Example

The example saves the old limit, installs a deliberately tiny threshold, and catches csv.Error for a nine-character cell. The finally block restores the old limit even when parsing fails, which matters because the setting is process-wide.

import csv, io
old = csv.field_size_limit()
csv.field_size_limit(8)
try:
    try:
        list(csv.reader(io.StringIO('value\n123456789\n')))
    except csv.Error:
        outcome = 'field-too-large'
finally:
    csv.field_size_limit(old)
assert outcome == 'field-too-large'
print(outcome)

Expected stdout:

field-too-large

Reading the result

The threshold of eight is only a test. Choose a production limit from the largest valid export cell and test it with actual representative data.

Set the limit before constructing the reader and restore it as soon as the bounded parse ends. That narrows the time during which unrelated CSV work in the same process sees the altered policy.

Catching csv.Error makes the oversized cell a controlled finding rather than a generic crash. The restored old limit is asserted by program structure, not assumed.

Sources

- Python csv module documentation

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.