Treat Unicode normalization in filenames as a portability concern

By Batu · English technical notes

Also published in our Blogger archive.

Direct answer

Two filenames can look alike while their Unicode strings differ. The example compares a precomposed accented character with an equivalent sequence containing a combining accent. The first assertion preserves the fact that the original strings are unequal; NFC normalization makes them equal for the selected comparison.

unicodedata.normalize returns a string. It does not rename a file or prove that a filesystem will store, compare, or display the two spellings in the same way. Keep the original spelling alongside a normalized lookup key when a report needs to explain a collision.

Choose normalization as an explicit application rule. Applying it after constructing a dictionary can hide that two distinct inputs have collapsed onto one key. Case folding, visually confusable characters, and filesystem-specific case sensitivity are different concerns and are not tested here. Before renaming real files, check the target filesystem and detect destination collisions separately. This snippet intentionally demonstrates text comparison without creating either filename.

Complete example

from pathlib import Path
from tempfile import TemporaryDirectory
with TemporaryDirectory() as d:
    import unicodedata
    a = 'café'
    b = 'café'
    assert a != b and unicodedata.normalize('NFC', a) == unicodedata.normalize('NFC', b)
    result = 'comparison=NFC'
    print(result)

Expected stdout (for a platform supporting the demonstrated operation):

comparison=NFC

Sources

- unicodedata.normalize

Prepared with AI assistance. The example uses synthetic data; its stated limits apply.