Choose a fallback charset for a text fixture
A byte fixture needs an encoding before it can become text. This example writes a small Latin-1 fixture into a TemporaryDirectory, then selects its declared charset when present and otherwise uses the deliberately chosen fallback, latin-1. The input bytes b"caf\xe9" therefore decode to café, which is asserted before the result is printed.
bytes.decode(encoding) converts bytes using the named codec; the encoding is a data-format decision, not something Python can reliably infer from arbitrary bytes. The Python documentation for bytes.decode describes that conversion. A fallback is appropriate when the fixture format has a documented legacy default or when the test itself controls the fixture. It should be recorded near the fixture so a later reader knows why it was chosen.
Do not use this pattern to silently recover from a UnicodeDecodeError by trying unrelated encodings. A successful fallback decode only shows that the byte sequence is decodable; it does not establish that the resulting characters reflect the producer’s intent. If a declared charset is malformed or unsupported, handle that as a separate validation decision. TemporaryDirectory, used here to keep the fixture isolated and automatically cleaned up, is available from Python 3.2 onward. AI assistance was used to draft this article.
from pathlib import Path
from tempfile import TemporaryDirectory
fixture_bytes = b"caf\xe9"
declared_charset = None
fallback_charset = "latin-1"
with TemporaryDirectory() as directory:
fixture = Path(directory) / "sample.txt"
fixture.write_bytes(fixture_bytes)
charset = declared_charset if declared_charset is not None else fallback_charset
text = fixture.read_bytes().decode(charset)
assert charset == "latin-1"
assert text == "café"
print(text)café