Decode a chunked HTTP body fixture

By Batu · English technical notes

Decode a chunked HTTP body fixture

Chunked transfer coding represents a body as hexadecimal length lines followed by byte sequences. This in-memory fixture contains Wiki as a four-byte chunk and pedia as a five-byte chunk; its second length line also includes an extension. The decoder separates a length line at CRLF, ignores an extension after ;, converts the remaining hexadecimal token with int(token, 16), and appends exactly that many bytes.

The assertions matter because a length alone is insufficient: each non-final chunk must have its own trailing CRLF, and the zero-sized chunk is followed here by the final blank line. The printed result is Wikipedia, decoded from the accumulated bytes only after the framing has been removed. This uses long-established bytes operations and int(); it has no newer API version requirement.

The routine intentionally accepts only a compact fixture format. It rejects missing delimiters through assertions, assumes no trailers, does not impose a maximum body or line length, and is not a streaming parser. Assertions are useful for demonstrating expected invariants, but applications should raise deliberate exceptions and enforce resource limits when processing untrusted messages. No connection is created; all input is a literal byte string.

The standard library’s HTTP client documentation describes HTTP protocol support and notes chunked transfer encoding. The hexadecimal conversion is specified by the int() documentation.

AI-assistance disclosure: AI assisted the drafting of this educational example.

def decode_chunked(data):
    position = 0
    decoded = bytearray()

    while True:
        line_end = data.find(b"\r\n", position)
        assert line_end != -1
        token = data[position:line_end].split(b";", 1)[0]
        assert token
        length = int(token, 16)
        position = line_end + 2

        if length == 0:
            assert data[position:position + 2] == b"\r\n"
            assert position + 2 == len(data)
            return bytes(decoded)

        chunk_end = position + length
        assert data[chunk_end:chunk_end + 2] == b"\r\n"
        decoded.extend(data[position:chunk_end])
        position = chunk_end + 2


fixture = b"4\r\nWiki\r\n5;note=part\r\npedia\r\n0\r\n\r\n"
body = decode_chunked(fixture)

assert body == b"Wikipedia"
print(body.decode("ascii"))

Wikipedia