Split an iterator into fixed-size batches
Also published in our Blogger archive.
Split an iterator into fixed-size batches
A stream does not need to be converted to a list before it can be processed in groups. fixed_batches calls iter once, then uses itertools.islice to pull at most size values for each tuple. It yields each non-empty tuple immediately, allowing the caller to process a long iterator incrementally.
The example feeds seven integers through a batch size of three. It emits two full three-value batches followed by (7,). Thus “fixed-size” means that each ordinary batch has three values and no batch exceeds three; the final remainder is intentionally smaller rather than silently discarded. The assertion records that contract precisely.
The function rejects zero and negative sizes because islice cannot make a useful forward batch from them. The returned object is a generator, so iterating it consumes the supplied iterator; call it again only with a fresh iterator if the input must be reread. Code that requires every emitted tuple to have exactly three values must handle or reject the final short tuple. Python 3.13's itertools.batched offers a strict option, but this small recipe makes its remainder policy explicit and works without relying on that newer parameter.
AI assistance was used to draft this article.
from itertools import islice
def fixed_batches(values, size):
if size < 1:
raise ValueError("size must be positive")
iterator = iter(values)
while batch := tuple(islice(iterator, size)):
yield batch
batches = list(fixed_batches(iter(range(1, 8)), 3))
assert batches == [(1, 2, 3), (4, 5, 6), (7,)]
print("batches:", batches)Expected stdout:
batches: [(1, 2, 3), (4, 5, 6), (7,)]