Use a lookahead to keep overlapping tokens
Ordinary regex matches do not overlap: after a match is found, the next search begins after that match. A positive lookahead provides a useful alternative when the desired tokens share characters. In ABABA, the three-character token ABA occurs starting at positions 0 and 2, but a plain findall("ABA", text) sees only the first occurrence.
The expression (?=(ABA)) is zero-width. At each position it checks whether ABA begins there, while the capturing group stores the token. Because the overall match consumes no characters, finditer() can test the next position and retain the overlapping occurrence. The example uses match start positions plus captured values, so the expected result makes that behavior explicit.
A lookahead is a matching technique, not a general tokenizer. It can create many candidate checks on long inputs, and more complex patterns inside it can be harder to reason about. If tokens have variable boundaries or need priority rules, an explicit scanner can be clearer. Also note that the token is obtained from group 1: match.group(0) is the empty string because the lookahead itself consumed nothing.
Lookahead and finditer() are available in supported Python 3 releases; no newer API is required.
AI assistance disclosure: This article was drafted with AI assistance and checked against the cited Python documentation.
Sources: Python regular-expression syntax: lookahead and Python re.finditer.
import re
text = "ABABA"
plain = re.findall("ABA", text)
overlapping = [(match.start(), match.group(1)) for match in re.finditer(r"(?=(ABA))", text)]
assert plain == ["ABA"]
assert overlapping == [(0, "ABA"), (2, "ABA")]
assert all(token == "ABA" for _, token in overlapping)
for start, token in overlapping:
print(f"start={start} token={token}")start=0 token=ABA
start=2 token=ABA