Iterators and generators

Iterable and iterator

An iterable is any object that can be traversed by an for (list, string, file). An iterator is the object that delivers the items one by one, with next(), and raises StopIteration when it ends. The for does exactly that underneath:

lista = [1, 2, 3]
iterador = iter(lista)
print(next(iterador), next(iterador), next(iterador))
try:
    next(iterador)
except StopIteration:
    print("acabou")

1 2 3
acabou

You can create your own iterator with a class that has __iter__ and __next__:

class Contagem:
    def __init__(self, limite):
        self.limite = limite
        self.atual = 0

    def __iter__(self):
        return self

    def __next__(self):
        if self.atual >= self.limite:
            raise StopIteration
        self.atual += 1
        return self.atual


print(list(Contagem(4)))

[1, 2, 3, 4]

Generator functions with yield

Writing a class just for that is tedious. A function with yield becomes a generator automatically. It pauses every yield and resumes from where it left off:

def contagem(limite):
    atual = 1
    while atual <= limite:
        yield atual
        atual += 1


print(list(contagem(4)))
gerador = contagem(2)
print(next(gerador), next(gerador))

[1, 2, 3, 4]
1 2

Laziness is economy

A generator does not store the items. It only keeps the point where it stopped. Therefore, generating one hundred thousand numbers takes almost nothing, while the equivalent list takes megabytes:

import sys

lista = [n for n in range(100_000)]
gerador = (n for n in range(100_000))
print(sys.getsizeof(gerador) < sys.getsizeof(lista))
print(sum(gerador))

True
4999950000

Since the generator has no defined end, it can even be infinite, as long as the consumer stops at the right moment:

from itertools import islice


def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b


print(list(islice(fibonacci(), 10)))

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

Pipelines

Generators chain together: each step receives the previous generator and processes one item at a time. I use this pattern to read large files: the line comes in, is filtered, transformed, and only then the next one is read:

def sem_vazias(linhas):
    for linha in linhas:
        if linha.strip():
            yield linha


def numerar(linhas):
    for numero, linha in enumerate(linhas, start=1):
        yield f"{numero}: {linha}"


texto = ["primeira", "", "segunda", "  ", "terceira"]
print(list(numerar(sem_vazias(texto))))

['1: primeira', '2: segunda', '3: terceira']

A generator can only be traversed once

Once exhausted, the generator does not return. This is the most common catch:

g = (n for n in range(3))
print(list(g), list(g))

[0, 1, 2] []

If you need to iterate twice, store it in a list or recreate the generator.

In batches

Since Python 3.12, itertools.batched divides any iterable into fixed-size groups, which is useful for sending data to an API in chunks:

from itertools import batched

print(list(batched("abcdefg", 3)))

[('a', 'b', 'c'), ('d', 'e', 'f'), ('g',)]