Chapter 48, Advanced
Performance and profiling
The golden rule: measure before optimizing. Almost always the bottleneck is in a different place than you imagine.
Code for this chapter: avancado/cap48_desempenho.py
Measure a snippet with timeit
The timeit runs a snippet multiple times and reduces noise. It answers "which of the two forms is faster?". Here, fetching an item in a list (scans everything, O(n)) versus fetching in a set (hash table, O(1) on average):
import timeit
setup = "dados = list(range(10_000)); conjunto = set(dados)"
na_lista = timeit.timeit("9_999 in dados", setup=setup, number=2_000)
no_conjunto = timeit.timeit("9_999 in conjunto", setup=setup, number=2_000)
print("conjunto mais rápido:", no_conjunto < na_lista)
conjunto mais rápido: True
I do not show the times because they change from machine to machine. The relationship between them is what matters, and it only grows with the size of the data.
Find the bottleneck with cProfile
When the entire program is slow, no one knows where. The cProfile shows how much time each function consumes. The columns that matter: ncalls (how many calls), tottime (time within the function, not counting internal calls) and cumtime (total time, counting everything it called):
import cProfile
import io
import pstats
def lento():
return sum(i * i for i in range(200_000))
def principal():
for _ in range(5):
lento()
perfil = cProfile.Profile()
perfil.enable()
principal()
perfil.disable()
relatorio = io.StringIO()
pstats.Stats(perfil, stream=relatorio).sort_stats("cumulative").print_stats(5)
print(relatorio.getvalue())
To profile an entire script, without changing the code:
python -m cProfile -o perfil.out meu_script.py
python -m pstats perfil.out
The algorithm wins over micro-optimization
Replacing for with comprehension gains a few percent. Replacing an O(n²) algorithm with O(n) gains orders of magnitude. See duplicate detection:
def tem_duplicado_lento(itens):
for i, a in enumerate(itens):
for b in itens[i + 1:]:
if a == b:
return True
return False
def tem_duplicado_rapido(itens):
return len(set(itens)) != len(itens)
amostra = list(range(2_000))
print(tem_duplicado_lento(amostra) == tem_duplicado_rapido(amostra))
t_lento = timeit.timeit(lambda: tem_duplicado_lento(amostra), number=3)
t_rapido = timeit.timeit(lambda: tem_duplicado_rapido(amostra), number=3)
print("a versão com set é mais rápida:", t_rapido < t_lento)
True
a versão com set é mais rápida: True
Memory
To measure memory consumption, the tracemalloc records the peak. The generator shows the advantage of not materializing the entire sequence:
import tracemalloc
def pico(funcao):
tracemalloc.start()
funcao()
_, maximo = tracemalloc.get_traced_memory()
tracemalloc.stop()
return maximo
pico_lista = pico(lambda: sum([n for n in range(200_000)]))
pico_gerador = pico(lambda: sum(n for n in range(200_000)))
print("gerador usa menos memória:", pico_gerador < pico_lista)
gerador usa menos memória: True
What I check before tweaking performance
| Symptom | Suspect | Tool |
|---|---|---|
| The whole program is slow | Algorithm, or I/O wait | cProfile |
| A specific snippet seems slow | Equivalent alternatives | timeit |
| High memory consumption | Complete lists instead of generators | tracemalloc |
| Program start is slow | Heavy imports | python -X importtime meu_script.py |
| Slow numerical calculation | Python loop over numbers | NumPy or another C library |
Order of actions
First, the algorithm and data structures. Then, avoid repeated work (
functools.cache, move calculations outside the loop). Next, replace pure Python work with C libraries. Concurrency and rewriting in another language come last, as they are the most expensive changes to maintain.
Exercise 1
Intersection without double loop
The function intersecao_lenta(a, b) is O(n by m). Write intersecao_rapida with the same result, using a set.