Skip to content

    Chapter 28, Advanced

    Cache

    Calculating or fetching again what hasn't changed is wasteful. The cache holds an answer for a time, and the hard question is the other: when does it stop being valid.

    A cache with validity (TTL)

    The idea: store the result along with the moment it was calculated, and reuse it while it hasn't passed the TTL (time to live, the validity period). After that, recalculate. I write the clock as a parameter, for the test to control the time and not need to wait for real:

    avancado/cap28_cache.pylinhas 10 a 61
    import threading
import time
from typing import Callable


class CacheTTL:
    def __init__(self, ttl: float, relogio: Callable[[], float] = time.monotonic):
        self.ttl = ttl
        self.relogio = relogio
        self._dados: dict[str, tuple[float, object]] = {}
        self._trava = threading.Lock()

    def obter_ou_calcular(self, chave: str, calcular: Callable[[], object]) -> object:
        with self._trava:
            agora = self.relogio()
            guardado = self._dados.get(chave)
            if guardado is not None and agora - guardado[0] < self.ttl:
                return guardado[1]
            valor = calcular()
            self._dados[chave] = (agora, valor)
            return valor

    def invalidar(self, chave: str) -> None:
        with self._trava:
            self._dados.pop(chave, None)


class RelogioFalso:
    def __init__(self):
        self.agora = 0.0

    def __call__(self) -> float:
        return self.agora


relogio = RelogioFalso()
cache = CacheTTL(ttl=60, relogio=relogio)
calculos = []


def cotacao_cara() -> float:
    calculos.append(1)
    return 5.43


print([cache.obter_ou_calcular("usd", cotacao_cara) for _ in range(3)], len(calculos))
relogio.agora = 59.9
cache.obter_ou_calcular("usd", cotacao_cara)
print(len(calculos))
relogio.agora = 60.1
cache.obter_ou_calcular("usd", cotacao_cara)
print(len(calculos))
    Output
    [5.43, 5.43, 5.43] 1
1
2

    Three calls, one calculation only. At 59.9 seconds, the cache is still valid. At 60.1, it expired and recalculated. Three details of the code matter: the clock is the time.monotonic (which does not go back or skip when the system clock is adjusted), there is a lock (the server handles requests in multiple threads, and a global variable without a lock corrupts the cache), and there is a invalidar, because sometimes you know that the data has changed and won't wait for the TTL.

    In a route

    avancado/cap28_cache.pylinhas 66 a 92
    from fastapi import FastAPI
from fastapi.testclient import TestClient

app = FastAPI()
chamadas_externas = []


def buscar_cotacao_externa() -> float:
    chamadas_externas.append(1)
    return 5.43


@app.get("/cotacao")
def cotacao():
    valor = cache.obter_ou_calcular("usd", buscar_cotacao_externa)
    return {"usd": valor}


relogio.agora = 1000.0
cache.invalidar("usd")
cliente = TestClient(app)
for _ in range(3):
    cliente.get("/cotacao")
print("chamadas à API externa:", len(chamadas_externas))
relogio.agora = 1061.0
cliente.get("/cotacao")
print("depois que o TTL passou:", len(chamadas_externas))
    Output
    chamadas à API externa: 1
depois que o TTL passou: 2

    `lru_cache` for pure functions

    For a function whose result depends only on the arguments (no I/O, no clock), the functools.lru_cache of Python solves it:

    avancado/cap28_cache.pylinhas 97 a 107
    from functools import lru_cache


@lru_cache(maxsize=128)
def fatorial(n: int) -> int:
    return 1 if n < 2 else n * fatorial(n - 1)


fatorial(50)
fatorial(50)
print(fatorial.cache_info())
    Output
    CacheInfo(hits=1, misses=50, maxsize=128, currsize=50)

    The cache_info shows the hits (hits) and the misses (misses). But, beware of the trap: the lru_cache is not valid, and should not to be used in a function that returns data that changes (a database query) or in a function that depends on who is logged in: the result of the first user would be served to all others.

    The HTTP cache: let the client store it

    Another form of cache is to ask the client (browser, CDN) to store the response. The header Cache-Control indicates how long. And the ETag allows for a cheap revalidation: the server returns a "fingerprint" of the content, and the next time the client asks "has it changed since this fingerprint?". If it hasn't changed, the server responds 304 Not Modified without a body:

    avancado/cap28_cache.pylinhas 112 a 135
    import hashlib
import json

from fastapi import Request, Response

CATALOGO = {"itens": ["caneta", "caderno", "régua"]}


@app.get("/catalogo")
def catalogo(request: Request, response: Response):
    corpo = json.dumps(CATALOGO, sort_keys=True).encode()
    etag = '"' + hashlib.sha256(corpo).hexdigest()[:16] + '"'
    if request.headers.get("if-none-match") == etag:
        return Response(status_code=304, headers={"ETag": etag})
    response.headers["ETag"] = etag
    response.headers["Cache-Control"] = "public, max-age=60"
    return CATALOGO


primeira = cliente.get("/catalogo")
etag = primeira.headers["etag"]
print(primeira.status_code, primeira.headers["cache-control"], len(primeira.content) > 0)
revalidada = cliente.get("/catalogo", headers={"If-None-Match": etag})
print(revalidada.status_code, revalidada.content)
    Output
    200 public, max-age=60 True
304 b''

    On the second call, the client sent the fingerprint it had, the server saw that nothing changed and responded 304 with an empty body: saved the transfer, although it still made the network trip.

    The problems of cache

    ProblemWhat happensHow to deal with
    Old dataThe cache serves what has already changedShort TTL, or invalidate when the data changes
    Multiple processesEach --workers has its own in-memory cache, and they divergeShared cache (Redis or equivalent)
    Data by userThe cache delivers data from one user to anotherInclude the user in the cache key of the cache
    Memory overflowThe cache grows without limitSize limit (maxsize) and removal of the oldest
    AvalancheThe cache expires and hundreds of requests recalculate at the same timeThe lock from the example above (each key calculates once)

    "There are only two hard things in computer science"

    The famous phrase is that they are cache invalidation and naming things. It is true: storing is easy, and knowing when it has become invalid is the real work. I start without cache, measure where the bottleneck is (chapter 48 of the Python course, "Performance and profiling"), and only then choose what to store, with the shortest TTL that the problem allows.

    Exercise 1

    Invalidate when the data changes

    Use the CacheTTL to store perfil:ana. After an "update", call invalidar("perfil:ana") and check that the next access recalculates. Use a custom fake clock for the test.