Memory: the right types

Where memory goes

Each column occupies linhas × tamanho do tipo. A 64-bit integer occupies 8 bytes even when it only holds values from 1 to 10. A text column repeats the same word thousands of times. Measuring with deep=True shows the real cost, including the content of the texts:

import numpy as np
import pandas as pd

pd.set_option("display.width", 170)
pd.set_option("display.max_columns", 20)

pedidos = pd.read_csv("dados/pedidos.csv", parse_dates=["data_pedido"])
grande = pd.concat([pedidos] * 250, ignore_index=True)
grande["id_pedido"] = np.arange(len(grande))
print(len(grande))
print((grande.memory_usage(deep=True) / 1024 / 1024).round(2).to_dict())

200000
{'Index': 0.0, 'id_pedido': 1.53, 'id_cliente': 1.53, 'id_produto': 1.53, 'quantidade': 1.53, 'data_pedido': 1.53, 'desconto': 1.53, 'canal': 2.25}

Reduce with care

A reduction function applies three ideas: smaller integers when the values fit, category for texts with few distinct values, and nothing in decimals that hold money (reducing float64 to float32 loses precision, and in monetary values this is a mistake):

def reduzir_memoria(tabela: pd.DataFrame) -> pd.DataFrame:
    nova = tabela.copy()
    for coluna in nova.select_dtypes("integer").columns:
        nova[coluna] = pd.to_numeric(nova[coluna], downcast="integer")
    for coluna in nova.select_dtypes(include=["object", "str"]).columns:
        if nova[coluna].nunique() / len(nova) < 0.5:
            nova[coluna] = nova[coluna].astype("category")
    return nova


menor = reduzir_memoria(grande)
antes = grande.memory_usage(deep=True).sum()
depois = menor.memory_usage(deep=True).sum()
print(menor.dtypes.astype(str).to_dict())
print("usa menos da metade da memória:", depois < 0.5 * antes)
restaurada = menor.astype(grande.dtypes.to_dict())
pd.testing.assert_frame_equal(grande, restaurada)
print("os valores são os mesmos")

{'id_pedido': 'int32', 'id_cliente': 'int8', 'id_produto': 'int8', 'quantidade': 'int8', 'data_pedido': 'datetime64[us]', 'desconto': 'float64', 'canal': 'category'}
usa menos da metade da memória: True
os valores são os mesmos

To prove that no value has changed, I convert the reduced table back to the original types and compare with assert_frame_equal. It is not enough to pass check_dtype=False: the assert_frame_equal still compares the container type, and a column category against a text one fails, even with the same values. The id_pedido dropped to int32, the id_cliente, the id_produto and the quantidade to int8, and the canal became category.

Be careful with automatic reduction

The downcast chooses the smallest type that fits the current values. If tomorrow a id_cliente larger than the limit arrives, a calculation with it can overflow (in NumPy, the integer wraps around without warning, as we saw in chapter 7 of the NumPy course). I reduce the types at the end of the preparation, to store and analyze, and not in the middle of a flow that is still receiving new data.

Read already with the right types

The best place to save is in the reading: with usecols you only read the columns you will use, and with dtype you give the right type from the start, without going through the expensive first:

from pathlib import Path

Path("saida").mkdir(exist_ok=True)
grande.to_csv("saida/pedidos_grande.csv", index=False)

padrao = pd.read_csv("saida/pedidos_grande.csv", usecols=["id_cliente", "quantidade", "canal"])
enxuto = pd.read_csv(
    "saida/pedidos_grande.csv",
    usecols=["id_cliente", "quantidade", "canal"],
    dtype={"id_cliente": "int16", "quantidade": "int8", "canal": "category"},
)
print(padrao.dtypes.astype(str).to_dict())
print(enxuto.dtypes.astype(str).to_dict())
print("a leitura enxuta usa menos de um quinto:", enxuto.memory_usage(deep=True).sum() < padrao.memory_usage(deep=True).sum() / 5)

{'id_cliente': 'int64', 'quantidade': 'int64', 'canal': 'str'}
{'id_cliente': 'int16', 'quantidade': 'int8', 'canal': 'category'}
a leitura enxuta usa menos de um quinto: True

Arrow types

For a read that already returns columns in the format of Arrow (which natively represents missing values and compresses well), there is the dtype_backend="pyarrow":

arrow = pd.read_csv("saida/pedidos_grande.csv", nrows=1000, dtype_backend="pyarrow")
print(arrow.dtypes.astype(str).to_dict())

{'id_pedido': 'int64[pyarrow]', 'id_cliente': 'int64[pyarrow]', 'id_produto': 'int64[pyarrow]', 'quantidade': 'int64[pyarrow]', 'data_pedido': 'string[pyarrow]', 'desconto': 'double[pyarrow]', 'canal': 'string[pyarrow]'}

The types end in [pyarrow]. They are a good option for data with many missing values and text, but some less common operations are still faster with classic types. Measure with your data before adopting. Type