Categories and ranges

The `category` type

Instead of storing the word Engenharia one hundred thousand times, the category stores one category table and, for each row, only the number of the category. The memory difference is significant, and the values are the same:

import numpy as np
import pandas as pd

pd.set_option("display.width", 170)
pd.set_option("display.max_columns", 20)

departamentos = ["Engenharia", "Marketing", "Finanças", "Ciência de Dados", "RH"]
grande = pd.Series(np.random.default_rng(0).choice(departamentos, 100_000))
como_categoria = grande.astype("category")

print(grande.dtype, como_categoria.dtype)
print(bool(como_categoria.memory_usage(deep=True) < 0.2 * grande.memory_usage(deep=True)))
print(como_categoria.cat.categories.tolist())
print(como_categoria.cat.codes.head(3).tolist())

str category
True
['Ciência de Dados', 'Engenharia', 'Finanças', 'Marketing', 'RH']
[4, 0, 2]

The category table is in .cat.categories, and the numbers per row in .cat.codes. The gain is greater the fewer distinct values and more rows there are. For a column of identifiers (almost all different), the category does not help.

Ordered categories

An ordered category knows that , and therefore allows correct comparisons and orderings. Sorting common text is sorting in alphabetical order, and alphabetical order is rarely the order of business ( Iniciante < Pleno < Sêniorwould come beforeAlto would be between the two): Baixo, and Médio keeps the

niveis = pd.CategoricalDtype(["Iniciante", "Pleno", "Sênior"], ordered=True)
equipe = pd.Series(["Sênior", "Iniciante", "Pleno", "Sênior"]).astype(niveis)

print(equipe.sort_values().tolist())
print((equipe >= "Pleno").tolist())
print(equipe.max(), equipe.value_counts(sort=False).to_dict())

['Iniciante', 'Pleno', 'Sênior', 'Sênior']
[True, False, True, True]
Sênior {'Iniciante': 1, 'Pleno': 1, 'Sênior': 2}

The value_counts(sort=False) order of the categories , and not the order of frequency.divides a number into ranges of

Ranges: `cut` and `qcut`

The pd.cut limits that you choose divides into ranges with. The pd.qcut the same number of elements (quartiles, deciles), and the limits come from the data: The ranges of the

funcionarios = pd.read_csv("dados/funcionarios_100.csv").drop_duplicates("id_funcionario", keep="last")
salario = pd.to_numeric(funcionarios["salario"].str.replace("R$ ", "", regex=False))

faixas = pd.cut(salario, bins=[0, 5000, 8000, np.inf], labels=["Baixa", "Média", "Alta"])
quartis = pd.qcut(salario, q=4, labels=["Q1", "Q2", "Q3", "Q4"])
print(faixas.value_counts().to_dict())
print(quartis.value_counts().sort_index().to_dict())
print(int(faixas.isna().sum()), int(salario.isna().sum()))

{'Alta': 60, 'Média': 29, 'Baixa': 4}
{'Q1': 24, 'Q2': 23, 'Q3': 23, 'Q4': 23}
7 7

The tracks of cut have different sizes (the limits are yours), and those of qcut have sizes almost equal. Missing salaries have no range (NaN), and therefore the counts sum to less than the total.

The limits of `cut`

By default, the ranges are closed on the right: (0, 5000] includes 5000 and excludes 0. With right=False, they become [0, 5000). A value exactly on the limit falls into different ranges depending on this choice, and this is a common mistake:

valor = pd.Series([5000])
print(pd.cut(valor, [0, 5000, 8000], labels=["A", "B"]).tolist())
print(pd.cut(valor, [0, 5000, 8000], labels=["A", "B"], right=False).tolist())

['A']
['B']

Group by category

In Pandas 3, the groupby by a column category only shows the categories that appear in the data (observed=True is the default). To also list the empty (useful in reports that need all rows, even the zeroed ones), request observed=False:

nivel = pd.Series(pd.Categorical(["Pleno", "Sênior", "Pleno"], categories=["Iniciante", "Pleno", "Sênior"]))
horas = pd.Series([10, 20, 30])
print(horas.groupby(nivel).sum().to_dict())
print(horas.groupby(nivel, observed=False).sum().to_dict())

{'Pleno': 40, 'Sênior': 20}
{'Iniciante': 0, 'Pleno': 40, 'Sênior': 20}

What a category does not accept

A column category only accepts values that are in the list of categories. Assigning another is an error (and this is a benefit: it validates the data):

try:
    nivel[0] = "Estagiário"
except TypeError as erro:
    print("TypeError:", str(erro)[:56])

TypeError: Cannot setitem on a Categorical with a new category (Est

And, when joining two columns of category with different categories, the result loses the type:

a = pd.Series(pd.Categorical(["x", "y"]))
b = pd.Series(pd.Categorical(["z"]))
print(pd.concat([a, b]).dtype)

str