Chapter 33, Advanced
Data testing and validation
A program has tests. A data analysis should too: of the code that cleans and of the data that arrives. Here are both things, and the tool that Pandas offers to compare tables.
Comparing tables: `pandas.testing`
A assert a == b with tables does not work (compares element by element and returns a table of booleans). The assert_frame_equal compares everything (values, types, index, column names), and says where the difference is. By default, decimals are compared with tolerance, and you can relax or tighten each thing:
import numpy as np
import pandas as pd
from pandas.testing import assert_frame_equal
esperado = pd.DataFrame({"a": [1, 2, 3], "b": [0.1, 0.2, 0.3]})
obtido = pd.DataFrame({"a": [1, 2, 3], "b": [0.1, 0.2, 0.30000001]})
assert_frame_equal(esperado, obtido)
print("passou, com a tolerância padrão")
try:
assert_frame_equal(esperado, obtido, check_exact=True)
except AssertionError as erro:
print("AssertionError:", str(erro).splitlines()[0])
passou, com a tolerância padrão
AssertionError: DataFrame.iloc[:, 1] (column name="b") are different
The other parameters I use the most:
inteiros = pd.DataFrame({"a": [1, 2]})
decimais = pd.DataFrame({"a": [1.0, 2.0]})
try:
assert_frame_equal(inteiros, decimais)
except AssertionError as erro:
print("AssertionError:", str(erro).splitlines()[0])
assert_frame_equal(inteiros, decimais, check_dtype=False)
ordem_a = pd.DataFrame({"x": [1, 2], "y": [3, 4]})
ordem_b = ordem_a[["y", "x"]]
assert_frame_equal(ordem_a, ordem_b, check_like=True)
print("passou sem olhar a ordem das colunas")
AssertionError: Attributes of DataFrame.iloc[:, 0] (column name="a") are different
passou sem olhar a ordem das colunas
| Parameter | What relaxes |
|---|---|
check_dtype=False | Accepts int against float |
check_like=True | Ignores the order of rows and columns |
check_exact=True | Tightens: requires exact equality in decimals |
rtol and atol | The relative and absolute tolerance (as in np.isclose) |
Validate the incoming data
Testing the code does not protect you from the file that arrives tomorrow with a new problem. A validation function describes what the data needs to be, and returns all the problems at once (and not just the first), which saves a lot of back and forth:
class ContratoViolado(ValueError):
pass
def validar_pedidos(pedidos: pd.DataFrame, clientes: pd.DataFrame) -> list[str]:
problemas = []
if pedidos["id_pedido"].duplicated().any():
problemas.append("id_pedido repetido")
for coluna in ("id_pedido", "id_cliente", "id_produto", "quantidade", "data_pedido"):
if pedidos[coluna].isna().any():
problemas.append(f"{coluna} com ausentes")
if (pedidos["quantidade"] <= 0).any():
problemas.append("quantidade não positiva")
if not pedidos["canal"].isin({"site", "loja", "app"}).all():
problemas.append("canal desconhecido")
if not pedidos["desconto"].dropna().between(0, 1).all():
problemas.append("desconto fora de 0 a 1")
orfaos = ~pedidos["id_cliente"].isin(clientes["id_cliente"])
if orfaos.any():
problemas.append(f"{int(orfaos.sum())} pedidos de clientes inexistentes")
return problemas
def exigir_contrato(pedidos: pd.DataFrame, clientes: pd.DataFrame) -> None:
problemas = validar_pedidos(pedidos, clientes)
if problemas:
raise ContratoViolado("; ".join(problemas))
clientes = pd.read_csv("dados/clientes.csv")
pedidos = pd.read_csv("dados/pedidos.csv", parse_dates=["data_pedido"])
print(validar_pedidos(pedidos, clientes))
['12 pedidos de clientes inexistentes']
The validation found the 12 requests from non-existent clients that chapter 21 discovered manually. With deliberately corrupted data, it shows several problems at once:
estragados = pedidos.copy()
estragados.loc[0, "quantidade"] = -3
estragados.loc[1, "canal"] = "telefone"
estragados.loc[2, "desconto"] = 1.5
estragados.loc[3, "id_pedido"] = estragados.loc[4, "id_pedido"]
print(validar_pedidos(estragados, clientes))
try:
exigir_contrato(estragados, clientes)
except ContratoViolado as erro:
print("ContratoViolado:", str(erro)[:60])
['id_pedido repetido', 'quantidade não positiva', 'canal desconhecido', 'desconto fora de 0 a 1', '12 pedidos de clientes inexistentes']
ContratoViolado: id_pedido repetido; quantidade não positiva; canal desconhec
Test the cleaning code
A cleanup function deserves testing like any other, and three properties apply to almost all: it any existing table. And this directly relates to the copy-on-write from chapter 27. the received table, is idempotent (cleaning twice gives the same result as cleaning once) and works on a small, hand-written table, where you know the answer:
def limpar(tabela: pd.DataFrame) -> pd.DataFrame:
return (
tabela.assign(nome=lambda d: d["nome"].str.strip().str.title())
.drop_duplicates("id")
.reset_index(drop=True)
)
def test_limpeza_com_resposta_conhecida():
entrada = pd.DataFrame({"id": [1, 1, 2], "nome": [" ana souza ", " ana souza ", "BRUNO LIMA"]})
esperado = pd.DataFrame({"id": [1, 2], "nome": ["Ana Souza", "Bruno Lima"]})
assert_frame_equal(limpar(entrada), esperado)
def test_limpeza_nao_altera_a_entrada():
entrada = pd.DataFrame({"id": [1, 2], "nome": [" a ", " b "]})
copia = entrada.copy()
limpar(entrada)
assert_frame_equal(entrada, copia)
def test_limpeza_e_idempotente():
entrada = pd.DataFrame({"id": [1, 2, 2], "nome": [" a ", " b ", " b "]})
assert_frame_equal(limpar(limpar(entrada)), limpar(entrada))
for teste in (test_limpeza_com_resposta_conhecida, test_limpeza_nao_altera_a_entrada, test_limpeza_e_idempotente):
teste()
print("passou:", teste.__name__)
passou: test_limpeza_com_resposta_conhecida
passou: test_limpeza_nao_altera_a_entrada
passou: test_limpeza_e_idempotente
In a project, these functions test_* are in files test_*.py and run with uv run pytest: the end-of-course projects do this. Here I called them directly to show the result.
What I always validate
Before any analysis: unique keys where they should be, no missing values where they cannot exist, values within reasonable ranges (positive quantity, discount between 0 and 1), known categories and referential integrity (every order has a customer). These five questions catch most of the corrupted files.
Exercise 1
A new rule
Add a rule to the validation: data_pedido it cannot be future (after 2025-12-31). Write datas_validas(pedidos), which returns the list of problems (empty if everything is correct), and check with the original table and with a future date inserted.