Missing data

Find

The isna() returns a boolean mask, and summing the mask counts the missing values by column. The isnull() is the same method with another name. The average of the mask gives the proportion:

import pandas as pd

pd.set_option("display.width", 170)
pd.set_option("display.max_columns", 20)
df = pd.read_csv("dados/funcionarios_15.csv")

print(df.isna().sum()[lambda s: s > 0])
print((df.isna().mean() * 100).round(1)[lambda s: s > 0])

departamento    1
cidade          2
idade           1
salario         2
dtype: int64
departamento     6.7
cidade          13.3
idade            6.7
salario         13.3
dtype: float64

The danger of `dropna()` without arguments

Discard any row that has any missing value seems simple. But just one problematic column can delete rows that were perfect in all others:

print(len(df), len(df.dropna()))
print(len(df.dropna(subset=["salario"])))
print(len(df.dropna(subset=["idade", "cidade"], how="all")))

15 9
13
15

The dropna() discarded 6 out of 15 employees (40% of the data) because of gaps in different columns. The subset restricts the decision to the columns that matter for the your question: if the analysis is about salary, only the missing salaries justify discarding. And how="all" only discards the row if all the columns of the subset are empty.

Fill: mean or median

The fillna replaces the missing values with a value. For numeric columns, the median is almost always the safest choice, because extreme values pull the mean away from what is typical, and the median hardly moves:

print(df["idade"].fillna(df["idade"].median()).tolist())

salarios = pd.Series([3000, 3200, 3100, 2900, 50000, None])
print(salarios.mean(), salarios.median())
print(salarios.fillna(salarios.mean()).iloc[-1], salarios.fillna(salarios.median()).iloc[-1])

[24.0, 25.0, 26.0, 27.0, 28.0, 29.0, 27.0, 31.0, 32.0, 33.0, 24.0, 25.0, 26.0, 27.0, 28.0]
12440.0 3100.0
12440.0 3100.0

A single salary of 50 thousand makes the average go from 12 thousand, and filling the missing value with it would invent a salary that no one has. The median (3,100) is a much more honest estimate.

For text columns, the filling is an explicit label, not a calculation: In data that hasorder

print(df["cidade"].fillna("Não informada").tolist())

['São Paulo', 'Belo Horizonte', 'Porto Alegre', 'Rio de Janeiro', 'Não informada', 'São Paulo', 'Belo Horizonte', 'Porto Alegre', 'Rio de Janeiro', 'Curitiba', 'São Paulo', 'Não informada', 'Porto Alegre', 'Rio de Janeiro', 'Curitiba']

Fill forward and backward

(a series over time), the previous value ( ) or the next one ( ) is usually a good estimate. The oldffillwasbfillremoved fillna(method="ffill") in Pandas 3: Filling erases the evidence

medicao = pd.Series([20.0, None, None, 23.0, None])
print(medicao.ffill().tolist())
print(medicao.bfill().tolist())

try:
    medicao.fillna(method="ffill")
except TypeError as erro:
    print("TypeError:", erro)

[20.0, 20.0, 20.0, 23.0, 23.0]
[20.0, 23.0, 23.0, 23.0, nan]
TypeError: NDFrame.fillna() got an unexpected keyword argument 'method'

Keep the information that was missing

that the data did not exist. If the absence can be informative (those who did not report their age may be different from the others), keep an indicator column before filling: You will see, in much material, . This pattern

tratado = df.copy()
tratado["idade_informada"] = tratado["idade"].notna()
tratado["idade"] = tratado["idade"].fillna(tratado["idade"].median())
print(tratado.loc[[5, 6, 7], ["nome", "idade", "idade_informada"]])

             nome  idade  idade_informada
5   Felipe Araújo   29.0             True
6  Gabriela Nunes   27.0            False
7  Henrique Costa   31.0             True

The `inplace=True` and chaining

no longer works df["idade"].fillna(0, inplace=True)in Pandas 3: the is a temporary copy, and the only alters it. Pandas warns, and the table remains as it was: df["idade"] The age remains missing (1 value), and a warning has been issued. The correct way is inplace to reassign

import warnings

teste = df.copy()
with warnings.catch_warnings(record=True) as avisos:
    warnings.simplefilter("always")
    teste["idade"].fillna(0, inplace=True)
print(int(teste["idade"].isna().sum()), [a.category.__name__ for a in avisos])

1 ['ChainedAssignmentError']

. I generally avoid the as it does not save memory in practice and breaks method chaining (chapter 30).: df["idade"] = df["idade"].fillna(0)Operation inplace Result

How the missing values behave

import numpy as np

print(np.nan == np.nan, pd.Series([1.0, None, 3.0]).sum(), pd.Series([1.0, None, 3.0]).sum(skipna=False))

False 4.0 nan