Chapter 6, Basic
Inspect before analyzing
Never analyze right after loading. Read the data first. Each function below answers a specific question, and together they reveal the problems that a "running without error" calculation would hide.
The questions and the functions
| Function | The question it answers |
|---|---|
.head() | What do the data look like? |
.tail() | What are the last few rows like? |
.sample() | What is a random slice like? |
.shape | How many rows and columns are there? |
.dtypes | What type of data is in each column? |
.info() | What is the structure, and where are the missing values? |
.describe() | What are the basic statistics like? |
import pandas as pd
pd.set_option("display.width", 170)
pd.set_option("display.max_columns", 20)
df = pd.read_csv("dados/funcionarios_15.csv")
print(df.shape)
print(df.head(3))
print(df.tail(2))
print(df.sample(3, random_state=1))
(15, 10)
id_funcionario nome departamento cidade idade salario experiencia nota_desempenho projetos_concluidos data_admissao
0 101 Ana Souza Engenharia São Paulo 24.0 4500 1 4 2 2021-01-10
1 102 Bruno Lima Marketing Belo Horizonte 25.0 R$ 4650 2 5 3 2022-02-11
2 103 Carla Mendes engenharia Porto Alegre 26.0 4800 3 6 4 2023-03-12
id_funcionario nome departamento cidade idade salario experiencia nota_desempenho projetos_concluidos data_admissao
13 114 Nelson Ribeiro Ciência de Dados Rio de Janeiro 27.0 6450 6 5 5 2022-05-23
14 101 Ana Souza RH Curitiba 28.0 6600 7 6 6 2023-06-24
id_funcionario nome departamento cidade idade salario experiencia nota_desempenho projetos_concluidos data_admissao
3 104 Diego Rocha Ciência de Dados Rio de Janeiro 27.0 NaN 4 7 5 2024-04-13
7 108 Henrique Costa Finanças Porto Alegre 31.0 R$ 5550 8 5 9 2024-08-17
6 107 Gabriela Nunes Marketing Belo Horizonte NaN 5400 7 4 8 2023-07-16
The head() always seems clean, because problems often lie deeper in the file. The sample() gives a more honest first impression of messy data, and the random_state fixes the sampling so the result is reproducible.
`info()`: the structure and the missing
The info() shows, by column, how many non-missing values there are and the type. Comparing with the total number of rows reveals where there are gaps:
df.info(memory_usage=False)
<class 'pandas.DataFrame'>
RangeIndex: 15 entries, 0 to 14
Data columns (total 10 columns):
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 id_funcionario 15 non-null int64
1 nome 15 non-null str
2 departamento 14 non-null str
3 cidade 13 non-null str
4 idade 14 non-null float64
5 salario 13 non-null str
6 experiencia 15 non-null int64
7 nota_desempenho 15 non-null int64
8 projetos_concluidos 15 non-null int64
9 data_admissao 15 non-null str
dtypes: float64(1), int64(4), str(5)
Each column should have 15 values. The departamento has 14, the cidade has 13, the idade has 14 and the salario has 13: the missing ones. And notice the type of the salario, which is str, and not a number.
`describe()` and the salary that disappeared
The describe() summarizes only the numericcolumns, by default:
print(df.describe().round(2))
id_funcionario idade experiencia nota_desempenho projetos_concluidos
count 15.00 14.00 15.00 15.00 15.00
mean 107.07 27.50 4.27 6.20 5.67
std 4.37 2.88 2.25 1.74 2.85
min 101.00 24.00 1.00 4.00 2.00
25% 103.50 25.25 2.50 5.00 3.50
50% 107.00 27.00 4.00 6.00 5.00
75% 110.50 28.75 6.00 7.50 7.50
max 114.00 33.00 8.00 9.00 11.00
Look at the columns of the table: the salario does not appear. It is not a Pandas error, it is a symptom: the column is text (because of the R$ 4650), and therefore does not enter the statistics. When a column you expected to see disappears from the describe(), the first suspect is its type.
Another observation: the id_funcionario appears, because it is a number, but an identifier is not a quantity, and its average means nothing. The describe doesn't know this, and who does is you.
To include the text columns, ask:
print(df.describe(include="str"))
nome departamento cidade salario data_admissao
count 15 14 13 13 15
unique 14 7 5 13 15
top Ana Souza Marketing São Paulo 4500 2021-01-10
freq 2 3 3 1 1
Count the values
Two questions come up all the time: what values exist, and how many times each one appears. The value_counts answers both, and the dropna=False includes the missing ones in the count (which, by default, are left out):
print(df["departamento"].value_counts())
print(df["departamento"].value_counts(dropna=False).head(3))
print(df["cidade"].nunique(), df["cidade"].unique().tolist())
departamento
Marketing 3
RH 3
Engenharia 2
Ciência de Dados 2
Finanças 2
engenharia 1
MARKETING 1
Name: count, dtype: int64
departamento
Marketing 3
RH 3
Engenharia 2
Name: count, dtype: int64
5 ['São Paulo', 'Belo Horizonte', 'Porto Alegre', 'Rio de Janeiro', nan, 'Curitiba']
The departamento shows seven groups that should be five: engenharia (with spaces) and Engenharia are the same department, and MARKETING and Marketing too. No error was raised, but any grouping by department would yield incorrect results. That's why it is inspected before analyzing.
An inspection script
These four lines, in this order, are what I run on every new dataset:
print(df.shape)
print(df.dtypes.value_counts())
print(df.isna().sum()[lambda s: s > 0])
print(df.duplicated().sum(), df.duplicated("id_funcionario").sum())
(15, 10)
str 5
int64 4
float64 1
Name: count, dtype: int64
departamento 1
cidade 2
idade 1
salario 2
dtype: int64
0 1
The size, the types, where values are missing, and if there are repetitions (whole or just by identifier). The two numbers at the end already point to a problem: no row is entirely repeated, but there is a one repeated identifier.
Performance and honesty
The
head()and thedescribe()are fast and give a sense of control. Thesample(), thevalue_counts()and theisna().sum()are the ones that find the problems. If you only have time to run three things, run these.
Exercise 1
A quality summary
Write resumo_qualidade(tabela), that returns a dictionary with linhas, colunas, ausentes (the total number of missing values) and ids_repetidos (how many ids appear repeated), and check for df.