Chapter 2, Environment
Installing, importing, and getting to know the data
Two commands get Pandas ready. The rest of the chapter presents the dataset that will accompany you throughout the Basic level, with its flaws.
Install and import
Pandas 3 requires Python 3.11 or higher. Within a virtual environment (like in the NumPy course):
uv add pandas pyarrow openpyxl
The pyarrow accelerates texts and supports the Parquet format, and the openpyxl reads and writes Excel files. For Pandas, the nickname pd is a convention as strong as np is for NumPy: all documentation and every answer on the internet assume it.
import pandas as pd
print(int(pd.__version__.split(".")[0]) >= 3)
True
To make the terminal display wide tables without breaking lines, I adjust the display width. It's just the presentation: it doesn't change any data.
pd.set_option("display.width", 170)
pd.set_option("display.max_columns", 20)
The Basic level dataset
From chapter 4 to 15, I use the same dataset, on purpose: you never relearn what each column means, only what to do with it. There are 15 employees of a company, in dados/funcionarios_15.csv. The first honest read:
df = pd.read_csv("dados/funcionarios_15.csv")
print(df.head(6))
id_funcionario nome departamento cidade idade salario experiencia nota_desempenho projetos_concluidos data_admissao
0 101 Ana Souza Engenharia São Paulo 24.0 4500 1 4 2 2021-01-10
1 102 Bruno Lima Marketing Belo Horizonte 25.0 R$ 4650 2 5 3 2022-02-11
2 103 Carla Mendes engenharia Porto Alegre 26.0 4800 3 6 4 2023-03-12
3 104 Diego Rocha Ciência de Dados Rio de Janeiro 27.0 NaN 4 7 5 2024-04-13
4 105 Elisa Fernandes RH NaN 28.0 5100 5 8 6 2021-05-14
5 106 Felipe Araújo MARKETING São Paulo 29.0 5250 6 9 7 2022-06-15
| Column | What it is | Expected type |
|---|---|---|
id_funcionario | Employee identifier | Integer |
nome | Full name | Text |
departamento | Engineering, Marketing, Finance, Data Science, or HR | Text |
cidade | Where they work | Text |
idade | Age in years | Number |
salario | Monthly salary, in reais | Number |
experiencia | Years of experience | Integer |
nota_desempenho | Rating from 4 to 9 | Integer |
projetos_concluidos | Number of projects | Integer |
data_admissao | Start date | Date |
The problems that are hidden
This dataset has flaws on purpose, and none of them cause a read error: Pandas loads everything without complaint, and the calculations run. You only discover them if you look. They are the subject of chapters 9 and 10 (the rows are counted from 0, like in the Pandas index):
| Problem | Where is |
|---|---|
Salary written as text (R$ 4650) | Lines 1 and 7, and this transforms the entire column into text |
| Missing salary | Lines 3 and 9 |
| Missing department | Line 8 |
Dirty text in department (engenharia, MARKETING) | Lines 2 and 5 |
| Missing city | Lines 4 and 11 |
| Missing age | Line 6 |
Repeated employee (same id_funcionario, different fields) | Line 14 repeats line 0 |
A quick count shows the missing without needing a cleaning line:
print(df.isna().sum())
id_funcionario 0
nome 0
departamento 1
cidade 2
idade 1
salario 2
experiencia 0
nota_desempenho 0
projetos_concluidos 0
data_admissao 0
dtype: int64
Note that the salario shows only 2 missing, not 4: the two lines with R$ 4650 and R$ 5550 are not empty for Pandas, they are texts. This is exactly the type of problem that an account that "runs without error" hides.
The data files
The CSVs were generated by a deterministic script (
dados/gerar_dados.py), so the same execution always produces the same files. If you accidentally delete one, run the script from within the folderdados/.
Exercise 1
Know before analyzing
Load dados/funcionarios_15.csv and check, with assert, that it has 15 lines and 10 columns, and that the id_funcionario 101 appears twice.