How Pandas stores data

One column, one type

In NumPy, the entire array has one dtype. In Pandas, each column has its own. This is what allows a table to mix numbers, texts, and dates:

import numpy as np
import pandas as pd

pd.set_option("display.width", 170)
df = pd.read_csv("dados/funcionarios_15.csv")
print(df.dtypes)

id_funcionario           int64
nome                       str
departamento               str
cidade                     str
idade                  float64
salario                    str
experiencia              int64
nota_desempenho          int64
projetos_concluidos      int64
data_admissao              str
dtype: object

The dtypes is the first thing I look at in any dataset. Three columns deserved attention: the salario is str (text), and not a number, which explains why it does not appear in a describe(); the idade is float64, although ages are integers; and the data_admissao is str, and not a date.

Why age became decimal

NumPy integers do not have a representation for "missing". When an integer column gains a missing value, Pandas converts it to decimal, because the NaN is a decimal value. You can see the rule happening:

print(pd.Series([24, 25, 26]).dtype)
print(pd.Series([24, 25, None]).dtype)
print(pd.Series([24, 25, None], dtype="Int64").dtype)

int64
float64
Int64

The third case uses the nullable Int64 (with I uppercase), which holds integers and accepts missing values without converting. Chapter 16 covers it in depth.

Text: `str` in Pandas 3, `object` before

Until Pandas 2, text columns were of type object, which holds any Python object and is slow and heavy. Pandas 3 has its own type, str, for text. A object only appears when the column truly mixes types:

print(pd.Series(["a", "b", "c"]).dtype)
print(pd.Series(["a", 1, 2.5]).dtype)
print(df["nome"].dtype, df["departamento"].dtype)

str
object
str str

If you read older material that says "text columns are object", know that this was true in previous versions. In this course, what matters is the same idea: a column that should be a number and appears as text (or object) is a cleaning problem, not a Pandas error.

The index is also an object

All Series and all DataFrame have an index, which labels the rows. By default, it is a sequence from 0 to n-1, but it can be anything: ids, dates, names. The operations align by label, not by position, and that is what most differentiates Pandas from NumPy:

a = pd.Series([10, 20, 30], index=["x", "y", "z"])
b = pd.Series([1, 2, 3], index=["z", "y", "x"])
print((a + b).to_dict())
print((a.to_numpy() + b.to_numpy()).tolist())

{'x': 13, 'y': 22, 'z': 31}
[11, 22, 33]

Pandas summed x with x (10 + 3), y with y (20 + 2) and z with z (30 + 1), despite the different order. NumPy summed position with position. Neither is wrong, but mixing the two models without realizing it is a classic source of swapped results.

How much does it cost

memory_usage(deep=True) shows the bytes of each column, including the content of the texts. Without the deep=True, it only counts the pointers, and underestimates the cost of the texts:

memoria = df.memory_usage(deep=True)
print(memoria.index.tolist())
print(bool(memoria["nome"] > memoria["idade"]))

['Index', 'id_funcionario', 'nome', 'departamento', 'cidade', 'idade', 'salario', 'experiencia', 'nota_desempenho', 'projetos_concluidos', 'data_admissao']
True

A text column costs more than a numeric one of the same size, and that is why, in large datasets, the right type for each column decides if the data fits in memory. Chapter 29 shows how to reduce this cost.