NumPy and pandas

A table is a set of arrays with names

A DataFrame is a table where each column is, underneath, a NumPy array (each column with its type), and the rows and columns have labels. It is exactly what the student project in chapter 41 teaches to feel: several columns of the same size describing the same people.

uv sync --group ecossistema

import numpy as np
import pandas as pd

df = pd.DataFrame(
    {"nome": ["Ana", "Bia", "Caio", "Davi"], "nota": [7.5, 9.0, 6.0, 8.2], "faltas": [1, 0, 4, 2]}
)
print(df)
print(df.dtypes.astype(str).tolist())

   nome  nota  faltas
0   Ana   7.5       1
1   Bia   9.0       0
2  Caio   6.0       4
3  Davi   8.2       2
['str', 'float64', 'int64']

Each column has its type. In pandas 3, text columns have their own type, str, and numeric ones maintain the dtype of NumPy.

From pandas to NumPy, and back

The to_numpy() extracts the values as an array. In a single column, the result is a 1D array with the column type. In a set of numeric columns, it is a 2D matrix (and integers become decimals, according to the promotion rules of chapter 7). In a DataFrame with mixed text and numbers, NumPy can only represent everything as object, and you lose speed:

notas = df["nota"].to_numpy()
print(type(notas).__name__, notas.dtype, notas.mean().round(1))
print(df.to_numpy().dtype, df[["nota", "faltas"]].to_numpy().dtype, df[["nota", "faltas"]].to_numpy().shape)

ndarray float64 7.7
object float64 (4, 2)

The rule I follow: select only the numeric columns before converting. An array object has none of the advantages of NumPy.

To go back, the DataFrame accepts an array and the column names:

matriz = np.array([[1.0, 2.0], [3.0, 4.0]])
print(pd.DataFrame(matriz, columns=["a", "b"]))

     a    b
0  1.0  2.0
1  3.0  4.0

The array that comes back is a protected view

In pandas 3, the to_numpy() returns, whenever possible, a view of the column data, without copying (and read-only), so writing to it does not corrupt the DataFrame without you noticing:

print(notas.flags["WRITEABLE"], notas.flags["OWNDATA"])
copia_editavel = df["nota"].to_numpy().copy()
copia_editavel[0] = 0.0
print(df["nota"].iloc[0])

False False
7.5

The array is not editable (WRITEABLE false) and does not own the data (OWNDATA false). To modify, make a copy, as in chapter 10, and the DataFrame remains intact.

The most dangerous difference: label alignment

NumPy sums by position. Pandas sums by label: it aligns the indices before operating. With the same values in a different order, the result changes:

s1 = pd.Series([1, 2, 3], index=[0, 1, 2])
s2 = pd.Series([10, 20, 30], index=[2, 1, 0])
print((s1 + s2).tolist(), (s1.to_numpy() + s2.to_numpy()).tolist())

[31, 22, 13] [11, 22, 33]

Pandas summed 1 + 30, 2 + 20 and 3 + 10, because the labels 0, 1 and 2 match. NumPy summed 1 + 10, 2 + 20 and 3 + 30, position by position. Neither is "wrong", but mixing the two without realizing it is a classic source of swapped results.

The ideas of NumPy, with another name

Almost everything you learned has a version in pandas, with the advantage of labels:

df["aprovado"] = np.where(df["nota"] >= 7, "sim", "não")
print(df["aprovado"].tolist(), df[df["nota"] > 7]["nome"].tolist())

vendas = pd.DataFrame(
    {"regiao": ["sul", "norte", "sul", "norte", "sul"], "valor": [10, 5, 20, 15, 30]}
)
print(vendas.groupby("regiao")["valor"].sum().to_dict())

['sim', 'sim', 'não', 'sim'] ['Ana', 'Bia', 'Davi']
{'norte': 20, 'sul': 60}

The groupby is the labeled version of what chapter 31 did with argsort and reduceat, only it accepts any type of key (here, text).

When to stay in NumPy, when to go to pandas

I stay in NumPy when the data is numbers of the same type and shape (images, signals, feature matrices) and speed matters. I go to pandas when there are columns of different types, labels, dates, missing data, and joins between tables. And I convert between the two without ceremony, at the point where the nature of the data changes.