Pandas and NumPy

From Pandas to NumPy and back

The to_numpy() extracts the array from a Series or from a DataFrame. In a table, the result has one type: NumPy does not accept one type per column, so Pandas chooses what accommodates all. To go back, the DataFrame receives the array, and you return the labels that the array does not carry:

import numpy as np
import pandas as pd

pd.set_option("display.width", 170)
pd.set_option("display.max_columns", 20)

CANONICO = {"engenharia": "Engenharia", "marketing": "Marketing", "finanças": "Finanças",
            "ciência de dados": "Ciência de Dados", "rh": "RH"}
df = pd.read_csv("dados/funcionarios_100.csv").drop_duplicates("id_funcionario", keep="last")
df["salario"] = pd.to_numeric(df["salario"].str.replace("R$ ", "", regex=False))
df["departamento"] = df["departamento"].str.strip().str.lower().map(CANONICO)
df = df.dropna(subset=["salario", "departamento"]).reset_index(drop=True)

X = df[["experiencia", "projetos_concluidos"]].to_numpy(dtype=float)
print(X.shape, X.dtype)
de_volta = pd.DataFrame(X, index=df.index, columns=["experiencia", "projetos_concluidos"])
print(bool(de_volta.equals(df[["experiencia", "projetos_concluidos"]].astype(float))))

(90, 2) float64
True

Missing when crossing the border

A nullable type (Int64) with missing values does not fit in a NumPy integer array. The to_numpy() returns an array of objects, unless you specify the type and the value that represents the missing value:

idade = pd.Series([30, None, 28], dtype="Int64")
print(idade.to_numpy().dtype)
print(idade.to_numpy(dtype="float64", na_value=np.nan))

float64
[30. nan 28.]

NumPy functions keep the labels

Passing a Series to a NumPy function returns a Series with the same index: the labels travel. On the other hand, the np.where returns a simple array, and you rewrap (or use the where from Pandas itself):

salario = df["salario"]
log_salario = np.log(salario)
print(type(log_salario).__name__, bool(log_salario.index.equals(salario.index)))

faixa = np.where(salario > salario.median(), "acima", "abaixo")
print(type(faixa).__name__)
faixa = pd.Series(faixa, index=salario.index)
print(faixa.value_counts().to_dict())

Series True
ndarray
{'abaixo': 45, 'acima': 45}

The standard deviation trap

The std Pandas divides by n - 1 (the sample variance divides by, ddof=1), and the np.std ). A z-score calculated in each way gives different numbers, and those who mix the two (one in the analysis, another in the model) have a small error that no one notices: n (ddof=0third case

valores = df["salario"]
print(round(float(valores.std()), 2), round(float(np.std(valores)), 2), round(float(np.std(valores.to_numpy())), 2))
print(round(float(np.std(valores.to_numpy(), ddof=1)), 2) == round(float(valores.std()), 2))

2433.44 2419.88 2419.88
True

Notice the , applied to: the np.std(valores), gave the Seriessame value of the array , 2419.88) and (ddof=0the one from the are not of the own .std() (2433.44). NumPy passes the Series to the method of the ddof=0 , and Pandas respects what it receives. In other words, the same data and Seriestwo ways of asking for the standard deviation in Pandas yield different results. The rule: write the explicitly ddof , in either of the two libraries, and ensure that the analysis and the model use the same.With NumPy, standardizing all columns at once is one line (the broadcasting from chapter 13 of the NumPy course), and gives the same result as Pandas when the

Standardizing with broadcasting

With NumPy, standardizing all columns at once is a line (the broadcasting from chapter 13 of the NumPy course), and gives the same result as Pandas when the ddof coincides:

Z = (X - X.mean(axis=0)) / X.std(axis=0, ddof=1)
z_pandas = (df[["experiencia", "projetos_concluidos"]] - df[["experiencia", "projetos_concluidos"]].mean()) / \
    df[["experiencia", "projetos_concluidos"]].std()
print(bool(np.allclose(Z, z_pandas.to_numpy())))

True

A regression with NumPy, checked with Pandas

Does salary increase with experience? The least squares line answers, and the slope has an exact relationship with the correlation of Pandas (inclinação = correlação × desvio_y / desvio_x):

exp = df["experiencia"].to_numpy(dtype=float)
sal = df["salario"].to_numpy(dtype=float)
A = np.column_stack([np.ones(len(exp)), exp])
(intercepto, inclinacao), *_ = np.linalg.lstsq(A, sal, rcond=None)
print(round(float(inclinacao)), round(float(intercepto)))

pela_correlacao = df["salario"].corr(df["experiencia"]) * df["salario"].std() / df["experiencia"].std()
print(bool(np.isclose(inclinacao, pela_correlacao)))

379 5044
True

Each year of experience adds about 380 reais to the salary in this dataset (which was generated this way, with more noise). The second line shows that NumPy and Pandas tell the same story in different ways.

Text variables as numbers

The get_dummies transforms a column of categories into columns of 0 and 1, the matrix that a model expects:

dummies = pd.get_dummies(df["departamento"], dtype=float)
print(dummies.shape, dummies.sum(axis=1).unique().tolist(), dummies.columns.tolist())
print(type(dummies.to_numpy()).__name__)

(90, 5) [1.0] ['Ciência de Dados', 'Engenharia', 'Finanças', 'Marketing', 'RH']
ndarray

Each row sums to 1, because each person is in one department.