From NumPy to machine learning

The convention that ties everything together

Scikit-learn expects the data as a matrix X of shape (n_amostras, n_características): one row per example, one column per attribute. And the targets in y, a vector with one value per example. This is the convention of chapter 41 (rows are students, columns are attributes), and it applies to almost the entire ecosystem.

uv sync --group ecossistema

The same regression, in two ways

First, data with known response: three features, true coefficients [2, -1, 0,5], an intercept of 3 and a bit of noise. The solution with NumPy uses least squares (chapter 23), with a column of ones for the intercept. The one from scikit-learn uses the LinearRegression:

import numpy as np
from sklearn.linear_model import LinearRegression

rng = np.random.default_rng(0)
X = rng.normal(size=(200, 3))
verdadeiro = np.array([2.0, -1.0, 0.5])
y = X @ verdadeiro + 3.0 + rng.normal(0, 0.1, 200)

Xb = np.hstack([X, np.ones((200, 1))])
theta, *_ = np.linalg.lstsq(Xb, y, rcond=None)

modelo = LinearRegression().fit(X, y)
print(np.allclose(theta[:3], modelo.coef_), np.isclose(theta[3], modelo.intercept_))
print(np.allclose(theta[:3], verdadeiro, atol=0.05))

True True
True

Both methods give the same answer, and both recover the true coefficients. There is no magic in the LinearRegression: it's this calculation, packaged with a standardized interface (fit, predict, score).

Measure the error by hand

The coefficient of determination (R²) compares the model's error with the error of simply predicting the mean. Writing it in NumPy takes one line, and the result matches the score from scikit-learn:

def r2(real, previsto):
    return 1 - ((real - previsto) ** 2).sum() / ((real - real.mean()) ** 2).sum()


print(round(r2(y, modelo.predict(X)), 3) == round(modelo.score(X, y), 3))

True

Evaluate on data that the model has not seen

Measuring the error on the same fitting data is too optimistic. The correct practice is to separate a part for testing (with the idea from chapter 19: shuffle and split):

ordem = rng.permutation(len(y))
treino, teste = np.split(ordem, [150])
Xb_treino = np.hstack([X[treino], np.ones((150, 1))])
theta_t, *_ = np.linalg.lstsq(Xb_treino, y[treino], rcond=None)
previsto = np.hstack([X[teste], np.ones((50, 1))]) @ theta_t
print(r2(y[teste], previsto) > 0.99)

True

Gradient descent: learning without the closed formula

For models without a closed-form solution (like neural networks), the way is to iterate: calculate the gradient of the error and take a small step in the opposite direction. For regression, the gradient of the mean squared error is 2/n · Xᵀ(Xw − y), and the iteration converges to the same solution as least squares:

w = np.zeros(4)
for _ in range(500):
    w -= 0.1 * (2 / len(y)) * Xb.T @ (Xb @ w - y)
print(np.allclose(w, theta, atol=1e-3))

True

The 0.1 is the learning rate: too large and the iteration diverges, too small and it takes too long. Chapter 43 builds an entire neural network with this same idea.

Standardization: what `StandardScaler` does

Many models work better with features on the same scale. The StandardScaler subtracts the mean and divides by the standard deviation of each column, exactly the z-score from chapter 13:

from sklearn.preprocessing import StandardScaler

print(np.allclose(StandardScaler().fit_transform(X), (X - X.mean(axis=0)) / X.std(axis=0)))

True

Data leakage

Calculate the mean and the standard deviation only on the training data and apply the same numbers to the test. Standardizing everything together before separating lets test information leak into the training, and the evaluation becomes optimistic. The fit learns from the training, and the transform only applies.

The path to deep learning

PyTorch and JAX have tensors with the same shape, the same indexing, and the same broadcasting rules as NumPy. The conversion between them is cheap because, on the CPU, they can share memory. What they add is GPU and automatic differentiation. Those who master chapter 17 can read a tutorial of theirs without finding the vocabulary strange.