Statistics: percentiles, histograms and correlation

Correlation between two variables

Correlation measures how much two variables move together, from -1 (move in opposite directions) to +1 (move together), with 0 meaning there is no linear. The np.corrcoef returns the correlation matrix, and the diagonal is always 1 (each variable with itself):

import numpy as np

rng = np.random.default_rng(42)
altura = rng.normal(170, 10, 500)
peso = 0.9 * altura - 85 + rng.normal(0, 5, 500)

matriz = np.corrcoef(altura, peso)
print(matriz.round(2))
print(0.8 < matriz[0, 1] < 0.95)

[[1.   0.86]
 [0.86 1.  ]]
True

I built the peso from the altura plus some noise, so the high correlation is expected (around 0.87, in theory). Correlation is not causation, and two numbers can move together by coincidence or because a third factor influences both.

Histograms

The np.histogram counts how many values fall into each range. You choose the ranges, with either an equal number of ranges or exact limits. Each range includes the left limit and excludes the right (except for the last one, which includes both):

notas = np.array([1, 2, 2, 3, 3, 3, 4, 4, 5])
contagens, limites = np.histogram(notas, bins=[0, 2, 4, 6])
print(contagens, limites)

contagens, limites = np.histogram(altura, bins=5)
print(contagens.sum(), len(limites))

[1 5 3] [0 2 4 6]
500 6

The number of limits is always one more than the number of ranges, and the sum of the counts is the total number of values.

Variations over time

The np.diff calculates the difference between consecutive elements, and dividing it by the previous value gives the percentage change. The np.cumsum accumulates a sum across the array:

precos = np.array([100.0, 102.0, 99.0, 105.0])
retornos = np.diff(precos) / precos[:-1]
print(retornos.round(4), np.cumsum([1, 2, 3, 4]))

[ 0.02   -0.0294  0.0606] [ 1  3  6 10]

The result has one element less than the input, because the first price has no previous.

Group with `bincount`

The np.bincount counts occurrences of each integer, and with the parameter weights sums values by group. It is a GROUP BY of SQL in one line, for numbered categories starting from 0:

categorias = np.array([0, 1, 1, 2, 2, 2])
valores = np.array([10, 20, 30, 1, 2, 3])
print(np.bincount(categorias), np.bincount(categorias, weights=valores))

[1 2 3] [10. 50.  6.]

Detect outliers

A classic rule uses the interquartile range (IQR, the distance between the 25th percentile and the 75th percentile): anything beyond 1.5 IQR from the quartiles is considered atypical. It is resistant to extremes, unlike a rule based on the mean:

dados = np.array([10, 12, 11, 13, 12, 95])
q1, q3 = np.percentile(dados, [25, 75])
iqr = q3 - q1
limite = q3 + 1.5 * iqr
print(dados[dados > limite])

[95]

Mean and standard deviation depend on extremes

The value 95 shifts the mean and standard deviation of this set significantly. Therefore, to find outliers and to describe skewed data, I prefer median and quartiles. Chapter 14 showed the same with salaries.