Missing values: NaN and masked arrays

The `nan` contaminates

The nan ("not a number") is a special decimal value. Any calculation involving it yields nan, and this propagation is what makes it dangerous: a single hole ruins an entire average. The "nan-aware" versions (nansum, nanmean, nanmax...) ignore the holes:

import numpy as np

x = np.array([1.0, np.nan, 3.0, 4.0])
print(x.sum(), np.nansum(x), np.nanmean(x), np.nanmax(x))

nan 8.0 2.6666666666666665 4.0

How to detect (and the equality trap)

The nan is different from itself: nan == nan is false. Therefore x == np.nan never finds anything, and the correct detection is with np.isnan:

print(np.isnan(x), np.isnan(x).sum())
print(np.nan == np.nan, np.isnan(np.nan))

[False  True False False] 1
False True

Remove or fill

Two common strategies: discarding the positions with nan, or filling with a reasonable value (the mean, the median, the previous value). Which to choose is a decision about your problem, not a rule of NumPy. Discarding loses information, and filling the inventing:

limpo = x[~np.isnan(x)]
preenchido = np.where(np.isnan(x), np.nanmean(x), x)
print(limpo, preenchido.round(2))
print(x > 2)

[1. 3. 4.] [1.   2.67 3.   4.  ]
[False False  True  True]

Notice the last result: a comparison with nan always yields False, and that’s why a mask like x > 2 silently discards the holes without warning.

Only decimals have `nan`

Integers do not have a representation for "missing". An array of integers cannot hold nan, and putting a nan in a list of integers converts it to decimal:

print(np.array([1, 2, np.nan]).dtype)
try:
    np.array([1, 2, 3])[0] = np.nan
except ValueError as erro:
    print(erro)

float64
cannot convert float NaN to integer

Masked arrays

The np.ma offers an alternative: an array with a mask that marks which positions should be ignored, also working for integers. It is useful when missing data is marked by a special value, such as -999:

m = np.ma.masked_invalid(x)
print(m, m.mean(), m.mask)

sentinela = np.ma.masked_equal([1, -999, 3], -999)
print(sentinela, sentinela.mean())

[1.0 -- 3.0 4.0] 2.6666666666666665 [False  True False False]
[1 -- 3] 2.0

The -- marks the ignored position, and the average considers only valid values.

Sentinel values are a trap

Values like -999 or 0 for "no data" are a danger: if someone forgets to handle them, they enter the calculations as real numbers. Convert them to nan (or mask them) at the input of the data, once, and do not trust the memory of those who use it later.