Read and write arrays

Text: human-readable and spreadsheet-friendly

The np.savetxt controls the formatting (without it, the default is a huge scientific notation), and the np.loadtxt writes a line of headers: fmt The text is universal (any spreadsheet opens it), but it has two costs: header writes a line of titles:

from pathlib import Path

import numpy as np

dados = np.array([[1.5, 2.0], [3.25, 4.0]])
np.savetxt("tabela.csv", dados, delimiter=",", header="a,b", comments="", fmt="%.2f")
print(Path("tabela.csv").read_text(encoding="utf-8"))

lido = np.loadtxt("tabela.csv", delimiter=",", skiprows=1)
print(np.array_equal(lido, dados))

a,b
1.50,2.00
3.25,4.00

True

The text is universal (any spreadsheet opens it), but it has two costs: loses precision (it fmt="%.2f" discarded digits) and is slow for large files, because converting text to number costs.

Binary: the `.npy` format

The np.save writes the array exactly as it is in memory, with the type and shape, to a file .npy. The np.load returns it identical. It is fast, precise, and compact:

np.save("matriz.npy", dados)
print(np.array_equal(np.load("matriz.npy"), dados))
print(Path("matriz.npy").stat().st_size > dados.nbytes)

True
True

The file is only slightly larger than the data (nbytes), due to a small header that describes the type and shape. To store multiple arrays in a file, use the np.savez (or np.savez_compressed, which compresses), which creates a .npz, a kind of named array folder:

np.savez("pacote.npz", x=dados, y=np.arange(3))
with np.load("pacote.npz") as pacote:
    print(sorted(pacote.files), pacote["y"])

['x', 'y'] [0 1 2]

Missing data in text

The np.genfromtxt reads text files with gaps, and converts them into nan. The loadtxt would fail in this case:

Path("faltando.csv").write_text("a,b\n1,2\n3,\n5,6\n", encoding="utf-8")
cru = np.genfromtxt("faltando.csv", delimiter=",", skip_header=1)
print(cru)

[[ 1.  2.]
 [ 3. nan]
 [ 5.  6.]]

Do not load pickle from someone you do not know

The np.load refuses, by default, Python object arrays (allow_pickle=False), and I I keep it this way. The format pickle can execute code arbitrary when read, and loading a file of unknown origin with it linked is a real security risk.

When the table has mixed text and numbers

NumPy alone is clumsy for tables with columns of different types (names, dates, numbers). For this, there is pandas (chapter 38), which reads CSV with pd.read_csv and returns a DataFrame with labels.