Files larger than memory: memmap

A file that behaves like an array

The np.memmap maps a binary file into memory. You read and write as in a regular array, and the operating system takes care of transferring the pages from disk on demand. Here, a file with a million float32:

from pathlib import Path

import numpy as np

caminho = Path("grande.bin")
mm = np.memmap(caminho, dtype=np.float32, mode="w+", shape=(1_000_000,))
mm[:] = np.arange(1_000_000, dtype=np.float32)
mm.flush()
print(caminho.stat().st_size, mm.shape)
del mm

4000000 (1000000,)

The file has exactly 1.000.000 × 4 bytes: the memmap is just the data, without a header, and therefore you need to specify the type and shape when opening it again. The flush ensures that the changes reach the disk, and the del releases the file.

Open only for reading

The most used modes are "r" (read-only), "r+" (read and write in an existing file) and "w+" (creates or deletes the file and opens it for writing). In read mode, the array is protected against accidental changes:

lido = np.memmap(caminho, dtype=np.float32, mode="r", shape=(1_000_000,))
print(lido[999_999], lido[:3], isinstance(lido, np.ndarray))
try:
    lido[0] = 1.0
except ValueError as erro:
    print(erro)

999999.0 [0. 1. 2.] True
assignment destination is read-only

A memmap is a ndarray, and therefore accepts everything you have already learned: slices, masks, aggregations.

Process in blocks

Even when mapped, a calculation over the entire array may need memory for intermediates. The solution is to traverse the file in blocks, accumulating the result. Here, the average of a million values with at most 250 thousand in memory at a time:

def media_em_blocos(mm, bloco=250_000):
    total = 0.0
    for inicio in range(0, len(mm), bloco):
        total += float(mm[inicio : inicio + bloco].sum(dtype=np.float64))
    return total / len(mm)


print(media_em_blocos(lido), np.arange(1_000_000, dtype=np.float64).mean())

499999.5 499999.5

The sum(dtype=np.float64) accumulates in double precision, even if the data is float32, so that the rounding error does not grow with size.

`np.load` with `mmap_mode`

A file .npy (chapter 24) already carries the type and shape in the header. With mmap_mode="r", the np.load opens it as memmap without you repeating this data:

np.save("grande.npy", np.arange(1_000_000, dtype=np.float32))
vista = np.load("grande.npy", mmap_mode="r")
print(type(vista).__name__, vista[10])

memmap 10.0

This is usually the most practical way: save in .npy and open with mmap_mode.

Cleanup

A memmap keeps the file open. On Windows, you cannot delete a file while some array is still mapping it, so release the references first:

del lido, vista
for nome in ("grande.bin", "grande.npy"):
    Path(nome).unlink()
print(Path("grande.bin").exists())

False