Skip to content

    Chapter 19, Intermediate

    Join, stack and split

    Data rarely comes in one block. It comes in pieces that you need to assemble, and then you need to separate into training and testing, in batches, in groups.

    Code for this chapter: intermediario/cap19_juntar_dividir.py

    Join along an axis

    The np.concatenate joins arrays along an existing axis. All other dimensions need to be the same:

    intermediario/cap19_juntar_dividir.pylinhas 10 a 15
    import numpy as np

a = np.array([[1, 2], [3, 4]])
b = np.array([[5, 6]])
print(np.concatenate([a, b], axis=0))
print(np.concatenate([a, b.T], axis=1))
    Output
    [[1 2]
 [3 4]
 [5 6]]
[[1 2 5]
 [3 4 6]]

    When the dimensions do not match, the error specifies exactly which:

    intermediario/cap19_juntar_dividir.pylinhas 17 a 20
    try:
    np.concatenate([a, b], axis=1)
except ValueError as erro:
    print(erro)
    Output
    all the input array dimensions except for the concatenation axis must match exactly, but along dimension 0, the array at index 0 has size 2 and the array at index 1 has size 1

    Stack on a new axis

    The np.stack creates a new axis and places each array in a position of it. This is what you want to combine several arrays of the same shape into a matrix, or several images into a batch. The shortcuts vstack and hstack stack vertically and horizontally:

    intermediario/cap19_juntar_dividir.pylinhas 25 a 28
    x = np.array([1, 2, 3])
y = np.array([4, 5, 6])
print(np.stack([x, y]).shape, np.stack([x, y], axis=1).shape)
print(np.vstack([x, y]), np.hstack([x, y]))
    Output
    (2, 3) (3, 2)
[[1 2 3]
 [4 5 6]] [1 2 3 4 5 6]
    FunctionWhat it doesShape of x and y with 3 elements each
    np.concatenate([x, y])Joins along an existing axis(6,)
    np.stack([x, y])Creates a new axis, in front(2, 3)
    np.stack([x, y], axis=1)Creates the new axis in the middle(3, 2)
    np.vstack([x, y])Stacks as rows(2, 3)
    np.hstack([x, y])Joins side by side(6,)

    Split

    The np.split splits into parts equal (and complains if it doesn't work), the np.array_split accepts parts of different sizes, and a list of positions indicates where to cut:

    intermediario/cap19_juntar_dividir.pylinhas 33 a 36
    dados = np.arange(10)
print(np.split(dados, 5))
print(np.array_split(dados, 3))
print(np.split(dados, [2, 7]))
    Output
    [array([0, 1]), array([2, 3]), array([4, 5]), array([6, 7]), array([8, 9])]
[array([0, 1, 2, 3]), array([4, 5, 6]), array([7, 8, 9])]
[array([0, 1]), array([2, 3, 4, 5, 6]), array([7, 8, 9])]

    The most common use is to separate training and testing: shuffle the positions (chapter 16) and cut. Shuffling beforehand is essential, because data stored in order (by date, by class) would give a biased test:

    intermediario/cap19_juntar_dividir.pylinhas 38 a 41
    rng = np.random.default_rng(0)
ordem = rng.permutation(10)
treino, teste = np.split(ordem, [8])
print(len(treino), len(teste), sorted(np.concatenate([treino, teste]).tolist()) == list(range(10)))
    Output
    8 2 True

    Repeat

    intermediario/cap19_juntar_dividir.pylinha 46
    print(np.repeat([1, 2], 3), np.tile([1, 2], 3))
    Output
    [1 1 1 2 2 2] [1 2 1 2 1 2]

    The repeat repeats each element, and the tile repeats the entire block.

    The `np.append` in a loop is a trap

    Each np.append creates a new array , copying everything that already existed. In a loop, this costs time proportional to the square of the size. The right pattern is to accumulate in a list (which grows cheaply) and convertonce at the end: Add the column of ones

    intermediario/cap19_juntar_dividir.pylinhas 51 a 68
    import timeit


def com_append(n):
    a = np.array([])
    for i in range(n):
        a = np.append(a, i)
    return a


def com_lista(n):
    return np.array([i for i in range(n)], dtype=float)


print(np.array_equal(com_append(500), com_lista(500)))
t_append = min(timeit.repeat(lambda: com_append(3000), number=1, repeat=3))
t_lista = min(timeit.repeat(lambda: com_lista(3000), number=1, repeat=3))
print("a lista foi pelo menos 5 vezes mais rápida:", t_append > 5 * t_lista)
    Output
    True
a lista foi pelo menos 5 vezes mais rápida: True

    Exercise 1

    that adds a column of ones

    Write adicionar_vies(X) at the beginning of a feature matrix . This is the standard step before fitting a linear regression with an intercept. XSorting an array is easy. What is really used is the **position** in which each element would be, because it allows reordering other arrays in the same order.