Text with the accessor str

Separate and extract

The split with expand=True returns columns instead of lists, and the extract uses a regular expression with named groups to extract parts of a text:

import pandas as pd

pd.set_option("display.width", 170)
pd.set_option("display.max_columns", 20)
df = pd.read_csv("dados/funcionarios_100.csv").drop_duplicates("id_funcionario", keep="last")

partes = df["nome"].str.split(" ", n=1, expand=True)
partes.columns = ["primeiro_nome", "sobrenome"]
print(partes.head(3))

extraido = df["nome"].str.extract(r"(?P<primeiro>\S+)\s+(?P<ultimo>\S+)$")
print(extraido.head(3))

  primeiro_nome sobrenome
0         Lucas     Nunes
1          João     Rocha
2        Olívia     Nunes
  primeiro ultimo
0    Lucas  Nunes
1     João  Rocha
2   Olívia  Nunes

The n=1 separates only at the first space, and the rest remains intact in the surname (important for compound names). The named groups (?P<primeiro>) become the names of the result columns.

Measure, search and test

nomes = df["nome"]
print(nomes.str.len().describe()[["min", "max"]].tolist())
print(int(nomes.str.startswith("A").sum()), int(nomes.str.contains(r"^[A-C]", regex=True).sum()))
print(int(nomes.str.count("a").sum()), nomes.str.upper().head(2).tolist())

[8.0, 16.0]
4 11
211 ['LUCAS NUNES', 'JOÃO ROCHA']

The contains accepts a regular expression (regex=True by default), and the count counts the occurrences of a pattern in each text. In all cases, a value missing passes without error and remains missing (or False, in boolean responses).

Assemble text

Summing text columns with + works, but any missing value contaminates the result: a text summed with NaN is NaN. The str.cat lets you choose what to do with it:

pequena = pd.DataFrame({"nome": ["Ana", "Bruno", "Carla"], "departamento": ["RH", None, "Marketing"]})
print((pequena["nome"] + " (" + pequena["departamento"] + ")").tolist())
print(pequena["nome"].str.cat(pequena["departamento"], sep=" - ", na_rep="sem depto").tolist())
print(df["id_funcionario"].astype(str).str.zfill(6).head(2).tolist())

['Ana (RH)', nan, 'Carla (Marketing)']
['Ana - RH', 'Bruno - sem depto', 'Carla - Marketing']
['001001', '001002']

The zfill fills with zeros to the left, useful for fixed-width codes (the 1001 becomes 001001).

Find the "same" value written in different ways

Two records, João Pereira and joao pereira, are the same person, but Pandas doesn't know. The solution is to create a normalized key: without accents, in lowercase, and with a single space between words. Duplicates appear in the key, and not in the original text:

def chave(textos: pd.Series) -> pd.Series:
    return (
        textos.str.strip().str.lower()
        .str.normalize("NFKD").str.encode("ascii", errors="ignore").str.decode("ascii")
        .str.replace(r"\s+", " ", regex=True)
    )


cadastro = pd.Series(["João Pereira", " joao  pereira ", "Maria Souza", "MARIA SOUZA"])
print(chave(cadastro).tolist())
print(int(cadastro.duplicated().sum()), int(chave(cadastro).duplicated().sum()))

['joao pereira', 'joao pereira', 'maria souza', 'maria souza']
0 2

Without normalization, the duplicated() finds none (0). With the key, it finds 2. It's the technique behind any "deduplicate customers", and it's also the principle of joins: two tables only match if the keys are written the same.

A warning about regular expressions

With the pyarrow installed, text operations of the type str use, when possible, the Arrow regular expression engine, and not the module re of Python. For common patterns (\s, \S, \d, \w, groups, anchors) the result is the same. But there are differences at the edges: in chapter 10, the escape \u0300 in Pandas 3: rejected by Arrow, while Python accepts it. The rule of thumb: prefer simple and portable patterns, and test on a small example when using an uncommon feature.

Text and performance

Operating on text is more expensive than operating on numbers, and the type str of Pandas 3 helps. In columns with few distinct values (department, city, channel), the type category from chapter 19 saves a lot of memory and speeds up groupings.