Chapter 35, Advanced

What changed in Pandas 3

Much of what is found on the internet was written for Pandas 1 or 2. This chapter gathers, in one place, what has changed, **each item proven by code**, and how to find what needs to be migrated in an old project.

What was removed

These names, very common in old material, no longer exist erases the evidence

avancado/cap35_pandas3.pylinhas 10 a 25

import inspect

import pandas as pd

removidos = {
    "DataFrame.append": hasattr(pd.DataFrame, "append"),
    "DataFrame.applymap": hasattr(pd.DataFrame, "applymap"),
    "DataFrame.iteritems": hasattr(pd.DataFrame, "iteritems"),
    "DataFrame.mad": hasattr(pd.DataFrame, "mad"),
    "DataFrame.ix": hasattr(pd.DataFrame, "ix"),
    "DataFrame.swapaxes": hasattr(pd.DataFrame, "swapaxes"),
    "DataFrame.first": hasattr(pd.DataFrame, "first"),
    "Series.view": hasattr(pd.Series, "view"),
}
print(removidos)
print("o substituto do applymap:", hasattr(pd.DataFrame, "map"))

Output

{'DataFrame.append': False, 'DataFrame.applymap': False, 'DataFrame.iteritems': False, 'DataFrame.mad': False, 'DataFrame.ix': False, 'DataFrame.swapaxes': False, 'DataFrame.first': False, 'Series.view': False}
o substituto do applymap: True

Before	Now
`df.append(outra)`	`pd.concat([df, outra])`
`df.applymap(f)`	`df.map(f)`
`df.iteritems()`	`df.items()`
`df.ix[...]`	`df.loc[...]` or `df.iloc[...]`
`df.first("5D")`	Slicing by date index
`s.view(...)`	`s.astype(...)` or NumPy

What changed in behavior

The most dangerous changes do not throw an error: the same code produces a different result. Each one was seen in a chapter of this course, and here they are together, with the proof:

avancado/cap35_pandas3.pylinhas 30 a 48

import warnings

import numpy as np

texto = pd.Series(["a", "b"])
print("1. texto é str:", texto.dtype)

s = pd.Series([1, 2, 3], index=["x", "y", "z"])
try:
    s[0]
except KeyError:
    print("2. s[0] em índice de texto: KeyError (use .iloc[0])")

grupos = inspect.signature(pd.DataFrame.groupby).parameters
print("3. groupby observed:", grupos["observed"].default, "| tem axis:", "axis" in grupos)

print("4. resolução de datas:", pd.to_datetime(pd.Series(["2025-01-01"])).dtype)

print("5. str.contains com ausente:", pd.Series(["a", None]).str.contains("a").tolist())

Output

1. texto é str: str
2. s[0] em índice de texto: KeyError (use .iloc[0])
3. groupby observed: True | tem axis: False
4. resolução de datas: datetime64[us]
5. str.contains com ausente: [True, False]

avancado/cap35_pandas3.pylinhas 50 a 64

try:
    pd.Series([1.0, None]).fillna(method="ffill")
except TypeError:
    print("6. fillna(method=...) removido: use ffill() e bfill()")

try:
    pd.Series([1, 2]).resample("M")
except (TypeError, ValueError):
    print("7. resample('M') não vale mais: use 'ME' ou 'MS'")

df = pd.DataFrame({"a": [1, 2, 3], "b": [10.0, 20.0, 30.0]})
with warnings.catch_warnings(record=True) as avisos:
    warnings.simplefilter("always")
    df[df["a"] > 1]["b"] = 0
print("8. atribuição encadeada:", [a.category.__name__ for a in avisos], df["b"].tolist())

Output

6. fillna(method=...) removido: use ffill() e bfill()
7. resample('M') não vale mais: use 'ME' ou 'MS'
8. atribuição encadeada: ['ChainedAssignmentError'] [10.0, 20.0, 30.0]

#	Before (Pandas 1 and 2)	Pandas 3	Chapter
1	Text was `object`	Text is `str`	3
2	`s[0]` fell to the position in text index	`KeyError`	4
3	`groupby(observed=False)` by default, with `axis`	`observed=True`, without `axis`	19 and 23
4	Dates in nanoseconds	Dates in microseconds (`datetime64[us]`)	18
5	`str.contains` returned `NaN` for missing (in `object`)	Returns `False` in type `str`	8
6	`fillna(method="ffill")`	Removed: `ffill()`	9
7	`resample("M")`	`"ME"` or `"MS"`	18
8	Chained assignment sometimes worked	Never alters the original	27
9	`groupby().apply` received the group column	No longer receives	23
10	`to_numpy()` returned a writable array (sometimes)	Read-only view	27

How to find what to migrate

An old project may have hundreds of uses of patterns that have changed. The ruff, with the rules of the set PD (pandas-vet), points out some of the most common. Consider this old file:

velho.py

import pandas as pd

df = pd.read_csv("x.csv")
df.dropna(inplace=True)
valores = df.values
df = df.append({"a": 1}, ignore_index=True)
antigo = df.iteritems()

Terminal

ruff check --select PD velho.py

Output

velho.py:4:11: PD002 `inplace=True` should be avoided; it has inconsistent behavior
velho.py:5:11: PD011 Use `.to_numpy()` or `.array` instead of `.values`

The ruff pointed two issues, and I checked that it are not points the append neither the iteritems: these already fail on their own, on the first run, with a AttributeError. The tool helps, but does not replace tests: the best safety net in a migration is a good suite, run with Pandas 3.

You can also do a simple scan, with a function that looks for old patterns in the code text:

avancado/cap35_pandas3.pylinhas 69 a 93

import re

PADROES = {
    r"\.append\(": "df.append foi removido (use pd.concat)",
    r"\.applymap\(": "applymap foi substituído por map",
    r"inplace\s*=\s*True": "evite inplace",
    r"fillna\(\s*method\s*=": "fillna(method=) foi removido (use ffill/bfill)",
    r"\.ix\[": ".ix foi removido (use loc ou iloc)",
    r"resample\(\s*[\"']M[\"']": "'M' virou 'ME' ou 'MS'",
    r"\]\[[^\]]+\]\s*=": "possível atribuição encadeada (use .loc)",
}


def auditar(codigo: str) -> list[str]:
    achados = []
    for numero, linha in enumerate(codigo.splitlines(), start=1):
        for padrao, aviso in PADROES.items():
            if re.search(padrao, linha):
                achados.append(f"linha {numero}: {aviso}")
    return achados


amostra = 'df = df.append(x)\ndf["a"].fillna(0, inplace=True)\nr = s.resample("M").sum()\n'
for achado in auditar(amostra):
    print(achado)

Output

linha 1: df.append foi removido (use pd.concat)
linha 2: evite inplace
linha 3: 'M' virou 'ME' ou 'MS'

Pin the version while you migrate

If a project still cannot migrate, pin the version, and migrate when there is time and tests:

Terminal

uv add "pandas<3"

The golden rule: migrate with tests. Behavioral changes (items 1 to 10 in the table) do not raise errors, and only a good suite, or a careful check of the outputs, shows that something has changed.

Exercise 1

The code auditor

Use the function auditar to check that it finds the three problems of amostra, on different lines, and that are not reports a code already migrated (pd.concat, ffill, resample("ME")).

What was removed

What changed in behavior

How to find what to migrate

Pin the version while you migrate

View solution