Dataframes

Stop walking rows one at a time: describe the rows you want and hand the whole description to the table.

The idea

A dataframe is a table you address as a whole. Rather than looping and testing each row, you build one column of True/False and give it to the table: df[mask] returns every kept row in a single operation.

The same shift covers summarising (groupby splits by key, applies once per group, and glues results back), joining (merge is a SQL join, duplicate keys and all), and arithmetic (column math replaces the loop you were about to write).

There is one trap. Some ways of selecting hand you back a new object, so a write there vanishes into a temporary. .loc is the address of the original.

Watch it happen

threshold for amount >= t 100

How it works

Every operation below is described once and executed over whole arrays in compiled code. Your job is to write the description, not the loop.

import pandas as pd

df = pd.DataFrame({
    "region": ["north", "south", "north", "east", "south", "north"],
    "rep":    ["ada",   "bo",    "cy",    "dee",  "eli",   "fin"],
    "amount": [120,      80,      200,     45,     160,     90],
})

# 1 — filter: a mask is a whole column of booleans; selecting with it is ONE op
mask = df["amount"] >= 100     # Series of 6 booleans, aligned to df's index
big  = df[mask]                # rows 0, 2, 4 — no Python-level loop

# 2 — split-apply-combine (sort=False keeps first-seen key order;
#     the default sort=True would give east, north, south)
totals = df.groupby("region", sort=False, as_index=False)["amount"].sum()
#   region  amount
# 0  north     410
# 1  south     240
# 2   east      45

# 3 — merge is a join: duplicate keys multiply rows
orders = pd.DataFrame({"order":  ["o1", "o2", "o3", "o4"],
                       "region": ["north", "south", "east", "north"],
                       "amount": [120, 80, 45, 200]})
mgr    = pd.DataFrame({"region": ["north", "north", "north", "south", "west"],
                       "mgr":    ["ada", "bo", "cy", "dee", "eli"]})

out = orders.merge(mgr, on="region", how="inner")
len(orders), len(mgr), len(out)      # (4, 5, 7)
# north: 2 left x 3 right = 6 | south: 1 x 1 = 1 | east: 1 x 0 = 0

# 4 — the write that silently misses
df[df["region"] == "north"]["amount"] = 0     # two lookups: lands on a temporary
df.loc[df["region"] == "north", "amount"] = 0 # one lookup: lands on df itself

# 5 — column math is already vectorized
df["net"] = df["amount"] * 0.93 - 2.50

Cost

operation	time	extra space	the trade-off
df[mask]	O(n)	O(n) mask + O(k) result	one pass in compiled code; you do materialise a full boolean column and copy the kept rows
for _, row in df.iterrows()	O(n), ~10 to 100× the constant	a fresh Series per row	same big-O, very different wall clock; each row is boxed and dtypes get coerced
groupby(k).sum()	O(n) average (hash)	O(g) result	default sort=True adds O(g log g); .apply(python_fn) puts you back at loop speed
merge(on=k)	O(n + m) average (hash join)	O(min(n,m)) table + O(r) result	r = Σ_key (left count × right count): output can dwarf both inputs
df["a"] * 2 - 1	O(n)	O(n) per temporary	memory for speed; long chained expressions allocate an array per step

Watch out for

Chained indexing on the left of an =. df[df.qty > 0]["price"] = 0 is two separate indexing calls: the first builds a new frame, the second writes into it, and that frame is thrown away. Historically this raised SettingWithCopyWarning; under copy-on-write (default in pandas 3.0) it simply never reaches df. Always: df.loc[df.qty > 0, "price"] = 0.
and / or inside a mask. Series don't have a single truth value, so use &, |, ~, and parenthesise, because & binds tighter than >=: df[(df.a >= 1) & (df.b < 9)].
Missing values quietly pick a side. NaN compares False to everything, so df[df.x > 0] drops missing rows, but NaN != 0 is True, so df[df.x != 0] keeps them. If it matters, be explicit with .isna() / .notna().
Merge fan-out and silent misses. A duplicated key on the right multiplies rows, so every sum afterwards is inflated. Assert your expectation with validate="m:1", and compare len() before and after. Also check key dtypes: "7" never matches 7, and a left join fills unmatched integer columns with NaN, upcasting them to float.
Groupby's quiet defaults. dropna=True means rows with a missing key disappear from the result entirely; observed=False on categoricals can invent empty groups. And groupby(...).apply(lambda g: ...) with a Python function re-enters the interpreter once per group: prefer the built-in aggregations when you can.

Worked example

You have 240,000 order rows and a small region lookup, and you want net payout per region for shipped orders only. First the filter: shipped = orders["status"] == "shipped" builds one boolean column, and orders.loc[shipped] selects in one sweep. Then the math: orders["net"] = orders["amount"] * 0.93 - 2.50. No loop, one new column.

Now the join. You merge in regions to get each region's manager, then groupby("region")["net"].sum(), and the total jumps from 1.2M to 1.9M. The lookup has three rows for north, one per manager, so every north order was tripled before you summed. Two habits catch it instantly: compare len() before and after, or write orders.merge(regions, on="region", validate="m:1") and let pandas raise instead of you shipping the number.

Finally you spot three orders with a negative amount and zero them. orders[orders.amount < 0]["amount"] = 0 runs without complaint and changes nothing: the exported CSV still has the negatives. orders.loc[orders.amount < 0, "amount"] = 0 lands on the frame you're actually exporting.

Check yourself

You ran df[df.qty > 0]["price"] = 0, saw no error, and df is unchanged. What happened?

The left frame has 3 rows with key "a"; the right frame has 2 rows with key "a". After an inner merge on that key, how many output rows carry "a"?