Time-series cross-validation shuffle bias

Code reviewML systemsSenior–Staff~20 min

Review this Python validation setup for a demand-forecasting model.

The CV MAE is great but the model misses badly in production. Find the leakage.

What a strong answer looks like

Separate real bugs from style. Rank issues by severity, point at the root cause rather than the symptom, and suggest a concrete fix, specific and kind.

0:00 of about 20 min

Mark a line and say what kind of problem it is.0 findings

1import pandas as pd

2from sklearn.model_selection import KFold

3from sklearn.ensemble import GradientBoostingRegressor

4from sklearn.metrics import mean_absolute_error

6def cv_forecast(df):

7 # df sorted by date; features include lag_1, lag_7, rolling_mean_30

8 X = df.drop(columns=["date", "demand"]).values

9 y = df["demand"].values

10 kf = KFold(n_splits=5, shuffle=True, random_state=0)

11 maes = []

12 for tr, te in kf.split(X):

13 m = GradientBoostingRegressor().fit(X[tr], y[tr])

14 maes.append(mean_absolute_error(y[te], m.predict(X[te])))

15 print("CV MAE:", sum(maes) / len(maes))

Which questions mattered is sealed until you submit. Telling you now would just be handing over the edge cases.