A/B test validator
Teams keep peeking at experiments and shipping on noise. You're asked to build an agent that analyzes A/B tests and tells teams whether results are real. This one can do real damage — walk me through the statistical guardrails you'd hard-code, what the agent may say versus what only a human data scientist decides, and how you'd validate its judgments against past experiments. Then make the case to your VP that an opinionated agent beats the current free-for-all.
classify_experiment_readout(pre_registered: bool, planned_sample_size: int, control_n: int, treatment_n: int, p_value: float, alpha: float) → str[true,1000,520,500,0.03,0.05]out"significant"[true,2000,500,500,0.004,0.05]out"interim significant"[false,1000,500,500,0.001,0.05]out"refuse: not pre-registered"Turn the fuzzy goal into a bounded agent task. Say what the agent reads, what it may do on its own versus draft for a human, and what it must never touch. Then earn the trust: how you’d verify it before rollout, and the pitch (the problem, what you built, why it matters).
Agent build: design the agent in plain language (or narrate it). The coach grades the decomposition, the guardrails, the verification plan, and the pitch.