Prep RoomSign inStart free

Metric diagnosis questions.

86 metric diagnosis questions from the bank, ready to practice out loud.

Practice metric diagnosis

Learn the ideas first · free with an account

Funnel metrics and conversion decompositionMetric anomaly triage: real move or artifact?STAR and CARL: structuring your storyAnswer-first: delivering findings under time pressureCohort analysis and retention curvesSegmenting a metric move: mix shift and Simpson's paradoxStatistical power and sample sizeUnit economics: CAC, LTV, and payback

All 86 questions

DAU dropped eight percentDAU dropped 8% on Tuesday. What do you check first?Segmentation strategyMid–senior
Investigating sudden revenue dropRevenue dropped 12% overnight in your dashboard, but customer support hasn't gotten any reports. Walk me through your investigation.Ruling out instrumentation issuesMid–senior
Sign-up conversion dropped 5%Sign-up conversion dropped 5% last week. New users from organic traffic are flat, but paid-traffic users are down 20%. What's your next step?Isolating cohortsMid–senior
Engagement spike no code changeEngagement on a feature spiked 30% after a release but no one shipped a change to it. What's most likely and where do you start?Hypothesis prioritizationSenior–staff+
Anomaly diagnosis in 30 minutesYour team is asked to explain a metric anomaly in 30 minutes but you're not sure of the cause yet. What do you say?Communicating uncertaintySenior–staff+
Session duration up, DAU flatAverage session duration grew 15% last month but DAU is flat. Walk me through what that combination tells you.Segmentation strategyMid–senior
Inconsistent metric results between teamsA new metric was added two weeks ago and it's showing inconsistent results between teams. Walk me through what to look for.Ruling out instrumentation issuesMid–senior
Retention dropped for new cohortsRetention dropped for users who signed up in the last 30 days, but earlier cohorts look fine. What does that point to?Isolating cohortsMid–senior
Conversion dropped, no deploys shippedConversion rate dropped, but no engineering changes shipped this week. What ten things are still possible?Hypothesis prioritizationMid–senior
Is Tuesday's dip concerningYour CEO asks if Tuesday's metric dip is a real concern. You're not sure. How do you answer?Communicating uncertaintySenior–leadership
Click-through rate fell on homepageClick-through rate fell on your homepage. Walk me through how you'd slice the data.Segmentation strategyMid–senior
Growth metric flat despite improvementsA growth metric has been flat for two weeks even though stakeholders swear the product has improved. What's worth checking?Ruling out instrumentation issuesSenior–staff+
Premium user churn spike investigationPremium users churn at 2% per month, free users at 5%. This quarter premium churn is at 3.5%. What do you investigate?Isolating cohortsSenior–staff+
Delayed discovery of retention dropYour retention dropped on a single day three weeks ago and you only just noticed. What do you do?Hypothesis prioritizationMid–senior
Briefing leadership on metric declineYour team needs to brief leadership that a metric is moving in the wrong direction but you only have a hypothesis. Walk me through your briefing.Communicating uncertaintySenior–leadership
Distinguishing real change from noiseA KPI improved this month. How do you tell whether it's a real change versus normal noise?Segmentation strategyMid–senior
Attribution shift after SDK releaseAfter a SDK release, conversion attribution shifted between two channels. What's worth checking?Ruling out instrumentation issuesSenior–staff+
Engagement up for power users onlyEngagement is up in your most-active 5% of users but down across everyone else. How do you read that?Isolating cohortsSenior–staff+
User growth without revenue growthYour monthly active users grew 4% but revenue is flat. Walk me through likely causes.Hypothesis prioritizationSenior–staff+
Anomalous metric worry assessmentYou're asked, 'should we be worried?' about an anomalous metric. Walk me through your answer.Communicating uncertaintySenior–staff+
Default slices for metric anomaliesWhen you see an anomaly, what slices do you reach for almost reflexively, in what order?Segmentation strategyMid–senior
Validating a data quality concernA teammate says 'the data is wrong'. Before you agree or argue, what do you check?Ruling out instrumentation issuesMid–senior
Feature adoption only power usersA new feature launched and total time-in-app went up — but only for the existing power users. What does that tell you?Isolating cohortsMid–senior
Pageviews down while sessions flatPageviews dropped while sessions stayed flat. What does that combination mean?Hypothesis prioritizationMid–senior
Explaining metric anomaly to executivesWalk me through how you'd present a metric anomaly to a non-technical exec without losing precision.Communicating uncertaintySenior–leadership
Simpson's paradox versus real changeWhat's the difference between a Simpson's-paradox-flavored metric move and a real change, and how do you tell?Segmentation strategySenior–staff+
Conversion discrepancy between toolsA change in conversion shows in your warehouse but not in the product team's event tool. Where do you start?Ruling out instrumentation issuesMid–senior
Engagement down, ad revenue upEngagement dropped 6% but ad revenue is up. What might be going on?Isolating cohortsSenior–staff+
Investigating a two-sigma KPI moveA weekly KPI moved 2 standard deviations above its mean. What's your read on whether to investigate?Hypothesis prioritizationSenior–staff+
Summarizing unresolved anomaly investigationWalk me through how you'd write the one-paragraph summary of a metric anomaly investigation when the cause is unknown after a day.Communicating uncertaintySenior–staff+
Android DAU drop investigationYour mobile app's daily active users dropped 10% yesterday, but only on Android. iOS is flat. What are your first three checks?Segmentation strategyEntry–mid
New metric with hourly volatilityA metric started tracking two days ago and shows wildly different values each hour. What do you check to rule out a logging problem?Ruling out instrumentation issuesEntry–mid
Regional sign-up drop investigationSign-ups dropped 15% last Thursday. You segment by country and see the US is down 40% but all other countries are up slightly. What's your next step?Segmentation strategyEntry–mid
Average order value doubled overnightYour dashboard shows average order value increased 20% overnight, but the raw event count also doubled. What might be wrong with the data?Ruling out instrumentation issuesEntry–mid
Cohort retention differenceUsers who signed up in January have 60% week-1 retention, but February sign-ups have 45%. Walk me through how you'd investigate this cohort difference.Isolating cohortsEntry–mid
Prioritizing conversion drop hypothesesYou have three hypotheses for why checkout conversion dropped: a payment bug, slower page load, or a marketing campaign change. How do you prioritize which to check first?Hypothesis prioritizationEntry–mid
Video plays dipped 7% yesterdayYour manager asks if yesterday's 7% dip in video plays is something to escalate. You haven't finished investigating. What do you tell them?Communicating uncertaintyEntry–mid
Tool and SQL query mismatchA metric looks correct in your analytics tool but shows different numbers in the SQL query you wrote. What are the first two things you check?Ruling out instrumentation issuesEntry–mid
Mobile-specific conversion declineDesktop users have 3% conversion and mobile users have 1.5%. This week mobile conversion dropped to 1.1%. How do you investigate whether this is a mobile-specific issue?Isolating cohortsEntry–mid
Simultaneous UI and backend changesYour product had a small UI change and a backend deploy on the same day. A metric moved. How do you figure out which change caused it?Hypothesis prioritizationEntry–mid
Communicating competing theoriesYou're writing a Slack update about a metric drop and you have two competing theories but no conclusion yet. How do you structure the message?Communicating uncertaintyEntry–mid
Correcting for duplicate loggingSearch queries increased 25% yesterday but you just learned the logging was accidentally duplicated for two hours. How do you correct your understanding of the real change?Ruling out instrumentation issuesEntry–mid
Metric jump after schema migrationYour activation rate jumped 20% the same week a tracking-schema migration shipped. The growth team wants to announce the win. What now?Ruling out instrumentation issuesMid–senior
Thumbs-up rate versus repeat usageYour AI assistant's thumbs-up rate has climbed for six weeks, but repeat usage of the assistant is falling. Walk me through the hypotheses you'd test, in order.Hypothesis prioritizationMid–senior
Detecting bot or crawler trafficPageviews and sign-up starts jumped 30% overnight, but completed sign-ups and activation are flat. You suspect bot or AI-crawler traffic. How do you confirm or rule that out?Ruling out instrumentation issuesMid–senior
Accounting for holiday impactYour weekly metric is down 10%, but the week included a public holiday in your largest market. How do you work out how much of the drop is just the calendar?Hypothesis prioritizationEntry–mid
Growth missed forecast reframingYour metric didn't actually drop — it grew, but missed forecast by 15%. Leadership is treating it like a decline. How do you investigate it, and how do you reframe the conversation?Communicating uncertaintySenior–leadership
LLM cost per user doubledYour LLM-powered feature's cost per active user doubled this week, but usage volume is flat. What do you check?Hypothesis prioritizationMid–senior
Model provider upgraded silentlyA third-party model provider silently upgraded the model behind your feature, and your task-success metric moved the same week. With no A/B test in place, how do you establish whether the upgrade caused it?Hypothesis prioritizationSenior–staff+
Mean up median down revenueAverage revenue per user rose 12% this quarter while median revenue per user fell. What's going on, and how do you confirm it?Segmentation strategyMid–senior
Conversion change after geographic expansionSign-up-to-paid conversion fell 3 points the same week you launched in three new countries. Is anything actually wrong? Walk me through your check.Segmentation strategyEntry–mid
Suspiciously flat weekly metricsYour weekly dashboard has shown exactly 0.0% change for three straight weeks. Why is that itself suspicious, and what do you check?Ruling out instrumentation issuesEntry–mid
Morning data dip patternEvery morning at 9am, yesterday's numbers look 20% low, then recover by noon. A stakeholder keeps escalating the morning dip. What's happening, and what do you do about it?Communicating uncertaintyEntry–mid
New surface retains 20% worseUsers acquired through your new AI-recommendation surface retain 20% worse than other channels, and the team wants to kill the surface. Is the surface bad, or is it selection? How do you tell?Isolating cohortsSenior–staff+
Rating drop newest version onlyYour app-store rating fell from 4.4 to 3.8 — but only among users on the newest app version. What do you investigate first?Isolating cohortsEntry–mid
NPS drop with halved responsesNPS dropped 12 points this quarter — but survey response rate also halved after an email-deliverability change. How do you decide whether sentiment actually fell?Segmentation strategySenior–staff+
Order value up orders downAverage order value jumped 18% yesterday, but total orders dropped 10%. Walk me through how you'd diagnose this.Segmentation strategyEntry–mid
Mobile conversion drop, desktop flatMobile conversion rate dropped 15% but desktop is unchanged. What dimensions would you segment by to narrow down the root cause?Segmentation strategyMid–senior
Page load time increased unexpectedlyYour analytics team reports that page load time increased 2 seconds last Thursday, but engineering says no deploys happened. What are the first three things you check?Ruling out instrumentation issuesMid–senior
Conflicting MAU numbers across dashboardsTwo dashboards show different MAU numbers for the same time period—one shows 2.1M, the other 2.4M. How do you resolve this discrepancy?Ruling out instrumentation issuesMid–staff+
Cost per paying user spikePaid acquisition cost per install stayed flat, but cost per paying user rose 40% this month. What cohort cuts help you understand why?Isolating cohortsSenior–leadership
Cohort retention dip then recoveryDay-1 retention for the March 15 cohort is 8 points lower than March 14, but March 16 bounced back. What does this pattern suggest and how do you verify?Isolating cohortsMid–senior
Checkout abandonment spikeCheckout abandonment spiked 25% at 3pm yesterday and stayed elevated for two hours, then returned to normal. Walk me through your hypothesis list.Hypothesis prioritizationMid–senior
Search CTR drop logged-in usersSearch result click-through rate dropped 6% globally, but when you filter to logged-in users it's only down 2%. What does that tell you and what's next?Segmentation strategySenior–staff+
Events up users flat instrumentationYour instrumentation library was upgraded three days ago. Since then, event counts are up 5% but user counts are flat. Should you be worried?Ruling out instrumentation issuesEntry–mid
Claimed activation improvement not visibleA product manager claims their new onboarding flow improved activation by 12%, but you see the metric is flat. How do you approach this conversation?Communicating uncertaintyMid–senior
WAU up but MAU downWeekly active users grew 3% but monthly active users dropped 2% over the same four-week period. What dynamics could explain this and how would you validate?Hypothesis prioritizationSenior–staff+
Statistical significance of conversion dropYour VP asks whether a 4% drop in trial-to-paid conversion is statistically significant. You have 2,000 trials this month. How do you respond?Communicating uncertaintyMid–senior
Notification opens down one countryNotification open rates are down 20% in one country but flat everywhere else. What country-specific factors would you investigate first?Segmentation strategyMid–senior
Events per session increased unexpectedlyYou're seeing 30% more events logged per session since last Tuesday, but product and engineering both say nothing changed. What's your debugging process?Ruling out instrumentation issuesMid–senior
Referral cohort retention gapCohorts who signed up via referral have 20% lower day-7 retention than other channels, but referral volume is growing. Do you raise this now or wait? Why?Communicating uncertaintySenior–staff+
Time-to-purchase increased, day-30 flatTime-to-first-purchase increased from 3 days to 5 days over the last month, but purchase rate at day-30 is unchanged. What might be happening?Isolating cohortsSenior–staff+
API errors iOS version specificAPI error rates doubled overnight but only for requests from iOS clients on version 2.1.3. How do you prioritize what to check?Hypothesis prioritizationMid–senior
Campaign signups don't match attributionMarketing reports that last week's campaign drove 10,000 signups, but you only see 7,500 new users in that attribution window. How do you reconcile this?Ruling out instrumentation issuesMid–senior
Per-user revenue growth math mismatchRevenue per user is up 15% month-over-month, but your finance team says total revenue is only up 8%. What's the first thing you investigate?Segmentation strategySenior–staff+
MAU dropped after deduplication changeThe data team rolled out a new user deduplication logic and MAU dropped 3%. The CEO wants to know if growth actually slowed. What do you tell them?Communicating uncertaintySenior–leadership
Mobile web bounce rate spikeBounce rate on your landing page increased from 35% to 55% on mobile web, but mobile app metrics are fine. What are your top hypotheses?Hypothesis prioritizationMid–senior
Experiment lift disappeared after shippingYour experiment showed a 10% lift in the treatment group, but when you shipped it to everyone, the metric stayed flat. Walk me through possible explanations.Hypothesis prioritizationSenior–staff+
Power users usage declinePower users (top 10% by activity) dropped their usage 15% last week, but the other 90% are unchanged. How would you slice this further?Segmentation strategySenior–staff+
Six-week metric decline diagnosisA metric has been trending down slowly for six weeks. Your manager asks if it's a data issue or a product issue. You're not certain. How do you answer?Communicating uncertaintyMid–senior
Annual subscription renewal rate declineSubscription renewal rate for annual plans dropped 5 points, but monthly renewals are steady. What cohort dimensions would help you understand the gap?Isolating cohortsSenior–staff+
Personal auto loss ratio spikeThe personal auto loss ratio jumped seven points quarter over quarter. Walk me through what you check first.Segmentation strategyMid–senior
Homeowners severity increase diagnosisHomeowners claim severity is up 12% year over year while frequency is flat. How do you prioritize the possible explanations?Hypothesis prioritizationMid–senior
Loss ratio spike after migrationTwo months after your claims-system migration, the reported loss ratio for the migrated lines spiked. How do you determine whether the book deteriorated or the data did?Ruling out instrumentation issuesSenior–staff+
Retention drop in long-term customersRenewal retention fell four points this quarter. First-term customers are renewing at normal rates, but customers with five-plus years of tenure are leaving at twice their usual rate. What's your next step?Isolating cohortsMid–senior
Early favorable loss ratio guidanceSix months into the accident year, your loss ratio is running six points better than plan, and the CFO wants to reflect it in guidance and slow the next rate filing. What do you say?Communicating uncertaintySenior–leadership