Metric diagnosis questions.
86 metric diagnosisquestions from the bank — open to read. Pick one and practice it out loud; a coach note comes back in seconds.
Learn the ideas first
Funnel metrics and conversion decompositionMetric anomaly triage: real move or artifact?Answer-first: delivering findings under time pressureCohort analysis and retention curvesSegmenting a metric move: mix shift and Simpson's paradoxStatistical power and sample sizeUnit economics: CAC, LTV, and paybackWhen you can't A/B test: quasi-experiments and holdouts
All 86 questions
DAU dropped 8% on Tuesday. What do you check first?Revenue dropped 12% overnight in your dashboard, but customer support hasn't gotten any reports. Walk me through your investigation.Sign-up conversion dropped 5% last week. New users from organic traffic are flat, but paid-traffic users are down 20%. What's your next step?Engagement on a feature spiked 30% after a release but no one shipped a change to it. What's most likely and where do you start?Your team is asked to explain a metric anomaly in 30 minutes but you're not sure of the cause yet. What do you say?Average session duration grew 15% last month but DAU is flat. Walk me through what that combination tells you.A new metric was added two weeks ago and it's showing inconsistent results between teams. Walk me through what to look for.Retention dropped for users who signed up in the last 30 days, but earlier cohorts look fine. What does that point to?Conversion rate dropped, but no engineering changes shipped this week. What ten things are still possible?Your CEO asks if Tuesday's metric dip is a real concern. You're not sure. How do you answer?Click-through rate fell on your homepage. Walk me through how you'd slice the data.A growth metric has been flat for two weeks even though stakeholders swear the product has improved. What's worth checking?Premium users churn at 2% per month, free users at 5%. This quarter premium churn is at 3.5%. What do you investigate?Your retention dropped on a single day three weeks ago and you only just noticed. What do you do?Your team needs to brief leadership that a metric is moving in the wrong direction but you only have a hypothesis. Walk me through your briefing.A KPI improved this month. How do you tell whether it's a real change versus normal noise?After a SDK release, conversion attribution shifted between two channels. What's worth checking?Engagement is up in your most-active 5% of users but down across everyone else. How do you read that?Your monthly active users grew 4% but revenue is flat. Walk me through likely causes.You're asked, 'should we be worried?' about an anomalous metric. Walk me through your answer.When you see an anomaly, what slices do you reach for almost reflexively, in what order?A teammate says 'the data is wrong'. Before you agree or argue, what do you check?A new feature launched and total time-in-app went up — but only for the existing power users. What does that tell you?Pageviews dropped while sessions stayed flat. What does that combination mean?Walk me through how you'd present a metric anomaly to a non-technical exec without losing precision.What's the difference between a Simpson's-paradox-flavored metric move and a real change, and how do you tell?A change in conversion shows in your warehouse but not in the product team's event tool. Where do you start?Engagement dropped 6% but ad revenue is up. What might be going on?A weekly KPI moved 2 standard deviations above its mean. What's your read on whether to investigate?Walk me through how you'd write the one-paragraph summary of a metric anomaly investigation when the cause is unknown after a day.Your mobile app's daily active users dropped 10% yesterday, but only on Android. iOS is flat. What are your first three checks?A metric started tracking two days ago and shows wildly different values each hour. What do you check to rule out a logging problem?Sign-ups dropped 15% last Thursday. You segment by country and see the US is down 40% but all other countries are up slightly. What's your next step?Your dashboard shows average order value increased 20% overnight, but the raw event count also doubled. What might be wrong with the data?Users who signed up in January have 60% week-1 retention, but February sign-ups have 45%. Walk me through how you'd investigate this cohort difference.You have three hypotheses for why checkout conversion dropped: a payment bug, slower page load, or a marketing campaign change. How do you prioritize which to check first?Your manager asks if yesterday's 7% dip in video plays is something to escalate. You haven't finished investigating. What do you tell them?A metric looks correct in your analytics tool but shows different numbers in the SQL query you wrote. What are the first two things you check?Desktop users have 3% conversion and mobile users have 1.5%. This week mobile conversion dropped to 1.1%. How do you investigate whether this is a mobile-specific issue?Your product had a small UI change and a backend deploy on the same day. A metric moved. How do you figure out which change caused it?You're writing a Slack update about a metric drop and you have two competing theories but no conclusion yet. How do you structure the message?Search queries increased 25% yesterday but you just learned the logging was accidentally duplicated for two hours. How do you correct your understanding of the real change?Your activation rate jumped 20% the same week a tracking-schema migration shipped. The growth team wants to announce the win. What now?Your AI assistant's thumbs-up rate has climbed for six weeks, but repeat usage of the assistant is falling. Walk me through the hypotheses you'd test, in order.Pageviews and sign-up starts jumped 30% overnight, but completed sign-ups and activation are flat. You suspect bot or AI-crawler traffic. How do you confirm or rule that out?Your weekly metric is down 10%, but the week included a public holiday in your largest market. How do you work out how much of the drop is just the calendar?Your metric didn't actually drop — it grew, but missed forecast by 15%. Leadership is treating it like a decline. How do you investigate it, and how do you reframe the conversation?Your LLM-powered feature's cost per active user doubled this week, but usage volume is flat. What do you check?A third-party model provider silently upgraded the model behind your feature, and your task-success metric moved the same week. With no A/B test in place, how do you establish whether the upgrade caused it?Average revenue per user rose 12% this quarter while median revenue per user fell. What's going on, and how do you confirm it?Sign-up-to-paid conversion fell 3 points the same week you launched in three new countries. Is anything actually wrong? Walk me through your check.Your weekly dashboard has shown exactly 0.0% change for three straight weeks. Why is that itself suspicious, and what do you check?Every morning at 9am, yesterday's numbers look 20% low, then recover by noon. A stakeholder keeps escalating the morning dip. What's happening, and what do you do about it?Users acquired through your new AI-recommendation surface retain 20% worse than other channels, and the team wants to kill the surface. Is the surface bad, or is it selection? How do you tell?Your app-store rating fell from 4.4 to 3.8 — but only among users on the newest app version. What do you investigate first?NPS dropped 12 points this quarter — but survey response rate also halved after an email-deliverability change. How do you decide whether sentiment actually fell?Average order value jumped 18% yesterday, but total orders dropped 10%. Walk me through how you'd diagnose this.Mobile conversion rate dropped 15% but desktop is unchanged. What dimensions would you segment by to narrow down the root cause?Your analytics team reports that page load time increased 2 seconds last Thursday, but engineering says no deploys happened. What are the first three things you check?Two dashboards show different MAU numbers for the same time period—one shows 2.1M, the other 2.4M. How do you resolve this discrepancy?Paid acquisition cost per install stayed flat, but cost per paying user rose 40% this month. What cohort cuts help you understand why?Day-1 retention for the March 15 cohort is 8 points lower than March 14, but March 16 bounced back. What does this pattern suggest and how do you verify?Checkout abandonment spiked 25% at 3pm yesterday and stayed elevated for two hours, then returned to normal. Walk me through your hypothesis list.Search result click-through rate dropped 6% globally, but when you filter to logged-in users it's only down 2%. What does that tell you and what's next?Your instrumentation library was upgraded three days ago. Since then, event counts are up 5% but user counts are flat. Should you be worried?A product manager claims their new onboarding flow improved activation by 12%, but you see the metric is flat. How do you approach this conversation?Weekly active users grew 3% but monthly active users dropped 2% over the same four-week period. What dynamics could explain this and how would you validate?Your VP asks whether a 4% drop in trial-to-paid conversion is statistically significant. You have 2,000 trials this month. How do you respond?Notification open rates are down 20% in one country but flat everywhere else. What country-specific factors would you investigate first?You're seeing 30% more events logged per session since last Tuesday, but product and engineering both say nothing changed. What's your debugging process?Cohorts who signed up via referral have 20% lower day-7 retention than other channels, but referral volume is growing. Do you raise this now or wait? Why?Time-to-first-purchase increased from 3 days to 5 days over the last month, but purchase rate at day-30 is unchanged. What might be happening?API error rates doubled overnight but only for requests from iOS clients on version 2.1.3. How do you prioritize what to check?Marketing reports that last week's campaign drove 10,000 signups, but you only see 7,500 new users in that attribution window. How do you reconcile this?Revenue per user is up 15% month-over-month, but your finance team says total revenue is only up 8%. What's the first thing you investigate?The data team rolled out a new user deduplication logic and MAU dropped 3%. The CEO wants to know if growth actually slowed. What do you tell them?Bounce rate on your landing page increased from 35% to 55% on mobile web, but mobile app metrics are fine. What are your top hypotheses?Your experiment showed a 10% lift in the treatment group, but when you shipped it to everyone, the metric stayed flat. Walk me through possible explanations.Power users (top 10% by activity) dropped their usage 15% last week, but the other 90% are unchanged. How would you slice this further?A metric has been trending down slowly for six weeks. Your manager asks if it's a data issue or a product issue. You're not certain. How do you answer?Subscription renewal rate for annual plans dropped 5 points, but monthly renewals are steady. What cohort dimensions would help you understand the gap?The personal auto loss ratio jumped seven points quarter over quarter. Walk me through what you check first.Homeowners claim severity is up 12% year over year while frequency is flat. How do you prioritize the possible explanations?Two months after your claims-system migration, the reported loss ratio for the migrated lines spiked. How do you determine whether the book deteriorated or the data did?Renewal retention fell four points this quarter. First-term customers are renewing at normal rates, but customers with five-plus years of tenure are leaving at twice their usual rate. What's your next step?Six months into the accident year, your loss ratio is running six points better than plan, and the CFO wants to reflect it in guidance and slow the next rate filing. What do you say?