Data Scientist Interview Kit

Question 1: Explain p-values to a non-technical stakeholder. Then discuss their limitations.

What to Look For

• • Clear, simple explanation for layperson
• • Understands null hypothesis framework
• • Mentions statistical vs practical significance
• • Discusses multiple testing problem
• • Understands p-hacking risks
• • Suggests alternatives or complements

Red Flags

• • Can't explain simply
• • Misunderstands p-value meaning
• • Thinks p-value = probability hypothesis is true
• • No awareness of limitations
• • Over-reliance on p-values

Follow-up Questions

• • What's the difference between Type I and Type II errors?
• • How do you handle multiple comparisons?
• • When would you use Bayesian methods instead?

Question 2: You're A/B testing a new feature. How do you determine sample size, and when do you stop the test?

What to Look For

• • Power analysis for sample size
• • Defines effect size of interest
• • Considers statistical power and significance
• • Addresses early stopping problem
• • Mentions sequential testing if appropriate
• • Considers practical constraints

Red Flags

• • No systematic approach
• • Stops at first sign of significance
• • Doesn't consider power
• • No mention of effect size
• • Ignores multiple testing issues

Follow-up Questions

• • What if you need results faster?
• • How do you handle multiple metrics?
• • What's the problem with peeking at results?