Skip to content

Quiz: Advanced Analytics and Experimentation

Test your understanding of experiment design, A/B testing, statistical significance, data pipelines, ETL processes, attribution modeling, customer segmentation, and predictive analytics with these review questions.

1. What is the standard p-value threshold used to declare statistical significance in A/B testing?

  1. 0.50 (50%)
  2. 0.05 (5%)
  3. 0.01 (1%)
  4. 0.10 (10%)
Show Answer

The correct answer is B. The chapter states that "a p-value below 0.05 (5%) is the standard threshold for declaring significance, meaning there is less than a 5% chance the result is due to random variation." This threshold controls the Type I error rate (false positives). While more stringent thresholds like 0.01 exist for high-stakes decisions, 0.05 is the conventional standard described in the chapter for product experimentation.

Concept Tested: Statistical Significance

2. Which component of a well-designed experiment ensures that observed differences in outcomes can be attributed to the change being tested rather than pre-existing differences between user groups?

  1. The hypothesis statement
  2. Random assignment of users to control and treatment groups
  3. The sample size calculation
  4. The guardrail metrics
Show Answer

The correct answer is B. The chapter explains that in A/B testing, "the randomization is critical because it ensures that differences in outcomes can be attributed to the change rather than to pre-existing differences between the groups." Without randomization, observed differences could reflect selection bias rather than the effect of the change. The chapter illustrates this with an example about button colors: without random assignment, more motivated users might simply prefer a particular option, creating misleading results.

Concept Tested: A/B Testing

3. What is the key difference between the ETL and ELT data processing patterns?

  1. ETL transforms data before loading it into the warehouse while ELT loads raw data first and transforms it inside the warehouse
  2. ETL uses cloud storage while ELT requires on-premises data centers
  3. ETL processes data in real-time while ELT operates only in batch mode
  4. ETL is designed for structured data while ELT can only process unstructured data
Show Answer

The correct answer is A. The chapter's ETL vs. ELT comparison table shows the core difference: in ETL, the transform happens "before loading (in pipeline)" while in ELT, it occurs "after loading (in warehouse)." The chapter explains that ELT gained popularity because "modern cloud warehouses like Snowflake and BigQuery are powerful enough to handle transformations at query time, reducing pipeline complexity." ELT offers more flexibility since you can "transform on demand" rather than redesigning the pipeline for new analyses.

Concept Tested: ETL Process

4. A colleague wants to stop an A/B test after two days because the treatment variant appears to be winning. Based on the chapter's experiment design principles, why is this problematic?

  1. A/B tests must always run for exactly 30 days to meet regulatory requirements
  2. The feature flag system requires a minimum runtime of one week before variants can be evaluated
  3. Two days is insufficient for the randomization service to assign users evenly across groups
  4. Stopping early violates the pre-calculated sample size and duration needed to capture full business cycles, risking a false positive conclusion
Show Answer

The correct answer is D. The chapter's experiment design table explicitly lists stopping an experiment early as a common mistake, noting that good practice requires experiments to run "through full business cycles (weekday + weekend)" while the common mistake is "stopped early when results 'look good.'" Additionally, the sample size must be "calculated before launch using power analysis" to ensure sufficient statistical power. Stopping early means the experiment has not accumulated enough data to distinguish real effects from random variation, increasing the risk of a Type I error (false positive).

Concept Tested: Experiment Design

5. In the position-based (U-shaped) attribution model, how is credit distributed across touchpoints in a customer journey?

  1. 100% credit goes to the first touchpoint only
  2. Equal credit is distributed to every touchpoint in the journey
  3. Credit increases progressively with each touchpoint closer to conversion
  4. 40% each to the first and last touchpoints, with 20% distributed among middle interactions
Show Answer

The correct answer is D. The chapter describes the position-based (U-shaped) model as assigning "40% credit each to first and last touchpoints, 20% distributed among middle interactions." This model "balances awareness and conversion" by giving significant weight to both the touchpoint that introduced the user (first touch) and the one that closed the deal (last touch), while acknowledging that middle interactions also contributed. This contrasts with first-touch (100% to first), linear (equal distribution), and time-decay (progressive increase toward conversion).

Concept Tested: Attribution Modeling

6. Your product analytics team builds a model predicting which users are likely to churn within 30 days. The model assigns an 85% churn probability to a specific user. How should you interpret this prediction according to the chapter?

  1. The user will definitely churn and should be removed from active customer counts immediately
  2. The prediction is unreliable because no model can accurately predict individual user behavior
  3. Among users who looked similar in the past, about 85% did churn, so this is a signal to take proactive retention action
  4. The user has already decided to leave and no intervention can change the outcome
Show Answer

The correct answer is C. The chapter explicitly addresses this scenario: "A churn prediction model that says a user has an 85% likelihood of churning is not guaranteeing that outcome. It means that among users who looked similar in the past, about 85% did churn." The chapter frames predictions as probabilities, not certainties, and describes them as "a signal to take action - perhaps trigger a personalized retention campaign - not a foregone conclusion." This understanding is essential for PMs who must communicate model limitations to stakeholders.

Concept Tested: Predictive Analytics

7. Which type of analytics does the chapter recommend for fraud detection and in-session personalization, and why does it caution PMs to evaluate whether these use cases truly justify the investment?

  1. Batch analytics, because it provides the most accurate results at the lowest cost
  2. Predictive analytics, because machine learning models can identify fraudulent patterns before they occur
  3. Cohort analytics, because grouping users by behavior reveals fraudulent segments efficiently
  4. Real-time analytics, because it processes data in seconds but comes with higher complexity and cost than batch alternatives
Show Answer

The correct answer is D. The chapter identifies real-time analytics as essential for "fraud detection, live dashboards, personalization, alerting, and operational monitoring." However, it explicitly cautions PMs about the trade-offs: "real-time analytics is more expensive, more complex to maintain, and more difficult to debug than batch analytics. Always ask whether the use case truly requires real-time data before requesting it." The comparison table shows real-time analytics has "high" complexity and cost versus "low" for batch processing.

Concept Tested: Real-Time Analytics

8. A product team segments its users into five groups based on feature usage patterns, engagement frequency, and purchase history. According to the chapter, what type of segmentation is this?

  1. Demographic segmentation
  2. Value-based segmentation
  3. Behavioral segmentation
  4. Psychographic segmentation
Show Answer

The correct answer is C. The chapter defines behavioral segmentation as grouping users by "feature usage patterns, engagement frequency, purchase history." It notes that for technical PMs, "behavioral segmentation is particularly powerful because it is derived from actual product usage data rather than self-reported characteristics." This contrasts with demographic (age, location, company size), psychographic (goals, motivations, pain points), and value-based (revenue contribution, lifetime value) segmentation dimensions.

Concept Tested: Customer Segmentation

9. Your A/B test shows a statistically significant conversion rate improvement of 0.01% with a p-value of 0.03. Based on the chapter's guidance, what should you conclude?

  1. The result is statistically significant but may not be practically significant, so you should evaluate whether the improvement justifies the cost of maintaining the change
  2. Ship the treatment variant immediately since the result is statistically significant and therefore valuable
  3. The result is invalid because conversion rate improvements below 1% cannot be detected reliably by A/B tests
  4. Run the test longer because a p-value of 0.03 is not below the standard 0.05 threshold
Show Answer

The correct answer is A. The chapter directly addresses this scenario: "A result can be statistically significant but practically meaningless. If your A/B test shows a conversion rate improvement of 0.01% with high confidence, the effect is real but may not justify the engineering cost of maintaining the change." The chapter advises PMs to "always pair statistical significance with a practical significance threshold defined before the experiment." A p-value of 0.03 is below the 0.05 threshold, confirming statistical significance, but the tiny effect size raises questions about business value.

Concept Tested: Statistical Significance / Conversion Rate

10. An event tracking implementation captures a "checkout_completed" event with properties including cart_value, items_count, payment_method, coupon_applied, and experiment_variant. Which common event tracking mistake does this implementation successfully avoid?

  1. Over-tracking by capturing too many irrelevant user interactions
  2. Missing properties by capturing the event without the context needed for meaningful analysis
  3. No versioning by failing to document schema changes over time
  4. Inconsistent naming by mixing camelCase and snake_case conventions
Show Answer

The correct answer is B. The chapter lists "missing properties - capturing the event but not the context needed for meaningful analysis" as a common event tracking mistake. The example event in the chapter includes rich contextual properties (cart_value, items_count, payment_method, coupon_applied, experiment_variant) that enable meaningful analysis beyond just knowing a checkout occurred. These properties allow segmentation by payment method, coupon impact analysis, and experiment result calculation. The chapter emphasizes that a well-designed event should answer four questions: who, what, when, and what was the context.

Concept Tested: Event Tracking