How to Choose the Right Statistical Method in A/B Testing: A Comparison Guide of Linear Regression and Other Tools

Beyond the Surface: Why Linear Regression Still Matters

In the wave of machine learning and deep learning, we often overlook a classic and powerful tool—linear regression. While LLMs and advanced architectures dominate headlines, linear regression continues to play a crucial role in data analysis, especially in A/B testing scenarios.

Let’s consider a real-world example: an e-commerce company launches a new banner design and needs to evaluate its impact on users’ average session duration. By collecting data through experiments and performing statistical analysis, we’ll explore various methods to interpret these results.

Quick Insights with T-Test

Start with the classic T-Test. The experimental data shows a significant effect: the difference in mean session duration between treatment and control groups is 0.56 minutes, meaning users spend an average of 33 seconds more on the product.

This metric looks promising, but does it truly reflect the banner’s real impact?

Linear Regression: Deeper Exploration

Now, re-analyze using linear regression. Use the treatment variable (whether the new banner is shown) as an independent variable, and session duration as the dependent variable. What does the model summary reveal?

The coefficient for the treatment variable is exactly 0.56—consistent with the T-Test result. Interestingly, the R-squared value is only 0.008, indicating that the model explains a very small portion of the variance in the data.

Is this just a coincidence? No.

Why do these two methods produce the same result? The answer lies in their mathematical foundation.

In linear regression, when the treatment variable is 1, it represents the average session duration for users who received the treatment; when 0, it represents the average for users who did not. Therefore, the treatment coefficient is essentially the difference between the two group means.

The null hypothesis tested by the T-Test (no difference between group means) is exactly the same as the null hypothesis for the treatment coefficient in linear regression. When the null hypotheses are the same, the calculated T-statistics and P-values from both methods will also be identical.

Why use linear regression at all?

A simple mean comparison might seem sufficient, but the real world is much more complex.

In reality, relying solely on the treatment variable may not explain all variations—systematic biases often exist. For example:

• Experienced users are more likely to encounter the new banner
• Users with different demographics respond differently to the banner

While random assignment helps mitigate this, it doesn’t eliminate it entirely. That’s why we need control variables (covariates).

By including users’ average session duration before the experiment as a control variable, the model’s performance improves immediately: R-squared jumps to 0.86, meaning we now explain 86% of the variance in the data.

The new estimated treatment effect is 0.47 minutes.

Which number is more accurate?

Now, two different treatment effects appear: 0.56 and 0.47. Which one reflects reality?

In simulated data, the true effect was set at 0.5. Clearly, the 0.47 estimate—obtained after adding control variables—is closer to the true value, with an error of only 0.03. This demonstrates that controlling key covariates can significantly improve estimation accuracy.