1. 基本问题

Classification of inspection indicators

ABTest indicators can be divided into two categories：绝对值指标、Proportional value indicator.The variance is calculated differently for the two.
The proportional values ​​can be divided into two categories according to the different denominators：The denominator is person times（留存率、转化率等）、The denominator is the number of actions（曝光点击率）.
The denominator is person times,The split unit and the analysis unit are the same,可以用$z$检验;The denominator is the number of actions,The units of analysis are not independent,要用$delta$检验.

Cumulative value over multiple days

The performance of indicators over multiple days is aggregated and calculated.For example, the per capita frequency of a certain behavior,Then the denominator is the total number of times that behavior occurred during the experiment,The numerator is the number of deduplicated people who entered the group during the experiment.

优点：To ensure independence between samples;增加样本量,Significance can increase with accumulation.

Multi-day accumulation of retention rate：Calculate the retention rate of new recruits every day,Then weighted according to the number of people.

不能用AB的情况

• When the intervention variable cannot be controlled（For example, the impact of watching live broadcasts on users,Some people cannot be forced to watch,Some people don't see it）
• Too much traffic is being used
• Policies can harm the user experience

AB实验步骤

Determine the experimental strategy;Develop experimental indicators of observation;计算样本量（显著性水平/统计功效/The minimum level of improvement in the indicator that needs to be observed/指标方差）;Experimental development online;数据回收.

AB不显著

• Whether the minimum sample size is reached
• DIDEliminate fixed differences
• Check the experiment link,See if everyone is reached by the strategy（渗透率低,可以PSM）

2. delta检验

见上篇,Applicable when the split unit and analysis unit are different.

3. 贝叶斯检验

优点：

• Sample size does not need to be considered.
• The distribution of the posterior parameters can be obtained,Then quantify the probability of the improvement of the index、The size of the metric lift.

贝叶斯派 VS 频率派 基本理论：

先验分布 $\pi (\theta )$ + 样本数据 $P(X|\theta )$ = 后验分布 $\pi (\theta |X)$

共轭先验分布：贝塔分布 与 二项分布

$\theta$~$beta(\alpha ,\beta )$,$X$~$Binomial(n,p)$, 则$\theta |X$~$beta(x+\alpha ,n-x+\beta )$

4. Different hypothesis tests

• $z$检验：Large sample data mean test（Distributions are not differentiated,中心极限定理;Does not distinguish whether the variance is known or not,n>30时t分布和z分布相似）

• $t$检验：Small sample normal data mean test(小于30,方差未知)

• $F$检验：方差齐性检验;单因素方差分析,Test the effect of the value of each level of a categorical variable.

• 卡方检验：The essence is to test whether the sample frequency is consistent with the expectation.Can be used to test the correlation between two sets of discrete variables（列联表）;Test the similarity between the actual distribution and the expected distribution,Nonparametric tests are mostly used for categorical variables.

  $X^2=\Sigma (X-E{)}^2/E$
  令$E=np$,The square of the normal distribution can be obtained.

• $k$-$s$检验：Whether the sample satisfies a specific distribution;Look at the maximum value of the difference between the sample cumulative distribution and the theoretical cumulative distribution.

DID

$y={\alpha }_1\ast treatment+\alpha 2\ast post+{\alpha }_3\ast treatment\ast post+u$

${\alpha }_3$represents the net effect of the policy

平行趋势检验

参考文献

ABexperimental interview
https://www.jiqizhixin.com/articles/2020-09-18-2
https://blog.csdn.net/deephub/article/details/112167937