You are using a browser not supported by the Wolfram Cloud
Supported browsers include recent versions of Chrome, Edge, Firefox and Safari.
I understand and wish to continue anyway »
WOLFRAM NOTEBOOK
Exploring Robustness of Mean-Difference Confidence Intervals
Exploring Robustness of Mean-Difference Confidence Intervals
This Demonstration examines confidence intervals at the 95% level for the difference in means in random samples ,…, and ,…, from either a normal, uniform, Laplace, or centered exponential distribution. The distributions all have mean 0. The variance of the distribution can be varied from 1 to 10 and the variance of is fixed at 1. Several methods available in the Mathematica function MeanDifferenceCI are examined. The "t (Welch)" method is the default method with MeanDifferenceCI, "t (pooled)" corresponds to the option setting , while "Z" and "Z (pooled)" correspond to setting the option KnownVariance to the sample variances of and or else to the pooled variance estimate. The fifth method is a conservative approximation using the two-sample t-statistic with degrees of freedom equal to .
X
1
X
n
X
Y
1
Y
n
Y
X
Y
EqualVariancesTrue
X
Y
min(,)-1
n
x
n
y
The scaling on the horizontal axis is in terms of =, the standard deviation of .
σ
δ
1+
2
σ
x
X-Y
Each iteration shows 100 confidence intervals, the nominal coverage probability , and the empirical coverage probability . By changing the random seed, more confidence intervals are produced and an increasingly accurate estimate of the true coverage probability is obtained. Try doing at least 10,000 simulations.
C
C
If the true coverage probability is greater than the nominal one, in this case set to 95% in our initialization code, the method is said to be conservative. Ideally the empirical coverage probability should be close to the nominal value. As an approximation, conservative confidence intervals are much more acceptable than ones that are not.
Using this Demonstration you can easily find that the "t (pooled)" method is not conservative when the variances are not equal. Also the -methods do not work well unless the sample sizes are quite large. Overall the "t (Welch)" works best even though it is also an approximation. The "t (pooled)" is exact for normal populations with equal variances but is not recommended since it is not robust when this assumption does not hold. The loss in degrees of freedom with the assumption of unequal variances is usually not as important.
Z
The 95% confidence interval for is equivalent to a test of the null hypothesis :δ=0 versus :δ≠0 at level . The empirical type I error rate is estimated by =1-. The red lines correspond to cases where is rejected at the 5% level.
δ=-
μ
Y
μ
X
H
0
H
a
α=0.05
α
C
H
0