The APP procedure for estimating the Cohen's effect size Open Access

Cohen (e.g. 1988) famously argued that researchers should be concerned not only with whether an effect is present but with the size of the effect too. Cohen discussed a variety of different effect size indices, and other researchers have added to the effect size toolbox. Nevertheless, for typical studies, where economic data for two groups are compared, economic data for two countries are compared, or where an experimental group is compared to a control group, Cohen's d remains by far the most popular effect size index. As will be explained in more detail in the subsequent section, Cohen's d denotes the difference in means divided by the standard deviation. Thus, Cohen's d provides valuable information about how much the means differ in standard deviation units.

One reason that scientists in the social and economic sciences have found Cohen's d useful is that many of the dependent measures these sciences do not have intrinsic meaning. For instance, whereas it might be reasonably clear what a dollar means, the meaning of a unit on an attitude scale might be less clear. If the mean attitude in the experimental condition is 2 and the mean attitude in the control condition is 1, is this a small difference or a large one? By converting the difference in means to Cohen's d, the researcher can gain an idea of the size of the difference in standard deviation units, even when the scale units are not themselves intrinsically meaningful. Another advantage of Cohen's d is that it facilitates comparisons across studies. Even if scale units are different for different studies, thereby rendering them seemingly impossible to compare, researchers still can compare in terms of standard deviation units. Many researchers have taken advantage of this, particularly in meta-analytic research. Despite the popularity of Cohen's d and its obvious usefulness, there remains an important limitation. Specifically, the Cohen's d that a researcher obtains in a particular experiment is a sample statistic. It is not a population value. Typically, researchers are not interested in sample statistics for their own sake, but because they provide useful estimates of population values. Thus, there is an important question that has not been properly addressed: how well does Cohen's d estimate the population effect size? Although researchers have long known how to compute traditional confidence intervals for Cohen's d, traditional confidence intervals do not properly address the question. This is because, for example, although 95% of 95% confidence intervals surround the population parameter, it is not the case that the population parameter has a 95% chance of being within a 95% confidence interval. This last is unknown. In addition, Trafimow and Uhalt (2020) have shown that sample confidence intervals tend to be inaccurate representations of population confidence intervals unless sample sizes are much larger than those typically employed. An alternative way to address the issue is to use the a priori procedure (APP) that has been employed previously in a variety of ways not pertaining to Cohen's d (e.g. Li et al., 2020; Trafimow, 2017, 2019; Trafimow and MacDonald, 2017; Trafimow et al., 2020a; Wang et al., 2020, 2021; Wei et al., 2020). Although the APP uses confidence intervals, it does so in a way that deviates importantly from traditional confidence intervals. To use APP thinking to address Cohen's d, the researcher would ask the two bullet-pointed questions below.

For example, the research might wish to have a 95% probability of obtaining Cohen's d within a tenth of a standard deviation of the population value. The present goal is to determine the sample size the researcher needs to collect to meet the precision and confidence specifications in the contexts of independent and matched samples experimental designs.

This paper is organized as follows. Definitions of Cohen's effect sizes for both populations and samples are given in Section 2, together with properties of noncentral t distributions. In Section 3, the APP methods are applied for estimating population effect size θ in independent case and θ_D in dependent case. In Section 4, the simulation study, the coverage rate, and real data examples are provided in supporting our main results given in Section 3. Conclusion remarks are given in Sections 5.

Effect size is a statistical concept that measures the strength of the relationship between two variables on a numeric scale. For example, in medical education research studies that compare different educational interventions, effect size is the magnitude of the difference between groups. The absolute effect size is the difference between the average, or mean, outcomes in two different intervention groups. The standard deviation of the effect size is of critical importance, since it indicates how much uncertainty is included in the measurement. For more details and applications, see, Sullivan and Feinn (2012), Schafer and Schwarz (2019), Bhandari (2020).

Cohen's d is one of the most common ways to measure effect size, which is known as the difference of two population means and it is divided by the standard deviation from the data. Mathematically, Cohen's effect size is denoted by:

where μ₁ and μ₂ are means of two populations, and σ is the standard deviation based on either or both populations.

Cohen's d is defined as the difference between two means divided by a standard deviation for the data obtained from both populations:

where $X̅1$ and $X̅2$ are sample means and S, defined by Jacob Cohen, is the pooled standard deviation (for two independent samples)

and n₁, S₁, n₂, S₂ are sample sizes and sample variances of two independent samples, respectively.

Note that confidence intervals of standardized effect sizes, especially Cohen's d, rely on the calculation of confidence intervals of noncentrality parameters. In order to find the minimum sample sizes for estimating the Cohen's effect size θ given in (2.1) by Cohen's d given in (2.2), we need the following definition.

It is easy to obtain the following properties of T ∼ t_m(λ) (see Nguyen and Wang, 2008).

1. The probability density function (pdf) of T is given by

1. The mean and variance of T are

and

respectively. For convenience, if we use the correction factor J(m) given by

then the mean and the variance of T are

and

We will use the following results in the proofs of our main results to be given in next section.

Proof. From the basic statistics, we know that.

1. $X̅1−X̅2∼N(μ1−μ2,(n1−1+n2−1)σ2)$ so that $Z≡n1n2n1+n2d∼N(λ,1)$⁠.

2. $(n1−1)S12σ2∼χn1−12$ and $(n2−1)S22σ2∼χn2−12$ so that $(n1+n2−2)S2σ2∼χn1+n2−22$ as $S12$ and $S22$ are independent.

Now by Definition 2.1, $T=n1n2n1+n2d∼tn1+n2−2(λ)$⁠, the desired result follows. □

For the dependent case, we have the following result.

Proof. Note that $D̅∼N(μ1−μ2,2(1−ρ)σ2n)$ so that $Z≡nD̅2(1−ρ)σ∼N(λD,1)$⁠, where $λD=n2(1−ρ)θD$⁠. Also it is easy to know that

and $D̅$ and $SD2$ are independent. Therefore, by Definition 2.1,

so that the desired result follows. □

In this section, we will apply the APP methods for estimating Cohen's effect size θ in independent samples' case and θ_D in matched sample case.

First, we consider two independent samples from two normal populations $N(μ1,σ12)$ and $N(μ2,σ22)$ with equal unknown variances: $σ12=σ22=σ2$⁠.

Proof. By Proposition 2.1, we know that $T1=n1n2n1+n2d∼tn1+n2−2(λ)$⁠, where $λ=n1n2n1+n2θ$⁠. Thus, the mean and variance of d, are given, respectively, by

and

Now it is easy to obtain the density of $H=d−E(d)σd$⁠, which is symmetric about 0 and given by

where $fT1$ is the density of T₁. Note that if we have two independent random samples of sizes n₁ and n₂, we can construct the 100c% confidence interval for θ based on (3.7). Now, let $T2=d12=X1−Y12S$⁠, which has a noncentral t distribution with n₁ + n₂ − 2 degrees of freedom and noncentrality parameter $θ/2$⁠. Thus, the unbiased estimator of θ is $2T2$⁠. Note that the variance of d₁ is

Therefore, we can setup the confidence interval for given confidence level c and precision f, which is given in (3.1). Since there are two unknowns n₁ and n₂, there are no solutions using Equation (3.1) so we need to modify this equation. Let n =  min{n₁, n₂} so that the degrees of freedom n₁ + n₂ − 2 ≥ 2n − 2. Suppose that we have two independent samples of size n in both; then, $S*2=S12+S222$⁠, and the distribution of $d*=(X̅1−X̅2)/S*$ is the noncentral t with 2(n − 1) degrees of freedom and the noncentrality parameter $λ*=n2θ$ so that its mean and variance are

Thus, the standardized random variable $H*=d*−E(d*)S*$ has the density given in Equation (3.3). Therefore, the required n can be obtained by solving Equation (3.2). Similarly to Remark 2.2, we know that $H2(n−1),(1−c)/2≥Hn1+n2−2,(1−c)/2$⁠, where H_m,(1−c)/2 is the critical value of the distribution H. It is easy to see that

so that the desired results follows. □

Proof. By Proposition 2.2, we know that $T=ndD∼tn−1(λD)$⁠, where $λD=n2(1−ρ)θD$⁠.

It is easy to see that the mean and variance of d_D are given by

Thus, the density of H is given in Equation (3.13). Now, we know that $dD1=D1/SD∼tn−1(λD1)$⁠, where $λD1=θD2(1−ρ)$⁠. Then, the variance of $dD1$ is

From the distribution of H, we can obtain the required sample size n which is given in Equations (3.12) - (3.14) so that the desired result follows. □

The input variables are the confidence level c, precision f, correlation coefficient ρ and $θD=θD0$ obtained from previous data information. The default value of θ_D = 0. The output is the desired sample size n required. Table 3 provides the n for c = 0.90, 0.95 for different values of f and different values of $θD0$'s. Also the relationship between n and θ_D is given in Figure 4.

In this section we will provide some simulation results and two real data examples to support our main results. Based on M = 100,000 runs, coverage rates of the confidence intervals and the corresponding point estimates of θ in independent case and θ_D for matched data are given in Tables 4 and 5. From these two tables, we can see that the performances of our APP procedures work very well and the biases are really small.

To evaluate our results, we provide a real data example for independent and match sample cases, respectively.

The estimated distributions based on the data set are approximately N(8.0868, 1.1099²) for population 1 and N(9.5273, 1.4115²) for population 2 (with unit $10,000) (see Figures 5 and 7). The Q–Q plots of the data sets are given in Figures 6 and 8, showing that the scatters lie close to the line with no obvious pattern coming away from the line in both Q–Q plots. Now we consider the 95% confidence interval of θ with precision f = 0.25 and default θ = 0. From Table 2, the minimum sample sizes n needed is 60, so we randomly choose samples of size 60 from both populations and obtain the value of Cohen's d is d = −1.1430. Then by Remark 3.1, the 95% confidence interval of θ is [−1.4900, −0.7814], which includes the true value of population effect size θ = −1.1284.