Credible Confidence: A Pragmatic View on the Frequentist vs Bayesian Debate

Suppose the interest lies in estimating the proportion of a given population that holds a specific property. This is a very general research question, applicable to many areas: the proportion of diabetes patients that respond positively to a certain treatment method, the proportion of voters expected to vote for a certain political party, the proportion of students passing an exam, etc.

To express the statistical uncertainty about the population proportion, a point estimate alone is not sufficient and an estimate in the form of an interval is preferred. Frequentists call such an interval a confidence interval, Bayesians call it a credible interval. These two types of intervals are, from a theoretical/philosophical point of view, fundamentally different. From a practical point of view, however, both intervals share a common feature: the interval is preferred over the point estimate to express uncertainty. Suppose one estimates a population proportion θ with the interval (.42, .78). This clearly provides different information about the population proportion than the interval (.59, .61), even though in both cases the interval is symmetrical around .60. Furthermore, when a certain value, say .50, is far from the interval, this gives the applied researcher confidence in believing that the unknown true value is unequal to .50: with the interval (.42, .78) one is not keen on rejecting the possibility that θ = .50, whereas with the interval (.59, .61) one can be much more confident about rejecting θ = .50. For this intuitive interpretation, it does not matter whether the interval is constructed using frequentist or Bayesian methods.

There are different frequentist and Bayesian approaches to generating such intervals, all based on a random sample of n objects, of which it is recorded that m objects hold the property of interest. These models differ in the mathematical way they are constructed, yet all are sensible approaches to estimating a proportion. Below, we outline three common frequentist approaches and two common Bayesian approaches. For sake of simplicity, we set the confidence/credible level at a fixed value of 95%. Furthermore, we assume that the population size is much larger than the sample size, such that we do not need to worry about finite population corrections.

When n, np and n(1–p) are all not ‘too small’, an approximate confidence interval is directly obtained from the normal approximation Bin(n, p) ≈ N(np, np(1–p)) due to the Central Limit Theorem. This gives the interval

where $p^=m/n$ is the observed proportion in the sample and 1.96 is the percentile of the standard normal distribution corresponding to the 95% level.

This asymptotic approach can be improved upon through the so-called plus-four method (Agresti & Coull, 1998). In this method, the estimate $p^$ in (1) is replaced, on all three instances, by $p˜=(m+z)/(n+2z)$⁠, where z = 1.96. Roughly, this method adds two successes and two failures to the sample, hence the name plus-four method. For large samples this change has little effect: the difference between $p˜$ and $p^$ is relatively small. For smaller samples Agresti and Coull have shown that their method constitutes an improvement.

Approach F1 is asymptotic and – even with the “plus four”-correction outlined – does not necessarily work well for small samples. However, it is frequently used, mainly because of its simplicity and the lack of alternative methods available in common software packages. Blyth (1986) discusses a method for computing the exact confidence interval, after Clopper & Pearson (1934):

with A = F_{0.025; 2m, 2(n – m + 1)} and B = F_{0.975; 2(m + 1), 2(n – m)} being percentiles from F-distributions.

This approach is based on the approximation (cf., Shao, 1998) that

which, after some derivations, leads to the interval

One of the instances where this approach is used is in the computation of Cohen’s h.

Bayesian approaches are specified through their prior distribution. The beta-distribution is a so-called conjugate prior of the Binomial distribution, which means that the posterior distribution is also Beta. In general, when using a Beta(a, b) distribution as prior, the posterior is given by the Beta(a + m, b + n – m) distribution. By taking the 2.5% and 97.5% percentile points of this distribution, one achieves the 95% credible interval.

Approach B1 is based on the prior assertion that all values for p between 0 and 1 are equally likely. This is achieved by using the uniform(0,1) distribution, which is identical to the Beta(1, 1) distribution, as prior. This results in a Beta(1 + m, 1 + n – m + 1) as posterior.

Jeffreys prior is a so-called non-informative prior (which means it is invariant under reparametrizations of the problem space), which is a desirable property of a prior. The Jeffreys prior for the current setting is the Beta(½, ½) distribution, yielding the Beta(½ + m, ½ + n – m) posterior.

Table 1 lists the intervals obtained by the five methods for various choices of m and n. It is clear that the methods are in general agreement, especially when n is large. Only exception is the arcsine method, that consistently provides wider intervals. In Table 2, we study the five approaches in more detail. For various choices for n, it lists the average overlap between approaches for all possible values of m (i.e. m = 0, 1, …, n). The arcsine method clearly has different behavior than the four others. For those other methods, even with n as low as 10, the overlap between any two approaches, whether one is Bayesian and the other frequentist, or whether both are from the same ‘school’, is at least 90%. For these methods, the agreement increases if n increases. Both Bayesian approaches are usually, but not always, somewhat more similar to each other than to the frequentist approaches, and the same can be said for the frequentist approaches F1 and F2. However, the differences are negligible. Thus, a frequentist might have the same level of agreement with a fellow frequentist as with a Bayesian. Similarly, it is entirely possible that two Bayesians agree less with each other than with a frequentist. In the words of the Bayesians Jaynes and Kempthorne (1976, p. 195): “The differences are so small that I could not magnify them into the region where common sense is able to judge the issue”.

For continuous data, the central limit theorem states that for any reasonable n, the sampling distribution of the sample mean is approximately normal. A frequentist 95% confidence interval for the population using the commonly used t-distribution is as follows

where $x¯$ is the sample mean, t_{n – 1} is the corresponding critical value from a t-distribution with n – 1 degrees of freedom, s is the sample standard deviation, and n is the sample size. We are going to contrast this standard frequentist confidence interval with a Bayesian credible interval, based on a default Cauchy prior on effect size, as this is currently implemented in e.g. the ‘point-and-click’ programmes JASP (JASP Team, 2018) and jamovi (jamovi project, 2018). The construction of such an interval proceeds as follows.

A prior is constructed for the population effect size delta, such that $δ~N(0,σδ2)$ and $σδ2~Inverse χ2(1)$⁠. Combining these two yields δ ~ Cauchy (Liang, Paulo, Molina, Clyde, & Berger, 2008). The next step is the construction of a likelihood function: L(data|δ). The posterior is proportional to the product of the prior and the likelihood. The 95% credible interval constitutes the middle 95% of this posterior.

With these restrictions in place, we conducted two sets of simulations. In the first set, we generate normally distributed data for a single group that varied along the following two dimensions:

1. Corresponding t-statistic: 0.5, 1, 1.5, and 2 (i.e., a sample of generated values was transformed such that the corresponding t-values exactly equaled these values, and that the sample standard devation equaled 1);¹

2. Number of participants: 10, 12, 14, 16, 18, 20, 22, 24, 26, 28, and 30.

Subsequently, we calculated 95% confidence and credible intervals for the resulting data.

In the second set of simulations, the data is artificially constructed such that the data vary on how skewed the underlying population distribution is. This was done by simulating data using the rsn function in R (from package sn, see Azzalini, 2017). Skew was manipulated by varying the ‘alpha’ parameter from 0 to 10 in steps of 1 (see Azzalini, 2014 for details). The number of participants was fixed to 20 for this set of simulations. Subsequent to sampling from the skewed normal distribution, the data was standardized and t/√20 was added to each data point to ensure all simulations varied only along the value of the t-statistics and the alpha parameter. Finally, we calculated 95% confidence and credible intervals for the resulting data.

Results for the first set of simulations, based on normally distributed data, are shown in Figure 1. The figure shows that frequentist confidence intervals and Bayesian credible intervals correspond closely. For lower sample sizes, the confidence intervals appear to be marginally wider than the credible intervals, but this difference quickly disappears for more realistic (but still small) sample sizes.²

Results for the second set of simulations, based on right-skewed data, are shown in Figure 2. The results of this second set of simulations mirror those of the first set of simulations in that there is no qualitative difference between the confidence and credible intervals. This is perhaps not so surprising: although the data itself deviates from normality, the central limit theorem implies that the sampling distribution of the sample mean is still approximately normal. As such, there is no reason to expect substantial differences between both sets of simulations.³

In the present paper, we have demonstrated by means of various examples that confidence intervals and credible intervals, in various practical situations, are very similar and will lead to the same conclusions for many practical purposes when relatively uninformative priors are used. The examples used here are based on small samples but are otherwise well behaved and could easily occur in practice. When sample size increases, the numerical difference between both types of interval will (usually) decrease.

So in what situations do the approaches yield more substantial differences? There are two main examples: (1) restriction of range of the data; (2) Bayesian methods based on a considerably more informative prior. As an example of the first point, consider 15 scores on a Likert scale ranging from 1 to 5. Suppose that ten scores are 1, four scores are 2, and one score is 5. Construction of a classical 95% confidence interval results in the interval (0.95, 2.12), an interval that includes values below the minimum possible value of 1. The Bayesian 95% credible interval is bounded by definition to not include values beyond the range of the parameter space. For a uniform prior on this interval, combined with the assumption that the sample standard deviation equals the population standard deviation, the resulting 95% credible interval is (1.08, 2.07) (see Figure 3).

The second point highlights the scope of our present findings: we have shown numerical similarities between frequentist and Bayesian methods for (relatively) uninformative priors. Depending on the research context, vastly different intervals can be obtained if one chooses a specific informative prior. Our paper meant to highlight similarities when relatively standard, off-the-shelve, methods are used for constructing intervals under both regimes, using ‘objective’ or fairly uninformative priors, in the simple common contexts of estimation of proportions and means.

Why then, in cases with little or no prior information, bother with Bayesian approaches, and not stick to the more traditional frequentist confidence interval? A good reason is that a Bayesian analysis is more in line with the way researchers actually interpret their results (whether frequentist or not). That is, researchers tend to interpret their results in explicit or implicit terminology indicating how certain they are about what the effect size truly (i.e. in the population) is. As many papers and text books emphasize, frequentist approaches cannot warrant such statements, but Bayesian approaches can: One can claim that there is a 95% chance that the true effect size is in the credible interval. Even stronger, one can accompany the credible interval with a full picture of the distribution from the true effect size by means of giving the full posterior distribution, see Figure 3 for an example. Similar frequentist approaches to distributional inference exist (Albers, 2003; Kroese & Schaafsma, 2004), but are neither straightforward nor often used in practice. A frequentist analogue to the rich information provided by the posterior distribution is the bootstrap (Efron & Tibshirani, 1994).

The frequentist approach works from the premise that only the data are prone to random fluctuations, while the true effect is fixed, and hence it makes no sense to specify probabilities for the (fixed) population effect size but only about the probability as to whether the confidence intervals estimated by means of the data will cover the true effect size. This is a subtle difference with the Bayesian credible interval interpretation, but as the way people like to interpret results is more in line with the latter, the Bayesian approach is better in serving researchers at their wishes. This comes with a price, however. The price is that the statements are always conditional upon the prior that one has specified. Fortunately, however, the exact location of credible intervals does not appear to vary strongly with variations in the prior. Indeed, in the case where we assume that the population variance is known, the confidence interval for means can be obtained by a particular choice of the prior, namely the uniform prior. This is implausible in practice, but can be seen as a limiting case of a flat prior. And as we have seen now, it does not lead to very different intervals than does the more realistic Cauchy prior.

For us the main message of our paper is as follows. Frequentist confidence intervals can be interpreted as a reasonable approximation to a Bayesian credible interval (with uninformative prior). This is reassuring for those who struggle with the formally correct interpretation of frequentist intervals. Additional insight can be obtained when these intervals are complemented (or replaced) by a full posterior distribution for the effect size measure under study. The posterior distribution will, conditionally upon a chosen prior, give the full picture of the uncertainty around its possible value. It can provide information on skewness, bimodality, and other properties – or the lack thereof, such as in Figure 3 – that a simple interval, with only a lower and upper bound, can not. Furthermore, it can estimate the probability that the parameter is larger or smaller than a fixed value, e.g. 0 or 0.5, or is within a certain interval. As such, posterior distributions can ideally work towards the enhancement of science.

All computations have been performed using R (R Core Team, 2018). All software code is available from https://osf.io/dgfht/.

The preprint of this paper has also been published on OSF.

The authors have no competing interests to declare.

CJA conducted the intervals for discrete data analysis and wrote the first version of the manuscript, HALK wrote a part of the paper, and critically commented on various versions of the manuscript, DvR conducted the intervals for continuous data simulations and critically commented on various versions of the manuscript. All authors contributed to writing of the final manuscript.

The author(s) of this paper chose the Open Review option, and the peer review comments are available at: http://doi.org/10.1525/collabra.149.pr