Garbage In, Garbage Out? Evaluating the Evidentiary Value of Published Meta-analyses Using Z-Curve Analysis

The methodology and analytic plan for this study were pre-registered and can be found, along with the R code and a reference list of all of the meta-analyses included in this review, at the Open Science Framework (OSF) page at this link: https://osf.io/gr3ax/?view_only​=​49a​2ad​228​222​477​ab​e99​6c​c62​7c​e69​b5. There are a few changes to note from this pre-registered method. First, in the pre-registered Method, it says that all of the meta-analyses coded for this project would be uploaded as PDFs on the OSF. However, it was decided not to do this, because that might end up infringing on copyright laws for those publications. Second, in the pre-registered Method, it is indicated that 34 meta-analyses were gathered for inclusion; however, only 21 were included in the final analysis. This is either because the z-curve analysis did not run (see Results for an explanation) or because the meta-analyses were found not to be suitable for inclusion upon closer inspection. See Inclusion Criteria below for more details on that.

The last difference between the pre-registered Method and the current Method is in how p-values were computed. The equations included below for how p-values were computed from effect sizes differ slightly from those in the pre-registered Method, because some further research on how to compute p-values from effect sizes was done, and a more valid way of doing so was found while double-checking the coding. Specifically, some coding had to be re-done in cases where studies included in meta-analyses manipulated their independent variables within-subjects, as the process for computing a p-value from a mean difference effect size (e.g., Cohen’s d) is slightly different depending on whether the independent variable in the primary study was manipulated within- or between-subjects (see Coding Procedure for more details on this).

Psychological Bulletin was chosen as the target journal, because it represents one of the most prestigious outlets for meta-analyses and because it publishes many meta-analyses. The time interval of 2010 to the present (i.e., spring 2020) was chosen so that the meta-analyses included would be reasonably current. Only meta-analyses that investigated experimental mean differences between two groups, a treatment and control, were included. This was done because the p-values taken from the effect sizes included in such meta-analyses were most likely to fulfill an important assumption of z-curve: that the p-values being entered into the z-curve analysis are the p-values from the statistical tests for the main hypothesis of each study. This is crucial to z-curve analysis, because it is the p-values from the test of the central hypothesis that are likely to be subject to publication bias (Simonsohn et al., 2014). This assumption is much more likely to be met in the case of p-values taken from the effect sizes included in meta-analyses of comparisons between two groups rather than meta-analyses of correlational effects, because correlational meta-analyses often include correlation coefficients that were only peripherally related to the main hypotheses of the studies from which they were taken. A comparison between two groups is much more likely to be the central test of a study.

Only meta-analytic reviews of experimental literatures that included tables with the authors’ coding for effect sizes and sample sizes, either in the main body of the article or in supplemental materials, were included in this review. The table also had to indicate whether each study came from a between- or within-subjects design, or the paper itself had to make it clear at some point in the text that 100% of studies were either between- or within-subjects designs.

According to the database PsycINFO, 568 pieces were published in Psychological Bulletin between January 1st, 2010, and December 31st, 2020. A total of 158 were meta-analyses (others were corrections, commentaries, and other kinds of pieces). Thirty-four were found to be tests of mean differences and to have tables from which effect sizes could be extracted, and were examined more in-depth for inclusion. From among the 34, a number were excluded because it was not clear which studies included were between-subjects or within-subjects. This information was necessary in order for the effect sizes to be properly converted to p-values. Others were excluded because they turned out not to be meta-analyses of mean differences manipulated experimentally, but were tests of quasi-experiments or cross-sectional studies (e.g., gender differences). Twenty-three meta-analyses ended up being coded and analyzed, although only 21 yielded data where a z-curve could be run successfully (see Results).

The major piece of coding was extracting the p-values from each individual effect size included in each meta-analysis. Because only meta-analyses that report their coding in a table of some sort were included in this project, this involved examining the effect sizes and sample sizes from these tables and computing test statistics—usually t-scores—from those values, and then generating p-values based on those test statistics. In a single case (Hu et al., 2020), the authors provided p-values alongside the effect sizes in their table, so these p-values were used for the z-curve in that case. In a second case (Weingarten et al., 2016), the authors performed a p-curve themselves and included the disclosure table in their supplemental materials with all of the test statistics from the studies included, so in this case, the p-values from their disclosure table were used for the z-curve.

P-values were calculated differently depending on how the authors of each meta-analysis presented their results, what effect size they reported, and the designs of the studies they included. In three cases (Fox et al., 2011; Landau et al., 2015; Sedlmeier et al., 2012), the authors reported effect sizes as Pearson r coefficients, even though the research area was in large part experimental. In these cases, the Pearson coefficients were entered into the p-value generator along with the sample size to acquire p-values at this link: https://www.socscistatistics.com/pvalues/pearsondistribution.aspx.

For all other meta-analyses included, calculating the p-values was more complicated. This procedure was different depending on whether the design of each study included in each meta-analysis used a between- or within-subjects design (Hedges, 1983). The design of included studies was typically noted explicitly in the text. For example, Fischer et al. (2011) noted that all included studies had a between-subjects design, while Cracco et al. (2018) noted that all included studies had a within-subjects design. Some authors (e.g., Coles et al., 2019) noted that the included studies were a mix of between- and within-subjects designs, and for these meta-analyses, the precise design for each included study was typically included in the meta-analytic table that listed the effect sizes taken from each sample. In these cases, the p-values from the studies listed as using between-subjects designs and the studies listed as using within-subjects designs were calculated differently according to the processes described below.

Converting the effect sizes into p-values was different depending on whether the authors presented their effect sizes as Cohen’s d, Cohen’s dz (d from a within-subjects design), or Hedges’ g. To compute p-values for between-subjects designs, it was often necessary first to convert Hedges’ g back into Cohen’s d. If the authors of the meta-analysis did not specify what method they used to convert Cohen’s d into Hedges’ g, the basic formula for the conversion given by Hedges (1983) was used: $d=g/c(m),$ where $c(m)=1−(3/((4∗m)−1)),$ where m is the degrees of freedom for the study. m was calculated as N – 2 for studies that used between-subjects designs and N – 1 for studies that used within-subjects designs. If the authors specified that they performed the conversion from Cohen’s d to Hedges’ g another way, then this method was used. If Cohen’s d was what the authors reported in their meta-analytic table, then no conversion was necessary and the Cohen’s d could be converted directly into a t-score.

Once a Cohen’s d was obtained, it was converted into a t-score. If the t-score was from a between-subjects design and sample sizes for the treatment and control conditions were provided in the meta-analytic table, it was converted to a t-score using the following formula: $t=d/(1n1)+(1n2).$ If the sample sizes for the treatment and control groups were not provided in the meta-analytic table, the t-score was calculated as follows: $t≈d2∗N$ . The degrees of freedom for each t-score was calculated as N – 2. In the case of within-subjects designs, the Cohen’s dz was converted to a t-score using the following formula: $t=dz∗n.$ The degrees of freedom was then calculated as N – 1. All t-scores were converted to p-values at the following link, assuming a two-tailed hypothesis for each: https://www.socscistatistics.com/pvalues/tdistribution.aspx. The formulae for calculating t-scores were modified using basic algebra from the formulae for calculating effect sizes from t-scores that Lakens (2013) provides.

Only one p-value was taken from each sample included in each meta-analysis, because each p-value entered into a z-curve must be independent, and if two p-values come from the same sample, they are not independent. If multiple effect sizes were included in a meta-analysis from a single sample, then the first one reported in the table was included in the z-curve and all others taken from that sample were excluded. This means that the number of p-values extracted from each meta-analysis may not equal the number of overall effect sizes included in the original meta-analysis.

If there were multiple meta-analyses reported in a meta-analysis paper, then separate z-curves were done on each meta-analysis for which the number of effect sizes was 20 or more. Note, however, that, as indicated in the Results below, z-curve will only run if at least 10 of those p-values are below .05.

There is a disclosure table inspired by those that Simonsohn et al. (2014) recommends for p-curves posted on the OSF. This disclosure table includes information about each meta-analysis and justification for some of the coding decisions. Included in this disclosure table is: (1) the number of z-curves that each paper resulted in; (2) how effect sizes were converted to p-values, with quoted text from each paper to justify it; (3) what subfield the paper was from; (4) the number of times the meta-analysis has been cited on PsycINFO; (5) whether or not the authors tested for publication bias; (6) which tests for publication bias were used; (7) how many different tests of publication bias the authors used; (8) the outcome of each test of publication bias; and (9) the average sample size per study included. There is a separate line of code for each meta-analysis on which a z-curve was performed.

Z-curve 2.0 was used for the main analysis (Brunner & Schimmack, 2020). This analysis takes the p-values from a set of studies of interest to the researcher, converts them to two-tailed z-statistics, and uses those z-statistics to calculate average power of all of the studies that have hypothetically been done using finite mixture modeling. Recall that power is the probability of finding a statistically significant result if there is a true effect (Cohen, 1992), so once the average power is estimated, the percent of all of the studies done in the area that would be predicted to be statistically significant can be estimated. The estimate that z-curve generates, however, is based on the full population of studies predicted to exist, including hypothetical unpublished studies that were not entered in the z-curve. The percent of all (hypothetical) studies predicted to be significant can then be compared to the percent of studies that show statistical significance in the published literature originally entered into the z-curve. If the number of statistically significant results in the published literature is larger than the predicted number of statistically significant results based on the average power of the studies, then this suggests the presence of biases in data-generating mechanisms in the published literature.

In addition to the expected discovery rate (EDR)—the percent of studies predicted to be significant based on the average power of published studies—and the observed discovery rate (ODR)—the percent of studies that show statistical significance in the published literature—z-curve produces several other estimates of interest. But one should note that all of the estimates listed hereafter are simply transformations of the EDR. That is, while they are all interpreted differently, they all basically provide the same information. With that said, the other estimates that z-curve provides are: the expected replication rate (ERR)—the average power of the studies entered, which is also an estimate of the percent of the studies that one would expect to replicate if one performed the studies in exactly the same way as they were done before; the Sorić false discovery rate (Sorić FDR)— the maximum percent of studies that could be false positives; and the file drawer ratio (FDR)—the ratio between the EDR and ODR that is expressed as the number of unpublished findings that are predicted to exist for every published effect size. The FDR is basically a snapshot of the presence of publication bias. For the purposes of reporting the results of this study, the FDR for each z-curve was converted to a percentage. This is done by dividing the FDR by the FDR plus one and then multiplying by 100. This makes interpreting the FDR more straightforward, as the percentage can be interpreted as the percent of findings missing from the published literature. Finally, the value of the EDR subtracted from the ODR was also examined, as the difference between the two is one of the crucial estimates when interpreting a z-curve. The higher this value is, the higher the likelihood of biases in data-generating mechanisms. The z-curves were calculated using the R code uploaded to the OSF (https://osf.io/gr3ax/?view_only=49a2ad228222477abe996cc627ce69b5).

Twenty-nine z-curves were attempted, and 25 of them ran successfully. Four of them did not run because there were fewer than 10 p-values below .05 in the data. The analysis in R will only run if there are at least 10 p-values below .05 in the data entered. Table 1 shows the detailed results for the 25 z-curves that ran successfully. All of the values—except the Significant N—are expressed as percentages, and include the 95% confidence intervals. The Significant N is included as a column in Table 1 to give an idea of how many p-values went in to each z-curve, as z-curves are run only with the significant p-values that are entered. Note that many of the confidence intervals for the estimates are large, so all interpretations should be cautious.

Out of the 25 z-curves, 19 (76%) showed an ODR larger than the EDR, as indicated by a positive difference between the two. Sixteen (64%) of the z-curves showed a difference greater than 10 percentage points; 11 (44%) showed a difference greater than 20 percentage points; 7 (28%) showed a difference greater than 30 percentage points; and 3 (12%) showed a difference of 50 or more percentage points. Figure 1 shows the values of the ODR – EDR arranged from the highest (70.1) to the lowest (-29.9) values. A negative difference indicates that the power of the published studies was such that one would predict fewer published studies than are available. The more positive the difference, the greater the estimates suggest the presence of biases in data-generating mechanisms.

Figure 2 shows the Sorić FDR for each z-curve, again arranged from lowest (0) to highest (83.8). Thirteen (52%) meta-analyses showed a Sorić FDR below 10; 5 (20%) showed a Sorić FDR between 10 and 20; and 7 (28%) showed a Sorić FDR above 20. The higher the Sorić FDR, the greater the proportion of findings that are predicted to be false positives.

Figure 4 shows the ERR for each z-curve, arranged from lowest (12.4) to highest (100). Eight (32%) of the z-curves showed an ERR below 50%; 13 (52%) showed an ERR below 60%; 19 (76%) showed an ERR below 70%; 21 (84%) showed an ERR below 80%; 23 (92%) showed an ERR below 90%; and 2 (8%) showed an ERR above 90%. The lower the ERR, the lower the percent of studies are predicted to replicate if they were conducted in exactly the same way as they were done the first time.

Because z-curve is a new analytic technique—certainly relative to the older and more widely used p-curve analysis—some may claim that its efficacy and trustworthiness has not yet been demonstrated sufficiently. Therefore, p-curve analysis was also used to examine the distribution of p-values from the three meta-analyses that z-curve analysis indicated to be characterized by the most biases in data-generating mechanisms (i.e., Fischer et al., 2011; Tannenbaum et al., 2015; Weingarten et al., 2016). The metric for determining the three weakest meta-analyses was the FDR, because this captures both QRPs and publication bias. It was decided only to include the three meta-analyses that could be analyzed in full; in other words, no meta-analyses were p-curved if they had multiple meta-analyses reported.

The p-curve analysis results for Weingarten et al. appear to corroborate the result of the z-curve. While the half p-curve indicates evidential value, Z = -2.89, p = .002, the 33% power test for the full p-curve is significant, Z = -3.82, p .001. This suggests that the studies included show less than 33% power, meaning that the studies included are not expected to replicate. Indeed, the p-curve estimates the power for the included studies at 10% [CI: 5%, 18%], which is close to the ERR of 12.4% that the z-curve predicted. Recall that the ERR is essentially the estimated power of the studies included in a z-curve, and so it should be equivalent to the average power estimated by p-curve.

The results for Tannenbaum et al. and Fischer et al. are more complicated. The full p-curve, Z = -18.24, p .001, and the half p-curve, Z = -20.66, p .001, are both significant for Tannenbaum et al.. The 33% power test also came out favorably for Tannenbaum et al. in both its full p-curve, Z = 10.75, p .999, and its half p-curve, Z = 19.66, p .999. Its estimated power is 91% [87%, 94%]. The p-curve results for Fischer et al. were similarly favorable. The full p-curve, Z = -7.51, p .001, and half p-curve, Z = -9.26, p .001, were both significant, and the 33% power test also came out favorably for both the full p-curve, Z = 3.99, p .999, and the half p-curve, Z =8.92 , p .999. The estimated power for Fischer et al. was 84% [68%, 93%]. The discrepancy in the outcomes of the z-curves and the outcomes of the p-curves is likely because the results of p-curves can be problematic when there is a lot of heterogeneity in the effect sizes in the population of studies being analyzed, as mentioned in the Introduction (Brunner & Schimmack, 2020). P-curve’s estimates can also be upwardly biased when there are a few highly-powered studies entered (Brunner & Schimmack, 2020). Thus, the discrepancy in the outcomes should not be taken as too strong a reason to question the outcomes of the z-curves reported above.

The z-curve analyses of the studies included in 21 influential meta-analytic reviews suggest the presence of questionable research practices (QRPs) in a majority of the examined literatures, and that a non-trivial proportion of studies included are predicted not to replicate. Three-quarters of the z-curves suggest that publication bias has resulted in one unpublished study for every published study, suggesting that only half of all studies have been published (i.e., FDR of 50% or higher). Around one-third of the z-curves suggest that up to 50% of the studies included are not expected to replicate as indicated by an ERR below 50%, and a similar proportion of the z-curves suggest that over 20% of studies included may be false positives. In aggregate, these results suggest a relatively widespread prevalence of biases in the data-generating mechanisms of the included primary studies—and hence suggest a similar bias in meta-analytic effect size estimates.

While the results show that QRPs appear to be quite widespread, they also appear to vary across literatures. Some meta-analyses were characterized by excellent estimates of the expected replication rate and low file drawer ratios such that the meta-analytic effect size estimates are likely to be relatively unbiased. The results of z-curve analyses for other meta-analyses were more concerning, suggesting low replicability and high file drawer ratios. While these data cannot speak to the reasons for this high variability in the results of the z-curve analyses, some possible reasons present themselves.

First, it may be that some literatures focus on effects that are inherently larger and more robust. In such settings, there may be no need to engage in QRPs. When studying a large effect, a researcher should not need QRPs to find an effect in their study (Cohen, 1992). Second, research and publication norms are likely to vary across subdisciplines. That is, in some subdisciplines, researchers may believe that QRPs are acceptable and widely practiced, and that journals have a strong preference for statistically significant findings. In others, researchers may believe that journals are more accepting of null results, or there may be strong research norms in place that prevent QRPs. Third, it may be that data collection is more time consuming, difficult, and expensive in some subdisciplines, which may make certain practices such as “data peeking” and optional stopping rules seem more acceptable. Researchers in these subdisciplines may not want to use up any more time, money, or resources than are totally essential, and so it may seem to make sense to check for statistical significance periodically and stop once it is achieved to avoid being wasteful by collecting more data than necessary.

The results of two of the three p-curves reported above were much more favorable than their corresponding z-curves. This is likely because an assumption of p-curve is that the studies included all investigate a single underlying effect; in other words, p-curve assumes effect size homogeneity. Brunner & Schimmack (2020) demonstrate that p-curves can overestimate evidential value when this assumption is not met—in other words, when there is effect size heterogeneity. Indeed, p-curve can overestimate statistical power by almost 25% under certain levels of effect size heterogeneity, whereas z-curve performs well under conditions of heterogeneity (Brunner & Schimmack, 2020). Every single meta-analysis coded for this work reported estimates of effect size heterogeneity, and all indicated the presence of heterogeneity. Thus, it stands to reason that p-curve may not be the ideal way to test the evidentiary value of those studies.

This work is not meant to encourage anyone to discount all published meta-analyses. Neither is it meant to point to any inherent problems with meta-analysis as an approach. It is merely to show that biases in the data-generating mechanisms of primary studies in some literatures appear to have affected the meta-analytic syntheses of those literatures. This, in turn, suggests that these meta-analytic results should be interpreted with some caution. In particular, it seems likely that effect size estimates are inflated, which has both practical implications (e.g., should an expensive intervention with a smaller effect size still be conducted) and methodological implications inasmuch as researchers may be basing their power calculations on inflated effect size estimates.

The first author came up with the idea for the project, performed all of the coding and analyses, and wrote the manuscript.

Grateful acknowledgment is extended to Marcus Credé for providing invaluable feedback on drafts of this manuscript.

The author has no competing interests to state.

All data analyzed for this project, as well as the R code used for the analyses, are available on the Open Science Framework at the following link: https://osf.io/gr3ax/?view_only=49a2ad228222477abe996cc627ce69b5.