Combinatorial meta-analysis

Combinatorial meta-analysis (CMA) is the study of the behaviour of statistical properties of combinations of studies from a meta-analytic dataset (typically in social science research). In an article that develops the notion of "gravity" in the context of meta-analysis, Travis Gee proposed that the jackknife methods applied to meta-analysis in that article could be extended to examine all possible combinations of studies (where practical) or random subsets of studies (where the combinatorics of the situation made it computationally infeasible).

Concept

In the original article, k objects (studies) are combined k-1 at a time (jackknife estimation), resulting in k estimates. It is observed that this is a special case of the more general approach of CMA which computes results for k studies taken 1, 2, 3 ... k − 1, k at a time.

Where it is computationally feasible to obtain all possible combinations, the resulting distribution of statistics is termed "exact CMA." Where the number of possible combinations is prohibitively large, it is termed "approximate CMA."

CMA makes it possible to study the relative behaviour of different statistics under combinatorial conditions. This differs from the standard approach in meta-analysis of adopting a single method and computing a single result, and allows significant triangulation to occur, by computing different indices for each combination and examining whether they all tell the same story. Gee's concept of 'gravity' was based on the extent to which removal of a study changed the overall effect size, foreshadowing influence statistics later implemented graphically in GOSH plots (see below).

Implications

An implication of this is that where multiple random intercepts exist, the heterogeneity within certain combinations will be minimized. CMA can thus be used as a data mining method to identify the number of intercepts that may be present in the dataset by looking at which studies are included in the local minima that may be obtained through recombination.

A further implication of this is that arguments over inclusion or exclusion of studies may be moot when the distribution of all possible results is taken into account. A useful tool developed by Gee was the "PPES" plot (standing for "Probability of Positive Effect Size", assuming differences are scaled such that larger in a positive direction is desired). For each subset of combinations, where studies are taken j = 1, 2, ... k − 1, k at a time, the proportion of results that show a positive effect size (either WMD or SMD will work) is taken, and this is plotted against j. This can be adapted to a "PMES" plot (standing for "Probability of Minimal Effect Size"), where the proportion of studies exceeding some minimal effect size (e.g., SMD = 0.10) is taken for each value of j = 1, 2, ... k − 1, k. Where a clear effect is present, this plot should asymptote to near 1.0 fairly rapidly. With this, it is possible then that, for instance, disputes over the inclusion or exclusion of two or three studies out of a dozen or more may be framed in the context of a plot that shows a clear effect for any combination of 7 or more studies.

It is also possible through CMA to examine the relationship of covariates with effect sizes. For example, if industry funding is suspected as a source of bias, then the proportion of studies in a given subset that were industry funded can be computed and plotted directly against the effect size estimate. If average age in the various studies was itself fairly variable, then the mean of these means across studies in a given combination can be obtained, and similarly plotted.

Recent applications of combinatorial and bootstrap methods in meta-analysis

Gee's original software for performing jackknife and combinatorial meta analysis was based on older meta-analytic macros written in the SAS programming language. It was the basis of one report in the area of arthritis treatment. While this software was shared with colleagues informally, it was not published. A later meta-analysis applied the concept in the context of the treatment of diarrhea.

A jackknife method was applied to meta-analytic data some years later but it does not appear that specialized software was developed for the task. Other commentators have also called for related methods, apparently unaware of the original work. More recent work by a software porting team at Brown University has implemented the concept in STATA.

The most widely used implementation of this idea is the Graphical Display of Study Heterogeneity (GOSH) plot, introduced by Olkin, Dahabreh, and Trikalinos in 2012. A GOSH plot is generated by fitting an equal-effects (or, in some implementations, random-effects) model to every possible subset of the studies in a meta-analysis and plotting the resulting effect-size estimates against a heterogeneity statistic such as I², H², or Cochran's Q. When the underlying studies are homogeneous, the resulting distribution of estimates tends to be roughly symmetric and unimodal; a multimodal or clustered distribution instead suggests the presence of outliers or distinct subgroups of studies. Because the number of possible subsets grows combinatorially with the number of studies, meta-analyses with more than about twenty studies typically rely on a large random sample of subsets (often on the order of 10⁵–10⁶) rather than exhaustive enumeration.

Recent applied meta-analyses have used GOSH plots to assess the robustness of pooled estimates and to detect influential studies. A 2025 systematic review and meta-analysis of resistance training for gait rehabilitation in people with stroke applied a GOSH plot after identifying studies with high heterogeneity, generating clusters of subset-based models to visualize how different combinations of studies affected the pooled effect size. Similarly, a meta-analysis of the relationship between flow states and athletic performance, published in Frontiers in Psychology, conducted over two million combinatorial iterations across all possible study combinations and used the resulting GOSH plots to confirm that no subset of studies produced a distorting cluster of effect estimates, supporting the robustness of the overall pooled result. Software support for this method is maintained in the widely used R package metafor, whose gosh function fits equal-effects models to all (or a large random sample of) subsets of a fitted meta-analytic model, and companion packages such as dmetar extend the technique with unsupervised clustering algorithms (k-means, DBSCAN, and Gaussian mixture models) to automatically flag which studies drive the heterogeneity patterns observed in a GOSH plot.

Bootstrap methods

Bootstrap resampling has also been applied in meta-analysis to relax the normality assumptions that underlie standard maximum-likelihood-based inference. Van den Noortgate and Onghena's foundational article proposed two parametric and two nonparametric bootstrap procedures for adjusting maximum-likelihood estimates in meta-analysis, and found via simulation that the parametric bootstrap methods, along with one nonparametric variant, generally outperformed the standard maximum-likelihood approach, albeit with a trade-off between bias and precision.

More recent work has extended bootstrap-based approaches to specific methodological problems in evidence synthesis. A study proposing a bootstrap resampling method for meta-analysis of magnetic resonance imaging (MRI) data collected across multiple scanners used individual participant data within each scanner and estimated the variance of the combined summary statistic via bootstrapping, comparing it against the DerSimonian–Laird and Hartung–Knapp random-effects methods as well as conventional pooling. In simulations, this bootstrap approach achieved better coverage probability and statistical power than the comparator methods when scanner-related differences were present, while performing comparably to conventional pooling when such differences were absent; the method was subsequently applied to an analysis of 787 participants examining the association between age and white matter hyperintensity volume.

Bootstrap methods have also been developed for diagnostic test accuracy meta-analysis. One approach uses bootstrap resampling to construct confidence intervals for the area under the summary receiver operating characteristic (SROC) curve and to derive data-adaptive thresholds for identifying influential studies, based on the change in AUC observed when a study is excluded; this method has been implemented in the R package dmetatools.

Building on this line of work, a 2026 article in the journal Stats introduced the R package boutliers, which provides a unified framework for bootstrap-calibrated outlier detection and influence diagnostics applicable to both fixed-effect and random-effects meta-analysis models. The authors note that bootstrap methods offer a natural way to evaluate the sampling variability of diagnostic statistics and to construct data-adaptive reference values for identifying outliers and influential studies, and that similar bootstrap-calibrated approaches have previously been used in influence diagnostics for multicenter clinical trials, diagnostic test accuracy meta-analysis, and network meta-analysis; the package aims to fill a gap in user-friendly computational tools for applying these methods in routine pairwise meta-analysis.

Limitations

CMA does not solve meta-analysis's problem of "garbage in, garbage out". However, when a class of studies is deemed garbage by a critic, it does offer a way of examining the extent to which those studies may have changed a result. Similarly, it offers no direct solution to the problem of which method to choose for combination or weighting. What it does offer, as noted above, is triangulation, where agreements between methods may be obtained, and disagreements between methods understood across the range of possible combinations of studies.

---