Accounting for Sample Overlap in Economics Meta-Analyses: The Generalized-Weights Method in Practice

Authors: Pedro Bom and Heiko Rachinger

Abstract: Meta-analyses in economics often feature a substantial degree of overlap among primary samples. If left unaddressed, sample overlap gives rise to efficiency losses and too high rates of false positives at the meta-analytical level. Bom and Rachinger (2020) propose a generalized-weights solution to sample overlap, which feasibly approximates the structure of correlation between primary estimates using information on sample sizes and degree of overlap in the primary studies. In this paper, we show how the generalized-weights method can be applied to economics meta-analyses by tackling several difficulties that are likely to be encountered in practice. In particular, we show how to account for different levels of data aggregation in primary studies, different estimation methods, and different effect size metrics, among other issues. We derive explicit covariance formulas for the different cases, study the accuracy of the approximations, and show, using Monte Carlo simulations, how the method improves efficiency and restores the rate of false positives to its nominal level.