The Trouble with Coarsened Exact Matching

Abstract

Balancing methods, which use matching or reweighting to improve the balance between treated and control units, are central tools for causal inference in the social sciences using cross-sectional observational data. We focus here on one method which has attained substantial popularity, especially in political science, Coarsened Exact Matching (CEM). We report evidence that CEM performs worse than a number of other popular balancing methods and explain why it does so. We report evidence both from simulations, and from replicating four recent studies that use CEM. Applied to real datasets, CEM drops substantially more observations than other methods; has larger standard errors; and produces average treatment effect estimates far from both other methods and from CEM itself. In simulations based on the real datasets, CEM is sensitive to adding noninformative covariates or varying which covariates one balances on, and can severely over-reject a true null. In simulations with generated data and heterogeneous treatment effects, and thus known truth, CEM has substantial bias and is much less precise than other methods. Our advice: never use CEM as the sole balancing method, and use it, if at all, with sensitivity checks for variable selection, binning choices, and evidence of low treatment heterogeneity. Those checks are rare in current practice.