Synthesizing evidence through bias estimation and adjustment

Tool Type






The Problem

In principle, the synthesis of all available evidence on an estimand provides high quality inferences because it draws on the entire body of accumulated scientific knowledge. In practice, however, much of the available evidence on an estimand is deemed at risk of unknown and heterogeneous forms of bias. In some cases, the only available evidence on some estimand may be at risk of completely unknown bias. This problem is routinely ignored by researchers. The ideal synthesis strategy is one in which the informative content of evidence at risk of bias is recovered and used to improve inferences. However, evidence at risk of bias is only informative about an estimand if the information used to learn about bias is not entirely derived from knowledge about the estimand. Predominant approaches to bias adjustment in meta analysis attempt to overcome this problem by gathering informative priors on the scale and location of bias terms through expert elicitation. However, this technique only works if the process through which experts form opinions is completely known. If the opinion formation process is unknown, rather than solving the problem of unknown bias, expert-informed priors simply displace it from the evidence on the estimand to the priors provided by experts.

The Solution

I show that it is possible to recover the informative content of evidence that is at risk of bias, even if the location and scale of the bias are completely unknown. The method exploits an assumption that the bias in evidence on an estimand of interest is similar to bias in another source of evidence about a different estimand -- an assumption equivalent to those routinely employed in the synthesis of evidence in regular meta-analysis. The use of empirical estimation procedures to adjust for bias in evidence synthesis presents an advantage over existing approaches, which derive information on bias from experts who may themselves form opinions with unknown systematic error.