The intention was to produce a system that could be used to capture evidence that would be useful to practitioners and policymakers interested in reducing crime. Evidence was therefore sought on a range of topics relevant to crime reduction. It was deemed important that the evidence collected was not restricted to purely whether an effect was present, but extended to information that would assist future implementation decisions in a particular local context. Results from studies adopting both quantitative and qualitative approaches were used alongside one another.A framework was developed to assess five dimensions of each systematic review, summarised by the acronym EMMIE. These refer to
For each of the five EMMIE dimensions, two components were distinguished: EMMIE-E relates to the 'Evidence' that emerges from reviews (see Stage 4 below) and EMMIE-Q relates to the 'Quality' of that evidence. Both are needed for users of systematic review evidence to gauge what is or is not known and with what confidence.
Rating the quality of evidence in terms of its reliability and validity is fairly well established for estimates of effect size presented in systematic reviews. For example, good reviews increase validity by having a transparent and well-designed search strategy, featuring a valid statistical analysis, sufficiently assessing the risk of bias (e.g. do published and unpublished studies report different effects), considering the validity of the way outcomes are measured and/or combined, and conducting separate analyses to examine the influence of distinct evaluation research designs on estimated outcomes.
They might also quantify an overall effect for unanticipated outcomes such as displacement caused by a geographically focused intervention. The extent to which systematic reviews address these issues was thus used to rate them in terms of the quality of the evidence presented with regards to the estimated Effect size of an intervention.
Less well established is the rating of quality on dimensions other than effect size. It was therefore important to consider what elements would be necessary to demonstrate sufficient care and attention to these. Taking Mechanism as an example, some reviews make no reference to how an intervention might work - simply assuming that it had been implemented and it would have some impact on crime. On the other end of the scale, reviews may be very detailed about the mechanism through which an intervention might bring about its effects.
Such reviews might (for example) map out a casual chain of events that should take place to lead from an action (e.g. installing a burglar alarm) through to an impact on crime (e.g. a reduction in burglary). They might discuss different possible mechanisms and make statements as to the plausibility of these. Exceptionally, they might make predictions about what would happen if different mechanisms were at play and use empirical data to test, or partially test these.
Many reviews fall somewhere between these two extremes. Hence, we use the rigor with which these concepts are explored to produce a system of calculating an EMMIE-Q score for Mechanism. Similar reasoning was used to produce Q scoring systems for the other elements of EMMIE.
More detail for each of these can be found in Johnson et al (2015). However, for transparency we provide a narrative guide to the Q-scores (for evidence quality) for each dimension below.
Go to Stage Three of Method Statement