Stage 2: Devising a system to capture and assess the quality of evidence from systematic reviews

The intention was to produce a system that could be used to capture evidence that would be useful to practitioners and policymakers interested in reducing crime. Evidence was therefore sought on a range of topics relevant to crime reduction. It was deemed important that the evidence collected was not restricted to purely whether an effect was present, but extended to information that would assist future implementation decisions in a particular local context. Results from studies adopting both quantitative and qualitative approaches were used alongside one another.
A framework was developed to assess five dimensions of each systematic review, summarised by the acronym EMMIE. These refer to

  1. Effect size/s produced (the magnitude of any impact on crime),
  2. Mechanism/s activated (how interventions work),
  3. Moderators/contexts for the activation of the mechanism/s, (where it works best)
  4. Implementation conditions that supported/obstructed delivery (how to do it)
  5. Economic assessment of interventions (what it costs).

For each of the five EMMIE dimensions, two components were distinguished: EMMIE-E relates to the 'Evidence' that emerges from reviews (see Stage 4 below) and EMMIE-Q relates to the 'Quality' of that evidence. Both are needed for users of systematic review evidence to gauge what is or is not known and with what confidence.

Rating the quality of evidence in terms of its reliability and validity is fairly well established for estimates of effect size presented in systematic reviews. For example, good reviews increase validity by having a transparent and well-designed search strategy, featuring a valid statistical analysis, sufficiently assessing the risk of bias (e.g. do published and unpublished studies report different effects), considering the validity of the way outcomes are measured and/or combined, and conducting separate analyses to examine the influence of distinct evaluation research designs on estimated outcomes.

They might also quantify an overall effect for unanticipated outcomes such as displacement caused by a geographically focused intervention. The extent to which systematic reviews address these issues was thus used to rate them in terms of the quality of the evidence presented with regards to the estimated Effect size of an intervention.

Less well established is the rating of quality on dimensions other than effect size. It was therefore important to consider what elements would be necessary to demonstrate sufficient care and attention to these.
Taking Mechanism as an example, some reviews make no reference to how an intervention might work - simply assuming that it had been implemented and it would have some impact on crime. On the other end of the scale, reviews may be very detailed about the mechanism through which an intervention might bring about its effects.

Such reviews might (for example) map out a casual chain of events that should take place to lead from an action (e.g. installing a burglar alarm) through to an impact on crime (e.g. a reduction in burglary). They might discuss different possible mechanisms and make statements as to the plausibility of these. Exceptionally, they might make predictions about what would happen if different mechanisms were at play and use empirical data to test, or partially test these.

Many reviews fall somewhere between these two extremes. Hence, we use the rigor with which these concepts are explored to produce a system of calculating an EMMIE-Q score for Mechanism. Similar reasoning was used to produce Q scoring systems for the other elements of EMMIE.

More detail for each of these can be found in Johnson et al (2015). However, for transparency we provide a narrative guide to the Q-scores (for evidence quality) for each dimension below.

EffectQSCORE

0 = Most forms of bias that could influence the study conclusions remain.
1 = Although the review was systematic, many forms of bias that could influence the study conclusions remain.
2 = Although the review was systematic, some forms of bias that could influence the study conclusions remain.
3 = The review was sufficiently systematic that many forms of bias that could influence the study conclusions can be ruled out
4 = The review was sufficiently systematic that most forms of bias that could influence the study conclusions can be ruled out

​MechanismQSCORE

0 = No reference to theory - simple black box
1 = General statement of assumed theory
2 = Detailed description of theory - drawn from prior work
3 = Full description of the theory of change and testable predictions generated from it
4 = Full description of the theory of change and robust analysis of whether this is operating as expected
​ModeratorQSCORE

0 = No reference to relevant contextual conditions that may be necessary
1 = Ad hoc description of possible relevant contextual conditions
2 = Tests of the effects of contextual conditions defined post hoc using variables that are at hand
3 = Theoretically grounded description of relevant contextual conditions
4 = Collection and analysis of relevant data relating to theoretically grounded moderators and contexts
ImplementationQSCORE

0 = No account of implementation or implementation challenges
1 = Ad hoc comments on implementation or implementation challenges
2 = Concerted efforts to document implementation or implementation challenges
3 = Evidence-based account of levels of implementation or implementation challenges
4 = Complete evidence-based account of implementation or implementation challenges and specification of what would be necessary for replication elsewhere
EconomicsQScore

0 = No mention of costs (and/or benefits)
1 = Only direct or explicit costs (and/or benefits) estimated
2 = Direct or explicit and indirect and implicit costs (and/or benefits) estimated
3 = Marginal or total or opportunity costs (and/or benefits) estimated
4 = Marginal or total or opportunity costs (and/or benefits) by bearer (or recipient) estimated


Go to Stage Three of Method Statement