Stage 4: Translating EMMIE into a practical tool

​The coding spreadsheet captures how each reviewer scores each review but does not present this in a way that is easy to follow. The College of Policing and UCL therefore worked together to develop an online toolkit to present this information in a more accessible form. This resulted in two formats: a summary or 'landing page', and an associated 'narrative' that provides fuller details. The toolkit presents evidence at intervention level. When an intervention is covered by more than one systematic review the results are presented together in the toolkit. This includes relevant findings found in reviews of multiple interventions. Where information was obtained from multiple reviews, the evidence reported under each EMMIE dimension comes from the review(s) that attained the highest evidence quality score on that particular item, as this evidence is the most reliable. Where evidence from additional reviews is seen as important to the narrative, the additional review is clearly indicated, along with its quality score for that EMMIE dimension.

Some features of the tool include:
1. A landing page which presents the rating and ranking of interventions by impact, quality of evidence, implementation information and costs. For each intervention this page summarises the available evidence against each of the five dimensions of EMMIE. The landing page provides, for example, an indication of the measured impacts of an intervention (see below), the presence or absence of information on variations in effectiveness by context (the 'moderator' dimension) and the quality of the evidence presented. In the case of the EMMIE dimensions, the tool indicates whether such information is available. For the Effect of intervention, a categorical scale is used to summarise whether the review(s) suggest that an intervention:

a) is associated with an overall reliable (in statistical terms) reduction (or increase) in crime across the studies reviewed;
b) has been shown to reliably (in statistical terms) decrease or increase crime in some studies, or under some conditions, but not to do so consistently across studies; or
c) never been shown to have a reliable impact upon crime.

The categorical Effect scale is presented in the form of ticks and crosses. A simple easy to read explanation of these ratings is provided on the website but the precise interpretation of these ratings, which was used to code the reviews, is shown in the table below. Many reviews employ statistical meta-analytic techniques to synthesise findings across studies. When conducted appropriately (see above), compared to simpler approaches (e.g. counting how many studies report a statistically significant effect), a statistical meta-analysis produces a more reliable estimate of the overall effect of an intervention by taking advantage of all of the available data. Thus, for studies that include a statistical meta-analysis, it is possible to indicate with some certainty if the intervention has had a reliable overall impact on crime across studies, and/or if it has done so in some studies or under certain conditions.

However, it is important to note that systematic reviews do not always include a statistical meta-analysis. For instance, many narrative reviews describe the results of individual evaluations but do not conduct a statistical analysis to compute an overall estimate of effect size. This can be for a variety of reasons, but in such cases any discussion of the overall effect of intervention may be unreliable. Hence, for such reviews, the coding applied is limited to an indication of whether statistically reliable effects have been observed in some studies or under some conditions. No indication is provided as to whether the intervention has had an overall effect on crime, as to do so would be misleading.

Number ​Rating ​Interpretation

 

 

It is important to note that the evidence presented and its interpretation is based on the systematic review on which the effects of an intervention are based alone.  Conclusions are not based on other sources of information.

     1.

XX✓✓

​No evidence to suggest that the intervention has had a statistically significant impact on crime.


     2.


XX
✓✓

​Statistical meta-analysis suggests that overall, the intervention has had a positive and statistically reliable effect on crime.

     3.

XX

​Overall, the intervention has not had a statistically significant effect on crime (or this was not tested), but there is evidence that it has had a statistically significant positive impact on crime in one or more studies, or under certain conditions.

     4.

XX✓

​Overall, the intervention has not had a statistically significant effect on crime (or this was not tested), but there is evidence from one or more individual studies that it has had either a statistically significant positive or negative impact on crime, depending upon the conditions.

     5.



 

XX✓✓

​Statistical meta-analysis suggests that overall, the intervention has had a statistically significant positive effect on crime, but it has also had a statistically significant negative effect on crime on one or more studies, or under certain conditions


     6.
XX✓✓

​Statistical meta-analysis suggests that overall, the intervention has had a negative and statistically reliable effect on crime.


     7.


XX✓✓

​Overall, the intervention has not had a statistically significant effect on crime (or this was not tested), but there is evidence that it has had a statistically significant negative impact on crime in one or more studies, or under certain conditions.

 

     8.​

XX✓

​Statistical meta-analysis suggests that overall, the intervention has had a statistically significant negative effect on crime, but it has also had a statistically significant positive effect on crime in one or more studies, or under certain conditions.

 

2.  A narrative for each intervention, summarising the evidence on each dimension of EMMIE. These narratives are constructed by at least two researchers and edited by staff from the College of Policing. The findings reported are limited to those included in the particular review in question. No additional searches were conducted to identify subsequent evidence that might challenge the findings of the review, nor was opinion sought about the reviews. Exceptionally, where additional important evidence is (or was made) known to the review team, this is usually included in a 'general considerations' section and the source is clearly marked.

3.  A filter is included (on the landing page), enabling users to find relevant interventions by type of crime outcome (e.g. violent crime), type of population targeted (e.g. locations, victims or offenders) or strength of evidence on effect size.

References
Bowers, K., Johnson, S., Tilley, N., Tompson, L. and Belur, J. (2013). Protocol for Work Package 1: Identifying Existing Systematic Reviews of Crime Reduction. Available here.

Bowers, K., Tompson, L. and Johnson, S. D. (2014) Implementing Information Science in Policing: Mapping the Evidence Base Policing; doi: 10.1093/police/pau058. Further details available here.   

Johnson, S.D., Tilley, N., and Bowers, K.J. (2015). Introducing EMMIE: An evidence rating scale to encourage mixed-method crime prevention synthesis Reviews. Journal of Experimental Criminology, September 2015, Volume 11, Issue 3, pp 459–473

Tilley, N. (2015) EMMIE as a framework for inclusive evidence appraisal.
Tompson, L., Bowers, K., Johnson, S.D., and Belur, J. (2015) EMMIE evidence appraisal coding tool and user guide. [Dataset] UCL Department of Security and Crime Science: London, UK.