Reporting guidance: Writing a systematic review for inclusion on the Crime Reduction Toolkit

The Crime Reduction Toolkit is an online repository for systematic reviews focused on crime reduction. The aim is for the toolkit to be the 'go to' place for decision-makers and funders when they want to see 'what works' to reduce crime. The College would like to continue to populate the toolkit with high quality systematic reviews and so has developed this guidance to help authors ensure their reviews will be included on the toolkit. The guidance covers:

  • the key criteria for a review to be considered for inclusion on the Crime Reduction Toolkit
  • the different elements of EMMIE (Effect, Mechanisms, Moderators, Implementation, Economics)
  • the evidence appraisal coding tool developed by academics at University College London.

Inclusion criteria

In order for a systematic review to be considered for inclusion in the Crime Reduction Toolkit, it must adhere to the following basic criteria.

  • Intervention: The intervention being reviewed must have a crime reduction (or prevention) focus with outcome measures that are clearly related to crime reduction or prevention (see below).

    Note that crime is defined as 'any behaviour prosecutable by law'. These interventions could be aimed at offenders, communities, or places. The intervention may be delivered in any setting or country.
  • Review type: Systematic reviews (meta-analysis, narrative review, integrative review, realist review, meta-review) which is defined as:

     - Having at least two included studies
     - Having an explicit methods section explaining how the review was conducted
     - States the sources searched and search terms used.
  • Outcome: The research must include a quantitative measure of crime reduction or prevention. These outcomes can include data collected at any point during the criminal justice process, including but not limited to:
     
     - Calls for service (response to emergency service calls)
     - Recorded Crime (official police records including local, regional and national data)
     - Victimisation (estimates or self-reported)
     - Arrests (official records)
     - Charge/ release data (official records)
     - Remand/ bail records (official records)
     - Conviction (official records)
     - Sentencing (official records)
     - Imprisonment (official records)

General measures of crime reduction/ prevention are also appropriate, including:

           - Re-offending/ recidivism (general)
           - Anti-social behaviour 
           - Self-reported offending


The EMMIE framework
Systematic reviews that meet the inclusion criteria outlined above will be assessed using the EMMIE evidence appraisal framework. The EMMIE framework is explained further below.

What is EMMIE?
Systematic review evidence is presented on the toolkit using the EMMIE framework. EMMIE is an evidence appraisal framework for grading and reporting systematic reviews. EMMIE was developed by academics at University College London (UCL). One of the aims of the framework is to help practitioners and decision-makers interpret evidence easily and quickly. EMMIE rates systematic review evidence against five dimensions: Effect, Mechanisms, Moderators, Implementation, and Economic cost.

The five dimensions of EMMIE
The table below provides a brief description of the five dimensions of EMMIE. It also outlines some of the key elements of a systematic review that are assessed using EMMIE. This is not an exhaustive list, rather it is intended to identify some of the important issues to consider in order to produce a review that scores highly when assessed using the EMMIE framework.

​Section
Description​
Guidance notes​
EFFECT​Impact on crime

This section focuses on whether the evidence suggests the intervention led to an increase, decrease or had no impact on crime.

The systematic review needs to include a meta-analysis, other quantitative method of synthesis or narrative summary of quantitative studies.

  • For meta-analyses, report overall pooled effect sizes for any findings as well as effect size for any sub-group analysis undertaken.
  • Report the analytic methods clearly, including confidence intervals and standard errors wherever possible.
  • Clearly state the number of participants in treatment and control groups.
  • Clearly state the individual study effects covered by the review.
  • Be clear about whether or not findings are statistically significant.
  • Be clear about whether any weighting was used when calculating the mean effect size.

Report on all assessments of bias of primary studies considered as part of the review.

MECHANISM​How it works

This section focuses on what it is about the intervention that could explain its effect.

  • Clearly state the theory of change, including an explanation of how the intervention is presumed to work and impact on the outcome, and the theoretical and empirical evidence that justifies this.

          i. How is the intervention presumed to work? 
          ii. What is needed to make it work?

  • What are the anticipated impacts on crime or on behaviour?

Clearly state what data are being used to test the mechanism/ theory of change and whether this analysis was planned as part of the original aims of the review (a-priori) or whether the data led to post-hoc analysis.

MODERATORS​Where it works

This section focuses on the circumstances and contexts in which the intervention is likely (or unlikely) to work.

Moderator analysis is the analysis of factors that are considered causally relevant to the activation of the mechanism(s) through which the intervention is expected to work. These are factors that are a pre-existing condition (for example, the age or gender of participants) rather than a factor introduced by the researchers/implementers (like data period).

  • Aim to pre-specify any theoretically or empirically derived factors and be clear when this has been done.
  • Report any moderator analysis alongside supporting effect sizes.
  • Report any literature that has been used to support the notion that the factors being tested are causally significant.
  • Report where it has been possible to consult with practitioners or other relevant bodies about the contextual factors which might be causally relevant.

Report any data enabling moderators (for example, collection of times and places that interventions were implemented or any environmental variables), and any sub-group analysis clearly.

IMPLEMENTATION​How to do it

This section focuses on the conditions that should be considered when implementing the intervention.

  • Describe the key stages involved in implementing the intervention. As much information as possible about the stages of implementation and the different people or groups that need to be involved is helpful. This also relates to the context, recipients of the intervention and components of the intervention itself.
  • Report any challenges and resolutions as well as factors which would aid implementation.
  •  Clearly identify any key components (of successful) implementation and how best to ensure they happen.

Clearly state the key components that are necessary in order to replicate the intervention elsewhere.

ECONOMIC COSTSWhat it costs​

This section focuses on the costs associated with the intervention, both direct and indirect, and whether there is any evidence of cost-benefit. 

  • Quantify the resources necessary for the intervention (for example, 100 hours of staff time or 200 items of hardware)
  • Quantify the outputs from the intervention (for example, the number of items installed)
  • As far as possible, estimate the direct and indirect financial cost of implementation (overall and by subgroup where appropriate) taking into consideration the cost of any unintended effects
  • As far as possible, estimate the cost-effectiveness of the intervention (for example, the estimated cost of reducing one crime versus the estimated cost of delivering one output) taking into consideration the cost of any unintended effects
  • Clearly state any sources used to estimate the cost of crime
  • Where appropriate, identify the stakeholders involved in an intervention and estimate how the costs and benefits are distributed across those stakeholders

 

If you have any queries relating to the content of this paper please contact whatworks@college.pnn.police.uk

This information has been compiled by staff from UCL Department of Security and Crime Science, University of Surrey and the College of Policing who have been involved in the EMMIE coding of the systematic reviews which are rated and summarised in the Crime Reduction Toolkit.