Our quality scale

How we rate the quality of research relating to interventions. ​​​

Key

No information
Limited quality
​Moderate quality
​Strong quality
Very strong quality


Where there are a number of systematic reviews relevant to a single intervention, EMMIE scores are based on the findings from all of the reviews and the strongest score (for each of the EMMIE elements) is identified and reported as the EMMIE Toolkit Quality/Effect score.  Therefore, ratings are not necessarily derived from a single review and might reflect the reported findings from multiple systematic reviews.

Ratings from individual reviews are also made available at the end of the intervention summary page to guide users to specific reviews if they are seeking further information on single EMMIE elements.  For example, one review might provide more detailed information on implementation and users may then use the link to read the individual systematic review which provides the most information on how to do the intervention.

Effect

Text to reflect specific review
​Although the review was systematic, many forms of bias that could influence the study conclusions remain
​Although the review was systematic, some forms of bias that could influence the study conclusions remain
​The review was sufficiently systematic that many forms of bias that could influence the study conclusions can be ruled out
​The review was sufficiently systematic that most​ forms of bias that could influence the study conclusions can be ruled out


Mechanism

​​No reference to theory - simple black box
​General statement of assumed theory
​Detailed description of theory - drawn from prior work
​Full description of the theory of change and testable predictions generated from it
​Full description of the theory of change and robust analysis of whether this is operating as expected


 

Moderator

No reference to relevant contextual conditions that may be necessary
Ad hoc description of possible relevant contextual conditions
Tests of the effects of contextual conditions defined post hoc using variables that are at hand
Theoretically grounded description of relevant contextual conditions
Collection and analysis of relvant data relating to theoretically grounded moderators and contexts


 

Implementation

No account of implementation or implementation challenges
Ad hoc comments on implementation or implementation challenges
Concerted efforts to document implementation or implementation challenges
Evidence-based account of levels of implementation or implementation challenges
Complete evidence-based account of implementation or implementation challenges and specification of what would be necessary for replication elsewhere


 

Economics

No intervention related costs and / or no synthesis of costs
Only direct or explicit costs (and/or benefits) estimated
Direct or explicit and indirect and implicit costs (and/or benefits) estimated
Marginal or total or opportunity costs (and/or benefits) estimated
Marginal or total or opportunity costs (and/or benefits) by bearer (or recipient) estimated