28 October 2016

Stop and search: Experimenting with training

The College of Policing has today released findings from its stop and search training experiment, the results of which have informed the development of the new national training, which has been launched today.

The College developed the pilot training with the Equality and Human Rights Commission following the home secretary's statement in 2014 to Parliament announcing a package of reforms to stop and search. The pilot training focused on legal decision making, unconscious bias and procedural justice and sought to have a positive impact on officers' knowledge, attitudes and behaviours around stop and search.

The training was piloted last year as a Randomised Control Trial (RCT) in six volunteer police forces around England. The use of an RCT design meant that it was possible to test the impact of the training.  The pilot also looked at how the training had been implemented.

What were the results of the research?
The Research Advisory Service worked with the College to test the impact of the training pilot on primary and secondary outcomes.  Their research found that, immediately after it was delivered, the pilot training had a range of small positive effects on the primary outcomes - officers' knowledge, attitudes and anticipated behaviours in hypothetical scenarios - measured by online officer surveys.  Importantly, some of these effects were sustained at a three-month follow-up.  The secondary outcomes were measured by officers' stop and search records. The pilot training had a more limited impact on these outcomes, which were less central to assessing the overall success of the pilot.  No effects were found on proportion of searches leading to arrest or on the quality of written grounds that officers recorded to justify their searches. Our infographic shows the results of the research.

RAND Europe worked with the College to explore how the pilot training had been implemented. Their research found that there was some variation in how the pilot training was delivered by each of the six pilot forces, possibly due to the guidance given to trainers not being prescriptive enough to ensure consistency across forces. They also found that the pilot training was generally well received by officers, with 83 per cent rating the course as 'good' or 'excellent', though some challenged aspects of the training that went against their accepted practices.

What is an RCT?
An RCT is a type of evaluation that tests whether or not a particular intervention, in this case pilot training, has an impact on outcomes. RCTs are often regarded as one of the most robust evaluation designs, allowing strong causal claims to be made about the effectiveness of an intervention.

RCTs involve selecting a number of people to take part in the study and assigning them at random to be in either the 'treatment group', which will receive the intervention, or the 'control group' which will not receive the intervention. Our infographic shows how we assigned 1,323 officers in our trial. 

Click here to see our design infographic

The use of random assignment is important. It means the two groups should be similar to one another except, of course, in terms of receiving the intervention. After the intervention has been delivered to the treatment group, the two groups are compared to see whether the intervention has had any effect, in this case in terms of officers' knowledge, attitudes and behaviour. Randomisation, therefore, allows any differences to be directly attributed to the intervention.

Why did you use an RCT design in the stop and search experiment?
We considered a range of evaluation designs but opted to use an RCT design to evaluate the effectiveness of the pilot training because it would allow strong causal statements to be made. Doing so also meant we were building on our earlier RCT in Greater Manchester Police,which tested the impact of procedural justice training. Given the general lack of evidence about the impact of police training, it was important for us to find out whether the pilot training made a difference to officers' knowledge, attitudes and behaviour around stop and search. An RCT can also be lower cost than other types of evaluation, as randomisation meant the treatment and control were equivalent to one another before the intervention was implemented, there was no need for us to gather data at the baseline.

What were the main challenges with the RCT?
Conducting an RCT can be particularly demanding, especially when it involves a large number of officers from several forces and tries to test an intervention in a real world setting. If our RCT was to be a 'fair test' of the training pilot, a good level of training implementation was required. This required us to stay in close contact with pilot forces to keep track of how many treatment officers had been trained. In the end, 87 per cent of officers received training. It was also important that the  experimental conditions were maintained by the pilot forces. We had to monitor the treatment and control group officers closely to make sure they remained in these groups throughout the evaluation period and completed the research activities – like our online surveys – even though they had demanding daily roles. While the design of the trial meant trained officers would have worked alongside untrained colleagues, which would have dampened the overall effect of the training, we needed to avoid direct contamination. We found out that control group officers in one force had been booked onto the pilot training by accident. Luckily, we spotted the issue before it became a problem.

Why did you also look at how the pilot training was implemented?
RCTs can tell you whether an intervention has a causal effect on outcomes in a particular context, but cannot tell you why. The evaluation aimed to find out how the training pilot was implemented and why it did or did not work. It explored the nature and quality of training implementation and the mechanisms of change. The research used a range of qualitative and quantitative methods – including interviews and observations – to get different perspectives on the training pilot. The study proved to be really helpful in thinking through how the redesigned national training was to be implemented.

How have the results of the experiment been used?
The findings of the evaluations have been taken into account in developing the new national stop and search training which is to be launched to the police service today. Knowing that the pilot training had a small but broadly positive effect on primary outcomes and less of an effect on secondary outcomes, and how the pilot training had been implemented, provided us with the evidence we needed to make substantial changes to the national training in order to increase its changes of having an impact. For example, the national training been expanded to accommodate a greater focus on procedural justice and includes a specific module for supervisors. 

This trial was added to the Policing and Crime Reduction Research Map, which includes summaries of ongoing and completed trials.

 Stop and search training experiment: Infographics