Case Study: WMP Operation Top Deck
continued...
Evaluation
One approach to evaluation is to
count how many conversations
were held and how many
tickets were issued, but this is a
measure of outputs not outcomes.
Theoretically, you could engage
in all this behaviour, claim some
impressive sounding numbers,
but make the problem worse,
so we need to think about a
better measure of impact. We
might, for example, find that
we issued less tickets/had less
educational conversations after
the intervention, but actually be
recording the fact that officers
and others were just taking
longer over each interaction.
During our KEC we discussed the plan to use designated bus routes at specific times, but the idea of the project
is that every bus becomes a potential source of surveillance. This means that we should expect higher levels
of use before the Operation is publicised, and lower levels of use in the vicinity of buses once the Operation
has been promoted. This relies on resource intensive observation before, during, and after the operation, and
could be compared with observed levels of use away from bus routes/buses themselves. This is likely to be
rather complicated and would need to be replicated each time the Operation ran. If it was to become Business
As Usual, then a different approach would be needed. Evaluation is a complex process, and it might be worth
seeking advice on how this can be done meaningfully and with the available resources.
The usual social media metrics (see p92) can also give an indication of whether the message was getting ‘out
there’. Likes, share and follows are not reliable on their own, but we can generally assume that higher numbers
are better than low numbers in terms of this approach being known about. If no-one is engaging, then we can’t
really assume anyone is being deterred.
121