Case Study : WMP Operation Top Deck continued ...
Evaluation
One approach to evaluation is to
count how many conversations were held and how many tickets were issued , but this is a measure of outputs not outcomes . Theoretically , you could engage in all this behaviour , claim some impressive sounding numbers , but make the problem worse , so we need to think about a better measure of impact . We might , for example , find that we issued less tickets / had less educational conversations after the intervention , but actually be recording the fact that officers and others were just taking longer over each interaction .
During our KEC we discussed the plan to use designated bus routes at specific times , but the idea of the project is that every bus becomes a potential source of surveillance . This means that we should expect higher levels of use before the Operation is publicised , and lower levels of use in the vicinity of buses once the Operation has been promoted . This relies on resource intensive observation before , during , and after the operation , and could be compared with observed levels of use away from bus routes / buses themselves . This is likely to be rather complicated and would need to be replicated each time the Operation ran . If it was to become Business As Usual , then a different approach would be needed . Evaluation is a complex process , and it might be worth seeking advice on how this can be done meaningfully and with the available resources .
The usual social media metrics ( see p94 ) can also give an indication of whether the message was getting ‘ out there ’. Likes , share and follows are not reliable on their own , but we can generally assume that higher numbers are better than low numbers in terms of this approach being known about . If no-one is engaging , then we can ’ t really assume anyone is being deterred .
123