Although well-received and popular , Kirkpatrick ’ s model has been challenged and criticized by scholars , researchers , and practitioners , many of whom developed their own variants of Kirkpatrick ’ s original model by addressing its shortcomings . Two of the most recent variants are Phillips ’ ROI Methodology and Kaufman ’ s Five Levels of Evaluation .
In 1994 , Jack Phillips expanded Kirkpatrick ' s model to include a fifth level – Return on Investment ( ROI ). ROI is a financial metric that compares the monetary net benefits of an investment . Relative to the investment cost the net progam benefits are the monetary benefits of the program minus the program ( Phillips & Phillips , 2016 ). ROI is calculated using the following equation :
ROI = ( net program benefits / costs ) x 100 %
Level 5 , ROI , of a learning program is calculated by collecting Kirkpatrick ’ s level 4 data , converting the data to monetary values , and comparing them to the cost of the program ( Phillips & Phillips , 2016 ).
When a course or program is implemented , a chain of impact should occur as participants react positively to the program ( level 1 ); acquire new knowledge , skills and / or attitudes ( level 2 ); apply the new knowledge , skills and / or attitudes to their job ( level 3 ); and , as a consequence , positively affect the education institution by improving key institutional metrics ( level 4 ). When selected institutional metrics are converted to monetary value and compared to the costs of the program , the ROI of the program is calculated ( level 5 ).
FICTICIOUS CASE SCENARIO
Over the past few years Hospital X has been experiencing abnormally high levels of healthcareassociated infections ( HAIs ). To help address this issue , the learning department at Hospital X was asked to develop and implement an e-learning module on hand hygiene , specifically the importance of hand hygiene compliance and proper hand washing techniques in healthcare . All physicians , staff and volunteers at Hospital X were asked to complete the e-learning module . The levels of evaluation and measurement focus for this fictitious educational program is summarized in Table 2 .
Table 2 . Summary of positive and negative reactions to virtual medical education based on themes .
Level Category Measurement Focus
Level 1 |
Reaction |
Measures participants ’ ( physicians , staff and volunteers ) reaction to the |
|
|
e-learning module |
Level 2 |
Learning |
Measures changes in participants ’ knowledge , skills , and / or attitudes |
|
|
as a result of participating in the e-learning module |
Level 3 Behavior
Measures the extent to which the newly acquired information influence participants hand washing behavior and extent to which participants are applying the proper handwashing techniques on the job
Level 4 |
Results |
Measures changes in the number of HAIs as a result of the application |
|
|
of learning to the work environment |
Level 5 |
ROI |
Compares the monetary benefits associated with a reduction in HAIs |
|
|
with the cost of developing and implementing the e-learning module |