Journal on Policy & Complex Systems Volume 1, Number 2, Fall 2014 | Page 71

��������������������������
sults , particularly confirming that the model deals reasonably with extreme values ( Gilbert , 2008 ). However , the policy analyst may become involved if some of the tests reveal problems with the design as implemented . For example , data analysis may suggest that an expected relationship between two variables is not supported when the model is run with new data .
In contrast , validation requires substantial input from the policy analyst because acceptability is a matter of judgment . Validation checks that the model behaves as expected using test cases , historical data , or other examples where the expected behavior is understood . Whether a model is valid depends strongly on the reasons for developing the model and different reasons can require substantially different criteria ( Hodges & Dewar , 1992 ; Klügl , 2008 ; Sargent , 2010 ). For example , a model to compare options is likely to include accuracy criteria ; computer and mathematical models should reproduce key behaviors such as cycles in growth or delay patterns ; and games to promote understanding may focus on whether behaviors feel realistic to the players .
Typically , the modeler will also do sensitivity analysis as part of the test phase . This analysis varies model settings within plausible bounds and examines the effect of that variation on the model outputs . If the results are reasonable ( that is , the model is qualitatively valid ), the analysis can be used to refine the model . Parameters that have little impact can be considered for removal from the model or , at least , removal from the interface . At the other extreme , parameters that give rise to substantial changes in model output should be considered for further research to reduce the narrow their plausible range and reduce the uncertainty in the model output .
�������������������

In the Use phase , primary responsibility returns to the policy analyst and the implemented model is applied to the original problem , but the modeler will often continue to be involved in analysis and demonstration activities . This phase has the greatest variability in the particular steps because of the variety of model functions . Typically , however , the model will be used in the policy development process for decision support and / or the model will be promulgated ( with documentation ) for use or comments by others .

Additionally , if the model is to be used and updated as part of ongoing practice , a training program for users and an update plan should be developed . Future updating to the model takes two forms : updating internal parameters to reflect current information , and changes to the model design as the real world system evolves or new policy is introduced .
���������������������������

As noted above , important functions

required for dealing with complex societal problems can be considered in the three broad domains of tasks : synthesize knowledge , manage unknowns , and support policy . In practice , there is some overlap as much of the support for policy and practice change arises from the greater understanding of the societal problem provided through synthesizing knowledge and managing unknowns ( Bammer , 2013 ).
����������������������������

Functions in the knowledge synthesis

domain are those that combine theoretical and empirical knowledge from relevant subject matter disciplines togeth-
69