Confronting Model Uncertainty in Policy Analysis for Complex Systems : What Policymakers Should Demand
4 . Elements of a Way Ahead
4.1 . A Framework for Uncertainty Analysis in Complex Policy Problems
Simulating consequences of policy actions depends on inputs to the simulation ( including the models to be used ). When each such input is specified , the result is a scenario , run , or case . Assessments from such work must obviously consider the uncertainties in those inputs . But how should those uncertainties be organized ?
We have used two similar frameworks . The first conceives the “ scenario space ” implied by uncertainties in groups labeled Context , Objectives and Strategies , Resources , Effectiveness , Environment , and Other Model Assumptions ( Davis , 1994 , p . 82 ). The second uses a framework and terminology , called XLRM ( Lempert , Popper , & Bankes , 2003 ) which refers to uncertainty factors X , levers of policy L , models and relationships R , and performance metrics M . For the purposes of this paper focused on model uncertainty , we have amended the XLRM framework as shown in Figure 1 . The stack of clouds at the top left of the diagram indicates “ meta models ,” fundamentally different models of the problem ( referred to as alternative perspectives in some past work ( Davis , Gompert , Johnson , & Long , 2008 ; Davis , Shaver , & Beck , 2008 )).
For each meta model , it is possible to specify X , L , R , and M , and to then explore by varying their values systematically . This may involve scores of variables and hundreds of millions of runs . Uncertainties can be represented with a range of discrete values , probability distributions , or both . The cases generated may be comprehensive or based on Monte Caro sampling .
Uncertainty factors ( X ) Uncertain factors not controlled by planning organization ( i . e ., exogenous factors )
Models and relationships ( R ) Models , including relationships among variables for evaluating outcomes , using metrics M , from using levers L across ensembles of assumptions about uncertain factors X
Policy levers ( L ) Factors that can be adjusted as part of a planning organization ’ s strategy
Performance metrics ( M ) Multi-attribute framework consisting of both metrics and acceptable / unacceptable values for those metrics as set by policy
Figure 1 . An adaptation of the XLRM Framework highlighting model uncertainty . 187