Confronting Model Uncertainty in Policy Analysis for Complex Systems : What Policymakers Should Demand
“ Models ” Inferred from Causal Statements
Aggregate Parameters
Have experts affirm important causal relationships among variables comprising a system lacking a formal model . Each set of statements acts as an alternative model . Example : in assessing the value of the United States supplying equipment to a regional security partner , require explicit statements about the perceived benefits and drawbacks for such objectives as joint effectiveness and human and institutional capacity building . Ask for estimates of the signs and magnitudes of effects , and for the shape of the combined-effect curve ( e . g ., linear , decreasing returns , or sigmoid ) ( O ’ Mahony et al ., 2018 ).
Forego modeling mechanisms and instead focus on aggregate consequences represented by parameters ( a type of minimalist modeling ). Example : New technologies could mitigate effects of climate change as characterized by sea level rise . Other aggregate measures are the decoupling rate between growths of the economy and energy production , and the decoupling rate between the growth of the economy and greenhouse gas emissions ( Lempert et al ., 2003 ).
Even with these methods , we cannot be comprehensive , but we can aspire to due diligence by considering model uncertainties of which we are or should be aware . That does not mean tossing in every possibility someone can think of . To do so would lead to paralysis . The practitioner analyst must employ a mix of art and science and must subjectively omit some possibilities ( e . g ., invasions from Mars ).
4.3 . Broadening the Concept of Analysis Campaign
Given methods for varying model structures , as above , the next element for a way ahead is approaching analysis with a multifaceted “ analysis campaign ,” perhaps with elements , as in Table 2 . We use the first column to indicate classic aspects of operations analysis ( Greenberger , Crenson , & Crissey , 1976 ; Walker , 2000 ), the second column to note features now coming into acceptance , and the third column to indicate additional features relating to model uncertainty .
Table 2 . Increasingly Ambitious Treatments of Uncertainty in Analysis Campaigns Early
Identify problem
Advanced ( primarily parametric uncertainty )
Understand system and problem area
More Advanced , Routinely Addressing Model Uncertainty and Multiple Perspectives
+ Recognize parties , stakeholders , and views
+ Understand broadened system , issues , questions , and tradeoffs
+ Recognize emergent phenomena 9
9 This is crucial for “ wicked problems ,” the norm in higher-level policy problem areas ( Rosenhead & Mingers , 2002 ). How to represent emergence with generative models is a frontier challenge in social-behavioral modeling ( Davis et al ., 2019 ).
189