Error Reasoning in Complex Systems : Training and Application Error for Decision Models
ceptable levels of error . These methods will leverage complex systems modeling better anticipate predictive performance in the context of application . In addition , we will hint at possible implications for a more holistic evaluation of a social modeling system wherein applications of policy frameworks can be adjusted based on model performance . We note that this a preliminary work with interesting findings and is anticipated to lead to further investigations of increased rigor .
2 . Background
The interaction of a predictive
model with an application such as a social system is itself a complex system . Complex systems are composed of numerous interacting entities whose aggregate behavior is nonlinear . It is argued that generating these macro-behaviors requires that we model the individual components of the system ( Miller & Page , 2007 ; Schelling , 2006 ). Agent-based modeling ( ABM ), which is a computational method that allows for the modeling of autonomous , heterogeneous , and interacting agents , is well suited for simulating such systems ( Gilbert & Troitzsch , 2005 ; Manson , Sun , & Bonsal , 2012 ). Within a computer simulation , agents interact with each other and the environment ( Macal & North , 2010 ). Whereas the interactions are known and specified , the cumulative outcomes are often not obvious .
In such a modeling environment , we can observe the impact that varying changes to a system have on meso- and macro-level outcomes . Examples of such changes might include new zoning plans or policies related to land use , the dismissal of poor performing employees within an organization , or strategies for determining student placement in a school where different class levels are available ( e . g ., honors , regular , advanced placement ). Because ABMs account for the inter-dependencies and feedback in a system , we may find that such changes in policies or strategies may in fact result in unintended consequences . For instance , the removal of a poor performing employee may hinder communication because of their prominent position in the organization ’ s social network . A machine learning approach to placing students in class may optimize for improved aggregate level measures of error but offer fewer guarantees of individual classification performance that impact a student ’ s ability to realize their full potential .
Making predictions about or within complex systems spanning multiple levels of inter- action is not a straightforward task . There can be different motives for model training and model implementation . Aggregate error methods are useful , because individual errors con- tribute to the overall error but allow enough robustness to outliers to prevent over-fitting ( c . f ., outside of the computer sciences ( Arya , Fellingham , & Schroeder ( 2004 )). This gives higher confidence that the model features are indeed useful and increase the chance that a model may be applicable in multiple implementation scenarios . However , applications can be high risk when individual predictions carry large consequences if incorrect ( e . g ., predic-
169