Error Reasoning in Complex Systems : Training and Application Error for Decision Models
Brian J . Goode * Fralin Life Sciences Institute of Virginia Tech , Arlington , VA , USA Corresponding author :
bjgoode @ vt . edu
Bianica Pires
Journal on Policy and Complex Systems • Volume 5 , Number 2 • Fall 2019
Fralin Life Sciences Institute of Virginia Tech , Arlington , VA , USA
Abstract
A conventional approach to assessing the feasibility of a predictive model is to compare the predictive performance using withheld training data to a baseline model . Colloquially termed a “ dummy model ”, the baseline model relies only on prior training outcomes ( e . g ., always choose the mean outcome ) and is not based on input features . Trained models that perform better than the baseline in terms of error demonstrate better predictive performance than random guessing based on prior outcomes . In the ideal case , this property will also be present when the model has finished training and is implemented . Although generally a useful threshold for model selection , we question if it is indeed enough of a guarantee when implemented in complex systems of high consequence , e . g ., social systems . This article presents a preliminary exploration of a scenario where a decision model is trained using one type of error , but the application is more aligned with another . Such circumstances arise when the convergence properties of training a model require convex loss functions , but the desired error is non-convex . A stylized classroom assignment simulation is presented where a machine learning model is trained to optimize for mean square error . Individual , student level , observations are used for training the model , but non-independent hierarchical effects compound the error at the class and school grouping levels . We demonstrate the case where a model might perform better than a conventional baseline , but still under- perform relative to the requirements of an application . Methods for using complex system modeling to establish a new baseline approach are explored .
Keywords : training error ; complex systems ; machine learning ; social system modeling ; decision making
165 doi : 10.18278 / jpcs . 5.2.10