Journal on Policy and Complex Systems
tions related to human lives ).
Further clouding the problem is that error has different meaning based on type . Ferri et al . ( 2009 ) identifies the interpretability of error metrics as : 1 .) qualitative metrics , 2 .) probabilistic metrics , and 3 .) ranking metrics . Qualitative metrics such as accuracy require thresholds and count the number of occurrences of correct or incorrect classifications . Probabilistic metrics ( e . g ., mean square error , MSE ) represent the degree of incorrectness as a distance . Ranking metrics focus on order but will not be discussed here . Despite different meanings , experimental evidence ( Gaudette & Japkowicz , 2009 ; Ferri et al ., 2009 ) shows a high degree of correlation between error measures like MSE and accuracy . Often , one type of error is chosen out of convenience for model training ( i . e ., convexity , Tseng ( 2010 )). However , there is no guarantee of in situ performance if the application considered is sensitive to another type of error . Therefore , our approach is to recast conventional baselines of performance in terms of the needs of intended applications .
Many classic agent-based models ( e . g ., Schelling , 1969 ; Epstein , 2002 ) have demonstrated the value of developing simple , stylized models of social phenomena . By uncovering some new relationship or testing some hypothesis , these models have shown the advantages of a computational approach . More recently , these classic models have also provided fruitful groundwork for more realistic , empirically-driven models ( e . g ., Pires & Crooks , 2017 ). In this paper , we present a stylized model of student placement at a school . We seek to demonstrate how methodological improvements in error representation can impact outcomes . Given the stylized nature of the scenario , we do not consider validation of the agent-based model itself . We can however simplify assumptions in the ABM to match those of the ML model and compare results , which is a form of validation .
3 . Framework
The model and simulation framework will be presented in terms of a dynamical system as shown in Figure 1 . We define a school as a bipartite graph that consists of N students , s ∈ { s 1
, s 2 , . . . s N
} paired with K classes c
∈ { c 1
, c 2 , . . . , c K
}. Each student is defined as a tuple , s := ( x , y ∗ ) where x is a vector of input features and y ∗ is the “ true ” resulting GPA of the student . In practice , the “ true ” GPA is only accessible for analysis during the training phase of model development and assumes that assumptions of correct class placement have been verified during the data collection process . Similarly , each class is a tuple , c := ( L , S ) that consists of a class level , L , and a set of students , S ⊆ { s 1
, s 2 ,
. . . , s N
}.
Predictive Model ( Machine Learning ). Central to the social system model is that the student placement into classes is adjudicated by the prediction of a student ’ s GPA , M P
: x 1→ y During ML training , the parameter vector , P , is optimized according to some convex loss function , l (·). Therefore , the chosen
170