Journal on Policy and Complex Systems
Figure 3 . Empirically derived Accuracy as a function of Mean Square Error ( MSE ). This figure compares two different types of error measures : qualitative and probabilistic . Qualitative error here is Accuracy and represents the proportion of individuals that are correctly assigned to a given class . Probabilistic error is mean square error and represents the aggregate distance that all of the samples are away from the ‘ true ’ value . In terms of computation , MSE is a convex function that allows for ease of training machine learning algorithms . However , in complex systems the choice of error metric depends on the interaction hierarchy of the agents . As shown in this figure , a single choice of MSE threshold for implementation can result in multiple realized accuracy values . Therefore , in developing an implementation , one has to decide not only the minimum desired accuracy , but that which is best suited to the application to which it is applied .
independent outcomes for training . Therefore , this framework enables one to simulate a variety of P ( l | e acc
) given an underlying process or set of assumptions .
For determining model training error , parameters N = 720 and decision model threshold , ∆ = 1 were set , and the result for P ( e acc
| e mse
, ∆ ) is shown in Figure 3 . This figure shows the expected value of accuracy given MSE , but is also subject to some error around the expected value . The Accuracy distributions were generated by sampling an N-sphere of radius e mse
N as described in Krauth ( 2006 ). As N decreases for smaller populations we expect this distribution to exhibit wider variances of Accuracy . This distribution is important because it gives meaning to model training error ( MSE ) in qualitative terms relevant to the application by assigning several expected incorrect observations to a given probabilistic error measure . Figure 3 also shows the expected accuracy of the mean baseline model which always predicts the mean GPA of the training data .
The ABM was run over range of parameter values of 100 replications each . We varied the percentage of students mismatched by 1 and 2 ,
176