Error Reasoning in Complex Systems : Training and Application Error for Decision Models
Once the classroom dynamics have been calculated , a predictive model is selected as de- scribed above to produce a set of residuals , r i
. Then the GPA for each student is calculated where w 1 and w 2 are weights on the school-wide disruptors and the assignment mismatch , respectively .
Application Error . In addition to the training error ( MSE ) discussed above , there is an additional error requirement induced by the choice of application . We can immediately see that the binning accuracy of the classroom assignments is going to be important as it impacts both the number of disruptors ,
n d and the individual class assignment | TrueL−AssignedL |.
In this case Accuracy is defined as where ]_(·) is the indicator function . Because it is not a convex function , it is more difficult to train a predictive model to maximize accuracy than the chosen MSE . It also does not penalize outliers as much as MSE . However , it is important for the application , because it directly factors into the GPA calculation . One could argue that the MSE also factors into the application as well , but despite having correlation to Accuracy , it is not as representative . Therefore , we seek to find an MSE baseline of acceptance for model training that satisfies certain Accuracy constraints .
4 . Analysis & Results
Our goal is to arrive at a formulation described by , where l is an application outcome . For this example , let us define the application outcome to be the mean GPA of the entire school , and let us set a target threshold of l ∗ = 2.25 . This target threshold is the minimum GPA we would like to see for the entire school . The distinction is made between P ( l | e acc
) and P ( e acc | e mse
, ∆ ) because the former is including effects that extend beyond individual observations such as group level events , path dependency , and renewal effects . In contrast , P ( e acc
| e mse , ∆ ) is subjugated to having
175