Error Reasoning in Complex Systems : Training and Application Error for Decision Models
Figure 1 . Social Modeling System . A group of N students is labeled with a set of features , x and true GPA outcomes y ∗ . A predictive model takes the features as input and generates a student-level GPA prediction , y . This prediction is then fed into a decision function that determines the class that the student should be placed into . The class dynamics are then modeled and GPAs are now assigned to each student in the class . We assume that the student GPA performance is a combination of the student ’ s innate ability ( captured by “ True GPA ”), class assignment , and the number of disruptors in the class . Finally , the error of the performance is measured both in aggregate and individually .
P during training is directly a result of the choice of minimizing training error . For this scenario , we want the model to minimize the distance between predictions and true GPA outcomes in training data using MSE
This is a probabilistic type of error that penalizes heavily for outliers which works to correct for overfitting . In this example , we do not specify the predictive model exactly , but identify predictions based on the error of the model . In other words , we assume that the model has been trained and validated to have a certain MSE .
Given a single MSE value , there are many combinations of residuals ,
r i
= y i
− y i
∗ , that could be constituted
to form that value . These residuals all form an N-Sphere of radius √ e mse
N . To produce a realization of the predictive model , then one only needs to select a residual vector from this spherical manifold . Methods for doing so are discussed in Krauth ( 2006 ).
Decision Model . Ultimately , the decision to place the students in a particular class falls to a decision model , D ∆
: y 1→
171