Conference Vs. Non-Conference

As a follow-on to this posting where I looked at how well the predictors worked over different periods of the season and noted some odd behaviors, I looked at conference versus non-conference games, to see if there was some difference in performance.  Here are the results:

Predictor	MOV Error	% Correct
TrueSkill -- Conference Games	10.689	72.10%
TrueSkill -- Non-Conference Games	11.539	76.39%
Govan -- Conference Games	10.583	72.17%
Govan -- Non-Conference Games	11.325	76.83%

This shows the same sort of split we see in early-season versus late-season predictions:  conference predictions are closer to the actual Margin of Victory, but more often on the wrong side of the contest.  Of course, most non-conference games are also early-season games, there's some interdependence in these results.  We can try to factor this out by looking at late-season non-conference games and early-season conference games:

Predictor	MOV Error	% Correct
TrueSkill -- Conference Games	10.689	72.10%
TrueSkill -- Early Conference Games	11.485	66.67%
TrueSkill -- Non-Conference Games	11.539	76.39%
TrueSkill -- Late Non-Conference Games	11.148	72.68%
Govan -- Conference Games	10.583	72.17%
Govan -- Early Conference Games	10.988	69.57%
Govan -- Non-Conference Games	11.325	76.83%
Govan -- Late Non-Conference Games	11.089	73.04%

This seems to show that it's really the early season versus late season that matters (and the predictors are particularly bad at early-season conference games).

One easy follow-up to this experiment is to train our predictor on only conference games (or only non-conference games) and see if this improves performance within the category.  I won't post the numbers, but there's no advantage in training on non-conference games when predicting non-conference games, etc.

None of this explains the oddity of MOV Error improving while % Correct simultaneously worsens, but at least there's some evidence that conference versus non-conference is probably not a big factor in prediction performance.