Correlation Between Predictors

Danny Tarlow was kind enough to give me some comments on a paper I'm writing about the work reported in this blog, and one of his suggestions was to look at whether the predictors I've tested are picking up on the same signals.  This is a significant question because if the predictors are picking up on different signals, then they can be combined into an ensemble predictor that will perform better than the individual predictors.  (Dietterich 2000) showed that

"...a necessary and sufficient condition for an ensemble of classifiers to be more accurate than any of its individual members is if the classifiers are accurate and diverse."

A classifier is accurate if is better than random guessing.  Two predictors are diverse if they make different errors.  Intuitively, an ensemble will perform better than the base predictors if the errors in the base predictors are uncorrelated and tend to cancel each other out.  Our predictors are all obviously accurate, but are they diverse?

To test this we can measure the correlation between the errors made by the different predictors.  If they are uncorrelated, then it is likely that we can construct an ensemble with improved performance.  I don't have the time and energy to test all combinations of the predictors I've implemented, but here are the correlations between the top two won-loss based predictors (Wilson, iRPI) and the top two MOV-based predictors (TrueSkill+MOV, Govan):

	Wilson	iRPI	TrueSkill + MOV
iRPI	0.99
Trueskill+MOV	0.93	0.93
Govan	0.95	0.95	0.98

Not unsurprisingly, the highest correlations are between the two won-loss predictors and the two MOV-based predictors.  But all of the predictors are highly correlated.  The least correlated (by a hair) are Wilson and TrueSkill+MOV.  Putting those two predictors into a combined linear regression or an averaging ensemble results in performance worse that TrueSkill+MOV alone.

On the other hand, perhaps using the best predictors is the wrong course.  Perhaps its more likely that the worst predictors are uncorrelated with the best predictors, and a combination of one of the worst with one of the best would be fruitful.

	Wilson	iRPI	TrueSkill + MOV	Govan
1-Bit	0.83	0.83	0.80	0.79
Winning Percentage	0.97	0.98	0.92	0.93

As this shows, even the 1-Bit predictor ("the home team wins by 4.5") is highly correlated with the better predictors, and using just the winning percentage shoots the correlation to 0.92+.  Adding these predictors to an ensemble with the better predictors also results in worse performance.

Of course, it's always possible that some combination of predictors will improve performance.  There's been some interesting work in this area -- see (Caruana 2004) in Papers.  But for right now I don't have the infrastructure to search all the possible combinations.

Predicting Using Winning Percentages

I'm working on a journal article summarizing my recent findings, and I realized that I never tested a predictor based upon winning percentages.  This is an unfortunate oversight, because winning percentages is the easiest obvious measure of a team's strength, and was the primary metric for selecting (at-large) Tournament participants before the development of RPI.  And RPI was developed primarily to address the criticism that not all winning percentages are equal.  So I've corrected that error by going back and testing a predictor based upon winning percentages:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
Winning Percentages	65.0%	11.63
RPI (infinite)	74.6%	11.33

Both results are interesting.  % Correct shows little improvement over simply picking the home team to win every game, which I take to be further confirmation of the HCA in college basketball.  On the other hand, MOV Error is nearly as good as the best variant of RPI.

Combinations & Other Models

Before we move on to MOV-based rating systems, it may be instructive to look at combining the various "RPI-like" rating systems to see if using them together can improve our prediction performance.  For these experiments, I'll be looking at our three best RPI-like ratings: TrueSkill, Improved RPI (iRPI) and the Iterative Strength Rating (ISR).

The first experiment we can try is to use more than one rating as an input to our linear regression.  The following table shows the performance using various combinations of the three ratings:

Predictor	% Correct	MOV Error
TrueSkill	72.8%	11.07
TrueSkill + Improved RPI	72.9%	11.01
TrueSkill + Iterative Strength Rating	72.7%	11.05
TrueSkill + Improved RPI + Iterative Strength Rating	73.0%	11.01
Improved RPI + Iterative Strength Rating	71.9%	11.31

The combination of TrueSkill and the Improved RPI improves performance modestly.  Adding in the Iterative Strength Rating does little (and in fact, the home team's ISR gets optimized out of the linear regression).

Another experiment we can try is to do a separate linear regression for each rating and then average their predictions (for regression tasks, this is done with the Vote operator in RapidMiner).  Here are some averaging results:

Predictor	% Correct	MOV Error
TrueSkill + Improved RPI (combined regression)	72.9%	11.01
TrueSkill + Improved RPI (averaged)	72.9%	11.04
TrueSkill + Iterative Strength Rating (averaged)	72.8%	11.17
TrueSkill + Improved RPI + Iterative Strength Rating (averaged)	73.0%	11.16

Averaging provides worse performance than using a single linear regression.

We can also try using a more sophisticated prediction model than a linear regression.  For cases where we're only using a single rating value for each team, we wouldn't expect this to provide significantly better performance than the linear regression.  Here are the performances of some alternative models for the TrueSkill ratings:

Predictor	% Correct	MOV Error
TrueSkill (neural network)	72.8%	11.07
TrueSkill (support vector machine)	72.7%	11.08
TrueSkill (k-NN, k=96)	72.7%	11.13

As expected, there is no improvement over a simple linear regression.  The alternate models also provide no benefit when we are using multiple ratings:

Predictor	% Correct	MOV Error
TrueSkill + Improved RPI (neural network)	68.8%	12.24
TrueSkill + Improved RPI (support vector machine)	72.9%	11.02
TrueSkill + Improved RPI (k-NN, k=96)	72.8%	11.13

SVNs do the best here, but still not an improvement over the linear regression.

So combining TrueSkill & Improved RPI into a single regression is an improvement, but generally more sophisticated models don't provide any value.  It's interesting to note that this performance is already as good as the best models reported in the literature.

Unless I get further distracted, I'll be moving on next to assessing ratings/models which make use of the margin of victory (MOV) next.  I have a small collection of ratings/models to assess, but I'm always looking for inputs, so if you have a favorite ranking that you'd like to see included, please let me know!

RPI-Like Summary

I've re-implemented the incorrect algorithms and results are below.   But before summarizing the results so far, I'll briefly mention a few others that I did not implement.

Glicko

The Glicko Rating system was developed by Mark Glickman (chairman of the US Chess Federation (USCF) ratings committee) as an improvement upon ELO.  The Glicko Rating system is very similar to Microsoft's TrueSkill rating system.  Both are based on Bayesian reasoning and provide both a rating and an uncertainty.  For two player games that don't produce ties (i.e., basketball) the only significant difference is that Glicko uses a logistic distribution of performance ratings rather than the Gaussian distribution used by TrueSkill.  It seems unlikely that this small difference would result in a significant difference in predictive performance.  (The Glicko-2 system adds a "volatility" factor, and this might be worth investigating at some point.)

Jon Dokter

The Prediction Tracker tracks the accuracy of various NCAA basketball rating systems both for won-loss and against the Las Vegas line.  The highest rated system as of the end of the 2010-2011 season belonged to Jon Dokter.  Dokter's ratings are intended to be predictive (he sells wagering advice for $10.99/week) but are not well-explained.  This page provides a general overview of his methods, but there is not enough detail to replicate his rating system for testing.

Bethel Rank

Roy Bethel proposed a ranking system based upon "Maximum Likelihood Estimation" specifically to address the problem of sports with unequal strength of schedules, i.e., where teams do not play a complete round-robin.  (His paper is available in the papers archive.) While Bethel's rating system appears interesting, it cannot handle winless or lossless teams.  Since both happen with some regularity in college basketball (and are certain to occur for a significant portion of the season) using this rating is problematic.

NCAA Tournament-Specific Ratings

A number of people have created rating systems specific to predicting the NCAA tournament, e.g., Bradley West and Nate Silver.  Most of these rely on seeding, human polls or other information that exists only for the tournament, and makes them unsuitable for predicting regular season games.

Other Systems

I'm interested in any pointers to other rating systems that I should investigate, particularly if they have a fundamentally different approach than the ones I've covered already.  Send me an email (srt19170@gmail.com) or leave a comment.

Summary of Results

In total, I tested about 110 algorithm variants.  (Some multiple times as I uncovered errors in my code!)  The following table summarizes the best performances for each algorithm:

Predictor	% Correct	MOV Error
Naïve	50.0%	14.50
1-Bit	62.6%	14.17
Random Walkers	71.0%	11.72
RPI	71.2%	11.62
ELO	71.8%	11.59
KRACH	71.5%	11.50
Colley	71.8%	11.33
Wilson	71.9%	11.32
ISR	71.9%	11.32
Improved RPI	72.1%	11.30
TrueSkill	72.8%	11.09

There are a couple of interesting points to be gathered from this.

First, the best performer (TrueSkill) represents about a 15% improvement over always picking the home team, but only about a 2% improvement over the standard RPI.  On MOV Error, it fares somewhat better, being a 22% improvement over picking the home team, and about 5% over RPI.  But given the complexity of TrueSkill compared to the 1-Bit algorithm (or even standard RPI), that isn't as much improvement as we might have hoped to see.

Second, we note that the performance of our implementation of ELO is very close to the tracked performance of Sagarin ELO at Prediction Tracker.  That gives us some confidence in these results.  (On the MOV Error side, there seems to be about a 1.5 point bias in MOV Error between my measurements and those on Prediction Tracker.  I don't know why that would be.)

Third, if we compare TrueSkill to the rating systems tracked at Prediction Tracker, we see it would have beaten all systems except Jon Doktor -- and his system makes use of MOV.  So even without making use of MOV, we have a system that is competitive with the best systems available.

Random Walkers

The next RPI alternative we'll look at is inspired by the LRMC rating system developed by Paul Kvam and Joel Sokol at Georgia Tech.  This rating system has gotten some publicity in recent years for predicting the NCAA tournament outcomes better than other rating systems.  The LMRC rating system is inspired by ranking system for NCAA football developed by Callaghan, Porter, and Mucha.  The core of their system is what they call a "simple random walker": 

Consider independent random walkers who each cast a single vote for the team they believe is the best. Each walker occasionally considers changing its vote by examining the outcome of a single game selected randomly from those played by their favorite team, recasting its vote for the winner of that game with probability p (and for the loser with probability 1-p).

So the random walker rating system simulates a large number of these random walker voters, and iterates until the votes across all the teams settles down to a steady state.

Callaghan, Porter, and Mucha derive a solution for this problem based upon linear algebra and differential equations that is beyond my mathematical abilities to comprehend (or program).  However, it is relatively straightfoward to calculate this with the sort of iterative solution we have used elsewhere.

Assume that a team T currently has W (weight) votes as the best team. (W need not be an integer.)  We update W by looking at the games that T has played, and distributing some of W to the winner and the loser of each game.  So some of the weight comes back to team T and some goes to its opponent.  We do the same thing for every other team until the weights of the teams settle down to a steady state.

If team T has played N games, we begin our distribution by splitting W into N buckets.  We then walk through the list of games that team T has played, and for each team we split the bucket (W/N) into two parts: (W/N)*p for the winning team, and (W/N)*(1-p) for the losing team.

The selection of "p" determines how the weights will be split between winning and losing teams.  If p=1.0, then the winning team in each game receives all the "votes" and the system is similar to RPI.  If p=0.50, then there is no bonus for winning games and every team tends to the same rating. In the casue of the Callaghan, Porter, and Mucha paper, they use p=0.80 to generate their ratings.  (LRMC, as we shall see, uses a more complex formula based upon MOV to determine the split.)  We will try a range of values for p=0.50 to p=1.00:

Predictor	% Correct	MOV Error
Wilson	77.7%	10.33
RW (p=0.60)	70.3%	11.87
RW (p=0.70)	71.0%	11.72
RW (p=0.80)	69.3%	12.09
RW (p=1.00)	64.7%	13.60

Performance is maximized somewhere in the neighborhood of p=0.70, but even so is considerably worse than our best predictor so far.

We'll revisit the "random walker" model later when we look at LRMC.  It's an intriguing framework for distributing weight between teams, and perhaps can be modified to provide better overall performance.

RPI Recap

I want to take a posting to recap the various approaches we've looked at over the past few weeks.  Although I've been applying them to the RPI algorithm, I believe they'll continue to be generally useful as we look at more complex prediction approaches.

(1) Measure

The initial insight was to have a clear notion of what we're trying to achieve and then select objective metrics to measure progress.  The importance of this was repeatedly evident as we looked at various RPI tweaks.  In many cases, "obvious" improvements to RPI turned out to be no improvement at all.  In another case, we corrected a math error in RPI that turned out not be an error at all (or at any rate didn't improve predictive performance). 

(2) Home Court Advantage

There is a proven and significant home court advantage in college basketball.  We looked at several ways to account for HCA, but in the end our predictive model captured it better than we could with an apriori solution.  For example, the linear regression for one of our RPI tweaks looked like this:

MOV = 85.414 * Hrpi - 80.508 * Arpi + 1.580

The different coefficients for the home team's RPI (85.414) and the away team's RPI (80.508), as well as the constant bias (1.580) combine to model the home court advantage.

The lesson here is that the HCA is important to accurate prediction, and we need to ensure that either our model accommodates it naturally (as in the case above) or that we otherwise account for it in our data.  For the latter case we looked at a number of possible tools: weighting the home record differently, applying a point bias to the home team, or splitting a team into a "home team" and "away team" component.

(3) Strength of Opponent

One of the paradoxical challenges of assessing a team's strength is that you need to know the strength of the teams it has played.  It's a classic Catch-22.

The RPI approach to breaking this death spiral is to base the strength metric on some other measure.  This is what RPI does -- RPI tells us how strong a team is, but RPI itself is eventually dependent on won-loss records.  By recursively finding the winning percentage for opponents, and opponents' opponents, RPI tries to estimate the true strength of a team.

One useful improvement we found on this technique is to carry out this recursion "infinitely" by using an iterative solution.  We can use another measure as an initial estimate of our strength metric, and then iteratively adjust the metric until we get the values that best match the actual performance so far.

(4) Data Filtering

Our prediction algorithms are based largely on past performance.  One approach we used with some success for RPI was to filter the past performance to eliminate games that reduced our prediction accuracy.  In the case of RPI, we found in it was useful to eliminate games where the MOV was 1 point.  In general, we may want to look at a variety of different filtering approaches (e.g., eliminate blow-out games, pre-season tournament games, etc.).

(5) Modeling Changing Performance

As we build predictive models, we have to consider whether a team's performance changes substantially over the course of a season.  (It clearly does so from season to season.)  If it does, then our predictive accuracy might be better if we discount older games when building our models.  Another intriguing possibility is whether we can identify specific events where a team's performance changed substantially, e.g., when we notice the minutes played by specific players changes signficantly due to an injury or other reason.

In the case of RPI, we weren't able to improve our accuracy by weighting recent games, but other methods or different approaches may yet prove valuable.

Next up we'll start taking a look at some alternative methods for rating teams that use only won-loss records.  There are a number of candidates, and we'll be looking to see if any of them  provide a significant advantage over our (tweaked) RPI.

The Recency (Non-)Effect

Our final RPI tweak (unless I'm lying again) will look at the impact of recent results on predictability.  It's reasonable to suppose that a team's level of performance might change during the season -- that is, it might get better or worse as the season goes along.  In this case,recent games might be a better predictor of future performance than older games

To test this notion, we can modify our RPI calculations so that they take into account only the last "N" games.  For some representative values of "N", that yields these results:

Predictor	N	% Correct	MOV Error
1-Bit		62.6%	14.17
RPI (nw, 15+15+70)		76.8%	11.46
RPI (nw, 15+15+70)	4	67.6%	12.52
RPI (nw, 15+15+70)	8	72.8%	11.89
RPI (nw, 15+15+70)	16	74.2%	11.47

Restricting the RPI calculations to the most recent games has a strong negative impact on predictive power.  Of course, this method is fairly drastic: it gives 100% value to the most recent games and 0% to anything older.  A more nuanced approach would count the most recent games more, but not discount entirely the older games.  Something like a weighted moving average would be ideal, but it isn't entirely obvious how to apply that to RPI.  Instead, we'll take the approach of counting the most recent games more than once.  That is, we'll treat each team as if it played it's most recent games multiple times (with the same results).  This will have the effect of weighting those games correspondingly more.

This table shows the impact of repeating some number of recent games some number of times (in addition to counting all games once):

Predictor	N	Repeats	% Correct	MOV Error
RPI (nw, 15+15+70)	4	1	72.8%	11.63
RPI (nw, 15+15+70)	8	1	74.2%	11.56
RPI (nw, 15+15+70)	16	1	74.6%	11.49
RPI (nw, 15+15+70)	4	1/3	73.6%	11.64
RPI (nw, 15+15+70)	8	1/3	75.4%	11.58
RPI (nw, 15+15+70)	16	1/3	75.2%	11.51
RPI (nw, 15+15+70)	2	2	70.6%	12.49
RPI (nw, 15+15+70)	1	2	66.2%	12.82

In no case that I could find did weighting recent games improve performance over the baseline.  Putting emphasis on a small number of recent games is particular bad; this suggests that if teams do change performance over the course of the season it is only slowly.

Next time (unless something shiny distracts me again), we'll sum up the various tweaks we've tried with RPI.

MOV Cutoff Filter

The next tweak we'll look at for RPI goes back to our previous discussion about the limits of prediction.  We noted there that the last possession of the game could swing the final game score by 6 points.  Even if it's hard to quantify exactly, there's a certain random component in final game scores.  That's particularly a concern for rating systems that rely only on won-loss records, because a swing of a few points in a close game could change a game from a win to a loss.  Intuitively at least, we might want to discount close games when calculating RPI, under the theory that they're not really good evidence that one team was better than the other.

The easiest way to do this is to filter out all games where the final MOV was less than some threshold when computing a team's RPI.  Making that change and applying it with various thresholds to our current best "% Correct" RPI variation gives these results:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unw,15+15+70)	75.4%	11.49
RPI (nw, 15+15+70, mov-cutoff=1)	76.8%	11.46
RPI (nw, 15+15+70, mov-cutoff=3)	75.4%	11.56
RPI (nw, 15+15+70, mov-cutoff=8)	73.4%	11.62
RPI (nw, 15+15+70, mov-cutoff=12)	70.0%	11.98

Filtering out all games that were decided by 1 point provides a big improvement in "% Correct" and a small improvement in "MOV Error". It's also interesting to note how resilient RPI is to removing games. MOV cutoff = 12 removes more than 40% of the games and only introduces a few percent more error.

We can try the same technique with our current best "MOV Error" variation (the infinitely deep RPI):

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (improved)	74.6%	11.33
RPI (improved, mov-cutoff=1)	74.2%	11.31
RPI (improved, mov-cutoff=3)	74.6%	11.43
RPI (improved, mov-cutoff=8)	73.4%	11.36
RPI (improved, mov-cutoff=12)	72.0%	11.79

Again, an MOV cutoff of 1 provides a (small) improvement in the MOV Error.

It turns out I lied in the last post when I said the MOV cutoff would be the last tweak we examined for RPI.  In the next posting, I'll take a quick look at calculating RPI using a rolling window that only counts the last "N" games, to see if there's a usable "recency" effect in teams' performances.

A Tale of Two Demons

We now turn our attention to one of the most vexing aspects of RPI, illustrated this season by the Tale of Two Demons:  the first being the DePaul Blue Demons and the second being the Northwestern State Demons.

DePaul University finished the season 7-24 and a miserable 1-18 in conference.  DePaul's wins included 4-25 Chicago St., 8-21 Northern Illinois and 9-21 Central Michigan.

In contrast, Northwestern St. finished the season 16-13 with a respectable 10-7 record in conference.  They split home-and-home with Southland East Division champion McNeese St. and came within 1 point of winning the conference tournament and advancing to the NCAAs.

Yet curiously, DePaul has an RPI of 0.4590 and Northwestern State an RPI of 0.4566!  How does this happen?

Recall the oft-cited formula for RPI: 

RPI = (WP * 0.25) + (OWP * 0.50) + (OOWP * 0.25)

The biggest factor in this equation is the Opponents' Winning Percentage.  But critically, the opponents' winning percentage is calculated from all of a team's opponents.  So DePaul University benefits more from their 18 losses to strong Big East opponents than Northwestern State does from it's winning record in the Southland Conference.

No doubt when the NCAA concocted this aspect of the RPI formula, they were thinking of the case where a team has run up a good record against a bunch of patsies.  In that case, the team's RPI gets docked because the OWP is low; and that makes sense because those are also (mostly) opponents the team has beaten.  (The NCAA might also have been intentionally motivating teams to play strong out-of-conference schedules.)  But it certainly seems counter-intuitive to give a team more credit for being beaten by good teams than for beating mediocre teams.  And surely it makes for worse predictability.

Doesn't it?

Well, time to roll out the code and test.  A reasonable first approach is to calculate OWP as the average of all the opponents a team actually beat, rather than all the opponents.  (A similar reasoning applies to OOWP.)  Let us try that approach using our current best RPI formula as well as the "Infinite Depth" RPI:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unw,15+15+70)	75.4%	11.49
RPI (unw,15+15+70,winners)	72.0%	12.20
RPI (infinite)	74.6%	11.33
RPI (infinite,winners)	74.0%	11.97

In both cases, this change makes for a worse predictor!  That's hard to comprehend.  Essentially, this says that losing to good teams makes a team more likely to win future games.  While I can certainly come up with a rationalization for that (e.g., playing good teams makes you a better team, even if you lose), it's hard to put much faith in it.  Still, the numbers don't lie.

As a second approach, we can break out the OWP for both the opponents we've beaten as well as the opponents that beat us.  So we'll have a "Winners OWP" and a "Losers OWP".  Instead of completely dropping the OWPs of the teams that beat us, we can include them as a less-weighted factor. Weighting the beaten opponents by about 3x the "lost to" opponents does improve performance over just the beaten opponents (to 369/11.47 for the unweighted, 15+15+70 version of RPI) but not enough to make it better than the standard versions.  So (at least for this approach) we have to conclude that RPI may be right about the relative strengths of the DePaul Blue Demons and the Northwestern St. Demons.  Apparently playing good opponents, even if you lose to them, makes you a stronger team.

We've about beaten RPI to death at this point, but we'll take a look at one more tweak before moving on to look at some other ratings that also make use of only won-loss records.

Infinitely Deep RPI

In the previous posting, we looked at extending the "depth" of RPI to an additional level -- that is, including OOOWP in our RPI calculation.  In this posting, we'll look at extending RPI to "infinite" depth.

We can view the RPI for a team as having two components.  The first is a measure of the team's strength, and the second is a measure of it's opponent's strength:

RPI = (Team's Strength) + (Team's Opponents' Strengths)

The standard RPI formula estimates the first term with WP and the second term with OWP and OOWP.  But a better estimate of a team's strength is the RPI itself, so we could use that instead, at least for the opponents:

RPI = (Team's Strength) + (Average of Team's Opponents RPIs)

But now we have a weirdly self-referential equation:  Calculating Duke's RPI will involve calculating UNC's RPI which will involve calculating Duke's RPI, ad infinitum.

We can solve this sort of self-referential formula iteratively -- that is, we can calculate an initial RPI estimate for all the teams and then repeat that process using the latest RPI estimates at each step until we reach an answer.  There are two requirements for this to be successful.  First, we need "bootstrap" values to get ourselves started, and secondly, we need to structure our formula so that the answers converge.

Conveniently for us, Andrew Dolphin has already done the hard work of determining a formula that meets those criteria:

RPI = (WP-0.5) + (Average of Team's Opponents RPIs)

so all we have to do is calculate.  Plugging this formula into our framework and testing gives us:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unw,15+15+70)	75.4%	11.49
RPI (infinite)	74.6%	11.33

This gives us slightly better performance (in MOV Error) than the RPI that stops at OOOWP, suggesting that the standard RPI formula depth (to OOWP) is probably sufficient for our purposes.

Next we'll look at one of the most perplexing oddities of RPI.

OOOWP

So far, we have explored various shortcomings of the RPI: home court advantage, averaging, and the distribution of the elements.  We've found a few tweaks that have improved the performance of RPI as a predictor.  We turn now to yet another potential area of improvement: the depth of evaluation.

Recall the (revised) formula for RPI:

RPI = 0.23*WP + 0.23*OWP + 0.54*OOWP

The last two terms of this formula can be thought of as a measure of a team's "Strength of Schedule" expressed as the winning percentage of a team's opponents and their opponents.  RPI arbitrarily stops evaluating this "Strength of Schedule" term at two levels.  Does extending this to more levels (e.g., OOOWP) add any predictive value?

The answer turns out to be yes and no.  With a formula of approximately:

RPI = 7*WP + 7*OWP + 7*OOWP + OOOWP

we get a performance of:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unw,15+15+70)	75.4%	11.49
RPI (+oowp)	74.6%	11.36

This reduces the MOV Error but doesn't improve % Correct.

So extending the depth of RPI another step provides at least some value.  This raises the natural question: is there value in extending it yet another step?  ..and another step?

While we could certainly manually explore those possibilities by calculating OOOOWP, etc., it's perhaps better to cut to the chase and ask whether we can extend the depth of RPI infinitely, and see what predictive value that has.  It may seem counter-intuitive, but it's possible to extend RPI to an "infinite" depth, but it requires a different computational approach.

RPI Distribution

In our previous ill-fated excursion triggered by Dick Vitale, we mentioned briefly the formula for combining the terms of RPI:

RPI = (WP * 0.25) + (OWP * 0.50) + (OOWP * 0.25)

This formula is pleasingly symmetrical but it isn't obvious at a glance why the terms are weighted as they are.  Andrew Dolphin provides a (possible) explanation on his web page:  Essentially, the numbers are chosen to make the formula the best approximation for an "ideal" RPI that went to an infinite depth, i.e., included terms for OOOWP, OOOOWP, etc.  The "proper" weightings are determined by the ratio of conference games to non-conference games, and for basketball Dolphin gives the following ideal formulas:

RPI = 0.27*WP + 0.46*OWP + 0.27*OOWP
              or
RPI = 0.23*WP + 0.23*OWP + 0.54*OOWP

The first is very close to actual RPI formula, so it's possible that the NCAA chose that weighting intentionally.  As an experiment we can try the second alternative to see if that provides better performance:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unweighted)	74.6%	11.53
RPI (unweighted, 23+23+54)	75%	11.37

And indeed it does, improving in both metrics over the unweighted RPI.  A quick hill-climbing experiment with other values for the distribution hits upon this alternative:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unweighted)	74.6%	11.53
RPI (unweighted, 15+15+70)	75.4%	11.49

which improves the number of correct predictions even further, at the cost of a slight degradation in MOV error.  At this point, I'm favoring improvements that increase the % Correct over the MOV Error, so we'll take this as our new highwater mark.

Of course, since we're using the RPIs of both teams as inputs to a predictor, there's no reason that we have to combine the elements of the RPI at all.  We can feed the elements of the RPI (e.g., the home team's WP, OWP and OOWP as well as the away team's WP, OWP and OOWP) directly into our predictor and let it choose the best weightings.  This also has the advantage of not requiring the same weightings for both home and away.  As it turns out in this case that doesn't result in signficantly better performance -- it drives the MOV error down slightly at the cost of % Correct -- but it's an option we can keep in our back pocket.

The Prophet Dick Vitale

We'll return for a moment to the issue of home court advantage.

RPI normally adjusts for home court advantage by weighting home games differently than away games and we looked at adjusting by points.  In both cases we found that adjusting for HCA actually worsened the performance of our predictor.

A radically different approach to dealing with HCA was suggested by a comment from the fount of all basketball knowledge, Dick Vitale.  During the 2011 season, commenting on St. John's -- a team that was hard to beat at home but weak on the road -- he said:

    "They're a different team at home, baybee!"

This keen observation suggests that we could account for the HCA by treating each team as two different teams: a "St. John's at home" team and a "St. John's on the road team".  To do this, we calculate a "Home RPI" based upon the team's winning percentage at home, the team's opponents' winning percentage on the road, and the team's opponents' opponents' winning percentage at home.  The "Away RPI" is calculated in the opposite manner.

Given that home teams win 2/3 of the games, you might expect teams to have better Home RPIs than Away RPIs.  In fact, just the opposite happens.  Recall that the formula for combining the parts of the RPI is:

    RPI = (WP * 0.25) + (OWP * 0.50) + (OOWP * 0.25)

The OWP is the largest part of this calculation, and that's largest when we're looking at the opponents' home records, i.e., when we are calculating the "Away RPI".  So in general, a team's "Away RPI" tends to be higher than its "Home RPI".

For the 2011 season, the top five away teams were:

Team	ARPI
San Diego St.	0.715
Kansas	0.710
Ohio St.	0.703
BYU	0.695
UNLV	0.668

Reviewing these teams' road records, this looks fairly accurate.

Unfortunately, breaking RPI down into home and away doesn't seem to have a lot of predictive value.  Using the home team's "Home RPI" and the away team's "Away RPI", we get this performance:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unweighted)	74.6%	11.53
RPI (home-away)	69.0%	12.5

...considerably worse than our previous best performance.  So Dickie V's advice turns out to be worthless.  (But we shall return to consider Mr. Vitale's wisdom again in the near future, where it may prove more valuable.)

Averaging RPI

Previously, we looked at improving the the Ratings Percentage Index (RPI) by fixing its treatment of the Home Court Advantage (HCA).  We found that the best results were had by eliminating the HCA adjustments.  There are some other approaches we can explore to improve RPI's treatment of home court advantage, but we'll turn now to another area of possible improvement. 

The RPI consists of the three terms: a team's winning percentage (WP), the winning percentage of the team's opponents (OWP), and the winning percentage of the team's opponents' opponents (OOWP).  These latter two terms are defined in a curious way.  They are not average values, but rather an "average of averages," e.g., OWP is computed by averaging the winning percentages of all the opponents.  Suppose, for example, that UCLA plays three opponents: USC (4-1), Arizona (6-0) and Oregon (0-1).  (USC and Arizona played in the preseason NIT.) OWP is calculated by averaging the WPs of these teams: (0.80+1.0+0)/3 = 0.60.  In contrast, the average OWP is (10-2) = 0.83. 

About this, Paul Kislanko says:

This would be equivalent to defining a batting average in baseball by the average of the BA for each game played.  A 0 for 5 day followed by a 3 for 4 day would give (.000 + .750) = .375 instead of 3 for 9 = .333.  In basketball, a player in a 3-game tournament who hits 2 of 10 shots, then 3 of 6, then 4 of 10 would have a shooting percentage of (.200 + .500 + .400)/3 = .433, when in fact for the tournament she was 9 for 25 = .360.   

There's no other formula in all of sports statistics that makes this mistake.

As far as I know, there's no reason that the NCAA chose to use an average of averages in calculating the RPI.  And I'm not aware of any reason why one method should be preferred over another, although Kislanko's argument is certainly compelling on its face.  It's certainly worth investigating which method provides the best predictions. 

If we substitute averaging into our RPI algorithm, we get this performance:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unweighted)	74.6%	11.53
RPI (unweighted+ave)	74.2%	11.63

This is worse performance, so the "average of averages" is the better choice (Kislanko's outrage over the poor math notwithstanding :-).

Another feature of how RPI calculates the OWP and OOWP is that if a team plays an opponent twice, that opponent's winning percentage is counted twice.  This makes some sense -- certainly if we played two different teams with identical WPs we'd want to count them both when figure the average strength of our opponents.  But perhaps it could be argued that playing a team a second (or third) time shouldn't affect the overall strength of your opponents.  Again, it is easier to test than to worry too much about a rationale.  If we eliminate duplicates opponents when calculating RPI (still using averages), we get this performance:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unweighted)	74.6%	11.53
RPI (ave)	74.2%	11.63
RPI (nodupes)	74.2%	11.59
RPI (ave+nodupes)	74.2%	11.63

No improvement over straight unweighted RPI.

Before we leave this topic, let's perform another experiment.  Instead of averaging, we could try using the median value for the OWP and OOWP.  Imagine a team whose opponents have records of 3-0, 0-1 and 0-1.  The average of averages of these is 0.33; the average of these is 0.60; and the median of the averages is 0.00.  We could certainly construct a rationale for why using the median might be a good idea, but again there's really no a priori reason to prefer one over the other.  But it seems worth a quick experiment:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unweighted)	74.6%	11.53
RPI (medians)	72.8%	12.14
RPI (medians+nodupes)	71.4%	12.57

Both medians and medians without duplicates per above underperform the unweighted RPI. So it appears that averages are better than medians, and averages of averages the best of all. 

The literature of sports ranking systems is full of long pages of carefully derived formula ensuring the best and most accurate math.  But the most sophisticated math in the world is of little use if it does not contribute to improving performance.  Taking the "average of averages" might not make any mathematical sense, but since it performs better than the (mathematically) superior alternatives, we're happy to use it!

The Home Court Advantage

In the previous post, we started looking at RPI and found that it was a considerably better predictor than the 1-Bit Predictor. However, RPI has several obvious shortcomings.  Will fixing these improve its performance as a predictor?  Let's see!

The first area we can look at improving is accounting for Home Court Advantage.  Recall that previously we showed HCA to give the home team about a 4.5 point advantage, or overall a +30% chance of winning.  In 1981, the NCAA added a correction to the RPI formula to account for this advantage.  The correction weights a team's wins and losses differently depending upon where they were played.  A home win is only worth 0.6 "wins", while a road win is worth 1.4 "wins".  Conversely, a home loss costs 1.4 "losses", while a road loss is only 0.6 "losses".

There are a couple of potential problems with this approach.  First, the RPI formula applies this weighting only to the winning percentage calculation of the team being rated.  It is not used in calculating the opponents' winning percentage (OWP) or the opponents' opponents' winning percentage (OOWP).  So the OWP and OOWP are potentially biased by the HCA.  Second, the weighting chosen (0.6/1.4) doesn't appear to reflect the actual home court advantage, which is closer to 30% than 40%. 

Let's see if changing the weightings in the calculation of WP to 0.7/1.3 (closer to the HCA I measured) results in any improvement. Making this change and testing gives this result:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI	73.2%	11.62
RPI (1.7/0.3)	73.4%	11.58

A very slight improvement.  This is not too surprising -- this change only has a small effect on Winning Percentage, which is only 25% of a team's RPI.  Perhaps the NCAA's approach to HCA doesn't have much impact at all?

I should have learned my lesson last time, let's pause a moment to run a test to make sure that the HCA really is a problem.  To do this, we'll run a quick experiment using no weighting (e.g., 1.0/1.0) to see how much improvement this approach to HCA is actually providing.  Performance with no weighting gives these results:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI	73.2%	11.62
RPI (1.7/0.3)	73.4%	11.58
RPI (1.0/1.0)	74.6%	11.53

Surprise!  The NCAA's correction for the home court advantage seems to have actually made the RPI's performance worse.  Hence the value of testing everything -- sometimes intuitively correct notions turn out to be incorrect. Further experimenting with a variety of weightings confirms that the unweighted RPI actually performs better than any weighted variety.

A different approach to accounting for HCA is to adjust game outcomes using the HCA in points.  That is, we'll subtract the HCA (say, 4.5 points) from the home team before determining who "won" the game.  So when Duke wins by 3 at home against North Carolina, that game will count as a win for North Carolina when calculating RPI.  (And we'll carry this through all levels of the RPI calculation to avoid that possible shortcoming.)  This is the equivalent of moving every game to a neutral site (but cheaper).

That change provides this performance:

Predictor	% Correct	MOV Error
1-Bit	62.6%	14.17
RPI (unweighted)	74.6%	11.53
RPI (HCA=4.5)	73.6%	11.61

This is not an improvement over RPI with no weighting.  Experiments with other values for the HCA also do not improve performance (HCA=3.5 does the best, though).  So it appears that the RPI does not benefit from adjustments to eliminate the HCA -- a somewhat surprising result!

My vague intuition about this result is that the HCA is essentially "washed out" of the RPI because the majority of teams play home-and-home series within their conferences.  So any home advantage is gained equally by every team, and any attempt to compensate within the RPI formula just adds error.

We'll return in a bit to an alternative approach for HCA suggested by Dick Vitale.  But since HCA doesn't appear to be a significant problem, first we'll detour into a couple of other possible improvements to RPI.

The Ratings Percentage Index (RPI)

As mentioned previously, one of the simplest and most accessible pieces of information that we can use for prediction is a team's won-lost record.  Naively, we might suppose that when Team A with a 6-2 record plays Team B with a 2-6 record, Team A will likely beat Team B.

But there's (at least) one significant problem with this supposition: we don't have any idea how Team A compiled its winning record, or Team B its losing record.  It could well be that Team A is a Big East team that played 8 patsies at home, while Team B is a mid-major team that has played 8 road games against the best teams in the country, and managed wins against both Duke and Kansas.  In that case we wouldn't be so certain that Team A could beat Team B.

The most widely known rating based on won-loss records is the Ratings Percentage Index (RPI).  RPI tries to address the shortcoming of using won-loss records by rating each team not only by its winning percentage, but also by the winning percentages of its opponents.  The assumption here is that opponents with good won-loss records are tougher opposition than those with poor records, so we should value wins (and losses!) against those opponents more highly.

Of course, you can extend this reasoning another level.  A team's opponents have winning records -- so what?  Again, we don't know if they compiled those records by playing good teams or bad teams.

And, in fact, the RPI addresses this concern by extending the rating another level, so that the RPI for a team is based upon:

1. The team's winning percentage (WP)
2. The team's opponents' winning percentage (OWP), and
3. The team's opponents' opponents' winning percentage (OOWP)

The RPI stops at this level, possibly because the NCAA had run out of the letter 'O'.

Previously we noted the significant impact of the home court advantage (HCA) on college basketball games.  The RPI accounts for this, too, by weighting a teams home wins less than its road wins, and its road losses less than its home losses.  The exact calculation of RPI is complicated, and  I refer the interested reader to the Wikipedia article for a more detailed explanation.  Studying that explanation for several days should lead to total enlightenment -- regarding RPI, anyway.

So how effective is RPI as a predictor?  Using my standard methodology, I get this performance:

Predictor	% Correct	MOV Error
Naive	50%	14.5
1-Bit	62.6%	14.17
RPI	73.2%	11.62

which shows a significant improvement over the 1-Bit Predictor (+11% correct, -2.5 points error).

The following plot shows the RPI characteristics of the test data:

RPI Characteristics (Click to Enlargify)

In this plot, each point represents a game.  The Y axis is the RPI of the home team, and the X axis is the RPI of the away team.  The color of each point indicates the winner of the game -- red for a home win, blue for an away win.  The diagonal line splits the field into games where the home team had the higher RPI (above the line) and games where the away team had the higher RPI (below the line).  While there are more blue points below the line and more red points above the line, the correlation is not overwhelming.

But wait.  I tricked you a bit back in the second paragraph of this posting, when I claimed that the won-loss record of a team isn't a good predictor because it doesn't take into account the quality of opponents.  Is that true?  I've always heard that claim, and it seems reasonable.  But perhaps we should take a minute to check it.  If we use winning percentages as inputs for a predictor, we get the following performance:

Predictor	% Correct	MOV Error
Naive	50%	14.5
1-Bit	62.6%	14.17
WP	72.4%	11.65
RPI	73.2%	11.62

Interesting! RPI is a better predictor than just the winning percentage, but not by a huge margin. 

So RPI provides a signficant improvement in prediction over the 1-Bit Predictor.  But there are several obvious shortcomings in RPI.  Can it be improved?  In the next few postings I'll examine the various shortcomings in RPI and perform various experiments to see if addressing these shortcomings improves performance.  Eventually, we'll also consider other schemes that make use of only won-loss records and see if those provide any significant advantage over RPI.