As mentioned in a previous post, Kaggle is sponsoring a March Madness contest. After some false starts, I managed to figure out the rules, scoring, and submission format. The first phase of the contest is scoring predictions for the past five tournaments. I entered a submission based on the Prediction Machine's point spreads and placed in the Top Ten of the leaderboard.
Some random thoughts about the contest in no particular order.
(1) The "typical" March Madness contest awards full points for predicting a game correctly, and no points for predicting a game incorrectly. Scoring is totally dependent upon the outcome of the game, so scoring for games between closely matched opponents is essentially random. Consequently, winning these contests usually comes down to getting a few late round upset picks correct, a topic I've previously explored.
The Kaggle contest is using an interesting alternative scoring method, the log loss, also called the predictive binomial deviance. Submissions give a likelihood from 0 to 1 for a particular outcome (e.g., Arizona will beat UNC Greensboro). The more certain (closer to 1) the prediction, the higher the reward (penalty) for getting the game right (wrong). For close games, you can predict an outcome around 0.50 and get a small reward if you are right but only a small penalty if you are wrong. This scoring metric does a better job of rewarding contestants who accurately judge the relative strengths of the teams in each game rather than the outcome of that particular game (if that makes sense).
You can also think of this scoring method as a betting strategy. When you place a high likelihood on a particular outcome, it's like betting a lot on the game. When you place an even likelihood on a particular outcome, it's like betting only a small amount on the game. The winner is the contestant who ends up with the most money at the end of the tournament.
(2) The problem with predicting "likelihood" is that there's no way to
measure the actual likelihood. If we made the teams play a 100 games,
we'd get a good approximation of the likelihood, but that's obviously
not reasonable. So there's really two parts to each submission: (a)
assessing the relative strength of the competing teams, and (b)
translating that into a likelihood of victory for one of the teams.
To see that these are two separate problems, imagine that every competitor had to base their entry on the RPI scores
of the teams. Every competitor would have the same relative strength
assessment. But they could translate that into a likelihood any way
they wanted. One competitor might use an exponential model with an
exponent of 15, another an exponential model with an exponent of 22,
another a logistic distribution, etc. The winner in this case would be
whomever happened to pick the best likelihood model for that year's
tournament.
To my mind, it would be a better test of the predictors to have them predict the point spread of each game. Point spread is directly measurable and is the best proxy we have for likelihood, so we'd eliminate that element of how well the competitors translated relative strength to likelihood. But this is probably a minor point -- predicting likelihood with a log loss evaluation is overall a pretty good approach.
(3) So what's the right strategy for this contest? The default strategy is obviously to make your best possible predictions for the games and enter that. But does it ever make sense to intentionally use something other than your best possible prediction?
In a traditionally-scored tournament pool, I believe it does make sense to pick against your best predictions. The reason is that most good predictors are going to have similar outcomes for almost all the games. In that situation, the best possible result for your best predictions might be to end up in a multi-way tie for first place. But in any decent size pool, the most likely result is that you're going to lose to someone who got lucky and picked one or more of the inevitable upsets. So if you want to win the pool, you need to pick upsets yourself, and hope to get lucky.
It isn't clear to me that the same reasoning applies with the log loss scoring method. Since it rewards accurate assessment more than game outcome, it may be that the best strategy is to simply use your best possible predictions.
(4) Phase One of this contest is essentially meaningless. The outcomes of the last five tournaments are known, so it is trivial to craft a "perfect" submission. No one has done that yet, but the top of the leaderboard is already filled with (what appear to be) unrealistic submissions. These submissions are probably "cheating" or are heavily tuned to do well on the Phase One test data.
(5) So what's the best "realistic" score for this contest? By this, I mean the score over a large number of tournament games.
On the point spread side, the best known predictor for college basketball game outcomes are the Vegas closing lines. This isn't an absolute bound on performance, but it's a good starting point. As I pointed out above, converting point spreads to likelihoods isn't straightforward, but with one reasonable approach, the lines have a log loss score of around 0.52 for the past few seasons of regular season games. So I'd be dubious of any approach that does significantly better than that.
(6) That said, it's important to remember that a single NCAA tournament is a very small set of data. It's perfectly reasonable to expect an approach that would be terrible on average over a large number of tournaments to do very well on any particular tournament (or vice versa). For example, my entry to Phase One had a score of about .54. When I look at how that entry scored on each individual season, I see that in some seasons it scored around .51. So the winner of Phase Two could easily be someone who just happened to get lucky with a good score this year.
It wouldn't be an entirely unreasonable approach to build a model to assess team strengths, an algorithm for translating that to likelihoods and then tune that to do particularly well on some past tournament (say, 2010). That's probably not the best general approach, but it might get lucky and do very well this year.
Showing posts with label march madness. Show all posts
Showing posts with label march madness. Show all posts
Monday, January 20, 2014
Monday, March 25, 2013
Upset Picks Review
I previously posted the Prediction Machine's top ten upset picks for the first round. How did it do? In the table below I've added a column for MOV -- a negative number indicating that the underdog (Away Team) won the game.
According to the PM, the Notre Dame-Iowa St. game had significantly higher upset chances than any other game, and in fact Iowa State won the game handily. The PM also liked Oklahoma to upset San Diego State, but State won that game handily.
The next tier of upset possibilities (> 2) was less likely, but the PM also went 50% in this tier. (Actually 75%. Although I marked it as a missed upset here, after the play-in game, the PM had St. Mary's as an outright favorite in the game against Memphis, so the Upset probability was for actually for Memphis to win! The Upset metric remained almost the same, by the way.)
The next tier is below the cutoff for consideration as an upset, although both the Wichita State and Cal upsets were identified here. (And Cincinatti, Missouri and Bucknell were all popular upset picks by pundits.)
Combined with last year's results, it appears that the PM's algorithm for detecting likely upsets works fairly well. (Note that whether the PM should include the upset in its bracket is a different question!)
| Home | Away | Upset | MOV |
|---|---|---|---|
| Notre Dame | Iowa St. | 17.4 | -18 |
| San Diego St. | Oklahoma | 7.1 | 15 |
| Memphis | St. Mary's | 2.5 | 2 |
| Oklahoma St. | Oregon | 2.4 | -13 |
| N.C. State | Temple | 2.2 | -4 |
| Illinois | Colorado | 2.1 | 8 |
| Colorado St. | Missouri | 1.8 | 12 |
| Creighton | Cincinnati | 1.6 | 4 |
| Pittsburgh | Wichita St. | 1.6 | -18 |
| UNLV | California | 1.5 | -3 |
| Butler | Bucknell | 1.5 | 12 |
| North Carolina | Villanova | 1.2 | 7 |
According to the PM, the Notre Dame-Iowa St. game had significantly higher upset chances than any other game, and in fact Iowa State won the game handily. The PM also liked Oklahoma to upset San Diego State, but State won that game handily.
The next tier of upset possibilities (> 2) was less likely, but the PM also went 50% in this tier. (Actually 75%. Although I marked it as a missed upset here, after the play-in game, the PM had St. Mary's as an outright favorite in the game against Memphis, so the Upset probability was for actually for Memphis to win! The Upset metric remained almost the same, by the way.)
The next tier is below the cutoff for consideration as an upset, although both the Wichita State and Cal upsets were identified here. (And Cincinatti, Missouri and Bucknell were all popular upset picks by pundits.)
Combined with last year's results, it appears that the PM's algorithm for detecting likely upsets works fairly well. (Note that whether the PM should include the upset in its bracket is a different question!)
Monday, March 11, 2013
Machine Madness?
In recent years, Danny Tarlow and Lee-Ming Zen have run the "Machine Madness" contest for computer predictors. They've gotten a late start this year and I'd forgotten to remind them, but we can probably still pull something together if there is interest.
If you are interested in competing, check out the Machine March Madness Google Group and indicate your interest.
I've posted an initial data dump for the season here. It's in the form of a Google Spreadsheet, but you can download it as Excel or a .csv file. This should be the team-level roll-up of the box score, e.g., http://scores.espn.go.com/ncb/
If you are interested in competing, check out the Machine March Madness Google Group and indicate your interest.
I've posted an initial data dump for the season here. It's in the form of a Google Spreadsheet, but you can download it as Excel or a .csv file. This should be the team-level roll-up of the box score, e.g., http://scores.espn.go.com/ncb/
Thursday, November 8, 2012
How to Pick a Tournament Bracket, Part 2
In the previous post, I looked at a strategy for picking a Tournament bracket. The basic idea is that to win a sizable Tournament challenge, you can't just pick the most likely outcome of each game. You're going to have to pick at least some of the inevitable upsets correctly. A reasonable way to do that is to decide how many points from upsets you think you'll need, and then pick some combination of upsets to reach that number. It turns out the best way to do that is to pick late-round upsets between closely-matched teams.
However, there are some concerns with that approach. One is that if you pick "likely" upsets (such as a #2 over a #1), it's reasonable to assume that many of your competitors might pick the same upset. So although the upset might be both likely and high-scoring, it might not do much to separate you from your competitors. That's an interesting problem, but one we'll leave for another day.
Another concern is that the strategy is "all or nothing." We are assuming that we'll need (say) G = 16 points to win the Tournament challenge and make picks accordingly. But in truth our chance of winning the Tournament challenge is more of an S-curve:
We have some guess at G that will give us a reasonable chance to win the challenge, but we might end up needing more or we might be able to win with less. With G = 16 the strategy I've outlined so far leads us to pick a single 16 point upset in a semi-final game. This is fine if the upset occurs. But if it doesn't, losing 16 points moves us a sizable distance to the left on the S-curve and greatly reduces our chances of winning. Something of this sort happened to my entry (the Pain Machine) in the last Machine Madness contest -- the PM predicted a Kansas-Kentucky upset that would have left it at G = 34. But since Kentucky won, the PM ended up at G = 2 and lost the contest to a predictor at G = 7. We'd really like a strategy that optimizes our chance to win under all possible scenarios.
Mathematically, this is the sum over all possible outcomes of the likelihood of the outcome times the likelihood of winning under that outcome. If we pick a single 16 point upset, then there are two possible outcomes: the upset happens or it doesn't. If L(n) is the likelihood of winning the challenge with n points, then expected value of that strategy is:
But if instead we picked two 8 point upsets, then there are four possible outcomes: neither of the upsets occur, the first upset occurs but the second doesn't, the first doesn't but the second does, or they both occur. The expected value of this strategy is more complicated:
Up until now, I've been implicitly assuming that the possible outcomes of an upset pick are either zero or n points. But that's not really true. The cost of an incorrect pick can be greater than just losing the points for that game.
For example, last year the Pain Machine correctly predicted that #15 Lehigh would beat #2 Duke. Given the rarity of 15-2 upsets, that was an amazing prediction. But even if it was a very likely upset, it would have been a bad pick, because there was a potentially high cost if the upset didn't happen. To see why, here is the bracket:
If
the prediction is correct, the Pain Machine picks up 1 point for the
correct first round prediction. But if the prediction is incorrect, the
Pain Machine is very likely to lose 2 points when Duke wins in round, 4 more points when they win in the third round, and so on. (Since as a #2 seed, we expect them to win until the round of eight.)
We can generalize this idea as a value formula for a win by U over V:
V(u,v) = (p(u,v) * roundi ) - (p(v,i+1) * roundi+1 ) - (p(v,i+1)*p(v,i+2)* roundi+2 ) ...
Here, p(u,v) represents the probability that U defeats V, roundi represents the scoring value for the Ith round of the tournament, and p(v,i+1) represents the probability of V defeating their likely opponent in round i+1 if they had not been upset by U. To return to the Lehigh-Duke example, the value is the probability of Lehigh beating Duke times 1 (the value of that round) minus the probability of Duke beating Notre Dame (their expected opponent) times 2 (the value of that round), and so on.
To maximize V(u,v) we must maximize p(u,v) and minimize p(v,i+1). And since roundi+1 = 2*roundi, it is twice as important to minimize p(v,i+1). To translate this into plain English, we want to pick upsets where the team being upset has very little chance to win its next round game. That's why the Lehigh upset was a poor pick -- because as a #2 seed Duke had a very good chance to win its second round game.
Instead, this formula will value upset picks like #10 Xavier over #7 Notre Dame. (Which also happened!) To see why, look again at the bracket:
Whichever
team wins the first round game -- Notre Dame or Xavier -- they are
likely to lose the second round game to the stronger Duke team. Thus the downside of the upset
pick is minimized -- if Notre Dame wins and then loses to Duke as expected, you'll only have lost one point for the incorrect upset pick.
This insight is nothing new. Canny pickers already look for upsets that are "firewalled" off in the next round by a strong opponent. However, the value formula above gives us an objective measure for comparing between possible upset picks. I suspect that most people incorrectly assess the p(u,v) vs. p(v,?) tradeoff. Because scoring doubles each round, it's much more important to consider the cutoff in the next round than the upset chance -- which most people probably find counter-intuitive.
Unfortunately, the strategy of "firewalling" upset picks runs counter to the strategy of picking high-scoring late round upsets, because (assuming most games are not upsets) the mismatches which make good firewalls primarily occur in the early rounds. If the tournament runs mostly true to the seedings, the late round games are usually between closely-matched teams and do not make good firewalls. An interesting exception is the Championship game itself. If you pick an upset in the Championship game incorrectly, you're guaranteed not to lose any additional points.
To summarize these thoughts about picking a tournament bracket:
However, there are some concerns with that approach. One is that if you pick "likely" upsets (such as a #2 over a #1), it's reasonable to assume that many of your competitors might pick the same upset. So although the upset might be both likely and high-scoring, it might not do much to separate you from your competitors. That's an interesting problem, but one we'll leave for another day.
Another concern is that the strategy is "all or nothing." We are assuming that we'll need (say) G = 16 points to win the Tournament challenge and make picks accordingly. But in truth our chance of winning the Tournament challenge is more of an S-curve:
Mathematically, this is the sum over all possible outcomes of the likelihood of the outcome times the likelihood of winning under that outcome. If we pick a single 16 point upset, then there are two possible outcomes: the upset happens or it doesn't. If L(n) is the likelihood of winning the challenge with n points, then expected value of that strategy is:
EV = L(0) * (1 - p(u,v)) + L(16) * p(u,v)
But if instead we picked two 8 point upsets, then there are four possible outcomes: neither of the upsets occur, the first upset occurs but the second doesn't, the first doesn't but the second does, or they both occur. The expected value of this strategy is more complicated:
EV = L(0) * (1 - p(u1,v1)) * (1 - p(u2,v2)) +Depending upon the probability of the various outcomes and the likelihood of winning, the expected value of this strategy might be higher than picking a single 16 point upset, even though the chances of scoring the full 16 points are reduced.
L(8) * p(u1,v1) * (1 - p(u2,v2)) +
L(8) * (1 - p(u1,v1)) * p(u2,v2) +
L(16) * p(u1,v1) * p(u2,v2)
Up until now, I've been implicitly assuming that the possible outcomes of an upset pick are either zero or n points. But that's not really true. The cost of an incorrect pick can be greater than just losing the points for that game.
For example, last year the Pain Machine correctly predicted that #15 Lehigh would beat #2 Duke. Given the rarity of 15-2 upsets, that was an amazing prediction. But even if it was a very likely upset, it would have been a bad pick, because there was a potentially high cost if the upset didn't happen. To see why, here is the bracket:
We can generalize this idea as a value formula for a win by U over V:
V(u,v) = (p(u,v) * roundi ) - (p(v,i+1) * roundi+1 ) - (p(v,i+1)*p(v,i+2)* roundi+2 ) ...
Here, p(u,v) represents the probability that U defeats V, roundi represents the scoring value for the Ith round of the tournament, and p(v,i+1) represents the probability of V defeating their likely opponent in round i+1 if they had not been upset by U. To return to the Lehigh-Duke example, the value is the probability of Lehigh beating Duke times 1 (the value of that round) minus the probability of Duke beating Notre Dame (their expected opponent) times 2 (the value of that round), and so on.
To maximize V(u,v) we must maximize p(u,v) and minimize p(v,i+1). And since roundi+1 = 2*roundi, it is twice as important to minimize p(v,i+1). To translate this into plain English, we want to pick upsets where the team being upset has very little chance to win its next round game. That's why the Lehigh upset was a poor pick -- because as a #2 seed Duke had a very good chance to win its second round game.
Instead, this formula will value upset picks like #10 Xavier over #7 Notre Dame. (Which also happened!) To see why, look again at the bracket:
Whichever
team wins the first round game -- Notre Dame or Xavier -- they are
likely to lose the second round game to the stronger Duke team. Thus the downside of the upset
pick is minimized -- if Notre Dame wins and then loses to Duke as expected, you'll only have lost one point for the incorrect upset pick.This insight is nothing new. Canny pickers already look for upsets that are "firewalled" off in the next round by a strong opponent. However, the value formula above gives us an objective measure for comparing between possible upset picks. I suspect that most people incorrectly assess the p(u,v) vs. p(v,?) tradeoff. Because scoring doubles each round, it's much more important to consider the cutoff in the next round than the upset chance -- which most people probably find counter-intuitive.
Unfortunately, the strategy of "firewalling" upset picks runs counter to the strategy of picking high-scoring late round upsets, because (assuming most games are not upsets) the mismatches which make good firewalls primarily occur in the early rounds. If the tournament runs mostly true to the seedings, the late round games are usually between closely-matched teams and do not make good firewalls. An interesting exception is the Championship game itself. If you pick an upset in the Championship game incorrectly, you're guaranteed not to lose any additional points.
To summarize these thoughts about picking a tournament bracket:
- A bracket consisting of chalk picks and true mis-seedings is not likely to win a sizable Tournament challenge.
- Picking late-round upsets between highly-seeded teams has the advantages of (1) scoring a lot of points, and (2) being relatively likely to occur.
- To maximize the overall chance of winning the challenge, it may be better to spread your upset picks rather than bet "all or nothing."
- Upset picks which are firewalled in the next round reduce the downside risk of an incorrect pick.
Wednesday, November 7, 2012
How To Pick A Tournament Bracket, Part 1
Pre-season is probably not the best time to be pondering the Tournament, but I've been recently thinking a bit more about the challenge of predicting the Tournament with the goal of winning something like the ESPN Tournament Challenge or the Machine Madness contest. These sorts of contests are a dilemma to a machine predictor, because most predictors try to determine who is most likely to win a particular matchup. But of course, that's exactly how the Tournament is seeded. So the machine predictors end up predicting almost entirely "chalk" outcomes.
The only time the machines don't predict a win for the higher seed is when they believe the teams have been mis-seeded -- that is, when the Committee has made a mistake in their assessment of the relative strengths of the teams. In last year's Machine Madness contest, Texas over Cincy and Purdue over St. Mary's were consensus upset picks by five of the six predictors -- strong evidence (to my mind, anyway) that those teams were mis-seeded. But, for all the grumbling by fans, the Committee does a pretty good job at seeding the Tournament, and you can't expect to find many true mis-seedings.
Neither chalk picks or mis-seedings are likely to win a Tournament challenge against a sizable field. That's because (1) a lot of your competitors will have made the same picks, (2) there will be a significant number of true upsets where a weaker team beats a stronger team (historically, 22% in the first round, and 15% for the Tournament overall), and (3) someone out there will have picked those upsets. So to win a Tournament challenge, the machine is going to have to pick some actual upsets -- and then hope that it gets lucky and those upsets are the ones that happen.
Knowing the historical frequency of upsets, my strategy last year was to force my predictor to pick 6 upsets in the first round and 5 more in the rest of the tournament. But is that the right way to pick upsets? How can we pick upsets to maximize (in some sense) our chance to win the Tournament challenge?
The first problem in answering this question is knowing how many points will be sufficient to win the challenge, because that will drive the selection of upsets. Obviously, it's impossible to know this number a priori. However, we could look at previous Tournament challenges and see how many points the competitors in the top (say) 1% had scored off correctly predicting upsets. That would provide a reasonable goal G for our upset calculations.
Sadly, ESPN, Yahoo, etc., seem to remove the Tournament challenge information from the Internets fairly quickly, so I can't actually research this. (If someone has some info on this, please let me know!) However, we do have the results of the last two Machine Madness contests. Last year, the winning entry scored 127 points and the "chalk" (baseline) entry scored 120 points, for G = 7. The year before, the winning entry scored 69 points and the chalk entry scored 57 points, for G = 12. (There's undoubtedly a correlation between the size of the field and G. G = 12 might be sufficient most years to win the Machine Madness contest, but probably wouldn't be enough to win the ESPN Tournament Challenge.)
If we adopt the notation that V(u,v) is the value of a victory of Team U over Team V, then we will want to pick upsets such that:
For example, a #8 seed beating a #1 seed is worth 2 points, because that matchup will necessarily occur in the second round.
If we adopt the notation that p(u1,v1) is the probability of u1 defeating v1, then the probability of G is:
Now imagine that we are predicting the tournament and we know that most games have a 0% chance of an upset. However, four of the third round games are very likely upsets -- 49%. And one of the semi-final games has a slight chance of an upset -- about 6%. If G = 16, which upsets should we pick?
The (possibly surprising) answer is that we should pick the very unlikely semi-final upset! To get 16 points we have to pick either the semi-final game, or all four of the third round games, and:
.06 > .49*.49*.49*.49
The joint probability equation combined with the typical Tournament challenge scoring means that it will almost always be better to pick unlikely late round upsets that score highly than multiple likely early round upsets that score poorly. Knowing this, it's easy to see that my strategy in previous years to force a certain number of upsets into my bracket was very non-optimal.
So, given a goal G and upset probabilities p(u,v) we have an approach for selecting upsets from our bracket. We've seen how to calculate V(u,v) and how to estimate G. How can we estimate the upset probabilities?
Many predictors will produce something that can be used to estimate upset probabilities. For example, in past years my predictor has used the predicted MOV to estimate upset probabilities -- the slimmer the predicted margin of victory, the more likely an upset. But lacking any information of that sort, we could estimate the upset probabilities based upon historical performance of seeds within the tournament:
This table shows the upset percentage for each seeding matchup for the last ten years. (I have left out matchups that have occurred 4 times or less.) Each upset percentage is shaded to indicate the value of the matchup in typical Tournament challenge scoring. For example, matchups between #1 seeds and #2 seeds have been won by the #2 seeds 52% of the time, and they are worth 8 points. With a few oddball exceptions (such as the 2-10 matchups), this table shows is that you should prefer to pick upsets of the #1 or #2 seeds by #2 or #3 seeds. These matchups are worth the most points and -- because the teams are closely seeded -- are nearly tossups.
So if G = 16, filling out your bracket with all chalk picks and two #2 over #1 upsets would give you the best chance to win the Tournament challenge.
More thoughts to follow in Part 2 at some later date.
The only time the machines don't predict a win for the higher seed is when they believe the teams have been mis-seeded -- that is, when the Committee has made a mistake in their assessment of the relative strengths of the teams. In last year's Machine Madness contest, Texas over Cincy and Purdue over St. Mary's were consensus upset picks by five of the six predictors -- strong evidence (to my mind, anyway) that those teams were mis-seeded. But, for all the grumbling by fans, the Committee does a pretty good job at seeding the Tournament, and you can't expect to find many true mis-seedings.
Neither chalk picks or mis-seedings are likely to win a Tournament challenge against a sizable field. That's because (1) a lot of your competitors will have made the same picks, (2) there will be a significant number of true upsets where a weaker team beats a stronger team (historically, 22% in the first round, and 15% for the Tournament overall), and (3) someone out there will have picked those upsets. So to win a Tournament challenge, the machine is going to have to pick some actual upsets -- and then hope that it gets lucky and those upsets are the ones that happen.
Knowing the historical frequency of upsets, my strategy last year was to force my predictor to pick 6 upsets in the first round and 5 more in the rest of the tournament. But is that the right way to pick upsets? How can we pick upsets to maximize (in some sense) our chance to win the Tournament challenge?
The first problem in answering this question is knowing how many points will be sufficient to win the challenge, because that will drive the selection of upsets. Obviously, it's impossible to know this number a priori. However, we could look at previous Tournament challenges and see how many points the competitors in the top (say) 1% had scored off correctly predicting upsets. That would provide a reasonable goal G for our upset calculations.
Sadly, ESPN, Yahoo, etc., seem to remove the Tournament challenge information from the Internets fairly quickly, so I can't actually research this. (If someone has some info on this, please let me know!) However, we do have the results of the last two Machine Madness contests. Last year, the winning entry scored 127 points and the "chalk" (baseline) entry scored 120 points, for G = 7. The year before, the winning entry scored 69 points and the chalk entry scored 57 points, for G = 12. (There's undoubtedly a correlation between the size of the field and G. G = 12 might be sufficient most years to win the Machine Madness contest, but probably wouldn't be enough to win the ESPN Tournament Challenge.)
If we adopt the notation that V(u,v) is the value of a victory of Team U over Team V, then we will want to pick upsets such that:
G < V(u1,v1) + V(u2,v2) + V(u3,v3) ...Because of the way the tournament is structured, the value of V(u,v) is determined by the seeding of the two teams. The following table has seedings down both axises and shows how many points an upset is worth:
For example, a #8 seed beating a #1 seed is worth 2 points, because that matchup will necessarily occur in the second round.
If we adopt the notation that p(u1,v1) is the probability of u1 defeating v1, then the probability of G is:
p(u1,v1) * p(u2,v2) * p(u3,v3) ...(because we must get all of our upset picks correct to score G points).
Now imagine that we are predicting the tournament and we know that most games have a 0% chance of an upset. However, four of the third round games are very likely upsets -- 49%. And one of the semi-final games has a slight chance of an upset -- about 6%. If G = 16, which upsets should we pick?
The (possibly surprising) answer is that we should pick the very unlikely semi-final upset! To get 16 points we have to pick either the semi-final game, or all four of the third round games, and:
.06 > .49*.49*.49*.49
The joint probability equation combined with the typical Tournament challenge scoring means that it will almost always be better to pick unlikely late round upsets that score highly than multiple likely early round upsets that score poorly. Knowing this, it's easy to see that my strategy in previous years to force a certain number of upsets into my bracket was very non-optimal.
So, given a goal G and upset probabilities p(u,v) we have an approach for selecting upsets from our bracket. We've seen how to calculate V(u,v) and how to estimate G. How can we estimate the upset probabilities?
Many predictors will produce something that can be used to estimate upset probabilities. For example, in past years my predictor has used the predicted MOV to estimate upset probabilities -- the slimmer the predicted margin of victory, the more likely an upset. But lacking any information of that sort, we could estimate the upset probabilities based upon historical performance of seeds within the tournament:
This table shows the upset percentage for each seeding matchup for the last ten years. (I have left out matchups that have occurred 4 times or less.) Each upset percentage is shaded to indicate the value of the matchup in typical Tournament challenge scoring. For example, matchups between #1 seeds and #2 seeds have been won by the #2 seeds 52% of the time, and they are worth 8 points. With a few oddball exceptions (such as the 2-10 matchups), this table shows is that you should prefer to pick upsets of the #1 or #2 seeds by #2 or #3 seeds. These matchups are worth the most points and -- because the teams are closely seeded -- are nearly tossups.
So if G = 16, filling out your bracket with all chalk picks and two #2 over #1 upsets would give you the best chance to win the Tournament challenge.
More thoughts to follow in Part 2 at some later date.
Wednesday, March 21, 2012
Upset Review
For the past three years that the Pain Machine has participated in the Machine Madness contest, I've maintained (without any real justification) that the proper strategy is to pick the correct upsets -- as opposed to simply picking the most likely outcome, which will be the higher seed in every case where the committee hasn't completely blown the seeding. In light of that, I wanted to review the PM's upset-picking strategy and see how it has worked out this year.
The PM predicts the Margin of Victory for each tournament game. With two exceptions this year, the predicted winner was the higher-seeded team. Historically, we know that the upset rate in the first round has been around 22%, and the upset rate for the whole tournament around 15%. (An upset is where a team seeded at least 2 lower than its opponent wins the game. A #9 over a #8 is not considered an upset.) In light of this, I force the PM's tournament picks to include 6 upsets in the first round and 5 more in the rest of the tournament.
The picking strategy is fairly straightforward. First of all, any games where the PM thinks an upset will happen are marked as upsets. After that, the PM marks the remaining of 6 games in the first round which have the lowest predicted MOVs as upsets and (after recalculating the rest of the bracket based upon those upsets) the remainder of 5 games in the rest of the bracket by the same criterion.
This year, that resulted in these upset picks (predicted MOV shown in parentheses, correct picks bolded) for the first round:
(11) Texas over (6) Cincinnati (-0.6)
(12) Cal/USF over (5) Temple (1.4)
(11) NC State over (6) SDSU (1.9)
(10) Purdue over (7) St. Mary's (3.3)
(10) WVU over (7) Gonzaga (3.3)
(9) UConn over (5) Iowa St. (3.6)
The PM picked 3 of these 6 upsets correctly: USF, NC State and Purdue. Texas shot just 16% in the first half and still managed to tie the game in the second half but couldn't finish the rally. The other two games were not very close. Still, getting 50% correct on upsets is probably pretty good performance.
The PM has the following upsets picked in later rounds:
(2) OSU over (1) Syracuse (-0.8)
(2) Kansas over (1) Kentucky (0.6)
(11) Texas over (3) FSU (2)
(5) New Mexico over (4) Louisville (2.7)
(6) Baylor over (2) Duke (3)
The FSU and Duke upsets cannot happen. The New Mexico upset did not happen. The other two games have not yet occurred.
We can also look at the (say) the most likely upsets at each seed position. These were:
(16) UNC-Asheville vs. (1) Syracuse (16.1)
(15) Lehigh vs. (2) Duke (12.8)
(14) Belmont vs. (3) Georgetown (6.3)
(13) Ohio vs. (4) Michigan (7.9)
(12) Cal/USF over (5) Temple (1.4)
(11) Texas over (6) Cincinnati (-0.6)
(10) Purdue over (7) St. Mary's (3.3)
(9) UConn over (8) Iowa State (3.6)
Again, the PM got 50% correct.
Of course, the PM also missed a number of upsets:
(12) VCU over (5) Wichita St. (9.6)
(10) Xavier over (7) Notre Dame (7.7)
(15) Norfolk St. over (2) Missouri (23.2)
(11) NC State over (3) Georgetown (5.4)
The Norfolk State win really stands out here as the outlier -- it was at least twice as unlikely as the Duke-Lehigh upset. I don't have the statistic handy, but 23 point upsets have to be greater 1 in a 1000 historically. (The beating Norfolk St. took in the next round is indicative of how anomalous the first round upset was.) VCU was a darling upset pick for many, in part due to their Cinderella status last year. This year's VCU team was considerably weaker, and the win over Wichita State was another very unlikely result. The Georgetown upset was the least surprising. The 5 point differential is well within the ~10 point error margin of the PM's predictions.
Overall, I give the PM a very positive grade for it's upset picks. It's clearly able to identify games where upsets are likely. I may have to work on how it selects upsets, though. There isn't a strong correlation between the magnitude of MOV and the likelihood of upset when MOV is under about 6 points, so it may not make sense to pick the games with the lowest MOVs. It may make more sense to pick upsets based upon other factors.
The PM predicts the Margin of Victory for each tournament game. With two exceptions this year, the predicted winner was the higher-seeded team. Historically, we know that the upset rate in the first round has been around 22%, and the upset rate for the whole tournament around 15%. (An upset is where a team seeded at least 2 lower than its opponent wins the game. A #9 over a #8 is not considered an upset.) In light of this, I force the PM's tournament picks to include 6 upsets in the first round and 5 more in the rest of the tournament.
The picking strategy is fairly straightforward. First of all, any games where the PM thinks an upset will happen are marked as upsets. After that, the PM marks the remaining of 6 games in the first round which have the lowest predicted MOVs as upsets and (after recalculating the rest of the bracket based upon those upsets) the remainder of 5 games in the rest of the bracket by the same criterion.
This year, that resulted in these upset picks (predicted MOV shown in parentheses, correct picks bolded) for the first round:
(11) Texas over (6) Cincinnati (-0.6)
(12) Cal/USF over (5) Temple (1.4)
(11) NC State over (6) SDSU (1.9)
(10) Purdue over (7) St. Mary's (3.3)
(10) WVU over (7) Gonzaga (3.3)
(9) UConn over (5) Iowa St. (3.6)
The PM picked 3 of these 6 upsets correctly: USF, NC State and Purdue. Texas shot just 16% in the first half and still managed to tie the game in the second half but couldn't finish the rally. The other two games were not very close. Still, getting 50% correct on upsets is probably pretty good performance.
The PM has the following upsets picked in later rounds:
(2) OSU over (1) Syracuse (-0.8)
(2) Kansas over (1) Kentucky (0.6)
(11) Texas over (3) FSU (2)
(5) New Mexico over (4) Louisville (2.7)
(6) Baylor over (2) Duke (3)
The FSU and Duke upsets cannot happen. The New Mexico upset did not happen. The other two games have not yet occurred.
We can also look at the (say) the most likely upsets at each seed position. These were:
(16) UNC-Asheville vs. (1) Syracuse (16.1)
(15) Lehigh vs. (2) Duke (12.8)
(14) Belmont vs. (3) Georgetown (6.3)
(13) Ohio vs. (4) Michigan (7.9)
(12) Cal/USF over (5) Temple (1.4)
(11) Texas over (6) Cincinnati (-0.6)
(10) Purdue over (7) St. Mary's (3.3)
(9) UConn over (8) Iowa State (3.6)
Again, the PM got 50% correct.
Of course, the PM also missed a number of upsets:
(12) VCU over (5) Wichita St. (9.6)
(10) Xavier over (7) Notre Dame (7.7)
(15) Norfolk St. over (2) Missouri (23.2)
(11) NC State over (3) Georgetown (5.4)
The Norfolk State win really stands out here as the outlier -- it was at least twice as unlikely as the Duke-Lehigh upset. I don't have the statistic handy, but 23 point upsets have to be greater 1 in a 1000 historically. (The beating Norfolk St. took in the next round is indicative of how anomalous the first round upset was.) VCU was a darling upset pick for many, in part due to their Cinderella status last year. This year's VCU team was considerably weaker, and the win over Wichita State was another very unlikely result. The Georgetown upset was the least surprising. The 5 point differential is well within the ~10 point error margin of the PM's predictions.
Overall, I give the PM a very positive grade for it's upset picks. It's clearly able to identify games where upsets are likely. I may have to work on how it selects upsets, though. There isn't a strong correlation between the magnitude of MOV and the likelihood of upset when MOV is under about 6 points, so it may not make sense to pick the games with the lowest MOVs. It may make more sense to pick upsets based upon other factors.
Friday, February 24, 2012
March Madness Contest
Danny Tarlow and Lee-Ming Zen over at This Number Crunching Life have announced their annual March Madness prediction contest. To compete, you use data from this season and past seasons (which Danny & Lee will provide), build a computer system that fills out a bracket, then pit yourself against the field of silicon competition. The posts from last season's tournament can be found here.
I personally know the winner from last year and the previous year, and I can only say that I have the utmost respect for their dedication, intelligence, and ruggedly handsome good looks.
I personally know the winner from last year and the previous year, and I can only say that I have the utmost respect for their dedication, intelligence, and ruggedly handsome good looks.
Subscribe to:
Posts (Atom)