While I like to focus on games where my predictor has been very accurate, it's sometimes entertaining to look at the games where it did the worst. And there are plenty of those. In fact, there were more than 100 games this year where my prediction was off by over 25 points.
The worst of the lot this year was the January 14 game between Nevada and Colorado State, which I predicted as a 15 point win for Colorado State. In fact, they won by 56 -- an error of over 40 points. Looking at the game, it isn't hard to see what went wrong. Nevada shot just 16% in the first half and was down 55-12 at halftime. Colorado State eased up in the second half or the damage might have been worse. As it was, the 56 point victory is the largest in Colorado State history.
But my worst prediction of all time was off by 61, so missing by "only" 40 is not so bad.
A more important game this year was the December 22nd matchup between Kansas and Temple. Kansas came into the game ranked #10 in the nation and was not expected to have much trouble with Temple, who was 8-4 and had already lost to the likes of UNLV and St. Joe's. I predicted a 10 point win by Kansas, and Vegas had them as 8 point favorites (climbing a point from the open). Instead, the Temple guards went crazy, pouring in 54 point en route to a 25 point victory -- an error of 35 points in my prediction. For the game, Temple shot 58% to Kansas's 32%.
Another terrible prediction occurred the night before the Kansas game when Harvard played Virginia. I had this as a 15 point Virginia win. Vegas favored Virginia by 10. Harvard was a good team -- they finished the season 22-8 after barely losing to UNC in the Tournament -- but Tony Bennett's team put on a defensive clinic for the ages. They held Harvard to a single basket in the first half and 8 of 50 (16%!) shooting for the night. Virginia won by 49 -- a 34 point error for me, but at least I was closer than Vegas.
These sorts of games demonstrate that predicting college basketball is really difficult. There's just a lot of variance and random factors that go into the final score, and you can't get too worked up over the occasional terrible result!
Friday, April 10, 2015
Tuesday, April 7, 2015
Year End Summary
Congratulations to the Duke Blue Devils on winning their 5th Championship and bringing the college basketball season to a great finish. Here's the season summary for Net Prophet:
For the first time, I submitted predictions to The Prediction Tracker. For the year, NetProphet ended up being the best predictor Against The Spread (ATS), and in the top two or three in a number of other categories. (Oddly enough, NetProphet is not very good predicting the winner straight up. Doktor Entropy is notable for doing very well both ATS and straight up.) Apparently some others took note that I was submitting picks to the Prediction Tracker, because when I didn't submit for one reason or another I got emails asking where they were!
And though I've said it before, many thanks to Todd for running The Prediction Tracker! It's a great service and a lot of work.
For The Prediction Tracker I submitted picks on all college games. I also privately identify "high confidence" picks that I think should perform well against the spread. This year those picks went 58.2% ATS, which is a very solid performance.
With the help of Monte McNair, I ran the 6th Annual Machine Madness competition. There were fourteen competitors, and the competition was won by BlueFool (aka Dr. Amanda Schierz) in her first year competing.
I don't put much effort into predicting the Tournament, but I did compete in both the Machine Madness competition and the Kaggle competition. In my Machine Madness entry, NetProphet identified an upset of Kentucky as the best value in pool play and selected Arizona over Kentucky as the most likely upset. The strategy was sound -- NetProphet was the only competitor with Arizona winning it all -- but as it turns out, Arizona couldn't even beat Wisconsin, much less Kentucky, so that did not pay off.
On the Kaggle side, my entry didn't do well the first day but climbed steadily into the 20s by the Championship game. The Duke win hurt me, however, and I fell back to 42nd. That's somewhat better than I did in 2014 (60th) and my log-loss score was also considerably lower (but that probably reflects a less upset-filled Tournament than any real improvement in the predictor).
Possibly more interesting is that I posted my entry to the Kaggle forum before the competition and invited other competitors to steal it. There were at least two interesting results from this. The aforementioned BlueFool submitted an entry that was an average of her entry and my entry. Her entry was 24th, mine was 42nd, but the average of them finished in 8th (!). That suggests that our models had cancelling independent errors. Second, Juho Kokkala submitted entries based upon my entry but with Kentucky's probabilities turned up to 1.0. Had Kentucky won out, Juho would have probably placed in the top two and collected some prize money.
The results of the Kaggle competition have me re-thinking some of my beliefs about predicting the Tournament, but I'll post more about that anon.
In terms of functional improvements to the predictor, I took a look this year at a number of different approaches, most of which I wrote up at some point or another. Some refactoring of my code was a significant improvement in terms of enabling me to try out new ideas more quickly. I took a look at boosting and other machine learning approaches to improving prediction accuracy, but I found that they did not result in any significant improvement over my base predictor. I also took a look at various approaches to creating statistics that are adjusted for schedule, and I adopted or tried out a number of different ideas from the literature or elsewhere. Unfortunately, the articles from the previous Kaggle competition didn't have any truly original ideas to steal.
On a bit of separate note, I had the chance this year to exchange ideas with Erik Forseth (who went on to create a predictor that seems to be every bit as good as my own), Mark Bashuk (SevenOvertimes.com), Scott Burrola, and several others. It's always interesting to exchange thoughts with other folks and see what insights and ideas they have. Feel free to send me an email -- the emails I get always start "Sorry to bother you..." but I'm actually always happy to discuss!
For the first time, I submitted predictions to The Prediction Tracker. For the year, NetProphet ended up being the best predictor Against The Spread (ATS), and in the top two or three in a number of other categories. (Oddly enough, NetProphet is not very good predicting the winner straight up. Doktor Entropy is notable for doing very well both ATS and straight up.) Apparently some others took note that I was submitting picks to the Prediction Tracker, because when I didn't submit for one reason or another I got emails asking where they were!
And though I've said it before, many thanks to Todd for running The Prediction Tracker! It's a great service and a lot of work.
For The Prediction Tracker I submitted picks on all college games. I also privately identify "high confidence" picks that I think should perform well against the spread. This year those picks went 58.2% ATS, which is a very solid performance.
With the help of Monte McNair, I ran the 6th Annual Machine Madness competition. There were fourteen competitors, and the competition was won by BlueFool (aka Dr. Amanda Schierz) in her first year competing.
I don't put much effort into predicting the Tournament, but I did compete in both the Machine Madness competition and the Kaggle competition. In my Machine Madness entry, NetProphet identified an upset of Kentucky as the best value in pool play and selected Arizona over Kentucky as the most likely upset. The strategy was sound -- NetProphet was the only competitor with Arizona winning it all -- but as it turns out, Arizona couldn't even beat Wisconsin, much less Kentucky, so that did not pay off.
On the Kaggle side, my entry didn't do well the first day but climbed steadily into the 20s by the Championship game. The Duke win hurt me, however, and I fell back to 42nd. That's somewhat better than I did in 2014 (60th) and my log-loss score was also considerably lower (but that probably reflects a less upset-filled Tournament than any real improvement in the predictor).
Possibly more interesting is that I posted my entry to the Kaggle forum before the competition and invited other competitors to steal it. There were at least two interesting results from this. The aforementioned BlueFool submitted an entry that was an average of her entry and my entry. Her entry was 24th, mine was 42nd, but the average of them finished in 8th (!). That suggests that our models had cancelling independent errors. Second, Juho Kokkala submitted entries based upon my entry but with Kentucky's probabilities turned up to 1.0. Had Kentucky won out, Juho would have probably placed in the top two and collected some prize money.
The results of the Kaggle competition have me re-thinking some of my beliefs about predicting the Tournament, but I'll post more about that anon.
In terms of functional improvements to the predictor, I took a look this year at a number of different approaches, most of which I wrote up at some point or another. Some refactoring of my code was a significant improvement in terms of enabling me to try out new ideas more quickly. I took a look at boosting and other machine learning approaches to improving prediction accuracy, but I found that they did not result in any significant improvement over my base predictor. I also took a look at various approaches to creating statistics that are adjusted for schedule, and I adopted or tried out a number of different ideas from the literature or elsewhere. Unfortunately, the articles from the previous Kaggle competition didn't have any truly original ideas to steal.
On a bit of separate note, I had the chance this year to exchange ideas with Erik Forseth (who went on to create a predictor that seems to be every bit as good as my own), Mark Bashuk (SevenOvertimes.com), Scott Burrola, and several others. It's always interesting to exchange thoughts with other folks and see what insights and ideas they have. Feel free to send me an email -- the emails I get always start "Sorry to bother you..." but I'm actually always happy to discuss!
Monday, April 6, 2015
Funny Things You Find in the Play-By-Play Data
3/10/2010 Nebraska versus Missouri, Big 12 Tourney:
12/14/2013 Louisville versus W. Kentucky:
5:11 The Captain was hereI'm not sure how to score that, Captain.
12/14/2013 Louisville versus W. Kentucky:
2:14 Louisville scored in own basket. Two points awarded to Team."Team"? That's not helpful.
Saturday, April 4, 2015
Congratulations!
Congratulations to Dr. Amanda Schierz (aka "Bluefool") on winning the 6th Annual Machine Madness Competition! Wisconsin's victory over Kentucky guaranteed her the championship. Only three of the predictors didn't have Kentucky as the champion, and although "Nothing But Neural Net" has Wisconsin to Bluefool's Duke, he cannot gain enough points to pass her even if the Badgers win on Monday night.
It's worth noting that Monte McNair came within a hair of repeating (back to back, no less) which no one has yet managed to do.
It's worth noting that Monte McNair came within a hair of repeating (back to back, no less) which no one has yet managed to do.
Subscribe to:
Posts (Atom)