After about two weeks of spare-time effort, I've translated about 75% of my core processing into Python. The result is about 6K SLOC. The Common Lisp version is about 14K SLOC, but it includes a lot of "dead code" that I edited out while translating and the remaining 25% of functionality.
Most of the translated functions have validated exactly against the Common Lisp originals, but one of the core algorithms is producing different (but still valid looking) numbers. I spent a little time trying to debug the difference but couldn't find any immediate culprits. So (after some fits and starts) I processed the entire historical game database through the new Python code and used that to train a model. The model had the same performance and accuracy as the model built off the Common Lisp processed data, so apparently the differences are irrelevant to the model. I suspect the Python version is probably "more correct" than the Common Lisp version, because this is a matrix-manipulation heavy part of the code, and it is expressed much more succinctly and clearly in Python.
At some point I'm going to have to decide whether I want to carry the Play-By-Play processing code forward. I spent a lot of time on the (original) code, and you can pull some interesting data out of the Play-By-Play. On the other hand, the coverage is poor (especially before about 2012) and the data is full of errors. Many (most?) games have statistics that don't agree between the Play-By-Play data and the box score. A surprising number of games have different final scores. So it's hard to put a lot of faith in the data.
Wednesday, September 16, 2015
Monday, September 14, 2015
Shot Clock Rule Change
Most of my (limited) free time is being spent porting Net Prophet to Python, but Eric Forseth sent an email asking about the impending shot clock change that I thought was interesting. For those who might not realize it yet, the NCAA is changing the shot clock this season from 35 seconds to 30 seconds (along with some other rule changes). The big question is whether this will have a big impact on the accuracy of game predictions.
My guess is that it won't. First of all, there are a limited number of teams (Virginia comes to mind) that intentionally play to use most of the 35 second clock. Second, I think most college coaches teach an offensive strategy that says (roughly) "Play our offense until there are 10 seconds left on the clock, then take the first decent shot opportunity." So while the reduced shot clock might cause that second phase to kick in more quickly, it won't be a fundamental difference in how teams play. Third, the modified rules were used in the 2014-2015 NIT and no real differences were observed. (However, that's a very small sample set of 31 games.)
But regardless of my opinion, I think there's a couple of interesting things you can do with your predictor to see how sensitive it would be to this sort of change and/or make it more robust.
First of all you have to realize that if teams play faster or differently in the coming season, that will show up in the game data. So the real question is whether the model you build on previous seasons that used different rules will work well for games played under the new rules.
One way you can investigate this question is to look at how well your model did predicting past games that you think are "most like" the games will be under the new rules. In this case, we might guess that average time of possession will go down in the new season, so we could look at how our model has worked historically on games where the teams had less-than-average times of possession. (That is, where both teams played at a "faster" pace.) If it does an acceptable job predicting those sorts of games, we have some assurance that it will be okay under the new rules.
(As an aside, this sort of analysis is a worthwhile effort even when there isn't an impending rule change. Slicing and dicing the games in different ways and looking at how your model works or doesn't can be very enlightening. You might find out that your model doesn't work on Fridays.)
You can also flip this on its head and try constructing a model using only the fast-paced historical games and see if that model performs better than the more general model. If it does, you might want to consider using the specialized model in the upcoming season (at least provisionally until you see how it actually performs).
If your model is sensitive to game pace (or any other game characteristic), how can you make it more robust? One possibility is to transform the source data (i.e., the statistics used to make predictions) to make it independent of the game pace. One possibility is to use standard scores.
The idea of a standard score is to transform a statistic from a raw number into the number of standard deviations that number is above (or below) the mean. So instead of "Duke averages 8 offensive rebounds a game" you use something like "Duke's offensive rebounding rate is 0.25 standard deviations above the mean". In theory, this approach permits direct "apples to apples" comparisons between seasons where the distribution has changed.
In practice, there are a couple of potential issues with this approach. First, the standard score transformation assumes that the underlying statistic follows a normal distribution. In my experience, this is usually true for NCAA college basketball statistics, but if the statistic does not follow a normal distribution applying this transform may introduce error. Second, the standard score is a non-linear transformation. If you use a linear machine learning model (e.g., a linear regression), this may impact your results -- possibly for the better, but possibly for the worse. Finally, one must be careful in setting the mean and standard deviation for the transformation not to "leak" future information into historical games. For example, it might seem reasonable to transform all games from the 2014 season using the season mean and standard deviation. But this would be an error, because at the beginning of the season you don't know the mean and standard deviation for the whole season.
Another possible transform is to use game ratios rather than raw statistics. In this approach, all the inputs to your model are expressed as ratios between the two competing teams. Instead of "Duke averages 8 offensive rebounds a game" and "Wake Forest averages 6 offensive rebounds a game" you use "Duke rebounds 1.33 times as well as Wake Forest." Like standard score, this is a non-linear transformation but avoids the need to estimate a mean and standard deviation.
My guess is that it won't. First of all, there are a limited number of teams (Virginia comes to mind) that intentionally play to use most of the 35 second clock. Second, I think most college coaches teach an offensive strategy that says (roughly) "Play our offense until there are 10 seconds left on the clock, then take the first decent shot opportunity." So while the reduced shot clock might cause that second phase to kick in more quickly, it won't be a fundamental difference in how teams play. Third, the modified rules were used in the 2014-2015 NIT and no real differences were observed. (However, that's a very small sample set of 31 games.)
But regardless of my opinion, I think there's a couple of interesting things you can do with your predictor to see how sensitive it would be to this sort of change and/or make it more robust.
First of all you have to realize that if teams play faster or differently in the coming season, that will show up in the game data. So the real question is whether the model you build on previous seasons that used different rules will work well for games played under the new rules.
One way you can investigate this question is to look at how well your model did predicting past games that you think are "most like" the games will be under the new rules. In this case, we might guess that average time of possession will go down in the new season, so we could look at how our model has worked historically on games where the teams had less-than-average times of possession. (That is, where both teams played at a "faster" pace.) If it does an acceptable job predicting those sorts of games, we have some assurance that it will be okay under the new rules.
(As an aside, this sort of analysis is a worthwhile effort even when there isn't an impending rule change. Slicing and dicing the games in different ways and looking at how your model works or doesn't can be very enlightening. You might find out that your model doesn't work on Fridays.)
You can also flip this on its head and try constructing a model using only the fast-paced historical games and see if that model performs better than the more general model. If it does, you might want to consider using the specialized model in the upcoming season (at least provisionally until you see how it actually performs).
If your model is sensitive to game pace (or any other game characteristic), how can you make it more robust? One possibility is to transform the source data (i.e., the statistics used to make predictions) to make it independent of the game pace. One possibility is to use standard scores.
The idea of a standard score is to transform a statistic from a raw number into the number of standard deviations that number is above (or below) the mean. So instead of "Duke averages 8 offensive rebounds a game" you use something like "Duke's offensive rebounding rate is 0.25 standard deviations above the mean". In theory, this approach permits direct "apples to apples" comparisons between seasons where the distribution has changed.
In practice, there are a couple of potential issues with this approach. First, the standard score transformation assumes that the underlying statistic follows a normal distribution. In my experience, this is usually true for NCAA college basketball statistics, but if the statistic does not follow a normal distribution applying this transform may introduce error. Second, the standard score is a non-linear transformation. If you use a linear machine learning model (e.g., a linear regression), this may impact your results -- possibly for the better, but possibly for the worse. Finally, one must be careful in setting the mean and standard deviation for the transformation not to "leak" future information into historical games. For example, it might seem reasonable to transform all games from the 2014 season using the season mean and standard deviation. But this would be an error, because at the beginning of the season you don't know the mean and standard deviation for the whole season.
Another possible transform is to use game ratios rather than raw statistics. In this approach, all the inputs to your model are expressed as ratios between the two competing teams. Instead of "Duke averages 8 offensive rebounds a game" and "Wake Forest averages 6 offensive rebounds a game" you use "Duke rebounds 1.33 times as well as Wake Forest." Like standard score, this is a non-linear transformation but avoids the need to estimate a mean and standard deviation.
Monday, September 7, 2015
Welcome to Another Year
"In the Fall, a young man's fancy lightly turns to thoughts of college hoops..."
-- Alfred Lord Tennyson
I may have gotten that quote slightly wrong but as the Summer has been winding to a close I've begun thinking again about college basketball. One of my goals for this year is to move my system into Python. This is a fairly major undertaking, so I thought I'd mention briefly why I've decided to do this.
The architecture of the NetProphet system currently looks something like this:
The top half shows the flow when building the predictive model. WebHarvest is used to fetch the historical data (this only happens once; from then on we can work from the saved copies). A Common Lisp program is then used to process this historical data in various ways. This includes calculating strength measures such as RPI as well as preparing the data for input into the machine learning algorithm. The processed data is then fed into Rapidminer, which uses this processed historical data to create a predictive model, which is saved for later use.
-- Alfred Lord Tennyson
I may have gotten that quote slightly wrong but as the Summer has been winding to a close I've begun thinking again about college basketball. One of my goals for this year is to move my system into Python. This is a fairly major undertaking, so I thought I'd mention briefly why I've decided to do this.
The architecture of the NetProphet system currently looks something like this:
The top half shows the flow when building the predictive model. WebHarvest is used to fetch the historical data (this only happens once; from then on we can work from the saved copies). A Common Lisp program is then used to process this historical data in various ways. This includes calculating strength measures such as RPI as well as preparing the data for input into the machine learning algorithm. The processed data is then fed into Rapidminer, which uses this processed historical data to create a predictive model, which is saved for later use.
The bottom half shows the flow when making a prediction. WebHarvest is used to fetch the schedule of upcoming games. This information is fed into a (slightly different) Common Lisp program which prepares the scheduled games for input into the predictive model. Finally, Rapidminer is used to apply the previously-saved model to the processed schedule to produce predictions.
This may seem complicated, but there's something to be said for breaking up the flow this way and applying the most appropriate tool at each step. However, executing these processes can be tedious, and unfortunately, the past few years have revealed some shortcomings in these tools.
At this point, WebHarvest is basically abandonware. It is open-source, so this isn't a showstopper. In fact, I've hacked the version I'm using to address some problems and add some additional features. However, it would be nice to use a supported tool with some more modern features.
I've used Common Lisp for many years. I'm very productive coding Common Lisp and I have a very efficient coding environment. But for this problem it has some shortcomings. Support for things like matrix mathematics and linear algebra is weak. I've managed to get an interface to BLAS working, but there are still areas where I'm limited by what is available.
Rapidminer has traditionally had some nice advantages for machine learning. The graphical interface makes it very easy to construct models and to try out many different processes. Saving models and applying them to create predictions is very easy. Unfortunately, a couple of years ago RapidMiner transitioned from being free & open-source to a tiered model. While there is still a free & open-source version, all development has stopped on that branch. The free "Community" version is hobbled in ways that make it unusable for me. And the "Professional" version is very expensive -- $2000/year (!). While I was once a strong advocate for Rapidminer as a beginning machine learning tool, due to these changes I can no longer recommend it.
Perhaps more importantly than any of the tool shortcomings, machine learning is a rapidly changing field, and few of the recent developments were available in Common Lisp or Rapidminer. As I've watched the field for the past few years, it seemed that Python is the most popular environment for machine learning research. There's certainly work in other languages, but more often than not interesting work finds its way into the Python environment. Python's also powerful enough to host all of the data processing components that are currently in Common Lisp. Python even has a very good web scraping functionality (Scrapy) that can replace WebHarvest.
The final factor was that I don't currently know Python, so this provides a chance for me to learn a new programming language.
I'm a few days into the rewrite and seem to be progressing fairly well. Updates to follow.
Rapidminer has traditionally had some nice advantages for machine learning. The graphical interface makes it very easy to construct models and to try out many different processes. Saving models and applying them to create predictions is very easy. Unfortunately, a couple of years ago RapidMiner transitioned from being free & open-source to a tiered model. While there is still a free & open-source version, all development has stopped on that branch. The free "Community" version is hobbled in ways that make it unusable for me. And the "Professional" version is very expensive -- $2000/year (!). While I was once a strong advocate for Rapidminer as a beginning machine learning tool, due to these changes I can no longer recommend it.
Perhaps more importantly than any of the tool shortcomings, machine learning is a rapidly changing field, and few of the recent developments were available in Common Lisp or Rapidminer. As I've watched the field for the past few years, it seemed that Python is the most popular environment for machine learning research. There's certainly work in other languages, but more often than not interesting work finds its way into the Python environment. Python's also powerful enough to host all of the data processing components that are currently in Common Lisp. Python even has a very good web scraping functionality (Scrapy) that can replace WebHarvest.
The final factor was that I don't currently know Python, so this provides a chance for me to learn a new programming language.
I'm a few days into the rewrite and seem to be progressing fairly well. Updates to follow.
Friday, April 10, 2015
The Worst Predictions of 2015
While I like to focus on games where my predictor has been very accurate, it's sometimes entertaining to look at the games where it did the worst. And there are plenty of those. In fact, there were more than 100 games this year where my prediction was off by over 25 points.
The worst of the lot this year was the January 14 game between Nevada and Colorado State, which I predicted as a 15 point win for Colorado State. In fact, they won by 56 -- an error of over 40 points. Looking at the game, it isn't hard to see what went wrong. Nevada shot just 16% in the first half and was down 55-12 at halftime. Colorado State eased up in the second half or the damage might have been worse. As it was, the 56 point victory is the largest in Colorado State history.
But my worst prediction of all time was off by 61, so missing by "only" 40 is not so bad.
A more important game this year was the December 22nd matchup between Kansas and Temple. Kansas came into the game ranked #10 in the nation and was not expected to have much trouble with Temple, who was 8-4 and had already lost to the likes of UNLV and St. Joe's. I predicted a 10 point win by Kansas, and Vegas had them as 8 point favorites (climbing a point from the open). Instead, the Temple guards went crazy, pouring in 54 point en route to a 25 point victory -- an error of 35 points in my prediction. For the game, Temple shot 58% to Kansas's 32%.
Another terrible prediction occurred the night before the Kansas game when Harvard played Virginia. I had this as a 15 point Virginia win. Vegas favored Virginia by 10. Harvard was a good team -- they finished the season 22-8 after barely losing to UNC in the Tournament -- but Tony Bennett's team put on a defensive clinic for the ages. They held Harvard to a single basket in the first half and 8 of 50 (16%!) shooting for the night. Virginia won by 49 -- a 34 point error for me, but at least I was closer than Vegas.
These sorts of games demonstrate that predicting college basketball is really difficult. There's just a lot of variance and random factors that go into the final score, and you can't get too worked up over the occasional terrible result!
The worst of the lot this year was the January 14 game between Nevada and Colorado State, which I predicted as a 15 point win for Colorado State. In fact, they won by 56 -- an error of over 40 points. Looking at the game, it isn't hard to see what went wrong. Nevada shot just 16% in the first half and was down 55-12 at halftime. Colorado State eased up in the second half or the damage might have been worse. As it was, the 56 point victory is the largest in Colorado State history.
But my worst prediction of all time was off by 61, so missing by "only" 40 is not so bad.
A more important game this year was the December 22nd matchup between Kansas and Temple. Kansas came into the game ranked #10 in the nation and was not expected to have much trouble with Temple, who was 8-4 and had already lost to the likes of UNLV and St. Joe's. I predicted a 10 point win by Kansas, and Vegas had them as 8 point favorites (climbing a point from the open). Instead, the Temple guards went crazy, pouring in 54 point en route to a 25 point victory -- an error of 35 points in my prediction. For the game, Temple shot 58% to Kansas's 32%.
Another terrible prediction occurred the night before the Kansas game when Harvard played Virginia. I had this as a 15 point Virginia win. Vegas favored Virginia by 10. Harvard was a good team -- they finished the season 22-8 after barely losing to UNC in the Tournament -- but Tony Bennett's team put on a defensive clinic for the ages. They held Harvard to a single basket in the first half and 8 of 50 (16%!) shooting for the night. Virginia won by 49 -- a 34 point error for me, but at least I was closer than Vegas.
These sorts of games demonstrate that predicting college basketball is really difficult. There's just a lot of variance and random factors that go into the final score, and you can't get too worked up over the occasional terrible result!
Tuesday, April 7, 2015
Year End Summary
Congratulations to the Duke Blue Devils on winning their 5th Championship and bringing the college basketball season to a great finish. Here's the season summary for Net Prophet:
For the first time, I submitted predictions to The Prediction Tracker. For the year, NetProphet ended up being the best predictor Against The Spread (ATS), and in the top two or three in a number of other categories. (Oddly enough, NetProphet is not very good predicting the winner straight up. Doktor Entropy is notable for doing very well both ATS and straight up.) Apparently some others took note that I was submitting picks to the Prediction Tracker, because when I didn't submit for one reason or another I got emails asking where they were!
And though I've said it before, many thanks to Todd for running The Prediction Tracker! It's a great service and a lot of work.
For The Prediction Tracker I submitted picks on all college games. I also privately identify "high confidence" picks that I think should perform well against the spread. This year those picks went 58.2% ATS, which is a very solid performance.
With the help of Monte McNair, I ran the 6th Annual Machine Madness competition. There were fourteen competitors, and the competition was won by BlueFool (aka Dr. Amanda Schierz) in her first year competing.
I don't put much effort into predicting the Tournament, but I did compete in both the Machine Madness competition and the Kaggle competition. In my Machine Madness entry, NetProphet identified an upset of Kentucky as the best value in pool play and selected Arizona over Kentucky as the most likely upset. The strategy was sound -- NetProphet was the only competitor with Arizona winning it all -- but as it turns out, Arizona couldn't even beat Wisconsin, much less Kentucky, so that did not pay off.
On the Kaggle side, my entry didn't do well the first day but climbed steadily into the 20s by the Championship game. The Duke win hurt me, however, and I fell back to 42nd. That's somewhat better than I did in 2014 (60th) and my log-loss score was also considerably lower (but that probably reflects a less upset-filled Tournament than any real improvement in the predictor).
Possibly more interesting is that I posted my entry to the Kaggle forum before the competition and invited other competitors to steal it. There were at least two interesting results from this. The aforementioned BlueFool submitted an entry that was an average of her entry and my entry. Her entry was 24th, mine was 42nd, but the average of them finished in 8th (!). That suggests that our models had cancelling independent errors. Second, Juho Kokkala submitted entries based upon my entry but with Kentucky's probabilities turned up to 1.0. Had Kentucky won out, Juho would have probably placed in the top two and collected some prize money.
The results of the Kaggle competition have me re-thinking some of my beliefs about predicting the Tournament, but I'll post more about that anon.
In terms of functional improvements to the predictor, I took a look this year at a number of different approaches, most of which I wrote up at some point or another. Some refactoring of my code was a significant improvement in terms of enabling me to try out new ideas more quickly. I took a look at boosting and other machine learning approaches to improving prediction accuracy, but I found that they did not result in any significant improvement over my base predictor. I also took a look at various approaches to creating statistics that are adjusted for schedule, and I adopted or tried out a number of different ideas from the literature or elsewhere. Unfortunately, the articles from the previous Kaggle competition didn't have any truly original ideas to steal.
On a bit of separate note, I had the chance this year to exchange ideas with Erik Forseth (who went on to create a predictor that seems to be every bit as good as my own), Mark Bashuk (SevenOvertimes.com), Scott Burrola, and several others. It's always interesting to exchange thoughts with other folks and see what insights and ideas they have. Feel free to send me an email -- the emails I get always start "Sorry to bother you..." but I'm actually always happy to discuss!
For the first time, I submitted predictions to The Prediction Tracker. For the year, NetProphet ended up being the best predictor Against The Spread (ATS), and in the top two or three in a number of other categories. (Oddly enough, NetProphet is not very good predicting the winner straight up. Doktor Entropy is notable for doing very well both ATS and straight up.) Apparently some others took note that I was submitting picks to the Prediction Tracker, because when I didn't submit for one reason or another I got emails asking where they were!
And though I've said it before, many thanks to Todd for running The Prediction Tracker! It's a great service and a lot of work.
For The Prediction Tracker I submitted picks on all college games. I also privately identify "high confidence" picks that I think should perform well against the spread. This year those picks went 58.2% ATS, which is a very solid performance.
With the help of Monte McNair, I ran the 6th Annual Machine Madness competition. There were fourteen competitors, and the competition was won by BlueFool (aka Dr. Amanda Schierz) in her first year competing.
I don't put much effort into predicting the Tournament, but I did compete in both the Machine Madness competition and the Kaggle competition. In my Machine Madness entry, NetProphet identified an upset of Kentucky as the best value in pool play and selected Arizona over Kentucky as the most likely upset. The strategy was sound -- NetProphet was the only competitor with Arizona winning it all -- but as it turns out, Arizona couldn't even beat Wisconsin, much less Kentucky, so that did not pay off.
On the Kaggle side, my entry didn't do well the first day but climbed steadily into the 20s by the Championship game. The Duke win hurt me, however, and I fell back to 42nd. That's somewhat better than I did in 2014 (60th) and my log-loss score was also considerably lower (but that probably reflects a less upset-filled Tournament than any real improvement in the predictor).
Possibly more interesting is that I posted my entry to the Kaggle forum before the competition and invited other competitors to steal it. There were at least two interesting results from this. The aforementioned BlueFool submitted an entry that was an average of her entry and my entry. Her entry was 24th, mine was 42nd, but the average of them finished in 8th (!). That suggests that our models had cancelling independent errors. Second, Juho Kokkala submitted entries based upon my entry but with Kentucky's probabilities turned up to 1.0. Had Kentucky won out, Juho would have probably placed in the top two and collected some prize money.
The results of the Kaggle competition have me re-thinking some of my beliefs about predicting the Tournament, but I'll post more about that anon.
In terms of functional improvements to the predictor, I took a look this year at a number of different approaches, most of which I wrote up at some point or another. Some refactoring of my code was a significant improvement in terms of enabling me to try out new ideas more quickly. I took a look at boosting and other machine learning approaches to improving prediction accuracy, but I found that they did not result in any significant improvement over my base predictor. I also took a look at various approaches to creating statistics that are adjusted for schedule, and I adopted or tried out a number of different ideas from the literature or elsewhere. Unfortunately, the articles from the previous Kaggle competition didn't have any truly original ideas to steal.
On a bit of separate note, I had the chance this year to exchange ideas with Erik Forseth (who went on to create a predictor that seems to be every bit as good as my own), Mark Bashuk (SevenOvertimes.com), Scott Burrola, and several others. It's always interesting to exchange thoughts with other folks and see what insights and ideas they have. Feel free to send me an email -- the emails I get always start "Sorry to bother you..." but I'm actually always happy to discuss!
Monday, April 6, 2015
Funny Things You Find in the Play-By-Play Data
3/10/2010 Nebraska versus Missouri, Big 12 Tourney:
12/14/2013 Louisville versus W. Kentucky:
5:11 The Captain was hereI'm not sure how to score that, Captain.
12/14/2013 Louisville versus W. Kentucky:
2:14 Louisville scored in own basket. Two points awarded to Team."Team"? That's not helpful.
Saturday, April 4, 2015
Congratulations!
Congratulations to Dr. Amanda Schierz (aka "Bluefool") on winning the 6th Annual Machine Madness Competition! Wisconsin's victory over Kentucky guaranteed her the championship. Only three of the predictors didn't have Kentucky as the champion, and although "Nothing But Neural Net" has Wisconsin to Bluefool's Duke, he cannot gain enough points to pass her even if the Badgers win on Monday night.
It's worth noting that Monte McNair came within a hair of repeating (back to back, no less) which no one has yet managed to do.
It's worth noting that Monte McNair came within a hair of repeating (back to back, no less) which no one has yet managed to do.
Subscribe to:
Posts (Atom)