Year End Summary

Congratulations to the Duke Blue Devils on winning their 5th Championship and bringing the college basketball season to a great finish.  Here's the season summary for Net Prophet:

For the first time, I submitted predictions to The Prediction Tracker.  For the year, NetProphet ended up being the best predictor Against The Spread (ATS), and in the top two or three in a number of other categories.  (Oddly enough, NetProphet is not very good predicting the winner straight up.  Doktor Entropy is notable for doing very well both ATS and straight up.)   Apparently some others took note that I was submitting picks to the Prediction Tracker, because when I didn't submit for one reason or another I got emails asking where they were!

And though I've said it before, many thanks to Todd for running The Prediction Tracker!  It's a great service and a lot of work.

For The Prediction Tracker I submitted picks on all college games.  I also privately identify "high confidence" picks that I think should perform well against the spread.  This year those picks went 58.2% ATS, which is a very solid performance.

With the help of Monte McNair, I ran the 6th Annual Machine Madness competition.  There were fourteen competitors, and the competition was won by BlueFool (aka Dr. Amanda Schierz) in her first year competing.

I don't put much effort into predicting the Tournament, but I did compete in both the Machine Madness competition and the Kaggle competition.  In my Machine Madness entry, NetProphet identified an upset of Kentucky as the best value in pool play and selected Arizona over Kentucky as the most likely upset.  The strategy was sound -- NetProphet was the only competitor with Arizona winning it all -- but as it turns out, Arizona couldn't even beat Wisconsin, much less Kentucky, so that did not pay off. 

On the Kaggle side, my entry didn't do well the first day but climbed steadily into the 20s by the Championship game.  The Duke win hurt me, however, and I fell back to 42nd.  That's somewhat better than I did in 2014 (60th) and my log-loss score was also considerably lower (but that probably reflects a less upset-filled Tournament than any real improvement in the predictor).

Possibly more interesting is that I posted my entry to the Kaggle forum before the competition and invited other competitors to steal it.  There were at least two interesting results from this.  The aforementioned BlueFool submitted an entry that was an average of her entry and my entry.  Her entry was 24th, mine was 42nd, but the average of them finished in 8th (!).  That suggests that our models had cancelling independent errors.  Second, Juho Kokkala submitted entries based upon my entry but with Kentucky's probabilities turned up to 1.0.   Had Kentucky won out, Juho would have probably placed in the top two and collected some prize money.

The results of the Kaggle competition have me re-thinking some of my beliefs about predicting the Tournament, but I'll post more about that anon.

In terms of functional improvements to the predictor, I took a look this year at a number of different approaches, most of which I wrote up at some point or another.  Some refactoring of my code was a significant improvement in terms of enabling me to try out new ideas more quickly.  I took a look at boosting and other machine learning approaches to improving prediction accuracy, but I found that they did not result in any significant improvement over my base predictor.  I also took a look at various approaches to creating statistics that are adjusted for schedule, and I adopted or tried out a number of different ideas from the literature or elsewhere.  Unfortunately, the articles from the previous Kaggle competition didn't have any truly original ideas to steal.

On a bit of separate note, I had the chance this year to exchange ideas with Erik Forseth (who went on to create a predictor that seems to be every bit as good as my own), Mark Bashuk (SevenOvertimes.com), Scott Burrola, and several others.  It's always interesting to exchange thoughts with other folks and see what insights and ideas they have.  Feel free to send me an email -- the emails I get always start "Sorry to bother you..." but I'm actually always happy to discuss!