tag:blogger.com,1999:blog-5902620336509647050.post1702571608600452081..comments2023-10-31T07:06:33.886-04:00Comments on Net Prophet: What Would a Perfect (Knowledge) Predictor Score in the Kaggle Competition?Scott Turnerhttp://www.blogger.com/profile/03393071448515738228noreply@blogger.comBlogger4125tag:blogger.com,1999:blog-5902620336509647050.post-44059265004881670272017-05-30T07:24:30.634-04:002017-05-30T07:24:30.634-04:00This comment has been removed by a blog administrator.Anonymousnoreply@blogger.comtag:blogger.com,1999:blog-5902620336509647050.post-44522275387802328842017-05-04T08:01:47.948-04:002017-05-04T08:01:47.948-04:00Good post, this will surely help me to predict sco...Good post, this will surely help me to predict scores of tournment starting in my office next week. Thank you for sharing it<a href="https://www.huffingtonpost.com/nancy-laws/the-shocking-truth-about-_5_b_7041934.html">essay best</a> https://www.blogger.com/profile/11825807784628551119noreply@blogger.comtag:blogger.com,1999:blog-5902620336509647050.post-85481530432270065922016-03-27T00:41:43.205-04:002016-03-27T00:41:43.205-04:00This comment has been removed by the author.melondonkeyhttps://www.blogger.com/profile/02818146576436658055noreply@blogger.comtag:blogger.com,1999:blog-5902620336509647050.post-80202561330366311372016-03-22T21:22:00.312-04:002016-03-22T21:22:00.312-04:00Scoring these kinds of things is hard because over...Scoring these kinds of things is hard because overconfidence can be rewarded (and it is more likely to be if you're on the right side of 50% in the long run). If you think of an infinite number of Kaggle competitions, anyway. It'd be interesting to see someone create a series of NCAA-like seasons of data with known true probability rates and run the various Kaggle submissions against them and see who picks the correct perfect knowledge.<br /><br />Something that I think is missing from Kaggle is model skill. This article covers it well for the uninitiated: http://fivethirtyeight.com/features/when-picking-a-bracket-its-easier-to-be-accurate-than-skillful/). Logs of probability get a little bit at skill because being more certain of the correct response is generally more skillful. However, if you just said you were 100% certain of 2v15 and a 3v14, you'd get the same score for both, when it's less impressive to predict the 2v15 (where you were only 6% more certain than naive historical perspectives). But of course there is the problem of picking a historically reasonable 'naive' baseline for skill estimation as you have pointed out. indpndnthttps://www.blogger.com/profile/10104482482233171142noreply@blogger.com