Basketball Power Index

Jeff Fogle over at Stat Intelligence has a new posting deriding ESPN's congratulatory self-coverage of the Basketball Power Index.  I won't comment much on what he says other than to note that as usual he's right on the mark with his criticism.

What I don't like about BPI that he doesn't mention is its "secret sauce" formulation.  Nobody except the stats gurus at ESPN know exactly what the formula is for BPI.  If you chase around the ESPN links trying to find a definition for BPI, you get to this page, which provides this sort of "explanation":

There are a number of small details that we have in our methodology to make it reflective of a résumé for a tournament team -- these are pretty technical and many people won't be interested, so we won't go into detail, but we think they improve how the tool works.

There's no way to check how BPI is calculated, whether all the small details are being applied consistently, whether ESPN is tweaking it weekly to inflate its performance, or to compare it to other methodologies.  (I have the same complaint about Ken Pomeroy, who is similarly vague about his actual calculations.)

Obviously, these folks have every right to keep their ratings formulas secret.  And by all means compare your rating performance to other ratings.  But to my mind, you're not a leader in sports rating systems if you're not willing to expose the details of your rating system and let others test and criticize it.