Tuesday, August 9, 2011

Clarifications About the JSM Poster

David Smith--from Revolutions--referred me to a criticism at Reddit regarding the poster my fellow grad student and I presented at JSM last week. This comment made me want to clarify what we are attempting in the analysis.

We ARE NOT attempting to find the most deserving players or the best players. We are attempting to use simple statistics to model the voting behavior and decision rules of those making the induction decisions. Many involved in baseball would argue that WAR is the best measure of overall player performance. I'd likely agree. But how many BBWAA voters make inductions based on that statistic (at least prior to, say, 2005)?

This is the idea we are presenting here: Hall of Fame voters are simplistic in nature when it comes to their voting. That doesn't mean they won't change, but it means that they will vote based on the information they have available. This likely includes Goals and Assists. We include Plus-Minus, but find it to be essentially useless in classification, which is probably a good thing: it shows that our model is making the decision rules correctly for this metric.

Now, I do think the thought about normalizing things like goals and assists is a valid one. It is something we are working on, but in baseball have generally found that aggregate milestones are most predictive of Hall induction. For example, using ERA+ did not improve upon the model with ERA. I'm not saying that it's the best way to go, but it seems to be the way the decision rules are made. I will double check this version of the model for hockey, of course.

Lastly, there was concern over including All-Star games in the analysis. Because there are other reasons for voting a player into the Hall--for example "integrity" is used specifically in the baseball induction requirements--the ASG totals are included in order to control for the popularity and general well-liked-ness (is that a word) of a player. We do not include it simply because we think it's a great measure of the best players. And there is certainly noise when it comes to ASG participation. The same goes for Stanley Cup Wins. But a player like Phil Rizzuto almost surely was inducted into the baseball HOF thanks to his appearance on so many World Series teams. It seems that some players are voted in based on their prominence in the media and on good teams. Again, I make no judgement as to whether or not that's the correct way to go.

I hope this clears up any confusion. Hopefully we will have a working version of the paper out in the coming months.

Monday, August 8, 2011

Request for Data (NHL Attendance)

This is a pleading, begging request for some help in collection of some data. I am working on a project looking at franchise-level hockey attendance for a chapter of my dissertation but for the life of me can't find certain years for certain teams. If anyone has the data below, I would be forever grateful to have your assistance. I need season-level attendance data by franchise.

I will even give you a mention in the acknowledgements of my dissertation so that you can live forever in print version in the dusty U of M Kinesiology dissertation library!

Anyway, below is what is needed. If you have anything, please let me know (bmmillsy AT umich DOT edu):

Boston Bruins: 1967-1971

Chicago Blackhawks: 1967-1972 and 1975-1983

Montreal Canadiens: 1967-1972 and 1975 to 1988

New York Rangers: 1967-1972 and 1975-1988

Toronto Maple Leafs: 1967-1972 and 1975-1987

And if you happen to run across it, any attendance data from before 1963, but that's not totally necessary (just always nice to have extra data). If anyone knows WHY these data are missing from just about everywhere possible, I'd also be interested in hearing that.


Friday, August 5, 2011

More on JSM

While my time at the 2011 Joint Statistical Meetings was short--I unfortunately missed some presentations I would have like to have attended--it was a great experience. The collection of academics and professionals is very different from the other conferences that I have attended (like Sport Management and Tourism conferences) and the interest in the methods themselves at JSM really forced me to be on my toes.

While there, I got the chance to put some faces with the names I have seen around the blogosphere. It was a pleasure to meet both Phil Birnbuam--of Sabermetric Research Blog--and David Smith--VP of Revolution Analytics Marketing and author of the Revolutions Blog. David asked about sharing my poster (joint with fellow graduate student, Steve Salaga) investigating Hockey Hall of Fame Induction using the R package "randomForest". While 'machine learning' can sound intimidating to some, Random Forests are actually quite a simple method for bootstrapping classification trees and allowing for random variable selection and a hold-out sample for each tree so that over-fitting is kept to a minimum. And what better way to implement it than with sports data!?!

As a side note, this is not the first time we have implemented randomForest for sports data. Steve and I have a forthcoming paper in the Journal of Quantitative Analysis in Sports identifying patters in BBWAA voting for the Baseball Hall of Fame. Our paper is similar to a recent work by Frieman (2011) in the same journal, but we add pitchers and a discussion of exclusions based on race. As a whole, it seems that there does not seem to be any negative effect of being a minority when it comes to BBWAA voting--at least according to the method we use.

So back to the Hockey Hall of Fame. For both this poster and the baseball paper, it is important to note that we are not attempting to gauge who should be in the Hall of Fame based on their performance as a player. Rather, we are attempting to gauge how well each player aligns with the views of the Hall of Fame Voting Committee and whether or not they were 'snubbed' based on how the committee would be predicted to vote. If the committee is terrible at gauging the best players, then our model will be as well. We are simply interested in the voting behavior and committee preferences, and not who the best players really are. This is an important distinction in attempting to find any exclusions based on qualitative variables like race or language, rather than attempting to rank the best players in the game.

We only include simple statistics--as we predict committee members to focus on these mostly--and goalies are not included in the analysis. Unfortunately, statistics for goalies are few and far between and the NHL has not kept Save Percentage for long enough to include in any worthwhile prediction model for goalies. Therefore, only skaters are included. We separate forwards and defensemen, but the only significant difference is the importance of Assists (they're higher for defensemen).

For example, classifying baseball player inductions on WAR or Win Shares gives us who probably should be the guys in the Hall based on their on-field performance. However, BBWAA voters do not necessarily use this metric when voting. Therefore, we want to train our data to what BBWAA voters do pay attention to. The same goes for hockey. The most important statistics for classifying players are what you would expect, and they are also presented using the Random Forest's "Variable Importance" metric.

This also allowed us to qualitatively evaluate the decision rule boundaries built by the forest and assess the possibility of certain players being discriminated against based on language. There is a line of (conflicting) economic literature--mostly in the 1980s and 1990s--that has made claims of language-based discrimination in the labor market for hockey, so we found the Hall of Fame voting to be another good test of this. Long story short, however, there does not seem to be anything systematic going on. But we leave that up to the reader, as we present each of the players near the boundaries of the decision rules from the forest.

For those interested in the full analysis, you'll have to wait for the paper. As always, there are further considerations for this sort of investigation, not the least of which include testing the RF algorithm against other classification techniques (like neural networks, discriminant analysis, simple classification trees, and others). We'll have to address those as well as other great comments from those that stopped by at the conference. However, a detailed summary of the current version is in THIS POSTER that we presented at JSM.

Thanks to all of those who stopped by. The conference was a great experience and I hope to return next year!