By Nick Citrone (@)
- North Dakota State prospect Carson Wentz is the top quarterback in the draft, ranking as one of the five best QB prospects in the last decade.
- Potential #1 Overall Pick Jared Goff is not predicted to be as strong as most believe, ranking as the 5th best QB in the draft.
- Wonderlic and College completion percentage are the statistics most correlated with NFL success.
- When trying to predict NFL QB Success using college & combine data, simple models work best. Predictive power is low, but significant.
NFL Free agency is flashy and exciting, but dynasties are built through the draft. Succesful selections in the NFL draft award teams with four years of premium play at a cheap cost, and with rare exceptions the draft is the only way for teams to land a franchise quarterback. What makes the draft so exciting is that it is as unpredictable as it is important; teams spend hundreds of hours studying incoming players but there is no guarantee college success or strong combine scores will translate to success at the professional level.
However, by modeling future NFL success using pre-draft data we can gain key insight on avoiding draft busts and identifying potential late-round steals. I have built two models: one for QBs and one for RBs which provide knowledge on which players are most likely to succeed and which measurements are most indicative of professional success.
One of the biggest challenges in this research was successfully merging three different types of data: collegiate (NCAA) player statistics, NFL player statistics, and NFL combine results. I obtained the NFL and college player statistics from sister-sites Pro Football Reference[http://www.pro-football-reference.com/] and College Football Reference[http://www.sports-reference.com/cfb/]. I pulled data on every player who had thrown a pass, caught a pass or had a rushing attempt at the NFL or Division 1 NCAA level between 2000 and 2015. I gathered the NFL combine data from two sources: nflcombineresults.com[http://nflcombineresults.com/nflcombinedata.php?] and NFLsavant.com [http://nflsavant.com/about.php].
My response variable is Approximate Value per season, which I originally discussed in an article on Maximizing Draft Value by Position.[https://tartansportsanalytics.com/2016/03/30/maximizing-draft-value-by-position/] Not all players in the dataset were drafted and some players do not end up on an NFL roster, but these entries are invaluable because they teach us exactly what NOT to look for in NFL QBs. The distribution of AV per season, shown below, is heavily skewed by the large number of quarterbacks who never played in an NFL game.
Prior to building the predictive models I first filtered my dataset to players who played D1 college football, attended the NFL Combine and declared for the draft between 2003 and 2014. Next, several players skip certain NFL Combine drills either by choice or due to injury, but incomplete values wreak havoc on many types of predictive modeling. To address this issue I substituted in positional mean drill scores for incomplete values.
For model selection, I constructed a function which computed leave-one-out cross validated mean squared errors for a given model. I tried fitting many different types of models to the data, including generalized linear regression, generalized additive models, regression trees, lasso regression, and cluster analysis. Surprisingly, the simplest models I tested performed the best, and the final model is a linear regression with a log-transformed response variable. Linear regression assigns a coefficient to each variable and then sums the variable terms to arrive at a predicted value. Log-transforming the response is done by adjusting the variable so the minimum is one, then taking the log and fitting a model. The non-transformed predicted values are then extracted by raising e to the power of the log-transformed predicted values.
The variables included in the Quarterback Predicted NFL Success model were college completion percentage, college TD pass percentage, player height and weight, 40 yard dash time, Wonderlic score and the player’s broad jump score. The below table shows the coefficients and p-values for the variables in the model; note that the y-variable is log-transformed and the model returns log(AVper) values.
To visualize the success of the model I have plotted the out-of-sample predicted AV per season values created by the model against the actual AV per season of the QBs in the data set. The blue line shows a smooth spline fit of Actual vs. Predicted AVper, with 95% error bars represented by the grey shading. The increasing smooth spline proves the predictive power of the model.
In the plot above I have highlighted a few individual players which demonstrate the strength of the model. Andrew Luck and Cam Newton were by a wide margin predicted as the two best prospects, and neither has dissapointed. The model also correctly predicted very little success for Washington QB Jake Locker, whose NFL career lasted just three seasons despite being drafted #8 overall by the Titans in 2011. Lastly, current Bills QB Tyrod Taylor’s recent success was forseen by the model which placed the 6th round draft pick among the top 20% of QBs in the data set.
Evaluating individual QB prospects is only half the benefit of the model I built. The extensive variable selection process resulted in the model only including terms which provide significant insight into success at the NFL level. There are a lot of potential takeaways from the model, but none are more exciting than the fact that the Wonderlic test is actually indicative of NFL QB success. The below graph shows the association between NFL success and Wonderlic scores; the blue line is a smooth spline fit which shows AV per season increases for higher Wonderlic.
People have debated the signifigance, or lack thereof, of the bizarre IQ test for years but the data shows wonderlic scores are more correlated with NFL QB success than any other combine score. It is unclear exactly why this is the case, and there are many possible reasons. QBs who perform better on the wonderlic might be able to better read defenses and make quick decisions, but it could also be because they are more thorough in their preparation for both the test and the NFL. Correlation does not imply causation, but it is still useful for making predictions.
Predicting the 2016 QB Draft Class
We can use the models created above to generate positional draft boards for the upcoming NFL Draft. The below barplot shows the predicted scores for each of the Quarterbacks who participated in the 2016 NFL Combine. Unfortunately, Wonderlic scores are not released by the NFL and only Carson Wentz (40) and Jared Goff (36) have had their scores revealed[http://larrybrownsports.com/football/carson-wentz-jared-goff-wonderlic-scores/298541] by former NFL scout John Middlekauff. To predict the rest of the draft class I substituted the average Wonderlic score since 2003 (26.1), so their is some accuracy loss. The predicted are as follows:
North Dakota State Quarterback Carson Wentz tops the board with an impressive 5.95 predicted AVper, but it is important to note the model has been trained solely on BCS college statistics while Wentz played in the weaker FCS division. Even so, Wentz’s score puts him in an elite group: every quarterback since 2003 with a predicted value above 4.50 has been drafted in the first half of round 1. The Rams will meet with both Wentz and Jared Goff prior to picking first overall on April 28th, and the QB model clearly believes Wentz is the better pick.
Jared Goff’s 2.54 initially seems terrible, but the young Cal QB’s predicted value ranks 33rd out of the 174 players used to build the model. The score is solid, but still underwhelming for a prospect many believe will be taken #1 overall. Goff’s score is pulled up by his impressive Wonderlic score, but his 40 yard dash and broad jump measurements were both mediocre. Goff likely won’t fall out of the top 10 in the draft, but the team that takes him may not find the success they seek.
Carson Wentz and Jared Goff are widely regarded as the top 2 QBs in the draft, but Louisiana Tech’s Jeff Driskel takes the #2 spot here with a predicted value per season of 3.58. Driskel’s jump up the draft board can be attributed to his impressive combine: his 4.56 40 yard dash and 122 broad jump were both the best of all QBs in 2016. Driskel is a very raw prospect, but he has the athleticism to wow if given time to develop. The third QB on the model’s draft board, Paxton Lynch has the height of a prototypical QB and was especially impressive in the broad jump.
Is the quarterback model a better predictor of NFL QB success than order drafted? Using Spearman’s rank correlation coefficient, I compared the correlation between ranked actual AVper and both predicted rankings and draft rankings for each of the 11 draft classes in the data. Both rankings have roughly similar success (rho 0.81 for Draft order, 0.72 for the model), but the draft order is ultimately a slightly better indicator. This is at least partially because QBs taken with early draft picks are afforded more time to develop and almost always given a chance to play at the NFL level. Even still, the QB prediction model outperformed actual draft order in five of the eleven draft classes (2003, 2008, 2010, 2013, 2014).
The model has significant predictive power, but it is far from perfect. The simple reality is if it was easy to predict NFL success with great success using college & combine data we would not see high profile picks bust at such a high rate. The reality is there are an immeasurable number of factors which dictate the success a player has at the NFL level including coaching, teammate ability, injuries, work ethic, size, and many more. This model is designed to be used as one of many tools in the arsenal of a draft-day General Manager.