.tg { margin: 20px auto; border-collapse:collapse; border-spacing:0; } .tg th, td { font-family: Arial, sans-serif; font-size: 12px; border-style: solid; border-width: 1px; overflow: hidden; word-break: normal; } .tg th { padding: 5px 20px; font-weight: bold; } .tg td { padding:3px 20px; } .tg .tg-yw4l { vertical-align:top; text-align: center; } .accuracies { margin: auto; text-align: center; }
After the first weekend of Madness, thereâs no need to remind our readers that game projections are notoriously difficult - just ask the Cavaliers, Musketeers, Bearcats, or Tar Heels what they think of the latest power rankings. The unpredictability is one of the reasons why we love the tournament, but it presents a conundrum when we sit down to fill out our brackets. For our readers who rely on public statistical systems (including ours) for bracket assistance, I wanted to dive deeper into how to evaluate projections, and how to avoid pitfalls hidden in the data.
I judged the accuracy of different projection systems by analyzing projected game outcomes, incorporating home court advantages when possible, and comparing those projections against actual game results. The dataset includes most Division 1 regular season and conference tournament games from the 2017-18 season. Because of data issues with the underlying data source, a small percentage of games are discarded.
I evaluated three different systems:
I also included a home court advantage factor. Moore and Sagarin provide specific home-court advantages that can be applied across all games, which assumes every team has an identical home court advantage. KenPom, on the other hand, applies team-specific home court advantages in his own predictions and I simply used the average of the advantages in place of the individual values to make predictions with his system. I also assumed home court advantage applied to the listed home team in neutral site games.
System Accuracy In Conf Out of Conf Home Away Games Home Court Pomeroy 73.50% 70.40% 78.40% 78.20% 62.80% 5286 3.12 Moore 72.40% 69.50% 76.90% 76.40% 62.00% 5315 4.00 Sagarin 73.70% 70.90% 78.30% 76.90% 65.30% 5283 3.17
I hypothesized that all three projection systems would begin the season with limited accuracy, with each system becoming more accurate over the course of the season (as more information about each team became available). Surprisingly, this proved false, each system enjoyed an early season peak, declined until final exams and the holiday season, after which performance leveled off or improved slightly until conference play started. Some of the early-season projection accuracy can be attributed to the non-competitiveness of âbuy gamesâ, with high major teams padding their win total against low major teams in need of money (including 2018 tourney darling, Texas Southern). I was also surprised to see projection accuracy continue to decline throughout the conference season, especially during February, when teams are usually facing opponents for a second time in a season. In the end, all three systems ended the season around 73% accuracy.
Next, I sliced the projections into common splits and analyzed how each system performed for these subsets:
In-conference versus out-of-conference and games
Picking home team winners versus picking away teams
In general, we would expect the accuracies for these splits to be randomly distributed around the overall average of ~73%. However, since we know that most out-of-conference games are played during the first half of the season, I expected the in/out conference splits to follow the overall time series patterns discussed above - and thatâs exactly what I saw. Since I expected richness of head-to-head matchup history, to improve projection accuracy, I was especially surprised by these results.
Looking at home/away splits provided us with the most surprisingly results of all. Our #1 ranked system, Jeff Sagarin, was over 10% more accurate when picking home teams to win, versus picking away teams to win. This pattern not only holds up for the other projection systems, but is even more pronounced for Moore (76.40% vs 62.00%, -13%) and KenPom (78.20% vs 62.80%, -16%). Seeing all three systems share the same bias toward over-picking away teams suggests it may be time to revisit how home court advantage is factored into these models.
Note: Since KenPom generates team-specific home court factors, it is likely possible that incorporating those values would slightly improve the home/away splits.
Kansas State vs Kentucky - South Region (Atlanta)
(5) Kentucky (9) Kansas St. KenPom 20.44 15.36 KenPom OOC 79.10% 83.50% Moore 82.35 78.43 Moore OOC 76.30% 81.30% Sagarin 88.21 84.31 Sagarin OOC 77.80% 82.30% BracketVoodoo 43.16 38.62 Vegas Spread -5.5 Vegas ML -255 +215
All three systems have Kentucky as a four or five point favorite, though the Atlanta location could make it a semi-home game for the Wildcats in blue. Interestingly, all three systems excel at correctly predicting outcomes for Big 12 out of conference games while having the lowest accuracy of any power conference with games involving an SEC team playing out of conference.
Loyola-Chicago vs Nevada - South Region (Atlanta)
(11) Loyola (IL) (7) Nevada KenPom 15.26 18.13 OOC 73% 77.20% Moore 74.98 78.4 OOC 75.40% 77.00% Sagarin 81.01 84.37 OOC 73.90% 80.20% BracketVoodoo 36.63 40.01 Vegas Spread -1.5 Vegas ML 105 -125 OOC Games 115 101 OOC Games 190 139
This game projects as a close game, with each system pegging the difference in teams to be about three points. It is also the only game where neither team comes from a power conference, which could be influencing why the systems rate them closely. The relative lack of accuracy for both teamsâ conferences in non-conference games suggests that the models might be less trustworthy in this matchup than others.
Florida State vs Gonzaga - West Region (Los Angeles)
(9) Florida St. (4) Gonzaga KenPom 16.21 24.35 OOC 82.00% 78.70% Moore 81.14 81.92 OOC 77.20% 76.20% Sagarin 85.45 89.67 OOC 80.20% 77.00% BracketVoodoo 40.92 44.55 Vegas Spread -5.5 Vegas ML 210 -250 OOC Games 206 122
This game features the most variety among the systems - KenPom and Sagarin have Gonzaga as varying degrees of favorites, whereas Moore has the game as basically a coin flip. But Moore has also had the least success with both conferences, so I would shade towards the other systems. On top of that, the LA crowd could also be slightly in the Bulldogsâ favor as a West Coast team.
Texas A&M vs Michigan - West Region (Los Angeles)
(7) Texas A&M (3) Michigan KenPom 16.91 23.41 OOC 85.30% 79.10% Moore 79.95 82.74 OOC 82.70% 76.30% Sagarin 85.43 89.01 OOC 83.80% 77.80% BracketVoodoo 42.22 44.41 Vegas Spread -2.5 Vegas ML 130 -150 OOC Games 185 190
Michigan is favored by all three systems, with KenPom favoring the Wolverines the most. The Big 10 enjoys the second most accurate out of conference predictions across the systems whereas the SEC has the least accurate, providing an interesting contrast in conference features. However, KenPom has the largest split between the two teams and performs the best for both conferences, increasing my confidence in his prediction of Michigan.
Villanova vs West Virginia - East Region (Boston)
(1) Villanova (5) West Virginia KenPom 31.7 22.29 OOC 86.60% 83.50% Moore 88.62 83.63 OOC 86.40% 81.30% Sagarin 94.79 90.2 OOC 85.90% 82.30% BracketVoodoo 51.07 45.71 Vegas Spread -5.5 Vegas ML -230 195 OOC Games 134 139
Villanova, the top remaining seed by both the committee and all three systems, is a five point favorite according to Moore and Sagarin. KenPom has the difference between the two teams as nearly double that at about nine and a half points. The crowd in Boston will likely be pro-Nova, which would further boost their chances. Furthermore, all three models do their best work with Big East out of conference games, making Villanova even more favored by them.
Texas Tech vs Purdue - East Region (Boston)
(3) Texas Tech (2) Purdue KenPom 21.99 26.99 OOC 83.50% 85.30% Moore 82.83 88.1 OOC 81.30% 82.70% Sagarin 88.45 92.35 OOC 82.30% 83.80% BracketVoodoo 44.02 47.76 Vegas Spread -1.5 Vegas ML 105 -125 OOC Games 129 185
Despite this being a 2/3 matchup in the East region, every system has Purdue as about a five point favorite over Texas Tech. However, the systems are likely unaware of Isaac Haasâ absence for Purdue due to a fractured elbow, which should help Texas Tech close the gap.
Kansas v Clemson - Midwest Region (Omaha)
(1) Kansas (5) Clemson KenPom 23.43 20.25 OOC 83.50% 82.00% Moore 85.6 80.84 OOC 81.30% 77.20% Sagarin 91.72 85.97 OOC 82.30% 80.20% BracketVoodoo 45.98 42.81 Vegas Spread -4.5 Vegas ML -210 180 OOC Games 139 206
This matchup between 1 seed Kansas and 5 seed Clemson is the closest any game comes to a true home/away situation in the Sweet 16, and could make us feel even more confident in the consensus pick of Kansas. KenPom rates the teams as closer than either Moore or Sagarin, which is a departure from some of our other matchups where KenPom had the largest differential.
Duke v Syracuse - Midwest Region (Omaha)
(2) Duke (11) Syracuse KenPom 28.95 14.04 OOC 71.10% 71.10% Moore 88.5 79.06 OOC 72.40% 72.40% Sagarin 93.64 83.15 OOC 73.80% 73.80% BracketVoodoo 50.11 39.02 Vegas Spread -11.5 Vegas ML 575 -795 OOC Games 149 149
Note: Since these are both ACC teams, for this game we looked at each systemâs accuracy for ACC conference games (IC)
Our final game holds the distinction of being the only game between two conference opponents for this round. The two ACC foes only met once during the season, however, so we donât have much history to go on. Duke was able to get past Syracuse by a score of 60-44 at Cameron Indoor, though Syracuse has succeeded thus far in the tournament by holding teams to 60 points or less. It will be interesting to see if the Orangeâs magic continues or if it will end in a double digit victory like the systems are predicting, despite the drop in confidence for in conference predictions by the models.
The biggest wild card region remains the South, based upon the lower accuracies for the remaining teamsâ out of conference games by the three models. Villanova is heavily favored by KenPom to come out of the East, but Moore and Sagarin think a potential matchup between the Wildcats and Boilermakers of Purdue is more of a toss-up. The West regional has similar matchups statistically, but two less predictable conferences involved with the SEC and WCC, making it tough to predict. In the Midwest, Duke is a clear favorite over Syracuse, but the Orangeâs 2-3 zone could be a confounding variable the models struggle to incorporate into their ratings and the in conference factor adds more uncertainty to the matchup of Hall of Fame coaches. If the higher seeds in both matchups advance, each system would favor Duke over one seed Kansas, though any home court advantage for the Jayhawks turns it into a toss-up.
Which Projections Are Best?
The answer, it turns out, is that the âbestâ projection varies from game-to-game. Thatâs not entirely unexpected, but now we have a toolkit for understanding how we can take advantage of those variations and pick the best brackets.
No matter how good the projections are, we still never would have picked UMBC to beat Virginia by 20 - and thatâs why we play the games.