Over the last couple of days I’ve posted on applying a simple statistical test to NHL goalies to gauge their starter potential and how this test performed on actual ‘good’ goalies.
Today I’ll be exploring an unanswered question from yesterday — does this test falsely characterize actually ‘bad’ goalies as being good, thereby leading to false conclusions about their talent-level?
Ground Rules, Revisited
Today’s cohort will be a bit harder to populate for a pretty simple reason –‘actually bad’ goalies usually don’t get to play for long in the NHL (again, unless you’re Ondrej Pavelec). I’ll be relaxing my criteria a bit: a) over 125 games played while having b) 0.907 or less SV%. I’ll be trying to find goalies who were given either starting duties or heavy backup duties at some point in their career, who also played the large majority of their careers after the 2012-13 lockout.
It turns out 9 goalies match this ‘bad goalie’ criteria pretty well:
- Jonas Gustavsson
- Anders Lindback
- Mike Condon
- Ben Scrivens
- Anders Nilsson
- Al Montoya
- Ondrej Pavelec!!
- Chad Johnson
- Keith Kinkaid
I’ll explore if any of these guys were falsely characterized as better than 0.9162 and if/when they were correctly classified as being worse than 0.9162 SV%.
Jonas Gustavsson
The Monster! In an almost identical progression to Koskinen, Gustavsson had a short spike, was rejectable as a starter by game 31, and terminally entered the red pit by game 56. He’d go on to play 179 games in the NHL, never coming close to being falsely characterized as being ‘better’ than an NHL starter.
Anders Lindback
Out of this cohort of 9, Lindback was the furthest from being a ‘starter’, maxing out at 28 GP in a season. But he played 130 total career games, spending almost half of which being rejectable as having NHL ‘starter’ quality. He was falsely classified after game 1 in the green zone, but dropped below having a 33% shot of being a true starter permanently after game 46.
Mike Condon
Condon follows a pretty typical trajectory — he played 8 hot games in a row to start his career that likely bought him a multi-year NHL career. In those 8 games he was 0.936, but he still was not misclassified as being better than a decent NHL starter. He was first rejectable as an NHL starter by game 37, at many points afterwards, and fell into the red pit permanently by game 110 of his career. He’s got 1 year left on a deal with a $2.4M cap hit. Ouch.
Ben Scrivens
Scrivezina! Scrivens will be two goalies in this cohort that push the boundaries of being a ‘bad’ goalie. At times I genuinely thought he was pretty good. He carried an 80% chance of being a decent NHL starter as late as game 61 of his career. He seemed like a legit goalie, even on the Oilers. Buuuut then the wheels fell off. He spent the next 60 games undoing his good work, finally being rejectable as a starter at game 124 of his career and never getting out of the red. After game 129 he was traded to Montreal, who got 15 more bad games out of him for the low, low price of $2.3M. But on the whole, Scrivens’ evaluation arc is perfect — he was given time to have some initial struggles, build a resume, eventually to perform well, given more time to unravel it, and was out of the league within 20 games of a terminal spiral.
Anders Nilsson
What was this guy’s name — The Moose? In any case, Nilsson has been pretty terrible since entering the league. (as an aside, a lot of this list have been Oilers oddly enough). He was rejectable as a starter by game 15 and has spent a huge portion of his 141 game career in the red. But he gets chances over and over again — Aaand the Ottawa Senators obviously see something they like — Nilsson just signed a 2 year $2.6M/year contract with them this summer! Condon and Nilsson — have fun Ottawa.
Al Montoya
Montoya is another goalie whose career started with some promise but hit the skids around game 40, dropped below 33% likelihood of being a decent starter by game 51, and was rejectable as a starter by game 58. He went on to play another 110 uneventful games, the last bit with the Oilers.
Ondrej Pavelec
We’re finally here! This feels great! Look at that chart! Pavelec is bad. His numbers were bad forever. On this test he was bad forever. He was rejectable by game 18 in his career and would go on to play 398 career games. 380 games that should not have occurred and lost his team a pile of games. He was hitting 0.6% on this test by game 46 of his career — 0.6% chance of being a decent NHL starter — that’s like a 1 in 116 shot of being a starter! He would fall into the red pit permanently by game 114 and was pinned near 0% for the rest of his like 285 game career. This has to be the worst goalie overcommitment in, what, the Bettman-era?
Chad Johnson
Now we get to the best of the worst. Chad Johnson played well for a pretty long time in the NHL. He had a Bobrovsky-ian slip around games 50-75, but regained his form to post great numbers, only to slip again, first being rejectable as a decent starter by game 181 of a 192 game career. So this test wouldn’t have really helped much — and that’s really the spirit of this test — as long as he was playing well and putting up pretty great numbers, how the hell else do you have anything to go by to suggest otherwise? Where this test would most aid is when a dip becomes a meaningful deviation from starting quality. If he’s playing well — play him! This tells you when to cut bait. And with Johnson, the NHL kind of collectively nailed it. They squeezed out his value, gave him enough leash to prove he wasn’t working out, and now he’s probably done when he should be.
Keith Kinkaid
Kinkaid is another goalie who avoided the red pit for quite a long time — not until game 146 of a 151 game career. He’s never been a clearcut starter, but he’s played 41 games for two years in a row now. If the Canadiens signed him to load manage Carey Price, they should have their eyes open on what to expect — likely non-starter quality talent. Then again, $1.75M for a non-crazy bet on a backup isn’t a really bad idea.
Koskinen vs the ‘Bad’ goalie cohort
Again we show Koskinen in thick black on this incredibly ugly chart versus the cohort of 9 ‘bad’ goalies. Koskinen is pretty middle of the pack here and, if anything, seems be even more rejectable than this peer group. Some observations:
- The last Type I error for the green rejection zone was Al Montoya at game 2. The last time a ‘Bad’ goalie was above 90% probability was Mike Condon at game 7. Scrivens was the only goalie to play above 80% probability (for 2 games) after game 40. Johnson and Scrivens were the only goalies to play above 50% probability after game 45. This test really does seem to steer bad goalies away from being falsely labeled as ‘better than decent starters’ after the first 10 games, but it also seems to push legitimately bad goalies down faster than 100 games played.
- They’re all rejectable as decent starters at some point (under 5% probability). Here’s a summary (other than spelling Kinkaid wrong):
- If you’re thinking Koskinen spent a pretty large chunk of his career under 33% probability of being a starter even compared to this crew, you’re right:
As mentioned, some of these guys started pretty hot but all eventually steadily careened downwards. Compared to yesterday’s post where (after a point) good goalie’s curves inexorably increased, this crew’s curves all grimly march lower. And which career curves are Koskinen most like? Pavelec, Gustavsson, and Nilsson. If you follow all three of those gentlemen’s curves to the right, I’d say we’re staring at about 120 additional games of sub-starter level goaltending from Koskinen.
Scrivens was at a similar place as Koskinen, but his yellow curve decoupled upwards pretty quickly towards a mid-career peak. Koskinen’s hasn’t broken back up yet and time is ticking.
Putting the cohorts together
So I’m semi-unsatisfied with the ugliness of the above charts — so let’s combine these two cohorts in at least a reasonably satisfying way:
The ‘Good’ cohort is in blue, ‘Bad’ cohort is in red, and Koskinen is in black. Where do you think the delineation happens between good and bad? You’d have to wait until game 135 to perfectly classify everyone. Separating the two groups after game 99 would only give you one goalie on the wrong side. Game 60 seems to generally drive the point home. But the really noisy confusion on the left seems to mostly resolve by about game 40.
What about the graph that shows the amount of careers spent above a 33% confidence level of being a starter? That one’s pretty now too…
Here we see rather starkly the difference between good and bad goalies. Bishop is the last good goalie to make his break upwards at around game 35. And while the blue lines are increasing, the red lines are all streaking downwards in a striking exponential pattern. By game 59, all of the good goalies are miles ahead of Koskinen — they’d spent huge portions of their careers with at least a 1/3rd shot of being starters by that point. The sad part is that many of the bad ones are ahead of him as well by game 59.
The X’s mark where you could have first rejected a Bad goalie on a red curve. I visualize just snipping off each red line at the X and watching the last meaningless number of games they played afterwards just falling off into the ether. For Pavelec’s curve I can actually hear a huge ‘thud’ sound and clouds of dust billowing upwards as 380 meaningless games hit the X-axis.
And remember, there are no X’s on a blue curve, as good goalies were never rejected after game 9.
Final Rules of Thumb
Say you have a goalie you’re interested in learning about. Is this kid a starter? Here’s some rules of thumb I’d propose after this Good vs Bad analysis:
- Give a new goalie 15-20 games in the NHL. Don’t reject the null hypothesis either way until he’s played that many.
- If your goalie drops below a 5% chance of being a 0.9162 goalie at any point after 20 games, you’ve got pretty compelling evidence that he’s not a starting goalie. Since NHL games are so rare and you only get to play one goalie at a time, personally I’d move on at this point and try another goalie out. If he’s cheap and your prospect pipeline sucks maybe keep him around, but in that case you can likely find someone who hasn’t failed this test yet for cheap in free agency every year.
- If your goalie gets above 95% probability on this test after 15-20 games, you’ve got pretty decent evidence that he’s at least a decent-starting talent, and perhaps better. I’d also be pretty damn interested in anyone above 90% after 40 games or 80% after 60. If you can catch them early enough, these are the candidates for a Ben Bishop-style $4.9M/year long-term deal when the time is right. Follow them closely, keep them happy and if they keep it up, sign them longer-term as quietly as possible before their bargaining position goes nutty.
- Anyone who doesn’t hit those marks and stays somewhere in the middle? As time goes on, if they’re “somewhere” in the middle they’ll be closer and closer to the 0.9162 SV% mark itself — if they stay that way you’ve found yourself a perfectly average decent NHL starting goalie. And who wouldn’t be happy with Devan Dubnyk?? Right.
In the end, this test feels most appropriate for ‘flunking’ people who string together improbably bad statistics after a certain number of games. You only get 82 games a season — playing a ‘likely’ bad goalie for any number of games more than you need to is imprudent — signing them to lucrative long-term deals is even more so.
Next in this series I’ll be examinig 2019 Free Agency and 2019-20’s starting goalies.