When you can tell your goalie isn’t a starter — Part II — Good Goalies

Yesterday I wrote about how you can apply a straightforward statistical test using the binomial distribution to accept or reject the notion that a particular goaltender has decent-NHL-starter talent as his career progresses. It turned out that Mikko Koskinen failed this test at game 37 of his career and then from games 56 and beyond — this led to the conclusion that he did not possess starting goalie talent (defined semi-loosely as 0.9162 SV%).

I’m sure some inquisitive people were reading that and wondering how other goalies fared under the same microscope. Surely, if actual good goalies were rejected by this test as possessing non-starting talent it would tarnish its conclusions. Similarly, if actual bad goalies were able to be rejected on the high side as possessing BETTER than 0.9162 talent, management might be boxed into a bad long-term contract decision based on this Type I error (or rejecting the null hypothesis when you shouldn’t).

Today’s post will delve into the former question — how do actual good goalies perform on this test throughout their careers?

Ground Rules

Firstly, I need to identify a cohort of actual good goalies to run through this test. Alright, so how are we going to define what a good goalie is? I thought that’s what we are doing with this test in the first place? Seem a bit circuitous. Yeah, probably — but bear with me.

At some point, we must be able to say that a goalie is good by some form of acclamation, right? Now matter how misguided an NHL franchise, they’re not going to run out a goalie for 5 years when there’s clear evidence he’s bad, right? (Unless he’s Ondrej Pavelec). So there must be something to calling some level of games played and some high standard of SV% as constituting someone who’s good. One season you turn around and Pekka Rinne is just Pekka Rinne: Good Goalie.

So I want to find this group of starting goalies who are uncontroversially good, or at the very least, uncontroversially not bad. A further condition is that I want goalies who became starting goalies after the 2012-13 lockout — in case there’s been some kind of sea change in goalie development or perhaps we could find evidence that goalies now do, in fact, improve over time. (ha!)

I’ve ended up building a ‘good’ cohort of 10 goalies that fit this description. At least 150 games played (most having 250+), at least 0.915 SV% over that span, who became clear starters after the last lockout. They are:

• John Gibson
• Ben Bishop
• Cory Schneider
• Sergei Bobrovsky
• Braden Holtby
• Robin Lehner
• Connor Hellebuyck
• Frederik Andersen
• Matt Murray
• Andrei Vasilevskiy

Let’s run our heroes through the gauntlet one-by-one and see how they do:

John Gibson

John Gibson would be in most people’s lists of the top 5 goalies in hockey today and his sterling performance since his rookie season shows it. He only briefly drooped below a 50% chance of being a decent starter after a single game in his career. After a brief dalliance in the green rejection zone in the first two games in his career, by game 164 he had firmly entrenched himself there. He never came close to being mischaracterized as a bad goalie by this test.

Ben Bishop

Ben Bishop had some interesting challenges in the first 50 games of his career, with a career-low measure of having an 8% chance of being a 0.9162 goalie at game 24. However, he never hit the red rejection territory and rocketed back up quickly to having over a 90% chance of being a 0.9162 goalie at games 79 and 91. He’s never actually hit our green zone, but on balance has maintained a level much closer to the good side. He’s had a better than 1/3rd shot of being an NHL starter every game of his career since game 45.

Cory Schneider

In a curve very reminiscent to Bishop’s above, Cory Schneider got off to a slow start in his NHL career, and would have been rejected as a non-NHL starter between games 7-9 of his career. This was short-lived, and after game 27 of his career he would never again fall under having a 33% probability of being an NHL starter. We had proof by game 65 of his career that he was better than a decent NHL starter (green zone) and he was basically pinned there for the next 300 games of his career, only recently falling slightly below.

Sergei Bobrovsky

Bobrovsky had a fairly hot start to his career, but had the latest meltdown of any goaltender in this cohort — his career game 75-100 stretch was disastrous, reaching a low of having a 7% probability of being a decent NHL starter at game 95. BUT — he never did reach our red rejection zone. After this, he’s steadily built up a resume to the point where he could be accepted as having better than decent-starter talent by game 348. He never again dropped below having a worse than 33% shot of being a decent starter after game 110.

Braden Holtby

After an early-career whipsaw that saw him dip into the red zone for games 5-6, Holtby quickly overcame that and by game 30 he never again dipped below having a 40% chance of being an NHL starter. He never hit our critical threshold for having better than 0.9162 talent (after game 1) but he’s come damn close a few times (peaking at 92% probability around game 282).

Robin Lehner

Lehner also saw some whiplashing action in his first 10 games, getting as low as having a 11% chance of being a decent starter by game 8, but by game 11 was back above having better than a 1/3rd chance and never again fell below that mark. In fact, he hit our green rejection zone evidencing having better than 0.9162 talent through games 32-38. He’s gone through some distinct phases of his career since then, but he’s very comfortably a good goalie.

Frederik Andersen

Andersen reminds me of John Gibson in that he’s maintained a high probability of being a decent NHL starter almost throughout his career. After being in the green rejection zone as late as game 6 of his career, he was within 1% of that green area again by game 37 of his career. He’s never had worse than a 40% chance of being a decent NHL starter.

Connor Hellebuyck

I wanted at least one goalie in this cohort that’s had to fight his way through multiple years of the “is this guy actually good” question. Just when you think his recent trajectory is leading you to some kind of relevant conclusion, this guy turns back the other way. The lowest he ever got was having a 12% chance of being a decent starter by game 80, but has spent the bulk of his career having at least a 1/3rd chance of being a decent starter. Today he’s just on the shy side of 0.9162, but he’s close enough to that that you’re almost equidistant from either alternative hypothesis of not being a 0.9162 quality goalie.

Matt Murray

Murray is going to be the shortest-tenured goalie in our ‘good’ cohort. He’s another full-career savant, only spending 4 games of his career having less than a 1/3rd shot of being a decent NHL starter. In fact, we were able to reject the null hypothesis to the green side as late as career game 30 and was over 90% probability as late as career game 62. He’s settled into a career of being ‘probably’ a decent NHL starter — nothing more, nothing less.

Andrei Vasilevskiy

Like Bobrovsky, Vasilevskiy saw a sophomore slump take him within proximity of our red rejection zone — by game 63 he only had a 18% chance of being an NHL quality starter — but this was a short-lived blip on an otherwise comfortable stay in the top half of this graph. He’s never been able to reject the hypothesis that he’s above a 0.9162 talent, but he’s gotten damn close a couple of times. Given the way this graph is trending I’d expect him to get there at some point.

Cohort vs Koskinen

If we throw all 10 ‘good’ goalies up on the same ugly-ass chart and compare them to Koskinen’s short career (thick black line), you get the above. What do you notice here — well, other than Koskinen’s trajectory clearly bifurcating from the good group at around game 30? I notice a couple of things:

• The latest a good goalie failed this test in his career was Cory Schneider at Game 9. No other good goalie falls into the red pit after game 9. Not bad. To apply this test in practice, I’d probably recommend avoiding at least the first 10 games of someone’s career.
• Bobrovsky comes the closest to failing at a late point in his career — he gets down to 7% probability at game 95. Next to this, Ben Bishop dropped to 8% briefly at game 24 of his career. This kind of validates our choice of alpha=0.05 (or our confidence level) — if we had been more lenient (say, alpha=0.1), we would have incurred some unforced Type I error.
• Every single one of the good goalies crested above 90% probability after game 15 of their careers at some point. Bishop didn’t get there until game 91, and Vasilevskiy didn’t do it until game 124 — but they all got there eventually at some point.
• 5 of the 10 good goalies crested 95% probability after game 15, meaning we could reject the hypothesis that they’re 0.9162 goalies and say that they’re better than that: Lehner, Murray, Schneider, Gibson, and Bobrovsky. Bishop is one good game away from getting there himself.
• Notice how the bulk of this group mostly stays above the 33% mark after, say, game 35? A few briefly fall below that mark afterwards, but most of a good goalies’ time is spent above that level. Let’s visualize that:

Good Lord that’s ugly. Sorry. I cut this off at 150 games since all of these good goalies’ lines just keep pew-pewing upwards after that. A few goalies overlap pinned at 100% — meaning they never dropped below 33%.

This simply asks: for how much of this goalie’s career so far have I been more than 33% confident that he’s a 0.9162 goalie? Of the good goalie group, only Bishop and Schneider had longer spells under this level of confidence than Koskinen at times, and both only briefly. By game 37, Koskinen drops below all of the good goalies and slid further beyond that — this dovetails nicely with my conclusion from the last post that we could have concluded he was not an NHL starter by game 37.

The latest goalie in our cohort to get above and stay above 50% of their careers having a better than 33% shot of being an NHL starter was Bishop, who reached that point by game 60. Schneider hit that level for good by game 40. Next is Lehner, by game 14. Good goalies spend most of their careers having a good chance of being NHL starters — and reach that point rather quickly. Koskinen never spent more than half of his career with a better than 33% shot — and if he ever gets there, will be the latest goalie to reach that level of excellence since the last lockout (by a huge distance). Looking above at where his curve is headed, I’m not confident he’ll ever get there (read: he won’t).

In my next post, I’ll be comparing Koskinen to a cohort of ‘bad goalies’.