≡Score for Score

I'm excited to share the results of my recent survey on inconsistency among US elites. Amazingly, 173 of you responded in just 11 hours -- I owe you a big thank you! (But like, am I really surprised that so many gymnerds jumped at the chance to share an opinion on gymnasts' consistency? Nope.) Particular thanks also to those of you who shared and retweeted!

Of course, "the people who happened to see this on Twitter and had the time and inclination to respond" are not a representative sample of the gymternet at large. Their opinions also probably don't represent the impressions, of, say the national team staff. It's just an opt-in online poll, and should be treated as such.

I asked one question about ten gymnasts: "Does this gymnast have a reputation for inconsistency?" Here are the raw survey results.

More usefully, the chart below shows each gymnasts' inconsistency reputation rating: of those respondents who answered yes or no, what percent thought, "Yes, this gymnast has a reputation for being inconsistent"?

Of course, we shouldn't just ignore the fact that some respondents indicated that they didn't know certain gymnasts, and others simply left some questions blank. This creates the possibility that the responses for lesser-known gymnasts would not be comparable to the responses for famous gymnasts: the former come from a narrower set of fans who may have a different view of gymnasts' reputations.

To check for this, I re-calculated the proportion of "yes" responses for each gymnast using only the responses from people who ranked every gymnast. None of the proportions changed significantly, so I felt good using the whole dataset going forward.

However, there were a handful of responses that seemed totally out of line with the others. I don't want to discount anyone's opinion - for example, you could argue that Simone Biles is inconsistent (#ThatDamnBarani). But when someone responds in opposition to the consensus on every gymnast with a clear-cut reputation, I have to think that a) they're trolling, or b) they misunderstood the question and accidentally marked which gymnasts have a consistent reputation instead of an inconsistent on.

I want to assume the best in people, so I did some post-survey data cleaning. For respondents that marked Biles as inconsistent and McCusker as consistent, I changed all their yes answers to no, and vice versa. It's fair to argue that Biles is not consistent and it's also fair to argue that McCusker is, but I think we can all agree that Biles is more consistent than McCusker. It follows that this particular combination of responses is a real red flag that something is wrong.

Below are the results after this alteration. This version of the data is what I present in the companion blog post.

If you haven't already, go ahead and check out my analysis of these results. Thanks again to all who participated!

Tags: Survey Says

As we draw closer to Worlds team selection, one question is being asked more and more: "Is she consistent?"

I've spent a lot of time thinking about how to measure consistency in gymnastics, and I've developed a metric for doing so. You'll see these consistency stats at the top of the page for every gymnast in my database who has enough information.

You can find the details behind this metric here, but there are two things that are important to understand. First, lower consistency stats are better; higher stats indicate a more inconsistent gymnast. Second, the consistency stats attempt to measure how well a gymnast hits in competition. That means we're looking at execution deductions and neutral deductions (out of bounds, over time, etc.) and not difficulty.

Below, I've pulled the current all-around consistency stats for ten top US elites. These draw upon all the scores in my database from the past year for which I have d-score information.

But with so much conversation around consistency ahead of team selection, I wanted to understand how these data-driven consistency metrics compare to the talk going 'round the gymternet. To do this, I shared a survey on Twitter asking fans to mark which gymnasts they think have a reputation for being inconsistent. The respondents are by no means a representative sample of any population -- but they are awesome group of 173 people who helped make this post possible.

The chart below shows the gymnasts' inconsistency reputation rating: of those respondents who answered yes or no, how many thought, "Yes, this gymnast has a reputation for being inconsistent." For the raw survey results and more details on the survey methodology, see this companion post.

Already, we can see some differences between the Score for Score consistency stats and the gymternet's inconsistency reputation rating. To visualize the differences, I plotted both metrics on the same chart.

As we'd expect, gymnasts with a higher consistency stat tend to have a greater reputation for inconsistency amongst the gymternet. The correlation is 0.38, indicating that 38% of the variation in gymnasts' inconsistency reputation ratings can be explained by the variation in their consistency stats.

The correlation is lower than I expected, largely due to two gymnasts. In particular, practically no one would ever consider Simone Biles an inconsistent gymnast. And she's not inconsistent - in terms of her win record. However, when we drill down and look at her execution scores and neutral deductions, her results are sort of all over the map. And she might have ended up atop the podium, but she did have a very off day at Worlds last year. It all shows up in her scores.

Aleah Finnegan is also statistically less consistent than her reputation would suggest. I'm guessing that that's largely because she's new on the elite scene and many fans haven't been following her performances much until recently. Plus, there's probably a "my sister is Sarah Finnegan" bump when it comes to having a reputation for consistent hits.

Of course, the reputations might be capturing important information that the scores themselves leave out. Which gymnasts came through in competition but looked scary in podium training? Which ones weren't put up in the team final because they weren't hitting in practice? The scores don't directly reflect any of this information.

But as we look towards the Worlds team selection, it's worth considering the numbers when we talk about a gymnast's consistency. You can't always believe what you read on Twitter.

Tags: Fun with Score Data, Survey Says