≡Score for Score

Today, we’re going back into the world of academia to discuss another study on how gymnastics is scored. I’d like to talk about a 2012 article by Alexandra Pizzera entitled “Gymnastic Judges Benefit From Their Own Motor Experience.”

This study tests the hypothesis that judges evaluate skills more accurately when they can perform that skill themselves - in technical terms, when they have “specific motor experience” (SME) with that skill. Because judges have to get a lot of information from a very quick movement, those who have done that movement are better prepared to do so.

This question is actually quite relevant to the ongoing conversation in the gymnastics community about who should be allowed to take the brevet judging test. USAG’s requirements are remarkably strict, and usually only former national team members are invited to take the test. I had assumed that this was largely a way to keep the elite circle small, and maybe it that’s true — but it’s also worth asking whether being an elite gymnast truly does make you better qualified to judge elite gymnastics.

The Experiment

To test her hypothesis, the author asked 58 gymnastics judges to score the angle of 31 different split leaps on beam. 22 of the judges were former gymnasts who reported that they could perform the skill themselves; the rest could not.

The judges were asked to judge whether the gymnasts hit the two angles required to get credit for the leap, taking the role of a d-panel judge. They were also asked to mark the total of all the deductions that could be taken for body shape, etc., taking the role of an e-panel judge. Each of the split leaps had been scored by an expert judge in slow motion to create a “correct” reference score with which to compare all 58 participants’ scores.

The results are shown in the figure above. The left-hand side of the graph, shaded in grey, shows that judges who could perform the skill themselves correctly identified whether a skill should or shouldn’t be credited much more often. They were correct on their judgements of Angle 1 77% of the time, while judges who couldn’t perform the skill were correct just 67% of the time. Similarly, the judges who could do the skill were right about Angle 2 87% of the time, compared to 82% for other judges. These differences are statistically significant at α = 0.05.

The same pattern holds for the e-scores. The author measured the difference between each judge’s total deductions on a skill and the reference judge’s total deductions on each skill as a percent of the total reference deductions. Judges who could do the skill were closer to the reference score: they deviated by an average of 80%, while the other judges deviated by an average of 97%.

All told, this evidence is pretty strong support for the original hypothesis: judges are more accurate when they’ve performed the skill themselves.

In Review

The findings in this study make a lot sense to me: former gymnasts understand the mechanics of a skill a little better than those of us who have only ever been observers. Of course, because there’s no way to randomize which judges can perform a split leap, these results aren’t causal. Maybe all that matters is that you spend 4 hours a day in a gym before the age of 12, even if you’re just sitting there - we can’t know for sure. But I’m basically convinced that gymnastics experience really helps a judge out.

This study also provides some evidence to back up policies that allow former elites to skip the early levels of judging on the path to becoming a brevet judge. While such policies already exist in at least the US and the UK, and other countries should consider implementing them as well.

I do have to wonder if the results are generalizable - this study doesn’t just test a single event, it tests one skill on one event. Things might look a little different when we take into account the full range of gymnastics that judges are asked to score.

Finally, I want to emphasize that these results in no way suggest that only former gymnasts should be allowed to judge the sport. There were certainly some non-gymnasts who judged more accurately than some former gymnasts in this experiment, and either way the last thing we want to do is shrink the pool of potential judges. But hopefully these results will encourage more former gymnasts to stay involved in the sport by becoming a judge. Or, just maybe, this paper will encourage some judges to try out an adult gymnastics class!

Check back next week for another review!

Tags: From the Academy