≡Score for Score

It's time for another post in which we dive into academic research on scoring in gymnastics. Today, I want to talk about “Judging the Cross on Rings” by Plessner and Schallies (2005).

This study looks at judges’ ability to recognize “shape constancy,” which is cognitive psychology language for the ability to recognize the same form from different angles. Specifically, the authors are interested in the iron cross on rings, which is judged based on the angle between the gymnasts arms and his body. It’s easy enough take the right deductions when you’re facing the gymnast head on, but it's a lot harder from other angles, as illustrated in the figure below.

Of course, a gymnast’s score shouldn’t depend on where the judges are sitting, and we’d all like to think that the training process teaches judges to take the correct deduction no matter what angle they have. But do they really? That’s the question that this paper investigates: are judges better than normal people at identifying the correct angles in an iron cross when we switch up their view of the gymnast?

The Experiment

To answer this question, the authors showed pictures of gymnasts holding an iron cross to a set of gymnastics judges and a set of laypeople. Each iron cross was photographed three times — head on, from a 30 degree angle, and from a 60 degree angle — and participants were given one of the three pictures of each iron cross. Participants were asked to judge the size of the deduction - 1 to 15 degrees, 16 to 30, and 31 to 45, which correspond to the deductions in the MAG code of points.

Unfortunately, both groups’ accuracy declined as the angle of the shot got worse. While the gymnastics judges took the wrong deduction 38% of the time when showed a head on shot for one second, they were wrong 44% of the time when showed a shot form a 60 degree angle. And the size of the increase in their error rate - that is, 6 percentage points - was almost as large as that of the laypeople, who performed 7 percentage points worse on the 60 degree shots than on the head on shots. The table below shows the error rates for laypeople and judges, separated by the angle of the photograph and the length of time that the photograph was shown for.

I should note that judges’ error rate when looking at ideal head on photographs — 38% — seems pretty high to me! About four fifths of the judges who participated in this study had more than ten years of experience judging, and they still only got the deduction right 62% of the time? The more I read these studies, the more curious I am about the progress toward robot judges.

The literature also suggests, unsurprisingly, that our ability to achieve shape constancy worsens when we are paying less attention to the shape. The authors therefore consider the possibility that counting the duration of the iron cross - another key component of judging rings - might make it harder for judges to correctly identify angles. To test this, they flashed the pictures of the iron crosses for different lengths of time and asked participants to judge the duration as well as the angle. Thankfully, this hypothesis was not born out — the professional judges, experienced in multitasking, performed no worse when evaluating the duration alongside the angle.

In Review

The evidence is in: angle matters. I’m sure any judge would be happy to back this up: we hear judges talk about angles all the time in response to fans’ critiques of the accuracy of their scores.

The FIG has guidelines for where judges ought to sit in relation to the apparatus, though both the MAG and WAG codes stipulate that "variations in the seating arrangement are possible depending on the conditions available in the competition hall." (Thanks to @romacastellini for pointing me to this part of the code.) MAG E-score judges are supposed to sit at wildly different angles with wildly different views of the gymnast - check out the following illustration from the 2017-2020 code of points.

This study tells us that such an arrangement is likely to create a situation in which different judges take different deductions for the angle of a gymnast's arms on his iron cross.

Now, if every gymnast at a given meet is being judged by the same judges sitting in the same positions, then the ranking will likely ultimately still be correct. But I could see this being a big issue in situations where the judges' seating arrangements are not so standardized, and where scores need to be compared across different meets - which sounds a lot like NCAA gymnastics. Of course, this probably isn't high on the list of things to address in NCAA judging, but it's something to keep in mind.

Anyway, that's all for this trip into academia! Stay tuned for more!

Tags: From the Academy