Assessing Writing Differently: Comparative Judgements

What if there was a way to assess that reduced our workload?
What if this assessment was also more accurate?
​And what if it improved the cohesion and culture of moderation?

It sounds too good to be true!

But this has been my experience with using comparative judgement to assess students' writing in a national project.

What is Comparative Judgement?

Comparative judgement is simply the process of comparing two pieces of work and identifying which one is 'better'. This means that we are making a relative judgement, rather than an absolute judgement.

"If someone walks into the room you're in at the moment and I say, 'How tall is that person?" That's and absolute judgement. If two people walk into the room you're in and I say to you, 'Who's taller? The person on the left or the person on the right?' That's a comparative judgement."
​- Daisy Christodoulou on Filling the Pail Podcast (Sept 30, 2021)

When I first heard this example, the skeptic in me thought 'If someone walked in the door, I reckon that I'd be able to work out their height.' 
Then I took a moment to work out how I would go about determining their height and realised that I was relying on the fact that I'd have the door frame as a reference point. Thus proving the point that comparative judgements are much easier to make then absolute ones! 

So the premise of comparative judgement is that teachers will look at two pieces of writing and select the one that is 'better'. This is completed numerous times until the pieces can be reliably ordered. And, thanks to technology, this can be achieved quickly!

Assessing Writing in Australia

Somehow, I stumbled across the Assessing Writing in Australia project run by No More Marking and Templestowe Heights Primary School. Our school signed up and our Grade 3 students duly sat the task early in Term 1, along with 1200 of their peers across 24 other schools. The amazing dynamo, Jeanette Breen outlined the project in a series of articles for Teacher Magazine (here & here).

The writing task was well set up. The stimulus was interesting, with lots of scope for different narratives. The process of setting students up on the No More Marking page was simple. Student pages were printed, and afterwards they are easily scanned and uploaded.
I appreciated the timing of the task, as it would give us a snapshot of our students after an incredibly disrupted 2020 thanks to multiple periods of remote learning.

About a week later we were ready to 'mark' the writing. Given the smaller size of our school, and the novelty of the process, all teachers  participated in the marking session. We sat in the staff room during our regular staff meeting, each with our own computer. We started selecting which piece of writing was better. Some judgements were very easy to make. Others took more thought.
I think it's important to note that we were never asked to judge one of our students pieces against a piece from another school. (I've now found out that 80% of judgements were our students' work, 20% of judgements involved students from other schools).
It only took us 20 minutes to complete all of our assigned judgements!
In this time I had reviewed the writing of most of our Grade 3s. I had also seen the writing of students from other schools. I can't remember the last time I had the opportunity to see writing from other schools!

A week or two after, we got the results-  a much quicker turnaround time than NAPLAN. Our students pieces were ranked against those of over 1200 others. We were provided a 'writing age' for each student. This data was fascinating as quite a few of our students performed better than we expected. This allowed us to examine our biases and how they could influence our decisions when marking using a traditional rubric.

At our next staff meeting, we laid out every piece of writing in the order they had been judged. Looking at the pieces laid out, it very quickly became obvious what factors where determining the final score. Some students used many run-on sentences, others were starting to use conjunctions accurately, and others showed a high level of control over sentences. Vocabulary was another major component. Interestingly, handwriting seemed to have less importance in the overall marking.

This session provided a myriad of rich discussions.  There was none of the typical defensiveness that can emerge in a traditional moderation session, when teachers can feel compelled to justify their gradings. The conversation was no longer focused on the teachers' judgements, rather it was focused on the students' writing.   

Participating in the Assessing Writing in Australia project provided our Grade 3 teachers with clear understanding of how to improve their students' writing. It also provided important information for the teachers of P-2, as we could clearly see what we'd like our students to be writing. There were also takeaways for our upper primary, who noticed an absence in interesting vocabulary and have been explicitly teaching this since.

In Term 2, we conducted a school-wide comparative judgement task for our students in Grades 1-6. We set a common task. Populating our students onto the No More Marking website was straightforward & then student response sheets were easily generated. I was able to do this in under an hour. Our task followed similar guidelines to those in the Assessing Writing in Australia task, and we conducted our task in the second week of Term 2. 
At the end of the day, teachers gave me their students work. I scanned it and uploaded it to the website. This took me less than 15 minutes!

Setting up the judging session was just as simple. Teachers were allocated a quota of judgements to make.
The judging session could be asynchronous, but we dedicated time in our staff meeting to complete (and to sort out any teething issues). I needn't have worried as the judging session was as uncomplicated as the rest of the process. Less than 30 minutes later we had collectively assessed the writing of every student. As teachers made each judgement I was able to observe how the students' writing was being ordered. Other data is provided including the strength of teachers' agreement in judgement, how many comparisons had been made, and (my personal favourite) the % of time a teacher chose the left hand piece.

The results were ready as soon as our judgement session had been completed. Teachers were able to view them immediately. At our next staff meeting we laid out the writing samples and looked for themes. Once again, we found this incredibly useful and led to another 'no-judgement' moderation session. I use the term 'no-judgement' because individual teachers didn't have to justify why they gave a certain mark. The scores had been allocated in a collective process, and again we found that we were able to dive into what elements of the composition had led to the mark. This in turn led to more rich discussions about how we can improve our students writing.

Our staff are keen to use comparative judgement as part of our assessment of student writing. We are also looking forward to the next round of Assessing Writing in Australia.

And the great news is that you can dip your toe in the waters of comparative judgement too! Check out www.nomoremarking.com/products/awa to find out more about participating, whether you're primary or secondary!
There is an information webinar on the 26th of October to learn more.
It'd be great to get more schools involved!

Photo by Sangga Rima Roman Selia on Unsplash