Designing Rubrics for Consistency of Marking in Large STEM Classes

Context: In science, technology, engineering, and mathematics (STEM) subjects, project-based assessment hasbeen a key component of the assessment landscape, due to its authenticity and effectiveness as a learning approach.Problem: Large classes require marking of student assignments to be distributed among several markers. This raises theproblem of consistency among markers, who may interpret assignment requirements and marking scales differently.Approach: We tried two approaches to rubric design, in attempt to provide markers with clear guidelines for assigning marks,and thereby yield consistent results among markers. The first approach, employed to mark final reports submitted byengineering students in a capstone project class, used a 10-point qualitative scale that markers used to mark components ofan assignment. The second, employed in marking a series of deliverables submitted by students as a group project in aMaster’s level computer science project class, used “Yes/No” questions to assessed components of each deliverable. Thesewere then aggregated, and converted into scores on the same 10-point qualitative scale. Results: We found that bothapproaches facilitated speedy but thorough marks: the 10-point rubric allowed markers to mark an entire final report in 60minutes or less, while the “Yes/No” rubrics required between 10 and 20 minutes per deliverable, depending on size.Consistency was good for both approaches, with the “Yes/No” approach producing higher consistency at the expense ofgreater up-front effort. Also, while markers appreciated the speed of marking enabled by “Yes/No” rubrics, some werefrustrated that there was no middle mark that they could apply to certain components. The 10-point rubric was easier tocreate, but produced lower initial consistency, which then had to be resolved in post-marking discussions between markers.Conclusion: Both approaches enable acceptable consistency when different markers mark the same subset of studentassignments. The 10-point rubric takes little time to prepare and deploy, but requires more effort on the part of markers toproduce acceptable consistency. The “Yes/No” rubric is easier for the markers to use, but requires more initial effort tocreate. Thus, the choice between one or the other is a trade-off between when effort is available to expend on consistency.