Monday, March 05, 2018

Let The Computer Grade the Projects

It seems like a great idea – have the computer automatically grade student projects. Save all that boring work. Looking at all those projects is a pain. Why not have the computer grade them? It’s what happens in a lot of courses. Harvard’s CS50x online version does it so it must be great. In fact  a lot of huge courses in major universities use automatic grading software. What could be wrong about all of that? I confess that its a very tempting idea at times.

I’ve played a little with software looking at projects with some simple HTML projects. For rote stuff like making sure all tags are present and that open tags have matching closing tags is pretty straight forward. I still want to open the pages in a web browser to see what they look like.

Programming projects? I’m not so sure about automated testing for students. Mike Zamansky blogged recently about a session at SIGCSE (Sigcse2018 - Malloc Lab) where a professor had to deal with students gaming the grading system. This is one of my biggest concerns with automatic grading software. It is easily “game able.”

Getting to the destination is only part of the story. It’s pretty important to understand how students get the results they get. We, as educators, only learn so much by seeing the results of a program. We really need to understand how the programs process inputs and generate outputs. When we read student code we learn a lot about our students. We learn what they know and don’t know.  We learn how they implement algorithms. We also see what they understand and don’t understand. In fact we also get to know our students better. We learn their coding the way English teachers learn student's writing

Students benefit as well. We can give them a lot more feedback than automatic grading routines can give. We can also modify how and what we teach based on what we learn from reading their code. So for now I think I will put up with the extra work. The extra benefits are worth it.


Mike Zamansky said...

Auto graders can be used for simple things but you really need to look at student work. Tools that help guide you and make looking at students work more efficient on the other hand can be terrific.

I wrote a bit about grading a year or so ago: here.

Garth said...

If all we wanted to do was grade the ability to type code then this is a grading solution. I could care less if a student can type code. The student could have his mother type it for all I care. I want to know if a student can design/develop a program. Can they start with a project, i.e. design and write a program that does a task, and then build the inputs, outputs and all the stuff in between. A programming course should not be a syntax and typing class. It should be a design class. Auto grades will not evaluate that.

Shriram Krishnamurthi said...

Our upcoming research paper (work led by Jack Wrenn) points to numerous methodological flaws with auto-grading that call the entire process, as conventionally followed, into question, but also shows how it can be made far more reliable. Odds are most people who are doing it now are doing it wrong even setting aside philosophical and moral issues, just on the technical front.