Students and faculty raise concerns over automated test proctoring

Since Miami University's contract with the online test proctoring service Proctorio began in 2017, some students have raised concerns about its effectiveness and security.

Proctorio is an online test proctoring software that uses machine learning and artificial intelligence to track students' activity while taking exams. This is done by monitoring them through their webcam and the activity on their laptop.

Tamise Ironstrack, senior lecturer of Spanish, has been teaching online courses over the summer for eight years. For the first few years, she had no way to monitor students online. She noticed students’ test averages were higher than they would be in person.

Eventually, Miami began using a software called Respondus LockDown Browser, which locked a student's computer to a single web page while taking a test.

"You could see right away that students' averages came back to normal, closer to what you would see in an in-person exam," Ironstrack said. "[I was] a fan of it in the sense that it seems to encourage students to do their own work, especially in foreign languages, where we have a lot of online translators, and it's really, really easy to use an online translator to do the work for you."

For many professors, there isn't another option for automated proctoring. Because of Miami’s contract with Proctorio, it's the only software readily accessible to them.

“We don't have a lot of choice as instructors over what we use,” Ironstrack said. “Miami says, ‘This is the product we paid for, so this what you should use.’”

Ironstrack said she noticed some students complaining about the remote proctoring system, but she’s beginning to understand their concerns.

James Porter is an English and Emerging Technology in Business and Design (ETBD) professor who has taught courses on digital ethics at Miami. He decided against using Proctorio in his classroom due to concerns over the ethics of the company and software.

Porter said one of his main concerns is the flawed nature of face and gaze detection software as a whole.

"The software ... is highly inaccurate," Porter said. "It's making judgments about people based on data that then becomes data for new judgments."

While fears of bias in face detection tech are prominent, Proctorio CEO Mike Olsen said the company has been working to eliminate these problems within its software. Over the past year, he said Proctorio has brought in third-party audits multiple times for guidance in ethics, bias and accomodations.

Enjoy what you're reading?
Signup for our newsletter

"The usage of the platform really drove that we did,” Olsen said. “We [proctored] over 20 million exams last year, all across the world. So that's a lot of diverse cultures, a lot of diverse skin colors, backgrounds and everything in between. And I don't see that slowing down, so we have to ensure that the software gets better and continues to be better. And then we can be more transparent about what those outcomes are."

Erik Johnson, a first-year computer engineering major, worked with ASG to prompt the University Senate to begin an investigation into the use of remote proctoring software after posting a series of tweets in September claiming that Proctorio's code included multiple strings referencing “live agents," which he believed meant people as opposed to the A.I. that Proctorio advertises the use of.

In response to the tweets, Proctorio filed a copyright takedown notice against Johnson under the Digital Millennium Copyright Act (DMCA), claiming he had illegally posted Proctorio's code.

While Johnson's lawyers said this is a clear example of fair use, Olsen said Johnson had included Proctorio's source code in the since-removed tweets, which could create security risks and allow students to circumvent the software more easily.

In response to Johnson's claims that Proctorio is not fully automated, Olsen said the "live agents" are not used by Miami, but are instead for commercial organizations who want to actively monitor test-takers. In most situations, Proctorio encrypts recordings before they are sent out, and only the university has access to the "key" to unlock the encryption.

"It's University-controlled data that we don't have," Olsen said. "As a result, all the data we store, all the video recordings, no one at our organization can watch them. There isn't a system admin anywhere within our organization who could log in and just start watching videos."

As a private corporation, Porter said there are countless concerns over how Proctorio uses collected data.

“[My concern is] that third-party services are collecting data, selling data, exchanging data, and there aren't really any strong controls, and we certainly don't know very much about how people are using the data,” Porter said. “[Proctorio] says they aren't selling data, but there's ways that data is shared that isn't directly selling, per se.”

Olsen said Proctorio has never sold or processed student data. Instead, its entire revenue comes from selling the product to different organizations, and all of its identification processes are through the students' learning management system – in Miami’s case, Canvas.

"We've always set very, very strict standards," Olsen said. "We never created a registration system, for example, and only use a Single Sign On (SSO) through the learning management system. The reason we do that is so we're not collecting email addresses, we're not collecting passwords, we're not storing those kinds of things. That's intentional by design."

When Annie Dell'Aria, an assistant professor of art history, first heard about Proctorio, she said it "just kind of bugged [her]."

Her primary concerns with Proctorio are over security, student comfort and issues for students with disabilities.

She worries students, especially those with disabilities or with concerns about remote proctoring, could both be made uncomfortable and perform worse on tests that use the software.

"I think that it's not just the culture of surveillance and the issues of that, implicating young people that they have to feel like they're being watched all the time,” Dell’Aria said. “That is a problem, but also the idea that we're letting third parties – for-profit companies – to have access into people's homes," Dell'Aria said.

Because of this, Dell'Aria has been working to create assessments for her classes that don't require remote proctoring. While this is an option for Dell'Aria, she acknowledged that it's often not for professors whose courses are structured differently.

For courses with large class sizes, or courses like Ironstrack's where students are tested on their ability to remember and apply concepts, it can be much more difficult to create tests that adequately assess students, prevent academic dishonesty and do not put a considerable workload on the instructor.

To prevent these issues of discrimination and over-proctoring, Olsen said Proctorio has been working to both give more control to professors and provide more training for the software, including a "Proctorio Academy" training course.

"There are aspects of this that you really need to understand before you turn it on and use," Olsen said, "and there are absolutely applications for them. If you have a nursing exam, where these people can literally kill people if they don't know what they're doing, you can't have wild cheating going on. And in those cases, yes, even sticky notes on the screen or a roommate behind the screen. Those are concerns and room scan is a perfectly valid setting. But a weekly exam for an art student? I don't know why you would need that. It's hard to justify."

First-year Wes Payne, ASG on-campus senator, has worked with Erik Johnson and Jannie Kamara, the outgoing student body president, to pass two bills in University Senate: One advocating for faculty to be trained in using Proctorio and another advocating for impartial investigations into the use of remote proctoring services at the university.

"I will say that the support from administration has been somewhat surprising, in a good way," Payne said. "I guess I wasn't anticipating them to be so receptive to it, but I think that given everything that's come out about not only Proctorio but remote proctoring services in general, they're worried that it could reflect poorly on the school, which is something that I've been worried about."

The bill for an investigation into remote proctoring, which was presented to University Senate on Feb. 22, was intentionally kept vague. By asking for an investigation of "remote proctoring services," Payne said the university would be looking at remote proctoring as a whole as opposed to only focusing on Proctorio.

Payne does not plan to stop addressing the use of Proctorio at Miami.

"I think it's important to keep in mind that Miami began their contract with Proctorio in 2017, and has been using the service since then," Payne said. "All of these [issues] have existed since 2017, we just noticed them more because we have to use [Proctorio] more, because of COVID."

Similarly, Olsen said he is willing to have conversations with the university and critics of the software to work on improvement.

"There's this notion that we're anti-critic, and I just want to make sure that's not true," Olsen said. "Critics make our company and our software better. We want to engage in those criticisms and in those sorts of conversations. If these kinds of things need to happen in a forum, absolutely. Organize it, have the school reach out to our success team manager, and we'll get things set up."

But even with improvements and changes, many professors like Dell'Aria feel the software is just not for them.

“Personally,” Dell’Aria said, “I think you have to start from an understanding of your students as people, not as potential infractions.”

robbinha@miamioh.edu

Correction: A previous version of this article incorrectly stated Proctorio used facial recognition software. This article has been updated to reflect that Proctorio actually uses face detection and gaze detection technology.