What is the best way to uphold academic integrity? Most professors and administrators incorrectly think that online proctoring and other technology-based tools (i.e., lockdown browsers) designed to stop the act of cheating are the best ways to uphold academic integrity in online assessments. It may seem like the right answer, but it’s not! Students know how to get around such technology. As a consequence, the level of cheating has increased dramatically as universities have transitioned more and more to online assessments. The best approach to maintaining academic integrity is through better assessment design using randomization techniques.
At EXAMIND, we have spoken to a lot of professors and administrators about this topic. There seems to be a lot of confusion and misunderstanding about randomization and its effectiveness. In this easy to follow guide, we will show you how using appropriate randomization techniques is one of the best assessment design strategies to maintain academic integrity. If you are not familiar with dynamic questions, we recommend that you first read our Dynamic Questions: The Beginner’s Guide before reading this post. It is critical that you understand the concept of a dynamic question in order to grasp the power of randomization.
Let’s get started with understanding what randomization means.
Randomization is a statistical term that refers to the process of making something random. The term random just means that each item of a set has the same probability of being chosen. If something is truly random it has the same chance of being selected as everything else. When I flip a coin, I have a 50% chance for heads and 50% chance for tails. It’s random.
Randomization, when appropriately applied, can eliminate the advantages students gain from cheating. Randomization is critical for maintaining academic integrity for 2 main reasons:
1) Minimizing the advantage of group collaboration
2) Making it hard to find solutions online
Randomization makes collaboration difficult because students receive different versions of the same question and/or in a different order. If students receive a question with randomized parameters, it is harder for students to find the exact question and answer posted online.
Then the natural question is, what should be randomized and how much? Let’s begin with discussing what should be randomized. We will then consider how much randomization is needed to eliminate the advantages gained from cheating.
When creating an assessment instructors should randomize the following:
1) the order questions are served to students
2) the questions served to students
3) the variables within those questions (dynamic questions)
Each of these randomization techniques used in isolation improves academic integrity but varies in their effectiveness in eliminating cheating. The most effective approach is to use all three techniques simultaneously.
We will discuss each randomization technique in turn and describe how instructors implement them in their assessments. We will begin with the most common technique, which also happens to be the most inefficient. We will then consider how using question pools improves upon the most basic approach but is still not as effective as randomizing both what questions are served to students and the variables within those questions.
The first technique is the most basic randomization strategy. Is it good at minimizing cheating? Not really! But it is better than doing nothing. In this technique, instructors randomize the ordering of their questions. The major flaw in this approach is that students still receive the same questions just in a different order. We often refer to this as creating exam versions.
Creating versions of your exam is most effective when the assessment is administered in-person because it is more difficult for students to look at their neighbors' work when the ordering of questions is different. In the online setting or in asynchronous in-person assessments this approach is less effective. If there is collaboration between students they can easily figure out what the corresponding question is or share with other students what was on the assessment. If exam questions are compromised (e.g. posted online), then students have access to the questions. It doesn't really matter what the order of the questions were, if the questions are compromised. This approach is particularly vulnerable when instructors are using third-party test bank questions or not updating their questions each term.
In summary, this randomization technique is not very effective at preventing cheating. Given these limitations, instructors start to consider randomizing different questions within their assessments. This brings us closer to the next technique we will discuss, the use of randomized question pools to serve students different but similar questions.
First of all, what is a question pool? A question pool is a collection of questions that an instructor wants to group together based on specific criteria. A common practice is to group questions based on the same topic or concept. For example, an accounting instructor may create several questions related to net income and group them together in the same question pool. An instructor could further group them by level of difficulty (easy, medium, difficult). You get the idea!
To make it more clear we will illustrate the concept of question pools visually:
The solid line box represents a question pool. Each grey box within the question pool represents a unique question that was manually created by the instructor.
Now that you understand what a question pool is, what do we mean by a randomized question pool? A randomized question pool is a question pool where the question or questions within the pool are served to a student randomly. When creating an assessment, learning management systems (LMS) generally have a feature that allows instructors to create question pools from questions that the instructor has already created in the LMS. When determining what questions to put into an assessment, the instructor can tell the LMS to pick randomly a certain number of questions from the instructor’s question pool to include in the assessment, hence the term randomized question pool.
Again, let’s look at this visually to make it more clear:
Here we are telling the LMS software to randomly pick one of the four questions in the question pool, represented as the red box. In this example each question has a 25% chance of being served to the student. If we had told the LMS software to select two questions from this pool, each question would have a 50% chance of being served to the student. The more questions you have in a randomized question pool and the fewer questions you choose from the pool to be served in an assessment; the more you increase the level of potential randomization. In other words, it will lower the probability that students will receive the same question on their assessment.
Serving students questions from a randomized question pool makes it harder for students to collaborate on their exams. However, if there is widespread collaboration by students or if questions are compromised, then students who cheat will still have an advantage.
Industry-leading research from the University of Illinois suggests that using randomized question pools is not enough to eliminate the advantages gained from collaborative cheating in asynchronous exams. Their research suggests that the best approach is to couple randomized question pools with randomized variables within each question. So let’s go into the details of this approach.
The best approach to eliminate the advantages gained through cheating is to use both randomized question pools and dynamic questions. The key point here is that we are including dynamic questions within randomized question pools. Just as a reminder in case you haven’t read our other Beginner’s Guide on Dynamic Questions, a Dynamic Question is a question that allows variables within the question to be randomly changed based on what you specify. In other words, it is a question with randomization built into its structure. You can randomize numbers, formulas, words, phrases, or anything else you can think of. Each dynamic question can have thousands of variations. At EXAMIND, we call each new variation of the question a new question variant.
Now let’s take a closer look at how using dynamic questions within randomized question pools works. We will build off our previous example of a randomized question pool with four questions. In such a pool, the instructor writes 4 separate questions and the student is served one of those questions. This is represented visually as follows:
Recall, that in this scenario, there is a 25% chance that this question is served to the student. But what would happen if we turned each one of these questions into a dynamic question that automatically created hundreds if not thousands of variants of the same question? How would that affect the probability that a student receives the same question?
By introducing dynamic questions into randomized question pools we can reduce the chance of a student receiving the same question as another student to almost 0%. In a randomized question pool with four dynamic questions the student is served a random new question variant from one of the randomly selected four dynamic questions in the question pool. Now the probability of the student receiving a unique question approaches 100%.
Once again let’s take a look at what this looks like visually:
The dynamic question within the randomized question pool (highlighted with the dark blue perimeter) can now be turned into many different new question variants represented by the small blue squares in the lower rectangle.
Each student will receive one of these new question variants, represented by the small red box below:
Now the student is served 1 of the 4 dynamic questions but receives one of the 1,000 or more possible variants of the question. Now it doesn't really matter if students try to cheat. When each student receives a unique question variant it makes it nearly impossible for students to collaborate because they have different questions.
To summarize, applying each randomization technique simultaneously is the best approach to maintain academic integrity. Now we need to identify the appropriate level of randomization.
Fortunately, the same research out of the University of Illinois has investigated this question. Based on their research, to eliminate the advantages gained from collaborative cheating in asynchronous exams randomized questions pools need to include at least 4 dynamic questions. In their study, only when at least 4 dynamic questions were included in each question pool were assessment scores between cheaters and non-cheaters not statistically different.
In summary, good assessment design is not only about what you randomize but also how much you randomize. Assessments need to have both randomized question pools and dynamic questions within those pools to eliminate the advantages gained from cheating. You can’t just do one and not the other. Hence, the best defence against cheating is to create randomized question pools with dynamic questions and to include at least 4 dynamic questions in the pool.
Thanks to this easy to follow Randomization: The Beginner’s Guide, you now have a better understanding of the concept of randomization and how it can be used to create better assessments. You know what needs to be randomized and how much is required to eliminate the advantages gained from cheating.
Recall that most instructors and administrators think that online proctoring and lockdown browsers are the best difference against cheating. In our guide we have shown you that the key to combat cheating is better assessment design. You should now have more clarity on how to start using randomization techniques to better maintain academic integrity.
Remember you don’t have to go about this alone. At EXAMIND, we are ready to help you start building dynamic questions and randomized question pools in a fraction of the time. Great assessment design is easier than you think and our mission at EXAMIND is to make it as easy as possible for you. Why? Because we believe that academic integrity can be made even better when we apply best practices based on proven academic research.
