Exams are part of the assessment processes that benchmark individuals’ educational progress and should be conducted in a way that promotes learning outcomes and upholds academic integrity. Ensuring academic integrity within online examinations has become a chief concern for educators. One such way of safeguarding academic integrity is by adopting methods to mitigate rampant breaches of the online examination procedures (Balasubramanian, DeSantis, & Gulotta, 2020; Dendir & Maxwell, 2020; Fask, Englander, & Wang, 2014), that were primarily developed due to the COVID-19 pandemic confinement (Clark et al., 2020; Dicks, Morra, & Quinlan, 2020; Jacobs, 2021). An underdeveloped sense of academic integrity and lax/absence of deterrence enforced by the educational institution preparing the examination can be a principal reason for cheating among students (Lang, 2014). Online examinations misconduct is accessible due to lack of faculty observation and prevalence of the internet – facilitating fact (i.e., answer) searching, especially if the actual examination questions were already available and gathered from online sources (Burgason, Sefiha, & Briggs, 2019; Kennedy, Nowak, Raghuraman, Thomas, & Davis, 2000).
In order to mitigate examination misconduct and question-sharing, a few educators suggested using proctoring technologies, such as webcams and microphones, to track and record students during the examination. Despite the effectiveness of such proctoring technologies in alleviating academic dishonesty during online examinations, they have limitations that are considered demanding in terms of not only cost and technical requirements, but also in terms of social and psychological implications on students (Karim, Kaminsky, & Behrend, 2014; Kharbat & Abu Daabes, 2021; Nigam, Pasricha, Singh, & Churi, 2021). Thus, to circumvent these drawbacks of procuring technologies, educators indicated the designing of examination questions to mitigate cheating and answer-sharing. Suggestions involved developing examination questions using open-ended questions or take-home examinations as effective solutions (Bengtsson, 2019; Schmidt-McCormack, Fish, Falke, Lantz, & Cole, 2019). These questions involve higher levels of student-thought and analysis, resulting in differing answers, enabling the instructor to analyze text-matching (similarity indexing) to safeguard academic integrity. Nevertheless, concerns are associated with examiner bias, thus offering a legal argument by a non-passing-graded student. Moreover, students can compromise the integrity of written essays (Bengtsson, 2019; A. M. Elkhatat, K. Elsaid, & S. Almeer, 2021b; Schuwirth & Van Der Vleuten, 2004).
Other suggestions include the development of examination questions ‘from scratch’, or paraphrasing a question that could prevent searching for questions (and related answers) online (A. Elkhatat, K. Elsaid, & S. Almeer, 2021a; Golden & Kohlbeck, 2020). Although this approach appears practical, students can breach the examination procedures by sharing the questions and answers with their classmates, searching on tutoring websites (e.g., Chegg) (Lancaster & Cotarlan, 2021; Steel, 2017), or hiring on-demand independent experts (e.g., tutors) to help students online. However, students mostly resort to sharing examination questions rather than tutoring services as tutoring services can have a different approach to solutions than what is taught in the class, which the instructor can consider an indication for academic misconduct (A. Elkhatat et al., 2021a). In contrast, classmates mostly use the same solution style taught. Furthermore, due to a typographical issue, tutoring websites may direct pupils to wrong responses (Donovan, 2020).
Hence, educators suggest using a test of randomly selected questions from a vast question bank (pool) as an effective solution to address question-sharing (A. Elkhatat et al., 2021a; Imran et al., 2019; Ware, Kattan, Siddiqui, & Mohammed, 2014). In a randomly- selected-questions examination (RSQE), the educator creates a question pool containing similar-value questions and specifies the number of questions from that pool to be given in the examination. In RSQE, every student gets a differing selection of questions - even if the examination allows multiple attempts, each attempt will probably contain a novel selection of questions.
Currently, all online-learning management systems allow for the creation of RSQEs. These learning management systems use differing names for the random selection feature, where RSQE is termed in Blackboard® (Blackboard, n.d.), USA as a ‘Random Block,’ and in Canvas®, USA, it is described as a ‘Question Group’. RSQE has plenty of advantages; it can be applied to any type of question, such as multiple-choice questions (MCQs), essay questions, true or false questions, among others. Educators can assign the correct answer for various questions (e.g., ordering, filling in the blank, matching, multiple answers, multiple-choice, Likert, true/false, etc.) using the online learning management system. As a result, without the intervention of the examiner, the examinations are evaluated automatically. Essay and file answer questions, on the other hand, need the examiner’s judgment and grading.
It is worth mentioning that RSQE interferes with students’ collective memory, which allows them to recall a recently finished test from memory and share the questions with other students who have not taken the exam yet (Persky & Fuller, 2021). Although RSQE allows randomly selected questions, online-learning management systems do not track the selected questions since the question-selection process follows mathematical probability concepts. Hence, a proportion of all questions in the question pool might appear to many students, while other questions do not appear at all. This inter-examination repetition of questions allows for question-sharing between students undertaking the same examination. Another major concern is that the RSQE might allow for the selection of sequential questions from the question pool. Consequently, sequential questions can lead to an unfair/skewed distribution of questions within the online examination paper (OEP). Accordingly, RSQE should be designed effectively in order to eliminate/minimize replicated inter-examination questions as well as sequential intra-examination questions.
Literature review
Although the definition of academic integrity is complex and primarily based on consensus, most universities define it as a commitment to several fundamental values, including honesty, trust, fairness, respect, and responsibility in learning, teaching, and research (“International Center for Academic Integrity. Fundamental Values Project.,” 2014; “Universities Australia. Academic Integrity Best Practice Principles,” 2017). Breaching of academic integrity includes breaches of the examination procedures (UniSA, 2022). Online examination misconduct can occur in a spectrum of manners, though the most predominant cheating practices are searching for the examination questions/question-related answers online together with question/answer-sharing between students.
Examination misconduct not only results in graduates with a shallow understanding of the subject knowledge, though such individuals are also more likely to engage in dishonorable behaviors to succeed throughout their future careers (Hodgkinson, Curtis, MacAlister, & Farrell, 2015). Multiple reasons encourage a student to breach integrity in the online examinations, including the shortage of understanding of the topic, lack of interest in studying, failure to manage the required examination time, immature feeling of academic integrity, and lack of rigorous deterrence against academic misconduct. (Lang, 2014). The rampant dishonesty incidents during online examinations have triggered educators and researchers to investigate cheating behavior and develop novel methodologies to prevent (or at least minimize) such educational loopholes to ensure academic integrity and assessment quality within online examinations.
It is noteworthy that fostering self-transcendent ideals through the existence of honor codes might minimize contract cheating (McCabe & Trevino, 1993); however, self-transcendence fails with ingroup loyalty. While students consider online searching for examination answers as cheating, their mindset is that question/answer-sharing constitutes ‘healthy collaboration’ and ‘ingroup loyalty’ among students (Jang, Lasry, Miller, & Mazur, 2017; Pulfrey, Durussel, & Butera, 2018). Due to the development of strong friendships, students experience a sense of ‘group loyalty to their peers’ (Wentzel, Barry, & Caldwell, 2004). Ingroup loyalty causes students to excuse collective cheating by claiming that “sharing is caring” (Pulfrey et al., 2018) and “good teamwork” (Jang et al., 2017) makes cheaters feel less ethically detached. Pulfrey and colleagues (Pulfrey et al., 2018) conducted an insightful study with 615 undergraduate university students to investigate how societal and individual competition affects collective cheating, respectively and how the degree of acquaintance with classmates affects collective cheating to understand the essential incentive of collective cheating better and share questions with classmates. The results showed that collective cheating fell dramatically by showing pupils a macro social competition image, albeit at the price of individual cheating. The individual competition also showed disengagement towards collective cheating at the expense of individual cheating. In addition, collective cheating increased among students who knew each other more than students of strangers. Another study explored students’ perceptions of cheating and its popularity (Honz, Kiewra, & Yang, 2010). The most prevalent and relevant findings of this study are that students consider sharing and giving information less of an ethical deviation than receiving information, and cheating outside campus is regarded by these students as less harsh of an ethical breach than cheating on campus.
Numerous studies suggested and developed different methodologies to mitigate such educational misconduct. The employment of proctoring technologies, such as webcams and lockdown browsers, to control cheating is one of the solutions that has been evaluated for such purposes (Karim et al., 2014; Kharbat & Abu Daabes, 2021; Nigam et al., 2021). The proctoring technologies can also include lockdown browsers that restrict the student’s computer, preventing the student from copying, pasting, or using other browsers until the end of the examination, or – alternatively – implement JavaScripts that can identify participant switching to additional browser/s. However, proctoring technologies obstruct students while taking the examination. Case in point, using lockdown browsers prevents students from using any other software on their computer terminal that might be required to answer the specific examination question at hand. Another concern relating to browser lockdown is that the examination-taker can cheat through the employment of a separate device, unless the examination is not proctored by camera/microphone surveillance. Karim and colleagues (Karim et al., 2014) conducted an exploratory study on 582 randomly-assigned participants for a remote technology-proctored examination. The results of this study implied that, although the approach effectively decreased cheating, it could unintentionally affect student reaction due to increased anxiety and privacy concerns. Another recent, systematic review on proctoring systems (Nigam et al., 2021), focused on artificial-intelligence-based and non-artificial intelligence-based proctoring systems, together with the essential parameters for their design. The study raises several ethical concerns related to proctoring technology, including the risk of reducing fairness levels – typically associated with artificial intelligence judgment - in addition to the attenuation of student privacy and autonomy. In agreement with these studies, Kharbat and Abu Daabes (Kharbat & Abu Daabes, 2021) analyzed 815 attempts within 21 online examinations to evaluate how well students performed under technology-proctored examinations. Their research findings highlighted the negative environmental and psychological factors that impact students, including feelings of stress and anxiety during the examination time-frame and students’ significant concern regarding privacy invasion. In essence, despite the effectiveness of such proctoring technologies in mitigating cheating during online examinations, previous literature reveals concerns on anxiety and privacy during the examination time-frame. Furthermore, limitations of the proctoring techologies in terms of cost and technical requirements, add additional challenges for proper implementation of such technologies.
Several studies have scrutinized written-assignment examinations as another approach to address cheating in such circumstances. Written-assignment examinations include open-ended questions or take-home examinations. Bengtsson (Bengtsson, 2019) conducted a systematic review on take-home examinations in higher education. The study concluded that take-home examinations are only recommended for higher-order Bloom’s taxonomy levels that involve higher-order thinking skills - including analysis, synthesis, and evaluation. Nevertheless, academic integrity might be breached by a proportion of students. Consequently, take-home examinations should be avoided for lowest-order Bloom’s taxonomy levels that involve knowledge and comprehension. The review addressed the advantages and disadvantages of take-home examinations, their risks, and how such risks could be mitigated. The benefits of take-home examinations consisted in reducing student anxiety and promoting the learning experience through assessment, which fostered the educational process beyond memorization. Notwithstanding, the majority of reviewed research articles agree that take-home examinations can be easily compromised by unethical student behavior, including the engaging of a third-party proxy to perform the examination instead. Elkhatat and colleagues (A. M. Elkhatat et al., 2021b) provided scenarios of student-employed methodologies for plagiarizing their written assignments without becoming flagged by similarity indexing software packages. This study analyzed the effectiveness of nine academic-level similarity indexing products against these unethical breaching of academic integrity through the plagiarism of previous literature.
In contrast to previous approaches to mitigate cheating and question-sharing, few articles discussed RSQEs. The merit of RSQE is that it applies to any educational level - primary, secondary, or tertiary, and for any study subject, such as mathematics, science, history, among others. Moreover, it helps instructors design both lower-order and higher-order thinking questions according to Bloom’s taxonomy (Bloom, 1956). Lower-order thinking questions include remembering information, demonstrating understanding, and using acquired information, while higher-order thinking questions include analyzing, discovering, and organizing information, integrating knowledge, and making judgments. Online learning management systems allow educators to design and develop essential and guiding questions to measure higher-order thinking (Blackboard). Ali (Ali, 2011) suggested randomly-selected questions with vast question pools as a strategy to counter cheating through question-sharing. However, as a method for mitigating question-sharing and student memorizing the bank questions, the researcher proposed a hybrid model of 30% randomly-selected questions and 70% non-randomly selected questions.
Notably, a vastly expanded question pool is not synonymous with a reduction in replicated inter-examination questions, since the frequency of one random event from multiple events could be higher than expected due to the probability – as described by the ambiguous issue recognized as the ‘Birthday Paradox’ (Swadling, 2019). Similarly, the frequency of sharing an identical question from a larger pool of questions can be higher than expected, leading to question repetition among students undertaking a specific, identical examination. The probability of sharing question-sharing can be calculated according to the following equation;
$$ P=1-\frac{N!}{\left(N-x\right)!\ast {N}^x} $$
Where, N is the pool size, x is the number of selected questions from the pool.
Based on probability calculations, Wentworth’s Institute of technology’s teaching and learning perspectives forum (Cookel, 2015) provides precious guidelines on designing RSQEs to minimize the number of replicated inter-examination questions. The study calculated the probability of five questions selected from differing question pool sizes (10, 25, 50, 100, and 200 questions). The study introduced the concept of the ‘Birthday Paradox’ to predict the likelihood of no repeated questions from question pools. Nevertheless, this study did not provide information on the frequency of replicated inter-examination questions, which is essential when considering methods to mitigate question-sharing among students. Moreover, such probability calculations assume that the selection of questions is a fair event, which might not be correct and consequently requires an experimental study to prove it. In addition, it does not provide statistical information on the issue of sequential questions.
Moreover, no studies have investigated the sequential questions that can lead to an unfair/skewed distribution of the exam questions. Case in point, if an examiner designed the examination to consist of 10 randomly-selected questions from a pool of 100 questions, there is the distinct probability of two (or more) sequential questions to be selected from the same question pool. Having sequential questions from a question pool might be a concern when the examiner follows a patterned order when creating the specific question pool. One scenario is when the first number of questions are derived from one specific lecture/lesson (e.g., lecture #1), followed by another set of questions from the next lesson (e.g., lecture #2), with this pattern building the entire online examination paper (OEP).
Consequently, although active research is currently underway within the field of online examination design, no previous literature has yet focused on the effectiveness of differing RSQE designs to address the issues of replicated inter-examination questions or sequential intra-examination questions. This study aimed to fill this research vacuum using the Monte Carlo approach (James, 1980), by conducting an empirical study through the development of 600 RSQEs - to investigate the impact of RSQE design in resolving such educational challenges.