Evaluating empowerment towards responsible conduct of research in a small private online course

Abstract

Teaching Responsible Conduct of Research (RCR) has gained recognition in recent years, with various organizations emphasizing the need to take responsibility in training their researchers in RCR. However, differing perspectives on RCR education result in a variety of practices, making it unclear what approach is most effective. As part of two European projects on research integrity education, we have developed an empowerment perspective on RCR education and incorporated this in an online course design. This paper presents the design and qualitative evaluation of a Small Private Online Course (SPOC) that aims to empower participants towards RCR. Results show the course enhanced individual aspects of empowerment, but it proved difficult to stimulate social aspects of empowerment in researchers day-to-day work.

Introduction: RCR education in need of a new perspective

Since the 1980s, universities and organizations like the National Academy of Sciences (NAS) and the American Office of Research Integrity (ORI) have stimulated the teaching of responsible conduct of research. More recently, European organizations like the League of European Research Universities (Lerouge and Holl, 2020) and All European Academies (ALLEA, 2017) and many universities have joined in. Several EU-funded projects have together initiated The embassy of good science, an online platform that supports educators to develop training on research integrity and ethics (https://embassy.science/wiki/Main_Page). As part of two European projects on research integrity education, we developed an empowerment perspective on responsible conduct of research (RCR) in education and incorporated it into a Small Private Online Course (SPOC) which we present in this paper.

The body of literature on RCR education has grown over the past 25 years and a wide variety in content, format and goals can be found (Kalichman 2013). Even though the ORI has provided guidelines on the topics that researchers and students need to be trained in (e.g., responsible publishing, peer review, mentoring, collaboration, data (acquisition, management, sharing, ownership), conflicts of interest, and research with human subjects or animals (Steneck 2007), a broad variety can be discerned in course formats, learning aims and evidence for effectiveness. It remains therefore an open question for educational developers what learning aims to prioritize and how to teach courses effectively.

Kalichman offers guidance on prioritization of learning aims and RCR topics. Instead of providing researchers with additional RCR knowledge and skills, he argues that it is more important to.

“arm them with a positive disposition toward RCR, with a sense that there are things they can do in the face of concerns, and with a belief that they are part of a culture that takes RCR seriously (Kalichman 2007). These attitudes are arguably more essential than any particular piece of knowledge or improvement in skills. In their absence, it would matter little if someone had perfect knowledge and skills” (Kalichman 2014, 70).

According to Kalichman, RCR education should be aimed at empowering researchers to handle complex issues in practice. The literature on RCR education however, still dominantly focusses on transfer of knowledge and specific do’s and don’ts and on developing specific (reasoning) skills (Watts et al. 2017; Steele et al. 2016; Medeiros et al. 2017). Building on the results of a meta-analysis by Watts et al. (2017), Mulhearn et al. (2017) developed a Predictive Modeling Tool for developing effective RCR courses. The tool reflects the dominant focus in the literature on knowledge and skills and not behavioral- or attitudinal changes of measures of effectiveness as it ‘strongly relied on descriptive, empirical findings rather than prescriptive “best practices” (Mulhearn et al. 2017, p. 206). In a more recent effort to make the available evidence on RCR education accessible, Krom & van den Hoven (2022) developed a quality checklist for RCR education. After analysis of eleven reviews on the efficacy of education for RCR, the authors conclude that quality criteria for effective RCR education are often narrowed to specific cognitive learning outcomes which can be measured quantitatively. At the same time, the considered meta-studies fail to present clear (and generalizable) conclusions on effective elements of effective teaching in RCR, such as participant characteristics, course aims or methods of assessment. Also, Katsarov et al. (2021) tested the robustness of eleven hypotheses on effective training strategies, whereby 75 effect sizes from 30 studies were taken into consideration. They found that most findings from previous studies on impact of RCR courses could not be replicated with multivariate analysis. On the other hand, Katsarov et al. (2021) did conclude that “a practical course orientation with an emphasis on experiential learning and an emotional engagement with ethical decision-making appears to be the best predictor of effective RCR education” (p18).

Based on these results and in line with Kalichman (2014), Krom & van den Hoven (2022) argue for a broader concept of quality of RCR education that reflects the complex nature of ethical decision making and requires a combination of skills, knowledge, attitudes and behavioral changes. Different approaches in courses will likely lead to different outcomes, so it is essential to define desired learning outcomes beforehand and align them with the course activities. To achieve this in our study, we will clarify what an empowerment perspective means in an RCR course context. Whereas in the past approaches to integrity education focused on misconduct such as falsification, fabrication, and plagiarism, Kalichman (2014, p. 69) argues that we are “to foster a research culture in which conversations about responsible conduct of research are expected and acceptable…”. Following this suggestion, we take the idea of empowering researchers towards fostering a positive research culture as the core of our RCR training approach.

View on empowerment towards RCR education

The concept of empowerment in education can be traced back to the Brazilian educator Paulo Freire, who stimulated people to strive for conscientizaçao, also described as critical consciousness (Freire 1970). His work is positioned in a political context of structural injustice and power disbalances in society, where groups were oppressed and dehumanized as a result. The idea of Paulo Freire's Pedagogy of the Oppressed is that traditional education systems are inherently oppressive, perpetuating a culture of domination and inequality. According to Freire, it is essential to view students as active agents of their own learning and development, rather than passive recipients of knowledge. Even though Freire developed his pedagogy in a context of severe social injustice, it provides important lessons for education on social issues, such as RCR. Transferring the pedagogy of the oppressed to RCR education involves a fundamental rethinking of the way we approach it, with a focus on collaboration, critical thinking, and capacity building. It involves incorporating more student-centered activities, such as group discussions, collaborative projects, and problem-solving tasks which involve critical thinking about social (integrity) issues and reflection on their own beliefs and assumptions. Rather than acting as the authority figure, teachers should become facilitators of learning, working alongside students to co-create knowledge and encourage critical thinking.

Lawson (2011) defines empowerment as a focus on the development of critical autonomy: “Critical autonomy includes the ability to think for oneself, the ability to use theory as a guide to action, and, crucially, the ability to evaluate the circumstances of one’s life, including the structural forces that surround us” (p.90). Critical autonomy thus includes cognitive aspects (e.g., reflection), skills (e.g., ability to think for oneself, ability to use theory as action guiding) and meta-competences (e.g., taking the circumstances of one’s life into account). In addition, Israel et al. (1994) state that at the individual level “empowerment refers to an individual’s ability to make decisions and have control over his or her personal life” (p.152). They emphasize that empowerment is a positive and proactive concept and distinguish three levels of empowerment, namely the individual level, the organizational level and the community level (ibid). A similar emphasis on multiple levels is argued for by Rappaport (1987), stating that empowerment is necessarily a multi-level construct:

“It is concerned with the study of and relationships within and between levels of analysis-individuals, groups, organizations, and other settings, communities, and social policies. It is assumed that there is a mutual influence process across levels of analysis, and that this process takes place over time.” (p.139)

Empowerment education should consider community and system perspectives, as well as of the perspective at the individual level (Bergsma 2004). These levels of empowerment align with research integrity discussions which often call for change at different levels, and research takes place in an (institutional) context where expectations and regulations can be demanding. At the individual level, responsible researchers are expected to have critical autonomy, take control, and be accountable for their research and practice.

Table 1 outlines our starting points for empowering RCR education: It should build capacities, i.e. (1) where knowledge, skills and attitudes enable participants to take control (2) develop a willingness to take responsibility for RCR in their daily practice, with courage (3) and stimulates a critical autonomy (4). Finally, empowerment stimulates a pro-active attitude which means to act upon decisions made (5). To achieve these goals, RCR education should focus on experiential learning and involve learners in determining what is taught. RCR education should stimulate researchers to analyze their social research context and help them to express their own beliefs and assumptions and determine their role in making changes, without assuming that integrity issues can be resolved easily. RCR starts in a non-ideal situation: grey areas and misconduct (will) continue to exist and institutional changes may be slow. Yet, with the premise that RCR is not solely an individual responsibility, RCR education could pave the way for institutional and systemic changes where conducting research in a responsible manner is the obvious and most attractive way of behaving in research practice.

Table 1 Starting points for RCR education

Online RCR education

The rapid development in educational technologies led to a shift from classroom-based instruction to online courses (Todd et al. 2017; Usher and Barak 2018) especially during the COVID-crisis. In the context of RCR education, there is an increase in the number of courses offered online (Todd et al. 2017; Watts et al. 2017). Online courses enable learning at one’s own time and pace and at the same time provide students the opportunity to study with peers from all over the world. MOOCs (Massive Open Online Courses) are a well-known example of online courses; they can accommodate many learners at the same time and require little intervention from the teaching staff. As a consequence, they involve individual learning and may lead to less (perceived) active involvement of students. SPOCS (Small Private Online Courses) are designed for a specific (and small) group of learners, to provide a personalized and interactive learning experience (Uijl et al. 2017). A sequence of smaller learning units helps participants to spread the workload over a predetermined course period. Instructors are present to guide the learning process, provide feedback and stimulate discussions in a safe learning environment. They can potentially meet up to the disadvantages that are linked to many online ethics training, i.e. that they lack interaction among learners and support of an expert teacher and that they sometimes deliver simple instructional content that has little effect on students’ awareness and practice (Todd et al. 2017).

The review studies presented in the introduction also include online training. Mulhearn et al. (2017) for example analyzed 35 online training modules which were marked by online instruction, self-directed learning, and web-based discussion. With respect to content, the online training predominantly placed emphasis on guidelines which can be tested in closed ended questions. Developers of online RCR courses still seem challenged by the need to promote complex competences such as ethical decision-making among students (Gross 2016; Phillips et al. 2018).

Empowering towards RCR via a small private online course

Within the context of the European project INTEGRITY and in cooperation with the company Elevate Health, we developed a Small Private Online Course (SPOC) on RCR for PhD candidates early 2020, titled Responsible conduct of research: how to do it right. Individual and group assignments were designed to actively engage participants and formative (peer)feedback was given at predetermined times.

At the end, students could obtain a certificate when (nearly) all assignments were completed with a pass. The four-week course comprised a total workload of 12 h for each participant. It was meant as a general introductory course on three main RCR topics: supervision and mentoring, data management, and publication and reviewing. The course goals focused on stimulating awareness and fostering a proactive attitude and skills in addressing and reflecting upon integrity issues. Each course week addressed a different topic in a so-called Learning Unit. Before week 1, participants were introduced to the course theme and asked to introduce themselves to all participants. Two synchronous (live) online meetings were included to engage participants more directly. Often, many SPOCs only assume asynchronous participation, but based on a pilot we decided that live meetings are helpful means to motivate participants. The first live session was an additional introduction to the course, the teacher and fellow participants. In the final week we scheduled a collaborative storytelling assignment. At the beginning of each week, the course materials of that week were made visible to participants, to enable synchronous working on the assignments and arranging groupwork on collaborative assignments. The course topics, each comprising one week, were:

• Doing research well. This learning unit introduced our positive approach to RCR and the ALLEA code of conduct. Integrity issues were portrayed as complex issues in a grey zone that need to be recognized and require reflection.

• Supervision and mentoring. This learning unit presented different supervisor roles to stimulate awareness and reflection on mutual expectations and responsibilities in supervisor/supervisee and mentor/mentee relations.

• Data management. The General Data Protection Regulation (GDPR) that is applicable in Europe was presented, and followed by recent developments in data management and intellectual property to raise awareness and discuss practical issues.

• Publication, reviewing and evaluating. Issues of authorship (e.g., giving credit, guest/ghost authorship, authorship order) and developments in peer review were used as input in writing a personal publication strategy followed by peer feedback.

• Upon finishing the course, participants were asked to reflect on the course and their learning process.

The concept of empowerment was not discussed explicitly in the course, but constituted a key feature of our course design, as illustrated by the following examples:

• a. After introduction of the ALLEA code of conduct in the first learning unit, participants were asked to actively interpret which and how principles from the code apply to their own project, addressing both positive aspects (what is taken care of well) and possibilities for improvement;

• b. Dialogue and argumentation about RCR were trained through multiple collaborative case discussions. An RCR reflection model was offered to help independent deliberation on integrity cases, which participants could derive from their own experience (preferred) or select from our course environment.

In week 3, a portfolio assignment asked participants to search for information within their institution, e.g. whom to turn to with questions on Ethics Review, the GDPR, data management, etc. This way we aimed to actively promote awareness of support options in one’s own research context and a proactive attitude to make use of these.

Having designed the online learning trajectory to stimulate empowerment towards responsible research behavior in participants, we were interested in whether participants perceived empowerment and increased ability to think, speak and act upon integrity issues they encounter in their own research practices. Therefore, we formulated the following research question: What aspects of empowerment towards RCR are stimulated in participants of the SPOC?

Method

Study design

We conducted an explorative evaluation study investigating participants’ experienced RCR empowerment and abilities they showed during and shortly after the course. A threefold approach was used to gain more insight into the participants’ experiences. First, we collected in-course data via questionnaires at the start and end of the course on respectively the perceived relevance and perceived achievement of the RCR learning aims. This was complemented with a learner report and course evaluation questionnaire focused on perceived empowerment. Second, we conducted semi-structured interviews after the course in which participants were asked about their view on RCR empowerment and whether and how the course contributed to their RCR empowerment. Third, a case deliberation assignment at the end of week two was used to grasp the participants ability to reflect on a complex integrity issue.

Participants

During this study, 92 PhD candidates from various European countries, subscribed to our course (12 in run one, 41 in run two, and 39 in run three) and agreed to take part in our study (see next section for the procedure). Fifty-eight participants (63%) finished the course and were eligible for a course certificate. Although SPOCs have generally higher completion rates than Massive Open Online courses (MOOCs), an average completion rate just above 50% is common (conf. Jordan et al. 2015; Guo 2017). We only included in-course data from participants of whom we could retrieve the complete set of answers from the questionnaires at the beginning as well as the end of the course. Leaving out uncomplete questionnaires, we analyzed in-course data from 36 PhD students.

Six participants, five females and one male, were interviewed. They were in different phases of their PhD study: first year (1), second year (1), third year (2), and fourth year (1) or unclear (1) and working in social sciences (3), health sciences (1), humanities (1), and natural sciences (1). Five of the participants had not received any education on research integrity before, and one reported that the topic was only briefly touched upon in research methodology courses.

The case deliberation assignments were collected from 48 participants. Student demographics, i.e. gender, stage of PhD project and research country, were not collected as data within the course environment. However, the online live meetings showed that participants worked in a broad range of disciplines (e.g. education, medicine, neuroscience) and countries (e.g. Malta, Lithuania, Portugal, The Netherlands, France, Switzerland, Ireland).

Data collection and analysis

After a pilot in June 2020, data for this study were collected in three runs of the course: September 2020, November 2020 and March 2021. Approval for the study was provided by the local ethics committee at Utrecht University (FETC-Hum, number 20–256-03). Participants were recruited with help of the partner universities of the two European projects. In a separate email, participants received an information letter about the study and were asked explicit consent for the use of their in-course data for research purposes. Although participants could opt out of their data being used without it having consequences, they all agreed to the pseudonymized use of their data for this study. The data were retrieved from the course environment and stored on a secure drive at Utrecht University. After the first run in September, some adjustments were made to the interview structure, as well as the empowerment questionnaire at the end of the course. As a result, we did not use these data from the first run in September. To prevent undue influence on students we maintained a clear distinction between researchers and teachers, yet all three authors were designers of the course and involved in analyses of the data.

In-course data

The in-course data consisted of questionnaires and a learner report to gain insight in students’ experiences. Course evaluation questionnaires were filled in at the start and end of the course. At the start participants were asked to indicate how relevant they deemed the courses’ learning aims, using a five-point Likert scale (“not at all relevant” to “very relevant”). At the end of the course, we asked participants to what extent the course helped them achieve these learning aims on a four-point Likert scale¹ (“not at all” to “to a great extent”). In this learning unit we also included 14 four-point Likert scale items on participants’ experienced empowerment after the course (“not at all” to “to a great extent”). The Likert scale items were analyzed using non-parametric descriptive statistics to provide a qualitative account of respondent’s course experiences and perceived RCR empowerment in particular.

Learner reports can provide valuable insights into student perceptions of their learning experiences in online courses (Kitsantas et al. 2021). We included four open ended sentences to allow participants to give more substantial feedback on their course experience, i.e., “What I learned about research integrity is…”; “What I valued about this course is…”; “What could be added to the course (what I missed) is…”; “What could be left out of the course is…”. The results of the course evaluation questionnaire were cross-checked with the answers to the open questions in the learner reports. First, meaningful fragments in the written answers were categorized in a) RCR aspects that participants mentioned to have learned and b) course characteristics. Next, these fragments were coded inductively.

Interviews

Two weeks before the start of the course, students were invited by email for an interview of approximately 45 min after finishing the course by a researcher who was not involved as a teacher in the course. Six students agreed to participate and returned the informed consent before the start of the interview. Interviews took place online through video calling software (Teams or Zoom) and were recorded by a separate voice recorder. The interviews focused on portraying the learning experiences of participants in more detail. Participants were asked about positive and negative course experiences and to elaborate on the questions in the course evaluation questionnaire. We also asked them to explain their view on empowerment and on what aspects they felt empowered through the course. The recordings were transcribed and stored on a protected server at the university. The transcriptions were pseudonymized before further analysis via Nvivo in two steps. First, the interviews were coded inductively by segmenting the transcripts and assigning relevant codes. All interviews were coded by at least two researchers and differences in coding were discussed until agreement was reached. Subsequently we ordered the codes hierarchically into main categories and subcategories, merged similar codes, and removed redundant codes.

Case deliberation assignment

In the second course week, participants individually deliberated on a case. These deliberations were downloaded and analyzed to grasp students' RCR reflective capacities. Participants could use a case from their own experience or a pre-given case which was derived from the Erasmus Dilemma Game with the title “First to the Mill” (Fig. 1). In the assignment, the RCR reflection model was used, which had been developed specifically for deliberating integrity issues (see Additional file 1). The model scaffolds participants to address the relevant considerations, including different perspectives of people involved, use the code of conduct and to come up with a decision and action strategy. The model resembles moral case deliberation models offering guidance for concrete situations, like the Utrecht Step-by-Step model (Bolt et al. 2015). Such ethical reflection models assume that in situations where a decision is necessary, as is often the case in integrity dilemmas, a structured approach helps in balancing ethical principles (as explicated in professional codes) against facts and regulations. The model was introduced in a video, made available as pdf-file and used as template to guide students through the subsequent reflection phases.

Fig. 1

Pre-given case

One case was incomplete and discarded, so 47 cases were analyzed, using a rubric to categorize the type of reflection. Categories included whether the participant chose the pregiven case or deliberated one’s own case, how lengthy deliberations were and whether all steps in the RCR reflection model were used in elaborating the case: What principles from the ALLEA code of conduct were considered and what perspectives of people involved? Did the deliberation reflect a simple case of misconduct or an integrity issue in the grey area? Did the deliberations include different options to act and a had a values/decision been made and/or an action strategy described|? The analysis was done by one of the researchers. The coding was checked by the other authors and discussed in a joint session.

Results

In this section, we combine the results from the different data sources and present them in a narrative way. The in-course data and interview data are considered as self-reporting data, because participants were asked how they perceived and experienced their learning process. The case deliberations on the other hand show how participants performed using the RCR reflection model, hence show the status quo of capacities of participants.

Perceived achievement of learning aims

Before the start of the course participants were asked about the personal relevance of the learning aims. The most relevant learning aims were those focused on one’s research practice (see Fig. 2). 89% of the respondents deemed it relevant or very relevant both to assess how research values apply to decisions and actions in their own research context and to be able to demonstrate what composes a good research practice in their discipline, followed by learning to discuss dilemmas in their own research practice (86%). The least relevant were considered learning aims regarding discussing responsibilities and expectations in supervision and mentoring (72%), knowing where to find support in handling issues with third parties (75%) and the learning aims focused on data management (76%). On average, in the questionnaire all learning aims were scored as relevant or very relevant with a mean score above 4.0.

Fig. 2

Percentage of deemed relevance of courses’ learning aims at the start of the course

Figure 3 shows that, upon completion of the course, the majority of the participants reported achieving each of the learning objectives to a moderate or great extent. Most positive evaluations concerned the communicative learning aims discussing dilemmas in one’s research practice where 86% of the participants scored to a moderate or great extent and discussing responsibilities and expectations in supervision and mentoring (89%). The participants also reported to have improved in their practical abilities, regarding demonstrating good research practice in their discipline (86%) and determining authorship order (89%). Relatively few participants, just more than half of them, perceived a moderate or substantial improvement in their understanding the values that underlie their own research project (58%) and where to find support in handling issues with third parties (61%).

Fig. 3

Percentage indicating to what extent the course has helped to reach the learning aims

RCR empowerment

In the interviews, participants were asked what empowerment means in their view. Four of the six participants were familiar with the concept of empowerment. Common explanations of the concept given by participants referred to being confident in yourself and in your abilities, having the skills and knowledge needed to solve issues, and acting when an integrity issue arises. According to one participant, a sense of disempowerment is experienced when you are aware of a problematic situation that requires action, but “you feel like either you don’t know what you should do, or you’re not able to do it, or you feel like other things are in the way.” On the other hand, as mentioned by another participant, someone who is empowered will “be brave in addressing issues when they arise and not... turn [their] blind eye”.

The course evaluation questionnaire asked more specifically on what aspects of RCR participants were more empowered (Fig. 4). The results indicate that on average participants perceived all aspects of empowerment to have increased to a moderate or great extent (mean scores all above 3.0) except for the ability to handle integrity issues in one’s own research project (2.83). The learner reports and interviews focused on a more nuanced understanding of participants experiences and perspectives regarding RCR empowerment within in the context of our course, as presented below.

Fig. 4

Percentage indicating to what extent the course helped to become empowered on RCR

Awareness and understanding of integrity issues

A starting point in RCR education is that participants learn to take control in integrity issues. This requires both awareness and understanding of RCR. In the course evaluation questionnaire, all participants stated that the course helped to increase their awareness of what integrity issues can arise in daily practice, and 42% said the course did so to a great extent. Similarly, all interview participants stated that the course made them think about issues they had never considered before. Also, all participants indicated an increase in their knowledge about integrity issues (50% to a great extent), rules and regulations (39% to a great extent), research values (33% to a great extent), and consulting others (28% to a great extent).

Two of the interview participants explained that the deliberation of cases contributed to their awareness of the various types of integrity issues they could encounter. Three interviewees said the information about authorship made them more aware of potential issues. According to one of them the course encouraged them to place integrity in a broader context and understand that a lack of integrity impacts not only the researcher but the scientific community and society at large as it may damage the public trust in science. This participant argued that the toleration of small acts of misbehavior could grow into more serious cases of misconduct: “in that sense I could say that I feel more empowered, like I know that it’s important and that I should try myself to be... as good as I can be in that sense and also if I encounter some kind of problems... to address them, not to ignore them.”

The learner reports show an increased awareness of the nature of integrity issues: 12 of the 36 respondents explicitly mentioned to have learned to recognize the complex or grey nature of integrity issues in their daily practice, either explicitly characterizing them as “grey” or “complicated”, “without clear-cut answers” or “involving multiple perspectives”, or stating that “research integrity is on the rise but although many of us have a good understanding of integrity and ethics in research, there are many grey areas in our day-to-day lives”.

This understanding of integrity issues as grey and multifaceted was expressed by all interviewees. One participant stated that the course “… gave a wider understanding of the various issues and grey zones, what actually integrity comprises of, what actors may be included, and who should take part in each decision …” This and another interviewee indicated that such knowledge helps making substantiated judgments about integrity issues, rather than responding based on initial emotions. Furthermore, all interviewed participants stated that the course had increased their knowledge on relevant rules and regulations and on how to interpret research values like transparency, honesty, and accountability. Similarly, the learner reports indicated that the most frequently (12 times) valued learning outcomes referred to the rules and regulations that were covered in the course, in particular the ALLEA Code of Conduct.

Although participants gained more insight in particular topics such as data management, authorship, and supervisor roles and responsibilities, half of the interviewees indicated online courses to have limitations as they cannot address all potential issues, or that the issues encountered in practice may vary depending on the specific discipline they are working. Five interviewees stated they missed a more in-depth discussion on a particular topic related to their specific research context.

Motivation & courage

As illustrated above, having increased awareness of integrity issues could be linked to increased motivation to tackle integrity issues, also a key element of RCR education: One interviewee explained that the course increased awareness and knowledge on what good conduct entails and “... if you don’t know what to do yourself and you have all these senior researchers, I don’t think that would motivate me to stand up.” Similarly, two interviewees expressed that the examples in the course about senior researchers who had committed misconduct, increased their desire to become a good researcher and “... when you decide to be a researcher, then you should do your best... to make the research good and to make it responsible.” On a more concrete level, two participants stated that the course stimulated them to talk about integrity more often, to discuss ethical issues with colleagues and supervisors and to share RCR information, such as provided in the course. It must be noted that four of the interviewees were already quite motivated before the start of the course and according to two of them the course did not further contribute. The questionnaire confirmed that the course fostered participants’ motivation to be or become a responsible researcher (69% to a great extent) and to work on a positive research culture (56% to a great extent). Most participants (95%) were also more motivated to become pro-active on RCR issues of which 42% to a great extent.

The course instilled more courage in addressing integrity issues for 97% of the participants, with 50% reporting an increase in their courage to a great extent. In the interviews, the social interaction during the course was mentioned as an important element as this created a feeling of relatedness within a community of researchers concerned about integrity issues, and not being alone. However, the interviews indicated their courage to be context-dependent: according to four interviewees, they feel primarily courageous to address integrity issues with students, friends, or colleagues. They explained that it is more challenging to discuss integrity issues with supervisors or those higher in the hierarchy, due to the higher risk of negative consequences if the relationship gets damaged. Two interviewees still struggled to discuss the issue of “undeserved or ghost authorship” in their research group because they lack the support in their research group and fear social rejection.

Reflection

Another starting point for RCE education was developing critical autonomy, of which (self-)reflection and deliberating RCR issues in ones’ research practice are key elements. In the course evaluation questionnaire, all participants reported to have improved their ability to reflect on integrity issues (47% to a great extent), to determine relevant considerations (39% to a great extent), and to choose between alternative actions or decisions when confronted with RCR issues (33% to a great extent). From the interview data, we identified three main ways in which the course had furthered participants reflection. First, four participants valued the practical experience in reflection through deliberation assignments, such as analyzing cases, answering reflection questions or discussing RCR topics collaboratively. One participant emphasized that they would have liked even more practice-based cases in the course. Second, the RCR reflection model was explicitly valued by two interviewees. The model fostered understanding of what steps to go through and what questions to consider when confronted with an integrity issue (1 participant), and helped to take some distance from a situation before deciding what to do (1 participant). Correspondingly, in the learner reports four participants explicitly mentioned that the RCR reflection model was a helpful tool to guide reflection or structure discussions. Thirdly, the course facilitated reflection by offering insights in how to overthink possible actions that can be taken in different situations(2 participants). However, one interviewee indicated to still experience difficulties in deciding what to do in particular circumstances, because “the principles discussed in the course do not seem to apply to all the dilemmas you can encounter”.

The case deliberation assignment showed participants reflection on an integrity issue using the RCR reflection model. Of the 47 cases analyzed, 34 participants chose the pregiven case (First to the Mill), and 13 used their own case (see Table 2). The RCR model was used in the majority of the deliberations (75,6%). On average, the 47 case deliberations involved 4.4 considerations. The majority of deliberations were brief, consisting only of short sentences (48.9%), to very brief (12.8%). An example of a brief type of consideration is “possible negative impact on future relationships” compared to a lengthier contribution, such as “Adam should be given a chance to explain himself, since it could have been a mistake or a distraction”. When comparing the case deliberations of participants’ bringing in their own example with participants choosing the pre-given case, the average number of considerations was slightly higher in the first group (5.1 vs 4.2). Particularly, the first step in the RCR model stimulated to elaborate the perspectives of the people involved and refer to (institutional) regulations, of which 27 used other perspectives of which but only 6 participants included and described interests from this other perspective. Comparing ‘own cases’ to ‘pregiven case’ participants, it is interesting to see that a slightly huger number using the pregiven case is able to consider multiple perspectives. Most of the participants (42) were able to indicate which principles are at stake in the cases: 30 participants only mentioned the principles, while 12 participants also explained why these principles from the Code of Conduct seemed relevant to them: “Respect and honesty. I would say there is definitely a lack of communication. This could be due to Adam’s focus on his own career and achievements, caring less about what someone else is doing or the consequences for them”.

Table 2 Characteristics of case deliberation

Respect for someone or their work was the value mentioned most frequently (28): “The first and foremost value at stake is respect. As colleagues working together, my fellow has to respect me and give the appropriate acknowledgement.”

In addition, giving credit for one’s contribution (18; “I think that her idea was stolen and she was given no credit for something important”), impact collaboration (13), in terms of consequences on the working relationship, and honesty (15) were considered frequently. Responsibility (1), rights (1) and intellectual property (2) were only mentioned occasionally. The RCR reflection model intends to steer participants towards a decision on what action should be taken. Interestingly, only 15 out of 47 participants reached a conclusion supported with arguments, while 37 out of 47 participants described a concrete action to be taken. This suggests that, despite offering the reflection model as a scaffold, only few participants explicitly balanced different considerations before arriving at a decision. Considering one’s own case or the pregiven case made a difference here. With respect to the pre-given case a smaller number (23.5%) reached a reasoned decision, compared to participants who considered their own case (54.8%).

Ability to handle integrity issues

RCR education should empower participants to act upon their decisions. Although the questionnaire indicated that the course improved participants’ ability to handle integrity issues in their own research project, only 19% agreed it helped to a great extent, and 36% to some extent. This modest score was reflected in the interviews, in which participants indicated they found it challenging to implement these learnings in real-world situations. Some explained that it can be challenging to address practical integrity issues as the general information provided by the course may not apply to all field-specific situations. Additionally, some problems may be complex and not easily solvable. Furthermore, the ability to handle integrity issues is not solely based on individual competence but also influenced by the social context. For instance, one interviewee mentioned that while she could now recognize when someone deserves authorship, addressing authorship issues in a research culture where guest authorship is considered acceptable remains difficult. Likewise, in the learner reports, four participants specifically noted that applying RCR in practice was challenging, due to factors such as complexity or their social institutional context:

“I find the code of conduct very helpful. I learned that there are rules and codes on how to perform research, however being in my final PhD year I notice that in practice it is not always like that. It was a really helpful course, also to recognize how things should be done (or not). Hope it helps me in discussions with my supervisor on topics like reviewing process, value of impact factor and publication pressure.”

This is also reflected in Fig. 4: 22% of the participants find it difficult to decide when to consult others on RCR issues and 25% do not feel courageous enough to address integrity issues with colleagues, fellow students, or supervisors.

From the interviews, we distilled three types of social or institutional challenges that may hinder RCR empowerment in the workplace. One concerns the dependency on others resulting from the hierarchical structure and power inequalities in academia, and the fear of negative consequences when speaking up. One participant stated: “in my university I [do] not feel free to say something. We have a really hierarchical system and everyone higher [has] more power than the lower ones and I think if I say something about integrity, they will shush me down”. Another participant admitted that they would not take any action if they witnessed any wrongdoing because “as a junior researcher you strongly depend on your supervisors... I need their help, and I need them to work for me sometimes, and I want them to like me and to make time for me to do stuff, so I would not like to get on their wrong side so to say.” Overall, participants referred to the power of supervisors that can decide whether students finish their PhD degree or not, as restraining students from speaking up.

A second challenge is the institutional research culture. Two participants explained how doing favors (e.g., granting authorship to someone who did not contribute to a paper) is part of that culture, and that, as a beginning researcher, it is hard to stand up against it. Furthermore, four interviewees were convinced that the research culture discourages to talk about integrity issues, and that it is considered unimportant or even inappropriate to do so. Three of these participants reported having experiences with working in an unsafe culture, i.e. being reproached when trying to address an integrity issue or witnessing someone else experiencing this. A final challenge concerned the competitiveness between scientists, that may lead individuals to prioritize career interests (such as publishing more) rather than focusing on the merit of their work. As a result, half of the participants expressed the belief that even if they had the motivation to speak up, it wouldn’t make a difference as others may not perceive the issue or may not be concerned enough to take action. One participant suggested that to change the institutional culture, research integrity should become a common interest: “... everyone needs to follow some sort of course like this so you don’t feel alone in your issues... if it would be the talk of the day a bit more then it wouldn’t be such an issue to bring up these issues.”

In contrast to these social and cultural challenges, participants were convinced to be better equipped in dealing in these challenging work environments after taking the course. The course made them feel less alone in their concerns about integrity; other PhD’s experience similar issues and there are more researchers who think integrity is important. One participant phrased it as follows: “... you see all this community and all [these] people working for integrity, so you feel... like there is someone behind your back. You see that there is a community who cares about it, so definitely you feel more courage.” Indeed, the collaborative nature of the course was one of the most valued aspects of the course as. seventeen participants explicitly mentioned in the learner reports the group discussions, receiving input from peers, or sharing dilemma’s when asked what they valued most about the course. Another way in which the course made students feel more empowered in dealing with their social environment, was by providing tools and information. As expressed by one interviewee: “I have some tools and some actual information I can share and show. It is not my opinion... it is a real thing. And I think that is how I feel empowered.” Another participant argued that they feel more prepared to deal with integrity issues in the future, because they know now where to find information regarding RCR. In addition, four of the interviewees mentioned as helpful the information they received on who to go to for help when encountering an integrity issue in their work environment. Thus, to conclude, it was easier for students to navigate challenging environments regarding integrity because they felt backed up by a community of people who care about integrity, they acquired information and tools, and knew people they can go to for help.

Discussion

The aim of this study was to investigate what aspects of empowerment towards RCR could be fostered in a Small Private Online Course, titled Responsible conduct of research: how to do it right? Based on a conceptual elaboration of empowerment, we determined five starting points for RCR education and used these to design and evaluate the course. As fully online courses are still the least-used approach in delivering integrity education among European universities (Abdi et al. 2021a), in this study we wanted to know whether RCR empowerment can be stimulated in an online environment. If our courses are successful, RCR courses that foster empowerment competences can be achieved more cost-effectively for universities.

Our findings suggest that the course indeed stimulated individual aspects of empowerment. Not only did participants indicate that they deemed empowerment competences relevant, but they also indicated via learner reports, Likert Scale evaluation items and interviews in what respects they considered themselves more capable to be a responsible researcher. Participants gained awareness of a variety of integrity issues, and knowledge about RCR, including rules and regulations. The RCR model proved a helpful tool to reflect on integrity issues and the online and interactive case-discussions provided a collective experience stimulating a positive attitude.

The increase in knowledge is quite common in (online) RCR education (Mulhearn et al (2017). Katsarov et al. (2021) also found “significant positive cognitive effects” … “for courses that emphasized the individual learning, experiential learning and application of ethical guidelines” (p. 14). Increase in knowledge as reflected in our course seemed to go beyond a simple understanding of the principles as stated in a code of conduct or regulations such as GDPR or about data management. Knowledge serves the purpose among participants of “feeling backed up” and knowing where to find support in their work environment, which are central in developing a critical autonomy. This awareness and understanding of ethical practice is an important first step toward responsible behavior and has indeed been reported to stimulate PhD’s to have conversations about research integrity, with fellow PhD students, someone close to them outside research practice, but also with their supervisor (Abdi et al. 2021b).

Our findings stress the importance of engaging learners in active and interactive experiences, and to facilitate discussion about how to deal with integrity cases that may arise in daily practice instead of preventing misconduct. Kalichman (2014) characterized this as taking a positive approach and Kalinovska, Koterwas & Dwojac-Matras (2020) as “helping people to determine how to act properly during each step of the research process” (p.43). In our approach, collaborative case-discussions proved a key element for participants. As Barak and Green (2020) also report, novice researchers identify the integration of collaborative and case-based learning as the key instructional design component that may foster not only ethical knowledge but also ethical practice in a fully online course.

Facilitating social interactions in online courses can be achieved through the utilization of online forums or live meetings, where one can actively engage in ethical discussions while receiving valuable support and suggestions. Also, online discussions provide a unique opportunity for novice researchers to openly express their thoughts in a manner that may not be possible in traditional classroom settings, as it creates a ‘safe zone’ for learners (Barak and Green 2020). Learners with similar research interest have the chance to exchange ideas and achieve meaningful learning outcomes (Usher and Barak 2018). Such open and honest discussions are crucial for ethics education, as certain research practices often reside in a gray area where even experienced researchers may struggle to make ethical decisions (Shamoo and Resnick, 2015). Deliberation on practical cases has been identified as a successful approach for promoting awareness and ethical behavior in daily practice (Watts et al. 2017).

In our study, it proved difficult to promote empowerment in the day-to-day collective, institutional context that novice researchers work in. Relatively few participants felt capable to handle integrity issues in daily practice and they scored relatively low on communicative aspects, such as becoming an active bystander, consulting others on RCR issues and being courageous to address integrity issues with colleagues. This seems related to the challenges that participants perceive in their social research environment, namely a perceived “lack of power to make changes” in the vulnerable position PhD candidates still have in the academic community (e.g., tenured contracts, lower in the hierarchy). Environmental factors, also indicated as research culture or research climate, have been reported as important factor influencing research behavior (e.g. Zwart & ter Meulen 2019; Haven et al. 2020). The idea that institutions should stimulate open dialogue on integrity issues, emphasizes the shift “from its dominant focus on individual actors and (….) to accept that the research culture has a role in sustaining integrity (and discouraging questionable research practices)” (Mejlgaard et al. 2020, p.359). In our view empowerment towards RCR is needed at the individual, group and institutional level, as suggested by Israel et al. (1994). Education of PhD’s in research integrity may contribute to a climate of responsible conduct of research in academia on the long run, but there are limits to its impact. More intensive training (e.g., for a longer period of time) and training at the group level is necessary (Haven et al. 2022). Developing the critical autonomy of supervisees’ asks for supervision practices and role modelling that reflect ethical behavior and there is an existing but growing demand to provide RCR training for supervisors alongside PhD students (Gray and Jordan 2012; Muthanna and Alduais 2021; Pizzolato and Dierickx 2022) along with the recognition of the importance of addressing research culture through workshops for departments and research groups to promote positive changes in research behavior.

The primary contribution of our study to the RCR education literature lies in our implementation of an explicit teaching philosophy centered around “fostering a positive research culture”. We operationalized this through the lens of empowerment. This aligns with the current trend of emphasizing RCR as a positive stance towards research practice and teaching novice researchers what responsible research requires (Steneck 2007; Kalichman 2014; van den Hoven & Krom 2020). This requires integration of specific skills, knowledge and attitudes as outcome measures which together shape novice researcher’s critical autonomy to address integrity issues in one’s project as well as in their broader institutional context. Our findings do not only indicate that an online course can indeed contribute to the RCR empowerment of PhD candidates, but also stresses its importance, as the experiences of PhD candidates reveal how a lack of empowerment can hinder research integrity. Future research could continue to build on this teaching philosophy by further investigating successful ways to empower PhD candidates beyond the individual level and foster a positive research culture.

Limitations to the study

First, course participants voluntarily subscribed to the course, indicating a higher level of interest and positive stance towards responsible conduct of research than the average PhD candidate. If the course had been mandatory, it is possible that a different view emerged on participants’ perceived empowerment. This potential bias is reinforced by the significant number of dropouts at the beginning of the course which occurred as soon as the participants realized the workload involved. As a result, those who remained active in the course likely had a relatively high motivation for RCR and a strong determination to finish the course. In fact, most interviewed participants expressed they were already motivated to be a responsible researchers before enrolling in the course.

An additional limitation of the study is the challenge to determine the extent of impact a four-week course can have on the empowerment of participants. While the interviews were intentionally conducted a few weeks after the course’s completion, this timeframe only provides a short-term impression of the course’s effects on participants. We consider the self-reported empowerment of participants an initial indicator of the potential of our course to stimulate RCR empowerment. However, further research is necessary to fully understand and establish the practical implications for young researchers within their own research practices. To further validate the findings of our exploratory study, it would be beneficial to conduct a follow-up study that incorporates quantitative elements, such as a questionnaire with a pre- and post-test design, to retest the findings obtained in this study. Furthermore, conducting a longitudinal intervention, such as a follow-up study one year later, would provide more insights into the ongoing development of course participants as they progress in their careers as professional researchers and how they prioritize research integrity.