Dan Stuart PhD, John Griswold MD, Cathy Lovett RN
Corresponding author: Dan Stuart
Contact Information: Daniel.Stuart@ttuhsc.edu
DOI: 10.12746/swrccc.v10i42.977
Introduction: The instruction of evidence-based medicine (EBM) in health sciences institutions is increasing. As used in medical schools, it looks to implement research-oriented approaches to clinical interventions and further develop the skills of medical students in evaluating and applying research.
Objectives: This study aimed to identify the effectiveness of an evidence-based medicine clerkship offered at a single institution to third-year medical students during their surgery rotation.
Methods: Two surveys were used to assess the quality of education provided during the third-year clerkship. The first was sent to practicing residents who had completed the clerkship as medical students. The second was sent to residency directors overseeing residents with prior enrollment in the clerkship.
Results: Twelve former students (out of total email list of 125) and six residency program directors (out of an email list of 13) completed this survey. Previous students felt confident about EBM fundamentals like question formulation and database navigation and, to a lesser degree, attributed their knowledge of databases and critical appraisal to their clerkship experience. Residency program directors acknowledged that residents had good EBM-related skills, but it was inconclusive as to whether this could be attributed to prior clerkship experience.
Conclusion: Both questionnaires attested to the retention of some EBM competencies and the effectiveness of librarian-led instruction. Feedback was particularly positive in areas such as navigating information resources and evaluating peer-reviewed research.
Keywords: Evidence-based medicine, libraries, clerkships, residents
Understood by those who practice it to be the “conscientious, judicious, explicit use of the current best evidence in making decisions about the care of individual patients,”1 Evidence-Based Medicine (EBM) is an important component in the education of medical professionals. Combined with other medical training, it seeks to provide a more qualitative approach to clinical care by providing more informed input for clinical decision-making.2–4 The ever-increasing amount of biomedical literature and information resources presents opportunities and challenges for medical professionals to apply relevant skills and further incorporate EBM education into existing curricula.
EBM teaching has become more established in health sciences institutions in the last two decades and has been integrated into undergraduate medical education in both the clinical and pre-clinical years.5–8 A 2014 survey of US and Canadian medical schools revealed that over 100 medical schools (out of approximately 170) incorporate some form of EBM teaching in their 4-year curriculum.9 Subsequent studies investigate both the prevalence of EBM instruction and modalities used in undergraduate medical education,10 and found a number of different approaches, including flipped classroom settings,1,11 online and interactive modules, hybrid courses,12 peer-assessment seminars,13,14 and longitudinal courses.15 The effectiveness of such training can vary but has increased overall competence in evidence-based medicine knowledge.
Librarians and information professionals have frequently participated in these efforts, contributing to the instruction of EBM core competencies and in curriculum development.16,17 Recent investigations into participation by librarians reveal their effectiveness at promoting EBM skills.18,19 They serve a valued role in introducing students to fundamentals, such as formulating PICO-informed clinical queries,20,21 developing search strategies, navigating information resources to acquire relevant research, and appraising the evidence.22 A recent evaluation of librarians in evidence-based medicine curricula showed improved physician EBM skills, though concern remains that students, having participated in EBM education as undergraduates, may not sufficiently retain the information going forward.23
Among the methods of EBM instruction, one used consistently has been its incorporation within clinical clerkships.8,24 Typical third-year curricula feature clinical rotations of six to eight weeks during which students gain clinical experience in various disciplines. Since the early 2000s, some clerkships have found ways to integrate EBM instruction with students’ clinical experience on select rotations. Supported by hybrid or blended-learning models, third-year clerkships are used by a sizable proportion of institutions to conduct formal EBM training25 with librarian-led components shown to be a vital part of the process. The present study seeks a similar objective: to answer the question about the effectiveness of third-year EBM clerkships during clinical rotations. Specifically, we investigated the retention of EBM competencies by residents who previously took the EBM clerkship as medical students. By assessing how well resident physicians incorporate EBM skills like evidence acquisition and appraisal, we can better understand the effectiveness of EBM instruction when delivered in the context of a clerkship.
Our investigation assessed the effectiveness of the EBM surgery clerkship conferences in residents of the Texas Tech University Health Sciences Center (TTUHSC), Lubbock, TX. Medical students in the institution’s School of Medicine average around 80–95 per cohort at the Lubbock Campus where the study was done.26 The Surgery clerkship, from which this study drew its data, has approximately 12 students per rotation.
During the eight-week rotation on surgery, students receive three in-person instruction sessions. An orientation at the beginning of each rotation briefs students on the clerkship’s criteria and lectures on the fundamentals of EBM, using criteria from Gordon Guyatt’s User’s Guide to Medical Literature for its textbook.27 This session also includes an introduction to information resources, mostly point-of-care tools, such as UpToDate, DynaMed Plus, and ClinicalKey Overviews. The second session is either a classroom lecture or 1-on-1 consultation with a paired librarian during which students receive tutoring on how to search in MEDLINE and other research databases. The final session involves a discussion with a surgeon on specific clinical scenarios. During each clerkship, students are responsible for devising their own PICO questions on a topic of their choice. The query is done with a template in which students must break down the question’s terms into synonyms and apply it to a clinical category (therapy, prevention, diagnosis, etc.). An article should be chosen that corresponds to this question and a corresponding appraisal worksheet must be completed in time for the discussion.
We targeted students taught during rotations occurring between 2016 and 2019. Following an established protocol to examine responses from former EBM students who were now residents on the Lubbock campus, two separate questionnaires of nine and ten questions were sent to 13 program directors and 125 residents. Both surveys were collaborated on and constructed by a reference librarian, current physician heading EBM Surgery clerkship instruction, and personnel from the Health Sciences Center Clinical Research Institute. The Institutional Review Board approved the protocol and study. A data coordinator input the surveys into Qualtrics (https://qualtrics.com) and equipped each with a permanent link to be emailed as formal recruitment letters to clerkship coordinators. Each recruitment letter detailed the purpose of the study and made clear it was being used as feedback for the current EBM curriculum. Residents and Program Directors were informed that they were granting consent to participate in the research study by submitting the survey.
One email at the start of the study was sent to residents and program directors. Three successive emails were sent to residents every six to eight weeks. The study concluded after four months. The resident survey (Appendix A) was first sent out in November of 2020 and queried practicing residents who were former medical students. Questions gathered res-ponses on department and clinical programs and then asked about information-seeking behavior when confronted with a clinical situation. Resident surveys then targeted feedback on specific resources used. A self-rating metric for clinical appraisal assessed the self-perceived proficiency in EBM appraisal skills. Program director surveys (Appendix B) were sent out simultaneously, again through clerkship coordinators. These gathered information on each director’s clinical specialty and determined the level of interaction with residents. This survey was emailed only once.
Existing clerkship rosters were used to collect resident names and contacts. Subsequent information was obtained from the institution’s match lists on the university website, and residency or clerkship coordinators were identified through residency program web pages. Recruitment letters in emails were initially sent to coordinators, who then submitted them to program directors or residents. Each incorporated a secure data platform with an embedded link for either the resident or program director survey. Assessment tools within Qualtrics were used to tabulate responses for exportation and data management, hosting two separate survey portals for questions and recorded input. Responses from all recruited participants were cataloged and used to create detailed response for all inquiries. Subsequent reports from both surveys were exported to Excel and Google Sheets platforms for further analysis. Research assessments were performed on selected questions and graphed accordingly.
Survey data were collected from 13 residency programs in the School of Medicine at the TTUHSC Lubbock campus. Clerkship coordinators, program directors, and residents were all involved in the process of data retrieval. Results were compiled following the third submission of the resident survey. Among former students who are now residents in TTUHSC programs at the Lubbock campus, 15 out of 125 responded with 12 completing the entire survey. Seven program directors out of 13 responded; these seven answered all questions.
Among the seven program directors, six answered that they were directly involved in the oversight of residents in clinical situations. The initial four questions of the survey inquired about evidence-based approaches to clinical interventions while two asked to rate resident ability at critical appraisal. Directors identified the relationship between evidence and clinical care as important, with all qualifying it as “Somewhat Important” or greater on a five-item scale; over half the respondents rated it as “Very Important” or “Essential.” All indicated their residents had at least an “Average” ability or better to locate relevant clinical evidence and integrate it with clinical situations (Table 1). The “Average” rating on these questions represented the middle index across all questions covering evidence-based approaches, notably on the topic of finding and acquiring evidence-supported research for clinical interventions.
Table 1. Program Director Assessment of Resident Ability to Find Evidence
| Count | Percentage | |
| Minimal | 0 | 0% |
| Below Average | 0 | 0% |
| Average | 2 | 33.33% |
| Above Average | 3 | 50% |
| Outstanding | 1 | 16.67% |
| Total | 6 | 100% |
For questions on information resources, program director answers were equally distributed across multi-option and ranked scales. Point-of-Care reference tools such as UpToDate or DynaMed were the highest selected resources with a six-count preference rating; five rated Research databases as the most preferred (Table 2). Practice guidelines and filtered research reports (Systematic Reviews, Meta-Analysis, etc.) were the most referenced study types. Program directors answered two questions related to appraisal. Both queries used Likert-scale surveys that, when tabulated together, showed half of the respondents indicating resident ability at “Above Average” or “Outstanding” (Table 3).
Table 2. Program Director Assessment of Resources Used by Residents to Gather Evidence for Clinical Situations
| Resource Type | Count | Percentage |
| Point-of-Care Tools (UpToDate, DynaMedPlus, Lexicomp, etc.) | 6 | 27.27% |
| Research Databases (PubMed, CINAHL, Embase, etc.) | 5 | 22.73% |
| Web Resources (Google, Wikipedia, Medscape, etc.) | 5 | 22.73% |
| Colleagues/Professional Input | 6 | 27.27% |
| Other | 0 | 0.00% |
| Total | 22 | 100% |
Respondents could select multiple options.
Table 3. Program Director Assessment of Resident Critical Appraisal Ability
| Answer | Count | Percentage |
| Minimal | 0 | 0.00% |
| Below Average | 0 | 0.00% |
| Average | 4 | 66.67% |
| Above Average | 2 | 33.33% |
| Outstanding | 0 | 0.00% |
| Total | 6 | 100% |
Resident survey respondents numbered 15 total out of a 125-person sample of physician-residents, of whom 12 answered all questions. Initially, all were identified as being alumni of the Lubbock campus’s School of Medicine through grad-year match lists that publicized residency and institution. This was later confirmed through a process of elimination in which email accounts were verified or unverified.
Respondents rated their ability at formulating relevant clinical questions at “Average” to “Above Average” with no outliers. When asked about acquiring the best evidence, results were varied, with only half denoting an “Above Average” or “Outstanding” qualification on a Likert-scale query. Research Data-bases and Point-of-Care tools ranked as the most utilized resources consulted. Responses for critical appraisal of evidence-based sources were evenly dispersed among “Average,” “Above average,” and “Outstanding.” Seven of the 12 participants attributed their own EBM knowledge to prior training in undergraduate medical education; these responses and did not specify the clerkship itself (Table 4). For skills attained during that time, “Resource Selection & Database Searching” was the most selected skill. No further comments were added to the survey on the question about the quality and further improvement of the EBM Surgery Clerkship.
Table 4. Resident Attribution of EMB Skills to Training During Undergraduate Medical Education
| Answer | Count | Percentage |
| None at all | 1 | 8.33% |
| To some extent | 4 | 33.33% |
| Quite a bit | 7 | 58.33% |
| Very much so | 0 | 0.00% |
| Total | 12 | 100% |
Both questionnaires attested to the relative competence of medical residents in EBM skills although clerkship training was not specified as the precursor. Despite this and other limitations including overall survey participation, EBM education was identified as an effective method for introducing students to research-based clinical initiatives. Prior clerkship students felt confident about incorporating EBM protocols like question formulation and database navigation. Program directors acknowledged that residents had a high proficiency at EBM-related skills, although this was not definitely correlated with prior clerkship experience.
The program director survey was instrumental in surveying senior physician perspectives on EBM integration. Quantitatively it assessed the degree of confidence physicians had in resident research abilities and provided some insight into the research methods and information resources used. As expected, physicians incorporated a hybrid approach to obtaining information pertinent to clinical interventions and found research data through multiple platforms and study categories. Directors felt that resident ability to appraise evidence was strong and indicated that application of appraised evidence was at a high level.
Residents also responded with favorable answers, but the number of completed surveys precluded statistical analysis. The survey responses indicated a variety of results when approaching, transcribing, and resolving a clinical situation. Self-rated scores hovered around “average” based on the distribution concerning the answering of clinical questions. Critical appraisal skills were likewise reported to be “average.” The scaled score identified “Average” as the most selected option with a slight skew toward “Above Average.” Respondents said they used a balanced variety of information resources to acquire evidence, revealing a near equivalent use of research databases, point-of-care tools, and further web resources. Databases, such as PubMed, MEDLINE, and Embase, had a higher priority than point-of-care platforms like DynaMed Plus or UpToDate. Other sources, though unspecified, were clearly the last choice among residents when navigating medical evidence.
More intuitive information was gathered through the final inquiries into the clerkship’s influence on EBM skills. Most residents reported attributing their knowledge to “prior training during undergraduate medical education” as “quite a bit” as opposed to “to some extent” and “not at all.” Further data collected on specific skills retained identified “Resource selection & database searching” as the most valued information attained during the surgery clerkship. “Evidence appraisal” and “question formulation” were also listed, counted as “still use[d]” by nearly half of respondents. Though residents in the open-ended section provided no further information, tabulated results identified positive indicators for EBM instruction and its use in clinical research.
Limitations to this study were significant. Participation by the program directors was around half, but only a small number of residents completed the survey. This restricted overall analysis as data acquired from the surveys was not enough for a thorough assessment. In addition, resident feedback seemed restricted by survey design as only self-perceived ability at EBM skills was examined. Program director surveys were similarly limited by the degree to which directors knew about resident approaches to the literature, use of information resources, and appraisal methods. Without a larger sample size, evaluation of EBM skills and the clerkship’s effectiveness remains inconclusive.
Responses from both questionnaires revealed average to above-average degrees of EBM skills competence of the participating residents. Both residents and program directors indicated a positive connection between EBM education and its use in clinical experience. Residents highlighted participation in the clerkship as a positive experience, although this could not be calculated with any statistical significance. Also, retention of EBM competencies from clerkship instruction could not be determined with certainty. And while limitations to the study included participation count and response rate, further limitations included the risk of bias related to the self-perception of residents about their own EBM proficiencies.
Residency program directors acknowledged that residents had good EBM-related skills, although this was not correlated with prior clerkship experience. Despite limitations, the data correlated with prior feedback from clerkship students. Post-clerkship surveys conducted following each surgery clerkship also reveal positive feedback. Ongoing EBM instruction should benefit from the information gathered. More objective assessment and more conclusive data could be obtained through on-site or workplace-based assessments of these residents’ use of information resources and appraisal skills. Hands-on observation of their integration of evidence into clinical decision-making might also prove valuable.
The protocol and IRB protocol for this study were developed with the assistance of the Health Sciences Center Clinical Research Institute (CRI). In conjunction with the CRI, survey construction and data capture were accomplished. A CRI biostatistician was consulted on how best to integrate survey results, but did not contribute to the final product. The university’s information technology department was consulted to ensure proper survey input into Qualtrics. Data for this report can be made available upon request.
Article citation: Stuart D, Griswold J, Cathy Lovett C. Effectiveness of evidence-based medicine instruction in medical school clerkships. The Southwest Respiratory and Critical Care Chronicles 2022;10(42):35–41
From: Libraries (DS), Department of Surgery (JG), Clinical Research Center (CL), Texas Tech University Health Sciences Center, Lubbock, Texas
Submitted: 12/20/2022
Accepted: 1/17/2022
Conflicts of interest: none
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.