Abstract

Predictors of Performance on USMLE Step 1

Winnie Wu BA*, Katy Garcia MA*, Sheila Chandrahas BS*, Arham Siddiqui MBA*, Regina Baronia MD, Yasin Ibrahim MD

Corresponding author: Yasin Ibrahim
Contact Information: Yasin.Ibrahim@ttuhsc.edu
DOI: 10.12746/swrccc.v9i39.813

ABSTRACT

Background: Ninety-four percent of program directors cited the USMLE Step 1 score as the most important factor in determining applicant competitiveness for residency. Thus, medical students are motivated to attain the highest possible scores. Since Step 1 recently switched to a pass/fail standard, it is important to analyze factors that predict meeting this goal.

Objective: To investigate the factors that can influence or predict performance on USMLE Step 1.

Methods: We conducted a systematic literature search on PubMed, Web of Science, Scopus and ERIC in 2019. Key words included “USMLE,” “Step-1,” “score,” “success,” and “predictors.” The search included articles published between 2005 and 2019. Studies that did not focus on Step 1 outcome or allopathic medical students in the United States were excluded.

Results: Two hundred seventy-five articles were found, 29 of which met our inclusion criteria. Analysis of these articles demonstrated that predictors of USMLE Step 1 score can be divided into unmodifiable and modifiable factors. Unmodifiable factors include gender, MCAT score, pre-clinical grades and NBME/CBSE scores. Modifiable factors include taking USMLE Step 1 within two months of completing pre-clinical courses, motivation from anxiety, multiple-choice questions completed, unique Anki cards seen, and complete passes of First Aid for the USMLE Step 1.

Conclusion: Our review suggests that although students can focus on modifiable factors to increase their score, the energy expenditure required to increase the Step 1 score by one point is unrealistic. This may have influenced the NBME’s decision to change Step 1 to a pass/fail exam.

Keywords: USMLE Step 1, performance predictors, medical school, medical student, test preparation, Step 1 modifiable predictors

INTRODUCTION

The license to practice medicine in the United States requires passing the United States Medical Licensing Examination (USMLE). USMLE Step 1 is the first of the three licensure examinations taken by medical students to assess the examinee’s understanding of, and ability to apply, basic science concepts to medical practice.¹ Since grading systems are inconsistent among schools, USMLE Step 1 provides a more objective method of assessment. In fact, 94% of residency program directors put an applicant’s USMLE Step 1 score as the most important factor in determining their competitiveness for admission.²

In February 2020, the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners^® (NBME^®), co-sponsors of the United States Medical Licensing Examination^® (USMLE^®), announced changing the Step 1 score reporting from a three-digit numeric score to reporting only pass/fail with a passing score of 194.¹ This decision is aimed at protecting medical students’ wellbeing by decreasing the stress they usually experience while struggling to achieve the highest possible score. In one to two years, when the decision goes into effect, the students’ focus will shift toward passing the test rather than achieving a high score. As is the case with the USMLE step 2 Clinical Skills (CS) exam, passing the Step 1 test on the first attempt will carry significant weight in assessing residency applicants’ competitiveness.

Due to the significance of Step 1 for both medical students and residency programs, multiple studies have examined factors that may predict Step 1 performance. The goal of this study is to conduct a review of prior research and identify which factors are significantly associated with passing Step 1. The predictors will be categorized as either unmodifiable or modifiable factors. Unmodifiable factors are concrete student characteristics associated with increased performance on USMLE Step 1. Modifiable factors are actions that students can take to increase their performance on USMLE Step 1. Particular emphasis will be placed on modifiable factors.

METHODS

A systematic literature search was conducted using PubMed, Web of Science, Scopus and ERIC. A systematic review was chosen to give a clear and inclusive overview of the relevant studies on USMLE Step 1 performance. Key words were a combination of the following: USMLE, Step-1, score, success, predictors. Our criteria specified that articles must be published within the last 15 years (2005–2019), with the most recent publication on May 22, 2019. The initial literature search yielded 275 articles, which were then narrowed to 45 articles. Additional criteria specified that selected articles must focus on the USMLE Step 1 outcome and allopathic medical schools located in the United States. All articles that did not meet these criteria were excluded, resulting in 27 articles that were reviewed and agreed upon by all authors of this study. An additional two studies were identified manually after the initial search process and included after agreement among study investigators about the new data. A total of 29 studies were included in this review. Figure 1 details the process of our systematic literature search based on the addressed criteria.

Figure 1. Systematic Literature Search Process.

RESULTS

Articles were selected based on the inclusion criteria and subject number with particular attention to those focused on passing/failing USMLE Step 1. While our focus is on passing/failing, the current pool of research focuses on scoring well since the implementation of a pass/fail Step 1 is recent. Of the 29 articles, 3 specifically looked at whether the outcome was pass/fail. Table 1 (located after references) summarizes our findings. There was an average of 9,139 (SD: 23,653) students included in the studies with a range of 48 to 97,445. Analysis of 29 articles meeting the inclusion criteria demonstrated that predictors of USMLE Step 1 performance can be divided into unmodifiable and modifiable factors. Unmodifiable factors are defined as factors that cannot be changed during the dedicated study period and include: MCAT score, gender, pre-clinical grades, NBME/CBSE (Comprehensive Basic Sciences Exam) scores. Modifiable factors are those that can be modified during the dedicated study time and include: number of multiple-choice questions completed, number of unique Anki cards seen, review book usage, test anxiety, perception of learning environment, timing of taking Step 1.

Table 1. Overview of Studies

Many of the studies examined similar variables and had overlapping findings, such as a positive correlation between MCAT score, multiple choice questions completed, unique Anki cards, review book usage and Step 1. In contrast, some factors did not follow those themes. At one school, students who participated in a peer-led Step 1 review course had higher average Step 1 scores than non-participants.^3,4 Andriole and Jeffe examined race and premedical debt and found a higher likelihood of not passing Step 1 on the first attempt for underrepresented minorities and those who had a premedical debt of at least $50,000.⁵ Test anxiety was negatively correlated with Step 1 performance.^6,7 Interestingly, students who had higher perceptions of adequacy had better performance.⁸ The perception of a meaningful learning environment, emotional climate and student-student interaction were positively associated with Step 1 performance.⁹

DISCUSSION

UNMODIFIABLE FACTORS

Unmodifiable factors are concrete student characteristics associated with performance on USMLE Step 1. For the purpose of this article, grades in years 1 and 2 as well as NBME/CBSE performance are considered unmodifiable.

MCAT SCORE

The MCAT score was a strong predictor of Step 1 performance.¹⁰ Individuals with MCAT scores ≤29 (pre-2015) were more likely to not pass Step 1 on the first try.⁵ In addition, as the number of times an individual took the MCAT increased, the Step1 score decreased.¹¹ The correlation between MCAT performance and Step 1 achievement is not consistent across all schools. Ghaffari-Rafi et al. found students at a university with lower MCAT scores outperformed their expected average on USMLE Step 1.¹² Therefore, MCAT scores cannot be said to be the sole predictor of performance as this effect may be mediated by curriculaspecific to individual medical schools (e.g., integrating early clinical exposure into years 1 and 2).

UNDERGRADUATE GPA

There is a positive relationship between the undergraduate GPA and unimpeded progress toward medical school graduation, such as passing USMLE Step 1; however, this effect levels off as the GPA increases above 3.5.¹³ There was also a statistically significant correlation between the average Step 1 score and the median GPA.¹²

PRE-CLINICAL GRADES

Other researchers found that performance in pre-clinical classes was a much stronger predictor of USMLE step 1 performance than pre-admission variables such as MCAT scores.¹⁴ In particular, Giordano et al. found a significant correlation with the USMLE Step 1 score and straight As during the first year of medical school.¹⁵

Achievement in individual courses was also strongly associated with Step 1 performance. Anatomy is a strong overall predictor, with both class rank in medical gross anatomy and score on a comprehensive gross anatomy final significantly correlating with passing Step 1.¹⁶ This effect is also seen among students who opted to take a gross anatomy course immediately prior to medical school matriculation.¹⁷ In addition, a specialized focus in Anatomical Sciences positively influences USMLE Step 1 performance. Fredieu and Snyder found that USMLE step 1 scores were significantly higher for students enrolled in their MD/MS program than for the MD program.¹⁸ This enhanced performance may be due to serving as anatomy TAs for the medical school by providing the opportunity to further increase their knowledge.

Performance in pre-clinical courses during year 2 also had a strong predictive ability. Performance in these courses accurately predicted 97.2% of those who passed and 80% of those who failed Step 1.¹⁹ Two courses with the highest failure rate, Organ Systems and Human Structure, were significantly correlated with USMLE Step 1 performance.²⁰ Pathology coursework was also found to be a predictor of USMLE step 1 scores.²¹

CBSE/NBME

Standardized exams created by the National Board of Medical Examiners were also found to be predictors of USMLE Step 1 performance. The Comprehensive Basic Sciences Exam (CBSE) allows students to gauge readiness for USMLE Step 1. Performance on USMLE Step 1 is significantly correlated with the performance on the CBSE taken immediately preceding the dedicated study period.¹⁵ In addition, CBSE results with scores less than 66 were significantly associated with failure of USMLE Step 1.²²

DEMOGRAPHICS

Gender, student debt, underrepresented minority status, age, and clinical experience were predictors of USMLE Step 1 performance.²³ After controlling for MCAT scores and undergraduate GPAs, men scored 1.5 points higher on Step 1 than women. In addition to female gender, underrepresented minorities in medicines, older matriculation age, and premedical debt of at least $50,000 were associated with increased risk of poor performance on Step 1.⁵ However, Giordano et al. found that financial need while in medical school is a statistically significant positive predictor of USMLE Step 1 score.¹⁵

Students with formal clinical experience prior to medical school perform better on USMLE Steps 1 and 2 than their counterparts without clinical experience.²⁴ Formal clinical experience is defined as employment in the healthcare field for any amount of time or scribing for at least 6 months prior to matriculation. Shadowing was not considered clinical experience. In conjunction, these studies suggest that if a student is going to take time off between college and medical school, working in the healthcare field may be beneficial.

The demographics of the medical school have also been shown to influence the USMLE 1score. National Institutes of Health research grants provided to either the medical school itself or to individual faculty members are associated with better student performance on Step 1.¹² In addition, students attending a private medical school scored, on average, 5 points higher than students attending public medical schools.

In contrast to many other studies, Giordano et al. did not find a relationship between the USMLE Step 1 Score and MCAT score, gender, prior medical training, or age.¹⁵

MODIFIABLE FACTORS

Modifiable factors are actions that students can take to increase their performance on USMLE Step 1. These include studying prior to the designated study period, utilizing a 700-page review book, multiple choice questions, and Anki cards, participating in peer led tutoring, timing of when Step 1 is taken, and psychological factors.

STUDY HABITS

Students who began studying prior to the dedicated study period and increased their study time by averaging 11.0 hours per day over a period of 35.3 days performed 4.2 points better on USMLE Step 1.²⁵ Each additional cover to cover reading of the review book was associated with a 2.3 point increase. These effects remained even after controlling for MCAT scores, pre-clinical exam performance, and self-identified score goal. Each additional 445 boards-style questions or 1700 unique Anki flashcards were associated with a one-point increase on Step 1.⁶ The percentage correct of USMLE World (UWorld) Question Bank was also associated with increased performance on Step 1.¹⁵ Commercial preparatory courses were not associated with higher USMLE Step 1 score.²⁶

PEER LED TUTORING AND GROUP STUDY

Participation in a peer-led review course was associated with higher Step 1 scores.³ Although many participants felt that the course did not decrease their anxiety or increase confidence in their performance, the presence of anxiety may have been a powerful motivating factor as demonstrated by the higher scores of participants than nonparticipants.

Although half of students surveyed reported group studying was helpful to them, it was not significantly correlated with exam performance.²⁷

TIMING OF USMLE STEP 1

The timing of when Step 1 was taken was not significantly associated with differences in scores as long as the exam was taken within two months of finishing second year courses. After two months, scores declined as time between finishing pre-clinical material and taking Step1 increased.²⁶

PSYCHOLOGICAL FACTORS

Psychological factors, such as burnout, feelings of depression, and test anxiety, were examined by Alcamo et al.³ Test anxiety, depression, and burnout were all significant negative predictors of exam performance.⁶ Another study by Green et al. supports this relationship by demonstrating a modest inversely correlation between test anxiety and USMLE step 1 scores.⁷ Anxiety of students who participated in a test-taking course decreased after completing Step 1 while anxiety among the control group increased. However, the control group still outperformed the intervention group on Step 1.

LIMITATIONS

One of the major limitations to the present study was the relatively few studies that focused on modifiable factors rather than unmodifiable factors (i.e., MCAT score, gender, pre-clinical grades, etc.). Studies that did focus on modifiable factors predominantly examined board preparatory resources, which were correlated with improved scores across the board for all students; however, it may be of greater benefit to instead categorize resources based on student learning styles. Categorizing board preparatory resources using these parameters may allow students to gain maximum benefit from their study efforts. For instance, Kim et al. found that students with reading/writing learning style preferences performed better on Step 1 than those with multimodal preferences.²⁸ This conclusion is logical considering that Step 1 requires students to read text-heavy questions and extract pertinent information in a time efficient manner. It follows that students with a predisposed affinity for reading/writing tend to score higher than students who do not have this learning style preference. However, Lynch et al. found a significant positive correlation between Step 1 scores and abstract thinking.²⁹ Considering that both of these studies show improved scores based on students’ innate mental qualities, future studies that correlate learning preferences with appropriately oriented preparatory resources could have important implications for students.

Furthermore, it is plausible that categorizing factors associated with modifiable factors is an unachievable task as many factors contribute to success on USMLE Step 1. These factors could include differing motivations for achievement, differing student instruction among medical schools, and emphasis on Step 1 success by different specialties.

CONCLUSION

Many studies focused on unmodifiable factors, such as MCAT score, gender and pre-clinical grades; however, there are also several modifiable factors that are strongly associated with a higher score on a student’s USMLE Step 1 score. Specifically, medical students can increase the number of multiple choice questions completed, unique Anki cards seen, and complete passes of First Aid for the USMLE Step 1 to increase their scores. However, the work effort needed to add one point to students’ Step scores can be inefficient, discouraging, and unrealistic. This finding may have influenced the NBME’s recent decision to change Step 1 to a pass/fail exam.

REFERENCES

1. Change to pass/fail score reporting for Step 1. United States Medical Licensing Examination. 2020.
2. Results of the 2018 NRMP Program Director Survey. The Match National Resident Matching Program. 2018.
3. Alcamo AM, Davids AR, Way DP, et al. The impact of a peer-designed and -led USMLE Step 1 review course: improvement in preparation and scores. Acad Med 2010; 85(10 Suppl):S45–48.
4. Schwartz LF, Lineberry M, Park YS, et al. Development and Evaluation of a Student-Initiated Test Preparation Program for the USMLE Step 1 Examination. Teach Learn Med 2018; 30(2):193–201.
5. Andriole DA, Jeffe DB. Prematriculation variables associated with suboptimal outcomes for the 1994–1999 cohort of US medical school matriculants. JAMA 2010;304(11):1212–1219.
6. Deng F, Gluckstein JA, Larsen DP. Student-directed retrieval practice is a predictor of medical licensing examination performance. Perspect Med Educ 2015;4(6):308–313.
7. Green M, Angoff N, Encandela J. Test anxiety and United States Medical Licensing Examination scores. Clin Teach 2016;13(2):142–146.
8. Khalil MK, Wright WS, Spearman KA, et al. Relationship between students’ perceptions of the adequacy of M1 and M2 curricula and their performance on USMLE step 1 examination. BMC Med Educ 2019;19(1):358.
9. Wayne SJ, Fortner SA, Kitzes JA, et al. Cause or effect? The relationship between student perception of the medical school learning environment and academic performance on USMLE Step 1. Med Teach 2013;35(5):376–380.
10. Casey PM, Palmer BA, Thompson GB, et al. Predictors of medical school clerkship performance: a multispecialty longitudinal analysis of standardized examination scores and clinical assessments. BMC Med Educ 2016;16:128.
11. Zhao X, Oppler S, Dunleavy D, et al. Validity of four approaches of using repeaters’ MCAT scores in medical school admissions to predict USMLE Step 1 total scores. Acad Med 2010;85(10 Suppl):S64–67.
12. Ghaffari-Rafi A, Lee RE, Fang R, et al. Multivariable analysis of factors associated with USMLE scores across U.S. medical schools. BMC Med Educ 2019;19(1):154.
13. Dunleavy DM, Kroopnick MH, Dowd KW, et al. The predictive validity of the MCAT exam in relation to academic performance through medical school: a national cohort study of 2001-2004 matriculants. Acad Med 2013;88(5):666–671.
14. Gohara S, Shapiro J, Jacob A, et al. Joining the conversation: Predictors of success on the United States Medical Licensing Examinations (USMLE). The Learning Assistance Review 2011;16:11–20.
15. Giordano C, Hutchinson D, Peppler R. A Predictive Model for USMLE Step 1 Scores. Cureus. 2016;8(9):e769.
16. Peterson CA, Tucker RP. Medical gross anatomy as a predictor of performance on the USMLE Step 1. Anat Rec B New Anat 2005;283(1):5–8.
17. Tucker RP. Performance in a prematriculation gross anatomy course as a predictor of performance in medical school. Anat Sci Educ 2008;1(5):224–227.
18. Fredieu JR, Snyder CW. Positive impact of a master of science in applied anatomy program on USMLE Step 1 performance. Anat Sci Edu 2015;8(1):31–36.
19. Coumarbatch J, Robinson L, Thomas R, et al. Strategies for identifying students at risk for USMLE step 1 failure. Fam Med 2010;42(2):105–110.
20. Gandy RA, Herial N, Khuder S, et al. Use of curricular and extracurricular assessments to predict performance on the United States Medical Licensing Examination (USMLE) Step 1: A Multi-Year Study. The Learning Assistance Review 2008;13:27–35.
21. Hu Y, Martindale JR, LeGallo RD, et al. Relationships between preclinical course grades and standardized exam performance. Adv Health Sci Educ Theory Pract 2016; 21(2):389–399.
22. Guiot HM, Franqui-Rivera H. Predicting performance on the United States Medical Licensing Examination Step 1 and Step 2 Clinical Knowledge using results from previous examinations. Adv Med Educ Pract 2018;9:943–949.
23. Cuddy MM, Swanson DB, Clauser BE. A multilevel analysis of examinee gender and USMLE step 1 performance. Acad Med 2008;83(10 Suppl):S58–62.
24. Shah R, Johnstone C, Rappaport D, et al. Pre-matriculation clinical experience positively correlates with Step 1 and Step 2 scores. Adv Med Educ Prac 2018;9:707–711.
25. Burk-Rafel J, Santen SA, Purkiss J. Study behaviors and USMLE Step 1 performance: implications of a student self-directed parallel curriculum. Acad Med 2017;92(11S Association of American Medical Colleges Learn Serve Lead: Proceedings of the 56th Annual Research in Medical Education Sessions):S67–S74.
26. Bonasso P, Lucke-Wold B, 3rd, Reed Z, et al. Investigating the impact of preparation strategies on USMLE Step 1 Performance. MedEdPublis 2015;4(1).
27. Kumar AD, Shah MK, Maley JH, et al. Preparing to take the USMLE Step 1: a survey on medical students’ self-reported study habits. Postgrad Med J 2015;91(1075):257–261.
28. Kim RH, Kurtzman SH, Collier AN, et al. The learning preferences of applicants who interview for general surgery residency: A multiinstitutional study. J Surg Educ 2016;73(6): e136–e141.
29. Lynch TG, Woelfl NN, Steele DJ, et al. Learning style influences student examination performance. Am J Sur. 1998; 176(1):62–66.

Article citation: Wu W, Garcia K, Chandrahas S, Siddiqui A, Baronia R, Ibrahim Y. Predictors of Performance on USMLE Step 1. The Southwest Respiratory and Critical Care Chronicles 2021;9(39):63–72
From: The School of Medicine (WW, KG, SC, AS), Department of Psychiatry (RB, YI), Texas Tech University Health Sciences Center, Lubbock, Texas
Submitted: 2/1/2021
Accepted: 4/3/2021
Reviewer: Kenneth Nugent MD
Conflicts of interest: none
This work is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License.

* Wu, Garcia, Chandrahas, Siddiqui are co-first authors.