Student-Faculty Co-Production of a Medical Education Design Challenge as a Tool for Teaching Health System Science

Posted 2021/04/08


AMA Accelerating Change in Education Innovation Grant program



What problem was addressed:

Medical schools prepare students to enter a complex health system with the knowledge to care for patients but provide little training on the health system they join. Health systems science (HSS) is an important topic that is starting to enter medical school curricula. The difficulty is how to teach this complex topic which is slow to gain traction with key stakeholders.1 We argue that HSS is not as difficult a concept to implement if presented in a familiar context that encourages active participation with the material. We present our educational innovation to teach HSS in an active learning setting that increased buy-in from medical students and faculty. 

What was tried:

We organized the Medical Education Design and Innovation Challenge (MEDIC), a competition that taught medical students HSS as they competed to design an educational innovation. We introduced 24 medical students from all years of training to HSS using the Shingo ModelTM 2 as a framework, the model successfully used by the University of Utah Health system. Students were divided into 6 teams and asked to identify an area for improvement and then design a program, course, or initiative utilizing this model. The Shingo ModelTM requires users to identify guiding principles, key stakeholders, and important outcomes as precursory steps to any innovative problem-solving design. This encouraged students to understand education system before proposing a solution to a perceived deficit. This event was divided over two days with a total of 8 hours of participation. Students were introduced to HSS and the Shingo ModelTM during an introductory dinner and then placed into teams. The following day teams had 4 hours to identify a deficit within their school system and design a solution (ex. Mentorship program for specialty exploration) using the Shingo ModelTM as a framework. Teams then pitched their proposals and were evaluated on their creativity, feasibility, and evidence of utilizing systems science in their design. Winners were determined based on majority vote from guest faculty judges, coordinators and participants.

What lessons were learned:

Two major challenges exist in teaching HSS to medical students: relevance to learners and incorporation into already full medical school curricula. Survey data from MEDIC suggest this project-focused approach to teaching HSS addressed both challenges. Participants (63% response rate) revealed that 100% of students felt that MEDIC was relevant to them. 93% of students thought the Shingo ModelTM was an appropriate framework for approaching medical education innovation and 73% were confident in their ability to apply the model after only four hours of team-based work. 80% of students found they developed new skills and had a change of perception of medical education design by participating in MEDIC. Additionally, 80% of students agreed or strongly agreed that all students would benefit from exposure to HSS in the core curriculum. This experience may be easily reproduced at other institutions. The positive response of the students and success in proposing innovative ideas for medical education encouraged us to continue to use this framework as we engage students and faculty in ongoing curricular reform.


  1. Gonzalo JD, Hawkins R, Lawson L, Wolpaw D, Chang A. Concerns and Responses for Integrating Health Systems Science Into Medical Education. Acad Med. 2018; 93(6):843–849

Personal, Social, Organizational, and Space Components of the Clinical Learning Environment: Variations in their Perceived Influence

Posted 2021/04/08


Purpose. Because of its impact on learning, interest in the learning environment continues unabated. This includes a framework which emerged from a Macy-sponsored conference which organizes factors that influence the quality of the learning environment into four components: personal, social, organizational, and space. This paper reports a study which assessed the relative influence of these components.

Methods. This study involved the secondary analysis of a subset of transcribed excerpts obtained from a study using an appreciative-inquiry style of questions in interviews and focus groups with faculty, residents and students from two departments at the University of Utah School of Medicine conducted in 2019. After all excerpts had been coded using a constant comparative method, those assigned the codes of “successes” or “challenges” were considered for the secondary analysis. For each selected excerpt, trained research assistants divided up 100 points among the four components according to their perceived relative influence on the ideas expressed in the excerpt. Differences in the average number of points assigned across components by type of excerpt (success versus challenge) and by type of speaker (faculty, resident, student) were examined.

Results. Overall, the social component received the highest average number of points, followed closely by personal, and then much less so by organizational, and space. In both successes and challenges, the four components followed this same rank order, Nevertheless, the average number of points assigned to organization was significantly greater for challenges than for successes. There were no statistically significant differences in points assigned based on speaker.

Conclusion. Our secondary analysis of excerpts from interview and focus group transcripts confirm the relatively stronger influence of the social and personal components in the learning environment than the organizational and space. Nevertheless, a consideration of the organizational component appears warranted when seeking to overcome challenges which impede learning in the clinical environment.


The clinical learning environment is receiving substantial and increasing attention in medical schools as stakeholders strive to optimize learning outcomes while overcoming longstanding challenges such as mistreatment and marginalization of learners.1-4

Given this level of interest, Gruppen and colleagues recently held a Macy-sponsored consensus conference on the learning environment from which was published a conceptual framework,5 hereafter in this article referred to as the Macy-Conference Framework. This framework integrates an extensive literature, including a recent review from Schonrock-Adema,6 with the intent to “facilitate health professions educators in understanding, studying, and designing interventions to improve the learning environment.”6

According to Gruppen and colleagues, the learning environment is “a complex psycho-social-physical construct that is co-created by individuals, social groups, and organizations in a particular setting.”5 The Macy-Conference Framework captures this complexity. It includes “five overlapping and interactive core components that form two dimensions: the psychosocial dimension and material dimension.”5 The psychosocial dimension comprises three components: the personal, social, and organizational. The material dimension encompasses physical and virtual spaces.5 The personal includes characteristics of participants in the learning environment which intrinsically shape behavior including knowledge, attitudes, perceptions, level of commitment, and a priori goals. The social focuses on the dynamics between individuals including interactions and social relations related to all aspects of the learning environment. The organizational includes norms, roles, and structures, which extrinsically shape behaviors (both individually or as a group). Space includes the physical and virtual characteristics of the environment in which learning and practice occur.

We encountered the Macy-Conference Framework after we had initiated an inquiry in 2019 into the learning environment at our institution, the University of Utah School of Medicine, with the intent of shifting focus from what’s wrong (e.g., reoccurring reports of mistreatment) to what’s working (e.g.,  stand out teaching moments). Our goal, consistent with the philosophy of appreciative inquiry,7-9 was to optimize the learning environment by first discovering and documenting existing examples which we could showcase and replicate. While we also explored challenges and concerns, these were not our primary focus, per se. Our inquiry included conducting, transcribing, and coding appreciative-style interviews8,10 and focus groups with faculty, residents, and students. We had reached the point in this inquiry of completing a first round of qualitative, inductive analyses, resulting in a robust set of coded excerpts, of which 189 had been coded as either “successes” or “challenges” in promoting learning in the clinical learning environment. 

As we began to consider how best to take advantage of the Macy-Conference Framework5 to analyze our data, we discovered that it wasn’t a matter of just using the domains or components as codes and assigning them to excerpts because all of the quotes simultaneously referred to the personal, social, organizational, and/or material components of the Macy-Conference Framework–explicitly or implicitly–with varying degrees of emphasis. This realization gave us the inspiration to initiate a secondary analysis of our data with the aim of using the Macy-Conference Framework as an interpretive lens to deepen our understanding of the conditions influencing the learning environment and ways to enhance that environment.

We guided our secondary analysis of these coded excerpts using the following questions:

  1. What is the relative influence of each component in the Macy-Conference Framework in how stakeholders talk about the learning environment?
  2. Are there differences in the relative influence of each component across excerpts coded as successes versus challenges?
  3. Are there differences in the relative influence of each component across excerpts made by faculty, residents or students?


After receiving IRB approval from the Institutional Review Board of the University of Utah, we collected data for the primary study upon which our secondary analysis was based through interviews and focus groups. We conducted individual interviews with Faculty and Residents in the Department of Surgery (n=7) and the Department of Obstetrics and Gynecology (n=6). We also conducted interviews with fourth-year medical students (n=4) and two focus groups with third-year medical students (n=20). Interview and focus group questions were designed utilizing an appreciative inquiry approach.7-9 Our primary interview/focus group questions (see Appendix) specifically asked participants to, 1) describe a successful learning moment and what they contributed to that moment, 2) identify their core values and how well these were reflected by our institution, and 3) recall an instance in which their values had been challenged. Additional, follow-up questions explored perceptions of mistreatment and barriers to an optimal learning environment.  We transcribed the interviews and focus group recordings verbatim. We used a constant comparative method11 to code the transcripts into categories that included, among other things, successful and challenging learning moments. Multiple members of the team participated in coding or reviewing codes and resolved any discrepancies through discussion.

It was at this point that we decided to initiate a secondary analysis of our qualitative data with the aim of using the Macy-Conference Framework5 as an interpretive lens. In response to time constraints, we randomly selected 122 excerpts for inclusion in our secondary analysis, or approximately 2/3 of excerpts coded as successes or challenges. The length of these selected excerpts ranged from 71 to 2783 words (average=790).

Our secondary analysis borrowed an approach used in previous research,12,13 in which study participants assign 100 points amongst a set of elements to indicate the relative influence of each element on the topic of interest. For example, Balmer and colleagues12 asked 4th-year students to divide 100 points among the explicit, implicit, and extra curriculum for each of 10 school-wide learning objectives to show the relative influence on each type of curricula in the student’s acquisition of the knowledge and skills required to achieve each objective. Differences in how the participants assigned points led to important insights about the positive influences of the implicit curriculum on student learning, particularly of learning objectives associated communication, teamwork, and professionalism.

Similar to Balmer and her colleagues,12,13 we developed, piloted, and refined a method to assign weights to the components of the Macy-Conference Framework to capture raters’ perceptions of the relative importance of each component in the content of each selected excerpt. Based on our pilot work in which raters tended to assign very few points to either of the two components of the material domain (i.e., physical and virtual),5 we combined them into one, which we called ‘space’. Two research assistants (CB and TD) first discussed definitions of each component, practiced using those definitions in assigning weights, and then shared their experience to optimize calibration. The raters then assigned weights to a subset of assigned excerpts, blinded as to whether the excerpt had been coded as a success or challenge and to whether the speaker was a faculty, resident, or student. In assigning weights, the raters carefully read each assigned excerpt and considered the key content being expressed. They then inferred the relative influence of each of the four components in each excerpt by dividing up 100 points among the components.  Both researchers assigned weights to 2/3 of the selected subset of excerpts so that 1/3 overlapped. To optimize congruence in their approach to assigning weights, the researchers reviewed and discussed each other’s weight assignments for the 1/3 overlapping excerpts before completing the task for all assigned excerpts. Table 1 contains example excerpts with point assignments.

Table 1. Example excerpts and relative weight assignment.

To address research question 1, BR averaged the weights assigned to the four components for the 1/3 overlapping excerpts and then computed overall averages and standard deviations of weights for each of the four domains across excerpts. For questions 2 and 3, BR computed separate averages and standard deviations for excerpts coded as success and challenges or for each stakeholder group.

We received help from the institution’s Center for Clinical and Translational Science Biostatistics Core14 to analyze our data and determine the significance of observed differences. We first summarized results descriptively, where subject (faculty, resident, student) and type (challenge vs. success) were summarized as frequency and percentage, and weight within each of the four components (personal, social, organizational and space) was summarized using mean and standard deviation (SD), median (25th and 75th quartiles), and range. Given that the weights across the four components summed to 100, we used analysis methods appropriate for compositional data.  We used a Friedman’s test to assess whether weight distributions differed across the 4 components. We used Dirichlet regression to assess whether weight distributions differed by type of excerpt (success versus challenge) and type of stakeholder (faculty, resident, or student).15 We reported Odds ratios (ORs) of weighting each component more than the Social component (reference) with 95% confidence intervals (CIs) and p-values. We assessed statistical significance at the p<0.05 level two sided tests. We conducted all analyses using Rv.3.6.16


Overall, we observed statistically significant differences in the relative weights assigned to personal, social, organization, and space (p<0.001, Friedman’s test). The social component received the highest average weights, followed closely by personal, and then much less so by organizational and space. See Table 2 for frequencies and percentages of stakeholders (faculty, resident, student) and types of excerpt (success versus challenge). See Table 3 for the mean, median, and standard deviation for weights assigned to each component.

Table 2. Number of excerpts coded by stakeholder and excerpt type
Table 3. Summary of weight distribution per component.

The average weight assignments of the four components follows the same rank order for Successes as well as Challenges (see Figures 1 and 2); however, the odds of an excerpt being weighted as an Organizational component more than the Social component is 40% lower for successes than for challenges OR=0.60 (95% CI 0.40, 0.89, p=0.012).

Average weights assigned by stakeholder to the four components again, fall in the same order of social, personal, organizational, and space. The weight distributions of the four components are not significantly different among stake holders (all p-values are greater than 0.05).

Figure 1. Average weight for each component stratified by Challenge vs Success with 95% confidence intervals.
Figure 1. Average weight for each component stratified by Challenge vs Success with 95% confidence intervals.
Figure 2. Odds ratios with 95% CIs for comparing each component to the social domain, by success vs. challenge.
Figure 2. Odds ratios with 95% CIs for comparing each component to the social domain, by success vs. challenge.


There is extensive literature that attempts to understand what it takes to optimize the learning environment (where successes occur and challenges are minimized), both generally2,4,5,17-22 and in Surgery23,24 and OB-GYN, specifically.25,26 The results of our study add to this literature, by using the lens of the Macy-Conference Framework to affirm the greater emphasis of the psychosocial domain compared to the material domain.

Within the psychosocial domain, our data highlights the relative greater influence of the personal and social factors on both success and challenges. This finding is intuitive and is consistent with the literature. Teaching and learning inherently emphasize interactions between individuals. As many authors have suggested, these interactions are shaped–whether good or bad–by the idiosyncratic characteristics of the participants (personal component) and by the dynamics of the interpersonal relationship (social component).2,23,27 Stakeholders in our study consistently referenced personal characteristics of faculty and learners. As illustrated in the sample excerpts included in Table 1, personal characteristics associated with successes and/or challenges include listening skills, a willingness to say ‘I don’t know’, a willingness to focus on teaching during surgery, an awareness of the learning climate, choosing to take the time to teach, and feeling safe enough to speak up.

The excerpts we included in our secondary analysis, as illustrated in Table 1, consistently highlight the interpersonal conditions central to promoting effective learning and responding to challenges. Examples include: involving others in teaching interactions, teaching styles, dynamics created with multiple levels of learners, communication patterns, giving and receiving feedback, and including or excluding learners. As a result, these data affirm the leading role of the social component in promoting success or in overcoming challenges in the learning environment, such as setting expectations, slowing down to explain whenever possible, providing timely feedback, promoting open communication, incrementally increasing learner autonomy, or protecting even a few minutes daily for teaching.1,25,28

Our data highlight the influence of organizational elements on the quality of the learning environment. While less than personal and social, the organizational elements deserve strong consideration in efforts to understand and shape the learning environment–particularly in terms of responding to challenges.2,18,22,25 As seen in the excerpts in Table 1, organizational elements that promote success, include working within the hierarchy of a team and establishing a culture which prioritizes interactions with learners. On the other hand, examples related to the organizational component that appear to create challenges to be overcome include patient care services that are perceived as devoid of any learning opportunities and the level of stress triggered by the broader environmental conditions.2,18,29

Our data suggest that the material domain of the Macy-Conference Framework has less influence in shaping the learning environment than the psychosocial domain. We consider this an interesting finding because a major addition of the Macy-Conference Framework, compared to the model proposed previously by Schonrock-Adema et. al.6 is the addition of the material domain. Reference to the material domain seldom occurred and thus raters consistently gave it little to no weight. As captured example 3 in Table 1, exceptions did occur, in which the speaker referenced an association between space (OR, clinical, floor) and learning.

Because the results of this study suggest that organizational factors may have greater influence for challenges than for successes, regardless of stakeholder group, we suggest leaders may want to look at interventions to improve the learning environment differently depending upon whether their intent is to promote successes versus overcome challenges. In particular, in order to promote more successes, leaders may want to first look for ways to influence social or personal aspects in the psychosocial domain using such interventions as faculty development30-34 and enhancing longitudinal relationships.35,36 On the other hand, to minimize challenges, leaders may want to first look for ways to modify organizational aspects of the learning environment, such as schedules, incentives, or policies.4,37 That said, because the Macy-Conference Framework assumes that the domains/components are continually overlapping and interacting, the development and application of interventions must be addressed at multiple levels.


This secondary analysis of existing qualitative data took place in a single institution and in two procedurally-oriented departments. The material domain was not well represented, possibly because space tends to influence our behavior without us consciously knowing.38 This phenomenon could also be a product of our explicit focus in Surgery and OBGYN, whereby the operating theater is the dominant clinical learning environment and the space where learning occurs in these disciplines is a given and is beyond the control of educators. Perhaps we would find greater variation in the space component if we were to expand to other more medically oriented specialties. While our study was able to affirm the relative importance of the various components of the Macy-Conference Framework, it was not designed to identify what may be missing from the Framework. Our primary study, is much better equipped to meet this challenge.


Clinical learning environments are composed of complex interactions between people, organizational structures, and physical factors which work dynamically to promote or impede learning. Such environments are in constant need of shaping and reshaping to best meet the needs of stakeholders–be they students, residents, or faculty. Using secondary analysis of transcribed excerpts from appreciative inquiry style interviews and focus groups, this study supports the value of the Macy-Conference Framework as a lens for better understanding the interacting components of the learning environment with an eye to continual improvement. Indeed, components are not all equally influential and their relative influence tends to change when one is focused on promoting successes versus minimizing challenges. In the former, the focus is most likely to be on social and personal components, with a secondary focus on organization and space. On the other hand, in the latter, the focus is more likely to also include the organizational component.

By using the Macy-Conference Framework as a lens for our secondary analysis, we provide evidence related to the validity of the framework and its potential utility in shaping the learning environment. Our data reinforces the presence of a common set of core components of the learning environment and their overall relative importance, particularly in the psychosocial domain, and should inform future efforts to optimize the clinical learning environment.


The research reported in this publication was supported in part by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002538. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Funding/Support. None

Disclosures. None

Ethical approval. University of Utah Internal Review Board

Disclaimers. None.

 Previous presentations.


1.         Brandford E, Hasty B, Bruce JS, et al. Underlying mechanisms of mistreatment in the surgical learning environment: A thematic analysis of medical student perceptions. American journal of surgery. 2018;215(2):227-232.

2.         Kilty C, Wiese A, Bergin C, et al. A national stakeholder consensus study of challenges and priorities for clinical learning environments in postgraduate medical education. BMC Med Educ. 2017;17(1):226.

3.         Mazer LM, Bereknyei Merrell S, Hasty BN, Stave C, Lau JN. Assessment of programs aimed to decrease or prevent mistreatment of medical trainees. JAMA Network Open. 2018;1(3):e180870.

4.         van der Goot WE, Cristancho SM, de Carvalho Filho MA, Jaarsma ADC, Helmich E. Trainee-environment interactions that stimulate motivation: A rich pictures study. Medical Education. 2019;54(3):242-253.

5.         Gruppen LD, Irby DM, Durning SJ, Maggio LA. Conceptualizing Learning Environments in the Health Professions. Academic medicine : journal of the Association of American Medical Colleges. 2019;94(7):969-974.

6.         Schonrock-Adema J, Bouwkamp-Timmer T, van Hell EA, Cohen-Schotanus J. Key elements in assessing the educational environment: where is the theory? Adv Health Sci Educ Theory Pract. 2012;17(5):727-742.

7.         Rama JA, Falco C, Balmer DF. Using Appreciative Inquiry to Inform Program Evaluation in Graduate Medical Education. J Grad Med Educ. 2018;10(5):587-590.

8.         Sandars J, Murdoch-Eaton D. Appreciative inquiry in medical education. Med Teach. 2017;39(2):123-127.

9.         Williams A, Haizlip JA. Ten Keys to the Successful Use of Appreciative Inquiry in Academic Healthcare. OD Practitioner. 2013;45(2).

10.       Bushe GR. Appreciative Inquiry is Not (Just) About the Positive

OD Practitioner. 2007;39(4):30-35.

11.       Glaser B SA. The discovery of grounded theory: strategies for qualitative research. . New York, NY: Aldine de Gruyter; 1967.

12.       Balmer DF, Hall E, Fink M, Richards BF. How do medical students navigate the interplay of explicit curricula, implicit curricula, and extracurricula to learn curricular objectives? Academic medicine : journal of the Association of American Medical Colleges. 2013;88(8):1135-1141.

13.       Balmer DF, Quiah S, DiPace J, Paik S, Ward MA, Richards BF. Learning across the explicit, implicit, and extra-curricula: an exploratory study of the relative proportions of residents’ perceived learning in clinical areas at three pediatric residency programs. Academic medicine : journal of the Association of American Medical Colleges. 2015;90(11):1547-1552.

14.       CCTS Population Health Research.

15.       Mazotti L, Adams J, Peyser B, Chretien K, Duffy B, Hirsh DA. Diffusion of innovation and longitudinal integrated clerkships: Results of the clerkship directors in internal medicine annual survey. Medical Teacher. 2019;41(3):347-353.

16.       Maier MJ. DirichletReg: Dirichlet Regression in R. R package version 0.7-0. 2020.

17.       Fried JM, Vermillion M, Parker NH, Uijtdehaage S. Eradicating medical student mistreatment: a longitudinal study of one institution’s efforts. Academic medicine : journal of the Association of American Medical Colleges. 2012;87(9):1191-1198.

18.       Gan R, Snell L. When the learning environment is suboptimal: exploring medical students’ perceptions of “mistreatment”. Academic medicine : journal of the Association of American Medical Colleges. 2014;89(4):608-617.

19.       House JB, Griffith MC, Kappy MD, Holman E, Santen SA. Tracking Student Mistreatment Data to Improve the Emergency Medicine Clerkship Learning Environment. The western journal of emergency medicine. 2018;19(1):18-22.

20.       Kulaylat AN, Qin D, Sun SX, et al. Perceptions of mistreatment among trainees vary at different stages of clinical training. BMC Med Educ. 2017;17(1):14.

21.       Olasoji HO. Broadening conceptions of medical student mistreatment during clinical teaching: message from a study of “toxic” phenomenon during bedside teaching. Advances in medical education and practice. 2018;9:483-494.

22.       Oser TK, Haidet P, Lewis PR, Mauger DT, Gingrich DL, Leong SL. Frequency and negative impact of medical student mistreatment based on specialty choice: a longitudinal study. Academic medicine : journal of the Association of American Medical Colleges. 2014;89(5):755-761.

23.       Castillo-Angeles M, Watkins AA, Acosta D, et al. Mistreatment and the learning environment for medical students on general surgery clerkship rotations: What do key stakeholders think? American journal of surgery. 2017;213(2):307-312.

24.       Kemp MT, Smith M, Kizy S, Englesbe M, Reddy RM. Reported Mistreatment During the Surgery Clerkship Varies by Student Career Choice. J Surg Educ. 2018;75(4):918-923.

25.       Lau JN, Mazer LM, Liebert CA, Bereknyei Merrell S, Lin DT, Harris I. A Mixed-Methods Analysis of a Novel Mistreatment Program for the Surgery Core Clerkship. Academic medicine : journal of the Association of American Medical Colleges. 2017;92(7):1028-1034.

26.       Baecher-Lind LE, Chang K, Blanco MA. The learning environment in the obstetrics and gynecology clerkship: an exploratory study of students’ perceptions before and after the clerkship. Med Educ Online. 2015;20:27273.

27.       Singh TSS, Singh A. Abusive culture in medical education: Mentors must mend their ways. Journal of anaesthesiology, clinical pharmacology. 2018;34(2):145-147.

28.       Furney SL, Orsini AN, Orsetti KE, Stern DT, Gruppen LD, Irby DM. Teaching the one-minute preceptor. A randomized controlled trial. J Gen Intern Med. 2001;16(9):620-624.

29.       Dyrbye LN, Thomas MR, Shanafelt TD. Medical student distress: causes, consequences, and proposed solutions. Mayo Clinic proceedings. 2005;80(12):1613-1622.

30.       Birman BF, Desimone L, porter AC, Garet MS. Designing professional development that works. Educational Leadership. 2000.

31.       Desimone LM, Porter AC, Garet MS, Yoon KS, Birman BF. Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study. Educational Evalution and Policy Analysis. 2002;24(2):81-112.

32.       Gast I, Schildkamp K, Veen JTvd. Team-Based Professional Development Interventions in Higher Education: A Systematic Review. Review of Educational Research. 2017;87(4):736-767.

33.       McLean M, Cilliers F, Van Wyk JM. Faculty development: Yesterday, today and tomorrow. Medical Teacher. 2008;30(6):555-584.

34.       Sklar DP. Moving From Faculty Development to Faculty Identity, Growth, and Empowerment. Academic Medicine. 2016;91(12):1585-1587.

35.       Hirsh D, Walters L, Poncelet AN. Better learning, better doctors, better delivery system: Possibilities from a case study of longitudinal integrated clerkships. Medical Teacher. 2012;34(7):548-554.

36.       Hirsh DA, Holmboe ES, ten Cate O. Time to Trust: Longitudinal Integrated Clerkships and Entrustable Professional Activities. Academic Medicine. 2014;89(2):201-204.

37.       Vanstone M, Grierson L. Social power facilitates and constrains motivation in the clinical learning environment. Med Educ. 2020;54(3):181-183. 38.       Harrouk C. Psychology of Space: How Interiors Impact our Behavior? 2020; Accessed 9/17/20, 2020.

Interview Questions

**This is a semi-structured interview protocol, adapted from a guide originally developed by Drs. Williamson and Suchman.1 Probes will be used as necessary to elicit additional pertinent information.

Introduction: This is going to be what we call an appreciative interview. I am going to ask you questions about times when you experienced educational things working at their best here at [institution]. Many times, we try to ask questions about things that aren’t working well—the problems—so that we can fix them. In this case, we are trying to find out about the things at their best—the successes—so that we can find out what works and why, and find ways to infuse more of it into our practice.
• As we get started, I’d like to know a little bit about you. Just so you know, this information will not be associated with any of your stories or quotes, but will just be used to provide context to our findings.
o What’s your role here at [institution] and how long have you been here?
• People do their best work when they are doing things that they find personally meaningful, and when they feel that their work makes a difference. During your time at [institution], there have no doubt been high points and low points. For now, I’d invite you to think of a teaching and learning moment that meant a lot to you, when things went right, a time that brought out the best in you.
o Please tell the story of that time. (If they are very general, try to probe for more specificity.)
o Without worrying about being modest, please tell me what it was about you—your unique qualities, gifts or capacities; decisions you made; or actions you took—that contributed to this teaching/learning experience?
o What did others contribute or do?
o What aspects of the situation made this a success (for example, the place, the time of day or year, recent events)?
• Now, think of a time at [institution] when you or your values were challenged.
o Please tell me a story about that time. (If participant needs clarification about what a value is, explain that a value is “a person’s principles or standards of behavior; one’s judgment of what is important in life.”)
• We each have different qualities, gifts and skills we bring to the world and to our work. Think about the things you value about yourself, the nature of your work and the university. At work, we’re always dealing with challenges and change.
• How have your strengths and values helped you deal with challenges and change?
o Your work: When you are feeling good about your work, what do you like about the work itself?
o Yourself: Imagine you’re at your retirement party. What do you think your colleagues would say they liked most about you?
o Yourself: Now what do you think your students would say they’ve liked most about you?
o How do your personal values match those of [institution]? (for example, honesty, compassion, teamwork)?
o Where have you seen examples of these values at [institution]?
• Where do you think these reports of mistreatment are coming from?

Improving the Promotions Dossier with the Enhanced CV

Posted 2020/10/06


In contrast to trends calling for the use of an educator portfolio to present evidence of accomplishment in the promotion dossier for faculty with a career focus on education or patient care, we propose use of an Enhanced Curriculum Vitae (CV). Rather than dedicate the time to create a portfolio that contains redundant information with the CV and is often ignored or treated as ‘second class’, we believe that, with a few simple additions, faculty can include nearly all of the content in the CV which they might have presented in a portfolio. We propose two types of additions. One is adding categories to the CV to be more inclusive of educational or clinical contributions (e.g., teaching, mentoring, and course leadership) not often included in the research-centric CV. The other is adding terse annotations to selected items listed in the CV to clarify quantity, quality, and scope of specific accomplishments (e.g., selection process of honors and awards, learner evaluation highlights).

We argue that, in lieu of peer-reviewed publication, educators be allowed to include other types of work products with the CV in the promotions dossier, such as a course syllabus, a representative instructional product, or clinical care guidelines.


Because academic medicine has become more accepting of educational and clinical accomplishments in support of rank advancement, in this Perspectives article, we argue that greater thought needs to be given to how information regarding such accomplishments can best be presented to participants in the promotions process (e.g. individuals asked to write letters of reference, members of promotions committees).  The goal, as always, is to ensure that participants in the promotions process pay attention to the evidence and then give it careful and fair consideration in light of institutional priorities and values

We make this argument–specifically as it applies to educational accomplishments—based on many years of experience working with a wide variety of clinician educators seeking rank advancement at institutions supportive of promotion based on educational accomplishments. Quite simply, we have learned that attempts to present a faculty member’s educational accomplishments in an educator portfolio tend to be less successful than presenting the same evidence using an annotated or Enhanced Curriculum Vitae (CV). For this to work, modest enhancements to the traditional CV are necessary. We also argue that as institutions become more accepting of accomplishments in education and clinical care as evidence of readiness for advancement, promotions dossiers need to allow educators to submit products such as syllabi or instructional materials in lieu of standard peer-reviewed publications.

The challenge to present diverse accomplishments fairly

In academic medical communities, faculty participate in clinical, educational, and investigational activities, with most faculty members participating in more than one of these areas and often in all three. In recent years, many institutions have broadened their promotions processes, as espoused by Boyer (1990) to include scholarly accomplishments in all these areas as legitimate domains for consideration in rank advancement (Simpson et al., 2007).  A manifestation of this trend is inclusion of different career tracks, most often including research, education, and clinical care.

The challenge, created by this trend, is the need for fair representation and consideration of academic accomplishments in promotions dossier across all of these areas. While Glassick’s (2000) criteria promote a similar standard to evaluate scholarship from any area, what has been challenging is finding the best way to present the unique types of evidence required for careful consideration and awarding of credit for the scholarship achieved in education and clinical care (Baldwin, Gusic and Chandran, 2010; Simpson et al., 2007).

Our experience suggests that the traditional CV does not capture this diversity and that a more inclusive format is needed. Generally, the standard promotions dossier consists of the traditional CV, copies of a limited number of the publications listed in the CV, and a short personal statement. The traditional CV has evolved to best capture the accomplishments of investigators, measuring success by peer reviewed grants and publications. Scientific peer review (i.e., NIH study sections, journal editorial boards), in effect, allows the promotions decision-makers to defer to the judgments of other scientists or experts, so little additional information about the items listed is needed. As a result, the listing of these accomplishments in the traditional CV tend to conform to very specific formats, designed to communicate needed information for each type of accomplishment most efficiently. For example, the listing of grants typically includes the funding agency, amount awarded, and the individual’s role in the funded project, while the listing of publications is limited to standard biographic information (authors, title, journal, date). These formats have become very familiar to individuals involved in the promotions process. Thus, the traditional CV and by extension the promotions process tends not to fully capture the accomplishments of educators and clinicians, whose activities are not peer reviewed in the same manner as researchers.

After working with numerous faculty educators who have been nominated for promotions, we have learned that simple lists of products produced is not sufficient to represent the wide diversity of their accomplishments. Most often, more explanation about the what, why, who, and how of the accomplishment, including the type of peer review, is usually needed. Furthermore, these accomplishments frequently do not typically result in the kinds of traditional peer-reviewed publications that are typically attached to a promotions dossier.

The educator portfolio – pros and cons

For well over 20 years, the use of an educator portfolio has been recommended as a companion to the traditional CV in the promotions dossier for faculty with substantive involvement in education (Simpson et al., 2007; Baldwin, Gusic and Chandral, 2010; Niebuhr et al., 2013; Shinkai et al., 2018). An educator portfolio is designed to detail educational accomplishments of faculty. It often includes a personal statement of philosophy and intent, and may include many of the same educational accomplishments listed in a traditional CV, but with greater detail. Furthermore, the portfolio often includes additional accomplishments because they do not fit into the categories of the traditional CV.

While portfolios privilege the attitudes and activities of educators, we have come to question whether educator portfolios offer the optimal way to present the accomplishments of non-research oriented activities for the purposes of promotion. One reason portfolios are suboptimal is that many individuals involved in promotions considerations are not familiar with them and may largely ignore them. In addition, generally only “educator track” faculty assemble portfolios, thus the educational contributions of non-educator track faculty, including clinicians, may remain underrepresented. The length and narrative quality of the educator portfolio may be alienating to non-educator faculty thus limiting their utility. Finally, having a supplement that only educators submit may paradoxically serve to further marginalize educators from the general community of medical faculty.

The Enhanced CV

At Columbia University Vagelos College of Physicians and Surgeons, these concerns led a group of educators from multiple clinical departments (led by BFR and DLC) to form a workgroup to think about the best way to: 1) organize educational achievements and products; 2) share them with promotions committees. We considered many models, and the model that was proposed and ultimately adopted was what we came to call the “Enhanced CV.” This is a standard CV that allows for more expansion than is traditionally used, particularly for educational accomplishments and materials; although this expansion can be adopted for other areas as well, such as clinical work. It brings the educational materials back into the CV and out of a separate educator portfolio, asking all faculty to highlight their educational work (not just those who identify as “educators”) and allows those on educator tracks to have room and flexibility to highlight their unique accomplishments. After similar discussions, the University of Utah School of Medicine has used a similar, yet less formal and less widely adopted Enhanced CV.

As faculty have adopted the Enhanced CV, we have learned first-hand about its effectiveness in capturing the variety and detail of a faculty member’s accomplishments. Including terse, bulleted annotations of selected accomplishments allows promotion’s decision makers to judge the magnitude of effort, degree of quality, and impact of accomplishments, including those where no external, standardized process of peer review was possible. Table 1 contains some representative examples from the Enhanced CV of RJG, which was included in a successful promotions dossier. The examples illustrate the type of annotation used as well as the types of content not typically included in the traditional CV.

Table 1
Table 1: Examples of Annotation from Enhanced CV of RG

The idea of annotation is not new per se, as they have been used in the traditional CV for grants (e.g. details about funding agency, grant amount, etc.) because the title of the grant alone fails to communicate sufficient information to appropriately weigh the impact and magnitude of each grant listed. The Enhanced CV can just as easily include annotations about other types of accomplishments which reviewers can read to fully understand and appreciate the impact of those accomplishments, such as quality improvement projects, public health guidelines, and educational innovations. For example, faculty at the University of Utah have found annotations particularly helpful in clarifying the different levels of selectivity in the peer review and acceptance processes of posters, oral presentations, and workshops at professional meetings.

The Enhanced CV offers individuals involved in the promotions process a common, systematically organized format for all faculty that flexibly and fairly presents their diverse scholarly accomplishments. As a result, the Enhanced CV is particularly useful as academic institutions move towards using multiple tracks rank advancement, each with unique forms of scholarly contributions.

Substitution of unique educational work products

Most promotions dossiers we have seen call for the faculty member to select a handful of publications (e.g., 5 at Columbia) from those listed in the CV to include in their entirety with the CV. Of course, such publications are representative work products that highlight the faculty member’s accomplishments. We believe that dossiers for educators need to allow substitutions of publications for more appropriate peer-reviewed work products generated by educational activities, such as a syllabus or chapter of an instructional text. We also believe that, similar to annotations in the CV, providing limited annotations within these documents helps provide important background information to enhance reviewer’s understanding and ability to judge the quality and merit of the work represented (providing its own form of peer review).

Experience with a promotions dossier with the Enhanced CV

In 2010, the Columbia University Medical Center launched a three-track structure for academic advancement: 1) investigator, 2) educational leadership and scholarship, and 3) applied health sciences. An important aspect of this launch included use of the Enhanced CV for all three tracks in the promotions dossier. It also included broadening the types of work products that could be included in the promotions dossier. No educational portfolio was encouraged, nor considered necessary.

As a result, the recommended promotions dossier for all faculty, regardless of track consists of 1) 1-2 page personal statement that clarifies and highlights the faculty member’s “story” as presented in the Enhanced CV, 2) The Enhanced CV and 3) Up to five appropriate work-products (not limited to publications). These documents are sent to individuals requested to write letters of reference and subsequently to members of the promotions committees. The University of Utah follows a similar process, without the five work-products. The personal statement is meant to capture the “impact” of the faculty’s work.

The Enhanced CV is designed to help promotions committee members understand the scope of the educational contribution, which is often central to promotions decisions. For example, committee members on other tracks may understand the number of hours that go into writing a grant, but might not appreciate the hours that go into conceptualizing, planning and executing a medical school course. In addition, educational work products such as websites, videos, and syllabi have been of great interest to promotions committee members and have helped bring the educator’s work to life. Given the confidential nature of promotion committee deliberations, it is difficult to cite specific promotions decisions where the improved dossier was critical to the outcome, nevertheless, feedback from committee members at both Columbia and Utah, who have seen the Enhanced CV, suggest that dossiers with the Enhanced CV and relevant work products have helped to encourage more careful and in depth consideration of the accomplishments represented–regardless of whether they were associated with research, education, or clinical practice—and that the evidence presented is appropriate to each track and considered equally valid.

Our experience with helping educators prepare for the promotions process also suggests that the Enhanced CV helps educators capture and understand the extent of the work they do and have done. Too often, educators do not capture educational activities, such as designing curricula, advising and mentorship, in their CVs, including only finished peer-reviewed products. Using the enhance CV not only helps educators to include these critical, often incredibly time-consuming activities, it helps them to better understand their roles as educators.

As would be expected, we estimate that Enhanced CVs are 20 to 50% longer than traditional CVs depending upon the additional content and annotations. While this may add to the time required to review the CV, it is still likely much less time than would have been required to review both a CV and a portfolio. Furthermore, because the Enhanced CV continues to emphasize the list-like format of the traditional CV, reviewers can easily scan and count the various types of listed content and thereby get a holistic view of the ‘story’ represented and then can re-review the same lists but this time pay attention to the details in the annotation and get a sense of the meaning and impact of specific items.

In light of the concern about adding length to the CV, over time, we have refined our understanding about the types and amount of annotations in the Enhanced CV that are most helpful to readers. For example, one common mistake has been to use annotations that are too lengthy, potentially overwhelming the reader and leading them to lose sight of the story represented in the chronological listing of accomplishments. Another mistake, not unique to the Enhanced CV, has been to try to include everything, rather than to prioritize and highlight with annotation the accomplishments that best represent the breadth and depth of a person’s accomplishments. As with many things, even with the Enhanced CV, “less can be more”.


While recent trends calling for the use of an educator portfolio in the promotion dossier for faculty with a career focus on education or clinical care, have been well meaning and thoughtfully led, we have argued that educators may be better served with the use of an Enhanced CV. This argument recognizes that the CV is a familiar format to all involved, including promotions committee members as well as individuals asked to write letters of reference. With a few simple additions, most notably brief annotations, faculty can include nearly all of the content in the CV that they might have presented in a portfolio. This model is designed to help both promotions committee members understand the work of clinician educators, and helps the educators themselves capture the breadth and depth of their work, and to even help define their critical roles in medical education. Our hope is that disseminating this model will help us take the next steps in testing the effectiveness of this alternate method for capturing the work of clinician educators.


The authors wish to recognize the general contribution to the development of the Enhanced CV made by members of promotions committees at Columbia University and the University of Utah and by faculty colleagues brave enough to be early adopters of the Enhanced CV format. We particularly want to recognize the contributions of Lisa Saiman, MD MPH and Susan Rosenthal, PhD who championed and helped shape these ideas from the outset.


Funding/Support: None.
Other disclosures: None.
Ethical approval: Not applicable.
Disclaimer: None.
Previous presentation: Emerging Solution: Managing Evidence of Educational Scholarship with the Enhanced CV. 2016 AAMC Learn Lead Succeed Meeting. Seattle WA.


Enhanced CV, promotions dossier, educator portfolio

Take home message:

  • A traditional CV typically does not include important non peer-reviewed educational work products
  • A traditional CV typically does not include important non peer-reviewed educational work products
  • Using an Enhanced CV can improve the promotions process
  • An educator portfolio is lengthy and less likely to reviewed fully by the promotions committee
  • An Enhanced CV is meant to include the essential information of a traditional CV and that of a portfolio


Boyer, E. L. (1990) Scholarship reconsidered: priorities of the professoriate. Princeton, NJ: Carnegie Foundation for the Advancement of Teaching.

Glassick, C. E. (2000) ‘Boyerʼs Expanded Definitions of Scholarship, the Standards for Assessing Scholarship, and the Elusiveness of the Scholarship of Teaching’, Academic Medicine, 75(9), pp. 877–880. doi: 10.1097/00001888-200009000-00007.

Niebuhr, V., Johnson, R., Mendias, E., Rath, L., et al. (2013) ‘Educator Portfolios’, MedEdPORTAL Publications. doi: 10.15766/mep_2374-8265.9355.

Shinkai, K., Chen, C. (A., Schwartz, B. S., Loeser, H., et al. (2018) ‘Rethinking the Educator Portfolio’, Academic Medicine, 93(7), pp. 1024–1028. doi: 10.1097/acm.0000000000002005.

Simpson, D., Fincher, R.-M. E., Hafler, J. P., Irby, D. M., et al.
(2007) ‘Advancing educators and education by defining the components and evidence associated with educational scholarship’, Medical Education, 41(10), pp. 1002–1009. doi: 10.1111/j.1365-2923.2007.02844.x.