Purpose. Because of its impact on learning, interest in the learning environment continues unabated. This includes a framework which emerged from a Macy-sponsored conference which organizes factors that influence the quality of the learning environment into four components: personal, social, organizational, and space. This paper reports a study which assessed the relative influence of these components.
Methods. This study involved the secondary analysis of a subset of transcribed excerpts obtained from a study using an appreciative-inquiry style of questions in interviews and focus groups with faculty, residents and students from two departments at the University of Utah School of Medicine conducted in 2019. After all excerpts had been coded using a constant comparative method, those assigned the codes of “successes” or “challenges” were considered for the secondary analysis. For each selected excerpt, trained research assistants divided up 100 points among the four components according to their perceived relative influence on the ideas expressed in the excerpt. Differences in the average number of points assigned across components by type of excerpt (success versus challenge) and by type of speaker (faculty, resident, student) were examined.
Results. Overall, the social component received the highest average number of points, followed closely by personal, and then much less so by organizational, and space. In both successes and challenges, the four components followed this same rank order, Nevertheless, the average number of points assigned to organization was significantly greater for challenges than for successes. There were no statistically significant differences in points assigned based on speaker.
Conclusion. Our secondary analysis of excerpts from interview and focus group transcripts confirm the relatively stronger influence of the social and personal components in the learning environment than the organizational and space. Nevertheless, a consideration of the organizational component appears warranted when seeking to overcome challenges which impede learning in the clinical environment.
The clinical learning environment is receiving substantial and increasing attention in medical schools as stakeholders strive to optimize learning outcomes while overcoming longstanding challenges such as mistreatment and marginalization of learners.1-4
Given this level of interest, Gruppen and colleagues recently held a Macy-sponsored consensus conference on the learning environment from which was published a conceptual framework,5 hereafter in this article referred to as the Macy-Conference Framework. This framework integrates an extensive literature, including a recent review from Schonrock-Adema,6 with the intent to “facilitate health professions educators in understanding, studying, and designing interventions to improve the learning environment.”6
According to Gruppen and colleagues, the learning environment is “a complex psycho-social-physical construct that is co-created by individuals, social groups, and organizations in a particular setting.”5 The Macy-Conference Framework captures this complexity. It includes “five overlapping and interactive core components that form two dimensions: the psychosocial dimension and material dimension.”5 The psychosocial dimension comprises three components: the personal, social, and organizational. The material dimension encompasses physical and virtual spaces.5 The personal includes characteristics of participants in the learning environment which intrinsically shape behavior including knowledge, attitudes, perceptions, level of commitment, and a priori goals. The social focuses on the dynamics between individuals including interactions and social relations related to all aspects of the learning environment. The organizational includes norms, roles, and structures, which extrinsically shape behaviors (both individually or as a group). Space includes the physical and virtual characteristics of the environment in which learning and practice occur.
We encountered the Macy-Conference Framework after we had initiated an inquiry in 2019 into the learning environment at our institution, the University of Utah School of Medicine, with the intent of shifting focus from what’s wrong (e.g., reoccurring reports of mistreatment) to what’s working (e.g., stand out teaching moments). Our goal, consistent with the philosophy of appreciative inquiry,7-9 was to optimize the learning environment by first discovering and documenting existing examples which we could showcase and replicate. While we also explored challenges and concerns, these were not our primary focus, per se. Our inquiry included conducting, transcribing, and coding appreciative-style interviews8,10 and focus groups with faculty, residents, and students. We had reached the point in this inquiry of completing a first round of qualitative, inductive analyses, resulting in a robust set of coded excerpts, of which 189 had been coded as either “successes” or “challenges” in promoting learning in the clinical learning environment.
As we began to consider how best to take advantage of the Macy-Conference Framework5 to analyze our data, we discovered that it wasn’t a matter of just using the domains or components as codes and assigning them to excerpts because all of the quotes simultaneously referred to the personal, social, organizational, and/or material components of the Macy-Conference Framework–explicitly or implicitly–with varying degrees of emphasis. This realization gave us the inspiration to initiate a secondary analysis of our data with the aim of using the Macy-Conference Framework as an interpretive lens to deepen our understanding of the conditions influencing the learning environment and ways to enhance that environment.
We guided our secondary analysis of these coded excerpts using the following questions:
- What is the relative influence of each component in the Macy-Conference Framework in how stakeholders talk about the learning environment?
- Are there differences in the relative influence of each component across excerpts coded as successes versus challenges?
- Are there differences in the relative influence of each component across excerpts made by faculty, residents or students?
After receiving IRB approval from the Institutional Review Board of the University of Utah, we collected data for the primary study upon which our secondary analysis was based through interviews and focus groups. We conducted individual interviews with Faculty and Residents in the Department of Surgery (n=7) and the Department of Obstetrics and Gynecology (n=6). We also conducted interviews with fourth-year medical students (n=4) and two focus groups with third-year medical students (n=20). Interview and focus group questions were designed utilizing an appreciative inquiry approach.7-9 Our primary interview/focus group questions (see Appendix) specifically asked participants to, 1) describe a successful learning moment and what they contributed to that moment, 2) identify their core values and how well these were reflected by our institution, and 3) recall an instance in which their values had been challenged. Additional, follow-up questions explored perceptions of mistreatment and barriers to an optimal learning environment. We transcribed the interviews and focus group recordings verbatim. We used a constant comparative method11 to code the transcripts into categories that included, among other things, successful and challenging learning moments. Multiple members of the team participated in coding or reviewing codes and resolved any discrepancies through discussion.
It was at this point that we decided to initiate a secondary analysis of our qualitative data with the aim of using the Macy-Conference Framework5 as an interpretive lens. In response to time constraints, we randomly selected 122 excerpts for inclusion in our secondary analysis, or approximately 2/3 of excerpts coded as successes or challenges. The length of these selected excerpts ranged from 71 to 2783 words (average=790).
Our secondary analysis borrowed an approach used in previous research,12,13 in which study participants assign 100 points amongst a set of elements to indicate the relative influence of each element on the topic of interest. For example, Balmer and colleagues12 asked 4th-year students to divide 100 points among the explicit, implicit, and extra curriculum for each of 10 school-wide learning objectives to show the relative influence on each type of curricula in the student’s acquisition of the knowledge and skills required to achieve each objective. Differences in how the participants assigned points led to important insights about the positive influences of the implicit curriculum on student learning, particularly of learning objectives associated communication, teamwork, and professionalism.
Similar to Balmer and her colleagues,12,13 we developed, piloted, and refined a method to assign weights to the components of the Macy-Conference Framework to capture raters’ perceptions of the relative importance of each component in the content of each selected excerpt. Based on our pilot work in which raters tended to assign very few points to either of the two components of the material domain (i.e., physical and virtual),5 we combined them into one, which we called ‘space’. Two research assistants (CB and TD) first discussed definitions of each component, practiced using those definitions in assigning weights, and then shared their experience to optimize calibration. The raters then assigned weights to a subset of assigned excerpts, blinded as to whether the excerpt had been coded as a success or challenge and to whether the speaker was a faculty, resident, or student. In assigning weights, the raters carefully read each assigned excerpt and considered the key content being expressed. They then inferred the relative influence of each of the four components in each excerpt by dividing up 100 points among the components. Both researchers assigned weights to 2/3 of the selected subset of excerpts so that 1/3 overlapped. To optimize congruence in their approach to assigning weights, the researchers reviewed and discussed each other’s weight assignments for the 1/3 overlapping excerpts before completing the task for all assigned excerpts. Table 1 contains example excerpts with point assignments.
To address research question 1, BR averaged the weights assigned to the four components for the 1/3 overlapping excerpts and then computed overall averages and standard deviations of weights for each of the four domains across excerpts. For questions 2 and 3, BR computed separate averages and standard deviations for excerpts coded as success and challenges or for each stakeholder group.
We received help from the institution’s Center for Clinical and Translational Science Biostatistics Core14 to analyze our data and determine the significance of observed differences. We first summarized results descriptively, where subject (faculty, resident, student) and type (challenge vs. success) were summarized as frequency and percentage, and weight within each of the four components (personal, social, organizational and space) was summarized using mean and standard deviation (SD), median (25th and 75th quartiles), and range. Given that the weights across the four components summed to 100, we used analysis methods appropriate for compositional data. We used a Friedman’s test to assess whether weight distributions differed across the 4 components. We used Dirichlet regression to assess whether weight distributions differed by type of excerpt (success versus challenge) and type of stakeholder (faculty, resident, or student).15 We reported Odds ratios (ORs) of weighting each component more than the Social component (reference) with 95% confidence intervals (CIs) and p-values. We assessed statistical significance at the p<0.05 level two sided tests. We conducted all analyses using Rv.3.6.16
Overall, we observed statistically significant differences in the relative weights assigned to personal, social, organization, and space (p<0.001, Friedman’s test). The social component received the highest average weights, followed closely by personal, and then much less so by organizational and space. See Table 2 for frequencies and percentages of stakeholders (faculty, resident, student) and types of excerpt (success versus challenge). See Table 3 for the mean, median, and standard deviation for weights assigned to each component.
The average weight assignments of the four components follows the same rank order for Successes as well as Challenges (see Figures 1 and 2); however, the odds of an excerpt being weighted as an Organizational component more than the Social component is 40% lower for successes than for challenges OR=0.60 (95% CI 0.40, 0.89, p=0.012).
Average weights assigned by stakeholder to the four components again, fall in the same order of social, personal, organizational, and space. The weight distributions of the four components are not significantly different among stake holders (all p-values are greater than 0.05).
There is extensive literature that attempts to understand what it takes to optimize the learning environment (where successes occur and challenges are minimized), both generally2,4,5,17-22 and in Surgery23,24 and OB-GYN, specifically.25,26 The results of our study add to this literature, by using the lens of the Macy-Conference Framework to affirm the greater emphasis of the psychosocial domain compared to the material domain.
Within the psychosocial domain, our data highlights the relative greater influence of the personal and social factors on both success and challenges. This finding is intuitive and is consistent with the literature. Teaching and learning inherently emphasize interactions between individuals. As many authors have suggested, these interactions are shaped–whether good or bad–by the idiosyncratic characteristics of the participants (personal component) and by the dynamics of the interpersonal relationship (social component).2,23,27 Stakeholders in our study consistently referenced personal characteristics of faculty and learners. As illustrated in the sample excerpts included in Table 1, personal characteristics associated with successes and/or challenges include listening skills, a willingness to say ‘I don’t know’, a willingness to focus on teaching during surgery, an awareness of the learning climate, choosing to take the time to teach, and feeling safe enough to speak up.
The excerpts we included in our secondary analysis, as illustrated in Table 1, consistently highlight the interpersonal conditions central to promoting effective learning and responding to challenges. Examples include: involving others in teaching interactions, teaching styles, dynamics created with multiple levels of learners, communication patterns, giving and receiving feedback, and including or excluding learners. As a result, these data affirm the leading role of the social component in promoting success or in overcoming challenges in the learning environment, such as setting expectations, slowing down to explain whenever possible, providing timely feedback, promoting open communication, incrementally increasing learner autonomy, or protecting even a few minutes daily for teaching.1,25,28
Our data highlight the influence of organizational elements on the quality of the learning environment. While less than personal and social, the organizational elements deserve strong consideration in efforts to understand and shape the learning environment–particularly in terms of responding to challenges.2,18,22,25 As seen in the excerpts in Table 1, organizational elements that promote success, include working within the hierarchy of a team and establishing a culture which prioritizes interactions with learners. On the other hand, examples related to the organizational component that appear to create challenges to be overcome include patient care services that are perceived as devoid of any learning opportunities and the level of stress triggered by the broader environmental conditions.2,18,29
Our data suggest that the material domain of the Macy-Conference Framework has less influence in shaping the learning environment than the psychosocial domain. We consider this an interesting finding because a major addition of the Macy-Conference Framework, compared to the model proposed previously by Schonrock-Adema et. al.6 is the addition of the material domain. Reference to the material domain seldom occurred and thus raters consistently gave it little to no weight. As captured example 3 in Table 1, exceptions did occur, in which the speaker referenced an association between space (OR, clinical, floor) and learning.
Because the results of this study suggest that organizational factors may have greater influence for challenges than for successes, regardless of stakeholder group, we suggest leaders may want to look at interventions to improve the learning environment differently depending upon whether their intent is to promote successes versus overcome challenges. In particular, in order to promote more successes, leaders may want to first look for ways to influence social or personal aspects in the psychosocial domain using such interventions as faculty development30-34 and enhancing longitudinal relationships.35,36 On the other hand, to minimize challenges, leaders may want to first look for ways to modify organizational aspects of the learning environment, such as schedules, incentives, or policies.4,37 That said, because the Macy-Conference Framework assumes that the domains/components are continually overlapping and interacting, the development and application of interventions must be addressed at multiple levels.
This secondary analysis of existing qualitative data took place in a single institution and in two procedurally-oriented departments. The material domain was not well represented, possibly because space tends to influence our behavior without us consciously knowing.38 This phenomenon could also be a product of our explicit focus in Surgery and OBGYN, whereby the operating theater is the dominant clinical learning environment and the space where learning occurs in these disciplines is a given and is beyond the control of educators. Perhaps we would find greater variation in the space component if we were to expand to other more medically oriented specialties. While our study was able to affirm the relative importance of the various components of the Macy-Conference Framework, it was not designed to identify what may be missing from the Framework. Our primary study, is much better equipped to meet this challenge.
Clinical learning environments are composed of complex interactions between people, organizational structures, and physical factors which work dynamically to promote or impede learning. Such environments are in constant need of shaping and reshaping to best meet the needs of stakeholders–be they students, residents, or faculty. Using secondary analysis of transcribed excerpts from appreciative inquiry style interviews and focus groups, this study supports the value of the Macy-Conference Framework as a lens for better understanding the interacting components of the learning environment with an eye to continual improvement. Indeed, components are not all equally influential and their relative influence tends to change when one is focused on promoting successes versus minimizing challenges. In the former, the focus is most likely to be on social and personal components, with a secondary focus on organization and space. On the other hand, in the latter, the focus is more likely to also include the organizational component.
By using the Macy-Conference Framework as a lens for our secondary analysis, we provide evidence related to the validity of the framework and its potential utility in shaping the learning environment. Our data reinforces the presence of a common set of core components of the learning environment and their overall relative importance, particularly in the psychosocial domain, and should inform future efforts to optimize the clinical learning environment.
The research reported in this publication was supported in part by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002538. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Ethical approval. University of Utah Internal Review Board
1. Brandford E, Hasty B, Bruce JS, et al. Underlying mechanisms of mistreatment in the surgical learning environment: A thematic analysis of medical student perceptions. American journal of surgery. 2018;215(2):227-232.
2. Kilty C, Wiese A, Bergin C, et al. A national stakeholder consensus study of challenges and priorities for clinical learning environments in postgraduate medical education. BMC Med Educ. 2017;17(1):226.
3. Mazer LM, Bereknyei Merrell S, Hasty BN, Stave C, Lau JN. Assessment of programs aimed to decrease or prevent mistreatment of medical trainees. JAMA Network Open. 2018;1(3):e180870.
4. van der Goot WE, Cristancho SM, de Carvalho Filho MA, Jaarsma ADC, Helmich E. Trainee-environment interactions that stimulate motivation: A rich pictures study. Medical Education. 2019;54(3):242-253.
5. Gruppen LD, Irby DM, Durning SJ, Maggio LA. Conceptualizing Learning Environments in the Health Professions. Academic medicine : journal of the Association of American Medical Colleges. 2019;94(7):969-974.
6. Schonrock-Adema J, Bouwkamp-Timmer T, van Hell EA, Cohen-Schotanus J. Key elements in assessing the educational environment: where is the theory? Adv Health Sci Educ Theory Pract. 2012;17(5):727-742.
7. Rama JA, Falco C, Balmer DF. Using Appreciative Inquiry to Inform Program Evaluation in Graduate Medical Education. J Grad Med Educ. 2018;10(5):587-590.
8. Sandars J, Murdoch-Eaton D. Appreciative inquiry in medical education. Med Teach. 2017;39(2):123-127.
9. Williams A, Haizlip JA. Ten Keys to the Successful Use of Appreciative Inquiry in Academic Healthcare. OD Practitioner. 2013;45(2).
10. Bushe GR. Appreciative Inquiry is Not (Just) About the Positive
OD Practitioner. 2007;39(4):30-35.
11. Glaser B SA. The discovery of grounded theory: strategies for qualitative research. . New York, NY: Aldine de Gruyter; 1967.
12. Balmer DF, Hall E, Fink M, Richards BF. How do medical students navigate the interplay of explicit curricula, implicit curricula, and extracurricula to learn curricular objectives? Academic medicine : journal of the Association of American Medical Colleges. 2013;88(8):1135-1141.
13. Balmer DF, Quiah S, DiPace J, Paik S, Ward MA, Richards BF. Learning across the explicit, implicit, and extra-curricula: an exploratory study of the relative proportions of residents’ perceived learning in clinical areas at three pediatric residency programs. Academic medicine : journal of the Association of American Medical Colleges. 2015;90(11):1547-1552.
14. CCTS Population Health Research. https://medicine.utah.edu/ccts/population-health/.
15. Mazotti L, Adams J, Peyser B, Chretien K, Duffy B, Hirsh DA. Diffusion of innovation and longitudinal integrated clerkships: Results of the clerkship directors in internal medicine annual survey. Medical Teacher. 2019;41(3):347-353.
16. Maier MJ. DirichletReg: Dirichlet Regression in R. R package version 0.7-0. 2020.
17. Fried JM, Vermillion M, Parker NH, Uijtdehaage S. Eradicating medical student mistreatment: a longitudinal study of one institution’s efforts. Academic medicine : journal of the Association of American Medical Colleges. 2012;87(9):1191-1198.
18. Gan R, Snell L. When the learning environment is suboptimal: exploring medical students’ perceptions of “mistreatment”. Academic medicine : journal of the Association of American Medical Colleges. 2014;89(4):608-617.
19. House JB, Griffith MC, Kappy MD, Holman E, Santen SA. Tracking Student Mistreatment Data to Improve the Emergency Medicine Clerkship Learning Environment. The western journal of emergency medicine. 2018;19(1):18-22.
20. Kulaylat AN, Qin D, Sun SX, et al. Perceptions of mistreatment among trainees vary at different stages of clinical training. BMC Med Educ. 2017;17(1):14.
21. Olasoji HO. Broadening conceptions of medical student mistreatment during clinical teaching: message from a study of “toxic” phenomenon during bedside teaching. Advances in medical education and practice. 2018;9:483-494.
22. Oser TK, Haidet P, Lewis PR, Mauger DT, Gingrich DL, Leong SL. Frequency and negative impact of medical student mistreatment based on specialty choice: a longitudinal study. Academic medicine : journal of the Association of American Medical Colleges. 2014;89(5):755-761.
23. Castillo-Angeles M, Watkins AA, Acosta D, et al. Mistreatment and the learning environment for medical students on general surgery clerkship rotations: What do key stakeholders think? American journal of surgery. 2017;213(2):307-312.
24. Kemp MT, Smith M, Kizy S, Englesbe M, Reddy RM. Reported Mistreatment During the Surgery Clerkship Varies by Student Career Choice. J Surg Educ. 2018;75(4):918-923.
25. Lau JN, Mazer LM, Liebert CA, Bereknyei Merrell S, Lin DT, Harris I. A Mixed-Methods Analysis of a Novel Mistreatment Program for the Surgery Core Clerkship. Academic medicine : journal of the Association of American Medical Colleges. 2017;92(7):1028-1034.
26. Baecher-Lind LE, Chang K, Blanco MA. The learning environment in the obstetrics and gynecology clerkship: an exploratory study of students’ perceptions before and after the clerkship. Med Educ Online. 2015;20:27273.
27. Singh TSS, Singh A. Abusive culture in medical education: Mentors must mend their ways. Journal of anaesthesiology, clinical pharmacology. 2018;34(2):145-147.
28. Furney SL, Orsini AN, Orsetti KE, Stern DT, Gruppen LD, Irby DM. Teaching the one-minute preceptor. A randomized controlled trial. J Gen Intern Med. 2001;16(9):620-624.
29. Dyrbye LN, Thomas MR, Shanafelt TD. Medical student distress: causes, consequences, and proposed solutions. Mayo Clinic proceedings. 2005;80(12):1613-1622.
30. Birman BF, Desimone L, porter AC, Garet MS. Designing professional development that works. Educational Leadership. 2000.
31. Desimone LM, Porter AC, Garet MS, Yoon KS, Birman BF. Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study. Educational Evalution and Policy Analysis. 2002;24(2):81-112.
32. Gast I, Schildkamp K, Veen JTvd. Team-Based Professional Development Interventions in Higher Education: A Systematic Review. Review of Educational Research. 2017;87(4):736-767.
33. McLean M, Cilliers F, Van Wyk JM. Faculty development: Yesterday, today and tomorrow. Medical Teacher. 2008;30(6):555-584.
34. Sklar DP. Moving From Faculty Development to Faculty Identity, Growth, and Empowerment. Academic Medicine. 2016;91(12):1585-1587.
35. Hirsh D, Walters L, Poncelet AN. Better learning, better doctors, better delivery system: Possibilities from a case study of longitudinal integrated clerkships. Medical Teacher. 2012;34(7):548-554.
36. Hirsh DA, Holmboe ES, ten Cate O. Time to Trust: Longitudinal Integrated Clerkships and Entrustable Professional Activities. Academic Medicine. 2014;89(2):201-204.
37. Vanstone M, Grierson L. Social power facilitates and constrains motivation in the clinical learning environment. Med Educ. 2020;54(3):181-183. 38. Harrouk C. Psychology of Space: How Interiors Impact our Behavior? 2020; https://www.archdaily.com/936027/psychology-of-space-how-interiors-impact-our-behavior. Accessed 9/17/20, 2020.
**This is a semi-structured interview protocol, adapted from a guide originally developed by Drs. Williamson and Suchman.1 Probes will be used as necessary to elicit additional pertinent information.
• Introduction: This is going to be what we call an appreciative interview. I am going to ask you questions about times when you experienced educational things working at their best here at [institution]. Many times, we try to ask questions about things that aren’t working well—the problems—so that we can fix them. In this case, we are trying to find out about the things at their best—the successes—so that we can find out what works and why, and find ways to infuse more of it into our practice.
• As we get started, I’d like to know a little bit about you. Just so you know, this information will not be associated with any of your stories or quotes, but will just be used to provide context to our findings.
o What’s your role here at [institution] and how long have you been here?
• People do their best work when they are doing things that they find personally meaningful, and when they feel that their work makes a difference. During your time at [institution], there have no doubt been high points and low points. For now, I’d invite you to think of a teaching and learning moment that meant a lot to you, when things went right, a time that brought out the best in you.
o Please tell the story of that time. (If they are very general, try to probe for more specificity.)
o Without worrying about being modest, please tell me what it was about you—your unique qualities, gifts or capacities; decisions you made; or actions you took—that contributed to this teaching/learning experience?
o What did others contribute or do?
o What aspects of the situation made this a success (for example, the place, the time of day or year, recent events)?
• Now, think of a time at [institution] when you or your values were challenged.
o Please tell me a story about that time. (If participant needs clarification about what a value is, explain that a value is “a person’s principles or standards of behavior; one’s judgment of what is important in life.”)
• We each have different qualities, gifts and skills we bring to the world and to our work. Think about the things you value about yourself, the nature of your work and the university. At work, we’re always dealing with challenges and change.
• How have your strengths and values helped you deal with challenges and change?
o Your work: When you are feeling good about your work, what do you like about the work itself?
o Yourself: Imagine you’re at your retirement party. What do you think your colleagues would say they liked most about you?
o Yourself: Now what do you think your students would say they’ve liked most about you?
o How do your personal values match those of [institution]? (for example, honesty, compassion, teamwork)?
o Where have you seen examples of these values at [institution]?
• Where do you think these reports of mistreatment are coming from?
Return to Table of Contents:Personal, Social, Organizational, and Space Components of the Clinical Learning Environment: Variations in their Perceived Influence by Chanta'l Babcock, MAS, Candace Chow, PhD, Brigette Smith, MD, Luke Buchman, MD, Tiffany Weber, MD & Boyd F. Richards, PhD