Volume 2

Acknowledgements

We acknowledge the support of the University of Utah Teaching and Learning Technologies, the University of Utah College of Nursing, and the University of Utah Consortium for Families and Health Research.

Funding

Program revisions were funded through a University of Utah Teaching and Learning Technologies Online Program Development Grant.

Declaration of Interest

We have no conflicts of interest to declare.

Abstract

The recent adoption of gerontology competencies for undergraduate and graduate education emphasizes a need for national standards developed to enhance and unify the field of gerontology. The Gerontology Interdisciplinary Program at the University of Utah revised all of the Gerontology course offerings to align with the Association for Gerontology in Higher Education’s (AGHE) Gerontology Competencies for Undergraduate and Graduate Education (2014), while also making improvements in distance instructional design. In this study, we examined student course evaluation scores and written comments in six Master of Science in Gerontology core courses (at both 5000 and 6000 levels) prior to and following alignment with AGHE competencies and online design changes. Data included evaluations two semesters prior to and two semesters following course revisions and was assessed using paired t-test and thematic analysis. No significant statistical findings were found between pre and post revisions. Qualitative comments post revision did show an increased focus in comments about interactive and engaging technology. These findings will be used for course and program quality improvement initiatives, including enhanced approaches to documenting and assessing competency-based education.

Keywords

Competency-based education, course evaluation, course revision, distance education

Background

Competency-based education (CBE) is growing in popularity and demand (Burnette, 2016; McClarty & Gaertner, 2015). Gerontology curriculum development has moved towards CBE with national standards developed to enhance and unify the field of gerontology (Association for Gerontology in Higher Education [AGHE], 2014; Damron-Rodriguez et al., 2019).  The Academy for Gerontology in Higher Education (AGHE) approved the Gerontology Competencies for Undergraduate and Graduate Education (AGHE, 2014); designed to serve as a curricular guide for undergraduate (i.e., majors, minors, certificates) and master’s degree level programs. Benefits of using competencies for curricular revisions, include: shifting focus to measurable outcomes (Burnette, 2016; Damron-Rodriguez et al., 2019; Wendt, Peterson, & Douglass, 1993), increasing program accountability for learning outcomes (Burnette, 2016; Damron-Rodriquez et al., 2019; McClarty & Gaertner, 2015), preparing students to graduate with necessary skills (McClarty & Gaertner, 2015), and training the gerontological workforce by bridging the gaps between aging services and gerontology education (Applebaum & Leek, 2008; Damron-Rodriquez et al., 2019).

At the same time CBE has increased, online teaching and learning are more accessible and in demand (Means, Toyama, Murphy, Bakia, & Jones, 2010; Woldeab, Yawson, & Osafo, 2020). For programs looking to enhance curriculum and program accessibility, considering both CBE and distance course design are vital. Quality course design for courses incorporating CBE, emphasize opportunities for student application and practice, active learning strategies, and timely instructor response and feedback (Krause, Dias, & Schedler, 2015). In a previous paper (Dassel, Eaton, & Felsted, 2019) we described an approach to the program-wide revisions to align with the AGHE competencies and to meet current recommendations in cyber-pedagogy. The University of Utah Gerontology Interdisciplinary Program (GIP) was in a position to make revisions to enhance both CBE and online instructional design using a course/credit model that embeds competencies within a traditional approach to higher education that offers credit hours towards a degree (Council of Regional Accrediting Commissions [C-RAC], 2015). The University’s Online Teaching and Learning Technologies (TLT) office released a funding opportunity for programs wanting to move completely online. The GIP took the opportunity to apply to use funds with the following purpose: 1) transition the Masters of Science program into a completely online format, and 2) improve the quality and consistency of existing gerontology courses through a full curriculum review by the experts at TLT. The goal was to make the fully online transition in a manner that allowed for dynamic online learning and to incorporate CBE within the program. In 2015, the GIP began the work to revise all program courses to meet best practices of online learning and map program curricula to National Competencies in Gerontology Education (AGHE, 2014).

            Course revisions were complete in 2017. We then applied for and received official UOnline Program status and accreditation as a fully online program through the Northwest Commission on Colleges and Universities (2020). This accreditation allows us to be recognized as an official UOnline Program at the University of Utah. The University is a member of the State Authorization Reciprocity Agreement (NC-SARA) which reduces the number of other state regulations to continually monitor, resulting in more efficiency in the authorization process. Through NC-SARA, GIP is able to offer and expand certain educational opportunities to students in and out of the state of Utah (National Council for State Authorization of Reciprocity Agreements, 2020). In 2017, we were also awarded Program of Merit (POM) status from the Academy for Gerontology in Higher Education (AGHE), at the master’s degree level. The process of curricula review, competency mapping, and online revisions/planning, facilitated our application, review, and award of the POM.

            Course revision and development followed a model that incorporated best practices in teaching pedagogy and online learning. These incorporated Fink’s (2003) approach to designing college courses, using the DREAM exercise, situational factors exercise, course alignment grid, and taxonomy of significant learning. A backward design approach (Wiggins & McTighe, 2005) helped faculty begin with competencies and learning objectives followed by identifying assessments that then measure those objectives. Bloom’s (1984) taxonomy was used to design assessments that evaluate the learning experiences accurately and active learning principles (Bonwell & Eison, 1991; Prince, 2004) guided choices to facilitate dynamic online learning. Instructional designers met individually with instructors to work through, enhance, and redesign courses to facilitate this work.

            Upon completion, the program continued to assess student learning using individual course assessments, grades, progress towards graduation, annual and exit student interviews, and alumni surveys. However, we wondered about the student experience and reaction to changes pre and post revision of the entire curricula. As this process spanned four years and courses, it became of interest to see if existing data might facilitate better understanding of the student experience pre compared to post program revision.

The purpose of this paper is to compare student course evaluations from six core courses of the Master of Science in Gerontology program before and after alignment with AGHE competencies and online design changes. The objective of this study is to analyze pre and post qualitative and quantitative student evaluations in order to assess indicators of program quality and improvement. We hypothesize that course evaluations will improve from pre to post revision. Testing of this hypothesis occurred through two aims:

Aim 1: Assess the changes in numerical course ratings provided by students pre to post course evaluation.

Aim 2: Assess the changes, pre to post course revision, in student open-ended feedback submitted with course evaluations.

Methods

Course Selection

For the purpose of the current study, we compared de-identified, anonymous student course evaluations in six of our Master of Science core courses before and after the course revision and alignment. The six core courses required in our Master of Science program are: 1) GERON 5001/6001: Introduction to Aging, 2) GERON 5370/6370: Health and Optimal Aging, 3) GERON 5002/6002: Services Agencies and Programs for Older Adults, 4) GERON 5500/6500: Social and Public Policy in Aging, 5) GERON 5604/6604: Physiology and Psychology of Aging, and 6) GERON 5003/6003: Research Methods in Aging (Note. 5000 and 6000 level courses are considered to be graduate level by the University of Utah). Two additional core courses, GERON 5990/6990: Gerontology Practicum and GERON 6970/6975: Gerontology Thesis/Project, were omitted from this study as they were newly created in an online format, are mentor-based (one instructor to one or two students), and don’t receive evaluations due to the small course size.

These six core courses underwent significant redesign across three consecutive semesters. Each instructor worked one on one with an instructional designer provided through the UOnline grant mechanism. Instructional designers, associated with the University of Utah’s TLT, aided course instructors in updating their courses with the latest technological media to provide online content in innovative and effective ways. 

Course Evaluations

Faculty were guided in assessing and revising courses through the use of AGHE competencies (2014) and Fink’s (2003) and Bloom’s (1984) taxonomies. AGHE competencies were first mapped across all gerontology courses, identifying redundancy, overlap, and missing content. A detailed description of this process is described in Dassel et al. (2019). Faculty noted recommended revisions based on competencies specific to objectives and modified content. These were incorporated as faculty worked with instructional designers on their assigned course. Next, instructors used the framework of taxonomies to redesign the student learning experience for an active online format. Fink’s taxonomy is a non-hierarchical model, which defines six major domains that need to be present for a complete learning experience – foundational knowledge, application, integration, human dimensions, caring, and learning to learn (Fink, 2003). Bloom’s taxonomy, revised posthumously by a group of cognitive psychologists in 2001, is a hierarchical model which defines and distinguishes six categories of learning (Bloom, 1984; Anderson & Krathwohl, 2001). Bloom’s six categories, which are each intended to be individually mastered before moving to the next category, are remember, understand, apply, analyze, evaluate, and create. These designations allow for the design of the accompanying assessment to accurately evaluate the learning experience by level.

Permission to analyze student course evaluations was submitted and reviewed by the Institutional Review Board (IRB) at the University of Utah. The IRB determined oversight was not required as this work does not meet the definition of Human Subjects Research. All student evaluations are completed on an anonymous basis. Evaluations are used as a tool of quality improvement to assess course outcomes and faculty instructions. In order to obtain a representative sample of student evaluations, we assessed evaluations from two consecutive semesters immediately prior to the course revision and the two consecutive semesters immediately following the course revision.

Course evaluations were emailed to students during the last month of the semester. Students were asked to voluntarily complete the anonymous course evaluations. The data, consisting of numerical scaled response options and open-ended comment sections, was summarized and provided to course instructors at the end of the semester once grades have been submitted. From the full list of course evaluation questions, we selected 10 quantitative questions that we felt were most relevant to course revision. The questions selected include: 1) Overall course evaluation, 2) The course objectives were clearly stated, 3) The course objectives were met, 4) The course content was well organized, 5) The course materials were helpful in meeting course objectives, 6) Assignments and exams reflected what was covered in the course, 7) I learned a great deal in this course, 8) As a result of this course, my interest in the subject increased, 9) Course requirements and grading criteria were clear, and 10) I gained an excellent understanding of concepts in this field. Response options were based on a 5-point Likert scale with 1= strongly disagree to 6 = strongly agree. Open-ended questions ask students to comment on: 1) course effectiveness, 2) the online components of the course, and 3) comments intended for the instructor.

Data Analysis

Data analysis occurred in two phases. Phase one focused on quantitative data from the course evaluations. Pre and post data were aggregated for each course. Since students do not take a course multiple times, analyzing data pre to post by individual student is impossible. Rather than focus on the individual student as the unit of analysis, we assessed pre and post evaluations using the course as the unit of analysis. The means of each sample were calculated for each of the course evaluation questions (e.g., overall course rating, course objectives) as a proxy for evaluating the effectiveness of curriculum revision and course mapping. We used univariate statistics to describe frequencies and mean responses for each evaluation question. Paired-samples t-tests were conducted on the course means to examine score changes from pre to post course revision. Each course was compared separately and then data was pooled for all courses to assess program change over time. For the qualitative portion of this study, we compiled and organized all of the course evaluation open-ended student responses by course and semester. Data was uploaded into NVivo (QSR International, 2018) and assessed in a two-phase process. First each comment was read and coded into four a priori codes: 1) pre-commendations, 2) pre-recommendations, 3) post-commendations, 4) post-recommendations. The second phase of coding used thematic analysis to assess the main themes presented by students (Saldaña, 2009). This allowed us to assess potential change in student thoughts pre- to post-revision.

Results

Data are anonymous and demographics were not gathered as part of student evaluations. However, we do have a general idea of student demographics within the GIP. During a recent fall semester, we had 189 unique students enrolled in gerontology courses. Students represented 6 master’s degree programs, and 3 doctorate programs; with 9 students undeclared and 4 nonmatriculated. The average age of students was 29; 137 female (72.5%), 50 male (26.5%), and 2 unknown (1.05%). The majority of students were white (67.72%), with others identifying as Hispanic/Latino (13.76%), Asian (7.40%), unknown ethnicity (4.23%), multi-racial (3.70%), international (1.59%), Black/African American (1.05%), and Native Hawaiian or other Pacific Islander (0.53%).

A summary of the t-test data results is found in Table 1. Some data were unavailable due to too few responses. One course, GERON 5500/6500, did not have sufficient data for analysis (less than 2 observations per class), as it was a newly developed course and did not have sufficient pre-revision data. This course was retained in the overall comparison of results pre to post. Paired t-tests comparing overall course ratings pre and post course revision revealed a trend in improvement in the GERON 5001/6001: Introduction to Aging (t=4.09; p=.05) course. Examination of aggregate data from all of the courses in relation to individual course evaluation questions showed trends in improvement for the following two areas: 1) “The course objectives were met” (t=1.47; p=.09), and 2) “I learned a great deal in this course” (t=1.36; p=.09). There were no significant differences on overall or individual course evaluation questions pre to post course revision.

Open-Ended Student Comments

Qualitative analysis summarizes both overall number of commendations and recommendations and the content of comments to assess change pre to post revision. A total of 298 codes were documented pre-revision (see Table 2). Of these 71% were commendations, focusing on positive feedback about course content, online teaching, and instructor efficacy. Comments focusing on recommendations for change comprised 29% of the total pre-revision codes. These recommendations centered on issues with course content, technology, and instruction. Comments in the recommendation’s category included both negative reviews and constructive ideas for change. Post-revision comments were coded 257 times. Seventy-three percent of these were commendations and 27% were recommendations (Table 2). Percentages are very similar pre to post, demonstrating that overall positive or negative comments did not alter much from pre to post revision.

The second phase of qualitative analysis assessed the content of the comments to understand the topics focused on pre to post revision. Student comments were evaluated for each course; pre-revision comments were analyzed first followed post the post-revision comments. After identifying themes within pre-revision comments, a summary was written of the main ideas. Following this, post-revision comments were read and coded for the same course. A summary was then written about the main themes for the post-revision codes. Representative quotes were included in each summary, in order to present examples of themes. At this time the pre and post revision summaries were compared for each course. Any major thematic changes were noted in a final course comparison summary. Once this process was complete for each course, all course comparison summaries were re-read and coded for similarities and differences across the group of courses. Table 3 includes a summary of each course, including representative quotes.

Summary of Qualitative Comments Pre to Post Revision

The following summarizes overall findings from qualitative analysis of student open-ended course evaluation comments. Student comments increased in two main areas post-revision when compared to pre-revision: 1) connection to the instructor, and 2) organized content.

            Connection to the Instructor. Students expressed not wanting all the extra technological features integrated into courses such as screen and video recorded Power Point lectures, interactive quizzes, and movie creation apps. A variety of apps (e.g., Flipgrid, Lucid chart, Pathbrite) led to confusion and overwhelmed students. However, students emphasized the importance of technology in helping them maintain connection with the instructor. For example, one student stated, “I especially liked the introduction videos before each module because it felt like the instructor was in constant communication with the class.” The adoption of video was particularly useful in helping students feel this connection.

            Organized Content. Comments emphasized the importance of balancing assignments, content, and the amount of work. Students noted that spreading assignments out throughout the semester helped them disperse their stress. This was most often mentioned when a course had multiple assignments due the last week of the semester. One student commented, “Assign one of the larger projects to be due at mid-term, to space out the stress.” Students value learning and in an online environment this requires incorporating moments of accountability to help students interact with the content. Students emphasized wanting these opportunities for accountability and when a course was lacking this, they acknowledged their lack of course interaction. “I have mixed feelings about the assignments. On the one hand, I feel that the small amount of assignments was nice, but also allowed for me to be less involved in the course than perhaps I should have.”

Discussion

In this mixed method, multi-year study examining student evaluations from pre to post course revisions, quantitative analysis did not produce statistically significant differences in mean course evaluation scores. This may be attributed to the small sample size, use of aggregate data rather than individual data points, missing data, and little variation in scores with most courses receiving high mean scores. In the qualitative analysis of student evaluations, we gained useful information. We found that students value technology that augments their connection to the instructor and course organization. Some students do not want all the extra features that come with a wide variety of technology (e.g., external sites to create blogs, mini podcasts, video creation). Students noticed video introductions, video lectures, and video summaries, often stating it made them feel connected to the instructor. This aligns with the quality indicators in CBE online courses that emphasize the importance of technology and navigation as one of seven recommended areas for measurement (Krause et al., 2015). Students want to learn. Learning online necessitates the incorporation of one or more forms of accountability (which the students want). In addition, students desire forms of accountability throughout the semester, rather than just at semester’s end. The balance of assignments, content, and amount of work matters to students. Instructional design is vital in quality online courses. Accountability should be an area that faculty and instructional designers collaborate on to facilitate enhanced quality in online CBE. Two quality indicators of accountability include 1) assessment and evaluation, and 2) competence and learning activities (Krause et al., 2015). We also observed an increase in student comments specific to a certain topic each time a major adjustment occurred, whether pre or post revision. This could be an outcome of the “growing pains” related to trying something new. Similar to piloting research, faculty pilot testing teaching strategies often need student feedback to improve changes in a manner that actually works for students. Checking in with students demonstrates the quality indicator of learner support, and allows faculty to assess and evaluate their course as part of quality assurance (Krause et al., 2015).

The information obtained from this study is relevant to course and program quality improvement. Strengths include the mixed method format and multi-year analysis. Limitations include data that did not allow for pre and post data from the same students, as it is impossible to require students to take a course twice. In some cases, there was not sufficient data for analysis, as t-tests require at least two observations per class (e.g., GERON 5500/6500). This insufficient data was attributed to new course development and changes in student evaluation questions that occurred across the University of Utah. This meant that questions were different pre to post revision for some courses. In addition, conducting a technology revision simultaneously with competency revisions makes it difficult to tease out changes due to course format versus curriculum. Instructors need to remind students which competencies are being covered and how they will expect to interact with this content during the course. Clear learning outcomes and student comprehension of the proficiencies they are working on enhances CBE (Burnette, 2016).

Mapping the entire GIP curriculum to the AGHE competency guidelines (Dassel et al., 2019) prepared us to apply for and receive Program of Merit designation through AGHE. This Program of Merit status has provided the foundation for future application for accreditation through the Accreditation for Gerontology Education Council (AGEC), which requires that the programs under review align with the AGHE Gerontology Competencies for Undergraduate and Graduate Education (2014). Students from all health science disciplines participate in undergraduate and graduate level certificates available through our program. Improving program quality and demonstrating the efficacy of such changes should strengthen the ability of students to work with older adults in community and health care settings.

Programs should build on CBE by developing measures to assess student achievement of competencies. This process can be used to improve the quality of the student learning experience (Damron-Rodriguez et al., 2019; McClarty & Gaertner, 2015). Our program is developing a tool that will allow faculty to assess program learning outcomes and AGHE competencies within each class. Data will be gathered every 3 years and will facilitate progress at both the course and program levels.  Tools, such as this, can be shared in an effort to develop tool-kits for other gerontology programs to build quality models of competency-based education (Damron-Rodriguez et al., 2019). It is our goal to enhance the ability of graduates to demonstrate the competencies and skills they have gained through high quality gerontology education as they work with employers and older adults. We will enhance our approach to CBE by assessing the path alumni take and their use of competencies to communicate their knowledge, skills, and contributions within the workforce. Advancing CBE in gerontology needs to happen through organizational leadership (Damron-Rodriguez et al., 2019). Our program benefits from being housed within a College of Nursing that follows a CBE model and process for accreditation. We can learn from this process of documentation, tracking, assessment, and quality improvement to enhance the rigor and approach we take to CBE in gerontology programs. Finally, plan to share our CBE strategies, assessment tools, and models gerontology programs  in the Utah State Gerontology Collaborative.

The results of this study have implications beyond the Gerontology Interdisciplinary Program to the larger Health Sciences campus where our program and college are housed. Many interprofessional health science students enroll in our courses. Thus, improving program quality and demonstrating efficacy ultimately strengthens students’ ability to work effectively with older adults in a variety of settings.

References

Anderson, L. W., & Krathwohl, D. R. (2001). A taxonomy for learning, teaching, and assessing: A revision of Bloom’s taxonomy of educational objectives. New York: Longman.

Applebaum, R. & Leek, J. (2008). Bridging the academic/practice gap in gerontology and geriatrics: Mapping a route to mutual success. Annual Review of Gerontology and Geriatrics, 28, 131-148. doi: 10.1891/0198-8794.28.131

Association for Gerontology in Higher Education [AGHE] (2014). Gerontology competencies for undergraduate and graduate education. Washington, DC: Association for Gerontology in Higher Education. Retrieved from: https://www.geron.org/images/gsa/AGHE/gerontology_competencies.pdf

Bloom, B. S. (1984). Taxonomy of educational objectives: The classification of educational goals. New York: Longman.

Bonwell, C. C., & Eison, J. A. (1991). Active learning: Creating excitement in the classroom. ASH-ERIC Higher Education Report. Washington, DC: School of Education and Human Development, George Washington University.

Burnette, D. M. (2016). The renewal of competency-based education: A review of the literature. The Journal of Continuing Higher Education, 64, 84-93. doi: 10.1080/07377363.2016.1177704

Council of Regional Accrediting Commissions [C-RAC]. (2015, June 2). Framework for competency-based education [Press release]. Retrieved from https://download.hlcommission.org/C-RAC_CBE_Statement_6_2_2015.pdf

Damron-Rodriguez, J., Frank, J. C., Maiden, R. J., Abushakrah, J., Jukema, J. S., Pianosi, B., & Sterns, H. L. (2019). Gerontology competencies: Construction, consensus and contribution. Gerontology & Geriatrics Education, 40(4), 409-431. doi: 10.1080/02701960.2019.1647835

Dassel, K., Eaton, J., & Felsted, K. (2019). Navigating the future of gerontology education: Curriculum mapping to the AGHE competencies. Gerontology & Geriatrics Education, 40(1), 132-138.

Fink, L.D. (2003) Creating significant learning experiences: An integrated approach to designing college courses. San Francisco: Jossey‐Bass.

Krause, J., Dias, L. P., & Schedler, C. (2015). Competency-based education: A framework for measuring quality courses. Online Journal of Distance Learning Administration, 18(1). Retrieved from https://www.westga.edu/~distance/ojdla/spring181/krause_dias_schedler181.html

McClarty, K. L. & Gaertner, M. N. (2015). Measuring mastery: Best practices for assessment in competency-based education. AEI Series on Competency-Based Higher Education. Washington, DC: Center on Higher Education Reform & American Enterprise Institute for Public Policy Research.

Means, B., Toyama, Y., Murphy, R., Bakia, M., & Jones, K. (2010). Evaluation of evidence-based practices in online learning: A meta-analysis and review of online learning studies. U.S. Dept. of Education, Office of Planning, Evaluation and Policy Development, Policy and Program Studies Service website. Retrieved from https://www2.ed.gov/rschstat/eval/tech/evidence-based-practices/finalreport.pdf

National Council for State Authorization Reciprocity Agreements [NC-SARA]. (2020). About NC-SARA. Retrieved from https://nc-sara.org/about-nc-sara

Northwest Commission on Colleges and Universities. (2020). Accreditation. Retrieved from https://www.nwccu.org/accreditation%20/

Prince, M. (2004). Does active learning work? A review of the research. Journal of Engineering Education, 93(3), 223-231.

QSR International Pty Ltd. (2018). NVivo qualitative data analysis software (version 12) [Software]. Retrieved from https://www.qsrinternational.com/nvivo-qualitative-data-analysis-software/home. Accessed May 17, 2020.

Saldaña, J. (2009). The coding manual for qualitative researchers. Thousand Oaks, CA: SAGE.

Wendt, P. F., Peterson, D. A., & Douglass, E. B. (1993). Core principles and outcomes of gerontology, geriatrics, and aging studies instruction. Washington, DC: Association for Gerontology in Higher Education and the University of Southern California.

Wiggins, G.P., & McTighe, J. (2005). Understanding by design. (2^nd Ed.).  Alexandria, VA: Association for Supervision and Curriculum Development.

Woldeab, D., Yawson, R.M, & Osafo, E. (2020). A systematic meta-analytic review of thinking beyond the comparison of online versus traditional learning. E-Journal of Business Education & Scholarship of Teaching, 14(1), 1-24.

Funding

AMA Accelerating Change in Education Innovation Grant program

Disclosures

None

What problem was addressed:

Medical schools prepare students to enter a complex health system with the knowledge to care for patients but provide little training on the health system they join. Health systems science (HSS) is an important topic that is starting to enter medical school curricula. The difficulty is how to teach this complex topic which is slow to gain traction with key stakeholders.¹ We argue that HSS is not as difficult a concept to implement if presented in a familiar context that encourages active participation with the material. We present our educational innovation to teach HSS in an active learning setting that increased buy-in from medical students and faculty. 

What was tried:

We organized the Medical Education Design and Innovation Challenge (MEDIC), a competition that taught medical students HSS as they competed to design an educational innovation. We introduced 24 medical students from all years of training to HSS using the Shingo Model^{TM 2} as a framework, the model successfully used by the University of Utah Health system. Students were divided into 6 teams and asked to identify an area for improvement and then design a program, course, or initiative utilizing this model. The Shingo Model^TM requires users to identify guiding principles, key stakeholders, and important outcomes as precursory steps to any innovative problem-solving design. This encouraged students to understand education system before proposing a solution to a perceived deficit. This event was divided over two days with a total of 8 hours of participation. Students were introduced to HSS and the Shingo Model^TM during an introductory dinner and then placed into teams. The following day teams had 4 hours to identify a deficit within their school system and design a solution (ex. Mentorship program for specialty exploration) using the Shingo Model^TM as a framework. Teams then pitched their proposals and were evaluated on their creativity, feasibility, and evidence of utilizing systems science in their design. Winners were determined based on majority vote from guest faculty judges, coordinators and participants.

What lessons were learned:

Two major challenges exist in teaching HSS to medical students: relevance to learners and incorporation into already full medical school curricula. Survey data from MEDIC suggest this project-focused approach to teaching HSS addressed both challenges. Participants (63% response rate) revealed that 100% of students felt that MEDIC was relevant to them. 93% of students thought the Shingo Model^TM was an appropriate framework for approaching medical education innovation and 73% were confident in their ability to apply the model after only four hours of team-based work. 80% of students found they developed new skills and had a change of perception of medical education design by participating in MEDIC. Additionally, 80% of students agreed or strongly agreed that all students would benefit from exposure to HSS in the core curriculum. This experience may be easily reproduced at other institutions. The positive response of the students and success in proposing innovative ideas for medical education encouraged us to continue to use this framework as we engage students and faculty in ongoing curricular reform.

References

1. Gonzalo JD, Hawkins R, Lawson L, Wolpaw D, Chang A. Concerns and Responses for Integrating Health Systems Science Into Medical Education. Acad Med. 2018; 93(6):843–849
2. https://shingo.org/model

Abstract

Purpose. Because of its impact on learning, interest in the learning environment continues unabated. This includes a framework which emerged from a Macy-sponsored conference which organizes factors that influence the quality of the learning environment into four components: personal, social, organizational, and space. This paper reports a study which assessed the relative influence of these components.

Methods. This study involved the secondary analysis of a subset of transcribed excerpts obtained from a study using an appreciative-inquiry style of questions in interviews and focus groups with faculty, residents and students from two departments at the University of Utah School of Medicine conducted in 2019. After all excerpts had been coded using a constant comparative method, those assigned the codes of “successes” or “challenges” were considered for the secondary analysis. For each selected excerpt, trained research assistants divided up 100 points among the four components according to their perceived relative influence on the ideas expressed in the excerpt. Differences in the average number of points assigned across components by type of excerpt (success versus challenge) and by type of speaker (faculty, resident, student) were examined.

Results. Overall, the social component received the highest average number of points, followed closely by personal, and then much less so by organizational, and space. In both successes and challenges, the four components followed this same rank order, Nevertheless, the average number of points assigned to organization was significantly greater for challenges than for successes. There were no statistically significant differences in points assigned based on speaker.

Conclusion. Our secondary analysis of excerpts from interview and focus group transcripts confirm the relatively stronger influence of the social and personal components in the learning environment than the organizational and space. Nevertheless, a consideration of the organizational component appears warranted when seeking to overcome challenges which impede learning in the clinical environment.

BACKGROUND

The clinical learning environment is receiving substantial and increasing attention in medical schools as stakeholders strive to optimize learning outcomes while overcoming longstanding challenges such as mistreatment and marginalization of learners.^1-4

Given this level of interest, Gruppen and colleagues recently held a Macy-sponsored consensus conference on the learning environment from which was published a conceptual framework,⁵ hereafter in this article referred to as the Macy-Conference Framework. This framework integrates an extensive literature, including a recent review from Schonrock-Adema,⁶ with the intent to “facilitate health professions educators in understanding, studying, and designing interventions to improve the learning environment.”⁶

According to Gruppen and colleagues, the learning environment is “a complex psycho-social-physical construct that is co-created by individuals, social groups, and organizations in a particular setting.”⁵ The Macy-Conference Framework captures this complexity. It includes “five overlapping and interactive core components that form two dimensions: the psychosocial dimension and material dimension.”⁵ The psychosocial dimension comprises three components: the personal, social, and organizational. The material dimension encompasses physical and virtual spaces.⁵ The personal includes characteristics of participants in the learning environment which intrinsically shape behavior including knowledge, attitudes, perceptions, level of commitment, and a priori goals. The social focuses on the dynamics between individuals including interactions and social relations related to all aspects of the learning environment. The organizational includes norms, roles, and structures, which extrinsically shape behaviors (both individually or as a group). Space includes the physical and virtual characteristics of the environment in which learning and practice occur.

We encountered the Macy-Conference Framework after we had initiated an inquiry in 2019 into the learning environment at our institution, the University of Utah School of Medicine, with the intent of shifting focus from what’s wrong (e.g., reoccurring reports of mistreatment) to what’s working (e.g.,  stand out teaching moments). Our goal, consistent with the philosophy of appreciative inquiry,^7-9 was to optimize the learning environment by first discovering and documenting existing examples which we could showcase and replicate. While we also explored challenges and concerns, these were not our primary focus, per se. Our inquiry included conducting, transcribing, and coding appreciative-style interviews^8,10 and focus groups with faculty, residents, and students. We had reached the point in this inquiry of completing a first round of qualitative, inductive analyses, resulting in a robust set of coded excerpts, of which 189 had been coded as either “successes” or “challenges” in promoting learning in the clinical learning environment. 

As we began to consider how best to take advantage of the Macy-Conference Framework⁵ to analyze our data, we discovered that it wasn’t a matter of just using the domains or components as codes and assigning them to excerpts because all of the quotes simultaneously referred to the personal, social, organizational, and/or material components of the Macy-Conference Framework–explicitly or implicitly–with varying degrees of emphasis. This realization gave us the inspiration to initiate a secondary analysis of our data with the aim of using the Macy-Conference Framework as an interpretive lens to deepen our understanding of the conditions influencing the learning environment and ways to enhance that environment.

We guided our secondary analysis of these coded excerpts using the following questions:

1. What is the relative influence of each component in the Macy-Conference Framework in how stakeholders talk about the learning environment?
2. Are there differences in the relative influence of each component across excerpts coded as successes versus challenges?
3. Are there differences in the relative influence of each component across excerpts made by faculty, residents or students?

METHODS

After receiving IRB approval from the Institutional Review Board of the University of Utah, we collected data for the primary study upon which our secondary analysis was based through interviews and focus groups. We conducted individual interviews with Faculty and Residents in the Department of Surgery (n=7) and the Department of Obstetrics and Gynecology (n=6). We also conducted interviews with fourth-year medical students (n=4) and two focus groups with third-year medical students (n=20). Interview and focus group questions were designed utilizing an appreciative inquiry approach.^7-9 Our primary interview/focus group questions (see Appendix) specifically asked participants to, 1) describe a successful learning moment and what they contributed to that moment, 2) identify their core values and how well these were reflected by our institution, and 3) recall an instance in which their values had been challenged. Additional, follow-up questions explored perceptions of mistreatment and barriers to an optimal learning environment.  We transcribed the interviews and focus group recordings verbatim. We used a constant comparative method¹¹ to code the transcripts into categories that included, among other things, successful and challenging learning moments. Multiple members of the team participated in coding or reviewing codes and resolved any discrepancies through discussion.

It was at this point that we decided to initiate a secondary analysis of our qualitative data with the aim of using the Macy-Conference Framework⁵ as an interpretive lens. In response to time constraints, we randomly selected 122 excerpts for inclusion in our secondary analysis, or approximately 2/3 of excerpts coded as successes or challenges. The length of these selected excerpts ranged from 71 to 2783 words (average=790).

Our secondary analysis borrowed an approach used in previous research,^12,13 in which study participants assign 100 points amongst a set of elements to indicate the relative influence of each element on the topic of interest. For example, Balmer and colleagues¹² asked 4th-year students to divide 100 points among the explicit, implicit, and extra curriculum for each of 10 school-wide learning objectives to show the relative influence on each type of curricula in the student’s acquisition of the knowledge and skills required to achieve each objective. Differences in how the participants assigned points led to important insights about the positive influences of the implicit curriculum on student learning, particularly of learning objectives associated communication, teamwork, and professionalism.

Similar to Balmer and her colleagues,^12,13 we developed, piloted, and refined a method to assign weights to the components of the Macy-Conference Framework to capture raters’ perceptions of the relative importance of each component in the content of each selected excerpt. Based on our pilot work in which raters tended to assign very few points to either of the two components of the material domain (i.e., physical and virtual),⁵ we combined them into one, which we called ‘space’. Two research assistants (CB and TD) first discussed definitions of each component, practiced using those definitions in assigning weights, and then shared their experience to optimize calibration. The raters then assigned weights to a subset of assigned excerpts, blinded as to whether the excerpt had been coded as a success or challenge and to whether the speaker was a faculty, resident, or student. In assigning weights, the raters carefully read each assigned excerpt and considered the key content being expressed. They then inferred the relative influence of each of the four components in each excerpt by dividing up 100 points among the components.  Both researchers assigned weights to 2/3 of the selected subset of excerpts so that 1/3 overlapped. To optimize congruence in their approach to assigning weights, the researchers reviewed and discussed each other’s weight assignments for the 1/3 overlapping excerpts before completing the task for all assigned excerpts. Table 1 contains example excerpts with point assignments.

To address research question 1, BR averaged the weights assigned to the four components for the 1/3 overlapping excerpts and then computed overall averages and standard deviations of weights for each of the four domains across excerpts. For questions 2 and 3, BR computed separate averages and standard deviations for excerpts coded as success and challenges or for each stakeholder group.

We received help from the institution’s Center for Clinical and Translational Science Biostatistics Core¹⁴ to analyze our data and determine the significance of observed differences. We first summarized results descriptively, where subject (faculty, resident, student) and type (challenge vs. success) were summarized as frequency and percentage, and weight within each of the four components (personal, social, organizational and space) was summarized using mean and standard deviation (SD), median (25^th and 75^th quartiles), and range. Given that the weights across the four components summed to 100, we used analysis methods appropriate for compositional data.  We used a Friedman’s test to assess whether weight distributions differed across the 4 components. We used Dirichlet regression to assess whether weight distributions differed by type of excerpt (success versus challenge) and type of stakeholder (faculty, resident, or student).¹⁵ We reported Odds ratios (ORs) of weighting each component more than the Social component (reference) with 95% confidence intervals (CIs) and p-values. We assessed statistical significance at the p<0.05 level two sided tests. We conducted all analyses using Rv.3.6.¹⁶

RESULTS

Overall, we observed statistically significant differences in the relative weights assigned to personal, social, organization, and space (p<0.001, Friedman’s test). The social component received the highest average weights, followed closely by personal, and then much less so by organizational and space. See Table 2 for frequencies and percentages of stakeholders (faculty, resident, student) and types of excerpt (success versus challenge). See Table 3 for the mean, median, and standard deviation for weights assigned to each component.

The average weight assignments of the four components follows the same rank order for Successes as well as Challenges (see Figures 1 and 2); however, the odds of an excerpt being weighted as an Organizational component more than the Social component is 40% lower for successes than for challenges OR=0.60 (95% CI 0.40, 0.89, p=0.012).

Average weights assigned by stakeholder to the four components again, fall in the same order of social, personal, organizational, and space. The weight distributions of the four components are not significantly different among stake holders (all p-values are greater than 0.05).

DISCUSSION

There is extensive literature that attempts to understand what it takes to optimize the learning environment (where successes occur and challenges are minimized), both generally^2,4,5,17-22 and in Surgery^23,24 and OB-GYN, specifically.^25,26 The results of our study add to this literature, by using the lens of the Macy-Conference Framework to affirm the greater emphasis of the psychosocial domain compared to the material domain.

Within the psychosocial domain, our data highlights the relative greater influence of the personal and social factors on both success and challenges. This finding is intuitive and is consistent with the literature. Teaching and learning inherently emphasize interactions between individuals. As many authors have suggested, these interactions are shaped–whether good or bad–by the idiosyncratic characteristics of the participants (personal component) and by the dynamics of the interpersonal relationship (social component).^2,23,27 Stakeholders in our study consistently referenced personal characteristics of faculty and learners. As illustrated in the sample excerpts included in Table 1, personal characteristics associated with successes and/or challenges include listening skills, a willingness to say ‘I don’t know’, a willingness to focus on teaching during surgery, an awareness of the learning climate, choosing to take the time to teach, and feeling safe enough to speak up.

The excerpts we included in our secondary analysis, as illustrated in Table 1, consistently highlight the interpersonal conditions central to promoting effective learning and responding to challenges. Examples include: involving others in teaching interactions, teaching styles, dynamics created with multiple levels of learners, communication patterns, giving and receiving feedback, and including or excluding learners. As a result, these data affirm the leading role of the social component in promoting success or in overcoming challenges in the learning environment, such as setting expectations, slowing down to explain whenever possible, providing timely feedback, promoting open communication, incrementally increasing learner autonomy, or protecting even a few minutes daily for teaching.^1,25,28

Our data highlight the influence of organizational elements on the quality of the learning environment. While less than personal and social, the organizational elements deserve strong consideration in efforts to understand and shape the learning environment–particularly in terms of responding to challenges.^2,18,22,25 As seen in the excerpts in Table 1, organizational elements that promote success, include working within the hierarchy of a team and establishing a culture which prioritizes interactions with learners. On the other hand, examples related to the organizational component that appear to create challenges to be overcome include patient care services that are perceived as devoid of any learning opportunities and the level of stress triggered by the broader environmental conditions.^2,18,29

Our data suggest that the material domain of the Macy-Conference Framework has less influence in shaping the learning environment than the psychosocial domain. We consider this an interesting finding because a major addition of the Macy-Conference Framework, compared to the model proposed previously by Schonrock-Adema et. al.⁶ is the addition of the material domain. Reference to the material domain seldom occurred and thus raters consistently gave it little to no weight. As captured example 3 in Table 1, exceptions did occur, in which the speaker referenced an association between space (OR, clinical, floor) and learning.

Because the results of this study suggest that organizational factors may have greater influence for challenges than for successes, regardless of stakeholder group, we suggest leaders may want to look at interventions to improve the learning environment differently depending upon whether their intent is to promote successes versus overcome challenges. In particular, in order to promote more successes, leaders may want to first look for ways to influence social or personal aspects in the psychosocial domain using such interventions as faculty development^30-34 and enhancing longitudinal relationships.^35,36 On the other hand, to minimize challenges, leaders may want to first look for ways to modify organizational aspects of the learning environment, such as schedules, incentives, or policies.^4,37 That said, because the Macy-Conference Framework assumes that the domains/components are continually overlapping and interacting, the development and application of interventions must be addressed at multiple levels.

LIMITATIONS

This secondary analysis of existing qualitative data took place in a single institution and in two procedurally-oriented departments. The material domain was not well represented, possibly because space tends to influence our behavior without us consciously knowing.³⁸ This phenomenon could also be a product of our explicit focus in Surgery and OBGYN, whereby the operating theater is the dominant clinical learning environment and the space where learning occurs in these disciplines is a given and is beyond the control of educators. Perhaps we would find greater variation in the space component if we were to expand to other more medically oriented specialties. While our study was able to affirm the relative importance of the various components of the Macy-Conference Framework, it was not designed to identify what may be missing from the Framework. Our primary study, is much better equipped to meet this challenge.

CONCLUSION

Clinical learning environments are composed of complex interactions between people, organizational structures, and physical factors which work dynamically to promote or impede learning. Such environments are in constant need of shaping and reshaping to best meet the needs of stakeholders–be they students, residents, or faculty. Using secondary analysis of transcribed excerpts from appreciative inquiry style interviews and focus groups, this study supports the value of the Macy-Conference Framework as a lens for better understanding the interacting components of the learning environment with an eye to continual improvement. Indeed, components are not all equally influential and their relative influence tends to change when one is focused on promoting successes versus minimizing challenges. In the former, the focus is most likely to be on social and personal components, with a secondary focus on organization and space. On the other hand, in the latter, the focus is more likely to also include the organizational component.

By using the Macy-Conference Framework as a lens for our secondary analysis, we provide evidence related to the validity of the framework and its potential utility in shaping the learning environment. Our data reinforces the presence of a common set of core components of the learning environment and their overall relative importance, particularly in the psychosocial domain, and should inform future efforts to optimize the clinical learning environment.

Acknowledgements

The research reported in this publication was supported in part by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002538. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Funding/Support. None

Disclosures. None

Ethical approval. University of Utah Internal Review Board

Disclaimers. None.

 Previous presentations.

REFERENCES

1.         Brandford E, Hasty B, Bruce JS, et al. Underlying mechanisms of mistreatment in the surgical learning environment: A thematic analysis of medical student perceptions. American journal of surgery. 2018;215(2):227-232.

2.         Kilty C, Wiese A, Bergin C, et al. A national stakeholder consensus study of challenges and priorities for clinical learning environments in postgraduate medical education. BMC Med Educ. 2017;17(1):226.

3.         Mazer LM, Bereknyei Merrell S, Hasty BN, Stave C, Lau JN. Assessment of programs aimed to decrease or prevent mistreatment of medical trainees. JAMA Network Open. 2018;1(3):e180870.

4.         van der Goot WE, Cristancho SM, de Carvalho Filho MA, Jaarsma ADC, Helmich E. Trainee-environment interactions that stimulate motivation: A rich pictures study. Medical Education. 2019;54(3):242-253.

5.         Gruppen LD, Irby DM, Durning SJ, Maggio LA. Conceptualizing Learning Environments in the Health Professions. Academic medicine : journal of the Association of American Medical Colleges. 2019;94(7):969-974.

6.         Schonrock-Adema J, Bouwkamp-Timmer T, van Hell EA, Cohen-Schotanus J. Key elements in assessing the educational environment: where is the theory? Adv Health Sci Educ Theory Pract. 2012;17(5):727-742.

7.         Rama JA, Falco C, Balmer DF. Using Appreciative Inquiry to Inform Program Evaluation in Graduate Medical Education. J Grad Med Educ. 2018;10(5):587-590.

8.         Sandars J, Murdoch-Eaton D. Appreciative inquiry in medical education. Med Teach. 2017;39(2):123-127.

9.         Williams A, Haizlip JA. Ten Keys to the Successful Use of Appreciative Inquiry in Academic Healthcare. OD Practitioner. 2013;45(2).

10.       Bushe GR. Appreciative Inquiry is Not (Just) About the Positive

OD Practitioner. 2007;39(4):30-35.

11.       Glaser B SA. The discovery of grounded theory: strategies for qualitative research. . New York, NY: Aldine de Gruyter; 1967.

12.       Balmer DF, Hall E, Fink M, Richards BF. How do medical students navigate the interplay of explicit curricula, implicit curricula, and extracurricula to learn curricular objectives? Academic medicine : journal of the Association of American Medical Colleges. 2013;88(8):1135-1141.

13.       Balmer DF, Quiah S, DiPace J, Paik S, Ward MA, Richards BF. Learning across the explicit, implicit, and extra-curricula: an exploratory study of the relative proportions of residents’ perceived learning in clinical areas at three pediatric residency programs. Academic medicine : journal of the Association of American Medical Colleges. 2015;90(11):1547-1552.

14.       CCTS Population Health Research.  https://medicine.utah.edu/ccts/population-health/.

15.       Mazotti L, Adams J, Peyser B, Chretien K, Duffy B, Hirsh DA. Diffusion of innovation and longitudinal integrated clerkships: Results of the clerkship directors in internal medicine annual survey. Medical Teacher. 2019;41(3):347-353.

16.       Maier MJ. DirichletReg: Dirichlet Regression in R. R package version 0.7-0. 2020.

17.       Fried JM, Vermillion M, Parker NH, Uijtdehaage S. Eradicating medical student mistreatment: a longitudinal study of one institution’s efforts. Academic medicine : journal of the Association of American Medical Colleges. 2012;87(9):1191-1198.

18.       Gan R, Snell L. When the learning environment is suboptimal: exploring medical students’ perceptions of “mistreatment”. Academic medicine : journal of the Association of American Medical Colleges. 2014;89(4):608-617.

19.       House JB, Griffith MC, Kappy MD, Holman E, Santen SA. Tracking Student Mistreatment Data to Improve the Emergency Medicine Clerkship Learning Environment. The western journal of emergency medicine. 2018;19(1):18-22.

20.       Kulaylat AN, Qin D, Sun SX, et al. Perceptions of mistreatment among trainees vary at different stages of clinical training. BMC Med Educ. 2017;17(1):14.

21.       Olasoji HO. Broadening conceptions of medical student mistreatment during clinical teaching: message from a study of “toxic” phenomenon during bedside teaching. Advances in medical education and practice. 2018;9:483-494.

22.       Oser TK, Haidet P, Lewis PR, Mauger DT, Gingrich DL, Leong SL. Frequency and negative impact of medical student mistreatment based on specialty choice: a longitudinal study. Academic medicine : journal of the Association of American Medical Colleges. 2014;89(5):755-761.

23.       Castillo-Angeles M, Watkins AA, Acosta D, et al. Mistreatment and the learning environment for medical students on general surgery clerkship rotations: What do key stakeholders think? American journal of surgery. 2017;213(2):307-312.

24.       Kemp MT, Smith M, Kizy S, Englesbe M, Reddy RM. Reported Mistreatment During the Surgery Clerkship Varies by Student Career Choice. J Surg Educ. 2018;75(4):918-923.

25.       Lau JN, Mazer LM, Liebert CA, Bereknyei Merrell S, Lin DT, Harris I. A Mixed-Methods Analysis of a Novel Mistreatment Program for the Surgery Core Clerkship. Academic medicine : journal of the Association of American Medical Colleges. 2017;92(7):1028-1034.

26.       Baecher-Lind LE, Chang K, Blanco MA. The learning environment in the obstetrics and gynecology clerkship: an exploratory study of students’ perceptions before and after the clerkship. Med Educ Online. 2015;20:27273.

27.       Singh TSS, Singh A. Abusive culture in medical education: Mentors must mend their ways. Journal of anaesthesiology, clinical pharmacology. 2018;34(2):145-147.

28.       Furney SL, Orsini AN, Orsetti KE, Stern DT, Gruppen LD, Irby DM. Teaching the one-minute preceptor. A randomized controlled trial. J Gen Intern Med. 2001;16(9):620-624.

29.       Dyrbye LN, Thomas MR, Shanafelt TD. Medical student distress: causes, consequences, and proposed solutions. Mayo Clinic proceedings. 2005;80(12):1613-1622.

30.       Birman BF, Desimone L, porter AC, Garet MS. Designing professional development that works. Educational Leadership. 2000.

31.       Desimone LM, Porter AC, Garet MS, Yoon KS, Birman BF. Effects of Professional Development on Teachers’ Instruction: Results from a Three-year Longitudinal Study. Educational Evalution and Policy Analysis. 2002;24(2):81-112.

32.       Gast I, Schildkamp K, Veen JTvd. Team-Based Professional Development Interventions in Higher Education: A Systematic Review. Review of Educational Research. 2017;87(4):736-767.

33.       McLean M, Cilliers F, Van Wyk JM. Faculty development: Yesterday, today and tomorrow. Medical Teacher. 2008;30(6):555-584.

34.       Sklar DP. Moving From Faculty Development to Faculty Identity, Growth, and Empowerment. Academic Medicine. 2016;91(12):1585-1587.

35.       Hirsh D, Walters L, Poncelet AN. Better learning, better doctors, better delivery system: Possibilities from a case study of longitudinal integrated clerkships. Medical Teacher. 2012;34(7):548-554.

36.       Hirsh DA, Holmboe ES, ten Cate O. Time to Trust: Longitudinal Integrated Clerkships and Entrustable Professional Activities. Academic Medicine. 2014;89(2):201-204.

37.       Vanstone M, Grierson L. Social power facilitates and constrains motivation in the clinical learning environment. Med Educ. 2020;54(3):181-183. 38.       Harrouk C. Psychology of Space: How Interiors Impact our Behavior? 2020; https://www.archdaily.com/936027/psychology-of-space-how-interiors-impact-our-behavior. Accessed 9/17/20, 2020.

Interview Questions

**This is a semi-structured interview protocol, adapted from a guide originally developed by Drs. Williamson and Suchman.¹ Probes will be used as necessary to elicit additional pertinent information.

• Introduction: This is going to be what we call an appreciative interview. I am going to ask you questions about times when you experienced educational things working at their best here at [institution]. Many times, we try to ask questions about things that aren’t working well—the problems—so that we can fix them. In this case, we are trying to find out about the things at their best—the successes—so that we can find out what works and why, and find ways to infuse more of it into our practice.
• As we get started, I’d like to know a little bit about you. Just so you know, this information will not be associated with any of your stories or quotes, but will just be used to provide context to our findings.
o What’s your role here at [institution] and how long have you been here?
• People do their best work when they are doing things that they find personally meaningful, and when they feel that their work makes a difference. During your time at [institution], there have no doubt been high points and low points. For now, I’d invite you to think of a teaching and learning moment that meant a lot to you, when things went right, a time that brought out the best in you.
o Please tell the story of that time. (If they are very general, try to probe for more specificity.)
o Without worrying about being modest, please tell me what it was about you—your unique qualities, gifts or capacities; decisions you made; or actions you took—that contributed to this teaching/learning experience?
o What did others contribute or do?
o What aspects of the situation made this a success (for example, the place, the time of day or year, recent events)?
• Now, think of a time at [institution] when you or your values were challenged.
o Please tell me a story about that time. (If participant needs clarification about what a value is, explain that a value is “a person’s principles or standards of behavior; one’s judgment of what is important in life.”)
• We each have different qualities, gifts and skills we bring to the world and to our work. Think about the things you value about yourself, the nature of your work and the university. At work, we’re always dealing with challenges and change.
• How have your strengths and values helped you deal with challenges and change?
o Your work: When you are feeling good about your work, what do you like about the work itself?
o Yourself: Imagine you’re at your retirement party. What do you think your colleagues would say they liked most about you?
o Yourself: Now what do you think your students would say they’ve liked most about you?
o How do your personal values match those of [institution]? (for example, honesty, compassion, teamwork)?
o Where have you seen examples of these values at [institution]?
• Where do you think these reports of mistreatment are coming from?