American Association of Neurological Surgeons Joint Sponsored Activities: A longitudinal comparison of learning objectives and intent-to-change statements by meeting participants

Abstract

Background: Continuing medical education (CME) activities are required for physician board certification, licensure, and hospital privileges. CME activities are designed to specifically address professional knowledge or practice gaps. We examined statements taken from participants of their “intent-to-change” as data to determine whether the CME activity content achieved a stated learning objective.

Methods: We performed a retrospective mixed-method thematic content analysis of written and electronic records from American Association of Neurological Surgery  (AANS) sponsored CME activities. Data was analyzed using a quantitative, deductive content analysis approach. Meeting objectives were examined to determine if they resulted in specific intent-to-change statements in learners’ evaluation of the CME activity on a direct basis for one year as well as longitudinally over 6 consecutive years. Intent-to-change data that did not align with meeting objectives were further analyzed inductively using a qualitative content analysis approach to explore potential unintended learning themes.

Results: We examined a total of 85 CME activities, averaging 12–16 meetings per year over 6 years. This yielded a total of 424 meeting objectives averaging 58–83 meeting objectives each year. The objectives were compared with a total of 1950 intent-to-change statements (146–588 intent-to-change statements in a given year). Thematic patterns of recurrent intent-to-change statements that matched with meeting objectives included topics of resident education, complication avoidance, and clinical best practices and evidence. New innovations and novel surgical techniques were also common themes of both objectives and intent-to-change statements.

Intent-to-change statements were not related to any meeting objective an average of 37.3% of the time. Approximately a quarter of these unmatched statements led to subsequent CME activity new learning objectives. However, the majority of intent-to-change statements were repeated over a number of years without an obvious change in subsequent meeting learning objectives. An examination of CME learning objectives found that 15% of objectives had no intent-to-change statements associated with those objectives.

Conclusion: An examination of CME learning objectives and participant intent-to-change statements provides information for examination of both meeting planner and learner attitudes for future CME activity planning.

INTRODUCTION

Providership Council, provides continuing medical education (CME) accreditation to approximately 20 CME activities each year. This is accomplished under the guidance of the Accreditation Council for Continuing Medical Education (ACCME). The ACCME stipulates that education activities should be designed to specifically address professional knowledge or practice gaps identified before the CME activity by organizers of that activity [1]. The prevailing concerns are to focus CME activities on improving practice rather than just disseminating information [2]. Accomplishing this requires a shift in how CME activities are evaluated, including going beyond measuring learner satisfaction and change in medical knowledge to the level of physician performance and patient outcome[3].

The planning of CME activities to meet the needs of learners participating in those activities can be a difficult task [1, 4]. The AANS staff, through the Joint Providership Application, review the CME activity for practice knowledge gaps, data sources, and needs of the meeting attendees. Practice gap data sources used on this application include previous evaluation results, program committee consensus, expert opinion, survey of target audience, journal articles and medical literature review, and outcomes data. This is ultimately distilled into CME learning objectives. Intent-to-change statements are described in the literature as statements of motivation to change [5, 6], commitment to change [7-14], and readiness to change [15]. These terms seem to be used interchangeably, but for this study we have used the convention of intent-to-change and defined by what learners engaged in an educational activity are asked to list as clinical practice changes they propose to make based on what they feel they gained from the activity [3, 16]. The AANS Joint Sponsorship Chair and Committee use the intent-to-change data from evaluations taken from each CME activity as a measure of the educational value of that activity. Furthermore, these intent-to-change data are given to meeting organizers to use as a framework for future planning and feedback of previous meeting outcomes. This study is an attempt to understand the relationship between intent-to-change data from given CME activities and the learning objectives set for those meetings. We used mixed-method content analysis of meeting objective data to explore how intent-to-change data are used by CME organizers to plan educational activities and discover how meeting objectives are formulated and how they evolve over time.

METHODS

Data Source and Setting

Each year, the AANS Joint Sponsorship Council sponsors 15–20 CME activities. These CME activities were multiple day regional and subspecialty organizational meetings. Specifics about actual meeting organization was available in general terms. CME activity evaluation data for each of these meetings were examined for the years 2011–2016 (supplemental table 1). The majority of these CME activities were sequential, occurring on a yearly basis, so trends in change were evaluated over time. The available data were in a format that was examined without significant editing to allow for the most robust examination of the research questions. This study (Protocol #2019-1152) was determined to be IRB exempt status because no human subject data were utilized in this study.

The data for this study comes from the AANS Joint Sponsorship collection kept by the AANS and is available to interested meeting planners who are members of the AANS. These data are in the form of free text as a list of intent-to-change statements gathered from each meeting and organized into Excel spreadsheets for a given year. The meeting objective data are taken directly from each meeting application and/or promotional material from that meeting. CME objectives were printed in the preconference brochures and a syllabus. Both of these sources of data were taken from all of the meetings sponsored in a given year and were examined using content analysis in line with the conceptual framework of meeting objective themes that could be used to map to intent-to-change data.

Research Design

Our study design is a retrospective mixed-method content analysis of written and electronic records from specific CME activity application records and corresponding CME activity evaluations. The data was anonymous and number of participants of the meeting and number of participants that submitted intent-to-change statements was unknown. It is possible that some participant submitted multiple intent-to-change statements while others may have not submitted any statements. Meeting learning objectives were those formulated for the entire CME activity and not for individual sessions of the meeting. The data were first examined by quantitative content analysis to examine whether meeting objectives result in specific intent-to-change statements in learners’ evaluations of the CME activity. This data was examined both on a direct basis for one year as well as longitudinally over many years for the same CME activity. Yearly objective data were compared to include the effect of intent-to-change data on future meeting planning. Intent-to-change data that failed to align with meeting objectives was noted and examines further using qualitative content analysis to identify patterns in the intent-to-change data that did not align with meeting objectives. The overall scheme of the research design and plan is found in Figure 1.

Data Analysis and Statistics

Data were examined by two independent coders (BD and RLJ) using quantitative content analysis [17] to determine whether meeting objectives result in specific intent-to-change statements in a learner’s evaluation of the CME activity. First, all meeting objectives were examined independently by each reviewer, and a set of themes based on each meeting objective were formulated. Next, the coders met and, through an iterative process, decided on a common set of meeting objective themes for each year of study. Meeting intent-to-change data was examined and coded for words or phrases that related to the derived themes by each observer and the data recorded using the computer program Dedoose (www.dedoose.com, Los Angeles, CA) or by manual grouping in Excel spreadsheets.

The frequency that each rater matched a given learning objective theme to a given intent-to-change statement was calculated. Inter-rater reliability of the raters was measured by weighted Cohen Kappa. These data was interpreted where values ≤ 0 indicate no agreement, 0.01–0.20 is interpreted as none to slight, whereas 0.21–0.40 is judged as fair, and 0.41– 0.60 is moderate, while 0.61–0.80 as rated substantial, and 0.81–1.00 as rated as almost perfect agreement [18]. The two raters’ frequency data were then averaged in the majority of cases. In the rare case that a large discrepancy was found for a given intent-to-change statement, the raters together examined the actual data set made by each observer and determined a consensus. The data were then analyzed for number and percentage of intent-to-change data that mapped to a specific objective theme for each year. Descriptive statistics were used to characterize frequency counts. Furthermore, the frequency with which a given objective theme was mentioned in intent-to-change statements was recorded. Comparisons between groups of aggregated data (usually comparing year to year data trends or match and unmatched data) were made by Chi squared test and Fisher’s exact test. Continuous variables are compared using the unpaired Student t-test with a two-tailed p value. Continuous variables are reported as the mean ± standard deviation unless otherwise specified. The alpha for significance is set to 0.05. All statistical analyses are performed using IBM SPSS software version 26 (IBM Corp., Armonk, NY).

RESULTS

Overall Study Descriptive Outcomes

Overall CME activity descriptive data are found in table 1. During the years 2011–2016, a total of 85 CME activities (yearly range = 12–16, yearly mean = 14, SD 1.3) were sponsored by the AANS Joint Sponsorship Committee (Supplemental table 1). Meeting objective data was taken from CME activity application forms. The total number of meeting objectives per year ranged from 58 to 83, with a mean of 71 (SD 8.6) objectives met each year. Learning objectives for a given CME activity ranged from 3-8 objectives per meeting. There were overlapping meeting objectives from separate CME activities that were consolidated into overall objective themes for a given year.

The intent-to-change statements were taken directly from the meeting evaluation data. The total number of intent-to-change statements submitted by participants ranged from 146 to 588 during this time period, with a mean of 325 (SD 146) in any given year. Intent-to-change statements outnumber meeting objectives by a ratio of 4.4 (range 2.5 to 7.1, SD 2.5) in any given year.

A measure of inter-rater reliability of the frequency of intent-to-change statements matched to a given learning objective theme between the two observers is presented in supplemental table 2. Weighted Cohen Kappa ranged from 0.9160 to 0.9735 when measured for a specific year and 0.9777 (standard error 0.0056, 95% CI 0,9665-0,.9889) when measured overall.

Quantitative Content Analysis of Intent-to-change Statements

CME activity objectives were coded as themes and mapped to participants’ intent-to-change statements from the years 2011–2016 For any given year, there were a number of overlapping objective themes among the different AANS-sponsored meetings, making examination as aggregate data the most effective way to assess overall alignment of meeting planner objectives and participant statements of intention to change. For example, for the year 2016 there were 45 unique objective themes that were identified from 16 AANS-sponsored CME activities. This included 25 major themes with 20 associated subthemes. There were 588 intent-to-change statements from participants available for evaluation during this year. In particular, 175 (29.8%) participant statements did not correspond with any of the 45 themes and subthemes. These will be examined in more detail a follow-up publication. Table 2 condenses the data by objective theme over time. This includes only objectives that matched to intent-to-change themes for more than one year during the study. The most frequent category is “no matched objective” meaning that a meeting attendee intent-to-change statement did not map to any of the stated meeting objectives. These statements are the subject of future work and only a brief summary of this work is included here.

Table 3 summarizes the meeting objective themes with the 5 highest number of intent-to-change statements associated with that objective. The most commonly repeated themes (in bold) include resident education, best practices and clinical evidence, socioeconomics, innovation and emerging technology, and complication avoidance. Not surprisingly, a common thread through all meeting objective themes was the dissemination of new therapeutic options contained in themes such as recent innovations, novel surgical approaches and techniques, recent progress, and advancements. Another common thread included methods of determining whether our current treatments are appropriate and adequate, expressed in themes such as guidelines and databases, practice change and controversy, surgical treatment and outcomes, and current treatment options.

Qualitative Content Analysis of Intent-to-change Statements Not Related to Meeting Objectives

Table 4 contains a summary of intent-to-change statements that did not map to any meeting objective for the years 2011–2016. In total, 728 of 1950 intent-to-change statements did not correspond with any CME meeting objective, a mean of 37.3% (range 29.8–48.6%, SD 6.8%). This represented the largest category of intent-to-change statements for all years studied. We next focused our examination on these unmatched intent-to-change statements made by attendees at these CME activities to determine the nature of the data and look for why they did not map to explicit objectives and whether there might be implicit learning to account for these statements. Furthermore, we wanted to know if these unmatched intent-to-change statement drive subsequent meeting planning in the form of meeting objectives. To accomplish this, we examined all unmatched intent-to-change statements using qualitative content analysis to discover themes that emerge from these intent-to-change statements on a yearly basis.

            Table 5 summarizes this data over the course of the study period. One notable recurrent theme is that of referrals. This theme came up in every year that we studied, but no meeting objectives were ever created by meeting planners to address this perceived need. The other intent-to-change statements not shown in this table are not sustained over multiple years, suggesting some fulfillment at least on an intermittent basis. In fact, 35/45 of the intent-to-change statement themes associated with no stated meeting objective occurred only in a single year suggesting resolution in subsequent years.

We next examined how these statements may have led to meeting objective changes by mapping intent-to-change statements from previous years to subsequent meeting objectives. Some of the unmatched intent-to-change statements appear to have led to a new meeting objective in the following year that had not been seen in the year of the original intent-to-change statement (table 6). For instance, in 2011 the unmatched intent-to-change statements of comprehensive and multidisciplinary care and minimally invasive surgery were found as meeting objectives in 2012. The same is true for stem cell and cellular transplantation in 2012/2013. In 2013, unmatched themes of tumor tissue biomarkers, minimally invasive surgery, neurocritical care, and outcomes and guidelines are possibly related to the same meeting objectives found in 2014. In the years 2014/2015, the same pattern was found for neurocritical care, and outcomes and guidelines are possibly related to the same meeting objectives found in 2014. In the years 2014/2015, the same pattern was found for concussion management, Chiari malformation management, and surgery for intraparenchymal hematoma. No such pattern was found in 2015/2016. Although these seeming relationships exist, table 7 shows that this is not the most common outcome of unmatched intent-to-change statement, as on mean only 22.6% (range 0–41.7%, SD 15.4%) of unmatched intent-to-change statements led to new meeting objectives.

DISCUSSION

Importance and Use of Data from This Study

The ACCME defines “joint providership as the provision of a CME activity by one accredited and one nonaccredited organization.” (https://www.accme.org).  In this study, the AANS is the accredited organization and the multiple meetings we have examined are cosponsored by nonaccredited organizations. The accredited provider is responsible for the conduct of the nonaccredited organization’s CME activity. Thus, the AANS has a special interest in the quality of the CME activities sponsored under its cooperation. AANS CME Mission Statement “ aims to achieve excellence in continuing medical education (CME) through educational activities built on evidence-based medicine and adult learning principles. The AANS CME program provides activities to meet the participants’ identified education needs and to support their life-long learning towards a goal of improving neurosurgeon’s competency skills with a measurable result” (https://www.aans.org/en/Education/CME-Accreditation)

The reaccreditation process required by the ACCME includes that the AANS verify that CME activities sponsored by the AANS meet all ACCME requirements. The AANS staff, AANS Joint Sponsorship Chair and Committee use the intent-to-change data from each CME activity as a measure of the educational value of that activity. Furthermore, these intent-to-change data are given to meeting organizers to use as a framework for future planning and feedback of previous meeting outcomes. It is important to understand the relationship between intent-to-change data from given CME activities and the learning objectives set for those meetings. Furthermore, one can use this data to explore how intent-to-change data are used by CME organizers to plan educational activities and discover how meeting objectives are formulated and how they evolve over time. In addition, by using more traditional inductive qualitative techniques we were able to show that many intent-to-change statements made by CME attendees did not map to any predetermined meeting objectives. As will be discussed below there are a number of ways to interpret this data but consider that at least one possibility is this represents unseen, unplanned learning that takes place at various AANS-sponsored CME activities. A close examination of this data has the potential to reveal themes of “hidden curriculum” or unmet needs that might represent “practice gaps” that meeting planners did not know about or did not think would be of interest to meeting attendees. Examination of this type of data over multiple years, as well as examination of whether practice gaps are changed over time, can be used to improve CME activities to better meet the needs of the participants of these events.

 CME for Improving Neurosurgical Practice

Continuing medical education (CME) is “defined as any activity that serves to maintain, develop, or increase the knowledge, skills, and professional performance and relationships that a physician uses to provide services for patients, the public, or the profession” [2]. CME appears to be effective in contributing to the “acquisition and retention of medical professionals’ knowledge, attitudes, skills, behaviors, and clinical outcomes” [19]. CME activities are required for physician board certification, state licensure, maintenance of certification, and hospital privileges. The most common form of CME activity used in neurosurgery is presented in the form of live meeting events. These types of CME opportunities are generally found to be effective in changing physician performance [4, 20].

It is important to understand the relationship between intent-to-change data from given CME activities and the learning objectives set for those meetings for several reasons. Learning objectives that align with participant’s intent for practice change should be included in future CME activities while those that align with few or no intent-to-change statements might be discarded as something CME attendees are not interested in pursuing. Alternatively, intent-to-change statements that have no relationship to any of the meeting objectives are especially interesting and deserve some exploration. There are several explanations of why this might occur. It is possible that there is an actual discrepancy between the stated meeting objectives and what is the actual content covered or what learning experience evoked. It could be that since there are a limited number of objectives that can be stated for a given CME activity these are not comprehensive of what the meeting planners actually hope to teach during this activity. The level of specificity of the meeting objectives might also account for intent-to-change discrepancies that are broader in nature, or the opposite might occur with broad objectives not seeming to be congruent with specific intent-to-change statements. Finally, and probably most interesting, is the case in which the learning is simply outside of the explicit (stated objectives) curriculum of the planned activity. It is well recognized that quite often there is implicit learning that takes place that is termed the “informal curriculum” or “hidden curriculum” [21] These intent-to-change statements that don’t seem to align with stated learning objectives might represent the unseen, unplanned learning that takes place at various AANS-sponsored CME activities. A closer examination of this data has the potential to reveal unmet needs for future meetings. A careful examination of these data could reveal “practice gaps” that meeting planners did not know about or did not think would be of interest to meeting attendees. Further, examination of this type of data over multiple years, as well as examination of whether practice gaps are changed over time, can be used to improve CME activities to better meet the needs of the participants of these events. Finding methods to drive CME activities to correspond with learner needs is an important unmet need for the Joint Providership Council of the AANS in particular and CME meeting planners across all disciplines in general.

Intent-to-change Statements as a Measure of Learning

This study is based on the concept of mapping CME activity evaluation data in the form of intent-to-change statements directly to CME activity objectives [22, 23]. A previous survey of physicians in the United States demonstrated that they feel confident in identifying their own learning needs [24]. Intent or motivation to change has been thoroughly studied at the level of the individual learner [5, 6, 25] There is strong evidence to suggest that an individual CME participant’s motivation to change leads to knowledge acquisition [5, 25], a process mediated by promoting self-efficacy—the belief that an individual has in his or her own capacity the ability to achieve a given goal. Williams et al. [5, 6] based these observations on social cognitive understanding of change behavior, where the CME activity leads to the motivation and confidence to put the new knowledge into the participant’s medical practice.

Intention-to-change data can be used to assess alignment of intended changes in physician behavior with program objectives, confirm and strengthen intended practice change and explore unanticipated learning outcomes [7]. One confounding problem with this type of analysis is that it has been demonstrated that it is possible to have “no significant difference in intention between a health care professional who later reported a behavior change and those who reported no change” [26]. Others have demonstrated just the opposite—that learners that indicated an intent-to-change immediately after a given lecture were more like to actually use that information in a change of their practice [9], although this does not always take place with a single CME activity [27].

Overton et al. used qualitative methods to “find that there can be a range of meanings underlying intention-to-change statement” and in fact, for some participants “commitment is too strong a word to describe their intention” [28]. Although many CME participants make changes to their practices, this study “highlights that merely asking learners to specify the changes that they intend to make does not necessarily imply that learners feel a sense of commitment towards the intended changes.” When there is a gap between knowledge acquisition and behavioral change, it can be attributed to a number of factors, but there are two factors that are known to drive behavioral change—a sense of urgency and a level of certainty that the behavior change is important [29]. Others have found that physician behavior after CME activities is expected to change if the practice alteration is congruent with values and sense of what the physician’s feels is important [8].

Intent-to-change statements are a means for promoting reflection on current practice and encouraging participants to identify and commit to specific planned practice changes [10, 25, 30, 31]. They can serve as a marker or proxy for actual practice change since physicians who make intent-to-change statements are more likely to follow through with making changes than those who do not  [7, 9, 11, 12, 25, 30, 32-36].

Quantitative Content Analysis of Intent-to-Change Statements

Our data over many years suggest that the majority of intent-to-change statements can be directly tied to stated meeting objectives. From a broad overview, we identified resident education, best treatment practices and treatment options, socioeconomic issues, databases and registries, complication avoidance, patient outcomes, innovation, and new surgical techniques and approaches as common threads from year to year. This is hardly surprising given the audience of a relatively like-minded and focused practice group of neurosurgical learners as the majority audience at these events. There are variations from year to year, but many of the differences are in terminology and not necessary in the intent of the meeting planners or the actual meaning of the individual participants’ statements.

As described previously, intent-to-change statements are acknowledged as a valuable evaluation tool for educational program evaluation. Some have opined that these statements are based on Locke’s goal-setting theory, which holds that behavior is affected by individual motivation and draws on the principle that adults learn what is relevant to their needs [12, 37]. The majority of the intent-to-change statement examined in this study support this notion. These statements all paint a picture of learners committed to practice change and improvement. The heavy reliance of the field of neurosurgery on new technology is reflected in these statements and has be found in the work of others examining CME meeting outcomes from meetings involving rapidly evolving technologies [38]. Our study data are not sufficient to demonstrate actual practice change by participants of the CME activities we have examined here. Others have criticized work similar to ours as incomplete without verification of data on the actual clinical practice change and impact on physician behavior. Our data, and most similar study data, involve information gathered from participants only at the immediate end of a CME activity with no follow up at a later date [3, 39]. Some have questioned the validity of this approach without adding follow-up at a later date to confirm that the commitment to change has been carried out [25, 40]. These authors have argued that the self-reported nature of the statements was the major limitation of this method [3]. Other have demonstrated that, in fact, self-reported intent-to-change statements can be a valid measure of changes in clinical practice behavior [32]. Spending time to complete a post-meeting questionnaire and write intent-to-change statements may, in and of itself, reflect a seriousness about the intent-to-change that may, in turn, predict action [8].

Meeting Objective Themes with No Correlated Intent-to-change Statements

One of the most interesting aspects of this study is the 30–48% of yearly intent-to-change statements that did not map to any meeting objectives. This was the largest single theme of intent-to-change statements for each of the years; however, admittedly it was simply a compilation of unmatched statements. This will be the focus of a subsequent study that we will publish to examine in detail. There are a number of reasons for the intent-to-change statements to not align with meeting objects. These include a difference in opinion between meeting planners and the participants’ importance of a given subject. It is also possible that none of the speakers chose to present information on these objective topics or that some of the intent-to-change statements did reflect actually teaching that took place during the meeting but was not stated as an actual meeting objective.  It is possible that our interpretation of the meeting objectives and coding into themes did not reflect the implicit intent of the meeting planners but the meeting participant’s intent-to-change statements were written with this in mind. That said it does appear that unmatched intent-to-change statements may represent unintended consequences of the CME activity.

Limitations of This Study

There are certainly limitations in our study. One is that this work relied on two coders to perform the content analysis of meeting objective themes and the actual thematic coding of the intent-to-change statements. We attempted to eliminate unintentional bias and errors by having both observers independently start this process. The initial examination of content analysis data involved independent generation of the deductive codes. We later met to review and edit codes through a collaborative, iterative process until final objective themes/codes were generated. Because the second observer was an undergraduate research assistant, it is possible that the senior author may have introduced more unintended bias on themes as a result of greater familiarity with the process of AANS-associated CME and with neurosurgical topics. When examining intent-to-change data that did not map to specific meeting objectives, we took a more inductive coding approach, looking for unknown themes of learning that took place in the CME activities examined in this study. This approach is subject to a similar source of unintended bias. It is possible that even more unintentional bias could be eliminated with participation of additional independent coders.

Another obvious limitation of this study in general is the indirect nature of the data. The data was collected prospectively but is limited by a retrospective analysis. Furthermore, the data was collected for meeting evaluation but not necessarily for the direct comparison to meeting objectives as we have done in this study. Since this was not the intended use of the intent-to-change data there are limitations of the “fit” to the meeting learning objectives. The opposite is true as well, the meeting objectives were not necessarily designed by the meeting planners for later comparison to learner intent-to-change statements. For the sake of simplicity, the data are aggregated from multiple CME activities for analysis. Attempts at a more granular examination of the relationship of meeting objectives and learner perceptions of take-home messages from the CME activity proved difficult because of the reduced number of both objectives and intent-to-change statements. Even more problematic were attempts at examining data from a particular meeting using classification by particular neurosurgical subspecialty. The data were available with this level of detail from the Joint Sponsorship Council but did not prove to be adequate for meaningful examination.

Since meeting evaluations and specific intent-to-change data is anonymous, it is not possible to know the number of participants for any given CME activity examined in this study. Furthermore, it is possible that an individual participant may have submitted multiple intent-to-change statements while other learners may have not participated in the evaluation process at all. A participant with a particular agenda or perception of the CME activity might skew the evaluation data in a certain direction. This can certainly add bias to the interpretation of the intent-to-change statements and not reflect the true overall outcome of a given CME activity.

While we felt that qualitative and quantitative content analysis methods were the best approach for these data, it is entirely possible that that some intent-to-change statements or learning objectives were sufficiently vague that our methods were not sensitive enough to categorize the true meaning of the participant and subsequently failed to capture the relationship between a given objective and intent-to-change statement. It must be acknowledged that there are certainly differences in implicit and explicit meaning for many learning objectives, and intent-to-change statements as well. This can complicate the process of alignment of the data in a study like that presented here. In a similar manner, when a meeting objective was overly broad or narrowly specific in theme, we may have not properly associated a given intent-to-change statement to that objective even though the participants intent was a fulfillment of that objective. This is most certainly possible when intent-to-change statements or learning objectives are more oriented to declarative knowledge compared to procedural knowledge. We recognize that is not possible for meeting planners to state every desired learning goal in their stated meeting objectives. It is likely that some of these unwritten objectives might be found in the intent-to-change statements that we categorized as unmatched or unintended learning and, in fact, represent topics very much included in the meeting planners hoped for learning outcome objectives. Finally, this work is an indirect measure of outcomes of CME activities and does not measure whether the intent-to-change statements, either matched or unmatched to meeting learning objectives, indeed led to physician practice change.

CONCLUSIONS

It appears that intent-to-change data can be useful to examine the relationship between a CME activity and whether it achieved a stated learning objective. The longitudinal examination of objectives and intent-to-change data over time is useful in understanding the efficacy of CME for closing identified knowledge gaps and for determining unmet needs for future CME planning. Intent-to-change statements can be mapped to meeting objectives in a majority of CME activities studied. Theme patterns of recurrent intent-to-change statements that matched with meeting objectives for neurosurgical CME activities are focused on resident education, reduction of patient complications, evidence-based practice change, and innovation of surgical procedures and technical advances. A little over a third of intent-to-change statements were not related to any meeting objective. Approximately a quarter of these unmatched statements led to subsequent CME activity new learning objectives. However, the majority of intent-to-change statements were repeated over a number of years without resolution. A small number of CME learning objectives had no associated intent-to-change statements. When these objectives went unmatched for multiple years, we found that the themes of these objectives tended to be somewhat general/declarative knowledge in topic, whereas objectives on specific/procedural topics were more likely to be unmatched for only a single year. A number of CME learning objectives are repeated for a number of subsequent years without change. This however, was not found to correlate with unmatched status to intent-to-change statements. An examination of CME learning objectives and participant intent-to-change statements is a rich source of information for examination of both meeting planner and learner attitudes and motivation for acquisition of medical knowledge.

Acknowledgments

            We would also like to thank Kristin Kraus for her editorial assistance throughout the preparation and completion of this text.  I would also like to thank Samantha Luebbering and Lorelei Garcia from the American Association of Neurological Surgeons for help collecting and compiling this data.

Disclosures

Dr Jensen served on the AANS Joint sponsorship committee as a member and/or Chair during the years data was collected for this study.

References

1.         Wittich, C.M., et al., Perspective: a practical approach to defining professional practice gaps for continuing medical education. Acad Med, 2012. 87(5): p. 582-5.

2.         Davis, N., D. Davis, and R. Bloch, Continuing medical education: AMEE Education Guide No 35. Med Teach, 2008. 30(7): p. 652-66.

3.         Shershneva, M.B., et al., Commitment to practice change: an evaluator’s perspective. Eval Health Prof, 2010. 33(3): p. 256-75.

4.         Davis, D., et al., Continuing medical education effect on practice performance: effectiveness of continuing medical education: American College of Chest Physicians Evidence-Based Educational Guidelines. Chest, 2009. 135(3 Suppl): p. 42S-48S.

5.         Williams, B.W., H.A. Kessler, and M.V. Williams, Relationship among knowledge acquisition, motivation to change, and self-efficacy in CME participants. J Contin Educ Health Prof, 2015. 35 Suppl 1: p. S13-21.

6.         Williams, B.W., H.A. Kessler, and M.V. Williams, Relationship among practice change, motivation, and self-efficacy. J Contin Educ Health Prof, 2014. 34 Suppl 1: p. S5-10.

7.         White, M.I., S. Grzybowski, and M. Broudo, Commitment to change instrument enhances program planning, implementation, and evaluation. J Contin Educ Health Prof, 2004. 24(3): p. 153-62.

8.         Mazmanian, P.E., et al., Effects of a signature on rates of change: a randomized controlled trial involving continuing education and the commitment-to-change model. Acad Med, 2001. 76(6): p. 642-6.

9.         Mazmanian, P.E., et al., Information about barriers to planned change: a randomized controlled trial involving continuing medical education lectures and commitment to change. Acad Med, 1998. 73(8): p. 882-6.

10.       Armson, H., et al., Is the Cognitive Complexity of Commitment-to-Change Statements Associated With Change in Clinical Practice? An Application of Bloom’s Taxonomy. J Contin Educ Health Prof, 2015. 35(3): p. 166-75.

11.       Domino, F.J., et al., The impact on medical practice of commitments to change following CME lectures: a randomized controlled trial. Med Teach, 2011. 33(9): p. e495-500.

12.       Jones, D.L., Viability of the commitment-for-change evaluation strategy in continuing medical education. Acad Med, 1990. 65(9 Suppl): p. S37-8.

13.       Lockyer, J.M., et al., Commitment to change statements: a way of understanding how participants use information and skills taught in an educational session. J Contin Educ Health Prof, 2001. 21(2): p. 82-9.

14.       Pereles, L., et al., Effectiveness of commitment contracts in facilitating change in continuing medical education intervention. Journal of Continuing Education in the Health Professions, 1997. 17(1): p. 27-31.

15.       Shirazi, M., et al., Applying a modified Prochaska’s model of readiness to change for general practitioners on depressive disorders in CME programmes: validation of tool. J Eval Clin Pract, 2007. 13(2): p. 298-302.

16.       Purkis, I.E., Commitment for changes: an instrument for evaluating CME courses. J Med Educ, 1982. 57(1): p. 61-3.

17.       Morgan, D.L., Qualitative content analysis: a guide to paths not taken. Qual Health Res, 1993. 3(1): p. 112-21.

18.       McHugh, M.L., Interrater reliability: the kappa statistic. Biochem Med (Zagreb), 2012. 22(3): p. 276-82.

19.       Marinopoulos, S.S., et al., Effectiveness of continuing medical education. Evid Rep Technol Assess (Full Rep), 2007(149): p. 1-69.

20.       Forsetlund, L., et al., Continuing education meetings and workshops: effects on professional practice and health care outcomes. Cochrane Database Syst Rev, 2009(2): p. CD003030.

21.       Balmer, D.F., et al., Learning across the explicit, implicit, and extra-curricula: an exploratory study of the relative proportions of residents’ perceived learning in clinical areas at three pediatric residency programs. Acad Med, 2015. 90(11): p. 1547-52.

22.       Dolcourt, J.L. and G. Zuckerman, Unanticipated learning outcomes associated with commitment to change in continuing medical education. J Contin Educ Health Prof, 2003. 23(3): p. 173-81.

23.       Lockyer, J.M., et al., Assessing outcomes through congruence of course objectives and reflective work. J Contin Educ Health Prof, 2005. 25(2): p. 76-86.

24.       Cook, D.A., et al., Professional Development Perceptions and Practices Among U.S. Physicians: A Cross-Specialty National Survey. Acad Med, 2017. 92(9): p. 1335-1345.

25.       Wakefield, J.G., Commitment to change: exploring its role in changing physician behavior through continuing education. J Contin Educ Health Prof, 2004. 24(4): p. 197-204.

26.       Legare, F., et al., Responsiveness of a simple tool for assessing change in behavioral intention after continuing professional development activities. PLoS One, 2017. 12(5): p. e0176678.

27.       Parochka, J. and K. Paprockas, A continuing medical education lecture and workshop, physician behavior, and barriers to change. J Contin Educ Health Prof, 2001. 21(2): p. 110-6.

28.       Overton, G.K., et al., Practice-based small group learning: how health professionals view their intention to change and the process of implementing change in practice. Med Teach, 2009. 31(11): p. e514-20.

29.       Kennedy, T., et al., Exploring the gap between knowledge and behavior: a qualitative study of clinician action following an educational intervention. Acad Med, 2004. 79(5): p. 386-93.

30.       Wakefield, J., et al., Commitment to change statements can predict actual change in practice. J Contin Educ Health Prof, 2003. 23(2): p. 81-93.

31.       Lowe, M., et al., The role of reflection in implementing learning from continuing education into practice. J Contin Educ Health Prof, 2007. 27(3): p. 143-8.

32.       Curry, L. and I.E. Purkis, Validity of self-reports of behavior changes by participants after a CME course. J Med Educ, 1986. 61(7): p. 579-84.

33.       Dolcourt, J.L., Commitment to change: A strategy for promoting educational effectiveness. Journal of Continuing Education in the Health Professions, 2000. 20(3): p. 156-163.

34.       Purkis, I.E., Continuing medical education: learning preferences of anaesthetists. Can Anaesth Soc J, 1982. 29(5): p. 421-3.

35.       Parker, F.W.I. and P.E. Mazmanian, Commitments, learning contracts, and seminars in hospital-based CME: Change in knowledge and behavior. Journal of Continuing Education in the Health Professions, 1992. 12(1): p. 49-63.

36.       Crandall, S.J.S., The role of continuing medical education in changing and learning. Journal of Continuing Education in the Health Professions, 1990. 10(4): p. 339-348.

37.       Locke, E.A. and G.P. Latham, Building a practically useful theory of goal setting and task motivation. A 35-year odyssey. Am Psychol, 2002. 57(9): p. 705-17.

38.       Dundar, Y., et al., Comparison of conference abstracts and presentations with full-text articles in the health technology assessments of rapidly evolving technologies. Health Technol Assess, 2006. 10(5): p. iii-iv, ix-145.

39.       Neill, R.A., M.A. Bowman, and J.P. Wilson, Journal article content as a predictor of commitment to change among continuing medical education respondents. J Contin Educ Health Prof, 2001. 21(1): p. 40-5.

40.       Mazmanian, P.E. and P.M. Mazmanian, Commitment to change: Theoretical foundations, methods, and outcomes. Journal of Continuing Education in the Health Professions, 1999. 19(4): p. 200-207.



Return to Table of Contents:

American Association of Neurological Surgeons Joint Sponsored Activities: A longitudinal comparison of learning objectives and intent-to-change statements by meeting participants by Blake Dunson, Yoon Soo Park, PhD, Boyd F. Richards, PhD, Laura Hirshfield, PhD & Randy L. Jensen, MD, PhD, MHPE

Boyd F. Richards, PhD

Boyd F. Richards is a Professor (Lecturer), Department of Pediatrics, University of Utah, Salt Lake City, UT. https://orcid.org/0000-0002-1864-7238