BC was a consultant for GE company and received research grant from Chiesi USA. BC did not receive any financial support specifically for this project. The other authors disclosed no conflict of interest. The research and REDcap database reported in this publication was supported (in part) by the National Center for Advancing Translational Sciences of the National Institutes of Health under Award Number UL1TR002538. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
Objective: To investigate the impact of an on-site pediatric and neonatal Point-of-care ultrasound (POCUS) course in long-term implementation of POCUS.
Methods: We hosted two pediatric and neonatal critical care POCUS courses in 2018 and 2019 using the Society of Critical Care Medicine curriculum (Critical Care Ultrasound: Pediatric and Neonatal), with local experts and infrastructure. We administered evaluation surveys based on a 5-point Likert scale before and after the course to assess the participants’ reactions, learning, and clinical behaviors. The final analysis incorporated Kirkpatrick’s evaluation model and descriptive statistics to compare confidence rankings and scanning behavior.
Results: A total of 32 on-site [JC1] clinicians from neonatal and pediatric critical care units attended the courses with a survey response rate > 72%. Respondents’ median satisfaction score was 4.0 (IQR 4.0-5.0). The median confidence rankings in their POCUS skills increased from 1.0 (IQR 1.0-2.0) pre-course to 3.0 (IQR 2.8-4.0) at 12 months after the course (p<0.0001). The proportion of respondents who reported an increased trend of performing > 4 scans in the prior month (12.5% vs. 30.4%, p=0.17). We discovered a decreased in institutional barriers, especially concerns over interdisciplinary conflicts.
Conclusions: An on-site pediatric and neonatal POCUS course utilizing local infrastructure and a reputable POCUS course effectively promoted POCUS implementation and addressed institutional barriers. Instead of having learners to seek off-site or online training, structuring an on-site course with multi-disciplinary local faculty in children’s hospitals that lack a robust POCUS program may be a feasible approach.
Point-of-care ultrasound, Kirkpatrick’s principle, adult learning, pediatric critical care, neonatal critical care
Neonatal and pediatric critical care point-of-care ultrasound (POCUS) training is in high demand. Recent national U.S. surveys showed that 83-90% of respondents thought that POCUS training should be a part of critical care fellowship education in Pediatrics1-3. However, only 67-90% of Pediatric Intensive Care, and 38% of Neonatal Medicine fellowship programs provide POCUS training1-3. Pediatric emergency medicine is the only pediatric subspecialty with established POCUS guidelines with professional endorsement4. None of the other pediatric subspecialties have a structured curriculum-based approach to POCUS training1-3.
However, evidence-based clinical implementation of POCUS is sparse. Furthermore, structured training programs for pediatric practicing clinicians (including post-graduate physicians, nurse practitioners, and physician assistants) are rare2. There is scant research that evaluates the best method to train practicing clinicians without prior POCUS experience and limited follow-up data on the impact of training courses on clinical implementation5.
Lack of a mature pediatric or neonatal critical care ultrasound program with limited skilled POCUS faculty remains a significant barrier to POCUS training for many institutions6. As a result, practicing clinicians are often encouraged to attend online or off-site courses at their discretion. After completing an off-site POCUS course, many clinicians report that integrating POCUS into their daily practice is challenging7, 8. Integration of POCUS into clinical practice varies widely across Pediatric Intensive Care Units (PICU), and only one-third Neonatal Intensive Care Unit (NICU) clinicians use POCUS1, 2.
To address the high demand for pediatric and neonatal critical care POCUS training by fellows and practicing clinicians in our institution, we implemented a nationally recognized and reputable POCUS course and curriculum. We hypothesized that an on-site POCUS training course that utilizes existing institutional infrastructure would enhance POCUS practice adoption by lessening implementation barriers.
We hosted two annual (in June 2018 and September 2019), pediatric and neonatal critical care 2-day POCUS courses for fellows and practicing clinicians in a free-standing university-affiliated children’s hospital and performed a 12-month prospective observational cohort study following course completion. The course curriculum was adapted from the 2-day “Critical Care Ultrasound: Pediatric and Neonatal” developed by the Society of Critical Care Medicine (SCCM)Ó (Mount Prospect, Illinois, USA). The course consisted of 12-hours of didactic lectures and 8-hours of hands-on training. The hands-on training was performed on pediatric volunteers, phantoms, and simulators (SonoSimÓ Ultrasound Trainings Solution, Santa Monica, CA). An adequate number of faculty is recruited to ensure proper 1:4 faculty to student ratios. One faculty each year was a POCUS expert from the SCCM faculty. Local POCUS faculty experts from PICU, NICU, Pediatric Emergency Department (PEM), and Radiology Department taught the course. The multidisciplinary approach enhances skill generalizability in different specialties. The local experts either had prior extensive POCUS training or fellowship, or were credentialed in echocardiography or sonography. Our institution supported the POCUS course financially and administratively. It was offered to fellows, attending physicians, nurse practitioners, physician assistants, nurses, and respiratory therapists from the PICU and NICU.
We intentionally designed the local course with POCUS faculty who could serve as champions within their individual units and departments to provide on-going support to participants after course completion. We also utilized the same ultrasound machines during the course that participants would continue to use in their own clinical practice.
Survey Development and Distribution
To evaluate our course effectiveness, we designed and distributed a pre-course and post-course survey. The post-course survey was given immediately post course (post) in paper form, and then a 3- (3mo), 6- (6mo), and 12-month (12mo) follow-up survey were given electronically (Supplementary Material 1). The surveys included multiple-choice, fill in the blank, and Likert-based questions similar to other published POCUS training surveys9-11. The survey collected information on the following: participant background information, clinical practice setting, and POCUS leadership or infrastructure in their respective practice. Additionally, we asked several questions regarding the frequency in scanning, confidence in interpreting, and barriers in integrating POCUS. The post-course survey addressed participants’ perception of the course and satisfaction scores with the various instructors as well as an opportunity for the participants to provide feedback and recommendations for future courses. We captured similar longitudinal data on the questions in the 3-, 6-, and 12-month follow-up surveys. Our data analysis here focused on comparing results between the pre-course and 12-month follow-up surveys.
Three POCUS experts (MSt, OK, BC) created questions for the surveys. Three other investigators (EH, SG, MSk), reviewed the questions and ranked them for clarity and completeness. After three iterations, the panel met again and reviewed each question for intention and brevity.
After the 2018 course, participant feedback prompted additional survey refinement for the 2019 course participants (Supplementary Material 2). Questions were either shortened or rearranged in numerical order to improve response rate and clarity. The investigator team reviewed the revised survey to ensure question fidelity and integrity. The concepts between the two survey versions were the same, even though the wording varied. For example, the question to assess the participant’s confidence in overall integrated POCUS skills, the 2018 survey asked “I am confident in my ability to acquire images and interpret them with POCUS putting it all together” (1=“strongly disagree” to 5=“strongly agree”). The 2019 survey asked “Ability to acquire and interpret images to clinically integrate into a diagnosis?” (1= “not confidence at all” to 5=“very confidence”). Data variation between the two survey versions were tracked to ensure internal validity.
The surveys were administered in person prior to the course (pre), and immediately at the end of the course (post). Follow-up surveys were sent via email to participants 3-, 6-, 12 months following course completion. The participants had one month to complete the survey with up to 4 email reminders.
Our primary outcome was to evaluate the on-site course’s effectiveness by comparing the pre-course and the 12-month follow-up survey results, based on the four-level Kirkpatrick’s evaluation framework12.
Level 1: assess the participants’ “reaction” based on their satisfaction of the course content, faculty teaching, and overall experience.
Level 2: assess the “learning” based on self-reported confidence of POCUS knowledge and skills.
Level 3: assess the education effect on the change of “behavior” based on the self-reported number of scans performed in clinical practice.
Level 4: assess the “results” based on the perceived institutional barrier resolution.
Study data were collected and managed using REDcap data capture tools hosted at the University of Utah13. Descriptive statistics, Student’s t-test, Mann-Whitney, and Wilcoxon test were used as appropriate. Data analysis was performed using GraphPad Prism versions 9.0.2 for Mac (GraphPad Software, San Diego, California, USA).
Institutional Review Board
After reviewing the study application, the University of Utah Institutional Review Board (IRB) exempted this study from full review and consent (IRB #_00112848).
The two-year combined survey response rates decreased over time from the course was taken. The response rates were 100% (pre), 100% (post), 94% (3mo), 94% (6mo), and 72% (12mo) respectively.
A total of 42 participants attended the two courses. Our analysis focused on the 32 on-site attendees from the NICU (50%) and PICU (50%). We excluded 10 participants because they worked at satellite community hospitals which lacked the same POCUS champions and infrastructure. Table 1 describes the clinical roles and years of practice for the 32 included participants. The majority were physicians (84%) who had completed a pediatric residency. 31% of participants had more than 10 years of clinical experience. 88% of participants reported having prior POCUS experience and training, through national conferences, online courses, medical school, or residency programs.
Kirkpatrick’s level 1 “reaction”
The course received good median satisfaction scores of 4 (IQR 4-5) on a 5-point Likert scale (1=disagree, 5=agree) in evaluating course content, objectives, and clinical relevance. The median course content satisfaction rating from the 2018 participants (n=12) was 4 (IQR 4-5) on didactic lectures, hands-on modules, and instructors. The median course content satisfaction score from the 2019 participants (n=20) was 5 (IQR 4-5), which was higher than the previous year (p<0.047). Participants from both years felt that the course met their learning objectives ranking a score of 4 (IQR 4-5) and was relevant to their field of practice ranking a score of 4 (IQR 4-5).
Kirkpatrick’s level 2 “learning”
The respondents reported increased confidence in POCUS image acquisition and interpretation over time (Figure 1). In the question regarding their overall integrated POCUS skill, the respondents reported their confidence increased from a median score of 1 (IQR 1-2) (pre) to 3 (IQR 3-4) (12mo), p<0.0001 on a 5-point Likert scale in combining both years. Looking the two years separately, the 2018 participants had reported median confidence score increased from 0.5 (IQR 0 to 2) to 2.5 (IQR 2 to 3), p<0.0017; the 2019 participants’ median scored had increased from 2 (IQR 1 to 2) to 4 (IQR 1 to 4), p<0.0001. The scoring trend was parallel between the two years, even the 2019 survey was modified (Supplementary Material 3).
In the pre-course survey, 73% of respondents felt that their lack of confidence in obtaining and interpreting images were the top POCUS implementation barriers. At the12-month follow-up survey, only 41% of respondents considered their personal confidence in POCUS skills as barriers.
Kirkpatrick’s level 3 “behavior”
After attending the on-site course, respondents reported an increase in the number of scans performed (Figure 2). The proportion of respondents who reported that they had performed more than 4 scans in the past month increased from 12.5% pre-course to 30.4% at 12-month follow-up (p = 0.17).
Of the 28 participants who had prior POCUS experience and training, 21% (n=8) reported in the pre-course survey that they had not performed any scans in the prior 6 months. At the 12-month follow-up survey, only 1 of 22 (4.5%) respondents reported not performing any scan in the prior 6 months.
Kirkpatrick’s level 4 “results”
The survey asked participants about barriers to POCUS implementation into clinical practice. Aside from their personal POCUS skills, the top 3 institutional barriers identified in 2018 were:
- lack of experienced POCUS faculty (33%),
- lack of quality assurance program to verify image acquisition and interpretation (25%),
- concerns of interdisciplinary conflicts (25%).
None of the 2019 course respondents reported concerns of interdisciplinary conflicts in their 12-month follow-up survey. A proportion of them still perceived the lack of a formal method to confirm image interpretation (40%), quality assurance program to review saved images (33%), and experienced POCUS faculty for hands-on training (20%) as top institutional barriers. Some participants (33%) also felt there was not enough time during their clinical day to perform POCUS.
We demonstrate that an on-site pediatric and neonatal POCUS course was effective based on Kirkpatrick’s four principles of reaction, learning, behavior, and results (Figure 3). To our knowledge, we are the first to describe how importing an off-site reputable course to an on-site pediatric and neonatal POCUS model could change POCUS clinical practice behavior. Participants ranked the course favorably and reported increased confidence in their POCUS skills. Although not statistically significant, participants seemed to incorporate POCUS more frequently into their clinical practice after the course, and this practice pattern was sustained. Most importantly, perceived institutional barriers to POCUS were reduced. This on-site pediatric and neonatal POCUS model utilizing nationally recognized ultrasound content while incorporating local expertise and strengthening infrastructure is an efficient way to expand POCUS clinical practice.
Due to limited POCUS expertise, pediatric and neonatal critical care clinicians have relied on online or off-site training courses. Even with many available online or off-site courses, an adequate POCUS knowledge translation into practice remains difficult. Firstly, gaining proficiency in POCUS requires complex training in image acquisition, interpretation, and clinical integration. Competency is best achieved with hands-on training, frequent practice, and integration into clinical practice. Although online or off-site training courses can enhance POCUS knowledge and promote confidence to attendees which meeting Kirkpatrick’s level 1 and 2, they often do not result in the behavior change essential to meet Kirkpatrick’s level 3 requirement. Online and off-site courses are unable to provide adequate post course hands-on training and timely feedback. This is evident in a study by Patrawalla et al. who showed that a 3-day regional POCUS was an effective educational model14. Still, it did not report detailed data on subsequent clinical practice use14. Secondly, institutional infrastructure is essential for clinical integration. National survey had reported the 5 top institutional barriers for POCUS clinical integration, including lack of equipment/funds, lack of personnel to train physicians, lack of time to learn, liability concerns, and cardiology or radiology resistance2. Successful clinical integration of POCUS requires both attaining expert knowledge and skill and overcoming local barriers. Historical online or off-site courses are unable to navigating the local practice environment requires more than distant expertise.
Integration of newly acquired skills and knowledge into clinical practice is challenging. Adhering to the principles of adult learning may help to enact positive behavior change15. As evidenced from our pre-course survey, some of our course participants did not utilize their previous POCUS skills in their clinical practice despite prior POCUS training and experience. After attending our on-site course, participants reported a trend of increasing POCUS usage behavior. Collins et al. described education techniques for lifelong learning that we utilized in this course15. Firstly, the adult learners valued the relevancy and practicality of this course15. Our course used the same ultrasound machines that the participants would use in their units, thereby enhancing the skills learned. Secondly, our participants attended with their own colleagues, which fostered an informal and personal environment in which adults learn best15. Another consideration is adults learn best by doing15. We found our respondents had an increased scanning frequency. The survey only assessed if more than four POCUS scans were performed per month, but the small increase was heading into the right direction of life-long behavior change. We suspect that more scanning now will translate to more scanning in the future. The scanning behavior will further be fostered by the on-site faculty who provide practice reinforcement and on-going feedback after course completion.
The off-site or online courses historically are unable to address the institutional barriers of integrating POCUS into daily practice. Prior to our on-site course, we identified similar barriers from the 2018 pre-course survey to those in other critical care programs1, 2, 16. Barriers included interdisciplinary conflicts, lack of local POCUS faculty, and lack of quality assurance programs. None of these barriers are solved by attending off-site courses. A few local POCUS experts first organized the course to fulfill the POCUS educational gap. As a result of the course, the institution recognized the need to strengthen the local infrastructure. Subsequently, a multi-disciplinary POCUS consortium was formed, including leaders from PICU, NICU, PEM, cardiology, radiology, and hospital administration. Additionally, PICU, NICU, and PEM champions became the ultrasound medical directors for their divisions, providing on-going education, leadership, and quality assurance. Our on-site pediatric and neonatal POCUS course model has fostered inter-departmental collaboration; thereby promoting transparency in POCUS practice, communication and eliminating concerns over multi-disciplinary conflict. By the time the 12-month follow-up survey was sent to the 2019 course participants, the POCUS consortium had been established for 28 months and the respondents reported no interdisciplinary conflicts concerns. We suggest this internal on-site infrastructure is essential to effect change for Kirkpatrick level 3 behavior and 4 results. Strengthening the POCUS infrastructure can help maintain an individual’s POCUS proficiency, expand education program, develop quality assurance processes, develop re-credentialing standards, and sustain POCUS integration.
Our model of importing a structured POCUS training curriculum is feasible and generalizable at hospitals with similar on-site champions. This pediatric and neonatal on-site POCUS model can be available to any institution. The prepared curriculum is nationally recognized and saves faculty time creating suitable education material. Pre-existing online education modules are important and helpful, but are limited by the lack of hands-on training on live human subjects. The on-site course could recruit local volunteers as scanning subjects. To support all clinicians within the institution to attend off-site courses is costly. This on-site pediatric and neonatal POCUS model is relatively more cost-effective. Clinicians can minimize travel time, reduce work schedule disruption, and balance work-life balance, encouraging more participation. We do recognize that many skilled clinician sonographers have developed excellent educational materials and although this paper used a specific POCUS course, many respectable courses could be used.
The limitations of our study include the small sample size and selection bias. As attendance was voluntary, motivated clinicians were more likely to incorporate POCUS into clinical practice and respond to the survey. The participants self-reported scanning pattern may introduce recall bias. As we did not have formal knowledge and technical skills assessments pre-course and during the follow-up periods, participants may have overestimated or underestimated their knowledge17. We felt that behavioral changes in adult learners were more important than knowledge assessments in adapting new skills. Post-course skill assessment is on-going via quality assurance process and expert faculty feedback.
In conclusion, our on-site pediatric and neonatal POCUS course transferred knowledge, positively changed clinician’s behavior, and broke down perceived barriers to POCUS integration at our institution. This model is exportable to other hospitals and clinical environments. Pediatric and neonatal critical care POCUS programs should consider distinctive education challenges and specific institutional barriers when designing their own educational programs. Further studies are needed to evaluate the long-term impact on patient outcomes from this training model.
We thank the following faculty for POCUS teaching and instruction during the two courses: R. Mart, R. Day, S. Ryan, E. Contreras, and J. Kim. Additionally, we thank T. Harbor, the research assistants from the Division of Pediatric Emergency Medicine, for helping with IRB maintenance, and creation, maintenance and distribution of the REDCap survey. Additionally, we thank Drs M. Johnson and R. Wilson for editing the manuscripts. We thank the Society of Critical Care Medicine for allowing the use of their education materials: critical care ultrasound: pediatric and neonatal.
1. Conlon TW, Kantor DB, Su ER, et al. Diagnostic Bedside Ultrasound Program Development in Pediatric Critical Care Medicine: Results of a National Survey. Pediatr Crit Care Med. Nov 2018;19(11):e561-e568. doi:10.1097/PCC.0000000000001692
2. Nguyen J, Amirnovin R, Ramanathan R, Noori S. The state of point-of-care ultrasonography use and training in neonatal-perinatal medicine and pediatric critical care medicine fellowship programs. J Perinatol. Nov 2016;36(11):972-976. doi:10.1038/jp.2016.126
3. Mosier JM, Malo J, Stolz LA, et al. Critical care ultrasound training: a survey of US fellowship directors. J Crit Care. Aug 2014;29(4):645-9. doi:10.1016/j.jcrc.2014.03.006
4. Marin JR, Abo AM, Arroyo AC, et al. Pediatric emergency medicine point-of-care ultrasound: summary of the evidence. Crit Ultrasound J. Dec 2016;8(1):16. doi:10.1186/s13089-016-0049-5
5. Matyal R, Mitchell JD, Mahmood F, et al. Faculty-Focused Perioperative Ultrasound Training Program: A Single-Center Experience. J Cardiothorac Vasc Anesth. Apr 2019;33(4):1037-1043. doi:10.1053/j.jvca.2018.12.003
6. Ahn JS, French AJ, Thiessen ME, Kendall JL. Training peer instructors for a combined ultrasound/physical exam curriculum. Teach Learn Med. 2014;26(3):292-5. doi:10.1080/10401334.2014.910464
7. Olgers TJ, Azizi N, Bouma HR, Ter Maaten JC. Life after a point-of-care ultrasound course: setting up the right conditions! Ultrasound J. Sep 7 2020;12(1):43. doi:10.1186/s13089-020-00190-7
8. Rajamani A, Miu M, Huang S, et al. Impact of Critical Care Point-of-Care Ultrasound Short-Courses on Trainee Competence. Crit Care Med. Sep 2019;47(9):e782-e784. doi:10.1097/CCM.0000000000003867
9. Webb EM, Cotton JB, Kane K, Straus CM, Topp KS, Naeger DM. Teaching point of care ultrasound skills in medical school: keeping radiology in the driver’s seat. Acad Radiol. Jul 2014;21(7):893-901. doi:10.1016/j.acra.2014.03.001
10. Stolz LA, Amini R, Situ-LaCasse E, et al. Multimodular Ultrasound Orientation: Residents’ Confidence and Skill in Performing Point-of-care Ultrasound. Cureus. Nov 15 2018;10(11):e3597. doi:10.7759/cureus.3597
11. Jones TL, Baxter MA, Khanduja V. A quick guide to survey research. Ann R Coll Surg Engl. Jan 2013;95(1):5-7. doi:10.1308/003588413X13511609956372
12. Kirkpatrick DL. Effective supervisory training and development, Part 2: In-house approaches and techniques. Personnel. Jan 1985;62(1):52-6.
13. Harris PA, Taylor R, Minor BL, et al. The REDCap consortium: Building an international community of software platform partners. J Biomed Inform. Jul 2019;95:103208. doi:10.1016/j.jbi.2019.103208
14. Patrawalla P, Narasimhan M, Eisen L, Shiloh AL, Koenig S, Mayo P. A Regional, Cost-Effective, Collaborative Model for Critical Care Fellows’ Ultrasonography Education. J Intensive Care Med. Dec 2020;35(12):1447-1452. doi:10.1177/0885066619828951
15. Collins J. Education techniques for lifelong learning: principles of adult learning. Radiographics. Sep-Oct 2004;24(5):1483-9. doi:10.1148/rg.245045020
16. Ben Fadel N, Pulgar L, Khurshid F. Point of care ultrasound (POCUS) in Canadian neonatal intensive care units (NICUs): where are we? J Ultrasound. Jun 2019;22(2):201-206. doi:10.1007/s40477-019-00383-4 17. Kruger J, Dunning D. Unskilled and unaware of it: how difficulties in recognizing one’s own incompetence lead to inflated self-assessments. J Pers Soc Psychol. Dec 1999;77(6):1121-34. doi:10.1037//0022-35188.8.131.521
Return to Table of Contents:On-site pediatric and neonatal point-of-care ultrasound (POCUS) course led by multi-disciplinary local experts may promote sustainable clinical POCUS integration. by Belinda Chan, MD, Matthew Steimle, DO, Olive Konana, MD, Sasha Gordon, BS, RDMS, RVT, Matthew Szadkowski, MD, Brandon Kirkland, MD & Ellie Hirshberg MD