Welcome to the Clinical Center Grand Rounds, a weekly series of educational lectures for physicians and health care professionals broadcast from the Clinical Center at the National Institutes of Health in Bethesda, MD. The NIH Clinical Center is the world's largest hospital totally dedicated to investigational research and leads the global effort in training today's investigators and discovering tomorrow's cures. Learn more by visiting us online at http://clinicalcenter.nih.gov OUR SPEAKER TODAY IS DR. MARIANNE GREEN, VICE DEAN FOR EDUCATION, THE RAYMOND H. CURRY, MD PROFESSOR OF MEDICAL EDUCATION, AND PROFESSOR OF MEDICINE AT THE NORTHWESTERN UNIVERSITY FEINBERG SCHOOL OF MEDICINE IN CHICAGO. DR. GREEN CURRENTLY OVERSEES THE UNDERGRADUATE, GRADUATE AND CONTINUING MEDICAL EDUCATION PROGRAMS OF THE FEINBERG SCHOOL OF MEDICINE. SHE'S ALSO CO-DIRECTOR OF THE CENTER FOR MEDICAL EDUCATION AND DATA SCIENCE AND DIGITAL HEALTH AT FEINBERG. DR. GREEN EARNED HER MEDICAL DEGREE FROM THE UNIVERSITY OF ILLINOIS AT CHICAGO SCHOOL OF MEDICINE AND WENT ON TO COMPLETE HER INTERNSHIP AND RESIDENCY TRAINING IN INTERNAL MEDICINE AT THE BETH ISRAEL HOSPITAL IN HARVARD MEDICAL SCHOOL IN BOSTON. DR. GREEN'S INTERESTS ARE IN COMPETENCY BASED MEDICAL EDUCATION, PORTFOLIOS AND NARRATIVES FOR PROGRAMMATIC ASSESSMENT OF COMPETENCE. SHE HAS HELD NUMEROUS LEADERSHIP ROLES IN MEDICAL EDUCATION AT NORTHWESTERN, AND RECEIVED SEVERAL TEACHING AWARDS INCLUDING THE OUTSTANDING TEACHER AWARD, AND THE 2020 ALPHA OMEGA ALPHA ROBERT J. GLASSER DISTINGUISHED TEACHER AWARD FROM THE ASSOCIATION OF AMERICAN MEDICAL COLLEGES. DR. GREEN IS A MEMBER OF THE AMERICAN COLLEGE OF PHYSICIANS AND THE SOCIETY OF GENERAL INTERNAL MEDICINE. SHE SERVES AS IMMEDIATE PAST CHAIR OF THE AMERICAN BOARD OF INTERNAL MEDICINE'S BOARD OF DIRECTORS, AND IS A MEMBER OF THE ABIM FOUNDATION BOARD OF TRUSTEES. SHE PREVIOUSLY SERVED ON THE ABIM COUNCIL AS COUNCIL DIRECTOR AND AS CHAIR OF THE INTERNAL MEDICINE SPECIALTY BOARD. IN ADDITION TO HER ACADEMIC ROLES, DR. GREENERY MAINS GREEN REMAINS CLINICALLY ACTIVE IN PATIENT CARE AS A GENERAL INTERNIST AT NORTHWESTERN MEMORIAL HOSPITAL. THE TOPIC OF DR. GREEN'S PRESENTATION TODAY IS: COMPETENCY BASED MEDICAL EDUCATION: THE WHY, THE HOW, AND THE FUTURE. AND WE ARE PLEASED TO WELCOME HER TO OUR VIRTUAL PODIUM. DR. GREEN. THANK YOU VERY MUCH. I VERY MUCH LOOK FORWARD TO SPENDING THE NEXT HOUR WITH ALL OF YOU TO TALK ABOUT COMPETENCY BASED MEDICAL EDUCATION, WHICH YOU'LL HEAR ME REFER TO AS CBME, AND I HOPE THAT BY THE END OF OUR TIME TOGETHER, I WILL HAVE CONVINCED YOU THAT THIS IS NOT ONLY A GOOD THING, BUT IT IS CERTAINLY ACHIEVABLE. I ENCOURAGE YOU TO SEND YOUR QUESTIONS AND HOPE TO HAVE SOME TIME TO BE ABLE TO GO OVER THOSE. I HAVE NO DISCLOSURES. SO I WANT TO START BY JUST SHARING WHAT I HOPE TO ACHIEVE IN THE NEXT HOUR. THE FIRST IS I UNDERSTAND THERE'S A WIDE VARIETY OF EXPERTISE IN THE AUDIENCE BUT I THINK IT'S GOOD TO HAVE A SHARED MENTAL MODEL AND I WILL GO OVER THE DEFINITION OF COMPETENCY BASED MEDICAL EDUCATION SO WE'RE ALL COMING FROM THE SAME FRAMEWORK. I WILL DESCRIBE THE RATIONALE FOR CBME AND HOPEFULLY YOU'LL UNDERSTAND WHY THIS IS THE HOT TOPIC OF CONVERSATION AMONG ALL EDUCATIONAL LEADERS IN BOTH THE UME AND GME SPACE. I'LL SPEND QUITE SOME TIME TALKING ABOUT THE BUILDING BLOCKS FOR A SUCCESSFUL CBME PROGRAM, WHICH IS REALLY APPLICABLE TO BOTH UME AND GME. AND THEN I'LL END WITH DESCRIBING SOME OF THE CURRENT CBME PILOTS ACROSS THE COUNTRY, AND MAYBE AN ASPIRATIONAL VISION FOR THE FUTURE. SO THERE ARE SEVERAL DEFINITIONS OF COMPETENCY BASED MEDICAL EDUCATION. THIS HAPPENS TO BE UP WITH OF ONE OF MY FAVORITE ONES: AN APPROACH TO PREPARING PHYSICIANS FOR PRACTICE THAT IS FUNDAMENTALLY ORIENTED TO GRADUATE OUTCOME ABILITIES AND ORGANIZED AROUND COMPETENCIES DERIVED FROM AN ANALYSIS OF SOCIETAL AND PATIENT NIEDZ. NEEDS. AND THAT'S UNDERLINED FOR A REASON. LET'S DECONSTRUCT THAT FOR A MOMENT. SO BASICALLY, COMPETENCY-BASED MEDICAL EDUCATION FOCUSES ON WHAT A PHYSICIAN SHOULD BE ABLE TO DO. AND IT'S DICTATED BY THE NEEDS OF PATIENTS IN SOCIETY WHICH CAN BE EVOLVING AS COVID-19 HAS CERTAINLY GIVEN US AN EXAMPLE. IT IS DRIVEN BY -- THE BEHAVIORAL OUTCOMES THAT DEFINE THE COMPETENCIES ONE CHOOSES THAT PHYSICIANS NEED TO ACHIEVE DRIVE THE CURRICULUM, AND THE INDIVIDUAL ADVANCEMENT, RATHER THAN TIME SPENT IN CLERKSHIPS OR ROTATIONS. IT USES A STANDARD RATHER THAN COMPARING TRAINEES TO ONE ANOTHER. AND A GOOD CBME SYSTEM OR PROGRAM ACTUALLY ALLOWS FOR TIME FOR LEARNER PROGRESSION AND SELF-DIRECTED LEARNING. SO THE LEARNING IS LARGELY IN THE HANDS OF THE LEARNER WITH TEACHER AS PARTNER, RATHER THAN THE MORE TRADITIONAL TEACHER-ORIENTED FRAMEWORK. SO THE FOLLOWING SCHE MAT A I THINK IS A GOOD WAY TO LOOK AT THE DIFFERENCE BETWEEN WHAT WE'VE LARGELY BEEN DOING, WHICH IS A TRADITIONAL EDUCATIONAL MODEL AND THEN CONTRASTING THAT WITH A COMPETENCY-BASED EDUCATIONAL MODEL. SO IN A TRADITIONAL MODEL, AND I'LL USE AN EXAMPLE HERE, IF I'M A PROGRAM DIRECTOR WHO HAPPENS TO HAVE A PARTICULAR PASSION FOR, SAY, DISASTER MEDICINE, AND I DECIDE THAT I'M GOING TO REQUIRE MY RESIDENTS TO ATTEND TWO OR THREE NOON CONFERENCES WITH LECTURES AND TOPICS AROUND DISASTER MEDICINE BE, I MIGHT CREATE SOME EDUCATIONAL OBJECTIVES AND I MIGHT EVEN OFFER A QUIZ TO MAKE SURE MY RESIDENTS HAVE LEARNED WHAT I THINK THEY NEED TO KNOW, BUT IT'S DRIVEN BY WHAT I THINK THEY NEED TO KNOW. IN A COMPETENCY BASED EDUCATION MODEL, IT'S THE HEALTH SYSTEM AND THE PATIENT NEEDS THAT ACTUALLY DRIVE WHAT THEY NEED TO KNOW. SO WHEN YOU THINK ABOUT THE EXAMPLE OF DISASTER MEDICINE ACTUALLY MAYBE THAT IS QUITE SALIENT, BUT THAT DECISION IS BEING MADE BECAUSE WE JUST LIVED THROUGH OR HOPEFULLY ARE ALMOST OVER A PANDEMIC WHICH NOW DICTATES THAT MY TRAINEES NEED TO BE COMPETENT IN THE APPLICATION OF EVIDENCE AND RAPIDLY EVOLVING EVIDENCE AT THE BEDSIDE. THEY NEED TO BE COMPETENT IN POPULATION HEALTH AND EPIDEMIOLOGICAL PRINCIPLES, AND THOSE THINGS, THE FACT THAT THIS PANDEMIC OCCURRED AND ACTUALLY THEN DRIVES OUR THINKING AS EDUCATORS TO WHAT DOES THE TRAINEE NEED TO ACTUALLY KNOW SO THAT FOR THE NEXT PANDEMIC, WHICH WILL PROBABLY OCCUR IN THEIR LIFETIME, THEY WILL HAVE THE SKILLS, KNOWLEDGE AND ATTITUDES TO BE SUCCESSFUL TO MANAGE THAT. THAT, THEN, LEADS ME TO DRIVE AND CREATE THE CURRICULUM, AND THE APPROPRIATE ASSESSMENTS TO MAKE SURE THAT THEY'VE ACHIEVED WHAT I BELIEVE THEY NEED TO ACHIEVE TO PROVIDE THOSE OUTCOMES. SO THAT'S THE FUNDAMENTAL DIFFERENCE BETWEEN THE TWO SYSTEMS. THE OTHER THING TO HIGHLIGHT IS THAT IN A TRADITIONAL MODEL, IT'S VERY TIME BASED, AND I USED HERE A UME EXAMPLE, ALTHOUGH IT APPLIES TO GME AS WELL, WHEN I WAS IN MEDICAL SCHOOL, THEN I WAS TOLD, OKAY, YOU NEED TO FINISH 12 WEEKS OF AN INTERNAL MEDICINE ROTATION, YOU NEED TO TAKE A COURSE IN ANATOMY, THAT HAPPENS TO BE 10 WEEKS, THERE'S SIX WEEKS OF PEDIATRICS. YOU SEE WHAT I MEAN. AND FOR SOME MAGICAL REASON, AT THE END OF THAT TIME, I WAS DEEMED READY TO GO ON TO RESIDENCY. WELL, IN EFFECT, A COMPETENCY-BASED MEDICAL EDUCATION SYSTEM ACTUALLY DOESN'T DEPEND ON TIME. IT DEPENDS ON SOMEONE MEETING THE OUTCOMES, AND THAT MAY BE TIME-VARIABLE. DIFFERENT LEARNERS WILL HAVE A DIFFERENT SET OF SKILLS THAT THEY BRING IN TO THE TRAINING, AND THEY MAY ACHIEVE AT DIFFERENT RATES. AND WE'LL GO OVER THAT A LITTLE BIT FURTHER IN THE FUTURE DURING THIS PRESENTATION. SO LET'S TALK A LITTLE BIT MORE ABOUT THE WHY THIS HAPPENED, WHY IS THIS GAINING SO MUCH TRACTION AMONG EDUCATORS? AROUND AND AND I THINK THE CRITICAL ELEMEN T IS THE FACT THAT WE ARE NOT NECESSARILY MEETING THE BEST OUTCOMES OF HEALTHCARE IN OUR COUNTRY. AS YOU ALL KNOW, IN 2006, THE INSTITUTE FOR HEALTHCARE IMPROVEMENT ARTICULATED THE AIMS OF HEALTHCARE SYSTEM SO THAT THEY COULD MEASURE PERFORMANCE OF OUR HEALTHCARE SYSTEM. AND THAT WAS DEFINED AS THE TRIPLE AIM WITH PER CAPITA COST FOR PATIENT CARE, THE EXPERIENCE OF CARE ON THE INDIVIDUAL LEVEL, AND THEN THE HEALTH OF A POPULATION WHERE THE OUTCOMES THAT THE IHI DERIVED. AND AS YOU KNOW, THE INSTITUTE OF MEDICINE ALSO THEN FORMALLY ADDED TO THAT ON THE EXPERIENCE OF CARE WITH THE CROSSING THE HEALTHCARE CHASM AND THE SIX AIMS OF THE INDIVIDUAL EXPERIENCE FOR CARE, WHICH ARE HIGHLIGHTED OVER HERE. BECAUSE BURNOUT BECAME SUCH A PROBLEM AND HAS RECEIVED SO MUCH ATTENTION IN THE LAST FEW YEARS, THIS HAS REALLY MORPHED INTO IMPROVED CLINICIAN EXPERIENCE AND WELLNESS AS ONE OF THE FOUR AIMS. SO IF THE ULTIMATE OUTCOME, IF WE AGREE THAT THE IHI'S FRAMEWORK IS A GOOD OUT COME FOR ASSESSING THE HEALTH AND SUCCESS OF OUR HEALTH SYSTEM, HOW WELL ARE WE DOING AT THAT? AND I THINK I'M NOT GOING TO SURPRISE YOU WHEN I SAY NOT SO WELL. THE U.S. REMAINS THE MOST EXPENSIVE HEALTHCARE SYSTEM WORLDWIDE, AND COVID-19 HAS CERTAINLY LAYED BARE OR FOCUSED EVEN MORE ATTENTION ON THE HEALTH DISPARITIES WHICH REALLY PRE-EXISTED COVID BUT BECAME EVEN MORE OBVIOUS. SO SIGNIFICANT HEALTH DISPARITIES ARE THERE. THERE'S VARIABILITY IN QUALITY AND PATIENT SAFETY THAT CONTINUES DESPITE MAJOR EFFORTS IN THIS DOMAIN. AND THEN BURNOUT CLEARLY REMAINS A SIGNIFICANT PROBLEM AMONG HEALTHCARE PROFESSIONALS, CERTAINLY WORSENED IN THE LAST YEAR. HERE'S JUST A QUICK SLIDE COMPARING OUR HEALTHCARE SPENDING AS A PERCENT OF GROSS DOMESTIC PRODUCT, WHICH U.S. IS HERE IN THE BLACK LINE AND THEN THE OTHER MAJOR DEVELOPED COUNTRIES ARE HERE IN THE MULTICOLORED LINES BELOW, AND YOU CAN SEE QUITE A SUBSTANTIAL DIFFERENCE. YET YOU WOULD HOPE THAT WITH THAT EXPENDITURE, WE WOULD RANK AMONG THE HIGHEST IN OVERALL HEALTHCARE RANKING WHEN YOU LOOK AT THINGS LIKE QUALITY, ACCESS, EFFICIENCY, EQUITY AND THINGS LIKE INFANT MORTALITY, AND IN FACT, WE RANK FAIRLY LOW. AND THEN AS YOU HOPEFULLY ALSO KNOW IN THE LAST FEW YEARS, WE HAVE COME TO RECOGNIZE THAT REALLY MOST OF WHAT WE DO IN THE HOSPITAL AND CLINICAL SETTING REALLY REPRESENTS THIS ABOUT 20% OF ALL HEALTH OUTCOMES. WHEN IN EFFECT, 80% OF HEALTH OUTCOMES ARE DRIVEN BY WHAT WE CALL THE SOCIAL DETERMINANTS OF HEALTH, THE SOCIOECONOMIC FACTORS, ACCESS TO FOOD, TRANSPORTATION, OTHER HEALTH BEHAVIORS, THE PHYSICAL ENVIRONMENT OF THE PATIENT. SO WE NEED TO MAKE SURE THAT OUR TRAINEES -- I KNOW THAT WHEN I WENT TO MEDICAL SCHOOL, MY TEACHING WAS ALL IN THIS 20% DOMAIN. TO MAKE AN IMPACT ON THE HEALTH OF OUR POPULATION, WE NEED TO MAKE SURE OUR TRAINEES ARE ALSO CLEARLY COMPETENT IN THESE OTHER AREAS AND DOMAINS. AND THAT'S JUST ONE EXAMPLE. THERE ARE MANY OTHERS. SO THE ULTIMATE -- IF THE ULTIMATE GOAL OF CBME IS TO IMPROVE THE HEALTHCARE OF OUR POPULATION AND OF OUR INDIVIDUAL PATIENTS, HOW DOES THAT ACTUALLY HAPPEN WITH AN EDUCATION SYSTEM? WELL, THE MISSION SEER IS THAT HERE IS THA T WE ARE EDUCATING STUDENTS AND TRAINEES TO BE ABLE TO MAKE AN IMPACT AND HAVE THE KNOWLEDGE, SKILLS AND ATTITUDES TO IMPROVE THESE HEALTHCARE OUTCOMES. SO WE DESIGN CURRICULUM, AND I'LL REFER BACK TO THE EXAMPLE OF THE PANDEMIC AND THE KINDS OF COMPETENCIES THAT THAT MIGHT ACTUALLY ENGENDER. SO IN OTHER WORDS, APPLICATION OF EVIDENCE-BASED MEDICINE AT THE BEDSIDE, CERTAINLY AN IMPORTANT COMPETENCY. APPLICATION OF EFFECTIVE POPULATION HEALTH AND EPIDEMIOLOGICAL KNOWLEDGE. THOSE KINDS OF THINGS. SO THAT'S GOING TO DRIVE THE KIND OF CURRICULUM WE CREATE AND THE KIND OF ASSESSMENT THAT WE DEVELOP, AND WE'LL TALK MORE ABOUT THESE EPAs AND MILESTONES. ALL OF IT IS A SYSTEM THAT KIND OF, YOU KNOW, FEEDS ON EACH OTHER AND ULTIMATELY IDEALLY IMPROVES OUR OUTCOMES. WE'RE TRYING TO MAKE AN IMPACT IN EDUCATION SO THAT OUR TRAINEES CAN CHANGE THAT TRAJECTORY THAT I SHOWED YOU IN THE PREVIOUS SLIDES TO WHERE THE U.S. IS NO LONGER RANKING AT THE BOTTOM. SO HOW DO WE DO THAT? I'M GOING TO SPEND A LITTLE TIME IN TALKING ABOUT AN EFFECTIVE COMPONENT OF A CBME SYSTEM. WE'VE ALREADY MENTIONED THAT IT IS AN OUTCOMES-BASED COMPETENCY FRAMEWORK, AND WHAT'S CRITICAL IS THAT WE PROVIDE PROGRESSIVE SEQUENCING OF THOSE COMPETENCIES ACROSS THE CONTINUUM OF MEDICAL EDUCATION, SO IN OTHER WORDS, WE DON'T NEED A GRADUATING FELLOW TO HAVE ONE OF HIS OR HER COMPETENCIES THAT YOU'RE ASSESSING THEM ON TO BE ON HOW TO TAKE A HISTORY AND DO A PHYSICAL, BUT WE DO NEED THAT TO BE THE CASE FOR AN EARLY MEDICAL STUDENT. AT THE SAME TIME, WE DON'T NECESSARILY WANT A MEDICAL STUDENT TO HAVE ONE OF HIS OR HER CURRICULA INVOLVED WITH COMPLEX PROCEDURAL TASKS NECESSARILY, BUT THAT MIGHT BE SOMETHING THAT A GRADUATING SURGICAL RESIDENT OR FELLOW SHOULD BE ASSESSED ON. THE LEARNING EXPERIENCES NEED TO BE TAILORED TO THOSE COMPETENCIES, SO IF YOU WANT TO BUILD IN CURRICULUM FOR A COMPLICATED PROCEDURAL TASK, YOU MIGHT BUILD IN A SIMULATION TIME INTO THE TRAINING CURRICULUM PRIOR TO PUTTING THE LEARNER IN A SITUATION WHERE THEY'RE DOING THAT PROCEDURE ON AN ACTUAL PATIENT, AS ONE EXAMPLE. THE TEACHING SHOULD BE TAILORED TO THE COMPETENCIES, AND THEN VERY IMPORTANTLY, THERE NEEDS TO BE A PROGRAMMATIC ASSESSMENT SYSTEM. AND I SAY THAT WORD ASSESSMENT SYSTEM VERY DELIBERATELY, AND WE'LL SPEND SOME TIME TALKING ABOUT THE GENERATION OF THAT KIND OF EFFECTIVE ASSESSMENT SYSTEM. SO AN EFFECTIVE SYSTEM IN CBME, WE'VE ALREADY MENTIONED, HAS THE DEFINED COMPETENCIES, EPAs, AND I'LL EXPLAIN WHAT THOSE ARE IN A MOMENT OR ENTRUSTABLE ACTIVITIES, OUTCOMES, AND THEN ESTABLISHED BENCHMARKS FOR THE ACHIEVEMENT OF THOSE COMPETENCIES OR OUTCOMES. OBVIOUSLY AS A TRAINEE MOVES THROUGH THEIR TRAINING, THE EXPECTATIONS FOR WHERE THEY SHOULD BE AND THE ABILITY TO EFFECTIVELY DERIVE A DIFFERENTIAL DIAGNOSIS MAY BE FOR A SIMPLE MORE STRAIGHTFORWARD PATIENT PRESENTATION EARLY IN THE TRAINING AND A MORE COMPLEX PATIENT LATER IN THE TRAINING. SO A LITTLE BIT MORE ON THESE DEFINITIONS OF THESE OUTCOMES. SO THE COMPETENCIES DEFINE THE CORE ABILITIES AND ATTRIBUTES OF THE INDIVIDUAL, THE EDUCATIONAL OUTCOMES. THE ACGME HAS DEFINED THOSE SIX COMPETENCIES WHICH I'LL SHOW YOU IN A MOMENT, HOPEFULLY YOU ALL ARE FAMILIAR WITH THEM. MILESTONES, I PROVIDE THIS OVERVIEW BECAUSE I DO THINK THAT THERE IS A I LOT OF CONFUSION IN THE VARIETY OF TERMS THAT ARE USED HERE. BUT MILESTONES DESCRIBE THE COMPETENCIES IN DEVELOPMENTAL NARRATIVES. IN OTHER WORDS, THEY ALLOW A DESCRIPTIVE SENSE OF WHAT, FOR EXAMPLE, PROVIDING URGENT AND EMERGENT MEDICAL CARE IS TO A PATIENT IN A WAY THAT IS DEVELOPMENTAL AND WILL RESONATE MORE WITH THE FACULTY WHO MIGHT BE AND THE TRAINEE WHO'S LOOKING AT WHAT DOES THAT ACTUALLY MEAN? WHAT DOES PRACTICE-BASED LEARNING MEAN? SO THE MILESTONES DESCRIBE IT IN DEVELOPMENTAL WORDS. EPAs ARE ACTUALLY AN ATTEMPT TO TRANSLATE THE COMPETENCIES INTO CLINICAL PRACTICE. I KNOW THAT WHEN I DEVELOPED OUR COMPETENCY-BASED SYSTEM AT THE UME LEVEL AT FEINBERG, I PRETTY SOON QUICKLY REALIZED THAT IT WASN'T CRITICAL FOR ALL OF OUR FACULTY TO TRULY UNDERSTAND. IT WAS FOR THOSE WHO WERE DESIGNING A SYSTEM AND CERTAINLY IMPORTANT FOR THOSE WHO WERE SERVING ON COMPETENCY COMMITTEES TO UNDERSTAND THE DETAILS AND THE DIFFERENCES BETWEEN WHAT DO THESE TERMS MEAN. BUT FOR MANY, MANY PHYSICIANS WHO ARE OUT IN THE SYSTEM WHO ARE ACTUALLY PROVIDING DIRECT OBSERVATION AND ASSESSMENTS ON ALL OF OUR TRAINEES, TRYING TO GET THEM ALL TO UNDERSTAND DIFFERENCE BETWEEN COMPETENCIES AND MILESTONES WAS CHALLENGING. AND EPAs WERE THE -- IS ONE SOLUTION TO THAT PROBLEM. SO BASICALLY IT DEFINES THE CORE WORK THAT A PHYSICIAN DOES. IT'S A UNIT OF PROFESSIONAL PRACTICE, IN OTHER WORDS, LET ME GIVE YOU AN EXAMPLE OF AN EPA, IT MIGHT BE THE ABILITY TO RUN A FAMILY MEETING. NOW THAT EPA HAS A NUMBER OF COMPETENCIES THAT MIGHT BE BENEATH IT. IN OTHER WORDS, THE TRAINEE WHO CAN EFFECTIVELY RUN A FAMILY MEETING HAS TO HAVE PROFESSIONAL -- THE PROFESSIONALISM, ASPECTS OF THE PROFESSIONALISM COMPETENCY, HAS TO HAVE COMMUNICATION COMPETENCIES, HAS TO HAVE THE ABILITY TO -- MEDICAL NEJ COMPETENCY MOST LIKELY. MAY HAVE TO HAVE SOME SYSTEM AWARENESS AS WELL, SYSTEMS-BASED PRACTICE IN UNDERSTANDING THE INTEGRATION OF THE ENTIRE MEDICAL TEAM. SO THE EPA IS RUNNING THE FAMILY MEETING, AND I AS THE EVALUATOR OF THAT TRAINEE WILL MAKE A DECISION AT THE END OF OBSERVING MY TRAINEE RUNNING THAT FAMILY MEETING ON WHETHER I TRUST THAT TRAINEE TO DO IT INDEPENDENTLY IN THE FUTURE OR WHETHER THAT TRAINEE NEEDS MY PRESENCE RIGHT THERE AND THEN, WHICH IS A LITTLE BIT DIFFERENT LEVEL OF ACHIEVEMENT FOR THAT TRAINEE. I HOPE THAT EXPLAINS THAT A LITTLE BIT MORE, AND WE'LL HAVE SOME MORE EXAMPLES AS WE MOVE THROUGH. SO HERE'S JUST A SCHEMATIC OF COMPETENCIES, AND THERE ARE MANY SUBCOMPETENCIES WHICH VARY BY DISCIPLINE OBVIOUSLY ALTHOUGH THE SIX ACGME COMPETENCIES FOR GME REMAIN THE SAME, AND THESE ARE THE MILESTONES ACROSS THE LEVELS THAT THE TRAINEE NEEDS TO ACHIEVE AND ULTIMATELY AT THE TIME OF GRADUATION FROM GME TRAINING, IT'S USUALLY AT LEVEL 4. THE SCHEMATIC SHOWING THE RELATIONSHIP BETWEEN EPAs, COMPETENCIES AND MILESTONES IS SHOWN HERE. SO ANOTHER EXAMPLE FOR AN EPA IS THE TRAINEE IS ABLE TO PROVIDE TELEPHONE ADVICE AND MANAGEMENT OF PATIENTS VIA TELEHEALTH. WHILE THE COMPETENCY DOMAINS, WHOEVER CREATED THIS ONE, FEELS REALLY ALL SIX COMPETENCIES ARE REQUIRED TO SUCCESSFULLY DO THAT, THERE ARE VERY LIKELY SUBCOMPETENCIES THAT WOULD BE FURTHER DESCRIBED, AND THEN THE MILESTONES WOULD BE DESCRIBED HERE AS TO INPATIENT CARE IF IT IS, YOU KNOW, BEING ABLE TO QUICKLY DERIVE THE PATIENT'S CRITICAL HISTORY OF PRESENT ILLNESS FOR WHATEVER IS GOING ON, THAT'S PROBABLY PATIENT CARE AND MEDICAL COMPETENCY COMBINED, YOU WOULD HAVE A MILESTONE AS AN EVALUATE TORE OF WHAT IS EXPECTED OF THAT TRAINEE AT THAT PARTICULAR COMPETENCY. I WANT TO MAKE SURE THAT THE PROGRAM DIRECTORS IN THE AUDIENCE AND REALLY ANYBODY WHO SERVES ON COMPETENCY COMMITTEES OR FACULTY WHO ARE DEEPLY EMBEDDED IN THE ASSESSMENT OF OUR TRAINEES IS AWARE THAT THE ACGME HAS PUBLISHED A SUPPLEMENTAL MILESTONES GUIDE 2.0. THE REASON I HIGHLIGHT THIS IS BECAUSE WHAT'S REALLY CRITICAL IN A LOT OF THESE COMPETENCY ASSESSMENT SYSTEMS IS THAT THERE IS A SHARED MENTAL MODEL OF WHERE THE TRAINEES SHOULD BE AND AT WHICH STAGE. SO THIS GUIDE ACTUALLY PROVIDES A STARTING POINT FOR HELPING FACULTY AND TRAINEES AND COMPETENCY COMMITTEES CREATE THAT SHARED MENTAL MODEL. IT GIVES SPECIALTY-SPECIFIC EXAMPLES FOR EACH LEVEL OF TRAINING OR MILESTONES WITH DESCRIPTIONS AND EXAMPLES, AND THEN REALLY IMPORTANTLY, ESPECIALLY FOR THOSE WHO ARE DESIGNING THESE SYSTEMS, IT ALSO HAS SUGGESTIONS FOR ASSESSMENT TOOLS THAT ARE LISTED HERE AND THEN SOME REFERENCES FOR SOME CURRICULAR TOOLS AS WELL. SO ENCOURAGE YOU TO TAKE A LOOK AT THAT, ESPECIALLY IF YOU'RE DESIGNING THESE KINDS OF CBME PROGRAMMATIC ASSESSMENT PROGRAMS. SO AN EFFECTIVE ASSESSMENT SYSTEM IN CBME ALSO HAS ASSESSMENTS THAT REFLECT THE CURRICULUM AND THE EXPECTATIONS OF THE LEVEL OF THE TRAINEE. SO HERE'S WHERE MILLER'S PYRAMID, WHICH WAS REALLY DEVELOPED FOR OUTCOMES-BASED EDUCATION CAN BE HELPFUL, WHEN YOU THINK ABOUT WHAT ARE THE KINDS OF ASSESSMENTS THAT YOU WANT TO BUILD INTO YOUR ASSESSMENT PROGRAM FOR CBME. SOMETHING AT THE BASELINE LEVEL OF KNOWS THAT YOU CAN SHOW THAT THE TRAINEE HAS KNOWLEDGE ABOUT A CERTAIN TOPIC MIGHT BE GOOD ON AN IN-TRAINING EXAM WITH MULTIPLE CHOICE QUESTIONS. IF YOU WANT TO KNOW THAT THE TRAINEE CAN APPLY THAT KNOWLEDGE, YOU MIGHT DEVELOP SOME CLINICAL VIGNETTES TESTING THEIR DIAGNOSTIC REASONING, AND EVEN SOME ADVANCED MULTIPLE CHOICE QUESTIONS MIGHT ACTUALLY WORK HERE AS WELL. IF YOU WANT TO SHOW THAT THE TRAINEE CAN APPLY THIS IN CERTAIN SETTINGS AS ACTUALLY A SKILL, YOU MIGHT PROVIDE A SIMULATION OR STANDARDIZED PATIENT FOR THIS LEVEL OF THE MILLER'S PYRAMID. AND THEN THE HOLY GRAIL IS ACTUALLY YOU WANT TO SHOW THAT YOUR TRAINEE ACTUALLY DOES THIS IN PRACTICE. THAT'S WHERE THE FIELD OF WORKPLACE-BASED ASSESSMENT COMES IN. THAT'S THE MOST IMPORTANT PART OF THE PROGRAMMATIC ASSESSMENT THAT YOU'RE DEVELOPING AND CREATING FOR YOUR TRAINEE. THAT'S WHERE YOU HAVE MULTISOURCE FEEDBACK, YOU MIGHT HAVE PATIENT SURVEYS, YOU MIGHT HAVE 36 360s WITH NURSES AND PHYSICAL THERAPISTS EVALUATING YOUR TRAINEE, AND VERY CRITICALLY, A NUMBER OF DIRECT OBSERVATIONS. NOT JUST INFERRED PERFORMANCE, BUT ACTUALLY DIRECT OBSERVATION OF PERFORMANCE. AND THEN HOPEFULLY THAT WILL DEMONSTRATE THAT THE TRAINEE IS PREPARED AND READY TO HAVE THE POSITIVE OUTCOMES ON ACTUAL PATIENT CARE. WHEN YOU'RE THINKING ABOUT WHICH ASSESSMENT TOOLS TO USE, THERE CERTAINLY ARE QUITE A VARIETY OF THEM, AND WHEN I FIRST GOT INVOLVED IN THIS FIELD, I GOT KIND OF OVERWHELMED THINKING I HAD TO PICK THE PERFECT TOOL FOR THE PERFECT SITUATION. AND I WILL SHARE WITH YOU THAT HAVING THE RIGHT PROGRAMMATIC ASSESSMENT SYSTEM ACTUALLY ALLEVIATES SOME OF THAT ANGST, BECAUSE REALLY WHAT YOU'RE DOING IS HAVING MULTIPLE TOOLS, EYE IDEALLY MULTIPLE TOOLS THAT ARE DEPLOYED THROUGHOUT YOUR ASSESSMENT OR TRAINEE'S TIME WITH YOU, AND THAT ARE USED IN DIFFERENT CONTEXTS, AND THAT ACTUALLY TAKES THE BURDEN OFF ANY ONE TOOL, BUT PROVIDES -- ONE TOOL PROVIDES A SNAPSHOT, BUT MULTIPLE TOOLS AGGREGATED TOGETHER PROVIDES A MUCH MORE -- MUCH BETTER AND CLEARER PICTURE OF COMPETENCE, WHICH I'LL SHOW YOU IN A MOMENT. BUT WHEN YOU THINK ABOUT INDIVIDUAL TOOLS, I LIKE TO USE CEES VANDER VLEUTEN'S UTILITY INDEX BECAUSE WHETHER WE THINK ABOUT NOT ONLY THE COST AND HOW GOOD A TOOL MIGHT BE, BUT THERE'S FEASIBILITY IMPLICATIONS. WHILE OBSERVATION IS CLEARLY IMPORTANT, THERE ARE RESOURCE LIMITATIONS. AND SO YOU MIGHT NOT HAVE A SYSTEM WHERE YOU CAN DEPLOY AS MUCH DIRECT OBSERVATION AS YOU'D LIKE. OR YOU MAY NOT HAVE A SIMULATION CENTER. I KNOW THE NIH DOES. SO HOPEFULLY THAT'S NOT A BARRIER. BUT THERE ARE INSTITUTIONS THAT DON'T. SO YOU WANT TO LOOK AT ALL OF THESE ASPECTS WHEN YOU'RE THINKING ABOUT PUTTING TOOLS INTO THE WHOLE PICTURE OF YOUR PROGRAMMATIC ASSESSMENT SYSTEM. SO THE ACGME DOES HAVE MINIMAL ASSESSMENT COMPONENTS THAT IT REQUIRES. SO YOU ARE MOST LIKELY ALL FAMILIAR WITH SUMMARY EVALUATIONS THAT WE ALL FILL OUT WITH MILESTONES NOW ON THEM AND WE RATE OUR TRAINEES FROM LEVEL ONE OR NOVICE TO EXPERT. BUT THOSE ARE REALLY PROBABLY THE LEAST OR AMONG THE LEAST VALID OF OUR ASSESSMENT TOOLS, BECAUSE IT'S INFERRED PERFORMANCE. WHAT'S MUCH BETTER IS DIRECT OBSERVATION. NOW IF INSTEAD YOU'RE FILLING ONE OF THOSE OUT AFTER A TRAINEE PRESENTS A CASE DURING ROUNDS, THAT REALLY IS A DIRECT OBSERVATION, NOT REALLY A SUMMARY EVALUATION. SO CORRECT OBSERVATIONS, ESPECIALLY THOSE THAT ARE BEHAVIORALLY ANCHORED, ARE REALLY VERY EFFECTIVE TOOLS TO UNDERSTAND THE TRAINEE'S ABILITIES CERTAINLY IN THE WORKPLACE. AND THEN AS I MENTIONED, THERE ARE MULTI-FORCE FEED FEEDBACKS. ACG CGME DOES REQUIRE THAT THOSE OCCUR, AND IDEALLY ALSO PROVIDING AUDIT AND PERFORMANCE DATA, INCLUDING PATIENT EXPERIENCE, WHICH RELATES TO THE PRACTICE-BASED LEARNING AND SYSTEM-BASED PRACTICE COMPETENCIES. IN TRAINING EXAMS, GOOD ASSESSMENT OF MEDICAL KNOWLEDGE, AND WHEN AVAILABLE, SIMULATION ESPECIALLY FOR PROCEDURAL COMPETENCIES. SO I MENTIONED THAT AN EFFECTIVE ASSESSMENT SYSTEM PLACES ASSESSMENTS ACROSS THE CONTINUUM OF EDUCATION AND TIME THAT THE TRAINEE IS WITH YOU. AND I WANT TO SHOW YOU -- I DIDN'T REALIZE ALL OF THIS WAS ANIMATED SO MY APOLOGIES FOR THIS. SO I WANT TO SHOW YOU REALLY JUST A SCHEMATIC OF WHAT THAT MIGHT LOOK LIKE. SO IF THIS IS THE BEGINNING OF TIME OF TRAINING AND THIS IS THE END, AND YOU HAVE TRAINING ACTIVITIES, MAYBE WARD MONTHS, MAYBE AN ICU BLOCK, MAYBE AN ER BLOCK, MAYBE TIME IN OUR LAB, WHATEVER THE TRAINING COMPONENTS ARE THAT ARE REPRESENTED BY THE TRIANGLES AT THE TOP, YOU WANT TO THINK ABOUT THIS DELIBERATELY AS A PROGRAM DIRECTOR AND A DESIGNER OF YOUR ASSESSMENT SYSTEM. YOU WILL SPRINKLE OUT THE RELEVANT AND IMPORTANT ASSESSMENTS ACROSS THE TIME THAT HOPEFULLY CORRESPOND TO THE TRAINING ACTIVITY. SO IN OTHER WORDS, IF YOU BUILD IN TIME FOR SIMULATION, THAT MIGHT BE WHERE YOU'RE GOING TO PUT IN A SIMULATION ASSESSMENT. IF YOU HAVE A WARD BLOCK OR TWO OF THEM, YOU MIGHT WANT YOUR TRAINEE TO COLLECT A CERTAIN NUMBER OF DIRECT OBSERVATIONS OF AN ENTRUSTABLE PROFESSIONAL ACTIVITY OR IF YOU DON'T USE THOSE, OF A PARTICULAR COMPETENCY THAT YOU WANT THE TRAINEE TO ACTUALLY DEMONSTRATE SKILL IN. AND THEN I'LL SAY MORE ABOUT THIS, BUT ANOTHER PIECE THAT IS A CRITICAL ELEMENT TO A SUCCESSFUL CBME PROGRAM, AND I MENTIONED EARLIER THAT THIS IS MUCH MORE LEARNER-DRIVEN THAN THE TRADITIONAL EDUCATIONAL MODELS THAT WE'VE ALL BEEN USED TO, IS YOU BUILD IN TIME FOR LEARNER REFLECTION AND PLANNING. IDEALLY, YOUR LEARNER IS LOOKING OVER THEIR ASSESSMENTS, HOPEFULLY WITH A COACH, I'LL SAY MORE ABOUT THAT IN A MOMENT, WITH A COACH AND HELPING THEM TO UNDERSTAND THEIR STRENGTHS, THEIR AREAS OF WEAKNESS, AND WHERE THEY MAY ACTUALLY WANT TO ASK SOMEONE FOR EVEN MORE DIRECT OBSERVATION BECAUSE THEY MIGHT FIND THAT THEY NEED TO HONE A CERTAIN SKILLSET. SO THAT MAKES THE LEARNER HAVE A MUCH BIGGER INVESTMENT IN THEIR OWN SUCCESS. AND THESE ARROWS REPRESENT TIMES WHERE YOUR COMPETENCY COMMITTEES MIGHT BE GATHERING TOGETHER, AND MAKING SUM MAT IVE DECISIONS AND JUDGMENTS TAKING A LOOK AT ALL OF THESE ASSESSMENTS THAT HAVE NOW BEEN AGGREGATED AND PAINT THAT PICTURE AND THEN THEY WILL BE MAKING DECISIONS OF WHETHER OR NOT A TRAINEE IS REGULAR. SO ANOTHER IMPORTANT ASPECT HERE, AND I DON'T HAVE TIME TO GET INTO MASTERY LEARNING VERY MUCH, ALTHOUGH CERTAINLY SOMETHING I FEEL QUITE STRONGLY ABOUT, BUT THIS IS JUST A SLIDE THAT WILL SHOW YOU THAT YOUR TRAINEES AND OUR LEARNERS ARE COMING IN WITH VERY DIFFERENT SKILLSETS, AND IF YOU, FOR EXAMPLE, INSTEAD OF SAYING THIS IS A SIMULATION ENVIRONMENT, WE SAY IT'S AN EPA, LET'S SAY IN PROVIDING THAT TELEHEALTH COMMUNICATION EFFECTIVELY ADVISING -- ASSESSING AND ADVISING MANAGEMENT PRINCIPLES FOR A PATIENT VIA TELEHEALTH, YOU'RE GOING TO HAVE LEARNERS COMING IN WITH VERY DIFFERENT ABILITIES AT THE BEGINNING OF TRAINING. IF YOU SET A MASTERY STANDARD HERE TO AN EXPECTATION OF WHERE YOU CAN EXPECT THAT TRAINEE TO BE ABLE TO SUCCESSFULLY HAVE THAT CONVERSATION WITH A PATIENT VIA TELEHEALTH WITHOUT YOU HAVING TO BE ON THE CALL AS WELL AND JUST REPORT TO YOU, YOU MIGHT SAY THAT'S YOUR MASTERY STANDARD. YOU CAN SEE HERE THAT DIFFERENT TRAINEES ARE GOING TO ARRIVE THERE AT DIFFERENT TIMES. YET ALL-OF-US GET THERE. AND THIS IS THE KIND OF THING YOU NEED TO BE THINKING ABOUT AS YOU BUILD YOUR ASSESSMENT SYSTEM, ALLOWING FOR THE TRAINEE THAT MIGHT BE COMING IN HERE WITH LOWER SKILLSETS TO HAVE A LITTLE BIT MORE TIME FOR PRACTICE THAN THIS TRAINEE, WHO MIGHT BE COMING IN WITH MAYBE A MEDICAL SCHOOL THAT HAD A TELEHEALTH CURRICULUM AND THIS RESIDENT IS NOW A LITTLE MORE ADVANCED AND IS GOING TO GET TO THIS MASTERY LEVEL A LITTLE SOONER. SO THAT'S SORT OF WHAT THAT LOOKS LIKE ABOUT SORT OF DELIBERATELY PLACING YOUR ASSESSMENTS. AND IT HELPS A LOT IF WE HAVE AN EFFECTIVE UME TO GME TRANSITION, WHICH REMAINS ONE OF OUR CHALLENGES IN CBME. SO I MENTIONED THIS EARLIER, THIS IS A REALLY IMPORTANT MESSAGE, THAT AN EFFECTIVE ASSESSMENT SYSTEM HAS THE DESIGN SO THAT MULTIPLE EVALUATORS ARE FILLING OUT ASSESSMENTS IDEALLY DIFFERENT TYPES OF ASSESSMENTS IN DIFFERENT CONTEXTS. AND THAT IS HOW YOU PAINT THE PICTURE OF COMPETENCE. SO I LIKE THIS SORT OF ANALOGY HERE. SO EARLY ON, ANY ONE POINT OF ASSESSMENT, ONE OR FIVE OR 10 OR HOWEVER MANY PIXELS MIGHT BE HERE, DIRECT OBSERVATIONS DON'T NECESSARILY GIVE YOU A VERY CLEAR PICTURE OF MONA HERE, AND THAT GETS BETTER WHEN YOU START TO COLLECT EVEN MORE MOMENTS OF OBSERVATION, SUPPLEMENTED, MAYBE CHART RECALL, CHART AUDIT, SUPPLEMENTED MAYBE BY A SIMULATION ACTIVITY, SUPPLEMENTED BY A PERFORMANCE IN TRAINING EXAM, AND EVENTUALLY YOUR COMPETENCY COMMITTEE TAKES ALL OF THOSE PIXELS AND PAINTS A MUCH MORE CLEAR PICTURE OF THE ABILITIES OF THAT PARTICULAR TRAINEE. THAT'S THE ULTIMATE GOAL. THAT ALSO, AS I MENTIONED, TAKES THE PRESSURE OFF OF ANY ONE TOOL BY ITSELF TO BE PERFECT. SO ANOTHER IMPORTANT ASPECT THAT CAN BE VERY CHALLENGING NOT ONLY TO GET OUR FACULTY TO PROVIDE FOR US BUT ACTUALLY ALSO TO ANALYZE AS COMPETENCY COMMITTEES IS THAT THE MOST EFFECTIVE FEEDBACK AND ASSESSMENT TOOLS INCLUDE NARRATIVE FEEDBACK. IF I'M A LEARNER, I REALLY NEED THAT NARRATIVE TO UNDERSTAND WHY I WAS RATED AT, SAY, A LEVEL 2 INSTEAD OF WHAT I THOUGHT I WAS, I MIGHT BE A LEVEL 3 OR A LEVEL 4. SO WE KNOW THAT INDIVIDUALS, ESPECIALLY IN MED SIRNTION ARE MEDICINE, A RE NOT PARTICULARLY GOOD AT SELF-ASSESSMENT. SO GETTING AS MUCH BEHAVIORALLY ANCHORED FEEDBACK BACK TO THAT INDIVIDUAL LEARNER WILL HELP THEM BE ABLE TO SEE THEIR STRENGTHS AND WEAKNESSES MUCH BETTER. ESPECIALLY IF THERE ARE DIFFERENT RATINGS. WE ALL KNOW THAT ALL OF US, AND CERTAINLY THE STUDENTS KNOW IN THE AUDIENCE, THAT THEY -- IT'S SOMETIMES VERY UNCLEAR WHY SOMEBODY RATES THEM A 5 ON 1 TASK AND SOMEBODY ELSE RATES THEM A 2. THERE'S DOVES, THERE'S HAWKS. AGAIN, IF YOU HAVE ENOUGH DATA POINTS AND IF YOU HAVE NARRATIVE FEEDBACK THAT IS BEHAVIORALLY ANCHORED, IT ALLOWS THE LEARNER TO SYNTHESIZE THAT DISPARATE FEEDBACK AND GET A MUCH CLEARER PICTURE. SO AN EFFECTIVE SYSTEM ALSO IN CBME HAS A PLACE FOR THAT LEARNER AND FOR YOUR COMPETENCY COMMITTEES TO REVIEW ALL OF THAT DATA THAT YOU'VE NOW PUT TOGETHER WITH ALL OF THESE WONDERFUL ASSESSMENTS THAT YOU'VE DEVELOPED. AND THAT MIGHT LOOK SOMETHING LIKE THIS. THERE ARE A VARIETY OF DIFFERENT TYPES OF DASHBOARDS THAT ARE USED, SOME WITH SPIDER PLOTS, SOME HOME GROWN, SOME NEW INNOVATIONS CERTAINLY HAS SOME AS WELL, AND HOPEFULLY YOUR LEARNER WHO INTERACTS WITH THIS INFORMATION CAN DRILL DOWN FURTHER TO INDIVIDUAL ASSESSMENTS TO READ THAT NARRATIVE FEEDBACK. AND IDEALLY, THESE KINDS OF SYSTEMS, WE CALL OURS PORTFOLIO SYSTEM, ACTUALLY HAS A PLACE FOR THAT LEARNER TO REFLECT ON THAT FEEDBACK AND MAYBE EVEN FOR THE COACH TO REFLECT ON THEIR TRAINEE'S FEEDBACK, AND CREATE LEARNING PLANS AND CREATE GOALS THAT THEY'RE SETTING FOR THEIR NEXT WARD ROUNDS MONTH AND WHAT THEY WANT TO IMPROVE ON. IN ADDITION, THE COMPETENCY COMMITTEES THAT ARE ASKED TO MEET TO MAKE DECISIONS ABOUT THOSE MULTIPLE ASSESSMENTS AND DECIDE WHETHER OR NOT THE TRAINEE HAS MET THE EXPECTED BENCHMARKS IN EACH COMPETENCY THAT'S BEING EVALUATED, HAS TO HAVE AN EFFECTIVE PLACE TO LOOK AT ALL OF THAT INFORMATION AND HAVE IT BE ORGANIZED AND EASY TO UNDERSTAND BY EPA IF YOU'RE USING THEM OR BY COMPETENCY OR BY MILESTONE. SO I MENTIONED HOW CRITICAL I REALLY DO BELIEVE THAT THIS SELF DIRECTED LEARNING IN IN A REALLY EFFECTIVE CBME SYSTEM. YOU'LL SEE WHEN I GO TO THE END AND TALK ABOUT SOME OF THE PILOTS THAT ARE BEING USED, WITH ENOUGH AUTONOMY, THE LEARNERS ARE REALLY STARTING TO EMBRACE ASSESSMENT IN A WAY THAT CERTAINLY ISN'T UNIVERSAL IF THEY FEEL THAT IT ACTUALLY HELPS THEM UNDERSTAND THEIR SUCCESS BETTER. SO REALLY, TAKING SOME TIME TO MAKE SURE THAT YOUR FACULTY, IF POSSIBLE, CAN PROVIDE THAT KIND OF DIRECTED FEEDBACK TO LEARNERS IS REALLY THE PLACE WHERE YOU'RE INCULCATING THE SKILLS OF ONGOING DESIRE FOR FEEDBACK ASSESSMENT THROUGHOUT OUR ENTIRE CAREERS. SO I MENTIONED ALREADY DUNN AND KRUEGER HAVE SHOWN THIS IN THEIR CLASS THAT WE KNOW DOCS THAT ARE REALLY QUITE GOOD OFTEN UNDERESTIMATE THEIR SKILLSET SO THERE'S BEEN A LOT OF LITERATURE IN COACHING AND WHEN POSSIBLE, GME PROGRAMS SHOULD INSTITUTE COACHING AS WELL. MANY PEOPLE ADVOCATE FOR THE COACH TO NOT BE INVOLVED IN DECISION-MAKING OR ON COMPETENCY COMMITTEES BUT ACTUALLY HAVING A FIREWALL BETWEEN THE COACH AND ANY JUDGMENT ABOUT THE TRAINEE'S PROGRESSION ALLOWS FOR A VERY TRUSTED AND HONEST RELATIONSHIP THAT THE TRAINEE CAN THEN ALSO BE VULNERABLE IN THEIR AREAS OF SUCCESS AND AREAS OF WEAKNESS. SO WE'VE BEEN TALKING QUITE A BIT ABOUT -- I'VE MENTIONED QUITE A BIT ABOUT COMPETENCY COMMITTEES, AND I WANT TO SPEND A LITTLE BIT MORE TIME ON THAT, BECAUSE CLEARLY THOSE ARROWS IN OUR SCHEMATIC OF TRAINING TIME, THOSE ARROWS ARE IMPORTANT AND THESE COMMITTEES ARE MAKING A DECISION THAT ALL OF THIS DATA IS AGGREGATED AND MAKING A DECISION WHETHER THE TRAINEE HAS DEMONSTRATED THAT HE OR SHE IS READY TO MOVE ON TO THE NEXT LEVEL OF TRAINING. WHETHER THAT BE AT THE CRITICAL DECISION AT THE TIME OF GRADUATION FROM TRAINING AND INTO INDEPENDENT PRACTICE OR EVEN AT THE TIME WHERE SOMEONE IS MOVING FROM THE SECOND YEAR TO THE THIRD YEAR OF THEIR RESIDENCY PROGRAM. SO THERE'S BEEN A LOT OF CRITIQUE ABOUT JUDGMENT IN THE OLD DAYS OF ASSESSMENT, AND SORT OF DISPARAGING OF SUBJECTIVE DECISION-MAKING ABOUT A TRAINEE'S PERFORMANCE, AND THAT'S REALLY CHANGED. AND IT'S CHANGED BECAUSE WE UNDERSTAND THAT THESE COMPETENCIES ARE COMPLEX, AND ACTUALLY THERE IS NO TOOL THAT ALLOWS YOU TO SAY -- YOU KNOW, THERE'S NO MAGIC TOOL THAT I CAN THINK OF THAT WOULD INTERGREAT >> INTEGRATE COMPETENCIES REQUIRED TO HOLD AN EFFECTIVE FAMILY MEETING INDEPENDENT OF A IDEALLY TRAINED PHYSICIAN OBSERVER, AND HIS OR HER JUDGMENT IS CRITICAL IN MAKING A DECISION ABOUT WHETHER OR NOT SOMEBODY IS DOING WHAT THEY SHOULD BE DOING IN OUR COMPLEX ENVIRONMENT OF HEALTHCARE AND PATIENT CARE. BUT OBVIOUSLY THERE'S BIAS, AND OBVIOUSLY NO ONE PERSON SHOULD BE RELIED UPON AND THIS JUST GOES BACK TO EVERYTHING I'VE ALREADY SAID, WHICH IS MULTIPLE OBSERVATIONS OVER TIME, IDEALLY BY DIFFERENT OBSERVERS PAINT THE BEST PICTURE. IN ADDITION, ON A COMPETENCY COMMITTEE, THERE IS DATA THAT SAYS, YOU KNOW, GROUP PROCESSES ACTUALLY CAN IMPROVE JUDGMENT AND OUT COME. THERE WAS A STUDY THAT LOOKED AT SURGICAL RESIDENCY PROGRAMS AND THAT MORE DEFICIENCIES, NEARLY 20% OF DEFICIENCIES WERE IDENTIFIED VIA A DISCUSSION OF THE PARTICULAR TRAINEE'S PERFORMANCE THAN BY JUST LOOKING AT THE RATINGS OR ONE INDIVIDUAL DECISION ALONE, WHICH THERE THEY FELT THE TRAINEE WAS READY TO MOVE ON. SO AS I MENTIONED EARLIER AGAIN, I'M GOING TO HIGHLIGHT THE IMPORTANT POINTS, SHARED MENTAL MODEL IS PARTICULARLY CRITICAL FOR COMPETENCY COMMITTEES. AND THERE ARE SOME FACTORS THAT HAVE BEEN IDENTIFIED WITH INCREASED EFFICACY OF COMPETENCY COMMITTEES. SO HAVING A DIVERSE MEMBERSHIP IS REALLY IDEAL. MORE INFORMATION, I'VE SAID THIS MANY, MANY TIMES. AS WITH ANY GROUP PROCESS, HAVING A LEADER WHO'S NOT TOO DOMINANT AND WHO INVITES ALL OPINIONS. THE IDEAL SIZE FOR COMPETENCY COMMITTEES IS PROBABLY AROUND FIVE TO 10 MEMBER. AND THEN A SHARED MENTAL MODEL WITH MAKING SURE THAT EVERYBODY UNDERSTANDS THE MILESTONES OR ENTRUSTABLE PROFESSIONAL ACTIVITIES IF YOU HAS HAPPEN TO BE USING THOSE IN YOUR PROGRAM. AND VERY KEY IS AN EFFECTIVE UNDERLYING TECHNOLOGY. THAT DASHBOARD I SHOWED YOU, A SYSTEM FOR THE COMPETENCY COMMITTEE NOT TO HAVE TO BE SEARCHING AND TO TRULY PAINT A PICTURE AND HELP THE COMMITTEE UNDERSTAND THE PERFORMANCE. AND IT SHOULD BE ORGANIZED BY COMPETENCY OR EPA AND NOT BY ASSESSMENT TYPE. IT'S MUCH RICHER THAN IF YOU DON'T JUST WANT TO LOOK AT ALL OF THE MULTIPLE CHOICE EXAM QUESTIONS OR ALL OF THE DIRECT OBSERVATIONS. IT'S MUCH BETTER TO HAVE THE VARIETY OF TOOLS THAT PAINT THE PICTURE IN EACH INDIVIDUAL COMPETENCY. FACTORS THAT INHIBIT THESE COMMITTEES INCLUDE A LOW QUANTITY OF DATA, OBVIOUSLY MONA LISA IS STILL PIXELATED THERE. A LOW QUALITY OF DATA, NOT ENOUGH NARRATIVE FEEDBACK TENDS TO BE THE MOST IMPORTANT HERE. AND THEN CLUNKY IT SYSTEMS, NO QUESTION, ARE A BIG BARE WE BARRIER. AND THESE COMMITTEES NEED TO BE RESOURCED. THEY NEED TO HAVE TIME TO BE ABLE TO -- AND WE NEED TO SUPPORT THEIR EFFORTS. THEY NEED TIME TO BE ABLE TO DO THIS WORK EFFECTIVELY. THERE ARE SOME RESOURCES THAT HELP WITH THIS, INCLUDING SOME STUDIES AND BOOKS, BUT HOPEFULLY THE PROGRAM DIRECTORS ARE ALL FAMILIAR WITH THE ACGME CLINICAL COMPETENCY COMMITTEE GUIDEBOOK, AND HOPEFULLY I WOULD ENCOURAGE THEM TO SHARE THAT GUIDEBOOK WITH ALL COMPETENCY COMMITTEE MEMBERS AS WELL. SO LET ME JUST PUT ALL OF THIS TOGETHER INTO OUR ASSESSMENT SYSTEM HERE. WE HAVE THE STRUCTURED PORTFOLIO IN THE MIDDLE WITH ALL OF THE VARIOUS ASSESSMENTS THAT YOU'VE CAREFULLY THOUGHT ABOUT PLACING ALONG YOUR TRAINING PROGRAM THAT MAKE THE MOST SENSE OR RELATING TO YOUR CURRICULUM AS WELL. YOU HAVE YO UR COACH YOUR COACH OR ADVISO R THAT WORKS WITH YOUR TRAINEE TO INTERACT WITH THAT STRUCTURED PORTFOLIO THAT LOOKS AT ALL OF THAT AND THEN CAN REFLECT ON THE CONTENTS AND DEVISE LEARNING PLANS. YOU HAVE YOUR COMPETENCY COMMITTEE THAT IS MAKING A DECISION PERIODICALLY REVIEWING ACGME REQUIRES AT LEAST TWICE YEARLY TO REVIEW THAT SYSTEM. I THINK ANOTHER IMPORTANT POINT HERE IS IF THIS IS DONE WELL, YOU'RE GOING TO IDENTIFY LEARNERS THAT ARE NOT QUITE WHERE THEY SHOULD BE, AND YOU NEED TO HAVE RESOURCES TO BE ABLE TO PROVIDE SUPPORT FOR THOSE LEARNERS. THAT BECOMES INCREASINGLY CLEAR IN A CBME SYSTEM, BUT THAT'S IMPORTANT. WE DON'T WANT TO BE GRADUATING THOSE LEARNERS BECAUSE MAYBE THAT IS PART OF WHY OUR OUTCOMES AREN'T BETTER. AND THEN IDEALLY YOUR PROGRAM LEADERS ARE TAKING A LOOK AT THE OUTCOMES OF THEIR TRAINEES, AND THAT DRIVES CURRICULUM. IF YOU SEE THAT THERE ARE DEFICITS IN THE TELEHEALTH COMPETENCIES, MAYBE YOUR TELEHEALTH CURRICULUM ISN'T EFFECTIVE ENOUGH. AND THAT WILL DEVELOP -- ALLOWS THE PROGRAM DIRECTORS TO HELP THINK ABOUT WHAT KIND OF CURRICULUM WILL DRIVE THE KINDS OF OUTCOMES YOU WANT TO SEE. AND ULTIMATELY, THOSE ARE THEN SHARED WITH LICENSING AND CERTIFICATION AGENCIES. SO, THAT WAS A WHIRLWIND TOUR, I KNOW, INTO A LOT OF THINGS. THERE'S A LOT MORE INFORMATION FOR THOSE WHO WANT TO DIVE DEEPER INTO EPA'S MASTERY LEARNING, PROGRAMMATIC ASSESSMENT, ALL OF THOSE THINGS. I JUST WANTED TO GIVE AN OVERVIEW OF WHAT I THINK TRULY ARE THE COMPONENTS OF AN EFFECTIVE SYSTEM, AND OBVIOUSLY I'M ALWAYS HAPPY TO BE A RESOURCE TO ANYBODY IN THE FUTURE AND HAVE LEFT MY EMAIL WITH EVERYONE THERE. I'D BE HAPPY TO TALK WITH ANYONE WHO WANTS MORE INFORMATION. BUT LET'S TALK FOR A MOMENT ABOUT WHERE I THINK CBME MIGHT BE HEADED. I DO THINK IT'S HERE TO STAY, NO QUESTION. BUT THERE ARE CERTAINLY CHALLENGES, AND I THINK THAT GME IS ACTUALLY FURTHER ALONG THAN UME IN TERMS OF IMPLEMENTATION, MOSTLY BECAUSE THE ACGME HAS MANDATED IT, BUT THAT DOESN'T MEAN THAT THE PROGRAMS ARE DOING THIS ARE DOING IT AS WELL AS IDEALLY I'VE JUST DESCRIBED. THAT REQUIRES A LOT OF RESOURCES. WE'VE ALREADY MENTIONED THE TRAINING OF RATERS TO INCREASE THE RELIABILITY. WE NEED TO DEPLOY MULTIPLE DIFFERENT TIMES OF ASSESSMENT TOOLS. THOSE ARE NOT EASY, NECESSARILY, THEY REQUIRE RESOURCES, AND YOU NEED TO BE ABLE TO SUPPORT YOUR COMPETENCY COMMITTEES AND THE INFRASTRUCTURE, THE TECHNOLOGICAL INFRASTRUCTURE, INCLUDING MAYBE EVEN MOBILE OBSERVATION TOOLS ON A FACULTY'S CELL PHONE WHICH WE'VE BEEN USING AT FEINBERG WITH A LOT OF SUCCESS FOR RAPID DIRECT OBSERVATION. SO THERE'S A LOT OF RESOURCES THAT ARE REQUIRED. IN ADDITION, ONE OF THE BIG IMPEDIMENTS HAS BEEN THE SORT OF CHALLENGE AND TENSION BETWEEN SERVICE OF TRAINEES VERSUS EDUCATION OF TRAINEES. AND THAT TRAINEES ARE EMPLOYEES OF HOSPITAL SYSTEMS, AND LARGELY REQUIRED TO SPEND MORE TIME IN SERVICE THAN IN EDUCATION. IDEALLY WE FIGURE OUT A WAY TO MELD THOSE TWO. REGULATORY AGENCIES IN THE UNITED STATES ARE MANY, AND THAT HOUSE OF MEDICINE BEING SILOED AND DIVIDED HAS ACTUALLY BEEN AN IMPEDIMENT TO MORE UNIFORM ADAPTATION OF SOME OF THE PRINCIPLES OF WHAT WE'VE TALKED ABOUT. AND THEN COMPLACENCY. CHANGE IS HARD. MANY OF THE LEADERS IN OUR DIFFERENT HEALTH SYSTEMS WERE TRAINED, AS I WAS, IN A MORE TRADITIONAL MODEL AND FEEL IT'S NOT BROKEN. SO ONE NEEDS TO MAKE A REAL ARGUMENT FOR CHANGE AND HOPEFULLY I'VE DONE THAT A LITTLE BIT AT THE BEGINNING OF THIS TALK AND YOU CAN USE SOME OF THOSE STATEMENTS WHEN NEEDED TO MAKE AN ARGUMENT FOR CHANGE. CANADA IS FAR AHEAD OF US. THE ROYAL COLLEGE OF PHYSICIANS AND SURGEONS IN CANADA ADOPTED A PLAN THAT FROM 2014 TO 2021, ALL OF THEIR GME PROGRAMS WOULD BE IN COMPETENCY -- IN A COMPETENCY BASED DESIGN AND FRAMEWORK. THEY HAVE HAD MORE SUCCESS WITH IMPLEMENTATION BECAUSE THEY DO REALLY HAVE ONE MAJOR REGULATORY ORGANIZATION AND NOT ALL OF THE MULTIPLE SILOS AND HOUSES OF MEDICINE THAT WE DO, SO THEIR INCREASING NUMBERS OF IMPLEMENTATION PAPERS COMING OUT OF DAN AFTER MORE AND MORE PROGRAMS HAVE MOVED OVER TO A CBME TRAINING MODEL. BUT THEY EXIST IN THE UNITED STATES AS WELL, AND SOME OF YOU MAY BE FAMILIAR WITH WHAT WE CALL EPAC. IT'S THE EDUCATION IN PEDIATRICS ACROSS THE CONTINUUM, THAT WAS STARTED A FEW YEARS AGO. THE PARTICIPATING INSTITUTIONS ARE BELOW, AND BASICALLY WHAT IT WAS IS THAT THE PARTICIPATING INSTITUTION TOOK MEDICAL SCHOOLS AND WHO WERE PART OF THIS PILOT PROGRAM. STUDENTS IDENTIFIED EARLY THAT THEY WERE INTERESTED IN PEDIATRICS AND THEN THEY WERE PART OF A COMPETENCY BASED EPA FRAMEWORK WHERE THEY HAD MULTIPLE OUTCOMES THAT WERE MEASURED, MULTIPLE ASSESSMENTS, EVERYTHING I JUST DESCRIBED TO YOU, AND THEN IF MEETING BENCHMARKS, WERE ABLE TO ENTER THE AFFILIATED INSTITUTION'S PEDIATRIC RESIDENCY PROGRAM EARLIER THAN THE TRADITIONAL TIME FOR UME TO GME TRADITION, AND CONTINUE THEIR DEVELOPMENT IN THOSE RESIDENCY PROGRAMS. SO BOB ENGLANDER WHO HAS PUBLISHED A LOT ON THIS TOPIC IS NOW THE HEAD PERSON, VICE DEAN FOR EDUCATION AT UNIVERSITY OF MINNESOTA, AND HAS SHARED THAT THIS IS SORT OF THE CURRENT STATE. SO THE TYPICAL TIME FOR TRAINING IN PEDIATRICS IS 36 MONTHS. IN THEIR FIRST COHORT, THEY HAD TWO MEDICAL STUDENTS WHO ENTERED THE PEDIATRIC RESIDENCY PROGRAM AFTER 33 TO 34 MONTHS. THEY HAD A FEW WHO ACTUALLY TOOK A LITTLE BIT OF TIME BECAUSE THEY WERE THINKING ABOUT DIFFERENT CAREER GOALS, AND THEN THE COHORT IS PROGRESSING WELL IN RESIDENCY. WE'RE GOING TO NEED TO WAIT TO SEE WHAT THESE OUTCOMES WILL BE. THE PROBLEM HERE IS THAT THIS IS ONE SPECIALTY, IT'S PEDIATRICS. IT'S RELATIVELY SMALL NUMBERS. FAIRLY HIGHLY RESOURCED, SO FEASIBILITY IS A CHALLENGE, BUT WE'RE GOING TO NEED TO WAIT AND SEE WHETHER OR NOT MORE AND MORE SUCCESS IS COMING FROM THESE EPAC PROGRAMS AND WHETHER THIS CAN BE SCALED TO LOTS OF OTHER TRAINING TYPES OF PROGRAMS. THERE'S ALSO PILOTS GOING ON IN GME, SO ACGME HAS THESE ADVANCING INNOVATION IN RESIDENCY EDUCATION PILOT PROGRAMS, THAT'S LISTED, THE ONES THAT I'M AWARE OF HERE. THERE'S ONE IN TRANSPLANT HEPATOLOGY WHERE GI FELLOWS WHO ARE INTERESTED IN TRANSPLANT HEPATOLOGY CAN BEGIN THAT WORK IN THE THIRD YEAR OF FELLOWSHIP IF THEY HAVE DEMONSTRATED COMPETENCE IN THEIR ASSESSMENT SYSTEMS THAT ARE EMPEDDED INTO EACH OF THESE PILOT PROJECTS AND MUST BE -- THE TRAINING MUST DEMONSTRATE ACHIEVEMENTS AT BENCHMARK BEFORE BEING ALLOWED TO PURSUE THIS. THERE'S ONE ALSO IN INTERNAL MEDICINE AND CARDIOLOGY WHERE THE THIRD YEAR OF INTERNAL MEDICINE RESIDENCY IS BLENDED WITH BEGINNING OF CARDIOLOGY FELLOWSHIP, AGAIN, IT'S DIFFERENT THAN A PSTP PROGRAM. THIS IS MORE OF AN ACTUAL JUST CLINICAL CARDIOLOGY TRAINING PROGRAM THAT IS BASED ON COMPETENCY OUTCOMES, ACHIEVEMENT BY THE TRAINEE. AND THEN ANOTHER ONE IN GERIATRICS AND PALLIATIVE MEDICINE, WHERE THEY'RE TRYING TO MELD TWO INDIVIDUAL YEARS OF TRAINING INTO ONE. SO LOOKING FORWARD, HOW CAN CBME IMPROVE? WELL, I THINK WE'VE ALREADY MENTIONED HOW CRITICALLY IMPORTANT WORKPLACE BASED ASSESSMENTS ARE, THE ADOPTION OF MORE ENTRUSTABLE PROFESSIONAL ACTIVITIES WHICH REALLY HAVE BEEN SHOWN TO ENHANCE THE RELIABILITY OF THAT WORKPLACE BASED ASSESSMENT AND THE OBSERVATION AS IT IS A COMMON FRAMEWORK FOR CLINICIANS TO UNDERSTAND THAT'S THE WORK THAT THEY DO, THEY'RE BEING ASKED TO MAKE A DECISION IF THEY TRUST A TRAINEE ON THAT WORK. OBVIOUSLY LEVERAGES -- ADVANCING TECHNOLOGY WILL CLEARLY HELP US. AT FEINBERG, WE HAVE ALREADY -- WE'RE IN THE MIDST OF A STUDY LOOKING AT WHETHER NATURAL LANGUAGE PROCESSING CAN HELP US ANALYZE THE MULTITUDES OF NARRATIVE DATA THAT EXIST IN OUR PORTFOLIO SYSTEM. I THINK THAT'S NOT TOO FAR DOWN THE ROAD. ARTIFICIAL INTELLIGENCE WILL BE USED, I'M SURE, TO AGGREGATE THAT AND ORGANIZE THAT DATA AND MAY HELP A LEARNER UNDERSTAND HIS OR HER TRAJECTORY OF LEARNING AS WELL. AS I MENTIONED EARLIER, GME IS FURTHER ALONG IN THIS AN UME IS, BUT THERE IS INTENSE EFFORT RIGHT NOW TO ENHANCE THE TRANSITION TIME FROM UME TO GME AND IMPROVE THAT INFORMATION GAP THAT EXISTS BETWEEN MEDICAL SCHOOLS AND RESIDENCY PROGRAMS RIGHT NOW. AND THEN THE HOLY GRAIL HERE REALLY IS FOR TIME VARIABLE EDUCATION. AS I SHOWED YOU THAT SLIDE THAT LOOKED AT THE MASTERY -- THE DIFFERENT LEVEL OF ACHIEVEMENT OF THE TRAINEE IN THE CURRICULUM, IDEALLY SOMEBODY WHO'S MEETING THEIR BENCHMARKS MIGHT BE ABLE TO ENTER FELLOWSHIP EARLY, MIGHT BE ABLE TO COMPLETE FELLOWSHIP EARLY. IT'S A COMPLICATED CONVERSATION BECAUSE THERE CLEARLY ALSO IS TIME IN THE DOMAIN THAT MATTERS. IT'S NOT JUST SOMEBODY COMING IN AND SHOWING THAT THEY MIGHT BE GOOD AT -- SOMETHING RIGHT AWAY. I THINK MOST OF US AS EDUCATORS FEEL THAT THERE IS VALUE IN TIME SPENT IN CERTAIN DOMAINS, BUT HAVING SOME VARIABILITY IN HOW THAT TIME IS USED IS ANOTHER WAY TO THINK ABOUT IT. SO A TRAINEE WHO MIGHT BE QUITE ADEPT AT CERTAIN COMPETENCIES MAY HAVE MORE TIME TO PURSUE A RESEARCH PROJECT OR PURSUE ANOTHER PASSION OF THEIRS. SO THAT CAN BE BUILT INTO THE TRAINING TIME BECAUSE OBVIOUSLY THE CONSTRAINTS AROUND RESIDENCY MATCH AND FELLOWSHIP MATCH AND THOSE KINDS OF REGULATIONS ARE GOING TO HAVE TO BE QUITE ADJUSTED IF WE WIND UP MOVING TO MUCH MORE AD HOC SYSTEM. AND THEN AGAIN, ULTIMATELY CONNECTING THE LEARNER OUTCOMES TO PATIENT CARE IS REALLY WHAT WE ALL WANT TO ACHIEVE, AND IF THIS MODEL IS ADAPTED AS EFFECTIVELY AS IT COULD BE, IF WE PUT IN THE RIGHT RESOURCES, REALLY, THIS IS A MODEL FOR CONTINUING CERTIFICATION AS WELL. I KNOW AT THE ABIM, WE ARE STARTING TO HAVE CONVERSATIONS ABOUT WHAT DOES THE FUTURE OF ASSESSMENT OF A PRACTICING PRACTITIONER, PRACTICING PHYSICIAN LOOK LIKE, AND HAVING SORT OF A COMPETENT BASED FRAMEWORK FOR THAT IS ALSO IN DISCUSSION AND IS A VISION OF THE AMERICAN BOARD OF MEDICAL SPECIALTIES AS WELL. SO THE POTENTIAL AS I MENTIONED, IT IS A TRUE CONTINUUM, MENTIONED THE TIME VARIABLE. THE POWER OF ASSESSMENT FOR LEARNING AND THE EMBRACING OF ASSESSMENT AND NORMALIZING FEEDBACK IS SOMETHING THAT A TRAINEE OR A PHYSICIAN OR ANYBODY IS SEEKING TO HAVE IS REALLY QUITE IMPRESSIVE. SO I'M GOING TO STOP THERE AND OPEN IT UP AND HOPE THAT THERE MIGHT BE SOME QUESTIONS. THAT YOU ALL MIGHT HAVE THAT I CAN HELP ANSWER. >> WELL, THAT'S GREAT, MARIANNE, WE DO HAVE ONE QUESTION FROM THE AUDIENCE TO START. I HAVE SOME QUESTIONS FOR YOU TOO. I WILL BE ASKING THEM AFTER THIS ONE. I THINK YOU ADDRESSED IN IN YOUR TALK BUT MAYBE YOU CAN PROVIDE A LITTLE SUMMARY TO ANSWER THE QUESTION HERE WHICH IS, HOW DO YOU TEACH AND ASSESS CRITICAL THINKING SKILLS WITHIN A CBME SYSTEM? >> THAT'S A GREAT QUESTION. I THINK IT'S MULTIMODAL, HOW YOU TEACH CRITICAL THINKING SKILLS, SO FOR EXAMPLE, THERE ARE CLINICAL VIGNETTE APPLICATIONS, ELECTRONIC RESOURCES THAT ALLOW A LEARNER TO WORK THROUGH A COMPLICATED CASE SCENARIO AND HAVE TO DERIVE AND EXPLAIN THEIR RATIONALE FOR A PARTICULAR DIFFERENTIAL DIAGNOSIS. CRITICAL THINKING SKILLS IS A BROAD TOPIC. IT'S ALSO THE APPLICATION OF EVIDENCE AT THE BEDSIDE, CLEARLY ASKING THE LEARNER TO ARTICULATE THAT, ROLE MODELING THAT, DOING IT IN A SIMULATION SETTING WITH DEBRIEF OPPORTUNITIES. THERE ARE MULTIPLE WAYS YOU CAN PAINT A PICTURE OF THAT CRITICAL THINKING. BOB, WAS THE QUESTION MORE ABOUT ASSESSING OR TEACHING? >> BOTH. >> BOTH. WELL, THE TEACHING OF CRITICAL THINKING, I THINK HAS HAD A LOT OF LITERATURE AROUND IT, AND I THINK ONE OF THE BIGGEST CHALLENGES AROUND THAT IS THAT YOU GET TO A LEVEL OF EXPERTISE AND TEACHERS DON'T SPEND ENOUGH TIME DECONSTRUCTING HOW THEY ARRIVE AT A CERTAIN DECISION. WHETHER IT BE A DIAGNOSIS OR A MANAGEMENT DECISION. AND SO HELPING YOUR FACULTY DECONSTRUCT THAT FOR A LEARNER IS REALLY IMPORTANT. THERE'S DATA AND CLINICAL REASONING LITERATURE AND CLINICAL REASONING THAT GETS AT THAT. BEYOND THAT, IT'S NOT AN EASY THING. >> RIGHT. SO WE'RE AWARE THAT IN PEDIATRIC TRAINING, THERE IS A BIG FOCUS ON INDIVIDUALIZED LEARNING PLANS, AND MAKING SURE THAT NOT ONLY DO PROGRAMS EMBRACE THAT, BUT ALSO THAT THE INDIVIDUAL LEARNING PLAN IS ACTUALLY FUNCTIONING. I'M SORT OF INTERESTED IN YOUR THOUGHTS ABOUT INDIVIDUALIZED DEVELOPMENT PLANS, WHICH ARE INCORPORATING INDIVIDUALIZED LEARNING PLANS WITHIN A COMPETENCY BASED TRAINING MODEL. I DO THINK THAT ONE OF THE -- I DON'T WANT TO SAY COMPLAINTS, BUT THE FEEDBACK THAT WE GET FROM OUR ADVANCED TRAINEES IS THAT, LOOK, WE NEED ALSO SOME GUIDANCE WITH HOW TO MOVE OUR CAREERS FORWARD. IT'S NICE THAT, YOU KNOW, YOU OFFER US AN OPPORTUNITY TO HELP US ASSESS SPECIFIC LEARNING NEEDS BECAUSE WE THINK IT'S IMPORTANT, BUT WE NEED PROFESSIONAL DEVELOPMENT GUIDANCE. AND QUITE FRANKLY, I'VE ACTUALLY LOOKED AT YOUR IDP TEMPLATE FOR YOUR JUNIOR FACULTY AT NORTHWESTERN, AND REALLY LIKE IT. WOULD YOU SAY A FEW WORDS ABOUT IDPs, ILPs, HOW YOU YOU A PLY IT APPLY IT IN A EXE COMPETENCY BASIS. >> IT CAN BE SOMETHING YOU REQUIRE YOUR LEARNER TO ACTUALLY HAVE. THERE'S A LOT OF DI KRETION IN THE DISCRETION OF COMPETENCIES THAT YOU DEMAND IN YOUR TRAINING AND YOUR TRAINING BEYOND THE SIX AGME COMPETENCIES. SO YOU COULD EXPECT YOUR LEARNER TO CREATE AN INDIVIDUAL DEVELOPMENT PLAN AS ONE OF YOUR COMPETENCIES. AND YOU KNOW, I THINK -- I THINK IT WOULD BE VERY SIMILAR. YOU NEED TO HAVE THE RESOURCES, THE CURRICULUM PROVIDED FOR THAT TRAINEE TO BE ABLE TO UNDERSTAND THE DIFFERENT PATHWAYS HE OR SHE MAY GO DOWN. YOU NEED MENTORSHIP, YOU NEED A COACH TO CLEARLY HELP THAT TRAINEE. SO I'M NOT QUITE SURE THAT IT'S ALL THAT DIFFERENT, OR MAYBE I'M NOT UNDERSTANDING YOUR QUESTION TOTALLY WELL, BOB. >> WELL, I THINK IT'S MORE HOW DO YOU OPTIMIZE IT WITHIN A COMPETENT SEAL BASED COMPETENCY BASED DEVELOPMENT SYSTEM. I KNOW THAT THERE TENDS TO BE A FOCUS ON OKAY, LET'S DO THIS FROM THE POINT OF VIEW OF IDENTIFYING VERY, VERY TANGIBLE, VERY, VERY SPECIFIC LEARNING ACTIVITIES THAT RELATE TO THE LEARNER, WHICH IS A GOOD THING. THE REAL QUESTION IS, HOW DOES THE PROGRAM -- THE PROGRAM DIRECTOR, THE ENGAGED FACULTY AND THE LEARNER COMBINED REALLY SORT OF ENGAGE IN A PROCESS THAT ULTIMATELY IS GOING TO BE REFLECTED IN A DEVELOPMENT PLAN FOR THAT INDIVIDUAL? IT SEEMS TO ME THAT WE'RE NOT DOING AS GOOD ENOUGH JOB COLLECTIVELY IN FOCUSING ON THAT GOING FORWARD FOR OUR INDIVIDUAL TRAINEES. I DON'T KNOW IF YOU AGREE OR DISAGREE. >> WELL, I GUESS I'M HAVING A HARD TIME UNDERSTANDING WHY IT IS ANY DIFFERENT FROM ACHIEVING ANY COMPETENCY. BECAUSE IF IT'S THE APPLICATION OF CLINICAL KNOWLEDGE, YOU KNOW, OBVIOUSLY YOU NEED A PLAN TO GET TO THAT POINT THAT YOU EXPECT THE TRAINEE TO BE AT. SO IT'S A COMBINATION OF HAVING THE TRAINEE ENGAGE WITH WHATEVER CURRICULUM THAT IS, WHETHER IT'S A RESEARCH CURRICULUM, WHETHER IT'S A POLICY CURRICULUM, WHETHER IT'S TIME SPENT IN A MEDICAL EDUCATION CERTIFICATE PROGRAM, IF THAT SHOULD BE WHERE THEIR CAREER GOALS ARE HEADED. YOU HAVE TO GIVE THEM THE OPPORTUNITY TO EXPLORE THAT. THEN YOU HAVE TO HAVE THE OPPORTUNITY TO ACTUALLY HAVE THEM DEMONSTRATE SOME -- WHETHER IT'S, YOU KNOW, ACHIEVEMENT IN THAT PARTICULAR DOMAIN, PROBABLY LESS SO, BUT WHETHER IT'S MORE REFLECTION AND DECISION-MAKING ABOUT -- YOU DON'T WANT THEM TO JUST FLOUNDER, TASTING ALL OF THE DIFFERENT CAREER PATHS THEY MAY MAKE. WE NEED TO ACTUALLY SHOW YOU THAT YOU'RE GOING THEY'RE GOING TO MOVE I N ONE DIRECTION OR ANOTHER. AND THAT HAS TO BE GUIDED. THAT HAS TO BE GUIDED BY THEIR COACH. SO I'M SEEING THAT AS VERY SIMILAR AS THE ACHIEVEMENT OF REALLY ANY COMPETENCY. >> GREAT. AND THE FINAL QUESTION RELATES, AGAIN, TO TIME LIMITED VERSES TIME VARIABLE TRAINING. BECAUSE IN THE END, FOLKS HAVE TO BE CERTIFIED BY THE AMERICAN BOARD OF MEDICAL SPECIALTIES. THEY ALSO HAVE TO SORT OF MEET THE -- SORT OF THE TRAINING PLAN LAYOUT FROM THE ACGME. DO YOU SEE ANY MOVEMENT ON THE PART OF THESE ORGANIZATIONS TO ENGAGE IN COLLECTIVE COLLABORATIVE DISCUSSION ABOUT MOVING THIS PROCESS FORWARD BECAUSE IT DOES SEEM THAT YOU'RE GOING TO NEED LOTS OF BUY-IN IN ORDER TO GET THIS IMPLEMENTED IN AN EFFECTIVE WAY GOING FORWARD. >> THE TIME VARIABLE ONE, YOU MEAN? >> YES. >> UNFORTUNATELY, I DON'T THINK IT'S ONLY THE ABMS. YOU ALSO HAVE THE LICENSING BUREAU FEDERATION OF STATE LICENSE BUREAU, SO THAT HAS -- ON THE UME LEVEL, AND THERE -- AND THE GME LEVEL, ACGME STILL HAS SOME TIME-BASED REQUIREMENTS, CERTAIN TIME SPENT IN EACH TYPE OF ROTATIONAL SETTING OR CLINICAL SETTING. THOSE ARE ALL THINGS THAT ARE GOING TO HAVE TO DISAPPEAR. AND THIS COLLABORATIVE WORK AS YOU POINT OUT IS SO CRITICAL. I DO THINK THAT THE BOARDS WILL MOST LIKELY COME TOGETHER WHEN SOME OF THESE PILOTS ARE A LITTLE BIT FURTHER ALONG, AND WE START TO GET SOME EVIDENCE EVEN OUT OF CANADA IN TERMS OF SOME OF THE SUCCESS OF THESE PROGRAMS, AND THEN THAT WILL ALLOW -- THAT DATA WILL ALLOW US TO ACTUALLY FEEL A LITTLE LESS -- FEEL A LITTLE MORE SECURE IN ABOLISHING SOME OF THESE TIME REQUIREMENTS. IT'S COMPLICATED BECAUSE OBVIOUSLY THE OTHER ISSUES ARE THE FACT THAT YOU HAVE AN RMP AND AAMC HAVING A STAKE IN THIS, AROUND WHETHER OR NOT AND WHETHER OR NOT WE ALLOW FOR DIFFERENT ENTRY POINTS INTO TRAINING AT THE G GME AND AT THE FELLOWSHIP LEVEL, THERE'S A LOT OF SYSTEM REDESIGN THAT HAS TO HAPPEN. I DO THINK IT CAN HAPPEN BUT IT NEEDS A BIG LEADERSHIP LEVEL. SOMEBODY MAYBE EVEN HIGHER THAN ACGME. I DON'T KNOW WHO THAT IS. WHETHER THAT'S CONGRESS WHO MAKES A DECISION THAT THIS IS A -- OR SOME OTHER ORGANIZATION EVEN BIGGER, THAT -- NOT BIGGER THAN CONGRESS -- THAT GETS AT THIS IS THE DIRECTION WE NEED TO MOVE IN SO WE IMPROVE OUR HEALTHCARE SYSTEM. YEAH, IT'S COMPLICATED. I THINK THE BOARDS IS JUST ONE PIECE OF THE PUZZLE. >> GREAT. WELL, WE ARE OUT OF TIME, BUT THIS HAS BEEN A FANTASTIC PRESENTATION AND QUITE INFORMATIVE. THANK YOU VERY MUCH FOR TAKING THE TIME AND SHARING NOT ONLY YOUR INFORMATION BUT YOUR INSIGHTS WITH US. >> MY PLEASURE. >> HAVE A GREAT AFTERNOON. >> THANK YOU. BYE-BYE.