>> HI, WELCOME, EVERYONE. THANK
YOU SO MUCH FOR JOINING US
TODAY. I AM YOUR HOST FOR
TODAY'S EVENT, ORGANIZED BY
NATURAL LANGUAGE PROCESSING AND
TEXT MINING SPECIAL INTEREST
GROUP AT NIH. WE ARE EXTREMELY
EXCITED TO INTRODUCE OUR TODAY
SPEAKER, DR. ANDREW TAYLOR,
BOARD CERTIFIED EMERGENCY
PHYSICIAN AND INFORMATICIST AND
PROFESSOR AND VICE CHAIR OF
RESEARCH AND INNOVATION IN
DEPARTMENT OF EMERGENCY MEDICINE
AT UNIVERSITY OF VIRGINIA. DR.
TAYLOR IS THE DIRECTOR OF THE
CODE LAW, STANDS FOR THE
CLABORATORY FOR DIGITAL HEALTH
AND MEDICAL SQUARE AND HIS
DESIGN, EVALUATION AND
IMPLEMENTATION OF DIGITAL HEALTH
TOOLS IMPROVE DECISION MAKING
AND REDUCE DIAGNOSTIC ERROR IN
REAL WORLD CLINICAL
ENVIRONMENTS. I CANNOT WAIT TO
HEAR HIS IDEAS AND VISIONS
TOWARDS ACHIEVING A BETTER
HEALTH OUTCOMES. OUR TALK TODAY
WILL BE AROUND 40 MINUTES LONG,
FOLLOWED BY 10 MINUTES FOR Q&A.
FOR Q&A YOU CAN USE MY EMAIL,
DISPLAYED ON THE BOTTOM OF THE
SLIDE, OR YOU CAN USE THE
BUTTON, SEND LIKE FEEDBACK
BUTTON ON THE TOP LEFT OF YOUR
VIDEO CAST SCREEN AND SEND IN
QUESTIONS. WITHOUT FURTHER
ADIEU, PLEASE WELCOME DR. ANDREW
TAYLOR.
>> THANKS. LET ME GET THE SCREEN
SET UP HERE. JUST LET ME KNOW IF
YOU ALL CAN SEE.
>> WE CAN SEE.
>> JUST MINIMIZING THIS BOX AND
GET STARTED. GREAT SICHLT
AWESOME TO BE HERE TODAY. WE ARE
GOING TO TALK ABOUT LEVERAGING
LARGE LANGUAGE MODEL FOR
DIAGNOSTIC SAFETY AND QUALITY
IMPROVEMENT. I'M GOING TO KEEP
THE SCOPE FAIRLY NARROW AND
MOSTLY FOCUS ON QUALITY
IMPROVEMENT ASPECTS OF THIS.
THERE ARE NUMEROUS APPLICATIONS
OF LARGE LANGUAGE MODELS RELATED
TO DIAGNOSTIC SAFETY AND
QUALITY. WE JUST OBVIOUSLY DON'T
HAVE TIME TO COVER ALL SO WE
WILL KEEP IT A LITTLE LIMITED.
THE OTHER THING I WANTED TO SAY
IS I WILL PROBABLY TALK AT A
HIGH LEVEL, PROBABLY COVER MORE
OF THE CLINICAL ASPECTS OF
THINGS, BUT THEN WE WILL GET
INTO A LITTLE AROUND KIND OF THE
INFORMATICS, NLP ASPECT OF
THINGS AS WELL. SO YOU HAVE DONE
THE INTRODUCTIONS. I WILL
LARGELY SKIP OVER THIS. MY
RESEARCH IS IN DIGITAL HEALTH,
KIND OF ACROSS A VARIETY OF
DIFFERENT DOMAINS. REALLY I
THINK -- I'M PRIMARILY FOCUSED
-- OUR RESEARCH LAB IS FOCUSED
ON MAKING A DIFFERENCE. IN
GENERAL, THAT IS THROUGH TWO
MECHANISMS OF ONE IS REALLY
UNDERSTANDING PATIENTS AND
POPULATIONS THROUGH DATA SCIENCE
AND SUPPORT IDENTIFICATION AND
FURTHER INTERVENTIONS. THE
SECOND IS REALLY PROVIDING
DECISIONS THROUGH TRANSLATIONAL
AI. SO THIS IS THE GENERAL
OUTLINE. I WILL LEAVE HOPEFULLY
PLENTY OF TIME FOR QUESTIONS AND
EVERYTHING ELSE. WE WILL GO OVER
A LITTLE CONTEXT. I WANTED GIVE
YOU A FEEL FOR THE EMERGENCY
ROOM CONTEXT, BECAUSE THAT IS
WHAT WE ARE TALKING ABOUT MOST
OF WHAT WE ARE TALKING ABOUT. IF
YOU HAVEN'T BE BUT MOST HAVE SO
YOU PROBABLY HAVE A SENSE HOW
BAD IT IS. WE WILL TALK ABOUT
CURRENT DIAGNOSTIC SAFETY
PROGRAMS TO FAVOR MORE OF THE
CONTEXT AND TALK ABOUT USE OF
LARGE LANGUAGE MODELS FOR
DIAGNOSTIC SAFETY.LY KIND OF
WRAP UP WITH A LITTLE DISCUSSION
AROUND FUTURE RESEARCH AND JUST
CONCLUSIONS AND OPEN IT UP FOR
QUESTIONS. SO WE WILL START WITH
SOME CONTEXT. I LOVE THIS QUOTE
BY ANTHONY BORDAINE TALK ABOUT
CONTEXT AND MEMORY PLAY POWERFUL
ROLES IN ALL THE GREAT MEALS IN
ONE'S LIFE. I THINK IT IS REALLY
IMPORTANT TO KIND OF UNDERSTAND
THE CONTEXT IN EMERGENCY CARE TO
REALLY UNDERSTAND WHY OFTENTIMES
WE STRUGGLE I THINK WITH
DIAGNOSTIC ACCURACY AND
EXCELLENCE. SO THE FIRST THING
TO REALIZE IN THIS KIND OF
DOMAIN IS THERE IS AN INCREASE
IN RISING PATIENT COMPLEXITY AND
MULTI -MORBIDITY. I LOVE THIS
GRAPHIC THAT WAS DONE. IT REALLY
KIND OF SHOWS YOU HOW PATIENT
POPULATIONS HAVE REALLY STARTED
TO SHIFT AND CHANGE. AND WHAT WE
ARE SEEING WITHIN THE EMERGENCY
DEPARTMENT. SOT THE BASIC IDEAS
WE ARE SEEING PATIENTS WITH LOTS
MORE COMPLEXITY, LOT MORE
MULTI -MORBID CONDITIONS AND
FROM A DIAGNOSTIC STANDPOINT,
REALLY WHAT THAT COMES DOWN TO
IS A HUGE AMOUNT OF INFORMATION
OVERLOAD WHEN WE ARE MAKING
DECISIONS. THE CLASSIC EXAMPLE I
GIVE IN THIS SPACE IS, YOU KNOW,
WE HAVE KIND OF THESE RUNNING
USER-FACING COMPONENTS THROUGH
THE EHR DISPLAYING DIAGNOSE S
AND MEDICATIONS. COULD BE 30, 40
DIFFERENT PROBLEMS LONG. IT CAN
BE MIXED UP. YOU CAN HAVE
MEDICATIONS THAT ARE, YOU KNOW,
20-LONG AND OFTEN ARE POORLY
MANAGE ED FROM THE STANDPOINT OF
WHICH ARE ACTIVE AND -- (AUDIO
ISSUE) TO GET A FULL VIEW AROUND
WHAT PROBLEM THEY ARE COMING IN,
WHETHER THAT IS CHEST PAIN OR
SOMETHING ELSE. THEN THE NEXT
THING I WANTED TO KIND OF JUST
HINT AT IS JUST THE GENERAL
WORKFORCE SHORTAGE AND BURNOUT
BECAUSE IT IS WELL PUBLICIZED
WITHIN THE LAY PRESS. I THINK
THE LAST WELL-PUBLICIZED IS HOW
MUCH IT AFFECTS ALL OF THE
EMERGENCY DEPARTMENT WORKFORCE.
THAT IS FROM NURSING (AUDIO
ISS
ISSUE) AROUND ACCURACY. IF WE
HAVE TIME AT THE END THERE ARE
INTERESTING CONVERSATIONS AROUND
THE APPLICATION OF AI WITHIN THE
EMERGENCY -- WITHIN HEALTH CARE
AT-LARGE AND WHETHER THAT IS
GOING TO FEED INTO FURTHER
DISPARITIES AND CARE. A LOT OF
TIMES WHAT I WORRY ABOUT MOST IS
SMALL RURAL HOSPITALS THAT
AREN'T ABLE TO DO THE THINGS
LIKE I'M GOING TO SHOW YOU LATER
ON AND REALLY NOT BE ABLE TO
IMPLEMENT A LOT OF THE AI
SYSTEMS THAT CAN IMPROVE CARE IN
THE FUTURE. OBVIOUSLY VERY HUGE
ISSUES AROUND MENTAL HEALTH AND
SUBSTANCE ABUSE CRISISES. MOST
IS THOUGHT WITHIN THE EMERGENCY
DEPARTMENT. A LOT OF THOSE
FACTORS ARE THEN FEEDING INTO
SIGNIFICANT CHALLENGES WITH
CROWDING AND BOARDING AND
THROUGH-PUT CHALLENGES IN THE
EMERGENCY DEPARTMENT SPACE. YOU
HAVE LOTS OF FINANCIAL PRESSURES
FROM THE HOSPITAL ENVIRONMENT.
WORKFORCE SHORTAGES. FROM
BURNOUT AND OTHER ISSUES. THEN,
YOU KNOW, LIMITED BED SPACE,
FAMILY STRESS, WORKPLACE
VIOLENCE OCCURRING. YOU CAN
THINK ABOUT YOU HAVE THIS KIND
OF CRAZY MORE CHAOTIC EMERGENCY
DEPARTMENT THAT'S BEEN
TRADITIONALLY IN THE PAST. THEN
WE RUN INTO THIS YEAR AROUND
DIAGNOSTIC ERRORS. THIS IS A
NUMBER THROWN OUT FROM AN HRQ
REPORT. I HAVE SEEN VARYING
NUMBERS WHEN COMES TO OVERALL
KIND OF DIAGNOSTIC ACCURACY AND
ERROR RATES WITHIN THE EMERGENCY
DEPARTMENT. YOU KNOW, ANYTHING
BETWEEN 1% AND 10%. THEN I THINK
THERE ARE A LOT OF ONGOING
STUDIES TO REALLY KIND OF NARROW
DOWN THIS NUMBER TO FIGURE OUT
WHAT THAT ACTUALLY NUMBER IS.
BUT IT IS NOT FIGURED OUT. AND
WE ARE MAKING DIAGNOSTIC ERRORS
IN THE DEPARTMENT ON A REGULAR
BASIS. THE LAST PART I WANT TO
GIVE YOU, JUST AROUND SOME
CONTEXT IS THINKING ABOUT
EMERGENCY CARE AS A CONTINUUM. I
LOVE THIS GRAPHIC FROM THE WHO
BECAUSE IT I THINK SHOWS THE
CARE JOURNEY OR PROCESS FOR
PATIENTS. I ALSO THINK IT
HIGHLIGHTS POTENTIAL
OPPORTUNITIES FOR AI AND LARGE
LANGUAGE MODELS AND CHATBOTS,
FROM STARTING AT A PARTICULAR
SCENE AND HAVING INTERACTIONS
WITH EMS AND BYSTANDERS AT THE
SCENE AND ISSUES WITHIN THE
EMERGENCY DEPARTMENT AND ALL THE
CARE POST EMERGENCY DEPARTMENT.
I WON'T TALK ABOUT IT TODAY BUT
I THINK THERE ARE REAL
OPPORTUNITIES FOR BETTER DATA
COLLECTION THROUGH LARGE
LANGUAGE MODEL APPLICATIONS AND
CHATBOT APPLICATIONS IN HELPING
PATIENTS NAVIGATE THAT CARE
PROCESS AND ALSO FROM A
FOLLOW-UP STANDPOINT. THE MAIN
THING, WHAT YOU WANT TO TAKE
AWAY, IS REALLY THINK ABOUT THE
WHOLE KIND OF EMERGENCY CARE
PROCESS IS REALLY A CONTINUUM IN
THIS SPACE. I WILL SWITCH GEARS
A LITTLE BIT. WE WILL TALK ABOUT
DIAGNOSTIC SAFETY. I'M GOING TO
FRAME IT FROM KIND OF THE
NEGATIVE ASPECT FROM DIAGNOSTIC
ERROR. SO THIS IS A DEFINITION
FROM THE NATIONAL ACADEMY OF
MEDICINE, THIS IS FAILURE TO
ACCURATELY DIAGNOSE A PATIENT'S
HEALTH PROBLEMS OR COMMUNICATE
THAT INFORMATION TO THE PATIENT.
I THINK THE SECOND PIECE IS
PARTICULARLY INTERESTING.
OFTENTIMES WE DON'T THINK ABOUT
THE COMMUNICATION PART THAT
REALLY NEEDS TO BE CONVEYED TO
THE PATIENT, NOT BE CONSIDERED A
DIAGNOSTIC ERROR. WITHIN KIND OF
THE DIAGNOSTIC ERROR SPACE,
THERE ARE MULTIPLE DIFFERENT
SUB-COMPONENTS. MISDIAGNOSIS.
PROBABLY THE TERM THAT WAS MOST
FREQUENTLY USED IN THE PAST WERE
CONDITIONS NOT IDENTIFIED AT
ALL. I MIGHT CHANGE THAT OR
MODIFY THAT A LITTLE TO SAY THE
CONDITION IS NOT IDENTIFIED
POINT WHERE INTERVENTION WOULD
HAVE BEEN HELPFUL. OTHERWISE IT
WOULD BE -- WE WOULDN'T KNOW IT
WAS A DIAGNOSIS TO HAVE TAKEN
PLACE. THEN THERE ARE WRONG
DIAGNOSES OR INCORRECT LABELS
APPLIED TO PATIENTS. MOST I
THINK OF THE LITERATURE WHEN WE
ARE TALKING ABOUT DIAGNOSIS AND
DIAGNOSTIC ERROR AT THIS POINT
IS REALLY FRAMED AROUND THIS
IDEA OF DELAYED DIAGNOSIS. I
WILL GIVE YOU AN EXAMPLE OF
THAT. SO THIS IS THE KIND OF
EXAMPLE I WILL GIVE THROUGHOUT
THE TALK. SO A PATIENT COMES IN
TO THE EMERGENCY DEPARTMENT.
THEY HAVE ABDOMINAL PAIN. THEY
ARE SEEN, THEY ARE EVALUATED,
THE PHYSICIAN THINKS ULTIMATELY
THAT LET'S SAY IT IS GASTRO
INTERITIS, THEY DISCHARGE AND
THREE DAYS LATER SHOW UP AT
PRIMARY CARE DOCTOR, HAVE PIPE,
THE PRIMARY CARE ORDERS THE CT,
THIS IS PERFORMED, THEY HAVE
APPENDICITIS AND NEED TO COME
BACK INTO THE HOSPITAL SYSTEM
THAT. WOULDN'T FALL UNDER A
MISDIAGNOSIS UNDER THIS SCENARIO
BUT DELAY THAT HAD INTERVENTION
OF SURGERY AFTER. SO A LOT OF
KIND OF DIAGNOSTIC ERRORS,
SAFETY SPACE IS REALLY FRAMED
FROM THAT STANDPOINT. THE OTHER
ASPECT OF THIS, JUST TO KEEP IN
MIND, IS WHETHER OR NOT THERE IS
THE POTENTIAL FOR HARM AND HOW
THAT FACTORS IN TO OPPORTUNITIES
AND MISSED OPPORTUNITIES WITHIN
THIS SPACE. OKAY. SO NOW WE HAVE
A LITTLE BIT OF KIND OF
BACKGROUND CONTEXT AROUND THIS.
SO I THINK WHAT I WANTED TO DIVE
INTO IS REALLY WHAT ARE THE
CURRENT PROBLEMS WITH DIAGNOSTIC
QUALITY IMPROVEMENT? WHY ARE WE
HAVING CHALLENGES? WHY IS THAT
NUMBER 6%, NOT MUCH LOWER. THE
THINGS I CAN REALLY BOIL IT DOWN
TO ARE THESE THREE COMPONENTS.
ONE IS WE HAVE A SYSTEM OF PO
CAPTURE. I WILL GO INTO THESE.
THE SYSTEMS AREN'T REALLY
ESCAPABLE AT THIS POINT. WE
DON'T DO A GREAT JOB OF
MEASURING THEM WHEN WE DO
CAPTURE THEM IN SOME WAY. I HAVE
BEEN IN AND THESE ARE THE TWO
PRIMARY FRAMEWORKS WHERE
DIAGNOSTIC ERRORS ARE NOTICED
WITHIN THE SYSTEM. FOR THE
EMERGENCY DEPARTMENT, MOST QI
PROGRAMS ARE GOING TO HAVE SOME
KIND OF 72-HOUR RETURNED A
ADMISSION SURVEILLANCE AND ARE
DISCHARGED, ADMITTED, CREATES A
FLAG, REVIEW THE POTENTIAL CASES
AND LOT OF US ARE NOT GOING TO
HAVE ANYTHING TO DO WITH
DIAGNOSTIC SAFETY OR ERRORS SO A
LOT OF NOISE FROM A DIAGNOSTIC
SAFETY STANDPOINT. THE OTHER IS
MANUAL REPORTING. THE ANNUAL
AROUND THE APPENDICITIS IS A
COMMON SCENARIO, WHERE IT IS
NOTICED WITHIN THE SYSTEM BY
SOME OTHER PROVIDER. THAT ERROR
PROCESS NEEDS TO BE THEN
COMMUNICATED BACK TO THE PRIMARY
TEAM OR DEPARTMENT. SO THAT MAY
OR MAY NOT OCCUR. THERE ARE
SYSTEMS THAT ALLOW FOR
ELECTRONIC REPORTING. THOSE
PROCESSES ARE IN PLACE BUT
INFREQUENTLY USED SO IN GENERAL
WE HAVE PRETTY POOR CAPTURE WHEN
COMES TO THESE WHAT I CALL A
NEAR KIND OF REAL-TIME BASIS
WITHIN A HEALTH CARE SYSTEM. THE
SECOND PART IS THE QI SYSTEMS
ARE NOT SCALEABLE SO YOU THINK
ABOUT THIS. A LOT WILL BE
FOCUSED ON ONE DISEASE PROCESS.
YOU MAY HAVE ROLLED OUT WITHIN
YOUR HOSPITAL SYSTEM SOMETHING
AROUND MISSTROKED DIAGNOSIS. SO
IT TYPICALLY IS VERY
DISEASE-ORIENTED. THESE ARE THE
WAYS THEY ARE CONSTRUCTED.
SOMETIMES NEEDS TEAM ORIENTATION
AROUND THIS AND YOU NEED
DIFFERENT STRUCTURES AROUND
DISEASE PROCESSES. THIS IMPACTS
THE QI SYSTEMS THAT ARE THERE. I
THINK AT THIS POINT WHEN WE WILL
GO INTO A LITTLE MORE DETAIL
AROUND SOME ELECTRONIC WAYS OF
CAPTURING THESE DIAGNOSTIC
ERRORS, IF YOU KIND OF OPENED
THE FLOOD GATES AND APPLIED MORE
SIMPLISTIC PROCEDURES TO
CAPTURING DIAGNOSTIC ERRORS
WITHIN THE SYSTEM YOU WOULD GET
THIS FLOOD OF WAY TOO MUCH NOISE
WITHIN THE SYSTEM. YOU WOULD
OVERWHELM QI TEAMS. SO YOU KIND
OF LOSE A LOT OF THAT DIAGNOSTIC
ERRORS BACK, UPDATING TEAM
STRUCTURES AND PROCESSES KIND OF
WITHIN THE SPACE TO IMPROVE
OUTCOMES. IF THOSE SYSTEMS
AREN'T ESCAPABLE YOU AREN'T ABLE
TO KIND OF LEARN BOTH SYSTEM
LEVEL ISSUES AND MAYBE PROVIDER
LEVEL ISSUES IF YOU ARE TALKING
ABOUT JUST A FEW DIAGNOSIS PER
PROVIDER PER YEAR. THEN THE LAST
PART ABOUT THIS IS THAT EVEN IN
THOSE SCENARIOS LOT OF TIMES
THEY ARE NOT WELL MEASURED SO
GOING BACK TO THAT INITIAL
EXAMPLE IF THE APPENDICITIS
OCCURS, IT GETS REPORTED BACK,
THERE MAY BE AN EMAIL EXCHANGE
BETWEEN -- TO DEPARTMENTAL
OPERATIONAL LEADERSHIP AND THE
PROVIDER REGARDING THE ERROR
THAT MAY NOT BE SYSTEMATICALLY
CAPTURED IN ANY STANDARDIZED
WAY. A LOT OF TIMES THERE IS NOT
GOING TO BE A PARTICULAR SCORING
SYSTEM AROUND THAT OR ANY TYPE
OF GRADING WHERE YOU CAN KIND OF
LEARN ADDITIONAL COMPONENTS OR
ASPECTS FROM THOSE TYPES OF
CASES. SO THERE ARE SOME SYSTEMS
THAT ARE IN PLACE THAT ARE A
LITTLE AT A HIGHER LEVEL OF
SCALE THAN WHAT I KIND OF
MENTIONED BEFORE. PROBABLY THE
MOST WELL KNOWN ARE E-TRIGGERS
FOR DIAGNOSTIC SURVEILLANCE.LY
TAKE TIME TO WALK THROUGH THIS
PROCESS. E-TRIGGERS ARE A
LOOK-BACK MECHANISM FOR
IDENTIFYING DIAGNOSTIC ERRORS.
SO SIMILAR TO THE EXAMPLE I
GAVE, I WILL KIND OF I'LL KIND
OF FRAME IT AROUND THAT. THE
BASIC ED IS IS YOU IDENTIFY
DIAGNOSIS THAT HAPPENED WITHIN A
SYSTEM. LET'S SAY A PULMONARY
EMBOLISM HAPPENED IN THE SYSTEM.
THEN YOU WOULD LOOK BACK IN TIME
TO SEE IF AN EMERGENCY
DEPARTMENT VISIT OR, YOU KNOW,
COULD BE A PRIMARY CARE OR
HOSPITAL VISIT OCCURRED. THEN
THERE NEEDS TO BE SOME TYPE OF
FLAG OR SYSTEM TO SAY WHETHER
THERE WAS A POTENTIAL
MISDIAGNOSIS ON THAT VISIT.
SOMETIMES THAT IS DONE THROUGH
SYMPTOM DISEASE PAIRS. FOR
INSTANCE, FOR PULMONARY
EMBOLISM, YOU LOOK BACK AND THE
EMERGENCY DEPARTMENT VISIT HAD A
CHIEF COMPLAINT OF SHORTNESS OF
BREATH OR CHEST PAIN THAT.
MIGHT BE THE TRIGGER FOR YOUR QI
TEAM TO DO FURTHER REVIEW WITHIN
THAT. SO YOU CAN, YOU KNOW,
THINK ABOUT MULTIPLE CHALLENGES
WITH THIS. ONE IS THAT THE
DIAGNOSIS THAT'S OCCURRING, IT
IS SOMETIMES DIFFICULT TO
IDENTIFY, YOU KNOW, TO GET AN
ACCURATE PHENOTYPE THAT THERE IS
AN ACCURATE DIAGNOSIS THAT IS
HAPPENING. FOR THE PULMONARY EM
BOLL LIS M, COULD BE A PATIENT
HAD A CHRONIC EMBOLISM AND FOR
BILLING PURPOSES OR OTHER GOT
PLACED ON A PROBLEM LIST WITHIN
THE HOSPITAL STAY, THAT THAT
DIAGNOSIS SHOWS UP AGAIN BUT IT
IS NOT A NEW DIAGNOSIS. SO
THERE'S CHALLENGES ABOUT GETTING
TO ACCURATE REPRESENTATIONS OF
NEW DIAGNOSES THAT ARE HAPPENING
WITHIN THE SYSTEM. ONE NATURAL
LANGUAGE PROCESSING APPLICATION,
OR IN THIS CASE MOSTLY LARGE
LANGUAGE MODEL APPLICATIONS,
WOULD BE LOOKING AT DIAGNOSTIC
RADIOLOGY NOTES WHERE MOST THESE
DIAGNOSES AT LEAST IN EMERGENCY
CONTEXT ARE GOING TO BE
HAPPENING. AND GETTING TO
ACCURATE REPRESENTATIONS OF
WHETHER THEY ARE ACUTE OR
CHRONIC PROCESS THAT IS ONGOING
THERE AND LABELING THE DIAGNOSIS
FROM THAT AS OPPOSED TO USING
DIAGNOSIS CODES. BUT THERE ARE
OBVIOUSLY ISSUES THAT ARE --
EVEN WITH KIND OF THAT SPACE AS
WELL. SO THAT IS, YOU KNOW, ONE
KIND OF CLASSIC ISSUE AROUND
THESE TRIGGER TOOLS. THE SECOND
PART IS THAT KIND OF NOISE
ASPECT WHEN YOU LOOK BACK. YOU
KNOW, A LOT OF QUALITY
IMPROVEMENT TEAMS, YOU ARE
ASKING -- OR EMERGENCY
PHYSICIANS, PRIMARY CARE
DOCTORS, WHATEVER IS REALLY PART
OF THAT DEPARTMENT THAT IS
LOOKING INTO DIAGNOSTIC SAFETY,
YOU ARE ASKING THEM TO GO
THROUGH CHARTS AND SPEND
CONSIDERABLE TIME TO CREATE OR A
RESOURCE ASK ON YOUR SYSTEM SO
YOU WANT TO MAKE SURE YOU ARE
PRETTY ACCURATE WITH YOUR
INITIAL TRIGGER IN
IDENTIFICATION OF THE CASE
THAT'S BEING PUSHED TO THE QI
TEAM. SO APPEARS SYMPTOM DISEASE
PAIR IS PROBABLY NOT GOING TO
WORK WELL FOR MOST OF THE CASES
WHERE YOU ARE GOING TO GET
EXCESSIVE AMOUNT OF NOISE WITHIN
THE SYSTEM FROM THAT. NOW YOU
CAN THINK ABOUT THESE TRIGGERS
FURTHER DEVELOPED USING AI. SO
INSTEAD OF DISEASE SYMPTOM PAIR
NOW WE WILL BUILD OUT AI MODELS
FOR EACH DIAGNOSIS SO WE WILL
TRAIN MODEL AT THE TIME OF THE
INIT
INITIAL E. D. DIAGNOSIS TO
DETERMINE IF THEY HAVE A
PARTICULAR CONDITION THAT. GETS
CHALLENGING BECAUSE OF WHOEVER
THE OUTCOME IS PRESENT OR NOT
AND HOW THAT GETS TRAINED, THOSE
ARE POTENTIAL OPTIONS FOR
IMPROVING THE INITIAL
IDENTIFICATION AND SENSITIVITY
AND SPECIFICITY AROUND SOME OF
THOSE ALGORITHMS. ALL OF THIS
FITS INTO THIS LARGER KIND OF
TECHNICAL WORK SYSTEM WHERE WE
ARE REALLY KIND OF THINKING
ABOUT FEEDBACK MECHANISMS AND
LOOPS. YOU MAY BE ASKING, YOU
ARE DOING TRIGGER SYSTEMS LIKE
WHAT IS THE ULTIMATE END GOAL. A
LOT OF THIS AT THIS POINT IS
ABOUT FEEDBACK MECHANISMS TO
PROVIDERS AND, YOU KNOW, I THINK
THAT HAS BEEN WHERE A LOT OF THE
KIND OF TAILORED RESPONSES FOR
DIAGNOSTIC SAFETY HAVE TAKEN
PLACE THERE ARE A LOT OF FOCUS
ON COGNITIVE BIAS WITH PROVIDERS
AND CHANGES TO THEIR APPROACHES
TO PATIENTS. SO THINKING ABOUT
IT FROM A SYSTEMS ASPECT OF, YOU
KNOW, ARE THERE ASPECTS OF THE
OVERALL ORGANIZATIONAL STRUCTURE
THAT MAY BE CONTRIBUTING TO
DIAGNOSTIC ERRORS. SO CLEAR
QUESTIONS WOULD BE IS BOARDING
AND CROWDING CONTRIBUTING OR
CAUSING DIAGNOSTIC ERRORS WITHIN
THE SYSTEM. YOU CAN'T REALLY DO
THAT UNLESS YOU ARE KIND OF
ALREADY AT SCALE ACROSS MULTIPLE
DIAGNOSES TO BE ABLE TO MEASURE
THOSE ASPECTS OF KIND OF THE
DIAGNOSTIC PROCESS. SO HOW CAN
LARGE LANGUAGE MODELS AND NLP
KIND OF HELP? THERE'S A COUPLE
OF DIFFERENT WAYS. THE FIRST IS
REALLY THINKING ABOUT SCALING
ACROSS DISEASE. SO, YOU KNOW,
WHEN WE TALK ABOUT THAT KIND OF
TRIGGER SYSTEM AND YOU ARE
EITHER USING DISEASE SYMPTOM
PAIRS, EVEN FOR THAT YOU NEED TO
DEVELOP OUT A FULL LIST OF
DISEASE SYMPTOM PAIRS FOR
WHATEVER DIAGNOSES YOU HAVE OF
PARTICULAR INTEREST. IF YOU ARE
GOING TO BUILD OUT AI MODELS
LIKE STANDARD MACHINE LEARNING
MODELS, YOU WOULD NEED TO
POTENTIALLY BUILD OUT MULTIPLE
DIFFERENT MODELS FOR ALL OF THE
DISEASE PROCESSES YOU ARE
INTERESTED AT AND -- BUT I THINK
WITH A LOT OF THE IN-BUILD
KNOWLEDGE COMPONENTS LARGE
LANGUAGE MODELS HAVE IT IS MUCH
EASIER TO FEED IN A PARTICULAR
DIAGNOSIS AND SAY OKAY, THIS WAS
THE DIAGNOSIS SEVEN DAYS LATER.
NOW I NEED YOU TO KIND OF LOOK
BACK AT THE PRIOR VISIT AND SEE
WHETHER OR NOT THE PRESENTATION
WAS CONSISTENT. WE WILL TALK
ABOUT THAT MORE IN JUST A
MINUTE. IT THEN ALLOWS THAT
GENERALIZATION OF THE SCREENING
ON THE PRIOR VISIT SO WE ARE
ABLE TO TAKE IN A LOT OF
DIFFERENT CONTEXT FROM VARIOUS
ASPECTS OF THE ELECTRONIC HEALTH
RECORD. WE CAN PULL IN NOTES,
STRUCTURED INFORMATION ALL IN
KIND OF A MULTI -MODAL FASHION
TO DO THE INITIAL SCREENING. IF
WITHIN THE PROMPT STRUCTURE -- I
WILL GO INTO MORE DETAIL ABOUT
THIS, YOU CAN THINK ABOUT
CREATING BETTER SCORING
MECHAN
MECHANISMS FOR DEEPER INSIGHTS
INTO WHY POTENTIALLY THERE WAS A
MISDIAGNOSIS, WHAT ASPECTS WERE
MISSED, WAS THERE A LACK OF
IMAGING, ORDERING, ALL THOSE
COMPONENTS. I THINK LAST PART
IS, YOU KNOW, FROM A LONG-TERM
FLEXIBLE ADAPTIVE COMPONENT OF
THESE LARGE LANGUAGE MODELS ARE
CHANGING, THEY ARE VERY FLEXIBLE
IN WHAT THEY CAN DO, HOW YOU CAN
CHANGE THEM FROM ONE PROCESS TO
ANOTHER. SO JUST TO KIND OF GIVE
YOU A FLAVOR FOR THIS WHEN I WAS
TALKING ABOUT -- THIS IS KIND OF
HOW WE HAVE SET UP OUR SYSTEMS
TO HELP WITH THIS. YOU CAN THINK
ABOUT KIND OF MULTI MODAL
CLINICAL DATA COMING IN, IN THE
EMERGENCY CONTEXT FROM AN NLP
ASPECT. MOST OF THE KIND OF
EMERGENCY CONTEXT IS COMING FROM
PROVIDER NOTES AND NURSING NOTES
BUT YOU COULD THINK ABOUT
PULLING IN -- IF IT WAS A CHEST
PAIN PATIENT OR MISSED MI,
PULLING IN MOST RECENT
CARDIOLOGY NOTES AND OTHER
AREAS. INJECTING STRUCTURE
INFORMATION LIKE LABORATORY
RESULTS AND VITAL SIGNS. THAT
PROVIDES YOUR OVERALL CONTEXT
FUSION THAT YOU ARE ABLE TO
FINALLY MAKE POTENTIALS ON TOP
OF THAT. SO WE'VE KIND OF --
THIS IS AN EARLIER VERSION OF
OUR KIND OF GENERAL PROMPT
STRUCTURE.LY GO INTO DIFFERENT
COMPONENTS OF THIS. THIS IS THE
OVERALL OUTLINE OF WHERE WE
INITIALLY STARTED. SO WE HAVE
SOME DETAILED INSTRUCTIONS. I
WILL SHARE THOSE IN A MINUTE.
THEN WE ARE APPLYING A SET OF
CRITERIA, CALLED A SAFER TX
CRITERIA, THAT HAS 11
COMPONENTS. REALLY LESS THAN
THAT. BUT THE NUMBER OF
INSTRUCTIONS ARE 11. AND THEN
THEY ARE -- WITHIN EACH OF THOSE
CRITERIA YOU HAVE POINTS ABOUT
WHETHER THERE IS AGREEMENT OR
NOT AGREEMENT FOR THAT
PARTICULAR ASPECT OF IT. THEN WE
WOULD INJECT NOTES WITHIN THE
PROMPT ITSELF. THEN GIVE THE
SYSTEM THE FINAL DIAGNOSIS. I
WILL SHOW YOU EACH OF THESE IN
TURN. SO I WON'T READ THROUGH
THIS, BUT YOU CAN JUST KIND OF
GET A GENERAL SENSE OF THE WAY
INSTRUCTIONS WERE. IT JUST KIND
OF LAYS OUT WHAT'S GOING TO BE
ASKED OF THE SYSTEM MOVING
FORWARD AND APPLYING THE SCORE
WITHIN THAT AND COMING UP WITH
KIND OF AN EVENTUAL ASSESSMENT
ABOUT A MISSED DIAGNOSTIC
OPPORTUNITY OR NOT THAT. IS THE
INITIAL FRAMING. THESE ARE THE
CRITERIA COMPONENTS. AGAIN, THIS
IS ONE OF THOSE ASPECTS THAT
ALLOWS US TO BE ABLE TO HAVE A
LITTLE MORE INSIGHT INTO THIS
KIND OF OVERALL SUB COMPONENT
THAT MAY BE DRIVING DIAGNOSTIC
ERRORS IN THESE DIFFERENT
CONTEXTS. THEN FOR THIS, WHEN WE
STARTED OUT, WE WERE BOTH
INJECTING IN THE ORIGINAL
VISITS. WHEN YOU DID THE
LOOKBACK PERIOD, WE WERE PUSHING
IN THOSE NOTES. THEN ALSO GIVING
THE FOLLOW-UP CONTEXT AROUND
THAT SO WHEN THE DIAGNOSIS
OCCURRED. I MENTIONED
APPENDICITIS CASE, WE WOULD THEN
LOOK AT THE NOTE ASSOCIATED WITH
THE PRIMARY CARE DOCTOR AT TIMES
TO GIVE ADDITIONAL CONTEXT
AROUND WHAT HAPPENED IN THE
MEANTIME. WITH THE IDEA BEING IF
YOU ARE AT A SEVEN-DAY WINDOW
AND LIKE THE PATIENT HAD FOUR
DAYS WITHOUT ANY PAIN AND FELT
PERFECTLY FINE, THEN ALL OF A
SUDDEN STARTED DEVELOPING
WORSENING PAIN AGAIN, MAKES IT
POTENTIALLY LESS LIKELY THE
INITIAL DIAGNOSIS WAS
APPENDICITIS. SO HAVING SOME OF
THAT CONTEXT HELPS. THEN WE FEED
IN THE FINAL DIAGNOSIS SO THE
LLM ALREADY KNOWS, NOT GOING DE
NOVO, IS TRYING TO MAKE
ASSESSMENT ABOUT THE INITIAL
VISIT AND WHETHER THERE WAS A
POTENTIAL MISSED DIAGNOSTIC
OPPORTUNITY WITHIN THAT SPACE.
WHERE DOES KIND OF ALL THAT GET
US? SO OVERALL IF YOU ARE
THINKING ABOUT THIS IN BUILDING
OUT KIND OF A FULL QI SPACE,
THIS IS KIND OF WHERE WE ARE
MOVING TOWARDS. SO INSTEAD OF
REALLY HAVING THE SYMPTOM
DISEASE PAIR OR SIMPLE LOOK-BACK
WE ARE USING THE LLMs TO DO THE
ASSESSMENT FOR THOSE PRIOR SIS
SITS. IT ALLOWS US TO BE ABLE TO
SCALE FROM ONE OR TWO DISEASE
PROCESSES TO -- WE HAVE USED A
LIST OF ABOUT 200. THAT IS JUST
BASED UPON HIGH ACUITY,
CAN'T-MISS DIAGNOSES WITHIN THE
EMERGENCY DEPARTMENT. THE IDEA
IS YOU WOULD IDENTIFY THOSE WITH
STRONG POTENTIAL FOR MISSED
DIAGNOSTIC OPPORTUNITY. THOSE
THEN ARE IN THAT LIST SO YOU ARE
NOT OVER-BURDENING YOUR QI TEAM
IN THIS PROCESS. I'M NOT SURE IF
WE WILL GET TO A SYSTEM WHERE
ATTRIBUTION OF A DIAGNOSTIC
ERROR IS FULLY AUTOMATED WITHIN
AN AI SYSTEM -- FROM JUST A
FEEDBACK MECHANISM, IT MAY BE
DIFFICULT TO GET ADAPTIVE
CHANGES IN PROVIDER S IF YOU ARE
ONLY USING AN AI SYSTEM AS
OPPOSED TO HAVING OTHER HUMANS
IN THE LOOP WITH THAT PROCESS.
SO THAT IS KIND OF WHERE WE ARE
NOW. THE NICE THING ABOUT THIS
IS BECAUSE YOU ARE MOVING OUT
REALLY TO SCALE AND LOOKING AT
HUNDREDS OF DIFFERENT DIAGNOSES
ACROSS THE ENTIRE SYSTEM, YOU
ARE ABLE TO BUILD OUT QUALITY
DASHBOARDS WITH A LOT MORE I
THINK DEPTH THAN YOU OTHERWISE
WOULD BE ABLE TO. SO HOW ARE WE
KIND OF DOING THIS? I SHOULD SAY
SO I KIND OF SIT IN TWO WORLDS
RIGHT NOW. I WAS RECENTLY AT
YALE, NOW AT THE UNIVERSITY OF
VIRGINIA. WE ARE DOING THAT
PROCESS AT BOTH THESE
INSTITUTIONS BUT THIS KIND OF
GIVES YOU A BASIC SETUP OF HOW
THIS PROCESS IS HAPPENING. I
WILL SAY THIS WAS TAKEN FROM
SOME OTHER WORK WE HAD AROUND
OPIOID USE DISORDER AND
PHENOTYPE INFORMATION BUT THE
INFRASTRUCTURE IS RELATIVELY THE
SAME. SO WE ARE USING A COME POE
TUIT
COMPUTATIONAL HEALTH PLATFORM
USING INFORMATION FROM CLARITY
AND KABOODLE WITHIN EPIC,
UPDATED WITHIN A 24-HOUR BASIS
THAT. IS IMPORTANT IN THINKING
THIS. WE HAVE NOT MOVED TO A
SYSTEM WHERE YOU HAVE AUTOMATIC
IDENTIFICATION OF DIAGNOSTIC
ERROR. A LOT OF REASONS THAT
WILL BE AT LEAST CURRENTLY
DIFFICULT IS BECAUSE DOCUMENTARY
OFTENTIMES DOES NOT OCCUR UNTIL
AFTER VISITS HAVE BEEN COMPLETED
AND SOMETIMES MULTIPLE HOURS
AFTER WARDS. WE WOULD LOVE IT IF
ALL THE DOCUMENTATION WAS DONE
PRIOR TO PATIENTS LEAVING, BUT
THAT IS JUST NOT THE REALITY.
THAT MAY CHANGE AS WE SEE MORE
AND MORE MBAI INTRODUCE BUD
LIMITS YOUR ABILITY TO DO TRULY
REALTIME ASSESSMENT FOR
POTENTIAL MISSED DIAGNOSTICS IN
THE SPACE SO WE ARE REALLY
WORKING ON A 24-HOUR WINDOW POST
THESE VISITS. BUT THAT
INFORMATION IS PROCESSED WITHIN
THE COMPUTATIONAL HEALTH
PLATFORM. FOR BOTH ENVIRONMENTS
WE ARE WORKING WITHIN
ACCOMODATIVE MODEL, O█MOF, THEN
RUN SCRIPTS FOR FINDING NOTES
AND OTHER STRUCTURED
INFORMATION. THERE IS AN
OPPORTUNITY TO PUSH THAT DATA
BACK INTO EPIC IN A COUPLE OF
WAYS. ONE IS THROUGH DATALINK.
BUT RIGHT NOW THE MAIN MECHANISM
WE ARE GOING TO WORK KIND OF
BUILDING IN FROM THE QI
STANDPOINT IS THAT AREA THAT I
KIND OF MENTIONED BEFORE,
BUILDING OUT DASHBOARDS AND
IDENTIFICATION FOR THE QI TEAMS
MOVING FORWARD AFTER WARDS. THE
OTHER THING, WITHIN THE EL
SYSTEM THERE'S A COUPLE OF
MODELS THAT ARE WITHIN THE
SYSTEMS. (?) IS ONE. THERE IS
ALSO AN OPPORTUNITY FOR USING
OPEN AI WITHIN AZURE
ENVIRONMENT. THAT IS THE SAME,
AT LEAST AZURE IS SAME AT EVA AS
WELL. SO WHAT ARE SOME OF THE I
THINK BROADER ISSUES AND
CHALLENGES? ONE IS THE OVERALL
SYSTEM TO BE ABLE TO WEIGH IN ON
POTENTIAL FOR MISSED DIAGNOSIS.
OVERSIGHT OF THAT SYSTEM. SO WE
GO FROM ONLY FINDING POTENTIALLY
A FEW MISSED DIAGNOSES, AND MOST
OF THE TIME WHEN I TALK TO I. T.
TEAMS THEY SAY THEY GET THREE,
FOUR A MONTH, MAYBE LESS, WHICH
IS NOT CONSISTENT WITH WHAT YOU
SEE FROM A RESEARCH STANDPOINT,
SO WE KNOW THERE'S A LOT OF
UNDERREPORTING. IF THAT NUMBER
GREATLY EXPANDS, FIGURING OUT
HOW YOU ARE GOING TO OVERSEE AND
MANAGE THAT BECOMES AN ISSUE,
ALSO BECOMES AN ISSUE FROM A
RESOURCE STANDPOINT. THIS ALSO I
THINK FEEDS BACK INTO ISSUES
AROUND, YOU KNOW, EQUITY AND
DISPARITIES AT THE SYSTEM LEVEL.
WE TALK ABOUT IMPLEMENTING THESE
TYPES OF AI SYSTEMS WITHIN RURAL
HOSPITALS, ET CETERA. THEN THE
LAST IS REALLY STANDARDIZING A
LOT OF THIS PROCESS, SO YOU MAY
HAVE DIFFERENT GROUPS CHOOSING
DIFFERENT SETS OF DIAGNOSES OR
USING DIFFERENT SCORING CRITERIA
AND WAYS OF DOING REPORTING
AROUND THIS. SO I THINK THERE
NEEDS TO BE CONTINUED WORK IN
THAT SPACE TO REALLY HELP OUT
THAT STANDARDIZATION PROCESS SO
THAT EVENTUALLY HOPEFULLY WE
COULD GET TO LARGE-SCALE
MULTI INSTITUTIONAL
AGGREGATIONAL AND GET TO BETTER
NUMBERS FOR DIAGNOSTIC ERROR
WITHIN THE EMERGENCY DEPARTMENT
SPACE AND ELSEWHERE. SO KIND OF
FUTURE RESEARCH FOR US, AS WE
ARE DOING BUILD-OUT FOR THESE.
BUT I THINK ONCE WE GET THERE,
THERE IS A REAL OPPORTUNITY FOR
DESCRIPTIVE CHARACTERIZATIONS
BECAUSE WE ARE NOW AT-SCALE AND
HAVE A LOT MORE KIND OF CAPTURE
OF DIAGNOSTIC SAFETY WITHIN THE
SYSTEM. I'M INTERESTED IN
LOOKING AT ORGANIZATIONAL AND
TEAM RISK FACTOR S SO THAT
EXAMPLE I MENTIONED ABOUT
BOARDING, YOU CAN THINK ABOUT A
LOT ARE OCCURRING AT NIGHTTIME
VERSUS DAY. IS IT OCCURRING ON A
PHYSICIAN'S FIFTH DAY OF WORKING
OR THE EIGHTH HOUR THEY ARE INTO
THEIR SHIFT. ALL THOSE ARE
REALLY INTERESTING QUESTIONS
THAT POTENTIALLY CAN MOVE THE
NEEDLE FROM A DIAGNOSTIC SAFETY
STANDPOINT. THEN I THINK MOVING
FORWARD WITH DISSEMINATION, HOW
CAN YOU PACKAGE THESE TOOLS UP
TO USE THEM ACROSS MULTIPLE
ORGANIZATION. THEN THE LAST IS
REALLY OPTIMIZATION. IT IS
IMPROVING THAT RECOGNITION OR
TRIGGER SYSTEM SO YOU BECOME
MORE EFFICIENT OVERALL WITH THE
PROCESS. I THINK WE WILL SEE
CONTINUED OPT MIZATION FROM
FURTHER DEVELOPMENT OF LARGE
LANGUAGE MODELS AS WE MOVE
FORWARD. SO THANK YOU VERY MUCH
THAT. IS MY LAB TEAM. I WILL
CLOSE DOWN THIS AND OPEN IT UP
FOR QUESTIONS.
>> THANK YOU SO MUCH. YOU
TOUCHED ON SO MANY IMPORTANT
POINTS. IT IS JUST SO
OVERWHELMING IN TERMS OF HOW
POWERFUL THE SYSTEMS ARE. HOW
DIFFICULT IT IS TO IMPLEMENT
THEM IN THE CLINICAL
ENVIRONMENT. SO YOU ARE FOCUSED
ON FINDING THIS DIAGNOSIS
QUALITY, QUALITY IMPROVEMENT
FROM THE QUALITY IMPROVEMENT
STANDPOINT. ARE YOU CURRENTLY
USING LLMS TO HELP YOU WITH
DIAGNOSIS?
>> YEAH, I THINK THAT IS THE
EASIEST PATHWAY INITIALLY INTO
THE DIAGNOSTIC KIND OF QUALITY
IMPROVEMENT SPACE AS OPPOSED TO
HAVING A LARGE LANGUAGE MODEL
WORKING IN PARALLEL WITH A
PROVIDER GENERATING A
DIFFERENTIAL AND EVERYTHING
ELSE. I THINK WE ARE GET THERE
YOU GO BUT IT IS SO DEPENDENT
UPON YOUR UNDERLYING
INFRASTRUCTURE AND THE WAY YOU
HAVE YOUR EHR SET UP AND THE EHR
INTEGRATION BECOMES VERY
DIFFICULT. I THINK OBVIOUSLY THE
HOPE IS YOU WILL HAVE THESE
RUNNING SIDE-BY-SIDE, LONG-TERM,
BUT WE ARE NOT THERE YET. BUT
HOPEFULLY WE WILL BE SOON.
>> I HAVE A QUESTION FROM ONE OF
OUR VIEWERS, FROM RACHEL SABO.
THE QUESTION IS HOW DO YOU
HANDLE PROTECTING PII DATA
INCLUDED INTO THE LLM. I THINK
FOLLOW-UP QUESTIONS. HOW DO YOU
COOLING DATA EFFICIENTLY? IS
SOMEONE MANUALLY COPYING AND
PASTING TO THE LLM PROMPT?
>> YES. FOR THE FIRST ONE, THE
AZURE OPENING AI ENVIRONMENTS,
THAT IS A SECURE HIPAA COMPLIANT
ENVIRONMENT, NO SHARED DATA IS
COMING OUT THAT. IS THE SETUP A
LOT ARE GOING WITH AT THIS
POINT. SO USING STATE LESS
SYSTEMS TO BE ABLE TO DO THAT
AND THEY ARE ALREADY AROUND KIND
OF THE HIPAA COMPLIANT ASPECTS
AND ASSURANCES. THE OTHER WAY IS
HAVE YOUR MODEL IN YOUR
INSTITUTION, IN YOUR
COMPUTATIONAL HEALTH PLATFORM
AND TO USE THAT MODEL SO ONE
EXAMPLE OF LIKE POTENTIALLY
USING A MODEL THAT'S BEEN
DEVELOPED WITH REALLY KIND OF A
MEDICAL FOCUS THAT EXISTS IN
YOUR -- THERE IS NO REAL
EXCHANGE OF DATA OUTSIDE OF
THAT. THE SECOND WAS -- SO THE
EMERGENCY DEPARTMENT IS A LITTLE
EASIER PROBABLY THAN OTHER
CONTEXT WHEN COMES TO DIAGNOSIS.
SO, YOU KNOW, IT CAN KIND OF
CONFINE THAT DIAGNOSTIC EVENT
AROUND THE ACTUALLY EMERGENCY
DEPARTMENT CARE. SO THERE IS A
MUCH MORE LIMITED SET OF NOTES
THAT I NEED TO BE ABLE TO SEARCH
THROUGH AND REALLY JUST THE
PROVIDER NOTE, NURSING NOTE TO
GET A GOOD CONTEXT AROUND THAT.
YOU COULD THINK LIKE IF YOU ARE
LOOKING AT IN-HOSPITAL
ENVIRONMENT IS MUCH MORE
CHALLENGING, RIGHT. YOU ARE
POTENTIALLY LOOKING AT MULTI,
MULTI DAYS OF NOTES ACROSS A
VARIETY OF DIFFERENT PROVIDERS
AND FIGURING OUT HOW TO BE
EFFICIENT ARE GOING TO BE
CHALLENGES MOVING FORWARD IN
THAT SPACE. SO I GUESS I WOULD
LIKE THE EMERGENCY DEPARTMENT
FOR THAT. WE HAVE THESE
DISCREET EVENTS AND TIME
PERIODS. IT IS EASIER TO LOOK
BACK AND ATTRIBUTE WHETHER THERE
WAS A POTENTIAL MISDIAGNOSIS OR
NOT. THAT IS PROBLEM LAY LOT
MORE DIFFICULT THAN OTHER
ENVIRONMENTS.
>> YEAH, WOULDN'T IT BE
FASCINATING WE COULD KNOW A
PATIENT HISTORY AND YOUR LLM IN
THE EMERGENCY DEPARTMENT WOULD
ACTUALLY HAVE THAT BACKGROUND
KNOWLEDGE WHEN HELPING YOU WITH
DIAGNOSIS FOR THAT PARTICULAR
PATIENT.
>> YEAH, THAT IS ANOTHER -- I
DIDN'T TALK ABOUT BECAUSE I
THINK THERE IS STILL A LOT OF
IMPLEMENTATION CHALLENGES. BUT
THE OTHER ASPECT OF THIS, I LIKE
TO CALL LIKE EITHER DECISION
HYGIENE OR THE DECISION
ENVIRONMENT THAT ALONG WITH
SUMMARIZATION. SO WE STRUGGLE
WITH MANAGING THAT VERY LONG
PROBLEM LIST SOR LOOK BACK
THROUGH NOTES TO SEE IF A
PROBLEM HAD A CATH OR PROCEDURE.
OBVIOUSLY THINGS GET MISSED
BECAUSE WE DON'T HAVE A LOT OF
TIME AND IT IS TOUGH TO DO THOSE
SEARCHES. I THINK AS WE MOVE
FORWARD THAT INFORMATION
SUMMARIZATION AND SYNTHESIS
ASPECT THAT WILL BE DRIVEN BY
LARGE LANGUAGE MODELS IS GOING
TO BE HUGELY HELPFUL TO
PROVIDERS BECAUSE IT WILL GIVE
THEM A BETTER CONTEXT AND
UNDERSTANDING OF THE PATIENT.
YOU CAN THINK ABOUT THIS. IT CAN
BE VERY CONTEXT DEPENDENT TOO.
IF THE PATIENT COMES IN WITH
CHEST PAIN IT IS ONLY LOOKING
FOR INFORMATION THAT WOULD BE
RELEVANT FOR CHEST PAIN
DIAGNOSIS OF SOME SORT. THAT CAN
BE -- YOU CAN COMPLETELY CHANGE
THE WHOLE USER INTERFACE BASED
ON THOSE TYPES OF KIND OF
CONTEXT COMPONENTS FROM THE
CHIEF COMPLAINTS AND WHATEVER
FOR THE PATIENT.
>> YEAH, YEAH. DO YOU HAVE A
QUESTION?
>> YEAH. FIRST OF ALL, THANK YOU
FOR THE TERRIFIC TALK. I HAVE
ONE CLARIFICATION QUESTION AND
ONE OTHER QUESTION.
>> SURE.
>> CAN YOU CLARIFY THAT. WHEN
YOU SAY WORK FLOW DIAGRAM, YOU
MENTIONED ABOUT 24 HOURS, DOES
THAT MEAN THE AI IS POSITIONED
TO DETEXT MISDIAGNOSIS WITHIN 24
HOURS TIME FRAME? IS THAT WHAT
IT IS SUPPOSED TO BE DOING?
>> IT WOULD BE. IT IS REALLY
DEPENDENT ON THE DATABASE
UPDATES, RIGHT. SO THE ETLs ARE
COMING OUT ARE ONLY ON A 24-HOUR
TIME FRAME. SO NOTES AREN'T
GOING TO BE UPDATED. YOU COULD
THINK ABOUT THE WINDOW TO BE
PROBABLY THE ABILITY OF THE
SYSTEM DO DIAGNOSIS AND
ATTRIBUTION IS PROBABLY IMPROVED
BY THE LONGER WINDOW THAT
HAPPENS, SO IT IS PROBABLY
EASIER TO SEVEN-DAY ON 24-HOUR
BASIS TO MAKE THATS ASSESSMENT.
MAINLY BECAUSE NOTES ARE STILL
IN FLUX. PEOPLE AREN'T
COMPLETING AND YOU MAY HAVE
UPDATES TO NOTES OR LABS AND
OTHER THINGS THAT AREN'T THERE.
BUT THAT IS THE -- THAT IS ONE
OF THE CURRENT --
>> YEAH, THE REASON I ASK THIS,
I WAS WONDERING, IS THIS MAINLY
FOR HELPING QUALITY ASSURANCE --
QUALITY IMPROVEMENT FOR THE
FUTURE PATIENTS VERSUS THIS IS
SOMETHING THAT CAN ALSO AFFECT
THE CURRENT PATIENTS? YOU KNOW
WHAT I MEAN?
>> RIGHT. THAT IS SOMETHING THAT
WE HAVE THOUGHT ABOUT, IS THIS
EARLY NOTIFICATION OF THIS
POTENTIAL RETURN. WE HAD THOSE
TYPE OF MECHANISMS IN EMERGENCY
DEPARTMENTS FOR OTHER THINGS SO
A PATIENT LET'S SAY GETS
DISCHARGED BUT HAVE A LAB THAT
COMES BACK INDICATIVE OF
SOMETHING MORE SERIOUS SO THEY
GET CALLED BACK. YOU COULD
THEORETICALLY HAVE SOMETHING
VERY SIMILAR IN THIS SPACE WHERE
THE LLM IS REVIEWED AT THE
24-HOUR MARK. SAYS OH, THERE IS
A POTENTIAL MISS HERE .
THE TROUBLE WITH THAT IS IT
DOESN'T HAVE THE FINAL DIAGNOSIS
YET. THERE IS A LIMITATION
BECAUSE YOU ARE NOT FEEDING IN
TO SAY THIS WAS THE DIAGNOSIS,
THIS WAS THE PRIOR CONTEXT. SO
THERE IS SOME LIMITATION SO I
TOTALLY AGREE THAT IS AN
OPPORTUNITY IN THE FUTURE. I
THINK THE QUESTION IS LIKE HOW
MUCH IMPACT DOES THAT HAVE ON
THE HEALTH SYSTEM AND HOW MANY
PEOPLE ARE YOU CALLING BACK? YOU
WOULD NEED A PRETTY SPECIFIC
SYSTEM SO YOU ARE NOT CALLING
BACK 10%, 20% OF PATIENT
POPULATION. ALL THOSE THINGS
WOULD HAVE TO BE WORKED OUT.
>> YEAH. IF I MAY, ANOTHER
QUESTION I HAVE IS, NOW
DEVELOPED THIS AI MODEL BASED
SYSTEM, I DON'T THINK I HEARD
MUCH ABOUT HOW ARE YOU GOING TO
EVALUATE THIS? HOW DO YOU KNOW
THIS IS GOOD ENOUGH TO BE
IMPLEMENTED IN YOUR HEALTH CARE
SETTINGS?
>> YEAH, SO THE PROCESS WE ARE
GOING TO GO THROUGH FROM AN
EVALUATION STANDPOINT IS A LOT
OF THE -- SO A LOT OF MANUAL
REVIEW, SO USING THE QI TEAM TO
MAKE ASSESSMENT ABOUT HOW WELL
THE AI IS IDENTIFYING THE
OPPORTUNITIES, THE DISADVANTAGE
IS THE FLAGS AND IDENTIFICATION
OF CASES, YOU ARE A LITTLE
UNBALANCED. IF YOU ARE ONLY
PULLING IN THE ONES THE SYSTEM
IS ALREADY IDENTIFYING, YOU ARE
ONLY CLARIFYING THOSE POTENTIAL
ONES THAT ARE THE FALSE
POSITIVES. THAT IS ONE THING WE
ARE WORKING THROUGH. SOME OF
THIS JUST REQUIRES SAMPLING. WE
HAVE ALSO STARTED THINKING ABOUT
EXPLORING DOING SOME FINE-TUNING
AND TRAINING OF SMALLER MODELS
TO BE ABLE TO ALIGN WITH SOME OF
THE KIND OF QI FEEDBACK AND THE
SCORING COMPONENTS. ONE OF THE
THINGS I MENTIONED THE SCORING
ASPECTS OF THINGS BUT WE HAVE
SEEN THE LARGE LANGUAGE MODELS,
THEY TEND TO GO TOWARD THE
EXTREMES. IT IS EITHER THE POINT
SCORE SYSTEM IS VERY LOW OR IT
IS VERY HIGH AND THERE IS NOT A
LOT IN THE NUANCE IN THE MIDDLE,
SO I THINK THERE'S DEFINITELY
WORK TO HAPPEN IN THAT SPACE.
THEN YEAH, I THINK ULTIMATELY IT
WILL COME DOWN TO THE QI TEAM IN
EVALUATION ABOUT HOW WELL WE ARE
DOING IN IDENTIFYING THEM AND
COMPARING IT TO EXISTING SYSTEMS
AS WELL. BECAUSE THEY WILL STILL
PROBABLY HAVE THE REPORTING
MECHANISMS AS WELL.
>> I HAVE A BUNCH OF QUESTIONS.
MAYBE I CAN DO ONE MORE BECAUSE
WE PROMISED TO WRAP UP AROUND
2:15. SO THE QUESTION IS, SEEMS
TO ME LIKE YOU NEED REAL-TIME
DATA TO OBTAIN DATA FROM
PATIENTS COMING TO THE ER. COULD
THEY BE PING ED WITH A FEW BRIEF
QUESTIONS THAT WOULD HELP
SHEPHERD THEM TO THE CORRECT
DOCTOR OR SPECIALIZATION OR FLAG
THE SEVERITY OF THE CASE?
>> SO YEAH, I WOULD -- THAT IS
ANOTHER -- DEFINITELY ANOTHER
AREA OF TOPIC AND DISCUSSION. SO
I CONSIDER THESE LIKE THE KIND
OF PRE-TRIAGE OR KIND OF -- LIKE
AN AI-BASED TRIAGE SYSTEM WHERE
IT IS REALLY FIGURING OUT IS THE
CASE AN EMERGENCY, CAN WE DO A
BETTER JOB OF GETTING PATIENTS
TO THE RIGHT SPACES, COULD THEY
POTENTIALLY GET URGENT CARE, DO
THEY NEED TO COME TO THE
EMERGENCY DEPARTMENT TO
FACILITATE THEM BY FOLLOWING UP
WITH THEIR PRIMARY CARE DOCTOR
THE NEXT DAY? I DEFINITELY THINK
WE ARE GOING TO SEE THOSE
SYSTEMS. THERE ARE A COUPLE OF
COMPANIES I KNOW THAT ARE
ALREADY IN THAT MARKET TO HELP
BETTER FACILITATE PATIENTS ALONG
THEIR CARE JOURNEY. SO YEAH, I
COMPLETELY AGREE THAT THAT IS
HAPPENING TO OTHER PIECE I ALL
LOUED TO A LITTLE BIT WAS
PATIENTS WE HAVE SEEN IN
EMERGENCY DEPARTMENT BUT IN
OTHER ENVIRONMENTS THERE, IS A
LOT OF DOWN TIME WITH PROCESS.
MOST PATIENTS IN OUR EMERGENCY
DEPARTMENTS ARE STAY THERE YOU
GO SEVERAL HOURS. THE ABILITY TO
BE ABLE TO COLLECT A DIFFERENT
-- OTHER INFORMATION ABOUT THEM
RELATED TO THE CARE PROCESSES,
WE ARE NOT REALLY DOING I THINK
A GREAT JOB OF THAT RIGHT NOW SO
EVEN TALK ABOUT SOCIAL
DETERMINANTS OF HEALTH OR, YOU
KNOW, DOING SCREENING FOR CANCER
AND OTHER THINGS, THOSE -- WE
DON'T REALLY HAVE SYSTEMS RIGHT
NOW. THE ASK CURRENTLY FOR A LOT
IS FOR NURSING TO DO THAT TYPE
OF SCREENING. AS I MENTIONED
PREVIOUSLY, NURSING STAFF IS
ALREADY OVERBURDENED AND BURNED
OUT. SO THINGS TO OFFLOAD THOSE
TYPES OF TASKS ON THEM WOULD I
THINK BE HELPFUL.
>> YEAH, YEAH. PERSONALLY, TO
ME, THOSE CASES WHERE
MISDIAGNOSIS HAPPENS THEY FIELD
LIKE A GOLD MINE BECAUSE THESE
ARE CASES WHERE WE HAVE
PARTICULARLY PARTICULAR
DIFFICULT TO DIAGNOSE SO WOULD
BE AMAZING TO HAVE A COLLECTION
FOR FINE-TUNING OF LLMS OR ANY
OTHER TRAINED MODELS. I WANT TO
THANK YOU SO MUCH FOR THIS VERY
EXCITING AND INTERESTING TALK.
CAN WE HOPE THAT YOU MAY COME
BACK TO US IN SOME NEAR FUTURE
AND UPDATE WITH THE PROGRESS IN
THE SYSTEMS IN THE EMERGENCY
CARE BASED ON THE NUMBER OF
QUESTIONS A LOT OF PEOPLE WERE,
YOU KNOW, VERY INTERESTED IN
THIS TALK. THANK YOU SO MUCH,
DR. TAYLOR.
>> YEAH, THANK YOU ALL VERY
MUCH.
>> THANKS, BYE. (TRANSCRIBING OF
THIS CALL HAS STOPPED.
)