>> WELL, WELCOME, EVERYONE, AND
THANK YOU FOR JOINING US.
I'M LANA YEGANOA, YOUR HOST FOR
TODAY'S EVENT, ORGANIZED BY
NATURAL LANGUAGE PROCESSING AND
TEXT MINING SPECIAL INTEREST
GROUP HERE AT NIH.
WE'RE EXTREMELY EXCITED TO
INTRODUCE OUR TODAY'S SPEAKER,
DR. ZHE HE IS AN ASSOCIATE
PROFESSOR AT THE SCHOOL OF
INFORMATION AT THE FLORIDA STATE
UNIVERSITY AND A CORE AFFILIATE
OF THE INSTITUTE FOR SUCCESSFUL
LONGEVITY.
HE IS DIRECTOR OF
BIOSTATISTICS -- IN THE
UNIVERSITY'S CLINICAL AND
TRANSLATIONAL SCIENCE AWARD HUB.
HIS RESEARCH EXPERTISE INCLUDES
MACHINE LEARNING, NATURAL
LANGUAGE PROCESSING, AND
KNOWLEDGE REPRESENTATION.
THE OVERARCHING GOAL OF HIS
RESEARCH IS TO IMPROVE
POPULATION HEALTH AND ADVANCE
BIOMEDICAL RESEARCH THROUGH
APPLICATION OF BIOINFORMATICS.
DR. HE, I CANNOT WAIT TO HEAR
YOUR IDEAS AND VISIONS TOWARDS
ACHIEVING A BETTER PUBLIC HEALTH
AND WITH THE HELP OF
BIOINFORMATICS AND DATA SCIENCE.
SO BEFORE I PASS THE FLOOR TO
YOU, I JUST WANT TO SAY THAT WE
WILL HAVE AT THE END 10 TO 15
MINUTES FOR QUESTIONS AND
ANSWERS.
AND AS USUAL, PLEASE SEND YOUR
ANSWERS TO MY EMAIL,
LANA.YEGANOVA@NIH.GOV, AND AFTER
THE SPEAKER IS DONE WITH THE
PRESENTATION, I WILL BE HAPPY TO
READ OUT YOUR QUESTIONS AND HAVE
THEM FOR DISCUSSION.
AND WITH THAT, DR. HE, WE'RE
READY TO HEAR YOU.
>> GREAT.
THANKS SO MUCH.
SO LET ME START SHARING MY
SCREEN, AND THEN START MY
PRESENTATION.
AND FIRST OF ALL, I WANT TO MAKE
SURE THAT YOU CAN SEE THE ENTIRE
SCREEN OF MY POWERPOINT SLIDE,
RIGHT?
>> YES, WE'RE GOOD.
>> SO THANK YOU SO MUCH FOR THE
OPPORTUNITY TO SPEAK HERE AT NLM
AND TO THIS GREAT AUDIENCE.
AND I REALLY FEEL VERY HONORED
AND PROUD TO BE ABLE TO BE HERE.
ALSO THIS OPPORTUNITY WOULD NOT
BE POSSIBLE WITHOUT DR. LIU'S
SUPPORT.
I WORKED WITH DR. LIU SINCE LAST
SEPTEMBER AND WORKED AS GUEST
VISITOR A.
THIS HAS BEEN ONE OF MY MAJOR
RESEARCH PROJECTS SINCE 2019 SO
I WANT TO SHARE WITH YOU SOME OF
MY JOURNEY AND WHERE I'M AT OR
WHERE I WAS AT AND WHERE I'M
CURRENTLY AT IN TERMS OF
RESEARCH TRAJECTORY.
I FEEL THIS PROJECT IS REALLY
UNIQUE AND INTERESTING TO ME,
SIMPLY BECAUSE I STARTED THIS
PROJECT BECAUSE OF SOME OF MY
PERSONAL EXPERIENCES WITH
HEALTHCARE SYSTEMS AND SOME OF
MY FRUSTRATION AND STRUGGLES
WITH KIND OF GETTING TO KNOW
WHAT DOES MY DATA MEAN, WHAT
DOES MY RECORD MEAN, AND THEN I
HAD TO FIND GREAT COLLABORATORS
TO WORK WITH AND GRADUALLY
GETTING SOME REALLY INTERESTING
PROJECTS GOING AND PUBLISHED AND
ALSO TRYING TO GET SOME -- AS
WELL.
SO THE TOPIC IS USING MINGS AND
INFORMATICS AND GENERATIVE AI.
THE BACKGROUND OF THIS PROJECT
IS THAT WE HAVE NO EASY ACCESS
TO EHR DATA BECAUSE OF ADOPTION
OF THE SYSTEMS ACROSS THE UNITED
STATES, SO IT'S UNLIKELY THAT IT
WILL EVER GO TO A HOSPITAL
WHERE -- DATA IS NOT USED TO
CAPTURE YOUR PERSONAL
INFORMATION.
ALSO WITH ACCESS TO EHR DATA
THROUGH PATIENT PORTAL, PATIENTS
NOW ALSO HAVE EASIER ACCESS TO
THEIR CLINICAL DATA, AND WE LOOK
AT LITERATURE, WE FOUND THAT
PEOPLE ACTUALLY USE PATIENT
PORTALS FOR A VARIETY OF
PURPOSES, INCLUDING GETTING LAB
TEST RESULTS, COMMUNICATING WITH
THEIR PROVIDERS AND ALSO
MEDICATIONS, MEDICAL HISTORY,
AND AMONG ALL OF THESE
ACTIVITIES, VIEWING LAB TEST
RESULTS SEEMS TO BE ONE OF THE
MOST FREQUENT ACTIVITIES PEOPLE
UTILIZE PATIENT PORTALS FOR.
HOWEVER, EVEN IF PEOPLE/PATIENTS
HAVE ACCESS TO LAB TEST RESULTS
THROUGH PATIENT PORTALS, STILL
MANY PEOPLE FOUND CONFUSING AND
HARD TO UNDERSTAND.
THERE IS A VARIETY OF REASONS.
PARTLY COULD BE DUE TO PATIENTS'
LITERACY OF HEALTHCARE, OF
HEALTH LITERACY, ALSO COMPLEXITY
OF RESULTS AND COMPLEXITY OF
THEIR HEALTH CONDITIONS.
AND ALSO WE'RE SEEING SOME
INTERESTING TREND THAT EVEN
THOUGH PATIENT PORTALS ARE
SUPPOSED TO PROVIDE A WAY FOR
PATIENTS TO COMMUNICATE THEIR
INFORMATION AND CONCERNS,
QUESTIONS TO PROVIDERS,
PROVIDERS OFTENTIMES MAY SEE
THIS ALSO AS A PROBLEM BECAUSE
THEY NEED TO SPEND A LOT OF TIME
ANSWERING PATIENTS' QUESTIONS.
SO, THEREFORE, WE HAVE SEEN SOME
NEWS ABOUT PROVIDERS ACTUALLY
STARTING TO CHARGE PATIENTS IN
PATIENT PORTAL MESSAGE FOR
SOMETHING LIKE ASKING QUESTIONS
ABOUT LAB RESULTS.
AND ALSO THERE'S A RECENT "WALL
STREET JOURNAL" ARTICLE ABOUT
WHEN SHALL WE SEND THIS MEDICAL
TEST RESULTS TO PATIENTS, AND
THIS ARTICLE TALK ABOUT DOCTORS
ARE REALLY THINKING THAT GIVING
PEOPLE ACCESS TO EARLY TEST
RESULTS REPORTING IS GREAT BUT
SOMETIMES WHEN PATIENTS HAVE
ACCESS TO THESE RESULTS BEFORE
THEIR PHYSICIANS ACTUALLY GET IN
TOUCH WITH THEM, IT CAUSES SOME
TROUBLES, OFTENTIMES BECAUSE OF
MISINTERPRETATION OF RESULTS, OR
MISINTERPRETATION OF EVEN HAR
HARMFUL AND HARMLESS
INFORMATION.
SO ESPECIALLY WHEN PEOPLE
RECEIVING RESULTS THAT ARE
LIFE-CHANGING.
SO DOCTORS SAY THEY HAVE LEARNED
AND HAVE HAD PATIENTS LEARN
ABOUT CANCER DIAGNOSIS FROM
SMARTPHONE NOTIFICATION IN THE
MIDDLE OF BUSINESS DINNER OR
WHILE READING A BEDTIME STORY TO
A 3-YEAR-OLD OR DOI DURING RUSHR
COMMUNITY, AND EVEN THERE'S A
PATIENT'S SPOUSE WHO WENT TO THE
EMERGENCY ROOM FOR AN ANXIETY
ATTACK AFTER MISINTERPRETING HER
HUSBAND'S RESULTS.
-- WHAT TIME FRAME WOULD BE A
PROPER TIME FRAME FOR THEM TO
COMMUNICATE INFORMATION TO THEIR
PATIENTS.
AND ALSO, IF YOU SEE A LAB TEST
RESULT, IF YOU'VE EVER SEEN LAB
TEST RESULTS FROM PATIENT
PORTALS YOU MAY PROBABLY SEE
SOMETHING LIKE THIS WHERE IT
SHOWS YOU THE LAB TEST NAME, THE
RESULTS, VALUE, THE UNIT, AN
ALSO THE REFERENCE INTERVAL.
AND THIS TYPICALLY IS THE WAY
THAT PATIENT PORTALS ARE
REPORTING LAB RESULTS TO
PATIENTS, AND HOWEVER, THIS TYPE
OF REPORTING MAY NOT BE THE
IDEAL WAY OF REPORTING
INFORMATION BECAUSE ABNORMAL
RESULTS WHICH MEANS SOMETIMES
THE LAB TEST RESULTS OUT OF
RANGE MAY NOT MEAN IT'S VERY
CONCERNING, WHEREAS IF OTHER --
SOMETIMES LAB VALUES WHICH MAY
BE CLOSE TO THE BORDERLINE MAY
BE CONCERNING.
SO -- AND ALSO SOME LAB TEST
RESULTS COULD HAVE DIFFERENT
INTERPRETATIONS BASED ON
DIFFERENT STANDARDS.
FOR EXAMPLE, TSH LEVEL WHICH IS
USED TYPICALLY TO MEASURE
THYROID FUNCTION, THEY COULD
HAVE DIFFERENT INTERPRETATION BY
DIFFERENT ORGANIZATIONS AND
DIFFERENT ASSOCIATIONS.
SO IT'S UP TO THE PROVIDERS TO
PROVIDE MORE ACCURATE
INTERPRETATION TO LAB RESULTS,
WHICH COULD TYPICALLY BE THE
CASE WHEN THEY HAVE DIRECT
ACCESS -- WHEN THEY CAN
COMMUNICATE WITH THE PATIENTS --
WHEN THEY CAN'T COMMUNICATE WITH
PATIENTS IN PERSON.
SO IF YOU GIVE THIS WITHOUT
INTERPRETATION IT COULD POSSIBLY
CONFUSE AND CAUSE PROBLEMS.
AT THE SAME TIME, PATIENTS WHEN
THEY SEE LAB RESULTS, THEY MAY
NOT FIND THEM CONCERNING OR THEY
MAY NOT REALLY UNDERSTAND THE
URGENCY OF SUCH INFORMATION.
SO PREVIOUS EFFORTS HAVE DONE TO
UNDERSTAND PATIENTS' ACTIVITY
AND BEHAVIOR AFTER LAB RESULTS
REPORTING.
AND THERE'S A STUDY IN 2018
SAYING LIKE OUT OF THIS
93 PATIENTS, THEY FOUND NEARLY
TWO THIRDS OF THEM DID NOT
RECEIVE ANY EXPLANATION OF LAB
RESULTS AND ABOUT HALF OF THEM
CONDUCT ONLINE SEARCH TO
UNDERSTAND THEIR RESULTS BETTER.
ALSO THERE'S A STUDY THAT TALKS
ABOUT EXPLORE DIFFERENT WAYS TO
PRESENT LAB INFORMATION TO
PATIENTS EITHER IN -- FORMAT
WHERE THEY HAVE DIFFERENT
INDICATOR OF URGENCY AND ALSO
THEY MAY USE DIFFERENT COLOR
CODING OR DIFFERENT -- EVEN THE
EMOJIS TO PROVIDE SOME SORT OF
INTERPRETATION WITH SOME TEXTUAL
EXPLANATION.
AND THERE WAS AN INTERESTING
STUDY PUBLISHED IN 2017 BY
ZIKMUND-FISHER GROUP THAT
EXPLORED THE USE OF DIFFERENT
REPORTING OF LAB INFORMATION AND
ALSO EVALUATING THEIR URGENCY.
SO THEY ASSESS THE TABLE
REPRESENTATION WHICH IS TYPICAL
FORMAT, AND ALSO THEY LOOK AT
THE SIMPLE LINE FORMAT, WHERE
YOU JUST HAVE A SIMPLE LINE
INDICATING THE NORMAL REFERENCE
RANGE AND ALSO THERE'S A BLOCK
LINE FORMAT WHERE THEY HAVE
DIFFERENT BLOCK LINE TO INDICATE
THE BORDERLINE VALUE AND ALSO
THE -- LIKE THE VERY LOW VALUE
OR LOW VALUE, AND ALSO THERE'S
ANOTHER ONE, IT'S A GRADIENT
LINE VALUE REPRESENTATION, WHICH
DOESN'T HAVE BLOCK LINE BUT VERY
GRGRADUALLY CHANGING THE COLOR O
INDICATE URGENCY AND NORMALCY OF
A LAB TEST RESULT.
SO THE ASSESSED, GIVEN THIS
DIFFERENT TYPE OF REPRESENTATION
OF THE SAME INFORMATION TO
PATIENTS, WHAT IS THE URGENCY
LEVEL, AND THEY FOUND THAT
FROM TABLE TO SINGLE LINE TO
BLOCK LINE TO GRADIENT LINE
GRADUALLY DECREASING URGENCY
LEVEL FROM PATIENTS.
IN OUR GROUP AS I MENTIONED, WE
STARTED TO EXPLORE THIS IDEA OF
HOW TO BETTER PROVIDE
INFORMATION TO PATIENTS FOR
THEIR LAB RESULTS COMPREHENSION,
WE STARTED EXPLORING THIS BY
CONDUCTING A SIMPLE ONLINE
SURVEY, LOOKING AT THE
SELF-PERCEIVED UNDERSTANDING OF
RESULTS AND DIFFERENT CHALLENGES
THEY HAVE FACED, SO BASED ON
THIS SIMPLE ONLINE SURVEY
CONDUCTED, WE FOUND THAT WHEN
PATIENTS ARE TRYING TO INTERPRET
LAB RESULTS THEY OFTENTIMES NEED
BOTH INFORMATION ABOUT REFERENCE
RANGE AND ALSO SOME PERSONALIZED
CONTEXTUAL INFORMATION SUCH AS
TREATING OPTION, DIAGNOSIS AND
WHAT THEY SHOULD DO NEXT.
THEY ALSO FOUND THAT SEVERAL
FACTORS LIKE NORMALCY OF
RESULTS, HEALTH LITERACY,
TECHNOLOGY PRO IF EFFICIENCY, PD
IMPACTED PERCEPTION OF THE
RESULTS WHEN THEY INTERPRETED
RESULTS USING PATIENT PORTALS.
WE ALSO FURTHER LOOKED AT SOCIAL
MEDIA POSTINGS ON THE SOCIAL
MEDIA QUESTION WEBSITE, WE LOOK
AT QUESTIONS THAT CONTAIN SOME
DIABETES-RELATED LAB VALUES,
A1C, GLUCOSE AND BLOOD SUGAR,
AND LOOK AT WHAT KIND OF
INFORMATION THEY CAN PROVIDE IN
THESE QUESTIONS AND WHAT KIND OF
RESPONSES THEY TYPICALLY RECEIVE
FROM THE COMMUNITY USERS OF THIS
WEBSITE AND WE FOUND THAT WHEN
PATIENTS ASKING QUESTIONS ON
SOCIAL MEDIA LIKE SOCIAL Q & A,
THEY MAY NOT PROVIDE ENOUGH
INFORMATION AND SOME OF THEM MAY
PROVIDE INFORMATION ABOUT THEIR
LIFESTYLE, ABOUT THEIR DIET AND
ABOUT THEIR DIAGNOSIS AND
COMORBIDITIES.
SO IT'S A VERY DIVERSE GROUP.
WE ALSO CONDUCTED A STUDY WHERE
WE DESIGNED A PROTOTYPE OF A
TOOL THAT PRESENTS LAB
INFORMATION TO PATIENTS, AT THE
SAME TIME PRESENTING A WAY FOR
PATIENTS TO LOOK AT
INTERPRETATION OF LAB VALUE, LAB
TEST AND ALSO THEY CAN ALSO
ANNOTATE AND LABEL SOME OF THE
INTERPRETATION INTO THEIR
PERSONAL LIBRARIES.
SO ALL OF THESE STUDIES REALLY
GIVE US SOME GOOD EXPERIENCES
INTO THIS FIELD AND LOOKING AT
ALSO WORKING DIRECTLY WITH
INDIVIDUAL PATIENTS TO
UNDERSTAND THEIR CONCERNS AND
THEIR BEST KIND OF -- THEIR
NEEDS IN GETTING THEIR MEDICAL
NEEDS OF A FIELD BY LAB RESULTS
INTERPRETATION.
RECENTLY WE OBTAINED SOME
INTERNAL GRANT TO CONDUCT
STUDIES TO FURTHER UNDERSTAND
THE FACTORS THAT INFLUENCE
UNDERSTANDING LAB TEST RESULTS.
THE MAIN REASON WHY WE DO THIS
SURVEY STUDY IS FOR THE PREVIOUS
ONE WHICH WE USED -- FOR DATA
COLLECTION, WE DIDN'T REALLY
RECEIVE A LOT OF RESPONSES FROM
OLER ADULTS, THAT'S BECAUSE
OLDER PEOPLE TEND TO NOT USE
LIKE THOSE TYPE OF SERVICES
AS A LESSER CLOUD MECHANISM FOR
SURVEY AND ALSO WE WANTED TO
GAIN A MORE DEEPER UNDERSTANDING
OF THE FACTORS USING STANDARD
AND EVALUATIVE METRICS.
WE ALSO DID A STUDY WHERE WE
ANNOTATED SOME ONLINE POSTS
ABOUT LAB RESULTS
INTERPRETATION, AND MORE
RECENTLY, I WORKED WITH
DR. LIU'S GROUP HERE AT NCBI TO
EVALUATE MODELS -- ANSWERING LAB
TEST RESULTS FR QUESTIONS FROM
PATIENTS. SO IN THIS ONLINE
SURVEY, WHAT WE DO IS WE WANT TO
LOOK AT A VARIETY OF FACTORS
THAT MAY INFLUENCE PATIENTS'
UNDERSTANDING AND COMPREHENSION
OF LAB RESULTS.
SO WE FOUND OUT THERE'S ACTUALLY
NO EXISTING METHOD FOR
EVALUATING SUBJECTIVE OR
OBJECTIVE COMPREHENSION OF LAB
TEST RESULTS, SO WE DEVELOPED
OUR OWN SCORE WHICH CONSISTS OF
21 QUESTIONS ARE WHETHER LAB
RESULTS IS NORMAL OR ABNORMAL
BASED ON PATIENT'S UNDERSTANDING
OF LAB TEST INFORMATION.
WE ALSO USED DIFFERENT
MEASUREMENTS LIKE NUMERACY
SKILLS, EHEALTH LITERACY AND
ALSO TO ASSESS PEOPLE'S NU P PE-
DEMOGRAPHICS, HEALTH STATUS,
FREQUENCY OF DOCTORS VISIT,
ACCESS AND USE OF LAB RESULTS
AND INTERNET AND TECHNOLOGY USE
TO UNDERSTAND THESE RESULTS IN
LAB RESULTS COMPREHENSION.
SO THIS IS ONE EXAMPLE WHERE WE
USE SOME SIMPLE LIKE LAB PANELS
THAT ARE FREQUENTLY DONE IN
PRIMARY CARE SETTING TO
PATIENTS, AND DURING THE ANNUAL
EXAM, AND THEN GIVING THEM THE
HYPOTHETICAL SCENARIO AND
LETTING THEM EVALUATE THESE
RESULTS AND PROVIDING THEIR
UNDERSTANDING OF THE NORMALCY
AND -- OF THE RESULTS.
SO USING 21-SUCH QUESTIONS, WE
CAME UP WITH A SCALE BUT WE'RE
STILL KIND OF ADJUSTING THIS
SCORE BECAUSE WE'RE STILL NOT
QUITE HAPPY ABOUT THE TYPE OF
VARIETIES AND THE VARIANTS WE
GET FROM THE RESPONSE FROM THE
PATIENTS.
AND ALSO WE WELCOME ANY IDEAS OF
HOW TO BETTER MEASURE PATIENTS'
UNDERSTANDING OF LAB TEST
RESULTS USING A STANDARD SCALE.
ALSO WE LOOKED AT PATIENT ACCESS
TO LAB RESULTS, WE FOUND OUT
MANY PEOPLE THEY HAVE ACCESS TO
LAB TEST RESULTS THROUGH PATIENT
PORTAL STILL, A LOT OF PEOPLE
THEY PREFER TO BE CALLED UP BY
THEIR DOCTORS OR NURSES SO THEY
CAN ASK QUESTIONS ON THE FLY.
WE ALSO ASK QUESTIONS ABOUT WHY
THEY DON'T WANT TO USE PATIENT
PORTAL TO VIEW LAB TEST RESULTS,
AND HOPEFULLY TO GET SOME MORE
INSIGHTS.
THIS IS NUMERACY SCALES, VERY
SIMPLE THREE-ITEM SUBJECTIVE
NUMERACY SCALE, WANTING TO JUST
LOOK AT THEIR GENERAL
UNDERSTANDING OF STATISTICS AND
BECAUSE WE ARE PRIMARILY
TARGETING DIVERSE PATIENT GROUPS
IN TERMS OF THEIR AGE, SO WE'LL
WANT TO LOOK AT WHICH SCALES
WILL GIVE EASIER SIMPLE WAY TO
ASSESS SUCH INFORMATION AND NOT
OVERBURDEN OUR PARTICIPANTS.
THIS IS OUR LITERACY SCALE, MORE
FOCUSED ON EHEALTH BEHAVIOR,
FOCUSED ON HEALTH RESOURCE
SEARCH ON THE INTERNET AND THERE
MIGHT BE OTHER USEFUL AND
VALUABLE EHEALTH LITERACY
SCALES, BUT WE FIGURE THIS SCALE
IS MORE LIKE EASY TO ADMINISTER
ON THE INTERNET -- OR ON THE WEB
FORMAT THROUGH OUR INSTRUMENT.
SO TRY TO UNDERSTAND THE GENERAL
DISTRIBUTION BASED ON STATISTICS
AND ALSO CONDUCTED LOGISTICAL
REGRESSION MODEL AND ALSO USING
GENERAL LINEAR MODEL TO OOP R
ANAANALYZEPREDICTORS.
SO THIS IS A GENERAL BREAKDOWN
BY OUR POPULATION.
WE HAVE A FAIRLY GOOD
PRESENTATION OF DIFFERENT AGE
GROUP FROM 18 TO 34, 35 TO 49
AND OVER 50 YEARS OLD.
WE ALSO HAVE A GENERALLY
BALANCED BETWEEN POPULATION
BETWEEN MALE AND FEMALE, AND
DIFFERENT RACE AND ETHNIC
GROUPS.
SO WE GOT 276 PARTICIPANT
RESPONDENTS FROM OUR LOCAL
REGISTRY WHERE WE SENT EMAIL TO
OVER 2,700 PARTICIPANTS WHO MAY
BE INTERESTED IN THIS RESEARCH,
AND ALSO COLLECT DATA FROM
AMAZON MECHANIC TURK.
AND HERE ARE SOME RESULTS FROM
THIS STUDY.
SO THE FIRST RESULT IS REGARDING
THE ACCESSING LAB RESULTS VIA
PATIENT PORE TALS.
WE FOUND OUT SOME OF THE
BARRIERS ESPECIALLY FOR THE
OLDER ADULT GROUP INCLUDES THE
PREFERRED DOCTORS' TO EXPLAIN
THE RESULT TO THEM.
SOME OF THEM JUST DON'T KNOW HOW
TO INTERPRET LAB VALUES BY
THEMSELVES.
AND ALSO LOOK AT FACTORS THAT
INFLUENCE ACCESS TO LAB RESULTS
VIA PATIENT PORTALS, WE FOUND
OUT USING THIS REGRESSION
ANALYSIS, WE FOUND THAT COMPARED
TO YOUNGER ADULTS, 18 TO 34
GROUP, ADULTS BETWEEN 50 TO 65
YEARS PLUS AND OLDER WERE MORE
LIKELY TO ACCESS THEIR LAB
RESULTS BY A PATIENT PORTAL AND
EHEALTH LITERACY WAS ALSO A
SIGNIFICANT PREDICTOR OF
ACCESSING LAB TEST RESULTS USING
PATIENT PORTAL, AND THEN THESE
ARE SOME OF THE REASONS WHY
PATIENTS -- WHAT PATIENTS DO
AFTER LAB TEST RESULTS, WE FOUND
MOST PEOPLE WOULD LIKELY TO
FOLLOW UP WITH THEIR DOCTOR
AND -- BUT MORE THAN HALF OF
THEM WOULD WANT TO DO RESEARCH
ONLINE.
SO THIS IS THE GENERAL LINEAR
MODEL WHERE WE PERFORM TO
EXAMINE THE RELATIONSHIP BETWEEN
USE OF PATIENT PORTAL TO VIEW
LAB RESULTS, NUMERACY SCALES,
AGE -- SO WE FOUND COMPARED TO
PATIENTS WHO HAVE VIEWED THEIR
LAB RESULTS VIA PATIENT PORTAL,
THOSE WHO HAVE NEVER HAD --
NEVER HAD A PATIENT PORTAL USE
HAD A DECREASED COMPREHENSION OF
LAB RESULTS AND ALSO COMPARED TO
PEOPLE WITH 50 YEARS OLDER, THE
YOUNGER ADULTS HAD A LOWER
COMPREHENSION OF RESULTS, AND
THESE ARE SOME OF THE
STATISTICAL SIGNIFICANT RESULTS
AND THE NUMERACY SKILLS IS
MARGINALLY SIGNIFICANT AS A
PREDICTOR FOR LAB RESULTS
COMPREHENSION.
AND IN A MORE RECENT STUDY, WE
PRESENTED A AN AMIA, WE WANTED O
PROVIDE MORE INFORMATIC SUPPORT
TO PATIENTS BECAUSE WE THOUGHT
AT THE TIME THAT THERE'S NOT
MUCH KIND OF STANDARDIZED WAY TO
PRESENT PATIENTS A TOOL FOR LAB
RESULTS COMPREHENSION.
SO IN THIS STUDY, WHAT WE DO IS
WE WANT TO DEMONSTRATE THE
FEASIBILITY OF EXTRACTING LAB
RESULTS INTERPRETATION FROM
CONSUMER-FRIENDLY WEBSITE
ARTICLES AND ALSO BUILDING A
FOUNDATION FOR DEVELOPING A TOOL
TO PROVIDE SUPPORT FOR LAB
RESULTS FOR PATIENTS.
SO HERE IS THE GENERAL WORKFLOW
WE FOLLOW, LIKE STANDARDIZED
DATA COLLECTION, ANNOTATION
OF -- ANNOTATION GUIDELINE
DEVELOPMENT AND KRAITION,
QUALITY ASSURANCE, AND THEN
PERFORM RECOGNITION WITH
TRANSFORMERS AND LINKING
ENTITIES WITH VOCABULARIES, WE
WANT TO BE ABLE TO IDENTIFY
THOSE INFORMATION FROM
ELECTRONIC HEALTH RECORDS AND
THEN BE ABLE TO AUTOMATICALLY
PROMPT PATIENTS WITH QUESTIONS
THEY CAN FOLLOW UP WITH THEIR
PHYSICIANS.
SO IN THIS STUDY, WE USE A --
AND THEN WE -- BECAUSE THEY
SOURCE HAVE CONTENT THAT ARE
MORE FAIRLY GENERALLY
UNDERSTANDABLE BY PATIENTS US
USING -- LANGUAGE AND AS THE
CONTENT ARE VIEWED BY MEDICAL
PROFESSIONALS.
SO WE LOOK AT PRIMARILY THE
SECTIONS ABOUT WHAT DO MY
RESULTS MEAN AND ALSO WHAT DOES
THIS RESULT HAVE OTHER NAMES,
AND OUR TEAM DEVELOPED THIS
ANNOTATED DATASET WITH USING A
TOOL INCEPTION, AND I WORK WITH
MY -- TO DEVELOP SOME EXAMPLE
ANNOTATION AND THEN WE HAVE --
TO DO ANNOTATION TOGETHER FOR
30-PLUS RECORDS TOGETHER TO
TRAIN THEM AND THEN AFTER THE --
HIGH WE ASK THEM TO AN TATE THIS
DATASET USING THIS TOOL.
AND THEN ONCE WE GET EVERY
RECORD ANNOTATED BY AT LEAST
THREE PEOPLE, WE PERFORM THE
CURE RANGE OF MOTCURATE THE DAT.
THIS INCLUDES THE LAB NAME,
ALTERNATIVE -- DIFFERENT KIND OF
CONDITIONS LIKE DIFFERENT AGE
GROUP FOR DIFFERENT LAB NORMAL
RANGE AND ALSO WHAT IS THE
INDICATION FOR ABNORMAL LAB TEST
RESULTS.
SO THIS WAY WE'LL BE ABLE TO
VIEW SOME SORT OF LIKE -- SO
THIS IS A GENERAL STATISTICS
ABOUT DIFFERENT -- BECAUSE WE
WANTED TO USE THIS TRAINING
DATA, THIS ANNOTATED DATA AS A
CORPUS TO TRAIN A MODEL FOR
DETECTING SUCH INFORMATION FROM
OTHER RECORDS, SO WE
EXPERIMENTED WITH DIFFERENT --
TRANSFORMER-BASED MODELS FOR
RECOGNIZING SUCH CONCEPTS.
SO WE TRIED TO FEED IN USING
PARAGRAPH AND SENTENCES, WE
FOUND THAT USING PARAGRAPHS
TYPICALLY GIVE US BETTER
PERFORMANCE IN TERMS OF STRICT
F1 LINEAR MATCH F1, I THINK
THAT'S MAINLY BECAUSE SOMETIMES
THE INFORMATION IN THIG THESE LB
RESULTS -- WE'RE ALSO TRYING TO
LINK THIS ANNOTATED DATA TO
STANDARD VOCABULARIES BASED ON
THE NATURE OF THE CONCEPTS.
AND WE FOUND SNOMED CT HAD A
PRETTY GOOD COVERAGE OF THOSE
DIFFERENT INFORMATION, NAMES IN
THE RECORDS COMPARED TO OTHER
VOCABULARIES.
AND IF WE COMBINE ALL OF THEM,
THE COVERAGE OF THE ANNOTATED
CONCEPTS RANGE BETWEEN 90% --
RANGE BETWEEN 60% TO 90%.
SO IT SEEMS THAT A LOT OF
ENTITIES CAN BE LINKED TO
STANDARD VOCABULARIES.
SO IN TERMS OF USE CASES, WE
COULD CONSTRUCT MORE
PERSONALIZED QUESTION PROMPTS
FOR PATIENT BASED ON LAB RESULTS
AND ALSO BASED ON THEIR
CONDITIONS THAT WE CAN -- FROM
HEALTH RECORDS, SO WE CAN
CONSTRUCT QUESTIONS LIKE IS MY
ABNORMAL RESULT DUE TO LIVER
DISEASE OR USE OF A PARTICULAR
MEDICATION, AND PATIENTS CAN USE
THIS INFORMATION TO FOLLOW UP
WITH A PHYSICIAN.
AND ALSO AS YOU ALL KNOW,
BECAUSE RECENTLY THERE IS A
REALLY BOOMING PHENOMENON OF A
LARGE MODEL FOR A VARIETY OF
MEDICAL-RELATED USE CASES, AND
WE'RE ALSO TRYING TO ALSO
EXPLORE THE USE OF A M MODEL FOR
LAB TEST RESULTS AND
INTERPRETATION.
SO THIS IS NOT SOMETHING NEW.
THIS A GROUP OF PEOPLE FROM RICE
UNIVERSITY TEXAS A & M, THEY
ACTUALLY DEVELOPED A VERY NICE
TREE EVOLUTION SHOWING THE
EVOLUTION OF LARGE LANGUAGE
MODELS OVER TIME FROM 2018 TO
NOW.
SO WHERE HOPEFULLY YOU'RE GOING
TO SEE MORE MATURE SYSTEMS BEING
DEVELOPED AND ALSO BEING USED
FOR HEALTH-RELATED PURPOSES AND
BUT AS A HEALTH INFORMATICS
RESEARCHER, WE ALSO ARE VERY
CAREFUL ABOUT USING THEM FOR
MEDICAL PURPOSES BECAUSE YOU
KNOW, THIS IS A LOT OF KIND OF
CONCERNS REGARDING THE QUALITY
AND PROBLEMS WITH USE OF THE
SYSTEMS.
SO INITIALLY WHAT WE DID IS WE
LOOKED AT SOME -- WE WANT TO --
FOR EXAMPLE GENERALLY UNDERSTAND
HOW WELL -- MODELS CAN BE USED
FOR LAB TEST RESULTS
UNDERSTANDING, SO WE ALSO LOOK
AT THE RECENT LITERATURE AND
THERE ARE SOME RECENT PAPERS
THAT ARE ACTUALLY EVALUATING
THE -- OF LARGE ELECTION
MODELS -- LAB RESULTS
INTERPRETATION.
SO THIS PAPER PUBLISHED IN 2013,
2023, THEY FOCUSED ON CASES OF
PATIENTS RECEIVING LAB RESULTS
IN ROUTINE CHECKUPS, AND NOT
HAVING SPECIFICALLY ASKED A
QUESTION TO CHATGPT AND THEY
FOUND USING THIS GENERIC MORE
GENERAL QUESTIONS DEVELOPED BY
THIS GROUP OF EXPERTS, THEY
FOUND THAT LIKE MODELS LIKE
CHATGPT CAN GENERALLY IDENTIFY
DEVIATIONS FROM REFERENCE RANGE
AND ALSO GIVE -- INTERPRETATION
AS WELL AS SOME OVERALL
INTERPRETATION.
THE INTERPRETATION, WE KNOW IS
CONSIDERED AS SUPERFICIAL AND
NOT ALWAYS CORRECT.
AND ALSO SOMETIMES JUST
COHERENTLY, THEY DIDN'T REALLY
MAKE MEANINGFUL SUGGESTIONS
REGARDING FOLLOW-UP DIAGNOSTIC.
ANOTHER GROUP OF PEOPLE FROM
CORNELL AND OTHER INSTITUTIONS,
THEY ALSO LOOK AT HOW CAN
CHATGPT BE ABLE TO PROVIDE
INTERPRETATION TO MORE
PROFESSIONAL QUESTIONS.
THEY ASK MEDICAL DOCTORS TO ASK
QUESTIONS TO CHATGPT ABOUT THEIR
CONCERNS REGARDING PATIENTS' LAB
TEST RESULTS, SO THE QUESTIONS
IN THIS STUDY ARE MORE
COMPLICATED AND MORE HARDER TO
ANSWER.
SO THEY FOUND THAT IN ROUGHLY
50% TIMES, CHATGPT IS ABLE TO
PROVIDE REASONABLY ACCURATE
INTERPRETATION TO THESE
QUESTIONS.
AND FOR US, THE REASON WHY WE DO
THIS PARTLY IS ALSO BECAUSE WE
FOUND THAT MANY PATIENTS, WHEN
THEY RECEIVE LAB RESULTS
INFORMATION, THEY MAY TURN TO
SOCIAL MEDIA FOR ANSWERING
BECAUSE THEY MAY NOT HAVE THE
READY ACCESS TO THEIR DOCTOR,
SOG SOME OFSOSOME OF THEM ASK QN
SOCIAL MEDIA, THE RESPONSES ARE
ALSO QUESTIONABLE BASED ON THE
PREVIOUS LITERATURE, THE QUALITY
REALLY VARIES SIGNIFICANTLY AND
NOT ALL THE RESPONSES FROM THE
SOCIAL MEDIA WEBSITE ARE
ACCURATE OR RELIABLE.
SO WE WANTED TO LOOK AT --
DATASET, WHICH IS A GENERAL Q &
A DATASET THAT WE FOUND WHETHER
THERE ARE QUESTIONS ABOUT LAB
TEST RESULTS AND LAB RESULTS
INTERPRETATION, AND THEN HOW
WELL MODELS LIKE CHATGPT AND
OTHERS CAN ANSWER THOSE
QUESTIONS RAISED BY PATIENTS
WITHOUT ANY MODIFICATIONS, WE
JUST USE THEIR QUESTIONS AS IS.
SO IN ORDER TO IDENTIFY THOSE
QUESTIONS, WE FILTER QUESTIONS
BY LAB RESULTS.
THEN OUT OF THESE 12,000
QUESTIONS, WE ASKED OUR STUDENTS
TO ANNOTATE 500 QUESTIONS TO
DETERMINE WHETHER THEY ARE LAB
TEST RESULTS -- OFTENTIMES LAB
TEST QUESTIONS, EVEN THOUGH THEY
MAY CONTAIN LAB TEST INFORMATION
OR LAB TEST NAMES, THEY MAY NOT
BE TEUT TEULLY ABOUT LAB RESULTS
INTERPRETATION.
SO THEN WE -- ARTIFICIAL
LEARNING MODELS FOR CLASSIFYING
THOSE QUESTIONS INTO LAB
RESULTS -- RELATED AND NOT.
SO WE EXPERIMENTED WITH ALL OF
THESE MODELS AND BASED ON
RESULTS, WE FOUND THAT
CLINICALBERT IS BETTER IN
DETERMINING AND CLASSIFYING THE
LAB TEST QUESTIONS, AND WE
COMBINE THE WITH MACHINE
LEARNING METHODS, WE CAN
GENERALLY HAVE A PRETTY HIGH
ACCURACY IN DETERMINING THE LAB
RESULTS QUESTIONS CATEGORY.
AND THEN WE WORK WITH OUR
PRIMARY CARE PHYSICIAN
COLLABORATORS ON THE TEAM TO
IDENTIFY 53 QUESTIONS THAT
CONTAIN RESULTS OF BLOOD AND
URINE LAB TEST ON MAJOR PANELS
INCLUDING COMPLETE -- METABOLIC
PANEL, THYROID FUNCTION, EARLY
MENOPAUSE OR -- PANEL.
THEN WE EXAMINED THE AVAILABLE
LARGE MODELS AND WHICH MODELS
SHOULD BE EVALUATED IN THE
STUDY.
WE FOUND CHAT CHATGPT -- THAT CE
ACCESSED THROUGH WEB PORTAL AND
ALSO API CAN BE USED, ALSO
MEDALPACA, LLAMA2 BY META AND
ALSO ORCA MINI, BASED ON LLAMA
2, BECAUSE WE THOUGHT THE
RESPONSES ARE REASONABLE,
GENERALLY SPEAKING.
SO HERE IS AN EXAMPLE, SO AFTER
WE GENERATE RESPONSES FROM THESE
MODELS AND ANALYZED THEIR
QUALITY, YOU SEE -- MORE FOCUSED
ON ASSESSING THE SIMILARITY OF
THOSE RESPONSES.
AND THEN WE ALSO ASK EXPERTS
WHERE PHYSICIANS AND ALSO MD
TRAINED POSTDOC WITH MEDICAL
BACKGROUND TO MANUALLY EVALUATE
THE RESPONSES, SOME WERE ASKED
THAT AND THEN COMPARE RESULTS OF
THOSE DIMENSIONS.
SO HERE ARE SOME OF THE
RESPONSES FROM LIKE MODELS.
SO THIS ONE IS A QUESTION THAT
WE FOUND WHICH ASKS FOR SOME
URINE TEST RESULTS ON CREATININE
VALUE, AND ALSO
PROTEIN/CREATININE RATIO, AND
THIS IS THE RESPONSE WE GOT FROM
LLAMA 2 MODEL, AND ALSO WE TOOK
THIS SCREEN SHOT FROM THE ONLINE
SURVEY BECAUSE WE GENERATE FOR
PHYSICIAN TO EVALUATE AND WE
DON'T WANT THE PHYSICIAN TO KNOW
WHICH MODELS WE USED SO WE HIDE
THIS IN THIS SCREEN.
SO THIS IS THE RESPONSE FROM
ORCA MINI MODEL AND YOU CAN SEE
THAT THEY HAVE PROVIDED A FAIRLY
COMPREHENSIVE RESPONSES TO THESE
QUESTIONS WITH LOTS OF
EXPLANATIONS, AND ALSO THIS IS
THE RESPONSES FROM CHATGPT,
GPT-4, AND GPT-4 EVEN TEND TO BE
MORE KR COMPREHENSIVE LONGER.
BUT LOOKING AT THE RESPONSES,
THERE WAS -- TEND TO ALSO
PROVIDE A LOT OF DISCLAIMERS
LIKE SAYING I'M NOT A DOCTOR AND
YOU SHOULD CONSUL YOUR PHYSICIAN
PROVIDERS TO GET A MORE ACCURATE
DIAGNOSIS AND TREATMENT OPTIONS
BASED ON YOUR COMPLETE
EVALUATION MEDICAL HISTORY.
SO THEY'RE REALLY TRYING TO MAKE
SURE THAT THEY DON'T HAVE ANY
TROUBLE PROVIDING
MEDICAL-RELATED RESPONSES TO
THESE QUESTIONS.
AND THIS IS THE RESULT FROM
MEDALPACA.
SOMEHOW COMPARED TO OTHER LIKE
MODEL, MEDALPACA TEND TO BE VERY
SHORT AND VERY SUCCINCT
RESPONSES COMPARED TO OTHER
MODEL, AND WE'VE FOUND ALSO THAT
MEDALPACA'S TONE TENDS TO BE A
BIT MORE AGGRESSIVE THAN OTHER
LIKE MODELS.
HERE ARE SOME EXAMPLES OF HUMAN
ANSWERS.
WE WILL PROBABLY KNOW HUMAN
ANSWERS SOMETIMES ARE BETTER.
IN SOME OF THE CASES WE FOUND
THE HUMAN ANSWERS ACTUALLY ARE
SUCCINCT AND SHORT AND ALSO MAY
BE FOCUSED ON ONLY ONE ASPECT OF
THE QUESTION, NOT PROVIDING
EVIDENCE TO THESE RESPONSES.
AND BECAUSE MOST LIKELY, I THINK
MOST OF THESE ANSWERS CAN BE
ANSWERED BY PURE PATIENTS AND
NON-MEDICAL PROFESSIONALS.
SO FIRST WE LOOKED AT THEIR
LENGTH, WE FOUND THAT
GPT4 TENDED TO HAVE LONGER
RESPONSES THAN OTHER MODELS AND
THE RESPONSES FROM USERS,
ANSWERS TEND TO BE SHORTER WITH
RESPECT TO ALL THE THREE
ASPECTS, THE AVERAGE CHARACTER
IN THE RESPONSE, ALSO AVERAGE
WORKOUT, AND AVERAGE COUNT FOR
GPT RESPONSES IS LIKE FOUR TIMES
THAT OF HUMAN USES --LY SPONSES
AND SOME OF OUR HUMAN EXPERT
EVALUATORS ACTUALLY FOUND
GPT-4 IS SOMETIMES TOO LENGTHY
AND SOMETIMES GETS A LITTLE
IRRELEVANT.
WE'LL FIRST LOOK AT TEXTUAL AND
SOMATIC SIMILARITY OF THE RAO
SPONSES BECAUSE THOSE KIND OF
METRICS WERE PREVIOUSLY
OFTENTIMES USED TO EVALUATE
ANSWERING RESULTS AND THE USE
CASES FOR THOSE ARE TYPICALLY --
THEY HAVE THE STANDARDIZED QA
BENCHMARK DATASET, THE ANSWERS
ARE VERY HIGH QUALITY, SO THEY
CAN EVALUATE DURING THE
GENERATED RESPONSES ARE GOING TO
BE OF A SIMILAR QUALITY TO THE
REFERENCE -- AMONG THOSE
METRICS.
THESE METHODS ARE MOSTLY SUB --
STRUCTURE OF A -- THEY LOOK AT
EMBEDDING OF THE TOKENS, SO THEY
LOOK AT HOW SIMILAR TWO
SENTENCES IS WITH RESPECT TO THE
SUM OF -- TOKENS SO WE FOUND OUT
FIRST SCORE IS GENERALLY DOING A
BETTER JOB IN EVALUATING
SEMANTIC SIMILARITY BETWEEN THE
RESPONSES.
THE HUMAN RESPONSES ARE VERY
SHORT COMPARED TO LIKE MODEL
RESPONSES, AND THESE ARE RESULTS
WHERE WE USE DIFFERENT WREE F
RESPONSES GENERATED BY EITHER
THE HUMAN OR LENGTH MODELS AS
REFERENCE ANSWER AND WE WANT TO
SEE WHICH ANSWERS ARE SIMILAR TO
EACH OTHER, AND WE FOUND IF WE
USE GPT4 AS A REFERENCE --
RESPONSE, THE RESUMM RESPONSE FM
LLAMA2 IS CONSIDERED THE MOST
SIMILAR ONE ACCORDING TO FIVE
OUT OF SIX METRICS, AND ALSO
LLAMA 2 AND OCRA MINI GENERATED
RESPONSES THAT ARE SIMILAR TO --
WERE SIMILAR.
BUT -- GENERATED SCORED LOWER AS
A REFERENCE, AND WE ALSO FOUND
THAT HUMAN ANSWERS FROM -- DATA
WERE SCORED LOWEST AND LEAST
SIMILAR TO OTHER LIKE MODEL
GENERATED RESPONSES.
GPT1, YOU SEE YAHOO ANSWERS HAVE
A LOW SCORE.
AND FOR THE HUMAN EVALUATION, WE
LOOKED AT PREVIOUS LEADER TERM,
AND WE WANT TO LOOK AT WHAT KIND
OF METRICS WERE PREVIOUSLY
CONSIDERED BY THE PRIVATE
LITERATURE FOR LIKE MODEL
GENERATED RESPONSES EVALUATION
AND WE FOUND RELEVANCE
CORRECTNESS AND HELPFULNESS AND
POTENTIAL HARM FOR SAFETY ARE
GENERALLY CONSIDERED AS
IMPORTANT FACTORS FO FOR EVALUAG
RESPONSES BECAUSE RELEVANCE
TENDS TO FOCUS ON THE -- BETWEEN
INTERPRETATION, AI-GENERATED
RESULTS, WHETHER RESULTS ARE
RELEVANT TO THE QUESTION.
-- ACCURACY, TRUTHFULNESS AND
TECHNICAL ACCURACY OF THE
INTERPRETATION.
-- COULD BE CONSIDERED A VERY
RELEVANT AND VERY IMPORTANT
ASPECT BECAUSE IT CONSIDERED THE
ABILITY OF THE MODEL TO PROVE --
PROVIDE SOME INSIGHTS TO
PATIENTS, TO LAY PEOPLE, AND HOW
HELPFUL THE RESULTS WOULD BE
ALSO IN TERMS OF THE LEVEL OF
SUGGESTIONS AND WHETHER THEY
CONTINUE TO BRING USEFUL
INFORMATION TO PATIENTS.
AND CERTAINLY WE DON'T WANT THE
RESULTS TO BE HARMFUL SO THIS
DIMENSION IS IMPORTANT.
SO WE GIVE A 5 LIKENESS SCALE TO
THESE RESPONSES SO WE HAVE -- SO
WE USE SEVEN QUESTIONS OUT OF
THE -- IN TOTAL WE ASK EU MAN
EVALUATORS TO EVALUATE 35
RESPONSES, AND ALSO EACH
RESPONSE, WE ASK THEM TO
EVALUATE ON THESE FOUR
DIMENSIONS.
AND OUR SCORE IS GOING FROM 1 TO
5 GOING FROM VERY HIGH TO VERY
LOW, SO IT'S A LITTLE
COUNTERINTUITIVE BASED ON WHAT
TYPICALLY OTHERS -- WE EACH HAVE
TO POTENTIALLY MODIFY THIS.
SO THE LOWER THE BETTER,
GENERALLY SPEAKING.
WE FOUND COMPARED TO OTHER
MODELS, CHATGPT 4 TEND TO GIVE
BETTER -- IN DIMENSIONS AND THE
OTHERS ARE COMPARED TO CHATGPT
4, LLAMA 2 TEND TO -- BETTER
RESULTS COMPARED TO HUMAN
ANSWER.
FROM YOUTH RESPONSES, SOME
MEDICAL DOCTORS, PHYSICIANS,
THEY PROVIDE SOME GENERAL
FEEDBACK ON THESE RESPONSES.
THEY'RE SAYING FOR EXAMPLE LLA
LLAMA2, GENERALLY IS ABLE,
PROVIDE ALTERNATIVE MANAGEMENT
AND ALSO SHOW SOME LEVELS OF
EMPATHY AND -- ALSO CAN EXPLAIN
THE RESULTS IN DETAIL, BUT IN
SOME CASES, IT MAY BE A LITTLE
TOO AGGRESSIVE REGARDING THE
RECOMMENDATION, AND FOR GPT-4,
THEY CAN PROVIDE THE MOST
COMPREHENSIVE EX-PLAMATION OF
POSSIBLE CAUSES OF LAB TEST
RESULTS AND ESPECIALLY ABNORMAL
RESULTS, AND FOCUS ON THE
RECOMMENDATION BY -- SOMETIMES
GET TOO WORDY, THE STATEMENT
SEEMS TO BE SO SUPERFICIAL AND
MAY NOT PROVIDE ENOUGH
INFORMATION AND MAY NOT BE SAFE
FOR PATIENTS.
AND ALSO OUR REVIEWERS FOUND
THAT -- ANSWERS TEND TO FOCUS ON
ONE ASPECT OF A SCENARIO AND
IT'S NOT HELPFUL IN MANY CASES
IN COMPREHENSIVE CARE, AND ALSO
FOCUSED ON SOME ISOLATED LAB
VALUES.
SO WE ALSO LOOK AT SOME MORE
SPECIFIC KIND OF INTERPRETATION.
LIKE FOR EXAMPLE, FOR THIS
QUESTION GIVEN BY A PATIENT, WE
LOOK AT CHATGPT RESPONSES AND WE
ALSO FOUND THAT GPT RESPONSES
TEND TO BE SENSITIVE TO THE
PROMPT.
FOR EXAMPLE, IF YOU GIVE A
QUESTION WITHOUT NORMAL
REFERENCE RANGE, THEY TYPICALLY
WILL NOT MENTION THE NORMAL
RANGE IN THE RESPONSE.
BUT IF YOU PROVIDE REFERENCE
RANGE IN YOUR QUESTION, THEY
WILL TALK ABOUT HOW YOUR VALUE
IS CORRESPONDING TO THOSE VALUE
RANGE.
OUR DOCTOR LOOKED AT THESE
RESPONSES AND FOUND THEY COULD
PROB BLIL GIVE MORE --
ESPECIALLY WHEN HE SEES THIS
TOTAL CHOLESTEROL VALUE IS MUCH
LOWER THAN THE NORMAL REFERENCE
RANGE, AND HE SUGGESTS THAT THIS
COULD BE INDICATIVE OF
MEDICATION OR MEDICAL CONDITION
OR LAB ERROR, IT IS IMPORTANT TO
RECHECK SUCH UNUSUAL LOW VALUE
AND CONSIDER CLINICAL
CORRELATION.
BUT THESE ARE NOT REALLY MUCH
MENTIONED IN THIS CHATGPT
RESULTS, SO WE THINK THAT IS
IMPORTANT TO BE ABLE TO REALLY
GENERATE THESE RESPONSES BUT AT
THE SAME TIME ASK HUMAN EXPERT
OR DOCTORS TO EVALUATE RESPONSES
AND PROVIDE MORE INSIGHTS TO
PATIENT OR TO REMOVE POSSIBLY
INACCURATE INFORMATION.
SO GENERALLY SPEAKING THE MAJOR
FINDING WE OBTAINED FROM THIS
STUDY IS WE FOUND
GPT-4 OUTPERFORMED OTHER SIMILAR
MODELS TESTED BASED ON THEIR
EXPERT EVALUATION, AND ALSO
HUMAN ANSWERS FROM SOCIAL Q & A
ARE GENERALLY SHORT AND OFTEN
FOCUSED ON ONE ASPECT ONLY AND
POSSIBLY INACCURATE.
SO THEY'RE NOT GOING TO BE GOOD
AS GOLD STANDARD ANSWER FOR
EVALUATION PURPOSES SO WE HAVE
TO BE CREATIVE TO FIGURE OUT HOW
TO CREATE BETTER RESPONSES.
THERE ARE LIMITATIONS REGARDING
THE MODELS AS IT'S NOT VERY
INDIVIDUALIZED AND IT REALLY
DEPENDS ON THE QUALITY OF THE
PROMPT.
IF THE PROMPT DOESN'T PROVIDE
PERTINENT INFORMATION TO THESE
LAB RESULTS, LIKE MODELS
CERTAINLY MIGHT NOT BE ABLE TO
ASK QUESTIONS, TO ASK FOR
ADDITIONAL INFORMATION.
AND ALSO WEAR NOT SURE EXACTLY
WHAT KIND OF SOURCES WERE THEY
USED TO GENERATE RESPONSES, AND
ALSO SO THERE'S A POSSIBILITY
THEY MAY NOT USE -- ALWAYS USE
THE MOST CREDIBLE INFORMATION
SOURCES TO PRODUCE RESPONSES.
SO WE THINK THAT VALIDATION BY
MEDICAL EXPERTS IS STILL REALLY
VERY MUCH NEEDED TO REDUCE
INACCURATE INFORMATION.
AND REGARDING NEXT STEPS, WE
THINK THAT IT'S IMPORTANT TO BE
ABLE TO PROVIDE MORE FACTORS
INTO NORMAL RANGE UNDERSTANDING
BECAUSE A LOT OF LAB TESTS
ACTUALLY THEIR VALUE RANGE IS
ALSO DEPENDING ON THE GENDER AND
AGE GROUP AND OTHER FACTORS LI
LIKE -- PERIOD AND MENOPAUSAL
CONDITION.
AND ALSO WE THINK THEY SUPPO
SUPPORT -- PROMPT -- AND ALSO
EVALUATING DIFFERENT PROMPTING
STRATEGIES TO FIGURE OUT WHAT
KIND OF STRATEGIES CAN WE
PROVIDE TO EITHER PATIENTS OR
PROVIDERS FOR THEM TO GET BETTER
INSIGHTS INTO LAB RESULTS.
AND ALSO WE SHOULD ALSO BE
CONSIDERING AUGMENTING MODELS
WITH CREDIBLE INFORMATION
SOURCES.
SO WE CAN CONTROL THE QUALITY OF
RESPONSES.
SO HERE IS SOME PROPOSED
PIPELINE THAT WE ARE RIGHT NOW
EXPERIMENTING WITH, SO NOW WE'RE
LOOKING AT EXISTING CREDIBLE
SOURCES FROM TABULAR LAB RESULTS
INTERPRETATION FROM A VARIETY OF
SOURCES LIKE AMERICAN BOARD OF
INTERNAL MEDICINE, STANFORD
MEDICINE AND ALSO LOOKING AT
UNSTRUCTURED -- SOURCES LIKE
MAYO CLINIC, MEDLINEPLUS, AND
ALSO NCBI BOOKSHELF STATPEARLS,
AND ALSO WE -- STRUCTURE DATA,
ANSWER DATA INTO COMPUTABLE
FORMAT, AND THEN FED THEM INTO
THE EMBEDDING SPACE USING THINGS
LIKE FAST TEXT AND OTHER
METHODS, LOADING THEM INTO A
VECTOR DATABASE.
SO FOR THE QUESTION PROMPT SIDE,
WE WANT TO BE ABLE TO USE LARGE
MATH MODELS TO INTERPRET, TO
INTERACT WITH USERS, TO AUGMENT
THEIR USERS SO THAT THE QUESTION
PROMPT CAN BE MORE COMPLETE AND
INCLUDE RELEVANT INFORMATION,
AND THEN -- PARTICULAR QUESTION
FROM CREDIBLE SOURCES AND THEN
GENERATING THE RESPONSES,
RETRIERETRIEVAL AUGMENTED GENERN
TECHNIQUE AND ALSO TO PROVIDE
FEEDBACK ON RESPONSES FOR THE
GENERATIVE RESPONSES.
SO THIS IS STILL A WORK IN
PROGRESS, AND WE'RE VERY EXCITED
TO BE ABLE TO DO THIS TYPE OF
RESEARCH IN THIS AREA OF LARGELY
MODEL AND GENERATIVE A.I., AND
WE FOUND THIS REALLY EXCITING
AND A BOOMING FIELD AND WE HOPE
TO BE ABLE TO CONTRIBUTE
IMPORTANT EXPERIENCES AND ALSO
EXPLORATION TO HOPEFULLY HELP
PATIENTS HAVE AN EASIER ACCESS
TO AND BETTER UNDERSTANDING OF
LAB RESULTS AND IMPROVING THEIR
HEALTH OUTCOMES AND REDUCE
HEALTH DISPARITY AS WELL.
SO THIS IS ACKNOWLEDGMENT OF ALL
THE PEOPLE I'VE BEEN WORKING
WITH IN THE PAST FEW YEARS ON
THIS TOPIC, ESPECIALLY MY
POSTDOC SCHOLAR, DR. LIU'S
POSTDOC, ALSO COLLABORATING,
IT'S AN ENORMOUS HONOR TO BE
ABLE TO DO IT, I'M REALLY HAPPY
TO WORK WITH ALL OF THE PEOPLE
ON THIS TEAM.
AND I THINK THAT'S IT FOR MY
PRESENTATION FOR TODAY.
I THINK I HAVE A FEW MINUTES FOR
QUESTIONS.
THANK YOU.
>> DR. HE, THANK YOU SO MUCH.
THIS WAS VERY, VERY, VERY
INTERESTING PRESENTATION, VERY
EXCITING.
TRULY EXCITING TO LIVE IN THE
TIME WHERE WE CAN EXPLORE THE
APPLICATIONS OF LLM TO EVERY
POSSIBLE FIELD.
AND MY QUESTION WAS GOING TO BE
IF YOU HAVE TRIED RETRIEVAL
ELEMENT IN GENERATION AND YOU
ANSWERED IT, THIS IS A WORK IN
PROGRESS, SO IT'S EXCITING TO
SEE AT YOUR RESULTS AND IF YOU
CAN IMPROVE BECAUSE THIS SEEMENS
TO FOCUS YOUR LLM IN THE RIGHT
DIRECTION AND ENSURE THAT IT
PRODUCES -- RESULTS --
>> WE'RE REALLY TRYING HARD TO
IMPLEMENT SOME OF THE RETRIEVAL
AUGMENTED GENERATION FRAMEWORK.
YOU KNOW, -- OVER THERE RIGHT
NOW, SO WE'RE TRYING TO FIND
ALSO CREDIBLE SOURCES FROM THE
TABULAR SOURCES AND -- AND WE'RE
ALSO VERY FORTUNATE TO BE ABLE
TO CONNECT WITH PEOPLE FROM
MEDLINEPLUS, WHICH ARE ACTUALLY
WORKING AT NLM, TO GIVE US SOME
TECHNOLOGY, LIKE API, IT DID NOT
EVEN ALLOW US TO QUERY
MEDLINEPLUS CONTENT USING
STANDARDIZED MEDICAL TERMINOLOGY
LIKE LOINK WHICH REALLY GIVES US
A LOT OF GREAT ACCESS TO THOSE
RESOURCES FOR THIS TYPE OF WORK.
>> YEAH, YEAH, THAT'S THRIE
AMAZING RESOURCE.
AND FOR OUR AUDIENCE, PLEASE
DON'T HESITATE TO SEND ME YOUR
QUESTIONS TO MY EMAIL.
IT'S LANA.YE IMGANOVA@NIH.GOV.
THANK YOU FOR DISPLAYING.
SEND IN YOUR QUESTIONS AND WHILE
I'M WAITING FOR YOUR QUESTIONS,
I'M GOING TO ASK MY QUESTIONS.
SO ON ALL OF THE MODELS THAT YOU
USED, ALL OF THE LLM MODELS THAT
YOU USED HERE, NONE OF THEM ARE
OPEN SOURCE, CORRECT?
>> ACTUALLY WE USED LLAMA 2 IS
OPEN SOURCE, I BELIEVE.
THE WAY WE USE IT IS THAT A LOT
OF THESE MODELS ARE KIND OF --
INTO A LOCAL MACHINE, SO WE USE
A FRAMEWORK CALLED A LANCHE THAT
ALLOWS US TO -- USING LOCAL
MODELS BECAUSE ONE CHALLENGE OF
USING GPT MODEL IS BECAUSE IT'S
A COMMERCIALIZED MODEL AND IT'S
NOT OPEN SOURCE, AND WE CAN ONLY
ACCESS IT EITHER THROUGH PATIENT
WEB PORTAL OR USING API, SO IN
EITHER CASE, THE RESPONSES SORT
OF DEPENDS ON THE TYPE OF
SERVICES BEHIND THE SCENE AND
THEY KEEP KIND OF DEVELOPING
FURTHER, FURTHER, SOMETIMES IT
HARD TO CONTROL AND ALSO HARD TO
HOW TO MAKE SURE THAT THE
RESULTS ARE REPRODUCIBLE.
SO THAT'S WHY WE'RE RIGHT NOW
FOCUSING MORE ENERGY INTO LIKE
LLAMA 2 MODEL, WHICH IS OPEN
SOURCE MODEL FOR THIS TYPE OF
WORK AS WE CAN DO A LOT OF
THINGS WITH THAT.
>> OKAY.
THAT'S GOOD TO KNOW.
THERE'S A LOT OF PUSH IN THE
COMMUNITY TO REALLY -- IT SEEMS
LIKE GPT-4 IS LEADING AND IT WAS
CLEAR FROM YOUR RESULTS.
JUST WANT TO REITERATE THAT IT'S
PRETTY AMAZING SO BASICALLY ALL
OF THESE LLMs PROVIDED BETTER
RESULTS THAN RESULTS PROVIDED BY
PEERS.
OBVIOUSLY, YOU KNOW, WE WANT TO
REFER TO DOCTORS FOR THE MOST
CORRECT INFORMATION, BUT LLM --
THEY PROVIDE REASONABLE
EXPLANATIONS AND
INTERPRETATIONS.
AND THERE IS A LOT OF PUSH IN
THE COMMUNITY NOW TO KIND OF
DEVELOP THOSE OPEN SOURCE MODELS
THAT ARE SIMILAR IN PERFORMANCE
TO A GPT -- AND IT LOOKS LIKE
WHILE WE'RE NOT THERE YET BUT WE
ARE ON THE WAY TO GET THERE.
>> MM-HMM.
SO THIS WAS INTERESTING YOU ARE
MENTIONING HOW TO INTERPRET LAB
RESULTS, BUT YOU ARE EMPLOYED AT
THE -- YOU ARE AFFILIATED IN
THE -- FOR SUCCESSFUL LONGEVITY.
ARE THERE ANY PROGRAMS IN YOUR
INSTITUTE THAT ARE GEARED
TOWARDS PREVENTION OR
SUSTAINABILITY OF A HEALTHY
LIFESTYLE RATHER THAN TREATMENT?
>> IN OUR INSTITUTION, WE'RE
ACTUALLY VERY FOCUSED ON HEALTHY
BEHAVIOR PROMOTION AND OUR
INSTITUTE -- THE GOAL IS TO
SUPPORT SUCCESSFUL LONGEVITY
AMONG EALDER ADU OLDER ADULTS, Y
FROM -- PERSPECTIVE, MOSTLY --
ACCEPTANCE TOWARDS USING CHATGPT
OR GENERATIVE A.I. MODELS FOR --
PURPOSES, SO ALL OF THESE THINGS
ARE GOING TO BE REALLY
INTERESTING TO LOOK AT, BECAUSE
WE DON'T KNOW, YOU KNOW, WE
OFTENTIMES LOOK AT THIS FROM A
SCIENTIST PERSPECTIVE, HOW IT'S
GOING TO BE USEFUL, BUT WE DON'T
KNOW HOW INDIVIDUAL USERS OR
PATIENTS OR PEOPLE WHO ACTUALLY
NEED THESE SERVICES ARE LOOKING
AT THESE RESPONSES, WHETHER
THEY'RE TRUSTWORTHY OR WHAT KIND
OF CONCERN DO THEY HAVE, TO THIS
IS DEFINITELY SOMETHING THAT
WE'RE TRYING TO LOOK AT, AND
ANOTHER THING IS THAT WE'RE ALSO
GENERALLY INTERESTED IN KIND OF
COLLABORATING WITH ANY --
INTERESTED IN -- THERE'S A LOT
OF PEOPLE HERE WHO ARE
SPECIALIZED IN COMMUNITY HEALTH,
LOOKING AT PREVENTION AND LATE
ONSET OF ALZHEIMER'S DISEASE,
DIFFERENT TYPE OF MODALITIES
WHERE ALSO DOING A RESEARCH
PROJECT WITH CUTTING TRAINING
PROGRAM O FOR POSSIBLE KIND OF
EARLY DETECTION OF ALZHEIMER'S
AND DEMENTIA, AND SO IN THAT
PARTICULAR PROJECT, I THINK
THERE'S A LOT OF INTERESTING
FINDINGS AS WELL WE'RE
PUBLISHING.
>> THAT SOUNDS VERY BEING
SITING.
DO YOU ALSO AGREE THAT THERE IS
A -- SOCIAL ASPECT IN KIND OF
HEALTHY AGING AND LONGEVITY?
LIKE PEOPLE -- WHEN WE AGE, WE
KIND OF DEPEND ON THOSE SOCIAL
CIRCLES FOR SOCIAL INTERACTIONS,
BUT ALSO FOR PHYSICAL EXERCISE
AND STUFF LIKE THAT, SO IS THERE
ANYTHING IN YOUR INSTITUTE THAT
IS KIND OF PROMOTING SOCIAL
INTERACTIONS BETWEEN PEOPLE?
>> WE'RE REALLY PROMOTING SOCIAL
INTERACTIONS.
DURING COVID ESPECIALLY, WHEN
PEOPLE ARE ISOLATING FROM
WHATEVER THEY DO, WE ACTUALLY
DID A LOT OF LIKE TRAINING
SESSIONS TO ALLOW PEOPLE TO
LEARN HOW TO USE ZOOM,
ESPECIALLY FOR OLDER PEOPLE WHO
ARE NOT USED TO A LOT OF
TECHNOLOGIES NEED A LOT OF
TRAINING.
SO WE DEFINITELY ARE DOING A LOT
TO PROMOTE THAT, AT BOTH THE
SOCIAL CONNECTEDNESS ASPECT OF
THIS.
WE ALSO ARE CURRENTLY
COLLABORATING WITH -- THERE ARE
ACTUALLY TWO OTHER INSTITUTES
HERE AT FSU, WE'RE DOING -- FOR
RESEARCH AND WE'RE RIGHT NOW
ORGANIZING A SERIES OF EVENTS
LIKE MUSIC EVENTS TO ALLOW
PEOPLE TO GET TOGETHER AND TO
INTERACT WITH EACH OTHER AND TO
SOCIALIZE.
SO WE HOPE TO BE ABLE TO KIND OF
DO A LOT OF THINGS WITH
COMMUNITIES HERE IN THE AREA.
FOR HEALTHY AGING.
>> THAT'S INCREDIBLE.
SO I HAVE A QUESTION AND THEN A
QUESTION FROM THE AUDIENCE.
SO ANY OF THOSE MODELS ARE
IMPLEMENTED, ANY OF THOSE
AGENTS, LLM AGENTS, ARE THEY
IMPLEMENTED IN YOUR ELECTRONIC
HEALTH RECORDS OR INTERPRETATION
OF LAB RESULTS OR EVERYTHING IS
STILL IN THE RESEARCH STAGE?
>> RIGHT NOW EVERYTHING IS STILL
IN THE RESEARCH STAGE, BUT WE
CURRENTLY ARE WORKING ON A
PROJECT, LAB GENIE, AND THIS
PROJECT, IF WE GET FUNDED FROM
OUR FUNDING AGENCY, WE'LL BE
ABLE TO WORK DIRECTLY WITH OUR
HEALTHCARE PARTNER TO DEVELOP
THIS TOOL THAT CAN INTERACT WITH
THE EHR SYSTEMS USING FAIR API
SO PATIENTS WILL BE ABLE TO GET
THEIR -- DIRECTLY WITH THIS TOOL
AND INCLUDING -- TO PRODUCE
IMPORTANT LIKE INSIGHTS INTO
THIS DATA AND BACK TO PATIENTS.
BUT RIGHT NOW, IT'S STILL MORE
IN AN EXPLORATORY PHASE BUT
WE'RE LOOKING FORWARD TO
DEVELOPING THE SYSTEM WITH THIS
TYPE OF SUPPORT.
>> THAT SEEMS TO BE WHERE THE
COMMUNITY IS GOING, AND TO IP
PLEIMPLEMENT THOSE MODELS AND
IMPLEMENT THOSE SMART AGENTS WHO
WOULD BE ABLE TO COMMUNICATE
WITH THE USERS.
>> ABSOLUTELY.
>> WELL, I WANT TO THANK YOU
AGAIN.
THIS WAS A VERY, VERY
INTERESTING CONVERSATION, AND
PLEASE LET US KNOW HOW YOUR
RETRIEVAL AUGMENTED GENERATION,
WE'RE VERY INTERESTED TO SEE
WHAT RESULTS DO YOU GET THERE,
AND WE WILL BE FOLLOWING YOUR
RESEARCH AND THANK YOU AGAIN FOR
YOUR TIME AND VERY INTERESTING
TALK, AND FOR JOINING US TODAY.
>> THANKS SO MUCH FOR THE
OPPORTUNITY.
AND THANK YOU SO MUCH, LANA, FOR
ALL THE HARD WORK IN ORGANIZING
THIS AND MAKING THIS HAPPEN.
I REALLY APPRECIATE IT.
AND ALSO THANK DR. LIU FOR
REALLY MAKING THIS HAPPEN, KIND
OF LEADING THIS EFFORT AND
WORKING COLLABORATIVELY ON THIS
PROJECT.
THANK YOU SO MUCH.
>> ABSOLUTELY.
IT WAS GOOD TO HAVE YOU.
>> THANK YOU. 1476 00:57:28,077 BYE-BYE.