ASSOCIATE DIRECTOR FROM
1965-1983, A CONTRIBUTOR AT THE
NATIONAL CANCER INSTITUTE,
LIBRARIANS AND INFORMATICS
ENGINEER.
THIS IS TO STIMULATE LIAISON
BETWEEN YOUR ORGANIZATIONS.
SO IN SPIRIT OF THAT GROWTH AND
COLLABORATION, YOU ARE ALL
INVITED TO PARTICIPATE IN A
CONVERSATION.
YOU WILL SEE A LIVE FEEDBACK
BUTTON UNDER THE VIDEO STREAM
SHORTLY.
PLEASE USE THIS FEATURE TO
SUBMIT QUESTIONS FOR OUR
SPEAKER.
FOLLOWING THE LECTURE, THERE
WILL BE A Q&A SESSION,
MODERATED BY NLM'S KEN KOYLE
WHO SERVES AS DEPUTY CHIEF,
PLEASE USE THE LIVE FEEDBACK
BUTTON BELOW THE VIDEO TO
SUBMIT YOUR QUESTIONS.
WITH THAT HOUSEKEEPING NOTE OUT
OF THE WAY, I'M HONORED TO
INTRODUCE OUR ESTEEMED SPEAKER
TODAY, DR. MAIA HIGHTOWER.
CEO AND CO-FOUNDER OF EYE
QUALITY AI AND CHIEF
INFORMATION OFFICER OF
UNIVERSITY OF CHICAGO, MEDICINE.
DR. HIGHTOWER IS A LEADING
VOICE IN THE INTERSECTION OF
HEALTHCARE, DIGITAL
TRANSFORMATION AND HEALTH
EQUITY.
SHE IS A CHAMPION FOR
RESPONSIBLE AI, ENSURING THE
DIGITAL FUTURE OF HEALTHCARE IS
EQUITABLE AND JUST.
HER COMPANY SEEKS TO DETECT AND
REMEDIATE AI BIAS IN HEALTHCARE
AND ALIGN AI STRATEGY WITH
HEALTH OUTCOMES AND EQUITY.
SHE IS A FOUR TIME C-SUITE
EXECUTIVE, SPANNING HEALTHCARE
IT, MEDICAL AFFAIRS AND
POPULATION HEALTH, CLINICALLY
INTEGRATED NETWORKS AND
HEALTHCARE TECH COMPANIES.
SHE RECEIVED HER B.A. IN
CORNELL AND M.D. AND MASTERS OF
PUBLIC HEALTH FROM ROCHESTER
SCHOOL OF MEDICINE.
INTERNAL MEDICINE AND
PEDIATRICS AT UNIVERSITY OF
CALIFORNIA, SAN DIEGO AND HOLDS
AN MBA FROM WHARTON SCHOOL OF
PENNSYLVANIA.
I'M VERY PROUD TO HAVE DR.
HIGHTOWER WITH US TODAY.
NLM PLAYS AN IMPORTANT ROLE IN
THE RESPONSIBLE USE OF AI IN
BIO MEDICINE.
AND RECOGNIZING AND ELIMINATING
AI BIAS TO DRIVE DISCOVERY AND
IMPROVE HEALTH FOR ALL PEOPLE.
WITH THIS IN MIND I LOOK
FORWARD TO LEARNING FROM DR.
HIGHTOWER'S INSIGHT ON
RESPONSIBLE AI.
PLEASE JOIN ME IN WELCOMING DR.
HIGHTOWER TO THE LECTURE.
>> Maia Hightower: HELLO,
THANKS FOR HAVING ME.
THIS IS ONE OF MY MOST FAVORITE
TOPICS TO TALK ABOUT.
LITTLE DID YOU KNOW WHEN YOU
CLICKED THAT JOIN BUTTON YOU
WERE SIGNING UP, YOU WERE
SAYING, HOST LET ME ON THE TEAM.
I WANT TO PLAY A ROLE IN MAKING
SURE THE DIGITAL FEATURE, THE
AI FEATURE OF HEALTHCARE IS
EQUITABLE AND FAIR.
SO ON THE ZOOM TODAY YOU WILL
KNOW YOU WILL HAVE A GOOD SENSE
OF WHAT YOUR ROLE IS TO ENSURE
THAT WHOEVER YOU REPRESENT, YOU
ARE -- SERVE EVERYONE AND NOT
JUST A SELECT FEW.
WE ARE IN THIS AMAZING TIME IN
HEALTHCARE.
UNLOCK UNPRECEDENTED VALUE.
WE MEASURE VALUE BY THE VALUE
EQUATION.
QUALITY OVER COST, MULTIPLIED
BY EXPERIENCE.
AND MANY OF THE INNOVATIONS ARE
REALLY DRIVEN BY IMPROVING
CARE, DRIVING DOWN THE COST AND
IMPROVING THE EXPERIENCE OF ALL
STAKEHOLDERS IN THE HEALTHCARE
ECOSYSTEM.
PROVIDERS, PATIENTS, CARE TEAM
MEMBERS.
AND A.I. HAS THAT POTENTIAL TO
DRIVE EACH OF THESE METRICS,
REALLY DEFINING HOW AI CAN BE
MEASURED IN TERMS OF THE VALUE
IT PROVIDES.
EXAMPLES INCLUDE IMPROVING
CLINICAL QUALITY WITH A.I.,
DIAGNOSTIC IMAGING, THEIR
INCREDIBLE NEW DIAGNOSTICS THAT
ARE ACTUALLY EMBEDDED WITHIN
DEVICES.
SOME OF WHICH ARE FDA APPROVED,
FDA REGULATED THAT IMPROVE THE
ABILITY TO DIAGNOSE SAY
DIABETIC RETINOPATHY USING
ARTIFICIAL INTELLIGENCE.
TREATMENT RECOMMENDATIONS, AI
HAS THIS ABILITY TO SIFT
THROUGH VAST AMOUNTS OF
CLINICAL DATA AND PROVIDE
SUGGESTIONS ON WHAT TREATMENTS
MAY BE BEST APPROPRIATE FOR A
PARTICULAR PATIENT, ESPECIALLY
WHEN IT COMES TO ONCOLOGY AND
CLINICAL TRIAL MATCHING.
AND PREDICTIVE ANALYTICS IN
GENERAL, PREDICTING WHO IS AT
RISK.
WHO MAY MOST BENEFIT FROM AN
INTERVENTION IN ORDER TO AVOID
AN ADVERSE OUTCOME.
AI, ESPECIALLY WITH GENERATIVE
AI HAS NOW ALSO INTRODUCED THE
PROMISE OF REDUCING
HEALTHCARE'S ADMINISTRATIVE
BURDEN.
WE KNOW HEALTHCARE HAS AN
INCREDIBLE AMOUNT OF
ADMINISTRATIVE BURDEN THAT MAY
BE DRIVING PHYSICIAN BURN OUT,
NURSE BURN OUT, CLINICIAN BURN
OUT, PATIENT BURN OUT.
AI HAS THE POTENTIAL TO REDUCE
THAT BURDEN.
IMAGINE THE DAY WHERE THE EHR
IS A JOY TO USE, AS JOYFUL AS
SAY ONE'S iPHONE OR SOME OF THE
OTHER DELIGHTFUL DIGITAL
EXPERIENCES.
NOW EHR IS A LITTLE BIT FAR
FROM THAT, BUT THE ADVANCES IN
CLINICAL DOCUMENTATION
ASSISTANTS REALLY OFFERS THAT
PROMISE.
THERE ARE OTHER ADMINISTRATIVE
BURDENS THAT AI CAN HELP
AUGMENT OR IMPROVE OPERATIONAL
EFFICIENCY, INCLUDING
SCHEDULING AND BILLING AND
CODE, WORKFLOW OPTIMIZATION.
BUT THERE'S GOT TO BE A CATCH,
RIGHT?
BECAUSE OF COURSE THAT'S TOO
GOOD TO BE TRUE.
IF IT'S ALL GOOD, WHAT ABOUT
ARE THERE ANY THINGS WE SHOULD
BE CONCERNED ABOUT.
OF COURSE THE SKEPTICS IN THE
ROOM ARE CORRECT.
THERE ARE THINGS WE DO HAVE TO
CONSIDER.
THERE ARE CHALLENGES. A.I
BRINGS ON A NEW SET OF
CHALLENGES WE NEED TO CONSIDER.
ONE IS THE CLINICIAN OR HUMAN
IN THE LOOP VERSUS AUTO PILOT.
HOW DO WE ENSURE THE HUMAN IN
THE LOOP WHO IS ACTUALLY THE
DRIVER IN THE DRIVER SEAT IS
PROPERLY EQUIPPED AND HAS THE
CAPABILITY TO GUIDE AI AND HAVE
IT AUGMENT AND NOT BECOME
OVERLY RELIANT ON AN AUTOMATED
SYSTEM, OR A SYSTEM THAT HASN'T
YET EARNED THAT TRUST.
REGULATIONS, THEY ARE MOVING
QUITE FAST.
I WILL TALK ABOUT THAT.
THEY MAY NOT BE MOVING FAST
ENOUGH, THE WHOLE GOAL OF
REGULATION IS TO CREATE SOME
FLOOR, A FOUNDATIONAL
ACCEPTABLE USE TO ENSURE
STANDARDS IN SAFETY IS
ESTABLISHED.
SO MOVING FAST BUT MAYBE NOT
FAST ENOUGH.
QUESTIONS ABOUT DATA PRIVACY,
INTELLECTUAL PROPERTY AND
ALGORITHMIC BIAS.
ALL THE WAY TO THE WAY AN AI
SYSTEM IS DEPLOYED INTO A
SHARED DELIVERY SYSTEM,
WORKFLOW.
ALL CAN PERPETUATE AND AMPLIFY
BIAS THAT ALREADY EXISTS. WHILE
AI OFFERS INCREDIBLE POTENTIAL
TO HEALTHCARE, THERE ARE THESE
CHALLENGES.
WHEN WE THINK OF DR. MARTIN
LUTHER KING, OVER 50 YEARS AGO,
HE HAD THIS QUOTE, "OF ALL THE
FORMS OF INJUSTICE INEQUALITY,
INJUSTICE IN HEALTHCARE IS THE
MOST SHOCKING AND INHUMANE."
HOW MUCH HAVE WE CLOSED THE GAP
IN THE LAST 50 YEARS?
WE HAVEN'T MADE THAT MUCH
PROGRESS.
INCREMENTAL AT BEST.
AS WE ENTER INTO THIS NEW AI
ERA, OUR TOOLS CAN EITHER WIDEN
HEALTH DISPARITIES, REALLY
AMPLIFY THEM.
OR THIS IS AN INCREDIBLE
OPPORTUNITY TO USE THESE NEW
TOOLS TO NARROW AND ELIMINATE
HEALTH DISPARITIES.
WHEN IT COMES TO AI, ALGO
RHYTHMIC BIAS IN MACHINE
LEARNING, AI --
WEIZENBALM'S BOOK OF COMPUTER
POWER AND HUMAN REASON SUGGESTS
THAT BIAS COULD ARISE FROM THE
DATA AND CODING ASSUMPTIONS
EMBEDDED IN COMPUTER PROGRAMS.
A MEDICAL SCHOOL IN THE U.K.
DEVELOPED AN ALGORITHM TO
AUTOMATE A PORTION OF THEIR
ADMISSIONS PROCESS THAT
RESULTED IN DISCRIMINATION
AGAINST WOMEN AND MEMBERS OF
ETHNIC MINORITIES.
THE OUTPUT OF THE MODEL OF THE
ALGORITHM DISCRIMINATED AGAINST
WOMEN AND MEMBERS OF ETHNIC
MINORITY GROUPS. NOT SURPRISING.
IN 2016, WE BECAME MORE AWARE
AS AI CONTINUED TO INCREASE IN
ADOPTION ACROSS INDUSTRIES,
PROPUBLICA FOUND BIAS IN THE
COMPAS ALGORITHM FOR CRIMINAL
RECIDIVISM.
A PAPER DISCOVERED BIAS IN
HEALTHCARE THAT USED A MODELING
ERROR COST THAT HAS A
PREDICTION OF RISK AND
AMPLIFIED HEALTH DISPARITIES
AND REAL QUESTIONS SINCE 2020
HOW WE USE RACE IN VARIABLES IN
CLINICAL ALGORITHMS.
SO TO DIVE DEEPER, ALGO
RHYTHMIC BIAS IS THE DARK SIDE
OF AI.
OBERMEYER'S PAPER, WHAT THEY
FOUND ON A NATIONAL SCALE THIS
MODEL HAD BEEN DEPLOYED IN
HUNDREDS OF HEALTH SYSTEMS THAT
BLACK PATIENTS WERE 50% LESS
LIKELY TO BE REFERRED TO CASE
MANAGEMENT, DESPITE BEING
EQUALLY SICK.
THEY COULD MEASURE THE DECLINE
IN CASE MANAGEMENT RATES ON A
NATIONAL LEVEL, USING A
NATIONAL DATABASE.
THEY WERE ABLE TO MEASURE THIS
ANOMALY WHERE RATES OF REFERRAL
WERE GOING DOWN AND THEY
COULDN'T FIGURE IT OUT UNTIL
THEY LOOKED AND SAW IT WAS AN
ALGORITHM.
THE GOOD NEWS IS THEY WERE ABLE
TO DETECT THE BIAS AND MITIGATE
THE BIAS.
SO THERE ARE GOOD TOOLS OUT
THERE IN MACHINE LEARNING WHERE
YOU CAN ACTUALLY MEASURE BIAS
AND YOU CAN MITIGATE THAT BIAS
AND CREATE A FAIR MODEL.
THAT'S WHAT THEY DID.
THEY MEASURED THE OUTPUT OF
MODEL, USING METRICS WE OFTEN
CALL THEM FAIRNESS METRICS.
THEY WERE ABLE TO BETTER
ESTIMATE WHAT THE MODEL SHOULD
LOOK LIKE AND THEN FIND THE
BETTER LABEL.
THE BETTER LABEL TO PREDICT WHO
WOULD BENEFIT FROM CASE
MANAGEMENT.
WHAT THEY DID INSTEAD OF USING
COST AS A PROXY FOR RISK, OR
RISK OF ADVERSE HEALTH OUTCOME,
THEY USED A NUMBER OF DISEASES
ON THE PROBLEM LIST.
IN A CLINICAL SENSE, WE ALL
INTUITIVELY KNOW, SOMEONE IS
HIGHER RISK IF THEY HAVE MORE
MEDICAL PROBLEMS.
SOMEONE IS HIGHER RISK IF THERE
ARE DISEASES LIKE DIABETES, OR
BLOOD PRESSURE ARE NOT
CONTROLLED.
AND SO THOSE WERE THE LABELS
THEY USED TO PREDICT WHO IS AT
RISK, VERSUS COST OR HOW MUCH
DOLLARS ARE SPENT IN HEALTHCARE.
BECAUSE WE KNOW THAT
SYSTEMICALLY, BLACK PATIENTS
SPEND LESS WHEN EQUALLY SICK,
FOR A WIDE VARIETY OF
SOCIOECONOMIC AND HISTORICAL
FACTORS.
SO THEY FIXED THE ALGORITHM,
USED A NUMBER OF PROBLEMS ON
THE PROBLEM LIST, WHETHER OR
NOT DISEASE WAS CONTROLLED AND
THEY CREATED A DE-BIASED MODEL
THAT WAS EQUITABLE TO BOTH
WHITE PATIENTS AND BLACK
PATIENTS.
SO THAT PARTICULAR MODEL WAS
EXPOSED TO AT ONE POINT
ESTIMATED 80 MILLION PATIENTS
EXPOSED TO THIS BIAS MODEL.
THE TOTAL LOST OPPORTUNITY
COST, $1 BILLION OF LOST
OPPORTUNITY.
AND SO, YES BIAS AND
ALGORHYTHMIC BIAS, HAVE A
FINANCIAL TOLL.
WE TALK ABOUT HUMAN TOLL.
I'M A PHYSICIAN, INTERNIST,
PRIMARY CARE DOCTOR, SO MANY OF
MY PATIENTS HAVE COMPLEX
DISEASES AND NAVIGATING THE
HEALTHCARE SYSTEM IS
CHALLENGING.
WHEN OFFERED CASE MANAGEMENT TO
HELP NAVIGATE, IT COULD BE SUCH
A RELIEF TO PATIENTS AND
FAMILIES, THAT OPPORTUNITY COST
WAS NOT MEASURED.
BUT WE COULD INTUITIVELY
CALCULATE IT WAS A BURDEN ON
PATIENTS THAT WAS UNFAIR AND
UNJUST.
FIRST QUESTION, ACTUALLY AT
THAT TIME THIS MODEL WAS
RELEASED I WAS CHIEF POPULATION
HEALTH OFFICER AT A LARGE
HEALTH SYSTEM.
AND THIS MODEL WAS MOST LIKELY
DEPLOYED IN MY HEALTH SYSTEM.
AND I DIDN'T HAVE THE TOOLS TO
MEASURE OR MITIGATE.
BUT THE DEVELOPERS FIXED THE
MODEL, AND I THOUGHT PHEW!
GOOD THING THEY FIXED THAT
MODEL.
BUT LATER ON, 2020, SINCE THEN,
THE QUESTION WAS THIS A ONE-OFF
OR A FREQUENT EXPERIENCE?
SO AHRQ AND NIH DID A STUDY.
THEY DID A BROAD LITERATURE
SEARCH, THAT SEARCH IDENTIFIED
OVER 11,500 CITATIONS.
AFTER FULL REVIEW, 58 ARTICLES
MET CRITERIA.
MANY OF THEM INCLUDED MODELING
WIDE VARIETY OF HEALTHCARE
MODELS, ALGORITHMS, ONLY 58 MET
INCLUSION CRITERIA AND ONLY 14
STUDIES HAD MET ENOUGH OF THE
DUAL MULTIPLE INCLUSION
CRITERIA TO ANSWER THIS FIRST
QUESTION ON THE IMPACT ON RACE
AND ETHNIC DISPARITIES, DOES
ALGORITHMS INCREASE DISPARITIES
AND DOES IT, THAT PARTICULAR
QUESTION.
SO CAN YOU IMAGINE, OVER 10,000
STUDIES AND ONLY 58 STRATIFIED
THEIR OUTCOMES BY POPULATION.
AND STRATIFIED THE MODEL
PERFORMANCE BY POPULATION.
SO THAT'S BASICALLY WHAT WAS
NEEDED TO KNOW WHETHER OR NOT
HEALTH INEQUITIES.
THINK FOUND 10 OUT OF 14
PERPETUATED, 4 OUT OF 14
DECREASED.
OF THE FOUR THAT DECREASED IT
WAS BY DESIGN.
SO THE TAKE-HOME MESSAGE WHEN
INTENTFUL, THEY CAN DECREASE
HEALTH DISPARITIES.
WHEN LEFT TO CHANCE, MOST
LIKELY THEY INCREASE HEALTH
DISPARITIES, BUT THE BIGGEST
TRAVESTY IS MOST DO NOT EVEN
MEASURE STRATIFIED PERFORMANCE,
LET ALONE AFFECT ON OUTCOMES.
SO OF COURSE THAT'S A DIFFERENT
ERA.
NOW THERE HAVE BEEN A LOT OF
CHANGES ON HOW EVEN THE
JOURNALS WILL REVIEW A NEW
MODEL, WHETHER IT'S WORTHY OF
PUBLICATION AND REALLY DEMAND
SOME OF THAT STRATIFIED
PERFORMANCE.
SO BIAS OCCURS THROUGHOUT THE
AI LIFE CYCLE.
IT STARTS IN THE MOMENT OF DATA
CREATION.
AND FROM THAT DATA, REAL WORLD
DATA IN MANY CASES, ESPECIALLY
IN HEALTHCARE ALGORITHMS WE ARE
USING HR DATA.
SOMETIMES WE ARE LUCKY TO
CURATE DATA FROM A CLINICAL
TRIAL WHICH IS HIGHER QUALITY
THAN ANY ELECTRONIC MEDICAL
DATA.
BUT THAT DATA CREATION,
DEFINITELY HAS REAL WORLD BIAS
THAT IS REFLECTED IN THE
DATASET.
NEXT PART OF THE AI LIFE CYCLE
FROM PROBLEM FORMULATION, WHO
GETS TO DECIDE WHAT KIND OF
PROBLEMS TO SOLVE?
RIGHT, DOES COMMUNITY DETERMINE
WHAT PROBLEMS ARE IMPORTANT
THAT WE ADDRESS IT WITH AI
SYSTEMS OR IS IT HEALTHCARE
SYSTEMS THAT DETERMINE.
AND WHAT ARE THE UNDERLYING
DRIVERS OF THOSE DECISIONS?
ONCE THAT DATA IS ACQUIRED AND
THE DEVELOPMENT IS UNDER WAY
FOR MODEL DEVELOPMENT, A MODEL
TO BE DEVELOPED, THEN IT'S LEFT
TO THE INDIVIDUAL DATA
SCIENTISTS AND THEY ARE MAKING
HUNDREDS, IF NOT THOUSANDS OF
DECISIONS DURING THE DATA MODEL
DEVELOPMENT.
MODEL EVALUATION, WHETHER OR
NOT THE PERFORMANCE OF THE
MODEL WAS STRATIFIED BY
DEMOGRAPHICS IN ORDER TO ASSESS
IF THE MODEL IS ACCURATE.
OR THE OUTCOME OF THE MODEL
FAIR ACROSS SUB POPULATIONS.
THAT COULD BE DONE AND OFTEN
TIMES IT ISN'T.
COULD HAVE THE PERFECT MODEL,
AND THEN YOU DEPLOY IT IN AN
IMPERFECT WORLD.
ALL ACROSS THE AI LIFE CYCLE,
BIAS CAN OCCUR.
THE GOOD NEWS AGAIN, THERE ARE
REALLY GOOD MITIGATION METHODS
THAT COULD BE DEPLOYED
THROUGHOUT THE AI LIFE CYCLE.
THINK OF THE TECHNICAL ONES,
IT'S EASY TO THINK AI, LET'S
JUMP TO TECHNICAL BIAS
MITIGATION METHODS.
BUT THERE ARE SOME SOCIAL
MITIGATION METHODS THAT ARE
VERY EFFECTIVE AS WELL.
ONE OF THE GOALS OF NLM IS
WORKFORCE DEVELOPMENT AND
DEFINITELY WORKFORCE
DEVELOPMENT, HAVING DIVERSE
TEAMS.
IT'S NOT EVEN DIVERSE TEAMS
WHEN IT COMES TO SAY
DEMOGRAPHIC DIVERSITY.
IT'S LIVED EXPERIENCE DIVERSITY.
IT'S DIVERSITY IN THE TYPES OF
SKILLS THAT PEOPLE BRING TO THE
TABLE.
SO THAT TEAM DIVERSITY CAN HELP
BALANCE OUT DIVERSITY AND BIAS
THAT'S IN THE DATASET OR IN THE
WAY THAT WE MODEL.
GOVERNANCE, MAKING SURE YOU
HAVE LOCAL POLICIES,
PROCEDURES, THE ROLE OF
REGULATION IS TO CREATE THAT
FLOOR.
AND SOME OF THE MORE TECHNICAL
MITIGATION METHODS.
FROM REGULATIONS, IT MOVES ONTO
FRAMEWORKS AND STANDARDS AND
ACTUAL METHODS WHERE YOU ARE
DEPLOYING STATISTICAL METHODS
AND MACHINE LEARNING METHODS TO
BOTH DETECT BIAS AND MITIGATE
THAT BIAS.
AND THEN MONITORING FOR
OUTCOMES OVERTIME.
A LOT OF TOOLS AS WELL ARE
EMERGING TOOLS ON VALIDATION
AND MONITORING MODELS OVERTIME
FOR ADRIFT OF DEGRADATION OF
PERFORMANCE AND MAKING SURE
THEY ARE CONTINUALLY REFRESHED
TO MEET INSTITUTIONAL
THRESHOLDS OF PERFORMANCE.
SO WHEN WE TALKED ABOUT
DIVERSITY AND WE ARE TALKING
ABOUT CONCERN ABOUT HEALTH
EQUITY, AND THAT REALLY IS ONE
OF THE KEY DRIVERS.
WHEN I TALKED TO HEALTHCARE
LEADERS, THEY ARE REALLY
WORRIED ABOUT AI BIAS AND
MAGNIFYING HEALTH INEQUITY WHEN
THEY DEPLOY A.I.
YET ONLY 80% OF HEALTH EQUITY
LEADERS REPORT THEY HAVE LITTLE
OR NO DECISION MAKING IN THEIR
ORGANIZATIONS STRATEGY.
SO THERE'S THAT DISCONNECT.
OFTEN YOU HAVE AI STRATEGY ON
ONE PILLAR, YOUR HEALTH EQUITY,
DIVERSITY, EQUITY INCLUSION ON
ANOTHER PILLAR, AND THE TWO
DON'T MEET.
SO IT'S REALLY IMPORTANT THAT
HEALTH EQUITY, IF NOT LEADERS
THEMSELVES BUT THOSE THAT HAVE
HEALTH EQUITY AS CORE PRINCIPLE
OF WHAT THEY ARE DOING ARE
INCLUDED IN AI DECISION MAKING
AND AI STRATEGY.
SO WE TALKED ABOUT THE RISKS,
NOW THE REGULATIONS.
IN THE LAST SIX MONTHS THERE'S
BEEN A NUMBER OF REGULATIONS
THAT HAVE STARTED TO PROVIDE
THAT FOUNDATIONAL FLOOR.
HTI ONE WAS THE FIRST ON THE
HEALTHCARE FRONT.
ONC RELEASED VERY MADE FINAL
RULE, QUITE A MONUMENTAL BIT OF
REGULATION REQUIRING AI
TRANSPARENCY.
THEY WOULD DESCRIBE IT AS SIX
FEET WIDE AND A MILE DEEP.
IT'S A VERY DEEP SET OF RULES
WHEN IT COMES TO AI
TRANSPARENCY REQUIREMENTS FOR
CERTIFICATION.
THE SCOPE IS NARROW, BASICALLY
IT'S JUST THE 600 OR SO EHR'S
THAT ARE CERTIFIED BY ONC.
CMS FOLLOWED SUIT IN FEBRUARY,
MOSTLY LIMITING AI USE FOR
MEDICAID ADVANTAGE COVERAGE
DETERMINATIONS.
AND SIMILARLY, ACA LAST WEEK
RELEASED THEIR FINAL RULE
FOCUSING ON AI
NON-DISCRIMINATION PROTECTIONS.
ONE OF THE OTHER -- SO ONE OF
THOSE OTHER MONUMENTAL TYPES OF
REGULATIONS WAS THE EU AI ACT.
THAT WAS MADE FINAL IN MARCH
13, 2024.
SO JUST A FEW MONTHS AGO.
AND THAT ONE IS A MILE HIGH AND
MILE DEEP.
THE PENALTIES ARE STIFF FOR
MULTI-NATIONAL CORPORATIONS. IT
HAS WIDE-REACHING IMPACT.
FOR MOST OF OUR PHARMA
INDUSTRY, THAT HAS BEEN AN
INCREDIBLY IMPACTFUL
LEGISLATION.
SYSTEM IS PER --
PERPETUATING THAT BIAS.
UNDER PERFORMANCE OF AI
INVESTMENTS.
AND SOLUTION THAT AI GOVERNANCE
AT THE CORE, MOST FUNDAMENTAL
AND HAVING THAT ABILITY OF
MONITORING YOUR AI MODELS,
ALGORITHMS, MAKING SURE THEY
COMPLY WITH INSTITUTIONAL
POLICY AND STANDARDS AND IF THE
INSTITUTION DEEMS HEALTH EQUITY
IS HIGH PRIORITY, ALL THESE
RESPONSIBILITIES ARE A
PRIORITY, THAT WOULD BE
REFLECTED IN AN AUDIT.
FOLLOWED BY VALIDATION, ONGOING
ABILITY TO PROVIDE THOSE
AUDITS, NOT JUST A ONE-OFF,
TWO-OFF, BUT EVERY MODEL BEING
CONTEMPLATED OR DESIGNED IN THE
HEALTH SYSTEM AND THEN
MONITORING OVER TIME.
THESE ARE SOME OF THE QUALITY
ASSURANCE GOVERNANCE
FRAMEWORKS. ALL THE WAY FROM
U-CHICAGO, OBERMEYER AND HIS
TEAM WROTE THE ALGORHYTHMIC
BIAS PLAY BOOK.
IT'S A COMPREHENSIVE FRAMEWORK
TO MITIGATE RISK WHEN IT COMES
TO AI.
AND ISO, THE INTERNATIONAL
STANDARD FOR ARTIFICIAL
INTELLIGENCE MANAGEMENT SYSTEMS.
THREADING THE NEEDLE FOR AI
STANDARDS, ISO BROADEST,
INTERNATIONAL STANDARD.
NIST IS NATIONAL INSTITUTE
STANDARDS.
INDUSTRY AGNOSTIC.
NATIONAL ACADEMY OF MEDICINE
FOR THE INDUSTRY.
AND WHEN IT COMES TO OUR HIGHLY
REGULATED HEALTHCARE, THOSE
INDUSTRY STANDARDS AS WELL AS
STANDARDS UP ABOVE ARE DRIVING
THE FEDERAL REGULATIONS AND HOW
THEY ARE FORMULATED.
AND THEN INSTITUTIONS, DEVELOP
INSTITUTIONAL STANDARDS THAT
LINEUP, THREAD THE NEEDLE FROM
THAT INSTITUTIONAL STANDARD ALL
THE WAY PERHAPS TO THE NIST
STANDARD.
SO EXAMPLES OF AI STRATEGY
ALIGNMENT IS PRIORITIZING
HEALTH EQUITY AND MAKING SURE
COMPLIANCE HAS THAT
FOUNDATIONAL COMPONENT.
BUT THAT'S JUST THE FOUNDATION.
WE ARE MORE THAN JUST
COMPLIANCE, WHICH IS IMPORTANT.
BUT WE ALSO WANT TO MAKE SURE
WE ARE LIVING UP TO OUR MOST
IMPORTANT PRIORITIES.
AN AUDIT WOULD HAVE THAT
TECHNICAL AUDIT OF MODELS USING
METRICS, AS WELL AS PERFORMANCE
STRATIFIED BY POPULATION.
THAT STRATIFICATION COULD BE
RACE, GENDER, IT COULD BE AGE,
IT COULD BE DISABILITY STATUS,
RURAL VERSUS URBAN.
THERE ARE SO MANY WAYS YOU CAN
STRATIFY LOOKING AT VULNERABLE
POPULATIONS WHERE A MODEL MAY
NOT WORK.
AND THEN MAKING SURE THAT YOU
HAVE EITHER ENOUGH DATA, YOU
HAVE ENOUGH INPUT, YOU HAVE
ENOUGH INFORMATION TO BE ABLE
TO PROVIDE, BUILD A BETTER
MODEL FOR ALL THAT THE MODEL
MAY HAVE IMPACT.
THE OTHER SIDE OF AN AUDIT AND
THIS IS DEFINITELY IN ALIGNMENT
WITH ISO OR NIST IS THAT
PROCESS AUDIT.
MAYBE AS AN INSTITUTION YOU CAN
TECHNICALLY DO SOMETHING, BUT
MORE IMPORTANTLY, ARE YOU
FOLLOWING YOUR POLICIES AND
PROCEDURES?
SO WHEN IT COMES TO RESPONSIBLE
AI, THERE ARE CERTAIN
COMPONENTS OF STEPS THAT YOU
CAN DO DURING THE MACHINE
LEARNING OPS PROCESS TO MEASURE
FAIRNESS, MEASURE
EXPLAINABILITY IN A MODEL.
AND SO IN A TYPICAL MACHINE
LEARNING PRACTICE, DEVELOPMENT
PRACTICE, YOU START WITH YOUR
DATA.
YOU TRAIN YOUR MODEL.
YOU CHECK THE CURVE,
PERFORMANCE.
AND SOMETIMES THAT'S WHERE THE
TRADITIONAL PROCESS KIND OF
ENDS.
OKAY, MY END OF THE CURVE IS
0.8 5, GREAT.
WELL IN A FAIRNESS OR
RESPONSIBLE AI FRAMEWORK YOU
WOULD DO THAT SUB POPULATION
STRATIFIED PERFORMANCE.
SO YOU ARE LOOKING TO MAKE SURE
THAT, OKAY, AT THE AGGREGATE
LEVEL, THE MODEL PERFORMS SAY
AT A CERTAIN ACCURACY LEVEL.
RACE, ETHNICITY, GENDER, ET
CETERA.
SO YOU STRATIFY AND SEE HOW
THAT MODEL PERFORMS.
YOU CAN ALSO DO IT NOT ONLY BY
PERFORMANCE BY ALSO BY FAIRNESS
METRICS WHERE YOU ARE ACTUALLY
MEASURING THE OUTPUT OF THE
MODEL.
THE MODEL ACCURACY IS THE
OUTPUT OF THE MODEL MAY BE
PREDICTING WHETHER OR NOT
SOMEBODY WOULD BE BENEFITING
FROM PARTICULAR GOOD OR
SERVICE, IS A DISTRIBUTION OF
THAT GOOD OR SERVICE EQUITABLE
ACROSS SUB POPULATIONS.
THEN IF IT'S NOT, IF THE
PERFORMANCE OF THE MODEL OR
OUTPUT OF THE MODEL FALLS BELOW
A THRESHOLD, THEN YOU COULD
APPLY VARIOUS MITIGATION
METHODS.
AND THESE, SOME ARE MACHINE
LEARNING METHODS.
SOME ARE STATISTICAL METHODS,
THERE ARE A LOT OF DIFFERENT
METHODS OUT THERE.
THERE ARE A LOT OF DIFFERENT
WAYS OF RECTIFYING YOUR MODEL.
AND THEN RECYCLING IT THROUGH
THIS WHOLE PROCESS AGAIN.
WE CAN'T DO A TALK ABOUT
FAIRNESS METRICS WITHOUT
TALKING ABOUT THERE ARE OFTEN
TRADE-OFFS. IT'S VERY DIFFICULT
TO FIND THAT PRECISE BALANCE
BETWEEN ACCURACY AND FAIRNESS.
OFTEN WHAT WE WILL FIND,
ESPECIALLY WITH A LARGE DATASET
THAT HAS ENOUGH OF YOUR SMALL
SAMPLE SIZE OR SUB POPULATION
REPRESENTED THAT YOUR
PERFORMANCE MAY NOT GO DOWN
SIGNIFICANTLY.
OR IF IT DOES, YOU MAY FIND YOU
NEED A LARGER POPULATION.
THERE'S A LOT OF DIFFERENT
THINGS YOU NEED TO DO.
BUT THE IMPORTANT THING IS
THESE FAIRNESS METRICS DO HAVE
TRADE-OFFS.
SO WE ARE GETTING CLOSE TO WHAT
ARE THE ROLE, IN THE VERY
BEGINNING YOU SAID I HAD A ROLE
TO PLAY.
I WANT TO KNOW WHAT MY ROLE IS.
WHAT AM I SUPPOSED TO BE DOING
TO MAKE SURE THE FUTURE OF AI
IN HEALTHCARE IS EQUITABLE AND
FAIR.
IF ARE YOU ANY CONTRIBUTOR TO
NLM AND LIBRARY SCIENCE THERE'S
AN INCREDIBLE AMOUNT OF
OPPORTUNITY THAT NLM LIBRARIANS
CAN CONTRIBUTE TO RESPONSIBLE
AI.
ONE IS THE SUSTAINABLE DIGITAL
ECOSYSTEM AND FOSTERING THAT
ROBUST TRANSPARENCY ENVIRONMENT
FOR RESPONSIBLE AI DEVELOPMENT.
THE INITIATIVE IS FRONT AND
CENTER AND OPEN SCIENCE IS
ABOUT ENSURING AI RESEARCH IS
TRANSPARENT, REPRODUCIBLE,
ACCESSIBLE.
THERE'S AN OPPORTUNITY FOR
INCREASED TRANSPARENCY.
IT'S MEETING INSTITUTIONAL
GUIDELINES ON STRATIFIED
POPULATIONS. SO INCREDIBLE
OPPORTUNITY TO MARRY SCIENCE
CONCEPT WITH RESPONSIBLE AI
WITH PRODUCTS THAT COULD HELP
THE NATION AT LARGE.
WE TALKED A BIT ABOUT WORKFORCE
DEVELOPMENT.
THIS IS ONE OF THE MOST AMAZING
THINGS BEING IN TECH AND
HEALTHCARE IT COMPARED TO IT IN
GENERAL.
YOU LOOK AT TECH COMPANIES.
ONE THING HEALTH IT IT IS THE
DIVERSITY.
AND THAT HISTORICALLY HAS BEEN
BECAUSE OF THE DIVERSE PATHWAYS
TO INFO MAT ICS.
NURSING INFORMATICS PROGRAMS
ACROSS THE COUNTRY PRODUCE SOME
OF THE MOST LIVED EXPERIENCE,
DIVERSE PEOPLE IN HEALTHCARE IT.
THEY DON'T SAY IN CLASSICAL
INFORMATIC ROLES, THEY ARE DATA
SCIENTISTS, THEY ARE
PROGRAMMERS. THEY ARE PART OF
THE BROADER TECH ECOSYSTEM AND
LIBRARIANS THAT ARE INFORMATICS
TRAINED AS WELL HAVE THE SAME
BROAD LIVED EXPERIENCE THAT
REALLY HELP TO AUGMENT OUR
TEAMS.
COMMUNITY ENGAGEMENT.
THE ROLE OF LIBRARY MEDICINE
AND SCIENCES WHEN IT COMES TO
DISSEMINATING INFORMATION, NOT
ONLY TO MEDICAL PRACTITIONERS
BUT TO PATIENTS, TO COMMUNITY,
THROUGH RELATIONSHIPS. AT THE
UNIVERSITY OF UTAH, WHERE I WAS
THE CHIEF INFORMATION OFFICER.
WE HAD A WONDERFUL PATIENT
LIBRARY THAT THE LIBRARIANS OF
COURSE, MEDICAL LIBRARY TRAINED.
AND THAT ABILITY TO TRANSLATE
DIGITAL LITERACY TO PATIENTS,
TO PROVIDE THAT CONDUIT WAS
INCREDIBLY IMPORTANT.
AND WE ARE GOING TO NEED THAT
IN THIS NEW AI WORLD.
PATIENTS DON'T UNDERSTAND.
THEY DON'T UNDERSTAND AND WE
NEED TO PROVIDE THAT CAPACITY
BUILDING TO GIVE BROADER
COMMUNITY THE RESOURCES TO
THRIVE IN AI DECISION MAKING.
AND THEN PARTNERSHIPS IN
GENERAL.
SO LOTS OF WONDERFUL
OPPORTUNITY FOR NLM, FOR
LIBRARIANS ACROSS THE WORLD,
REALLY, TO DRIVE RESPONSIBLE AI
PRACTICES.
FOR DEVELOPERS, IT REALLY IS
DESIGNING YOUR MACHINE LEARNING
MODELS WITH SUB POPULATIONS IN
MIND.
TESTING THOROUGHLY, DON'T BE IN
A BOX.
MAKING SURE YOU ARE LISTENING
TO STAKEHOLDERS AND ADOPTING
NEW METHODS.
NEW METHODS ARE BEING DEVELOPED
FREQUENTLY OFTEN REALLY TO
ADDRESS FAIRNESS AND TO ADDRESS
RESPONSIBLE AI PRINCIPLES,
EXPLAINABILITY, TRANSPARENCY
AND KEEPING UP-TO-DATE.
FOR HEALTH EQUITY ADVOCATE,
GOING FROM CHANGE AGENTS TO
INNOVATORS. OFTEN WHEN WE THINK
OF HEALTH EQUITY, ADVOCATE ARE
IN THE ROOM.
BUT THERE'S THIS GREAT
OPPORTUNITY TO PARTICIPATE IN
AI AND TECHNOLOGY GOVERNANCE,
MAKING IT AN INCREDIBLY HIGH
PRIORITY, HAVING THAT SEAT AT
THE TABLE.
WHEN HAVE THAT SEAT AT THE
TABLE, MAKING SURE YOU ARE
EXPANDING THAT SEAT TO OTHERS,
INCLUDING DIVERSE COMMUNITY
VOICES IN RECOGNIZING WHO IS
MISSING FROM A LOT OF THE
DECISION MAKING WHEN IT COMES
TO AI.
MEASURING INEQUITY AND FAIRNESS.
OWN SURING THAT EQUITY GAPS
DRIVE-IN NOVATION.
WHEN YOU SEE THAT GAP, MAKING
SURE WE ARE ACTUALLY
ADDRESSING, USING OUR
INNOVATIVE ENGINE TO ADDRESS
THAT GAP.
FOR COMMUNITY MEMBERS IT'S
DEMANDING TRANSPARENCY AND
ACCOUNTABILITY.
SO OFTEN THE CURRENT STATE OF
AI GOVERNANCE IS ACTUALLY
PRETTY MATURE.
YET AI GOVERNANCE IS THE WAY
FOR AN INSTITUTION TO BE
ACCOUNTABLE FOR A SYSTEM THEY
DEPLOYED IN THEIR SYSTEM.
PATIENTS NEED TO DEMAND THAT
ACCOUNTABILITY AND PROVIDE THAT
FEEDBACK.
IT COULD BE GENTLE FEEDBACK OR
LIKE RED-TEAMING TYPE OF
FEEDBACK.
I WON'T GO INTO RED-TEAMING
RIGHT NOW.
BUT SOME OF THE MOST
INTERESTING WAYS OF DRIVING
CHANGE HAVE COME FROM PATIENT
ADVOCATE THAT HAVE FIND,
ESSENTIALLY HACKED AND MADE
VISIBLE VULNERABILITIES THAT
PUT PATIENT PRIVACY AND SAFETY
AT RISK.
AND THEN ADVOCATING FOR
RESPONSIBLE AI.
THE ROLE OF CLINICIANS,
UNDERSTANDING LIMITATIONS OF
AI, GETTING INVOLVED IN
DEVELOPMENT.
AND THEN MAKING SURE THAT WE
CLINICIANS REMAIN THE HUMAN.
WITH THAT, THANK YOU VERY MUCH.
I REALLY LOOK FORWARD TO
ANSWERING YOUR QUESTIONS.
THAT WAS KIND OF A WHIRLWIND
TOUR.
DID A LITTLE OF EVERYTHING
THERE.
AND LOOK FORWARD TO YOUR
QUESTIONS.
> KEN KOYLE: THANK YOU, WE
APPRECIATE IT.
WE DO HAVE TIME FOR QUESTIONS
FROM OUR AUDIENCE.
I'M KEN KOYLE, I WILL BE
MODERATING THE Q&A SESSION.
AS A REMINDER FOR THOSE
WATCHING THE NIH VIDEOCAST,
PLEASE CLICK THE LIVE FEEDBACK
BUTTON TO SEND ANY QUESTIONS
FOR DR. HIGHTOWER.
THE FIRST QUESTION WE HAVE,
COMES FROM ONE OF OUR VIEWERS,
ARE YOU SEEING ANY POSITIVE
SWING ALREADY IN HEALTH EQUITY
AS AI IS IMPLEMENTED?
CAN AI HELP PREVENT OUR HUMAN
BIAS, AND IF YES, HOW IS THAT
DATA BEING SHARED WITH
CLINICIANS?
HOW DO WE POINT OUT THE
BENEFITS TO HELP PREVENT
CLINICAL BIAS, THINGS LIKE
CREATING PATIENT NOTES WHERE
THE CLINICIAN MAY INADVERTENTLY
INTRODUCE A BIAS TONE.
>> Maia Hightower: THAT WAS
LIKE FIVE QUESTIONS IN THERE.
>> SORRY ABOUT THAT.
STARTING WITH, ARE YOU SEEING,
SO AI IS RELATIVELY NEW IN
CLINICAL CARE.
BUT ARE YOU SEEING A POSITIVE
TREND SO FAR?
>> Maia Hightower: YES,
DEFINITELY I'M SEEING A
POSITIVE TREND WHEN IT COMES TO
STRATIFIED MEASUREMENT OF
PERFORMANCE OF MODELS.
RIGHT NOW AI, THERE'S YET TO BE
-- AI IS A RELATIVELY NEW
SCIENCE WHEN IT COMES TO
CLINICAL TRIALS AND
OUTCOMES-BASED MEASUREMENT OF
THE IMPACT OF AI IN HEALTHCARE,
EVEN THOUGH IT IS BEING
DEPLOYED.
ITS STILL IN ITS INFANCY IN
CLINICAL TRIALS, THAT'S A WHOLE
OTHER TALK.
PEOPLE TALK ABOUT THE NEED FOR
MORE CLINICAL TRIALS. AS THEY
ARE BEING ACTUALLY RIGHT NOW
DEPLOYED THEY ALMOST
UNIVERSALLY HAVE HEALTH EQUITY
IN MIND, ESPECIALLY THAT ARE
NIH FUNDED THAT ARE FUNDED BY
--
MAJOR DONORS. ALMOST ALL OF
THEM LAST TWO YEARS, INCLUDING
AS HEALTH EQUITY BY DESIGN
COMPONENT, WHICH IS EXTREMELY
PROMISING.
WE HAVE BETTER MECHANISMS TODAY
FOR AI MOVING FROM SAY THE LAST
YEAR FORWARD, THAN WE DID TWO
OR THREE YEARS AGO.
SO THE TREND IS POSITIVE.
THE CHALLENGE THOUGH IS THAT A
LOT OF OUR MECHANISMS FOR
MEASURING HEALTH EQUITY REALLY
WERE AND ARE BASED ON
PREDICTIVE MODELS.
LESS SO ON NLM'S GENERATIVE AI.
WE DON'T REALLY HAVE GOOD
MECHANISMS FOR MEASURING
IMPACTS, OUTCOMES OF GENERATIVE
AI.
THEY ARE BEING DEPLOYED FOR
CLINICAL DOCUMENTATION, SORT OF
LOWER RISK OUTPUT.
BUT EVEN TO YOUR CALLER'S
QUESTION ABOUT TONE OF THOSE
GENERATIVE AI MODELS AS THEY
ARE HELP WITH CLINICAL
DOCUMENTATION, EVEN CHAT BOTS
FOR PATIENTS, THOSE STUDIES ARE
STILL YET TO BE DONE.
AND THEY CAN BE DONE.
WE HAVE GOOD METHODS FOR TAKING
A CLINICAL NOTE AND BEING ABLE
TO DETECT SENTIMENT.
A GOOD PROBABILITY THAT STUDY
IS UNDER WAY.
CONCURRENT WITH THE DEPLOYMENT
OF GENERATIVE MODELS IN
HEALTHCARE.
PRETTY EASY IN OUR MIND --
YOU CAN ALREADY THINK ABOUT
WHAT METHODS YOU CAN USE,
AGGREGATE OF NOTES AND ASSESS
SAY A SENTIMENT TONE, SOME OF
THE WORK AT U-CHICAGO THEY HAVE
DONE ON THAT SENTIMENT ANALYSIS
BY RACE AND DEMOGRAPHICS.
>> OKAY, WONDERFUL, THANK YOU.
OUR NEXT QUESTION, THIS WAS A
FASCINATING AND IMPORTANT TALK,
THANK YOU.
TO WHAT EXTENT, IF AT ALL, DO
YOU SEE VALUE IN CONTRIBUTIONS
OF THE FIELD OF THE HEALTH
HUMANITIES AND HELPING US
NAVIGATE THE CURRENT AND FUTURE
LANDSCAPE OF A.I.
>> Maia Hightower: HEALTH
HUMANITIES, WHEN I THINK OF
HEALTH HUMANITIES I WAS A
SCIENCE AND TECHNOLOGY MAJOR IN
COLLEGE.
HEALTH AND HUMANITIES THAT'S A
BIG COMPONENT OF MY
UNDERGRADUATE TRAINING.
WHEN I THINK OF HEALTH
HUMANITIES I THINK OF THE
PHILOSOPHY OF SCIENCE, THE
ANTHROPOLOGY OF SCIENCE.
THERE'S LIKE TWO OTHERS,
SOCIOLOGY OF SCIENCE.
SO THOSE ALL PLAY A BIG ROLE IN
HOW WE TRAIN THE WORKFORCE THAT
ARE DEVELOPING MODELS AND AI
GOVERNANCE AND HAVING THAT
BROAD PERSPECTIVE.
WHEN IT COMES TO OUR AI
GOVERNANCE ALMOST ALL HAVE
EMPHASIS, DEFINITELY ON THE
NATIONAL PANELS AND WORK GROUPS
I'VE BEEN A PART OF, EMPHASIS
PLAY A HUGE ROLE IN AI
GOVERNANCE AND DEFINING
PRINCIPLES THAT ARE IMPORTANT.
AND ENSURING THAT HEALTH EQUITY
IS ONE OF THE DRIVING
PRINCIPLES BY DESIGN.
SO I THINK FROM WHETHER IT'S
ETHICS, ANTHROPOLOGY AND
UNDERSTANDING HUMAN BEHAVIOR,
SOCIOLOGY AND HOW WE WORK
TOGETHER, THEY PLAY AN
IMPORTANT ROLE, EITHER AS
WORKFORCE OR HOW MOD --
MODELS ARE DEVELOPED AND
DEPLOYED.
>> KEN KOYLE: THANK YOU.
NEXT QUESTION.
THIS IS A LITTLE MORE DIRECT
AND PERHAPS LESS COMPLEX.
GENERAL QUESTION, HOW ARE THE
HEALTHCARE LANGUAGE MODELS
TRAINED?
HOW DO WE DECIDE WHAT TO TRAIN
WITH.
>> Maia Hightower: GREAT
QUESTION.
MOST ARE GENERAL PURPOSE MODELS.
THEY CAN BE FINE-TUNED BY
HEALTHCARE SYSTEMS.
BUT ESPECIALLY THE FIRST
GENERATION, SECOND GENERATION
OPEN AI, LARGE LANGUAGE MODELS,
THEY ARE GENERATIVE, NOT FOR
PURPOSE, GENERAL PURPOSE MODELS.
THERE ARE NEWER ONES BEING
DEVELOPED THAT ARE SPECIFICALLY
TO HEALTHCARE SPECIFIC.
BUT THEY HAVE YET TO BE AS
WIDELY DEPLOYED OR TESTED.
AT THE HEALTH SYSTEM LEVEL MANY
ARE DOING LOCAL FINE TUNING
USING THEIR OWN, WHATEVER DATA
THEY HAVE ACCESS TO TO IMPROVE
THE ACCURACY FOR HEALTHCARE.
>> KEN KOYLE: OKAY, THANK YOU.
THIS ONE IS MAYBE A LITTLE MORE
COMPLICATED.
GREAT TALK, THIS WRITER WRITES.
WHEN THERE'S LOW-HANGING FRUIT
IN HEALTHCARE, MAKING SURE
EVERYONE WHO NEEDS INSULIN CAN
AFFORD IT, PROBLEMS WE KNOW HOW
TO SOLVE IF WE JUST SPENT THE
MONEY, WHY SHOULD WE EXPLORE AI?
DOES AI HAVE A ROLE IN SOLVING
SOME OF THESE PROBLEMS THAT, I
THINK ARE WIDESPREAD AND EASILY
RECOGNIZED BY HARD TO SOLVE?
>> Maia Hightower: YEAH, WHEN
IT COMES TO UNHOUSED AND THE
INCREDIBLY HIGH COST OF
MEDICATIONS FOR THOSE WHO MAY
BE UNDER INSURED OR LACK
INSURANCE, AI MAY HAVE A ROLE
TO PLAY.
BUT IS THAT THE BEST, SHOULD WE
BE FOCUSING ON THESE MORE
COMPLEX ISSUES VERSUS AI.
IS IT COMPETING PRIORITIES THAT
RESULT IN DEVIATION OF
RESOURCES.
THAT IS MY INTERPRETATION, THIS
IS A RESOURCE ALLOCATION
QUESTION.
AND MORE RESOURCE DEPLOYMENT
TYPE OF QUESTION.
I THINK THAT'S ONE OF THE
CHALLENGES WITH HEALTHCARE.
YOU HAVE SO MANY STAKEHOLDERS
DRIVING THEIR STRATEGIC
PRIORITIES.
AND ULTIMATELY DECISIONS ARE
MADE BY THOSE THAT ARE DRIVING
THEIR WHATEVER INSTITUTION THEY
ARE PART OF, THOSE STRATEGIC
PRIORITIES.
SO FOR HEALTHCARE DELIVERY
SYSTEM, HOUSING THE UNHOUSED
ISN'T A STRATEGIC PRIORITY, BUT
IT IS A STRATEGIC PRIORITY TO
UNDERSTAND SOCIAL DETERMINANTS
OF HEALTH MAY BE AFFECTING OR
IMPACT INTO A UNIT THAT IS
ADDRESSABLE FOR THAT COMPONENT
OF THE HEALTHCARE DELIVERY
SYSTEM.
AND ACTUALLY, AI MAY BE ABLE TO
HELP IDENTIFY AND CONNECT
INFORMATION.
THERE MAY BE THAT ABILITY, SAY
A PATIENT THAT NEEDS SOME SORT
OF SOCIAL DETERMINANT OF HEALTH
RISK FACTOR ADDRESSED.
AND BE ABLE TO CONNECT IT TO
THE APPROPRIATE RESOURCES.
THAT'S WHERE AI MAY ACTUALLY BE
BENEFICIAL.
NOW WHETHER OR NOT, SINCE OFTEN
IN DECISION MAKING AT THE
HEALTH SYSTEM LEVEL, THE LEVEL
I CAN MOST TALK ABOUT --
TYPES OF DECISION MAKING.
BUT THAT'S AT THE VERY LOCAL
HEALTHCARE DELIVERY SORT OF
UNIT LEVEL.
ON A NATIONAL LEVEL, THAT'S
VERY DIFFERENT, RIGHT?
THAT'S OUR POLICY MAKERS.
THAT'S THE FUNDING MECHANISM
FOR NIH AND WHO IS DRIVING THAT
DECISION MAKING ON WHAT GETS
FUNDED.
AND SO, I THINK THE POLITICAL
REALITY IS, DETERMINING OUR
HAVING INFLUENCE ON WHO IS
MAKING THOSE POLITICAL
DECISIONS ON HOW WE SPEND BROAD
TAXPAYER DOLLARS IS SOMETHING
WE ALL NEED TO RECKON WITH AND
TRY TO MAKE SURE THAT OUR
POLICY MAKERS REALLY ARE
DRIVING THE DECISIONS THAT ARE
MOST IMPORTANT TO US AS A
COLLECTIVE.
>> KEN KOYLE: THANK YOU.
THANK YOU, DR. HIGHTOWER.
OUR NEXT QUESTION, I THINK,
DRILLS DOWN A LITTLE BIT INTO
THE ROLES YOU TALKED ABOUT IN
YOUR LECTURE.
THIS IS A QUESTION REGARDING
THE CONSTITUENCY REPRESENTED BY
THE TWO SPONSORING
ORGANIZATIONS OF THIS LECTURE,
NLM AND MLA.
QUESTION IS, AS LIBRARIANS AND
INFORMATIONISTS, WE SUPPORT AND
ADVISE RESEARCHERS WHO WILL BE
INCREASING RELYING ON AI
GENERATED SEARCH RESULTS FOR
THEIR RESEARCH.
YOU TALKED ABOUT THE
INSTITUTIONAL ROLE FOR
INSTITUTES BUT HOW CAN WE MAKE
SURE THEIR QUERIES RETURN
EQUITY AND, UNBIASED RESULTS.
>> Maia Hightower: THAT'S A
GREAT QUESTION.
WHEN IT COMES TO QUERYING, ONE
IS THE TOOL, LIKE UNDERSTANDING
THE LIMITATIONS OF THE TOOLS.
SO YOU'RE RESEARCHERS OR EVEN
IN YOUR LIBRARY, ARE YOU
ACCESSING LARGE LANGUAGE MODELS
TO HELP WITH THAT RESEARCH
PROCUREMENT, LIKE THE
LITERATURE PROCUREMENT PROCESS,
UNDERSTANDING THE LIMITATIONS
OF SUCH TOOLS.
AND THEN UNDERSTANDING HOW TO
PROMPT.
THERE NEEDS TO BE A HUGE AMOUNT
OF TRAINING ON PROMPTING TO
ENSURE THAT THE OUTPUT OF A
PROMPT ACTUALLY MATCHES THE
INTENTION OF THE INVESTIGATOR
OR RESEARCHER OR LIBRARIAN.
THERE'S THIS WHOLE EMERGING
METHODS AROUND HOW BEST TO
PROMPT ENGINEERING, WHO EVER
HEARD OF PROMPT ENGINEERING AS
OF ONE YEAR AGO.
THIS WHOLE THING, PROMPT
ENGINEERING.
I THINK THOSE TOOLS WILL BE
INCREASINGLY IMPORTANT,
DEPLOYED UNDERSTANDING, NOT
ONLY AT THE LIBRARY LEVEL OR
INDIVIDUAL LIBRARIANS HELPING
RESEARCHERS. BUT EVEN WHEN IT
COMES TO STUDENTS AND GENERAL
PUBLIC.
LIKE WHO IS GETTING THE MOST
BENEFIT OUT OF GENERATIVE A.I.
AND BEING ABLE TO DETECT.
BECAUSE REALLY THE WHOLE POINT
OF GENERATIVE A.I. IS THE HUMAN
IN THE LOOP SHOULD HAVE ENOUGH
KNOWLEDGE ABOUT THE SUBJECT
MATTER THEY ARE RESEARCHING TO
BE ABLE TO IDENTIFY WHEN
SOMETHING MAY BE A
HALLUCINATION OR MAY BE A
FALSEHOOD.
THAT'S THE HUMAN IN THE LOOP.
THE ROLE OF THE LIBRARIAN AS
WELL IS TO BE THAT BIT OF A
FILTER, ESPECIALLY IF YOU HAVE
A NOVICE DOING THE QUERY.
I THINK THERE'S A LOT OF
BENEFIT TO BE HAD AT
UNDERSTANDING THE TOOLS,
UNDERSTANDING PROMPTING,
UNDERSTANDING THE LIMITATIONS,
THERE'S MORE, EVEN EVOLVING,
LIKE HOW DO WE USE CORPUSES OF
TRUST, KNOWLEDGE BANKS OF
TRUST, VERSUS A WHOLE
REPOSITORY LIKE A LARGE
LANGUAGE MODEL THAT IS
UNRESTRICTED.
AND DEPENDING ON HOW YOUR
HEALTH SYSTEM OR LIBRARY IS
DEPLOYED, THAT GENERATIVE MODEL
WITHIN YOUR ENVIRONMENT, I
THINK STEPHEN TALKED A LITTLE
ABOUT KNOWLEDGE GRAFTS. AND I
THINK THERE'S GOING TO BE AN
INCREASING ROLE IN MAKING SURE
WE ARE LEVERAGING CURATED
KNOWLEDGE GRAFTS WHEN IT COMES
TO THE DATA, THE OUTPUT OF SOME
OF THESE QUERY SEARCHES.
>> KEN KOYLE: OKAY, THANK YOU.
NEXT QUESTION.
CAN YOU SPEAK A LITTLE ABOUT
THE POTENTIAL NEGATIVE EFFECTS
OF A.I. ON PRIVACY AND KFL
CONFIDENTIALITY OF PATIENTS AND
POPULATIONS AT LARGE?
>> Maia Hightower: ABSOLUTELY.
RIGHT NOW THERE'S A LOT OF
CONCERNS ABOUT PRIVACY.
HIPAA STILL STANDS.
THERE STILL ARE HUGE CONCERNS
ABOUT ESPECIALLY GENERATIVE
MODELS, WHAT HAPPENS TO
PERSONAL HEALTH DATA WHEN IT
ENTERS INTO, WHEN YOU JUST
ENTER THAT INFORMATION INTO A
QUERY, INTO A PROMPT.
SO THERE ARE HUGE CONCERNS
ABOUT PRIVACY, SECURITY.
I WOULD SAY, IT DEPENDS ON WHO
YOU ARE.
IF YOU ARE A PATIENT,
DEFINITELY WITH CARE, YOU WOULD
WANT TO LIMIT THE AMOUNT OF
INFORMATION YOU PROVIDE.
SAY A GENERATIVE MODEL, SAY
OPEN AI, CHATGPT, OR SOMETHING
LIKE THAT.
AND THEN FOR HEALTH SYSTEMS,
MAKING SURE YOU ARE CREATING
SECURE ENVIRONMENTS, AS SECURE
AS POSSIBLE, THERE'S DIFFERENCE
BETWEEN OPEN AI AND WITHIN
AZURE OR MICROSOFT OR CLOUD
ENVIRONMENT, AN
ENTERPRISE-GRADE, LARGE
LANGUAGE MODEL THAT IS MORE
LOCKED DOWN.
IT'S NOT 100%.
BUT ANYWAYS, THE DATA DOESN'T
LEAVE, OR ISN'T FURTHER REUSED
OR REPURPOSED FOR FURTHER
FINE-TUNING OF THE MODEL.
SO THE WAY THAT THE MODEL MAY
BE DEPLOYED IN YOUR SYSTEM,
WHETHER YOUR LIBRARY OR YOUR
HEALTH SYSTEM IS REALLY
IMPORTANT TO MINIMIZE SOME OF
THE PRIVACY CONCERNS AND
POTENTIAL FOR HIPAA VIOLATION.
THERE ARE A LOT OF SORT OF DO'S
AND DON'T'S, IT'S EASY FOR
PATIENTS AND CLINICIANS TO
ACCIDENTALLY CROSS THAT LINE
AND CREATE SOME PRETTY BLATANT
HIPAA VIOLATIONS. SO BE CAREFUL.
TALK TO YOUR ITT BEFOREHAND.
IF YOU ARE JUST OPENING UP A
WEB BROWSER, JUST KNOW THAT'S
THE SAME AS OPENING IT UP TO
THE WORLD, RIGHT?
YOU JUST OPENED YOUR WHOLE
FRONT DOOR.
IF YOU ENTER YOUR INFORMATION
INTO AN OPEN WEB BROWSER, JUST
ASSUME YOU HAVE NO PRIVACY.
>> KEN KOYLE: OKAY, UNDERSTOOD,
THANK YOU.
THIS NEXT QUESTION REFERS BACK
TO WHAT YOU WERE SAYING, IN
YOUR RESPONSE A MOMENT AGO
ABOUT THE RESEARCHERS HAVING
THE EXPERTISE TO EVALUATE WHAT
THEY ARE SEEING FROM GENERATIVE
BY AI.
THIS QUESTION COMES IN, IF WE
HAVE IN PRACTICE ACCEPTED THAT
MOST INDIVIDUAL PROVIDERS DON'T
HAVE THE TIME TO CRITICALLY
APPRAISE PUBLIC EVIDENCE, POINT
OF REFERENCE TOOLS UP-TO-DATE
AND GUIDELINES BUILT INTO THE
ELECTRONIC HEALTH RECORDS, IS
IT REALISTIC TO EXPECT
PROVIDERS WILL HAVE THE TIME
AND DEVELOP THE SKILL TO
EVALUATE THE VARIOUS
IMPLICATIONS OF A.I.
AND MAYBE THIS GETS BACK TO THE
PEOPLE WHO ARE IN THE
PROFESSION OF HELPING THE
PROVIDERS TO USE THESE TOOLS
EFFECTIVELY AND EQUITABLY.
>> Maia Hightower: YEAH, THAT'S
A GREAT QUESTION.
SO FOR GOOD OR BAD, MANY OF THE
HEALTH SYSTEMS THAT ARE
EVALUATING THE OUTPUT OF
GENERATIVE AI RECOGNIZE DOCTORS
COULD BE A LITTLE LAZY AT TIMES.
BEING A DOCTOR MYSELF, I WILL
ADMIT IT AS WELL.
THEY ARE ON THE LOOK OUT.
MANY OF THE EVALUATION METHODS
ARE LOOKING FOR WHETHER DOCTORS
REVERT TO AUTOMATICITY AND
AUTOMATICALLY ACCEPTING THE
OUTPUT OF GENERATIVE MODELS.
THERE WAS A PAPER IN THE LAST
COUPLE WEEKS, THEY WERE LOOKING
AT THE OUTPUT, I CAN'T REMEMBER
EXACTLY WHAT THE OUTPUT WAS,
BUT WHAT THEY FOUND WAS A LOT
OF AUTOMATICITY.
HENCE THAT, BUT ALSO A LOT OF
ERRORS.
AND HOW DO WE THEN CREATE SOME
GUARDRAILS. SO THAT BE THE NEXT
STEPS, LIKE IF SOMEBODY IS
AUTOMATICALLY ACCEPTING EVERY
OUTPUT OF THE MODEL.
THAT KIND OF TELLS YOU, THEY
ARE AUTOMATICALLY ACCEPTING THE
OUTPUT WITHOUT QUESTION.
VERSUS THOSE MODIFYING THE
OUTPUT, MY CHART MESSAGES, YOU
HAVE ONE DOCTOR WHO JUST 100%
ACCEPTS AND SENDS. OR ANOTHER
DOCTOR 60% OF THE TIME IS
MODIFYING THE MESSAGE AND
THERE'S VARIATION IN THERE.
THAT VARIATION IN PERFORMANCE
MAY LATER ON RESULT IN
PROMPTING THE CLINICIAN TO
REVERT BACK TO THE MEAN, CHECK
A FEW MORE MESSAGE RESPONSES
BEFORE SENDING THEM ALONG.
RESEARCHERS ARE AWARE OF THAT
PROBLEM, AND TRYING TO BOTH
MEASURE IT AS WELL AS DEVELOP
METHODS FOR MITIGATING IT.
AND IT MAY BE SOME SORT OF
PROMPT OF THOSE THAT VARY FROM
THE MEAN, WE DON'T KNOW YET.
>> KEN KOYLE: IT'S A NEW WORLD,
I THINK THERE ARE, AS YOU SAID,
THERE ARE SOME BOUNDARIES AND
KIND OF GUIDE RAILS WE NEED TO
ESTABLISH.
THIS IS CERTAINLY ONE OF THEM,
ONE AREA WHERE I THINK THERE'S
A LOT MORE THOUGHT AND MAYBE
THIS IS WHERE THOSE HUMANISTS
COME IN TO REALLY MANAGE THAT.
AND INSURE THE QUALITY IS STILL
THERE, AND LAZINESS DOESN'T
COMPROMISE THAT.
KIND OF RELATED AND I THINK
PERHAPS A VERY LOADED AND
COMPLEX QUESTION.
ARE THERE ANY LEGAL SYSTEMS IN
PLACE TO ADDRESS USING AI FOR
PATIENT TREATMENT?
FOR EXAMPLE, IF AI WERE TO
SUGGEST A SPECIFIC TREATMENT
FOR A PATIENT AND THEY TAKE
THAT ADVICE BUT EXPERIENCE
ADVERSE RESULTS WHO IS
RESPONSIBLE?
I THINK WE ARE TALKING ABOUT A
MEMBER OF THE PUBLIC TAKING
MEDICAL ADVICE FROM AN AI TOOL
DIRECTLY AND NOT, OBVIOUSLY IF
A CLINICIAN IS INVOLVED, THE
CLINICIAN IS RESPONSIBLE.
BUT IF THERE'S NO CLINICIAN IN
THE LOOP, IS THERE ANY ANYTHING
TO KEEP THE AI MODELS, KIND OF
KEEP THEM HONEST?
>> Maia Hightower: YEAH, YOU
ARE RIGHT.
THE QUESTION WAS MORE AT THE
PATIENT LEVEL, BUT I THINK IT'S
IMPORTANT TO ADDRESS THE
CLINICIAN.
SO FOR THE CLINICIAN, LET'S SAY
RIGHT NOW, THERE ARE A COUPLE
LAWSUITS THAT ARE ACTUALLY
UNDER WAY.
AND ONE OF THE LAWSUITS IS A
CLINICIAN THAT FOLLOWED AN
ALERT, AN A.I. GENERATED ALERT,
THAT PREVENTED A PATIENT FROM
GETTING AN INTERVENTION, OR THE
CARE THAT WAS PROBABLY LIKELY
NEEDED.
BUT THE PATIENT FELL OUTSIDE
THE PERFORMANCE OF THE MODEL,
AGE RANGE.
A YOUNG PERSON WITH CHEST PAIN.
SO THE PHYSICIAN MAY STILL BE
FOUND LIABLE.
WE DON'T KNOW YET.
BECAUSE THE MODEL SUGGESTED
SOMETHING THAT WAS OUTSIDE
STANDARD OF CARE.
AND SO THE SORT OF FOUNDATION
FOR PHYSICIANS IS STILL TO
FOLLOW STANDARD OF CARE.
IF A MODEL TELLS YOU TO DO
SOMETHING OUTSIDE THE STANDARD
OF CARE AND YOU FOLLOW IT
BLINDLY, YOU MAY BE HELD LIABLE.
IF THE MODEL TELLS YOU TO DO
SOMETHING WITHIN STANDARD OF
CARE AND YOU IGNORE IT, YOU MAY
BE STILL HELD LIABLE.
STANDARD OF CARE STILL TRUMPS,
OR TRIUMPHS. FOLLOW TRAINING
AND STANDARD OF CARE.
AND IN AN IDEAL SITUATION, THE
CO-PILOT EFFECT WHERE THE MODEL
IS PROVIDING YOU INFORMATION
WHERE YOU MAY HAVE MISSED THE
STANDARD OF CARE, THEN YOU
PAUSE, YOU HAVE TO THINK ABOUT
IT AND THEN SAID OH I MIGHT
HAVE MISSED THAT.
AND GRAY AREA.
THERE ARE A LOT OF DECISIONS
THAT ARE WITHIN THE STANDARD OF
CARE.
THAT'S SORT OF NOW HOW THAT IS
AUGMENTED PHYSICIAN ROLE WORKS
IN A LEGAL FRAMEWORK.
AS FAR AS FOR PATIENTS, NOW
THIS IS A WHOLE OTHER THING,
RIGHT?
WHERE A PATIENT IS GETTING
MEDICAL ADVICE FROM AN
AI-DERIVED SYSTEM.
THIS IS WHERE, EVEN FOR OPEN
AI, IF YOU PUT A LOT OF
INFORMATION IN CHATGPT, IT
DOESN'T MAKE A DIAGNOSIS, WHAT
IT SUGGESTS IS THIS IS
SOMETHING YOU MAY WANT TO
CONSIDER, BUT I'M NOT A
CLINICIAN, AND TALK TO YOUR
DOCTOR.
SO WHAT PATIENTS REALLY NEED TO
DO, FROM A LEGAL PERSPECTIVE, A
LOT OF THE DIRECT TO CONSUMER,
DIGITAL HEALTH SOLUTIONS, WHEN
IT COMES TO AI, IF THEY ARE FDA
APPROVED, THEY HAVE AN
INCREDIBLY HIGH BAR.
AND USUALLY ARE AUGMENTED.
I DON'T KNOW OF MANY SYSTEMS
THAT ARE FDA APPROVED DIRECT TO
PATIENTS.
AND THEN YOU ARE LEFT WITH THIS
WILD WEST OF CONSUMER HEALTH.
SO IT ACTUALLY FALLS INTO A
DIFFERENT SORT OF LEGAL
FRAMEWORK THAN ACTUALLY
DIAGNOSIS OR TREATMENT.
BUT WHAT DO YOU CALL IT,
DIGITAL HEALTH AND WELLNESS.
SO YEAH, BUT IT'S A VERY GRAY
AREA, WHAT PATIENTS WILL
INTERPRET AS BEING MEDICAL
ADVICE, RIGHT.
EVEN THOUGH THERE MAY BE SOME
SMALL DISCLAIMER THAT SAYS THIS
IS NOT MEDICAL INFORMATION,
MEDICAL DIAGNOSIS OR TREATMENT,
THIS IS JUST INFORMATION YOU
SHOULD BE PERHAPS SHARING WITH
A CLINICIAN.
AND RIGHT NOW, I THINK THAT IS
A BIT OF A GRAY AREA BECAUSE
THERE USUALLY ARE SOME, FROM A
LEGAL PERSPECTIVE, GRAY AREA.
THERE ARE OFTEN THESE
DISCLAIMERS THAT IT'S NOT
ACTUALLY MEDICAL ADVICE, OR
MEDICAL INFORMATION.
IT'S KIND OF LIKE FOR THOSE WHO
MAY GET A STOCK TIP OR
SOMETHING, THIS IS NOT
FINANCIAL ADVICE, THIS IS FOR
INFORMATION PURPOSES ONLY.
BOTH FINANCE AND HEALTHCARE ARE
HIGHLY REGULATED SO TO GIVE
MEDICAL ADVICE, THERE'S ONLY
LIMITED ROLES WHERE THAT CAN
OCCUR LEGALLY.
>> KEN KOYLE RENT CONTROL RIGHT.
: RIGHT.
GREAT ANSWER TO THAT QUESTION
AND ALL THE QUESTIONS.
THANK YOU VERY MUCH.
THAT IS ALL THE TIME THAT WE
HAVE.
I WILL NOTE THAT WE HAD QUITE A
FEW MORE QUESTIONS THAT CAME IN.
SO, ON YOUR LAST SLIDE, YOU
PROVIDED SOME CONTACT
INFORMATION.
MAYBE WE COULD PUT THAT BACK UP
AGAIN.
WOULD YOU BE OKAY, DR.
HIGHTOWER IF THE PEOPLE WHO
WEREN'T ABLE TO GET THEIR
QUESTIONS ANSWERED IF THEY CAN
SEND THEM TO YOU ON THAT
CONTACT INFORMATION ON THAT
SLIDE?
>> Maia Hightower: ABSOLUTELY.
>> KEN KOYLE: WE WILL PUT THAT
SLIDE BACK UP.
THANK YOU FOR THE WONDERFUL
TALK AND ANSWERING THE
QUESTIONS FROM OUR AUDIENCE, I
WOULD LIKE TO THANK DR. STEPHEN
SHERRY FOR INTRODUCING TODAY'S
SPEAKER AND THANK YOU TO THE
NLM AND MLA, THEY PRODUCE AND
SUPPORT SEVERAL OTHER LECTURE
SERIES, SO PLEASE VISIT THEIR
WEBSITES TO LEARN MORE.
THANK YOU FOR ATTENDING AND
HAVE A SAFE AND PLEASANT DAY.
GOODBYE.