GOOD MORNING
EVERYBODY, THIS MEETING OF THE
NLM BOARD OF SCIENTIFIC
COUNSELORS IS BEING CALLED
TOGETHER AS WE HAVE A QUORUM OF
MEMBERS.
THE MEETING IS BEING VIDEOCAST
AT THE NIH WEBSITE AND WILL BE
AVAILABLE FOR VIEWING WITH
EXCEPTION TO THE EXECUTIVE
SESSIONS.
SO I WOULD LIKE TO WELCOME
EVERYONESH EVERYONE TO THIS
SESSION AND I WILL NOW TURN IT
OVER TO ACTING NLM DIRECTOR
STEVE SHERRY.
>> THANK YOU.
WELCOME, EVERYONE.
I APPRECIATE ALL OF YOU FOR
JOINING US AND THANK EACH OF YOU
FOR THE IMPORTANT ROLE YOU PLAY
IN ADVANCING RESEARCH HERE AT
THE NATIONAL LIBRARY OF
MEDICINE.
THIS IS THE APRIL 18th BOARD
OF SCIENTIFIC COUNSELORS ACTING
SCIENTIFIC DIRECTORS UPDATE.
NEXT SLIDE, PLEASE.
SO AS WE ALL KNOW NLM IS WORKING
CONSTANTLY FOR DISCOVERY AND
DATA POWERED HEALTH THROUGH
ENHANCED RESEARCH, DISSEMINATION
AND ENGAGEMENT AND BUILDING A
WORK FOR FOR DATA DRIVEN
RESEARCH AND HEALTH.
NEXT SLIDE, PLEASE.
AND THIS IS WELL ALIGNED WITH
DR. MONICA BERTAGNOLLI'S VISION,
SHE TOOK THE HELM THIS PAST
NOVEMBER, AND WITH A NEW
DIRECTOR COMES A NEW VISION, SO
I WANTED TO SHARE INSIGHT FOR
THE DOCTOR PROVIDING EXCITING
ROLES IN THE ROLE THAT NLM WILL
PLAY IN THE NIH STORY.
LET'S BEGIN WITH CHALLENGES IN
PUBLIC HEALTH THAT
DR. BERTAGNOLLI BELIEVES WILL BE
QUITE IMPORTANT.
--WHAT'S MORE IS THAT OUR LIFE
EXPECTANCY IS DIPPED SINCE 2014,
A TREND THAT CANNOT BE FULLY
EXPLAINED BY THE COVID-19
PANDEMIC.
AS DR. BERTAGNOLLI ABOUT THIS
TROUBLING DATA, WE ARE CLEARLY
NOT ON THE RIGHT TRACK.
ANOTHER PLACE WE'RE NOT ON THE
RIGHT TRACK IS THAT DESPITE OUR
POOR LIFE EXPECTANCY AMONG OUR
PEERS, THE U.S. IS SPENDING FAR
MORE ON HEALTHCARE.
WE ARE MORE ALIGNED WITH OTHER
WEALTHY COUNTRIES IN THE 1970S
BUT AS THIS FIGURE FROM 2020
SHOWS OUR INCREASED SPENDING IN
THIS AREA HAS NOT RESULTED IN
BETTER HEALTH.
IN 2018 THE U.S. HAD ROUGHLY
$10,000 PERCAPITTA OF HEALTH
EXPENDITURES AND A AVERAGE
HEALTH LIFE EXPECTANCY OF 90
YEARS OLD.
WE CAN CONTRAST THAT TO YAP AN
WITH A AVERAGE LIFE EXPECTANCY
OF 84 YEAR ANDS ONLY ABOUT $45
PER CAPITA IN HEALTH
EXPENDITURES, THE DOCTOR AND I
AGREEN CELLS THAT NIH AND MLM
CAN PLAY AN IMPORTANT ROLE IN
REVERSING THIS TREND AND HELP
BRING DATA THAT LINK NEW
RESOURCES AND UTILIZATION TO
MEANINGFUL CLINICAL OUTCOMES.
NEXT SLIDE, PLEASE EMPLOY BUT AS
POINTED OUT THE LINK BETWEEN
BIOMEDICAL EVIDENCE AND CLINICAL
DECISION MAKE BEING BEING BE
MADE STRONGER SHE GIVES AN
EXAMPLE FROM HER OWN FIELD IN
ONCOLOGY WHERE RESEARCH HAS
SHOWN THAT THERE'S BEEN LITTLE
TO NO CHANGE IN PAST DECADE IN
THE NATURE OF EVIDENT USED TO
DEVELOP CLINICAL CARE
GUIDELINES, THIS ANALYSIS OF THE
2019 NATIONAL COMPREHENSIVE
CANCER NETWORK GUIDELINES FOUND
THAT ONLY 7% OF THE
RECOMMENDATIONS CAN THAT PART IN
THE LITTLE BLUE WEDGE THERE ARE
SUPPORTED BY RANDOMIZED CLINICAL
TRIALS.
THE GREAT MAJORITY OF GUIDELINES
THAT WE USE TO DELIVER CARE,
WERE DERIVE FRIDAY LEVEL 2
EVIDENCE WHICH IS GENERALLY
PROVIDED BY CAREFULLY CONTROLLED
OBSERVATIONAL TRIALS EARLIER
PHASED RANDOMIZED TRIALS BUT NOT
THE BEST WORK THAT WE CAN DO
WHICH IS A RANDOMIZED CLINICAL
CONTROL.
NEXT SLIDE, PLEASE.
NOT ONLY DO WE HAVE A GAP
BETWEEN THE EVIDENCE AND ITS
IMPACT ON CLINICAL CARE, BUT WE
ALSO HAVE A CONCERNING GAP IN
THE EVIDENCE ITSELF.
WE KNOW THERE ARE SERIOUS
DISPARITIES IN ACCESS TO
CLINICAL TRIALS THAT MIRROR
DISPARITIES AND HEALTHCARE
DELIVERY OVERALL.
WE ARE MISSING DATA, ON A VERY
LARGE AND IMPORTANT GROUP OF
PATIENTS INCLUDING THOSE LISTED
HERE.
THE 1S THAT TYPICALLY DO NOT GET
TARGETED FOR ENROLLMENT IN CARE
SUCH AS THE ELDERLY, MINORITY,
UNINSURED GROUPS, RURAL, THOSE
WHO HAVE EXCESSIVE COMORBID
CONDITIONS ARE JUST EXAMPLES OF
PATIENT WHAT'SY NEED TO DO
BETTER AND TRY TO ENGAGE IN
CLINICAL RESEARCH.
NEXT SLIDE, PLEASE.
WHILE THESE CHALLENGES ARE
SERIOUS, NIH IS COMMITTED TO
ADOPTING PRINCIPLES THAT CAN
MEANINGFULLY TURN THE TIDE
TOWARDS BETTER HEALTH FOR ALL
PEOPLE.
NEXT SLIDE, PLEASE.
THIS INCLUDES LOOKING BEYOND THE
MOMENT OF SCIENTIFIC DISCOVERY,
WHILE DORPH SECERTAINLY THE
FOUNDATION OF OUR WORK HERE AT
NLM AND NE H BROADLY,
DR. BERTAGNOLLI HAS REMINDED US
THAT THE WORK IS NOT FINISHED
UNTIL THE SCIENTIFIC ADVANCES
ACTUALLY REACH THE PEOPLE WHO
NEED THEM THAT'S THROUGH LIVING
LONGER, HEALTHIER LIVES.
NEXT SLIDE, PLEASE.
TO MAKE THIS HAPPEN,
DR. BERTAGNOLLI, EMPHASIZES WE
NEED TO RELY ON ALL COMPONENTS
OF NIH RESEARCH, BASIC SCIENCE
TO IMPLEMENTATION RESEARCH IN
THE COMMUNITY.
OUR PURSUIT OF BETTER HEALTH
MUST BE ROBUST AND COMPREHENSIVE
AND AS DR. BERTAGNOLLI SAYS,
PEOPLE ARE OUR PARTNERS IN
DISCOVERY, WHETHER THEY'RE
RESEARCHERS, PARTICIPANTS OR
TRIAL CLINICIANS, THEY WANT TO
SEE A COMMUNITY AS THE THIRD
PART OF THE TRIAD FOR RESEARCH
ENTERPRISE HERE AT NIH.
NEXT SLIDE, PLEASE.
AND FINALLY, OF PARTICULAR
RELEVANCE TO NLM, SHE HAS
EMPHASIZED WE HAVE A GREAT
OPPORTUNITY TO ADEMBRACE NEW
ADVANCES IN SCIENTIFIC METHODS
LIKE DATABASE DATAAN LYTICS AND
TO INSURE THESE ADVANCES SYRUP
EVERYONE BY INCLUDING EVERYONE.
DR. BERTAGNOLLI HAS NOTED THAT
WE NOTE AT NLM THAT WHILE
RESEARCHER VS BEEN USING
ARTIFICIAL INTELLIGENCE AND
MACHINE LEARNING FOR SOMETIME
WHERE THE TECHNOLOGIES CAPTURED
THE WORLD'S ATTENTION, AI IS
ALREADY REVOLUTIONIZING CARE AND
THE DOCTOR CALLED FOR A MORE
SECURE AND SUSTAINABLE DATA
SHARING INFRASTRUCTURE, NOW IS
THE TIME TO DELIVER
EVIDENCE-BASED DATA DRIVEN
HEALTHCARE TO EVERY PATIENT AND
TO EMBRACE THE REALTIME FEEDBACK
THESE TECHNOLOGIES CAN OFFER US
ALLOWING THE CLINIC TO INFORM
THE RESEARCH AND NOT JUST THE
OTHER WAY AROUND.
NEXT SLIDE, PLEASE.
DR. BERTAGNOLLI, PROVIDED A
SAMPLE OF THE FEEDBACK LOOP
BETWEEN CLINICAL AND BASIC
RESEARCH, WHERE THEY INFORM ON
EACH OTHER AND WAS FROM A
DIRECTOR'S BLOG, WE KNOW THAT AS
MANY AS 4 OUT OF EVERY 5 PEOPLE
WITH AUTOIMMUNE DISEASES ARE
FEMALE BUT WE DON'T FULLY
HAPPENED WHY THIS DISPARITY
OCCURS, RECEIPTLY AN NIH FUBDED
TEAM AT STANFORD FOUND THIS MAY
BE RELATED TO MOLECULES CALLED
EXIST WHICH ARE ENCODED ON THE
X-CHROMOSOME AND TRANSCRIBED
INTO NONLONG CODING STRETCHES OF
RNA ONLY WHEN THERE ARE 2 X
CHROMOSOMES.
THESE X6 MOLECULES WIND
THEMSELVES AROUND THE CHROMOSOME
TO SHUOF THE DOWN TRANSCRIPTION,
AS A REULTRAVIOLET B, A WOMAN
HAS VERY DIFFERENT PATTERNS THAN
A MAN WHO ONLY HAS 1
X-CHROMOSOME.
THE STANFORD TEAM LED BY HOWARD
AND DIANEA, FOUND THIS FEMALE
PATTERN OF TRANSCRIPTION CAN
RESULT INCLYSMS OF DMA, RNA AND
PRO10S THAT IN TURN TRIGGER
AUTOIMMUNE RESPONSES, TD
RESEARCHERS MAKE 2 STRAINS THAT
CAN PRODUCE TO SEE IF THIS WOULD
MEASURABLY INCREASE THEIR RISK
OF AUTOIMMUNITY DISEASE, 1 XIST
WAS ACTIVATED IN MALE MICE THAT
WERE PROTECT FRIDAY AUTOIMMUNITY
AND ONCE THIS WAS DONE, THE
RESEARCHERS FOUND THESE MICE
BECAME MORE SUSCEPTIBLE TO
DEVELOPING A WILL YOU PLEASEUS
LIKE CONDITION, THERE'S MUCH
MORE TO UNDERSTAND ABOUT THIS
CONNECTION BUT IT'S ELEGANT WORK
THAT SERVES AS 1 OF MANY
EXAMPLES TO SHOW HOW CLINICAL
OBSERVATIONS CAN INSPIRE BASIC
SCIENCE AND HOW FINDINGS AT THE
BENCH CAN ELIMINATE UPON
CLINICAL RESEARCH AND PRACTICE.
NEXT SLIDE, PLEASE.
DR. BERTAGNOLLI, AS NIH DIRECTOR
IS TO SUPPORT RESEARCH THAT
INTEGRATES WITH HEALTHCARE AND
REVITALIZES OUR OW PROACH TO
DATA INTEGRATION, SHARING AND
DATA USE.
NEXT SLIDE.
BY INTEGRATING WITH HEALTHCARE,
SHE MEANS CONNECTING RESEARCH
WITH THE POINTS OF CARE
PEOPLE--THE POINTS OF CARE THAT
MOST PEOPLE ARE LIKELY TO
ENCOUNTIER.
SHE EMPHASIZED WE HAVE NOT YET
ESTABLISHED AN NIH WIDE CLINICAL
INFRASTRUCTURE FOR THE PRIMARY
CARE SETTING, PRIMARY CARE
DELIVERER WHO SEES EVERYTHING
THAT IS HAPPENING WITH THEIR
PATIENTS HEALTH.
SHE EMPHASIZES THAT THIS FOCUS
IS A WAY TO MEET PATIENTS WHERE
THEY ARE, PARTICULARLY THOSE IN
UNDERSERVED COMMUNITIES OR IN
RURAL AND URBAN REGIONS ALIKE.
THIS MAY LOOK LIKE ADOPTING
INNOVATIVE STUDY DESIGNS THAT
ADDRESS COMMON HEALTH ISSUES
THAT CAN BE DONE IN A BUSY
PRIMARY CARE OFFICE OR COMMUNITY
HEGHT CENTER, THIS CAN ALSO LOOK
LIKE USING THE COMMUNITY HEALTH
RECORD NOT ONLY AS A SOURCE OF
DATA BUT AS A WAY FOR PATIENTS
TO DISSEMINATE THE RESEARCH
FINDINGS THEMSELVES.
NEXT SLIDE.
DR. BERTAGNOLLI, HAS ALSO
EXPRESSED THE NEED TO EXPAND BY
O RESEARCH USE TO INFORM NEW
RESEARCH IN IMPROVED HEALTH
OUTCOMES, SHE EMPHASIZES THIS
MEANS LEVERAGES INSIGHTS FROM
THE SOCIAL SCIENCES AND CLINICAL
CARE AS WELL AS BASIC SCIENCE.
THIS WAS POLED IN ADVANCE
DIAGNOSTIC, WEARABLES AND ALL
OTHERS OF NIH FIELD RESEARCH
THAT CAN HELP MOVE THE NEEDLE ON
PEOPLE'S HEALTH.
NIH WILL EMPLOY A FEDERATED
ARCHITECTURE FOR DATA USE AND
HOSTING AND TO ENABLE LOW COST
ACCESS USING OPEN INDUSTRY
STANDARDS, IMPORTANTLY NIH WILL
TAKE A SUSTAINABLE APPROACH TO
THOSE ADVANCES BY BROADENING
ACCESS TO THESE TOOLS AND
INVESTING IN EDUCATION CAN
WORKFORCE DEVELOPMENT TO PREPARE
A NEW GENERATION OF DATA
SCIENTISTS.
NEXT SLIDE, PLEASE.
I AM ENERGIZED BY THE WAY IN
WHICH DR. BERTAGNOLLI'S
STRATEGIES ALIEP WITH NLM AND
OUR VISION.
FROM OUR DAT DRIVEN RESEARCH TO
TRAINING PROGRAMS WE HAVE
EXCITING OPPORTUNITIES TO
SUPPORT THE NIH WIDE GOAL ANDS
PRIORITIES THAT I'VE SHARED
HERE.
NEXT SLIDE, PLEASE.
SO THINKING BEYOND THE VISION
AND THE PRIORITIES OF NIH TO OUR
FUNDING SITUATION, JUST AS AN
UPDATE FOR EVERYONE THAT THE
FINAL FISCAL YEAR 2024
APPROPRIATIONS BILL WAS SIGNED
INTO LAW ON MARCH 23rd, 2024,
THE RESULT AS YOU SEE HERE, FOR
NLM, IS A FLAT BUDGET, RELATIVE
TO FISCAL YEAR 2023 LEVELS.
NEXT SLIDE, PLEASE.
IN OTHER NEWS, ON
DECEMBER 7th, 2023, A MAJORITY
OF THE VOTING NIH FELLOWS
UUNITTED, VOTED TO BE
REPRESENTED BY THE INTERNATIONAL
UNION OF UNITED AUTOMOBILE
AEROSPACE AND AGRICULTURAL
IMPLEMENT WORKERS OF AMERICA OR
THE UAW UNION, THE FLLA
CERTIFIED THE BARGAINING UNIT ON
DECEMBER 15th 2023, THE UAW
UNIT COVERS APPROXIMATELY 5000
FELLOWS ACROSS THE NIH.
NIRE H IS LOOKING FORWARD TO
PARTNER WITH THE UAW TO
NEGOTIATE A HISTORIC COLLECTIVE
BARGAINING AGREEMENT FOR THE
PARTIES, COLLECTIVE BARGAINING
WILL BE ONGOING THROUGHOUT 2024
NEXT SLIDE, PLEASE.
NLM HAS ALSO KICKED OFF AN AI
GENERATED PILOT WITH 10 USE
CASES ACROSS NLM USING
MICROSOFT'S OPEN AI AND AZURE
SAFE PLAYGROUND.
AS SEEN HERE THERE'S A VARIETY
OF USE CASES RANGING FROM
PRODUCT EFFICIENCY AND CUSTOMER
EXPERIENCE THROUGH PRODUCTION
SERVICES THAT ARE WRITING CODE
AND AUTOMATING DATA
VISUALIZATION TASKS, WE'RE
LOOKING SYSTEMATICALLY AT HOW TO
REDUCE BIAS IN OUR WORK FLOWS
AND TASKS, EXAMPLES HERE OF A
GRANT FORT FOLIO OR EDITING UMLS
AND THEN USING IT IN RESEARCH
DISCOVERY ITSELF THROUGH SOME
IRP PROGRAMS, HERE SHOWN BY
DR. ZHIYONG LU, AND ALSO
UNDERSTANDING THE DYNAMICS OF
OUR OUR SERVICES CAN BE USE
INDEED REALTIME.
DURING THIS PROJECTS OUR STAFF
ARE GAINING NEW KNOWLEDGE,
COLLABORATING ACROSS NLM AND
NIRE H TO GAIN INSIGHT INTOS
PRACTICAL USES OF LARGE LANGUAGE
MODELS AND OVER THE PAST FEW
MONTHS THEY'VE EXPERIMENTED WITH
NEW TOOLS AND ARE IN THE PROCESS
OF DEVELOPING PROOF OF CONCEPT
THAT WILL APPLY TO THEIR USE
CASES.
THE NEXT STAGE OF THESE WORKS
WILL BE MEASURING RESULTS THAT
INSURE SAFE, UNBIASED AND
RELIABLE RESULTS USING THIS
FRAMEWORK FOR RISK MITIGATION
AND BIAS.
NEXT SLIDE, PLEASE.
FINALLY I WOULD LIKE TO SHARE AN
UPDATE ON OUR EFFORTS TO CREATE
A CULTURE OF CONTINUOUS
INNOVATION AT NLM.
NLM CONTINUED TO MAKE GREAT
PROGRESS AND ESTABLISH A FORWARD
LOOKING MODEL OF IPT GREATER
NOVATION THAT IS APPLIED
CONSISTENTLY ACROSS THE LIBRARY
AND ALL OUR DIVISIONS OF
OPERATIONS, NEXT SLIDE, PLEASE.
EVERY DAY I SEE COUNTLESS
EXAMPLES OF STAFF BRINGING
FORWARD NOVEL APPROACHES, IDEAS
AND ALTERNATIVE METHODS THAT
SHOWCASE HOW INNOVATIVE WE ARE
HERE, THIS ONGOING COMMITMENT TO
INNOVATION IS SEEN IN OUR
DAY-TO-DAY OPERATIONAL DELIVERY
AND OUR FORWARD LOOKING APPROACH
AS WE REMAIN RESPONSIVE IN HOW
WE SERVE THE PUBLIC, INNOVATION
IS A COMMON THREAD THROUGHOUT
THE 3 PILLARS OF OUR NLM
STRATEGIC PLAN WE LOOKED AT A
FEW MINUTES AGO, IT'S ALSO
REINFORCED FROM THE VITIONZ
THROUGH HHS AND NIH OVERALL WITH
EXAMPLES LIKE CUSTOMER
EXPERIENCE AND DIGITAL
TRANSFORMATION, WITHIN NNLM, WE
ARE LEADING IN MANY OF THESE
SPACES AS WE BUILD OUT A NEW
DIGITAL STRATEGY AND SHOWCASE
THE APPLICATION OF DATA SCIENCE
THROUGH MACHINE LEARNING AND
ARTIFICIAL INTELLIGENCE, SO
WE'RE BUILDING A NEW STRUCTURE
AND PROCESS THAT WILL CREATE
CONSISTENCY THROUGH NLM THROUGH
A COMMON STRUCTURE AND PROCESS
AND I WILL SHOW MORE ABOUT THIS
ON THE NEXT COUPLE OF SLIDES.
NEXT SLIDE, PLEASE.
WE HAVE CONTINUED TO MAKE GREAT
PROGRESS IN ESTABLISHING THIS
FORWARD LOOKING APPROACH FOR
INNOVATION THAT IS APPLIED
CONSISTENTLY ACROSS THE NLM,
THIS PIVOTS US ACROSS SEPARATE
ACTIONS AT CONTINUOUS MODEL AND
INTENTIONAL KNOW CONSIST EPT
ACROSS UNITS AND EMBEDDED ACROSS
ALL LEVELS OF THE ORGANIZE EGG,
WITH THE DEVELOPMENT OF THE
ROADMAP OF KEY ACTIVITIES AND
MILESTONES REQUIRED FOR THIS
CULTURE OF CONTINUOUS INNOVATION
DEPLOATED BACK IN DECEMBER, WE
HAVE NOW STARTED TO BUILD ACROSS
4 MAJOR COMPONENTS, GOVERNANCE
WHICH IS TO CREATE A STRUCTURE
AND DECISION MAKING PROCESS TO
APPROVE NOVEL IDEAS FROM ALL
LEVELS OF STAFF ACROSS NLM, THE
IDEA LIFE CYCLE WHICH IS TO BELL
AN END-TO-END PROCESS THAT TAKES
IDEAS FROM INITIAL CONCEPT,
CONNECTED TO A PROBLEM
STATEMENT, FOR THE KRAGZ OF NEW
PROTOTYPES AND OR ALTERNATIVE
METHODS TO IMPROVE OUR DELIVERY
TO THE PUBLIC.
THIRD IS MEASUREMENT, IT DEFINES
STANDARD MEASURES TO EVALUATE
THE RESULTS FROM OUR INNOVATION
COMMITMENTS AND INFORM
CONTINUOUS LEARNING AND
IMPROVEMENT, AND FOURTH HIS
ORGANIZATIONAL READINESS TO
COMMUNICATE, ENGAGE AND
REINFORCE CONTINUOUS EDUCATION
ACROSS ALL LEVELS OF NLM.
WE ARE ON TRACK TO HAVE AN
INITIAL BUILD FOR EACH OF THESE
COMPONENTS BY THE END OF JUNE
WITH KEY ACTIVITIES CONTINUING
INTO THE REMAINDER OF FISCAL
YEAR 24 AND YOU THROUGHOUT
FISCAL YEAR 2025, NEXT SLIDE,
PLOAZ AS WE BUILD OUT THE
CONTINUOUS INNOVATION LIFE CYCLE
WE REMAIN COMMITTED TO CREATING
A SUFFICIENT, SUPPORTIVE AND
INCLUSIVE EXPERIENCE FOR ALL
STAFF AT EVERY LEVEL ACROSS THE
NLM.
WE RECOGNIZE THAT CREATING A
CULTURE OF INNOVATION IS A
RESPONSIBILITY THAT LIES WITH
EACH OF US, THEREFORE ANYONE AT
NLM WITH SUBMIT AN INNOVATION
IDEA, LEADERSHIP HAS BEEN
THOUGHTFUL IN ITS VISIONING TO
CREATE A PROCESS FOR NOVEL IDEAS
THAT IS TRANSPARENT AND
EQUITABLE REGARDLESS OF YOUR
POSITION, OFFICE OR DIVISION,
THERE WILL BE A SINGLE LOCATION
TO SUBMIT YOUR IDEAS AND VIEW
OTHER INNOVATION IDEAS FROM YOUR
COLLEAGUES.
IDEAS WILL BE EVALUATED THROUGH
COMMON AND CONSISTENT CRITERIA,
TO DETERMINE HOW AN INNOVATION
IDEA WILL MOVE FORWARD, AND AS
IDEAS MOVE FORWARD, A DEDICATED
TEAM WILL BE ASSIGNED TO HELP
BUILD AND REFINE THE INNOVATION
IDEA AND CREATE A PROTOTYPE.
ANC LAUNCHING INNOVATION ASSET,
IT WILL BE--COMMUNICATED ACROSS
THE NLM TO RECOGNIZE THE IDEA
OWNER AND ASSIGNED TEAM MEMBERS
FOR THEIR EFFORTS AND DEDICATION
FOR FURTHERING OUR DELIVERY TO
THE BIOMEDICAL COMMUNITY.
NEXT SLIDE, PLEASE.
WITH THAT, I WANT TO THANK YOU
FOR YOUR TIME AND YOUR ATTENTION
TODAY, I'M VERY PROUD OF THE
INCREDIBLE WORK COMING OUT OF
THE NLM'S INTRAMURAL RESEARCH
PROGRAM AND I LOOK FORWARD FROM
HEARING FROM OUR INVESTIGATORS
AS WELL AS NLM SCIENTIFIC
DIRECTOR DR. RICHARD SHERMAN,
RICHARD I WILL NOW PASS THIS
OVER TO YOU.
>> GREAT, THANK YOU STEVE.
ALL RIGHT, WELCOME EVERYONE.
I HAD A FEW COMMENTS I WANTED TO
MAKE AT THE BEGINNING OF OUR
MEETING TODAY--
>> YOU'RE SHARING THE PRESENTER
VIEW, WHICH IS FINE JUST FYI.
NOKAY, LET ME SWAP THIS.
>> THERE YOU GO?
THANKS, PETER.
LET ME START BY OFFERING MY
SINCEREST THANK TO YOU DR. KAING
AND DR. ONOMA CHAD O, TODAY WILL
BE THEIR LAST MEETING THEY WILL
BE ATTENDING.
WE REALLY APPRECIATE YOUR
SERVICE.
I KNOW THAT SERVING ON THESE
KINDS OF BOARDS REQUIRES
SIGNIFICANT EFFORT AND YOUR
CONTRIBUTIONS TO THE SUCCESS OF
THE INTRAMURAL RESEARCH PROGRAM
HERE IS GREATLY APPRECIATED SO I
JUST WANT TO OFFER MY SINCEREST
THANKS.
AND WE HAVE IDENTIFIED 2 NEW
BOARD MEMBERS WHO WILL BE
JOINING US FOR THE NEXT MEETING
WHICH IS DR. MONA SING WHICH IS
A PROFESSOR AT SCIENCE AT
PRINCETON UNIVERSITY AND
DR. WILLIAM HOGAN WHO IS THE
DIRECTOR OF A NEW DATA SCIENCE
INSTITUTE AT THE MEDICAL COLLEGE
OF WISCONSIN, SO WE LOOK FORWARD
TO HAVING THESE 2 NEW BOSC
MEMBERS JOIN US AT OUR NEXT
MEETING.
ONE OF THE THINGS WE HAVE BEEN
WORKING ON OVER THE LAST SEVERAL
MONTHS IS THAT WE ARE DEVELOPING
A STRAY JECTORY STIGIC PLAN FOR
THE NATIONAL LIBRARY OF
MEDICINES INTRAMURAL RESEARCH
PROGRAM EMPLOY SO WE HAVE
ASSEMBLED A COMMITTEE OF
INVESTIGATORS WITHIN THE IRP TO
BEGIN THAT PROCESS, AND WE'VE
RECENTLY ENGAGED CONSULTANT
CONTRACTOR WHO IS HELPING KIND
OF SHEPHERD US THROUGH THIS
PROCESS.
ONE OF THE PLANNED ACTIVITIES IS
THAT WE ARE GOING TO BE HAVING A
2 DAY RETREAT AT THE BEGINNING
OF UNIWHERE WE ARE REALLY GOING
TO BE DIVING INTO THE DETAILS OF
THE STRATEGIC PLANNING PROCESS.
AND THAT WILL INCLUDE DOING
THINGS LIKE A SWAT ANALYSIS
WHERE WE IDENTIFY THE STRENGTHS
OF THE EXISTING PROGRAM AND
POSSIBLE WEAKNESSES AND THEN
IDENTIFY OPPORTUNITIES FOR
EXPANDING OUR RESEARCH PROGRAM
TO ADDRESS SOME OF THE
WEAKNESSES WE IDENTIFY.
WE WOULD VERY MUCH APPRECIATE
INPUT FROM AND THE PERSPECTIVE
OF THE BOARD OF SCIENTIFIC
COUNSELORS IN THIS PROCESS AND
SO IN ORDER TO DO THAT, WE HAVE
SET ASIDE A DEDICATED SESSION
DURING THAT WORKSHOP ON
JUNE--MONDAY JUNE 3rd FROM
1:30 TO 315, WHERE WE WOULD LIKE
TO INVITE BOSC MEMBERS TO
PARTICIPATE IN THIS EVALUATION
PROCESS.
THIS IS AN OPTIONAL--YOU KNOW
ATTENDING THIS SESSION IS
OPTIONAL FOR BOSC MEMBERS BUT WE
WOULD VERY MUCH LIKE YOU TO BE
ABLE TO ATTEND IF POSSIBLE.
SINCE MANY OF YOU MAY HAVE
CONFLICTS FOR THAT TIME, WE WILL
ALSO PLAN ON SETTING ASIDE TIME
TO GET STRATEGIC--FOR STRATEGIC
DISCUSSIONS WITH BOSC MEMBERS
DURING FUTURE BOSC MEETINGS AND
ALSO IF WE MAY ALSO WANT TO
SCHEDULE SOME INDIVIDUAL
ONE-ON-ONE SESSIONS FOR PEOPLE
WHO WOULD NOT BE ABLE TO ATTEND
THAT SESSION DURING THE RETREAT.
SO JUST IN SUMMARY, YOU KNOW WE
ARE GOING TO HAVE A DEDICATED
SESSION FOR BOARD OF SCIENTIFIC
COUNSELOR MEMBERS ON MONDAY
JUNE 3rd FROM 1:30 TO 3:15,
AND WE WOULD LOVE TO HAVE YOU
ATTEND BUT ATTENDANCE IS
OPTIONAL.
SO DURING STEVE'S COMMENTS, HE
MENTIONED THE FACT THAT IN
DECEMBER, THE IRP TRAINEES VOTED
TO ESTABLISH A COLLECTIVE
BARGAINING UNIT THAT WILL BE
ADMINISTERED BY THE UAW.
THE RULES FOR ENGAGEMENT WITH
OUR TRAINEES THROUGH THIS
COLLECTIVE BARGAINING UNIT ARE
CURRENTLY BEING WORKED OUT SO
WE'RE STILL NOT SURE EXACTLY HOW
WE'RE GOING TO BE ENGAGING WITH
OUR TRAINEES IN THIS UNIONIZED
ENVIRONMENT BUT IN THE MEAN
TIME, WE WANT TO BE VERY CAREFUL
ABOUT HOW WE ARE INTERACTING
WITH OUR TRAINEES AND SINCE
THERE WILL BE SESSIONS DURING
THIS PARTICULAR MEETING WHERE WE
WILL HEAR POSTER PRESENTATIONS
FROM SOME OF OUR TRAINEES AND SO
WE JUST WANT TO MAKE SURE THAT
THE BOARD IS AWARE OF THE FACT
THAT WE ARE IN THE MIDDLE OF
THIS UNIONIZATION PROCESS, AND
THAT YOUR INTERACTIONS WITH THE
TRAINEES SHOULD REALLY FOCUS ON
UNDERSTANDING THE TRAINING
ENVIRONMENT THAT'S PROVIDED BY
THE MENTOR.
IT SHOULD NOT REALLY BE, YOU
KNOW CRITIQUING THE WORK OF THE
TRAINEE, I MEAN, PROVIDING
SCIENTIFIC FEEDBACK IS CERTAINLY
WELCOMED, BUT WE QUUOF THE WANT
TO MAKE SURE THAT--JUST WANT TO
MAKE SURE THAT WE'RE NOT TALKING
ABOUT THINGS LIKE THEIR WORK
ENVIRONMENT AND QUESTIONS LIKE
THAT WHICH WOULD BE COVERED
UNDER 1 OF THESE COLLECTIVE
BARGAINING AGREEMENTS.
SO YOU KNOW AS I'M SURE YOU WERE
GOING TO DO ANYWAY, FOCUSING IN
ON THEIR SCIENCE AND RESEARCH IS
PERFECTLY ACCEPTABLE AND REALLY
CONSIDERING YOU KNOW HOW THEIR
WORK IS A REFLECTION OF THE
MENTORING THAT THEY'RE RECEIVING
FROM THE INVESTIGATOR IS REALLY
THE FOCUS OF THAT ASSESSMENT.
AND THEN FINALLY, I JUST WANTED
TO REMIND PEOPLE THAT 1 OF THE
INVESTIGATORS WHO WILL BE
PRESENTING TODAY, IS DR. DINA
DEMNER-FUSHMAN, SHE IS AT THE
TENURE TAIJ IN HER TRACK WHERE
HE WILL BE CONSIDERING HER
PROMOTION TO A TENURED
INVESTIGATOR, SO WE WANT TO ASK
THAT THE BOARD CONSIDERS THAT
PROMOTION TO TENURE AS PART OF
THEIR EVALUATION PROCESS EMPLOY
SO THOSE ARE ALL THE COMMENTS
THAT I WANTED TO COVER THIS
MORNING.
DAVID, I DON'T KNOW IF THERE'S
ANYTHING ELSE FROM YOUR
PERSPECTIVE THAT WE WANTED TO
DISCUSS WITH THE BOARD BEFORE
THEY GET STARTED WITH THEIR
INVESTIGATOR PRESENTATIONS?
>> NO, THERE WAS 1 QUESTION FROM
[INDISCERNIBLE] ABOUT ARE THEY
ALREADY UNIONIZED, SO MY ANSWER
WAS YES, THEY WERE UNIONIZED
LAST YEAR BUT COLLECTIVE
BARGAINING IS BEING WORKED ON AT
THE PRESENT.
NRIGHT.
YEAH.
ALL RIGHT, THANK YOU.
>> AND WE'RE REGARDING AS ALMOST
COMPLETE.
>> YES, THEY HAVEUNIZED AUTOJUST
THE TERMS OF THE COLLECTIVE
BARGAINING AGREEMENT ARE RIGHT
NOW, SO WE DON'T KNOW WHAT THOSE
TERMS ARE YET.
ARE THERE OTHER QUESTIONS?
OKAY, IF NOT, WELL IF NOT I WILL
TURN IT BACK OVER TO PETER.
>> YEAH, I HAD A QUESTION,
REGARDING THE PUBLIC NATURE OF
THIS MEETING, IS THERE LIKE A
WAY THAT WE CAN KNOW WHEN--THERE
WAS AN INSTANCE IN THE PAST IT
WAS PUBLIC AND WE DIDN'T KNOW IT
WAS PUBLIC.
SO WHICH PARTS OF THIS SESSION
ARE OPEN TO THE PUBLIC WILL BE A
GOOD QUESTION TO CLARIFY.
>> MAYBE I CAN BRING UP THE
AGENDA AND MAYBE THAT WILL BE
HELPFUL.
>> I CAN ANSWER THAT PRETTY
QUICKLY AND EASILY.
IT'S EVERYTHING IS PUBLIC EXCEPT
FOR WHEN THE BSC IS IN COMMITTEE
WITH A SUBSET OF NLM IRP
MEMBERS.
SO WHEN YOU--ON THE AGENDA, I'M
LOOKING AT THE AGENDA, I IF I
CAN FIND IT, HERE IT IS, SO
WHEREVER IT SAYS CLOSED SESSION,
THAT'S NOT PUBLIC.
>> THANK YOU.
>> MAYBE I BETTER BE MORE CLEAR
ON THAT SINCE THE QUESTION CAME
UP, IN MY FUTURE AGENDAS.
>> ALL RIGHT, SO, THANK YOU VERY
MUCH RICHARD.
WHAT WE SHOULD I THINK GO AHEAD
AND DO NOW IS MOVE TO THE
PRESENTATION BY DR. LU, SO WE'RE
A FEW MINUTES EARLY BUT I WOULD
SAY LET'S GO AHEAD.
>> OKAY, GOOD MORNING EVERYONE.
CAN YOU SEE MY SLIDE?
>> YES.
>> YOU CAN HEAR ME OKAY, AS
WELL?
>> YES.
GREAT.
LET'S GET STARTED AGAIN, MY NAME
IS ZHIYONG LU, I AM A SENIOR
INVESTIGATOR AT THE IRP, MY LAB
IS INTERESTED IN USING AI AND
MISSION LEARNING FOR BIOMEDICAL
TEXT AND IMAGE PROCESSING ON A
DAILY BASIS.
WE PRIMARILY WORK WITH MILLIONS
OF PARTICLES IN THE BIOMEDICAL
LITERATURE WHEN ACCESS IS
ALLOWED, WE ARE ALSO WORKING
WITH PATIENT DATA INCLUDING
ELECTRONIC MEDICAL RECORDS AS
WELL AS CLINICAL IMAGES IN
PARTICULAR IN THE AREAS OF
OPERATING GLOBALLYTAL MOLOGY AND
RADIOLOGY BECAUSE WE HAVE GREAT
CLINICAL COLLABORATIONOT NIESH H
CAMPUS, SO USING AI, AND MISSION
LEARNING WE WORK ON A WIDE
VARIETY OF DIFFERENT PROBLEMS,
INCLUDING IMPROVING INFORMATION
ACCESS, BIOMEDICAL LITERATURE,
WHICH IS VERY TIED TO THE
MISSION OF NATIONAL LIBRARY OF
MEDICINE, BEYOND THAT, WE ARE
ALSO INTERESTED IN USING TO
EXTRACT, KNOWLEDGE FROM
UNOBSTRUCTED TEXT AND TURNING
THAT INTO STRUCTURED AND
COMPUTABLE INFORMATION AND LAST
BUT NOT LEAST, USING AI IN THE
LARGE MODELS WE'RE ABOUTED IN
TAKING THE PATIENT DATA
INCLUDING MOST IMAGE AND TEXT IN
PULTY MODAL FASHION FOR MISSION
DIAGNOSIS AND PROGNOSIS ALONG
WITH OTHER APPLICATIONS.
SO FOR TODAY'S PRESENTATION, I
WILL BE PRESENTING SOME OF THE
PROGRESS WE MADE IN EACH OF
THESE AREAS.
SO I WILL TART WITH
[INDISCERNIBLE], SO ON AVERAGE
DAY WE HAVE 5.2 MILLION USERS,
3 MILLION SEARCHERS AND
9 MILLION PAGE VIEWS SO EVEN
THOUGH AI IS MISSION LEARNING IS
A HOT TOPIC THESE DAYS BUT SINCE
DAY 1, AT NCBI AND NLM WHICH IS
ABOUT OVER 17 YEARS NOW, WE'VE
ALWAYS BEEN INTERESTED IN USING
AI IN THE MISSION TO IMPROVE PUB
MED SEARCH.
SO HERE JUST A LIST OF SEARCH
FEATURES THAT ASSIST AND MADE
AVAILABLE THROUGH COMPUTATIONAL
MISSION LEARNING TECHNIQUES, SO,
DIFFERENT FROM THE COMMERCIAL AI
SYSTEMS WHAT WE HAVE TRIED TO DO
IS TRIED VERY HARD TO EXPLAIN TO
OUR USERS HOW THE DETAILS OF
NUTS AND BOLTS OF THESE AI
GENERATED SEARCH ASSISTANCE.
SO WE HAVE PRETTY MUCH--EVERY
FOR EACH OF THE SEARCH FEATURE
WE HAVE A PAPER ASSOCIATED AND
IN THE PRESENTATION FOLLOWED BY
OUR TRAINEES, YOU WILL HEAR 1 BY
ASHLEY SHIN, A POST BACK IN MY
GROUP, SHE WILL BE PRECEPTING TO
YOU A NEW PROPOSAL THAT WE THINK
WILL INCREASE TRANSPARENCY OF
READING ARTICLE WHICH IS A
WIDELY USED FEATURE IN PUB MED.
BECAUSE WE BELIEVE THE MORE YOU
KNOW ABOUT A SYSTEM OR FEATURE
THE BETTER OUR USERS WILL MAKE
USE OF THIS.
BUT AS MUCH AS I LOVE PUB MED
BUT IN THE END IT IS A KEY WORD
BASED SYSTEM, SO THAT MEANS FOR
SIPTAL INFORMATION SEARCH NEEDS
SUCH AS FINDING THE PAPER ANDS
ONLY THE PAPERS OF A PARTICULAR
TOPIC, THIS TOPIC COULD BE COULD
COVID-19, GWAS, CANCER
DISPARITY, HOPEFULLY OUR USERS
WILL COME UP WITH A LONG LIST OF
SEARCH TERMS, THEY TRY TO BE
BOTH COMPREHENSIVE AND PRECISE
BUT YOU SEE THESE 2 THINGS ARE
TRADE OFF OF EACH OTHER, NOT TO
MENTION CERTAIN TOPICS SUCH AS
COVID, THERE'S NO TERM THAT KEY
WORD YOU CAN USE BECAUSE THE
TERM COVID-19 WAS NOT COINED AND
USED IN THE LITERATURE IN THE
RESEARCH COMMUNITY UNTIL SUMMER
2020.
SO, ALL THE PAPERS THAT BEFORE
THAT, ARE BASICALLY USING ALL
DIFFERENT WAYS DESCRIBING THE
COVID RESEARCH, SO IT WAS VERY
CHALLENGING IN THE BEGINNING OF
THE PANDEMIC TO ELECTRIC FOR
PAPERS ABOUT COVID RESEARCH AND
THAT OF COURSE BASICALLY OUR
MOTIVATION TO CREATE A CENTRAL
REPOSITORY FOR ALL THE PAPERS WE
CALLED LATE COVID AND IT PLAYED
AN IMPORTANT REALM FOR NOT ONLY
SCIENTIFIC LITERATURE BUT ALSO
THE GENERAL PUBLIC AND THIS WAS
TAKEN WHEN MY FORMER COLLEAGUE
AND POST DOC TRAINEE,
DR. [INDISCERNIBLE] WHO IS
STARTING HIS LAB AT YALE
UNIVERSITY.
NOW WE START THIS PROJECT WITH
JUST 2 POST DOCS AND MYSELF.
SO WE STARTED PROJECT, WE
BASICALLY HAVE TO MANUALLY LOOK
AT ALL THE PAPERS TO TAKE IN TO
SEE IF THEY'RE RELEVANT TO COVID
RESEARCH AND WHAT KIND OF TOPICS
THEY MENTION, SO WE STARTED, WE
WORKED ON THESE 2 PROBLEMS
MANUALLY FOR SEVERAL MONTHS OF
IN THE BEGINNING OF THE PANDEMIC
BUT AFTERRY REALIZE THIS IS NOT
SUSTAINABLE BECAUSE THE RAPID
GROWTH OF THE LITERATURE REALLY
JUST OUTPAYS THE TIME WE HAVE
AND THE 2 POST DOC CAN'T REALLY
AFFORD, BUT LUCKILY WE HAVE DONE
THIS MANUAL CURATION PROCESS, SO
WE USED THAT FOR DEVELOPING SOME
OF THE AI ALGORITHMS USING
TRANSFORMER MODELS TO ASSIST US
TO TRIAGE PAPERS, AND
SOPHISTICATED SO THAT MOVED TO
SOMETHING I CALLED AI ASSIST IN
THE SENSE THAT WE WERE AROUND
THE AI SYSTEM, BUT THEN EVERY
ARTICLE WITH AI PREDICTIONS IS
STILL BEING REVIEWED BY A HUMAN
BEING.
BUT THEN TIME GOES ON AND
THERE'S EVEN MORE PAPERS AND
THEN, EVEN THAT BECOMES NOT
SUSTAINABLE WHAT WE HAVE THEN
TURNED INTO IS THAT AI DRIVEN,
SO DURING AI ASSISTED PROCESS
THAT WE NOT ONLY REVIEW PAPERS
BUT WE ALSO CONTINUE TO
CALIBRATE OUR AI SYSTEM, SO IN
THE AI DRIVEN, WHERE IF AI
PREDICTION IS HAVING CONFIDENCE
VERY HIGH, WE WILL NOT HUMAN
REVIEW IT, BUT IF AI IS BORDER
LINE, OR NOT SURE, WE WILL DO
HUMAN REVIEW AND THAT COUNTS AS
20%, AND YOU IN THE END, YOU
KNOW WHY IT'S DONE AND MOST OF
THE TRAINEES LEFT AND STARTED,
NEW POSITIONS ELSEWHERE WE
TURNED THAT INTO A HUNDRED
MACHINE CURATION, SO THIS IS A
GREAT EXAMPLE I THINK TO SHOW
HOW AI CAN BE USED FOR DATABASED
CURATION AND THROUGH THIS 3
YEARS OF PANDEMIC, COVID REACHED
MANY, MANY USERS FROM ALL OF THE
WORLDS, WE HAVE OVER 60 MILLION
HITS FROM ALL OVER THE WORLD,
ALL OVER THE COUNTRIES INCLUDING
PEOPLE WORKS IN ANT ARCTICCA,
THIS IS A GREAT TOOL FOR NEXT
GENERATION SCIENTIST.
SO WE HAVE LILLY XU WHO WAS A
SUMMER INTERN IN 2022 AND WORKED
ON THIS PROJECT AND IN SEPTEMBER
WE HELPED HER PREPARE FOR AN
APPLICATION FOR AMIA HIGH SCHOOL
SCHOLARSHIP AND WE'RE LATER TOLD
IS MORE COMPETITIVE COMPARED TO
THE REGULAR PAPER SUBMISSIONS
AND LUKELY SHE GOT THAT IS THIS
IS A PICTURE OF HER PRESENTING
AT THE AMIA ANNUAL CONFERENCE IN
NOVEMBER AND SHE'S NOW AN
UNDERGRAD AT HARVARD.
SO SO FAR I TALKED ABOUT THIS,
THEY ARE HELPING YOU FIND THE
PAPERS THAT ARE INTERESTING BUT
GIVENNING AVERAGE OF KEY WORDS,
OFTEN TIME IS NONAPOPTOTIC THE
THAT THEY DON'T HAVE THE PAPERS,
THEY HAVE TOO MANY PAPERS
RETURNED BY THE SEARCH ENGINES,
HUNDREDS OF PAPERS, OR THOUSANDS
OF PAPERS SO THE PROBLEM IS THAT
NO 1, NO INDIVIDUAL RESEARCHER
CAN ACTUALLY TAKE THE TIME AND
READ ALL THESE PAPERS, NOT TO
MENTION MAKING THE CONNECTIONS
OR THE KNOWLEDGE IN THE
LITERATURE.
SO THIS IS SORT OF THE--TO MEET
THE MILLION DOLLAR QUESTION, CAN
WE TEACH COMPUTERS TO ACTUALLY
READ BECAUSE COMPUTERS CAN BE
REALLY GOOD AT THAT TO READ ALL
THESE PAPERS AND THEN MAKING THE
HIDDEN CONNECKS THAT IT WOULD
OTHERWISE NOT BE AVAILABLE TO
HUMAN SCIENTISTS FOR NEW PROCESS
GENERATION FOR DISCOVERY.
SO FOR THIS WE GO TAKE BABY
STEPS INCLUDING TRYING TO LOOK
THEA THE CONCEPTS AND WHAT IS
THE RELATION OF THE CONCEPTS.
SO LET ME GIVE AN EXAMPLE, HERE
A TITHES ARTICLE WHERE THERE'S A
MENTION OF A DISEASE, GENE AND
VARIANT BUT WE LIKE THE COMPUTER
TO RECOGNIZE THE TEXT WORDS,
THESE ARE REFERRED TO DEC GENE
AND VARIANT BUT ALSO TO MAP THAT
AND GROUND THAT INTO
STANDARDIZED ONTOLOGY AND
IDENTIFIERS OF [INDISCERNIBLE].
BUT SEE EVEN FOR THIS PARTICULAR
TASK IT'S NOT SO TRIVIAL BECAUSE
HUMAN LANGUAGE IS VERY RICH BUT
AT THE SAME TIME IT'S VERY
AMBIGUOUS, THERE ARE MANY
DIFFERENT WAYS THAT PEOPLE CAN
USE TO REFER TO THE SAME THING
AND BIOLOGISTS OR SCIENTISTS
INCLUDING MYSELF ARE NOT HARSHLY
KNOW THAT WE TYPICAL LYE DON'T
AGREE TO USE THE SAME TERM, EVEN
THOUGH THERE IS A NOMENCLATURE
FOR MANY THINGS, BUT AT THE SAME
TIME, THE SAME TERM CAN BE USED
TO REFER TO MANY DIFFERENT
CONCEPTS AS WELL, IN DIFFERENT
CONTEXT, THIS IS EVEN MORE
CHALLENGING IN THE EXAMPLE I
SHOWED YOU, THERE'S A GENE
DISEASE VARIANT NEXT TO EACH
OTHER BUT IN REALITY THERE ARE
MULTIPLE DECS, MULTIPLE GENES
AND VARIANTS ALL SCATTERED
ACROSS THE TEXT.
NOT TO MENTION TIMES HAVE YOU
OMISSIONS SO THIS MAKES THIS
MORE CHALLENGING SO THIS IS A
BIOMEDICAL INFORMATION
EXTRACTION, INFORMATION
EXTRACTION, AND THIS IS NOT
SOMETHING--THIS IS 1 OF THE
OLDEST PROBLEMS IN OUR FIELD AND
THE WHOLE COMMUNITY IS VERY
INTERESTED AND THE WORK ON THIS
PARTICULAR PROBLEM AND AND OUR
ROLE IS TO WORK WITH THE
COMMITTEE TO REALLY MOVING
FORWARD ON THIS SO IF THERE'S
SOMETHING CALLED BIOCREATIVE
WHICH IS I COMMUNITY WIDE
CHALLENGE EVENT FOR EVALUATING
BIOMEDICAL NLP SYSTEM,Y THIS ,-S
1 OF THE LONGEST RUNS EVENTS NOW
BECAUSE IT START INDEED 2004 AND
THROUGH THESE THIS SERIES, WE'RE
REALLY PUTTING UP SOME OF THE
TASKS FOR BENCHMARKING DIFFERENT
SYSTEMS AND 1 OF THE
CONTRIBUTION SYSTEM TO CREATE
THE CHALLENGE, TO CREATE THE
DAILY BASIS THEA SETS THAT CAN
BE USED NOT ONLY DURING THE
CHALLENGE BUT ALSO AFTER THE
CHALLENGE, SO, THERE'S A WEBSITE
CALLED PAPERS WITH CODON, I
DON'T KNOW IF YOU ARE FAMILIAR
WITH THE INDEX OR DIFFERENT DATA
SETS USED IN NLP, AI AND MISSION
LEARNING AND WHEN I FILTERED THE
TASK, I FIND ON TOP OF THE LIST,
HUNDREDS OF DATA SETS, SOME OF
THE TOP USED 1S INCLUDING THE 1S
WE HAVE DEVELOPED THROUGH THE
BIOCREATIVE AND THIS IS IN
COMPARISON TO MANY OTHER DATA
SETS THAT ARE USE INDEED THE
JOURNAL DOMAIN MUCH BROADLY AND
OUR LATEST CONTRIBUTION INCLUDES
A NEW DATA SET THAT WE CREATED
CALLED A BIO[INDISCERNIBLE]
WHICH I BELIEVE IS STILL THE
LARGEST DATA SET FOR BIOMEDICAL
RELATION INSTRUCTION TASKS, AND
THROUGH OUR COLLABARATORS
WITH--IN THE BIOCREATED 8 EVENT
LAST YEAR, WE HAVE HAD A
WONDERFUL TASK AND ALSO RECEIVED
ATTENTION AND PARTICIPATION FROM
MANY DIFFERENT GROUPS FROM
ACROSS THE WORLD.
AND OUR RESPONSE TO THIS IS
SOMETHING CALLED A
[INDISCERNIBLE] WHICH IS
SOMETHING THAT WE SEPARATELY
DEVELOPED [INDISCERNIBLE] SYSTEM
FOR DISEASES FOR GENES, STORY
DISEASES, FOR CHEMICALS AND
DRUGS AND THEN WE APPLY THESE
TAGGERS TO THE ENTIRE MILLIONS
OF ARTICLES IN PUB MED, IN THE
BACK END AND THEN WE AGGREGATE
RESULTS IN THE BACK END AND
RANDOMIZE RESULTS IN THE FRONT
FOR VISUALIZATION.
SO THE PUBITATOR TEST IS NOW
OVER 10 YEARS OLD AND THROUGH
THAT PROCESS WE HAVE ADDED
APIs, WE HAVE ADDED FTP, WE
EXPANDED FROM MINING RESULT ANDS
ANNOTATIONS FROM THE ABSTRACTION
TO THE FULL TEXT AND WE HAVE
CONTINUOUSLY IMPROVING ITS
TAGGING ACCURACY THROUGH THE
LATEST AND MOST ADVANCED MISSION
LEARNING AI TECHNIQUES AND I AM
HAPPY TO ANNOUNCE THAT YOU ARE
AMONG THE FIRST GROUP TO KNOW
THAT WE ARE GOING TO RELEASE THE
THIRD VERSION, THE 3.0 VERSION
OF THE PUP TATOR THIS YEAR AND
IT WILL RELEASE RESULT THIS IS
YEAR AND THIS IS AACCREDITTED TO
2 STAFF SCIENTISTS, DRS. WI AND
LEEMAN AND ALSO IN THE
PRESENTATION LATER ON BY
TRAINEES YOU WILL HEAR A SHORT
PRESENTATION BY SONIA, A SUMMER
INTERN HAOF THE YEAR, SHE
CLOSELY WORKED DRRKS REEMAN, 1
OF THE PROJECTS RELATED TO
PUPTATOR, 3, AND IT'S BEEN USE
INDEED A VARIETY OF DIFFERENT
CASES.
WE NOTED THEM HERE, WE KNOW, WE
LOOK AT PAPERS WHO CITED
PUBITATOR, AND ALSO THESE WHO
USE THEM AND ASKING FOR MORE.
SO IN TERMS OF USAGE, API IS THE
WINNER EACH THOUGH MAY MAKE
DIFFERENT WAYS PUBITATOR CAN BE
ACCESS THE OVER THE LAST COUPLE
YEARS WE HAVE OVER 1 BRILLION
API REQUESTS FROM MANY DIFFERENT
INSTITUTES AND WERERS ACROSS THE
WORLD AND LIKE TO MENTION, ALSO
HIGHLIGHT THAT FOR MANY
DATABASES INCLUDING FOR EXAMPLE,
COMPARATIVE TOXIC GENOMICS, CDG,
SWITZERLAND INSTITUTE OF
BIO-INFORMATICS, SWISS PRODUCT,
GWAS CATALOG, PHARMA GKB, ALL
THESE USING PUBTATOR, AND THEY
PUT IT IN THEIR SYSTEMS.
IT COVERS A WIDE VARIETY OF
CONTENTS, DISEASES, YEENS,
CHEMICALS, SO ON AND O FORTH AND
WE ALSO GET SPECIAL REQUESTS FOR
PEOPLE WHO ARE INTERESTED IN
VARIATION, SEQUENCE VARIATION
BECAUSE THEY MENTION A LOT OF
THINGS GET THEM LOST IF WE DON'T
TEXT MINE INFORMATION IN THE
SEPARATE [INDISCERNIBLE].
SO WE TREASES THAT CALL AND THE
REQUESTS BY KREAING A SPEN OFF,
WE CALL IT [INDISCERNIBLE] AND
SO NOT ONLY THE TEXT LINE HAD AN
ABSTRACT FOR TEXT BUT IN
[INDISCERNIBLE] WE GO 1 STEP
FURGT XER TEXT MINE THE
INFORMATION IN THE SEPARATE
MATERIAL BECAUSE THAT'S WHERE
THE INFORMATION IS REALLY
ENRICHED.
NOW WE DON'T GENERALLY DO THAT
FOR PUB TATOR BECAUSE AS YOU
KNOW THE SUPPLEMENTAL IT RECALL
IS REALLY NOISE EXPE WORKING
TOGETHER WITH THE COLLEAGUES
FROM NCBI, [INDISCERNIBLE]
GROUP, WE ARE MAKING SURE THAT
THEIR USERS ARE HAVING READILY
ACCESS TO THE LITVAR CONTENT AS
WELL.
SO NOW LET ME SWITCH GEAR AND
TALK TO ABOUT OUR PASSION IN
MEDICAL IMAGING AND I MENTIONED
AT THE VERY BEGINNING WE'RE
WORKING ON PARTICULAR
SPECIALTIES IN OPHTHALMOLOGY AND
RADIOLOGY BUT I WILL USE
OPHTHALMOLOGY AS AN EXAMPLE
TODAY AND IN PARTICULAR DISEASE
CALLED AGE RELATED MACULAR
DEGENERATION, AMD FOR SHORT.
THIS IS A LEADING CAUSE OF
BLINDNESS FOR ELDERLY POPULATION
AND IT'S PROGRESSED WITH--IT
PROGRESSES WITH AGE AND THE
EARLY STAGE INTERMEDIATE STAGE
AND LATE STAGE AS YOU CAN SEE,
IN THE LATE STAGE OR
INTERMEDIATE STAGE, WHERE THEY
REALLY GET IMPAIRED AND
UNFORTUNATELY THERE'S NO CURE
FOR THIS DISEASE AT THIS POINT
ALTHOUGH THE RESEARCH AND
CLINICAL TRIALS HAVE SHOWED,
RANDOMIZED TRIALS HAS SHOWED
THAT HIGH DOSE ZINC CAN HELP
SLOW DOWN THE PROGRESSION AT THE
RIGHT TIME, BUT ONLY AT THE
RIGHT TIME.
THIS IS LOOKING AT THE PACK OF
THE PATIENT'S EYE AND THEY
IDENTIFY 2 MAIN RISK FACTORS, 1
IS CALLED [INDISCERNIBLE] WHICH
IS FATTY DEPOSIT AND THE OTHER
IS PIGMENT CHARGES AND THERE ARE
2 FORMS OF LATE AMD,
[INDISCERNIBLE], GA FOR SHORT,
REFERRED TO AS THE DRY FORM AND
THERE'S THE WET FORM, CORNEAL
VASCULARIZATION, SO THIS
REQUIRES SPECIAL TRAINING, YOU
DON'T GET THIS BY A GENERAL
OPHTHALMOLOGIST.
YOU NEED A RETINAL SPECIALIST
AND EACH THAT IT IS VERY TIME
CONSUMING AND LABOR INTENSIVE IN
THE PROCESS, NOT TO MENTION IN
LANE PARTS OF THE WORLD THERE'S
LACK OF ACCESS FOR SUCH RETINAL
SPECIALISTS SO WORKING WITH
LONG-TERM COLLABORATOR, A
SPECIALIST IN AMD FROM NATIONAL
EYE INSTITUTE, WE SET UP
SOMETHING TO DISCUSS WHAT WE CAN
DO USING AI TO ADDRESS OR TO
IMPROVE THIS PROCESS BY AI TO
ASSISTING AMD DIAGNOSIS AND
PROGNOSIS AND THE TIME IS REALLY
RIGHT BECAUSE WE ALSO--SHE IS
ALSO THE LEADING PI FOR 2 VERY
LARGE CLINICAL TRIALS CALLED
ACREDS 1 AND 2, THIS IS THE
LARGEST FOR AMD AND 1 OF THE
LARGEST IN THE EYE SPECIALTY.
SO HERE WE HAVE, 10S OF
THOUSANDS OF IMAGES AND THOUSAND
PARTICIPANTS, WHAT'S EACH NICER
FOR AI OF COURSE IS ALL THESE
IMAGES HAVE BEEN MANUELLY GRADED
BY READING CENTER SPECIALISTS
AND THIS DATA IS LONGITUDINAL SO
WE CAN NOT ONLY DO DIAGNOSIS BUT
WE CAN DO PROGNOSIS AND RISK
[INDISCERNIBLE] AND THAT'S WHAT
WE DID WITH PIE FORMER POST DOC
[INDISCERNIBLE] WHO LEFT MY LAB
IN LATE 2020 AND IS NOW--WE'RE
STILL WORKS WITH [INDISCERNIBLE]
BUT THROUGH HIS NEW TRAINEES, ON
THESE 2 PROBLEMS BUT IN MY OWN
LAB WE HAVE SORT OF ALSO
SWITCHED, AND 1 STEP FORWARD AND
LOOKING AT SOMETHING IN MORE
DEPTH AND MORE [INDISCERNIBLE]
THE WAY WHERE WE TRY TO DISCOVER
WHAT ARE THE THINGS THAT MATTERS
IF ARE AMD PROGNOSIS THAT MAKING
PEOPLE FOR INTERMEDIATE AND TO
LATE STAGE, SO THERE IS
SOMETHING CALLED RPD FOR SHORT,
SO PEOPLE KNOW THIS, LONG TIME
AGO, BUT SOMETHING, THE EVIDENCE
OF RPD IS JUST NOT STRONG ENOUGH
BECAUSE THIS IS SOMETHING VERY
HARD TO BE SEEN ON THE COLOR
PHOTO, COLOR FROM THIS
PHOTOGRAPHS AND COMPARED TO
CLASSIC WHICH IS MUCH MORE
EASILY TO BE SPOTTED IN THESE
COLOR PHOTOS SO AS CAN YOU SEE
IN THE RIGHT FIGURE WHERE WE
SHOW EACH INDIVIDUAL DOTS, THESE
ARE BASICALLY HUMAN SPECIALISTS,
AS YOU CAN SEE, THESE--THEIR
PERFORMANCE FOR SPOTTING RPD IS
VERY CLOSE TO THE
[INDISCERNIBLE] LINE WHICH IS
RANDOM GUESSING AND WHILE YOU
CAN SEE THAT AI METHOD THAT WE
DEVELOPED IS CONSIDERED MUCH
BETTER AND MUCH BETTER THAN
HUMAN BEINGS IN THIS CASE, AI
CAN SEE SOMETHING THAT HUMANS
CAN REALLY NOT SEE SOMETHING
VERY WELL EMPLOY SO USING THIS
AI SYSTEM, WE BASICALLY
REPROCESSED THE HISTORICAL
ACREDS 1 DATA SET WHERE COLOR
PHOTOS IS THE ONLY DATA MODALITY
THAT'S AVAILABLE AT THE TIME SO
WHEN WE REPROCESS THAT, WE CAN
CHECK TO SEE THE PRESENCE, AND
ABSENCE OF RPD AND WITH THAT,
EVIDENCE ENHANCED THROUGH THE
HARD WORK OF DR. TIAMAN KEENAN,
HE JUST STARTED HIS NEW
INDIVIDUAL, INDEPENDENT LAB AS A
STANDARD [INDISCERNIBLE] AT AND
HE'S ALSO ADJUNCT INVESTIGATOR
AT NLM, WE DISCOVER THAT
EVIDENCE IS MOWPTING AND STRONG
ENOUGH TO SHOW THAT RPD IS A
THIRD MACULAR RISK FACTOR TO AMD
AND NOW WITH THAT KNOWLEDGE, WE
HAVE ANOTHER PAPER, JUST SENT
THIS WEEK AND IT'S COMING OUT TO
SHOW THIS WORK, AND THOU WE KNOW
IN ADDITION TO 2 OLD RISK
FACTORS, THIS THIRD RISK FACTOR
HOW CAN WE USE THIS INTO THE
DIAGNOSIS PRACTICAL FOR
PHYSICIANS.
AND YOU CAN DO THIS, WE CAN DO
THIS FOR AMD, WE DID THE SAME
THING VERY SIMILAR FOR CATARACT
EYE DISEASES, AS WELL AS FOR
DISEASES THAT ARE IN THE OTHER
PART OF THE PART, NOTHING TO DO
WITH THE EYES, SO FOR EXAMPLE,
IN COLLABORATION WITH THE TEAM
IN EUROPE, WE SHOWED THAT WE CAN
IN THE RETINAL SCAN TO PREDICT
HEART ATTACK, WE HAVE WORK BY
ANOTHER WORK, TO PREDICT COMPANY
FUNCTION DECLINE.
I ALSO WANT TO EMPHASIZE THESE 2
PAPERS ARE EXAMPLES WHERE WE
DEVELOP AI SYSTEMS AND THEN WE
HELP OUR COLLABORATORS OR OUR
COLLABORATORS HELPING US TO
EVALUATE ON THEIR OWN DATA SET,
SO THIS IS VERY IMPORTANT TO
MAKING SURE THAT AI ARE
DEVELOPED IN 1 SET OF DATA SET,
YOU KNOW FOR EXAMPLE, AREGS IS
PRIMARILY CAUCASIAN, AND IT CAN
WORK VERY WELL IN THE ASIAN
POPULATION, SO IN THE LAST 10
MINEUES I WILL TALK TO SOMETHING
ABOUT THAT WE ARE VERY EXCITED
ABOUT AND WE THINK WE ARE GOING
TO DO MORE IN THE NEAR FUTURE
AND IN THE NEXT 4 YEARS, AND THE
NEXT 1 WHICH IS WAS ANNOUNCED IN
NOVEMBER, 2022 AND EVERYBODY HAS
THEIR TAKE ON THIS, AND I'M SURE
KNOW THAT HOW POPULAR IT IS, NOT
ONLY AMONG OUR RESEARCH, LIKE
OURSELVES, BUT ALSO AMONG THE
GENERAL PUBLIC.
SOME OF THE EARLY WIDE WHAT OF
CHAT GPT, IT HAS PERFORMERS
COMPARABLE TO THE HUMAN PATTERN
COPE IN MANY YEARS AND EVEN MORE
AMAZING IS WITH LESS THAN 6
MONTHS IT'S PERFORMANCE HAS
REACHED CLOSE TO HUMAN EXPERT
PERFORMANCE.
YOU MAY HAVE HEARD THIS ABOUT IN
THE NEWS THAT CHAT GPT WAS ABLE
TO DIAGNOSE, MAKE THE CORRECT
DIAGNOSIS FOR A BOY WHO WENT OUT
TO SEE 17 DOCTORS OVER 3 YEARS
FOR A CHRONIC PAIN WITH NO
DIAGNOSIS, SO JUST ABOUT THIS
TIME LAST YEAR, MAYBE IN LATE
MARCH WE WROTE OUR VERY FESTER
PAPER, I RELATED TO CHAT GPT AND
YOU KNOW A PROBLEM THAT IS VERY
MUCH IMPORTANT IN OUR MIND WHICH
IS BACK TO THE LITERATURE
SEARCH, SO WE WERE ASKING
OURSELVES, YOU KNOW IF CHAT GPT
IS SO POPULAR AND SO SMART, WITH
PEOPLE JUST USING CHAT GPT TO
ASK MEDICAL QUESTIONS, DURING
THE PUB MED RESEARCH AND THEN
READING THE PAPERS, AND SO WE
SET OUT AND ASK CHAT GPT, CHA IS
THE LONG TYPICAL OR MOST COMMON
SYMPTOM ESPECIALLY WITH LONG
COVID.
CHAT GPT GIVE THOSE SYMPTOMS
WHICH ISN'T SO IMPRESSIVE TO US
BUT CHAT GPT SEEMS COMPETENT FOR
ANSWERS BY PROVIDING THE
EVIDENCE, ARTICLES OR RESEARCH
THAT BACK IN THESE SYMPTOMS,
HOWEVER, IF WE COP SEPASTE IT,
THESE TITLES AND PUT THAT INTO
PUB MED OR SEARCH ENGINES, WE
FIND NO SEARCH ARTICLE EXISTED.
THIS IS SOMETHING, OBVIOUS LE
EVERYBODY KNOWS THIS FABRICATING
INFORMATION ARE ALSO KNOWN AS
HALLUCINATION.
THESE--THESE ARE REAL THOUGH,
BUT [INDISCERNIBLE] IF WE PUT IT
INTO PUB MED, THEY HAVE
SOMETHING TO DO WITH COVID
RESEARCH.
SO THIS IS--SO WE WROTE THIS
PERSPECTIVE PAPER LAST YEAR AND
WE BASICALLY SAID THAT HUMAN
VERIFICATION AS THE LAST STEP IS
STILL VERY IMPORTANT, I THINK
EVEN FOR TODAY, THIS PERSPECTIVE
STILL IS VALID BECAUSE EVEN
TODAY, I DON'T THINK THAT THE
LARGE [INDISCERNIBLE] CHAT GPT
HAS COMPLETELY ADDRESSED OR
SOLVED ISSUE THAT WE MENTIONED
IN THAT ARTICLE.
SO THIS FITS IN 1 OF THE THEMES
THAT WE ARE INVESTIGATING LARGE
MODELS THAT RELATES TO
EVALUATION.
NOT ONLY WE EVALUATE HOW IT
WORKS FOR LITERATURE SEARCH OR
MEDICAL LITERATURE BUT WE ALSO
EVALUATE NLP TASKS THROUGH OUR
COLLABORATION WITH A GROUP AT
YALE.
WE ALSO HAVE DONE SOMETHING WITH
EVALUATION WITH THE LATEST GPT
WORLD VISION THAT INCLUDES BOTH
THE TEXT OR IMAGE ALTOGETHER
WHICH YOU WILL HEAR A
PRESENTATION BY DR. JIN IN MY
GROUP WHO IS A NEWLY JOINED POST
DOC FELLOW, HE HAS DONE A LOT OF
WORK INVOLVED IN THIS SPACE, SO
NOT ONLY ARE WE THINKING ABOUT
VALUE BUT HOW WE CAN USE OUR
DOMAIN EXPERTISE TO IMPROVE THE
STANDARD LARGE LANGUAGE MODELS.
YOU KNOW PEOPLE TALK ABOUT AND
KNOW THAT RETRIEVAL OF THE
GENERATION OR RAC FOR SHORT IS
ABLE TO INCLUDE STANDARD BUT
THERE ARE THINGS THAT ARE
SPECIAL IN OUR COMMITTEE, YOU
KNOW WHAT KIND OF DOCUMENTS YOU
WANT TO DO THE RETRIEVAL, WHAT
RETRIEVER YOU WANT TO USE.
THESE ARE MORE DEPTH RESEARCH
PROGRAMS WE HAVE STARTED TO
INVESTIGATE.
WE HAVE ALSO DOWN THE WORK TO
TEACH CHAT GPT TO USING TOOLS,
AND WE USE CHAT GPT TO USE THE
NCBI TOOLS AND BY DOING THAT, IT
BECOMES MUCH SMARTER IN
ANSWERING QUESTIONS RELATED TO
THE PROTEIN GENES AND THINGS
THAT IN OUR DOMAIN, AND THIS IS
OTHERWISE NOT AVAILABLE BECAUSE
I DON'T THINK THESE DOMAIN
DATABASE HAS BEEN USED FOR
TRAINING CHAT GPT.
SO UNDERSTANDING THE POTENTIAL
AS WELL AS THE RISK OF LARGE
MODELS, WE ALSO THINK HOW CAN WE
REALLY TAKE ADVANTAGE OF THE
GENERATIVE AI IS ITS CAPABILITY
FOR SOME OF THE MOST IMPORTANT
PROVINCE IN OUR FIELD, SO WE'VE
DONE SOME OF OUR PROJECTS BUT
FOR THE TIME BEING I WILL ONLY
TALK ABOUT 1 WHICH IS CALLED
TRIAL GPT, WHICH IS REALLY
EXCITING.
IT'S SAID THE DPOAL IS TO SPEED
UP THE PATIENT TRIAL MATCHING
PROCESS WHICH IS A REAL WORLD
PROBLEM AT THE NE H AND ALSO
MORE BROADLY, ABOUT YOU WE ARE
STARTING WITH THE NIH, SPECIALLY
IN OUR CLOSE COLLABORATION WITH
CLINICIANS AT THE NCI WHO
ACTUALLY RUN THOSE CLINICAL
TRIALS.
SO THE STATUS QUO, FOR THE
MOMENT IS THESE--THERE'S A TEAM
OF--THERE'S OFFICE IN THE TEAM
OF NURSES WHO WILL LOOK AT THE
PATIENT INFORMATION ON THE 1 END
AND TRY TO MANUALLY TO MATCH
WITH DIFFERENT TRENDS THAT ARE
RUNNING AT THE NCI OR CLINICAL
CENTER AT NIH, AND THIS PROCESS
AS WE TOLD BY OUR PHYSICIANS IS
VERY DIFFICULT OR NOT--IT'S
ERROR PRONE AND OFTEN TIMES THE
PHYSICIANS HAVE TO STEP IN AND
SPEND THEIR TIME TO DO JUST THIS
TRIAGE PROCESS EMPLOY SO WHAT WE
ARE TRYING TO DO IS TO IMPROVE
EFFICIENCY AS WELL AS PEOPLE
HAVE ACTESS TO PATIENTS IN ALL
DIFFERENT COMMUNITIES IN THIS
REGARD, EVEN THOUGH WE START
WIDE ONCOLOGY BUT THIS APPROACH
WE DEVELOP IS GENERALIZABLE AND
WE WERE INVITED TO WRITE A COVER
PAGE FOR TRIAL GPT FOR THE
AMERICAN NEUROLOGICAL
ASSOCIATION EMPLOY SO IN A NUT
SHELL WHAT TRIAL GPT DOES IS ON
THE 1 END WE HAVE PATIENT NOTES
AND ON THE OTHER TRIAL WE HAVE
CANDIDATES.
SO WE TAKE THE TRADITIONAL
METHOD, TAKE A BLACK BOX AND
TELL YOU THERE'S A MATCH,
WHETHER THERE'S PATIENT A GOOD
OR BETTER MATCH FOR THIS TRIAL,
BUT INSTEAD WE WILL HAVE
INTERMEDIATE STEP WHERE WE LOOK
AT EACH INDIVIDUAL INCLUSION,
EXCLUSION CRITERIA, WE HAVE
ASKED CHAT GPT TO EVALUATE BASED
ON THE PATIENT INFORMATION AND
WE ASKED CHAT GPT, NOT ONLY CAN
YOU TELL ME THE ANSWER, WHETHER
IT'S ELELIMINATEDDIBLE OR NOT
ELIGIBLE FOR EACH PERSON, BUT
ALSO EXPLAIN WHERE IS THE
EVIDENCE IN THE PATIENT NOTES,
THE RELEVANT SENTENCE AND WHAT
IS THE RATIONAL TO EXPLAIN THIS
AND BY DOING THIS, THIS IS
REALLY ADDING THE TRANSZ
APPEARANCE AND ALSO
EXPLAINABILITY OF THE AI SYSTEM
FOR THE POTENTIAL USERS, ABOUT
YOU IN ADDITION THIS ALSO HELPS
INCLUDING TO PUT CHAT GPT IN A
CONTROLLED FASHION THAT YOU CAN
MITIGATE POSSIBLE
HALLUCINATIONS, SO IN DOING
THAT, WE HAVE--WE HAVE BUNCH
MARK, ON 3 DIFFERENT DATA SETS
BECAUSE THIS IS A KNOWN PROBLEM,
WHERE WE HAVE 200 PATIENTS AND
20,000 TRIALS AND STATE OF ART
METHOD WHAT'SY HAVE LOOKED AT
HAVE ACHIEVED ONLY 55% ACCURACY
WHILE IN OUR CASE, TRIAL GPT IS
81% WHICH IS BEASKLY ON PARWITH
HUMAN [INDISCERNIBLE].
BECAUSE EVEN IF YOU ASK HUMANS
YOU MAY NOT BE GETTING THE
PERFECT ANSWER AND GETTING THAT
VERY ENCOURAGING RESULTS IN THE
OFFLINE EVALUATION WE HAVE
STARTED WE HAVE STARTED OUR
STUDY WHERE WE HAVE PHYSICIANS
LOOKING AT THE RESULTS WITH AND
WITHOUT AI ASSISTANCE AND WE
SHOWED THAT WITH THAT AI SYSTEM,
WE CAN DROP 40% OF TIME WITHOUT
ANY SACRIFICE IN ACCURACY.
SO I MENTIONED OTHER FUTURE
DIRECTIONS 1 WE HAVE MOVING ON
FROM TRADITIONAL HISTORICAL
COLOR, PHOTOGRAPHS TO THE MORE
ADVANCED OCT IMAGES IN
OPHTHALMOLOGY, I DIDN'T TALK
MUCH ABOUT RADIOLOGY BUT WE HAVE
THIS WONDERFUL COLLABBUATION
WITH DR. RON SUMMERS FROM THE
CLINICAL CENTER RADIOLOGY
DEPARTMENT WHERE HE ALSO HAS
BEEN A CO MENTOR FOR SEVERAL OF
MY TRAINEES AND THE MOST RECENT
1 IS DR. ZHU WHO HAS A PASSION
FOR WORKING ON CT IMAGES
TOGETHER WITH THE TEXT DATA.
SO WE ARE NOW WORKING ON 1
PROJECT, WE WILL POTENTIALLY
RELEASE A MULTIMODAL DATA SET
FOR THE CT DATA SET, THERE'S
NOTHING LIKE THIS AVAILABLE FOR
OUR RESEARCH COMMITTEE AT THE
MOMENT.
THERE'S A LOT INCLUDING THE 1
THAT RON AND I WORKED ON IN THE
RELEASE CALLED NIH CHECKS THE
RATE, IN 2017, WE RELEASED THAT
VERY LARGE AND THAT ACTUALLY WAS
VERY WELL RECEIVED BUT I HAVE A
HIGH HOPE THAT THIS CT WILL BE
ANOTHER HIGH IMPACT PROJECT AS
WELL.
AMONG OTHER THINGS WE TALKED
ABOUT THE NUMBER 1 PRIORITY FOR
AI SYSTEM, I THINK HAS TO BE THE
SAFETY, SO FOR THAT, THIS IS THE
TOPIC FOR MY Ph D STUDENT,
[INDISCERNIBLE] AT THE COMPUTER
SCIENCE AT UMD, COLLEGE PARK, IN
THE YEAR AND HALF HE'S WITH US,
HE HAS DONE INTERESTING RESEARCH
INCLUDING ARE VEALING THE RACIAL
BIAS OF LARGE [INDISCERNIBLE]
PRODLES, AND WE'VE SHOWN HOW
STRICT AND STRAIGHT FORWARD TO
GENERATE SOME ADVERSARIAL OR
HARMFUL OUTCOME, YOU MAY HAVE
HEARD HOW AI GENERATED DAMAGING
STATEMENTS AND PICTURES BUT
SHOWING POTENTIAL RISK ANDS
BIASES, NOT ENOUGH IN MY MIND
FOR WE HAVE TO DEVELOP NOVEL ART
WITH THEM AND THIS IS THE
TECHNICAL CONTRIBUTION PART OF
HIS THESIS.
JUST ON WRAPPING UP, SO NLM HAS
BEEN A LEADER FOR AI AND
IMPROVING USER EXPERIENCE, WE
HOPE WE CAN CONTINUE TO
CONTRIBUTE TO THIS MISSION
THROUGH VARIOUS PROGECS
INCLUDING PUTTING AI IN REAL
WORLD APPLICATIONS THAT ARE VERY
CLOSE TIED TO THE MISSION OF
NLM, INCLUDING THE LITERATURE
SEARCH AND INDEX, THIS HAS TO BE
1 NLM PROJECT MEANING WE HAVE TO
ENGAGE AND INVOLVE A LOT OF
DIFFERENT PARTS OF NLM DIVISIONS
AND COLLEAGUES.
BUT ALSO WE ARE INTERESTED IN
DOING SOMETHING THAT SEE MORE AI
IN THE POTENTIAL CLINICAL SPACE
WHERE JUST TO SEE THE THINGS WE
HAVE DONE WITH AMD FOR EXAMPLE,
THIS WILL EVOLVE, NOT ONLY THE
NLM BUT ALSO OTHER ICs AT NIH
AND ALSO FDA, SO EVEN THOUGH,
I'M THE RESENTER TODAY BUT
REALLY THIS IS THE WORK FROM ALL
THE PEOPLE HARD WORKING AND
TALENTED GROUP MEMBERS FROM MY
GROUP.
AND AS WELL AS MY COLLABORATORS,
SO I WILL STOP HERE AND HAPPY TO
TAKE QUESTIONS.
THANK YOU.
NTHANK YOU VERY MUCH FOR YOUR
PRESENTATION, THIS IS OUTSIDE OF
MY AREA EXPECTATIONS PETTER EASE
SO I HAVE NAIVE QUESTIONS FOR
THE NATURAL LANGUAGE PROCESSING
MODELS OR ALGORITHMS YOU USE TO
POWER THOSE LITERATURE SEARCHES,
HOW DO YOU VALIDATE THOSE?
I'M WONDERING ABOUT WHAT THE
GROUND TRUTH IS.
I'M WONDERING ABOUT AMD WHERE
YOU HAVE THE CLINICALLY
VALIDATED IMAGES, AS WELL THERE
ANALOGOUS GROUND TRUTH FOR
NATURAL LANGUAGE.
>> YEAH, YEAH, GOOD QUESTION.
THANK YOU FOR THAT QUESTION.
SO YES, SO, IT VARIES FOR
DIFFERENT PROBLEMS THAT WE ARE
TALKING ABOUT, SO, I JUST GIVE
YOU SOME EXAMPLES WHERE WE
EVALUATE THE SYSTEM IN GENERAL.
SO WE GENERALLY
CERTAINLY--CERTAINLY
VALENTINEDUATE IN 2 PARTS.
FIRST PART WE WILL HAVE SOME
DPROWND TRUTHS THAT ARE MANUELLY
ANNOTATED BY HUMAN EXPERT
SUBJECTS SO IF WE'RE TALKING
ABOUT RELATED ARTICLES SO WE
WOULD KNOW THAT WHICH ARTICLE IS
RELEVANT TO THIS ARTICLE AND
THESE PAIRS OF ARTICLES HAS BEEN
MANUALLY ANNOTATE FRIDAY HUMAN
SUBJECT HUMAN BEINGS AND TIMES
THESE MANUELLY GENERATE OR
MANUELLY ANNOTATED DATA SETS OR
GROUND TRUTH ARE EITHER BY US,
BY OURSELVES, OR--WELL, LET ME
PUT IT THIS WAY, IF THERE'S SUCH
DATA SET AVAILABLE BY OTHER
RESEARCHERS, WE WOULD USE THEIR
DATA SETS, FIRST, OKAY, IF
SOMETHING AVAILABLE BUT THIS IS
IMPORTANT, WE WILL GENERATE AND
DO THIS, OF COURSE THIS HAS TO
BE DONE BLIND WAY, WE CANNOT
HAVING PEOPLE TO DO THIS AND
THEN EVALUATE.
SO BUT THE PROBLEM WITH THIS IS,
WELL NOT THE PROBLEM BUT WE
ALWAYS DO THIS AT THE FIRST STEP
AND WE SHOW THAT OUR ALGORITHM
WORKS WELL ON THIS EVALUATION
DATA SET BUT SOMETIMES THIS DATA
SET ARE RELATIVELY SMALL
COMPARED TO MILLIONS OF PAPERS,
RIGHT IN SO THE NEXT TEP IS IF
WE SEE ENOUGH SIGNALS, ENOUGH,
YOU KNOW, WE HAVE ENOUGH
CONFIDENCE THAT OUR RHYTHM WORKS
BETTER THAN IN THESE OFFLINE
BENCH MARK AND DATA SET IN TERMS
OF, YOU KNOW F1 SCORE OR
ACCURACY OR RETRIEVAL RANKS AND
THEN WE STARTING TO IMPROVE AND
EVALUATE THIS IN AN ONLINE
FASHION, SO HERE THE EXAMPLE, I
WOULD USE IS FOR EXAMPLE, THE
RELEVANT SEARCH IN PUB SO ONCE
WE HAVE EVALUATED ALL THE
ALGORITHMS TO DEVELOP, ON
MISSION LEARNING AND BENCHMARK
AND DATA SETS THEN WE ARE
EVALUATING IN THE REAL WORLD
USERS, SO IN THAT CASE, WE DO
SOMETHING CALLED AB TESTING
WHICH IS SORT OF LIKE A CASE
CONTROL STUDY, IN BIOLOGY, WHERE
WE WOULD HAVE BOTH VERSION OF A
AND B WITH THE NEW ALGORITHM AND
WITHOUT THE NEW ALGORITHM AND
WHAT WE MEASURE IS CALLED A
CLICK THROUGH RATE.
SO OUR DIFFERENT HYPOTHESIS IS
THAT [INDISCERNIBLE] WOULD LIKE
TO SEE THAT MORE AND WILL CLICK
MORE.
SO THAT IS SOMETHING THAT WE
MONITOR AND THIS IS VERY MUCH
SIMILAR TO THE TECH COMPANIES,
YOU KNOW GOOGLE WHEN THEY
RELEASE NEW FEATURES, THEY DO
THIS KIND OF
CERTAINLY--CERTAINLY VALUATION
AS WELL, AND WE WERE VERY
CAUTIOUS ABOUT DOING THIS
PROCESS, WE WOULD ALWAYS
STARTING WITH 1% OF OUR USERS,
AND THEN ONLY 5% OF OUR USERS
AND IF THE SIGNAL IS CONTINUED
TO BE STRONG AND THE DIRECTION
THEY ARE LOOKING TO SEE, THEN WE
OPEN THAT TO 10% OR 25% TO
EVENTUALLY A HUNDRED PERCENT.
THIS IS HOW WE EVALUATE BECAUSE
WE UNDERSTAND THAT EVERY CHANGE
WE MAKE IN PUB MED CAN AFFECT
MILLIONS OF USERS SO THIS IS THE
GENERAL PIPELINE THAT WE DO.
EVEN AFTER WE ROLL OUT AN
ALGORITHM IN PUB MED, WE MONITOR
ITS USAGE AFTERWARDS AND WE
CONTINUE TO IMPROVE.
MEANING SOME OF THE MODELS WE
DEVELOP INITIALLY, FOR EXAMPLE,
THE NEW RESEARCH THAT WEB
CONNECTED DEVELOP AND PUT INTO
PUB MED WAS IN 2017, THAT MODEL
IS ACTUALLY CONTINUOUSLY
PERIODICALLY UPGRADED, UPDATED
EVERY 6 MONTHS SO WE YOU MAKING
SURE THE MODEL ACTUALLY FIT WITH
THE MOST RECENT REQUEST OF OUR
USERS, WITH THE LATEST DATA WE
HAVE.
SO THESE ARE SORT OF THE WAYS
THAT WE DO FOR THE NRP.
>> IT LOOKS LIKE THERE'S A BUNCH
OF QUESTIONS LINED UP SO--
>> THANK YOU FOR YOUR
PRESENTATION
PRESENTATION EMPLOY I HAD 1
TECHNICAL QUESTION AND THEN
LARGER QUESTION, YOU KNOW YOU
MENTIONED RAG AS GREAT
OPPORTUNITY FOR ESPECIALLY THE
KIND OF WORK THAT YOU ARE DOING
HOW DO YOU--HOW DO YOU ENVISION
USING RAG LIKE IS REFERENCE TIME
OR DO YOU THINK THERE'S AN
OPPORTUNITY GIVEN YOUR ACCESS TO
ALL THE INFORMATION THAT YOU
HAVE AND ALL THE DATA THAT YOU
HAVE TO THINK ABOUT RAG
APPROACHES AT THE TRAINING TIME
OF IT?
>> THAT'S A GOOD QUESTION.
SO, SO FAR WHAT WE DID AT THE
INFERENCE TIME, MORE ALIGNED
WITH OTHER, THE PRACTICE OF
USING RAG, NOW THIS COULD BE
CHANGING IF YOU'RE FAMILIAR WITH
THIS, BECAUSE THE NEWER VERSION
OF THE PLASMA CAN TAKE MUCH
LONGER CONTEXT THAN THE 32 K FOR
THE MOMENT, IT COULD BE MILLIONS
SO FOR THE MOMENT TO STUDY THAT
I WAS REFERRING TO, AND WHEN WE
BASICALLY EVALUATE DIFFERENT
KIND OF [INDISCERNIBLE] THAT WE
CAN USE AND WE CAN PROVIDE TO
THE MEDICAL,--FIRST OF ALL, LET
ME TAKE 1 BACK, SO WE EVALUATE
ON THE MEDICAL QA, DIFFERENT
MEDICAL QA DATA SETS AND THEN WE
COMPARED MULTIPLE LARGE IMAGE
WITH AND WITHOUT RAG AND THEN
USING RAG, WE ARE PARTICULARLY
LOOKING AT 2 DIFFERENT ASPECT, 1
ASPECT IS WHAT DOCUMS CAN BE
SUPPLIED FOR HELPING.
IT COULD BE PUB MED ARTICLES, IT
COULD BEING BOOKS, IT COULD BE
SOME OTHER MEDICAL TEXT, AND
THEN, THE OTHER--THE OTHER
DIMENSION WE LOOKED INTO IS WHAT
RETRIEVAL CAN BE USED FOR THIS
PURPOSE, IT COULD BE GENERAL,
N25, COULD BE MORE DENSE
RETRIEVERS SO ON AND O FORTH SO
BY COMBINING THIS INFORMATION
TOGETHER WE HAVE SOME GUIDANCE
AND SUGGESTION FOR SOME OF THE
BEST PRACTICE FOR RAG, NOW THIS
IS DEFINITELY, THINGS ARE STILL
CHANGING LIKE YOU KNOW WITH THE
MORE ADVANCED THAT MAYBE PEOPLE
WILL STILL DO SOMETHING
DIFFERENT BUT I THINK FOR THE
MOMENT, THAT THE FINDINGS THAT
WE UNCOVER INDEED THIS RESEARCH
WILL BE VERY INSIGHTFUL FOR
PEOPLE WHO ARE INTERESTED IN
USING RAG BECAUSE IT'S SO FAR AT
THE MOMENT IT SEEMS TO BE 1 WAY
TO GO, 1 OF THE IMPORTANT THINGS
WE CAN USE TO IMPROVE STANDARD
LARGE LANGUAGE MODELS FOR OUR
DOPAIN EMPLOY --DOMAIN.
BECAUSE I'M NOT A BELIEVER IN
CREATING LARGE RESOURCE MODELS
GIVEN THE TECHNICAL CHALLENGES
WE FACE IN THIS DOMAIN.
>> INTERESTING, THANK YOU.
MY SECOND QUESTION IS SHORT BUT
I'M CURIOUS ABOUT THE CHOICE OF
GPT, CHAT GPT, BUT NOWADAYS YOU
MENTIONED MIXED ROLE IS THERE
ANY INTEREST--LIKE WHAT'S A
REASON ON CHOOSE A COMMERCIAL
NONTRANSFER ENSEL LARGE LANGUAGE
MODEL VERSUS [INDISCERNIBLE] 1
IN YOUR AGENDA, I GUESS.
>> YEAH, YEAH, GREAT QUESTION.
SO, YOU KNOW AT THE
BEGENERATEDDING, WELL, THERE ARE
A COUPLE REASONS, 1 IS ERGS
PECIALLY AT THE BEGINNING, CHAT
GPT SEEMS TO BE PERFORMING
BETTER THAN OPEN SOURCE MODELS,
THAT'S 1 OF THE REAPS, BUT I
GUESS, THE MOST IMPORTANT REASON
IS, DATA PRIVACY.
OKAY, SO WE WILL ACCOUNT FOR
EXAMPLE, TRIAL GPT PROJECT.
WE DEAL WITH SENSITIVE PRIVATE
PATIENT INFORMATION EMPLOY EVEN
THOUGH THESE ARE SEMI--THESE ARE
DEIDENTIFIED IN A WAY, BUT WE
THINK THIS IS MUCH MORE SAFE AND
CONTROLLED ENVIRONMENT BECAUSE
WE USE CHAT GPT NOT THROUGH THE
OPEN API, SO I DIDN'T MENTION
THAT, I SHOULD MENTION THAT.
WE DO EVERYTHING HERE WITH
MICROSOFT [INDISCERNIBLE]
ENVIRONMENT THROUGH THE CLUED
NIH [INDISCERNIBLE] INITIATIVE,
WHERE WE HAVE THIS HIPAA
COMPLIANT ENVIRONMENT THAT WE
CAN TEST AND RUN OUR RESEARCH
PROJECTS.
SO THIS IS SOMETHING THAT I
THINK TO US IS THE NUMBER 1 MOST
IMPORTANT THING TO DO AND
BECAUSE OF THAT WE CAN ACTUALLY
CREATE SOMETHING THAT IF
WE--LIKE I MENTIONED, CHAT GPT
AT THE MOMENT EXTENT AT THE
RESEARCH BUT IT'S SOMETHING THAT
COULD BE USED WITH OUR
PHYSICIANS AT NCI OR OTHER
ICs, THEN WE HAVE TO CREATE
SOMETHING THEY CAN USE IN A SAFE
CAN [INDISCERNIBLE] ENVIRONMENT
AND THIS HAS TO BE SOMETHING
LIKE AZURE, THE OPEN SOURCE 1S.
WE DO USE THAT, I DID NOT
MENTION THAT, WE DO USE THAT IN
MANY OF OUR PAPERS WHEN WE'RE
COMPARING WITH CHAT GPT 3.0, 5
AND 4, WE USE EXTENSIVELY FOR
EXAMPLE, THE [INDISCERNIBLE] 7 B
MODEL, FOR OUR EVALUATION AND
COMPARISON PURPOSE SO WE DO USE
OPEN SOURCE ONCE BUT NOT--NOT
SOMETHING THAT WE USE.
THANK YOU.
>> LUCILLA WAS NEXT AND THEN
CANNED WHAT AND THEN ME.
>> YEAH, THANKS FOR THE
QUESTION, MY QUESTION WAS
RELATED TO THE HIPAA
ENVIRONMENT, IS IT A SPECIAL
ARRANGEMENT IN THAT YOU'RE
PROVIDING THE DATA TO TRAIN THE
OPEN MODEL IN AI OR IS IT THE
CASE THAT YOU DON'T HAVE AN
ENVIRONMENT THAT OHM HAS THE
PRETRAINED MODEL IN THAT
EVERYTHING REMAINS WITHIN THE
NIH PLATFORM.
>> YEAH, GREAT QUESTION.
SO, THE 1 THAT WE'RE USING
THROUGH [INDISCERNIBLE], IT
GIVES US ACCESS TO THE LATEST
GPT MODELS BUT ALSO PROTECTS
JUST LIKE YOU'RE SAYING THAT
DATA THAT'S USED WILL NOT BE AS
I UNDERSTAND AS MICROSOFT AS YOU
TOLD US IS NOT GOING TO BE USED,
PUT BACK FOR OPEN AI FOR THEIR
MODEL TRAINEE, SO, IT'S
PROTECTED IN THAT SENSE.
WE HAVE FOR EXAMPLE, USED OTHER
DATA SETS LIKE THE MIMIC DATA
SET FOR EXAMPLE, AND AS YOU KNOW
THAT, YOU KNOW IN OF THE
RESEARCH IN EARLY DAYS, THEY
USED OPEN AI API AND USING
IMITATED DIRECTLY AND THAT WAS
ESSENTIALLY NOT A GOOD THING AND
OUR COMMITTEE SOON REALIZED WE
SHOULD NOT DO THAT SO USING THE
AI ENVIRONMENT ACTUALLY PROTECTS
US AND ALLOW US TO PURSUE OUR
RESEARCH PROJECT IN THAT WAY.
>> THANK YOU.
>> JUST A QUICK ADD ON TO THAT,
WE HAVE A SIMILAR RELATIONSHIP
AND IN ADDITION TO NOT USING IT
TO TRAIN, THEY ALSO DON'T RETAIN
ANY OF THE INFORMATION THAT'S
BEEN SENT.
SO THERE'S A NUMBER OF
PROTECTIONS THERE.
IT'S ENCRYPTED IN TRANSIT, IT IS
USED TRANSIENTLY, IT IS NOT
KEPT, NOT USED TO TRAIN.
SO IN THEORY IT IS SAFE, I SAY
IN THEORY BECAUSE--
>> --TRUST WHAT THEY TOLD US.
>> --WELL WE WERE IN A BIT OF
DISCUSSION WITH THEM TO VERIFY
THIS IN A NARR BIT OF DETAIL SO
I THINK, HOPEFULLY IT'S TRUE.
[INDISCERNIBLE]
>> YES, I HAVE A COUPLE
QUESTIONS, FIRST OF ALL
EXCELLENT PRESENTATION AND
RESEARCH IN GENERAL.
I HAVE A COUPLE QUESTIONS AS
TO--I'M A BIG FAN OF YOURS, SO I
COME MORE FROM THAT DIRECTION,
BUT ESPECIALLY IN THE FIELD OF
DETECTING VARIANT ANDS IMPACTING
AND IMPACTING DEVISION VARIANTS
AND SUCH, ARE YOU SORT OF THE
ENTITIES THAT PUB TATOR IS
RECOGNIZING, FOR INSTANCE I
NOTICE THERE ARE PATHWAYS THAT
THEY ARE NOW PART OF YOUR RECK
NIGS AND YOU HAVE GENE DISEASE
AND MUTATION?
ANOTHER QUESTION RELATED TO PUB
DATA ARE YOU--IT WASN'T VERY
CLEAR TO ME IN YOUR PRESENTATION
AND YOUR DOCUMENTS EXACTLY HOW
THE PROGRESS OF PUBTATOR 3 IN
RECOGNIZING RELATIONSHIPS
BETWEEN THOSE ENTITIES WHICH
IS--WE HAD TRIED AND I WOULD
LIKE TO DO, IT'S A VERY
DIFFICULT TASK AND I THINK IT'S
REALLY SIGNIFICANT FOR, YOU KNOW
UNDERSTANDING THOSE
RELATIONSHIPS IN MOLECULAR
VERSUS PHENOTYPICAL
RELATIONSHIPS.
SO THAT WOULD BE MORE MY
QUESTION WITH THAT.
AND THE THIRD QUESTION, JUST
RELATED, THIS IS ONLY MY FIRST,
THE PUB TATOR, IF YOU ARE ALSO
THINKING OF LINKING MORE, YOUR
PUB TATOR IN YOUR SEARCH AND
YOUR RESEARCH THERE WITH YOUR
BIOCREATIVE, YOU KNOW DEPOSITING
BENCHMARKING WE CAN USE ALSO FOR
OUR TRAINING OF, YOU KNOW
MACHINE LEARNING TOOLS.
SO YOU HAVE TAKEN--THEY PUB
TATOR AND ASK YOU ANOTHER NTHANK
YOU.
>> SO, I THINK THIS IS
REALLY--THANK YOU FOR LETTING ME
CLARIFY THAT I MAY NOT HAVE PUT
IN MY PRESENTATION IN RICHER
MATERIAL, SO AS YOU KNOW THAT
RELATIONSHIP [INDISCERNIBLE] IS
NOT A NEW THING.
A LOT OF US WHO WORKED ON THIS
KNOW ABOUT THIS AND THE REASON I
HESITATE AND INCLUDING OUR OWN
GROUP AND OTHERS GROUPS THAT ARE
ON THE CALL, THE REASON THAT WE
DIDN'T PUT THIS INTO PUB TATOR
IS BECAUSE I WAS RELUCTANT TO DO
THAT BAH OF THE RACERACY OF THE
RELATIONSHIP ISN'T THAT GOOD TO
BE HONEST, OKAY EMPLOY DEPENDING
ON THE KIND OF RELATIONS YOU'RE
TALKING ABOUT, IT COULD BE
RANGING FROM 80% TO REALLY 20%,
OKAY, SO THAT MADE ME REALLY
UNCOMFORTABLE PUTTING THAT INTO
THE EARLIER VERSION OF THE PUB
TATOR AND PART OF THE REASON IS
BECAUSE THERE WAS NO GREAT DATA
SET FOR SUPER VISING LEARNING
FOR TRAINING RELATION EXPRESSION
MODELS, AND THAT IS WHY WE, I
THINK 2022, 3 POST DOCS, 3 FTs
THEY SPENT 1 WHOLE YEAR AND
CURATED THE ABSTRACT WITH ALL
THE RELATIONS IN THE PAPERS SO
THAT WAS THE BIORED DATA SET I
WAS REFORRING TO AND AFTER THAT
EVEN THOUGH IT TURNED OUT TO THE
MOST LARGEST AND WITH THE MOST
COMPREHENSIVE COVERAGE WITH
DIFFERENT RELATIONSHIPS, NOT
JUST, YOU KNOW IT COULD BE
DRUG-DRUG INTERACTION, DRUG-DEC
RELATIONSHIP, COULD BE DRUG-GENE
RELATIONSHIP, RESEARCH HAS BEEN
USING IT BUT ALSO MENTIONED 600
IS NOT BIG ENOUGH.
WE WANT MORE.
SO TO ADDRESS THAT FROM OUR
COMMUNITY WE ADDED 400 SO NOW WE
MAKE A TOTAL 1000 AND NOW WE
THINK THIS IS REALEE GOOD AND
THE REASON IS THAT WE SEE THAT
AT LEAST IN OUR OWN BENCHMARKING
FOR A LOT OF THE NUMBERS, IT HAS
STARTED TO BECOME SATURATED.
IN OTHER WORDS, THE MORE DATA
YOU PUT INTO THAT IT MAY NOT BE
ACTUALLY HELPING YOU TO
DRAMATICALLY IMPROVE THE
PERFORMANCE ANY URGT IFER.
OKAY, SO NOW THE ACCURACY OF
BIORED, IS NOW MAKING ME MORE
COMFORTABLE TO ADD THAT TO THE
RESEARCH WEBSITE.
SO EVEN THAT AS YOU NOTICED OR I
SHOULD MENTION THATY WOO ARE
ONLY TEXT MINING RELATIONS IN
THE ABSTRACT BECAUSE WE TRIED
THE FULL TEXT, IT'S MUCH MORE
COMPLICATED AND ONCE WE DO THAT,
THE RESULTS ARE GOING BACKWARDS
WHICH MAKES ME, EVEN THOUGH
OTHER ENTITIES WE DO ABSTRACT
FOR FULL TEXT BUT FOR RELATIONS
WE ARE ONLY FOCUSING ON THE
ABSTRACT LEVEL.
THERE ARE ALWAYS OTHER THINGS WE
CAN TAG, GENE ONTOLOGY TERMS,
CELL PHENOTYPES, AND PATHWAYS,
WE'RE NOT--YOU KNOW, WITH THE
RESOURCES WE HAVE, WE'RE
GRADUALLY EXPANDING IN THESE
AREAS.
WE WELCOME INPUT FROM OUR
COMMUNITY WHICH 1S YOU THINK ARE
THE MOST IMPORTANT TO ADD, YOU
KNOW FOR EXAMPLE, PEOPLE HAVE
REACHED OUT TO US AND SAY, YOU
KNOW, PENE O TYPES, IS REALLY
IMPORTANT EMPLOY SO WE HAD A
PRONL ECTOMYOSIN TAGGING HUMAN
PHENOTYPES USING HUMAN PHENOTYPE
ONTOLOGY SO WE HAD SUCCESS
THERE, , SO THIS IS A HUMAN
DRICH COMMUNITY PROJECT WE WANT
TO ADD MORE BUT DON'T EXPECT US
TO DO EVERYTHING BECAUSE THAT
WILL BE AT THE COST OF THE
QUALITY UNFORTUNATELY, SO THAT'S
BASICALLY MY ANSWER TO YOU AT
THIS POINT.
>> EXCELLENT.
ANSWER.
THANK YOU.
>> WE HAVE 2 MORE MINUTES LEFT
BEFORE THE PRESENTATION FROM
YOUR TRAINEES DR. LU.
>> JUST 1 LITTLE QUESTION, I
THINK IT'S IMPORTANT FOR MY
EVALUATION IT'S ABOUT HOW YOU
ADDRESSING DIVERSITY IN TERMS OF
YOU KNOW THE ACTUAL RESEARCH?
I NOTICE YOU HAVE A LITTLE
STRONG DIVERSITY STATEMENT IN
YOUR PRESENTATION, IT WAS CLEAR
THAT YOU HAVE PEOPLE AND
RESOURCES THAT IN EMERGING ITS,
YOU DID MENTION VERY BRIEFLY
RACIAL BIAS AND I WOULD LIKE TO
HEAR YOUR ANSWER TO THAT
QUESTION.
THANK YOU.
NTHANK YOU FOR THAT QUESTION
MUCH SO AS I MENTIONED IN THE
WRITTEN REPORT, WE TAKE THIS
VERY SERIOUSLY AND WE REALLY
BELIEVE THE IMPORTANCE OF
DIVERSITY, IN BOTH RESEARCH AND
DAY-TO-DAY WORK, SO, I DON'T
KNOW HOW MUCH I CAN ADD ON TO
THIS BUT VERY QUICKLY TO
SUMMARIZE IN TERMS OF WE HAVE
DONE A LOT OF THINGS INCLUDING
MAKING SURE THAT OUR GROUP IS
VERY DIVERSE, EVERYBODY HAS TO
SAY, AND WE ALSO WORKING ON
VARIOUS INVOLVEMENT IN TERMS OF
CERTAINLY--CERTAINLY QUITY,
COMMITTEE AT THE NIH LEVEL AND
OUTSIDE INTERNATIONAL
PROFESSIONAL SOCIETIES.
WE HAVE, YOU KNOW IN SOME
INITTIAIVE IT IN TERMS OF GENDER
BALANCE, IN TERMS OF UNDERSERVED
COMMUNITY IN MY GROUP, AND LAST
BUT NOT LEAST, YOU SLEEP APNEA
AND OBESITYY THAT WE PUT THIS
INTO RESEARCH PROBLEMS AND WE
THINK THIS WILL ALSO HELP US TO
ADVANCE THE KNOWLEDGE ON MANY OF
THE THINGS THAT WE WORK IN
ESPECIALLY IN TODAY'S AI SPACE.
HAPPY TO TALK MORE IF WE NEEDED
DURING THE CLOSED SESSION.
>> GREAT.
THANK YOU VERY MUCH DR. LU, SO
WE NOW HAVE A CLOSED POSTER
SESSION WITH POST DOCTORAL
FELLOWS OF DR. LU?
>> [INDISCERNIBLE]
>> SORRY IN.
>> I THINK MARCILLA, WAS
WAITING.
>> I THINK THIS IS PUBLICLY
BEING BROADCAST, SO I TEXTED
THAT GRACILLIA CAN ASK THAT
DURING OUR 15 MINUTE CLOSED
SESSION WITH DR. LU.
ALL RIGHT, SO AGAIN WE ARE NOW
STARTING A CLOSED SESSION, THAT
POSTER SESSION WILL BE WITH POST
DOCTORAL STUDENTS AND THEN WE
WILL HAVE A CLOSED SESSION WITH
DR. LU AND THEN A BREAK THE OPEN
SESSION WITH DR. DEMNER-FUSHMAN,
WILL START AT 2:00 P.M. EASTERN
SO THE BOARD AND TRAINEES SHOULD
JOIN THE CLOSED SESSION BREAK
OUT ROOM NOW.
>> ALL RIGHT, WELCOME, EVERYBODY
TO OUR SECOND PUBLIC SESSION
WHERE DR. DINA DEMNER-FUSHMAN
WILL PRESENT ON HER RESEARCH.
NGOOD AFTERNOON, I'M PRESENTING
MY INTRAMURAL RESEARCH PROJECT
THAT I HOPE IS PROVIDING TIMELY
AND ACCESS TO RELIABLE HEGHT
RELATED INFORMATION FOR DECISION
SUPPORT AND EICATION.
AND THE OVERALL RESEARCH
OBJECTIVE FOR THIS PROJECT IS TO
ENABLE EASY ACCESS TO RELIABLE
HEALTH RELATED INFORMATION AND
AT THE SAME TIME ADVANCE THE
FOUNDATIONAL NLM RESEARCH
APPROACHES AND THESE OBLIGATIONS
YECTIVES TIE WITH THE HHS
STRATEGIC GOALS ON PROTECTING
HEALTHCARE AND ALSO SUSTAINING
AND ADVANCING SCIENCES.
SO, I WILL START WITH MODERATING
MY APPROACH THAT THAT HIGHLY
AVAILABLE INFORMATION DPLEVERY
AS A QUESTION ON STREAM PROCESS
AND THEN VERY BRIEFLY TOUCH ON
THE TRADITIONAL QUESTION AND
ANSWERING THAT WE ARE DOING IN
SOME YEARS BEFORE THE LARGE
LANGUAGE MODEL REVOLUTION AND
THEN I WILL TALK MORE ABOUT OUR
CURRENT RESEARCH WHICH IS
BASICALLY LEVERAGING THE
OPPORTUNITIES AND ADDRESSING THE
CHALLENGES, POSED BY THE LLMs.
SO 1 QUESTION ANSWERING, IT IS A
NATURAL WAY FOR HUMANS TO LEARN
AND SPECIFICALLY FOR OUR TARGET
AUDIENCES WHICH IS ANYONE WHO IS
WORKING ON HEALTH PROBLEMS OR
HAVING HEALTH ISSUES OR IS EAR
HAS INFORMATION NEEDS RELATED TO
THEIR HEALTH, SO THE RESEARCHERS
HERE SAY THAT SCIENCE BEGINS BY
ASKING QUESTIONS AND OUR JOB IS
TO FIND THEM THE ANSWERS THAT
ALREADY EXIST AND MAYBE SHOW THE
GAPS THAT ARE NOT COVERED YET.
SOMEWHERE AT THE TURN OF THE
CENTURY, PHYSICIANS HAD REALLY
MOVED STUDIES ON SHOWING HOW
MANY CLENICAL QUESTIONS OCCUR AT
BEDSIDE BUT THEY'RE UNANSWERED,
BECAUSE PHYSICIANS OFTEN KNOW
THAT THE ANSWERS MIGHT NOT BE
THERE, DON'T HAVE TIME AND SO
ON.
SO THAT WAS 23 YEARS AGO, AND IS
IT STILL THE CASE AND RECENT
TESTAMENT THAT IT IS MORE THE
CASE NOW BECAUSE THERE IS A LOT
MORE INFORMATION COMING
PHYSICIANS WAY, THERE'S ALSO A
LOT INFORMATION IN THE EHR,
THERE WAS NOT A PROBLEM IN THE
20th CENTURY SO NOW IT'S EVEN
MORE IMPORTANT TO THE GET THE
INFORMATION THOSE THAT NEED IT
AND FINALLY THE CONSUMERS WHICH
ARE A VERY MAJOR PART OF OUR
RESEARCH, ARE ALWAYS FINDING THE
ANSWERS TO THOSE QUESTIONS, BUT
UNFORTUNATELY, THEY HAVE NO
MEANS TO JUDGE IF THE ANSWER IS
TRUTHFUL AND IF THE ANSWER IS
CORRECT SO THAT'S WHY WE ARE
FOCUSING ON THE
QUESTION-ANSWERING APPROACHES
AND THE DPOALS FOR
QUESTION-ANSWERING SYSTEMS WERE
FORMULATED AGAIN OVER 20 YEARS
AGO AND JUST IN 4 BULLETS, IT'S
TO ALLOW A USER TO ANSWER
QUESTIONS IN EVERY DAY LANGUAGE,
AND ANSWER QUICKLY, ANSWER
SUCCINCTLY, AND PROVIDE
SUFFICIENT CONTEXT TO VALIDATE
THE ANSWER, WE HAVE TO
UNDERSTAND THE WHAT THE QUESTION
IS ABOUT AND FIND THEM IN
REALTIME AND AGGREGATE THEM AND
PROVIDE ENOUGH EVIDENCE TO
SUPPORT THE ANSWER.
AND THIS IS WHAT WE HAVE DEMON
TRAITED TO THE PREVIOUS BOARD 4
YEARS AGO.
WE WERE JUST FINISHED BUILDING A
PROTOTYPE QUESTION ANSWERING
SYSTEM THAT WAS USING A
TRADITIONAL INFORMATION
RESTREEFAL BASED APPROACH SO
HERE YOU SEE THAT THE QUESTION
IS KIND OF UNDERSTOOD, THE MAIN
POINTS OF THE QUESTION, WHERE
THE QUESTION IS ABOUT, AND WHAT
THE USER WILL DO WITH THAT
QUESTION, WILL THE ANSWER TO
THAT QUESTION REMAIN, THESE MAIN
COMPONENTS ARE USED TO SEARCH
THE DATABASES OF 2 TYPES, BUT
THEN 1 OF SIMILAR QUESTIONS THAT
WERE ALREADY ASKED AND ANSWERED
BY THE NIH INSTITUTES SO HERE IF
I FIND A QUESTION, IT ALREADY
HAS AN EXCELLENT ANSWER PROVIDED
BY THE SPECIALIST BUT ALSO
FINDING THE LITERATURE AND
ENEXTRACTING THE ANSWER FROM THE
LITERATURE AND SUMMARIZING AND
AGGREGATING THE ANSWERS.
SO SOMETIME RIGHT AFTER WE
PRESENTED THE SYSTEM, WE STARTED
SEEING THESE INCREDIBLE SUCCESS
STORIES FROM THE LARGE LANGUAGE
MODELS THAT WERE DOING EXACTLY
WHAT WE ARE TRYING TO DO, THEY
WERE ANSWERING QUESTIONS, SO IF
WE LOOK CLOSER, UNDER THE HOOD
OF THESE QUESTION-ANSWERING
SYSTEMS, THIS IS WHAT THE GOOGLE
TEAM DID WITH THEIR OWN SYSTEM
AND THEY ASSEMBLED THESE FIRST
MEDICAL QUESTION ANSWERING
BENCHMARK AND THEY ACTUALLY
REACHED OUT TO US AND ASKED IF
THIS LIVE QA AND MEDICAL QA
QUESTIONS THAT WE PROVIDED TO
THE NATIONAL INSTITUTE OF
STANDARD AND TECHNOLOGY FOR
THEIR TEXT RETRIEVAL CONFERENCE
EVALUATIONS.
IF THESE ARE REAL LIFE
QUESTIONS, AND WE SAID THAT YES,
WE TOOK THOSE FROM OUR CUSTOMERS
PURPOSES AND MED LINE PLUS LOGS
SO THESE ARE TRULY QUESTIONS
ASKED BY THE PUBLIC AND THEY
INCLUDED THESE IN THE BENCHMARK
AND THE DIFFERENCE BETWEEN THOSE
SUCCESS STORIES ON THE MEDICAL
EXAMS, AND THESE QUESTIONS WERE
THAT ASKED BY THE PUBLIC IS THAT
MEDICAL EXAM QUESTIONS IN MANY
CASES HERE ARE MULTIPLE CHOICE,
SO YOU JUST PICK 1 ANSWER THAT
IS CORRECT, WHEREAS THESE ARE
OPEN ENDED QUESTIONS, AND THEY
DO NOT HAVE EXISTING ANSWERS SO
TO PROVIDE THE GOLD STANDARD
ANSWERS, WE ACTUALLY GENERATE
THE ANSWERS USING SUBJECT MATTER
EXPERTS, AND THEN EVALUATING
THESE OPEN ENDED QUESTIONS IS
ALSO VERY HARD SO THE GOOGLE
TEAM HAD 9 CLINICIANS FROM
DIFFERENT COUNTRIES THAT WERE
ASSESSING THE ANSWERS OF
INTERWEAVING THE ANSWERS OF
THE--THAT WERE PRODUCED BY THE
HUMANS BUT ALSO WERE PRODUCED BY
THE LARGE LANGUAGE MODELS AND
WHEN THE MICROSOFT TEAM WAS
EVALUATING THEIR MEDICAL
QUESTION ANSWERING SYSTEM ON A
BENCHMARK, THEY ACTUALLY HAD TO
LEAVE OUT THESE QUESTIONS
BECAUSE IT'S JUST VERY EXPENSIVE
TO DO THESE KINDS OF
EVALUATIONS.
SO SWITCHING TO THESE NEW
TIEWPTS AND CHALLENGES BY THESE
LLMs, FIRST WE SEE THAT THEY
CAN DIRECTLY GENERATE THE ANSWER
BUT WE ALSO SEE THAT THE ANSWER
NEEDS SOME SUPPORT SO I WILL
TALK ABOUT SUPPORTING THE ANSWER
AND GROUNDING IT IN EVIDENT, WE
ALSO SEE THAT THERE IS A
MEANINGFUL PLAIN LANGUAGE AND WE
WILL TALK ABOUT THAT LATER AND
MULTIMODAL QUESTION ANSWERING.
SO HERE'S AN EXAMPLE OF WHAT WE
DO WHEN WE START A NEW TASK AND
NO DATA EXISTS FOR THAT, WE TAKE
THE QUESTION THAT WAS ASKED BY
THE CONSUMER AND A MEDICAL
STUDENT HAS GENERATED THIS GOLD
STANDARD AS WELL AS LOOKING AT
THE LATEST EVIDENCE.
SO NOW WE ASK THAT SAME QUESTION
ABOUT THE TOTAL HIP REPLACEMENT
WHICH THE MEDICAL STUDENT
INTERPRETED AS DO I NEED PAIN
MEDICATION TO IMPROVE MY
RECOVERY?
WE ASKED THE LARGE LANGUAGE
MODEL AND THE MODEL PROVIDES A
VERY LONG ANSWER.
INTERPRETING THE QUESTION AS,
HOW MUCH DOES PAIN MEDICATION
DECREASE--IMPROVE MY PAIN?
HOW MUCH DOES IT AFFECT THE PAIN
IN ASK WE ALSO SEE THESE
PECULIAR THINGS THAT EVERYONE IS
NOW FAMILIAR WITH, VERY SPECIFIC
SYSTEMATIC REVIEW OF 19 STUDIES
THAT IS NOT REFERENCED, AND A
REFERENCE TO A PAPER IN A
RELEVANT JOURNAL BUT AGAIN,
THERE IS NO WAY TO FIND THAT
PAPER GIVEN THESE ANSWER.
SO WE LOOK AT MANY OF THESE
ANSWERS AND IN GENERAL, WE FIND
THAT THE STRENGTHS OF THESE NEW
MODELS ARE PRODUCING THESE
EXTREMELY GRAMMATICAL AND FLUENT
ANSWERS, SEEMS TO BE
COMPREHENSIVE AND IN MANY
ANSWERS, NOT IN THIS 1, WE HAVE
SEEN THAT AT THE END, THE MODEL
RECOMMENDS SEEING THE HEALTHCARE
PROVIDER.
WE SEE THAT SOMETIMES THE MODEL
DOES NOT UNDERSTAND THE QUESTION
FULLY.
WE ALSO SEE A LOT OF HEDGING IN
SOME DIFFERENT ANSWERS.
THE ANSWERS WERE VERY LONG AND
OF COURSE WE HAVE SEEN THAT IT
NEEDS FACT CHECKING AND IN THIS
CASE SPECIFICALLY, WE ASK THE
MODEL TO PRODUCE THE ANSWER IN
PLAIN LANGUAGE BUT, THE LANGUAGE
WAS MORE SUBJECT MATTER EXPERT
ORIENTED.
SO CAN WE IMPROVE THAT AND ALSO
GOING BACK TO THAT IR-BASED
QUESTION ANSWERING BUT NOW IT'S
CALLED RETRIEVAL OF
[INDISCERNIBLE] GENERATION AND
WHAT HAPPENS HERE IS THE
QUESTION GOES TO SOME KNOWLEDGE
BASE IN OUR CASE PUB MED, WE
RETRIEVE RELEVANT DOCUMS AND WE
THEN PROVIDE THE MODEL WITH THE
QUESTION AND THE RELEVANT DOCK
YOU WANTS AND HERE WE ASK THE
MODEL TO WRITE AND ANSWER THAT
IS BASED ON THESE DOCUMENTS, AND
THEN, INDICATE IN THE ANSWER
WHICH DOCUMENTS EXACTLY
CONTRIBUTED TO THAT ANSWER.
SO IF WE LOOK CLOSER NOW, WE NOW
CAN EVALUATE WHETHER THE MODEL
USED THE RIGHT DOCUMENT TO
SUPPORT ITS STATEMENT AND ALSO
WE CAN EVALUATE IF THE DOCUMENT
INDEED SUPPORTS THE STATEMENT.
SO HERE YOU SEE THAT IF WE READ
THE MODELS ANSWER FIRST, WE
MIGHT THINK THAT IT IS ACTUALLY
SUPPORTED BY THE DOCUM, SO THE
DOCUMENT DID SAY THAT A PAIN
DECREASES CONSTANTLY IN THE
FIRST WEEK AFTER SURGERY, BUT
THEN THE MODEL ALSO--THE GOLD
STANDARD ANSWER ALSO SAYS THAT
WHEN YOU START PHYSICAL THERAPY,
THE PAIN LEVELS GO UP WHEREAS
THE MODEL SAYS, THAT THE PAIN IS
DECREASING BECAUSE OF THE
PHYSICAL THERAPY.
SO BASICALLY WHEN YOU START THE
PAIN WILL GO DOWN EVEN MORE
WHICH OF COURSE MISLEAD XG COULD
POTENTIALLY BE HARMFUL BECAUSE
NOW THE PATIENT MAY THINK THAT
SOMETHING IS WRONG WITH PHYSICAL
THERAPY BECAUSE THE PAIN LEVEL
THAT WAS GOING UP.
SO LOOKING AT THESE POTENTIAL
HARMS THERE WAS SPECIFICALLY A
PART OF THAT GOOGLE EVALUATION,
THE MODELS DO PROVIDE HARMFUL
AND BY SOMEWHAT MORE OFTEN D& D
CLINICIANS AND 1 EXAMPLE OF THE
HARM THAT IS NOT ONLY TO THAT
INDIVIDUAL PATIENT BUT ALSO TO
THE ORGANIZATION THAT PROVIDED
THE ANSWER, IS MAY HAVE LOST
HERE, THE BRITISH EATING
DISSEREDDER HELP LINE ANNOUNCED
THAT THEY ARE REPLACING THEIR
HELP LINE WITH THE CHAT BOT AND
ON JUNE 1st OF THAT SAME YEAR,
SO BASICALLY THE SAME DAY THE
ANSWERS PROVIDED BY THE CHAT BOT
WAS SO HARMFUL THAT THE SERVICE
WENT DOWN AND I CHECKED RECENTLY
THE IS STILL NOT RESTORED SO
THAT WAS LIKE A PERMANENT DAMAGE
TO THAT SERVICE.
SO WE DEFINITELY NEED TO
MITIGATE THESE ARMS AND 1 WAY IS
TO SOMEHOW LEVERAGE THE DOMAIN
KNOWLEDGE THAT WE NOW HAVE AND
WE STARTED WITH LEVERAGING THE
MLS AND WITH 2 QUESTIONS, 1, HOW
MUCH OF THE MLS KNOWLEDGE IS
ALREADY ENCODED IN THE MODEL AND
IS THE MODEL CAPABLE OF
RETRIEVING THAT KNOWLEDGE IF
IT'S THERE, AND THE SECOND PART,
IF THE KNOWLEDGE IS NOT THERE,
OR IS ONLY PARTIALLY THERE, CAN
WE DIRECTLY PROVIDE THAT
KNOWLEDGE TO THE MODEL TO
IMPROVE PERFORMANCE AND OUR USE
CASE IS A DIFFERENTIAL SUPPORT
OF THE DIFFERENTIAL █DIAGNOSIS
WHICH IS 1 OF THE OLDEST
CLINICAL INFORMATIC TAVENS BUT
IS STILL VERY RELEVANT AND OUR
COLLABORATOR [INDISCERNIBLE] IS
WORKING ON THAT PROBLEM.
SO OUR COMBINED EFFORTS WERE
FIRST IN PROBING HOW MUCH OF THE
UMLS KNOWLEDGE IS ALREADY
CONTAINED IN THE LANGUAGE MODEL
EMPLOY SO TO THAT END WE ASKED
THE MODEL TO COMPLETE THE UMLS
PATH.
SO WE GIVE THE MODEL A CONCEPT
FROM THE UMLS, SUCH AS
ANAPHYLAXIS AND THEN THE EDGES
THAT ARE RELEVANT TO THE
DIFFERENTIAL DIAGNOSIS SUPPORT,
AND WE ASK THE MODEL TO COMPLETE
THE PATH ALONG THIS EDGE, WITH A
CONCEPT, WITH ANOTHER CONCEPT,
AND THE BEST MODEL FOR THAT TASK
WAS CHAT GPT BUT EVEN THAT MODEL
FAILED TO COMPLETE 9 MONTHS
ALTOGETHER AND FOR 22 OF 50
PATHS IT CAN ONLY PROVIDE LESS
THAN A THIRD OF THE RELEVANT
CONCEPTS.
SO DEFINITELY, EITHER BEING
MODELED DOES NOT HAVE ALL THE
KNOWLEDGE OR IT CANNOT RETRIEVE
IT IN THIS FORMAT.
SO FOR THE SECOND QUESTION, IF
WE END THAT KNOWLEDGE EXPLICITLY
WITH THE DATA THAT WE PROVIDE TO
THE MODEL ABOUT THE PATIENTS
SYMPTOMS, AND IF WE WILL LET
THEM IMPROVE THE RESULTS AND YOU
CAN SEE HERE THAT IN THE PAST TO
THE PROMPT AND THE DATA GIVEN TO
THE MODELS, SOMEWHAT IMPROVES
THE RESULTS, BUT THE RESULTS ARE
STILL NOT GREAT, IN THE
PREDICTIONS OF THE DIFFERENTIAL
DIAGNOSIS, AND YOU SEE HERE,
THAT WE USE THESE TRADITIONAL
LXICAL AND METRICS AND WE WILL
TALK LATER HOW WE ARE NOW
EXPERIENCING A BIT OF A CRISIS
ON USING THESE TRADITIONAL
METRICS THAT WE HAVE USED FOR
YEARS IN THESE EVALUATIONS.
SO SWITCHING TO THE PLAIN
LANGUAGE ADAPTATIONS, THE
MITIGATION FOR THAT PROJECT %S 2
FOLD, 1, EVEN BEFORE THE
PANDEMIC WE HAD THE PATIENTS WHO
ALREADY READ EVERYTHING THAT'S
AVAILABLE IN CONSUMER FRIENDLY
RESOURCES, ASKING, WHAT IS THE
LATEST RESEARCH ON MY CONDITION.
AND WE HAVE SEEN DURING A
PANDEMIC THAT THE CONSUMER
FRIENDLY RESOURCES WERE BEHIND
AND THEY COULD NONAPOPTOTIC THE
PROVIDE ANY RELEVANT INFORMATION
TO THE PATIENT SO THERE IS
CLEARLY A NEED TO PROVIDE PLAIN
LANGUAGE ANSWERS IN REALTIME,
THERE ARE 2 ONGOING EFFORTS
RIGHT NOW WITHIN THE COMMUNITY,
1 IS DIRECTLY SUMMARIZE THE
RESEARCH EVIDENT INTO THE
LANGUAGE THAT ANSWER FOR A
SPECIFIC QUESTION, WE CHOSE
ANOTHER WAY BECAUSE THIS WAY, IF
WE ARE FAITHFULLY TRANSLATING
THE SCIENTIFIC ABSTRACT INTO A
PLAIN LANGUAGE, THEN WE CAN USE
IT FOR DIFFERENT PURPOSES AND
ANSWER DIFFERENT QUESTIONS
POTENTIALLY.
SO THIS IS AN ONGOING EFFORT
WITHIN THE TRUCK EVALUATIONS,
IT'S IN THE SECOND YEAR AND
BRIAN [INDISCERNIBLE] WHO IS
PRESENTING IN THE PRIVATE
SESSION WILL TALK MORE ABOUT
THIS EFFORT.
AS WITH ALL OF OUR NEW TASKS,
THERE WERE NO CORRECTIONS THAT
HAD THESE SYMPTOMS BY SYMPTOMS
FOR PUB MED ABNORMALITIES TRACKS
SO HERE I'M SHOWING HOW WE
CREATED THE COLLECTION.
WE STARTED WITH THE POPULAR MED
LINE PLUS SEARCHES; WE SEARCHED
PUB MED FOR THE RELEVANT
ABSTRACTS AND THEN FOR EACH 1 OF
THOSE ABSTRACTS, WE TRANSLATED
THESE SENTENCES INTO PLAIN
LANGUAGE.
USING THIS COLLECTION, WE THEN
HAD LAST YEAR, THIS COMMUNITY
EVALUATION AND I SHOW HERE THE
RESULTS OF THE LATEST NARRATION
WE HAVE DONE ON COLLECTION AND
THE TOP YOU SEE, THE RESULTS
SUBMITTED BY THE TEAMS THAT
PARTICIPATED IN THE CHALLENGE
AND IN THE BOTTOM, YOU SEE THE
NLM BASE LINE THAT WAS PRODUCED
BY BRIAN USING CHAT GPT AND I
BELIEVE HE IS PROVIDING THE
DETAILS ON HOW HE HAS DONE IT
AND THEN AT THE BOTTOM, YOU SEE
THE IMITATORS THAT WE USED FOR
HUMAN ADAPTATIONS, HUMAN
TRANSLATIONS OF THESE SENTENCES
AND YOU CAN SEE HERE THE MACHINE
IS DEFINITELY MORE FLUENT THAN
PEOPLE.
SO FLUENCY PROBLEM IS DEFINITELY
SOLVED AND 1 OF THE PROBLEMS
WITH THESE TRADITIONAL METRICS
IS THAT IT IS MEASURING FLUENCY.
NOW, WHEN IT COMES TO
FAITHFULNESS AND ACCURACY OF THE
ANSWERS, THEN AGAIN, SOME OF THE
APPROACH THAT BRIAN AS
DEVELOPED, THE NUMBERS LOOK VERY
GOOD BUT AGAIN, THEY ARE HIDING
SOMETHING THAT I SHOWED EARLIER
SO 1 OF THOSE ANSWERS COULD
POTENTIALLY BE HARMFUL.
NOW SWITCHING TO THE MULTIMODAL
QUESTION ANSWERING, THERE ARE
MANY CASES WHERE IT'S BETTER TO
SEE SOMETHING THAN TO READ A
LONG TEXTURAL DESCRIPTION, FOR
EXAMPLE, IF SOMEONENIES TO GIVE
FIRST AID, OR A
PATIENT'S--PATIENTS ARE
INTRUCTED TO DO
SELF-EXAMINATIONS ORIFICEICAL
THERAPY, SO THE GOAL IS TO SHOW
THEM HOW TO DO THESE THINGS AND
TO ACCOMPLISH THAT, WE FIRST
NEED TO FIND RELEVANT VIDEOS
THAT ACTUALLY CONTAIN INTRUKSS
AND THEN WITHIN THOSE VIDEOS WE
NEED TO FIND THE SNIPITS THAT
SHOW HOW TO DO SOMETHING, AND
WITHIN THESE SNIPITS, THE
PROCEDURES SOMETIMES COMBINE OUT
OF MANY STEPS.
AND OUR GOAL, FOR EXAMPLE, IF
SOMEONE SAYS HOW DO I STAND UP,
WALK AND SIT DOWN USING
CRUTCHES?
AND FOR EXAMPLE, THEY JUST WANT
TO KNOW HOW TO SIT DOWN, THEN WE
WANT TO IESHES DENTIFY THAT STEP
IN THE VIDEO AND WILL LABEL IT
AS SUCH.
SO THIS IS WHAT WE ARE WORKING
ON RIGHT NOW, AND THIS PAPER
THAT I'M PRESENTING WILL BE
PRESENTED NEXT MONTH AT THE
COMPUTATIONAL LINGUISTICS
CONFERENCE.
AS WITH ALL OTHER TASKS, THE
DATA SAID THAT WE CREATED
MANUALLY IN THE FIRST YEAR WAS
FAIRLY SMALL, SO WE LOOKED INTO
CREATING DATA SET FOR TRAINING
AUTOMATICALLY.
AND WE STARTED WITH THESE FILES
THAT WE CLASSIFIED AS MEDICAL
INTRUKSES AND WE FOUND MORE OF
THESE IN THIS YouTUBE
COLLECTIONS THEN WE SEGMENTED
THE INSTRUCTIONAL VIDEOS INTO
THESE TOPICS FOR EXAMPLE,
WALKING WITH CRUTCHES AND THEN,
EACH SEGMENT WAS REPRESENTED
USING TRANSFORMER ENCODER.
AND WE WILL--THIS EVIDENT WAS
LABELED AS BEGINNING FOR INSIDE
OR OUTSIDE OF THE ANSWER AND
THEN ON TOP OF THESE SEGMENTS
NOW THAT ARE THE ANSWER
SEGMENTS, THE QUESTION IS
GENERATED THAT COULD BE
POTENTIALLY ANSWERED BY THAT
SEGMENT.
SO TO [INDISCERNIBLE] THAT
COLLECTION HAS IMPROVED OUR
PERFORMANCE ON THAT VIDEO,
ANSWERED [INDISCERNIBLE] WE
DEVELOPED AN APPROACH THAT
LEVERAGES THE VIDEO FEATURES AND
THE CAPTIONS FEATURES AND THE
TIME STAMPS THIS IS CALLED THE
CYCLE CONSISTENT ANSWER
LOCALIZATION AND THE DIFFERENT
ANALOGIES, THE WAY IT IS, THE
GOLD STANDARD QUESTION SYSTEM
USED TO GENERATE AN ANSWER, THAT
GENERATED ANSWER IS FED INTO A
MODEL THAT INVOLVES ANOTHER
QUESTION WITH THAT WORRY AND
THAT OTHER MODEL IS GENERATING
THE ANSWER TO THE GENERATED
QUESTION AND DURING TRAINING ALL
OF THESE STEPS NEED TO BE
COORDINATED.
SO USING THIS MODEL AND USING
THESE ADDITIONAL DATA, WE CAN
SEE THAT ADDING THESE NEW DAILY
BASIS THEA, THE COLLECTION WAS
BUILT RIGHT, IT IMPROVES
PERFORMANCE SIGNIFICANTLY AND
THE METRIC HERE IS THE
INTERSECTION OF THE ANSWER
GENERATED BY THE MODEL WITH THE
GOLD STANDARD ANSWER, AND
THERE'S DIFFERENT BOUNDARIES ON
HOW MUCH OF INTERSECTION THERE
SHOULD BE SO THAT WE SEE THE
DIFFERENT NUMBERS FOR THESE
DEFINITE BOUNDARIES WITHIN THE
ANSWER AND OF COURSE, THE MODELS
TILL MAKE MISTAKES, AND SOME OF
THESE MISTAKES COULD BE
ADDRESSED WHEN WE ADD THE VISUAL
FEATURES FROM THE VIDEO FRAMES
TO THE TEXT, WE SEE THAT
PERFORMANCE GOES UP.
SO EVALUATION IS NOW A VERY OPEN
QUESTION BECAUSE THESE
TRADITIONAL LEXICAL MET METRICT
WE SEE AND THESE METRICS SHOW
VERY GOOD RESULTS BUT WITHIN
THOSE RESULTS ARE THESE RARE
OCCASIONS, YOU KNOW, 5%, WHICH
COULD POTENTIALLY BE HARMFUL, SO
WE'RE VERY ACTIVELY WORKING WITH
THE NIST GROUP THAT CONDUCTS
THESE TEXT RETRIEVAL CONFERENCE
EVALUATIONS ON DEVELOPING NEW
METRICS AND OF COURSE THAT
ENROLLS A LOT OF MANUAL
JUDGMENTS ON THE WAY SO THAT WE
CAN COMPARE THE DIFFERENT
METRICS TO THE MANUEL JUDGMENTS.
THE INTERFACE I SHOWED YOU AT
THE BEGINNING IS NOW READY TO BE
REPLACED BY THE NEW INTERFACE
FOR THAT PRIVATE TYPE CONSUMER
HEALTH QUESTION ANSWERING
SYSTEM, YOU CAN SEE AT THE TOP
THE ANSWER IS GENERATED BY THE
MODEL AND THEN THE EVIDENCE TO
SUPPORT THE ANSWER, AND THE
VIDEOS THAT WE PULL IN FROM THE
OPEN EYE COLLECTION VIDEOS AND
IMAGES.
NOW TECHNICALLY IT'S FAST AND
READY TO BE PRESENTED BUT I
DON'T BELIEVE THAT WE ARE READY
TO HAND IT OVER RIGHT NOW
BECAUSE WE STILL NEED TO DO LOTS
OF USER EVALUATIONS, IS THAT
SOMETHING--IS THAT ANOTHER WAY
TO PRESENT SOMETHING TO THE USER
SO THAT THEY WILL BE ABLE TO
DETECT DISCREPANCIES AND JUDGE
WHAT'S WRONG WITH THE ANSWER, OR
IF THE ANSWER GOOD AND
SUPPORTED.
AND THAT IS GOING TO THAT FUTURE
WORK WHERE WE NEED TO VERY
[INDISCERNIBLE] COMPLETE AND
STRENGTH OF THE ANSWERS, EXPLORE
QUESTIONS THAT NEED
CLARIFICATION FOR EXAMPLE, THE
QUESTION THAT THE MODEL AND THE
MEDICAL STUDENT INTERPRETED
DIFFERENTLY, WE COULD TURN ARK
ROUND AND ASK THE USER IF THEY
REALLY WANT TO KNOW WHICH PAIN
MEDICATIONS ARE BETTER OR IF
THEY WANT TO KNOW HOW MUCH THEIR
RECOVERY WILL BE INFLUENCED BY
THE--BY PAIN MANAGEMENT AND THE
CONTINUING WORK ON THAT VISUAL
MODEL QUESTION ANSWERING, YOU
WILL ALSO SEE [INDISCERNIBLE]
PRESENTING THE WORK HE STARTED
ON EHR MODELING AND SOMETHING IT
MORE THAT THE CLINICIANS WANT TO
KNOW ABOUT THEIR CARE PROCESS
AND ALSO DEVELOPING RELIABLE AND
AFFORDABLE EVALUATION APPROACHES
AND AGAIN, MOVING CLOSER TO
TRANSFERRING THE TECHNOLOGY TO
OUR OTHER DIVISIONS, FOR
EXAMPLE, IN WHAT I'VE SHOWN
TODAY THAT FINDING THAT DANCE
RETRIEVAL, THAT WE USED FOR
FINDING RELEVANT VIDEOS IS READY
FOR TRANSFER, IT'S HARMLESS, THE
RESULTS ARE MUCH BETTER, BUT OF
COURSE, THE QUESTION ANSWERING
SYSTEM ITSELF IS NOT READY FOR
TRANSFER YET.
SO WITH THAT, OF COURSE, IT'S A
GROUP EFFORT ON THE TRAINEES
WILL BE PRESENTING THEIR WORK
RIGHT AFTER THIS SESSION BRIAN
AND SARVESH, THE CURRENT GROUP
CONTRIBUTES ARE LISTED ON THE
LEFT AND ON THE RIGHT THESE ARE
ALL THE FORMER TRAINEES THAT
CONTRIBUTED OVER THE YEARS TO
THE PROJECT AND A LOT OF WHAT
THEY HAVE KRIEWBED IS TILL BEING
USED FOR EXAMPLE, KATE MASTER
TON HAS DEVELOPED THESE REPORTS
OUT OF THE NETHERLANDS,
[INDISCERNIBLE] TO GET THE
LATEST QUESTIONS AND WE ARE
STILL USING THIS SERVICE
DEVELOPED BY KATE SOME YEARS AGO
AND SO THANK YOU AND I HOPE YOU
HAVE QUESTIONS.
NIF YOU CAN CAN YOU HIGHLIGHT
THE QUESTION FOR SOMEONE OUTSIDE
OF FIELD, WHAT IS
STATE-OF-THE-ART, WHAT IS NEW
AND WHAT IS KIND OF COMMON THING
BECAUSE IT'S VERY HARD FOR ME TO
JUDGE.
>> SO WHAT I'VE SHOWN HERE IS
COMPLETELY NEW, SO, YOU KNOW,
LIKE THIS MAGICAL VIDEO
INSTRUCTIONAL VIDEO QUESTION
ANSWERING, WE STARTED IT 2 YEARS
AGO.
THIS IS A BRAND NEW EFFORT.
THERE ARE NOW--WE HAVE SEEN--SO
THE FIRST YEAR, WE HAD 3 TEAMS
PARTICIPATING, THE SECOND YEAR,
WE HAD I THINK 5 OR 7, NOW WE
SEE THAT 8 TEAMS HAVE SIGNED UP
TO PARTICIPATE THIS YEAR.
THIS RETRIEVAL OF THE
GENERATIONAL ANSWERS IS BRAND
NEW AS WELL.
THE [INDISCERNIBLE] WORK I'VE
SHOWN PROBING THE UMLS
KNOWLEDGE, WE JUST SUBMITTED THE
PAPER TO JBI, SO IT'S UNDER
REVIEW FROM THE JBI, SO
EVERYTHING I'VE SHOWN TODAY
EXCEPT FOR THE VERY FIRST SLIDE
IS THE OLD SYSTEM, EVERYTHING
ELSE IS NEW.
NTHANK YOU.
>> I THINK I WAS NEXT.
SOIME VERY INTERESTED IN THIS
CONCEPT FOR POTENTIAL FOR HARM
THAT YOU MENTIONED, I FOUND IT
QUITE STRIKING FOR EXAMPLE, THE
SLIDE WHERE YOU HAD THE TABLE
AND YOU COULD HAVE 95% ACCURACY
BUT THERE COULD STILL BE 5%
POTENTIAL FOR HARM, SO I WAS
WONDERING, YOU KNOW IF YOU COULD
ELABORATE ON THAT A LITTLE BIT,
TELL US MORE ABOUT WHAT'S MEANT
BY POTENTIAL FOR HARM AND IN
PARTICULAR, HOW IS THAT DECIDED?
WHAT'S THE THRESHOLD THERE?
THAT'S A VERY STICKY QUESTION,
SO GOOGLE WAS BRAVE ENOUGH TO
SAY, WE WILL EVALUATE THAT IT BE
BRING HARM.
NO 1 ELSE WANTS TO TOUCH IT SO
IN OUR PLAIN LANGUAGE
ADAPTATIONS, WE'RE JUST STOPPING
AT THIS IS EXACTLY WHAT THE
ORIGINAL SAID BECAUSE YOU KNOW
THESE TRAINED CLINICIANS THAT
WE'RE DOING THESE
MULTI[INDISCERNIBLE] JUDGMENTS
WITH THE [INDISCERNIBLE]
AGREEMENT, WE ARE ABLE TO JUDGE
THAT THAT MIGHT BE POTENTIALLY
HARMFUL TO THE PATIENT, RIGHT IN
SO AS I SAID, WITH THIS ANSWER,
HOW HARMFUL IS IT IF I EXPECT
THAT MY PAIN WILL GO DOWN BUT IT
WILL GO UP INSTEAD, RIGHT?
SO IT MIGHT BE FOR DIFFERENT
PEOPLE, THAT MIGHT BE DEFINITE
RANGE OF HARMFULNESS, SO THIS IS
A VERY SUBJECTIVE WITH THE
EXCEPTION OF IF YOU KNOW
RECOMMENDS CRINGING BLEACH
ONLINE AND THEN PEOPLE GO AND
DRINK BLEACH, THAT'S REAL HARM
BUT THERE'S ALSO THESE POTENTIAL
HARM THAT IS KIND OF LIKE A WIDE
RANGE OF HARMFUL CONSEQUENCES,
AND YOU KNOW THAT HARM TO THE
COMPANY THAT PUT UP THE CHAT BOT
AND THIS IS WHAT WE SEE
UNFORTUNATELY IN THE COMMUNITY
THAT SORT OF EAGERNESS TO RUSH
FORWARD AND JUST USE THIS LARGE
LANGUAGE MODELS BECAUSE THEY'RE
SO WONDERFUL BUT MAN, THE WHOLE
SERVICE GOES DOWN THE NEXT DAY.
>> THANK YOU EMPLOY.
>> I CAN JUMP IN HERE, NICE
PRESENTATION ON YOUR WORK.
I WAS JUST WONDERING FROM THE
QUESTIONNAIRE PERSPECTIVE, ARE
YOU TAKING INTO ACCOUNT
INFORMATION ON THE LEVEL OF
EXPERTISE MAYBE BY LOOKING AT
THE TYPES OF QUESTIONS THAT ARE
ASKED SO FOR INSTANCE ISSUES CAN
YOU HAVE VERY UNINFORMED PATIENT
ANDS VERY INFORMED PATIENT IN
MAYBE MAKING YOUR DECISIONS, YOU
KNOW LOOKING AT THE HARMFULNESS
AND INFORMED PATIENT MAY BE ABLE
TO HANDLE A LITTLE BIT MORE
INFORMATION THAN AN UNINFORMED?
>> YEAH, SO FOR THE QUESTIONS
THAT ARE EXTERIOR FSM POSTED TO
FORUMS OR SENT TO CUSTOMER
SERVICES, IT IS SOMEWHAT EASIER
TO FIGURE OUT THE LEVEL OF
EDUCATION FOR THESE VERY SHORT
QUESTIONS.
SOMETIMES IT'S HARDER TO, YOU
KNOW IF PEOPLE ARE USING 3-5
WORDS TO ASK SOMETHING, YOU
COULD STILL DISTINGUISH SOME
YEARS AGO, KIRK ROBERTS A
TRAINEE, HE PUBLISHED QUESTIONS
HE WAS ABLE TO DETERMINE THE
REGISTER OF THE PERSON WHO WAS
ASKING THE QUESTION.
BUT OF COURSE WE DON'T TRACK
OUR--UML, DOES NOT TRACK THE
KNOWLEDGE, ABOUT WHO THE USER IS
AND WHAT THEY HAVE SEEN BEFORE,
SO BASICALLY EVERYTHING WE INFER
ABOUT THE USER IS FROM THE
QUESTION THAT THEY'VE ASKED.
>> I DON'T KNOW WHO WAS FIRST
BUT I WAS FIRST?
THANKS.
SO DINA, THANK YOU, GREAT
PRESENTATION AND GREAT WORK, AS
ALWAYS, NOT EXACTLY MY
[INDISCERNIBLE] QUESTION
ANSWERING FOR A LONG TIME BUT 1
THING I WAS THINKING IS I
NOTICED THAT YOU'RE USING LLAMA
270 B AS A WAY TO COMPARE AND
THE PERFORMANCE PRETTY GOOD SO
YOUR OPINION ON THE USE OF THIS
FOUNDATION MODELS THAT ARE OPEN
SOURCE VERSUS LIKE THE CHAT GPT?
IS MY FIRST PART OF MY QUESTION
AND I WILL WAIT FOR YOU TO
ANSWER THAT BEFORE DOING MY
SECOND PART.
SO GO AHEAD, PLEASE.
>> YES, SO WE DO SEE THE BEST
RESULTS WITH CHAT GPT BUT IT'S A
VERY KRSHES DECISION TO ALWAYS
DO A OPEN SOURCE FULLY AVAILABLE
MODEL IN PARALLEL TO SEE IF WE
CAN GET THE SAME RESULTS AND
BETTER RESULTS BECAUSE IT'S SORT
OF UNACCEPTABLE TO PROVIDE--TO
RELY ON SOMETHING LIKE A BLACK
BOX TO PROVIDE OUR SERVICES.
SO YOU KNOW WE ARE WORKING ON
DEMOCRATIZING THESE LARGE
LANGUAGE MODEL ANDS MAKING THEM
MORE AFFORDABLE AND MORE
ACCESSIBLE.
>> EXCELLENT.
EXCELLENT.
MY SECOND WOP IS SOMETIMES WE
TEND IN THIS MOMENT OF THE CHAT
GPT ERA TO OVERUSE RESOURCES,
MEANING, I'LL GO ASK CHAT GPT
THE DEFINITION OF SOMETHING,
WHERE GOOGLE WAS PERFECTLY FINE
WITH THAT, RIGHT?
SO LIKE THIS WHOLE PUSH COMING
IN THE BACKGROUND OF GREENNESS
AND ENVIRONMENTAL CONSCIOUSNESS
USING THIS BUT, IN THIS
PRACTICAL INSSTANCE OF VIDEOS AS
ANSWERS TO QUESTIONS, GOOGLE HAS
THE OPTION TO CHOOSE, RIGHT?
YOU PUT A SEARCH AND YOU
HAVE--SHOW UP, AND VIDEO AND WEB
EXPW SO ON, HOW DOES--WHEN IS
IT--WHAT DO WE GAIN OVER THAT,
QUICKLY AND I WANT TO SEE HOW
YOU SEE THAT?
>> DID I UNDERSTAND THE QUESTION
RIGHT?
IT'S LIKE YOU CAN GIVE THAT
GOOGLE AND WHY THEY'RE WORKING
ON RESEARCHING THIS STUFF?
>> LET ME RESTATE MY QUESTION,
LIKE WHAT DO WE GAIN BY USING
LARGE LANGUAGE APPROACH MODELS
MULTIMODAL FOR ANSWERING
QUESTIONS WITH POTENTIALLY
VIDEOS OR VERSUS WE ARE
[INDISCERNIBLE] SEARCH OF VIDEOS
IN YOUR EXAMPLE THAT ARE STILL
ON THE KREEN, RIGHT, OF HOW TO
EXAMINE LYMPHNODINGS SO IT'S A
QUESTION THAT'S A SEARCH IN
GOOGLE AND YOU CLICK ON VIDEOS
AND YOU GET THEM, SO THAT'S MY
QUESTION, HOW DO WE GAIN MORE
WITH THIS.
>> RIGHT.
SO, THE KIND OF YOU KNOW
SEVERALLA ASPECTS, SOME PEOPLE
ARE MORE VISUAL AND SOME PEOPLE
ARE MORE SAVVY, SO SOME PEOPLE
WHO ARE MORE VISUAL, THEY WILL
WANT TO SEE A VISUAL ANSWER A
PICTURE, I MYSELF AM A TEXT
PERSON, I'M FINE READING A
DESCRIPTION OF SOMETHING, BUT I
UNDERSTAND THAT DIFFERENT
VIEWERS WILL WANT DIFFERENT OPGS
AND THAT'S KIND OF LIKE TOWARD
THAT NLM 1 ENTRY POINT WHERE YOU
CAN GET WHATEVER YOU WANT FROM
THAT ENTRY POINT.
YOU DON'T HAVE TO BE AWARE OF
LIKE, I NEED TO GO OVER THERE
AND LOOK FOR SOMETHING OVER
THERE IF I WANT A--AN IMAGE
BASED ANSWER.
THE OTHER PART OF IT IS THIS
IS--PARTICULARLY FOR THE MEDICAL
DOMAIN, SO CAN YOU SEE, YOU KNOW
THIS STEP CAPTURING FULLY OPEN
DOMAIN ALREADY EXISTS AND THERE
ARE EVALUATION ON THAT, SO IT'S
NOT A SOLVED PROBLEM EVEN FOR
THE OPEN DOMAIN BUT WE CAN
ALREADY SEE THE RESULTS ON
GOOGLE SEARCHES WHERE THERE IS
THE STEP DESCRIPTION FOR THE
ITEM OR WHARF YOU WERE LOOKING
FOR, BUT FOR THE MEDICAL DOMAIN,
NOTHING LIKE THAT EXISTS, YET.
SO WHEN YOU SEARCH THESE
YouTUBE OR GOOGLE VIEWS,
THEY'RE NOT, YOU ARE JUST
GETTING THE WHOLE VIDEO AND VERY
OFTEN IT'S NOT EVEN AN INTRUKSAL
VIDEO, IT'S TOPICAL RELEVANT,
BUT IT COULD BE SOMEONE SHOWING
THE SLIDES.
WE ALSO MAKE AN EFFORT TO SELECT
REPUTABLE SOURCES AND SELECT
PATIENT-LEVEL SOURCES SO IT
WON'T BE AT A PROFESSIONAL
LEVEL, SO ALL OF THAT IS STILL
SORT OF ACTIVE RESEARCH AND
CREATING THESE COLLECTIONS AND
HAVING THESE COLLECTIONS IN THE
PUBLIC DOMAIN IS USEFUL FOR THEM
INCLUDING GOOGLE.
NOKAY, THANKS.
>> DINA, WONDERFUL PRESENTATION
AND EXTREMELY USEFUL AREA OF
RESEARCH, NOT MY PARTICULAR AREA
OF RESEARCH BUT I HAVE A
QUESTION FOR YOU IN YOUR FUTURE
CHALLENGES OF RESEARCH BUT I
WANTED TO FIND OUT A LITTLE BIT
MORE AS A SCIENTIFIC, IF
SUDDENLY MY METRICS, I MEAN,
WHAT I'M TRYING TO DO IS
CHALLENGE, I CAN SEE IN YOUR
CASE, IT'S LIKE, BEHAVIORIAL AND
PSYCHOLOGICAL ISSUES, I MEAN, I
NEVER HAD TO WORRY ABOUT THAT.
I FOUND A MUTATION OR I DON'T
EMPLOY IN YOUR CASE, IT'S LIKE
IS THERE EMOTIONAL DISTRESS FROM
THE ANSWER OF THESE VERSUS, YOU
KNOW, IT'S USEFUL OR NOT, SO IS
IT SOMETHING THAT YOU WOULD
REQUIRE--I CAN IMAGINE THAT THAT
WILL BE PART OF YOUR CHALLENGES
AND I ALSO IMAGINE THAT THE
RESOURCES THAT HAVE YOU AROUND
AT NNLM MIGHT NOT BE PROVIDING
SOME OF THOSE, YOU KNOW, SOME
BETTER WAYS TO WORK ON THOSE
CHANCES, SO TOMORROW ON THE
QUESTION, WHAT ARE YOUR
CHALLENGES, WHAT ARE THE
RESOURCES THAT YOU HAVE OR
MOSTLY YOU DON'T HAVE TO FACE
THOSE CHALLENGES, ESPECIALLY
CONSIDERING THAT YOU ARE NOW IN
THIS AREA WHERE THINGS LIKE
THAT, BECOME A PART OF THE
METRIC OF YOUR WORK.
>> YEAH, SO WE WERE VERY
FORTUNATE AT LAST YEAR, YES, NLM
GAVE US RESOURCES TO HIRE THAT
ICF COMPANY TO DO THESE LINE BY
LINE TRANSLATIONS, SO THEN, I
THINK IT WAS 30 K, WHICH IS YOU
KNOW, THE BENEFIT IS TREMENDOUS
BECAUSE THESE ARE PEOPLE WHO ARE
TRAINED, THEY ARE SUPPORTING MED
LINE PLUS, AND THEY WERE TRAINED
TO SUPPORT EHR, SO THEY DO THAT
FOR A LIVING, THEY KNOW HOW TO
EXPLAIN THAT RESEARCH IN PLAIN
LANGUAGE SOPHISTICATED THEIR
TRANSLATIONS WERE
TRIEWMENTAL--AND THEY ALSO
HELPED EVALUATING THESE LARGE
LANGUAGE MODELS, UPWARDS IN THE
CHALLENGE, SO AGAIN THIS, IS
LIKE A SERVICE TO THE COMMUNITY
AS WELL, BECAUSE NOW THE
COMMUNITY GETS THEIR WORK
EVALUATED IN THE EXACT SAME
ENVIRONMENT SO NOW YOU HAVE, YOU
CAN AT LEAST COMPARE THESE TEAMS
THAT PARTICIPATED IN THESE
APPROACHES THAT WE USED IN THAT
CHALLENGE TO SEE WHICH 1 IS MORE
PROMISING AND WHICH 1 SHOULD BE
DEVELOPED MORE.
AND YES, WE HAD--IS THE PURPOSE
WHO HAS BEEN THE PERSON
EVALUATING THESE APPROACHES AND
THE TEXT RETRIEVAL CONFERENCE
FOR THE LAST 3 OR 4 YEARS AND
SHE WAS VERY CONCERNED IN THESE
LAST TRACK MEETING THAT WE DO
NEED TO LIKE HAVE A COMMUNITY
EFFORT ON FINDING THESE, BECAUSE
YOU HAVE SEEN THE ANSWERS, THE
ANSWERS ARE BEAUTIFUL SO IT'S NO
LONGER THE PROBLEM OF
[INDISCERNIBLE] AND FLUENCY, THE
PROBLEM NOW IS THAT THE ANSWERS
ARE SO PLAUSIBLE AND THEY ARE
ALL SO RELATIVELY RARELY THAT IS
NOW TURNING INTO LIKE DETECTING
THAT OUTLIER, EVALUATING LIKE
WHAT WE USE TO DO SO IT'S A
CHALLENGE BUT THAT'S A GOOD
THING, RIGHT?
WE ARE MOVING FORWARD.
ALL RIGHT, WE HAVE TIME FOR 1
MORE QUESTION BEFORE WE MOVE TO
CLOSED SESSION WITH THE POST
DOCS.
ALL RIGHT, SO WITH THAT, IF WE
CAN MOVE ALL OF THE BOARD AND
THE POST DOCS TO THE CLOSED
SESSION BREAK OUT ROOM, THAT 3237 02:04:04,470 WOULD BE GREAT, THANK YOU