>> WELCOME, EVERYONE, AND
THANK YOU FOR JOINING US
THIS AFTERNOON.
I'M LANA YEGANOVA, I'M
YOUR HOST FOR TODAY'S
EVENT ORGANIZED BY
METROLANGUAGE PROCESSING
AND SPECIAL INTEREST GROUP
HERE AT NIH.
WE'RE EXTREMELY EXCITED TO
HAVE TODAY DR. TRISTAN
NAUMANN JOIN US FOR THE
SEMINAR.
BEFORE I INTRODUCE OUR
WONDERFUL SPEAKER, I WOULD
LIKE TO MENTION THAT AS
USUAL WE'LL HAVE A FEW
MINUTES AT END OF THE TALK
FOR QUESTIONS AND ANSWERS.
PLEASE SEND YOUR QUESTIONS
IN BY THIS E-MAIL TO ME.
THIS IS WHERE YOU GET
NOTIFICATIONS FROM.
YOU WOULD LIKE TO
INTRODUCE OUR SPEAKER.
WE ARE EXTREMELY EXCITED
TO DOCTOR DR. TRISTAN
NAUMANN, HE'S A PRINCIPAL
RESEARCHER AT MICROSOFT
RESEARCH'S HEALTH FUTURES.
HIS RESEARCH FOCUSES ON
PROBLEMS OF INTERSECTION
OF MACHINE LEARNING AND
HEALTH EXPLORING
RELATIONSHIP IN COMPLEX
AND STRUCTURED HEALTH DATA
USING NATURAL LANGUAGE
PROCESSING AND
UNSUPERVISED LEARNING.
HE VALUES SUPPORTING BRO
BROADER MACHINE LEARNING
COMMUNITY THROUGH ACADEMIC
SERVICE AND HAS SERVED AS
GENERAL CHAIR AND A
VARIETY OF OTHER ROLES AT
THE NEUROINFORMATION
PROCESSING SYSTEMS
CONFERENCE, HLI, ON HEALTH
INFERENCE AND LEARNING.
WITH THAT, TRISTAN, THE
FLOOR IS YOURS.
>> YEAH, THANK YOU, I
REALLY APPRECIATE THE KIND
INTRODUCTION AND I THINK
IN THE SAME LIGHT AS IT
WAS MENTIONED I ENJOY
SUPPORTING BROADER
ACADEMIC COMMUNITY, I CAN
THINK OF FEW OTHER PLACES
THAT SUPPORTED
INTERSEC
INTERSECTIONAL COMMUNITY
AND I'M DELIGHTED TO BE A
PART OF THIS MORE THAN
NIH, I MEAN, I THINK FROM
MY OWN HISTORY, I CAN SAY,
I WOULD NOT BE HERE
WITHOUT SUPPORT OF NIH.
I WAS ON RESEARCH TRAINING
GRANT DURING MY GRADUATE
STUDIES AND SO MANY OF US
REALLY WERE, I'M REALLY
QUITE HONORED TO BE HERE
TODAY AND DELIGHTED TO
TELL YOU A BIT ABOUT THE
WORK GOING ON IN OUR
GROUP, REAL-WORLD EVIDENCE
AT MICROSOFT RESEARCH
HEALTH FUTURES AND HOW IT
IS SHIFTING IN THE AGE OF
L
LLM'S.
LET ME GET STARTED AND
PROVIDE AN IDEA HOW DID I
GET INTO THIS SPACE.
AS I SAID, I STARTED OFF
AS A GRADUATE STUDENT WITH
THIS IDEA THAT NLP WOULD
BE KEY TO UNLOCKING THE
FUTURE OF MEDICINE FROM
CLINICAL TEXT AND THE
INTUITION WAS THIS NOTION
FOR A LOT OF THINGS IN
MEDICINE, MAYBE WE KNOW
THAT HEART RATE IS A VERY
GOOD INDICATOR OF
SOMETHING LIKE
CARDIOVASCULAR HEALTH.
WE COLLECTIVELY AS SOCIETY
PUT FORWARD AN EFFORT TO
MEASURE THAT AND
INSTRUMENT WAYS OF MEASURE
THAT WITH HIGH FREQUENCY,
HIGH FIDELITY.
ALL THE OTHER THINGS WHERE
WE THINK ARE IMPORTANT, WE
DON'T KNOW THEY ARE
IMPORTANT FOR FUTURE
VERSION OF OURSELVES OR
SOMEBODY ELSE IN THE CARE
CHAIN, ALL THAT COULD GET
DUMPED INTO THE NOTES
SOMEWHERE.
INTUITION WAS THIS IDEA IF
WE WANT TO EXPAND OUR
CLINICAL CAPACITY MAYBE WE
SHOULD BE LOOKING TO SOME
THINGS IN THE NOTES.
IN SORT OF SAME LINE OF
THINKING, WE PROBABLY HAVE
MANY DIFFERENT TYPES OF
DATA ABOUT NOTES AND WHILE
I WAS REALLY FOCUSED ON
NATURAL LANGUAGE AND TEXT
AND ENJOY THAT AREA, IT
WOULD BE MAYBE MISLEADING
TO FOCUS ON ONE ASPECT OF
A PATIENTS RECORD OR
COLLECTIVE THINGS THEY
GENERATE.
WHAT WE WANT TO SEE IN IT
IS FUTURE IS NOTION OF
MULTI MODAL MODELS, IF
TEXT WAS IDEA THE BASIS
FOR UNLOCKING FUTURE OF
MEDICINE, MULTI MODAL
WOULD BE BASIS FOR
UNLOCKING NEW
OPPORTUNITIES IN MEDICINE,
AS WELL.
OF COURSE, YOU KNOW, AS WE
ALL MAYBE HAVE TOP OF
MIND, THERE MAY BE TWO
THINGS WE CARE ABOUT, THE
FIRST ONE IS OF COURSE,
DOES IT ACTUALLY WORK?
AND IS IT SAFE TO USE?
RIGHT.
YOU KNOW, IF WE THINK
ABOUT WHAT REAL WORLD DATA
OFFERS US, IT CAN PROVIDE
CRUCIAL EVIDENCE THAT IS
COMPLIMENTARY TO EXISTING
EVIDENCE GENERATION IN
MEDICINE INFORM THIS SAME
VEIN, WHETHER OR NOT IT IS
SAFE, WE MIGHT REALLY LOOK
TO SOLUTION SPECIFICALLY
THAT HELP MITIGATE RISK
AND CONTINUOUSLY IMPROVE
AND I WANT TO ACKNOWLEDGE
THAT I THINK IF I WERE
GIVING THIS SIMILAR TALK A
YEAR AGO, I WOULD PROBABLY
BE TALKING ABOUT VERY
DIFFERENT THINGS, MAYBE
SOME OF THEM OVERLAPPING,
MANY OF THEM VERY
DIFFERENT, STARTING A YEAR
AGO, WE HAD ESPECIALLY IN
THIS NLP SPACE THIS CALL
TO START -- HOW WE
APPROACH MACHINE LEARNING
AND ARTIFICIAL
INTELLIGENCE FOR HEALTH.
NOT SURPRISING THAT GPT
HAD IMPACT IT DID.
I THINK SPECIFICALLY ONE
OF THE MOST IMPORTANT
ASPECT OF RELEASE OF THIS
WAS ASSESS IBLT, A LOT OF
CLINICIANS I WORK WITH
WERE REACHING OUT, HEY,
HAVE YOU TRIED THIS, LOOK
AT THESE ANECDOTAL POINTS
OF EVIDENCE IT IS DOING
INCREDIBLY WELL ON CERTAIN
TYPES OF THINGS.
HERE ARE EXAMPLES I THINK
IT ISSIL FAING.
I THINK INITIAL ENGAGEMENT
WAS MORE EXCITING THAN
MANY OTHER THINGS I'VE
SEEN IN RECENT TIMES
TRYING TO MOVE MACHINE
LEARNING INTO PRACTICE AND
MEDICINE AND WE WERE
SEEING IMMEDIATE DEMANFOR
HOW CAN WE USE THIS AND
THAT IS EXCITING PLACE TO
SEE IT.
YOU KNOW, MAYBE
UNSURPRISINGLY, A LOT OF
INITIAL STUDIES, ONE I
LIKE FROM A YEAR AGO,
LOOKING AT HEY, LIKE HOW
DO SOME OF THESE TOOLS
COMPARE TO HUMANS AND SORT
OF SIMILAR SETTINGS AND I
THOUGHT THIS WAS
SURPRISING ONE IN PART.
IF YOU ASKED ME
BEFOREHAND, I WOULD NOT
GUESS THIS WAS THE CASE,
MAYBE A TOOL WOULD
ACTUALLY GET HIGHER PR
PREVALENCE OF PERCEIVED TO
BE HIGH-QUALITY RESPONDS
THAN A HUMAN WOULD.
IT MAKES SENCE IN VARIETY
OF CONTEXT, HUMANS ARE
TIME BOUND AND TOOLS ARE
NOT.
SIMILARLY, NOW IS TIME
MORE THAN EVER WE NEED TO
BE THINKING ABOUT SOME OF
THESE TOPICS, THERE HAVE
BEEN MANY ANNOUNCEMENTS
THAT INDICATED PREVALENCE
THEY ARE HOLDING WITHIN
THIS BROADER SPACE FCHL WE
LOOK BACK TO HIMMS EARLIER
THIS YEAR, PRIMARY EHR
VENDORS ABOUT "HOW WE SEE
IT" TOOLS HAVE BEEN USED
TO GENERATE MANY, MANY,
MANY MESSAGE DRAFTS BEING
REVIEWED BY HUMANS, AS
WELL, IT IS CLEAR SOME
TOOLING IS MAKING ITS WAY
INTO THIS SPACE.
I THINK ALSO, WHEN WE
THINK ABOUT THIS SPACE, WE
THINK ABOUT SOME OF THE
CONCERNS THAT WE MIGHT
HOLD.
I THINK THERE IS SORT OF
EMERGING LINE OF WORK
HIGHLIGHTING IN ADDITION
TO SOME CONCERNS, REALLY
INCREDIBLE OPPORTUNITIES,
IN PARTICULAR, WITH
RESPECT TO IMPROVING
DETECTION OF BIAS,
CREATION OF SORT OF
DATASET THAT HAVE EQUITY
RELEVANT INFORMATION.
I LIKE THIS WORK FROM P
PIERSON, ETAL, LOOKING AT
HOW TO ACTUALLY GENERATE
DATA WE WISH WE HAD RATHER
THAN MAYBE DATA WE HAVE
CURRENTLY AND WE'RE
LEARNING FROM.
I THINK WHEN WE THINK
ABOUT THIS PROBABLY IN
CONTEXT OF REAL-WORLD
EVIDENCE, THERE IS THIS
BROADER CONSIDERATION
AROUND HOW WE MIGHT BE
ABLE TO USE SOME DATA THAT
IS NATURALLY COLLECTED TO
ACCOMMODATE EXISTING
METHOD OF EVIDENCE
GENERATION IN MEDICINE AND
ONE MAY BE MOTIVATING
EXAMPLES I PRESENT UP
FRONT BEFORE DIVING IN,
SPECIFICALLY IF WE LOOK AT
ONCOLOGY SPACE, WE SEE
THIS SORT OF PARADIGM THAT
DOESN'T MAKE A LOT OF
SENCE AT FIRST GLANCE.
IF WE THINK ABOUT THE
NUMBER OF PATIENTS ENROLL
IN CLINICAL TRIAL, THERE
IS DEMAND FOR MORE
ENROLLMENT, WE SEE SO MANY
TRIAL FAILURES.
ON THE SAME SIDE, WE SEE
THIS OTHER SIDE OF THE
EQUATION THAT IS LIKE WE
HAVE FAILING TRAILS, WE
DON'T HAVE ENOUGH PATIENTS
TO BE IN TRIALS.
MAYBE WE CAN START TO
CLOSE THAT GAP.
OUR GROUP, COMING FROM NLP
BACKGROUND LOOK TO HOW
MIGHT WE USE LARGE
LANGUAGE MODELS TO REALLY
QUICKLY PROVIDE A
MECHANISM FOR UNIVERSAL
STRUCTURING OF A PATIENT
RECORD AND TRY TO UNLOCK
SOME OF THAT VALUE.
MAYBE I'LL PRESENT SOME
INITIAL WORK WE'VE DONE IN
THIS SPACE WE BROUGHT TO
MACHINE LEARNING FOR
HEALTHCARE LAST YEAR
AROUND SPECIFICALLY LIKE A
CLINICAL TRIAL FROM
CLINICAL TRIALS.GOV,
DIFFERENT ELIGIBILITY
CRITERIA.
ONE THING AS COMPUTER
SCIENTIST, CAN WE TURN
THIS INTO A LOGIC PROBLEM
OR LOGIC STATEMENT?
WE LOOKED TO LLMS TO SAY
CAN WE ACTUALLY TRY TO
CREATE SORT OF NORMAL
LOGIC STATEMENTS, LOGIC
STATEMENTS FORMED FROM
ANDS, ORS OR NOTES IN A
VERY SPECIFIC MANNER FROM
INPURT FROM CLINICAL
TRIALS.GOV, TO CONSTRUCT A
QUICK MATCHING TREE LIKE
THIS.
AND WHAT WE WERE QUITE
SURPRISED TO FIND, THE
THERE -- YOU CAN DO THIS
USING SOME TOOLS OUT OF
THE BOX AND OUT OF THE BOX
AND YOU CAN DO IT AT A
PERFORMANCE LEVEL THAT IS
COMPARABLE OR BEATS
ENTIRELY TOOLS
SPECIFICALLY DESIGNED FOR
THIS TASK.
TO SORT OF MOTIVATE WHERE
YOU MIGHT IMAGINE I'M
GOING WITH THIS, ON THE
OTHER SIDE, WE HAVE
PATIENTS WHO GENERATE T
TREMENDOUS AMOUNTS OF
DATA, AS WELL.
I SHOULD CLARIFY UP FRONT,
NO REAL PATIENT DATA IS
SHOWN, THESE ARE EXAMPLES
SYNTHETIC IN NATURE.
IF YOU THINK ABOUT DATA
PATIENT GENERATES MIGHT
GENERATE TONS OF TEXT DATA
ACROSS DIFFERENT TYPES OF
NOTES.
HERE IS COUPLE OF EXAMPLES
SYNTHETIC IN NATURE, YOU
MIGHT IMAGINE TRYING TO
FIND SOME ELIGIBILITY
CRITERIA ACROSS.
SOME EARLY WORK IN THIS
SPACE LOOKED AT CAN WE
CREATE SPECIFICALLY MODELS
AROUND THIS TYPE OF
INFORMATION EXTRACTION, IT
WAS TUMOR STAGING KRI
TERIO, TNM SPECIFICALLY,
IT SEEMS LIKE YES, WE CAN
EXTRACT INFORMATION VERY
QUICKLY AND MAKE IT
ACCESSIBLE TO CLINICIANS
OR MEDICAL EXTRACTORS OR
WHOEVER ELSE NEEDS THIS
INFORMATION AND WAY THAT
IS TRANS FORMATIVE TO WORK
THEY ARE DOING.
YOU CAN IMAGINE PERHAPS
HOW YOU MIGHT ON THE ONE
HAND BE ABLE TO DO THE
TRIALS, MIGHT BETTER
MOTIVATE MATCHING PROCESS
IF YOU ARE ABLE TO EXTRACT
SAME INFORMATION FROM
PATIENT DATA.
AND YOU KNOW, I THINK FROM
THESE PRELIMINARY RESULTS,
THEY SEEM RATHER
PROMISING, IT IS A NICE
IDEA YOU CAN TAKE
SOMETHING LIKE CRITERIA
WE'RE LOOKING FOR AND
QUICKLY IDENTIFY THAT
INFORMATION IN LITERATURE,
PATIENT RECORD AND OTHER
PLACES BECAUSE FOR
EXAMPLE, GPT OR LARGE
LANGUAGE MODEL THESE DAYS
MIGHT BE ABLE TO QUICKLY
IDENTIFY THIS INFORMATION
IN WAY THAT WAS REQUIRED
TO HAVE HUMAN INTERVENTION
PREVIOUSLY.
WE SORT OF AGAIN MOCKED UP
A FEW THINGS AROUND HOW
YOU MIGHT BE ABLE TO
BETTER TRIAGE SOME OF THE
PATIENT DATA AND IDENTIFY
BASED ON THINGS THAT SHOW
UP IN THE RECORD, MAYBE
TRIALS THEY WOULD BE
ELIGIBLE FOR.
I THOUGHT ACTUALLY I'D
TAKE A MOMENT HERE TO SORT
OF QUICKLY COVER FROMECK
IT CAL STANDPOINT HOW DID
WE GET WHERE WE ARE TODAY.
THIS HAS BEEN A RATHER
LARGE SHIFT, I'VE BEEN
LIVING FROM THE NLP SIDE
OF THINGS, I DON'T WANT TO
ASSUME EVERYBODY ELSE HAS
BEEN DOING THE SAME 6789
MAYBE I'LL MENTION AT HIGH
LEVEL, THERE HAS BEEN
CONSOLIDATION WE SEE
ACROSS AI AND LLP, AS
WELL.
ARCHITECTURE TRANSFORMED
ARCHITECTURE JOIN ENCODER
AND ENCODER NETWORK AND
ENCO
ENCODER PORTION IS
RELATIVELY SIMILAR TO
THINGS WE HAD SEEN AND
ADDS THIS PIECE OF
SELF-ATTENTION, AS WELL.
ONE THING WITH COMBINATION
OF SELF-ATTENTION AND
NUMBER OF OTHER ADVANCES
WAS THIS NOTION THAT
CERTAIN WORDS IN A
SEQUENCE RELATE TO OTHER
WORDS IN THE SAME
SEQUENCE.
IN TURNS YOU CAN IMAGINE
CONSTRAINT THE ON
DIRECTIONALITY AND THAT
OPENED BROAD VARIETY OF
DIFFERENT TYPE OF
ARCHITECTURE THAT BIRD OR
GPT OR ELMO, SESAME STREET
CHARACTERS THAT APPEAR IN
THE NAMES.
MORE GENERALLY, ACROSS ALL
OF THEM, SELF-ATTENTION
PROVIDES THIS FORM EVER
CONTEXTUALIZATION THAT IS
POWERFUL IN A NUMBER OF
SETTINGS AND GIVES US IF
WE THINK ABOUT TRADITIONAL
MASS MODEL I'LL TALK ABOUT
LATER, GIVES BROADER
CONTEXT WINDOW.
-- MENTION BEYOND USE IN
TEXT, WHAT WAS NICE TO
SEE, NOT ONLY USEFUL FOR
TEXT, YOU CAN USE THIS FOR
OTHER PLATFORMS, THINGS IN
VISION.
YOU CAN USE IT
INCREASINGLY FOR VARIOUS
ASPECTS IN MOLECULAR
SIMULATION AND MOLECULAR
DESIGN, AS WELL.
AND I WANT TO SAY, THE
OTHER THING THAT HAS BEEN
SORT OF KEY AND SHOWN UP
IN NLP FOR A LONG TIME,
NEWER TO THE SPACE,
ESPECIALLY IN MEDICINE, WE
HAVE THIS SIMILAR
CONUNDRUM, TO NLP, WE HAVE
LABEL DATA, WE DON'T HAVE
THAT MUCH AND WE WANT TO
BUILD THIS APPLICATION
MODEL.
AT THE SAME TIME, WE
PROBABLY HAVE AS WE DO
OFTEN IN MEDICINE, YOU
KNOW, A HUGE AMOUNT OF
DOMAIN KNOWLEDGE THAT TASK
SPECIFIC AND WE TYPICALLY
HAVE HUGE AMOUNT UNLABELED
DATA THAT IS TEST
AGNOSTIC.
ONE ADVANCE IN THIS SENSE
HOW DO WE BUILD ON THE
OTHER SIDE?
WALK THROUGH EXAMPLE OF
HOW THIS WAS DONE IN NLP
SPACE SPECIFICALLY
REALIZATION WE HAVE A TON
OF UNLABELED TEXT AND CAN
MAKE UP OUR OWN TRAINING
TASKS WITH THIS.
SPECIFIC TASK WAS THIS MAS
LANGUAGE OBJECTIVE WHERE
YOU EFFECTIVELY SAY LET'S
COVER UP SOME WORDS M THIS
TEXT, GIVEN ONE MASKED
WORD, WHAT ARE THE OTHER
MASKED WORDS?
IF YOU CONSTRAIN THAT
PROCESS TO HAVING THE NEXT
WORD ONLY EVER BE THIS
GPT-STYLE NEXT-WORD
PREDICTION, THIS IS LIKE
AUTO AGGRESSIVE FORM OF
WHAT I JUST DESCRIBED, YOU
GET AN INTERESTING THING
WHERE WE END UP WITH
MODELS WE HAVE TODAY.
THAT IS MAYBE ANOTHER
SHIFT THAT I'LL DETAIL A
LITTLE BIT MORE WAS WE HAD
THIS SHIFT SORT OF FROM
SPECIFICALLY MASKED
LANGUAGE MODEL OR LANGUAGE
MODEL IN GENERAL AT A
DIFFERENT SCALE TO HOW CAN
WE ACTUALLY USE PROMPTS
AND HAVE THIS GENERALIVITY
FORM OF GENERATIVE AI TO
SUPPORT SOME OF OUR TASKS
LIKE TRANSFORMATION AL TASK
MOVING FROM WHAT WE HAD
BEFORE TO WHAT WE HAVE
NOW.
A LOT OF WORK HAS BEEN DIN
SINCE THEN ON SPECIFIC
FORMS OF SUPERVISED FINE
TUNING, REINFORCEMENT
LEARNING FROM HUMAN
FEEDBACK, PREFERENCE
OPTIMIZATION, CALL OUT
THINGS IN THE SPACE THAT
CONTRIBUTE THAT
TRATRANSFORMATION.
TWO THINGS THAT HAVE
DRIVEN THIS, ONE, T
TREMENDOUS GROWTH INNED
MOEL SIZE, THIS NEEDS TO
BE UPDATED, REINFORCEMENT
LEARNING ADVANCE IN MODEL
SPACE, AS WELL.
THIS IS MOVE TOWARD
INCREASINGLY LARGE MODEL
SIZES AND THEN ALSO LARGE
AMOUNT OF DATA
CONTRIBUTING TO EACH OF
THE MODELS, AS WELL.
AND ONE OF THE THINGS WE
SORT OF SEE THIS SORT OF
PIECE WHERE AS WE SCALE
LLMS TO CERTAIN SIZES, WE
CAN START TO SEE LIKE
EME
EMERGENT ABILITIES, THIS
NOTION THERE IS
PERFORMANCE JUMP IN DOWN
STREAM TASKS WE WERE NOT
EXPECTING, RIGHT.
ASIDE FROM THAT, YOU OFTEN
SEE SORT OF FEW SHOT
ABILITY WHERE YOU PROVIDE
A FEW EXAMPLES OR CHAIN OF
THOUGHT ABILITIES WHERE IF
YOU INCLUDE SOME OF THOSE
IN THE CONTEXT, PRIOR
PROMPTS, FOR EXAMPLE, YOU
GET BETTER AND BETTER
RESULTS, THIS IS LIKE
UNEXPECTED MAYBE AN
INTERESTING THING.
THE REAL WAY MAYBE TO SEE
THIS, THOUGH, I OFTEN
THINK ABOUT LANGUAGE
EXAMPLES FROM MY OWN WORK.
TO SEE THIS, YOU CAN SEE
THE EFFECTIVE SCALE AND
VISION A LITTLE BETTER.
THIS IS JUST SINGLE PROMPT
TRYING TO PRODUCE SPECIFIC
IMAGE AND YOU CAN SEE LIKE
350 MILLION PARAMETER
MODEL, MAYBE IT IS
KANGAROO, MAYBE NOT QUITE
THERE.
DEFINITELY LOOKING A
LITTLE BETTER, THERE IS
THIS OTHER KANGAROO IN THE
FOREGROUND AND YOU CAN SEE
HOW THIS IS INCREASING
BOTH SORT OF WITH ADVANCE
IN DATA SCALE AND MODEL
SIZE.
SO I'LL TRANSITION FROM
NOW TO THAT LEVEL SET TO
MAYBE WHAT THIS LOOK LIKE
IN THE BIOMEDICAL SPACE,
THE SPACE I CARE ABOUT AND
I THINK THAT ONE OF THE
THINGS THAT HAS BEEN
INTERESTING IS WE'VE OFTEN
HAD THIS CHALLENGE MAYBE
TWO CHALLENGES IN
BIOMEDICINE OR CLINICAL
DOMAINS AT LARGE, THAT OUR
DATA IS NOT GENERAL DOMAIN
DATA.
WHEN WE THINK ABOUT
ESPECIALLY LIKE THE
COUCHES WE ALL DO TO HELP
GENERATE SOME LABELS FOR
GENERAL DOMAIN, IF WE
SHOWED LAY USERS THIS
CAPTURE OVER HERE WITH
AMMONIA, THEY WOULD BE
HARD-PRESSED TO DO IT.
MAYBE SOME IN THE OAUDIENCE
COULD DO IT.
I WOULD FAIL A NUMBER OF
THEM, IF YOU CHANGE DOMAIN
I WOULD FAIL ALL OF THEM.
THAT IS ONE LARGE
DIFFERENCE.
THE OTHER ONE DIFFERENT IN
MY MIND, WAY WE USE
LANGUAGE IS OFTEN
DIFFERENT, AS WELL.
IT IS VERY FREQUENT THAT
WE USE NEGATION IN THIS
DOMAIN DIFFERENTLY THAN
GENERAL DOMAIN.
WE OFTEN LOOK AT ONE OF
THESE CHEST X-RAYS, FOR
EXAMPLE, LOOKING FOR
SOMETHING SPECIFIC,
DESCRIPTION, IT EXISTS OR
DOESN'T EXIST AND THIS TO
DRIVE THAT EXAMPLE HOME,
IF YOU LOOK AT THIS IMAGE
OF TWO COWS GRAZING IN A
FIELD, CLASSIC IMAGE
EXAMPLE, YOU WOULD NEVER
SAY NO PRESENCE OF
DINOSAUR IN THIS LANGUAGE.
THAT APPEARS IN MEDICINE,
MAYBE WE NEED DIFFERENT
TOOL FOR BIOMEDICAL
RELATIVE TO GENERAL DOMAIN
AND MOTIVATES IT,
COMPARABLE DATASET I LIKE
THIS EXAMPLE, GIVEN A
NUMBER OF YEARS AGO,
OBSERVING A LOT OF
DATASETS HAVE ARRIVED IN
CLINICAL AND BIOMEDICAL
SETTINGS OVER A DECADE
LATER.
NIH HAS DONE INCREDIBLE
AMOUNT TO MAKE DATA
AVAILABLE IN THESE
DOMAINS.
BUT BY AND LARGE, THE
ORDER OF MAGNITUDE AND
SCALE THAT WE SEE ARRIVES
ABOUT A DECADE LATER,
THINGS LIKE I2B2 ARRIVED
OVER A DECADE AFTER PEN
DATASET SIZE.
SO I THINK THIS IS MAYBE
TWO OF THE CHALLENGES AND
ONE THING I WANT TO RUN
THROUGH IS WHAT ARE SOME
PARADIGM SHIFTS WE'VE HAD
IN BIOMEDICINE?
WE'VE LANDED IN A SPACE
NOW FOR GENERAL DOMAIN AND
WE LOOK TO HAVING LAYER
WHERE LLM IS ABLE TO
PROVIDE BOTH LANGUAGE
UNDERSTANDING AND
GENERATION.
SO IF I SORT OF BREAK DOWN
WHAT SOME PARADIGM SHIFTS
WITH LLMS ARE, THERE ARE
MAYBE THREE.
I'LL DIVE INTO THESE
BRIEFLY, IT IS MAYBE THIS
SHIFT FROM SPECIALIST TO
GENERALIST MODEL, SHIFT
FROM CLOSED SET TO
OPEN-ENDED GENERATION AND
SHIFT FROM REPRESENTATION
LEARNING TO PROMPTABLE
MALLEABLE INTERFACES.
IN THE FIRST ONE, REALLY
QUICKLY, GO BACK AND THINK
ABOUT WHAT WERE WE DOING
IN LLP FOR A LONG TIME, WE
MIGHT HAVE FOCUSED ON
SUMMARIZATION,
SUMMARIZATION MODELS OR
MAYBE FOCUSED ON ENTITTY
RECOGNITION AND I THINK A
LOT OF COMMUNITIES ARE
DIVIDED AROUND SPECIFIC
AREAS AND HOW TO DO THEM
AND A LOT OF OVERLAP AND
WE WORKED MANY OF THEM
MUCH OF THE TIME.
IT WAS REALLY THIS IDEA
YOU WOULD HAVE THESE SORT
OF SPECIALISTS MODELS AND
I THINK WHAT WE STARTED TO
SEE WITH LARGE LANGUAGE
MODEL WORK, THINGS LIKE
BIRD MAYBE NO LONGER
LARGE-LANGUAGE MODEL, AT
THE TIME, THEY WERE
CERTAINLY LARGE, SHIFT TO
A SET OF HEADERS THAT WERE
SPECIALIST IN NATURE.
YOU WOULD HAVE TASK
AGNOFTIC BASE AND SHIFT TO
TASK SPECIFIC HEADER FOR A
SPECIFIC SET OF TASKS.
WHAT WAS NICE, BOTH WAS
REUSABLE AND ACCELERATED
RESEARCH AND SHARED SOME
OF THE ADVANCES ACROSS
ONES THAT WERE NOT SEEING
AS MANY ADVANCES, AS WELL.
ONE THING THAT HAS BEEN
PRETTY MAJOR SHIFT SINCE
IS LIKE I SAID, MAYBE GOES
INTO LLM AND COMES OUT THE
OTHER SIDE FOR EACH OF THE
TASKS.
SECOND ONE IS DIFFERENCE
BETWEEN CLOSED SET
CLASSIFICATION AND OPEN
ENDED, GOING BACK TO
PREDEFINED SUMMARIZATION
YOU HAD PREDEFINED SET OF
TEMPLATES OR LIMITED LABEL
OR SPACE IS CONSTRAINED IN
SOME WAY OR FOR ENTITY
RECOGNITION, PREDEFINED
SET OF LABELS, THIS
SHIFTED OVER TIME WITH
LLMS TO BE YES, YOU CAN
PROVIDE CLASSIFICATION,
YES, YOU CAN DO VERY
SPECIFIC CLASSIFICATIONS
FOR SPECIFIC DATASET AND
START TO DO SUMMARIZATION,
OPEN-ENDED GENERATION
,
YOU CAN DO STRUCTURED
FORMS OF VARIOUS THINGS
AND DO CLASSIFICATION AND
GENERATION SIMULTANEOUSLY
RATHER THAN DIVIDING THEM
OUT IN SEPARATE MODELS.
LAST ONE WAS REALLY SHIFT
FROM REPRESENTATION
LEARNING WE WERE DOING
PREVIOUSLY TO MORE
PROMPTABLE INTERFACE AND
SPECIFICALLY LIKE WHEN I
DESCRIBED IT BUYEFORE, WE
HAD ARCHITECTURE THAT WAS
REUSABLE AND LOOKING AT
HOW CAN WE BUILD OUT TASK
SPECIFIC HEADERS TO MOVING
TOWARD SOMETHING THAT IS
MUCH MORE ORIENTED AROUND
BUILDING OUT PROMPTABLE
INTERFACE UNIVERSAL IN
SOME SENSE, AND DOES TASKS
WE WOULD LIKE.
UNSURPRISINGLY BECAUSE OF
SUCCESS AND EASE OF USING
SOME TOOLS THERE HAS BEEN
EME
EMERGENCE OF HUGE NUMBER
OF DIFFERENT PIECES IN
THIS SPACE.
WE HAD ACTUALLY LOOKED AT
THIS A LONG TIME AGO WITH
PROJECT RENAMED BUT ONE OF
THE THINGS WE WERE LOOKING
AT WAS NOTION OF HOW
ACCOUNT WE DO DOMAIN
SPECIFIC, LIKE PUB MED,
INCREDIBLE RESOURCE AND
MAYBE SUFFICIENT AMOUNT OF
TEXT WE CAN GO FROM THERE
AND ACTUALLY TRAIN OUT
DOMAIN-SPECIFIC MODEL AND
WE LOOKED AT THIS AND WE
THOUGHT, YOU KNOW, WHY
WOULD WE WANT TO DO THIS?
MAYBE WE COULD DO THIS,
WHY WOULD WE WANT TO DO
THIS?
THE REALIZATION IN BOUNDED
RESOURCE, MAYBE WE WANT TO
HAVE MORE EFFICIENT
LEARNING BY FOCUSING ON
IN-DOMAIN DATA CO
COMPREHENSIVE APPROACH OF
TRYING TO UNDERSTAND
EVERYTHING.
INITIALLY LOOKING AT
THINGS SPECIFICALLY AROUND
VOCABULARY THAT SHOWS UP,
YOU KNOW, THIS IS EXCERPT
FROM THE PAPER RENAME YET,
WE WERE SPECIFICALLY
LOOKING AT MAYBE THE WORD
BREAKS AND TOKENIZATION
SCHEME DON'T MAKE SENSE
FOR OTHER DOMAIN MODEL, IF
WE SPECIFICALLY TRAIN ON
THIS DOMAIN, PRESERVE
INTEGRITY OF THINGS WE
CARE ABOUT FOR DOWN-STREAM
TASK IN BIOMEDICINE, YOU
KNOW COMPARED TO WHAT
OTHER MODELS WERE DOING
SHATTERING TOKENS AND
WORDS INTO MANY TOKENS.
WE MADE THIS AVAILABLE SO
EVERYONE COULD USE IT,
FANTASTIC RESOURCE FOR
EVERYONE.
AGAIN, WHILE THERE IS A
LOT OF UTILITY IN HAVING
SMALLER MODELS ACROSS THE
BOARD WE START TO SEE
SHIFT TOWARD MORE
GENERALIST MODELS TO ALLOW
US TO WORK IN BIOMEDICINE,
AS WELL.
ONE THING INTERESTING, I
LOVE SEEING EVOLUTION
TREES LIKE THIS.
I ENJOY THEM IN PART SO
MUCH BECAUSE THEY JUST
ADVANCED SO QUICKLY AND
THIS IS NOT EVEN CO
COMPREHENSIVE ANYMORE LIKE
COMMAND R AND OTHERS NOT
THERE, AS WELL.
MAYBE I'LL QUICKLY TALK
ABOUT SPECIFICALLY ONE OF
THE THINGS ON ENCODER
STYLE ONLY ARCHITECTURE
REFLECTED IN SEVERAL OF
THESE MODELS AND MAYBE THE
ENCODER DECODER STYLE
ARCHITECTURE, MAYBE THE
MOST IMPORTANT THING TO
SAY HERE, THERE IS CLASS
OF MODELS LIKE T-5 MODELS,
MOSTLY TRYING TO PROVIDE
KEYWORD SO PEOPLE
UNDERSTAND THESE ARE TYPE
OF ARCHITECTURE WE SEE
WHEN TALKING ABOUT THESE
TERMS.
MAYBE THE THIRD TYPE THAT
IS MORE COMMON TO EVERYONE
NOW IS THIS BROADER NOTION
OF LIKE DECORD ONLY STYLE
ARCHITECTURE, YOU ARE
SPECIFICALLY TRAINING FOR
NEXT TOKEN PREDICTION FROM
THE PRIOR TOKENS.
AND AGAIN, THESE HAVE BEEN
SHOWN TO WORK REALLY,
REALLY WELL SPECIFIC WHEN
TRAINED ON SPECIFICALLY
BIOMEDICAL AND DOMAIN SORT
OF DATA, IN I THINK THINGS
LIKE BIO-GPT RELATIVE TO
HUM
HUMANANOTATION AND
RELATIVE TO OTHER
OFFERINGS IN THIS SPACE,
AS WELL.
MAYBE WHAT I'LL DO IS I
THINK MANY OF US HAVE SEEN
SORT OF SOME RESULTS, I
AUN TO ACKNOWLEDGE THEM
AND ACKNOWLEDGE THAT IN
SOME CASES WE ASKED THIS
QUESTION OF HOW WELL DOES
AI PERFORM CLINICALLY AND
LOOK AT OUT OF BOX
EXPERT-LEVEL COMPETENCY
ON -- DEBATE AROUND LIKE
SEEING THIS TEST IS NOT
SUFFICIENT FOR CLINICAL
CLAIMING VICTORY BUT IT IS
REALLY INTERESTING
QUESTION IN MANY WAYS
BECAUSE I THINK AGAIN, IF
I GO BACK MAYBE A COUPLE
YEARS AND YOU ASK ME, IS
SOMEBODY AN MLP, WE WANT
TO DO THESE TYPES OF
QUESTIONS LIKE ALLEN
INSTITUTE ENDEAVOR AND
DOING PROGRESSIVELY HARDER
AND HARDER MAC QUESTIONS
OR WRITTEN EXAM QUESTIONS
IN OTHER DOMAINS.
COULD YOU BUILD A TOOL TO
DO THIS?
I WOULD HAVE GIVEN YOU A
LOT OF REALLY SORT OF
ANSWERS THAT WERE HEDGED
IN DOUBT AND MAYBE WE
COULD DO IT, IT WILL TAKE
QUITE A BIT.
I THINK WHAT IS
INTERESTING, INDEPENDENT
IF WE THINK THIS IS USEFUL
FOR LIKE CLINICAL BASIS IN
MEDICINE, IS REALLY THIS
NOTION THAT THIS IS
SOMETHING THAT
FUNDAMENTALLY COUPLE YEARS
AGO MAYBE WE COULD HAVE
DONE IN WELL-SCOPED
SCENARIOS AND SEEMS TO BE
SOMETHING THAT
INCREASINGLY LARGE
LANGUAGE MODELS ARE ABLE
TO DO OUT OF THE BOX TODAY
AND THAT IS A REALLY
FUNDAMENTAL SHIFT IN THIS
UNDERLYING NLP SPACE.
SO I THINK ONE OF THE
MAYBE ONE OF THE QUESTIONS
THAT COMES UP THIS
QUESTION AROUND GPT FOR
SHARING GENERAL DOMAIN SO
LIKE WHY WOULD WE EXPECT
IT TO WORK FOR
BIOMEDICINE?
MAYBE WE EXPECT IT TO WORK
WELL FOR BIOMEDICINE
BECAUSE IT CONTAINS
BIOMEDICAL TEXT.
LOOK AT SOME COMPONENTS IN
LARGE RESOURCES LIKE FILE,
PUB MED CENTRAL IS
SIGNIFICANT PORTION OF
THAT 825 GIGA BYTES
TOGETHER WITH ABSTRACT
OVER 100 GIGS, AN EIGHTH
OF THE DATA IN MANY LARGE
LANGUAGE MODELS.
THIS IS REALLY INTERESTING
IDEA OF, YOU KNOW, MAYBE
SOME GENERALIST MODELS
BEING TRAINED, NONE OF US
KNOW WHAT ARE IN SOME OF
THESE, BUT BASED ON
OPEN-SOURCE MODELS TRAINED
ON SOME OF THESE DATASET
MAYBE GOOD IMPRESSION WHAT
TYPES OF THINGS ARE IN
THERE.
MAYBE WAY TO THINK ABOUT
SOME OF THIS IS IN
BIOMEDICINE, WHAT WE'LL
HAVE THIS SET OF
GENERALIST MODELS, FOR
EXAMPLE, THINGS LIKE
GPT-4, THEN WE'LL ALSO
HAVE SET OF LIKE TASK
SPECIFIC FINE TUNING
MODELS LIKE MED PALM, IF
WE'RE LOOKING AT MULTI
MODAL THINGS OR LAVA OR
MED PUB-M.
WE'RE GETTING MORE
SPECIALIZED AND IT IS
HARDER TO STEER DOWN TO
DOMAIN SPECIFIC
PRETRAINING OF SPECIFIC
TYPES OF MODELS.
THERE ARE MORE POWERFUL
MODELS, EASIER TO STEER
AND MAYBE THE TRADEOFF TO
THINK ABOUT.
WITH THAT IN MIND AND SORT
OF AN IDEA WHERE WE ARE IN
THE BIOMEDICAL LLM SPACE I
WANT TO HIGHLIGHT WORK OUR
GROUP HAS DONE AND REALLY
TALK THROUGH SOME OF MAYBE
LIKE FRONTIERS WE'VE BEEN
TRYING TO EXPLORE.
QUICKLY, ONE OF THE FIRST
IS IN BROAD SPACE OF
PROMPT
PROMPTING.
AS I MENTIONED BEFORE, A
LOT OF MODELS HAVE SHIFTED
US TO THINKING MORE AND
MORE ABOUT HOW DO WE
PROMPT THE MODEL TO DO THE
THING WE'D LIKE TO DO OR
IN TURN, HOW DO WE HAVE A
MOD
MODEL START TO ACT ON THE
SET OF THINGS WE WOULD
LIKE IT TO DO?
I'LL SORT OF TAKE INITIAL
STEP IN THIS DIRECTION.
IF WE THINK ABOUT SORT OF
MAYBE LIKE COMPARISON WITH
THINGS LIKE MED PALM M,
THEY WERE ABLE TO
DEMONSTRATE COMP TENSZY ON
U.S. MEDICAL EXAM, ONE
THING INTERESTING ABOUT
THE WORK NOT ONLY WAS MED
PALM 2 ABLE TO IMPROVE
OVER MED PALM AND BROAD ER
RESULTS, BUT IF YOU THINK
ABOUT LIKE HOW THEY WERE
ABLE TO DRIVE SOME OF THAT
IMIMPROVEMENT, ONE THING
THEY WERE LOOKING AT WAS
HOW CAN WE COME UP WITH
APPROPRIATE PROMPTS AND
RESTYLE STRATEGY.
LOOK AT RESULT ON MED QA,
SOME OF THE PROMPTS WERE
LOOKING AT UP TO 40-45-ISH
API CALLS PER SPECIFIC
PROBLEM.
THIS IS INTERESTING FOR
VARIETY OF REASONS AROUND
HOW TO BREAK TASKS DOWN
INTO THINGS LLMS CAN
ANSWER MORE EASILY AND
ALSO INTERESTING IN TERMS
OF WHAT SHOULD WE EXPECT
OUT OF BOX WITH ONE PROMPT
VERSUS MULTIPLE PROMPTS,
AS WELL.
WHAT WAS INTERESTING IN
THIS CASE, AS WELL, WHEN
WE START LOOKING AT OTHER
RESULTS IN THE SPACE,
MIGHT BE THINKING WHAT DO
WE GET OUT OF THE BOX?
WE STARTED TO SEE OTHER
RESULTS THAT STARTED TO
SUGGEST ACTUALLY WE COULD
GET OUT OF THE BOX
PERFORMANCE WITHOUT MORE
EXPENSIVE FINE-TUNING
MECHANISM WITH
PERFORMANCE.
QUESTION WAS COULD WE
START, IF WE COULD GET
THAT WITH NO FINE TUNING
WITH CALL WITH FEW SHOT
COULD WE PUSH THAT EVEN
FURTHER WITH MORE
ELABORATE PROMPTING
STRATEGY?
LET'S SEE WHAT WE CAN DO.
WE SORT OF CAME UP WITH
THIS COLLECTIVELY OR
OTHERS IN THE GROUP COME
UP WITH COLLECTIVELY HOW
CAN WE PUSH THIS FORWARD
EVEN FURTHER?
AND DO SORT OF SET OF
ABLATION, WHAT IS ZERO
SHOT PERFORMANCE, DID
RANDOM FEW SHOT IDEA AND
INTRODUCE ED CHAIN OF
THOUGHT AND INTRODUCED
NOTION WE WANT TO FIND
EXAMPLES SORT OF SPECIFIC
TO THE PROMPT THAT YOU --
THE QUESTION YOU ARE
TRYING TO ANSWER IN THIS
PROMPT?
COULD WE COME UP WITH
SPECIFIC FEW SHOT EXAMPLES
USING KLM AND TRAIN OF
THOUGHT TO MOVE INTO
USEFUL CONTEXT FOR THE
LLM?
COULD WE ASSEMBLE ALL
THESE CHOICES TOGETHER TO
COME UP WITH A SOLUTION
THAT ACTUALLY PUSHES THIS
FURTHER AND I THINK WHAT
WAS NICE ABOUT THIS WORK
WAS REALLY COMBINATION OF
THE ABLATION AND
UNDERSTANDING PROMPTING
STRATEGIES MOVING US
FORWARD AND IT GIVES US
SIGNIFICANT LIFT ACROSS
THE BOARD WITH RESPECT TO
SOME OF THE RESULTS AND
AGAIN, I WANT TO
ACKNOWLEDGE THAT U.S. MLE,
NOT ONLY BENCHMARK FOR
DOING THIS, LOOK AT BROAD
SET OF BENCHMARKS TO
UNDERSTAND MULTIPLE POINTS
OF EVIDENCE AROUND WHERE
MIGHT NEEDLE MIGHT BE
GETTING MOVED EFFECTIVELY.
I ALSO WANT TO HIGHLIGHT
WORK OTHERS IN THE GROUP
ARE DOING AROUND SELF-
SELF-VERIFICATION.
I THINK FOR MANY OF US,
THERE ARE NUMBER OF THINGS
WE MIGHT CARE ABOUT, WE
MIGHT CARE ABOUT CLOSED
NATION AND DATA AND
INCLUESEFITY AND THINGS
LIKE THAT.
MANY SERIOUS ISSUES AND
RISKS.
I WANT TO TALK ABOUT
SPECIFICALLY LIKE
HALLUCINATIONS AND MAYBE
TO A LESSER DEGREE MATH
AND LOGIC ERRORS AND
REASONING ABILITY.
BECAUSE I THINK WHAT WE
ARE LOOKING AT WAS
INITIALLY THIS SORT OF
CONCERN WE HAD HAD AROUND
IF YOU HAD A PROMPT LIKE
THIS, YOU KNOW, CAN YOU
TELL ME ABOUT A SPECIFIC
USE OF A DRUG AND CITE
YOUR SOURCES.
YOU WOULD GET LIKE A QUICK
SORT OF ANSWER AND SOME
SET OF SOURCES, RIGHT?
IT IS VERY EASY FOR GPT TO
HALLUCINATE.
THIS SPECIFIC SOURCE IS
NOT WHAT IT SEEMS TO BE
BASED ON THE SOURCE THAT
IS THERE.
IF I CLICK THIS LINK, I GO
TO SOURCE AND IT IS NOT
THE CITATION WE THOUGHT IT
WAS.
OBVIOUSLY A NUMBER OF
TOOLS -- GENERATION BEING
MAYBE KEY TO SORT OF LIKE
TRYING TO REDUCE THIS.
ONE THING WE WANTED TO
COVER IN SOME OF THIS WORK
WAS REALLY THIS IDEA HOW
COULD WE TRY TO REDUCE
SOME OF THESE AND I WANT
TO ACKNOWLEDGE THAT THERE
IS ACTUALLY SOME
SIMILARITY HERE IN THE WAY
THAT MAYBE LIKE WE RECALL
FACTS, WHICH IS
INTERESTING.
IF I, I ASKED FOR EXAMPLE,
DID YOU READ THIS REALLY
LONG BOOK, YOU KNOW,
PERHAPS LIKE LIMITS OR
LIKE A PLAY OR SOMETHING.
YOU MIGHT SAY YES.
IF I ASKED YOU TO RECITE
VERBATIM THE FIRST PAGE OR
WHATEVER MEANINGFUL UNIT
PROBABLY NONE OF US COULD
DO IT.
WE REMEMBER BROADER
THEMES.
MAYBE THERE IS SOMETHING
INTERESTING HAPPENING
HERE, BUT PRACTICALLY WHAT
WE WOULD LIKE IS LIKE A
SOURCE THAT ACTUALLY IS
THE CORRECT FACTUAL
SOURCE.
ONE THING WE THOUGHT ABOUT
FROM PURE SCIENCE STAND
POINT ANALOGY TO VERIFKSZ
BEING EASIER THAN -- WE
LOOKED AT COUPLE DIFFERENT
METHOD AROUND HOW CAN WE
START USING PROMPTS TO
SELF-FACT CHECK.
SO WE MIGHT WANT TO GO
FROM A PATIENT CHART LIKE
I'M SHOWING ON LEFT TO
SOMETHING LIKE THE
SPECIFIC PROBLEM LIST THAT
IS THERE.
AND SO WE CAME UP WITH
THIS PIPELINE OF CAN YOU
HAVE SORT OF PROMPT THAT
REPROMPTS TO SAY FIND ALL
THINGS YOU HAVE MISSED AND
BUILD EVIDENCE FOR THINGS
YOU MISSED AND RESULTS,
MAYBE COMING FROM CLASSIC
INFORMATION RETRIEVAL
PIPELINE, WHERE YOU WANT
TO BOOST RECALL AND
RESTORE PRECISION
AFTERWARD.
EXAMPLE OF WHAT THIS MIGHT
LOOK LIKE, YOU HAVE A
PROMPT TEMPLATE AND YOU
EFFECTIVELY GO THROUGH AND
SAY EXTRACT DISEASE FROM
THE TEXT AND IT DOES IT.
FINE ALL EMISSIONS AND YOU
SAY GENERATE EVIDENCE FOR
THIS OMISSION AND ONE OF
THE -- MAYBE TWO POINTS
USEFUL HERE.
ONE, IF YOU GET SOMETHING
BACK FOR AN LLM THAT IS
EQUALLY HAND WAVING AND
GENERATING EVIDENCE, IT IS
LESS LIKELY IT EXISTS,
ESPECIALLY IF YOU ASK
SEVERAL LLMS, IT IS IDEA
IF YOU WALK DOWN THE
STREET AND ASK FOR
DIRECTIONS, THREE PEOPLE
GIVE SAME SET OF
DIRECTIONS, PROBABLY
HEADING RIGHT.
IF THEY ALL GIVE DIFFERENT
DIRECTIONS, PAUSE AND
FIGURE OUT.
WE SEE SAME THING HERE.
SECOND POINT, BY
GENERATING EVIDENCE, MUCH
EASIER FOR HUMAN TO CHECK
IS THIS THING THERE OR
NOT.
IN A LOT OF HUMAN IN THE
LOOP USE CASES WE IMAGINE,
FOR EXAMPLE, THIS IS
REALLY USEFUL IN PROVIDING
EVIDENCE HUMANS CAN
VERIFY.
WE DO THIS PROCESS AND
CONTINUE THIS PROCESS
UNTIL WE LAND ON SOMETHING
THAT IS A USE FUL PROBLEM
LIST.
AND WHAT WE FOUND WHICH IS
INTERESTING, JUST ADDING
SIMPLE PROMPTING EXERCISE
AND FORMULATING INTO
PIPELINE WE GET COMPARABLE
TO STATE OF THE ART,
PRETTY INTERESTING RESULT
FOR SOMETHING IN ADVERSE
DRUG EVENT DETECTION, WE
START TO SEE NOT ONLY CAN
WE DO THIS, THROUGH
DISTILLATION, WE SEE
THINGS SMALLER IN THIS
CONTEXT, AS WELL.
MAYBE BROADLY, WAY I WOULD
THINK ABOUT LIKE SOME OF
THE EMERGING BEHAVIOR IN
THIS SPACE, YOU CAN USE
LARGE LANGUAGE MODEL LIKE
GPT-4 TO PROVIDE THIS
MECHANISM FOR UNIVERSAL
STRUCTURING, YOU CAN DO
UNIVERSAL SUMMARIZATION,
IF YOU START THINKING
ABOUT OTHER THINGS WE CARE
ABOUT IN PRACTICE, COST
AND EFFICIENCY, WHITE BOX
AND BLACK BOX ACCESS,
CUSTOMIZATION AND CONCERNS
WE MIGHT HAVE, YOU MIGHT
WANT TO LOOK AT ASPECTS OF
KNOWLEDGE DISTILLATION TO
MODELS YOU CREATE
YOURSELF.
I'LL PAUSE THERE AND SHIFT
AND MOVE TO MULTI MODAL
MODELS.
AS I MENTIONED BEFORE,
THERE IS A LOT OF THINGS
WE CARE ABOUT AT LARGE IN
THE LANGUAGE SPACE AND I
THINK LANGUAGE HAS BEEN
REALLY DRIVING FORWARD A
LOT OF SPACE.
PATIENTS GENERATE MANY,
MANY DIFFERENT MODALITIES
OF DATA, RIGHT?
OUR HOPE WOULD BE MULTI
MODAL MODELS ARE ABLE TO
UNDERSTAND AND REASON
ABOUT MANY OF THESE
MODALITIES, RIGHT?
AND I THINK THIS IS REALLY
INTEREST
INTERESTING EMERGING SPACE
IN PART BECAUSE SO MUCH
WORK IN THIS SPACE AND SO
MUCH POSSIBILITY, A LOT OF
WORK GOING ON WITH
MODALITIES DOING WELL,
SPECIFICALLY MULTI MODAL,
WE OFTEN MEAN SPECIFICALLY
LIKE IMAGING AND TEXT
RATHER THAN OTHERS WHERE
THERE IS OPEN NEED AND
AGAIN, JUST TO PROVIDE
CONCRETE USE AND MOTIVATE
SOME OF THE REASONS WHY WE
MIGHT CARE ABOUT MULTI
MODAL, WE HAVE CONCRETE
USE CASES.
WE LOOK AT FOR EXAMPLE
LIKE CASE STUDY IN
IMMUNOTHERAPY, WE MIGHT BE
INTERESTED IN FINDING
RESPONDERS.
WE NEED TO KNOW ABOUT THE
ENVIRONMENT, WE CAN'T LOOK
TO TEXT, WE PROBABLY NEED
TO LOOK TO SOME FORM OF
PATHOLOGY AND SOME FORM OF
TEXT OR PERHAPS ENTIRE
PATIENT RECORD.
AND SO WHEN WE START
THINKING ABOUT THE
CHALLENGES OF BRINGING
MULTI MODAL TO DOMAIN
MAYBE FOREMOST IS GOING
BACK TO DATA PROBLEM I
MENTIONED ABOUT
DIFFERENCES IN -- AND
BIOMEDICAL.
IF I ASK FOR A PHOTO OF
LUNG CT SCAN, I MIGHT GET
THINGS LIKE THIS FROM A
LOT OF IMAGE GENERATION
TOOLS AND THESE ARE
HELPFUL AND SUPER COOL,
THEY LOOK GREAT IN
PRESENTATIONS.
BUT FOR ANYONE WHO IS RA
RADIOLO
RADIOLOGIST, NONE OF THESE
LOOK REAL, ESPECIALLY ON
LOWER RIGHT, WE EXPECT
SOMETHING CLOSER TO THIS.
WE LOOKED AT MULTI MODAL
MODELS WITHIN OUR GROUP
HOW CAN WE DO THESE FORMS
OF GENERATION, FACTUAL
GENERATION BASED ON RERNS
EXAMPLES THAT ARE MORE
REAL IN NATURE THAN THINGS
WE SEE TODAY AND WHEN WE
THINK ABOUT THIS BROAD
LANDSCAPE OF BIOMEDICAL
LARGE MULTI MODAL MODELS,
THERE IS A LOT OF EFFORT
IN THIS SPACE, VISION TR
TRANSFORMERS AND THINGS
LOOK AT HOW WE FORM BASIS
OF GLOBAL AND MODAL
ALIGNMENT LIKE GLORIA AND
PROJECTS LIKE WE HAD
WORKED ON REALLY AROUND
NOT ONLY CAN WE USE THAT
GLOBAL AND LOCAL
ALIGNMENT, CAN WE DO LIKE
RADIOLOGY SPECIFIC
LANGUAGE MODELING TO TRY
TO ADVANCE STATE OF THE
ART THERE AND OTHER THING
WE CAN ALSO IMAGINE DOING
IS LOOKING TO VAST AMOUNT,
AGAIN, LOOKING TO THINGS
LIKE PALM MED CENTRAL
BASIS OF THIS BASIS FOR
GENERATING DATA COMPARABLE
TO GENERAL DOMAIN AND SO
ONE THING WE MIGHT LOOK TO
FIRST IS IF WE HAVE
ADVA
ADVANCES IN CLIP DOING
CONTRASTING LANGUAGE AND
YOU NEED IMAGE AND
CORRESPONDING TEXT COULD
WE GATHER FROM
BIOMEDICINE?
IT LOOKS LIKE WE CAN.
THERE ARE 30 MILLION PLUS
FINGER CAPTION PAIRS IN
PALM MED AND EXAMPLE BEING
RIGHT HERE.
IF WE HAD THAT, COULD WE
IMAGINE TAKING VARIETY OF
DIFFERENT FIGURES AND
ADMITTEDLY SOME WILL BE
PATHOLOGY, CHARTS,
MOLECULAR DESIGN,
RADIOLOGY, RADIOGRAPHY,
BROAD ARRAY OF DIFFERENT
DATA, CAN WE USE AS BASIS
FOR CREATING FOUNDATION
MODEL IN CLIP STYLE
FOUNDATION MODEL IN SPACE?
WE WENT AND DID THAT AND
LOOKS LIKE WE CAN AND WHAT
IS INTERESTING ABOUT IT,
RELATIVE TO SOME OF THE
OTHER ENTRIES PREVIOUSLY
IN THIS SPACE, WE'RE ABLE
TO DO QUITE WELL USING
SOME OF THIS DATA AS BASIS
FOR THE FOUNDATION MODEL.
AND I THINK WHAT IS REALLY
SPECIFICALLY INTERESTING
TO SEE, NOT JUST ACROSS
COMMON LIKE TEXT IMAGE
RETRIEVAL, CLASSIFICATION
AND BQA BENCHMARK IN THIS
SPACE, WE START TO SEE
REALLY WE'RE STARTING TO
APPROACH SORT OF REALLY
HIGH ZERO SHOT IMAGE
CLASSIFICATION PERFORMANCE
USING SOME OF THESE TOOLS,
AS WELL.
IF WE GO BACK TO THINKING
ABOUT MODEL SIZE
INCREASES, DATA SIZE
INCREASES, THINGS LIKE,
FOR EXAMPLE, PALM MED
CENTRAL AND THIS
CONSTRUCTION OF FIGURE
CAPTION FROM PALM MED AND
SIGNIFICANTLY LARGER,
ORDER OF MAGNITUDE LARGER
THAN SOME OTHER DATA
AVAILABLE THAT IS CLINICAL
IN NATURE IN THIS SPACE.
EVEN THOUGH THEY ARE NOT
CLINICAL, WE SEE INCREASES
IN ZERO SHOT PERFORMANCE
ON CLINICAL TASKS, AS
WELL.
SO THIS MOTIVATED US TO GO
A STEP FURTHER, IMAGE,
TEXT, PAIR, COULD WE BUILD
OUT SOMETHING THAT LOOKED
MORE SPECIFICALLY AT SOME
INTERACTIVE MULTI MODAL
MODELS?
WE WANTED TO START WITH
THINGS LIKE BIOMED CLIP,
START WITH THAT DATA
SOURCE AND THEN USE
ANOTHER TOOL, GPT-4 OR
LARGE LANGUAGE MODEL OF
YOUR CHOICE TO SORT OF DO
UNIVERSAL ANNOTATION AND
BUILD OUT A DATES.
IDEA YOU CAN SAY QUITE
QUICKLY, BUILD OUT A
CONVERSATION AROUND THIS
CAPTION WITHOUT EVER
HAVING SEEN THE C
CORRESPONDING IMAN, IT IS
NOT MULTI MODAL IMAGE OF
GPT-4, BUILD AROUND THIS
KAP
CAPTION AND USE FOR IMAGE.
IF WE THINK ABOUT
DIFFERENT TYPES OF DATA
THAT EXIST IN SOME OF THE
DOMAINS, YOU HAVE A
BREADTH OF DATA AND MIGHT
EXPECT TO BE ABLE TO DO
THIS QUITE WELL FOR SOME
INSTRUCTIONS AND C
CORRESPONDING R
CORRESPON
CORRESPONDING RESPONSES.
CALL OUT FRAMING THIS
ENTIRE SPACE, IN THIS
MULTI MODAL SPACE, IF YOU
SORT OF BREAK DOWN OR
MODULARIZE THIS OUT, ONE
INSIGHT, MANY OF THESE
MODELS ARE FORMED FROM
INTRODUCING PROJENTHION
LAYER TO CONVERT IMAGE TO
TEXT EMBEDDING SPACE.
THERE ARE A LOT OF
QUESTIONS, I SHOULD SAY
ABOUT WHETHER OR NOT
LANGUAGE IS REALLY THE
BASE THAT WE SHOULD BE
WORKING IN.
I OF COURSE AM VERY HAPPY
WITH IT, THAT IS SORT OF
MY BACKGROUND, FOR IMAGES,
MAYBE NOT THE RIGHT BASIS
OR LAAT ANY TIME SPACE TO
BE WORKING IN.
IT IS FOR LLMS.
IF WE THINK ABOUT SORT OF
WHAT THIS FAMILY OF MODELS
MIGHT LOOK LIKE, WE CAN
ACTUALLY LOOK AT DIFFERENT
ENTRIES IN THIS SPACE AND
LOOK AT THINGS LIKE, YOU
KNOW, LAVA, WHICH IS A
MULTI MODAL MODEL THAT
USES INSTRUCTION TUNING.
THEY BASICALLY TAKE A
LANGUAGE MODEL, WHICH IN
THEIR CASE LLAMA, VISION
ENCODER, SOMETHING TO
ENCODE IMAGES LIKE CLIP
AND PROJECT THAT VISION
SPACE INTO LANGUAGE SPACE
IN ORDER TO DO THE BUILD
THE FINAL MODEL.
IF WE THINK ABOUT THAT FOR
THAT SAME SORT OF
CONSTRUCTION MECHANISM FOR
MEDICAL SPACE ONE WORK WE
LOOKED AT WAS LOOKING AT
ONE I DESCRIBE IS LOOKING
AT CAN WE TAKE THE
UNDERLYING LANGUAGE MODEL
FROM LLAVA AND USE BIOMED
CLIP AND THEN AGAIN DO
LINEAR PROJECTION IN.
THERE ARE NUMBER OF OTHER
WORKS MAYBE NOTABLY MED
PALM M THAT FOLLOW SIMILAR
CONSTRUCTION MECHANISM,
THEY START WITH PALM AS
LANGUAGE MODEL AND THEY
USE BIT FOR VISION ENCODER
AND DO LINEAR PROJECTION
AND THIS IS INTERESTING
BECAUSE IT GIVES A SIMILAR
SET OF TASKS THEY ARE
REPORTING ON AND ABLE TO
DO WITH SLIGHTLY DIFFERENT
VAR IANT OF WHAT EACH
COMP
COMPONENT LOOKS LIKE AND
ELIXIE
ELIXIR SIMILAR TO ENCODER
AND CHOOSE ARGUABLY MORE
POWERFUL PROJECTION
MECHANISM MORE SPECIFIC IN
SOME SENSE, THERE IS MED
FLAMINGO, COVER SIMILAR
SET OF IDEA AND IT IS MORE
POWERFUL PROJECTION
MECHANISM AND LOOK AT
SPECIFIC FORM OF DATA TO
PUSH THE PERFORMANCE
UPWARD AND MAYBE SORT OF
CLOSING UP HERE, I AM
AWARE OF TIME, WE MIGHT
START THINKING IN SAME WAY
WE THOUGHT ABOUT LLM,
MECHANISM FOR UNIVERSAL
STRUCTURING, HOW WE MIGHT
LOOK TO SORT OF TAKE MULTI
MODAL DATA AND ACROSS MANY
DOMAINS, MAP IT TO LIKE
THIS NATURAL LANGUAGE
EMBEDDING SPACE BECAUSE
THIS IS WHERE OCCURRENCELY
WE HAVE MOST SOPHISTICATED
CAPABILITIES.
BUT, YOU KNOW, IDEALLY THE
SENSIBLE GOAL OF DOING
SOMETHING LIKE THIS IS TO
EMPOWER A LOT OF THE OTHER
IN THIS SPACE TO DO MORE
THAN THEY HAVE BEEN ABLE
TO DO CURRENTLY WITH SOME
DATA THEY HAVE ACCESS TO.
AND SO MAYBE WE EMERGED
AND DONE A GUILTY WORK ON
SOME OF THESE OTHER
SPACES, AS WELL AND A LOT
OF EXPLORATION IS OPEN.
ONE THING I'VE HEARTENED
BY AND THINK ABOUT OR
THOUGHT ABOUT QUITE A BIT
IS WHAT ARE SOME OF THE
THINGS WE MIGHT WANT OUT
OF POPULATION MODEL,
HEALTH OR BIOMEDICAL
KNOWLEDGE LARGE-LANGUAGE
MODEL, TONS AND TONS OF
OPPORTUNITY WITHIN CONTEXT
OF PATIENT AND PATIENTSES
GENERATE DATA AND WHAT
WOULD SERIALIZED TOKEN
SEQUENCE LOOK LIKE?
WHAT DO THINGS LIKE
CONTINUING TO TRAIN FROM
GENERAL DOMAIN ON SOME OF
THESE HEALTH SPECIFIC DATA
LOOK LIKE.
TWO REAL QUESTIONS BEHIND
THIS, IF WE THINK ABOUT
SCALING FROM BEFORE, LIKE
WHAT IS MULTI MODAL HEALTH
SCALING LAW WE SEE AND
WEREN'T EXPECTING, THAT
WOULD BE QUITE EXCITING
AND IN SAME VEIN, WILL
THERE BE SORT OF EMERGENT
HEALTH CAPABILITY.
WITH THAT, I'LL AGAIN
THANK EVERYONE WHO HAS
BEEN INVOLVED OVER MANY
YEARS, THIS IS HUGE EFFORT
THAT CROSSED MANY
INSTITUTIONS AND MANY
PEOPLE, AS WELL.
SO I'LL STOP THERE AND
THANK YOU ALL.
>> THANK YOU SO MUCH FOR
THIS UNBELIEVABLY INTER
INTERESTING PRESENTATION.
INDEED WE ARE ALL LIVING
THROUGH THIS LLM
REVOLUTION AND IT IS VERY
EXCITING TO SEE THAT ALL
THIS APPLICATIONS YOU ARE
WORKING ON AND IN OUR
GROUP, WE HAVE ALSO WORKED
IN THE SIMILAR DIRECTION,
FOR EXAMPLE, WORKED ON
PROBLEM OF MATCHING
PATIENTS TO CLINICAL
TRIALS AND THE TOOL THAT
CAME OUT OF THE WORK IS
CA
CALLED TRIAL GPT, WE HAVE
ALSO WORKED ON MULTI MODAL
DATA, IN PARTICULAR
CONSIDERING OPHTHALMOLOGY
IMAGES WITHANNOTATION AND
HAD INCREDIBLE ADVANCES
AND AMAZING HOW MUCH YOU
CAN BUILD WITH THIS.
THANK YOU FOR SHARING YOUR
SIDE OF THE RESEARCH.
I HAVE A QUESTION FROM AN
O
AUDIENCE AND THE QUESTION
IS FROM JANE, IN THE BIRTH
ERA DOMAIN SPECIFIC
BIOBIRD, PALM MED BIRD ARE
IMPORTANT MODELS, THEY
PERFORM BETTER THAN
GENERAL DOMAIN
COUNTERPARTS IN THE ERA OF
LLMS WILL THERE BE DOMAIN
SPECIFIC MODEL SUCH AS
WIRE GPT 4 THAT CAN BE
BETTER IN BIOMEDICAL
TASKS?
WHAT IS YOUR TAKE ON IT?
>> I THINK THIS IS REALLY
OPEN QUESTION AND REALLY
INTERESTING ONE AULSZ.
I THINK THERE IS A COUPLE
OF DIFFERENT WAYS TO THINK
ABOUT THIS IN GENERAL, I
THINK ONE OF THEM IS MAYBE
QUALIFICATION OF LIKE WHAT
DO YOU THINK OF BIOGPT 4,
WOULD IT MEAN GPT-4 ONLY
TRAINED FROM HEALTH DATA
OR MEAN SOME SORT OF FORK
THAT BUILDS OFF THE BASIS
OF GPT-4 SPECIFICALLY FOR
HEALTHCARE OR MEAN
EXPOSING SOMETHING LIKE
GPT-4 TO HEALTHCARE DATA
IN WAY THAT SORT OF
PROVIDE FINETUNING
MECHANISM OVER TIME AND
DIVERGES FROM LIKE BASE
MODEL IN THAT SORT OF
CREATION.
SO THERE ARE COUPLE
DIFFERENT WAYS TO THINK
ABOUT OR ALTERNATIVELY
JUST USING GPT-4 ENTIRELY
ALONE AS IT IS IN SOME
SETTINGS.
I THINK THERE IS IT
POINTED TO OR ELUDED TO,
THERE IS TENSION BETWEEN
SOME OF THE LIKE SMALLER
DOMAIN-SPECIFIC MODELS WE
HAVE CONTROL OVER AND ARE
MORE LIKE MAYBE A LITTLE
BIT HARDER TO WORK WITH AT
TIMES, MAYBE MORE TASK
SPECIFIC AND SORT OF LIKE
THIS GENERAL CAPABILITY
AND SO THERE ARE A LOT OF
SYSTEMS YOU CAN IMAGINE
AROUND BASICALLY TRYING TO
FIRST WORK WITH ONE OF THE
SYSTEMS THAT IS MORE
SPECIFIC TO YOUR TASK AND
THEN BACKING OFF TO SORT
OF MORE GENERAL SYSTEMS.
YOU CAN MATCH IT THE OTHER
WAY OF MAYBE THE INTERFACE
IS ACTUALLY MORE GENERAL
SYSTEM AND THEN MORE
GENERAL SYSTEM IS WORKING
WITH AND IN COORDINATION
WITH NUMBER OF TOOLS THAT
ARE SPECIFIC TO THAT
SPACE.
THERE ARE A COUPLE
DIFFERENT FORMATIONS I'VE
HEARD ALL OF THEM HAVE AND
ARE COMPELLING AND HAVE
PROS AND CONS.
I THINK ONE OF THE MORE
INTERESTING THINGS IS THIS
NOTION AROUND MAYBE THE
THING THAT WE ALL ACTUALLY
ENJOY ABOUT USING GPT OR
FAVORITE LLM IS LIKE THIS
DISCOURSE IT PROVIDES AND
ENABLING IT WITH SOME C
CORRESPONDING TOOLS WE
WANT IN OUR SPECIFIC
DOMAIN MAYBE GENERAL PIECE
IS ACTUALLY THIS INTERFACE
THAT ALLOW US TO INTERACT
DIRECTORY WITH SPECIFIC
TOOLS RATHER THAN JUST
ALL-OUT REPLACEMENT.
THAT SAID, I ALSO, IT IS
HARD TO PREDICT THE FUTURE
IN ANY MEANINGFUL WAY, I
THINK MAYBE JUST CALLING
OUT WHERE I THINK I'VE
BEEN WRONG, I THINK FOR A
LONG TIME I REALLY HAD
REALLY RESISTED THIS IDEA
FOR BIOMEDICAL MODELS THEY
WOULD DO AS WELL AS THEY
DID FOR CLINICAL TASKS
RATHER BIOMEDICAL TASKS
AND WAS REALLY SURPRISED
TO FIND OUT THAT IN MANY
CASES JUST HAVING ACCESS
TO MORE DATA ALLOWED THEM
TO OUT PERFORM ON CLINICAL
TASKS SOME CLINICAL
SPECIFIC TOOLS THAT WE HAD
SEEN AND SO MAYBE WE'LL
SEE REPEAT OF THAT HERE,
AS WELL.
AND I CALL THAT ONE OUT
BECAUSE I HELPED ORGANIZE
CLINICAL NATURAL LANGUAGE
WORKSHOP AT ACL VENUE,
WHICH IS DIFFERENT FROM
B
BIONLP WORKSHOP ALONG WHAT
TEXT DATA ARE YOU LOOKING
AT AND SOME DAY MAYBE
THEY'LL BE THE SAME
BECAUSE HOW MUCH DATA YOU
HAVE RATHER THAN SPECIFIC
TYPES OF DATA.
>> SO IT IS A LOT UP TO
HUMAN IMAGINATION HOW WE
CAN USE THESE MODELS AND
STACK THEM.
IT IS TRULY INCREDIBLE
SOMEBODY CAN GET ALL
FIGURES AND FROM THE PALM
MED CENTRAL PAPERS AND USE
IN SUCH WAY TO ELEVATE THE
MODEL TO LEVEL THAT WE SEE
TODAY.
WELL, WE HAVE COUPLE MORE
MINUTES, IF I COULD ASK MY
QUESTION.
I HAVE QUESTION ABOUT THE
PROMPTS.
IT IS INTERESTING, I GOT
INTERESTING PERSPECTIVE
FROM YOUR TALK THAT
REPROMPTING IS A GOOD
TECHNIQUE.
SO IF DO YOU HAVE ANY
PRACTICAL SUGGESTIONS OF
HOW TO PROMPT OR WHAT IS
THE BEST WAY OF PROMPTING
THAT HAS WORKED FOR YOU?
>> YEAH, SO THIS IS ALSO
PRETTY OPEN.
I THINK MED PROMPT WORK
AND THERE IS CORRESPONDING
RESOURCE FOR MED PROMPT
WORK CALLED PROMPT BASE ON
GITHUB THAT PROVIDES CODE
TO RUN SOME SAME
REPROMPTING EXERCISES.
I THINK IN SOME WAYS, SOME
OF THE IT DEPENDING ON
YOUR DOWN-STREAM TASK,
WHAT ARE YOU ASKING IT TO
DO, FOR CLASSIFICATION
TYPE PROBLEMS OR SOME LIKE
CLASSIC SORT OF QA STYLE
PROBLEMS, ONE THING
HELPFUL WAS THIS LIKE KNN
FEW SHOT PROMPTING AND BY
THAT I MEAN, IF YOU KNOW
WHAT THE UNDERLYING
QUESTION IS GOING TO BE
AND YOU'RE ABLE TO PROVIDE
SOME NOTION OF SIMILARITY
OF OTHER LIKE TWO OTHER
QUESTIONS, CAN YOU SORT OF
INTELLIGENTLY SELECT WHAT
QUESTIONS YOU PROVIDE IN
CONTEXT IN POSITIVE
EXAMPLE AND NEGATIVE
EXAMPLE IF THAT IS THE
TYPE OF CLASSIFICATION YOU
ARE DOING AND PROVIDE
THOSE AS CONTEXT SO YOU
CAN HELP ELIS EIGHT THE
POSITIVE AND NEGATIVE
EXAMPLE AND THAT CONTEXT
AND GUIDE PROMPT TO
PROVIDE RIGHT ANSWER FOR
SPECIFIC QUESTION YOU
HAVE.
THERE ARE THINGS LIKE THAT
THAT ARE INTERESTING AND
ALSO LIKE FOR SELF-
SELF-VERIFICATION PIECE,
NOEGSZ NOTION I DISCUSS
AROUND DOING MANY TIME
REPROMPTING AND
POTENTIALLY WITH DIFFERENT
AGENTS THAT HAVE DIFFERENT
CONTEXTS CAN BE
INTERESTING IN IDENTIFYING
IS THIS THING ACTUALLY
REAL OR IS THIS
HALLUCINATION BECAUSE
SEVERAL OF THE METHODS
SEEM TO BE VERY GOOD AT
SELF-DETECTING
HALLUCINATION.
>> I PERSONALLY FIND
PROMPTING INTRIGUING, IT
MAKES YOU ACTUALLY THINK
ABOUT HOW HUMANS THINK.
IF YOU CAN EXPLAIN TO
YOURSELF WHAT IS THE
THOUGHT PROCESS YOU GO
THROUGH TO MAKE A
DECISION, WHICH VERY
FREQUENTLY WE DO
INTUITIVELY, ARE TWO
SENTENCES RELATED?
DECIDE WHETHER THEY ARE
RELATED OR NOT, THIS WILL
NOT, WORK WITH CHAT GPT,
GIVE EXAMPLE HOW TO JUDGE
TWO SENTENCES AND HELPFUL
TO UNDERSTAND HOW WE THINK
ABOUT PROBLEMS AND
NORMALIZE HUMAN THINKING
PROCESS AND PROVIDE THIS
AS DIRECTIONS TO LLMS.
>> YEAH.
TALK ABOUT THIS THIS TIME,
THERE IS WHOLE VARIETY OF
WORK IN RETRIEVAL
AUGMENTED GENERATION,
CONSTRAINED DECODING COME
FROM LIKE CODE CREATION
LITERATURE, BOTH OF WHICH
ARE FASCINATING FOR HOW
CAN WE ALSO NOT JUST TUNE
PROMPT SIDE ALTER OUTPUT
SIDE, AS WELL AND COME UP
WITH USEFUL RESPONSES.
SGLL YEAH, UNFORTUNATELY
IT IS 3 O'CLOCK AND I HAVE
TO LET YOU GO.
THANK YOU SO VERY MUCH,
THIS IS VERY INTERESTING,
THIS TALK IS RECORDED AND
WHOEVER DIDN'T HAVE A
CHANCE WE'LL BE SHARING
THIS LINK AND I'M SURE
I'LL COME BACK TO THIS
LINK FOR THIS PRESENTA
PRESENTATION.
THANK YOU SO MUCH, THIS
WAS WONDERFUL, THANK YOU
FOR YOUR TIME.
>> THANK YOU AGAIN FOR
HAVING ME.
>> OKAY, BYE.