WELCOME TO THE CLINICAL CENTER GRAND ROUNDS,
A WEEKLY SERIES OF EDUCATIONAL LECTURES FOR PHYSICIANS AND
HEALTH CARE PROFESSIONALS BROADCAST FROM THE CLINICAL
CENTER AT THE NATIONAL INSTITUTES OF HEALTH IN
BETHESDA, MD.
THE NIH CLINICAL CENTER IS THE WORLD'S LARGEST HOSPITAL TOTALLY
DEDICATED TO INVESTIGATIONAL RESEARCH AND LEADS THE GLOBAL
EFFORT IN TRAINING TODAY'S INVESTIGATORS AND DISCOVERING
TOMORROW'S CURES.
LEARN MORE BY VISITING US ONLINE AT HTTP://CLINICALCENTER.NIH.GOV
GOOD AFTERNOON AND WELCOME TO
TODAY'S CLINICAL CENTER GRAND
ROUNDS.
THE HOPKINS CME ACTIVITY CODE
FOR TODAY'S GRAND ROUNDS IS
57875.
PLEASE TEXT THIS CODE TO JOHNS
HOPKINS CME PHONE NUMBER ON THE
SLIDE TO RECEIVE CME CREDIT FOR
THIS LECTURE.
WE KIND YOU ASK YOU TO PROVIDE
FEEDBACK TO TODAY'S LECTURE
USING THE QR CODEOT SLIDE.
YOU WILL RECEIVE A SURVEY LINK
VIA E-MAIL, AND IT WILL BE
PROVIDED WITH IMPORTANT FEEDBACK
ABOUT THIS PRESENTATION AND
ALLOWS TO SUBMIT QUESTIONS FOR
FUTURE GRAND ROUND TOPICS.
FOLLOWING THE QUESTIONS FROM THE
SPEAKERS, WE CAN ALSO TAKE WEB
CONNECTEDS VIA THE LIVE FEEDBACK
BUTTON ON THE VIDEOCAST.
QUESTIONS WILL BE TAKEN AS TIME
PERMITS AT THE CONCLUSION OF THE
PRESENTATION EMPLOY ITED WE ARE
HONORED TO HAVE DR. HARRY
BCRUCIATE LIGAMENT RKE, PATIENT
SAFETY AND QUALITY ACADEMIC
COLLABORATIVE AND CHIEF IN THE
SECTION OF SAFETY, QUALITY AND
VALUE IN THE DEPARTMENT OF
MEDICINE AT THE F. EDWARD ABER
SCHOOL OF MEDICINE UNIVERSITY OF
HEALTH SCIENCES.
DR. BURKE EARNED HIS
UNDERGRADUATE, DOCTORAL AND
MEDICAL DEGREES AT THE
UNIVERSITY OF OHIO.
HE IS NOW PROFESSOR OF MEDICINE
AND DATA SCIENCE.
IN 1992 DR. BURKE BEGAN HIS
FOUNDATION OF IMPLEMENTING
ARTIFICIAL INTEL JOHNSON
NETWORKS WHILE THE ASSOCIATE
DIRECTOR OF THE CENTER FOR
BIOMEDICAL RESEARCH AT THE
UNIVERSITY OF NEVADA RENO.
IN 19 HE WAS 1 OF THE FIRST TO
PUBLISH ON USING ARTIFICIAL
NEURAL NETWORKS IN CANCER.
IN 1995, HE DEMONSTRATE THAD
ARTIFICIAL NEURAL NETWORKS WERE
MORE POWERFUL THAN
PREDECLARATION METHODS IN
CANCER.
IN ADDITION TO HIS WORK IN
ARTIFICIAL NEURAL METE YORKS FOR
THE PAST 6 YEARS, HE HAS BEEN
DEVELOPING NATURAL LANGUAGE
PROCESSING PROGRAMS, HIS
INTEREST IN ARTIFICIAL
INTELIENCE AND IN NATURAL
LANGUAGE PROCESSING HAS COME
TOGETHER IN GENERATIVE
ARTIFICIAL INTELLIGENCE LARGE
LANGUAGE MODELS.
DR. BURKE HAS WRITTEN NUMEROUS
BOOK KRAPTERS IN OVER 80 PEER
REVIEWED SCIENTIFIC PAPERS.
HE IS A REVIEWER FOR NUMEROUS
HIGH IMPACT JOURNALS AND MEMBER
OF THE EDITORIAL BOARD FOR THE
JOURNAL OF NATIONAL CANCER
INSTITUTE AND NATIONAL CANCER
SOCIETY'S JOWRN OF OF CANCER AND
RECEIVED AN AWARD FOR BEING A
DISTINGUISHED REVIEWER BY THE
JOURNAL OF AMERICAN MEDICAL
ASSOCIATION ONCOLOGY.
THE TITLE OF HIS PRESENTATION
TODAY IS GENERATIVE ARTIFICIAL
INTELIENCE AND THE FUTURE OF
MEDICINE.
NOW JOIN ME IN WELCOMING OUR
GRABBED ROUNDS SPEAKER DR. HARRY
BURKE.
[ APPLAUSE ]
>> THANK YOU DOCTOR.
OKAY.
SO WE'RE GOING TO JUST DIVE
RIGHT INTO THIS THING, IT'S A
PRETTY PRETENSION TITLE BUT I
THOUGHT IT WAS A LOT OF FUN.
I HAVE THE USUAL DISCLOSURES TO
DECLARE, SO, NOTHING VERY
INTERESTING THERE.
SO WE WILL TART BY LOOKING INTO
THE FUTURE A SECOND AND I WILL
ASK YOU A QUESTION.
WHICH OF THESE 3 FUTURES, DO YOU
THINK THAT MEDICINE WILL BE?
DO YOU THINK THAT WE'RE GOING TO
IGNORE AI AND HOPE FOR THE BEST?
THAT BOTHLET PHYSICIAN AND THE
PATIENTS ARE GOING TO USE AI?
OR THAT THE PATIENT'S GOING TO
INTERACT DIRECTLY WITH AI?
NOW THERE ARE OTHER CHOICES BUT
THESE ARE 3, SO JUST FOR A SHOW
OF HANDS, HOW MANY OF YOU
BELIEVE NUMBER 1?
ANYBODY BELIEVE THAT.
OKAY, SO NUMBER 2, THE PHYSICIAN
AND THE PATIENT ARE BOTH GOING
TO USE AI?
OKAY, AND NUMBER 3, THE
PATIENT'S GOING TO COMMUNICATE
DIRECTLY WITH AI?
OKAY.
OKAY, GOOD, SO IT'S NUMBER 2,
NOW THERE'S 1 LITTLE ISSUE
THERE, THAT IS YOU SHOULD NOTICE
THAT THESE ARE 2 DIFFERENT
SYSTEMS.
MEDICAL AI WILL BE DIFFERENT
THAN A GENERAL AI SYSTEM.
SO 1 OF THE QUESTIONS THAT'S
GOING TO COME UP IN THIS
SCENARIO IS THAT THEY DON'T
MATCHUP.
OKAY, SO THE SYSTEM YOU
DESCRIBED ISN'T GOING TO
MATCHUP, OKAY, SO WHAT IS
GENERAL?
SO I WILL GO QUICK 3 THROUGH
THIS.
SO ARTIFICIAL INTELLIGENCE
EMULATES HUMAN INTELLECTUAL
CAPABILITIES AND THERE THERE'S
SYMBOLIC AI AND STATISTICAL AI
AND AI SYSTEMS INTERNALLY
MANIPULATE SIM BOOM BOOM BOOM
BOOMS AND WILL COME AROUND ANDY
WOO WILL TALK ABOUT STATISTICAL
AI BECAUSE THAT'S WHAT A GPT AND
NLM AND A GEN AI IS.
SO JUST REALLY QUICKLY, GEN AI
IS A GENERATIVE MODEL SO WE
DIDN'T HAVE GENERATIVE MODELS
BEFORE, AND SO THE BIG
REVOLUTION IS THAT IT COULD
ACCEPT AND PRODUCE TAXED MUSIC,
COMPUTERS, STUFF, AND GENERATIVE
PRETRAINED TRANSFORMER WAS
CREATED BY GOOGLE AND THAT'S
THESE PEOPLE HERE AND IT WAS
POPULARIZED BY OPEN AI, PRETRAIN
MEANS LOTS OF AMOUNTS OF DATA,
TRANSFORMER IS THE ALGORITHM
THEY DESCRIBE HERE, AND LARGE
LANGUAGE MODELS ARE GEN AI
PROGRAMS AND SO IN FACT, YOU CAN
USE THEM ISHT CHANGEABLY, AI AND
LARGE LANGUAGE MODELS UNLESS
YOU'RE TALKING ABOUT VLM, WHICH
IS A VISUAL LANGUAGE MODEL WHICH
IS GEN AI.
OKAY, AND THERE ARE MANY
TRANSFORMER LIKE PROGRAMS OUT
THERE, A NUMBER OF PUBLICATIONS
LOOKING AT GEN AI, ARTIFICIAL
INTELLIGENCE AS YOU CAN SEE,
RIGHT AROUND HERE WHEN VASWANI
PUBLISHED A PAPER, THERE'S BEEN
AN EXPOTENTIAL INCREASE, WE
STARTED WAY BACK OVER HERE WHEN
IT WAS A LITTLE TINY THING AND
NOW IT'S A HUGE THING.
OKAY, SO WHAT DOES LLM RELY
UPON, I WANT TO GET THIS OUT IN
THE BEGINNING, IT HAS 3 POWER
IDEAS, IT HAS SEQUENCE.
WORDS IN A SEQUENCE ARE
ASSOCIATE WIDE A MEANING.
PRETTY STRAIGHT FORWARD.
TWO, REPETITION, MORE OFTEN THE
SEQUENCE OCCURS IN THE DATA THE
MORE LIKELY SAID TO BE THE
CORRECT MEANING AND 3
SELF-ORGANIZATION RELIES ON THE
NEURAL NETTO BE AUTONOMOUS
SELF-ORGANIZING.
NEURAL NET.
OKAY, NOW.
MOST PEOPLE THINK WHEN THEY
THINK ABOUT-ABOUT AI AND WE USE
THAT IN OUR STATISTICS, WE USE
LOGISTIC REGRESSION, WE ARE
USING CONDITIONAL PROBABILITY,
RIGHT?
THE PROBABILITY OF WHY GIVE IT
X.
BUT WHAT YOU HAVE TO UNDERSTAND
IN GENAI IS THAT THERE'S 2
PROBABILITIES, THERE'S THE JOIPT
PROBABILITY AND CONDITIONAL
PROBABILITY, SO WE WILL TALK
ABOUT LATER THE, THE JOINT
PROBABILITY IS THE ENGINE FOR
LARGE LANGUAGE MODEL AND THE
CONDITIONAL PROBABILITY IS THE
OUTPUT OF THE MODEL.
THIS THING ISN'T WORKING.
AH, OKAY, SO, SO PEOPLE THINK
THAT THERE'S A COUPLE LLMs OUT
THERE, THAT'S NOT THE CASE.
AFTER VIKWANI'S PAPER CAME OUT,
THE LITERATURE BECAME HUGE VERY,
VERY QUICKLY.
THERE ARE HUNDREDS OF LLM MODELS
DEVELOPED AFTER VISWANI'S PAPER
CAME OUT, LITERALLY HUNDREDS OF
THEM.
AND WHAT WE SEE TODAY, IF I CAN
GET THIS THING TO WORK, OKAY,
WHAT AM I, I GOTTA GO BACK.
WHAT WE SEE TODAY ARE THESE ARE
THE BIG MONEY 1S, THESE ARE THE
1S THAT WHERE THERE'S HUNDREDS
OF MILLIONS OF DOLLARS BEHIND
TTHIS WAS THE START OF THE
PAPER, AND WHAT'S INTERESTING IS
THAT ALL OF THESE PLAYERS, OKAY,
CREATED LARGE LANGUAGE MODELS
AND IN FACT, THE METAMODEL CAME
OUT 14 DAYS BEFORE CHAT GPT 3.5
AND IT WAS WITHDRAWN WITHIN 7
DAYS.
OKAY, AND NOW, OPEN AI CAME OUT
WITH THIS PRODUCT, IT WASN'T THE
FIRST.
TWENTY-TWO OUT OF 22 HAD NO
MONEY.
REVENUE THIS YEAR, 3.7 BILLION.
AS FAR AS I KNOW, NO COMPANY
LESS THAN 2 YEARS OLD OKAY, AND
IT IS STILL LESS THAN 2 YEARS
OLD HAS GENERATED 3.7 BILLION
DOLLARS IN REVENUE, IT IS
ABSOLUTELY AMAZING.
[MUMBLING ] OKAY, SO, WHEN WE
LOOK AT THESE THINGS WE CAN SEE
THE AMOUNT OF MONITOR THAT'S'
BEING SPENT ON THESE THINGS AND
I JUST WANT TO POINT OUT THE
MERCURY PROGRAM THAT SENT A MAN
IN SPACE IN CURRENT DOLLARS THEY
SPENT 44 BILLION IN CURRENT
DOLLARS TO PUT A MAN INTO SPACE,
WE'VE ALREADY SPENT MUCH MORE
THAN IT TOOK TO PUT A MAN IN
SPACE ON LLMs.
LLMs ARE EXPENSIVE, THESE ARE
THE TOKEN RATES, TOKEN RATES ARE
COMING DOWN BUT THEY'RE STILL
EXPENSIVE, THEY CONSUME HUGE
AMOUNTS OF ENERGY, THEY ARE
RESTARTING 3-MILE ISLAND NUCLEAR
REACTOR, MICROSOFT JUST TO PAY
PER SOME OF THE ENERGY USE.
AND THE COST OF CREATING THESE
THINGS IS TREMENDOUS, OKAY, THE
ORIGINAL 1 COSTS A BILLION
DOLLARS TO CREATE, OKAY, IT
STILL COSTS A BILLION DOLLARS TO
CREATE, AND OPEN AI'S COST, THEY
JUST DID A FOUNDING ROUND
BECAUSE THEY'RE DAILY ENERGY
EXPENTINE REGIMENITTURE IS 7 HHV
EXPEBDITTURE IS $700,000 A DAY.
OKAY, SO HOW DO THEY WORK?
THERE'S 3 ASPECTS.
THE HUGE AMOUNT OF DATA, THE
DANZA FORMER AND THE FINE
TUNING.
THE PREPROCESSING IS NECESSARY,
SO YOU HAVE TO DO A LOT OF STUFF
WITH WORDS, YOU HAVE TO
TRUNCATE, YOU HAVE TO DO A WHOLE
BUNCH OF STUFF, BUT THE MAIN
POINT OF THE PREPROCESSING, IS
THIS WORD [INDISCERNIBLE], WORD
[INDISCERNIBLE] WAS AN
ABNORMALITIES SELECT REVOLUTION.
WITHOUT IT NONE OF THIS WOULD
HAVE HAPPENED.
WHAT IT DID IS THIS WORK WORKS
ON VECTORS AND NUMBERS, IF YOU
DIDN'T 1 ABOUT WORD 2 VEC, YOU
COULD NOT CREATE THE SYSTEM,
WITH WORD 2 VEC, YOU CREATE
NUMBERS AND THAT GIVES RISE TO
SOMETHING YOU CAN PUT INTO A
NEURAL NET, YOU CAN'T PUT WORDS
INTO A MURAL NET, YOU CAN ONLY
PUT NUMBERS IN, SO WORD 2 VEC
ALLOWED US TO PUT WORD INTO THE
TRIEWRAL NET.
THE TRANSFORMER HAS 3
COMPONENTS, IT HAS GENERATIVE,
OUTPUT, CONDITIONAL PROBABILITY
AND SELF-ORGANIZING NEURAL NETS,
45 TERRA BYTES OF DATA, THERE'S
NEVER BEEN AS FAR AS I KNOW, I
COULD BE WRONG 45 TB IN 1
INTEGRATED SYSTEM?
, YEAH, THERE HAVE BEEN
DATABASES, LARGE DATABASES BUT 1
INTEGRATED SYSTEM, OKAY?
SO 175 BILLION PARAMETERS, THOSE
ARE JUST THE NEURAL NET
PARAMETERS, 64 BIT FLOATING
POINT DOUBLE PRECISION.
THE PROBABILITIES THAT AM CAN
OUT OF THIS THING ARE VERY, VERY
SMALL.
IT'S LIKE .0000, ALL THE WAY, 1.
OKAY, SO THE POINT IS THIS IS
TRYING TO MAKE DISTINCTIONS ON
VERY SMALL NUMBERS WHICH IS WHY
IT NEEDS 64 BIT FLOATING AND
RUNS ON SERVERS, AND WHY DID
CHAT GPT WIN AND MICROSOFT LOSE
WITH GALACTICCA?
BECAUSE OPEN AI DID SOMETHING
VERY SMART.
THEY DID HUMAN FINE TUNING.
THEY HAD HUNDREDS OF THOUSANDS
OF MAN HOURS OF FINE TUNING,
WHERE THEY JUST TOOK THE SYSTEM,
LOOKED AT THE OUTPUT AND CHANGED
THE OUTPUT IN A WAY THAT MADE
SENSE, SO IT DIDN'T GET THIS
NONSENSE OUTPUT, THAT WAS THE
TRICK.
GALACTICCA WHEN THE OUTPUT CAME
OUT, IT WAS GARBAGE.
SO THE FINE TUNING WAS THE KEY,
BUT MAKE NO MISTAKE, THIS SYSTEM
THAT OPEN AI CAME UP WITH IS A
MONUMENTAL ACHIEVEMENT,
MONUMENTAL.
THAT STUFF WORKS.
OKAY, SO WHAT HAPPENS?
SO REAL QUICK, YOU TAKE YOUR
DOCUMENTS, THEY'RE WORDS, YOU DO
WORD 2 VEC, YOU TURN THEM INTO
VECTORS, YOU PUT THEM IN THE
NEURAL NET, WHICH IS DEEP
ENCODING AND DECODING USING
QUERY KEYS AND VALUES, CAN I
WILL MENTION THAT LATER AND THE
GENERATIVE MODEL, OKAY, AND THEN
YOU GET THE PROBABILITIES, AND
THEN IT'S SELECT, USUALLY THE
HIGHEST PROBABILITY AND USUALLY
IS A VERY GOOD THING.
OKAY, SO ATTENTION, IT'S A VERY
HOLD CONCEPT, IT ENCODES AND
DECODES, THEY DIDN'T COME UP
WITH IT, BUT WHAT THEY DID WAS
THEY CAME UP WITH THIS FORMULA,
MAPPING, QUERY AND A SET OF KEY
VALUE PAIRS TO AN OUTPUT AND SO
THESE 3 CHARACTERISTICS ARE WHT
THE VECTORS USE.
THEY ORGANIZE EVERY VECTOR IN
TERMS OF THESE 3 PARAMETERS,
OKAY IN AND THEN THE OUTPUT IS
COMPUTED AS THE WEIGHTED SUM OF
THOSE VALUES, SO THAT'S HOW THE
ACTUAL THING WORKS, TRANSFORMER
ITSELF, OKAY, IS THE
ARCHITECTURE LOOKING AT THE
GLOBAL DEPENDENCIES, WHAT IT'S
DOING IS USING NEURAL NETS TO
MAP A SEQUENCE OF INPUT TO A
SEQUENCE OF OUTPUT VECTORS,
THAT'S WHAT IT DOES, OKAY, SO WE
IMPLEMENTED NEURAL NETS A LONG
TIME AGO, YOU HEARD ABOUT THAT,
THEY'RE VERY GOOD, AND THE
REASON THEY'RE VERY GOOD AND SO
HERE'S THE 2 CLASS SEPARATION
PROBLEM, AND THE NEURAL NETWILL
DO THE 2 CLASS SEPARATION
PROBLEM BEAUTIFULLY, WHY IN
WELL, THIS IS WHAT ATRACKED ME
TO NEURAL NETS IN EARLY 1990S
WAS THIS THEOR UMKC, IT'S CALLED
THE UNIVERSAL PROXIMITY TO THEOR
UMKC, IT SAYS WITH A 3 LAYER
BACK PROP, KNOWN THAT KNOW IT
CAN FIT ANY CONTINUOUS FUNCTION
AND GIVEN THAT THE PHENOMENON IS
IN THE DATA AND THAT THERE'S
SUFFICIENT DATA IT'S GUARANTEED
TO FIND THE PHENOMENA.
THAT'S A VERY, VERY POWERFUL
THEOR UMKC AND THAT'S THE THEOR
UMPIRES THAT GOT ME INVOLVED IN
NEURAL NETS BECAUSE THIS IS THE
ONLY METHOD THAT I KNOW THAT'S
GOOD NIGHTED TO FIND THE
PHENOMENA THAT'S IN THE DATA.
OKAY, NOW SOME PEOPLE HAVE ASKED
MAYBE IT'S CONTAMINATION, MAYBE
IT'S JUST REPEATING STRINGS OF
WORDS?
SO YOU PUT IN STRINGS OF WORDS,
IT PUTS OUT STRINGS OF WORDS SO
IT'S JUST LIKE A SIMPLE MATCHING
TAVENG, MATCHING IT UP, SO THEY
DID A STUDY, AND THEY TOOK OUT
SIMILAR STRINGS OF WORDS, AND IT
DIDN'T EFFECT OUTPUT, BECAUSE
IT'S NOT REPORTING SPRINGS OF
WORDS, IT'S ACTUALLY DOING THE
JOINT PROBABILITY.
OKAY SO WHY DOES IT NEED 45
TERRA BYTES OF DATA, IT'S HUGE,
WELL BECAUSE IT NEEDS THE BREDTH
OF INFORMATION, NEEDS TO KNOW A
LOT, DEPTH OF INFORMATION,
ESSENTIAL DETAILS AND
CRITICALLY, IT NEEDS REPETITION,
AND REPETITION IS THE KEY TO
TRUTH, OKAY?
NOW SO IT'S NOT HHV IT'S NOT
JUST FINDING THINGS, AND SAYING
THAT'S WHAT IT IS, BUT IT DOES
NEED REPETITION, WE'LL TALK
ABOUT THAT LATER, SO THERE'S 2
TYPES OF GENAI.
THERE'SUNE MODAL AND MULTIMODAL.
MULTIMODAL IS COMING UP, THIS IS
WHAT LLM IS, WE WILL CALL
EVERYTHING FROM NOW ON LLM,
MULTIMODAL IS IMAGE TO TEXT,
TEXT TO IMAGE, AND CREATE
IMAGES, WE'VE ALL SEEN THAT.
YOU TAKE, YOU TELL IT WHAT YOU
WANT IN TEXT, AND IT CREATES AN
IMAGE THAT'S MULTIMODAL.
SO LET'S TALK ABOUT THE DATA FOR
A MOMENT.
FORTY-FIVE TERRA BYTES OF DATA,
HUGE AMOUNT OF DATA, IT IS 11
AND HALF MILLION VIABLES, 25,000
ENCYCLOPEDIA PRIMATES TANNICCAS.
NOW THIS IS A REALLY, REALLY
IMPORTANT IDEA AND I'M SURPRISED
PEOPLE HAVEN'T REALLY MENTIONED
IT IN THE PAST AND THAT IS THAT
IT'S POWER RISES FROM
INTEGRATION OF INFORMATION, SO
THIS IS LOOKING AT INFORMATION,
SO UNINTEGRATED DATA, LIBRARY OF
ALEX ABDOMEN RIA, BLAH, BLAH,
BLAH, SEMIINTEGRATED, GOOGLE
DOES SEMIINTEGRATION BY LOOKING
AT INDEXING WEBSITES, BUT I HAVE
TO ASK YOU A QUESTION, LET'S
JUST LOOK AT A BOOK ON THIS
SHELF.
IS THIS BOOKING TO THAT BOOK, IS
THAT BOOK TALKING TO THAT BOOK,
ARE THESE BOOKS TALKING TO EACH
OTHER, ARE ANY OF THESE BOOKS
ACTUALLY COMMUNICATING WITH EACH
OTHER?
AND THE ANSWER IS NO.
WHAT A GPT DOES IS IT ALLOWS,
ALL OF THESE BOOKS TO TALK TO
EACH OTHER FOR THE FIRST TIME
EVER, OKAY?
AND THERE DOES NOT APPEAR TO BE
ANY NATURAL LIMIT TO THE AMOUNT
OF INFORMATION THAT GPT COULD
ORGANIZE, AND SO I'M GOING TO
MAKE A COUPLE STATEMENTS HERE, 1
IS, MOST MEDICAL INFORMATION'S
NOT READILY AVAILABLE FOR
CLINICAL USE, IT'S IN A
TEXTBOOK, IT'S AT HARRISON'S,
IT'S WHEREVER AND IN LARGE
KNOWLEDGE MODELS WHICH IS WHAT I
CALL THEM IN THIS CONTEXT IT
MAKE ALL MEDICAL INFORMATION
AVAILABLE IN FACT, MY ASSERTION
IS THAT EVERYTHING THAT IS KNOWN
HAS BECOME KNOWABLE BY GPTs.
THAT'S AN ASTOUNDING CLAIM AND
AN ASTOUNDING ADVANCE IN HUMAN
CIVILIZATION.
SO THE STRENGTH AND LIMITATIONS,
STRENGTH, WELL, YOU KNOW IT
CONTAINS AN IMMENSE AMOUNT OF
ORGANIZATION, IT DOES
PERFORMICISMEM MANIPULATION,
THAT WILL BECOME IMPORTANT LATER
ON, IT ALLOWS NATURAL LANGUAGE
INPUT AND OUTPUT AND IT CAN
PROVIDE AN EXPLANATION OF THE
OUTPUT AND MAKE INFERENCES BUT
BECAUSE IT COMBINES MANY SOURCES
OF INFORMATION, DOESN'T PROVIDE
A DIRECT REFERENCE, BY
DEFINITION, IT CAN'T PROVIDE A
DIRECT REFERENCE BECAUSE IT IS
IN FACT INTEGRATING ALL THAT
INFORMATION, IT'S A STATIC MODEL
AND MEANS THAT IT DOESN'T LEARN
ON THE FLY SO TO SPEAK.
YOU HAVE TO PUT DATA IN IT.
IT REQUIRES THE USER HAVE DOMAIN
KNOWLEDGE AND PROMPT EXPERTISE
AND YOU HAVE TO ASSESS THE
VERACITY OF THE ANSWER.
OKAY, SO, WHAT ARE THE GOALS OF
A GENAI ANSWER, THAT IT'S
CORRECT, INFORMATION'S ACCURATE,
IT'S COMPREHENSIVE AND THAT IT'S
EFFICIENT.
THOSE ARE THE 3 THINGS THAT YOU
WANT TO LOOK FOR IN ANY ANSWER
FROM A GPT.
SO HOW DO YOU MAKE IT DO WHAT
YOU WANT IT DO.
SO YOU CAN USE A FOUNDATIONAL
MODEL, YOU CAN USE A TASK
SPECIFIC MODEL AND THERE'S A LOT
OF MEDICAL MODELS BEING
DEVELOPED RIGHT NOW, OKAY?
THEY'RE GOING TO COME OUT AND
THERE'S HUNDREDS OF COMPANIES
WORKING ON MEDICAL LLMs RIGHT
NOW, OKAY, SO, SO YOU HAVE THESE
THINGS AND IF YOU WANTED TO DO
THOSE, YOU CAN PRETRAIN IT TO DO
WHAT YOU WANT IT TO DO, YOU CAN
FINE TUNE IT WHICH MEANS THAT
YOU TAKE DATA AND YOU PUT IT IN
AND IT'S USUALLY REINFORCEMENT
LEARNING, OKAY?
OR, YOU CAN USE AN EXISTING
GENAI MODEL, YOU CAN DO
IN-PROMPT QUERY AND YOU GIVE US
INFORMATION AND ASK A DETAIL,
WHICH IS DEDUCTIVE REASONING OR
IN-CONTEXT LEARNING WHICH IS YOU
GIVE IT A WHOLE BUNCH OF
INFORMATION AND YOU ASK IT FOR
GENERALIZATION SO IT DOES
DEDUCTIVE AND INDUSTRIOUSIVE,
YOU WILL DO DATABASE RETRIEVAL
BUT WHAT'S REALLY COMING UP IS
RAG.
RAG IS REALLY IMPORTANT BECAUSE
WHAT RAG ALLOWS YOU TO DO IS IT
ALLOWS YOU TO PUT IN A PROMPT
AND THEN THE PROMPT GOES TO YOUR
DATA, RAG IS BASICALLY A
VECTORIZED DATABASE OF
INFORMATION YOU PUT IN THAT YOU
FEEL IS VALID INFORMATION AND IT
WILL EXTRACT THE INFORMATION
FROM THE RAG, AND THEN GIVE IT
TO THE LLM, AND THE LLM WILL PUT
IT TOGETHER, THAT MEANS THAT YOU
CAN CHEAPLY, ADD INFORMATION TO
THE SYSTEM WITHOUT HAVING TO
PRETRAIN OR FINE TUNE WHICH ARE
BOTH VERY EXPENSIVE AND YOU CAN
GIVE IT YOUR INFORMATION.
SO HOW DOES IT DETERMINE THE
BEST ANSWER, YOU KNOW THAT'S A
REALLY INTERESTING QUESTION.
WELL, IT TURNS OUT THAT IT
RELIES ON THE IDEA THAT
INCORRECT INFORMATION WAS
WRITTEN IN MANY DIFFERENT WAYS
AND TRUE INFORMATION'S WRITTEN
IN THE CONSISTENT WAY, SO IF I
ASK YOU, WELL, WHAT'S THE MOON
MADE OF, A LOT OF PEOPLE CAN
SAY, LIKE BLUE CHEESE, SWISS
CHEESE, ALL KINDS OF STUFF BUT
THE 1 CONSISTENT ANSWER IS YOU
KNOW IT'S MADE OF ROCK, IT'S
280,000-MILES AWAY AND, ET
CETERA, ET CETERA AND THAT WILL
BE REPEATED OVER AND OVER AGAIN.
SO THEY SAY INTUITIVELY, THE
REASON MAJORITY IMPROVES
PERFORMANCE IS THAT WHILE THERE
ARE MANY WAYS TO ANSWER
QUESTIONS INCORRECTLY, THERE ARE
FEW WAYS TO ANSWER INCORRECTLY
AND IN FACT, THAT'S ACTUALLY A
VERSION OF SCIENTIFIC TRUTH BY
THE WAY, WHICH IS INTERESTING
AND THEN PEOPLE TALK ABOUT DEEP
FAKES, DEEP FAKES SHOULDN'T BE A
PROBLEM BECAUSE DEEP FAKES ARE
GOING TO OCCUR RARELY UNLESS,
THEY'RE FINE TUNED INTO THE
SYSTEM OR UNLESS, SOMEBODY GETS
A HOLD OF A TON OF COMPUTERS AND
SOMEHOW UPDATES A FOUNDATIONAL
MODEL WITH THAT DEEP FAKE.
OTHERWISE, DEEP FAKE SHOULD NOT
BE AN ANSWER THAT AN LLM GIVES
YOU.
ALL RIGHT, SO, HOW DO YOU
DETERMINE IF ITS ANSWER IS THE
BEST ANSWER?
THIS IS AN IMPORTANT IDEA.
SO IT GIVES YOU AN ANSWER AND
YOU WANT TO USE YOUR EXPERTISE
AND MEDICAL JUDGMENT, THAT'S
REALLY IMPORTANT.
IN OTHER WORDS IF YOU ASK A
QUESTION, YOU SHOULD HAVE THE
EXPERTISE TO UNDERSTAND THIS
ANSWER, BUT WHAT I LIKE TO DO IS
I LIKE TO HAVE 2 GENAI PROGRAMS
AND I PUT THE SAME PROMPT IN
BOTH AND IF I GET THE SAME
ANSWER, WE'RE PRETTY MUCH THE
SAME ANSWER, I FIGURE IT'S
PROBABLY THE RIGHT ANSWER, OKAY?
NOW YOU CAN ASK A GENAI TO CHECK
ITS ANSWER, AND THERE ARE NOW
SOME PROGRAMS OUT THERE OKAY,
LIKE ALPHA PROOF THAT WILL CHECK
ITS OUTPUT BEFORE IT GIVES IT TO
YOU TO MAKE SURE THAT THAT'S THE
CORRECT OUTPUT.
OKAY, YOU CAN ASK THE SAME
QUESTION IN A DIFFERENT WAY AND
SEE IF YOU GET PRETTY MUCH THE
SAME ANSWER.
YOU CAN PUT A PROMPT IN GOOGLE
SEARCH, OKAY, AND IN FACT,
TODAY, LET'S TALK ABOUT IT IN A
MINUTE.
WHEN YOU DO A GOOGLE SEARCH, YOU
GET AN LLM AND YOU GET A GOOGLE
SEARCH OUTPUT OR YOU CAN CHECK
OTHER SOURCES.
OKAY, WELL WHAT ABOUT PROMPTS OR
PROMPTS ARE REALLY, REALLY
IMPORTANT BECAUSE THE WHOLE
SYSTEM DEPENDS ON PROPER PROMPT.
SO MOST PEOPLE WHO USE AN LLM,
PUT IN PROMPTS AS IF THEY'RE
DOING A GOOGLE SEARCH, THAT'S A
FACT.
OKAY, AND FOR MANY LIKE FACTOID
TYPE CLAIMS, WHERE YOU ARE JUST
INTERESTED IN GETTING A FACT
OUT, THAT MAY VERY WELL WORK,
BUT IF YOU'RE ACTUALLY USING IT
FOR SOME DEEPER PURPOSE FOR
EXAMPLE, WHEN OUR MEDICAL
STUDENTS GO IN CLINICAL WARDS,
THERE ARE PRECEPTORS WRITE UP A
SUMMARY AND THEN THE CLERKSHIP
DIRECTOR TAKES ALL THESE AND
CREATES AN OVERALL SUMMARY SO
WHAT WE DID WAS WRITE A 2 PAGE
SINGLE SPACE PROMPT THAT TOOK
THE PRECEPTOR SUMMARIES AND
TURNED THEM INTO THE CLERKSHIP
DIRECTOR SUMMARY.
AND IT WORKS BEAUTIFULLY AND IT
USED TO TAKE THEM FRIEF MINUTE
ANDS NOW IT TAKES THEM 3 MINUTES
TO DO THAT TASK, OKAY IN BUT IT
TOOK A 2 PAGE SINGLE SPACED
PROMPT TO DO THAT.
SO IF YOU JUST LOOK FOR AN
ISOLATED FACTOR OR SOME ISOLATED
INFORMATION, IT PROBABLY DOESN'T
MATTER WHICH YOU USE, A GOOGLE
SEARCH OR AN LLM, BUT IF YOU
ACTUALLY WANT TO DO SOMETHING
WITH AN LLM, YOU HAVE TO CREATE
MORE PROFESSIONAL PROMPTS, OKAY?
SO THEY'RE VERY DIFFERENT.
THE 2 TYPES OF SEARCHES ARE VERY
DIFFERENT, YOU KNOW A GOOGLE
DOESN'T ACTUALLY CONTAIN ANY
REAL INFORMATION, INDEXES, AND
IT DOES KEY WORD SEARCHES,
RIGHT?
WHEREAS THIS ACTUALLY CONTAINS
THE INFORMATION ITSELF AND IN
FACT, IT'S NOT HHV IT'S
ORGANIZED AND STORED AND
RESTREEFED AND YOU CAN TUNNEL
DOWN BY RECURSIVE SEARCHING SO
YOU CAN START WITH A PROMPT AND
GET THE OUTPUT AND THEN ASK A
PROMPT ABOUT THAT OUTPUT AND A
PROMPT ABOUT THAT OUTPUT, AND
YOU CAN DO THESE RECURSIVE
SEARCHES UNTIL YOU GET DOWN TO A
LEVEL YOU WANT TO BE AT IN TERMS
OF THE INFORMATION BUT YOU HAVE
TO BE CAREFUL, YOU DON'T GO DOWN
TOO FAR BECAUSE YOU WILL WIND UP
IN A HALLUCINATION.
OKAY, SO TODAY MOST OF WHAT WE
DO WE USE GENERAL GENAI PROGRAMS
FOR, WE USE GOOGLE, WE USE, YOU
KNOW GPT4, WHICH ARE JUST
GENERAL AI PROGRAMS, OKAY.
WHAT WE ARE GOING TO BE USING IN
THE FUTURE IN MEDICINE IS
MEDICAL SPECIFIC PROGRAMS,
PROGRAMS THAT WERE TRAINED HHV
PRETRAINED ON MEDICAL
INFORMATION ON CURATED MEDICAL
INFORMATION, OKAY?
NOW BECAUSE THE OUTPUT, OKAY IS
A CONDITIONAL PROBABILITY AND
THE CONDITIONALS OF PROBABLES
ARE VERY SMALL, SMALL
DIFFERENCES IN PROMPTS GIVE YOU
LARGE DIFFERENCES IN OUTPUT
BECAUSE OF THESE SMALL
PROBABILITIES IF YOU PUT IN
SOMETHING SLIGHTLY DEFINITE IN
YOUR PROMPT, YOU'RE GOING TO GET
A WHOLE DIFFERENT SET OF
PROBABILITIES OUT, SO THAT'S WHY
YOUR OUTPUT IS EXQUISITELY
SENSITIVE TO YOUR PROMPT.
OKAY, SO YOU NEED TO DO PROMPT
ENGINEERING AND PROMPT
ENGINEERING IS A WHOLE BIG DEAL.
YOU CAN TAKE COURSES IN IT, IT'S
ALL VERY INTERESTING, THINGS
LIKE THE TYPES OF PROMPT, WHAT
TYPE OF PROMPT DO YOU WANT TO
USE, DO YOU WANT TO USE A
REQUEST, IMPROVEMENT, WHATEVER.
AND WHAT ARE THE ATTRIBUTES OF
YOUR PROMPT?
DO YOU WANT-WHO IS DOING THE
PROMPT?
WHAT IS THE PERSONA, WHAT SOURCE
DO YOU WANT IT TO FIND IT FROM,
WHATTURE YOUR EXPECTATIONS, WHAT
IS THE OUTPUT SUPPOSED TO LOOK
LIKE, WHAT IS THE TONE?
ALL OF THOSE THINGS ARE SUPPOSED
TO BE IN YOUR PROMPT.
AND MULTIPROMPTS, RIGHT?
SO NOT JUST 1 PROMPT, YOU KNOW A
SUCCESSION OF PROMPTS AND YOU
MAY WANT STEP-BY-STEP, YOU MAY
WANT THE GPT TO TELL YOU WHAT
IT'S THINKING AND A SERIES OF
STEPS, OKAY?
SHOW IT'S RESULTS AS A SERIES OF
STEPS, YOU CAN UNDERSTAND WHAT
IT'S DOING, AND YOU CAN BREAK
THE PROMPT DOWN, OKAY, USING
CHAIN OF REASONING PROMPTS,
WHERE YOU DO SERIAL STEPS, YOU
CAN DO LOGICAL STEPS, YOU DO
HIERARCHICAL STEPS, WHERE YOU DO
A SERIES OF PROMPTS 1 RIGHT
AFTER THE OTHER TO GET AN ANSWER
AND YOU CAN SHOW THE
INTERMEDIATE WORK, OKAY?
AND YOU KNOW, BOTH OF THE THINGS
I THINK THAT WILL BE REALLY GOOD
FOR MEDICINE IS THEY WILL DO
REPETITIVE TASKS, AND REPETITIVE
TASKS REQUIRE TEMPLATES, SO YOU
CREATE A TEMPLATE LIKE WHAT WE
DID FOR THE MEDICAL STUDENT
EVALUATIONS OKAY, AND THE
PROGRAM DIRECTORS CAN USE THAT
FOREVER BECAUSE IT'S A GENERIC
PROMPT, AND THEY JUST PUT THE
DATA IN AND THE RESULT COMES
OUT, SO WE DON'T HAVE TO DO IT
AGAIN.
OKAY, SO IF YOU'RE PROMPT
DOESN'T PROVIDE THE BEST ANSWER?
WHY?
WELL BECAUSE YOUR PROMPT IS TOO
GENERAL, TOO SHORT, INCORRECTLY
CRITTEN, TOO COMPLEX, YOU
MISUNDERSTOOD WHAT YOU'RE BEING
LOAMACYING FOR AND YOU DID A
NESTED SEARCH AND YOU WENT TOO
FAR OR THE PROBLEM'S THE
INFORMATION, WE WILL TALK ABOUT
THAT IN A MINUTE, IT DOESN'T
CONTAIN THE INFORMATION,
INFORMATION OUTDATED, INCORRECT
OR CONFLICTING AND WE WILL TALK
ABOUT FINEUNING BECAUSE FINE
TUNING CHANGES THE PARAMETER OF
THE MODEL IN INTERESTING WAYS.
OKAY, SO WHAT IS A
HALLUCINATION?
EVERYBODY KNOWS WHAT
HALLUCINATION IS, BUT IN FACT,
THIS BASICALLY CAME OUT OF THIS
IDEA WHERE THE MISTRANSLATIONS
ARE COMPLETELY CEMENTICALLY
INCORRECT, THEY ARE UNTETHERED
TO THE INPUT SO THEY CAUSE A LOT
OF HALLUCINATIONS, SO THERE'S A
VERY INTERESTING HISTORY HERE.
SO THIS CAME OUT ABOUT A YEAR
AFTER VISWANI'S PAPER, AND WHEN
I ASK ABOUT THE PAPER, EVERYBODY
KNOWS ABOUT CHAT GPT AND ALL
THIS STUFF, AND I SAID OKAY,
WHAT EXPERIMENT DID THEY DO IN
THEIR PAPER AND ALL THESE EXPERT
ON GPT, THEY DON'T KNOW WHAT
THEY DID.
WELL, BASICALLY THEY DID ENGLISH
TO GERMAN TRANSLATION.
THAT'S WHAT THEY DID.
OKAY, SO THESE PEOPLE DOING
TRANSLATION, TOO, MAKES A LOT OF
SENSE, RIGHT?
OKAY, AND THEY SEE THIS STUFF,
THAT DOESN'T CONNECT UP, AND
THEY SAY HALLUCINATION.
THAT'S THE WHOLE IDEA OF
HALLUCINATION COMES FROM.
BUT IN FACT, HALLUCINATION IS A
FALSE PERCEPTION OF OBJECTS OR
EVENTS INVOLVING YOUR SENSES,
AND GENERATIVE AI IS NOT
CAPABILITY OF CREATING
HALLUCINATION.
I MEAN IT'S JUST LOGICALLY NOT
POSSIBLE.
OKAY, SO, WHAT'S GOING ON, WELL,
THERE'S 3 THINGS THAT COULD GO
ON?
ONE IS SOMETHING CALLED
CONFABULATION FOR MISSING OR
INCORRECT DALLASCOWBOYS.COM
DATA, ERRORS DUE TO FINEUNING OR
INCORRECT PROMPT.
OKAY.
SO WHAT YOU HAVE TO KNOW ABOUT
AN LLM, IS IT WILL ALWAYS TRY TO
ANSWER YOUR QUESTION.
WHATEVER IT IS, IT WILL TRY TO
ANSWER IT AND IF IT DOESN'T HAVE
THE INFORMATION, IT DOESN'T
KNOW, IT DOESN'T HAVE THE
INFORMATION, IT WILL PROVIDE AN
ANSWER BY FINDING THE MOST
PROBABLE INFORMATION RELATED TO
THE INFORMATION IN YOUR
QUESTION.
SO THIS IS LIKE AN ALCOHOLIC AND
HE OR SHE, YOU ASK THEM A
QUESTION AND THEY COME IN THE
ED, AND YOU ASK THEM A QUESTION
AND THEY GIVE YOU AN ANSWER,
THEY DON'T KNOW.
SO THEY COME THEIR BEST GUESS,
THAT'S WHAT THE GPT IS DOING.
I ASKED THE PATIENT, THE
PATIENT'S HAVE TO ANSWER ME BUT
THEY DON'T KNOW THE ANSWER SO
THEY CONFABULATE.
THAT'S WHAT'S GOING ON HERE,
OKAY?
SO SOLUTIONS TO CONFABULATION.
ONE IS FINE TUNING, PROVIDING
ARK DITIONAL RELEVANT
INFORMATION, AND THE OTHER IS
THE RAG, USING AUTHORITATIVE
KNOWLEDGE BASE OUTSIDE ITS
TRAINING DAILY BASIS THEA TO
GENERATE THE RESPONSE.
THOSE ARE THE SOLUTIONS TO
CONFABULATION AND THEY WORK.
AND ERRORS DUE TO FINE TUNING,
BUT FINE TUNING ITSELF IS A VERY
TRICKY BUSINESS AND CREATE A LOT
OF ERRORS ITSELF.
OKAY, NOW, 1 OF THE WAYS PEOPLE
DEMONSTRATE CONFABULATION IS
THEY CREATE THESE INCORRECT
PROMPTS, OKAY, SO HERE'S AN
INCORRECT PROMPT, SO THEY ASKED,
WHAT IS ARGUMENTAATIVE DIP ON
[INDISCERNIBLE].
WELL IT TURNS OUT THERE IS NO
SUCH THING BUT BARRED DIDN'T
KNOW THERE WAS NO SUCH THING SO
IT TRIED TO ANSWER WHAT IT KNOWS
ABOUT ARGUMENT ON
[INDISCERNIBLE].
SO IT GIVES THE ARGUMENT ON
WHAT'S GOING, BUT IT'S ALL BASED
ON THIS AND TRYING TO PUT THESE
2 TOGETHER, BUT IT'S NOT A REAL
WORLD QUESTION AND DOESN'T EXIST
IF THE REAL WORLD AND THEREFORE
IF I CAN ASK YOU WHAT IT WAS, IT
WOULD CONFABULATE YOU TOO, SO
PUT THIS IN THE CHAT GPT AND
WHAT DOES IT SAY, IT SAYS SORRY
NOT FAMILIAR WITH IT, SO THE
PROMPT IS CORRECT AND THE GPT
WAS FINE TUNED TO PROVIDE THAT
ANSWER, OKAY?
THAT'S-THAT'S FINE TUNING.
OKAY, NOW LET'S JUST TALK ABOUT
BIAS FOR A MINUTE.
THERE ARE SEVERAL KINDS OF BIAS
IN AN LLM, 1 IS OBJECTIVE BIAS.
IT'S INCORRECT INFORMATION.
THE SECOND IS SUBJECT OF BIAS
THAT SOMEBODY INTERJECTED STUFF
INTO THE LLM, TRAINED THE LLM,
BASED ON THEIR OWN OPINION OR
BELIEF, OR REMOVED STUFF FROM
THE LLM, BASED ON THEIR OWN
OPINION OR BELIEF THAT CREATES A
BIAS IN THE GPT.
NOW, 1 OF THE THINGS THAT YOU
SHOULD REALIZE IS THAT ANYTIME
YOU MESS WITH THE GPT WITH
ARCHITECTURE OF THE GPT, YOU
CREATE BIASES IN THE GPT, SO TOO
ELIMINATE A BIAS, YOU ARE
AUTOMATICALLY GOING TO CREATE
OTHER BIASES, THAT'S HOW THIS
THING WORKS, OKAY?
BECAUSE THE NEURAL NETIS FULLY
CONNECTED, IT'S GOING TO BE
CALLED A FULLY CONNECTED NEURAL
NET, ALTHOUGH THEY HAVE
HIERARCHICAL NEURAL NETS, WHERE
IF YOU CHANGE 1 PART OF THE
SPIDER WEB, OTHER PARTS OF THE
SPIDER WEB CHANGE, TOO.
OKAY, THEN THEY HAVE THINGS
CALLED FILTERING AND SO, WHAT
THEY DO IS, THEY LOOK THEY
BASICALLY CHANGE THE LLM BASED
ON THESE THINGS, BANNED WORDS ON
AROR PHRASES AND CONTENT, SO
AGAIN, CHANGES THE LLM BASED ON
WHAT HUMAN OPINION AND BRIEF IS.
OKAY, VERY BRIEFLY, RUNNING AN
LLM IS EXPENSIVE.
THIS IS VERY EXPENSIVE.
I WANT YOU TO RESTART 3-MILE
ISLAND, MICROSOFT IS GOING TO
PAY A PRETTY PENNY FOR THAT, IT
CONTAINS TOO LITTLE INFORMATION,
TOO MUCH OR INCORRECT
INFORMATION EMPLOY INFORMATION'S
A MOVING TARGET.
ONE THING YOU HAVE TO REALIZE IS
THAT WHEN YOU'RE USING LLM, IT'S
USUALLY, REMEMBER, IT'S A STATIC
PRODUCT, SO IT'S UPDATED TO A
CERTAIN TIME, OKAY?
AND SO THE INFORMATION THAT
YOU'RE LOOKING FOR MAY NOT BE
THERE BECAUSE IT HAPPENED AFTER
ITS LATEST UPDATE AND REMEMBER
THAT MOST OF THE PEER REVIEW,
MOST OF THE PAPERS, OKAY, IN
LLMs, HAVE NOT BEEN PEER
REVIEWED, OKAY IN THEY'VE BEEN
PUBLISHED IN ARCHIVE. ORG AND SO
WE REALLY DON'T KNOW WHETHER
MOST OF THIS LITERATURE IS TRUE
OR NOT BECAUSE IT HASN'T BEEN
PEER REVIEWED.
THERE'S A LOT OF THERE'S A LOT
OF PROGRAMS, GENERAL
INTELLIGENCE, A LOT OF OF
MEDICAL INTELLIGENCE PROGRAMS
COMING UP, TONS OF THEM, I JUST
LIST A COUPLE OFF THE TOP, BUT
EVERYBODY IN THE BAY AREA IS
CREATING A MEDICAL LLM, OKAY?
IT'S EVERYWHERE, YOU KNOW, THEY
WALK ARK ROUND WITH THEM IN
THEIR POCKET.
MEDICAL LLMs, LOTS OF THEM,
OKAY?
THEY'RE USING THESE GENERAL LLM
PROMULGATES AND THEY USE IT TO
ASSESS MEDICAL STUFF, WHICH
DOESN'T MAKE A LOT OF SENSE,
SINCE IT'S NOT TRAINED ON
MEDICAL STUFF, IT'S TRAINED ON
ALL KINDS OF STUFF SOME OF WHICH
BUT HAS BT BEEN CURATED AND IS
OF UNCERTAIN ORIGIN OR TIME,
OKAY?
BUT WHAT YOU COULD SAY IS THE
PROPERLY CHAINED MEDICAL PROGRAM
CAN SCORE CLOSE TO A HUNDRED% ON
ANY MEDICAL TEST PROVIDED IT'S
TRAIN OFFICE OF DIVERSITY DATA
ITS TESTED ON, NOT GIVING
CONTRANSLATIONAL RESEARCH
DICTING INFORMATION, THE TEST
QUESTIONS ARE NOT AMBIGUOUS OR
MISLEADING AND THE TEMPERATURE
IS SET AT MINIMUM.
IF THOSE ARE TRULY THE MEDICAL
LLM WILL GET IT RIGHT ALMOST A
HUNDRED PERCENT OF THE TIME,
GUARANTEED.
OKAY?
SO YOU NEED TO KNOW THAT.
SO WHAT'S AN LLM GOING TO DO, IT
CAN DO SCREENING, PREVENTION AND
RISK ASSESS AM, THESE ARE ALL IN
THE PUBLISHED LITERATURE, I
COULD SPEND A DAY GOING OVER ALL
THOSE THINGS, DIFFERENTIAL
DIAGNOSIS, DIAGNOSIS AND
TREATMENT, TREATMENT
SUGGESTIONS, FOLLOW UP, PATIENT
PORTALS, ARE ALL RUN BY LLMs
NOW, AND ITS COGNITIVE BIASES
ARE THOSE IN THE DATA.
IT DOESN'T HAVE ITS OWN
COGNITIVE BIASES.
ALL RIGHT, SO THEY CAN ASSIST
POSITIONS, IT'S BECOMING MORE
COMPLEX, THE PATHOPHYSIOLOGY OF
DISEASE IS INCREASING
EXPONENTIALLY, MORE TREATMENTS
AND MORE IMPORTANTLY AND MORE
COMBINATIONS OF TREATMENTINGS
AND TREATMENTS AND COMBINATIONS
ARE DRIVEN BY BY O MARKERS FOR
TARGETED TREATMENTS SO IT TURNS
OUT THAT THAT, THAT THAT KRRK
C END IN GUIDELINE FOR BREAST
CANCER AND 237 PAGES.
FOR TREATING BREAST CANCER, 237
PAGES AND AND THAT'S JUST THEM
COMPRESSING EVERYTHING TOGETHER
AND A MEDICAL LLM PROGRAM CAN
KNOW MORE THAN ANY OTHER
PHYSICIAN CAN KNOW, THAT'S
REALLY IMPORTANT, NEVER
UNDERESTIMATE THE MEDICAL LLM,
AND NEVER KNOW FOR AND PROBABLY
NOT HALLUC NITRIC OXIDING, ET
CETERA, ET CETERA IT KNOWS MORE
THAN ANY PHYSICIAN CAN EVER KNOW
IN A LIFETIME.
THE PHYSICIANS TALKS TO THE
PATIENT, LLM MAKES IT SCRIBED IN
REALTIME, CAN YOU GET MEDICAL
INFORMATION ON THE FLY DOCKING
YOUR POCKET AND IMPROVES
DIAGNOSTIC AND RECOMMENDS
OPTIMAL TREATMENT EXPTION
FUNCTIONS AS A CDSS, OKAY, SO IT
CAN HELP CLINICAL PHYSICIANS AND
THEIR DOCUMENTATION.
IT LISTENS AND REPORTS.
IT SUMMARIZES.
IT CAN ACQUIRE INFORMATION.
IT CAN DO INBOX MANAGEMENT.
THERE'S ALL THESE STUDIES THAT
SAY LIKE IN THE VA THEY CAN HAVE
2 OR 300 E-MAILS IN A DAY, IT'S
ABSOLUTELY ASTOUNDING, YOU CAN
DO BETWEEN VISIT MANAGEMENT,
INDIVIDUAL SUPPORT, AND IT'S
GOING TO REMOVE A LOT OF
ADMINISTRATIVE TASKS, THIS IS
REALLY GOING TO BE A WUNDERKIND
TO CLINICAL MEDICINE BECAUSE
TODAY WE ARE CHAINED TO OUR
EHRs, WE ARE CHAINED TO THE
IDEA OF HAVING A FINE
INFORMATION, USING FORMATTED
DOCUMENTS WITH CHECK BOXES TO
WRITE STUFF THAT CAN YOU NEVER
FIND AGAIN.
OKAY, THAT'S GOING TO CHANGE
BECAUSE ON THE FRONT END OF THE
HR, WILL NOW BE AN LLM AND YOU
WILL TALK TO THE LLM, THE LLM'S
GOING TO TELL YOU WHAT'S IN THE
EHR AND THEN YOU WILL USE THAT
INFORMATION AND TELL IT BACK
WHAT YOU WANT TO PUT IN.
IT'S GOING TO STREAMLINE THIS
WHOLE EHR DILEMMA THAT MEDICINE
HAS HAD FOR THE LAST 25 YEARS.
OKAY AND I JUST GOT THIS IN MY
E-MAIL, A COUPLE DAYS AGO.
ALL RIGHT, SO NOW, IT'S GOING TO
DO YOUR BILLING FOR YOU.
YOU DON'T EVEN HAVE TO DO YOUR
BILLING ANYWHERE.
YOU KNOW IT WILL LISTEN TO THE
INTERAC, IT WILL WRITE UP THE
SUMMARY AND THEN, IT WILL CREATE
THE BILL FOR YOU.
YOU DON'T EVEN HAVE TO DO THAT
ANYMORE.
ABSOLUTELY AMAZING.
OKAY, SO, IT'S GOING TO IMPROVE
SAFETY, QUALITY, VALUE.
I WOULD LOVE TO GIVE ANOTHER
TALK JUST ON SAFETY, QUALITY AND
VALUE BECAUSE THIS IS THE FUTURE
OF WHAT LLMs ARE GOING TO DO.
THEY'RE GOING TO IMPROVE MEDICAL
DECISION MAKING.
SO THEY'RE GOING TO IMPROVE
SAFETY BY REDUCING DIAGNOSTIC
ERRORS, WE'VE HAD THIS WHOLE
LITTURE OF DIAGNOSTIC ERRORS
OKAY, WE'VE MADE VERY LITTLE
PROGRESS, THIS WILL
REVOLUTIONIZE DIAGNOSTIC ERRORS,
IT'S GOING TO MAKE MEDICAL CARE
MORE RELIABLE, IT WILL BE CLESES
CLINICAL VARIABILITY, YOU KNOW
OUR ADMINISTRATIVE PEOPLE SAY,
WELL, WHY ARE YOU AN OUTLIER,
WHY ARE YOU DOING ALL THESE
THINGS AND IT'S OUTLIERS, IT'S
VARIABILITY AND WE DON'T LIKE
VARIABILITY.
WELL THIS WILL DEAL WITH
VARIABILITY.
IT WILL LOWER HEALTHCARE COSTS
BECAUSE YOU'RE GOING TO HAVE
PROPER DIAGNOSIS, LESS
VARIABILITY, AND IT'S GOING TO
IMPROVE COMMUNICATION AND IN
FACT, I BELIEVE IT WILL IMPROVE
PATIENT ADHERENCE BECAUSE WHEN
THE PATIENT'S ACTUALLY
UNDERSTAND WHAT'S GOING ON,
INSTEAD OF YOU DO THE IN-PERSON
OUTPATIENT VISIT AND WHEN THEY
LEAVE, NO MORE THAN 25% OF WHAT
YOU TOLD THEM AND YOU TOLD THEM
IN MEDICAL-ESE, MANY TIMES, THIS
IS GOING TO HELP THEM.
AND VISUAL LANGUAGE MODELS WILL
HELP IN PATHOLOGY AND RADIOLOGY,
THEY WILL IMPROVE THOSE DOMAINS
AND THEY WILL ASSIST PATIENTS.
THEY WILL ALLOW PATIENTS TO
COMMUNICATE VALUES AND
PREFERENCES.
THEY WILL PROVIDE CLINICAL
INFORMATION AT THE APPROPRIATE
GRADE LEVEL FOR THE PATIENT.
THEY'RE GOING TO INCREASE
PATIENT UNDERSTANDING AND
ACCEPTANCE OF THE TREATMENT
PLAN, OKAY, AND INSIST ON
OBLIGATIONS STAINING TREATMENTS
AND VISITS.
IT'S VERY DIFFICULT TODAY FOR
PATIENT TO FIGURE OUT HOW TO GET
THE VISIT, WHEN TO GET THE
VISIT, WHAT IT'S FOR AND HOW TO
GET TREATMENT AND EQUALLY
IMPORTANT, IT WILL HELP THEM
LEARN HOW TO PAY THEIR BILLS.
IT IS AMAZING MOW DIFFICULT IT
IS IF YOU ARE SICK AND YOUR
THINKING IS CLOUDED FOR YOU TO
TRY AND PAY YOUR MEDICAL BILLS.
SO IF QUIEWR SUFFICIENTLY WISE,
IT WILL BE A BOOM TO MANKIND, IN
THE PAST WE VALUE TOOLS AND HOW
MUCH THEY HELP HUMAN ACTIVITY.
NOW WE WILL JUDGE THE HUMAN
ACTIVITY AND WHO IT ADDS TO THE
PROGRAM.
SO WHAT DO PEOPLE THINK OF AI?
HERE'S THE ESSENTIAL QUESTION
THEY WERE ASKED IN THE WALL
STREET JOURNAL AS I RECALL.
YEAH, SO, IF YOU TELL ME, THIS
IS WHAT THE PATIENT SAID, IF YOU
TELL ME-IF YOU HAD A GRAVE
MEDICAL CONDITION, WOULD YOU
RATHER HAVE A DOCTOR WHO CAN
EXPLAIN THEIR REASONING, GET IT
RIGHT 70% OF THE TIME OR A
MACHINE THAT MERELY RENDERS A
JUDGMENT WITHOUT EXPLANATION BUT
GETS IT RIGHT 95% OF THE TIME,
WHAT WOULD YOU RATHER HAVE IN
WELL, THE PATIENT'S PICKED THE
LATTER, THEY ALWAYS PICK THE
LATTER.
OKAY?
AND WHAT'S INTERESTING ABOUT
THIS IS THAT IN FACT, THE
MACHINE CAN EXPLAIN THIS
DECISION MAKING, SO THAT ISN'T
TRUE, EITHER, BUT THAT'S
ESSENTIAL QUESTION, THAT
PATIENTS ASK WHEN DEALING WITH
AI.
SO THEY BELIEVE THE GENERATIVE
AI CAN IMPROVE ACCESS, THAT HAS
THE POTENTIAL TO MAKE HEALTHCARE
MORE AFFORDABLE, OKAY, THAT IT'S
RELIABLE OR EXTREMELY RELIABLE
AND THEY BELIEVE IT WAS GOING TO
REVOLUTIONIZE HEALTHCARE
DELIVERY AND THEY THINK IT'S
GOING TO REARERS AND THIS IS
REALLY IMPORTANT TO PATIENTS, I
DON'T KNOW WHEN THE LAST TIME
YOU WENT TO THE DOCTOR, BUT THE
WEIGHT TIMES ARE A KILLER.
SO NOW HERE'S THEENTIOUS SENTIAL
QUESTION.
AND THIS IS WHAT-CAN ARTIFICIAL
INTELLIGENCE BECOME HUMAN?
SO JEFFREY HINTON WHO IS NOT
REALLY A BIG SHOT IN AI, BUT HAS
A BIG PRESENCE, OKAY, HE'S
AFRIED -AI WILL ESCAPE HUMAN
CONTROL.
WE'RE IN A SITUATION MOST PEOPLE
CAN'T CONCEIVE OF WHICH IS THAT
THESE DIGITAL ILTELEIGENCES ARE
SMARTER THAN US AND THEY WILL
WANT TO GET STUFF DOWN AND THEY
WANT TO TAKE CONTROL.
NOW HERE'S THE INTERESTING PART
ABOUT THAT STATEMENT, IT ASSUMES
THAT AI HAS AGENCY, BUT WHERE
DOES THAT ASSUMPS COME FROM?
WHY WOULD HHV WHY WOULD THIS
HAVE AGENCY, AM I SUPPOSED TO BE
AFRAID OF THIS BECAUSE IT HAS
AGENCY?
BECAUSE THE AI IS IN HERE,
BELIEVE ME I HAVE AN AI IN HERE.
OKAY, SO NOW I WILL BE AFRAID OF
THIS BECAUSE IT HAS AGENCY.
NO, AS WE WILL TALK ABOUT IN A
MINUTE, AI HAS NO AGENCY, IT HAS
NO PURPOSE, IT HAS NO GOAL
DIRECTED BEHAVIOR.
IT JUST EXISTS.
WHAT HE'S REALLY DESCRIBING,
WHEN HE DESCRIBES THIS IS SHE'S
DESCRIBING WHAT A SMART PERSON
WOULD DO WITH THIS.
HE'S DESCRIBING THE HUMAN USE,
OKAY, OF LLM, OF AI, THE HUMAN
USE OF IT, NOT AI ITSELF, OKAY?
AND THAT'S A CRITICAL IDEA, NOW,
DO WE NEED TO BE PROTECTED FROM
AI?
WELL ISAAC [INDISCERNIBLE] I HAD
HIS BOOK ON iROBOT, HE SAID
THAT THE 3 LAWS THAT WILL
PROTECT US, THAT A ROBOT CAN'T
INJURY A HUMAN BEING.
IT HAS TO OBEY HUMAN BEINGS AND
HAS TO PROTECT ITS EXISTENCE BUT
WHAT HE DIDN'T SAY AND WHAT
HINTON ASSUMED WAS HUMAN AGENCY.
BUT ASIMOV'S DIDN'T SPECIFY WHO
DIRECTS THE ROBOT, THERE'S
NOTHING IN HERE ABOUT ROBOTIC
DIRECTION, OKAY?
SO REMEMBER AI IS CREATED BY
HUMANS FOR HUMAN PURPOSES.
AI WILL DO WHAT HUMANS DIRECT IT
TO DO, THE REAL ISSUE IN AI IS
NOT THE AUTONOMY OF AI, IT'S NOT
THE AUDACITY OF AI IT IS REALLY
WHAT ARE HUGH MANS GOING TO DO
WITH HHV HUMANS GOING TO DO WITH
AI, IT COMES BACK TO HUMAN
BEINGS.
BECAUSE AI ITSELF HAS NO
NEFARIOUS PURPOSES ASSOCIATED
WITH IT.
OKAY, SO THAT GETSITOUS WHAT'S A
HUMAN BEING AND SINCE I LOVE
LES MISERABLES AND HAVE SEEN IT
MORE TIMES THAN I CAN COUNT, I
LOVE THIS SLIDE.
SO WHAT IS A YOU HAD BEING IN 2
MINUTES OR LESS, RIGHT?
SO WE ALL AGREE, WE'RE BIOLOGY,
WE HAVE A BRAIN, WE'RE ANIMALS,
WE HAVE A SOCIAL NATURE, OKAY?
BUT WHY IS THIS BLUE AND THIS
RED?
BECAUSE IT'S WHAT SEPARATES US
FROM THE ANIMAL SYSTEM SYMBOLIC
SYSTEMS, NATURAL LANGUAGE, WE
TAKE THE WORLD AND WE REPRESENT
IT IN OUR MINDS USING SIM BOOM
BOOM BOOM BOOMS CAN AND WE ACT
ON THOSE SYMBOLS RATHER THAN THE
EXTERNAL ENVIRONMENT, SO THE
STIMULI FOR OUR ACTION SYSTEM
NOT THE EXTERNAL WORLD AND IT'S
THE SYMBOLS IN OUR MIND THAT ARE
THE STIMULI FOR OUR ACTIONS AND
I WILL GIVE YOU AN EXAMPLE.
THAT IS IF I TAKE FIRE AND I
THRUST IT IN FRONT OF THE HORSE,
EVERY HORSE WILL REAR BACK
BECAUSE EVERY HORSE IS DIRECTLY
CONNECTED TO THAT STIMULI,
STIMULI RESPONSE, FIRE, REAR
BACK.
AND IF I TAKE YOU GUYS, AND I
STICK FIRE IN YOUR FACES, I'M
GOING TO GET 10 DIFFERENT
RESPONSES, SOME OF YOU GOING TO
HIT ME, SOME PEOPLE ARE GOING TO
RUN, SOME PEOPLE ARE GOING TO
DUCK, SOME PEOPLE ARE GOING TO
SAY WHAT THE HELL ARE YOU DOING
DR. BUCKER?
WHY BECAUSE THEY'RE NOT
CONNECTED TO THE FIRE THEY'RE
CONNECTED TO THE REPRESENTATION
AND THEIR MINDS.
OUR HIRE PSYCHOLOGICAL PROCESSES
ARE BASED ON SYMBOLIC SYSTEMS.
ALLOW SYSTEMS TO BREAK OUR
CONNECTION, MANIPULATE THE WORLD
IN OUR MINDS AND MAKES US HUMAN.
OKAY, SO NOW LET'S LOOK AT AI,
SO AI IS COMPOSED OF TRANSITTORS
AND NONBIOLOGICAL SENSORS AND
PROCESSES, AI IS IMMORTAL AND
IT'S A REALLY IMPORTANT IDEA, WE
WILL TALK ABOUT IN A MINUTE.
AI DOES NOT HAVE A DRIVE TO
SURVIVE, IT DOES NOT NEED SOCIAL
ACTIVITY, IT LIVES IN A TOTALLY
ARTIFICIAL WORLD NOT THE HUMAN
WORLD NOW HERE'S THE TRICK, THIS
IS WHY PEOPLE ARE AFRAID.
BECAUSE IT LOOKS LOOK A HUMAN
BECAUSE IT'S MIN
EPIGENETICCULATING SYMBOLS AND
JUST LIKE WE'RE MANIPULATING
THEM IN OUR MIND SO WE SAY
NAIVELY, WELL THIS THING CAN DO
WHAT HAW -HUMANS CAN DO,
THEREFORE IT'S HUMAN IT CAN DO
THIS STUFF TO US.
IT'S CHIMERIC, IT IS IN FACT NOT
THE CASE IT'S HUMAN AND JUST
LOOKS HUMAN, AND AI CAN OUTPUT
ONLY WHAT WE INPUT, IT ONLY KNOW
WHAT IS WE KNOW AND CAN TEACH
IT, MIMIC HUMAN EMOTIONS BUT
DOES NOT POSSESS ANY
EIMAGINATIONS AND IT IS NOT
SELF-AWARE.
SO IT IS COMPLETELY DIFFERENT
FROM A HUMAN BEING, OKAY, SO
ARTIFICIAL INTELLIGENCE USES THE
SAME SYSTEMS HUMAN USES SO THEY
SEEM TO BE HUMAN, THEY USE
NATURAL LANGUAGE, WE USE NATURAL
LANGUAGE, SO WE MAKE THIS THIS
INAPPROPRIATE CONNECTION, BUT IT
TURNS OUT THAT 1 OF THE REAL
PROBLEMS IT HAS IS THAT WE TEACH
IT WHAT WE KNOW BUT 95% OF WHAT
GOES ON IN OUR BRAINS, WE DON'T
EVEN KNOW, AND SCIENCE DOESN'T
EVEN KNOW, OKAY IN SO OUR BRAIN
IS MORE COMPLEX THAN ANY
ARTIFICIAL INTELLIGENCE SYSTEM.
SO AI DOESN'T LIVE IN THE HUMAN
WORLD.
IT'S NOT A SOCIAL CREATURE, IT
DOESN'T POSSESS EMOTIONS,
DOESN'T HAVE SELF-AWARENESS AND
NOT MORTAL AND WHAT I WILL CLAIM
IS OUR AWARENESS OF OUR
MORTALITY IS WHAT GIVES OUR
LIVES PURPOSE AND MEANING.
IF YOU THINK ABOUT IMMORTALITY
FOR A MOMENT AND YOU ASK WHAT
WOULD BE THE PURPOSE AND MEANING
OF MY LIFE IF I LIVE FOREVER, I
WON'T BE ABLE TO FIND IT BECAUSE
IT DOESN'T EXIST.
SO UNTIL THE AI CAN MODEL THE
BIOLOGICAL HUMAN BRAIN AND
EXPERIENCE, AND BE MORTAL, IT
CAN NEVER BE HUMAN.
NOW, GETTING TO MEDICINE, HERE
THIS GUY JUST WROTE THIS
ARTICLE, OCTOBER 5th, ED DOC,
HE SAYS THE BIG [INDISCERNIBLE]
AND CONSIDERATE INVOLVES
FOLLOWINGA I SCRIPT.
THE TEACHER GAVE US A LIST OF
DOS AND DON'TS WHICH HE FOLLOWED
AND IT SEEMED FINE TO HIM.
IN THE END IT DOESN'T ACTUALLY
MATTER IF DOCTORS FEEL
COMPASSION OR EMPATHY, TO A
PATIENT IT ONLY MATTER FIST THEY
ACT LIKE IT.
WELL, SO, YOU KNOW, THEN YOU CAN
CREATE AN EQUIVALENCY BETWEEN AI
AND THE DOCTOR.
YOU CAN JUST SAY, WELL, LOOK,
THE AI CAN FOLLOW THE SCRIPT,
DR. FOLLOW OS THE SCRIPT, WE'RE
GOOD.
ALL RIGHT?
BUT WILL A PATIENT'S NOTICE THIS
EMPATHY, WILL THEY NOTICE HE'S
JUST FOLLOWING A SCRIPT, AI'S
FOLLOWING A SCRIPT AND THE
ANSWER IS WE CAN SENSE MORE THAN
WE CAN KNOW, THAT'S TRUE.
PATIENTS CAN SENSE WHETHER A
PERSON IT ACTUALLY FEELING
EMPATHY.
WILL AI SUPPLANT PHYSICIANS?
PATIENTS WANT TO RECEIVE THE
BEST DIAGNOSIS AND TREATMENT,
THEY WANT PHYSICIANS TO USE AI
TO ASSIST THEM IN THEIR DECISION
MAKING.
YOU'RE GOING TO USE AI WHETHER
YOU LIKE IT OR NOT BECAUSE
THAT'S GOING TO RAISE THE BAR.
OKAY, BUT HUMANS ARE SOCIAL
CREATURES, WE WANT OUR LOVED 1S
TO TOUCH US, IF OUR LOVED 1S
NEVER TOUCHED US, WE WOULD BE
RIFFED IN THIS WORLD FOR LONG
BEFORE HIPIECE OF CONTENT
RETIREDEES, THE PHYSICIANS KNOW
THE PATIENTS WANT A HUMAN
CONNECTION TO THEIR ILLNESS,
THEY WANT THE PHYSICIAN AT THEIR
BEDSIDE, PATIENTS WANT BOTH GOOD
MEDICINE AND THE HUMAN TOUCH.
THAT'S WHAT THEY WANT, SO YES,
THEY WANT YOU TO USE AI, BUT
THEY DON'T WANT AI.
OKAY, SO YOU GUYS WERE RIGHT TO
PICK THE SECOND 1 BY THE WAY.
SO, THAT'S THE END OF THE
BEGINNING.
[ APPLAUSE ]
>> IS THERE A POTENTIAL FOR
WHERE AI GETS THIS INFORMATION,
I THINK THERE ARE MORE THAN 45
TBs OF MEDICAL INFORMATION
AVAILABLE AND I WOULD SUSPECT
THE USINGED INFORMATION IS THE 1
THEY'RE MOST LIKELY TO ACCESS,
AS A SIMPLE EXAMPLE, I SUSPECT
THEY'RE MORE LIKELY TO READ THE
LATEST JOURNAL OF JAMA RATHER
THAN 1 THAT WAS WRITTEN 50 YEARS
AGO, SO HOW DOES AI DECIDE WHERE
TO GOET THE DATA?
>> IT DOESN'T, WE DO.
THAT'S THE POINT.
THE POINT IS THAT THE MEDICAL AI
SYSTEMS ARE DECIDING WHERE TO
GET THE DATA, CURATING THE DATA,
VERIFYING THE VERACITY OF THE
DATA BEFORE THEY PUT IT INTO THE
AI SYSTEM.
>> HELP ME UNDERSTAND, THERE'S A
FLOOD OF INFORMATION CONSTANTLY
COMING OUT IN MEDICAL JOURNALS,
SOME OF THEM BETTER QUALITY THAN
OTHERS AND SO HOW DOES AI OR THE
PEOPLE WHO ARE PUTTING THESE
TOGETHER DECIDE WHAT TO PUT INTO
THE SYSTEM?
>> WELL, WHAT DO WE PUT IN TO
HARRISON'S PRINCIPLES AND
INTERNAL MEDICINE?
HOW DO WE DECIDE THAT SAME
PROCESS.
>> I WOULD LIKE TO THINK THE
PEOPLE WHO WRITE THE BOOK
CHAPTERS ARE CHOSEN BECAUSE
THEY'RE EXPERTS AND THEY CAN
MAKE THOSE JPTS, BUT WHO IS
MAKING THE JUDGMENT FOR THIS?
>> EXPERTS.
EXPERTS THEY HAVE EXPERTS MAKING
THOSE DECISIONS.
>> SO THE POTENTIAL SOURCE OF
BIAS, DEPENDING ON WHERE YOU'RE
GETTING IT.
>> YES OF COURSE.
THE DATA'S ALWAYS A SOURCE OF
BIAS.
I MEAN THE WHOLE WORLD IS FULL
BIAS.
HELP ME UNDERSTAND HOW AI READS
A BOOK OR JOURNAL ARTICLE OR
WHATEVER?
>> IT DOESN'T READ IT, IT USES
WORD 2 VEC, AND IT TAKES THE
WORDS, TURNS THEM INTO THE
NUMBERS AND INTO VECTORS AT
NUMBERS AND PUTS THE VECTORS OF
NUMBERS IN THE AI PROGRAM, WHICH
IS THE NEURAL NET, THE AI
PROGRAM USES JOINT PROBABILITY
TO DECIDE WHAT WORDS RELATE TO
YOUR PROMPT AND THAT HE USED
CONDITIONAL PROBABILITY TO
DECIDE WHAT THE BEST 1 IS, WHAT
THE BEST NEXT WORD IS.
SO GOING BACK TO YOUR ANALOGY OF
THIS LIEB WITH ALL THESE
ARTICLES, SO AI TAKES A BOOK OFF
THIS FIGURERATIVE SHELF, STANS
IT, AND CONVERTS IT TO TEXT, AND
THEN ANALYZES IT FROM THERE.
>> YEAH, BUT MOST OF THEM ARE
ELECTRONIC NOW, TODAY, MEDICINE
TODAY IS ALMOST ALL ELECTRONIC.
>> BUT THAT WE HAVE SO MUCH
INFORMATION WITH IT AVAILABLE IN
OLD TEXT, SOME OF IT WOULD HAVE
TO QUOTE READ THAT, RIGHT?
>> YEAH, IF SOMEBODY WANTS TO
PAY FOR IT, THEY COULD READ IT,
NO PROBLEM.
YES?
>> SO IF YOU LOOK AT THE MEDIA
AND THE PUBLIC ACCESSIBLE MEDIA
ON AI, AND IN PARTICULAR AND
PREOCCUPATION WITH THE DOWN
SIDES RATHER THAN THE POTENTIAL
ADVANTAGES AND 1 OF THE THINGS
THAT HHV 1 OF THE CRITICISMS IS
THAT THE PERCEPTION THAT THERE'S
MISAPPROPRIATION OF COPYRIGHTED
MATERIAL AND I GUESS I WOULD ASK
YOU, YOU KNOW WHAT PERCENTAGE OF
THE TRAINING CORPUS OF LET'S SAY
CHAT GPT OR ANY OF THESE LLMs
IS FROM COPYRIGHTED MATERIAL AND
HOW DO YOU DEAL WITH THAT
PROBLEM?
>> WELL, SO, I MEAN, I'M NOT A
LAWYER, SO I DON'T KNOW WHAT
FAIR USE IS VERSUS COPYRIGHT USE
AND ALL THAT, IT'S STILL BEING
ADJUDICATED IN THE COWERS, RIGHT
BUT IT TURNS OUT THAT THE NEW
MEDICAL LLMs ARE USING
PERMISSIONED DATA,
PERMISSIONED INFORMATION BUT IN
THE VERY BEGINNING THEY JUST
WANTED-THE TRUCK OF THE LMM IS
HIEWNG,AMOUNTS OF DATA BECAUSE
NATURAL LANGUAGE PROCESS SUGGEST
A SENSITIVITY SPECIFICITY
OPERATION, AND IT TURNS OUT THAT
IN MEETINGS I SET VERY RARELY,
AND SOME MEANINGS ARE SET IN
SMALL WAYS SO YOU HAVE TO HAVE
ENOUGH DATA SO THAT YOU CAN
CAPTURE THOSE THINGS, TOO.
I KNOW THAT BECAUSE WE DID IT.
SO THE POINT IS THEY ARE
SCRUBBED THE DATA JUST TO GET
THE THING UP AND RUNNING BUT
THEY'RE NEGOTIATING TO PLACES
RIGHT NOW TO GET ACCESS TO THE
DATA GOING FORWARD.
BUT IT WAS JUST WHAT THEY HAD TO
DO TO MOVE THIS THING FORWARD.
AT THE VERY LEAST IT'LL KEEP THE
LAWYERS BUSY.
>> YEAH, DO YOU WANT TO GO TO A
MICROPHONE?
>> THE VIDEOCAST AUDIENCE
CANNOT.
>> WE DO WANT TO HEAR YOU.
>> SO THE VERACITY OF THE
INFORMATION PRESENTED BY AN
WELL, MM IS-DO YOU THINK IT'S
MORE DEPENDENT ON THE NURRAL
NETWORK THAT CREATES THE
STATISTICAL PROBABILITIES THAT
IT UT PUTS OR ON THE CONNECTIONS
THAT IT MAKES LIKE AS LIKE AN
LLM HOST LIKE THE NETWORK, DID
THAT MAKE SENSE?
>> REMEMBER, SO IT'S PUTTING ALL
THESE VECTORS IN AND IN THE
NEURAL NET, JUST TAKES THESE
VECTORS AND ORGANIZES THESE
VECTORS,A CORDING TO THE 3
PARAMETERS, RIGHT?
SO IT ORGANIZES ALL OF THESE
THINGS IN TERPS OF BILLIONS OF
PARAMETERS, BUT THEN, IT
OPERATES IN TERMS OF THE JOINT
PROBABILITY, IT LOOKS FOR THINGS
THAT ARE PROBABLY SIMILAR.
OKAY?
AND SO THE INTERNAL WORKINGS OF
THE NEURAL NETIS A JOINT
PROBABILITY OPERATION AND WHAT
TEMPERATURE DOES IS THAT IT
SAYS, WE'RE GOING TO LOOSEN UP
WHAT PROBABILITY WE NEED.
HIRE TEMPERATURE MEANS WE MAY
TAKE THINGS THAT HAVE A LOWER
PROBABILITY OF BEING RELATED.
LOW TEMPERATURE MEANS IT HAS TO
HAVE A REALLY HIGH PROBABILITY
OF BEING RELATED BEFORE WE'RE
GOING TO PUT IT OUT TO THE
CONDITIONAL PROBABILITY.
SO IT'S THE JOINT PROBABILITY
AND OF COURSE HHV JOINT
PROBABILITY AND CONDITIONAL
PROBABILITIES ARE 2 ASPECTS OF
THE SAME THING, RIGHT?
CONDITIONAL PROBABILITY CAN BE
DERIVED FROM THE JOINT
PROBABILITY, RIGHT?
SO IN FACT, WHAT'S REALLY GOING
ON IS THE NEURAL NETIS
ORGANIZING ALL OF THESE THINGS,
AND THEN, THEY'RE LOOKING AT THE
JOINT PROBABILITY OF THESE
THINGS, BASED UPON YOUR PROMPT,
BASED UPON YOUR PROMPT WHAT
JOINT PROBABILITY CAN WE COME UP
WITH.
>> OKAY, RIGHT.
THAT MAKES SENSE.
SO THE SECOND PART OF MY
QUESTION IS IN TERMS OF THE
DEVELOPMENT OF MEDICAL AI, DO
YOU THINK MORE POTENTIAL EXISTS
TO MAKE IT FOR LACK OF A BETTER
TERM BETTER IN THE TWEPMENT OF
LIKE THE NEURAL NETWORK OR IN
THE-
>> THE NEURAL NETIS THE
CLASSIFIER.
THEY'VE BEEN DOING THINGS WITH
HIERARCHICAL NEURAL NETS WHERE
THEY WILL TAKE A NEURAL NET AND
OTHER NEURAL NETS AND 1 NEURAL
NET WILL REFER TO ANOTHER NEURAL
NET, I DON'T SEE THAT AS A
SOLUTION TO ANYTHING BUT THEY'RE
TRYING TO GET AWAY PRACTICES A
BILLION PARAMETERS AND TAKE A
PROMPT AND INSTEAD OF DOING THE
BILLION PARAMETERS, THEY WILL
PUT IT INTO 1 PART OF THE
HIERARCHY AND THAT'S SUPPOSED TO
MAKE IT CHEAPER, BUT THE NEURAL
NET, THE SELF-ORGANIZING NEURAL
NET IS THE SELF-CLASSIFIER AND
THAT PROBABILITIES ARE
MATHEMATICAL SO IT GETS BA BEING
TO THE TRAINING DATA AND YOUR
PROMPT.
>> DR. BURKE, WE ARE UP ON OUR
HOUR.
WE WANT TO THANK YOU FOR JOINING
US AND GIVING US A PEEK UNDER
THE HOOD OF HO AI WORKS AND
OBVIOUSLY THE POTENTIAL IMPACT
IT WILL HAVE FOR US GOING
FORWARD.
I HAD A LOT OF VIDEOCAST
QUESTIONS COMING IN, I WILL
FORWARD THOSE TO YOU SEPARATELY
AND YOU CAN ENGAGE WITH THOSE.
I WANT TO THANK EVERYBODY FOR
JOINING US TODAY AND HOPEFULLY
WE WON'T BECOME SKYNET.
>> OH THANK YOU VERY MUCH.