WELL, HELLO, EVERYONE,
I AM REALLY DELIGHTED TO BE HERE
TO KICKOFF THE J. EDWARD RALL
CULTURAL LECTURE AND FIRESIDE CHAT.
WELCOME EVERYONE, IT IS MY
DISTINCT PLEASURE TO INTRODUCE
OUR SPEAKER TODAY
DR. VINT CERF, A TRUE PIONEER
IN THE DIGITAL WORLD.
BUT FIRST I WILL NOTE THAT THE
ANNUAL NIH RALL CULTURAL LECTURE IS
HOLD IN HONOR OF DR. J. EDWARD RALL,
WHO HELPED TO DEFINE THE
MODERN INTRAMURAL RESEARCH
PROGRAM AT NIH.
IN THE 1950S HE ESTABLISHED A
STABLE, ACADEMIC MINDED AND
CULTURAL RICH COMMUNITY WITHIN
OUR RAPIDLY EXPANDING GOVERNMENT
AGENCY.
DR. RALL, WAS KNOWN AS A
CONSUMMATE SCIENTIST, A VERY
CHARISMATIC MENTOR AND A
RENAISSANCE MAN AND IT IS HE WHO
RECOMMENDED IN 1984 THAT NIH ADD
A CULTURAL LECTURE TO ITS
DIRECTOR'S LECTURE SERIES TO
ENRICH OUR SCIENTIFIC COMMUNITY.
I AM TRULY DELIGHTED THAT
DR. CERF AGREED TO BE OUR
SPEAKER THIS YEAR.
HIS TITLE IS VICE PRESIDENT AND
CHIEF INTERNET EVANGELIST FOR
GOOGLE.
HE WILL TALK TO US ON THE TOPIC OF THE
PROMISES AND PERILS OF AI IN
BIOMEDICAL RESEARCH AND
HEALTHCARE DELIVERY.
THIS IS A TOPIC AS MANY OF YOU
KNOW THAT IS VERY CLOSE TO MY
HEART.
BEFORE I CAME TO NIH IN 2022, I
WAS A SURGICAL ONCOLOGIST AND
RESEARCHER IN BOSTON.
OVER TIME, I BECAME INCREASINGLY
INVOLVED IN LEADING CLINICAL
TRIALS AND WORKED ON THE EFFORTS
TO INTEGRATE STANDARDIZED
BIOMARKERS IN TRIALS, A
COMPLICATED ENTERPRISE TRYING TO
MIX BASIC SCIENCE END POINTS
INCLUDING COMPLEX GENOMICS AND
CLINICAL DATA AND NOW WE HAVE
ANOTHER--YET ANOTHER COMPLEX
HORIZON FOR US THAT IS, THAT OF
USING ELECTRONIC HEALTH RECORDS
AS WELL AS MACHINE LEARNING FOR
CLINICAL RESEARCH.
AT NIH I VERY MUCH HOPE FOR ALL
OF US TO WORK TOGETHER TO
DELIVER EVIDENCE-BASED
HEALTHCARE TO ALL PEOPLE.
AND I'M VERY EXCITED ABOUT HOW
NEW TECHNIQUES IN MACHINE
LEARNING AND ARTIFICIAL
INTELLIGENCE CAN HELP US ADVANCE
TOWARD THAT GOAL SO OF COURSE, I
LIKE ALL OF YOU AM VERY MUCH
LOOKING FORWARD TO HEARING
DR. CERF'S INSIGHS ON THIS
TOPIC.
DR. CERF HAS BEEN AT GOOGLE
SINCE 2005 AND CONTRIBUTES TO
TO GLOABAL POLICY DEVELOPMENT AND
CONTINUED STANDARDIZATION AND
SPREAD OF THE INTERNET.
PREVIOUSLY IN HIS CAREER HE
WORKED AT MCI, THE CORPORATION
FOR NATIONAL RESEARCH INITIATIVES AND
DARPA, HE IS WIDELY KNOWN AS THE
FATHER OF THE INTERNET HAVING CO
DESIGNED THE INTERNET'S
ARCHITECTURE AND THE TCP/IP
PROTOCOLS, THE SETS OF RULES FOR
MESSAGE FORMATS AND PROCEDURES
THAT ALLOW MACHINES AND
APPLICATION PROGRAMS TO EXCHANGE
INFORMATION.
HE HAS RECEIVED TOO MANY AWARDS
TO LIST HERE TODAY BUT I WILL
NAME A FEW NOTABLE, THE U.S.
NATIONAL MEDAL OF TECHNOLOGY,
THE ALAN TURING AWARD, THE
PRESIDENTIAL MEDAL OF FREEDOM,
MEMBER OF THE BRITISH ROYAL
SOCIETY AND RECENTLY THE MARCONI SOCIETY
LIFETIME ACHIEVEMENT AWARD.
HE SERVES AS AN ADVISOR TO THE U.S.
GOVERNMENT AGENCIES, INCLUDING THE NSF, NASA,
DEPARTMENT OF DEFENSE, ENERGY
AND COMMERCE, AND I HOPE US.
HE HOLDS BACHELOR OF SCIENCE
DEGREE IN MATHEMATICS FROM STANFORD,
A MASTER IN SCIENCE AND Ph.D.
DEGREES IN COMPUTER SCIENCE
FROM UCLA AS WELL AS MANY, MANY
HONORARY DOCTORATES SO I VERY
MUCH LOOK FORWARD TO HEARING
DR. CERF TODAY AND THEN ALSO TO
OUR CONVERSATION WHICH WILL
OCCUR AFTER THIS TALK, WHICH
WILL ALSO GO OVER SOME OF THE
QUESTIONS THAT ALL OF YOU HAVE
SUBMITTED SO PLEASE JOIN ME
IN WELCOMING
DR. CERF.
[ APPLAUSE ]
>> WELL, GOOD AFTERNOON,
EVERYONE, THANK YOU VERY MUCH FOR JOINING
US TODAY, I CONSIDER THIS AN
EXTRAORDINARY HONOR, I AM NOT
SURE I DESERVE IT BUT YOU KNOW I'LL TAKE IT
ANYWAY.
I'M EXCITED ABOUT WHAT WE ARE
GOING TO TALK ABOUT BUT I NEED
TO WARN YOU AHEAD OF TIME THAT
I'M NOT NECESSARILY THE WORLD'S
BEST EXPERT ON ARTIFICIAL
INTELLIGENCE.
I'M A NETWORKING GUY BUT I WORK
FOR A COMPANY AT GOOGLE WHERE WE
MADE INVESTMESTS IN ARTIFICIAL
INTELLIGENCE FOR QUITE A LONG
TIME.
MAJOR PROGRESS HAS BEEN MADE
ESPECIALLY IN MACHINE LEARNING
WHICH IS A THIRD FORM OF
ARTIFICIAL INTELLIGENCE, THE
TERM WAS DEVELOPED IN THE 1960S
BY JOHN MCCARTHY AND HIS
COLLEAGUES AT MIT AND THEN
THEY STARTED OUT WITH HEURISTIC
PROGRAMMING WHICH ROUGHLY
SPEAKING MEANT SOMETIMES IT
WORKS AND SOMETIMES IT DOESN'T.
IF IT WORKS ALL THE TIME, THAT'S
ENGINEERING, IF IT DOESN'T,
THAT'S ARTIFICIAL INTELLIGENCE.
[LAUGHTER]
THEN THEY HAD SOME MORE
ORGANIZED TECHNOLOGY CALLED
EXPERT SYSTEMS, THESE ARE
ROUGHLY SPEAKING COLLECTIONS OF
IF, THEN, ELSE PHRASES, YOU KNOW
IF THIS, THEN THAT.
THEY WOULD CREATE LARGE NUMBERS
OF THESE IF, THEN, ELSE PHRASES
AND THEY WOULD TRIGGER EACH
OTHER, IF THIS CONDITIONS HOLDS
YOU THEN YOU SHOULD GO DO THAT,
AND IF YOU DID THAT, THEN SOME
OTHER CONDITION WOULD SHOW UP
AND YOU WOULD KEEP TRAINING YOUR
WAY THROUGH.
THE PROBLEM WITH THAT IS THAT AS
THE NUMBER OF IF, THAN ELSE
PHRASES GOT BIG, THE SYSTEM
STARTED TO BE INCONSISTENT AND
IN SOME CASES COLLIDING WITH
EACH OTHER.
SO IT DIDN'T SCALE ALL THAT
WELL, SO THEN THE NEXT THING
THAT HAPPENS IS WHAT WE NOW CALL
NEURAL NETWORKS.
THEY HAD THEIR ORIGINS IN THE
1960S BUT THE NEURAL NETWORK WAS
ONLY ONE LAYER DEEP AND IT DIDN'T
DO VERY MUCH.
AND IN FACT, MCCARTHY AND SOME
OF HIS ACTUALLY, MINSKy AND PAPERT AT MIT WROTE A
BOOK CALLED PERCEPTRONS WHICH
TRASHED THE IDEA OF NEURAL
NETWORKS ON THIS THE BASES OF
THIS ONE LAYER DEEP PERCEPTRON
AND THAT INVOKED A NEW CLEAR
WINNER FOR THE PERIOD OF ABOUT A
DECADE.
BUT THERE SOME PEOPLE WHO HUNG ON AND SAID YOU IF
WE COULD BUILD A DEEPER NETWORK
WE AUGHT TO BE ABLE TO CAPTURE
MORE STATE INFORMATION THAT
WOULD ALLOW US TO DETECT AND
SENSE CORRELATIONS THAT WOULD BE
OF INTEREST AND THAT'S WHERE WE
HAVE COME AS A RESULT IN MACHINE
LEARNING.
NOW ALL OF YOU I'M SURE HAVE
BEEN EXPOSED TO ONE OF THE ASPECTS
OF MACHINE LEARNING CALLED LARGE
LANGUAGE MODELS AND I SUSPECT WE
WILL GET INTO THIS DURING OUR
DISCUSSION BUT I WILL TELL YOU
THAT TODAY'S LARGE LANGUAGE MODELS ARE
SIMULTANEOUSLY UNBELIEVABLY AND
UNXEBTEDLY BRILLIANT AND EQUALLY
CONFUSING.
IN FACT, YOU KNOW I LOOK A
LITTLE BIT LIKE SIGMUND
FREUD SO I'VE BEEN THINKING OF
RECORDING A LITTLE VIDEO.
>> WELL WHAT WE HAVE NOW IS THE
ARTIFICIAL HEAD ON THE
ARTIFICIAL EGO AND WHAT THEY'RE
MISSING ARE THE ARTIFICIAL SUPER
EG0 TO CONTROL THE
UNCONTROLLABLE PULSES OF THE
ARTIFICAL HEAD.
[ APPLAUSE ]
AND [LAUGHTER]
AND IF I HAD ONE I WOULD HOLD UP A
CIGAR AND SAY SOMETIMES A CIGAR
IS JUST A CIGAR.
[LAUGHTER]
SO WHAT IS IMPORTANT THOUGH FOR YOU TO KNOW IS
THAT QUITE APART FROM THE LARGE
LANGUAGE MODELS AND THE KIND OF
HALLUCINATIONS THAT THEY STILL
EXHIBIT THEY STILL SHOW
REMARKABLE CAPABILITY, THERE IS
SOMETHING GOING ON IN THOSE
MODELS THAT IMPLY A LOT OF
SEMANTICS ARE HIDING IN THE
STATISTICS OF WORD
RELATIONSHIPS.
THAT'S HOW THEY WORK.
THEY GENERATE OUTPUT BASED ON
THE STATISTICAL RELATIONSHIP OF
WORDS OCCURRING ADJACENT TO EACH
OTHER OR SOME DISTANCE APART.
BUT THOSE WORDS HAVE COME FROM
SOMETHING THAT HAD MEANING AND
SO THERE IS A GHOST INSIDE THE
LARGE LANGUAGE MODEL AND WE
DON'T FULLY UNDERSTAND OR
APPRECIATE ALL THAT IT CAN DO
BUT WE DO KNOW THERE ARE SOME
THINGS IT DOESN'T DO WELL, SO
THE CONSEQUENCE OF THAT HOWEVER
SHOULD NOT DISTORT YOUR
UNDERSTANDING AND APPRECIATE FOR
WHAT WILL BE POSSIBLE HERE AT
NIH WHEN IT COMES TO GATHERING
LARGE AMOUNTS OF DATA,
UNDERSTANDING CORRELATIONS THAT
ARE EXHIBITED BY THE THAT DATA
AND THEN LOOKING FOR THE CAUSAL
MODEL THAT WILL EXPLAIN WHAT
THOSE CORRELATIONS ARE.
AND THAT'S SOME OF WHAT I HOPE
WE'LL GET TO TALK ABOUT THIS
AFTERNOON.
SO I'M GOING TO STOP HERE WITH
MY LITTLE SERMON WHICH IS WHAT
CHIEF INTERNET EVANGELISTS DO AND
LET'S GET STARTED WITH OUR
DISCUSSION.
[ APPLAUSE ]
I NEED TO WARN EVERYBODY, I WEAR
HEARING AIDS AND I DON'T KNOW
HOW GOOD OR BAD THE ACOUSTICS
ARE HERE SO I'M GOING TO PUT A
HEADSET ON.
I AM NOT LISTENING TO THE LATEST
FOOTBALL GAME; AT LEAST I WON'T
ADMIT IT EVEN IF I WERE.
I AM HOPING TO GET GOOD QUALITY
SOUND BECAUSE I WOULD LIKE TO
MAKE SURE I HEAR THE QUESTIONS.
SO HERE WE GO, AND I HAVE TO
ADJUST MY HEARING AIDS IN THE
PROCESS.
THERE WE GO.
THE THEORY IS I CAN HEAR YOU
WELL.
LET'S TRY THAT.
>> ARE YOU READY?
>> YEAH, WELL I DON'T KNOW IF
I'M READY BUT I CAN HEAR YOU.
>> EXCELLENT.
OKAY, HERE WE GO, WELCOME
DR. CERF AND THANK YOU FOR
JOINING ME FOR THIS
CONVERSATION.
A MAJOR PROBLEM WITH HEALTHCARE
IS THAT ACTUALLY VERY, VERY
LITTLE OF IT IS BASED UPON
RIGOROUS CLINICAL DATA.
>> OH MY GOD, THAT'S SCARY AS
HELL.
>> ISN'T THAT SCARY?
[LAUGHTER]
AND JUST TO GIVE YOU AN IDEA
RIGOROUS ANALYSIS OF AT LEAST I
KNOW FROM THE FIELDS OF ONCOLOGY
AND CARDIOLOGY, ABOUT BETWEEN 5
AND 10% OF THE DECISIONS THAT
ARE MADE IN THE CLINIC ARE
ACTUALLY BASED ON RANDOMIZED
TRIALS.
THAT'S IT.
>> CAN I GET A CHANGE OF
UNDERWEAR NOW?
>> EXACTLY.
SO IT REALLY IS FRIGHTENING AND
IT DOES SAY SOMETHING ABOUT
THE NEURAL NET THAT IS THE
HUMAN BRAIN OF A DOCTOR THAT OUR
CLINICIANS REALLY ARE ABLE TO
PUT TOGETHER THESE REALLY
DIVERSE COLLECTIONS OF DATA
INCLUDING THEIR OWN PERSONAL
EXPERIENCE AND COME UP WITH
SOMETHING THAT REALLY DOES
IMPROVE PEOPLE'S HEALTH TO A
GREAT DEGREE.
SO WE'RE NOT GOING TO DENIGRATE
THE CONTRIBUTION OF THE HUMAN
BRAIN OR THE EXPERIENCE OF A
CLINICIAN, THOSE ARE REALLY
CRITICAL BUT WE ALL THINK WE CAN
DO BETTER.
WE CAN INTRODUCE BETTER DATA TO
MAKE BETTER DECISIONS.
>> YOU KNOW IT JUST OCCURS TO ME
THAT IT'S A GOOD THING THIS IS
CALLED THE NATIONAL INSTITUTES OF
HEALTH AND NOT THE NATIONAL
INSTITUTES OF DISEASE BECAUSE I
DON'T KNOW IF ANYBODY WOULD FUND
THAT, YOU KNOW?
>> EXACTLY.
[LAUGHTER]
SO MY BIGGEST AMBITION AS NIH
DIRECTOR IS TO PRODUCE MORE
EVIDENCE-BASED DATA DRIVEN
HEALTHCARE --
>> I LOVE THAT.
>> ACROSS THE WORLD
AVAILABLE TO EVERY PATIENT.
AND I WOULD LIKE EVERY DAY
VISITS TO THE DOCTOR, THIS IS
THE IMPORTANT PART, EVERY DAY
VISITS TO THE DOCTOR TO
CONTRIBUTE TO OUR EVIDENCE-BASE.
>> YES, ACTUALLY, ACTUALLY, THIS
IS A CONVERSATION, THIS ISN'T
GOING TO BE JUST Q&A.
THROW AWAY THE QUESTIONS.
OKAY, SO, VISIT TO THE DOCTOR,
IF I WERE A DOCTOR AND MY
PATIENTS ONLY CAME IN WHEN THEY
HAD SYMPTOMS AND THEY WERE SICK,
MY STATISTICS WOULD SAY, MY
PATIENT IS ALWAYS SICK BECAUSE I
NEVER SEE THEM WHEN THEY'RE
WELL, SO ONE OF THE THINGS THAT'S VERY
IMPORTANT I THINK IS FOR US IN
TERMS OF EVIDENCE-BASED MEDICINE
IS TO COLLECT INFORMATION ABOUT
MY PATIENTS WHEN THEY'RE WELL,
ALSO, SO I HAVE SOME SENSE FOR
WHAT'S NORMAL FOR THEM AND THAT
WILL HELP ME FIGURE OUT WHETHER
THEY HAVE GONE IN SOME EXCURSION
AWAY FROM THE NORM.
SO INSTRUMENTING OURSELVES MIGHT
TURN OUT TO BE VERY BENEFICIAL
FOR OURSELVES AND FOR MEDICINE
IN GENERAL AND I'M NOT SURE WHAT
KINDS OF INSTRUMENTATION WE
MIGHT WANT BUT YOU'D KNOW THAT
BETTER THAN I DO.
I DO THINK WE NEED TO THINK
ABOUT THAT.
WE NEED TO HAPPENED PEOPLE'S
STATE OF AFFAIRS WHEN THEY'RE
HEALTHY AS WELL AS WHEN THEY'RE
NOT.
>> COULDN'T AGREE MORE AND SO
MANY OF OUR DISEASES THAT EMERGE
ARE PRESENT FAR, FAR BEFORE EVEN
THE PATIENT IS AWARE OF THEM.
>> NO SYMPTOMS. WE COULD NAME SO MANY.
NO SYMPTOMS.
>> AND THIS IS ANOTHER HOPE THAT
SOMEHOW COLLECTING BETTER DATA
MIGHT HELP US UNCOVER THINGS
THAT MAYBE EVEN THE PATIENT
DOESN'T REALIZE.
>> I LOVE TO HEAR YOU SAY THIS
BECAUSE THINK ABOUT OUR MEDICINE
UP UNTIL THE POINT WHERE WE HAD
THE ABILITY TO DO DNA SEQUENCING
AND IT WAS BASED ON JUST ON OBERVATIONS
AND WE WERE GETTING DATA, BLOOD
TESTS AND WE COULD GET URINE
SAMPLES AND ALL THESE OTHER
KINDS OF THINGS BUT THE DNA
SEQUENCING LED US TO THINK THAT
THAT WAS THE TOP OF THE
MOUNTAIN, YOU KNOW, ONCE WE GOT
THAT, WE KNEW EVERYTHING WE
BEING POSSIBLY KNOW AND THEN WE
DISCOVERED PROTEOMICS AND THEN
AFTER THAT WE DISCOVERED METALLOMICS
AND YOUR POINT THOUGH
WHICH IS
I TAKE VERY
SERIOUSLY IS THAT WE CAN CAPTURE
INFORMATION THAT LATER WILL TELL
US A CORRELATION, WE MIGHT
DISCOVER AFTER WE DISCOVERED THAT
THE PATIENT HAS AN ILLNESS OR
DISEASE, WE CAN LOOK BACK AT THE
DATA THAT WE COLLECTED AND SAY,
IS THERE ANYTHING, ANY
CORRELATION IN THERE OR A CAUSAL
MODEL THAT WE CAN ASSOCIATE WITH
IT
AND THAT WOULD SPEED UP OUR
ABILITY TO RECOGNIZE MEDICAL
CONDITIONS WOULDN'T IT.
>> ABSOLUTELY AND THOSE ARE THE
DATA SETS WE LACK TODAY.
WE DON'T HAVE A LONGITUDINAL
RECORD, WE DO IN SOME, WE DO
HAVE SOME REALLY WONDERFUL
LONG-TERM OBSERVATIONAL COHORTS
BUT WE HAVE IT IN PIECES AND WE
DON'T HAVE THEM FOR EVERYONE IN
THE DIVERSITY THAT WE NEED IT SO
WE'RE NOT EVEN ON TO OUR FIRST
QUESTION YET, OKAY.
SEE WHAT I MEAN.
OKAY, SO LET'S DO OUR FIRST
QUESTION AND THIS FITS WITH WHAT
WE'VE BEEN TALKING ABOUT.
WHAT DO YOU SEE AS THE
CHALLENGES AND OPPORTUNITIES ON
THIS FRONT, BASICALLY FOR MAKING
HEALTHCARE TRULY EVIDENCE-BASED?
>> WELL, WE JUST TALKED A LITTLE
BIT ABOUT THAT AND I THINK AS I
IMAGINED ALL OF US ARE NOT
STATIONARY, AND SO WE MOVE
AROUND, AND AS WE MOVE AROUND WE GO TO DIFFERENT
DOCTORS, DIFFERENT CLINICS
AND HOSPITALS, OUR HEALTH
RECORDS ARE SCATTERED ALL OVER
EVERYWHERE, SOME OF THEM ARE
STILL ON PAPER, IN FACT MAYBE
MOST OF THEM AND SO THE SYSTEM
ISN'T CONTRIBUTING TO EVIDENCE AS
WE WOULD LIKE IT, SO ONE THING WE
MIGHT COLLECTIVELY WORK ON IS
HOW TO CAPTURE THAT EVIDENTIAL
INFORMATION AND MAKE IT COHERENT
AND COLLECTED IN SOME SENSE AND
A LOT OF PEOPLE ARE VERY NERVOUS
ABOUT INFORMATION, HIPAA IS
WITH US AND IN A FUNNY WAY, IT
MAY BE INHIBITING OUR ABILITY TO
DO EVIDENCE-BASED MEDICINE.
SO MAYBE WE SHOULD RETHINK THAT
AND RECOGNIZE THAT EACH OF US
HAS THE OPPORTUNITY TO
CONTRIBUTE TO A LARGER DATABASE
OF HEALTH KNOWLEDGE AND CLINICAL
KNOWLEDGE THAT WOULD HELP
EVERYONE INCLUDING OURSELVES.
AND ONE OF THE THINGS WE TALKED ABOUT AT ONE POINT
IN THE TEAM WAS WHAT IF EVERYTIME SOMEONE
WHO CAME INTO THE DOCTOR'S OFFICE
WITH A PARTICULAR DISEASE,
DIABETES, WE'LL JUST PICK THAT,
THEY COULD BE ASKED TO BE PART
OF A CLINICAL TRIAL.
THEY MAY SAY YES OR SAY NO, BUT
WHAT IF EVERYBODY WHO HAS
DIABETES IS PART OF THE DATA
COLLECTION AND THE ONES WHO--SO
NOW YOU HAVE LARGE OBSERVATIONAL
COHORTS WITH THE CLINICAL TRIALS
NESTED WITHIN THEM AND THE
ABILITY TO LOOK AT THE
DIFFERENCE AND ONE OF THE MOST
CRITICAL THINGS THAT WE THINK
ARE ALL UNDERSTANDING, IS WHO
SAYS YES TO THE TRIAL AND HOW
DIFFERENT THOSE PEOPLE ARE AND
EVEN WHEN PEOPLE HAVE THE SAME
DISEASE, RIGHT?
>> YEAH, BUT THIS IS VERY
RELEVANT TO ME, MY 2 BROTHERS
ARE DIABETIC SO THAT WOULD BE
HELPFUL TO DO.
>> OKAY, HOW DO YOU THINK AI AND
MACHINE LEARNING WILL HELP MAKE
THE CONNECTIONS WE NEED BETWEEN
THE BENCH AND THE BEDSIDE
PRACTICALLY?
>> WELL AS A PRACTICAL MATTER
WHAT WE'VE LEARNED ABOUT MACHINE
LEARNING IS THAT IT'S CAPABLE
OF INGESTING UNBELIEVABLE AMOUNT
OF DATA AND THEN DISCOVERING
RELATIONSHIPS.
NOW SOMETIMES THOSE THEY'RE NOT CAUSAL
AND THEY'RE HAPPEN STANCE, BUT
WHERE WE CAN IDENTIFY A CAUSAL
MODEL THEN HAVING COLLECTED ON
ALL THAT DATA GIVES US
CONFIDENCE THAT THE CAUSAL MODEL
IS LET'S SAY ACCURATE, IT'S
USEFUL FOR PREDICTIONS OR FOR
DIAGNOSIS, BUT WE RIGHT NOW, WE
DO NOT HAVE A REGULAR PRACTICE
OF ACCUMULATING THAT INFORMATION
AND PUTTING IT INTO A FORM WHERE
WE CAN PERFORM THOSE KINDS OF
MACHINE LEARNING ALGORITHMS SO
WE MAY HAVE TO FIND METHODS BY
WHICH PEOPLE ARE--HAVE INCENTIVE
TO PARTICIPATE IN THE DATA
COLLECTION PROGRAM.
YOU JUST SUGGIESTED VERY
INTERESTING WAYS OF DOING THAT.
AND I THINK THE PEOPLE
UNDERSTOOD THAT SHARING OF THAT
DATA IS ALMOST AS IMPORTANT AS
GIVING BLOOD.
KNOWLEDGE IS REALLY IMPORTANT.
>> SO TECHNICALLY DO WE HAVE THE
LEVEL OF TECHNOLOGY THAT ALLOWS
US TO DO THIS WITH ENOUGH
SECURITY AND SAFETY TO CONVINCE
PEOPLE TO CONTRIBUTE THEIR
HEALTH DATA.
LIKE LET'S SAY SOMEBODY HAS A
DISEASE THAT HAS SOME STIGMA
ASSOCIATED WITH IT.
OF COURSE, THAT WOULD GIVE THEM
PAUSE, DO WE HAVE THE TECHNOLOGY
TO ALLOW THAT LEVEL OF SECURITY
AND COULD SOME OF THESE
ALGORITHMS HELP US WITH THAT.
>> I'M PRETTY SURE THE ALGORITHMS
WILL HELP BUT I WILL SAY
THAT IT'S SOFTWARE AND SOFTWARE
HAS BUGS AND BUGS GET EXPLOITED.
AND WHEN PEOPLE WAVE THE
CRYPTOGRAPHY FLAG AROUND, WHICH
IS NOT UNREASONABLE, WE CAN
PROTECT DATA FROM EXPOSURE BUT
SOMETIMES THE BUGS GET PAST SOME
OF THOSE MECHANISMS.
WE SPENT AN AWFUL LOT OF TIME AT
GOOGLE DOING EVERYTHING WE CAN
TO KEEP DATA FROM BEING EXPOSED
OTHER THAN TO AUTHORIZED
PARTIES.
WE MAKE HEAVY USE OF
CRYPTOGRAPHY TO DO THAT, WE
INSIST ON TWO FOCTOR
AUTHENTICATION AND OTHER
METHODS, AND SO I THINK THE
ANSWER IS THAT TECHNOLOGY EXISTS
TO PROTECT INFORMATION BUT IT
HAS TO EXIST EVERYWHERE WHERE
THE PROTECTION AND NEEDED SO
THERE'S STILL WORK TO BE DONE.
>> UNFORTUNATELY.
OKAY.
HERE'S A NEW ONE, YOU'RE KNOWN FOR
STANDARDIZING INFORMATION TO
ALLOW FOR MORE WIDE SPREAD
COMMUNICATION.
WHAT ARE THE SOME OF THE LESSONS
YOU'VE LEARNED FOR THOSE OF US
IN BIOMEDICAL RESEARCH WHERE
STANDARDIZATION IS PRETTY MUCH
NONEXISTENT, UNFORTUNATELY.
>> WELL, IN A FUNNY WAY, THERE'S
HL7 AND MANY OF YOU KNOW ABOUT,
MANY OF YOU MUST KNOW AS WELL, I
WOULD SAY THAT STANDARDIZATION
IS REALLY IMPORTANT NOT ONLY FOR
CLASSIFYING THE INFORMATION THAT
WE HAVE, SO THAT WE KNOW WHEN WE
HAVE PIECES OF INFORMATION THAT
ARE RELATED, IT ALSO CREATES, AS
YOU SUGGESTED INTEROPERABILITY
AMONG SYSTEMS.
STANDARDS ALSO HAVE AN
INTERESTING POTENTIALLY
HAZARDOUS COMPONENT TO IT, LET
ME TRY TO EXPLAIN WHAT THAT IS,
IN OUR HEALTHCARE SYSTEM, WE
HAVE HEALTHCARE HMOs, HEALTH
MANAGEMENT ORGANIZATIONS, WE
HAVE INSURANCE COMPANIES, THE
WAY THE HEALTHCARE SYSTEM IN THE
U.S. WORK SYSTEM THAT WHEN THE
PATIENT COMES TO SEE THE DOCTOR, IF
THE DOCTOR WANTS TO BE COMPENSATED
BY ONE OF THE HEALTHCARE SYSTEMS,
THEY HAVE TO ASSIGN A PARTICULAR
CODE TO EITHER THE VISIT, THE
DIAGNOSIS OR THE TREATMENT.
NOW THAT STANDARDIZATION IS
THERE TO HELP I GUESS ASSESS
WHAT THE VALUE OF THE TREATMENT
WAS OR WHAT THE ACCURACY OF THE
DIAGNOSIS WAS AND WHETHER OR NOT
IT ALL MEETS STANDARDS OF CARE
BUT IT ALSO HAS THIS VERY
PECULIAR PROPERTY THAT IT PIGEON
HOLES AND SHOE HORNS THINGS INTO
PARTICULAR DIAGNOSES AND THAT
SEEMS LIKE IT'S THE ANTITHESIS
OF MEDINE THAT TAKES INTO ACCOUNT THE PERSON AND
THE PERSON'S SPECIFICS.
NOW I'M NOT A DOCTOR AND I COULD
HAVE JUST SAID SOMETHING NOT VERY
PRECISE, BUT IT FEELS TO ME LIKE
THAT STANDARDIZATION WHICH IS
THERE FOR REASONS THAT WE CAN
UNDERSTAND THE FINANCIAL
BUSINESS AND EVERYTHING ELSE,
INTERFERES WITH OUR ABILITY TO
DELIVER HEALTHCARE--I'M GOING TO
USE THE WORDS PERSONALIZED WAY.
SO WE HAVE TO FIGURE THIS OUT
BETTER THAN WE HAVE SO FAR
BECAUSE WE WANT PEOPLE AND WE
WANT DOCTORS SPECIALLY TO
BE ABLE TO TREAT PATIENTS IN
ACCORDANCE TO THE EVIDENCE THAT
THE PATIENT'S CONDITION
EXHIBITS.
AND WE'RE NOT THERE, OUR
STANDARD OF CARE IS TOO
STANDARDIZED.
>> BOY, YOU HAVE HIT PERFECTLY ONE
OF THE MOST DIFFICULT PROBLEMS
IN ALL OF ELECTRONIC HEALTH RECORD
DATA COLLECTION AND THAT IS THE
THING WE CALL THE PROBLEM LIST
WHICH EVERY CLINICIAN OUT THERE
WILL TELL YOU THE PROBLEM LIST
IS A MESS.
YOU KNOW--SOMEONE--SOME
DIAGNOSIS WILL GET PUT ON THAT
LIST AND IT NEVER LEAVES, NEVER
LEAVES THE LIST AND IT MIGHT NOT
EVEN BE TRUE AND IN FACT OFTEN
IS NOT SO IT'S ONE OF THE BIGGEST
ISSUES THAT YOU'VE HIT RIGHT
AWAY.
>> WELL AND YOU WERE TELLING ME
THAT I HIT ON SOMETHING THAT I
DON'T UNDERSTAND BECAUSE WHEN
YOU SAID IT'S ON THE PROBLEM
LIST, HELP ME UNDERSTAND, WHAT
IS IT THAT GETS SOMETHING ON TO
A LIST THAT YOU CALL A PROBLEM
LIST.
WHAT'S THE PROBLEM?
>> SOME CLINICIAN HAS THE PATIENTS
COME IN AND THEY'VE SAID YOU
HAVE HYPERTENSION, IT GOES ON
THE PROBLEM LIST, OH MEAN--
>> OH THIS IS THE LIST OF
PROBLEMS THAT THE PATIENT HAS
OH, OKAY, THANK YOU.
THANK YOU.
NOW I UNDERSTAND.
AND NOW IT'S STITCHED ON YOUR
FOREHEAD NOW.
>> AND THEN FOREVER THAT PERSON
IS LABELED AND IT MIGHT BE THAT
THEY CAME IN AND THEY GOT REALLY
NERVOUS AND EXCITED BECAUSE THIS
DOCTOR TERRIFIED THEM SO THEIR
BLOOD PRESSURE WAS HIGH.
>> AND THEY HAD SOME SYMPTOM AND
THEIR BLOOD PRESSURE WAS UP AND
NOW HYPERTENSION IS ON THEIR
DIAGNOSIS LIST FOREVER.
WOW, SO YOU TRIGGERED ANOTHER
REACTION FOR ME ANYWAY, I THINK
SOMEDAY I WANT TO WRITE AN ESSAY
CALLED YOU ARE NOT YOUR DISEASE.
YOU KNOW AND I DON'T MEAN TO
OFFEND ANY DOCTORS IN HOUSE HERE
BUT I THINK SOME DOCTORS HAVE A
TENDENCY TO THINK OF THEIR
PATIENTS AS THE DISEASE OPPOSED
TO THE PATIENT AND THAT'S PART
OF THE PROBLEM LIST.
>> PERFECT.
OKAY, WE HAVE ANOTHER QUESTION
FOR SOMEONE.
AT NIH WE DEFINITELY BELIEVE IN
THE POWER OF UNLEASHING
INFORMATION, FOR A LITTLE OVER A
YEAR NOW, WE HAVE HAD A NEW DATA
MANAGEMENT AND SHARING POLICY
THAT REQUIRES EVERY
INVESTIGATOR AND INSTITUTION WHO
RECEIVES FUNDING FROM THE NIH TO
PLAN FOR HOW THEY WILL SHARE
THEIR DATA AND HOW THEY WILL NOT
ONLY SHARE THEIR DATA BUT MANAGE
THEIR DATA SO THAT IT CAN BE
SHARED.
SO WHAT ARE SOME OF THE ROAD
BLOCKS THAT YOU FORSEE IN
GETTING USEABLE DATA FROM
RESEARCHERS, AND HOW DO YOU
RECOMMEND WE OVERCOME THOSE
DIFFICULTIES?
>> MAN, HOW MANY HOURS DO WE
HAVE?
THIS IS A HARD PROBLEM.
FIRST OF ALL, IS THE DATA THAT
THEY HAVE COMMENSURATE WITH ANY
OTHER DATA, HOW COULD WE LABEL
IT IN SUCH A WAY--EVEN THE UNITS
OF DATA, IT'S A NUMBER, OKAY,
25.
WELL IS THAT 25 POUNDS OR
25 OUNCES OR YOU KNOW SOMETHING
ELSE?
25 KILOGRAMS, SO FIRST
OF ALL WE HAVE TO FIGURE OUT HOW
WE HAVE COMMON LABELS FOR THE
DATA OR WE HAVE TO TRANSLATE IT
INTO SOMETHING THAT EVERYBODY WILL
UNDERSTAND.
THAT'S PROBLEM NUMBER ONE.
SECOND, THERE COULD BE
A LOT OF IT.
SO WHEN YOU TELL SOMEBODY, YOU
HAVE TO FIGURE OUT HOW TO SHARE
YOUR DATA, WE BETTER HAVE
IN MIND SOMETHING TO TELL THEM
TO HELP THEM DO THAT.
IF SOMEBODY SAYS I'M HAPPY TO DO BUT I
DON'T KNOW WHERE TO DO IT OR PUT
IT, THERE'S MORE THAN I CAN
AFFORD TO STORE AWAY, SO YOU
HAVE TO OVERCOME SCALE.
YOU HAVE TO EVERYCOME SEMANTICS
OF THE DATA IN THE STANDARDIZED
REPRESENTATIONS AND OH, BY THE
WAY, WE DIDN'T EVEN GET TO THE
OTHER HARD PART WHICH IS
PRESERVING IT OVER PERIODS OF
TIME.
FORTUNATELY OUR LIFETIMES ARE
GETTING LONGER BUT WHAT THAT
MEANS IS THAT THE DATA ABOUT US
OUR HEALTH DATA SHOULD BE
PRESERVED OVER ENTIRE LIFETIME
BECAUSE IT COULD TELL US A GREAT
DEAL TOWARDS OF END OF OUR LIVES
ABOUT WHAT PROBLEMS MAY HAVE
ARISEN THAT THEY MAY HAVE
ORIGINS MUCH EARLIER, MIGHT EVEN
HAVE BEEN ANALYTICALLY
DETECTABLE.
SO HOLDING ONTO THAT DATA OVER A LONG PERIOD OF TIME
FOR DIGITAL PRESERVATION IS A BIG
CHALLENGE.
IT USED TO BE THAT WE WROTE OUR
STUFF ON LITTLE CLAY TABLETS IN
CUNEIFORM THAT WAS 6000 YEARS
AGO AND THE FUNNY THING IS THOSE
TABLETS ARE STILL AROUND IF YOU
CAN READ CUNEIFORM THAT'S NOT
TOO COMMON, BUT THEY WERE
WAREHOUSE RECORD ANDS THE
WAREHOUSES BURNED DOWN AND BAKED
THE CLAY, SO THE CLAY LASTED
6000 YEARS AND THEN THERE WAS
VELUM AND CALF SKIN AND THE LIKE
AND THAT LASTS A COUPLE THOUSAND
YEARS, AMAZINGLY BUT IT'S ONLY A
COUPLE THOUSAND COMPARED TO SIX
AND THEN COMES RAG PAPER, WHICH
LASTS 500, 600 YEARS AND WE'RE
GETTING WORSE AND WORSE AT
STORING OUR INFORMATION.
SO HOW MEAN PEOPLE HAVE 3 AND
HALF INCH FLOPPY DISKS OR 5 AND
A QUARTER INCH FLOPPY DISKS AND
EVEN IF THE BITS ARE STILL
THERE, DO YOU HAVE A READER THAT
CAN READ IT.
SO ONE OF THE BIG PROBLEMS
ASSOCIATE WIDE YOUR QUESTION IS
NOT ONLY THE FORMAT AND
COMMONALITY AND STANDARDIZATION
BUT IT'S ALSO HOW THE HELL DO I
PRESERVE ALL THAT DIGITAL
INFORMATION LONG ENOUGH AND AT
SCALE. SO THAT'S WHERE
COMPANIES LIKE GOOGLE ARE HARD
PRESSED TO INCREASE THE TOTAL
AMOUNT OF ABILITY TO STORE
INFORMATION.
>> IT'S A TOUGH PROBLEM.
>> BUT I LOVE THAT PROBLEM.
SCALE IS FUN TO TACKLE.
YOU KNOW, MAKE IT BIGGER.
WELL, I STILL HAVE MY OLD
LABORATORY NOTEBOOKS WITH ALL
THE WRITING IN THEM BUT I ALSO
HAVE SOME 3-INCH FLOPPY DISKS
AND MY LABORATORY NOTEBOOKS I
CAN STILL READ.
I APPRECIATE THAT PROBLEM.
>> EVERYTHING THAT'S ON THOSE
DISKS ... THE PROBLEM, THE DIFFERENCE THOUGH IS THAT
YOU CAN READ THE LAB NOTEBOOKS
BUT THE COMPUTER MIGHT NOT BE
ABLE TO READ THEM AND IF YOU
WANTED TO EXERCISE THE KIND OF
TOOLS THAT WE'VE BEEN TALKING
ABOUT, WE NEED TO TURN THAT INTO
DIGITAL INFORMATION.
>> OKAY, NIH IS INVESTED IN
ADVANCING POINT OF CARE
TECHNOLOGIES WHICH HELP MEET
PEOPLE WHERE THEY ARE, WHERE
THEY LIVE, WHERE THEY WORK,
INCLUDING IN THEIR OWN HOME ANDS
WE ALSO HAVE A CRITICAL NEED, WE
HAVE TRUST ISSUES WITH OUR
PATIENTS, THE PEOPLE WERE TRYING
TO SERVE SO WE ALSO WANT TO
INVOLVE PEOPLE MORE IN THEIR OWN
DECISION MAKING AND IN SELF-CARE
AS WELL AS IN THE RESEARCH
PROCESS.
SO THIS CAN INCLUDE DEVICES THAT
CONNECT TO THE INTERNET AND
HEALTH SYSTEMS, THE INTERNET OF
THINGS, HOW DO YOU THINK THE
INTERNET OF THINGS DEVICES CAN
IMPROVE HEALTHCARE.
>> WELL, FIRST OF ALL THERE'S NO
DOUBT IN MY MIND THAT THEY CAN.
PART OF THE REASON FOR THIS IS AGAIN
GOES BACK TO MY EARLIER
OBSERVATION THAT BEING ABLE TO
OBSERVE A PATIENT 24 HOURS A
DAY FOR DAYS ON END OR WEEKS OR
MONTHS MEANS THAT WE CAN
CALIBRATE THAT PATIENT'S
CONDITION RELATIVE TO THAT WHICH
IS BEING MEASURED.
THAT'S REALLY IMPORTANT
WE CAN DO ALERTS AND THINGS LIKE
THAT IF WE SEE SOME VARIATION
THAT IS EXTREME FOR EXAMPLE, SO
BUILDING A MODEL OF AN
INDIVIDUAL PATIENT BASED ON THAT
KIND OF DATA IS REALLY QUITE
ATTRACTIVE.
SOME OF YOU PROBABLY WORN
RECORDING DEVICES FOR EXAMPLE,
THAT STRAP TO YOUR WAIST FOR A
PERIOD OF DAYS OR EVEN WEEKS,
THAT INFORMATION COULD BEING
COLLECTED MORE READILY BY WHAT
WE CALL WEARABLES AT GOOGLE,
YOUR WRIST WATCH CAN PICK UP A
LOT.
IT'S AMAZING HOW MUCH YOU CAN
TELL BY STICKING SOMETHING IN
YOUR EAR, IT SOUNDS KINDS OF
SILLY BUT IT'S TRUE.
THERE'S A LOT OF THE BODY'S
CONDITION IS LITERALLY
DETECTABLE INSIDE THE EAR CANAL
SO I CAN EASILY IMAGINE WEARING
DEVICES THAT HELP US, KEEP TRACK
AND WARN US IF SOMETHING'S
HAPPENING.
THOSE OF YOU WHO ARE OR KNOW
ABOUT DIABETICS THAT HAVE
GLUCOSE DETECTORS AND AUTOMATIC DETECTION AND IN SOME
CASES AUTOMATIC INJECTIONS OF
INSULIN IN ORDER TO RESPOND TO
THE CURRENT STATE OF GLUCOSE,
WILL KNOW THAT THOSE DEVICES ARE
REALLY QUITE EFFECTIVE.
NOW WE WORRY A LOT, ANYWAY,
ABOUT THE CORRECT OPERATION OF
THE DEVICE, SO WHATEVER SOFTWARE
IS INVOLVED IN THOSE DEVICES
ABSOLUTELY HAS TO BE RIGIDLY
TESTED AND CHECKED AND VALIDATED
AND EVERYTHING ELSE BUT I SEE,
THE MONITORING OF SENSORY
INFORMATION IS A VERY IMPORTANT
FUTURE FOR OUR ABILITY TO
PROVIDE BETTER HEALTHCARE.
CAN I JUST MENTION THAT I WAS IN
SAUDI ARABIA RECENTLY AND
VISITED A VIRTUAL HOSPITAL WHERE
THE PATIENTS ARE REMOTE BUT THEY
HAVE LOCALLY IN THEIR HOMES AND
ON THEIR PERSON, SENSORY DEVICES
THAT ALLOW A DOCTOR AT THE
CENTRAL VIRTUAL HOSPITAL TO
GATHER THEIR DATA, HAVE
DISCUSSIONS AND VIDEO CALLS AND
SO ON WITH THE PATIENTS SO THAT
THEY CAN GET HEALTHCARE THAT
THEY WOULDN'T OTHERWISE GET
BECAUSE THERE ARE NO CLINICS
LOCALLY AVAILABLE.
AND THERE ARE THINGS THAT ARE
STARTING TO HAPPEN, EXAMPLE
SONOGRAPHY, SONOGRAMS ARE
BECOMING MUCH MORE RELIABLE AS
DIAGNOSTIC TOOLS.
I SOUND LIKE I KNOW WHAT I'M
TALKING ABOUT WHICH IS REALLY
AMAZING, SOUNDS LIKE I'M A LARGE
LANGUAGE MODELING PERSON HERE
SO SONOGRAM RESOLUTION HAS
GOTTEN MUCH, MUCH, BETTER AND
THOSE ARE MUCH LESS EXPENSIVE
THAN AN FMRI, OR MRI-TYPE
MACHINE OR CT SCANNERS.
SO I SEE THESE KINDS OF SMALL
PORTABLE DEVICES AS BEING A
SIGNIFICANT IMPROVEMENT IN OUR
ABILITY TO DIAGNOSE ILLNESS AND
TO TRACK HEALTH STATES.
>> AND SOON.
>> YES.
>> THIS ISN'T THE FUTURE THIS,
IS HERE NOW. THIS IS NOW.
>> ALRIGHT.
THESE ARE GETTING HARDER,
MISINFORMATION, DISINFORMATION
AND DECLINING TRUST REMAIN HUGE
CHALLENGES FOR THE SCIENTIFIC
COMMUNITY ESPECIALLY FOR
GOVERNMENT SCIENTISTS.
DO YOU THINK THIS IS GETTING
BETTER OR WORSE, DO YOU THINK
THIS WILL GET BETTER OR WORSE IN
IN THE FUTURE AND HOW DO YOU
THINK WE CAN COMBAT THE PROBLEM
AND DEVISE SOLUTIONS TO
MISINFORMATION.
I TOLD YOU THEY'RE GETTING
HARDER.
>> YOU DON'T HAPPEN TO HAVE
ANOTHER QUESTION, DO YOU?
>> I DO.
WE CAN SKIP THAT.
>> NO, NO, I WILL NOT DUCK THAT
BECAUSE THAT IS A VERY
IMPORTANT PROBLEM.
SO THERE ARE SEVERAL THINGS THAT WE'RE GOING TO HAVE TO DO, FIRST
OF ALL WITH REGARD TO
INFORMATION, PROVIDENCE OF THE
INFORMATION IS INCREASINGLY
IMPORTANT, WE HAVE TO KNOW WHERE
STUFF COMES FROM, WE HAVEN'T
DONE AS GOOD A JOB AS I THINK WE
WERE ARE GOING TO HAVE TO DO IN ORDER TO
IDENTIFY WHERE THINGS COME FROM.
EVEN IF WE KNOW WHERE THEY COME
FROM, THEN WE HAVE TO ASSESS
WHETHER OR NOT THEY COME FROM A
SOURCE THAT I DON'T CARE TO
TRUST OR NOT.
MAYBE IT'S A SOURCE THAT WE HAVE
TRUSTED AND SUDDENLY WE HAVE TO
DECIDE WE DON'T ANYMORE BECAUSE
THE INFORMATION HAS BECOME POLLUTED.
FOR SOME KINDS OF MISINFORMATION
AND DISINFORMATION, THERE IS A
DEFENSE, IT'S CALLED CRITICAL
THINKING AND IT TAKES A FAIR
AMOUNT OF WORK TO DO THAT AND A
LOT OF PEOPLE ARE UNWILLING TO
DO THE WORK TO THINK CRITICALLY
ABOUT WHAT THEY SEE AND HEAR, I
ACTUALLY THINK THAT YOUNG PEOPLE
SHOULD BE TRAINED TO THINK
CRITICALLY, THEY SHOULD BE
ENCOURAGED AND CONGRATULATED FOR
THINKING CRITICALLY ABOUT WHAT
THEY SEE AND HEAR.
NOW THERE ARE SOME PARENTS THAT
DON'T LIKE TO HEAR THAT BECAUSE
THE KID MIGHT COME HOME AND
DISAGREE WITH THEIR PARENTS AND
SOME PARENTS DON'T TAKE THAT TOO
LIGHTLY.
ON THE OTHER HAND, I THINK
THAT'S HOW YOU HAVE TO COPE WITH
A WORLD WHERE SOME INFORMATION
IS NOT TO BE TRUSTED.
SO I THINK THERE'S STILL A LOT
OF WORK TO BE DONE TO LABEL
INFORMATION, TO IDENTIFY TRUSTED
SOURCES AND FRANKLY USE CRITICAL
THINKING.
YOU KNOW EVERY ONCE IN A WHILE,
I THINK--YOU KNOW HOW WE HAVE TO
TAKE A TEST BEFORE YOU GET YOUR
DRIVER'S LICENSE?
I'VE OFTEN THOUGHT THAT PEOPLE
SHOULD HAVE INTERNET DRIVER'S
LICENSE TEST, WHERE YOU HAVE TO
LEARN ABOUT THE HAZARDS OF BEING
ON THE INTERNET ROAD AND THERE
ARE PARTIES OUT THERE ON THE
REAL ROADS THAT DON'T KNOW HOW
TO DRIVE VERY WELL AND THERE ARE
PEOPLE ON THE VIRTUAL ROADS THAT
DON'T HAVE THE TRUTH AT THEIR
FINGERTIPS AND YOU NEED TO
UNDERSTAND THAT AND BE--FIGURE
OUT HOW TO BE DEFENSIVE ABOUT
THAT.
OUR SCIENTISTS ESPECIALLY
THOUGH, HAVE TO BE SKILLED IN
THIS ART BECAUSE OTHERWISE THEY
WILL INGEST BAD INFORMATION
WHICH WILL LEAD TO BAD DIAGNOSES
AND BAD TREATMENTS.
>> SO WHO REGULATES ALL THIS?
HOW DO WE MAKE THIS HAPPEN?
AND WELL THE ME BACK UP TO WHERE
WE WERE BEFORE ABOUT, YOU KNOW
THE INTERNET, SOMEONE HAD TO
DECIDE, THIS IS THE STANDARD
WE'RE GOING TO USE BECAUSE
THERE'S ALWAYS COMPETITION FOR
THESE THING ANDS SOMEBODY HAS TO
SAY, OKAY ENOUGH SILOS, WE'RE
GOING WITH THIS ONE.
YOU KNOW, THINGS LIKE THIS,
OKAY, WE'VE HAD THIS FREE AND
THIS OPEN AND NOBODY KNOWS WHERE THIS COMES FROM, NOW WE NEED
TO SEE AN IDENTIFIER, WHO
CAN--WHO SHOULD BE SETTING--WHAT
ORGANIZATION, HOW SHOULD THESE
STANDARDS COME ABOUT?
>> OKAY, THAT IS A HARD
QUESTION.
>> I KNOW.
>> WELL, FIRST OF ALL THERE IS
INCREASING ATTENTION BEING PAID
TO IDENTITY IN THIS ENVIRONMENT
AND SOME OF US THINK THAT YOU
COULD HAVE A DISTRIBUTED
COLLECTION OF PARTIES WHO WOULD
VERIFY YOUR IDENTITY, AND LET ME
USE THAT WORD IN A VERY BROAD
SENSE TO VERIFY THE ORIGIN OF
INFORMATION THIS IS BACK TO
THINKING ABOUT WHERE DO THINGS
COME FROM, WHAT IS THEIR
PROVIDENCE?
SO HAVING A PARTY THAT CAN
VERIFY A SOURCE AND THEN HAVING A WAY
FOR THAT SOURCE TO LABEL THE
INFORMATION AT THE SOURCE IS
PROVIDED SO WE CAN TRACE IT BACK
TO THE SOURCE, JUST TO BORROW THAT
TERM FOR A MOMENT, TRACEABILITY
IS SOMETHING THAT THE NATIONAL
INSTITUTES OF HEALTH--I'M SORRY,
THE NATIONAL INSTITUTES OF
STANDARDS AND TECHNOLOGY ARE
VERY BIG ON TRACEABILITY.
WHAT THEY MEAN BY THAT IS THE
SCALE AT THE STORE WHERE YOU
WEIGH A POUND OF BANANAS HAS
BEEN CALIBRATED AGAINST A STANDARD
AND YOU CAN TRACE THAT ALL THE
WAY BACK TO PHYSICAL STANDARDS
THAT NIST MAINTAINS, SO THAT
TRACABILITY
IS OUR FRIEND THERE
BECAUSE WE CAN TRUST THE
INSTRUMENTS THAT ARE DOING WEIGHTS ANDS
MEASURES.
THAT'S WHAT THE NATIONAL BUREAU STANDARDS ORIGINALLY STARTED, SO
THE SAME ARGUMENT MIGHT BE MADE
FOR INFORMATION IF WE CAN
IDENTIFY THE ORIGIN OF THE
INFORMATION JUST BECAUSE
WE KNOW WHERE IT CAME FROM
DOESN'T NECESARRILY MEAN IT'S CORRECT OR
ACCURATE, BUT AT LEAST WE KNOW
WHERE IT CAME FROM.
AND IN TERMS OF WHO WOULD
PROVIDE THAT KIND OF VALIDATION,
YOU CAN IMAGINE JUST AS WE HAVE
GENERALLY ACCEPTED ACCOUNTING
PRACTICES, AND WE HAVE COMPANIES
THAT DO AUDITS, YOU CAN IMAGINE HAVING
GENERALLY ACCEPTED IDENTITY
PRACTICES THAT SOME COMPANIES
WOULD BE CERTIFIED TO PROVIDE
THAT LABEL SO THAT YOU COULD USE
A CRYPTOGRAPHIC SIGNATURE SO TO
SPEAK ON THE CONTENT THAT YOU'RE
TRYING TO ESTABLISH AS VALID.
SO I THINK IT DOESN'T HAVE TO BE
CENTRALIZED NECESSARILY, BUT
THERE HAVE TO BE STANDARD
PRACTICES THAT WE ALL AGREE TO
AND WE HAVE TO AGREE TO BE
AUDITED WITH REGARD TO OUR
EXERCISE OF THOSE STANDARDS.
MAYBE THAT'S A PLACE TO START.
>> SCIENTIFIC METHOD, GROUNDED
IN SCIENCE AND GROUNDED IN
RESEARCH IS CERTAINLY FOR
MEDICINE, IS OUR STANDARD,
RIGHT?
>> YEAH, YOU REMEMBER THAT THE
SCIENTIFIC METHOD, IT'S
BEAUTIFUL, I MEAN THERE IS SOMETHING UTTERLY
ELEGANT ABOUT THIS, IT STARTS
OUT BY SAYING, YOU HAVE
OBSERVATIONS AND YOU TRY TO
FIGURE OUT WHAT THE POSSIBLY
HAVE PRODUCED THAT OBSERVATION,
YOU GET A THEORY, AND THEN YOU
TEST THE THEORY IF YOU'RE A GOOD
SCIENTIST AND THEN, IF THE DATA
THAT YOU COLLECT DOESN'T MATCH
WHAT THE THEORY PREDICTED, THEN
YOU HAVE TO ADMIT THE
POSSIBILITY THAT YOUR THEORY IS
WRONG AND THAT'S REALLY HARD FOR
SOME SCIENTISTS, THIS IS MY
THEORY, I'VE BEEN WORKING ON
THIS, FOR 25 YEARS AND YOU KNOW,
YOUR INSTRUMENTATION IS BROKEN.
NOW SOMETIMES THAT'S ACTUALLY
TRUE, SOME OF YOU MAY REMEMBER
READING ABOUT AT CERN WHERE SOMEBODY
THOUGHT THE PHOTONS WERE GOING
FASTER THAN THE SPEED OF
NEUTRINOS WE GOING FASTER
THAN THE SPEED OF LIGHT, WELL IT WAS AN
INSTRUMENTATION PROBLEM BECAUSE
THERE WAS A CONNECTOR THAT
WASN'T CONNECTED PROPERLY AND
THERE WAS A DELAY THAT WAS
SUPPOSED TO HAPPEN BUT SOMETIMES THE
MEASUREMENTS REALLY DO TELL US
THAT OUR THEORIES ARE WRONG.
THIS IS THE BEAUTY OF THE
SCIENTIFIC METHOD IF WE'RE GOOD
ABOUT IT.
HERE IS ONE OTHER THING TO OBSERVE HERE,
AND THAT HAS TO DO WITH
MEASUREMENT, ITSELF, SOME OF THE
MOST FUNDAMENTAL THINGS IN
SCIENCE COME ABOUT BECAUSE WE'VE
SUDDENLY BEEN ABLE TO MEASURE
SOMETHING WE COULDN'T MEASURE
BEFORE, THINK ABOUT DNA
SEQUENCING, SOMETHING WE COULD
NOT MEASURE BEFORE, WE CAN NOW
MEASURE.
THINK ABOUT SOME OF THE OTHER
THINGS THAT WE CAN MEASURE NOW
THAT WE COULDN'T BEFORE.
ONE OF THE MOST INTERESTING
QUESTIONS, ANY SCIENTIST AND
MAYBE EVEN ANY DOCTOR COULD ASK,
IS WHAT IT THAT I CAN'T MEASURE
NOW THAT IF I COULD MEASURE
WOULD MAKE A BIG DIFFERENCE IN
MY UNDERSTANDING OF DISEASE
OR IN MY TREATMENT OF
PATIENTS.
THERE ARE MILLIONS OF QUESTIONS,
MILLIONS OF THINGS
THAT WE CAN'T MEASURE NOW THAT
ARE EVENTUALLY GOING TO ALLOW US TO TREAT
PATIENTS BETTER.
>> SO, WHILE THE IMPORTANT THING
WOULD BE TO FIGURE OUT WHICH OF
THOSE THINGS WE DON'T KNOW HOW
TO MEASURE RIGHT NOW, WE SHOULD
TRY TO TACKLE FIRST.
AND TO GIVE AN EXAMPLE OF
SOMETHING THAT'S REALLY AMAZING
AND NSF THERE WAS THIS
LIGO EXPERIMENT,
LASER INTERFEROMETER GRAVITY
OBERSERVATION, IT WAS A 40 YEAR
PROJECT, WHEN IT WAS FIRST
PROPOSED, THE SCIENTIST WHO
PROPOSED IT KNEW THAT IF THEY BUILT
THE MACHINE IT WASN'T GOING TO
WORK BUT THEY HAD TO BUILD IT TO
FIGURE OUT HOW FAR OFF THE
MEASUREMENT WAS TO FIGURE OUT
WHAT IT WAS TO CHANGE TO GET
BETTER MEASURES, THEY WENT
THROUGH THREE CYCLES OF DESIGN AND
IMPLEMENTATION, EACH ONE GETTING
CLOSER AND CLOSER AND BETTER
REFINEMENT AND MORE ACCURATE
MEASUREMENT UNTIL FINALLY, A FEW
YEARS AGO, ONE OF THEM WAS IN TEST
MODE, AND IT DETECTED TWO BLACK HOLES
CIRCLING EACH OTHER AND
COLLIDING AND A BILLION LIGHT
YEARS AWAY.
AND THAT WAS AN UNMISTAKABLE
SIGNAL, FORTUNATELY THERE WERE TWO
SUCH MACHINES AND THE SECOND ONE
ALSO DETECTED THE SAME SIGNAL
AND THERE WAS NOTHING MESSY
ABOUT IT AT ALL, I MEAN IT WAS A
PERFECT TRACE OF WHAT IT WOULD
PREDICT FROM THE--IN TERMS OF
THE GRAVITATIONAL FLUCTUATION.
SO THAT'S AN EXAMPLE, WHERE WE
KNEW THAT WE COULDN'T MEASURE
SOMETHING, BUT WE KNEW THAT WE
HAD TO BUILD SOMETHING TO FIND
OUT HOW FAR OFF WE WERE, THAT'S
PROBABLY GOING TO BE TRUE FOR A
LOT OF OTHER THINGS.
>> WELL, IT SOUNDS LIKE
ARTIFICIAL INTELIENCE TO ME.
>> WELL, IT'S CERTAINLY A THING,
WHICH IS NOT WORKING THE WAY WE
WOULD LIKE IT TO AND SO WE KNOW
WE HAVE WORK TO DO.
GOOD POINT.
>> YEAH, OKAY.
WHAT ADVICE DO YOU HAVE FOR
THOSE HERE WHO ARE JUST STARTING
THEIR SCIENTIFIC CAREERS?
>> WELL, LET'S SEE THE STOCK
ANSWER THAT I HAVE TO THAT IS YOU
SHOULD GO INTO ASTROPHYSICS.
[LAUGHTER]
I WILL EXPLAIN.
THERE'S A RATIONAL FOR THAT.
YOU KNOW A HUNDRED YEARS AGO, WE
THOUGHT THAT WE KNEW PRETTY MUCH
HOW THE UNIVERSE WORKED, WE HAD
THIS GREAT UTONNIAN MODEL AND IT
MADE GREAT PREDICTIONS AND WE
FIGURED WELL IF WE JUST MEASURE
THE CONSTANTS MORE ACCURATELY,
WE WILL BE ABLE TO PREDICT
EVERYTHING AND THEN EINSTEIN
COMES ALONG AND HE BLOWS THAT
WHOLE THING UP.
AND THEN QUANTUM PEOPLE COME
ALONG AND THEY BLOW UP EINSTEIN
AND YOU KNOW, AT THIS POINT, NOW
A HUNDRED YEARS LATER, WE KNOW
LESS ABOUT THE UNIVERSE THAN WE
THOUGHT WE DID A HUNDRED YEARS
AGO.
WE KNOW ABOUT 5% OF THE UNIVERSE
IS NORMAL MATTER, THAT INCLUDES
ANTIMATTER, AND THEN, 70% IS
DARK ENERGY.
WHAT'S THAT?
I DON'T KNOW BUT IT'S CAUSING
THE EXPANSION OF THE UNIVERSE AND
IT'S ACCELERATING AND THIS IS
SORT OF WTF?
[LAUGHTER]
AND THEN 25% IS DARK
MATTER--I'M SORRY,I SAID THAT WRONG IT'S DARK
ENERGY AND 70% DARK MATTER IS
OTHER STUFF, WHAT'S THAT, WE
DON'T KNOW.
SO WE KNOW ABOUT 5%, SO IF YOU
GO INTO ASTROPHYSICS AND YOU DO
ANYTHING, THE CHANCES OF A NOBEL
PRIZE ARE SKY HIGH.
[LAUGHTER]
SO HONESTLY, TO BE FAIR ABOUT
THE ANSWER HERE, IF I WERE
YOUNG, I'M NOT OBVIOUSLY, BUT IF
I WERE YOUNG, I THINK I WOULD
WANT TO GET DEEPER AND DEEPER
INTO MICROBIOLOGY.
I HAVE BECOME UTTERLY FASCINATED
BY HOW COMPLICATED CELLS ARE AND
I USED TO THINK THE CELL WAS
THIS LITTLE BAG FULL OF
CHEMICALS AND THEY BUMPED INTO
EACH OTHER AND STUFF HAPPENED.
WELL THEN I GOT AHOLD OF THE
MICROCHEMISTRY OF THE CELL, YOU
KNOW THAT BIG 1,700 PAGE THING
AND DISCOVERED THAT THE CELL IS
ABOUT AS COMPLICATED AS DOWN
TOWN MANHATTAN.
AND THERE'S ALL KINDS OF STUFF
GOING ON IN THERE--STUFF GOING
ON IN THERE EMPLOY THERE'S THESE
MICROTUBULES AND THESE PROTEIN
THINGS THAT WALK ALONG THE
TUBULES CARRYING OTHER PROTEINS
THAT ARE LABELED JUST
LIKE PACKETS ON THE INTERNET AND
THEY GET DELIVERED TO THE RIGHT
PLACE INSIDE THE CELL.
I MEAN IT'S--SOMEBODY FROM
UC SAN DIEGO SENT ME A NOTE THAT SAID, HI, THERE'S
AN INTERNET INSIDE THE CELL AND
THEY TALK TO ME ABOUT EXACTLY
THAT.
SO MAN, THAT WOULD BE SO MUCH
DAMN FUN, I WOULD LOVE TO DO
THAT.
>> OKAY.
SO WE'RE NOW GETTING TO THE
SECOND MOST POPULAR QUESTION
RECEIVED FROM ALL OF YOU.
159 UPVOTES FOR THIS ONE. IS THIS ONE
CALLED WHAT WERE YOU THINKING?
>> YEAH.
THIS ONE'S PRETTY PRACTICAL
THOUGH, HOW DO WE ENSURE, FOR
BIOMEDICAL RESEARCH, THE BROAD
AND APPROPRIATE
REPRESENTATIVENESS OF THE DATA
BEING USED TO TRAIN OUR MODELS
GIVEN THE HISTORICAL INEQUITIES
OF LARGE DATA COLLECTION?
>> SO THIS IS REALLY HARD
BECAUSE WE KNOW WE HAVE ENORMOUS
SDIVERSITY IN THIS WORLD BUT
NOT EVERYONE IN THIS DIVERSE
GLOBE HAS ACCESS TO MEDICAL CARE
THAT WOULD CAPTURE THE DATA IN
THE FIRST PLACE.
AND SO, THE IMPORTANT THING IS
WHAT YOU JUST DID, WHICH IS TO
RECOGNIZE THAT WE DON'T HAVE
THAT DATA AND THAT WE SHOULD
TAKE STEPS IF WE POSSIBLY CAN TO
GET THAT DATA. IT MAY BE AS
THE COST OF GATHERING
INFORMATION ABOUT PEOPLE'S
HEALTH GETS LESS AND LESS AND
MORE AFFORDABLE AND MORE
FEASIBLE AND MORE SUPPORTABLE,
WE MAY BE ABLE TO GET MORE PEOPLE
WILLING TO CONTRIBUTE DATA ABOUT
THEIR--THE WAY THEIR BODIES WORK
OR ARE WORKING.
IN FACT, IF I CAN MAKE A GENERAL
OBSERVATION.
SOMETIMES YOU TAKE THE BIGGEST
AND MOST EXPENSIVE TECHNOLOGY
AND IT'S ONLY USED BY PEOPLE
THAT CAN AFFORD IT, HOW ABOUT
COMPUTING IN THE EARLY
DAYS, ONLY VERY BIG COMPANIES COULD
AFFORD TO HAVE A COMPUTER AND
THEY WERE SO PROUD OF IT THEY
PUT IT IN A GLASS BALLROOM AND
SAID LOOK, WE HAVE
COMPUTERS AND YOU DON'T.
WELL, NOW WE HAVE THEM IN OUR
POCKETS.
I HAVE ONE IN NIGH EAR, AND SO IT BECOMES
SMALLER, LESS ENERGY DEPENDENT AND
AFFORDABLE SO MANY
TECHNOLOGIES HAVE THAT
CHARACTERISTIC. SO ONE THINK WE CAN DO IS TO TRY TO RECOGNIZE EXACTLY THE PROBLEM THAT
YOU SAID AND LOOK FOR WAYS OF GATHERING THAT DATA
OF LOWER AND LOWER COST
TRYING TO PROVIDE BENEFIT TO A LARGER
AND LARGER POPULATION AND
ACCUMULATE AS MUCH DATA AS WE
CAN TO BUILD BETTER MODELS BUT
IT HAS TO BE SOMETHING WE
CONSCIOUSLY DECIDE TO DO.
>> YEAH.
AND I WILL ADD TO THAT THAT WE
ALSO RECOGNIZE THAT SEGMENTS OF
OUR SOCIETY WHO HAVE--FOR WHOM
WE HAVE THE LEAST DATA ARE
ALSO THE ONES WITH THE WORST
OUTCOME WHO ARE ALSO THE ONES THAT
HAVE LEAST TRUST IN MANY WAYS
AND SO, WE'VE GOT A SOCIAL
PROBLEM.
SCIENCE ISN'T GOING TO FIX THIS.
IT'S THE ONLY THING THAT'S GOING
TO FIX THIS IS THE HUMAN TO
HUMAN INTERACTION AND THE
EARNING TRUST.
>> WELL, SOCIAL AND ECONOMIC
RESPONSE IS NEEDED.
THIS IS--A LOT OF PROBLEMS WE
SEE, ARE A CONSEQUENCE OF BEING
IN A LOWER SOCIOECONOMIC LAYER
THAN OTHERS AND SO WE HAVE TO
RECOGNIZE THAT.
SO YOUR POINT'S EXTREMELY WELL
TAKEN AND ALL WE CAN DO IS SHOW
BY OUR DELIBERATE ACTION THAT WE
ARE TRUSTABLE.
>> OKAY, SO THIS ONE SHOULD BE
EASY.
IN WHAT WAYS COULD TECHNOLOGY
LIKE CHATGPT AND BARD BE
LEVERAGED, AND THEY ARE THERE, BE
LEVERAGED TO IMPROVE MEDICAL
RESEARCH IN PATIENT CARE AND IF
THEY ARE USED THERE WHAT
MEASURES NEED TO BE TAKEN TO
ENSURE THE ACCURACY AND
RELIABILITY OF THE INFORMATION
GENERATED?
>> WELL I'M GOING TO BORROW FROM
ONE OF MY DISTANT RELATIVES, AT
LEAST WE THINK WE WERE RELATED
BENNETT CERF, AND SOMEBODY WILL REMEMBER BENNETT CERT AND WHAT'S MY LINE
AND RANDOM HOUSE AND THE FACT THAT HE NEVER
MET A PUN HE DIDN'T LIKE. AND
HE WROTE A BOOK CALLED LAUGHTER
IS THE BEST MEDICINE AND MY
REACTION RIGHT NOW IS MAYBE WE
SHOULD USE BARD AND GEMINI
AND CHATGPT FOR ITS LAUGHTER
ELEMENT MORE THAN ANYTHING ELSE,
IT'S GREAT MEDICINE.
THEY ARE GENERALLY NOT READY FOR
PRIME TIME DIAGNOSTICS OR
ANYTHING LIKE THAT OR EVEN
MEDICAL ADVICE.
LET ME SET ASIDE THE LARGE
LANGUAGE MODEL WHICH IS A
QUESTION YOU ASKED AND ARGUE
IT'S NOT READY TO BE HELPFUL TO
US OTHER THAN FOR I WOULD SAY
ENTERTAINMENT.
IT DOES DO--LET ME BE FAIR ABOUT
THIS, THERE IS ONE THING FOR SURE
THAT THE LARGE LANGUAGE MODELS
HAVE BEEN ABLE TO DO WHICH I
FIND ASTONISHING AND THAT'S
LANGUAGE TRANSLATION, SPEECH
UNDERSTANDING, SPEECH SYNTHESIS,
IT'S ASTONISHING HOW WELL THESE
MODELS WORK AND I USE THEM ALL
THE TIME BECAUSE I GET E-MAIL
FROM ALL OVER THE WORLD AND A
LOT OF IT IS NOT IN A LANGUAGE
THAT I SPEAK, TRANSLATIONS ARE
NOT PERFECT BUT THE LARGE LANGUAGE
MODELS HAVE THEIR UTILITY BUT
MACHINE LEARNING TOOLS WHICH ARE
NOT LARGE LANGUAGE MODELS MAY
VERY WELL BE OUR STRONGEST
AVENUE TO ANALYZING LARGE
QUANTITIES OF DATA LOOKING FOR
CORRELATIONS AND THEN TRYING TO
UNDERSTAND WHETHER THERE IS AN
EXPLANATION FOR WHAT THOSE
CORRELATIONS EXHIBIT.
AND THIS IS WHY MY FRIEND JUDEA PEARL
AT UCLA WHO IS
ANOTHER TURING AWARD WINNER WROTE TWO
BOOKS.
ONE OF THEM'S CALLED THE BOOK OF
WHY, AND THE OTHER ONE WAS CALLED
CAUSALITY AND HIS POINT IS THAT
UNLESS YOU HAVE A CAUSAL MODEL,
HAVING CORRELATIONS DOESN'T GIVE
YOU MUCH INSIGHT.
SO I BELIEVE THAT THE MACHINE
LEARNING TOOLS CAN IN FACT HELP
US IN JUST--AND LOOK FOR
CORRELATIONS WHICH POINT WE NOW
CAN START ASKING OURSLEVES
SERIOUS QUESTIONS ABOUT WHAT
POSSIBLE MODEL COULD PRODUCE
THAT CORRELATION AND DOES THAT
GIVE US INSIGHT INTO METABOLISM
AND THE WAY IN WHICH BODIES
WORK.
>> OKAY.
ONE OF THE BIGGEST CHALLENGES IN
INSURING THE RELIABILITY AND
VALIDITY OF AI MACHINE LEARNING
ALGORITHMS HAD HEALTHCARE AND
HOW CAN WE ADDRESS THOSE
CHALLENGES?
>> WELL YOU'VE ALREADY IMPLIED
SEVERAL WAYS IN WHICH TO DO
THAT, ONE OF THEM IS TO
DELIBERATELY SET ABOUT
COLLECTING THE DATA THAT WE KNOW
WE NEED AND DON'T HAVE AS WITH
THE COHORTS THAT DON'T GET
ADEQUATE ATTENTION FOR EXAMPLE
AND THEN PROCEED, YOU ALSO
SUGGESTED FOR EXAMPLE THAT WE
PROVIDE INCENTIVES NOT ONLY TO
PATIENTS BUT TO DOCTORS AND TO
RESEARCHERS FOR COLLECTING AND
SHARING THEIR DATA, FOR
PROVOIDING THEM WITH THE TOOLS
FOR DOING THAT, FOR PROVIDING
THEM WITH STANDARDS SO THAT THE
DATA IS COMMENSURATE.
ALL OF THOSE THINGS ARE STEPS
THAT NIH IS IN A UNIQUE POSITION
TO TAKE, I THINK TO SHOW LIGHT
IN DARK CORNERS.
AND I HAVE THE SENSE THAT YOU
WANT TO DO THAT.
SO I WOULD ENCOURAGE THAT AND I
HOPE THE LEGISLATORS UNDERSTAND
THIS IS AN IMPORTANT WAY
FORWARD.
>> RELATED.
GIVEN YOUR EXPERIENCE WITH THE
EVOLUTION OF THE INTERNET AND
DIGITAL TECHNOLOGIES, WHAT ARE
THE ETHICAL CONSIDERATIONS AND
SECONDARY PRIVACY CONCERNS, DO
YOU THINK WE SHOULD ALL BE
THINKING ABOUT AS WE PRIORITIZE
MACHINE LEARNING TO BECOME
INTEGRATED INTO HEALTHCARE?
>> THAT'S AN INTERESTING KIND OF
SANDWICH YOU JUST PUT TOGETHER
BECAUSE YOU STARTED OUT WITH THE
INTERNET AND THEN YOU RAN OFF ON
THE MACHINE LEARNING STUFF, THE
INTERNET IS NOT A MACHINE
LEARNING SYSTEM EXCEPT IN ONE OR TWO
SENSES.
THE ROUTING SYSTEM, FOR EXAMPLE,
ROUTING A PACKAGE GOING THRU THE
NETWORK COULD BE HELPED WITH
MACHINE LEARNING TOOLS AND WE
USE THOSE.
BUT THE INTERNET ITSELF IS
BASICALLY, SIMPLY A MECHANISM FOR
TRANSMITTING DIGITAL INFORMATION FROM ONE
PLACE TO ANOTHER
BUT THEN YOU GET TO LAYER A BUNCH OF STUFF ON TOP OF
THAT, WHEN TIM BURNERS-LEE INVENTED THE
WORLD WIDE WEB HE PUT A
A LAYER OF PROTOCOL ON TOP OF THE
TCP/IP WORK THAT BOB KAHN AND I DID
IN ORDER TO BUILD WHAT WE NOW
THINK OF AS THE WORLD WIDE WEB.
I'M TRYING TO FIGURE OUT
HOW--OKAY, SO YOU MENTIONED
ETHICAL CONCERNS.
THIS IS A PLACE WHERE SOME
PEOPLE COME UP AND SAY, YOU KNOW
IT'S ALL YOUR FAULT, AND WHAT
WERE YOU THINKING, LOOK AT
ALL THE BAD STUFF THAT HAPPENED
AND MY REACTION TO THIS IS,
WELL, WAIT A MINUTE, PEOPLE
SOMETIMES TAKE TOOLS THAT HAVE
BEEN INVENTED FOR CONSTRUCTIVE
REASONS AND THEY USE THEM FOR
OTHER THAN CONSTRUCTIVE
PURPOSES, A HAMMER CAN POUND A
NAIL AND IT CAN POUND A HEAD, SO
I KIND OF RESIST BEING ACCUSED
OF ENABLE--WELL, I COULD BE
ACCUSED ALONG WITH MY COLLEAGUES
OF ENABLING SOME BAD BEHAVIORS BECAUSE HONESTLY
WE TRIED TO REDUCE THE BARRIER TO
ACCESS TO INFORMATION TO A MINIMUM.
THAT MEANT REDUCING THE BARRIER
FOR SHARING INFORMATION TO A
MINIMUM REDUCE THE BARRIER TO
FINDING INFORMATION TO A
MINIMUM, ALL OF WHICH HAS HAD
VERY POSITIVE BENEFIT BUT IT'S
ALSO CREATED AN OPPORTUNITY TO
SPREAD MISINFORMATION AND
DISINFORMATION BUT THE DECISION
TO DO THAT FOR THE MOST PART IS
SOMEONE'S DECISION, IT'S NOT THE
NETWORK DECIDING TO DO THAT.
SO I THINK WE HAVE TO HOLD
PARTIES RESPONSIBLE FOR BAD
BEHAVIOR WHICH TRANSLATES INTO
ACCOUNTABILITY AND I WOULD SAY
THAT WHAT IS MISSING FROM THE
ENVIRONMENT THAT WE'RE IN RIGHT
NOW IS A SUFFICIENT DEGREE OF
ACCOUNTABILITY FOR DECISIONS
THAT ARE MADE, AND IN ORDER TO
HAVE ACCOUNTABILITY, YOU HAVE TO
HAVE TO IDENTIFY, ABILITY TO
IDENTIFY.
NOW HERE'S WHERE WE GET INTO
REAL TENSION BECAUSE SOME PEOPLE
WILL SAY, MY PRIVACY IS
IMPORTANT TO ME, I DON'T WANT TO
BE KNOWN WHEN I'M USING THESE
TOOLS AND YOU CAN UNDERSTAND WHY
SOME PEOPLE MIGHT SAY THAT, AND
SOME PEOPLE ARE AT REAL RISK, IF
THEY WERE IDENTIFIABLE, BUT EVEN
IF YOU INVOKE THE WHISTLEBLOWER
SCENARIO WHICH IS A COMMON ONE TO
JUSTIFY PRIVACY AND ANONYMITY,
USUALLY THE WHISTLE BLOWER BLOWS
THE WHISTLE TO SOMEONE WHO HAS
TO FIGURE OUT WHO THEY ARE IN
ORDER TO KNOW WHETHER THEY'RE
CREDIBLE BUT WHO IS RESPONSIBLE
FOR PROTECTING THEIR IDENTITY.
REPORTERS GO TO JAIL SOMETIMES
IN THE PROTECTION OF THESE
SOURCES THAT THEY CHOOSE.
SO I THINK IN THE LONG RUN,
ABSOLUTE ANONYMITY IS NOT OUR
FRIEND IN THIS ONLINE
ENVIRONMENT AND WE HAVE TO HAVE
A WAY OF PIERCING THE VEIL UNDER
THE RIGHT CIRCUMSTANCES, WITH
THE RIGHT AUTHORITIES.
JUST LIKE LICENSE PLATES THAT
LOOK LIKE GOBBLEDY-GOOK TO US
BUT SOME PEOPLE HAVE THE
ABILITY, THE POLICE, HAVE THE ABILITY TO FIND OUT WHICH OWNER IS
ASSOCIATED WITH THAT LICENSE
PLATE.
THAT IS NOT NECESSARILY THE
DRIVER OF THE CAR THAT JUST WENT
120-MILES AN HOUR DOWN THE ROAD
BUT AT LEAST YOU IDENTIFY THE
OWNER OF THE CAR AND THAT'S A
START.
SO I THINK THESE ARE ALL THINGS
ACCOUNTABILITY IS
REALLY IMPORTANT NOW
THERE'S ONE OTHER THING THAT'S
EQUALLY IMPORTANT AND THAT'S
AGENCY IN THE SENSE THAT WE NEED
TO GIVE PEOPLE THE TOOLS TO
PROTECT THEMSELVES, TO PROTECT
THEIR PRIVACY, TO USE THESE
SYSTEMS WITHOUT REVEALING
ANYMORE THAN THEY ABSOLUTELY
HAVE TO IN ORDER TO LEGITIMATELY
USE IT.
LIKE SOMETIMES YOU HAVE TO PAY
FOR THINGS SO THAT MEANS YOU
HAVE TO SHARE YOUR CREDIT CARD
INFORMATION.
IF YOU DON'T WANT TO DO THAT,
YOU MAY NOT GET ACCESS TO WHAT
IT IS YOU'RE TRYING TO GET.
SO AGENCY AND ACCOUNTABILITY, I
THINK ARE GOING TO BE OUR
FRIENDS HERE AS WE TRY TO MOVE
FORWARD.
SO THIS TAKES THAT IN A
LITTLE BIT OF A DIFFERENT
DIRECTION BUT I THINK IT'S AN
IMPORTANT QUESTION, SO, WE HAVE
LAB TESTS, A LAB TEST, THERE'S A
GOVERNING APPROVING AGENCY THAT
SAYS THIS TEST IS SAFE, THIS
TEST IS REPRODUCABLE, IT PERFORMS
THE SAME ON SAMPLES EVERY TIME,
IT CAN SAFELY BE USED IN A
CLINICAL ENVIRONMENT TO MAKE
DECISIONS FOR PEOPLE, CLIA,
APPROVAL OF LABS, ET CETERA.
DO WE NEED TO THE SAME KIND OF
KNOWLEDGEABLE REGULATORY AGENCY
OR BODY FOR MACHINE LEARNING AND
AI BUILT TOOLS AND TECHNOLOGIES?
>> SO, I ACTUALLY WOULD SUGGEST
THAT WE SHOULD EXPAND RATHER
THAN CONTRACT HERE.
AND I WOULD SAY GENERALLY
SPEAKING FOR SOFTWARE, WHETHER
IT'S AI OR YOU KNOW MACHINE
LEARNING OR ANY OTHER KIND OF
SOFTWARE, THERE SHOULD BE
THOUGHTFUL ABILITY TO AUDIT THE
SAFETY AND SECURITY OF THAT
SOFTWARE, WHETHER IT WAS AI OR
MACHINE LEARNING OR ANYTHING
ELSE BECAUSE WE'RE SO DARN
RELIANT ON SOFTWARE NOW.
I MEAN THE PHONE IN YOUR POCKET,
THE MOBILE, WHEN IT DOESN'T
WORK, IT'S CASCADE FAILURE TIME,
ALL KINDS OF THINGS CAUSE YOU
TROUBLE, SO THE IDEA OF HAVING
STANDARDS FOR DEVELOPING
RELIABLE SAFE AND SECURE
SOFTWARE, INCLUDING AI AND
MACHINE LEARNING MAKES A LOT OF
SENSE.
NOW IN THE ACCOUNTING WORLD
THERE'S SOMETHING CALLED GENERAL ACCEPTED
ACCOUNTING PRINCIPLES, MAYBE WE NEED TO HAVE GENERALLY ACCEPTED SOFTWARE
PRINCIPLES WHERE
MAYBE GENERALLY ACCEPTED, IT'S
GAS, GENERALLY ACCEPTED AI
PRINCIPLES WHICH DOESN'T
PRONOUNCE WELL BUT I THINK WE
SHOULD BE LOOKING FOR PRINCIPLES
THAT WE CAN ALL ADOPT AND APPLY
IN THE DEVELOPMENT AND USE OF
THESE KINDS OF TECHNOLOGIES.
BUT I WANT TO EMPHASIZE
SOMETHING HERE. IT'S THE USE THAT'S IMPORTANT HERE. THE RISK
FACTORS HAVE TO DO WITH THE WAY THESE
TECHNOLOGIES ARE APPLIED SO IT'S
NOT THE SPECIFIC DETAIL OF THE
NEURAL NETWORK THAT IS AT ISSUE
HERE AS MUCH AS IT IS THE
DECISIONS TO USE THAT SYSTEM FOR
PARTICULAR APPLICATION.
SO IF I USE A CHATBOT, JUST FOR
ENTERTAINMENT, THAT'S A
RELATIVELY LOW RISK THING,
UNLESS YOU HAVE A BIG ARGUMENT
OVER INTELLECTUAL PROPERTY AND
WHERE DID THE INFORMATION COME
FROM FOR TRAINING THE MODEL THAT
WE CAN HAVE QUITE A LONG
CONVERSATION ABOUT THAT BUT WHEN
YOU START TO GET INTO
APPLICATIONS WHERE WE'RE SAYING,
WE'RE GOING TO USE THIS THING TO
GIVE YOU FINANCIAL ADVICE, OR
WE'RE GOING TO DO DIAGINOSIS OF
YOUR SYMPTOMS OR WE'RE GOING TO
MAKE RECOMMENDATIONS FOR
TREATMENT, BOY OH BOY ARE YOU
EVER IN HIGH RISK TERRITORY
THERE SO HOW DO YOU DEMONSTRATE
TO A BODY THAT'S GOING TO AUDIT
YOUR PRACTICE THAT HAVE YOU
TAKEN PRUDENT STEPS TO PROTECT
THE SAFETY AND INTEREST OF THE
PARTY WHO'S GOING TO BE
FOLLOWING THE ADVICE, THE
DIAGNOSIS AND THE TREATMENT.
SO WE NEED TO HAVE THAT KIND OF
AUDIT, I DON'T KNOW THAT WE ARE
ALREADY PREPARED TO LAY OUT WHAT
THOSE AUDIT TERMS SHOULD BE
BUT WE'VE HAD THE LONG TIME TO
DEVELOP THAT FOR ACCOUNTING.
MAYBE IT'S TIME TO GET STARTED
ON THE SAME THING FOR SOFTWARE
AND FOR MACHINE LEARNING AND FOR
LARGE LANGUAGE MODELS, AND
GENERALLY MACHINE LEARNING AND
AI, AND WE JUST NEED TO START
WORKING ON THAT.
I SUSPECT WE WILL NEED TO HAVE
SOME OVERSIGHT FOR THESE JUST
LIKE WE DO FOR EVERYTHING IF
THEY END UP HELPING US MAKE
CLINICAL DECISIONS.
THESE ARE LIFE AND DEATH
DECISIONS SO
WE NEED TO UNDERSTAND AND HAVE
SOME KIND OF QUALIFICATION
SYSTEM SO IT'S NOT THE WILD WEST
SO AN INTERESTING QUESTION IS
WHETHER WE HAVE ANY WORKED
EXAMPLES.
WE HAVE THE FEDERAL DRUG
ADMINITRATION FOR EXAMPLE, AND
THEY HAVE RULES ABOUT HOW WE
DECIDE WHETHER OR NOT A DRUG
TREATMENT IS SAFE, OR A TEST IS
ACCURATE OR NIST HELPS
US WITH SOME OF THOSE THING TOO BECAUSE THEY
HELP US CALIBRATE WHETHER THE
TESTS ARE PRODUCING RELIABLE
RESULTS, SO WE NEED SOMETHING
LIKE THAT, I DON'T KNOW IF IT
NEEDS TO BE CENTRALIZED EXACTLY
BUT WE DO HAVE TO HAVE A
COMMON SET OF AGREED PRACTICES
BECAUSE WE WOULD LIKE TO BE ABLE
TO RELY ON WHOEVER IS PROVIDING
THOSE PRODUCTS AND SERVICES TO
HAVE MET THE SAME CERTIFICATION
STANDARDS.
>> GREAT.
AND IF THESE THINGS ARE
REGULATED AS DEVICES IT WILL BE
A VERY DIFFERENT SITUATION--SOME
OF THE CHALLENGES WE HAVE IN THE
DEVICE WORLD IT HASN'T BEEN
SOLVED THERE.
>> AND THE INTERESTING THING
ABOUT DEVICES THESE DAYS IS THAT
MOST OF THEM HAVE A LOT OF
SOFTWARE IN THEM.
SO THAT'S WHY I GET NERVOUS
ABOUT SOFTWARE IN GENERAL IS
THAT WE REALLY NEED TO PAY
ATTENTION TO HOW DOES THIS STUFF
WORK AND WHAT IF IT DOESN'T?
WHAT ARE THE EDGE CASES THAT
HAVE TO BE CATERED TOO?
>> SO WE'RE OUT OF TIME.
WOW, THAT WAS JUST WONDERFUL.
>> GEEZ, THAT WENT FAST.
OOF.
>> CAN YOU BELIEVE THAT?
>> NOPE, THERE'S A TIME WARP IN
HERE.
>> BUT WE CAN'T END TALKING
ABOUT REGULATORY PROCEDURES,
THAT'S NOT FAIR.
SO I WILL GIVE THE FLOOR OVER TO
YOU ONE LAST--FIRST OF ALL I'M
GOING TO SAY THANK YOU FOR THIS
INCREDIBLE CONVERSATION, WE'RE
SO GRATEFUL YOU ARE HERE FOR
THIS VISIT, I COULDN'T THINK OF
A MORE SPECTACULAR RALL
CULTURAL LECTURE THAT YOU ARE
PROVIDING FOR ALL OF US.
>> WELL, THANK YOU.
>> THIS HAS BEEN SO THOUGHTFUL
AND INSPIRING BUT I WILL TURN IT
OVER TO YOU BECAUSE WE WOULD
LOVE TO HEAR IF YOU HAVE ANY
PARTING IDEAS FOR US?
>> WELL, THANK YOU FIRST OF ALL
FOR THE OPPORTUNITY TO HAVE THIS
CONVERSATION.
I WILL TAKE AWAY A BUNCH OF
IDEAS THAT I MIGHT NOT HAVE
OTHERWISE THOUGHT ABOUT.
SO I APPRECIATE THE OPPORTUNITY.
I HOPE THOSE OF YOU SITTING HERE
IN THE AUDIENCE RECOGNIZE WHAT
AN OPPORTUNITY NIH HAS TO
IMPROVE THE STATE OF HEALTHCARE,
NOT ONLY HERE IN THE U.S. BUT
ELSEWHERE, WITH THE KIND OF
RESEARCH THAT YOU'RE DOING AND
THE ATTEMPT TO EXERCISE THESE
POWERFUL COMPUTING TOOLS THAT
ARE EMERGING.
YOU KNOW PEOPLE, YOU ASKED
EARLIER, YOU KNOW WHAT WOULD I
HAVE SUGGESTED PEOPLE STUDYING
GO INTO, AND I SAID
ASTROPHYSICS BUT GENERALLY SPEAKING
SOMETHING THAT'S HAPPENED OVER
THE PAST 20 YEARS AND I WILL
CALL IT COMPUTATIONAL X FOR
EVERY VALUE OF X, YOU CAN THINK
OF LINGUISTICS, CHEMISTRY,
PHYSICS, ECONOMICS, WE ARE USING
COMPUTING TOOLS INCREASINGLY TO
BUILD MODELS, TO TEST THE
MODELS, TO TRY TO UNDERSTAND HOW
THE WORLD WORKS, MAKE
PREDICTIONS AND NIH IS INVESTED
HEAVILY IN TRYING TO DEVELOP
THOSE KINDS OF TOOLS AND YOU
HAVE AN OPPORTUNITY TO MAKE A
LOT OF PROGRESS, I THINK BECAUSE
YOU'RE THE SOURCE OF A FABULOUS
RESEARCH, GREAT DATA AND NOW,
ALSO POTENTIAL FOR USING
COMPUTER BASED TOOLS TO MAKE
YOUR PREDICTIONS EVEN MORE
PRECISE.
SO IF I WERE IN YOUR SHOES, I
WOULD BE SITTING HERE THINKING,
BOY, DO WE HAVE A LOT OF WORK TO
DO, AND A HUGE AMOUNT OF
OPPORTUNITY TO DO GOOD, SO THANK
YOU FOR RUNNING THE SHOW.
>> THANK YOU.
[ APPLAUSE ]
[APPLAUSE ]