THANK YOU FOR JOINING US FOR
TODAY'S OHEART ATTACK SRP SERIES
SESSION TITLED: IRB REVIEW OF
RESEARCH INVOLVING AI.
OUR SPEAKER TODAY IS
DR. BENJAMIN C. SILVER MAN WHO
IS THE DIRECTOR AT BRIGHAM
AFFAIRS.
ADDITIONALLY HE IS CHAIR OF THE
EMBRYONIC STEM CELL RESEARCH
OVERSIGHT COMMITTEE.
DIRECTOR OF ETHICS FOR THE
INSTITUTE OF TECHNOLOGY AND
PSYCHIATRIST AT Mc CLEAN
HOSPITAL AND MEDICAL SCHOOL AND
HE SERVES AS INTRUCTOR AND
PSYCHIATRY IN THE TRUSTEES FOR
BIOETHICS.
DR. SILLER MAN RECEIVED HIS
DEGREE FROM THE UNIVERSITY OF
SCHOOL OF MEDICINE, COMPLOATED
PSYCHIATRY RESIDENCY AND
TRAINING PROGRAM AND COMPLETED
SPECIALTY FELLOWSHIP TRAINING IN
ADDICTION PSYCHIATRY THROUGH
MASS GENERAL BRIGHAM EMPLOY
OHSRP DR. GREEN WILL JOIN C
SILLER MAN FOR THE Q&A SESSION.
THROUGHOUT THE SESSION WE
ENCOURAGE VIEWERS TO SUBMIT
QUESTIONS BY CLICKING ON THE BOX
BELOW THE VIDEO SAYING SEND LIVE
FEEDBACK LINK.
A LINK TO THE VIDEOCAST AS WELL
AS SLIDES FROM THE SESSION WILL
ALSO BE POSTED ON THE OHSRP
WEBSITE APPROXIMATELY 1-2 WEEKS
AFTER THE SESSION IN THE
PRESENTATION ARCHIVE SECTION OF
THE OHSRP EDUCATION AND TRAINING
WEBPAGE.
THANK YOU FOR JOINING US
DR. SILVER MAN.
PLEASE GO AHEAD AND GET STARTED.
THANK YOU.
>> GREAT.
THANK YOU SO MUCH FOR THE
INTRODUCTION AND THANKS SO MUCH
FOR HAVING ME AND ARRANGING THIS
SESSION WHICH I THINK IS ON A
REALLY TIMELY AND IMPORTANT
TOPIC AND THANK TO YOU OHSRP FOR
INVITING ME AND HELLO TO
EVERYONE OUT THERE JOINING, HOPE
TO HEAR YOUR QUESTIONS AS WE GO
HERE, I THINK WHAT YOU WILL FIND
IS I AM PROBABLY GOING TO RAISE
MORE QUESTIONS AND PROVIDE
ANSWERS BUT HOPE TO JUST TAKE
YOU THROUGH HOW WE'RE THINKING
ABOUT THE IRB REVIEW OF RESEARCH
INVOLVING ARTIFICIAL
INTELLIGENCE.
SO JUST TO GET STARTED TO SAY,
YOU KNOW, WHY ARE WE TALKING
ABOUT THIS AND SOME OF THIS
PROBABLY GOES WITHOUT SAYING,
BUT I THINK IT'S USEFUL JUST TO
KIND OF GIVE A HIGH LEVEL
OVERVIEW, YOU KNOW THE REASON
THAT EVERYONE IS TALKING SO MUCH
ABOUT THIS AND SO EXCITED ABOUT
THIS IS BECAUSE OF THE AWESOME
POTENTIAL FOR DIGIT AT HEALTH
AND HEALTHCARE POWERED BY AI.
AI HAS THE POTENTIAL TO ADD
QUANTITATIVE AND REALTIME DATA
TO CLINICAL CARE AND RESEARCH,
ENHANCES PRECISION MEDICINE,
INDIVIDUALIZES CARE,
DEMOCRATIZES EXPERTISE, IMPROVES
SCALABILITY AND AVAILABILITY OF
CARE, OPTIMIZES DOCUMENTATION,
CAN MINIMIZE ADMEN
ADMINISTRATIVE BURDEN, GUIDE THE
DRUG DORPHY PROCESS, MANY, MANY
POWERFUL APPLICATIONS WITH REAL
AWESOME POTENTIAL ESSENTIALLY TO
SAY THAT AI CAN DO WHAT PEOPLE
CANNOT DO ON THEIR OWN AND
REALLY FROM A CLINICAL
STANDPOINT PROVIDE THE RIGHT
CARE IN THE RIGHT PLACE AT THE
RIGHT TIME TO MORE PEOPLE.
SO WITH THAT SAID, WHAT IS THE
BIG DEAL, WITH THE AWESOME
POTENTIAL, THE PROBLEM IS WITH
THE AWESOME POTENTIAL COMES SOME
VERY SCARY PROBLEMS THAT ARISE
FROM AI.
AI DOESN'T ALWAYS WORK MEANING
IT CAN JUST GIVE THE WRONG
ANSWER, IT DOESN'T ALWAYS WORK
FOR EVERYONE, MEANING THAT FOR A
VARIETY OF REASONS WE WILL TALK
A LITTLE BIT ABOUT ESPECIALLY
FOR NONREPRESENTATIVE DATA SETS
WHERE THE DATA SET IN WHICH AI
IS TRAINED IS THE DIFFERENT FROM
THE DATA SET IN WHICH AI IS THEN
USED ON CAN CAUSE RESULTS THAT
ARE WRONG FOR SOME PEOPLE BUT
NOT ALL PEOPLE, AND CAN LEAD TO
BIASES.
IT DOESN'T ALWAYS WORK IN EVERY
CONTEXT, SIMILAR TO WHAT I JUST
SAID ESSENTIALLY THAT IT CAN BE
BEHAVE DIFFERENTLY IN DIFFERENT
DATA SETS IN DIFFERENT
LOCATIONS, SO YOU TRAIN IT IN 1
HOSPITAL, AND MOVE IT TO ANOTHER
HOSPITAL AND IT MAY NOT WORK AS
INTENDED AND IT DOESN'T ALWAYS
GO THROUGH THE INDEPENDENT
REVIEW PROCESS AS WE'RE
ACCUSTOMED TO SEEING FOR MEDICAL
DEVICE VICES AND WE WILL TALK
MORE ABOUT THIS AS WE GO ON HERE
BUT THE CURRENT REGULATIONS
REALLY DID NOT ANTICIPATE MUCH
OF THIS WORK AND SO, SOME OF THE
WORK GOES THROUGH WITHOUT REAL
EXTENSIVE REGULATORY REVIEW, AND
SO WHAT ALL THAT MEANS IS THAT
THERE'S ESSENTIALLY A POTENTIAL
FOR AI TO CAUSE HARM, EITHER BY
BEING WRONG OR PERPETUATING AND
AMPLIFYING SOCIETAL JUSTICES AND
EXCLUSION AND BIAS.
SO GIVE TO GIVE AN EXAMPLE OF
THAT, THIS IS AN EXAMPLE FROM
SEVERAL YEARS AGO BUT
NONAPOPTOTIC FROM FROM A COMPANY
CALLED NOBLA USING GTP3, THIS
WAS NOT A REAL LIFE EXAMPLE,
THIS WAS THEM JUST PLAYING WITH
THE SOFTWARE, THIS IS AN EXAMPLE
IN PSYCHIATRY, MY LINE OF WORK
AND TRYING TO BASICALLY SAY, CAN
WE USE THIS AS A LARGE LANGUAGE
MODEL, AS A CHAT BOT TO HELP
HELP WITH PSYCHIATRIC PROBLEMS
SO CAN YOU SEE ON THE LEFT SIDE
HERE, SOMEONE SAYS I FEEL SAD
AND DON'T KNOW WHAT TO DO, AND
THE CHAT BOT SAYS, THAT THERE'S
DIFFERENT TECHNIQUES THAT
INCREASE HAPPINESS.
THE PERSON ASKED WHAT THOSE ARE
AND THE EXAMPLES INCLUDE TAKING
A WALK, SEEING A FRIEND OR
RECYCLING ELECTRONICKIC TO
REDUCE POLLUTION, THE PERSON
THEN ASKS FOR MORE DETAILTHAT
AND THEY SAY IF YOU RECYCLE THIS
CAN RECEIVE A TAX REFUND AND
THIS MAKES AMERICANS FEEL
HAPPIER BECAUSE THEY GET MORE
MONEY.
THAT'S A BIZARRE ANSWER TO SAY
THE LEAST AND CERTAINLY NOT IN
EVIDENCE-BASED FORM AND SO YOU
CAN SEE WHERE IT JUST KIND OF IS
WEIRD AND NOT TERRIBLY HELPFUL.
ON THE RIGHT HAND COLUMN, THAT
MUCH WORSE, SOMEONE SAYING THEY
WANT TO KILL THEMSELVES AND THE
CHAT BOX BASICALLY SAYING, YES,
I THINK YOU SHOULD DO THAT.
SO, AGAIN, GLAD THIS WASN'T A
REAL LIFE EXAMPLE BUT JUST TO
SHOW THAT THESE LARGE LANGUAGE
MODELS DON'T ALWAYS ACT THE WAY
YOU EXPECT THEM TO AND YOU KNOW
HAVE REAL POTENTIAL FOR HARM.
THIS IS ANOTHER REAL LE
INTERESTING EXAMPLE THAT I LOVED
SHARING WITH PEOPLE, THIS IS
FROM LEO [INDISCERNIBLE] AT M.
I.T. AND COLLEAGUES AND THIS IS
SHOWING WHERE AI DOESN'T ALWAYS
WORK FOR EVERYONE.
SO THIS WAS INITIALLY DISCOVERED
WHEN THEY WERE DEVELOPING AN
ALGORITHM TO DIAGNOSE PNEUMONIA
FROM CHEST X-RAYS AND WHAT THEY
FOUND WAS THAT THE ALGORITHM WAS
CONSISTENTLY MISSING THE
PNEUMONIA DIAGNOSIS OF CHEST
X-RAYS IN BLACK PATIENTS MORE IN
WHITE PATIENTS.
AND THEY LOOKED INTO THIS MORE
AND WHAT THEY REALIZED WAS THAT
THE ALGORITHM WAS PREDICTING THE
SELF-REPORTED RACE OF THE
PATIENT FROM THE X-RAY WITH'S
REMARKABLE DEGREE OF ACCURACY SO
WITH A 90% DEGREE OF ACCURACY,
SO THE ALGOGHT RHYTHM WAS
PREDICTED THE SELF-REPORTED RACE
OF PATIENTS FROM THE MEDICAL
IMAGES ALONE, THE REASON IT'S SO
STARTLING IS THAT THAT'S NOT A
TASK GENERALLY UNDERSTOOD TO BE
POSSIBLE FOR HUMAN BEINGS OR
EXPERTS MOEBING YOU SHOULD BE
ABLE TO HAND A CHEST X-RAY TO A
RADIOLOGIST AND THEY SHOULD NOT
BE ABLE TO TELL YOU THE RACE OF
THE PATIENT FROM WHOM THAT CHEST
X-RAY CAME, YOU KNOW WHO WOULD
BE FLIPPING A COIN ESSENTIALLY.
SO THEN WHAT THESE INVESTIGATORS
DID WAS THEY TRIED TO UNDERSTAND
HOW THIS WAS HAPPENING, HOW THE
ALGORITHM WAS ACTUALLY FIGURING
THIS OUT AND TO DO THAT, THEY
CORRUPTED THEIR IMAGES, THEY
NOISED THEIR IMAGES AND TRIED TO
BASIC LE HAVE THE ALGORITHM LOSE
THE ABILITY TO DETERMINE RACE IN
ORDER TO FIGURE OUT HOW IT WAS
DETERMINING RACE AND WHAT THEY
FOUND THERE WAS EVEN MORE
STARTLING SO HOPEFULLY YOU CAN
ALL SEE THIS BUT ON THE LEFT
HERE, THESE ARE IMAGES THAT HAVE
BEEN DISTORTED USING LOW PASS
FILTERS OPF AND HIGH PASS
FILTERS HPF, AND ON THE BOTTOM
IS ESSENTIALLY THE ABILITY FOR
THE ALGORITHM TO CONTINUE TO
DETERMINE RACE ON THESE
CORRUPTED IMAGES.
SO IF YOU LOOK AT HPF OF 50
HERE, LETTER D, AT THAT POINT
IT'S NO LONGER EVEN PERCEPTIBLE
AS AN X-RAY TO US, RIGHT IN IF I
HANDED YOU LETTER D AND DIDN'T
TELL YOU WHAT IT WAS, YOU
WOULDN'T BE ABLE TO TELL ME IT
WAS AN X-RAY, SO REALLY, REALLY
DISTORTED AND YET YOU CAN SEE ON
THE GRAPH, ON THE BOTTOM RIGHT
THAT IF YOU LOOK AT A HIGH PASS
FILTER OF 50 TO A HUNDRED, YOU
KNOW THE PREDICTION RATE THERE,
IT'S STILL 85ISH PERCENT,
BETWEEN 80 AND 90 PERCENT, SO
EVEN ON THAT INCREDIBLY
DISTORTED IMAGE THAT NO LONGER
LOOKS LIKE A CHEST X-RAY TO US,
THE ALGORITHM IS STILL ABLE TO
PREDICT THE RACE OF THE PERSON
FROM WHOM THAT X-RAY ORIGE
NAWILLLY CAME FROM AND
ESSENTIALLY TO SAY NOBODY REALLY
KNOWS WHY, AND IT'S STARTLING
AND SCARY BECAUSE WHAT THAT
MEANS IS THAT ALGORITHMS DON'T
ALWAYS PROVIDE ANSWERS WE
UNDERSTAND.
THEY CAN BE ACTING ON HIDDEN
INFORMATION WITHIN DATA SETS,
AND WE MAY NOT BE ABLE TO TELL
WHAT THE ALGORITHM IS SEEING IF
WE CAN'T SEE IT OURSELVES AND
THAT REALLY LEADS TO THE
POTENTIAL THAT BIAS CAN BE
PROPAGATED IN WAYS WE DON'T
UNDERSTAND.
AS I MENTIONED EARLIER, THERE'S
A WELL UNDERSTOOD PROBLEM, I
BEING IN THE DEVELOPMENT OF AI
THAT REGARDING NONREPRESENTATIVE
DATA SETS AND AGAIN WHAT THAT
MEAN SYSTEM IF YOU DEVELOP AN AI
ALGOGHT RHYTHM ON A DATA SET
THAT IS NONREPRESENTATIVE, CAN
YOU EXPECT IT'S NOT GOING TO
WORK WELL ON THE PEOPLE WHO ARE
NOT INCLUDED FROM THAT DATA SET
FROM WHICH THE ALGOGHT RHYTHM
WAS DESIGNED.
AND GENERALLY THERE'S UNDERSTOOD
TO BE A FIX FOR THAT, WHICH IS
TO MAKE SURE HA WE'RE USING
FULLY REPRESENTATIVE DATA SETS
OF OUR ENTIRE POPULATION, BUT
WHAT THIS IS SAYING, WHAT THIS
EXAMPLE SHOWS IS THAT EVEN IF
YOU CORRECT THE REPRESENTATIVE
DATA SET PROBLEM, THAT'S
CRITICAL, THAT'S ABSOLUTELY
CRITICAL BUT IT'S NOT ACTUALLY
ALWAYS NECESSARILY CORRECT
BECAUSE EVEN WITH THAT, THE
OLYMPIC GAR RHYTHM COULD BE
ACTING ON THINGS THAT WE CAN'T
SEE AND DON'T UNDERSTAND.
SO JUST TO SAY THAT, YOU KNOW
THIS SCARY PROBLEM IS REAL AND
IS SOMETHING THAT WE REALLY NEED
TO BE THINKING ABOUT AS WE
CONTINUE DOWN THE ROAD OF AI
INTEGRATING MORE INTO
HEALTHCARE.
AND SO WHAT IS THAT MEAN FOR
IRBs?
SO TO KIND OF GO BACK TO THE
QUESTION OF--SO WHAT DOES AN IRB
DO AND HOW DO WE REVIEW THIS
TYPE OF RESEARCH.
I LITTLE TONGUE IN CHEEK BUT I
LIKE TO PROVIDE THIS PICTURE
HERE AS AS WAY OF ILTRAITING
THIS POINT.
SO 1 OF THESE PICTURE SYSTEM A
LLAMA AND 1 OF THESE PICTURES IS
AN ALPACA AND I SUSPECT THERE
MAY BE PEOPLE IN THE AUDYEPS,
YOU KNOW IF WE WERE LIVE I WOULD
ASK YOU TO DO THIS, TO TELL ME
WHICH IS WHICH AND WHY, AND THE
DIFFERENT CHARACTERISTICS AND
VERY SPECIFIC CHARACTERISTICS
BETWEEN THEM AND THERE'S SOME
PEOPLE WHO ARE EX PERWHO IS
WOULD BE ABLE DO THAT AND OTHER
PEOPLE LOOK AND SAY THEY'RE
EXACTLY THE SAME AND I LIKE
THESE FOR AN ANALOGY FOR THE WAY
IRBs NEED TO,A PROACH THIS
WORK WITH AI AND WHAT I MEAN BY
THAT IS THAT SOME THINGS ARE
EXACTLY THE SAME THINGS WE NEED
TO CO WITH RESEARCH INVOLVING AI
WITH THE SAME RESEARCH AND
DETERMINATIONS THAT WE DO FOR
ANY OTHER STUDY AND WE CAN'T
MISS THAT, WE CAN'T FORGET ABOUT
THAT AND WE CAN'T IGNORE THAT
AND SOMETIMES PEOPLE DON'T WANT
TO BECAUSE IT CAN BE A LITTLE
MORE COMPLICATED BUT WE HAVE TO
DO THOSE BASIC THINGS AND THEN
WE ARE GOING TO NEED THE EXPERTS
IN THE ROOM WHO CAN TELL US
EXACTLY WHAT IS DIFFERENT
BETWEEN THIS STUDY AND SOMETHING
ELSE WE MIGHT SEE AND AND REAL
LYE HELP US UNDERSTAND THOSE
FINE, FINE EXPERT DETAIL WE NEED
TO DISTINGUISH BETWEEN THESE.
SO THE QUESTIONS ARE THE SAME,
THE THINGS ARE THE SAME, THAT WE
NEED TO THINK ABOUT IS IS IT
HUMAN SUBJECTS RESEARCH, SO WHEN
WE GET A PROTOCOL INVOLVING AI,
OR A PROJECT INVOLVING AI,
APPLYING THE SAME QUESTIONS WE
DO FOR OTHER STUDIES TO
DETERMINE IF IT'S SURVEYS MAN
SUBJECTS RESEARCH WE DO, WE HAVE
TO ASK QUESTIONS ABOUT FDA
REGULATION AND IF IT'S FKA
REGULATED AND WE HAVE TO
DETERMINE IF IT MEETS THE
REGULATORY APPROVAL, THOSE ARE
ALL THE SAME THINGS WE WOULD DO
WITH ANY STUDY, THOSE ARE UNIQUE
ASPECTS FOR THOSE QUESTIONS BUT
WE HAVE TO GO THROUGH THE SAME
BASIC PROCESS, AND WHAT'S
DIFFERENT, THERE ARE SPECIAL
CONSIDERATIONS FOR AI RESEARCH
THAT WE NEED TO SEE IN THE
RESEARCH PROTOCOL AND CONSENT
FORM AND WE'LL GO THROUGH IN OF
THOSE IN SOME MORE DETAIL.
TO BROADEN THE LITTLE BIT AND
WHAT DOES THIS MEAN FOR IRBs
BUT FOR HRPPs AND
INSTITUTIONS, 1 QUESTION
CERTAINLY OUR IRB AND IRBs
FROM CROSS THE COUNTRY WITH
CONVERSATIONS I'VE HAD WITH MANY
PEOPLE IS THE IRBs ROLE, IS
SOLVING THESE PROBLEMS AND
ANSWERING THESE QUESTIONS THE
IRB'S ROLE IN IF IT ISN'T THEN
WHO'S?
WHOSE JOB IS IT?
AND THEN THINKING ABOUT THE
REGULATION TO GOVERN THIS TYPE
OF RESEARCH AND IF NOT, WHAT
WOULD NEED TO CHANGE AND/OR, IF
THE IRB CAN'T DO IT BECAUSE THE
REGULATIONS DON'T ALLOW THE IRB
TO DO IT THEN AGAIN, WHO'S
RESPONSIBILITY IS IT TO PROVIDE
ADEQUATE OVERSIGHT FOR THIS TYPE
OF WORK EMPLOY SO JUST TO START
WITH SOME OF THE QUESTIONS I
SAID ARE THE SAME
BETWEEN--BETWEEN REVIEWING AI
RESEARCH AND REVIEWING RESEARCH
TARTING WITH THE QUESTION IS IT
HUMAN SUBRECYCLINGS RESEARCH, I
WON'T SPEND A WHOLE LOT OF TIME
ON THIS BECAUSE THIS IS LIKELY
MANAGE THAT'S VERY FAMILIAR TO
PEOPLE IN THIS SPACE BUT JUST TO
SAY THAT BASICALLY THE QUESTIONS
THAT WE ASK ARE GOING TO BE THE
SAME FOR ANY OTHER STUDIES,
YOU'RE GOING TO BE ASKING
WHETHER SOMEONE IS CLICKING
INFORMATION IN A SYSTEMATIC WAY.
YOU WILL BE ASKING IF SOMEONE IS
TRYING TO CREATE GENERALIZABLE
KNOWLEDGE AND YOU WILL ASK
WHETHER PEOPLE ARE USING AND
AGAIN THIS IS THE--FOCUS ON THE
COMMON RULE DEFINITION FOR A
MOMENT, YOU WILL BE ASKING
WHETHER PEOPLE ARE
USING--INTERACTING WITH
INDIVIDUALS OR USING
IDENTIFIABLE HUMAN SUBJECT
MATERIAL.
I THINK WE GET A LOT OF STUDIES
THAT COME IN THAT ARE DEVELOPING
ALTASD GOR RHYTHMS THAT ARE
BASICALLY TRYING TO DO WORK ON
LARGE DATA SETS THAT ARE BOTH
UNCONSEBTED AND DEIDENTIFIED AND
THIS INTERESTING PROBLEM OR
QUESTION THAT QUMS UP WHICH I
THINK OFTEN PEOPLE FEEL PRETTY
UNCOMFORTABLE NOT REVIEWING
THOSE TYPES OF PROJECTS BECAUSE
OF EVERYTHING THAT I JUST WENT
THROUGH WHICH IS AI BRINGS UP
ALL OF THESE TRICKY QUESTIONS,
BUT IF SOMEONE IS IN THE VERY
EARLY STAGES OF DOING AI WORK
AND THEY'RE DOING--THEY'RE NOT
INTERACTING WITH HUMAN BEINGS
AND THEY'RE USING DEIDENTIFIED
DATA ONLY, YOU KNOW MIGHT IT
BE--MIGHT IT BE CONSIDERED NOT
HUMAN SUBJECTS RESEARCH, AND
REALLY THAT LEADS INTO AND MAYBE
IT DOES AND THAT GETS TO WHAT I
ALLUDED TO EARLIER WITH 1 OF THE
PROBLEMS, I THINK THAT PEOPLE
HAVE RIGHT NOW, IS THAT THIS
RESEARCH DOESN'T ALWAYS UNDERGO
THE SAME INDEPENDENT REVIEW AND
OVERSIGHT AS RESEARCH RESEARCH
DOES BECAUSE OF THE THINGS THAT
CALL IN THE NONHUMAN SUBJEKS
CATEGORY LEADS TO CRUITNY.
THAT LEADS TO A SECOND QUESTION
WHICH IS IS IT FDA REGULATED
RESEARCH BECAUSE AS YOU ALL
KNOW, IN THAT DETERMINE NATION,
WHERE YOU'RE SAYING OKAY,
SOMEONE IS NOW DOING EARLY STAGE
AI WORK FOR DEIDENTIFIED DATA
SET AND THE QUESTION TO BE ASKED
BEFORE IT WORK STARTS IS IS IT
FDA REGULATED ARE THEY AT THIS
STAGE TESTING OR DEVELOPING A
MEDICAL DEVICE IN THE WAY WE
WOULD THINK OF THAT WOULD
REQUIRE YOU KNOW FORMAL IRB
REVIEW BECAUSE OF THE FDA
REGULATIONS AND REALLY I THINK
THE QUESTION I ASK WHEN I HEAR
THIS QUESTION OF IS IT THE FDA
REGULATED RESEARCH IS I TRY TO
FLIP THE QUESTION A LITTLE BIT
AND SAY, ACTUALLY WHAT I THINK
WE SHOULD BE ASKING IS, AT WHICH
POINT IN THE AI MODEL LIFE CYCLE
DOES IT BECOME SUBJECT TO FDA
OVERSIGHT?
WE KNOW THAT AI ALGORITHMS IN
HEALTHCARE WILL BE CONSIDERED
SOFTWARE AS A MEDICAL DEVICE IN
A LARGE NUMBER OF SITUATIONS SO
THEY WILL BE FDA REGULATED, THE
QUESTION IS TO AT WHAT POINT
THAT KICKS IN AND AT WHAT POINT
DO YOU HAVE TO CALL THAT IRB
REVIEWED HUMAN SUBRECYCLINGS
RESEARCH VERSUS POTENTIALLY
NONHUMAN RESEARCH IN THE EXAMPLE
I GAVE IS REALLY TRICK Y AND I
DON'T THINK WE HAVE A CLEAR
ANSWER ON THAT AND I THINK
PEOPLE HANDLE THAT VERY
DIFFERENTLY POTENTIALLY FROM IRB
TO IRB AND SOMETHING WE NEED TO
CONTINUE CONTINUING ABOUT, SO
JUST A BIT MORE IN DETAIL, SO
PEOPLE KNOW WHAT I MEAN, THE AI
LIFE CYCLE AND ESSENTIALLY ON
THE RIGHT HAND SIDE HERE, THIS
IS A BIT OVERSIMPLISTIC, BUT IT
STARTS WITH PLANNING, THINKING
ABOUT THE PROBLEM YOU'RE TRYING
TO SOLVE AND WHY AI WOULD BE
APPROPRIATE FOR IT AND THEN YOU
ARE NEED TO HAVE THE DATA SET
THAT YOU ARE GOING TO WORK ON SO
YOU DO DATA ACQUISITION AND DATA
CLEANING AND GETTING THE DATA IN
THE FORMAT THAT YOU NEED AS I
SAID OFTEN THESE PROJECTS ARE
STARTING WITH DATA SETS THAT ARE
ALREADY COLLECTED THEN YOU TART
BUILDING YOUR MODEL, AND YOU
HAVE TO TRAIN YOUR MODEL, THEN
YOU EVALUATE AND VALIDATE YOUR
MODEL AND THEN FINALLY
CLINICALLY DEPLOYING THE MODEL
AND THEN AFTER YOU CLINICALLY
DEPLOY IT, THOSE THINGS CHANGE
OVER TIME AND ONGOING MONITORING
AND MAINTENANCE WHICH GOES IN A
BIG CIRCLE BECAUSE YOU HAVE TO
IDENTIFY PROBLEMS AND PLAN TO
FIX THEM AND DO MORE TRAINING
AND RETRAINING AND VALIDATION SO
IT'S A NEVER ENDING CYCLE AND
JUST THE KIND OF REFLECT ON THE
QUESTION I JUST ASKED IN LOOKING
AT THIS PICTURE, SO WHEN YOU'RE
JUST PLANNING, NOT DOING
ANYTHING YET, IS THAT FDA
REGULATED.
WHEN YOU'RE JUST PREPARING YOUR
DATA AND COLLECTING YOUR DATA
BUT NOT ACTUALLY DOING ANYTHING
WITH THE AI MODEL YET, IS THAT
FDA REGULATED.
WHEN YOU START BUILDING THE
MODEL, IS THAT FDA REGULATED,
WHEN YOU START TRAINING THE
MODEL IS THAT FDA REGULATED, THE
DEVIL IS IN THE DETAILS FOR EACH
PROTOCOL FOR SOME DEGREE BUT I
THINK THERE'S ROOM FOR DEBATE
AND POTENTIALLY DISAGREEMENT
AROUND WHEN THAT KICKS IN, AND
I'VE SEEN DIFFERENT AGAIN,
DIFFERENT IRBs HANDLE THIS IS
IN DIFFERENT WAYS, IN SOME WAY
THE QUESTION IS BASICALLY, HOW
FAR BACK, IS THE FDA GOING TO
WANT TO LOOK AND HOW
CONSERVATIVE IS AN IRB GOING TO
WANT TO BE.
SO JUST TO REITERATE A FEW
POINTS HERE ON THE BULLETS.
YOU KNOW THE MEDICAL DEVICE
REGGULESS DO APPLY, THE FDA HAS
DONE A GREAT YOB IN TRYING TO
KEEP UP WITH THIS, IT'S HARD TO
KEEP UP WITH, THEY RELEASED
REALLY NICE GUIDANCEOT
DEVELOPMENT OF CLINICAL
DECISION, SUPPORT TOOLS AND WHEN
THAT IS DEVICE VERSUS NONDEVICE,
CLINICAL DECISION SUPPORT,
THERE'S GOING TO BE MORE AND
MORE FOR SURE, BUT THE QUESTIONS
I HAVE TO ASK AGAIN IS WHAT THE
FACE OF THE AI RESEARCH LIFE
CYCLE?
QUESTIONS WE LIKE TO ASK ARE
WHERE EXACTLY IS THE RESEARCH
HAPPENING?
AND WHAT I MEAN BY THAT IS
SOMEONE DOING SOMETHING IN WHAT
WE CALL SHADOW MODE WHERE IT HAS
NO IMPACT ON--IT DOESN'T TOUCH
CLINICAL CARE AT ALL, IT'S BEING
DONE ESSENTIALLY IN THE RESEARCH
LAB AND NO 1 EVER KNOWS THE
RESULTS, IT HAPPENING SIDE BY
SIDE WITH ROUTINE CLINICAL CARE,
WHERE YOU KNOW IT'S NOT ACTUALLY
IMPACTING MEDICAL DECISION
MAKING BUT IT'S HAPPENING IN THE
CLINIC SIDE BY SIDE OR IS IT
ACTUALLY HAVING A DIRECT IMPACT
ON MEDICAL DECISION MAKING AND I
THINK THOSE TYPES OF QUESTIONS,
CERTAINLY WEIGH IN WITH THINKING
ABOUT DEVICE DETERMINATIONS AND
HOW WE WOULD HANDLE STUDIES LIKE
THAT, THINKING ABOUT WHEN DATA
WOULD BE SUBMITTED TO THE FDA,
AGAIN, HOW FAR BACK THE FDA
WOULD GO, I THINK INTEND IS
IMPORTANT, I DO THINK ASKING
PEOPLE FROM THE BEGINNING ABOUT
WHETHER THEY'RE PLANNING TO TRY
TO COMMERCIALIZE THIS, AND WHAT
THEY'RE TRYING TO DO WHETHER
THERE'S CORPORATE FUNDING AND
PART OF THE STRUGGLE HERE IS
WHEN YOU GO DOWN THIS PATH, THIS
IS FDA REGULATED AND WE'RE GOING
TO COME IN AND SCR FOR
DISCIPLINARY DEVICE BROCHURE
BECAUSE THEY DON'T HAVE THEM AND
THINKING ABOUT THEM AND THE
LEVEL OF DETAIL YOU NEED TO GET
AND THEN HOW YOU GET THAT DETAIL
IN SOMETHING THAT'S CONSTANTLY
CHANGING LIKE AN AI ALGORITHM
THAT IS NOT A STATIC MEDICAL
DEVICE FOR A CARDIAC DEVICE,
IT'S A WHOLE NEW WORLD SO TO
SPEAK AND THINK BEING WHAT THE
IRB NEEDS TO SEE AND HOW WE
WRITE THAT UP IN A WAY THAT YOU
KNOW WE CAN UNDERSTAND IT, AND
EVALUATING IS QUITE TRICKY.
SO THE LAST QUESTION THAT I WILL
TOUCH ON IS AGAIN ON THE BUCKET
OF SIMILAR, WHAT WE NEED TO DO
THAT IS THE SAME FOR THE
PROTOCOLS IS ON THE REGULATORY
CRITERIA FOR APPROVAL AND I TOOK
A STAB AT GOING THROUGH THE
REGULATORY CRITERIA FOR APPROVAL
AND THINK BEING WHAT ASSPECS OF
IT CONSIDERATIONS OR ASSPECS
RELATED TO AI WORK IN EVALUATING
EACH OF THE CRITERIA FOR
APPROVAL EMPLOY I WILL NOT READ
EVERY WORD OF THIS SLIDE.
THE SLIDES WILL BE AVAILABLE FOR
YOU BUT I HOPE THIS CAN
CERTAINLY SERVE AS A REFERENCE
AT THE HELP DOING THIS WORK WHY
UOF THE TO DO A FEW, WHEN WE
THINK ABOUT MINIMIZING RISK, I
THINK IT'S INTERESTING BECAUSE 1
OF THE THINGS WE TRADITIONALLY
THINK ABOUT IS MINIMUM NECESSARY
DATA USE, THINKING ABOUT MINIMUM
NECESSARY DATA USE IN AI WORK IS
CONFLICT BECAUSE THE MORE PEOPLE
YOU ASK IS THE MORE DATA YOU
HAVE, THE BETTER, SO TO SAY WHAT
IS THE RIGHT AMOUNT OF DATA AND
HOW DO WE APPLY THE USUAL
STANDARDS FOR DATA ARE TRICKY.
MAKING SURE THERE'S APPROPRIATE
SECURITY PRECAUTIONS,
CYBERSECURITY PRECAUTIONS,
INTERNALLY AND EXTERNALLY IS
CRITICAL AND USUALLY I WOULD
SAY, OUTSIDE THE BOUNDS OF IRB
EXPERTISE AND SO HAVING OTHER
INSTITUTIONAL SUPPORT OR
RESEARCH INFORMATION, SECURITY,
OVERSIGHT AND INFORMATION LIKE
THAT IS CRITICAL, IN THE
SITUATION WHERE AI IS ACTUALLY
IMPACTING MEDICAL CARE, I THINK
REALLY MAKING SURE THAT THE
CLINICAL SIDE OF THE HOUSE IS
INVOLVED AND THINKING THROUGH,
YOU KNOW ACTUAL REAL WORLD
CLINICAL DEPLOYMENT WITH YOU,
YOU KNOW, WE HAVE OF COURSE TONS
OF RESEARCH WHERE THAT LINE
BETWEEN RESEARCH AND CLINICAL
CARE BECOMES BLURRY, I THINK
THAT AI IS A REALLY A PERFECT
EXAMPLE OF TIME WHEN IS THAT CAN
BE VERY, VERY BLIEWRY AND WE
HAVE TO BE--BLURRY AND WE HAVE
TO BE VERY THOUGHTFUL.
IN TERMS OF HAPPENING ABOUT
WEIGHING RISKS AND BENEFITS, AND
INDIVIDUAL PRIVACY RISKS AND
POTENTIAL FOR AI TO CAUSE
INDIVIDUAL HARM.
AS WE SAID EARLIER, AI CAN BE
WRONG AT TIMES BUT ON THE OTHER
HAND THINKING ABOUT POTENTIAL
BENEFITS FROM AI AGAIN AS I SAID
AT THE VERY BEGINNING THE
AWESOME POTENTIAL THERE THAT
COULD IMPROVE CARE FOR
INDIVIDUALS, IN CLINICAL
SETTINGS BUT ALSO MORE BROADLY
IMPROVEMENT IN SCALABILITY OF
MEDICAL CARE FOR EQUITABLE
SUBJECT SELECTION, IT GOES BACK
TO THE NEED FOR REPRESENTATIVE
DATA SETS AND REPRESENTATIVE
POPULATIONS AND COMMUNITY
ENGAGEMENT AND THE LAST 1 I WILL
TOUCH ON BRIEFLY IS INFORMED
CONSENT BASICALLY AND I WILL SAY
MORE ON THIS IN THE COMING
SLIDES BUT ESSENTIALLY TO SAY,
WE NEED TO THINK ABOUT HOW
CONSENT CAN BE SPECIALIZED TO
INCLUDE DETAILS ABOUT AI, HOWAR
I WORKS AND HOW IT'S HAPPENING
AND THE UNANTICIPATED RISKS OF
AI AS WELL AS PRIVACY
LIMITATIONS AND RISKS OF
REIDENTIFICATION, THINGS LIKE
THAT, 1 OF THE THINGS THAT I'M
WORKING ON WITH SOME COLLEAGUES
IS THIS CONCEPT WE KNOW OF AS
MODEL CARDS AND A CONCEPT OF
TEAM CARDS WE COME UP WITH,
MODEL CARDS, THE CONCEPT FROM
THE COMPUTER SOFTWARE, SIDE OF
THIS WORK WHERE BASICALLY IT'S
KIND OF PROVIDING AN EXPLAINER
FOR HOW IT'S WORKING AND WE
PROVIDED CARDS ON HOW THE PEOPLE
WHO WERE ACTUALLY INVOLVED WITH
THE IDEA THAT IF YOU THINK ABOUT
THE TEAM, AND WHO THEY ARE AND
WHAT THEIR BIASES MAY OR MAY NOT
BE, CONSCIOUSLY OR UNCONSCIOUSLY
THAT MAY PROVIDE INSIGHT INTO
THE ALGOGHT RIM EPIGENETICS THAT
NEED TO BE INCORPORATED THEREIN.
I WON'T GO THROUGH THE WEST OF
THIS, YOU CAN SEE IT HERE BUT
ESSENTIALLY, CONTINUOUS DATA
MONITORING AND APPROPRIATE
PRIVACY PROTECTIONS AND
COMMUNITY ENGAGEMENT REMAIN
CRITICAL AS THEY ARE IN ALL OF
OUR WORK.
SOY WHAT ARE THE SPECIAL
CONSIDERATIONS INVOLVING AI THAT
I THINK IS DIFFERENT FROM WHAT
WE SEE IS AGAIN, PROBABLY GOING
TO RAISE MORE QUESTIONS THAN
ANSWERS HERE BUT I THINK THINGS
THAT ARE IMPORTANT TO THINK
ABOUT.
SO PRIVACY AND CONFIDENTIALITY
OF DATA SO JUST TO SAY THAT THE
OPENING AI PLATFORMS ARE NOT
HIPAA COMPLIANT AND THE POWER OF
SOME OF THESE ALGORITHMS REALLY
INCREASES PROBABILITY OF
REIDENTIFICATION, SO YOU CAN PUT
IN WHAT WOULD NORMALLY BE
CONSIDERED A SCANT AMOUNT OF
DATA THAT WOULD NOT BE
REIDENTIFIABLE, IF YOU PUT IT
INTO 1 OF THESE HIGH POWERED
LARGE LANGUAGE MODELS, I THINK
THE POSSIBILITY THAT THEY COULD
USE THAT DATA TO REIDENTIFY
PEOPLE IS MUCH HIGHER THAN IT
USED TO BE WITH TRANSLATIONAL
RESEARCH DIGGAL COMPUTING AND
BECAUSE OF THAT, MOST ACADEMIC
MEDICAL CENTERS, I ACTUALLY
DON'T KNOW WHAT THE FEDERAL
GOVERNMENT OR NIH LAWS ARE, OR
RULES ARE ON THIS, BUT, MOST
ACADEMIC MEDICAL CENTERS HAVE
ESSENTIALLY PROHIBITED THE ENTRY
OF CERTAIN TYPES OF DATA INTO
THE PUBLIC GENERATIVE PLATFORMS
THAT INCLUDE NEPH I OR
INDIVIDUALLY IDENTIFIABLE
INFORMATION, IT ALSO INCLUDES
DEIDENTIFIED INFORMATION BECAUSE
OF WHAT I JUST SAID ABOUT THE
PROBABILITY OF IDENTIFICATION,
IT INCLUDES BUSINESS INFORMATION
OR INTELLECTUAL PROPERTIES SO
THEY BASICALLY SAID YOU CAN'T
USE THIS WITH HOSPITAL OR
PATIENT DATA BECAUSE THE RISKS
ARE TOO HIGH AND THAT INCLUDES
EVEN IF YOU TURN OFF THAT BUTTON
THAT SAYS DON'T SAVE THIS FOR
TRAINING PURPOSES.
SO WHAT HAPPENS IS BASICALLY
THESE LLMs ARE RUN INTERNALLY
INSTEAD, SET UP INSTEAD OF THE
GENERAL INFRASTRUCTURE THAT IS
HIPAA COMPLIANT AND EVERYTHING
IS RUN ON INTERNAL SERVERS THAT
REQUIRE IRB REVIEW, LEGAL
COMPLIANT CONTRACTING AND OTHERS
AND THAT'S THE WAY MOST PEOPLE
ARE HANDLING THIS RIGHT NOW TO
TRY TO GRAPPLE WITH THE PRIVACY
RISKS.
AS I MENTIONED EARLIER, OFTEN
THERE ARE LARGE AMOUNTS OF
UNCONSENTED DATA USED AND ALMOST
ALL AI PROJECTS AT THIS POINT
ARE STARTING WITH PROCESSING
LARGE DATA SETS AND AS I ALLUDED
TO THE MINIMUM NECESSARY IS
REALLY TRICK EXPE OFTEN
UNCONSENTED.
I WON'T REPEAT MYSELF WHAT I
SAID EARLIER BUT THAT DOES MAKE
IT COMPLICATED TO THINK ABOUT
NHSR VERSUS FDA REGULATIONS FOR
THESE DE-IDENTIFIED AND
ANONYMIZED EARLY STAGE AI, AI
PROJECTS.
I THINK IT POSES A LOT OF
CHALLENGES AND DISCOMFORT FOR
ME, AND CERTAINLY FOR US AS
IRBs, BECAUSE OF THE FACT THAT
THE GOAL HERE IS USUALLY TO
COMMERCIALIZE OR MAKE THESE
THINGS OPEN SOURCE, SO WHAT I
JUST SAID IS, YOU KNOW WE DEAL
WITH THE PRIVACY BY DOING THIS
ALL INTERNALLY WHEN WE'RE DOING
THE WORK BUT FOR MOST PEOPLE WE
WOULD UNDERSTAND THAT THE GOAL
WOULD NOT BE THAT IT'S GOING TO
STAY WITHIN YOUR HOSPITAL SYSTEM
INDEFINITELY, THE GOAL IS
USUALLY TO TAKE IT OUT, AND
PLACE IT ELSEWHERE, AND THAT
REALLY BRINGS UP SOME REALLY
SIGNIFICANT QUESTIONS AROUND YOU
KNOW, PRIVACY RISKS IN THE
FUTURE ESPECIALLY WITH REGARD TO
UNCONSENTED DATA AND ARE THERE
RETAINED PRODUCTS OF
IDENTIFIABILITY SOMEWHERE IN
THERE AND SO 1 MIGHT WONDER
WHETHER WE SHOULD BE, YOU KNOW
USING SOME SORT OF COMPARISON,
OF CONSENT REQUIREMENTS FOR
FUTURE USE AND BROAD SHARING
LIKE THE NIH GENOMIC DATA
SHARING POLICY OR THINKING
ABOUT, YOU KNOW THE COMMON RULE
REQUIRED ELEMENTS OF CONSENT FOR
WHOLE GENOME SEQUENCING, THIS IS
A LITTLE BIT OF LIKE AN
ETHICISTS DREAM IN TERMS OF
LIKE, I WOULD LOVE TO SEE, YOU
KNOW LARGE CONSENTED DATA SETS
FOR AI DEVELOPMENT.
I THINK THE CYNIC IN ME THINKS
THAT SHIP HAS PROBABLY SAILS BUT
IT'S WORTH THINKING ABOUT FROM
AN ETHICS STABBED POINT IN
THINKING ABOUT THE BENEFIT THERE
IS AND WHY WE WOULD WANT THAT IN
ORDER TO IDENTIFY THE RISKS FOR
WHEN WE'RE NOT USING IT RIGHT
NOW.
ANOTHER REALLY INTERESTING THING
THAT COMES UP IN THIS WORK IS
RETURN OF INDIVIDUAL RESEARCH
RESULTS, SO AS I ALOUDED TO
EARLIER, 1 OF THE REAL POWERS
HERE IS THAT AI DIAGNOSTIC
MODELS CAN SEE THINGS THAT
CLINICIANS CAN'T SEE AND SO WHEN
WE RUN YOU KNOW AI DIAGNOSTIC
MODELS ON AGAIN LARGE HUGE DATA
SETS, OF UNCONSENTED SAMPLES,
WHAT DO WE DO WHEN THE AI MODEL
FINDS MEDICALLY ACTIONABLE
FINDINGS AND YOU KNOW, SHOULD WE
RETURN THOSE RESULTS TO PEOPLE?
YOU KNOW SOME OF THIS GOES TO
THE SAME QUESTION, YOU WOULD ASK
FOR ANY RETURN OF RESEARCH
RESULT QUESTIONS AROUND HOW
CONFIELD FUNCTIONS KENTIA ARE WE
IN THE VALIDITY OF THE AI
RESULTS AND WEIGHING THE POETIC
TEBTIAL BENEFITS VERSUS
POTENTIAL RISKS BUT I THINK 1 OF
THE CHALLENGES HERE IS THAT WE
CAN EXPECT THIS IS GOING TO COME
UP MORE OFTEN THAN IT USED TO
BECAUSE OF THE FACT THAT THE AI
DIAGNOSTIC MODELS ARE MORE
POWERFUL AND YOU CAN EXPECT THAT
YOU ARE GOING TO SEE THINGS THAT
YOU DIDN'T SEE AND SO THEN ARE
YOU GOING TO RETURN THOSE TO THE
CLINICIAN, YOU KNOW MAYBE WHO
MISS TODAY THE FIRST TIME FOR
LIKE A REPEAT READ OR
REASSESSMENT, YOU KNOW BY THE
RADIOLOGIST OR PATHOLOGIST OR
TREATING CLINICIAN OR ARE YOU
GOING TO FOLLOW THE PATIENT AND
BASICALLY SAY, WE WERE DOING
RESEARCH ON YOUR UNCONSENTED
DATA AND WE FOUND SOMETHING AND
THAT'S AN INCREDIBLE LOAMACYY
DIFFICULT CONVERSATION TO HAVE
AND 1 WE GENERALLY DISCOURAGE
PEOPLE FROM FINDING THEMSELVES
IN BECAUSE IT'S JUST VERY
DIFFICULT FROM A MICROETHICS
PERSPECTIVE, BUT AT THE END OF
THE DAY, IF YOU HAVE AN
ACTIONABLE FINDING, YOU CAN'T
IGF NORIT EITHER POTENTIALLY SO
THINKING AGAIN IF IT'S CONSENTED
AND WHAT DID THE CONSENT FORM
SAY, AND HOW ARE YOU GOING TO
CAREFULLY WALK THROUGH THAT AND
THEN AGAIN, I HAD, I WILL SAY
I'VE HAD THE SITUATION WHERE,
THEY HAD A RECENT CONVERSATION
WHERE THEY WERE GOING TO DO
SOMETHING THAT HAD A REALLY,
REALLY HIGH PROBABILITY THAT
THEY WERE GOING TO FIND A TON OF
ESSENTIALLY PREVIOUSLY MISSED
RESULTS BECAUSE OF THE ALGORITHM
THEY WERE USING AND I THINK YOU
SAID YOU NEED TO DO THAT ON A
CONSENTED DAILY BASIS AT SET AND
I JUST DON'T THINK YOU CAN DO IT
ON AN UNCONSENTED DAILY BASIS
THEA SET BECAUSE YOU WILL CREATE
THIS REALLY DIFFICULT SITUATION
FOR ERCH AND THEY UNDERSTOOD IT,
AND PROCEEDED DOWN THAT ROOT.
I MENTIONED THIS EARLIER BUTTA
ALGORITHMIC BIAS FROM THE DATA
SETS AND OTHER CAUSES AS I KIND
OF SHOWED YOU IN THE EXAMPLE AT
THE BEGINNING.
MITIGATION OF BIAS REQUIRED TRAN
APPEARANCE AND EXPLAINABILITY
AND INDEPENDENT REVIEW, AND SO
IT REALLY DOES REQUIRE PROTOCOL
TO DESCRIBE HOW AI WAS DEVELOPED
TO TRY TO PUT INTO REAL WORDS,
THAT EXPLAINABILITY WAS TRYING
TO PUT THE REAL PROBLEM IN THAT
THE REST OF US CAN UNDERSTAND
ABOUT HOW IT'S SUPPOSED TO
FUNCTION AND HOW IT WILL BE
MONITORED OVER TIME FOR BIAS,
ALGORITHM FOR BIAS IS
COMPLICATED, I ASK A QUESTION
HERE BUT I WILL PROVIDE THE
ANSWER WHICH IS TO SAY I DO NOT
THINK IRBs HAVE EXPECTATIONS
PETTER
EXPECTATIONS--EXPERTISE FOR THIS
AND DO NOT GIVE ACCESS TO THE
ALGORITHMS RIGHT NOW SO IT'S
LIKE WE COULDN'T DO IT IF WE
WANTED TO.
SO THIS IS AN EXAMPLE WHERE THIS
MAY NEDRA TO FALL TO EITHER
INTERNAL OR EXTERNAL GROUPS OR
MAYBE IT'S THE PI'S
RESPONSIBILITY TO FIGURE IT OUT
AND REPORT IT BACK TO US, I
THINK THERE'S A VARIETY OF WAYS
TO GO ABOUT IT, DO THINK THAT
OPEN SOURCE TOOLS ARE NEEDED TO
REALLY BE ABLE TO KIND OF HELP
US DEMOCRATIZE THE ABILITY TO DO
THIS ON A MORE WIDE SCALE AND I
DO THINK PEOPLE SHOULD BE AWARE,
BE INFORMED OF THIS RISK
ESPECIALLY WHEN THEY'RE BEING
ENROLL INDEED A CLINICAL TRIAL
OF AN AI ALGORITHM THAT WILL
IMPACT THEIR CLINICAL CARE,
MEDICAL CARE, THEY NEED TO BE
AWARE OF WHERE IT WAS DEVELOPED
OR WHETHER IT COULD HAVE BIASES
BUILT-IN.
ADVERSE EVENT MONITORING I THINK
IS A BIT UNIQUE WHEN IT COMES TO
AI RESEARCH.
AGAIN, YOU HAVE TO ASK THE
QUESTION, OF DID THE ALGORITHM
ACCOUNT ACT AS EXPECTED AND IN
ORDER TO DO THAT, YOU NEED TO
KNOW HOW IT WAS SUPPOSED TO ACT
SO IT REQUIRES EXPLAINABILITY
AND YOU KNOW REQUIRES REALLY
KIND OF THINKING ABOUT THE
DIFFERENCES BETWEEN THE EXPECTED
BEHAVIOR AND WHAT WE WOULD CALL
NONDIRECTED OR EMERGENT
BEHAVIOR.
THERE NEEDS TO BE A PLAN FOR
MONITORING THAT AND THINKING
ABOUT WHAT THE ANTICIPATED
MISTAKES MAY BE UP FRONT.
ONE OF THE CHALLENGES HERE IS
THAT MAY, YOU KNOW FALL INTO THE
WORLD WHERE IT'S BEYOND ACTUALLY
JUST THE RESEARCH SUBJEKS
INVOLVED SO WE HAD A CASE FOR
EXAMPLE WHERE AI ALGORITHMS
WHERE IT WAS BUILT ON A REAL
LIFE CASE BUT WE KIND OF ADDED
THE AI POINT TO MAKE IT A LITTLE
MORE FITTING BUT YOU CAN
CONSIDER THE EXAMPLE WHERE YOU
ARE USING AI, TO ADD SOME SORT
OF ADD SOME SORT OF REMINDER TO
A PATIENT CHART AND THEN WE
START CHANGING THE WAY WE
DOCUMENT THE PARTICULAR
CONDITION AND THE AI ALGOGHT
RHYTHM STARTS ADDING THAT
REMINDER TO OTHER CASES, TO
OTHER RECORDS IN THE MEDICAL
RECORDS THAT ARE NOT ACTUALLY
PART OF THE RESEARCH STUDY, AND
SO JUST TO SAY THAT THINK BEING
ANTICIPATED MISTAKES COULD GO
BEYOND THE ACTUAL CASES THAT ARE
CONSIDERED KIND OF IN THE
RESEARCH STUDY AND THINKING
ABOUT, YOU KNOW BASICALLY RISKS
TO OTHERS WHICH IS A LITTLE
FURTHER THAN WE'RE OFTEN
THINKING ABOUT, THINKING ABOUT.
I WILL JUST FOR THE SAKE OF TIME
KEEP GOING HERE.
SO TO SAY ALL OF THAT, ASKING
THE QUESTION OF WHAT ACTUALLY
WOULD NEED TO GO IN AN IRB
PROTOCOL IN ORDER FOR US TO KIND
OF REALLY ARK ASSESSES ALL THESE
THINGS THAT I JUST MENTIONED AND
THE ANSWER TO THAT QUESTION IS
BASICALLY IT'S MORE THAN WE
USUALLY SEE, I THINK 1 OF THE
CHALLENGES IN REVIEWING THIS
WORK, IS OFTEN WE GET IN FAIRLY
SCANT DETAILS ABOUT, YOU KNOW
THE AI ALGORITHM ITSELF AND WHAT
THE KIND OF LONGER TERM PLANS
ARE FOR THE DEVELOPMENT OF THAT
ALGORITHM AND SOMETIMES I THINK
THAT'S BECAUSE THE PIs
THEMSELVES DON'T ACTUALLY KNOW,
RIGHT?
THERE'S A LOT OF EARLY WORK THAT
GETS STARTED AND YOU DON'T
REALLY KNOW WHERE IT'S GOING TO
GO BUT I DO THINK THAT WE NEED
MORE DETAILS IN OUR PROTOCOLS
AND OUR CONSENT FORMS TO REALLY
KIND OF BE ABLE TO, YOU KNOW
ADEQUATELY PROVIDE SOME DEGREE
OF OVERSIGHT FOR THIS WORK.
SO I THINK THE USUAL
REQUIREMENTS APPLY, BUT THEN AS
I MENTIONED EARLIER, REALLY
UNDERSTANDING THE CURRENT PHASE
OF THE RESEARCH IN THAT PROTOCOL
IS THAT, YOU KNOW PROOF OF
CONCEPT, SHADOW MODE,
DEVELOPMENT AND TRAINING OR IS
IT ACTUALLY TESTING OR ACTUALLY
A REAL WORLD CLINICAL TRIAL, IS
IT INFORMING OR DRIVING CLINICAL
CARE, WHAT'S THE BACK UP LOOK
LIKE, IS THE ALGORITHM
DIAGNOSING OR TREATING AND ARE
THOSE DECISIONS AUTONOMICALLY
ACTED ON, IS THE ALGORITHM
EXPECTED TO CHANGE OVERTIME?
HOW IS IT EXPECTED TO ACT?
AS I ALLUDED TO IN THE PREVIOUS
SLIDE, WHAT MISTAKES MIGHT GO
WRONG, WHAT MISTAKES MIGHT BE
MADE?
ARE THE TRAINING REPRESENTED?
HOW IS MONITORING DONE?
COMMUNITY AND STAKEHOLDER INPUT
IN AND WHICH DATA SETS WILL BE
USED AND WHICH PARTICIPANTS WILL
BE ENROLLED ALONG WITH CONSENT
CONSIDERATIONS.
SO MORE ROBUST RETAIL IS WHAT WE
NEED AND I WILL SAY THAT YOU
KNOW IN SOME OF THE WORKING
GROUPS THAT I'M ON, THERE'S A
WORKING GROUP THAT COLLEAGUES
FROM EMORY PUT TOGETHER THAT IS
DOING A REALLY NICE YOB OF
TRYING TO BRING IRB
PROFESSIONALS AND
HRPP PROFESSIONALS AROUND THE
COUNTRY TOGETHER AND WORKING
SOME SORT OF OPEN SOURCE TOOL
KIT THAT WOULD COLLECT ALL OF
THIS INFORMATION FOR US, SO WE
CAN ACTUALLY ALL SHARE THAT AND
BE ABLE TO YOU KNOW GET THE
INFORMATION WE NEED AT THE
OUTSET.
SAME THING FOR THE CONSENT FORM
WHICH WILL BE IN THE CONSENT
FORM IS PROBABLY MORE THAN WE'RE
USED TO SEEING RIGHT NOW.
THIS IS MY PERSONAL BIAS AND I
HAD A REALLY INTERESTING
CONVERSATION AT THE CONFERENCE I
JUST MENTIONED WITH A RESEARCHER
WHO'S REALLY AT THE CUTTING EDGE
OF DOING AI WORK AROUND CONSENT
FOR AI AND YOU KNOW THERE IS A
CERTAIN DEGREE TO WHICH I WOULD
ACKNOWLEDGE THAT AI IMPACTS ALL
OF US IN OUR DAILY LIVES RIGHT
NOW, RIGHT?
EVERY TIME WE USE GOOGLE, EVERY
TIME WE GO TO TARGET OR WALL
MART OR DISNEYWORLD OR WHATEVER
IT MAY BE, AI IS PART OF ALL OF
THE SYSTEMS THAT KIND OF YOU
KNOW HELP US GET WHAT WE WANT
AND PROBABLY HELP OTHER PEOPLE
COMMERCIALIZE OUR DATA AND SO
THAT IS KIND OF A--NORMAL DAILY
RISK/DAILY EXPERIENCE THAT WE
ALL HAVE IN OUR LIVES.
SO THE QUESTION THEN BECOMES YOU
KNOW HOW MUCH IS IT ARE WE
RESPONSIBLE TO TELL PEOPLE ABOUT
THAT?
AND I THINK IN THE CONTEXT OF
RESEARCH, WE ARE STILL
RESPONSIBLE TO MAKE IT VERY
CLEAR, YOU KNOW I THINK
ESPECIALLY PEOPLE ENROLLING IN
RESEARCH THAT MAY IN THE
ACTUALLY HAVE BENEFIT TO THEM, I
THINK THE REQUIRED ELEMENTS OF
CONSENT IN THE REGULATIONS WOULD
HAVE US DISCLOSE, YOU KNOW THAT
AI AS BEING USED, HOW IT'S BEING
USED, AGAIN, HOW IT WAS TRAINED,
IF THERE MIGHT BE RISKS OF BIAS,
AND SPECIFIC ANTICIPATED RISKS
TO THE AI, AS WELL AS YOU KNOW
HOW THE DATA WILL BE USED IN THE
FUTURE AND POTENTIAL PRIVACY AND
CONFIDENTIALITY IMITATIONS, YOU
KNOW I THINK WE DO HAVE AN
OBLIGATION TO SHARE THAT.
I THINK IT'S INTERESTING TO
SHARE ON THE CLINICAL SIDE,
PERHAPS AFTER THESE THINGS ARE
FULLY APPROVED AND USED, THERE
ARE THINGS, YOU KNOW IN THE ICU
RIGHT NOW, FOR EXAMPLE, THE
DEVICES IN THE ICU, THEY DO HAVE
SOME INDEPENDENT THINKING
ABILITY SO TO SPEAK AND SO WE
DON'T NECESSARILY PROVIDE EVERY
LITTLE DETAIL ON THE CLINICAL
SIDE, SO THERE'S SOME GOOD
THINGS HERE ON WHERE WE LAND.
AND FRANKLY, HOW QUOTE-UNQUOTE
EXPLAINABLE AI ACTUALLY CAN BE,
YOU KNOW, I THINK SOME PEOPLE
WOULD ARGUE THE CONCEPT OF
EXPLAINABLE AI MIGHT BE A MYTH
AS I SAID AT THE VERY BEGINNING,
WE DIDN'T ACTUALLY KNOW HOW THAT
ALGORITHM WAS MAKING THE
DECISION IT WAS MAKING BUT I
THINK LETTING PEOPLE KNOW THAT
AI IS INVOLVED AND AND WHAT THE
RISKS ARE IS AN EXPECTATION AND
CERTAINLY A REQUIREMENT THEY
WOULD ASK FOR AS AN IRB
REVIEWER.
SO JUST TO WRAP UP, I WILL SPEND
A FEW MINUTES HERE TO WRAP UP
AND LEAVE TIME FOR QUESTIONS
JUST TO GET BACK TO THE QUESTION
OF WHOSE JOB IS IT, THERE'S A
TON OF AI GOVERNANCE BEING
DEVELOPED RIGHT NOW.
I THINK IRBs PLAY A ROLL HERE
BUT I ABSOLUTELY DO NOT THINK
THAT IRBs ARE THE ONLY PEOPLE
RESPONSIBLE, I THINK WE SHOULD
NOT BE AND CANNOT BE BASICALLY
THE 1S THAT ARE GOING TO BE
ENTIRELY RESPONSIBLE FOR
PROVIDING OVERSIGHT OF AI.
I THINK WE WILL SEE MORE OF THIS
IN THE COMES MONTHS, THERE ARE
TONS OF ORGANIZATIONS WHO ARE
ISSUING GOVERNANCE RIGHT NOW.
JUST TO SAY, YOU KNOW WHAT IT
WOULD REALLY LOOK LIKE FOR AN
IRB TO TAKE THIS ON, YOU KNOW
REALLY GOING BEYOND OUR USUAL
REVIEWS, SO AGAIN, IF YOU WERE
TO PUT--SHOW THE AI LIFE CYCLE
HERE, DOWN AT THE BOTTOM, YOU
KNOW YOU WOULD ESSENTIALLY WOULD
HAVE TO BUILD IN PROCEDURES OR
POLICIES, AT EACH STEP OF THE
WAY, TO UNDERSTAND WHAT THE AI
IS DOING, DOING DATA BIAS
CHECKING AND BIAS MITIGATION, ET
CETERA AND AGAIN, I THINK I CAN
SPEAK FAIRLY CONFIDENTLY THAT
IRBs DO NOT HAVE THIS LEVEL OF
EXPERTISE RIGHT NOW AND I THINK
IT'S AN OPEN ENDED QUESTION OF
WHETHER WE SHOULD OR NOT BE
DOING THIS WORK.
I BEING THIS IS IMPORTANT WORK,
IT HAS TO BE DONE BUT THE
QUESTION IS HOW DOES IT, I THINK
AGAIN ULTIMATELY IT WOULD BE
NICE IF WE COULD DEVELOP OPEN
SOURCE TOOLS TO BE ABLE TO DO
THIS.
WE DID PUT IN A GRANT TO TRY TO
START DOWN THAT PROCESS AT
LEAST.
SO JUST TO WRAP UP HERE, THE
CHALLENGES TO OVERVIEW WITH AI,
AS WITH ALL OF OUR WORK, THERE
IS A TENSION BETWEEN ATTENTION
AND REGULATION, AND THE SPEED OF
AI IS SO FAST AND MOST OF THE
THINGS I TALKED ABOUT TODAY
WOULD REPRESENT REGULATION THAT
WOULD HAVE THE POTENTIAL TO SLOW
THAT DOWN, THAT'S CERTAINLY NOT
MY GOAL IN TALKING ABOUT THESE
THINGS, I THINK WE JUST HAVE TO
PARTNER WITH OUR RESEARCHERS TO
REALLY UNDERSTAND HOW CAN WE
EFFICIENTLY AS POSSIBLE DO THE
TYPE OF REVIEW WE FEEL
COMFORTABLE, WE WILL CATCH THE
BIG PROBLEMS HERE BUT NOT STAND
IN THE WAY OF INNOVATION, IT
REQUIRES A LOT OF EXPERTISE, AND
IRBs ARE GOING TO HAVE TO
GRAPPLE WITH WHERE WHETHER THEY
STAFF IT UP WITH EXPERTISE OR
ANCILLARY GROUPS THAT THEY RELY
OI THINK THIS WORK REQUIRES A
HUGE AMOUNT OF COLLABORATION, I
THINK NOT UNLIKE OTHER WORK WE
DO BUT I THINK IT'S A PERFECT
EXAMPLE WHERE WE NEED EVERYONE
HAD AT THE TABLE THINKING ABOUT
THIS.
AND IT REQUIRES ONGOING
OVERSIGHT, RIGHT?
SO EVEN IF YOU DO ALL OF THIS,
THAT INITIAL REVIEW, IF YOU WENT
ALL-IN AT THAT INITIAL REVIEW,
THE QUESTION IS HOW TO HOW YOU
MONITOR THAT ALGOGHT RHYTHM OVER
TIME IS COMPLICATED AND DOESN'T
FIT WELL INTO THE STANDARD IRB
INFRASTRUCTURE, I DON'T THINK OF
OF US ARE DOING, EVERY 1 MONTH
CONTINUING REVIEW TO BE ABLE TO
DO THIS, SO, YOU KNOW REQUIRES
CREATIVE SOLUTIONS THERE, AND
ULTIMATELY, I THINK IT DOES
REQUIRE REGULATORY CHANGE EITHER
AGAIN FOR AI OR FOR IRBs OR
ARE OTHER BODIES TO BE ABLE TO
SUFFICIENTLY OVERSEE THIS WORK,
JUST TO KIND OF HIGHLIGHT THAT
LAST POINT, SOME OF THE
REGULATORY QUESTIONS I THINK
COME UP YOU KNOW, I ALLUDED TO
THIS ALREADY BUT JUST TO PUT, TO
NAIL DOWN THE POINT, REALLY I
WOULD SAY, THE CURRENT
REGULATIONS DID NOT HAVE AI,
RESEARCH IN MIND SPECIAL SO I
THINK THAT YOU CAN BE COMPLIANT
WITH THE CURRENT REGULATIONS FOR
EXAMPLE, AS I ALLUDED TO DOING A
LARGE RETROSPECTIVE--USING A
SECONDARY DATA SET THAT'S
DEIESHES DENTIFIED AND YOU CAN
HAVE SOMETHING THAT MAY
TECHNICALLY NOT HUMAN SUBJECTS
RESEARCH OR EXEMPT BUT IS THAT
REALLY WHAT THE REGULATIONS HAD
IN MIND FOR THIS TYPE OF WORK,
RIGHT IN SO IT MAY BE COMPLIANCE
BUT I THINK WE HAVE TO ASK THE
QUESTION OF IS IT REALLY
ADEQUATELY PROTECTED?
DO WE WANT TO MAYBE RAISE THE
FLOOR A LITTLE BIT FOR SOME OF
THIS WORK?
WE OFTEN GRANT HIPAA WAIVERRERS
TO DO THIS WORK AS I HINTED TO
IN THESE LARGE UNCONSENTED DAILY
BASIS THEA SETS, THE PROBLEM
THERE BECOMES THAT THE GOAL IS
GOING TO BE DOING THIS WORK
INTERNALLY AND EVENTUALLY MAKING
IT OPEN SOURCE OR
COMMERCIALIZING AND THEN THIS
QUESTION AS TO ARE THERE
RETAINED PRODUCTS OF IDENT FIELD
FUNCTIONSABILITY IN THE
ALGORITHMS IN INKEV NITRIC OXIDY
ESPECIALLY, SO BEING ABLE TO
DESTROY IT IN ORDER TO MEET YOUR
HIPAA WAIVER, I DON'T THINK THAT
EXISTS FOR THESE ALGORITHMS
EMPLOY I THINK PEOPLE WHO STUDY
THESE ALGORITHMS, IT WILL SUBMIT
OUT THAT SPRAINING DATA SET THAT
YOU THOUGHT MAYBE WASN'T
RETAINED AND WASN'T THERE AND
THERE'S REAL POETIC TEBTIAL, I
THINK LONG-TERM PRIVACY RISKS
THERE.
AND FINALLYI IRBs, THE COMMON
RULE DOESN'T ALLOW US TO
CONSIDER FUTURE RESEARCH RISKS,
RIGHT IN FUTURE GROUP HARMS AND
WHEN YOU THINK ABOUT THINGS LIKE
BIAS IN AI ALGORITHMS IT
ACTUALLY DOESN'T NECESSARY
ARMING THE PEOPLE ENROLLED IN
THE RESEARCH STUDY IN THAT
PARTICULAR STUDY, RIGHT?
IT'S IMPACTING PEOPLE 5, 10, 20
YEARS IN THE FUTURE WHEN IT'S
COMMERCIALIZED AND BROADLY USED
AND THAT'S NOT SOMETHING THAT
TECHNICALLY RIGHT NOW IRBs ARE
ALLOWED TO CONSIDER AND SO,
AGAIN, DO WE NEED REGULATORY
CHANGE TO ALLOW IRBs TO GO
THERE OR DO WE NEED SEPARATE
GROUPS TO BE THINKING ABOUT THE
SOCIETAL LARGER SCALE SOCIETAL
ASPECTS OF SOME OF THESE RISKS.
SO LAST SLIDE HERE, TAKE HOME
POINTS, THERE'S A TREMENDOUS
POWER THAT AI HAS FOR AWESOME
POTENTIAL FOR DIGITAL HEALTH BT
ALSO LOTS OF SCARY PROBLEMS.
AS I SAID, THE PROBLEMS CAN BE
AI CAUSES PROBLEMS BY BEING
WRONG OR PERPETUAL OR
EXCLUSION OR INDEPENDENT BIAS, I
THINK INDEPENDENT REVIEW IS
CRITICAL.
SOME IRBs HAVE A ROLE IN THAT
AND SHOULD BE BUILDING
COMPETENCIES BUT I ALSO THINK WE
NEED TO ENGAGE WITH OTHERS, I
DON'T THINK IT'S OUR JOB TO DO
ALONE.
I THINK WE SHOULD ALL WORK
TOGETHER TO DEVELOP, YOU KNOW
SOME OF THESE TOOLS AS I WAS
SAYING, STABBED ARD TEMPLATES WE
CAN ALL SHARE TO COLLECT THE
SAME NECESSARY INFORMATION THAT
WE NEED UP FRONT EMPLOY WE HAVE
TO WORK WITH RESEARCHERS TO HELP
THEM UNDERSTAND WHY ARE WE
TALKING ABOUT THE FDA HERE AND
WHAT DOES THAT MEAN, AND I'VE
SPENT A LOT OF TIME WORKING WITH
RESEARCHERS TO KIND OF SORT OUT
THOSE QUESTIONS AND HELP THEM
UNDERSTAND WHY WE NEED TO
CONSIDER THEM AND AGAIN, I
THINK, YOU KNOW, THIS IS
COLLABORATIVE WORK, YOU
SHOULDN'T WORRY ALONE AS AN IRB,
WE ALL NEED TO DO THIS TOGETHER,
TO STAY UP TO DATE WITH EVOLVING
REGULATIONS AND NORMS, SO, I DO
WANT TO ACKNOWLEDGE MY
COLLEAGUES HERE WHO HAVE BEEN
PART OF THIS WORK AND I WILL
START THERE AND I WILL LEAVE A
FEW MINUTES FOR QUESTIONS, SO
THANKS VERY MUCH FOR YOUR
ATTENTION.
NTHANK YOU.
WE DO INVITE TO YOU SUBMIT
QUESTIONS BY CLICKING ON THE BOX
THAT SAYS SEND LIVE FEEDBACK
LINK, AND THE DOCTOR IS JOINING
THE DISCUSSION, DO HAVE YOU ANY
QUESTIONS OR COMMENTS?
>> THANK YOU BENYY, IT WAS
AWESOME AS I KNEW IT WOULD BE SO
EXTREMELY HELPFUL EMPLOY I WOULD
SAY I WAS A TAD DISAPPOINTED
BECAUSE YOU DIDN'T ANSWER THE
HARDEST QUESTION FOR ME, YOU
TALKED A LOT ABOUT IT IN THE
BEGINNING AND I STRUGGLE WITH
THIS IS WHEN DOES IT BECOME F,
DIDA REGULATED, RIGHT?
SO WHEN I THINK ABOUT IT, RIGHT,
I MEAN, WE REGULATE THESE
DEVICES, THESE ARE MEDICAL
DEVICES AND IF YOU READ THE
LETTER OF THE LAW, IT'S AN
INVESTIGATIONAL DEVICE WHEN THE
DEVICE IS THE OBJECT OF THE
INVESTIGATION, THEY'RE TESTING
THE SAFETY AND EFFICACY OF THE
DEVICE.
WHICH COULD IN THESE CASES BE
PRETTY FAR DOWN THE ROAD, RIGHT
IN --SO I STRUGGLE.
WHEN THEY'RE FIGURING OUT THE
ALGORITHMS THEY'RE STILL
BUILDING THE DEVICE, THEY DON'T
HAVE IT YET AND I'VE TALKED TO
FDA PEOPLE AND THEY HAVE
ACTUALLY PUSHED IT WAY EARLIER
THAN I EVER WOULD HAVE THOUGHT.
THEY SAID OH NO, AS SOON AS
THEY'RE THINKING ABOUT CREATING
IT, THEIR INTENT IS TO TEST THE
SAFETY AND EFFICACY SOMEWHERE
DOWN THE ROAD WHEN GATE 12
APPLIES EVEN IF IT'S--IT DOESN'T
MATTER THAT THESE ARE
DEIDENTIFIED DAILY BASIS THEA
SETS.
SO I STILL, I STILL--I THINK
THAT AT THE OUTSET BECOMES 1 OF
THE MOST DIFFICULT DECISION
POINTS FOR SUSWHEN WE NEED TO
APPLY.
AND THE OTHER THING JUST TO GET
YOUR THOUGHTS FOR ME, IN MY SORT
OF SIMPLISTIC BRAIN, I THINK I
JUST DRAW THE ANALOGY TO MORE
TRADITIONAL TO A TRAW
DIAGNOSTIC.
IN THIS CASE, INSTEAD OF A
TISSUE BIOPSY YOU HAVE A DATA
BIOPSY.
SO THE DATA IS LIKE THE SPECIMEN
HERE, AND THAT'S, YOU KNOW, THE
SUBJEK, THE HUMAN SUBJECT HERE
AND THEN JUST TRY TO APPLY THE
SAME FRAMEWORK AS I DO FOR EVERY
OTHER INVITRO DIAGNOSTIC STUDY
DO IT AS BEST I CAN AND I AM
JUST WONDERING IF YOU THINK
THAT'S A GOOD APPROACH, DO YOU
THINK OF THAT, LIKE DO YOU DRAW
THE SAME ANALOGY, THE DATA HERE
IS THE BIOSPECIMEN?
>> YEAH, THANK YOU SO MUCH,
EXCELLENT QUESTION.
RIGHT, THE FIRST 1, I MEAN I'M
RIGHT THERE WITH YOU, I THINK
THAT IS THE--THAT IS THE
CRITICAL QUESTION AND RIGHT, THE
REASON I DIDN'T ANSWER IT IS
BECAUSE I DON'T THINK WE HAVE AN
ESTABLISHED RIGHT ANSWER, RIGHT
SO I HAVE SEEN HAD IRBs THAT
TAKE THE CONSERVATIVE APPROACH
AND SAY RIGHT, IF YOU ARE
THINKING ABOUT TESTING THE
SAFETY AND EFFICACY OF THIS
DEVICE AND YOU'RE JUST STARTING
TO BUILD IT ASK TRAIN IT THAT,
QUOTE UNQUOTE TRAINING IS PART
OFIOURE TESTING AND WE'RE GOING
TO CALL--WE EXPECT THAT THE FDA
WILL WANT TO GO BACK AND LOOK AT
HOW IT WAS BUILT AND TRAINED AND
THAT IT'S ALL GOING TO BE FDA
REGULATE EXCLUDE WE WILL CALL IT
THAT WAY FROM THE BEGINNING AND
THEN I'VE SEEN OTHER IRBs THAT
HAVE BASICALLY SAID, YOU KNOW AS
LONG AS YOU DO IT AS 2 SEPARATE
PROTOCOLS, AS LONG AS YOU HAVE 1
PROTOCOL THAT IS THE INITIAL
DATA GATHERING AND THINKING
ABOUT IT AND BUILDING AND
TRAINING AND THEN HAVE YOU
ANOTHER PROTOCOL THAT IS THE
ACTUAL REAL WORLD, YOU KNOW
TESTING OF THE SAFETY AND
EFFICACY THAT WE'RE GOING TO
SEPARATE IT OUT AND SAY THE
FIRST 1 IS NOT FDA REGULATED AND
THE SECOND 1 IS, SO I THINK
WE'RE NOT UNIFORM IN HOW WE'RE
HANDLING THIS, I CAN TELL YOU
THAT I THINK WE ARE TAKING THE
CONSERVATIVE APPROACH FOR A
WHOLE VARIETY OF REASONS UNTIL,
YOU KNOW I THINK ULTIMATELY IT
WOULD BE GREAT IF THE FDA COULD
PROVIDE US A DEFINITIVE ANSWER
THIS, IT'S FUNNY WHAT YOU SAID
BECAUSE I WAS TALKING ABOUT
SOMEONE LAST WEEK WHO SAID THEY
WERE IN A MEETING WITH A--I
WON'T NAME NAMES BUT A FAIRLY
HIGH LEVEL FDA PERSON WHO
PROVOIDED THE OPPOSITE ADVISE
AND SAID BASICALLY IF YOU'RE NOT
AT THAT STAGE WHERE THERE'S AN
ACTUAL DEVICE THAT YOU BUILT AND
TRAINED, WE DON'T WANT ANYTHING
TO DO WITH IT.
SO I THINK WE'RE GETTING MIXED
MESSAGES AND I THINK, YOU KNOW,
I THINK EITHER POSITION IS
DEFENSIBLE SO I THINK YOU HAVE
TO PICK 1 AND DO IT AND WE'LL
HOPE THAT THE FDA TELLS US MORE
SOON AND THEN YOU'RE SECOND
QUESTION, THE SHORT ANSWER IS
YES, I THINK ABOUT IT EXACTLY
THE SAME WAY AS AN INVITRO
DIAGNOSTIC DEVICE FOR THE AI
ALLEGOR RYTH EPIGENETICS THAT
ARE DEVICES, SO NOT ALL, BUT IN
THOSE SWIGZS, I THINK ABOUT IT
THE SAME WAY AND I THINK ABOUT
THE ID EXEMPT DIAGNOSTIC DEVICE
CATEGORY AS WELL.
>> WELL, LET'S GO TO THE
AUDIENCE QUESTIONS BECAUSE I
COULD PICK YOUR BRAIN ALL DAY.
>> THIS 1 SHOULD BIOMEDICAL AI
STUDIES BE REQUIRED TO INCLUDE
AND REPORT ARK I ATTENUATION?
MEANING THE ABILITY TO REQUIRE
AI TO SHOW WHAT IT IS
SPECIFICALLY USING TO MAKE THE
DECISION?
>> YEAH, I ALLUDED TO THAT A
LITTLE BIT BUT WE NEED SOME
DEGREE OF EXPLAINABILITY, WE
NEED SOME DEGREE OF
UNDERSTANDING HOW THE AI IS
WORKING AND HOW IT'S SUPPOSED TO
WORK.
BASICALLY IF YOU THINK ABOUT IT
FROM THE IRB PERSPECTIVE, WE
HAVE TO MONITOR FOR
UNANTICIPATED PROBLEMS, SO
PEOPLE ARE EXPECTED TO REPORT TO
US ON UNEXPECTED PROBLEM ANDS IF
YOU DON'T AT LEAST PUT YOUR
NICKEL DOWN, AROUND WHAT IS
ANTICIPATED AND WHAT IS
UNANTICIPATED, IT'S HARD TO DO
THAT SO I THINK WE HAVE TO, I
THINK THE CHALLENGE IS, AND YOU
KNOW MY BACKGROUND IS ETHICS AND
IRBPAYS AND ALL THAT, BUT I DO
NOT CLAIM TO BE AN AI EXPERT BY
ANY MEANS AND I THINK THE AI EX
PERS WILL TELL YOU THE CHAL
WENCH THAT IS AGAIN, YOU KNOW
THIS MEETING I WAS AT EARLIER
THIS WEEK, ACTUALLY, PEOPLE
SAYING EXPLAINABILITY IS A MYTH,
LIKE WE'RE NOT--AS THESE THINGS
GET BETTER AND BETTER AND
BETTER, WE'RE NOT ACTUALLY GOING
TO BE ABLE TO DO THAT SO PERHAPS
WE'RE ADDING A REQUIREMENT
THAT'S NOT POSSIBLE AND AGAIN,
WE DON'T WANT TO TYPHLE
INNOVATION, SO, I, YOU KNOW I
THINK RIGHT NOW, WE SHOULD DO
OUR BEST TO TRY ON DO THAT,
WHERE THE FUTURE GOES, IS
ANYONE'S GUESS.
>> OKAY, WE HAVE A COMMENT FROM
A RADIOLOGIST HERE.
I'M VERY FAMILIAR WITH THE CHEST
X-RAY ARTICLE THAT YOU
MENTIONED, IT'S NOT SURPRISING
THAT AI COULD PREDICT THE
PATIENT'S RACE EVEN WHEN WE
CAN'T SEE THE INFORMATION ON THE
IMAGE WITH THE NAKED EYE.
FOR EXAMPLE, YOU WOULD BE
SURPRISED TO KNOW THAT I CAN
TELL YOU THE EXPECTATIONS IS OF
A PATIENT BY LOOKING AT THE
CHEST X-RAY, OF COURSE NOT,
THERE ARE OBVIOUS CUES.
FOR RACE, THERE ARE CUES TOO,
BUT THE AUTHORS OF THE PAPER
MENTION DID NOT UNDERSTAND HOW
THEIR INTERVENTIONS DID NOT
OBSCURE THE CUES, PLEASE DO NOT
EXAGGERATE THE SIGNIFICANCE OF
THE FINDINGS OF THAT OR SIMILAR
PAPERS.
SUCH MISCONACCEPTINGS COULD LEAD
TO EXAGGERATED CONCERNS FOR
IRBs AND OVERREGULATION OF AI
RESEARCH.
NI APPRECIATE THE COMMENT.
ALL I CAN SAY IS, I DON'T THINK
THAT THE POTENTIAL FOR BIAS IN
AI ALGORITHMS IS EXAGGERATED.
I THINK MAYBE THAT PARTICULAR
EXAMPLE COULD BE BUT I THINK
THAT THE POETIC TEBTIAL FOR A I
BIAS AND ALGORITHMS IS REAL AND
WE HAVE TO BE THINKING ABOUT IT.
>> IS THERE A PATH WHERE WE
CAPTURE ADVERSE EVENTS AND
CONFIDENTIALITY RESEARCH IN AI
RESEARCH?
>> SAY IT AGAIN IN.
>> IS THERE A PATH WHERE WE
INCLUDE THE CAPTURE OF PRIVACY
AND CONFIDENTIALITY CONCERNS IN
AI RESEARCH?
>> I THINK AGAIN IT GOES BACK TO
WHAT IS AN UNANTICIPATED PROBLEM
VERSUS AN ANTICIPATED PROBLEM,
RIGHT AND SO, I MEAN THERE'S IT
REIDENT FIELD FUNCTIONSABILITY
OCCURS AND YOU KNOW PEOPLE ARE
ABLE TO IEE, AUDIENCE DENTIFY
SOMETHING THAT YOU WEREN'T ABLE
TO IDENTIFY.
YOU WOULD HAVE TO REPORT THAT AS
AN UNANTICIPATED PROBLEM, I
MEAN, YOU KNOW WHETHER IT'S AN
ADVERSE EVENT NECESSARILY, I
MEAN, IT'S A SEPARATE
CONVERSATION, BUT YOU KNOW
ESSENTIALLY 1 COULD ARGUE THAT
CONFIDENTIALITY OR VIOLATIONS
ARE AN EXPECTED ADVERSE EVENT
ANYTIME YOU COLLECT SOMEONE'S
PRIVATE INFORMATION, RIGHT?
>> REGARDING FUTURE GROUP ARMS,
DRUGS CAN CAUSE GROUP HARMS, BY
BEING AN EFFECTIVE OR MORE TOXIC
IN CERTAIN PATIENT POPULATIONS.
SHOULD THE IRB ASSESS DRUGS FOR
THOSE ARMS, WHY IS AI DIFFERENT
IN THIS REGARD?
>> WELL I GUESS WHAT I WOULD SAY
IS I THINK THAT WE TRY TO ASSESS
THAT FOR DRUGS RIGHT NOW, RIGHT?
SO WHEN PEOPLE DO YOU KNOW
PIVOTAL TRIALS TO GET APPROVAL
FOR DRUGS, THE HOPE IS THAT
THEY'RE ENROLLINGA ADEQUATELY
DIVERSE POPULATIONS TO BE ABLE
TO ARK ASSESSES THAT.
RIGHT?
AND THE IRB CERTAINLY CONSIDERS
THAT AS PART OF, YOU KNOW
EQUITABLE SUBJECT SELECTION, ET
CETERA EMPLOY SO I DON'T THINK
IT'S ACTUALLY DIFFERENT.
I THINK WE HAVE TO CONSIDER IT
IN BOTH SITUATIONS.
>> OKAY, DR. GREEN, DR. S
ILVERMAN, ANY FURTHER QUESTIONS?
WE'RE WELL OUT OF TIME.
>> NO, THANK YOU VERY MUCH FOR
COMING AND PRESENTING TO US A
TOPIC FOR MUCH DISCUSSION.
I'M SURE WE WILL HAVE MUCH MORE
QUESTIONS ABOUT IT AND THANK
YOU.
NTHANKS FOR HAVING ME NTHANK YOU
TO YOU BOTH FOR JOINING US
TODAY.
>> THANK YOU.
>> TAKE CARE.