>> WELL, GOOD MORNING, EVERYONE. IT GOT QUIET ALL OF A SUDDEN SO THAT'S A SIGN IT'S TIME TO GET START. I'D LIKE TO WELCOME YOU TO THE MEETING ON MEASURE INTELLIGENCE IN HEALTH CARE AND DR. BRUCE TROMBERG AND I ARE THE WELCOMING COMMITTEE SO WE'D LIKE TO THANK YOU VERY MUCH FOR BEING HERE TODAY AND AN ALSO UNDERSTAND THERE'LL BE A LARGE CONTINUE GENT ON THE WEBCAST SO I IMAGINE THERE'LL BE PEOPLE JOINING US THROUGH THE DAY. I'M JONI RUTTER AND I'M HERE TO WELCOME YOU TO THIS MEETING AND TALK A LITTLE BIT ABOUT THE APPLICATION OF MACHINE INTELLIGENCE IN HEALTH CARE. THE APPLICATION OF CUTTING EDGE TECHNOLOGIES IN THE HEALTH ECO SYSTEM NOT REALLY NEW AND WE'VE HEARD OF MACHINE LEARNING AND FIELD LEARNING IN THE FIELD OF HEALTH CARE BUT MACHINE USED IN THE CONTEXT OF THIS RM - WORKSHOP AND IT IS DEFINED AS THE ABILITY OF A TRAINED COMPUTER SYSTEM TO PROVIDE RATIONAL UNBIASSED GUIDANCE TO HUMANS IN SUCH A WAY THAT ACHIEVE OPTIMAL OUTCOMES IN A RANGE OF ENVIRONMENTS AND CIRCUMSTANCES. SO AS YOU THINK ABOUT THAT DEFINITION OF MACHINE INTELLIGENCE WE'RE AIMING TO GAME PERSPECTIVES AND FEEDBACK FROM YOU ON THE ASSOCIATION OF THE INCORPORATION OF SUCH TOOLS INTO HEALTH CARE. WHAT ARE THE RODE -- ROAD BLOCKS TO ADVANCE FURTHER AND THE APPROPRIATE GUIDANCES AN UNIQUE CHALLENGES FOR ETHICAL APPLICATION OF MACHINE INTELLIGENCE IN HEALTH CARE SETTINGS. SO THESE ARE BIG AND BROAD QUESTIONS WE'RE ASKING TO YOU THINK ABOUT TODAY AND THE OUTPUTS AND THE DISCUSSIONS FROM THIS WORKSHOP AND THE DIFFERENT SESSIONS THAT WE'LL BE HAVING WILL BE USED TO DEVELOP A WHITE PAPER ON TRANSLATING MACHINE INTELLIGENCE FOR CLINICAL APPLICATIONS AND THE ASSOCIATED IMPROVEMENTS THAT ARE GOING TO BE NEED AS WE THINK ABOUT THIS MOVING FORWARD. THIS IS AN AREA THAT IS OF OUTMOST IMPORTANCE TO US AT MCATS AND THE PARTNERS AT THE NATIONAL INSTITUTE OF BIOIMAGING AND ENGINEERING AND THE NATIONAL CANCER INSTITUTE AND BROADER AT THE NIH. THERE'S AN ADVISORY COMMITTEE TO THE DIRECTOR ON ARTIFICIAL INTELLIGENCE, FOR EXAMPLE. THIS IS A BIG ISSUE THAT NIH IS WANTING TO GET MORE INPUT ON AND START FIGURING OUT WAYS IN WHICH WE FIELD IT'S BEING IMPLEMENTED IN A TRANSPARENT AND ETHICAL MANNER TO ENABLE US TO REACH MORE PATIENTS MORE EFFICIENTLY. SO AS THE DAY UNFOLDS, YOUR CHALLENGE IS TO THINK BIG, PUT ON YOUR POSSIBILITY THINKING HATS. TODAY, NOTHING IS IMPOSSIBLE AS YOU THINK ABOUT THE DIFFERENT QUESTIONS YOU'LL BE GRAPPLING WITH SO FOR EACH SESSION THINK OF AT LEAST ONE AREA THAT YOU'D LIKE TO CONTRIBUTE, ONE THOUGHT YOU'D LIKE TO CONTRIBUTE AND WHAT YOU THINK IS CRITICAL FOR MOVING THE FIELD FORWARD. THIS IS A GREAT GROUP OF PEOPLE HERE THE GROUP HAS BROUGHT TOGETHER AND WE'RE REALLY COUNTING ON YOU TO SET THE STAGE FOR THINKING ABOUT MACHINE INTELLIGENCE TOOLS, RESOURCES AND NEEDS TO ADVANCE THE FIELD. I'D ALSO LIKE TO TAKE A MOMENT TO THANK CARLI SHARMA AND CHRISTINE CARTILLO WHO HELP ORGANIZE THE MEETING AND THE OTHER FOLKS INVOLVED IN THE WORKING GROUP TO PUT THIS GREAT GROUP TOGETHER, SO THANK YOU, GUYS. AND FOR NOW I'LL TURN IT OVER TO DR. TROMBERG THE DIRECTOR OF NIBIB FOR A FEW WORDS AS WELL. >> THANKS, JONI. GOOD MORNING, EVERYONE. SO I'VE BEEN HERE, WELL, FIRST I'D LIKE TO THANK ALL YOU HAVE FOR COMING. IT'S REALLY GREAT THERE'S SO MUCH EXCITEMENT AND INTEREST AND EVERYONE WHO ORGANIZED AND PUT ENERGY INTO THIS AND MY TEAM AT THE NIBIB AND BRUCE SHABASTARI WHO LEADS OUR INFORMATICS BASED PROGRAM AND GRACE IS OUR MODELING EXPERT, A PIONEER IN THIS AREA OF MATHEMATICAL MODELING AND I'VE BEEN AT THE NIN NOW FOR SEVEN MONTHS -- NIH. AND I FEEL SINCE I GOT HERE THE NIH HAS MAYBE DISCOVERED MATHEMATICS SO IT'S A COOL THING. I HADN'T SEEN THAT IN MY PREVIOUS 30 YEARS OF MY ACADEMIC CAREER. WHAT I'D LIKE TO DO AND I'LL PIVOT A LITTLE BIT FROM JONI'S REMARKS AND GIVE YOU A FEW OF MY IMPRESSIONS OF THAT DISCOVERY THAT'S JUST NOW BEEN MADE WHERE I THINK THERE'S KEY AREAS TO CONCENTRATE ON AS WE EMBARK ON THIS INTERESTING EXPANSION OF MACHINE INTELLIGENCE, ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING AND SO FORTH. AS YOU KNOW THE NIBIB SUPPORTS ENGINEERING AND PHYSICAL SCIENCE AND THAT'S THE BREAD AND BUTTER OF THE COMPUTATIONAL METHODS AND TRYING TO DEVELOP SYMBOLIC AND MATHEMATICAL REPRESENTATIONS OF THOSE IDEAS AND USING THAT MATHEMATICAL FRAMEWORK WITH A NUMBER OF OBSERVATIONS OF A SYSTEM TYPICALLY PHYSICAL SYSTEMS OR AT LEAST THE PHYSICAL COMMUNITY LIKES TO CALL THE PHYSICAL SYSTEMS AND THERE'S AN EQUATION AND THERE'S A BUNCH OF OBSERVATIONS AND THERE'LL BE SOME PARAMETERS IN THAT EQUATION THAT WE TRY TO RECOVER IN ORDER TO PREDICT THE FUTURE OF THAT BIOLOGICAL SYSTEM. MAYBE IT'S HOW YOUR PARTICULAR ORGAN IS WORKING. MAYBE IT'S THE FUTURE OUTCOME OF A PATIENT. MAYBE IT'S IDENTIFYING A REGION OF AN INTEREST IN AN IMAGE THAT CAN TELL YOU WHETHER SOMETHING IS DISEASED OR THE EXTENT OF DISEASE AND SO FORTH. AND BROADLY SPEAKING THERE'S TWO TYPES OF MATHEMATICAL APPROACHES. THERE'S WHAT WE CALL ANALYTICAL APPROACHES WHERE THERE'S AN EQUATION WITH A CLOSED-FORM SOLUTION AND THAT EQUATION IS BASED ON SOME SORT OF AN ABSOLUTE TRUTH AND THEN THERE ARE THESE GENERALIZED METHODS THAT ARE NUMBER HUNGRY. THEY'RE NUMERICAL SOLUTIONS THAT ARE REQUIRED AND WE HAVE TO FEED THEM NUMBERS. IN THOSE STATISTICAL TECHNIQUES, THEY GENERALLY FALL INTO CORRELATIVE APPROACHES AND THEN THERE'S LEARN-BASED APPROACHES WHERE YOU MAY HAVE NUMBERS TO FEED BUT THEN YOU HAVE TO HAVE A WHOLE BUNCH OF EXPERIMENTS AND KNOW THE ANSWERS AS WELL AND YOU FEED THE NUMBERS AND THE ANSWERS AND YOU HAVE SOME LEARNING GOING ON AND OUTCOMES WHAT ALL THE EXCITEMENT IS RIGHT NOW. SO JUST SOME OBSERVATIONS ABOUT THOSE, DON'T FORGET THE ANALYTICAL MODELS BECAUSE THOSE ARE VERY POWERFUL APPROACHES TO SOLVING PROBLEMS. I THINK IT'S INSTRUCTIVE AS WE THINK ABOUT THE BEAUTY OF THOSE ANALYTICAL MODELS AND I KNOW IT'S AN ABSTRACTION AND HERE'S A REALITY. EVERYBODY HEARD OF GRAVITATIONAL WAVES AND THE NOBEL PRIZE WAS AWARD BASED ON A 100-YEAR-OLD MODEL ON MANY KNOW IT AS THE RELATIVE THEORY BY EINSTEIN AND IT'S AN ABSOLUTE MODEL WITH COMPONENTS WE DON'T DO MUCH IN BIOLOGY IN THOSE TERMS IN MEDICINE SO ONE THOUGHT I'D LIKE TO PLANT IN YOUR HEADS IS TO PERHAPS USE THIS OPPORTUNITY, THIS INTEREST IN MATHEMATICAL TECHNIQUES TO BEGIN TO THINK ABOUT MODELS THAT COULD BE TESTABLE OF BIOLOGIC AND PHYSIO LOGIC SYSTEMS WHERE YOU HAVE MODELERS DEVELOPING IDEAS AND THEN TESTS, MEASUREMENTS, TO PROF -- PROVE OR DISPROVE THIS FRAMEWORK AND SOME MAY HAVE FOLLOWED THE EXPOSON. THERE'S MULTIVERSE AND SUPER SEM TRY AND WHEN YOU MEASURE THE ENERGY IT WAS SUPPOSED TO FALL INTO ONE MODEL OR ANOTHER AND THE IT FELL IN THE MIDDLE SO NOBODY KNEW WHICH WAS THE RIGHT MODEL BUT THEY'RE INSTRUCTIVE WAYS OF EXPANDING OUR MINDS AND THINKING ABOUT WHAT WE CAN DO IN OUR FIELD RATHER THAN JUST CONTINUE TO DRILL DOWN AND THINK SMALLER AND SMALLER ABOUT MACHINE LEARNS APPROACHES. LET'S STEP BACK AND THINK MORE BROADLY AND WHAT THE BIG OPPORTUNITIES ARE. SO IN ORDER TO GET TO THOSE BIG OPPORTUNITIES WHERE WE CAN GAIN ENTIRELY NEW INSIGHT AND POSSIBLY HAVE VIRTUAL HUMANS AND ORGANS AND MODEL THEM ENTIRELY AND DO CLINICAL TRIALS VIRTUALLY WITH WELL-TESTED AND VALIDATED MODEL-BASED SYSTEMS, WE NEED TO MAKE SURE IN OUR EXPERIMENTS, ALL OUR OBSERVATIONS, WE HAVE REALLY GOOD DATA. WE NEED TO UNDERSTAND THE DATA BOTH THE SOURCES OF THE SIGNALS AND THE ORIGIN OF THE NOISE IN THE DATA BECAUSE IF YOU'RE FEEDING BAD DATA INTO YOUR MACHINE LEARNING APPROACH WITH A LOT OF NOISE AND THE DATA ELEMENTS AREN'T TRULY ORTHOGONAL YOU'LL WIND UP WITH THE WRONG ANSWER. IT WON'T BE ULTIMATELY PREDICTIVE FOR ALL TIME WHICH IS ABOUT WHAT WE'RE HOPING FOR WITH ALL THE APPROACHES. THE SECOND THING I'D MANY OF IS THINKING OF -- I'D EMPHASIZE IS LOOKING AT THE EQUATIONS, THEY'RE ALL GOING ON IN YOUR BODY BUT BIOLOGICAL SYSTEMS ARE TOO COMPLICATED TO MODEL EVERY COMPONENT WITH ALL OF THAT BUT WE CAN INTRODUCE PHYSICAL AND MECHANISTIC CONSTRAINTS THAT ARE BASED ON THESE UNDERLYING, GROUNDED PHYSICAL PRINCIPLES THAT WILL IMPROVE OUR MACHINE INTELLIGENCE, MACHINE LEARNING APPROACHES AND PUT BOUNDARY CONDITIONS, IN A SENSE, ON HOW WE GET OUTPUTS OR ANSWERS FROM WHAT WE'RE DOING. SO THOSE ARE EXTREMELY IMPORTANT, PHYSICAL AND MECHANISTIC CONSTRAINTS. AND I THINK ONE OF THE BIG PAYOFFS IS THAT FROM DOING ALL OF THIS, WE'LL GAIN EVEN PERHAPS MORE INSIGHT INTO HOW BIOLOGIC SYSTEMS WORK AND DEVELOP CONCEPTUALLY HOW FRAME WORKS WORK AND HOW DRUGS AND COMPLEX PARAMETERS WILL VARY WITH THE ADDITION OF THOSE AND WHAT WE CAN INTRODUCE TO HELP HEAL PATIENTS. I THINK THOSE ARE SOME OF THE BIG PAYOFFS. WE HAVE TO PAY ATTENTION TO SOME OF THE TECHNICAL DIFFICULTIES AND FEATURES THAT NED TO BE ADDRESSED AND I KNOW THAT YOU WILL TALK ABOUT SOME OF THOSE THROUGHOUT THE DAY. I KNOW BERKMAN WILL TALK ABOUT SOME OF THOSE THINGS TODAY AND MY LAST KIND OF PITCH HERE, BECAUSE I'M THE DIRECTOR OF THE NIBIB AND TRYING TO MAKE SURE ALL BIOENGINEERS HAVE MANY EMPLOYMENT OPPORTUNITIES, I THINK IT'S IMPORTANT BECAUSE THIS COMMUNITY BIOENGINEERING, WORKS IN THIS SPACE. IT'S INTUITIVE AND MULTILINGUISTIC TRAINING AND YOU NEED TO UNDERSTAND NOT JUST THE COMPUTATION BUT THE PHYSICAL PRINCIPLES UNDERLYING COMPUTATION AND NOT JUST SOME ASPECTS OF BIOLOGY BUT HOW COMPONENTS IN BIOLOGIC SYSTEMS WORK TOGETHER TO FORM COMPLEX SYSTEMS. IT'S THAT LANGUAGE ACROSS ALL THOSE SCALES, SPACE AND TIME SCALES, BIOLOGY TO PHYSICS THAT'S EXTREMELY IMPORTANT TO DEVELOP. NO SINGLE PERSON KNOWS ALL THAT STUFF SO THAT'S WHY IT'S SO GREAT THAT ALL THE TEAMS ARE COMING TOGETHER TO REALLY THINK ABOUT THIS AND DISCUSS IT. BIOENGINEERS ARE REALLY GOOD PEOPLE IN THE MIDDLE WHO CAN REACH TO ALL THE SIDES AND LINK ALL THIS TOGETHER SO I THINK THEY'RE ESSENTIAL TO BUILD INTO YOUR TEAMS WHICH MAY HAVE PHYSICISTS AND COMPUTATIONAL EXPERTS AND PHYSICIANS AND BIOLOGISTS. VERY EXCITING TO SEE. I'M HOPING THAT I CAN EVEN CONVERT THESE REMARKS BECAUSE I SEE I HAD NO IDEA EVERYONE WAS SO EXCITED ABOUT ALL THIS STUFF AND I'LL TRY TO WORK WITH ALL OF YOU TO CONTINUE TO CONVERT AND CAPTURE THESE CONCEPTS INTO ARCHIVAL MATERIAL TO HELP WITH GUIDE POSTS. SO YOUR OUTPUT FROM THE DAY WILL HELP ME AS I GO ON AND TRY TO EXPRESS THESE THOUGHTS AND HELP SHAPE WHAT I THINK ARE WONDERFUL OPPORTUNITIES FOR OUR ENTIRE FIELD. SO, WITH THAT, CAN WE START? THANK YOU VERY MUCH. >> SO I'M JUST GOING TO SAY A FEW HOUSEKEEPING AND REMINDER RULES. SO THANK YOU BRUCE AND JONI. AND THANK YOU ALL FOR PARTICIPATING TODAY. WE'RE HOPING FOR THIS ROBUST DISCUSSION. I'M CHRISTINE KATILO AND IF YOU HAVE ANY QUESTIONS THROUGHOUT THE DAY JUST FIND ONE OF US OR DEBBIE NEAR THE FRONT DOOR. YOUR FEEDBACK AND PARTICIPATION IS CRUCIAL AS JONI SAID SO FOR IN-PERSON THAT MEANS PARTICIPATING AND FOR ONLINE FOLKS FEEDBACK CAN BE SENT THROUGH THE COMMENT BOX IN THE VIDEOCAST OR THE E-MAIL BOX WHICH IS ROTATING ON THE SLIDES BUT ALSO NCATS WORKSHOP AT NIH.gov. WE MAY NOT BE AIBLE -- ABLE TO INCORPORATE ALL QUESTIONS BUT ALL FEEDBACK RECEIVED IN PERSON AND ONLINE IN THE WHITE PAPER PROCEEDINGS REGARDLESS OF THE MECHANISM SO LET'S GET STARTED AND I'LL INTRODUCE THE CHAIR OF OF OUR FIRST SESSION TRUCE WORTHINESS LUCA FOSCHINI THE CO-FOUNDER AND CHIEF DATA SCIENTIST AT EVIDATION HEALTH. >> THANK YOU EVERYONE. EXCITED TO BE HERE. I'LL INTRODUCE THE OTHER PANELISTS WE HAVE BRIAN ALPER VICE PRESIDENT OF INNOVATION EVIDENCE-BASED DEVELOPMENT FOR EBSCO HEALTH AND WE HAVE MARTIAL LAW CLARK AND THANK YOU FOR HAVING ME AND LET'S GET STARTED. SO THIS IS A REVIEW FROM A COUPLE MONTHS AGO TOUCHING UPON ALL THE A.I. DEVELOPMENTS THAT TOUCH THE MEDICAL DOMAIN MOSTLY BASED ON DEEP LEARNING AND COVER THE GAMUT OF DIFFERENT OUTCOMES YOU CAN PREDICT SEPSIS, SUICIDES, DELIRIUMS AND COVER THE SPAN OF A LIME TIME OF A PERSON. THIS IS NOT ONLY ABOUT RESEARCH THOUGH. THIS IS A NICE INFO GRAPHIC THAT SHOWS THE RECENTLY IN THE LAST FEW YEARS FDA CLEARED DEVICE THAT HAVE AN M.I. COMPONENT AND THERE'S MORE IN THE RECENT YEARS THAN IN THE PAST. SO THESE SYSTEMS ARE AROUND US. SO HOW DO WE TRUST THEM? HOW CAN WE TELL WE TRUST M.I. AND I'LL USE M.I. AND A.I. INTERCHANGEABLY IN THE TALK. IT'S HARD TO DEFINE TRUST. IT'S EASY TO DEFINE SO AT LEAST WE CAN SAY A NECESSARY CONDITION TO TRUST AN M.L. STOYSTEM IS TO TRUST ALL THE COMPONENTS. THE TRAINING DATA AND MODEL BUILDING AND WHEN IT'S DEPLOYED. SO FOR THIS TALK I'M GOING TO TOUCH SPECIFICALLY ON THE TRAINING DATA PART. IN FIELD MORE MATURE IN THE MEDICAL AND HEALTH DOMAIN IT'S SEEN AS THE CRUCIAL PART OF PERFORMANCE OF AN M.L. SYSTEM. PEOPLE THINK OF ENGINEERING DATA SETS THERE THAT ARE VERY SPARSE AND DIVERSE THAN COMMODITIES IN THE FIELD WHERE MACHINE LEARNING IS MORE ADVANCE. WHAT DOES IT MEAN TO TRUST THE DATA THEN? I THINK WE HEAR THE PHRASE BAD DATA ALL THE TIME BUT MOST THE TIME WE FAIL ON WHAT THAT MEANS AND WITHOUT A CLEAR ARTICULATION OF WHAT BAD DATA MEANS FOR A SPECIFIC SUBJECT WE CAN'T DEFINE GOOD DATA THE DATA WE CAN TRUST. HERE WE CAN TAKE A PAGE FROM THE BOOK OF PEOPLE THAT WORK ON REAL WORLD EVIDENCE. THIS IS A DISCIPLINE IN WHICH PEOPLE LOOK AT DATA COLLECTED FOR OTHER PURPOSES SUCH AS MEDICAL CLAIMS COLLECTED BY INSURANCE AND TRYING TO MAKE MEDICAL INFERENCES OUT OF THAT SUCH AS VACCINE EFFECTIVENESS OUT OF CLAIMS DATA. REALLY HAVE TO TRUST YOUR DATA THERE BECAUSE THE DATA IS NOT BEING COLLECTED FOR THAT KIND INFERENCE AND COULD BE WORSE IN MANY WAYS. SO THIS DOMAIN HAS BEEN THINKING ABOUT HOW TO REALLY SCORE DATA QUALITY FOR A LONG TIME AND THIS IS AN INFO GRAPHICS FROM A WORKING GROUP TO DEFINE WHAT DOES IT MEAN TO USE REAL WORLD DATA FOR REGULATORY PURPOSES. SO A VERY HIGH BAR OF TRUST. THEY HAVE LOOKED AT THE PANORAMA OF HOW DATA CAN BE BAD AND NOT ONLY BAD DATA. IT CANNOT BE FIT FOR USE IN MANY WAYS. IT COULD HAVE BAD RELIABILITY OR QUALITY CONTROL. TODAY I'M GOING TO FOCUS ON THE VERIFICATION AND THE SUBSET CONFORMANCE AND PLAUSIBILITY IN THE CONTEXT OF PATIENT-GENERATED DATA WHICH IS WHAT OUR COMPANY WORKS ON. SO PATIENT-GENERATED HEALTH DATA TERMED AND IT'S NOT ONLY THE PATIENT THAT GENERATES THAT IS IN THE REAL WORLD THROUGH DEVICE COULD BE ANYTHING FROM A FITBIT TO A MEDICAL GRADE DEVICE TO P.R.O.s AND OUR STUDIES ARE MOSTLY USING THIS KIND OF DATA SO FOR US BEFORE WE START DOING ANY ANALYSIS IT'S IMPORTANT TO ASK IS THIS BAD DATA BECAUSE IT COMES FROM REAL WORLD FROM DATA THAT COULD EASILY NOT BE BAD. I'M GOING ILLUSTRATE CONFORMANCE AND PLAUSIBILITY AND COMPLETENESS. CONFORMANCE IS THE EXPECTATION OF THE FORMAT OF THE DATA. IF YOU'RE EVER DEALT WITH A FITBIT AND TRYING TO GET DATA ABOUT SOMEONE'S SLEEP THAT'S BEEN CONSENTED TO BE SHARED WITH YOU WILL KNOW IT WILL BE RETURNED TO YOU WITH A NOTION OF STAGING UNLESS THE EVENT IS SHORTER THAN A CERTAIN AMOUNT AND THERE'S ANOTHER WAY OF CLASSIFYING THAT IF YOU'RE EXPECTING SCHEMA AND YOU'RE IN LUCK BECAUSE IT NEEDS TO BE AWARE OF THE DATA CONDITION. COMPLETENESS IS ONE OF THE MOST IMPORTANT THINGS FOR PERSON GENERATED DATA. MISSING DATA CAN APPEAR IN MANY WAYS. AND ERRORS IN THE E.P.I., DEVICE MISSION AND PARTICIPANT NOT REALLY WEARING IT AND ILLUSTRATING THE EXAMPLE WITH FITBIT PATIENT GENERATED DATA IT CAN GIVE COMPLETENESS ACROSS CHANNELS IF YOU SEE ZERO STEPS BUT YOU SEE A HEART RATE YOU KNOW THE PERSON IS WEARING THE DEVICE AND NOT WALKING VERSUS WEARING THE DEVICE AND PLAUSIBILITY IS THE EXPECTATION OF THE VALUE YOU'RE GETTING IS RIGHT IF YOU SEE 1,000 STEPS WELL, THERE COULD BE AN ULTRA MARATHON SO IT'S A LITTLE BIT UNPLAUSIBLE BUT NOT COMPLETELY BOGUS AND IF YOU SEE A NUMBER BETWEEN 199 FOR TWO HOURS THAN YOU KNOW THERE'S A PROBLEM THERE. SO WHAT CAN WE DO WITH THIS? THERE'S BEEN AN ONGOING EFFORT TO DEFINE WAYS OF DOCUMENTING DATA SETS. THIS IS ONE EXAMPLE IS CALLED DATA SHEET FOR DATA SET PUBLISHED LAST YEAR AND PRESENTED AT A CONFERENCE AND IT'S A WAY TO CAPTURE HOW DATA IS PROCESSED AND DESIGNED BUT I THINK WE HAVE AN OPPORTUNITY TO BRING THE LEARNINGS INTO OUR FIELD AND EXTEND THEM IN THE DIMENSION OF THE REAL WORLD DATA PLAUSIBILITY FRAMEWORK ALONG RELEVANCE,ACCRUAL AND VALIDATION AND COMPLETENESS IN MEDICAL DOMAINS. WE WANT TO SCORE DATA SET AND WE SPEND A LOT OF TIME SCORING WHERE WE GET THE EDUCATIONS WE SHOULD BE SCORING WHERE THE MODELS GET EDUCATION TOO WHICH IS TRAINING DATA. FOR THE LAST COUPLE MINUTES I'M GOING TO TOUCH ON THE REMAINING PART OF THE PIPELINE QUICKLY TO GIVE REFERENCE TO TALK ABOUT INDEPENDENT LATER ON. MODEL BUILDING. HOW DO YOU TRUST IT? AGAIN, TRUST DEFINITION COULD BE MANY BUT IN SCIENCE TRUST IS REPRODUCIBILITY. THIS IS RECENT WORK WITH AN INSTITUTE AND A STUDENT AT M.I.T. THAT COMPARES PAPERS FOR HEALTH AS COMPARED TO OTHER MORE MATURE DOMAIN OF MACHINE INTELLIGENCE SPECIFICALLY NATURAL LANGUAGE PROCESSING AND COMPUTER VISION AND WE SAW MACHINE LEARNING FOR HEALTH SCORES POORLY COMPARED TO OTHER DOMAINS IN PRODUCIBILITY AND CODABILITY AND SPECIFICALLY TESTING THE MODEL ON MULTIPLE DATA SET IS THE MOST CONCERNING ONE. TRUSTING MODEL AN EXAMPLE I WON'T TALK ABOUT THAT BECAUSE I'M SURE EVERY PRESENTATION WILL COVER THAT TODAY SO I WON'T TELL YOU HOW TO STRATEGICALLY 3-D PRINT A TURTLE BUT I CAN TELL YOU THE EXAMPLES ARE BEING DEVELOPED IN THE MEDICAL DOMAIN AND WE WERE ABLE TO SHOW YOU CAN TAKE A STATE OF THE ART DEEP LEARNING CLASSIFIER FOR ARRHYTHMIA VERSUS NORMAL AND ADD A LITTLE BIT OF STRATEGY TO FOOL IT INTO BELIEVING ONE CLASS IS THE OTHER ONE. SO WHAT DOES IT MEAN TO TRUST? IT MEANS TO BE ABLE TO PROVE YOUR MODEL IS ROBUST TO ADVERSARIAL EXAMPLE AND PROVING SAY HARD THING TO DO -- IS A HARD THING TO DO AS WE'LL TALK IN THE PANEL AND THE LAST PART IN PUTTING UP THE SLIDES FOR YOU IS HOW DO YOU TRUST THE MODEL OUTPUT. IT'S BEEN TRAINED AND BUILT AND IT'S RUNNING IN THE REAL WORLD. THIS IS A HOT ACTIVE RESEARCH IN MACHINE LEARNING COMMUNITY AND I REALLY SUGGEST YOU CHECK OUT THE EXCELLENT TUTORIAL THAT DEVILVES INTO THIS AND THE GENERAL IDEA IS TO HAVE AN ADDITIONAL SYSTEM THAT CHECKS THE DATA BEING MADE A PREDICTION ON BY THE SYSTEM AND TRYING TO UNDERSTAND HOW CLOSE IT RELATES TO THE TRAINING DATA IS THE DATA POINT THE SYSTEM IS TRYING TO MAKE A PREDICTION ON IS SIMILAR TO SOMETHING THE MACHINE HAS SEEN IN THE PAST, IF SO IT COULD BE CREDIBLE AND THEY GIVE A CREDIBILITY SCORE AND TRUST SCORE OTHERWISE THEY DON'T. FINALLY, I WANT TO CONCLUDE WITH A THOUGHT, EVEN IF YOU TRUST ALL THE STEPS OF THE M.I. PIPELINE IT'S STILL A LITTLE BIT OF A TINY BLACK BOX WITHIN A LONGER COMPLEX SYSTEM OF A REAL WORLD LEARNING SYSTEM AND THIS IS FROM GOOGLE AND IN ORDER TO TRUST THE SYSTEM WE HAVE TO TRUST THE COMPLEX SYSTEM END TO END. IT WOULD BE HELPFUL TO GET INPUT FROM FOLKS THAT HAVE BUILT THEM IN OTHER DOMAINS IN NUCLEAR PLANT AND AUTOMOTIVE AND WE STEP ON MAN MADE DEVICES EVERY DAY IN THE SKY AND TRUST IT AND THERE'S COMPLEX ENGINEERING AND RELIABLE ENGINEERING THAT KNOWS HOW TO DEAL WITH RISK ASSESSMENT AND FOLD DETECTION AND PREVENTION AND MAINTENANCE AND WE SHOULD LEARN FROM THOSE FIELDS AS WELL. THANK YOU. >> NEXT UP IS MICHELLE CLARK. I GUESS WE TAKE QUESTIONS AT THE END? >> YEAH. >> ALL RIGHT. HELLO, EVERYBODY. SO I AM I STATISTICAL SCIENCE FOR THE RADY CHILDREN'S INSTITUTE FOR GENOMIC MEDICINE AND BEEN INVOLVED IN EVALUATING MACHINE INTELLIGENCE TOOLS FOR DIAGNOSING DISEASES IN CRITICALLY ILL INFANTS. THESE ARE TOOLS WE HAVEN'T DEVELOPED OURSELVES BUT PROVIDED BY EXTERNAL COLLABORATORS. I WANT TO SHARE WHAT WE'VE LEARNED SO FAR USING THESE TOOLS AND THE MOTIVATION BEHIND USING THEM AND WHAT WE FEEL IS NEEDED TO ESTABLISH TRUST. JUST FOR SOME BACKGROUND, GENETIC DISEASES ARE THE LEADING CAUSE OF INFANT MORTALITY IN THE UNITED STATES. DISEASE PROGRESSION IS REALLY RAPID IN INFANTS SO THE DIAGNOSIS NEEDS TO BE EQUALLY FAST TO AVOID MORBIDITY AND MORTALITY. APPROXIMATELY 55% OF INFANT UNDER THE AGE OF 1 IN THE INTENSIVE CARE UNIT HAVE A DISEASE OF UNKNOWN ORIGIN AND GETTING TREATMENT MODIFICATION AND THE DIAGNOSIS CAN TAKE WEEKS TO MONTHS LEADING TO OUTCOMES SUCH AS DELAYED DIAGNOSIS, UNNECESSARY MORBIDITY AND DELAYED PALLIATIVE CARE AND UNNECESSARY COST. WE KNOW THE STANDARD OF CARE DOES NOT WORK WELL FOR CHILDREN WIN GENETIC DISEASES. APPROXIMATELY 60% OF INFANT IN NEONATAL AND PEDIATRIC INTENSIVE CARE UNITS ARE EITHER MISDIAGNOSED OR MISTREATED. WE'RE USING SEQUENCES ON THE PATIENTS TO GET A DIAGNOSIS WITHIN TWO TO SEVEN DAYS. CLINICAL STUDIES ARE STARTING TO SUBSTANTIATE WHOLE GENOME SEQUENCING. YOU CAN SEE THE STATS ON THE RIGHT BUT IN OUR HOSPITAL WE FIND 1 IN 3 PATIENTS A GENETIC DISEASE DIAGNOSIS AND 1 IN 4 RECEIVE A CHANGE IN CARE AND 1 IN 5 EXPERIENCE IMPROVED OUTCOMES. BUT THERE ARE BARRIERS TO ADOPTION. I HAVE A FEW BARRIERS LISTED HERE. WE WANT TO MAKE THIS TECHNOLOGY AVAILABLE TO ALL HOSPITALS AND INFANT. THE APPROXIMATELY 80,000 CHILDREN A YEAR WHO COULD BENEFIT IN THE UNITED STATES. I'LL FOCUS ON THE TOP ONE WHICH IS UP UNTIL RECENTLY THE MESSAGE FOR ULTRA RAPID DIAGNOSIS HAD IMMENSE CAPITAL AND LABOR INTENSITY. THEY REQUIRE EXPERTS OF WHICH THERE ARE A SHORTAGE INCLUDING MEDICAL GENETICISTS AND COUNSELLORS AND DIRECTORS. WE FEEL WE HAVE BEEN ABLE TO KNOT KNOCK DOWN THIS BARRIER USING MACHINE INTELLIGENCE. THE SOLUTION HAS BEEN AN AUTOMATED DIAGNOSTIC WITH A MULTI-DISCIPLINARY TEAM IN FEBRUARY WE WERE ABLE TO SHORTEN THE TIME TO DIAGNOSIS AND REMOVE THE NEED FOR HUMAN ANALYSIS AND INTERVENTION. THIS ALLOWED US TO RECEIVE THE WORLD RECORD FOR THE FASTEST GENETIC DIAGNOSIS IN 19 1/2 HOURS. HOW DID WE DO THAT? WE AUTOMATED TWO OF THE LABOR INTENSIVE STEPS. THE FIRST OF WHICH IS DEEP PHENOTYPING. USUALLY AN EXPERT HAS TO GO THROUGH THE MEDICAL RECORD AND HAND PICK SYMPTOMS AND CHARACTERISTICS LIKELY RELEVANT TO THE CHILD'S DISEASE. THIS CAN TAKE ANYWHERE FROM 15 MINUTES TO TWO HOURS BUT USING NATURAL LANGUAGE PRFTSING WE'RE ABLE TO -- PROCESSING WE'RE ABLE TO GET 40 FOLD MORE IN 20 MINUTES AND THE LABOR INTENSIVE PIECE COMES AFTER SEQUENCING WHERE AN EXPERT HAS TO GO THROUGH VARIANTS AND IDENTIFY, ASSESS AND INTERPRET THEM TO SEE WHICH ARE THE MOST LIKELY CAUSING OF THE CHILD'S CONDITION. TO SPEED UP THIS ANALYSIS WE USED A MACHINE LEARNING SOFTWARE CALLED MOON. MOON IS ABLE TO PAIR THE PHENOTYPING WITH THE SEQUENCING DATA WITH NATURAL LANGUAGE AND PRODUCE JUST ONE VARIANT THAT'S LIKELY DISEASE CAUSING AND THIS IS OUT OF THE 5 MILLION VARIANTS ASSESSED. THE PROCESS MANUALLY TAKES ABOUT 10 HOURS. WITH THE MACHINE LEARNING PLATFORM IT TAKES JUST FIVE MINUTES. SO WE ARE A SMALL INSTITUTE. WE'RE JUST FOUR YEARS OLD SO WE DON'T HAVE A LOT OF DATA SO WE JUST STARTED DOING A NUMBER OF STUDIES EVALUATING THIS DIAGNOSTIC PLATFORM. THE FIRST OF WHICH IS IN A RETROSPECTIVE COHORT WITH 95 CHILDREN, WITH 95 GENETIC DISEASES. -- 97 GENETIC DISEASES AFTER RECEIVING GENOME SEQUENCING. NO INCIDENTAL FINDINGS WERE FOUND IN THE STUDY AND WE GOT 97% RECALL WITH THIS AUTOMATED PLATFORM. THESE METHODS WERE SUBSEQUENTLY APPLIED TO SEVEN PROSPECTIVE PATIENTS IN PARALLEL WITH OUR METHODS AND THE TIME SAVINGS WAS APPROXIMATELY 22 HOURS. ALL THREE DIAGNOSES WERE MADE BY BOTH METHODS AND THE AUTOMATED METHOD HAD NO FALSE POSITIVES. IN EVALUATING THIS PLATFORM RETROSPECTIVELY WE IDENTIFIED THERE WAS POTENTIAL GAINS IN USING IT FOR RE-ANALYSIS PERIODICALLY SO IN A STUDY OF 48 UNDIAGNOSED CHILDREN, TWO DIAGNOSES WERE MADE UPON RE-ANALYSIS. FOUR ADDITIONAL CASES WERE FLAGGED AS POSSIBLE DIAGNOSES WE'LL COME BACK TO DURING PERIODIC RE-ANALYSIS AND WHAT'S MOST NOTEWORTHY IS AN UNTRAINED ANALYST AND A SUMMER INTERN AND THE SPECIFICITY WAS 83% AND THE SENSITIVITY WAS 76%. SO LAST BUT NOT LEAST, WE'VE JUST STARTED WRAPPING UP A STUDY OF 50 PATIENTS. 16 OF THOSE RECEIVED A DIAGNOSIS WITH A STANDARD DIAGNOSTIC WORK FLOW AND ALL 16 WERE ALSO CORRECTLY DIAGNOSED WITH OUR AUTOMATED ANALYSIS. WE HAVE THIS PLATFORM AND WE'RE A SMALL INSTITUTE SO THE EVALUATION HAS BEEN LIMITED BUT IN TALKING TO OUR CLINICAL LAB DIRECTORS ABOUT MOVING THIS TECHNOLOGY FROM RESEARCH TO THE BEDSIDE, THERE IS HESITATION. ONE OF THE BIGGEST HESITATION IS THE TOOLS UNDERGO RAPID UPDATES AND IT GOES AGAINST HOW CLINICAL LAB DIRECTORS WERE TRAINED AND TRAINED TO HAVE HIGH VALIDATION STANDARDS SO THEY NEED SUFFICIENT WARNING PRIOR TO UPDATES AND REQUESTED INCREASED TRANSPARENCY. I KNOW THERE'S GOING TO BE A SESSION ON THIS LATER BUT I WANT TO TOUCH UPON IT BECAUSE IT IS INTERTWINED WITH TRUSTWORTHY SO MOON OUR INTERPRETATION SOFTWARE WAS LARGELY A BLACK BOX AND IT'S A PROPRIETARY SOFTWARE AND HAVE NO SAY IN WHAT'S UNDER THE HOOD AND RESPONDED TO OUR REQUEST BEEN ANNOTATING THE VARIANTS WITH A DATABASE AND YOU CAN SEE HOW MANY PUBLICATIONS MENTION THE SPECIFIC VARIANT AND YOU CAN CLICK ON IT AND REVIEW THE LITERATURE AND INCLUDE CONSERVATION SCORES WHERE IT WAS ONLY AVAILABLE IN THE WEB INTERFACE BUT YOU CAN NOW DOWNLOAD A SPREAD SHEET WITH THE INFORMATION AND ALL THE RARE VARIANTS REGARDLESS OF THEIR SCORES FOR FUTURE RESEARCH. AND FINALLY, THEY ALSO ALLOW FOR CLINICAL REPORTS TO BE CREATED. IT'S AUTOPOPULATED WITH THE VARIANT INFORMATION SO THIS AGAIN IS SAVING OUR LAB DIRECTORS TIME AND MAKING THEM CONFIDENT WHEN SCORING VARIANTS BASED ON THE GUIDELINES. SO THERE'S A FEW MORE ISSUES. THERE'S MANY MORE ISSUES BUT A FEW MORE THAT HAVE COME UP WITH OUR CLINICAL LAB DIRECTORS, WE KNOW WE'VE SEEN THERE'S HIGH SENSITIVITY BUT UNSURE ABOUT THE SPECIFICITY. WE NEED LARGER STUDIES FROM EXTERNAL PARTNERS WITH HUNDREDS OF THOUSANDS OF CASES AND THEY'RE ALSO USED TO USING GENOME IN A BOTTLE FOR CLINICAL VALIDATION OF GENOME SEQUENCING SO THEY'RE PUSHING FOR A PUBLICLY AVAILABLE BENCHMARK TO VALIDATE THE METHODS AFTER EVERY UPDATE. IN CONCLUSION THIS DIAGNOSTIC SYSTEM IS HANDS FREE BUT SUPERVISED AT EVERY STEP BY MOLECULAR GENETICISTS AND CLINICAL LAB DIRECTORS AND WILL BE FOR THE FORESEEABLE FUTURE AND IN OUR PAPER WE USE THE AFA AFALL -- ANALOGY OF THE AUTOPILOT THERE'S TWO PILOTS EMPLOYED IN EVERY COCKPIT OF EVERY COMMERCIAL AIRPLANE LIKEWISE THERE'LL ALWAYS BE A NEED FOR A SKILLED TEAM TO LOOK AT THE M.I. WORK FLOWS TO LOOK AT DIFFERENT CASES, DO MANUAL CURATION OF VARIANTS AND CLINICAL REPORT VARIATION. THIS IS THE HUGE TEAM OF PEOPLE WHO HELP CREATE, DEPLOY AND EVALUATE THIS SYSTEM AND HOPEFULLY WE'LL BE ABLE TO GET THE DOCTORS THE INFORMATION THEY NEED SO THEY CAN GET THEIR BABIES AN ANSWER AS SOON AS POSSIBLE AND AVOID BAD OUTCOMES. THANK YOU. >> HI, I'M BRIAN ALPER AND HAPPY TO TALK IN GUIDING HEALTH DECISIONS. I'M A PHYSICIAN AND I CREATED DINO MED WHICH NOW EMPLOYEDS HUNDREDS OF PEOPLE TO HAND CURATE MEDICAL EVIDENCE AND GUIDANCE AND PROVIDE THIS FOR USE AT THE POINT OF CARE IN CLINICAL MEDICINE AFFECTING HEALTH CARE DECISIONS FOR MILLIONS OF PEOPLE. I'M ALSO LEAD PROJECT EXTENDING FIRE FAST HEALTH CARE INTEROPERABILITY RESOURCES TO THE WORLD OF EVIDENCE-BASED MEDICINE AND MAKING EVIDENCE AND STATISTICS COMPUTABLE SO VERY MUCH HAVE TO PAY ATTENTION TO HOW WE TRUST THE DATA FOR HUMANS OR COMPUTERS IN LOOKING AT THIS. SO WHY ARE WE DOING ALL OF THIS? OUR GOAL IN MEDICINE IS TO PROVIDE THE BEST CARE. PROVIDE THE BEST INFORMATION TO GUIDE HEALTH CARE DECISIONS, IMPROVE THE HEALTH OUTCOMES AND NEED TO SEPARATE THE TRUE MEDICAL KNOWLEDGE FROM THE FOLKLORE AND BELIEFS WE HAVE. EVIDENCE-BASED MEDICINE IS THE INTEGRATION OF BEST RESEARCH EVIDENCE WITH CLINICAL EXPERTISE AND PATIENT VALUES. THE BEST RESEARCH EVIDENCE ON ITS OWN DOES NOT DO IT BUT WE NEED TO HAVE THE BEST RESEARCH EVIDENCE. IF YOU COULD SIT THROUGH ALL OF OUR PEER REVIEWED ESTEEMED DOCUMENTS OF GUIDELINES AND REVIEW ARTICLES AND RESEARCH ARTICLES AND SYSTEMATIC REVIEWS AND SOMEHOW PUT THAT TOGETHER AND THE GET THAT SINGLE SYNTHESIZED ANSWER YOU MAY DISCOVER NEW INSIGHTS. YOU MAY PROVIT THE MOST USEFUL INFORMATION OR MAY AMPLIFY FALSE SIGNALS MISINTERPRETED OR MISREPRESENTED REPEATEDLY ACROSS OUR MEDICAL LITERATURE. THE GREATEST ENEMY OF KNOWLEDGE IS NOT IGNORANCE, IT'S THE ILLUSION OF KNOWLEDGE OR PERHAPS BETTER SAID BY MARK TWAIN, IT AIN'T WHAT YOU DON'T KNOW THAT GETS YOU INTO TROUBLE, IT'S WHAT YOU KNOW FOR SURE THAT JUST AIN'T SO. SO THIS IS EVIDENCE-BASED MEDICINE. WE WANT TO INCORPORATE BEST RESEARCH EVIDENCE WITH CLINICAL EXPERTISE AND PATIENT VALUES BUT MISUNDERSTOOD BEST RESEARCH EVIDENCE CAN CAUSE SERIOUS HARM. FOR 30 YEARS WE WOULD RECOMMEND HORMONE REPLACEMENT THERAPY AFTER MENOPAUSE FOR HEART DISEASE AND FOUND IT INCREASED BREAST CANCER AND STOPPED DOING IT. WE USED TO USE ANTI-ARRHYTHMICS TO PREVENT PROBLEMS AFTER HEART ATTACK IF YOU HAD PVCs OR EXTRA SQUIGGLES IN THE EKG, THEN WE LEARN THE SIDE EFFECTS WERE WORSE AND KILLED MORE PEOPLE THAN WHAT WE THOUGHT WE WERE PREVENTING. THIS HAS BEEN REPEATED IN MANY SITUATIONS WHERE WHAT WE THOUGHT WE KNEW AND MADE BEST GUESS JUDGMENTS WERE NOT THE BEST EVIDENCE AND UNDERSTANDING OF IT. SO WHAT IS BEST RESEARCH EVIDENCE? IT CAN ONLY BE BEST TO KNOW WHICH EVIDENCE IS BEST. EVERY DAY NEW RESEARCH MAY BE THE BEST SO IT HAS TO BE COMPREHENSIVE AND CURRENT. AND WE NEED SYNTHESIS NOT JUST THE RESULT FROM ONE STUDY BUT LOOKING AT ALL STUDIES THAT ARE RELEVANT. SCIENCE REQUIRES REPLICATION OF RESULTS FOR CONFIRMATION BUT OUT OF RESEARCH PUBLICATIONS IN HEALTH CARE, FEW WERE REPLICATED. MANY THINGS HAPPENED BUT IN HEALTH CARE WE OFTEN MAKE OUR DECISIONS AND CHANGE OUR PRACTICE BASED ON SINGLE STUDIES. SO IT ALSO NEEDS TO BE VALID. CRITICAL APPRAISAL DEVELOPED THE POTENTIAL FOR BIAS AND WHAT IS PUBLISHED IS OFTEN WRONG, MISLEADING, MISINTERPRETED OR INCOMPLETE. WE ACCEPT ANDCYTE CONCLUSIONS OF -- CITE CONCLUSIONS AND REPEAT ERRORS AND CITE WHAT IS PUBLISHED INSTEAD OF GOING BACK TO THE ORIGINAL PUBLICATION WHERE THE DATA CAME FROM. WE SELECTIVELY SUMMARIZE AND CITE BASED ON BIASES OR FAMILIARITY AND WE FULL TEXT ESPECIALLY IF IT'S NOT FREELY AVAILABLE AND INTERPRET CHANGES IN SURROGATE MARKERS AS IF THAT MEANS CHANGES IN CLINICAL OUTCOMES. THE REASONS FOR THOSE LIFE-CHANGING DIFFERENCES IN WHAT WE THOUGHT WE KNEW WAS THE DIFFERENCE BETWEEN SURROGATE OUTCOMES AND WHAT ACTUALLY WAS THE EVIDENCE ON CLINICAL EFFECTS. SO BEST RESEARCH EVIDENCE NOT ONLY NEEDS TO BE COMPREHENSIVE, CURRENT, SYNTHESIZED VALID WITH CRITICAL APPRAISAL, IT NEEDS TO BE SYSTEMATIC. THE EVALUATION OF THE EVIDENCE DONE BY PROTOCOLS TO REDUCE THE BIAS OF THE AUTHORS, THE INVESTIGATORS, THE EDITORS, WHOEVER IS PUTTING THIS TOGETHER OR EVALUATING IT. SO IN THE WORLD OF EVIDENCE-BASED MEDICINE WE HAVE A SOLUTION CALLED THE SYSTEMATIC REVIEW. WE CAN THINK OF THE HIERARCHY OF BEST TYPES OF EVIDENCE FOR A SPECIFIC QUESTION. THIS IS AN EXAMPLE FOR AN INTERVENTION QUESTION, CAUSE AND EFFECT AND THEN A SYSTEMATIC REVIEW OF THE BEST EVIDENCE FOR IT WHERE YOU SYSTEMATICALLY SEARCH FOR THE EVIDENCE, SELECT THE BEST EVIDENCE, CRITICALLY APPRAISE IT AND PUT IT TOGETHER. BUT WE DO NOT HAVE SYSTEMATIC REVIEWS FOR MOST CLINICAL QUESTIONS AND THE SYSTEMATIC REVIEWS WE HAVE ARE OFTEN OUTDATED BECAUSE NEW EVIDENCE GETS PUBLISHED AND IT'S SO MUCH WORK TO CREATE THE SYSTEMATIC REVIEWS. SO FOR DYNA MED IT TAKES MUCH OF THE METHODS OF SYSTEMATIC REVIEWS AND START WITH SYSTEMATIC SEARCH ACROSS THE MOST VALID SOURCES BUT WHEN SELECTING THE BEST AVAILABLE EVIDENCE INSTEAD OF ASKING DOES IT ANSWER ONE QUESTION AT A TIME, WE SAY DOES IT ANSWER A RELEVANT QUESTION IN THIS CONTENT RELEVANT FOR CLINICAL PRACTICE THEN WE CRITICALLY APPRAISE THE RESEARCH OBJECTIVE AND REPORT THE FINDINGS, SYNTHESIZE IT AND PUT IT TOGETHER, BASE THE CONCLUSIONS ON THAT EVIDENCE AND YOU'D BE AMAZED HOW OFTEN PEOPLE DO EVIDENCE-BASED WORK AND MAKE FINAL ANSWERS ON OTHER THINGS THAN THE EVIDENCE WORK AND YOU CAN SEE WHEN THEY DON'T CONNECT THE DOTS. A SYSTEMATIC REVIEW GETS UPDATED IT DEPENDS. IT'S AN ONGOING DAILY ACTIVITY. TO UNDERSTAND THIS CONCEPT OF CRITICAL APPRAISAL WHAT ARE WE DOING TO MANUAL CURATION? IT'S DIFFERENT FOR DIFFERENT TYPES OF CONCLUSION. THESE ARE THE CRITERIA FOR AN INTERVENTIONAL CONCLUSION. A CONCLUSION THAT DOING SOMETHING LIKE A TREATMENT WILL LEAD TO A CHANGE IN OUTCOMES. WE HAVE TO HAVE FULL TEXT REPORT. WE CAN'T JUST ACCEPT ABSTRACT OR MEETING REPORTS. WE LOOK FOR CLINICAL OUTCOMES IN THIS CONTEXT NOT SURROGATE OUTCOMES AND EVERYTHING WITH THE STUDY, THE POPULATION, THE INTERVENTION, THE COMPARITOR, THE OUTCOME STUDIED NEEDS TO BE REPRESENTATIVE OF REAL PRACTICE NOT JUST THE RESEARCH ENVIRONMENT AND THAT'S A CONCEPT OF DIRECTNESS OR RELEVANCY THAT GOES INTO THE TRUSTWORTHINESS OF RESULTS. WE LOOK FOR TRUE RANDOMIZATION, BLINDING, IF POSSIBLE, ACROSS THE DIFFERENT PARTIES THAT CAN EFFECT THINGS AND FOLLOW-UP AND ACCOUNTING FOR DROP OUTS AND TO THE MEANINGFUL AND LESS MEANINGFUL RESULTS. WE HAVE DETAILED CRITERIA BASED ON THE TYPE OF RANDOMIZATION IF THERE WAS EARLY TRIAL DETERMINATION AND OTHER FACTORS THAT MAY BIAS THE CONSIDERATIONS. SO HOW CAN A MACHINE LEARN THIS AND HAVE MACHINE NATURAL GAS IF IT TAKES -- INTELLIGENCE FOR THE EFFECTS OF TREATMENTS? IF YOU USE THE PUBLISHED EVIDENCE AS-IS, WE KNOW GARBAGE IN, GARBAGE OUT AND WE AMPLIFIED THE PROBLEM AND THE MANY ARTICLES THAT CITED THE SAME MISTAKE YOU SEE MANY TIMES AND INTERPRET IT AS A STRONGER SIGNAL. WE CAN LIMIT OUR REFERENCE STANDARD TO THE PROVEN. WE'LL HAVE AN INADEQUATE SOURCE OF TRUTH AND OBSERVATION OF SAMPLES. WE CAN LIMIT THE REFERENC STANDARD TO THE MOST TRUSTWORTHY SOURCES. WELL, WHO DECIDES WHAT THOSE ARE AND WHAT'S THE CUT OFF FOR WHAT IS TRUSTWORTHY AND DO WE STILL END UP WITH INADEQUATE SOURCE OF TRUTHS BECAUSE THERE'S LESS TRUSTWORTHY DATA OR WILL IT TAKE A LARGE AMOUNT OF HUMAN EFFORT TO TRAIN THE MACHINE? AND WHAT IS IT THE MACHINE NEEDS TO LEARN. WE CANNOT ANSWER THE QUESTION WHAT ARE THE EFFECTS OF DIFFERENT MANAGEMENT OPTIONS BECAUSE WE DON'T HAVE A DIRECT ANSWER FOR THAT QUESTION. WHAT WE CAN COMMUNICATE PRECISELY ARE THE LIKELY EFFECTS AND BENEFITS AND HARMS OF THE DIFFERENT MANAGEMENT OPTIONS BUT HOW DO WE COMMUNICATE OUR CERTAINTY OR OUR CONFIDENCE IN THAT LIKELIHOOD PREDICTION OR EFFECT ESTIMATES AND HOW WE DO THAT IS CRITICAL FOR PUTTING ALL THIS TOGETHER. OTHERWISE WE CAN HAVE A MISMATCH WHERE THE LARGE EFFECT AND UNLIKELY TO OCCUR CAN GET CONFUSED AS IMPORTANT. THIS IS A SMALL SLICE OF THE EVIDENCE-BASED RESOURCE WE'RE BUILDING INTO FIRE WHICH TRANS FORMS THESE CONCEPTS INTO COMPUTABLE EXPRESSIONS THE WAY TO TAKE THAT KNOWLEDGE AND PUT IT INTO THE SIM BOWLS FOR MORE ALGORITHM ALGORITHMIC INTERPRETATION AND WHAT IS THE POPULATION GROUP IT WAS STUDIED IN AND WHAT IT'S INTENDED TO BE ABOUT AND CODE HOW CLOSE OR FAR HOW THAT MA MATCH OF THE TWO GROUPS ARE AND LOOK AT THE EXPOSURES AND OUTCOMES. STATISTIC HAS A LOT OF VARIABLES OR ELEMENTS FOR HOW YOU CODE THAT AND THEN YOU CAN SEE EXTENSIVE METHODS FOR CODING THE CERTAINTY OF THE CONCEPTS. IT'S IMPORTANT, IT'S NOT JUST MATH, IT'S ALSO CODABLE CONCEPTS ABOUT OUR CERTAINTY IN THE MATHEMATIC PREDICTION. THANK YOU. >> THANK YOU FOR HAVING ME HERE AND LUCA AND I WERE JOKING I HAVE A FUNNY LOOKING BADGE BECAUSE OF THE BREAKDOWN OF THE MACHINE AND WE HAVE A LONG WAY TO GO IN MACHINE INTELLIGENCE. THAT WAS HIS QUOTE NOT MINE. SO WHAT I WILL TRY TO DO IS WALK YOU THROUGH SOME IDEAS I HAVE AROUND HOW CAN WE ENGENDER TRUST IN THE USE OF ALGORITHMIC AGENT AND I CAME UP WITH THIS CUTE LOOKING TITLE RECOGNZED IN THE EAST COAST BUT ON THE WEST COAST I STILL GET BLANK STARES. TO FRAME OUR THINKING, IMAGINE A PERSON'S JOURNEY TIME FROM LEFT TO RIGHT. THE RED DOTS IS WHEN SOMETHING HAPPENS ON THAT PERSON'S JOURNEY AND DEPENDING ON WHO YOU ARE, YOU MIGHT HAVE ACCESS TO THEIR MEDICAL CLAIMS, CODES, MEDICATIONS, PROCEDURES DONE TO THEM, LAB TEST RESULTS OR DIAGNOSTIC SYSTEMS OR PROCESSED TEXT CONTENT FROM THEIR CLINICAL DOCUMENTS BEDSIDE MONITORS IF YOU'RE IN THE ICU, WEARABLES, GENE EXPRESSION DATA, PHONE USAGE, BROWSING HISTORY, SOCIAL MEDIA, AUDIO RECORDINGS OF CONVERSATIONS. THESE ARE TRACKS OF INFORMATION THAT ALL OF WHICH I HAVE PERSONALLY USED IN MY OWN RESEARCH AND THEN THERE'S OTHER STUFF AND YOU COULD HAVE 50 OTHER ROWS HERE. THE TWO THINGS THAT REMAIN TRUE ABOUT THIS IS NO ONE HAS ALL MODALITIES. AND THE OWNER OF A GIVEN MODALITY DOESN'T HAVE IT COMPLETE FOR ALL HUMANS TO HAVE DATA ON. THESE TWO FACTS REMAIN REGARDLESS OF DATA SOURCE, COUNTRY OR HEALTH SYSTEM. SO THE CHALLENGES WE HAVE THIS PATIENT OBJECT SPARSE AND WE HAVE IT ON MILLIONS OF HUMANS. SO MIN WORK DEPENDING ON WHICH TRACK WE'RE WORKING WITH, I END UP WORKING WITH ABOUT 200 MILLION PATIENT OBJECTS SOMETIMES TO 28,000 PATIENT OBJECTS DEPENDING ON HOW DENSE I WANT THE DATA AND THE MODALITY. SO HOLD ON TO THIS IMAGE. THEN, BY AND LARGE WHAT WE DO WITH THE DATA SETS OR VARIATION PORTIONS OF THEM IS TO SOME SORT OF RISK ASSESSMENT AND IF IT'S ABOVE ATHRESHOLD WE HAVE THEM APPLYING TO A CLINICAL SITUATION AND NO ONE HAS TESTED ALL OF THEM IN THE EXACT SAME SITUATION SO YOU'RE FACED WITH TAKING EVIDENCE PRODUCED IN A SLIGHTLY DIFFERENT ENVIRONMENT MAYBE SHAKY, MAYBE NOT AND THEN RECONCILE EVERYTHING IN YOUR HEAD. ON THE RISK ASSESSMENT SIDE, IT'S USUALLY A DECISION ABOUT WHO TO TREAT. SHOULD I TAKE ACTION. AND IT'S FROM MATHEMATICAL STANDPOINT IT'S OFTEN CLASSIFICATION AND LESS PREDICTION PROBLEMS. YOU MAY BE SURPRISED BUT BETWEEN 1965 TO 2015, SO FOUR YEARS AGO, THERE'S 250,000 RE-STRATIFICATIONS PUBLISHED IN MED LINE. 250,000. ASSUMING ONLY 10% ARE NEW AND 90% WE'RE RE-EVALUATING SOMEONE ELSE'S WORK, WHICH I FIND SHOCKING, BUT EVEN THEN, A LOWER BOUND ON THE NUMBER OF NEW RISK STRATIFICATION SCORES WE ALREADY HAVE IS 25,000 AND THAT'S AS OF FOUR YEARS AGO. FOR THE PRACTICING CLINICIANS, HOW MANY DO YOU ACTUALLY USE? SO WE DO A LOT ON THIS SIDE. AND THEN ON THE ACTION SIDE WE JUST HEARD AN EXCELLENT PRESENTATION ON WE DO THINGS THAT 20 YEARS LATER LIKE MEDICAL REVERSAL IS A THING. AND THIS IS ABOUT DECIDE THOUG TREAT. -- DECIDING HOW TO TREAT. MACHINE AND ALGORITHMS CAN HELP ON BOTH SIDES. ACTUALLY WHEN WE LOOK AT THIS IN THE PRECISION HEALTH, PRECISION MEDICINE WE GET GENETIC MARKERS, LESS OFTEN GEOGRAPHIC AND SOCIOECONOMIC STATUS WE MAY LOOK AT RECORD AND BEHAVIOR DATA AND SO ON AND AND THE ACTION SIDE AGAIN THE SAME THING BUT OFTEN WE'LL START LOOKING AT SIMILAR PATIENTS AND WE MAY HAVE SOME MECHANISTIC UNDERSTANDING OF DISEASE AND WHAT'S AVAILABLE AND HOW MUCH TIME WE HAVE ON OUR HAND TO GET THE WORK DONE. THESE ARE PRACTICAL CONSTRAINTS. SO WHEN WE THINK ABOUT MODELS OR WHEN WE THINK ABOUT A MACHINE LEARNING ENTITY THAT IS GOING TO BE APPLIED IN HEALTH CARE AN IDEALISTIC VERSION IS WE HAVE SOME FEATURE OR OUTCOME OR TRAINING DATA AND THEN THERE'S A LEARNING ALGORITHM THAT GIVES US THIS THING WE CALL A MODEL WHICH IS A FUNCTION. WE USE THIS ON NEW DATA. THIS IS THE WORK FLOW THAT GIVEN THE MODEL NOW GIVES ME PREDICTION OR SOME CLASSIFICATION. THE BELIEF IS THERE IS UNDERLYING DATA PROCESS THE MODEL IS CAPTURING AND THAT ASSUMPTION USUALLY GOES UNTEST. -- UNTEST. IMAGINE A WORK FLOW IN THIS CASE BECAUSE ABOUT FIGURING OUT WHO WOULD BENEFIT FROM PALLIATIVE CARE AND WE HAVE A WHOLE BUNCH OF RECORDS. I CAN HAVE AN ALGORITHM READ THE RECORDS AND FLAG IT BY A PHYSICIAN AND THE PERSON MAY BENEFIT AND THE PERSON THEN REACHES OUT TO THE TREATING DOCTORS SAYING I MAY BE ABLE TO HELP THIS PERSON, WHAT DO YOU THINK? THIS IS A CARTOONISH WORK FLOW. IN REAL LIFE IT LOOKS LIKE THIS. IT'S A REAL WORK FLOW FROM STANFORD HOSPITAL FOR GOALS OF CARE CONVERSATIONS WITH PATIENTS. IT HAS 21 STEPS SPANS, THREE LEVELS OF HIERARCHY IN THE WORK CHART AND HAS SEVEN HANDOFFS AND TAKES 48 HOURS. I CAN TAKE THIS PIECE AND THAT'S THE PART WHICH IS THIS FX. HOW DO I KNOW I CAN TRUST IT? SO TRUSTWORTHINESS WHEN WE COME TO THAT THE QUESTION IN MY MIND IS IT OF THE MODEL OR WORK FLOW AROUND IT OR BOTH? WE TALKED ABOUT TRUSTWORTHINESS OF THE DATA. I'M ASSUMING WE SOLVED THAT PROBLEM BUT WE NEVER MAKE THE DISTINCTION OF THE MONDAYEL OR -- MODEL OR WORK FLOW AND TRUST IS IT DOES WHAT IT CLAIMS TO DO AND EARNED OVER TIME. TYPICALLY IN THE MACHINE LEARNING COMMUNITY WE TALK ABOUT TRUSTWORTHINESS THERE'S INTERPRETABILITY AND EXPLAINABILITY. THIS STATE BILL HOW A MODEL GIVES YOU THE OUTPUT IT GIVES YOU AND EXPLAINABILITY IS ABOUT THE WHY AND THE ASSUMPTIONS ABOUT THE UNDERLYING DATA PROCESS. THIS CUTE LITTLE FUNCTION, GX. I'M HAVING MACHINE ISSUES TODAY. ALL RIGHT. SO HOW AND WHY. SO IN THE USE CASE I SHOWED ABOUT PREDICTING MORTALITY, NOW THE ACTUAL EFFECTS WAS 3 TO 12 MONTH MORTALITY BUT LET'S CONSIDER FOR -- 20 FOR OUR MORTALITY. INTERPRETABILITY IS A POOR DATA SET FOR TRUST PART OF A FAMOUS PAPER AND THE MODEL CAN TELL YOU EXACTLY HOW IT ARRIVED AT THE PROBABILITY OF DEATH IN 24 HOURS IT DOESN'T HELP ME DECIDE WHAT TO DO. EXPLAINABILITY IS A POOR DATA SET BECAUSE IN THAT CASE THE MODEL CAN'T EXPLAIN AND TELL ME THE REASON THE PERSON'S GOING DIE IS BECAUSE THEY HAVE MALIGNANT LUNG CANCER AND HAVE A THIS IN THEIR CHEST AND BED SORES AND THEY'RE 95. WHAT I NEED AT THAT POINT IS SOMETHING THAT HELPS ME STEP OUT OF THAT MIND SET AND SAYS IT'S TIME TO CALL THE FAMILY. SO IN THAT CASE, KNOWING THAT A MODEL'S PREDICTION HAS HELPED ME MAKE GOOD DECISION OR JUDGMENT CALLS IN THE PAST YEARS IS WHAT WE MEAN BY TRUSTWORTHINESS. THESE TWO THINGS ARE SURROGATES AND WE KIND OF GET LOST IN TALKING ABOUT THEM ABOUT WHAT WE REALLY WANT. SO BUILDING TRUSTWORTHY AND USEFUL MODELS AND USE CASE AND TECHNICAL FORMULATION WHERE SOME EFFORTS ARE SPENT ON THIS IN THE COMMUNITY AND THEN THE MODEL AND TECHNICAL VALIDATION WHERE 95% OF THE M.L. COMMUNITY SPENDS THEIR EFFORTS ON WHICH IS HOW TO GET THE BEST FUNCTION AND THEN THE UTILITY, DESIGN AND DEPLOYMENT ASSESSMENT WHERE NOT MUCH EFFORT RIGHT NOW WE ASSUME IF A MODEL EXISTS AND IT'S ACCURATE THERE'S NET BENEFIT AND IF THERE'S NET -- BENEFIT AND UNLESS YOU KNOW ABOUT IT IN ADVANCE THE MODEL WILL NOT REALIZE THE NET BENEFIT AND HAVING A RUNNING SYSTEM WHERE THE MODEL IS APPLIED THE WORK FLOW EXECUTES AND OVER TIME WE CONVINCE OURSELVES THE THING WORKS. THAT'S TRUSTWORTHINESS. I THINK THAT'S MY LAST SLIDE. ACKNOWLEDGEMENT TO MEMBERS OF MY TEAM AND FUNDING FROM A LOT OF INSTITUTIONS INCLUDING NCATS AND I'LL STOP HERE. THANK YOU. [APPLAUSE] >> WE'LL START THE PANEL DISCUSSION. SO THIS MEANS QUESTIONS FROM THE AUDIENCE AND REMOTE QUESTIONS, PLEASE. [OFF MIC] >> I SHARE MY FITBIT WITH MY WIFE AND SOMETIMES MY KIDS TAKE IT SO HOW DO WE KNOW WHAT WE ARE READING IS EXACT GENUINE PATIENT DATA. >> I GUESS THAT QUESTION A LOT AND IT'S A VALID QUESTION. I THINK THAT PROBLEM IS CALLED ATTRIBUTION IN THE LITERATURE IN DATA QUALITY AND FALLS IN THE ACCRUAL BOX I HAD THERE. THE SOLUTION IS REALLY -- THERE'S NOT REALLY A SOLUTION RIGHT NOW BUT MANY OF THE DEVICES BEING DEVELOED NOW HAVE SOME SORT OF BIOMETRIC AUTHENTICATION YOU KNOW A PERSON IS WEARING IT BY THEIR PHYSIOLOGICAL PATTERNS UNIQUE TO THEM TO IDENTIFY WHO IT IS. WE'RE BUILDING FOR THE FUTURE IN THAT PARTICULAR CASE. [OFF MIC] >> HAVE YOU USED MACHINE LEARNING TO HARVEST THE DATA IN THE PATIENT PROCESS ITSELF [INDISCERNIBLE]. >> IT IS POSSIBLE. YOU TEND TO I THINK IT'S MORE HELPFUL TO HAVE SOME SORT OF A SCORECARD LIKE HUMANS HAVE LOOKED AT THE DATA AND MADE SOME DETERMINATION ABOUT IT. IT'S TRUE IF YOU TRAIN AN END TO END MODEL MAYBE EVEN LIKE THE DATA CAN STILL GIVE A SOLUTION BUT IT DOESN'T REALLY TELL YOU WHETHER IT'S FOR OTHER PURPOSE YOU MAY WANT TO USE DOWN THE ROAD OR WHAT ARE THE UNDERLYING DATA DISTRIBUTION CHANGES AND KNOW WHAT TO DO WITH THAT. HOW DO WE MODERATE QUESTIONS FROM REMOTE? >> WE'LL TAKE SEPARATELY. >> OKAY. WITH THE UNDERCURRENT OF LUCA'S CONVERSATION WHICH THE COMPONENT IS A SMALL COG IN A BIGGER MACHINE AND YOU SHOWED THE SLIDE FROM GOOGLE THE UNIVERSE OF COMPLEXITY IS MUCH BIGGER THAN THE BLACK BOX OF MACHINE CODE AND THE WORK FLOW PROCESS OF MACHINES IS MUCH MORE COMPLICATED IN WHICH WE TRIED TO CAPTURE A SMALL PART WITH A MODEL AND THE VALIDATION SHOULD LOOK TO THE OVERALL COMPLEXITY AND MY QUESTION IS, DOES IT MAKE SENSE TO VALIDATE THE MODEL RATHER THAN VALIDATE THE ENTIRE WORK FLOW PROCESS. FOR ME IT SOUNDS LIKE THEY'RE TRYING TO VALIDATE THE SIDE MIRROR OF A CAR IN TERMS OF SAFETY OF DRIVING THE CAR. IT'S INTERETING BUT PROBABLY NOT WHERE WE SHOULD BE FOCUSSING. WE SHOULD BE FOCUSSING ON THE LONGER TERM IMPACT WHICH IS WHAT YOU WERE TRYING TO SAY AS THE DEFINITION OF THE TRUSTWORTHINESS. >> SO I LOVE THE ANALOGY AND USING THE SIDE MIRROR OF A CAR TO PROVE THE CAR IS SAFE I THINK IT'S RIGHT AND NECESSARY BUT NOT SUFFICIENT. LIKE YOU CAN HAVE A CAR IN PERFECTLY GOOD CONDITION AND DRIVE IT WITHOUT A SIDE MIRROR YOU CAN HAVE GOOD QUALITY HEALTH CARE WITHOUT ANY MACHINE LEARNING IN IT AND WE NEED TO BROADEN THE PERSPECTIVE OF ALGORITHMS AND HOW WE USE THEM IN MEDICAL CARE. THE OTHER SIDE NOTE I WOULD MAKE AND WE DIDN'T TOUCH UPON IT IN OUR PANEL BUT WE KIND OF ASSUME ALGORITHMS AND THEY'RE USE IF WE'RE ABLE TO USE THEM TO ADVANCE THE SCIENCE OF MEDICINE, WE'LL ADVANCE HEALTH CARE. AND THOSE ARE TWO VERY DIFFERENT ANIMALS. AND NIH-FUNDED RESEARCHERS TYPICALLY FOCUS ON ADVANCING THE SCIENCE OF MEDICINE WHILE WE LEAVE IT TO OTHERS TO ADVANCE THE PRACTICE OF MEDICINE AND HEALTH CARE AND THAT DIALOGUE NEEDS TO INCREASE. I CAN'T FIT IT IN THE CAR ANALOGY. >> THE BIGGEST DRIVER OF BEHAVIOR IN HEALTH CARE IS [INDISCERNIBLE] >> MAYBE KIND OF CONTINUING ALONG THOSE LINES, I THINK A REALLY INTERESTING POINT BRIAN RAISED IS EYE THE FINITE LIFE TIME OF SO-CALLED MEDICAL FACTS. THERE'S A HALF LIFE AND IT COULD BE RELATIVELY SHORT FOR THE SO-CALLED TRUTHS. AND I WONDER AND NIGAM YOU POINTED OUT YOU HAVE SOME FINITE NUMBER OF S IN MILLIONS AND -- OBSERVATIONS IN MILLIONS AND MILLIONS OF PEOPLE AND I WONDER WHAT YOUR THOUGHTS ARE ON THAT INTRINSIC BIOLOGIC DISPERSION AND ITS CONTRIBUTION TO THE FINITE LIFE TIME OF SO-CALLED MEDICAL FACTS. IT'S EASY TO BE SEDUCED OR LED DIRECTLY TO A CONCLUSION BUT YOU KNOW, YOU LOOK AT ENORMOUS POPULATIONS WHERE WE UNDERSTAND THAT THERE'S ENORMOUS VARIABILITY WITHIN THOSE POPULATIONS POP SO WILL MACHINE INTELLIGENCE APPROACHES ALLOW TO US GET MORE ABSOLUTE TRUTHS OR ARE WE GOING TO NEED BETTER DATA COLLECTION APPROACHES AND TECHNOLOGIES TO LOOK AT CONTINUOUS VARIABLES OTHER THAN DISCREET VARIABLES MEASURED OVER EXTREMELY LARGE TIME INTERVALS IN MILLIONS AND MILLIONS OF PATIENTS? >> IT'S A GREAT QUESTION. MY TAKE ON THAT RIGHT NOW IS THAT FIRST THE ALGORITHMS WILL LEAD US TO THOSE DATA MODALITIES WHERE MEASUREMENT ERROR IS WHAT YOU'RE GETTING AT, IS A BIG PROBLEM. AND MORE THAN NOT IT WILL BE A PROBLEM IN THE DISCREET PROBLEMS BECAUSE IT'S A HUMAN ENTERING THE CODE. IT'S INCENTIVIZED BASED ON WHAT CMS IS PAYING FOR TODAY BUT YOU CAN PICK THAT UP AS TRENDS IN TERMS OF NON-STATION ADDING IN THE DATA SETS AND THERE'S BEEN A COUPLE PAPER ALREADY THAT SHOW BASED ON HOW CODING CHANGES RATES OF SEPSIS RELATED AND PNEUMONIA RELATED DEATHS IN ICUs VARIES WILDLY BY 15X. THERE'S NO WAY THAT'S BECAUSE OF MEDICAL CARE. THOSE KINDS OF ANALYSES CAN BE DONE VERY FAST AT SCALE USING MACHINES POINTING US TO THOSE MODALITIES THAT ARE UNRELIABLE BECAUSE OF MEASUREMENT ERROR ARISING FOR WHATEVER REASON. >> THEY SAY HALF OF WHAT IS TAUGHT IN MEDICAL SCHOOL IS WRONG WE JUST DON'T KNOW WHICH HALF. THERE'S A DIFFERENCE BETWEEN MACHINE INTELLIGENCE TO SUPPORT DIAGNOSTIC IMAGE INTERPRETATION OR NAVIGATION HOW DO I GET FROM HERE TO THERE. TELL ME WHEN IT TURN AND TELL ME WHICH ROUTE TO TAKE AT THIS MOMENT IN TIME. WHAT SHOULD WE DO TO TREAT THE PROBLEM? ONE OF THE REASONS WE KEEP HAVING TO RELEARN AND IT'S NOT AN ABSOLUTE TRUTH, WE ACTUALLY KEEP PRODUCING NEW OBSERVATIONS, SOMETIMES NEW APPROACHES TO TREAT THE PROBLEM AND THEN THAT CHANGES WHAT THE ANSWER IS. 100 YEARS FROM NOW, WHAT TO DO TO TREAT THESE ABSOLUTE DIAGNOSTIC CLASSIFICATIONS IS LIKELY DIFFERENT THAN WHERE WE ARE NOW AND WHAT WILL CHANGE OVER TIME WILL CONTINUE TO CHANGE. SO FOR ANSWERING THE QUESTIONS OF WHAT TO DO AS AN INTERVENTION, WE NEED ANOTHER WAY TO UNDERSTAND WHAT IS THE REFERENCE STANDARD AND WHAT ARE THE FACTS AND IT'S NOT AN ABSOLUTE TRUTH BUT WHAT IS THE ABSOLUTELY BEST APPROXIMATION FOR NOW WITH OPENNESS AND RECOGNITION OF WHEN THE DATA CHANGES TO ANSWER THAT QUESTION. >> IF I CAN JUMP IN A LITTLE BIT. IF WE RECALL THE IMAGE OF DATA TRACKS LIKE A GENOME BROWSER BUT DIFFERENT MODALITIES OF DATA, IMAGING WASN'T ON IT. IT WASN'T ON IT BECAUSE TYPICALLY THE ANALYSIS OF IMAGES REMAINS INTRA-ENCOUNTER. AS IN AN EKG SCAN OR MRI AND THAT'S ONE ENCOUNTER AND YOU HAVE AN ALGORITHM READING IT USUALLY TRYING TO REPLACE THE RADIOLOGIST AND AUGMENT THEM AND THERE'S VERY FEW EFFORTS THAT ATTEMPTS TO TAKE THE IMAGE AND PRIMARY MEDICAL RECORD AND SAY SOMETHING ABOUT THE FUTURE. WHEN WE GET TO THAT LONGITUDINAL, THAT'S WHEN ALL THE ISSUES OF IMAGING MODALITIES CHANGES, THINGS YOU LEARN FROM A T1 SCANNER VERSUS A T3 SOUTHEASTERN IS NIGHT AND -- SCANNER IS NIGHT AND DAY AND IT'S EASIER AND SAFER IN THE SHORT TERM BUT WE MAY NOT CAPTURE THE LONG-TERM BENEFITS OF MACHINE USE INTELLIGENCE. FROM A RESEARCH FUNDING PERSPECTIVE WE NEED TO START LOOKING OVER TIME OPPOSED TO WITHIN TIME. WITHIN TIME THE COMPANIES ARE ON IT AND WE CAN LET GO OF THAT. >> ONE POINT I WANTED TO MAKE SI THINK IN OUR EXPERIENCE THERE'S DATA THAT GOES UNUSED WITHIN UNSTRUCTURED TEXT WITHIN THE MEDICAL RECORD THAT'S A HUGE RESOURCE LARGELY UNTAPPED. AND I THINK THE USE OF MACHINE INTELLIGENCE IS GOING TO PUSH US TO DIG IN THAT WELL OF INFORMATION AND THAT'S DEFINITELY GOING TO BE NEEDED AS WE MOVE FORWARD. >> NEXT QUESTION. >> THANK YOU. I WAS INTERESTED ABOUT YOUR TALK WHERE YOU'RE TALKING ABOUT TRUSTWORTHINESS AS BEING HAS THE ALGORITHM PREDICTED MORTALITY OVER THE PAST TWO YEARS. ONE OF THE QUESTIONS IS HOW WE BUILD THAT SAME TYPE OF TRUSTWORTHINESS FOR CLINICAL TRIALS APPLICATION WHERE A DRUG MAY ONLY HAVE TWO OR THREE LARGE CLINICAL TRIALS AND SO INSTEAD OF HAVING TWO YEARS OF EXPERIENCE WITH MAYBE HUNDREDS OF PATIENTS, WE HAVE MAYBE OVER FIVE YEARS, THREE TRIALS AND DON'T KNOW WHETHER THE DRUG DIDN'T WORK OR WE DON'T KNOW WHETHER WE DIDN'T HAVE AN ALGORITHM WE CAN TRUST AND DO YOU HAVE THOUGHTS IN HOW TO BUILD TRUSTWORTHINESS IN THAT APPLICATION? >> HISTORICALLY, THE PEOPLE WHO UPLOAD THEIR RESULTS TO CLINICALTRIALS.gov AND UPLOAD TO THE JOURNALS OF OBSERVATIONAL RESEARCH SO TO SPEAK, DON'T TALK TO EACH OTHER. THEY'RE USUALLY AT ODDS FOR THE MOST PART. BUT INCREASINGLY, I THINK WHAT YOU'RE GETTING AT IS WE FUNDED A TRIAL AND THE TRIAL HAS SOME RESULTS AND THE TRIAL'S ONLY PURPOSE WAS TO SHOW THAT CHEMICAL IS BETTER THAN WATER. ONCE THAT IS DONE AND WE START USING IT FROM GENERALIZING OR EXTRAPOLATING FROM 30,000 HUMANS TO 200 MILLION HUMANS, ALL THAT DATA IS GOING TO DATA WAREHOUSES FOR THE PURPOSES OF THIS CONVERSATION ARE THE PCOR NET KIND OF BUCKET. WE NEED MORE CONNECTIONS BETWEEN APROFL APROFLSZ -- APPROVES AND REGULATORY OBSERVATIONS AND ABOUT WHAT HAPPENED AND WE DON'T HAVE THAT RIGHT NOW. WE HAVE ALL THE PIECES IN PLACE AS A NATIONAL INFRASTRUCTURE BUT NOT AS MUCH CROSS TALK. THE MARGOLIS CENTER WILL TALK ABOUT REAL WORLD EVIDENCE FOR DECISION MAKING BUT STAY AWAY FROM TAKING THE SAME PCOR NED NET FOR THE EFFICACY OF MEDICAL PRODUCTS. I DON'T KNOW WHY THAT HAPPENS. MAYBE THERE'S LEGAL REASONS OR MAYBE WE HAVEN'T ARRIVED THERE YET AS A COMMUNITY. >> IF YOU ARE PROMOTING TRIAL EVIDENCE YOU POINT OUT THE RISK OF BIAS IN OBSERVATIONAL EVIDENCE AND SAY YOU DON'T TRUST BECAUSE OF THAT. AND IF YOU'RE PROMOTING REAL WORLD EVIDENCE YOU POINT TO THE LACK OF DIRECTNESS OR EXTERNAL VALIDITY OF THE RANDOMIZED TRIALS AND HOW MUCH OF THAT IS WHAT YOU'RE TRYING TO PROMOTE VERSUS WHAT YOU'RE REAL TRUST ISSUE IS SO MAYBE IF WE COME UP WITH A COMMON LANGUAGE FOR HOW TO TRUST THE DATA THAT IS NOT DIRECTLY DEPENDENT ON WHAT TYPE OF DATA BUT CAN EXPRESS THE DIFFERENT CONCEPTS, ALL THE DATA HAS ISSUES OF TRUSTWORTHINESS. IT'S JUST DIFFERENT ISSUES FROM DIFFERENT SOURCES. AND TO USE LUCA'S LANGUAGE, CLINICAL TRIAL DATA HAS ISSUES OF RELEVANCY AND THERE'S ISSUES OF RELIABILITY SO PICK YOUR EVIL. >> A MORE GENERAL POINT THERE IN OTHER INDUSTRY TRUSTWORTHINESS OF MACHINE INTELLIGENCE SYSTEM COMES FROM MONITORING THEM AFTER THEY'RE DEPLOYED BECAUSE THERE'S ONLY SO MUCH YOU CAN DO UP FRONT WITH A COMPLEX SYSTEM. A DRUG VERSUS NO DRUG BUT WHAT IF THE DRUG HAS A THERAPEUTIC COMPANION THAT IS AN APP WITH A LOT OF POSSIBLE BRANCHES AND LOGIC AND CAN POSSIBLY TEST THEM IN A SINGLE SEPARATE ARM SO IT'S MORE FEFEFFECTIVE TO SEE HOW IT'S WORKING IN THE REAL WORLD SO POST-MARKET VALIDATION WILL BE FUNDAMENTAL LATER ON I HOPE AND BELIEVE. >> I HAVE ONE SHORT COMMENT AND A QUESTION, IF I MAY. SO THE COMMENT IS ABOUT THE VERY NICE OBSERVATION THAT MICHELLE MADE ABOUT TRUST TO THE SYSTEM AFTER UPDATES AND WHAT HAPPENS WITH SYSTEM UPDATES AND I JUST WANT TO MENTION YOU HAVE IT FROM A POINT OF VIEW AT THE FDA WE HAVE THE SAME PROBLEM FROM A DEVICE REGULATION POINT OF VIEW BECAUSE TO MAXIMIZE THE BENEFIT OF THE DEVICES YOU MAY WANT TO UPDATE THEM OFTEN WHEN NEW EVIDENCE IS GENERATED BUT THEN THE TRADITIONAL WAY IS THEN TO DO THOSE UPDATES IN A DISCREET MANNER AND SOMETIMES IF NECESSARY, COME TO THE FDA WHEN YOU DO THE UPDATES. SO THE FDA RECENTLY RELEASED A DISCUSSION PAPER ON UPDATES OF SOFTWARE MEDICAL DEVICE. I WANT TO MENTION THAT AND THOUGH THE FORMAL FEEDBACK PERIOD IS OVER, IT CAN STILL DOWNLOAD THE PAPER AND LOOK FOR UPDATES TO SOFTWARE AS A MEDICAL DEVICE AND STILL SEND YOUR FEEDBACK BECAUSE WE WANT TO INVOLVE A WHOLE COMMUNITY ON HOW THIS PROCEEDS. THEN THE QUESTION IS TO LUCA, YOU MENTIONED A PERSON-GENERATED DATA AND I COME FROM THE IMAGING WORLD WHERE THESE ARE MEDICAL IMAGING DEVICES AND THEY GO THROUGH CONTINUOUS SEEN BY MEDIC MEDICAL PHYSICISTS LOOK FOR HIGH QUALITY MEDICAL DATA AND EVEN THEN WE SEE A LOT OF DISCUSSION AND SO WHAT ARE THE STEPS FOR MAKING SURE THE DEVICE IS CALIBRATED. I'M NOT TALKING ABOUT THE DEVICE BEING COMPLETELY BAD OR OUT OF WHACK BUT YOU KNOW, SLOWLY DRIFTS FROM ITS CALIBRATION AND HOW CAN WE, WHAT ARE SOME OF THE EFFORTS IN WHAT THE COMPANIES AND PRODUCER AND MANUFACTURERS OF THESE DEVICES ARE DOING AND ALSO ON THE USER SIDE. >> THANK YOU FOR YOUR QUESTION AND I'LL FOLLOW-UP ON YOUR FIRST POINT LATER REGARDING DEVICE VERIFICATION AND VALIDATION, THAT'S A WHOLE WORLD BY ITSELF. I WAS LOOKING AT THE DATA COLLECTED BY SOME DEVICE AND VERIFICATION AND IS THE DATA COLLECTED RIGHT AND VALIDATION AND IS THE RIGHT DATA BEING COLLECTED AT THE DEVICE LEVEL AND IS THIS SOMETHING COMPANIES ARE WORKING ON ACTIVELY. THERE'S A LOT OF DISCUSSION ANYTIME YOU GO TO AN NIH OR ANY PANEL LIKE EAST OF THE WEST COAST THIS QUESTION COMES UP AND IT'S A VERY RELEVANT ONE WHERE WE ARE IN THE SPECTRUM OF TECHNOLOGY, IMAGING IS MATURE. YOU CAN THINK OF GENETICS AND GENOMIC SEQUENCING ALSO HAVING GONE THROUGH THE GROWTH OF HAVING DIRTY DATA AND SOMEHOW THE QUALITY'S BEEN FIGURED OUT OVER TIME. FOR PERSON-GENERATED HEALTH ARE DATA WE'RE JUST AT THE BEGINNING AND DECIDING THE CHECKLIST OF WHAT SHOULD LOOK LIKE AND FOR THE FDA FRAMEWORK, THAT LETTER WITH HELP FROM RESEARCHERS AND SOME IN THIS ROOM, THANK YOU, HOW DO YOU UPDATE THE SYSTEMS WHEN THEY'RE DEPLOYED IN THE REAL WORLD IS A PROBLEM BECAUSE THE TRAINING DATA IS IMPORTANT AND WHAT ABOUT THE DATA THAT COMES DURING THE DEPLOYMENT AND SHIFTS WITH RESPECT TO THE TRAINING DATA AND HOW DO YOU DETECT THAT AND TAKE CARE OF THAT OVER TIME. THERE'S A LOT OF IDEAS BUT I'M HAPPY TO FOLLOW-UP IN DISCUSSION LATER. >> THANKS EVERYONE FOR INTERESTING TALKS. I WANT TO COME BACK TO LEW AH -- LUCA, ONE OF YOUR FIRST SLIDE IS BUILDING TRUSTWORTHINESS. I WANTED TO ASK BECAUSE WE HAVE A GREAT PANEL HERE WITH A RANGE OF EXPERTS WHAT IS YOUR PERSPECTIVE ON THAT IN THE SENSE THAT IN ORDR TO UNDERSTAND TO VERIFY AND TRUST EACH COMPONENT OF THIS YOU KIND OF HAVE TO UNDERSTAND EACH COMPONENT AND I'M A STATISTICIAN SO I HAVE NO IDEA OF COLLECTING ACCELEROMETER DATA AND I FALL SOMEWHERE IN THE MIDDLE SO I CAN LOOK AT VALIDATION AND TRAINING OF ALGORITHMS AND TRUST THAT PART BUT I REALLY DON'T UNDERSTAND THE PARTS. SO THE POINT IS IT TAKES A PANEL OF ACADEMIC EXPERTS TO REALLY UNDERSTAND EVER Y PART OF THIS PROCESS. GIVEN THAT, HOW CAN WE START TO EXPLAIN AND BUILD TRUSTWORTHINESS OF THE WORK FLOWS TO LAYMEN AND PATIENTS AND I DON'T WANT TO SOUND CONTROVERSIAL SAYING THIS BUT I HAVE FRIENDS WHO ARE WOMEN AND DOCTORS AND DELIVER A DIAGNOSIS TO THE PATIENT AND THEY'LL SAY THAT'S NICE CAN YOU HAVE A DOCTOR TELL ME THAT. PEOPLE EXPECT TO RECEIVE INFORMATION IN A WAY AND WHAT CAN WE DO TO SMOOTH THAT TRANSITION AND BUILD TRUST IN LAY COMMUNITIES? >> A LOT OF POINTS THERE. TRUST AND EXPLAINABILITY ARE TWO DIFFERENT THINGS LIKE NIGAM WAS SHOWING. I WOULD SAY TODAY WE STEP INSIDE THE CAR AND DRIVE WITHOUT UNDERSTANDING HOW THE CLUTCH AND ENGINE WORKS AND RELY ON THE FACT THE MANUFACTURER HAS CHECKED CERTIFICATION OF THE PARTS FROM VENDORS AND HAVE DONE 50 YEARS AGO PROBABLY YOU WOULD OPEN THE HOOD AND SEE IF THE PARTS WERE AS YOU EXPECT OR SHOULD BE AND I THINK THIS IS LIKE A COMPONENT THAT TAKES TIME TO GET USED TO NEW MODALITIES AND HOW YOU TRUST THAT. IN TERMS OF DOES IT TAKE A PANEL TO UNDERSTAND ALL THE POSSIBLE PARTS MAYBE AS YOU DEVELOP SCIENCE IT DOES BUT AS YOU DEVELOP TECHNOLOGY LIKE IF YOU'RE A COMPANY, YOU THINK OF THE TRUST OF THE SYSTEM IN TERMS OF IN A MODULAR WAY IN HOW SYSTEM ENGINEERING WILL TEACH YOU TO DO AND TRUST EVERY SINGLE PART AND DEL GET THE TRUST TO WHATEVER IS TRUSTING THE PART AND THE VERIFICATION IN THE FRAME BUILDING TYPES OF SYSTEMS. ANYONE ELSE? >> IN THEORY, THEORY IN PRACTICE THEY'RE THE SAME IN PRACTICE THEY'RE NOT. IF YOU'RE TRYING TO USE IT FOR A PRACTICAL USE A FUNCTIONAL USE THEN AS YOU WERE SHOWING TRUST IN THE RESULTS AT THE END IS WHAT YOU NEED TO TRUST AND IF YOU DON'T KNOW WHY OR HOW, YOU MAY STILL FIND YOU TRUST IT ENOUGH FOR FUNCTIONAL USE. IF YOU DIG BACK INTO THE WHY AND HOW AND DISCOVER AND THERE'S A MISSTEP OF ASSOCIATION BUT NOT CAUSATION YOU MAY DISCOVER NEW INSIGHTS TO LEAD TO BETTER WAYS TO GET THE RESULT BUT UNTIL THOSE DISCOVERIES OCCUR FOR FUNCTIONAL USE, YOU DON'T HAVE TO HAVE ALL THE THEORY LINED UP AND REALLY WELL UNDERSTOOD. >> I WOULD SAY THAT AT SOME LEVEL WE HAVE TO TRY AS IN IT'S VERY HARD TO COME UP WITH THE CRITERIA FOR WHEN WE SHOULD TRUST ALL THE COMPONENTS OF THE WORK FLOWS SITTING IN AN OFFICE AND THINKING. -- AND THINKING. THERE HAS TO BE SOME EXPERIMENTATION. WE ARE AT A WEIRD PLACE WHERE YOU ARGUE WITH IRBs WHEN YOU WANT TO VALIDATE A MACHINE-LEARNING ALGORITHM THAT'S GIVING YOU A PROBABILITY THAT IS NOT GOING ANYWHERE OTHER THAN YOUR EXCEL SPREAD SHEET YOU HAVE TO ARGUE WHY TUSKEGEE DOESN'T APPLY AND THE BELMONT REPORT DOESN'T APPLY. AT SOME POINT IT'S LUDICROUS. YOU DON'T HAVE TO NEED PERMISSION TO TRY A NEW BRAND OF COFFEE BUT THAT'S THE STAGE WE'RE AT IN THE PRACTICE MUCH RESEARCH WHERE IN ORDER TO DO ANY INNOVATION LIKE TRY THE STRONG BLEND VERSUS THE LIGHT BLEND, I NEED IRB APPROVAL. AS A SYSTEM WE NEED TO EVOLVE OUR PROCESSES SO WE HAVE SAFE SPACE TO EXPERIMENT WITHIN THE END TO END SYSTEMS. WE CAN'T VALIDATE THE CLUTCH BOX AND ENGINE PRISTINELY IN DIFFERENT ORGANIZATIONS AND SHIP EVERYTHING TO ONE PLACE AND HOPE THE CAR DRIVES. IT'S NOT GOING HAPPEN. THAT'S THE MISSING GREENT -- INGREDIENT WE NEED PLACES TO DO END TO END TESTING OF SIMPLE SYSTEMS THAT INCREASINGLY GET COMPLEX. WHENEVER YOU SAY CLINICAL TRIALS IT IMPLIES FDA CERTIFICATION AT SOME POINT SO THAT'S WHY I DON'T USE THE WORD. PROSPECTIVE STUDY. >> ONE PIECE OF THAT QUESTION WAS HOW DO WE GET PEOPLE TO TRUST THIS SORT OF TECHNOLOGY. ONE POINT I DIDN'T GET TO MAKE IN OUR USE CASE, MEDICINE IS ALWAYS GOING TO START AND END WITH THE PHYSICIAN. ERIC TOPEL FROM THE SCRIPPS INSTITUTE PUBLISHED A BOOK CALLED DEEP MEDICINE AND HOW TO USE ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING TO MAKE HEALTH CARE MORE HUMAN. THERE'S ALWAYS GOING TO BE THAT NEED FOR A HUMAN COMPONENT AND A SKILLED EXPERT TO TAKE THE INFORMATION AND DISTRIBUTE IT TO THE PATIENTS AND SO IN OUR WORLD THAT'S ALWAYS GOING TO BE KEY. HE MAKES AN ANALOGY OF SLF-DRIVING CARS. OF COURSE THE ULTIMATE GOAL IS TO HAVE A CAR THAT REQUIRES NO AND DOESN'T ALLOW ANY HUMAN INTERVENTION THAT'S ABLE TO DRIVE BY ITSELF BUT THERE'S ALWAYS GOING TO BE CONDITIONAL AUTOMATION AND A NEED FOR A HUMAN TO STEP IN AND SAME THING WITH ARTIFICIAL INTELLIGENCE IN MEDICINE. IT'S NEVER GOING TO BE 100% ARTIFICIAL WHERE THE PATIENT IS GETTING AN AUTOMATED RECORD. THERE'S GOING TO BE A PHYSICIAN LOOKING AT THE RESULTS AND STEPPING IN WHEN NEEDED. I JUST WANTED TO MAKE THAT POINT. >> I'M FROM UAB. I WAS REALLY STRUCK BY THE CONVERSATION ABOUT TRUSTWORTHINESS BEING OVER TIME. I CAN THINK OF EXAMPLES WHERE IT'S GONE THE OPPOSITE DIRECTION AND TRUSTWORTHINESS HAS BEEN PULLED BACK IN THE LAST FEW DAYS THERE WAS A PAPER THAT SHOWED THERE WERE EXTREMELY HIGH FALSE POSITIVE RATES IN THE GENOTYPES TESTS FOR RARE VARIANT DIAGNOSIS IN DIFFERENT PATIENT POPULATIONS. FOR SOMEONE THAT'S A RESEARCHER AND THINKING OF INCORPORATING THE GENOMICS AND GENETICS AND THE OTHER WEARABLES AND IMAGING AND THINKING HOW WE HELP PATIENTS PARTICULARLY THOSE IN CRITICAL NEED WITHOUT OVERSTEPPING ETHICAL BOUNDARIES OR MOVING TOO QUICKLY. I WONDER IF YOU HAVE THOUGHTS ON HOW TO EMPLOY MACHINE INTELLIGENCE TO HELP AUTO-CORRECT OR SELF-CORRECT GIVEN THE DIFFICULTIES IN THE HEALTH SYSTEM. THANK YOU. >> YOU'RE MAKING THE POINT I WAS TRYING TO MAKE WHICH IS TRUST ACCRUES OVER TIME. THE FACT THAT THE VARIANTS AND UTILITY OVER TIME HAS BEEN DISPROVEN IS PRECISELY THE POINT. IT SHUNT HAVE BEEN DECLARED AT TRUSTWORTHY WITHOUT BEING VALIDATED OVER TIME AND THAT'S SOMETHING THAT COMES UP OFTEN IN DEAN OMICS. -- GENOMICS. I REMEMBER BEING IN GRAD SCHOOL IN 2000 AND OPENED UP THE NEWSPAPERS AND CANCER WAS SUPPOSED TO BE CURED IN 2010. THAT'S WHAT THE HEADLINE SAID. SO THE ISSUE IS THAT WE NEED TO STEP BACK A LITTLE BIT FROM OVER PROMISING AS WELL. VARIABLES ARE NOT GOING TO CURE CANCER AND NEITHER IS GENOMICS. >> AGAIN, RESEARCH CONNECTING TO REAL WORLD PRACTICE AND THERE'S A PAPER, PEOPLE KNOW ABOUT IT, AND WHEN YOU START USING IT IN PRACTICE IN 200MAN -- 200 MILLION AND 3 MILLION PEOPLE IT CAN HAVE A LOCAL LEARNING SYSTEM. THERE'S NO LEARNING THAT GOES BACK AT THE MACRO LEVEL. SO PART OF IT IS FIGURING OUT THE CONNECTIONS THAT HOW WE PUT TOGETHER TEAMS AND PLACES WHERE END TO END FEEDBACK CAN HAPPEN. FOR VARIABLES AND ACROSS MODALITIES. PEOPLE STUDY RADIO GENOMICS AND DO VARIABLES AND IMAGES BUT NOBODY ACROSS ALL OF THEM. IF YOU WANT TO EFFECT HEALTH CARE, WE NEED TO START DOING THAT. >> SO YOU ANTICIPATED MY QUESTION. I WANTED TO CONNECT THE DOTS BETWEEN TRUST OCCURRING OVER TIME AND THE FACT THAT HUMANS, CLINICIANS, THE CLINICAL TEAM AND DATA SCIENTISTS CAN OUTRIGHT THE OUTPUT OF THE SYSTEM AND WHAT ABOUT THE FEEDBACK LOOP ABOUT THE LEARNING HEALTH SYSTEM? HAVE YOU SEEN CONCRETE EVIDENCE OF GOOD FEEDBACK LOOPS TO CONNECT THE FEEDBACK WHEN THE SYSTEM HAS TO BE OVER RIDDEN AND WHY? I HOPE WE CAN AVOID [INDISCERNIBLE] PROBLEMS IN HEALTH CARE. >> NOT YET. I'M SURE ACROSS OUR 6,000 HOSPITALS AND THE MULTIPLE HEALTH INSURANCE SYSTEMS SOMEBODY, SOMEWHERE IS DOING THIS BUT THEY'RE TALKING ABOUT IT AND NOT HITTING THE POPULAR PRESS OR THE RESEARCH LITERATURE. MOST OF THE LEARNING HEALTH SYSTEMS RELATED CONVERSATIONS TEND TO BE POWER POINT DIAGRAMS. >> AND MICHELLE, WHAT IS YOUR SYSTEM DOING WHEN IT'S OVER RIDDEN? >> WE DON'T CURRENTLY HAVE A SYSTEM AND WORKING WITH EXTERNAL PARTNERS TO HELP THEM TUNE THEIR TOOLS BUT REALLY THERE'S NO SYSTEM IN PLACE. WE'RE JUST AT THE RESEARCH LEVEL RIGHT NOW. >> EVEN WITH MACHINE SUPPORT THAT NAY -- MAY NOT BE CLASSIFIED THAT WAY BUT YOU SET UP A RULE AND SAY IF THIS OCCURS SEND THIS MESSAGE AND ALERT. THERE HAVE BEEN REPORTS WHERE PEOPLE HAVE BEEN ABLE TO GO BACK AND LOOK AT THE REASON FOR OVERRIDING THE ALERT AND USE THAT AS FEEDBACK AND THAT'S IN LESS INTELLIGENT SYSTEMS AND IT SEEMS LIKE THAT'S A HANDFUL OF REPORTS THAT HAVE BEEN PUBLISHED OF USING IT THAT WAY. WE'RE STILL VERY EARLY IN THE REVERSE AUTO CORRECT. HUMAN CORRECT TO THE AUTOMATED EFFORTS. >> I WAS HAVING THIS CONVERSATION ON OUR CAMPUS. STUDIES OF CLINICAL DECISION SUPPORT SYSTEMS THAT WORK IN HIGH ACUITY ENVIRONMENTS, RULE-BASED THINGS THAT GET OVERRIDDEN, THERE'S A DECENT BODY OF LITERATURE AROUND THAT BUT THERE IS PRACTICALLY NO LITERATURE ON THE STUDY OF THE ENTIRE END TO END CARE SYSTEMS THAT HAVE A FEW ALGORITHMIC AGENT AND A FEW ACTORS IN THE MIX AND YOU'RE STUDYING NET OUTCOME AND A FEEDBACK LOOP. I BELIEVE THERE'S CAREERS TO BE MADE. >> IN THINKING ABOUT THE GENETIC EXAMPLES, THERE ARE AND I THINK WE'VE DONE WELL IN GENETICS WHEN IT COMES TO KNOWING THE RIGHT PHENOTYPE AND WE'RE BETTER AT IDENTIFYING GEE GENETIC IM -- KEY GENETIC IMPACT AND FOR COMMON DISEASE IT'S HARDER FOR PHARMACO GENETICS AND IN THINKING OF THE IMPORTANCE OF THE PHENOTYPES TAKES ME TO THE CLINICAL OUTCOMES THROUGH THE HEALTH RECORD SYSTEMS AND THE IMPORTANCE OF VARIABLES WITHIN THE RECORDS AND THE SIN TACTIC AND SEMANTIC ERRORS THAT CAN COME ABOUT LOOKING ACROSS ALL HEALTH CARE SYSTEMS THEN PROVIDING THE FEEDBACK LOOP MAY HELP IN ONE SYSTEM BECAUSE OF THE SAME SYSTEM THEY'RE USING BUT IT MAY NOT TRANSFER ACROSS ALL AND THAT VARIABILITY IS GOING TO BE IMPORTANT. I WAS WONDERING IS A.I./M.I. THE ANSWER OR JUST AS IMPORTANT TO MAKE SURE WE'RE THINKING ABOUT THOSE VARIABLES IN THE TRAINING SYSTEMS WE'RE LOOKING AT AND DEVELOPING. A.I. IS NOT THE ANSWER. AND THERE'S RESEARCH THAT SHOWS MACHINE LEARNING SYSTEM IN ONE PROVIDER EVEN WHEN GIVEN THE SAME SCHEMA WILL TRANSFER POORLY IT'S ABOUT THE SPECIFIC PROVIDER OF WHICH THERE ARE MANY. A.I. IS NOT THE ANSWER AND THERE'S A PAPER ABOUT IT I WOULD ADD. >> I WANT TO CLARIFY THAT A LITTLE BIT. THERE'S THIS DESIRE TO HAVE MODELS THAT WORK ACROSS SITES. LET'S SAY WE'RE LOOKING AT A MORTALITY OPERATION MODEL. WHY SHOULD A MORTALITY PREDICTION MODEL THAT WORKS AT STANFORD HOSPITAL IN CALIFORNIA WORK THE EXACT SAME WAY AT A HOSPITAL IN MUMBAI. WHAT'S THE REASON FOR THAT REQUIREMENT? WHEN I DIG INTO THIS A LITTLE BIT, WHAT I DISCOVER IS THAT IN MEDICINE MOST OF US ARE TAUGHT UNDER WHAT'S CALLED THE CAUSAL MODELING LITERATURE. WE WANT MECHANISTIC UNDERSTAND INGS OF THINGS AND WE TRY TO ENSURE EXTERNAL VALIDITY OF OUR MODELS BUT INHERENTLY WHEN WE'RE USING A CLASSIFIER OR PREDICTOR TO MAKE A WORK FLOW MORE EFFICIENT AT A SITE, IT'S A LOCAL PROBLEM. AND SO WHY WOULD WE WANT THIS ADDITIONAL CONSTRAINT THAT MY MODEL IS GLOBALLY TRUE? WE DON'T NEED THAT. WHAT WE NEED, RATHER, IS THE ABILITY TO TRAIN MODELS THAT WORKS ANYWHERE BUT YOU ALWAYS TRAIN A LOCAL MODEL. IF NOT, YOU START WITH THE GLOBAL MODEL AND THEN RETRAIN IT AND REFIT IT AND MODIFY IT SO YOUR LOCAL DESTRUCTIONS. AS A COMMUNITY, I THINK WE NEED TO STEP OUT OF THIS GLOBAL VALIDITY OF MODELS MIND SET WHEN WE DO REGRESSIONS WITH FIVE VARIABLES THAT MADE SENSE BECAUSE IT WAS USEFUL TO HAVE THAT QUALITY CHECK AND WHEN WE DEAL WITH MODELS WITH 50,000 VARIABLES I THINK IT'S A MISTAKE TO HAVE THE SAME MODEL WORK IN CALIFORNIA AND MUMBAI. >> THERE'S OTHER REASON FOR THE VALIDITY CONCEPT. WORKING IN A COMMUNITY HOSPITAL I CLINICALLY WANT TO BE ABLE TO PREDICT THE MORTALITY FOR A PATIENT WITH A PARTICULAR CONDITION AND I WANT TO TRUST THAT PREDICTION MODEL. IF THERE IS A GLOBAL PREDICTION MODEL THAT'S BEEN USED IN MANY OTHER PLACES AND FOUND TO BE VALID IN SOME WAY, THEN I FEEL LIKE I CAN TRUST IT TO USE IT AND I DO NOT HAVE THE RESOURCES. DO YOU NEED A STATISTICAL MODEL OR NEED ADVANCED SOFTWARE OR INTERPRETATION OF THE DATA, I DID NOT HAVE THE RESOURCES TO DETERMINE A LOCAL PREDICTIVE MODEL LET ALONE TO KNOW IF I CAN TRUST THE LOCAL PREDICTIVE MODEL THAT EXISTS. UP TO NOW, THERE'S NOT BEEN THAT TYPE OF RESOURCE FOR LOCAL APPLICATION SO HAD THERE BEEN REASONS OTHER THAN THE EDUCATIONAL BACKGROUND WHERE IT'S NECESSARY FOR RESOURCE BALANCING TO HAVE A GENERIC MODEL I CAN USE IN MY LOCAL INSTANCE VERSUS CREATING A LOCAL MODEL. NOW, HOW DO YOU INTRODUCE A NEW CULTURE THAT YOU CAN TRUST A LOCAL MODEL FOR THESE THINGS. ONE, AREA THAT MAY HAVE SIMILAR ACCEPTANCE ISANTI-MICROBIAL PATTERNS ALWAYS LOCAL AND YOU'D LOOK MORE TO THE LOCAL EVEN THE INSTITUTION'S PATTERN CROSSING THE STREET MAY HAVE A DIFFERENT PATTERN SO THERE'S SOME PRECEDENCE FOR LOCAL MODELS BUT IT'S GOING TO TAKE A LOT OF EFFORT TO CHANGE WHAT WE NEED BY A MODEL AND WHAT WE'RE TRYING TO UNDERSTAND FOR A LOCAL APPLICATION. >> IT'S LOCALITY FOR A SURROGATE FOR TRUST. IF WE UNDERSTAND THE CAUSALITY WE CAN DECIDE TO TRUST IT BUT IN THE CASE OF M.L. DRIVEN WORK FLOWS THAT CAN FALL PART BADLY. >> A QUICK POINT TO THAT, I THINK BOTH YOUR POINTS ARE POINTED TOWARDS THE FACT THAT WE MIGHT BE LOOKING AT THE LEADING EDGE OF SCIENCE IN DOING ENGINEERING. ALL BRIDGES ARE DIFFERENT. THERE'S TO SCIENCE OF BRIDGES. THERE'S MATERIAL SCIENCE THAT TELLS YOU HOW MATERIALS BUILDING BRIDGES ARE SUITED WITHIN THE GENERAL PRINCIPLE BUT THE GOLDEN GATE BRIDGE IS NOT SOMETHING YOU CAN PUT IN THE HUDSON RIVER. IT HAS TO BE LOCAL AND THE GLOBAL PART OF BUILDING BRIDGES COMES FROM THE PROCESS SYSTEM ENGINEERING HAS TO BUILD BRIDGES PREVENTION AND RISK ASSESSMENT. THAT'S THE LEVEL WHERE THINGS BECOME GLOBAL AGAIN BUT NOT THE MODEL ITSELF. >> AND WE USED TO BUILD BRIDGES BEFORE THERE WAS CIVIL ENGINEERING AND MODEL SCIENCE. >> AND YOU PRESENTED EXAMPLES OF PRACTICE SUBSEQUENTLY SHOWN TO BE INCORRECT. WHAT I'M WONDERING IS, DO WE HAVE THE POSSIBILITY WHERE BECAUSE AND THAT CORRECTNESS WASN'T BASED ON ARBITRARY OR WHIMSICAL CONCLUSION BUT THE BEST AVAILABLE EXPERTS AND THEY WERE DEALING WITH INCOMPLETE DATA SETS WHICH IS WHY IT WAS SUBSEQUENTLY OVERTURNED. IS THERE A CONCERN WITH MACHINE LEARNING WE'RE JUST GOING TO AMPLIFY CREATING A LOT OF POTENTIAL PRACTICES AND ASPECTS OF MEDICINE THAT WILL SUBSEQUENTLY BE OVER TURNED WHEN NEW DATA IS AVAILABLE? AND WHAT I'M WONDERING IS, HAS THERE BEEN ATTEMPT TO GO BACK AND SEE WHETHER OR NOT WITH THE AVAILABLE DATA AT THE TIME, DOES THE MACHINE LEARNING APPROACH, WOULD THEY HAVE COME TO THE SAME CONCLUSIONS DONE AT THE TIME TO DEMONSTRATE WHETHER THERE IS INSIGHTS WE WERE MISSING AT THAT TIME OR WHETHER THEY'RE HOBBLED BY THE SAME LIMITATIONS WE WERE OPERATING UNDER AT THAT TIME IN TERMS OF WHAT THAT MEANS FOR THE FUTURE? >> I'D LIKE TO EXPLAIN A COUPLE OF THOSE EXAMPLES BECAUSE IT'S NOT AS SIMPLE AS INCOMPLETE DATA SET AND THEN WE GOT MORE COMPLETE DATA AND THEN IT LED TO THE REVERSAL. IT GETS TO THE HEART OF THE TRUSTWORTHINESS ISSUE AND WHAT WE PUT TRUST IN AND THE DATA SETS DIDN'T HAVE THE TRUSTWORTHY DATA AT THE SIME. THE HORMONE -- TIME. THE HORMONE REPLACEMENT EXAMPLE WE HAD 30 YEARS OF OBSERVATIONAL DATA THAT SAID WOMEN WHO TOOK HORMONES AFTER MENOPAUSE HAD LESS HEART DISEASE SO THAT ASSOCIATION OR OBSERVATION WAS CORRECT BUT WHAT WAS HAPPENING WAS WOMEN WERE DOING THAT BECAUSE WE WERE TELLING THEM TO. AND THE WOMEN WHO WERE DOING IT BECAUSE WE WERE TELLING THEM TO WERE MORE LIKELY TO DO OTHER THINGS WE WERE TELLING THEM TO DO, DIET, EXE EXERCISE AND NOT SMOKING AND SO FORTH AND THE ADJUSTMENT OF THE DATA DIDN'T ACCOUNT FOR THAT SO RANDOMIZED TRIALS, HORMONE VERSUS PLACEBO AND EVERYTHING ELSE CONTROLLED, WE SAW A DIFFERENT STORY. NOW, AT THE TIME THERE WAS A LOT OF SELECTIVE INTERPRETATION AND PROMOTION OF WHICH WAY TO SEE IT AND INFLUENCE HOW THINGS GET RECORDED AND UNDERSTOOD. THAT WAS A DIFFERENT TYPE OF EVIDENCE HOW WE WERE LEARNING THE INFORMATION. THEN A DIFFERENT EXAMPLE THE ANTI-ARRHYTHMICS AFTER A HEART ATTACK WE KNEW FROM OBSERVATION IF YOU HAD THESE PVCs, EXTRA SQUIGGLES ON THE EKG YOU HAD HIGHER MORTALITY. WE KNEW CERTAIN DRUGS WOULD GET RID OF THE PVCs. SO WELL-MEANING PHYSICIANS SAID YOU HAVE PVCs AFTER A HEART ATTACK, LET ME GIVE YOU THIS MEDICINE AND DID A RANDOMIZED TRIAL MEDICINE VERSUS PLACEBO AND THE MEDICINE WORKED BUT INCREASED MORTALITY BECAUSE THE SIDE EFFECT WAS A FATAL ARRHYTHMIA. IT TURNS OUT THE PVCs WERE NOT THE CAUSE OF INCREASED MORTALITY, THEY WERE ASSOCIATED WITH A SICKER HEART AND IF YOU HAD MORE HEART DAMAGE YOU HAD HIGHER MORTALITY AND YOU HAD THE PVCs. SO THE PROBLEM IS OR CONFUSION OF ASSOCIATION WITH CAUSATION AND THE RANDOMIZED TRIAL IS A METHOD TO SAY, IS IT IN THE CAUSAL PATH VERSUS NOT. SO IF WE APPLY MASSIVE AMOUNTS OF MACHINE INTELLIGENCE AND IDENTIFY A LOT OF ASSOCIATIONS, WE CAN ACTUALLY AMPLIFY AND FIND SIGNALS THAT ARE FALSE AND AMPLIFY THEM BUT IF WE LEARN HOW TO APPLY THE MACHINE INTELLIGENCE TO MAKE THOSE ADJUSTMENTS AND CONSIDER THOSE ISSUES MAYBE IT CAN HELP WITH THAT. >> A QUICK TIME CHECK AND ONE MORE QUESTION AND ANSWER AND ALSO THERE'S A THING CALLED CAUSAL MACHINE LEARNING WE'LL TALK ABOUT LATER. >> MY NAME IS JUNE LIAM. I'M A CHAIR OF AN ARTIFICIAL INTELLIGENCE GROUP. AND AT THE BEGINNING, WE FOCUS ON PREVENTIVE MEDICINE LIKE SCREENING AND X-RAYS AND MRI, THEN WE FOCUS ON DIAGNOSTICS SO EVERYBODY TALKING ABOUT DATA RIGHT NOW IS A BIG REFERENCE FOR DIAGNOSTIC AND NOW WE'RE FOCUSSED ON AUTONOMOUS THERAPEUTICS SO EACH STEP WE WORK ON, WE FIND OUT I AGREE THERE'S A BIG GAP BETWEEN A.I. SPECIALISTS AND PHYSICIAN. BIG GAP BECAUSE WE'RE THINKING DIFFERENTLY. SO EVENTUALLY THAT'S WHY OUR SCIENTISTS OVER SPECIALISTS AND WE TRY TO MERGE THIS GAP. BECAUSE I'LL GIVE YOU AN EXAMPLE. AUTONOMOUS CAR, EVERYBODY KNOW AND UNDERSTAND BUT FOR MEDICINE, THE TREATMENT, THE CORE OF THE TREATMENT IS A P.K.P.T. MODEL. YOU HAVE TO KNOW EXACTLY HOW IT WORKS AND WHAT IS THE DOSAGE. WHAT IS THE SIDE EFFECT. WHAT IS THE EFFICACY IN THERE. IN FACT, PHARMACOLOGISTS NOW FOR A.I. SPECIALISTS IT TAKES A LONG TIME TO LEARN THIS. WHY -- THAT'S WHY I AGREE IN THE NEAR FUTURE THE MEDICAL SCHOOL STUDENT SHOULD LEARN ALGORITHMS. I DESIGNED 250 ALGORITHMS FOR TREATMENT STILL NOT ENOUGH. WE NEED A LOT OF DETAIL WORKS AND THEY NEED ALL THE SPECIALISTS TO WORK TOGETHER AS A BIGGER GROUP, BIG TEAM AND THEN WE CAN SOLVE THE PROBLEM. OTHERWISE, BECAUSE THE DETAILED WORK NEVER BECAUSE WE THINK DIFFERENTLY. FOR EXAMPLE, THE DATA RIGHT NOW EVERYBODY IS TALKING ABOUT THE DATA. DATA IS VERY IMPORTANT FOR DIAGNOSTIC. FOR THERAPEUTIC TREATMENT WE NEED REAL-TIME DATA NOT DATABASE. THE SYSTEM NEEDS TO BE GEARED UP BECAUSE WHEN EACH MODEL BUILD UP, EVERY TIME WHEN PATIENTS COME IN YOU CAN TREAT IMMEDIATELY. FOR EXAMPLE, I'M A ROBOT, IF THE PATIENT COMES IN, HEART ATTACK, I HAVE TO KNOW EXACTLY THE DOSAGE WHAT IS HIS AGE AND OTHER FACTORS IN THERE. THAT'S CALCULATION DEPENDS ON A.I. SUPER COMPUTING. THERE'S NO TIME TO ANALYZE ALL THE DATA IN THERE. THAT'S ONLY FOR THOSE GAPS AND THE REFERENCE WILL BUILD UP AND THE ROBOTIC SYSTEM. RIGHT NOW THE DATA, EVERYTHING IN THERE THERE'S A LOT OF CLUSTERS THAT WASTE A LOT OF TIME. THOSE ARE GOOD FOR DIAGNOSTIC ONLY FOR A CERTAIN TIME PERIOD EVENTUALLY IN THE NEAR FUTURE ALL THE REAL DATA COLLECT WILL BUILD UP FRESH NEW DATABASE EVERY TIME ROBOT WILL USE THE NEW DATABASE. THANK YOU VERY MUCH. >> THANK YOU ALL VERY MUCH FOR YOUR PANEL. WE CAN GIVE A ROUND OF APPLAUSE. WE'RE GOING TO BE STARTING RIGHT BACK UP AT 10:30. >> GOOD MORNING, EVERYBODY. THANK YOU FOR BEING HERE. IT'S A PLEASURE TO BE HERE IN THE COMPANY OF SO MANY PEOPLE INTERESTED IN THE FIELD MOVING IT FORWARD, SHAPING IT. I'M EXCITED FOR THE NEXT SET OF TALKS I'M SHINJINI KUNDU. YOU MAY WONDER WHERE MY INTEREST IN A.I. COMES FROM BEFORE I WENT TO MEDICAL SCHOOL I WENT TO ENGINEERING AND HAVE A Ph.D. IN ENGINEERING AND I MERGED MY INTEREST IN ARTIFICIAL INTELIGENCE AND RADIOLOGY AND FINDING NEW EVIDENCE OF DISEASE AND PROGNOSTICATION FROM IMAGES BUT HUMANS HAVE NOT BEEN ABLE IT FIND IT THROUGH CONVENTIONAL METHODS. I'D LIKE TO INTRODUCE THE OTHER PEOPLE IN THE SESSION WE HAVE COLIN WALSH HE'S AN ASSISTANT PROFESSOR OF BIOINFORMATICS AT VANDERBILT UNIVERSITY MEDICAL CENTER. WE HAVE MERKMAN SAHINER AND WE IS A SENIOR VICE PRESIDENT SANJI FERNANDO. THAT SAID, TODAY I'LL TALK ABOUT NOT SO MUCH A ROAD MAP FOR A.I. BUT I'M INTERESTED IN EXPLORING WHAT WOULD BE LEFT IF WE WERE ABLE TO GET EXPLAINABILITY FOR HEALTH CARE APPLICATIONS. IT'S NOT MEANT TO BE A FINAL RECOMMENDATION BUT MORE SO EXPLORING DIFFERENT AREAS. SO ARTIFICIAL INTELLIGENCE OBVIOUSLY A VERY BROAD FIELD AND COMPRISES MANY DIFFERENT AREAS OF WHICH AND DEEP LEARNING IS ONE OF THE MOST EMERGING AREAS AND DEEP LEARNING IS A CATEGORY OF NEURAL NETWORKS. NEURAL NETWORKS IS A CATEGORY OF MACHINE LEARNING. MACHINE LEARNING IS KIND OF A CATEGORY WITHIN ARTIFICIAL INTELLIGENCE. SO WE'RE TALKING ABOUT A BROAD FIELD HERE BUT I THINK IT'S FITTING TO BE HERE AT THE NIH TALKING ABOUT NEURAL NETWORKS AND MACHINE LEARNING BECAUSE ACTUALLY THE FIRST DIAGRAM WAS CREATED OF BY A PHYSICIAN OF THE NEURAL NETWORK AND ABOUT 100 YEARS AGO HE WAS A PATHOLOGIST AND INTERESTED IN THE STRUCTURE OF THE MUM -- HUMAN BRAIN AND LOOK AT PATHOLOGY SLIDES OF THE BRAIN AND HISTOLOGY AND FOUND THE BRAIN TISSUE IS COMPRISED OF A HIGHLY CONNECTED MESH OF STRUCTURES WHICH WE KNOW TODAY ARE CALLED NEURONS BUT AT THE TIME IT WAS AN UNKNOWN FACT IT WAS COMPRISED OF NEURONS AND DENDRITES AND DEN DRONES. AT THE TIME HE COULDN'T HAVE IMAGINED THE APPLICATION OF HIS DISCOVERY TO OTHER FIELD BUT ABOUT HALF A CENTURY LATER INDEPENDENTLY THE FIELD OF COMPUTER SCIENCE WITH ADVANCES IN ELECTRONICS AND COMPUTER SCIENCE -- SCIENTISTS WERE APPLYING MODELS AND MODELS WERE MORE SIMPLISTIC AND WERE FOCUSSED ON IF STATEMENTS AND IN THE ANALYSTS WERE LOOK TRYING TO FIGURE OUT WAYS TO EXTRACT PATTERNS AND TRY TO FIND MEANING FROM INFORMATION IN WAYS THAT HUMANS COULD DO AND HUMANS WERE ALREADY GOOD AT. THEY LOOKED AT THE STRUCTURE OF THE HUMAN BRAIN AND DESIGNED A KNEW ALGORITHMIC APPROACH CALLED THE NEURAL NETWORK AND THAT REALLY DESCRIBES THE WAY INFORMATION IS SYNTHESIZED AND COMBINED COMPUTATIONALLY TO ARRIVE AT CERTAIN RESULTS SO WE STARTED HAVING NEURAL NETWORKS IN COMPUTER SCIENCE. SINCE THEN FIELD OF NEURAL NETWORKS HAS GROWN AND DEEP LEARNING AS WE MENTIONED ONE OF THE MOST EMERGING TECHNOLOGIES AND APPLIED WIDELY IN FIELD FROM NATURAL LANGUAGE PROCESSING TO VIDEOS AND IMAGES AND SO WE HAVE SEEN THIS FIELD TAKING OFF IN COMPUTER SCIENCE AND WHAT WE'RE HERE FOR TODAY IS TO CLOSE THE LOOP BACK TO MEDICINE AND SEEING HOW THESE ADVANCES COULD COME BACK AND HELP PATIENTS IN HEALTH CARE. WHY WE CARE ABOUT A.I. EXPLAINABILITY ESPECIALLY IN THE CONTEXT OF HEALTH CARE. I THINK IT COMES DOWN TO PRACTICING MEDICINE AND HEALTH CARE IN A SAFE WAY. I DON'T THINK IT'S A QUINCE DENSE THE FIRST THREE LAWS -- COINCIDENCE IS THE FIRST THREE LAWS ARE DO NOT HARM TO A HUMAN THROUGH ACTION OR INACTION OF THE ROBOT AND THE HIPPOCRATIC OATH WHICH IS FIRST DO NO HARM IS THE SAME. I THINK WE'RE AT A CONVERGENCE OF THE TWO FIELDS WHERE MITIGATING HARM AND RISK TO PEOPLE IS AT THE FOREFRONT. NOW, IN TERMS OF A.I. EXPLAINABILITY, IT'S A BROAD TERM. IT'S A BIG NOTION BUT WHAT I THINK OF A.I. EXPLAINABILITY IT'S EXPLAINABLE IF YOU LOOK AT HOW THE MATH WORKS ON THE INSIDE. A NEURAL NETWORK CAN BE LOOKED AT AND THAT'S EXPLAINABLE BUT THERE'S NO WAY FOR HUMANS TO INTERPRET THAT AND IT'S JUST BEYOND EXPLAINING MATHEMATICALLY. IT COMES DOWN TO A TRANSLATION BETWEEN MACHINE LOGIC TO HUMAN LOGIC IN A WAY HUMANS CAN UNDERSTAND IT'S ABOUT A TRANSLATION. WITH THAT EXAMPLE I WANTED TO WANTED TO TALK ABOUT EXPLAINABILITY AND THE FIELD I WORK ON HERE WE LOOK AT OSTEOARTHRITIS AND 1 IN 10 OF US WILL DEVELOP THIS IN OUR KNEES. CURRENTLY DOCTORS HAVE NO WAY TO DIAGNOSIS IT UNTIL IT'S DEMONSTRATABLE IN THE X-RAYS BUT IF WE GO BACK THREE YEARS WHERE A PERSON IS HEALTHY AND LOOK AT CARTILAGE IMPATHS THEY LOOK LIKE THIS. SO THERE'S ONE IMAGE PER PERSON AND THESE WERE OBTAINED BY AN OPEN SOURCE DATA SET. ALL THEE PEOPLE ARE COMPLETELY HEALTHY TODAY BUT WE KNOW 1 WIN DEVELOP IT IN THREE YEARS AND OTHERS WON'T BUT AT THE TIME POINT ZERO WE'RE LOOKING AT THE IMAGES IT'S NOT EASY TO TELL AND IT'S NOT REALLY AN EASY PROBLEM. NOW, THIS KIND OF PROBLEM IS NOT UNIQUE TO OSTEOARTHRITIS IN CARTILAGE. WE SEE THIS GENERAL IDEA WHERE WE HAVE A SET OF DATA AND HAVE MAYBE OUTCOMES OF THE FUTURE AND WE'RE TRYING TO LEARN THE RELATIONSHIP BETWEEN THE TWO. WE SEE THAT IN MANY FIELD. WE CAN SEE THAT IN CANCER DETECTION FOR MOLDS AND RETIN OPATHY. WE DEVELOPED A TECHNIQUE WHICH IS A GENERATIVE TECHNIQUE THAT USES MACHINE LEARNING TOO TRY TO BASICALLY LEARN WHAT A COMMON PATTERN IS IN DIFFERENTIATING THE GROUP AND WE FOUND AFTER TRAINING THE MODEL IT WAS ABLE TO PREDICT WITH 86% ACCURACY WHETHER SOMEONE WOULD DEVELOP THIS IN THREE YEAR'S TIME. IT SOUNDS IMPRESSIVE BECAUSE IF YOU CAN PREDICT A DISEASE BEFORE IT DEVELOPS MAYBE YOU CAN CHANGE THE TRAJECTORY. THE NEXT STEP GOES TO EXPLAINABILITY. WHAT IS IT LOOKING AT HUMANS AREN'T ABLE TO SEE FROM THE IMAGE. THAT'S A QUESTION AT THE FOREFRONT OF EVERYBODY'S MINDS. WAS IT'S GENERATIVE THIS IS THE DECISION BOUNDARY. THIS IS THE RULE THE ALGORITHM HAS LEARNED TO DIFFERENTIATE THOSE WHO IMMIGRATION TO OSTEOARTHRITIS AND THOSE WHO DO. THE MIDDLE IMAGE IS THE MEAN AND EACH IS TWO STANDARD DEVIATIONS TOWARDS EACH RESPECTIVE SIDE. THE FAR MOST IMAGE ON THE RIGHT DEVIATIONS TOWARDS THE NON-AFTER YO -- OSTEOARTHRITIS SIDE AND THE RED REPRESENTS INCREASED WATER AND SO WE KIND OF SEE THERE'S A POOLG OF WATER IN THE CENTER OF THE IMAGE AND WE CAN USE THAT TO PREDICT WHO WILL DEVELOP OSTEOARTHRITIS IN THE FUTURE. I'D LIKE TO TALK ABOUT IS A.I. EXPLAINABILITY AS THE FINAL DESTINATION. I ARGUE I THINK IT OPENS UP A COUPLE NEW QUESTIONS I WANT TO TOUCH ON. THIS IS A PICTURE OF CHIHUAHUAS AND MUFFINS AND MACHINE LEARNING SYSTEMS MAY HAVE DIFFICULTY DIFFERENTIATING THESE. WE AS HUMANS HAVE THE CONTEXT THAT DOGS ARE ANIMATED CREATURES THAT MAKE NOISE AND MUFFINS ARE EDIBLE THINGS. TO MACHINE LEARNING THESE ARE STATELESS IMAGES AND HAVE NO OTHER CONTEXT THAT N WHAT IS SHOWN. CONTEXT IS EVERYTHING. HOW DO WE MAKE SURE WHAT IT PICKED ON IS THE RIGHT THING TO PICK UP ON IN THE CONTEXT OF WHAT'S IMPORTANT? FINALLY, WE TALKED ABOUT CAUSALITY IN THE LAST SESSION BUT WE NEED TO BE CAREFUL WHEN WE TALK ABOUT A.I. EXPLAINABILITY TO NOT CONFUSE IT AS A CAUSAL EXPLAINABILITY. INE A.I. SYSTEM MAY COME UP WITH A WAY TO EXPLAIN THE DATA. IT DOESN'T CAUSE IT AND THERE MAY BE A DEEPER EXPLANATION WE NEED TO UNCOVER STILL. THOSE ARE TWO AREAS WHICH WE NEED TO EXPLORE EVEN BEYOND A.I. EXPLAINABILITY. THAT I THINK WOULD BE THE NEXT DESTINATION AFTER A.I. EXPLAINABILITY AND WITH THAT I HAVE NO DISCLOSURES AND THANK YOU FOR YOUR ATTENTION. I'D LIKE TO INTRODUCE THE NEXT SPEAKER BERKMAN SAHINNER. >> THANK YOU VERY MUCH. IT'S A GREAT PLEASURE TO BE PRESENTING HERE. I'M WITH THE FDA THE Ph.D. SENIOR BIOMEDICAL RESEARCH AND I HOPE THIS IS IMPORTANT AND HOPE IT WILL BE COVERED IN OTHER TALKS. SO NEW TECHNOLOGY IN SCIENCE OFTEN STATES A NEW VOCABULARY AND NOT NECESSARILY THE CREATION OF NEW WORDS BUT MORE OFTEN APPROPRIATION OF EXISTING WORDS AND SPECIFIC TECHNICAL CONTEXT. FOR EXAMPLE, BEFORE THE TIME OF KNUDSEN THERE WAS NOT THE PRECISE MEANING WE ATTRIBUTE TODAY LIKELY YOU SEE WORDS ON THE SCREEN PEOPLE IN A.I. AND MACHINE LEARNING TALK ABOUT AND SOME OF THEM DO NOT YET HAVE VERY CRISP ZEV -- DEFINITIONS SO IT'S IMPORTANT FOR MEETINGS LIKE THIS TO BRING TOGETHER PEOPLE FROM DIFFERENT FIELD AND MEDICINE, ENGINEERING, COMPUTER SCIENCE, SO TOGETHER WE CAN PROVIDE MORE CRISP DEFINITIONS SO TWO WORDS YOU SEE ARE EXPLAINABILITY AND INTERPRETABILITY AND IN EARLY TALKS NIGAM MADE A DISTINCTION AND IN MY TALK I WILL NOT. WE HAVE TO DISCUSS AND COME TO A DEFINITION ABOUT WHAT WE MEAN BY TERMS LIKE THIS. ROUGHLY I'LL MEAN A.I. REVEALING UNDERLYING CAUSES TO THE DECISION MAKING BOTH FOR EXPLAINABILITY AND INTERPRETABILITY. I'LL MAKE A DISTINCTION BETWEEN DESTINATION AND EXPLAINABILITY. AS NIGAM SAID AN EXPLANATION IS TYPICALLY AN ANSWER TO A WHY QUESTION. FOR EXAMPLE, WHY DID THE A.I. SYSTEM DIAGNOSE THIS CASE WITH PNEUMONIA AND WHY QUESTIONS ARE CONTRASTED SO WHEN YOU HAVE THE FIRST QUESTION AT THE TOP YOU'RE ASKING PERHAPS WHY DID THIS A.I. SYSTEM DIAGNOSE THIS WITH PNEUMONIA BUT NOT PLEURAL INFUSION. AN EXPLANATION CAN BE EVALUATED IN TWO WAYS ACCORDING TO INTERPR INTERPRETABILITY AND COMPLETENESS AND INTERPRETABILITY IS A WAY THAT'S UNDERSTANDABLE TO HUMANS SO HUMANS IS KEY AND TIED TO THE COGNITION AND BIAS OF THE USER. COMPLETENESS IS DESCRIBING THE COMPLETENESS OF A SYSTEM IN AN ACCURATE WAY AND YOU CAN ALWAYS PROVIDE A COMPLETE DESCRIPTION OF THE SYSTEM BY REVEALING ALL THE MATHEMATICS BUT IT COULD BE INTERPRETABLE AND EXPLAINABILITY SO THERE'S A TRADE-OFF BETWEEN EXPLAINABILITY OR INTERPRETABILITY AND COMPLETENESS. SO THE EXAMPLE THAT I GAVE EARLY YOU CAN HAVE VERY COMPLETE AT THIS BOTTOM RIGHT BUT IT CAN BE VERY INTERPRETABLE. THERE'S THIS PROBABLE TRADE-OFF. SO WHEN ASKED TO PROVIDE AN EXPLANATION, PEOPLE OFTEN PROVIDE DIFFERENT LEVELS OF EXPLANATION DEPENDING ON WHO WAS ASKING THE EXPLANATION AND WHAT THEY BELIEVE ARE THE MOST RELEVANT CAUSES. SO LIKEWISE IN HEATH A.I. WE HAVE DIFFERENT USERS AND THERE ARE MAYBE A NEED FOR DIFFERENT DEPTH FOR EXPLANATION SO THE USER CAN BE THE PATIENT AND CAREGIVER AND SCIENTIFIC COMMUNITY AND REGULATORY AGENCIES AND THE EXPLAINABILITY. THE LEVEL OF EXPLAINABILITY MAY BE DIFFERENT DEPENDING ON WHO THE USER IS. NOW, I WILL GO INTO A LITTLE BIT OF TECHNICAL DETAIL TO GIVE SOME DISCUSSION ABOUT TYPES OF EXPLANATION IN DEEP NETWORKS. I SAID DEEP NETWORKS BECAUSE THAT'S ALL THE RAGE TODAY BUT I THINK IT ALSO APPLIES TO THE MORE CONVENTIONAL A.I. AND MACHINE LEARNING SYSTEMS TOO. IT'S MAINLY BASED ON THE PAPER THAT GAVE A TAXONOMY OF EXPLANATIONS IN TERMS OF EXPLAINING THE PROCESSING OF THE DATA AND REPRESENTATION OF THE DATA AND CREATING EXPLANATION PRODUCING SYSTEMS AND TRY TO USE EXAMPLES OF IMAGING. FIRST IS THE PROCESSING OF THE DATA. THIS IS MOST COMMON AND INCLUDES PROXY MODELS AND INTERPRETABLE MODEL THAT BEHAVES TO A MORE COMPLEX MODEL IN A LOCAL NEIGHBORHOOD AND INCLUDES SYSTEMS LIKE OCCLUSION STUDIES AND SALIENCY AND CLASS ACTIVATION MAP THAT POINTS TO WHICH COMPUTATION IS MORE RELEVANT AND THIS IS THE LINEAR PROXY MODEL. I DON'T THINK I HAVE HERE IN THE HERE YOU SEE THE ORIGINAL IMAGE AND THE SYSTEM IS TRYING TO CHARACTERIZE THEM AS BELONGING TO DIFFERENT ANIMALS AND THIS SYSTEM WHAT IT DOES IS FIRST DECOMPOSES THE ORIGINAL IMAGE INTO INTERPRETABLE COMPONENTS AND RANDOMLY PICK THE COMPONENTS AND INPUTS THEM IN THE NEURAL NETWORK AND THEN SEES WHAT THE OUTPUT IS. SO YOU DEVELOP A DATA SET OF THE INPUTS AND THE OUTPUTS LIKE THIS AND THEN YOU FIT A LINEAR MODEL TO THIS AND THEN KEEP ONLY COMPONENTS THAT HAVE A HIGH WEIGHT IN THE LINEAR MODEL. THEN THE EXPLANATION IS MOSTLY THIS FROG'S FACE BUT IT ALSO CHOOSES THESE REGIONS WHICH ARE MAYBE NOT SO RELATED TO BEING A FROG. THIS MAY ALSO LEAD YOU TO ASK QUESTIONS ABOUT WHY WERE THOSE REASONS PICKED OCCLUSION VISUALIZATION. THIS HAS PLEURAL INFUSION AND THE DISEASE REGIONS. YOU TAKE THE NETWORK AND OCCLUDES PART OF THE IMAGE AND SEE WHAT THE NETWORK GIVES WHEN THOSE PARTS ARE OCCLUDED. YOU CAN SEE WHEN THIS PART OF THE IMAGE IS OCCLUDED THERE'S A DROP IN THE SYSTEM SAYING THIS IS THE DISEASE AND IT GIVES YOU AN IDEA WHERE THE SYSTEM LOOKS AT TO GIVE THAT DECISION. SALIENCY MAPS AGAIN WE HAVE AN INPUT IMAGE AND DEEP LEARNING SYSTEM THAT GIVES US A PREDICTION OF A TRACTOR. THIS IS THE FORWARD PREDICTION AND IN NEURAL NETWORKS WE KNOW WHERE TO DO IT WITH RESPECT TO WEIGHTS HERE WITH TO THE PIXELS THIS IS THE IMAGE YOU GET AND THIS REGION AT THE BOTTOM CONTRIBUTED MOSTLY TO DECIDING THIS SO THIS IS SOME KIND OF EXPLAINABILITY. ANOTHER IS CLASS ACTIVATION MAPS. I WON'T GO THROUGH IT FOR THE SAKE OF TIME AND EXPLAINING THE DATA AND OTHER METHODS OF EXPLAINABILITY TO UNDERSTAND THE ROLE AND STRUCTURE OF THE NETWORK COMPONENTS BY LAYER, UNIT OR ACTOR THIS IS THE VISUALIZATION OF THE FIRST LAYERS OF SOME POPULAR ARCHITECTURE FOR THE IMAGE NET DATA AND THE FIRST LAYER YOU CAN SEE THERE'S MULTIPLE FILTERS THAT DO THINGS LIKE DETECTION ETCETERA. IT GIVES SOME IDEA WHAT THE NETWORK IS DOING AND YOU CAN GO TO MORE DEPTH AND DO NETWORK DIE SECTION AND DO THAT FOR MANY LAYERS. I WON'T GOU THROUGH DETAILS AND THE FINAL IS CREATING EXPLANATION PRODUCING SYSTEMS. THIS COULD BE DISTANGLED REPRESENTATIONS SIMILAR TO ANALYSIS OR INDEPENDENT COMPONENT ANALYSIS OR EXPLAINING BETWEEN NETWORKS TO GENERATE EXPLANATIONS SO THE REPRESENTATIONS WHAT YOU DO IS YOU TRAIN AT A MODIFIED LOSS FUNCTION THAT FORCES DIFFERENT UNITS TO FOCUS ON DIFFERENT PARTS OF A GIVEN CLASS AND THEN THESE ARE THE OUTPUTS. THESE ARE THE RECEPTIVE FIELD OF SOME COMPONENTS OF THOSE FILTERS WHEN YOU TRAIN WITH AN INTERPRETABLE CNN AND WITHIN A REPRESENTATION YOU CAN SEE THE RECEPTIVE FIELDS FOCUS MOSTLY ON THE FACES AND FEATURES THAN THE RANDOM NETWORK AND FINALLY NETWORKS TRAINED TO GENERATE EXPLANATIONS. THIS IS A SYSTEM DESIGNED TO CLASSIFY DIFFERENT KINDS OF BIRDS. AS INPUT YOU NOT ONLY GIVE AN IMAGE AND WHAT CLASS IT BELONGS TO BUT A BRIGHT RED BIRD WITH AN ORANGE BEAK AND THE SYSTEM CAN GENERATE OUTPUTS LIKE THIS BECAUSE IT HAS A WHITE BIRD WITH ORANGE FEET AND FINALLY I WANT TO TALK ABOUT EVALUATING EXPLANATIONS AT THE FDA WE'RE FOCUSSED ON EVALUATION AND WHEN YOU CAN MEASURE WHAT YOU'RE SPEAKING ABOUT YOU KNOW SOMETHING ABOUT IT SO HOW CAN WE MEASURE THE EXPLANATION GIVEN BY AN A.I. SYSTEM. THIS IS TAKEN FROM THIS PAPER AT THE BOTTOM JUST TO GIVE YOU AN IDEA ABOUT SOME FRAMEWORK FOR DOING THIS AND IT HAS THREE LAYERS AND I LIKE THIS BECAUSE IT'S VERY SIMILAR TO WHAT WE DO AT THE FDA FOR EVALUATING SYSTEMS DEPENDING ON THE RISK AND ALSO HOW WELL THE DEVICE AND THE CONNECTION BETWEEN THE DEVICE AND THE MEDICAL CONDITION IS UNDERSTOOD. AT THE TOP IS APPLICATION GROUNDED, REAL USERS, REAL TASK. EVALUATE THE QUALITY OF AN EXPLANATION IN THE CONTEXT OF THE TASK. FOR EXAMPLE, WHETHER SHOWING THOSE EXPLANATIONS TO THE USERS REALLY RESULTED IN BETTER IDENTIFICATION OF ERRORS, ETCETERA. THIS IS THE HIGHEST LEVEL. IT'S VERY GOOD BUT IT ALSO CAN BE VERY RESOURCE INTENSIVE. SECOND, IT'S HUMAN GROUNDED NOT REAL USER BUT ANY HUMAN AND SIMPLIFIED TASK AND THE THIRD IS PROXY TASK SIMILAR TO, FOR EXAMPLE, MEASURES OF FIZZ -- PHYSICS TO LOOK AT THE PERFORMANCE OF SOME DEVICES BECAUSE SOME ARE WELL UNDERSTOOD AND THE CONNECTION BETWEEN THE MEASUREMENTS AND THE DISEASE CONDITION IS WELL UNDERSTOOD. SO THERE CAN BE A HIERARCHY OF EVALUATIONS LIKE THIS BUT I THINK EVALUATION OF EXPLANATIONS IS VERY IMPORTANT SO IN SUMMARY, CRISP DEFINITIONS OF THE TOPICS DISCUSSED IN THE WORKSHOP WILL ADVANCE THE FIELD AND DIFFERENT LEVELS OF EXPLAINABILITY MAY BE NEEDED DEPENDING ON THE USERS AND THE DESIRED DEPTH OF EXPLANATION AND THE LITERATURE FOR EXPLAIN TO HUMANS AND INTERPRETABILITY AND COMPLETENESS ARE COMPETING FACTORS AND METHODS TO EVALUATE EXPLAINABILITY NEED FURTHER ATTENTION. SO WITH THAT I BELIEVE THE NEXT SPEAKER IS DR. FERNANDO. >> GREAT. I'M SANJI FERNANDO AND I WILL TALK ABOUT OUR JOURNEY IN DEPLOYING DEEP LEARNING MODELS. TODAY I LEAD OUR A.I. ANALYTICS PLATFORM TEAM. UP UNTIL TWO, THREE WEEKS AGO I WAS WITH OPTUM LABS WHERE WE FIRST BUILT OUR DEEP LEARNING MODELS. AS I GO INTO MY TALK I WANT TO SET A LANDSCAPE. WE'VE BEEN THINKING HOW MACHINE INTELLIGENCE WILL CHANGE HOW WE MAKE CLINICAL DECISIONS AND IMPORTANT AND SOMETHING WE'RE PURSUING AT OPTUM BUT WE STARTED LOOKING AT HOW TO APPLY THIS FOR ADMINISTRATIVE PURPOSES STILL PART OF OUR WORK FLOW TODAY AND SYSTEM AND SEEMS TO BE THE RIGHT PLACE TO START. WE HAD LOTS OF DATA AND NEED. AT THE SAME POINT WE HAD A STRONG NEED TO UNDERSTAND AND EXPLAIN HOW THE MODELS WORK. SO THE CASE STUDY I'LL TALK ABOUT IS WHAT WE BUILT TODAY IN PRODUCTION WITH A NUMBER OF POSSIBLE SYSTEMS AS PART OF OUR REVENUE CYCLE BUSINESS. OUR SOMEWHAT ARCANE PROCESS IS WHEN YOU'RE ADMITTED FOR AN IN-PATIENT STAY IN CERTAIN CIRCUMSTANCES A PAYER WILL ASK FOR A THIRTY-PARTY TO REVIEW THE CHART FOR A LEVEL OF REIMBURSEMENT. THIS IS A SERVICE WE HAVE TODAY AND SO IT CAN BE A LABORIOUS EFFORT WE ASKED SKILLED CLINICIANS TO PROVIDE THOSE INDEPENDENT ASSESSMENTS AND THEY COMBINE NOT SIMPLY A SET OF CLINICAL GUIDELINES SOMETIMES REPRESENTED BY RULES BUT THEIR OWN REAL WORLD EXPERIENCE. IT'S AN INVALUABLE PART OF OUR HEALTH CARE REIMBURSEMENT PROCESS AND WE ASKED THE QUESTION, CAN WE MAKE THE WORK MORE EFFICIENT AND TAKE ADVANTAGE OF THEIR COLLECTIVE KNOWLEDGE DIFFERENTLY AND SOME ASPECTS OF DEEP LEARNING WE SAW AS IT FIRST EMERGED, IT'S REALLY BROKEN OUT IN THE LAST FIVE YEARS. FOR US, OUR JOURNEY STARTED TWO AND A HALF YEARS AGO AND IT STRUCK US WE HAD A GREAT DEAL OF INFORMATION. WE HAD THE MEDICAL RECORDS AND MORE IMPORTANTLY THE CLINICAL DECISIONS MADE ON THE MEDICAL RECORDS THEY FELT THEY WERE THE RIGHT INGREDIENTS FOR US TO TAKE ADVANTAGE OF THE NEW BREAKTHROUGHS. SO WE TRAINED A DEEP LEARNING MODEL ON OVER 200,000 MEDICAL NODES. THE MODEL'S DESIGN TO PRE-READ EVERYTHING FOR PHYSICIANS TO DECIDE THE MOST IMPORTANT CHARTS TO REVIEW. AS OTHERS HAVE TALKED ABOUT WE'RE LEARNING FROM THE NODES THE INPUT, AND WE HAVE THE ANSWER KEY ALMOST BY WHAT THE PHYSICIAN HAS DECIDED PREVIOUSLY. WE'VE USED A VARIATION OF WHAT'S CALLED THE RECURRENT NEURAL NETWORK WHICH OFTEN TIMES WORKS WELL WITH UNSTRUCTURED TEXT THOUGH WE'RE EXPLORING OTHER MODEL ARCHITECTURES LIKE CNNs FOR THIS TODAY. AND AS YOU THINK ABOUT HOW YOU USE MACHINE INTELLIGENCE YOU SHOULD UNDERSTAND WE'RE GENERATING A MATHEMATICAL SCORE. AT SOME POINT YOU'LL HAVE TO SET A THRESHOLD TO DECIDE WHAT TO DO WITH THE SCORE. WILL IT BE A BINARY CLASSIFICATION, SOMETIMES A MULTI-LABEL CLASSIFICATION. ULTIMATELY THEY'RE MATHEMATICAL EQUATIONS WHERE YOU HAVE TO DECIDE WHAT YOU'LL DO AND WHAT ACTION YOU'LL TAKE. THAT'S WHAT WE RAN INTO. THE WORK FLOWS RELY ON SKILLED PHYSICIANS AND WE HAVE TO UNDERSTAND HOW MODELS WORK BUT IN TERNTLY WE HAVE THIS -- INHERENTLY WE HAVE THE BLACK BOX PROBLEM AND HAVE TO TURN THE NODES INTO THESE PASSING THROUGH A SET OF INTERCONNECTED EQUATIONS AND THEY'RE EVALUATED AND PASS THEIR OUTPUT TO ANOTHER SET OF EQUATIONS. IT BECOMES VERY COMPLEX FOR I THINK AS SOMEONE SAID EARLIER FOR A HUMAN TO UNDERSTAND HOW THAT NETWORK OF EQUATIONS IS WORKING. THE MODEL ITSELF DOESN'T NECESSARILY IDENTIFY FOR US WHAT SPECIFIC ASPECTS OF THE MEDICAL NOTES OR THE MEDICAL GUIDELINE IT MAY HAVE REFERRED TO FOR THE DECISION. ALL WE HAVE IS THE INPUT FOR THE NOTE AND THE PHYSICIAN'S DECISION. AS WE ASKED OUR CLINICIANS TO HELP EVALUATE WHETHER THE MODEL IS HELPFUL IN THE WORK FLOWS THE QUESTION IS IT LOOKS LIKE IT'S WORKING BUT HOW DO WE KNOW IT'S WORKING? HOW DO I KNOW IT'S NOT KEYING OFF SOME RANDOM CHARACTER IN THE NOTES THAT MAY HAVE BEEN LEFT BEHIND BECAUSE OF SOME INFORMATION INFORMATIONAL LEAKAGE IN THE COMPUTER SYSTEM. THEY DOESN'T UNDERSTAND THAT. FOR US TO BUILD THE TRUST THIS COULD IMPROVE THE WORK FLOW WE HAD TO FIND A WAY TO HIGHLIGHT WHAT WAS ACTUALLY HAPPENING WITHIN THE NETWORK OF EQUATIONS. WE IMPLEMENTED A MODEL CALLS ATTENTION. WE TALKED IN THE PREVIOUS DISCUSSION ABOUT A COUPLE DIFFERENT WAYS TO OFFER EXPLAINABILITY THROUGH COMPLETENESS AND WITH TENSION WE'RE TRYING TO GET A WINDOW IN HOW THE INPUTS WE'RE PASSING THROUGH THIS COMPLEX SET OF EQUATIONS IS ESSENTIALLY WEIGHTING ITSELF AND WHAT ARE THE INTERIMMEDIATE -- INTERMEDIATE WEIGHTS AND SOME MAY SEIZE THAT INFORMATION IN A NUMERIC FORMAT BUT NOT COMBINE IT AND WE HAD VISUAL TOOLS TO HELP PEOPLE UNDERSTAND WHAT WAS HAPPENING IN SCORING THE NOTES. WE CREATE THE INTERFACES THAT BEGAN TO TAKE THE ATTENTION WEIGHTS AND THE TESTS THAT WERE DRIVING SOME OF THE DECISION OR SCORING IN OUR NEURAL NETWORK AND THIS WAS AN IMPORTANT BREAKTHROUGH FOR US BECAUSE AT THIS POINT WE FINALLY WERE ABLE TO EN GENDER THE TRUST OF THE CLINICIANS WHOSE WORK FLOW WE WERE TRYING TO IMPROVE. WE LOOKED AT THE ASPECTS OF THE MEDICAL NOTE TEXT THEY HAD TO READ EVERY DAY AND IT WOULD KEY ON THE SAME THING I WOULD SEE WHEN APPLIED THE CLINICAL EXPERTISE AND EVALUATE THE NOTES. HERE YOU COULD SEE WE USE AGAIN A WHOLE BUNCH OF SCORES AND APPLY A GRADIENT OF COLOR TO UNDERSTAND WHAT MIGHT BE FIRING WITHIN THE NETWORK. WE ALSO CREATED VISUAL TOOLS TO HELP PEOPLE UNDERSTAND HOW THEEL PERFORMED AND WHEN IT GOT THINGS RIGHT AND WRONG AND THIS IS ANOTHER PART OF EXPLAINABILITY BECAUSE PEOPLE WANT TO UNDERSTAND WHAT THE TRADE-OFFS WERE. IN A REIMBURSEMENT SETTING UNDERSTANDING THE IMPACT SHOULD BE APPROPRIATELY REIMBURSED OR NOT. OFTEN TIMES FOR MANY DATA SCIENTISTS IT'S A COMMON FRAMEWORK AND WE'RE ABLE TO HELP NON-DATA SCIENTISTS UNDERSTAND THE TRADE-OFFS AND MAKE DECISIONS HOW TO USE THE MODEL. >> PROVIDING A BROAD SET OF UNDERSTANDING DRIVING FOR SOME OF THAT COMPLETENESS SO PEOPLE CAN UNDERSTAND HOW THE DATA'S TRAINED AND HOW WE TRAIN THE MODEL AND WHAT ARE THE LABELS AND MODELS WE USE ARE IMPORTANT AND ATTENTION OR LIE OR SHAFT OR MANY OTHER MATHEMATICAL FRAMEWORKS CAN HELP US GET A SENSE OF WHAT FEATURES ARE DRIVING DECISIONS BUT IT'S NOT TRUE IN INTERPRETABILITY BUT IT'S ESSENTIAL EVERYONE'S CONFIDENT IT'S ACTING THE WAY INTENDED TO AND FOR THOSE WHO NOT BE SCIENTISTS TO UNDERSTAND THEY'RE GENERATING SCORES AND ULTIMATELY A PERSON AND AN EXPERT WILL NEED TO MAKE A DECISION AS TO WHERE TO SET THE THRESHOLD. WE HAVE TOOLS AND MATHEMATICAL WAYS TO OPTIMIZE FOR THAT BUT ULTIMATELY THEY'RE DECISIONS THAT FALL IN THE HANDS OF PEOPLE AND WE HAVE A LOT OF WORK AHEAD OF US. EXCITING WORK THAT'S PROBABLY GOING TO GIVE US MORE AND WE'VE BEEN SEGMENTING THE TRAINING OF THE MODEL AND LOOK TO SEE WHAT'S CONSISTEN ACROSS MULTIPLY TRAINED MODELS AND LOOKING AT A THEOREM THAT'S CENTURIES OLD. SO WE'RE VERY OPTIMISTIC WE'LL GET INTERPRETABILITY AND THAT WILL ALSO HELP US NOT ONLY WITH THE ADMINISTRATIVE WORK FLOWS AND THE CLINICAL DECISION MAKING WE KNOW IS IMPORTANT. >> BY PROCESS OF ELIMINATION I THINK IT'S MY TURN SO I'LL INTRODUCE MYSELF. I'M COLIN WALSH FROM VANDERBILT AND THANK YOU FOR INTRODUCING ME BEFORE, TO INTRODUCE MYSELF BY SHARING THREE PERSPECTIVES THAT HIGHLIGHT THE IMPORTANCE OF THE SESSION. MY FIRST PERSPECTIVE IS OF THAT A PRACTICING PRIMARY CARE PROVIDER WHICH MEANS WE'RE EXCITED ABOUT MACHINE INTELLIGENCE BUT INSOFAR WE CAN GET IT TRANSLATED TO THE POINT OF CARE AND BRING THE PERSPECTIVE OF A BIOINFORMATICS MATHEMATICIAN AND THAT BRINGS POWER AND POSSIBILITY AND POTENTIAL UNINTENDED CONSEQUENCES AND FINALLY BEFORE MY MEDICAL TRAINING I WAS TRAINED AS AN ENGINEER AND THAT'S A WORLD WHERE PERFECT IS OFTEN THE ENEMY OF GOOD. WITH THAT I'LL SHARE MY OBJECTS. THE FIRST IS TO HIGHLIGHT THE RELEVANCE OF EXPLAINABILITY AND THE DIFFERENCES BETWEEN INTERPRETABILITY AND EXPLAINABILITY AND I'LL MAKE EXPLICIT UP FRONT ESPECIALLY IN BEHAVIORAL HEALTH IT'S MORE ON INTERPRETABILITY AND HOW INSTEAD OF WHY. YOU'LL SEE THE WORLD EXPLAINABILITY BUT INTERPRETABILITY IS WHY WE'RE LIVING THESE DAYS. I'LL GIVE YOU EXAMPLES OF OUR STUDIES TO BRIDGE GAPS AND FINALLY I'LL SUGGEST TOPICS FOR DISCUSSION. THE ACTIVE INGREDIENT IS WE HAVE EXPERTS FROM VARIOUS DISCIPLINES AND MY HOPE IS I CAN LEARN MANY THINGS IN THIS CHALLENGING AREA WITH RESPECT TO MACHINE INTELLIGENCE. SO I'M GOING TO FOCUS ON A PARTICULAR PROBLEM. A PROBLEM WE'VE SPENT A LOT OF TIME THINKING ABOUT SUICIDE. BY THE END OF THE DAY IT STILL BLOWS ME AWAY EVERY DAY 123 MILLION AMERICANS DIE TODAY FROM SUICIDE. THAT'S DESPITE THE BEST EFFORTS OF PEOPLE AND IT'S A HUMBLING PROBLEM. THE RATES OF SUICIDE ATTEMPTS, FATAL OR NON-FATAL ARE HIGH. THE RATES OF CONTEMPLATION OF SUICIDE BRINGING ITS OWN MORBIDITY ARE HIGHER THAN THAT AND WE'VE PAR NERRED WITH -- PARTNERED WITH THE U.S. NAVY IN THINKING OF THE RISK OF SUICIDE AT VANDERBILT AND RATES CONTINUE TO GO UP LIKE THEY DO IN CIVILIANS DESPITE OUR BEST EFFORTS. WE CARE A LOT ABOUT IT AND WE'RE TRYING TO THINK THROUGH PREDICTIVE APPROACHES TO ENABLE PREVENTION AND I'LL USE THE INSTANCE TO PROVIDE PINS -- PRINCIPLES HOPEFULLY RELEVANT FOR A NUMBER OF PROBLEMS YOU ALREADY HEARD ABOUT. HIGHLIGHTING OUR APPROACH, BROAD STROKES, WHAT WE'VE DEVELOPED ARE A SET OF ALGORITHMS THAT TAKE THE ROUTINE TENDS TO BE STRUCTURED HEALTH CARE DATA WE COLLECT EVERY DAY TO DISCOVER GAPS AND SOMEBODY HAS TO THINK TO DO SCREENING TO ASK THE RIGHT QUESTIONS AND HOPEFULLY MAKE THE RIGHT INTERVENTION. SOME INDIVIDUALS WERE FALLING THROUGH THE CRACKS. OUR KNOWN SUICIDE ATTEMPTERS WERE NEVER SCREEN AND IT'S NOT A FAILURE OF THE CLINICIANS SOMETIMES IT'S A SYSTEMS LEVEL FAILURE. I HIGHLIGHT AN ARTICLE ABOUT THIS PARTICULAR WORK WHERE WE TAKE MACHINE INTELLIGENCE PLUS ROUTINE HEALTH CARE DATA TO DEVELOP A TORHORNY AND PREDICTABLE PROBLEM AND ALGORITHMS PREDICT RISK MORE THAN CLINICIANS. WE NEVER COMPARED. THE MACHINE AND HUMAN WERE PIVOTED AGAINST EACH OTHER AND WE USED CLINICIAN JUDGMENT TO TRY TO MAKE IT SMARTER. WE'RE FIRM BELIEVERS THE HUMAN WILL BE BETTER THAN THE COMPUTER BUT MAYBE THE COMPUTER PLUS THE PERSON IS BETTER THAN EITHER ALONE. I'M GOING TO TAKE THIS ONE SLIDE AND ONE TRADE-OFF YOU HEARD FROM MY COLLEAGUES IN TRADE-OFFS IN THINKING ABOUT INTERPRETABILITY ONE TRADE-OFF WE TEND TO THINK ABOUT IS THE TRADE-OFF BETWEEN PERFORMANCE AND HANDLING COMPLEXITY AS WELL AS EXPLAIN AND INTERPRETABILITY WHERE I CAN UNPACK THE MATH AND GIVE YOU AN EQUATION WE CAN UNDERSTAND THAT AND AS YOU GET CLOSER TO AS THEY SAY THE BLACK BOX WHICH IS MAYBE BETTER AT HANDLING COMPLEX ASSUMPTIONS AND INTERACTIONS WE HAVE TO THINK THROUGH WHERE WITH ARE WE ON THE OPERATING CHARACTERISTIC CURVE AND I WANT TO HIGHLIGHT THAT HERE AS TO WHY WE'RE TALKING ABOUT IT SO MUCH AND IN MENTAL AND BEHAVIORAL HEALTH. ONE OF THE THEMES IS PREDICTION IS NOT PREVENTION. WHAT DO WE DO NOW THAT WE HAVE ALGORITH ALGORITHMS THAT PEOPLE CAN PREDICT AND IT RAISES GREAT POTENTIAL AND PERIL AS WELL. SO THE WHY. TO MY MIND IT SETS UP THE CHALLENGES IN TERMS OF WHAT WE'RE DEALING WITH AND I THINK IT MAKES IT REAL TO UNDERSTAND WHY EXPLAINABILITY IS SO DARN IMPORTANT ESPECIALLY IN BEHAVIORAL HEALTH. I'M AN ADVOCATE OF A WRITTEN PAPER BY A STATISTICIAN IN 1976 ROTE THE PAPER AND THE -- WROTE THE PAPER AND IT'S REFERENCED ON THE RIGHT. WE KNOW MODELS ARE WRONG AND HOPE THEY'RE USEFUL. A KEY POINT IS IN THE EXCITEMENT AROUND MACHINE INTELLIGENCE AND DATA IT'S EASY TO PUT THE DATA BEFORE THE PROBLEM. YOU HAVE TO REMEMBER THE PROBLEM ALWAYS HAS TO COME BEFORE THE DATA. IN BEHAVIORAL HEALTH, MISCLASSIFICATION, ERRORS IN PREDICTION DOESN'T MEAN WE'LL TRY AGAIN NEXT TIME. IT CAN LEAD TO SELF-HARM OR LOSS OF LIFE. SOMETHING WE MAY FEEL OKAY ABOUT BASED ON A RECOMMENDATION CAN CAUSE SIGNIFICANT RISK. IT MAY LEAD ON THE FLIP SIDE TO UNNECESSARY TREATMENT. WE HAD TO PARTNER WITH AN ETHICISTS BECAUSE WE FELT WE WERE RUNNING INTO THORNY ISSUES AND THE TITLE OF THE PAPER IS BASICALLY PROTECTING LIFE WHILE PRESERVING LIBERTY INTEREST IF THERE'S UNVOLUNTARY HOSPITALIZATION WE WANT TO BE THOUGHTFUL IN HOW ALGORITHMS ARE ARRIVING AND THAT'S A STATE WE CAN ARRIVE IN IF WE'RE NOT THOUGHTFUL AND THERE'S STIGMA AND SCALE. IF WE CAN USE IT TO SAY THIS IS YOUR POTENTIAL PROBABILITY OF SUICIDE RISK AND YOU SHOWED UP BECAUSE YOU BROKE YOUR FOOT THERE'S A STIGMA AND POTENTIAL HARM. WE LIKE TO TALK ABOUT IT TO TACKLE THE STIGMA BUT WE HAVE THE ABILITY TO BE WRONG AND CAUSE HARM AT SCALE JUST LIKE WE CAN CAUSE GOOD. IN THE NAVY, THERE'S POTENTIAL CAREER IMPACTS. IF DEPRESSION AND SUICIDE IS IN SOMEBODY'S MEDICAL CHART IT CAN BE INCORPORATE INTO COMMAND DECISIONS AND WE HAVE TO BE THOUGHTFUL AND SOMETHING DISCUSSED IN THE FIRST PANEL IS ABOUT TRUST. AS NIGAM SAID THERE'S A SUR CAT FOR TRUST THAT'S NOT TRUST WE HAVE TO REBOUND IT DOES CONTRIBUTE. I WANT TO HIGHLIGHT THAT. A NUMBER OF OUR THOUGHTS IN THIS SPACE I'M NOT GOING TO UNPACK THOSE I WANT TO FOCUS ON TWO EXAMPLES OF STUDIES FOCUSSED ON THE INTERPRETABILITY TO UNPACK THIS BECAUSE FROM OUR PROVIDERS AS WE PARTNER WITH PROVIDERS WE FIND THERE'S A NEED TO SAY YOU'LL MAKE THIS RECOMMENDATION AND IT'S A HIGH STAKES RECOMMENDATION HELP ME UNDERSTAND HOW WE GET THERE. IN THIS STUDY WE'RE APPLYING TO ACTIVE DUTY MEMBER SAMPLE AND THERE'S AN APPETITE TO UNDERSTAND HOW WE GET THERE. AND THE MOST COMMONLY IMPLEMENTED MODELS ARE UNDER FIT AND THE THINGS YOU CAN DO IN THE BACK OF A NAPKIN FOUR OR FIVE RISK FACTORS I CAN LOOK DO IN MY HEAD AND GIVE RECOMMENDATIONS TO PEOPLE WITH SEVERE IMPACTS AS I MENTIONED BEFORE. ONE PERSON WE TRY TO PARSE THE BLACK BOX IS TOOK THE ALGORITHM AND LOOKED AT THOSE WHO SHARED A COMMON SYMPTOM FIBROMYALGIA AND WHAT WE'RE INTERESTED IN UNPACKING THIS FURTHER AND EXPLAIN PERFORMANCE. WE FOUND THERE WERE RISK AND PROTECTIVE FACTORS AND A TREND IN THE DATA AND I'M GLOSSING OVER THE DETAILS BUT HAPPY TO COMMENT AFTER THE FACT IF THAT'S INTERESTING WERE DRIVEN BY THE CLINICAL EVENTS IN OUT-PATIENT CARE AND PRESCRIPTION MEDS. WE WERE STRUCK BY THE DICHOTOMY AND TIME PEOPLE IN AN OUT-PATIENT OR IN-PATIENT SETTING WITH THIS PARTICULAR DIAGNOSIS IN THIS PROBLEM SPACE. WE FOUND FOR THOSE WHO ENDED UP WITH A SUICIDE ATTEMPT IN THIS COHORT, THEY WERE SPENDING ON AVERAGE LESS THAN AN HOUR IN A CLINICAL BECAUSE THEY WERE BEING SEEN IN OTHER CARE CENTERS WHERE THOSE WHO NEVER PRESENTED WITH SIMILAR RISK FACTORS WERE SPENTING ALMOST 50 AND OUR CLINICIANS AND OUR CLINICIAN NOT SURPRISED BY THAT FINDING AND THIS IS NOT CAUSALITY THIS IS A CORRELATION WE'RE INTERESTED IN. IT SUGGESTS TO US AND IT'S A HYPOTHESIS WE'RE TESTING FURTHER DOVETAILING BY WORK AT THE VETERANS AFFAIRS BY ENGAGING PEOPLE IN AN OUT-PATIENT CARE PROCESS IF THEY DIDN'T HAVE ONE BEFORE MAY BE HELPFUL IN THEIR PREVENTION AND FINDING THAT THROUGH THE DATA BUT UNPACKING THE BLACK BOX SHINED A LIGHT ON THIS POTENTIAL PATH WHICH RAISES THE QUESTION IS SOMETHING WE CAN THINK THROUGH WITH RESPECT TO A CARE PROCESS. AND ONE PAPER BY A Ph.D. STUDENT IN MY LAB IS THE IDEA WE CAN USE VARIOUS APPROACHES TO UNPACK THE ALGORITHMS AND ONE OF MY THEMES IN THE LAST SEVEN YEARS IS THE WAY WE ASK THE QUESTION CAN CHANGE THE ANSWER. WE'RE COMPARING THREE DIFFERENT ALGORITHMS TO PARSE A PART THE BLACK BOX AND SOME RISK FACTORS DIFFERENT SOME CLINICALLY ARE DIFFERENT AND WE HAVE RISK FACTORS SO THERE'S LOTS OF WORK TO DO AND IT'S A CALL TO ACTION FOR ALL OF US INTERESTED IN THE SPACE. FINALLY, I'LL GLOSS OVER THIS ONE BECAUSE IT MAY BE USEFUL FOR DISCUSSION AND WE'RE ALWAYS THINK OF PROVIDING THE RIGHT INFORMATION TO THE RIGHT PERSON IN THE RIGHT FORMAT AT THE RIGHT TIME. IT'S FUNDAMENTAL TO INFORMATICS AND MY FINAL SLIDE SAY SET OF -- IS A SET OF QUESTIONS AND THINGS I STRUGGLE WITH INFORMABLE IS NOT ACTIONABLE. WE WANT RISK AND WHAT YOU MAY BE ABLE TO DO ABOUT IT AND IF YOU DO, THIS IS WHAT MIGHT HAPPEN. WHEN DOES IT MATTER MORE OR LESS. INTERPRETABILITY OF AGE ALGORITHM AND UNDERSTANDING WHAT HAPPENS AT THE POPULATION LEVEL IS NOT THE SAME AS THE PERSON IN FRONT OF YOU AND WE HAVEN'T BRIDGED THAT GAP YES AND CAUSALITY IS A REAL CHALLENGE IN THE SPACE ESPECIALLY BEHAVIORAL MENTAL. SO LOTS OF PEOPLE TO THANK AS WELL FUNDING SOURCES AND HAPPY NOW TO TRANSITION TO THE PANEL. THANK YOU. >> THANK YOU FOR NICE PRESENTATIONS. I HAVE SEVERAL QUESTIONS I'LL ASK ONE BY ONE. THERE'S A CODE TO ERROR IS HUMAN AND ALL MODELS ARE WRONG. THAT MAKES THEM HUMAN I GUESS. MY POINT IS I HAVE SOME FUNDAMENTAL PROBLEM TALKING ABOUT EXPLAINABILITY OR INTEROPERABILITY WITH THE NOTION MODELS CAN BE WRONG. HUMANS ARE WRONG ALL THE TIME. AND I DON'T REALLY THINK TALKING ABOUT THE HUMAN EXPLAINABILITY MUCH AND WE SHOULD LOOK INTO THE BODY OF LITERATURE THAT SHOWS THE COGNITIVE SCIENCE AND BIAS AND HOW MANY TIMES MODELS LEARNING WRONG CLASSIFICATION OR ERRORS FROM US, HUMANS. I THINK WE SHOULD LOOK INTO EXPLAINABILITY COMPARATIVELY AND NOT JUST BECAUSE WE CAN KILL PEOPLE BY WRONG DECISIONS MADE BY ALGORITHMS WE SHOULD LOOK INTO HOW THE DECISIONS ARE BEING MADE PRACTICALLY. IS IT MAKING THE STATE OF THE ART BETTER, SAFER AND PROBABLY EVEN MORE HUMAN FOR PEOPLE THAT HAVE AILMENTS OR ARE AT RISK OF SUICIDE OR OTHER PROBLEMS AND THEN WE CAN UNDERSTAND THE REAL BENEFIT OF THESE MODELS. EXPLAINABILITY AS HUMAN BEINGS WE TRY TO EXPLAIN WHAT'S HAPPENING BILLIONS OF MILES AWAY WHEN BLACK HOLES COLLIDE INTO EACH OTHER AND TRYING TO EXPLAIN IS NOT A BAD THING. THE CONTEXT IS MORE IMPORTANT. I'D LIKE TO SEE WHAT THE PANEL LIKED TO THINK -- THINK OF EXPLAINABILITY AND WAYS MODELS CAN SHED LIGHT ON INNER WORKINGS. I GUESS MODELS ARE EXPLAINABLE AND THERE'S WAYS TO DO THAT. IT'S A MATTER OF WHAT WE'RE EXPLAINING IT TO IS IT FOR THE MATHEMATICIAN OR DOCTOR IN THE ROOM OR JUDGE IN THE COURT AND THAT WOULD MAKE A DIFFERENCE IN WHAT METHODOLOGY WE USE. THE POINT IS HOW IT'S BETTER OR WORSE THAN WHAT THE STATE OF THE ART IS. >> I'M HAPPY TO START. THANK YOU FOR YOUR THOUGHTFUL QUESTIONS AND COMMENTS. PARTLY IT'S A LIBERATING CONCEPT IF WE KNOW IT'S GOING TO BE WRONG LIKE I'LL MAKE MISTAKES AS A CLINICIAN. THERE'S SOME INTUITION WHEN I TRY TO DOCUMENT HOW I GOT THERE AND THAT'S HELPFUL AND THAT'S ONE OF THE FIRST PLACES I LIKE WHEN I TRY TO UNDERSTAND A COLLEAGUE'S DECISION AND IT'S LIBERATING AND WE THINK THROUGH THE TRANSPARENCY TO MAKE IT CLEAR TO USERS AND IT DEPENDS QUITE A BIT IN TERMS OF WHAT OUR BAR FOR EXPLAINABILITY AND ONE QUESTION WE GET A LOT IS WHAT WILL YOU DO IF THE ALGORITHM DISAGREE THE CLINICIAN AND WE NEED TO EMPOWER OUR USERS NOT TO ACT THOUGHTLESSLY SAYING THE COMPUTER TOLD ME SO AND THERE'S LITERATURE TO SUPPORT THAT BECAUSE SOME ARE IN A VERY FAST MOVING ENVIRONMENT AND WE CAN IMPROVE THE LARGER PROCESS. WE CARE ABOUT THE TECHNOLOGY BUT IT'S THE PEOPLE AND PROCESS THAT MAKES IT IMPORTANT. YOU CAN'T UNHINGE THAT FROM A MORE COMPLEX SYSTEM. >> IT'S A MORE NUANCED DECISION HOW CAN THE MODEL HELP TO WORK TO A FULL LICENSE. LIKE AUTO PILOTS. THERE'S STILL TWO PILOTS IN THE COCKPIT BUT MAYBE DON'T HAVE THEIR HAND ON THE STICK OVER THOUSANDS OF MILES OF THE ATLANTIC OCEAN BUT IN THE TAKEOFF AND LANDING THEIR WORKING AT THEIR FULL LICENSE AND ABLE TO FOCUS. THAT'S MY PERSPECTIVE. IT SHOULD BE EXPLAINABLE FOR ME TO UNDERSTAND HOW TO USE IT IN MY WORK FLOW. >> I THINK AND I AGREE WITH BOTH POINTS YOU BRING UP AN INTERESTING DICHOTOMY OR TO ERROR IS HUMAN AND WE EXPECT HUMANS TO MAKE MISTAKES AND WHEN A MODEL MAKES MISTAKES IT SOMEHOW RENDERS UNUSABLE OR YOU DON'T TRUST IT. SO PART GOES BACK TO TRUST AND IT GOES AND WE CAN SEE THIS WITH SELF-DRIVING CARS. IT'S A COMMON QUESTION THAT HOW COME DRUNK DRIVERS KILL PEOPLE EVERY DAY WHEN A SELF-DRIVING CAR TELLS SOMEBODY LIKE ON A NATIONAL NEWS IT GOES BACK TO TRUST. I THINK THE EXPLAINABILITY AND CORRECTNESS AND ACCURACY AS A SURROGATE FOR TRUST AND I THINK THERE'S ANOTHER THING GOING ON WHICH IS THAT MODELS CAN PROVIDE DECISIONS AND I THINK IT ALSO HAS TO DO WITH DOES IT REFLECT OUR VALUES AS HUMANS. FOR EXAMPLE, I CAN GET AN EXAMPLE NOT RELATED TO MACHINE LEARNING AND LIVER TRANSPLANTS AND THERE'S A SHORTAGE OF LIVER AND SOME PATIENTS HAVE TERRIBLE LIVER FROM DRINKING TOO MUCH AND THERE'S CRITERIA YOU HAVE TO MEET. AND AMONG PHYSICIANS, SHOULD ANYBODY BE ABLE TO GET A LIVER WHEN YOU KNOW THEY WON'T STOP DRINKING BUT IT MAY PROLONG THEIR LIFE OR HAVE A RATIONING SSTEM? I THINK THERE'S A LOT OF ISSUES WHERE THERE'S NOT CONVERGENCE AMONG HUMANS. IT'S NOT SO MUCH HUMANS ARE WRONG BUT THEY HAVE DIFFERENT VALUES. THAT'S ALSO PART OF IT IS MACHINE LOGIC IS TECHNICALLY REFLECTING A VALUES NUMBERS SYSTEM WHICH I DON'T KNOW HUMANS WILL ALWAYS ACCEPT. THAT'S ANOTHER PART OF IT. >> TO ERROR IS HUMAN AND LIKEWISE FOR THE MACHINE. ALL THE MACHINES ARE STATISTICAL DECISION METHODS IN THE END AND THEY WILL MAKE ERRORS AND THEY'LL MAKE CORRECT DECISIONS. SOMETIMES IT'S IMPORTANT TO UNDERSTAND THOSE CORRECT DECISIONS ARE MADE FOR THE WRONG REASON AND WHY EXPLAINABILITY IS SO IMPORTANT BECAUSE MAYBE IT WILL POINT OUT TO SOME ARTIFACT IN THE DATA THAT THE MACHINE JUST ZOOMED INTO AND GAVE A DECISION BECAUSE OF THAT. IT'S ALSO IMPORTANT LIKE IT WAS SAID BEFORE TO BUILD TRUST AND UNDERSTAND THE MACHINE IS MAKING THE RIGHT DECISION FOR THE RIGHT REASON >> BOSTON CHILDREN'S HARVARD MEDICAL SCHOOL. GREAT PANEL. I WANTED TO TALK ABOUT A POINT COLIN MADE TANGENTIAL BUT IMPORTANT TO EXPLAINABILITY AND THAT IS WHO IS THE EXPLAINER. SO FACEBOOK DOES REALLY GOOD SUICIDE PREDICTION ALGORITHM BECAUSE AS THE 10 AWKWARD MINUTES YOU MAY SPEND WITH AN ADOLESCENT IN THE OFFICE, FACEBOOK HAS EVERY POST THAT KID HAS MADE AND EVERY MESSAGE AND ALL THE LIKES AND EVERYTHING. THAT IS EXTRAORDINARILY REVEALING. THAT SAID, WHERE DOES IT GO? . FOR AN OPTUM OR OTHER HEALTH ORIENTED ANALYTIC EFFORTS, SURE, WE CAN PARSE THE POPULATION IN MANY DIFFERENT WAYS AND IDENTIFY WHO'S AT RISK BUT WE KNOW THAT VISIT IS A VERY CONSTRAINED TIME AND WE KNOW AND THEY CAN'T DEAL WITH ALL RISKS AS THEY COME UP BUT WE HAVE THE ABILITY TO DO POPULATION-LEVEL IDENTIFICATION. CAN PEOPLE LOOK AT WHO IS THE EXPLAINER OF THESE AND WHY DO YOU SEE THEM COMING AT. RADIOLOGY IS ONE OF THE EASIEST CASES BECAUSE IT'S PRETTY MUCH THE MACHINE AND RADIOLOGIST BUT IN THE HEALTH CARE ISSUE APPLICATIONS, WHAT DO WE THINK IS THE FUTURE? IT'S NOT JUST AN EXPLAINER BUT WHO HAS THE DATA TO SUPPORT AND UNDERSTAND THE PROBLEM BETTER. WHEN WE LOOKED AT SUICIDE RISK PREVENTION AND WE'RE A MULTI-FACETED HEALTH ORGANIZATION IT FEELS SPARSE AND WE DON'T HAVE ENOUGH DATA TO BE CONFIDENT THAT WE HAVE A MODEL THAT ACHIEVE SOME OF THE ETHICAL CONCERNS YOU RAISED ABOUT THE STIGMA OF IDENTIFYING AND STRATIFYING THE RISK. THEN YOU LOOK AT FACEBOOK OR YOUR CELL PHONE COMPANY AND SAY DO I HAVE A BETTER PERSPECTIVE ON ALL THE FACETS THAT MAY DRIVE YOU TO THAT RISK SO IT STARTS WITH DATA AND THEN A QUESTION OF THE ORGANIZATIONS, WHAT DOES FACEBOOK WANT TO DO? DO THEY WANT TO BE A PARTNER IN POPULATION HEALTH MANAGEMENT OR DO CRYPTO? IT'S THEIR MISSION AND WHERE THEY WANT TO GO. FOR US AS A POPULATION HEALTH MANAGEMENT WE'RE CONSTANTLY ASKING OURSELVES CAN WE ACQUIRE THE DATA AND PERSPECTIVE BECAUSE WE'RE THE STEWARDS OF THAT SUPPORT FOR MEMBERS. THANK YOU FOR YOUR QUESTION, I THINK THERE'S A LOT OF TO IT. -- A LOT TO IT. WE KNOW WE HAVE SUCH A SHORT MEASURE TO MEASURE ANYTHING IN HEALTH CARE SYSTEM AND PEOPLE LIVE THE VAST MAJORITY OF THEIR LIVES ONLINE AND ONE MODEL WILL NOT FIT ALL. THERE'S AN ALGORITHMIC LAYER AND OUR ALGORITHM BECAUSE IT CAN SCALE CAN GIVE SOME SENSE OF DISTRIBUTION OF RISK AND WE CAN SEE HOW DISPARATE IT IS AND GATHERED 25 FOLKS FROM IN-PEANUT BUTTER, SOCIAL WORKERS AND -- FROM IN-PATIENT AND SOCCER -- SOCIAL WORKERS AND HOW WE TRANSITION PATIENTS TO THE BEHAVIORAL HEALTH HOSPITAL AND THE TRANSITION PLACE IS WHERE I FIGURED OUT TWO HOURS DETERMINING WHAT INSURANCE WILL COVER FOR MY PARTICULAR PATIENT AND THOSE AT HIGHEST RISK ARE NOT WHERE COLLEAGUES SAY THEY'RE AT HIGH RISK IT'S THE MEDIUM RISK AND IT'S TIME CONSUMING AND TO FOLLOW AND SURVEY AND DO WE KNOW IF THEY'RE DOING THE THINGS WE ASKED THEM TO DO AND THERE'S VIEWS THAT MAY BE DRIVEN BY IT THAT MAY BE RELEVANT FOR THE GIVEN PARTNERS WILLING TO INTERACT WITH THE DATA. AND IF YOU DON'T HAVE THE PERSON IF WE'RE THROWING THE RISK PRE PREDICTION OF THE FOOTBALL WILL THEY CATCH IT? OUR APPROACH IS TO START SMALL AND HOPEFULLY FAIL FAST IF WE CAN AND OUR APPROACH IS TO DO THAT IN PARTNERSHIP WITH THE PEOPLE THAT WILL BE THERE AND WE DESIGN WHAT THAT IS. WE TRY TO MEET THEM WHERE THEY LIVE. >> WHEN WE TALK ABOUT ALGORITHMS EMBEDDED IN A DEVICE IT'S INCUMBENT ON THE DEVICE MANUFACTURERS TO DEMONSTRATE TO THE FDA AND SATISFY THAT AND WHEN WE TALK ABOUT STAND ALONE CLINICIAN SUPPORT ALGORITHMS, DOES THE FDA REGARD THAT AS MEDICAL PRACTICE WHICH YOU DON'T OR ARE YOU GOING TO CALL IT ALL DEVICES NOW SUBJECT TO REGULATION? THAT'S THE FIRST QUESTION AND YOU'RE TRYING TO AUTOMATE SOMETHING THAT ULTIMATELY IS A SUBJECTIVE ASSESSMENT AND WE HAVE MULTIPLE PEOPLE COMING UP WITH SUBJECTIVE ASSESSMENTS. WE TURNLY TALK ABOUT THE INTER AND INTRAOPERATING SYSTEMS. HAVE YOU LOOKED THAT IN YOUR POPULATION OF ASSESSORS? AND IF YOU'RE TRYING TO AUTOMATE WHAT IS A SUBJECTIVE INTERPRETATION, IT SEEMS TO BE A LITTLE BIT MORE OBJECTIVE. IF YOU JUST AGREE WITH IT YOU'RE NOT DOING BETTER BUT FASTER THAN WHAT'S ALREADY BEING DONE BUT YOU'RE MAKING THE ASSUMPTION WE'RE ALREADY AS GOOD AS IT GETS. >> I'LL START WITH THE FDA QUESTION. >> THE 21st CENTURIES CURES ACT pTRIED TO CLARIFY IT A BIT. THERE'S AN FDA DRAFT GUIDANCE THAT TRIES TO SAY FIRST OF ALL, WHAT'S THE DEVICE AND NON-DEVICE AND EVEN IF IT'S THE DEVICE, DOES THE PHAT WANT TO REGULATE IT -- FDA WANT TO REGULATE IT AND DEPENDING ON WHERE THE DATA COMES FROM AND IS THE ALGORITHM TRANSPARENT TO THE USER OR NOT? IT MAY OR MAY NOT BE A MEDICAL DEVICE OR MAY OR MAY NOT BE ACTIVELY ENFORCED. I KNOW I DIDN'T ANSWER CRISPLY BUT THERE'S A SPECTRUM. >> LET ME START BY SAYING IT'S A GREAT QUESTION AND I LOVE ANSWERING IT. FASTER IS THE BESTER ACHIEVER. WHAT DO I MEAN BY THAT? SOMETIMES THE DECISIONS CAN BE VERY SUBJECTIVE. THEY WARRANT REAL EXPERTISE FOR REVIEW AND WHEN WE THINK ABOUT AUTOMATING WITH A.I. AND IF WE CAN AUTOMATE THE DECISION AND IF WE CAN AUTOMATE THE DECISION WHERE BOTH COUNTERPARTIES AGREE ON THE ANSWER AND IF IT CUTS OUT TIME, EFFORT, COST TO COME TO THAT AGREEMENT, WE THINK THAT'S A GREAT OPPORTUNITY. IF IT'S GOING TO BE BE AN IN-PATIENT INITIATIVE AND SUPPORTED LET'S GET EVERYONE ALIGNED. IT'S SURPRISING HOW MUCH TIME AND EFFORT THAT CAN SAVE IN HEALTH CARE. AND IT'S IMPORTANT TO LOOK AT HOW I LOOK AT OPPORTUNITIES TO USE A.I. BECAUSE IT MAY SEEM VERY PEDESTRIAN AND MUNDANE BUT WE SPEND SO MUCH TIME COMING TO THAT AGREEMENT IN OUR WORK FLOWS TODAY. >> THANK YOU FOR THE WONDERFUL PRESENTATIONS. COLIN MENTIONED THIS ONCE AND IT WAS ALSO ALLUDED TO ACTIONABILITY. WE PRESUME EXPLAINABILITY HAS TO BE VERIFIED AND TO DISPROVE THAT AND EXPLAINABILITY DOES NOT LEND ITSELF TO ACTIONABILITY. THAT'S ONE POINT I'D LIKE THE PANEL'S COMMENT ON AND THE SECOND POINT IS SORT OF A COUNTEREXAMPLE FOR THE NEED FOR EXPLAINABILITY. THERE'S A STUDY DONE BY PEOPLE AT MICROSOFT RESEARCH AND IN A SIMPLIFIED USE CASE OF CHOOSING REAL ESTATE IN MANHATTAN AND TURNS OUT WHEN YOU HAVE EXPLAINABLE MODELS OR INTERPRETABLE MODELS WITH VARIABLES LIKE THE NUMBER OF ROOMS IN THE HOUSE AND SQUARE FOOTAGE THE DEGREE TO WHICH THEY CAN SPOT A MODEL'S MISTAKE GOES DOWN BECAUSE THEY TRUST IT AND IT'S INTERPRETABLE. BUT WHEN THEY'RE FACED WITH A BLACK BOX MODEL AND SHOULD THEY BUY THE HOUSE AT THAT PRICE THEY MAKE FEWER MISTAKS. THAT'S SOMETHING TO THINK ABOUT IN HOW WE PUT THINGS IN A WORK FLOW THE PURSUIT OF INTERPRETABILITY MAY GIVE A FALSE SENSE OF SECURITY WHERE THE HUMAN NEURAL DISENGAGES AND YOU TRUST THINGS. HAVING THE BLACK BOX MAY BE HELPFUL BECAUSE IT KEEPS PEOPLE VIGILANT TO THE MODEL'S MISTAKES. SO TWO QUESTIONS, ONE IS THE NEED FOR AC -- ACTIONABILITY AND EXPLAINABILITY AND DESTRUCTIO DISTRACTION. >> WE'RE STILL AT THE PHASE WHERE WE WANT TO BE AT THE PHASE WHERE WE CAN LOOK BACK TWO YEARS AGO AND SAY PEOPLE HAVE BEEN USING THESE INTEGRATED INTO CARE AND DID YOU MAKE A BETTER DECISION BECAUSE THE MODELS ARE THERE. YEAR NOT AT UPTAKE YET AND WE HAVE CONVERSATIONS AND IT BOLSTERS YOUR POINT THAT WE BUILT AN ALGORITHM AND TAKE THAT TO OUR CLINICAL PARTNERS WHO UNDERSTAND THAT SPACE THE BEST WE THINK AND SHOW THE RESULTS AND TYPICALLY THEY LOOK AT US SAYING WE KNEW THAT ALL ALONG AND WHAT WILL I DO ABOUT IT? IT'S PARTLY WHY WE CONFLATE THE TWO. EXPLAINABILITY AND AC -- ACTIONABILITY AND HELP PEOPLE UNDERSTAND WHAT THE DECISION SPACE MAY BE AND IT'S IMPORTANT TO UNHINGE THOSE AS ONE CONCEPT AND I'LL ADD A CONCEPT WHICH MAKES IT HARDER. MY STUDENT HAS A PERSPECTIVE COMING OUT SOON AND THE THEME IS IF WE'RE EVEN SUCCESSFUL, SAY WE HAVE JUST A BLACK BOX MODEL AND TOO WE'RE SUCCESSFUL, WE'LL START TO SEE WE'VE DONE SUCH A GOOD JOB OUR MODEL'S WORKING WORSE OVER TIME BECAUSE WE'RE CHANGING THE GOALPOSTS. WE HAVE TO THINK THAT THROUGH FROM MODEL INCEPTION AND HOW WE DO THAT IS HARD BUT WE HAVE TO THINK ABOUT THE WHOLE SPACE AND INCLUDE THAT MODEL SPECIFICATION SO WE DON'T GET LEFT BEHIND WHERE WE DID OUR JOBS AND IT LOOKS LIKE WE SCREWED UP. >> IN RESPONSE TO YOUR SECOND QUESTION, I THINK HAVING AN EXPLAINABLE SYSTEM IS JUST THE BEGINNING AND THAT'S WHERE MOST SYSTEMS ARE AT NOW. YOU NEED TO DO EXPERIMENTS WITH OR WITHOUT THE EXPLAINABLE COMPONENT OR EXPLAINABLE SIMPLE OR MORE COMPLEX MODEL AT THE HANDS OF THE USERS TO SHOW YOUR EXPLANATION IS HELPING WHETHER IT'S HELPING THE USER OR NOT. IT MAY DEPEND ON WHAT THE RISKS ARE MAYBE FOR SOME SYSTEMS YOU WOULDN'T NEED SUCH A DETAILED EXPERIMENT BUT IF YOU REALLY WANT TO PROVE THAT YOU'RE EXPLANATION HELPS, PROVIDING THOSE EXPLANATIONS IS THE FIRST STEP AND YOU HAVE TO DO THE PRUDENT WORK. >> I WANT TO RESPOND TO EXPLAINABILITY AND I THINK WE DON'T NEED TO KNOW HOW IT WORKS BECAUSE WE HAVE DATA AND YEARS OF CLINICAL VALIDATION PEOPLE GET BETTER AND THAT'S SAFE AND PEOPLE ARE COMFORTABLE USING IT AND DOCTORS AND PATIENTS AND IT'S EFFECTIVE AND WE KNOW THE SIDE EFFECTS AND HOW IT TREAT THE SIDE EFFECTS AND WE HAVE THAT EXPERIENCE WHICH WE JUST DON'T HAVE WITH THESE NEW A.I. TECHNOLOGIES. WHEN WE HAVE THAT BODY OF WORK THAT'S THE MOST IMPORTANT THING TO HAVE AND THE EXPLAINABILITY IS IMPORTANT INSOFAR AS IT HELPS VALIDATE THESE ARE USEFUL OVER TIME AND THESE ARE CLINICALLY APPLICABLE. THE SECOND QUESTION YOU HAD WAS ABOUT EXPLAINABILITY AND A BLACK BOX AND BEING VIGILANT. IT DOESN'T HELP HUMANS LEARN FROM NEW MODELS. IT'S NOT JUST TO AUTOMATE IN MEDICINE WE TRY TO HAVE QUANTITATIVE UNDERSTANDING. THERE'S A NEW RISK VALUE EVERY MONTH AND YOU PUT THEM IN AND IT WILL TELL YOU THIS PERSON'S EIGHT RISK OF THIS. KNOWS AREN'T PERFECT AND THEY'RE AROUND TRARY MODELS -- ARBITRARY MODELS BUT HELP GIVE A SENSE HOW TO GRADE AND ASSESS THE SITUATION AND ALSO GIVE US A SENSE OF WHAT THE MODIFIABLE RISK FACTOR SHOULD BE. INNING MORTALITY CALCULATIONS THERE'S A CALCULATOR VALIDATED TO BE ACCURATE AND WE LOOK AT WHAT THE PATIENT IS DOING AND IT CLOSES THE LOOP AND GIVES THE CLINICIAN OKAY, LET'S LOOK AT THESE THINGS FOR THE PATIENT IT'S NOT GOOD IF THEY'RE NOT EATING OR TALKING OR WAKING UP LET'S WORK ON THIS. THOSE ARE AREAS WHERE EXPLAINABILITY COULD BE HELPFUL. >> YOUR POINT IS SEMINOLE FOR THIS GROUP 30% TO 40% IS GETTING PREDICTION RIGHT AND DECIDING WHAT TO DO NEXT IS SUPER HARD IN HEALTH CARE. DON'T HAVE A GREAT ANSWER BUT MAYBE USE A LITHIUM OR STATIN AND I DON'T NEED A MODEL TO TELL ME I'M OVERWEIGHT AND IT PUTS ME AT RISK FOR DIABETES BUT WHAT'S THE RIGHT WAY TO APPROACH IT, I DID KETO AND MEDITERRANEAN AND GOT A PELOTON. YOUR POINT IS WELL RECEIVED. THE QUESTION OF WHAT TO DO NEXT IS THE BIG CHALLENGE FOR ALL OF US IN THE ROOM AND IN THE WEBCAST AND ALL OF US IN HEALTH CARE AND THAT'S EXCITING. >> THANK YOU. I THINK MY QUESTION IS NOT IRRELEVANT AND I APPRECIATE THAT EXPLANATION IS NOT EQUAL TO TRUSTWORTHINESS. COMING BACK TO EXPLANATION. I'M STILL IN THE DARK WITH EXPECTATION IF YOU COMPARE EXPLANATION THE WAY WE EXPLAIN OURSELVES AS HUMAN DECISION MAKERS, AGAIN, THERE'S A LOT OF COGNITIVE SCIENCE TO SHOW WE FIRST MAKE A DECISION AND THEN BECOME CONSCIOUS ABOUT IT AND THE BRAIN AREA USED TO MAKE DECISIONS IS DIFFERENT THAN EXPLANATION AND WE COME UP WITH THE SXHEXPLANATION FOR DITCH -- DIFFERENT THINGS AT DIFFERENT TIMES. THERE'S BODIES OF LITERATURE THAT SHOWS WE DON'T HAVE THE FACILITY TO EXPLAIN OUR DECISIONS. THEY'RE COMPLETELY SEPARATE TASKS OF BRAIN FUNCTIONALITIES. IF THAT'S NOT THE WAY WE'LL EXPLAIN MODELS WHAT DOES THE PANEL THINK OF WHAT WOULD BE A DESIRABLE EXPLAINABLE MODEL FROM YOUR PERSPECTIVE FROM SOCIAL JUSTICE TO HUMANITIES TO DIAGNOSTICS AT THE FDA AND EVERYTHING IN BETWEEN. WHAT'S THE FRAMEWORK FOR AN EXPLAINABLE MODEL HUMANS CAN ACCEPT? >> I THINK WHAT YOU SAID IS TRUE ABOUT MAYBE HUMANS NOT REALLY PROVIDING EXPLANATIONS BUT JUSTIFICATIONS AFTER THE DECISION HAS BEEN MADE. BUT NEVERTHELESS, WE MAKE EXPLANATIONS ALL THE TIME BECAUSE THERE'S A NED FOR IT. THE NEED MAY BE A PERSON NEEDING KNOW WHERE THEY WERE DENIED BY INSURANCE BASED ON A COMPUTER DECISION AND IN EUROPE THE HUMAN NEEDS THE INFORMATION TO BE ABLE TO CHALLENGE WHAT EVER AUTOMATED DECISIONS ARE MADE BY THE DATA. >> SO A JUSTIFICATION MECHANISM? INFORMATION IS CONTEXTUAL TO WHO WE ARE EXPLAINING WHALE DICTATE WHAT THE SCOMPEXPLANATION. IF A CLINICAL DECISION IS BEING EXPLAINED TO A FAMILY OF A PATIENT WILL BE DIFFERENT THAN IN A LAWSUIT TO FOR A JOURNAL CLUB. THOSE ARE DIFFERENT THINGS. I DON'T KNOW WHAT WE'RE EXPECTING FROM THE MODEL TO DO WHEN THE CONTEMPT OF THOSE EXPLANATIONS AND JUSTIFICATIONS ARE DIFFERENT IN DIFFERENT CONTEXTS. HOW DO WE TALK TO THE FAMILY AND TALK TO THE COURT OF LAW AND OTHER THAN BUILDING THE SYSTEM THAT CAN IMPROVISE AND ANSWER ON THE SPOT THAT HAS EVIDENCE BEHIND IT. >> I DON'T THINK THERE IS AN ANSWER TO YOUR QUESTION RIGHT NOW. BUT I THINK THAT'S THE REASON YOU'RE DISCUSSING IT AND AS A PRESENTED IN MY PRESENTATION THERE'S DIFFERENT LEVELS OF EXPLANATION DEPENDING ON HOW COMPLETE YOU WANT TO BE. I THINK THIS IS SOMETHING THAT SHOULD BE DEVELOPED IN TIME AND CURRENTLY MOST EXPLANATIONS ARE TARGETED AT THE URGS -- USER OR MEDICAL PERSONNEL BUTS THINK AS THEY'RE DEVELOPED THERE'S MUFFLE -- THERE'S MULTIPLE LEVELS OF EXPLANATION AND WE'RE NOT THERE YET BUT IT SHOULD BE ONE OF THE CONSIDERATIONS. >> ONE OF THE MORE PRACTICAL ASPECTS IF YOU THINK OF ARTIFICIAL INTELLIGENCE WE ASCRIBE DIFFERENT EXPECTATIONS BECAUSE THERE'S THE WORD INTELLIGENCE IN THERE AND WE THINK SOMETHING INTELLIGENT SHOULD BE ABLE TO ADAPT AND COMMUNICATE PERFECTLY TO A FAMILY MEMBER AND THE COURT BUT ACTUALLY IF YOU THINK OF THESE IT'S LIKE A CALCULATOR. IT DOESN'T INHERENTLY HAVE THE ABILITY TO GAUGE THE SITUATION AND UNDERSTAND WHAT THAT PERSON NEEDS IT'S ULTIMATELY A TOOL TO GET YOU FROM ONE POINT TO ANOTHER POINT AND TAKE THE INFORMATION IT'S GIVEN YOU IN A WAY YOU WANT TO USE IT FOR YOUR PURPOSE. THAT'S WHY HUMANS WILL STILL BE IMPORTANT TO INTERPRET THE CONTEXT. IF WE THINK OF THE SYSTEMS AS AUTOMATING ONE TASK OR FOCUSSING ON ONE TASK THERE'S OTHER COMPONENTS WHERE YOU NEED THE HUMAN TO MAKE A JUDGMENT ABOUT. YOUR POINT ABOUT A JUSTIFICATION, YES, BUT IT'S ON YOU JUST LIKE ANY JUSTIFICATION A HUMAN WOULD PROVIDE IT'S UP TO YOU TO DECIDE IF IT'S ACCEPTABLE OR WANT TO MOVE FORWARD. >> THERE REMAINS IN 2019 AN APPETITE TO IN TER GAT THE PROCESS -- INTERROGATE THE PROCESS AND WE'VE DONE IT OURSELVES OR SEEN SOMEONE WHERE YOU DEVELOP THE MODEL HOLDING THE DATA THE SAME AND DO IT AGAIN AND CHANGING ONE THING AND THE PROCESS OF RATIONALIZATION AND THE RISK FACTORS MAY HAVE CHANGED AND THE RISK FACTORS MAY HAVE CHANGED AND YOU SEE WHY IT MAKES SENSE IN BOTH CASES AND WITH WE HAVE CO EN COUNTER THAT AND KNOW THAT EXISTS. AND PART OF THE REASON THE QUESTION IS CHALLENGING AND YOU'RE MISSING CONTEXT. TO THE CONTEXT YOU'RE TRYING TO SOLVE AND WHAT'S NECESSARY IN A COURT OF LAW IS MAY BE DIFFERENT THAN A PROVIDER-PATIENT CONFERENCE ABOUT SHARED DECISION MAKING AND I THINK OVER THE NEXT THREE YEARS WE MAY DEVELOP MORE TRUST IN THE SYSTEMS TO SAY I CAN'T BUILD THE STETHOSCOPE BUT I KNOW HOW IT WORKS AND I CAN DEVELOP THE ALGORITHM BUT IT WORKS. >> LIKE HE MENTIONED A STETHOSCOPE. THAT'S A TOOL TOO AND THE HEART SOUND IS INTERPRETED DIFFERENTLY DEPENDING ON THE PATIENT AND WE NEED TO THINK OF IT THAT WAY AND NOT HOLD ARTIFICIAL INTELLIGENCE TO A STANDARD NOT WE HOLD TO HUMANS. >> I APPRECIATE ALL YOUR TALKS. I THINK I LEARNED A LOT FROM A LOT OF PERSPECTIVES SO THE TALKS HAVE BEEN GREAT. ONE THING I I COME BACK TO WE USE THE LARGE SCALE NEURAL IMAGING DATA SETS TO MAKE PREDICTIONS IN PATHOLOGY IN CHILDREN AND DO WE TRUST THE MODELS AND WE THINK WITH THE REPEATABLE AND ROBUSTNESS AND PRODUCIBILITY AND RELIABILITY. BEFORE WE CAN GET TO REALLY TRUSTING WHAT THE MODEL IS GIVING US AS A RESULT WE NEED TO TRUST THAT THE MODEL IS GOING TO REPRODUCE NEW SAMPLES. WE HEARD A LITTLE BIT OF TALK ABOUT GLOBAL VERSUS SPECIFIC. I THINK IT'S IMPORTANT IN SPECIFIC AREAS BUT THE IDEA WE COULDN'T CREATE GLOBAL MODELS IS TROUBLESOME AND I'M NOT HEARING A LOT OF TALK ABOUT THE PROBLEM OF OVERFITTING. OVER FITTING IS PROBABLY GOING TO BE ONE OF THE MOST IMPORTANT PARTS AND THE CRISP DIMENSIONALITY AS AN IMPORTANT PART OF TRUSTING IT AND IF THE PHYSICIANS ARE GOING TO TRUST US WE FIGURED OUT ALL THE OTHER THINGS. THE TRUTH IS WE HAVEN'T. WE HAVE A LOT OF ROOM TO GROW. I DON'T TRUST MACHINE LEARNING MODELS AND I BUILD THEM FOR A LIVING. IAGING MACHINES NEED TO BE ABLE TO PASS A LEVEL OF RADIATION AND IT'S GETTING THE IMAGE YOU WANT IT TO GET. YOU KNOW WHAT THE ANSWER SHOULD BE AND WHAT YOU'RE FEEDING IT AND MAKE SURE THE MODEL WORKS AND THAT PIECE IS SAFE. I THINK YOU HAVE TO DRAW THE DISTINCTION THAT IF YOU USE IT IN THE HOSPITAL AT STANFORD VERSUS MUMBAI THE ANSWERS WILL BE DIFFERENT BUT IS THAT A FAILURE OF THE MODEL OR NOT EVEN A FAILURE BUT THE FACT THE EPIDEMIOLOGY IN THOSE PLACES ARE DIFFERENT. I THINK YOU NEED TO THINK ABOUT THE TWO PHASES. ONE IS DEVELOPING A TECHNOLOGY OVER FITTING. IS THAT A PROBLEM WITH THIS PARTICULAR METHOD, IF IT IS, MAYBE USE A DIFFERENT KIND OF -- USE SOMETHING LIKE A VARIANCE TRADE-OFF AND YOU EXPERIMENT WITH DIFFERENT PARAMETERS OF THE MODEL. THEN ONCE YOU DO THAT AND APPLY IT IN THE REAL WORLD, AND YOU'RE CONFIDENT YOUR MODEL WORKS, I THINK YOU CAN FOCUS ON LOCAL PROBLEMS. ONE GOES BACK TO LIKE THE SPECIFICATIONS OF THE TECHNOLOGY AND ONE GOES TO HOW YOU'RE USING IT. I THINK THAT THOSE ARE LIKE TWO SEPARATE PROBLEMS BUT THEY'RE A LITTLE BIT RELATED. >> I THINK YOU RAISE VERY IMPORTANT POINTS AND GENERALIZABILITY IS SOMETHING I HEARD TODAY AND I THINK IT'S VERY IMPORTANT AND WHEN WE TALK ABOUT GENERALIZABILITY AND I WANT TO GO BACK TO ONE POINT BRUCE MADE EARLIER THIS MORNING ABOUT THE PHYSICS AND THE MORE FUNDAMENTAL SCIENCE WE KNOW ABOUT IS LOOKING AT THE GAPS AND I WANT TO USE THE MACHINE AND IF I KNOW THE PHYSICS AND HOW THE DATA IS GENERATE AND PROCESSED, THEN MAYBE I CAN SAY, WELL, THESE HAVE THESE CORRECTORS BUT THEY'RE SO SIMILAR THEY'LL GENERALIZE. BECAUSE I KNOW THIS ISN'T YOUR POINT BUT I WANT TO SAY JUST BECAUSE WE HAVE THIS GREAT NEW TECH NOKES IN DATA ANALYSIS, WE CAN'T FORGET WHAT WE KNOW IN FUNDAMENTALS BECAUSE COMBINING BOTH WILL HELP A LOT FOR GENERALIZABILITY. >> TO YOUR POINT ABOUT FIT. AS A GENERAL PRINCIPLE IF YOU THINK OF THE 250,000 PAPERS IN MED LINE AND THE TITLE OF THAT PAPER HAS MACHINE LEARNING IT. THE PRIOR PROBABILITY AS THE OVER FITTING IN THAT PAPER IS PROBABLY HIGH BUT THE CLINICAL REALITY IS THE MODELS LIKELY TO BE IMPLEMENTED CLOSE TO THE POINT OF CARE POINT TO UNDER FITTING BECAUSE THEY'RE SIMPLE AND THINGS YOU CAN DO WITH BASIC MULTIPLICATION AND ADDITION AND WE SEE THIS BEING PUT TOGETHER AND AS LONG AS YOU COMMIT TO RIGOROUS EVALUATION EVERY STEP YOU'LL BE ALREADY AS LONGER AS YOU THINK THROUGH CLINICAL IMPACT AND IT'S A SMALL PART OF A COMPLEX WHOLE THEY'LL ONLY HAVE CLINICAL IMPACT IF THEY DO THE THINGS DOWN STREAM AND COMMITTING TO BE RIGOROUS AND A LITTLE BIT CYNICAL I THINK IS A LITTLE BIT HEALTHY BUT THERE'S HUGE GAPS AND THE FACT THEY CAN SYNTHESIZE 20 YEARS OF DATA, THERE'S POWER THERE. THAT'S A GOOD THING. >> WE'RE ALREADY OVER TIME. WE'LL TAKE THE LAST TWO QUESTIONS BUT TRY TO KEEP IT SOMEWHAT BRIEF. >> THANK YOU. I REALLY LIKE YOU'RE PLOT COLUMN OF THE TRADE-OFF BETWEEN INTERPRETABILITY AND PERFORMANCE IN A WAY. THAT'S GRANTED IN RESEARCH AND A RESEARCH PAPER SOLVING A PROBLEM NEEDS TO HAVE A BASELINE AND IT NEEDS TO BE STATISTICALLY SIGNIFICANT. I WONDER HOW MUCH THAT'S A FALSE DICHOTOMY IN REAL WORLD APPLICATION. HOW MANY EXAMPLES DO WE HAVE IN WHICH THE PERFORMANCE OF AN EXPLAINABLE SIMPLE MODEL WAS DEEMED INSUFFICIENT AND DEMANDED EXPLAINABILITY FOR A MORE COMPLEX MODEL TO BE BUILT IN THE VOID. >> I CAN GIVE A BRIEF COMMENT ON THAT SO THANK YOU, LUCA. I THINK THAT'S HARD. I LIKE THE IDEA THERE'S TRANSPARENCY ABOUT THAT EXPLAINABILITY EVEN AT THE LEVEL OF THE PUBLICATION AND HOW WE EVALUATE THAT WHETHER IT'S IN A TRIPOD STATEMENT EQUIVALENT IS INTERESTING. WE'RE THINKING OF HYBRIDIZING AND MAYBE WE STAND UP NEXT TO AN ALGORITHM THAT HELPS INTERROGATE THE PROCESS AND SAMPLE OR DESCRIBE IT AND THESE LEAD TO THE SOLUTION. WE NEED TO THINK IT THROUGH SO IT'S A GREAT CHALLENGE YOU RAISE. >> AND I THINK IT'S PART OF THE PRACTICE WE TALKED ABOUT EARLIER. WHEN I THINK HOW WE DESIGN AND TRAIN MODELS WE ALWAYS START WITH THE BASIC AND SAMPLE AND SEE IF WE CAN OUTPERFORM THAT BY A NUMBER OF STANDARD DEVIATIONS. SO THE SIMPLY BECAUSE IT'S STRAIGHTFORWARD BUT THERE'S COMPUTATIONAL COSTS AND THINGS TO MAINTAIN WITH MODELS WITH HIGHER LEARNING PERFORMANCE. IF WE MEET THAT HURDLE IF I'M TRYING TO DECIDE WHO I MAY DO OUTBOUND CALLING TO IS DIFFERENT THAN DECIDING IF SOMEONE SHOULD BE ELIGIBLE FOR A COMPLEX CARE OR CLASSIFIED AS A SUICIDE RISK AND THINGS LIKE THAT. YOU NEED TO UNDERSTAND THE USE CASE AND WHAT YOU'RE TRYING TO ACHIEVE FROM THE PERFORMANCE OF THE MODEL WHAT THE IMPACTS AND TRADE-OFFS ARE. WE TALKED A LITTLE BIT ABOUT DECLINING OR NOT MAKING RESOURCES VAIN -- AVAILABLE TO FOLKS AND FAST TRACK THINGS. IT PLAYS INTO HOW YOU USE THE MODEL AND EFFECTIVELY AND HOW YOU TRAIN AND BUILD THEM. >> OKAY, THANK YOU. WE'LL TRY TO CLOSE IT OUT. THIS IS MOSTLY A THOUGHT ABOUT THE SESSION BECAUSE I THINK IT REPRESENTS ONE OF THE BIG CHALLENGES WE ARE ALL FACING. SO WE HAVE SHINGINI AND BERKMAN THAT REPRESENTATIVE THE MACHINE INTELLIGENCE APPROACHES AND EMBEDDED IN EVERYTHING THEY'RE SAYING IS PHYSICAL AND MECHANISTIC CONSTRAINTS. THEY'RE BASED IN A PHYSICS REALITY AND CONSISTENT -- INTERPRETABILITY FLOWS FROM THAT BECAUSE YOU HAVE SOME BASIS WITH WHICH TO INTERPRET HOW YOUR MODEL IS GIVING YOU AN OUTPUT. AND THAT WORKS WELL FOR THAT COMMUNITY AND OUR COMMUNITY HAS BEEN DOING THIS FOR A LONG TIME. ARE THERE LESSONS LEARNED? I SEE THERE'S NOT A LOT OF COMMUNICATION BETWEEN THAT DOMAIN AND COMMUNITY AND THE ONE THAT OVERWHELMS ALL OF US WE'RE A SMALL COMMUNITY IN ENGINEERING AND COMPUTER SCIENCE. CAN WE HELP OUR COLLEAGUES SEE WHAT THE VALUE OF IMPOSING PHYSICAL AND MECHANISTIC CONSTRAINTS AND CAN WE MOVE AWAY FROM CONDUCTION EQUATIONS AND THINK OF TRUTHS IN THE BIOLOGY WORLD. YOU MENTIONED LITHIUM. THAT'S -- THAT HAS EQUIVALENTS. YOU HAVE DATA OVER TIME. ARE THERE WAYS TO INTRODUCE MECHANISTIC CONSTRAINTS MAYBE NOT PHYSICS BUT IT WOULD IMPROVE THE MORE COMPLICATED PROBLEMS THAT ARE REALLY DIFFICULT TO SOLVE THAT ARE KIND OF AT THE OTHER END OF THE THINGS WE'RE LOOKING AT IN THE SESSION. I THINK THE NATURE OF MEDICINE IS THAT IT'S SOMETHING THAT'S JUST PRACTICAL KNOWLEDGE ACCUMULATED OVER TIME. SO MOST OF MEDICINE IS THAT AND YOU DO RESIDENCY AND TRAINING BECAUSE YOU CAN'T LEARN THAT FROM A BOOK. YOU CAN TRY TO LEARN IT FROM A MECHANISTIC POINT OF VIEW BUT MOST IS IT WORKS AND THERE'S DATA THAT IT WORKS AND SO THAT'S WHAT WE DO. WE HEAR 50% OF WHAT WE DO IS WRONG AND WE'LL LEARN THAT LATER WHEN WE HAVE MORE DATA AND THAT WILL CHANGE BUT IT REFLECTS A DIFFERENCE IN HOW MEDICINE IS PASSED DOWN KNOWLEDGE WISE AND HOW THE ENGINEERING SCIENCE ARE PASSED DOWN THROUGH EQUATIONS AN PHYSICAL CONSTRAINTS. AND BECAUSE I'VE WORKEDED IN BOTH FIELD -- WORKED IN BOTH FIELD I KNOW WHERE YOU COME FROM. TO GET MORE CONVERGENCE BOTH PEOPLE NEED TO CHANGE THEIR FRAME OF VIEW A LITTLE BIT. I DON'T THINK IT'S EASY. >> THAT'S A BIG OPPORTUNITY. >> I CAN GIVE EXAMPLES FROM WHERE IT'S CLOSER TO DEVICES THAN PHYSICS BUT WE ALREADY INCLUDE THOSE CONSTRAINTS. IF I'M DESIGNING A MACHINE LEARNING ALGORITHM I KNOW THE CT NUMBER CANNOT BE LESS THAN SOMETHING I ALREADY INCLUDE IT BUT AS YOU MOVE AWAY FROM MEDICINE IT GETS HARDER AND SOMETHING WE NEED TO EMPHASIZE AND THINK ABOUT. >> IN THINKING ABOUT YOUR ANALOGY OF PHYSICAL SCIENCE AND IN UNDERSTANDING QUANTUM, EVERYBODY'S GOT TO FIGURE OUT THESE DICHOTOMIES. >> THANK YOU SO MUCH. LET'S GIVE ANOTHER ROUND OF APPLAUSE. SO HOPEFULLY IF YOUR NOT FROM THIS AREA THERE IS A PIECE OF INFORMATION IB -- IN YOUR FOLDERS TO GIVE YOU INFORMATION ABOUT THE RESTAURANTS AND EATING OPTIONS. I WOULD LIKE TO GO AHEAD AND HAVE THE CHAIRS MEET ME AND WE'LL RECONVENE AT 1:15. >> THANK YOU FOR COMING BACK ON TIME, WE APPRECIATE THAT. SO WE'RE GOING TO DIRECTLY GET STARTED. I WILL INTRODUCE THE CHAIR OF OUR THIRD SESSION ON USABILITY, KEN MANDL, PROFESSOR PEDIATRICS AT BIOINFORMATICS AT HARVARD MEDICAL SCHOOL. >> GREETINGS, EVERYONE. HOPE LUNCH WAS GOOD. BEAUTIFUL DAY. SURPRISED HOW MANY PEOPLE ARE BACK HERE. THAT'S GREAT. SO LET'S JUMP IN. IT'S GOING TO BE I THINK A VERY INTERESTING SESSION FOLLOWING ON A REALLY INCREDIBLY STRONG MORNING. SO THE NUMBER OF DIFFERENT TYPES OF DATA SOURCES WE WANT TO BRING IN TO HEALTHCARE IS LARGE. BOTH FOR DECISION MAKING, AND ALSO FOR DISCOVERY. THIS IS A GRAPH FROM AN ARTICLE WE PUBLISH IN JAMA, GRIFFIN WEBER DREW THIS, JAMA LIKED IT SO MUCH THEY WERE LITERALLY SELLING POSTERS OF THIS THING. HOPE THEY MADE SOME MONEY. AND IT SHOWS YOU THERE'S A WHOLE SPECTRUM OF INFORMATION FROM SOCIAL MEDIA, TO THE PHARMACY SUPPLY CHAIN INFORMATION TO GROCERY SALES, THAT THERE'S A REALLY INCREASING LITERATURE AROUND MEASURING THE VALUE OF THESE THINGS AND OFTEN THE VALUE IS HIGHER THAN STRUCTURED CODES WE PUT INTO ELECTRONIC RECORDS. WE DON'T -- WE ARE NOT YET ABLE TO USE THESE THINGS IN POINT OF CARE SETTINGS. ONE OF THE REASONS FOR THAT IS THAT THE POINT OF CARE IS A WALLED GARDEN. WE MADE A $48 BILLION INVESTMENT IN ELECTRONIC HEALTH RECORDS. ABOUT 36 BILLION OF THAT WAS ESSENTIALLY PAY OFFS TO DOCTORS AND HOSPITALS. SO $44,000 PER DOC UNDER MEANINGFUL USE, MORE FOR HOSPITALS, TO PURCHASE ELECTRONIC MEDICAL RECORD SYSTEM. THE IDEA WAS THAT ONCE THEY HAD PURCHASED IT, THEN THEY WOULD FALL IN LOVE WITH ELECTRONIC FLAVORED CARE SO MUCH THAT THEY WOULD JUMP IN. THOSE SYSTEMS UNFORTUNATELY WE DIDN'T REALLY AS A NATION LOOK VERY CLOSELY UNDER THE HOOD AT WHAT THOSE SYSTEMS DID. HOW THEY INTEGRATED WITH OTHER SOFTWARE, WHETHER THEY COULD STANDARDIZE THEIR DATA AND WHETHER THEY COULD SUPPORT ANY FUNCTIONS THAT WE WANTED IN THE FUTURE LIKE AI, GENOMICS, DECISION SUPPORT. DATA VISUALIZATION. SO HOW DO WE GET THERE? IF YOU TOLD ME EVEN SIX YEARS AGO THAT I WAS GOING TO BE WORKING IN THE FIELD, I WORK IN MACHINE LEARNING DIRECTLY TOO BUT IF I WAS GOING TO BE WORKING IN THE FIELD OF INTEROPERABILITY, I WOULD HAVE KILLED MYSELF. SO IT'S GOOD STUFF. IT'S SO -- BUT TRADITIONAL APPROACHES TO INTEROPERABILITY, REALLY OFTEN NEVER CONVERGED ON TRUE INTEROPERABILITY. SO I WANT TO LOOK AT ANOTHER MODEL OF INTEROPERABILITY, THAT'S THE WORLD WIDE WEB. SO I'M NOT TELLING YOU ANYTHING YOU DON'T KNOW BUT LET'S JUST REMEMBER WHAT WAS ACHIEVED BY WHAT WAS ACHIEVED. OKAY? SO FIRST OF ALL, ALL WE WANTED TO DO, OKAY, WAS TO BEGIN WITH TO SHARE PREPRINT ARTICLE FOR PHYSICS. THAT WAS THE USE CASE. THAT IS A FOCUSED USE CASE. SO YOU SHARE PRE-PRINT ARTICLES, REQUIRE WE HAVE A WAY TO MARK UP THE TEXT SO WE CAN HAVE HEADERS AND COLUMNS AND THINGS LIKE THAT. SO THAT'S HTTP. WE NEED AD WAY TO MOVE THOSE THINGS AROUND TCP AND IP PROTOCOLS, WE NEEDED A WAY -- WE NEEDED HTTP. AND WE NEEDED A WAY TO DISPLAY THESE DOCUMENTS. WORLD WIDE WEB BROWSER. THEN WE NEEDED A WAY TO SERVE THEM UP. THESE FEW COMPONENTS WHICH WERE DESIGNED FOR ONE USE CASE, OKAY, BECAME THE FUNDAMENTAL STRUCTURE OF AN ENTIRE INFORMATION ECONOMY FOR THE WORLD. WHY WAS THAT SUCCESSFUL? PART OF IT WAS THE WORLD WIDE WEB CONSORTIUM CAME IN AFTER AN STANDARDIZED A FEW PARSE MOANIUS ROLES, THEY DIDN'T GO COMPLETELY CRAZY, THEY STARTED WITH THE FEW THINGS WE NEED SO IN HEALTHCARE PARTICULARLY WITH RESPECT TO A LOT OF PAIN POINTS WE HEARD IN THE MORNING. WHAT ARE A FEW THINGS THAT WE NEED TO GET INTO THE POINT OF CARE, MACHINE LEARNING, INTO CLINICAL WORK FLOWS IN SOME WAY. I'M GOING TO NAME FOUR. THERE MAYBE OTHER -- THERE WOULD BE A DIFFERENT FOUR THAN OTHER PEOPLE THINK ARE IMPORTANT. CLEARLY EACH OF THESE HAS MANY SUBHEADS BUT LET'S JUST SAY THAT SUBS TUESDAYABLE APPS -- SUBSTITUTABLE APPS, TRIGGERED PATIENT SUPPORT, GENERATED DATA AND PUSH BUTTON POPULATION HEALTH SO YOU CAN GET THE POPULATION DATA SETS. IF YOU HAVE THOSE FOUR THINGS THERE'S A LOT YOU CAN DO ON TOP OF WHAT WE HAVE NOW. LET ME TELL YOU WHAT I MEAN BY THESE THINGS. WHEN THE $48 BILLION WAS ABOUT TO BE MADE, I WROTE A PIECE WITH ZACK IN THE NEW ENGLAND JOURNAL AND SAID WHY NOT THINK ABOUT A PROGRAMMING INTERFACE ON TOM OF EHRs SO THE FORM OF INTEROPERABILITY THAT YOU WANT IS WHAT WE CALLED SUBSTITUTABILITY. THAT MEANT YOU CAN ADD OR DELETE THE APP. SO THAT'S REALLY FUNCTIONAL. IT'S DIFFERENT THAN SAYING WELL, THESE -- WE HAVE THE STANDARD, EVERYTHING IS INTEROPERABLE AND YOU SAY I IMPLEMENT IT, IT DOESN'T WORK. I CAN'T TALK TO THE OTHER GUY. THEY SAY WELL BECAUSE YOU DIDN'T IMPLEMENT THE SAME WAY. BUT IT'S A STANDARD. SO IF YOU GET TO A FUNCTIONAL DEFINITION, THE APP WILL RUN OR NOT RUN, YOU START TO THINK ABOUT THESE THINGS IN A DIFFERENT WAY IN AN ENGINEERING WAY. SO IF YOU CAN RUN APPS EVERYWHERE AND THE SAME APP COULD RUN ACROSS DIFFERENT SYSTEMS IN VENDOR INDEPENDENT WAY, THEN YOU ARE ABLE TO ACTUALLY YOU ARE ABLE TO ACTUALLY CHANGE THE FACADE ON TOP OF ELECTRONIC MEDICAL RECORDS, IT WAS CLEAR THE EMRs WE WERE IMPLEMENTING AT THE TIME WERE ALREADY 30 YEARS OLD. THAT WAS TEN YEARS AGO. SO IF YOU WERE TOLD YOUR IPHONE HAD TO BE -- HAD TO BE RETROFITTED WITH A MODEL FROM FIVE YEARS AGO, YOU WOULD BE VERY DEPRESSED. THIS IS 30-YEAR-OLD TECHNOLOGY, WE NEED TO ACKNOWLEDGE LAYER ON TOP WHERE MORE STUFF HAPPENS. SO THE BASIC IDEA HERE IS CAN WE CREATE AN APP STORE FOR HEALTH WHERE EMR BECOMES APPS PLATFORMS AND WHERE THE END USERS CAN SELECT DIFFERENT FUNCTIONALITY AND WHERE THE FOLKS WHO ARE DRIVING INNOVATION CAN GET THEIR INNOVATIONS TO THE POINT OF CARE. IN WAY THAT SUBSTITUTABLE ADD OR DELETE. PROBLEM ONE, THE AI APPS DON'T CONNECT TO HEALTH SYSTEMS DATA. TO MAKE A LONG STORY SHORT, THIS PROJECT ENDED UP BEING FUNDED AT THE BEGINNING OF THE OBAMA ADMINISTRATION, SMART, OR NOW CALLED SMART ON FIRE. WE GOT IT IN TO MEANINGFUL USE, WE GOT IT INTO THE REGs, THERE'S OOH A 600 PAGE ONC RULE WHICH SMART BECOMES INSTANTIATED AS THE APPS CONNECTION STANDARD, IT ALSO DEFIANCE, IT'S INTERESTING READING. 600 PAGES. GOES FAST. INTERESTING READING BECAUSE IT DEFIANCE ALONG WITH API REQUIREMENTS WE GOT INTO 21st CENTURY CURES IT DEFIANCE INFORMATION BLOCKING NO KNOWS LIKE THIS WILL BE ILLEGAL, THAT WILL BE ILLEGAL. THIS WILL BE ILLEGAL, ALL THE THINGS THAT HAPPENED TO YOU. WHEN YOU TRIED TO MOVE INFORMATION AROUND, IT IS REALLY GOOD. THE MODEL OF SUBSTITUTAL APPS CREATES COMMERCIAL ENVIRONMENT. AND HOW THIS FIRST APP THAT WON OUR CONTEST IS EIGHT YEARS AGO. IT DOES MEDICATION RECONCILIATION, MEDICATION INSTRUCTIONS FOR PATIENTS ACROSS 22 DIFFERENT LANGUAGES IN DYNAMICALLY GENERATED USER FRIENDLY WAYS. NICE LITTLE COMPANY TOO. ACQUIRED BY FIRST DATA BANK, THEY INTEGRATEDD WITH EPIC SMART SCRIPTS USING ATICS AND WE PUBLISHED A PAPER IN JMIR EARLIER THIS YEAR DEMONSTRATES ENORMOUS EFFICIENCIES THEY ACHIEVE IN INTEGRATION AND PROMOTING THE COMMERCIAL SUCCESS. WE HAVE AN APP GALLERY WHICH IS OPEN AND FEDERALLY FUNDED, EACH EMR VENDOR REPLICATED THIS APP GALLERY FOR THEIR OWN MORE CLOSELY HELD APPS AND THERE'S AN INTERESTING ECONOMY THAT'S DEVELOPING AROUND THERE, THAT IS ALSO ADDRESS IN THE RULE. SOME OF THE DISTORTED MARKET FORCES THAT WILL BE ADDRESSED. PROBLEM TWO, YOU CAN'T LAUNCH APPS AT THE RIGHT TIME, YOU DON'T WANT TO TELL OUR DOCS OH REMEMBER TO LAUNCH THE APP WHEN YOUR PATIENT IS BEHAVING BADLY BECAUSE IT'S A BEHAVIORAL APP. YOU DON'T WANT THAT. YOU WANT IT TO COME IN AT THE RIGHT MOMENT. SO IT TURNS OUT THERE'S ANOTHER STANDARD THAT ALSO IS MENTION IN THE RULE CALLED CDS HOOKS WHICH TELLS YOU AND EMR VENDORS HAVE IMPLEMENTED THIS IN THEIR PRODUCTS, IT LETS YOU ESSENTIALLY TRIGGER DECISION SUPPORT WHEN THINGS HAPPEN LIKE A MEDICATION IS PRESCRIBED OR SOMETHING LIKE THAT. PROBLEM NUMBER 3 IS THAT PATIENT GENERATED DATA ARE NON-STANDARDIZED AND ARE IN A SEPARATE SILO. THERE'S ALL KINDS OF PATIENT GENERATED DATA. I TALKED TO A NUMBER OF HEALTH INSTITUTIONS, SOMEWHERE ELSE. I THINK THAT'S A WAY TO GET THAT IN. THESE DIGITAL BIOMARKERS WE ARE SEEING INCREASINGLY USED IN MEDICINE, THESE MODULARLY PUT TOGETHER COMPONENTS, THIS IS LIKE FIVE DIFFERENT WAYS TO CREATE A ATRIAL FIBRILLATION DETECTOR, MANY, SEVERAL USING THE APPLE WATCH, PUTTING TOGETHER OPERATING SYSTEMS, PUTTING TOGETHER FIRMWARE SOFTWARE, ALGORITHMS INTERPRETATIONS, THESE THINGS ARE STILL OUTSIDE, HOW DO WE BRING THOSE IN TO PATIENT WORK FLOWS, ONE THING WE ARE WORKING ON THAT WILL PUT UP A WHITE PAPER ON SOON IS A SMART MARKERS FRAMEWORK TO BRING PATIENT GENERATED DATA IN TO THIS INTEROPERABLE FRAMEWORK AS WELL. I TELL YOU THAT THE FIRST -- SO WHERE IS THE -- THE LAST ONE IS GETTING REFERENCE POPULATION DATA OUT OF EHR INTO ANALYTIC PLATFORMS, IT'S HARD, TAKES TEAMS OF ETL EXPERTS, TERMINOLOGY EXPERTS, A COMMUNITY HOSPITAL CANNOT DO IT. A PHYSICIAN PRACTICE CANNOT DO IT. AT ALL. SO DON RUBBINGER WHO RUNS THE ONC CAME TO US AND YOU DID THIS THING FOR SMART ONE PATIENT AT A TIME, CAN YOU DO SOMETHING FOR ALL THE PATIENTS AT A TIME, COHORTS, SO WE DESIGNED THIS THING THE ONC CALLS BULK DATA EXPORT AND WE CALL FLAT FIRE, IT'S A WAY TO ACTUALLY COMPLETELY STANDARDIZED WAY TAKE FIRE AND EXPORT INTO A JASON FLAT FILE WHICH CAN BE INGESTED INTO AN ANALYTICS ENGINE. WE THINK THERE'S A COMMUNITY OF DEVELOPMENT AROUND FLAT FIRE FORMATS. THIS ALSO HAS BEEN THROUGH THIS THING CALLED THE ARGUE NAUGHT PROCESS, SO THE EMR VENDORS PUT THIS INTO THEIR PRODUCTS ALREADY. AND DEVELOPING THE NEXT SET OF USE CASES ON THIS IS INTERESTING. I TELL YOU -- WHAT COMPANY HAS -- WHAT COMPANIES HAVE TAKEN ADVANTAGE OF THIS EMERGING ECOSYSTEM, WHICH IS ONLY BEEN THERE IN KIND OF PRODUCTION FOR THE LAST TWO OR THREE YEARS, APPLE, TOOK ITS HEALTH APP AND CONNECTED IT TO FIRST 500 HEALTH SYSTEMS, ANY HEALTH SYSTEM THAT REGISTERS, USING THE SMART API. SO THE SMART API WAS THERE BECAUSE OF MEANINGFUL USE 3 AND APPLE CONNECTED TO 500 HEALTH SYSTEMS SO ANY PATIENT THOSE HEALTH SYSTEMS CAN DOWNLOAD A COPY OF HEALTH RECORD IN JASSON FOUR FORMAT ON TO THEIR PHONE. AND THEN APPLE APPS CAN ACTUALLY USE THOSE DATA AS FIRST ORDER DATA ELEMENTS FROM THE DATA CORE SO IT'S INTERESTING GOOGLE AND MICROSOFT ARE PUTTING FIRE SERVERS AND WITH SMART LAUNCH CAPABILITIES, INTO THEIR CLOUD SERVICES TO DIFFERENTIATE FOR HEALTHCARE WHAT THE CLOUD SERVICES CAN DO. THOSE WILL BE REALLY USEFUL ENGINES THAT SIT NEXT TO ELECTRONIC MEDICAL RECORDS, BUT WITH OBVIOUSLY REALLY GOOD SERVER TECHNOLOGY AND I THINK AS THOSE MATURE, WE WILL HAVE A LOT MORE OPPORTUNITIES TO DO COOL COMPUTATIONAL THINGS WHAT WE CALL THE SIDE CAR MODEL, NEXT TO THE ELECTRONIC MEDICAL RECORD AS OPPOSED TO NECESSARILY EVERYTHING HAPPENING INCITE THESE 40-YEAR-OLD STACKS. THEY WILL STILL HAVE A ROLE FOR A LONG TIME. HOPEFULLY THOSE COMPANIES CONTINUE TO INNOVATE AND BECOME A MODULAR HEALTH SYSTEM SO LET ME HOLD TON TO PANEL, THE REASON THIS USABILITY TALK IS BECAUSE I THINK THE -- THESE ELEMENTS ALLOW US FIRST OF ALL TO FOCUS ON USABILITY RATHER THAN THE -- RATHER THAN ON JUST ALL THE INTEGRATION STUFF BUT ALSO IF YOU HAVE THESE SUBSTITUTABLE APPLICATIONS WHERE YOU CAN DELETE THEM WHICH IS DIFFERENT THAN THAT ONE, YOU CAN'T DELETE STUFF FROM EPIC, IT'S THERE. THAT'S WHAT YOU GOT. BECAUSE YOU CAN DELETE IT AND BECAUSE THERE ARE THESE EMERGING APP STORES THEY WILL COMPETE WITH EACH OTHER. IT'S BEEN MY CONTENTION ALL ALONG WHAT THEY WILL COMPETE ON IS USABILITY. SO IF YOU GOT AN ML ALGORITHM THAT HAS ALL THOSE GREAT FEATURES THAT WE TALKED ABOUT, IF IT'S EXPLAINABLE INTERPRETABLE AND USABLE AND ACTIONABLE, YOU CAN ADD IT TO YOUR EHR. AND THE OTHER ONE DOCTOR SAYING I HAVE NO IDEA. WHAT I'M SUPPOSED TO DO WITH THIS INFORMATION. THAT WILL GET DELETED. SO I THINK THAT THEY COULD BE REALLY COMPETITIVE MARKETPLACE THAT OUT OF WHICH COMES AN EMERGENT PROPERTY OF USABILITY. SO LET ME HAVE THE NEXT PRESENTER COME ON UP. [APPLAUSE] >> MY NAME IS CHRIS DYMEK, I'M DIRECTOR OF HEALTH IT DIVISION FOR AGENCY FOR HEALTHCARE RESEARCH AND QUALITY. FIRST A FEW WORDS ABOUT AHRQ HEALTH IT AND HOW USABILITY FITS INTO OUR WORK. AHRQ'S MISSION IS TO PRODUCE EVIDENCE TO MAKE HEALTHCARE SAFER, HIGHER QUALITY, MORE ACCESSIBLE, AFFORDABLE AND EQUITABLE. WE TYPICALLY WORK WITHIN THE U.S. DEPARTMENT OF HEALTH AND HUMAN SERVICES AND OTHER PARTNERS TO MAKE SURE OUR EVIDENCE IS UNDERSTAND AND USED. IN THE HEALTH IT DIVISION AT AHRQ WE AIM TO PRODUCE AND FULFILL AHRQ VISION BY PRODUCING EVIDENCE HOW HEALTH IT CAN MAKE HEALTHCARE SAFER, HIGHER QUALITY MORE ACCESSIBLE, EQUITABLE AND AFFORDABLE. OVER THE YEARS, WE HAVE INVESTED IN A RANGE OF RESEARCH AROUND THE USABILITY OF HEALTH IT. THIS INCLUDES NOT ONLY GRANTS BUT ALSO TOOL KITS AND AROUND CONSUMER HEALTH IT AND NOT JUST EHR USABILITY. SO A QUICK AUDIENCE POLL BECAUSE IT'S GOOD TO MOVE YOUR BODY EVEN JUST YOUR HAND AFTER LUNCH, LEAVING ASIDE WHAT KEN TALKED ABOUT WITH RESPECT TO SIDE CARS TO EHR, HOW MANY THINK EHR SYSTEMS CAN BE MORE USABLE? SEW WHAT THAT SAYS TO ME IS WE STILL HAVE A LOT OF WORK TO DO TO HAVE OUR EVIDENCE WE PRODUCED BE UNDERSTOOD AND USED. BEING USED IS THE KEY HERE. I'M HOPING WE CAN DO BETTER WITH MI SYSTEMS. I WAS HAPPY TO RECEIVE THIS INVITATION ON THIS PANEL BECAUSE IT GAVE AN OPPORTUNITY TO REFLECT ON USABILITY WORK WE HAVE BEEN AND ARE CURRENTLY SUPPORTING. THE USABILITY LESSONS THAT COULD BE APPLICABLE TO MI. IN THE INTEREST OF TEN MINUTES ALLOTTED TIME I HAVE ONLY THREE LESSONS TO SHARE WITH YOU, I WILL TRY TO ILLUSTRATE THOSE BY REFERRING TO CURRENT RESEARCH THAT WE ARE SUPPORTING. HERE IS LESSON ONE, USABILITY ISSUES ARE ASSOCIATED WITH PATIENT HARM. WE WANT USABLE IT APPLICATIONS. I LOVE THE USABILITY OF THE IPHONE AND APPS AND ORDERING ON AMAZON, TOO MUCH SO. IT'S EASY AND NON-ANNOYING. BUT FROM MY PERSPECTIVE, FOR PROFESSION WHOSE MEMBERS PLEDGE TO DO NO HARM, WE HEARD THAT THIS MORNING, THIS IS THE KEY REASON FOR FOCUSING ON USABILITY OF MI SYSTEMS AND HEALTHCARE FROM MY HUMBLE PERSPECTIVE. THERE'S BEEN MANY CASES RECOUNT IN THE LITERATURE ABOUT E HEALTHCARESR USABILITY SYSTEMS ISSUES IN PATIENT HARM, AHRQ IS FUNDING A STUDY WHICH IS INTERESTING. RAJ, SOME OF YOU KNOW HIM, THE STUDY PI HUMAN FACTORS EXPERT AND DIRECTOR OF MED SCAR HEALTH NATIONAL CENTER FOR HUMAN FACTORS IN HEALTHCARE H. -- IN HEALTHCARE. WHAT RAJ AND TEAM HAVE FOUND IS THAT IF YOU LOOK AT PEDIATRIC PATIENT SAFETY EVENT REPORTS THAT HAD HIGH LIKELIHOOD OF BEING RELATED TO EHR USAGE AND MEDICATION ERRORS ONE-THIRD OF THOSE EVENTS THOSE PATIENT SAFETY EVENTS HAD TO COWITH USABILITY ISSUES. TO MOST COMMON USABILITY CHALLENGES WERE ASSOCIATED WITH SYSTEM FEEDBACK AND VISUAL DAYS PLAY ERRORS. THE MOST COMMON MEDICATION ERROR WAS IMPROPER DOSING. SO WHAT RAJ AND TEAM ARE CALLING SYSTEM FEEDBACK ERRORS I THINK IS KIND OF INTERESTING TO NOTE AND TALK ABOUT THESE A BIT. ONE EXAMPLE FROM THIS FIRST CATEGORY HERE SUBOPTIMAL CLINICAL DECISION SUPPORT OR ERROR PREVENTION, HAD TO DO WITH THE CASE WHERE A PHYSICIAN ORDERED FIVE TIMES THE RECOMMENDED DOSE OF A MEDICATION WITHOUT RECEIVING AN ALERT FROM THE EHR. THOUGH THE PRESCRIBED DOSE WAS OUTSIDE THE RECOMMENDED RANGE. WE ALSO NEED TO CONSIDER WHAT THE SYSTEM IS DOING BEHIND THE SCENES. LIKE CALCULATING DOSAGE BASED ON WEIGHT FOR EXAMPLE AND WHAT HAPPENS IF THE WEIGHT ENTERED IS WRONG. BOTH VENDOR DESIGN AND DEVELOPMENT AS WELL AS IMPLEMENTATION AND CUSTOMIZATION SOFTWARE AT THE SITE CONTRIBUTE TO THE CHALLENGES ASSOCIATED WITH SYSTEM FEEDBACK P. I THINK THE VISUAL DISPLAY ISSUES NOTED ON THIS SLIDE ARE SELF-EXPLANATORY. BUT HOW DOES THIS APPLY TO MI. WE TALKED ABOUT MI EXPLAINABILITY THIS MORNING P. I THINK IT'S A KEY FACTOR THAT WILL AFFECT USABILITY FOR MI APPLICATIONS. AS WE NOTED THIS MORNING MAYBE NOT ALL MI APPLICATIONS BUT CERTAINLY A GREAT DEAL OF THEM. ALSO MI NEEDS TO PLAN FOR UNFORESEEN CIRCUMSTANCES. LIKE RECOGNIZING THAT A DEVICE READING OR PIECE OF DATA IS INCORRECT. MY VISUAL DISPLAY IS INTERFACE NEED TO BE USER CENTERED TO AND LIKELY NEED EXPLANATION INTERFACES AS WELL. THAT IS MY SUGGESTION. MANY OTHER ELEMENTS AFFECT USABILITY BEYOND THE SOFTWARE AND USER INTERFACE DESIGN WHICH LEAD ME TO LESSON TWO. CONTEXT MATTERS WE HEARD ABOUT THIS THIS MORNING AS WELL. SO HEALTHCARE IS A COMPLEX SOCIO TECHNICAL CONTEXT OF SYSTEMS AND WHEN WE TRY TO FIX IT WITH TECHNOLOGY TOO OFTEN WE ARE PLAYING WHACK A MOLE, WE FIXED ONE THING AND THE UNINTENDED CONSEQUENCES OF THAT FIX RAISED ITS UGLY HEAD AND CAUSED ANOTHER NASTY ISSUE TO POP UP. NOT CONSIDERING THE SYSTEM, THE TECHNOLOGY WILL BE EMBEDDED IN, COMPROMISE SAFETY AND EFFECTIVENESS. SO THERE ARE FRAMEWORKS USED TO ASSIST IN TAKING A SYSTEMS APPROACH. DR. PASKAL CARRION, GRANTEE FROM THE UNIVERSITY OF WISCONSIN WAS INSTRUMENTAL IN DEVELOPING SUCH& AN APPROACH, CALLED SEEPS. WHICH STANDS FOR SYSTEMS ENGINEERING INITIATIVE FOR PATIENT SAFETY. THE BASIC IDEA HERE IS THAT WORK SYSTEMS SHOULD SUPPORT WORK PROCESSES IN SUPPORT OF DESIGNED OUTCOMES. THE WORK SYSTEM CONTAINS NOT ONLY TECHNOLOGIES BUT ALSO PEOPLE, ORGANIZATIONAL POLICIES AND PHYSICAL SURROUNDINGS THAT NEED TO BE FACTORED IN. WHAT I WOULD LIKE TO CUT HERE IS THAT OUR THINKING WILL SHIFT IF INSTEAD OF ASKING THE QUESTION WHAT FUNCTIONALITY DOES THIS TECHNOLOGY NEED TO EXHIBIT, WE ASK HOW CAN WE DESIGN A WORK PROCESS AND WORK SYSTEM INCLUDING ITS TECHNOLOGY, TO SUPPORT THE CARE PROCESSES THAT WILL RESULT IN DESIRED OUTCOMES. FOR EXAMPLE, DR. CARRION IS USING SEEPS TO STUDY HOW TO IMPROVE VTE PROPHYLAXIS CLINICAL DECISION SUPPORT IN HOSPITAL SETTINGS. MANY OF THE EXISTING VTE RISK ASSESSMENT ALGORITHMMENTS EXHIBITED THROUGH CKS DON'T TAKE INTO ACCOUNT FOR EXAMPLE, A CRITICAL CARE UNIT MAY NEED A DIFFERENT WORK FLOW DESIGN THAN ANOTHER HOSPITAL UNIT. ALSO YOU MAY NEED TO ASSESS VTE RISK AT SEVERAL POINTS DURING PATIENT HOSPITAL STAY AND NOT JUST DURING ADMISSION WHICH IS TYPICALLY DONE RIGHT NOW. SO DR. CARRION IS DEVELOPING DESIGN REQUIREMENTS FOR CDS THAT SUPPORTS COGNITIVE AND TEAM WORK FOR PREVENTING AND MANAGING VTE IN HOSPITAL ENVIRONMENTS AT VARIOUS POINTS IN THE PATIENT STAY. A LONG TERM GOAL OF THIS EFFORT IS TO DEVELOP AN APPROACH FOR DESIGNING HEALTH IT APPLICATIONS INCLUDING MI APPLICATIONS THAT CAN BENEFIT FROM IMPROVED INTEGRATION AND CLINICAL WORK FLOWS. AS I THINK ABOUT THIS TYPE OF APPROACH,'S FOLLOWING QUESTIONS FOR MI. WHEN SHOULD WE USE A SOCIO TECHNICAL APPROACH, DOES A DIAGNOSTIC MI SYSTEM THAT SIMPLY READS AN IMAGE AND SUPPLY AS DIAGNOSIS TO RADIOLOGIST REQUIRE THIS APPROACH? MAYBE. IT DEPENDS ON THE WORK SYSTEM, THE RADIOLOGIST IS EMBEDDED WITH AND HOW COMPLEX THE NEEDS ARE WITH RESPECT TO THAT SYSTEM. WHICH FRAMEWORKS SHOULD WE USE? SEEPS? THE CDS FRAMEWORK WAS MENTIONED THIS MORNING SOME OF YOU MAY BE FAMILIAR WITH. DO WE NEED DIFFERENT FRAMEWORKS FOR DIFFERENT PURPOSES? I JUST CAME ACROSS A REALLY INTERESTING ARTICLE IN JAMA, THE REFERENCE IS DOWN BELOW. THAT SUGGESTS A FRAMEWORK FOR ASSESSING THE UTILITY OF PREDICTED ANALYTICS AND HEALTHCARE. SO MAYBE WE NEED USABILITY SLASH SOCIO TECHNICAL FRAMEWORK FOR DESIGNING MI IN WORK FLOWS, AND ANOTHER FRAMEWORK ASSESSING UTILITY BECAUSE UTILITY IS AN EQUIVALENT TO USABILITY. I THINK THESE ARE ALL QUESTIONS THAT REQUIRE FURTHER RESEARCH. ANOTHER ISSUE THAT'S INTERESTED ME FOR MOST OF MY CAREER IS HOW DO WE EXPLOIT TECHNOLOGY FOR TRANSFORMATIONAL RATHER THAN INCREMENTAL GAIN? IN SOME CASES WE MAY WANT TO ASK HOW CAN THE WORK SYSTEM INCLUDING THE MI SUPPORT THE COGNITIVE WORK WE NEED TO DO BUT AREN'T DOING? I RECENTLY READ ABOUT PERSON WHO DIED AT AGE 62 FROM CHRONIC KIDNEY DISEASE. THIS PERSON HAD HISTORY OF MEASURES OF DECLINE OF STANDARD FUNCTION BUT THIS PERSON'S INDIVIDUAL TRIPS TO THE ER REGISTERED ABOVE THIS THRESHOLD FOR CONCERN THOUGH THE LONGITUDINAL TREND INDICATED THAT MEASURES OF KIDNEY FUNCTION WERE CONSISTENTLY TRENDING DOWNWARD. EVENTUALLY HE WAS REFERRED TO A KIDNEY SPECIALIST WHEN A KIDNEY FUNCTION MEASURE INDICATING CONCERNING LOSS OF FUNCTION BUT THEN TOO LATE AND NEEDED DIALYSIS. I'M LOOKING FORWARD TO THE DAY WHEN WE'LL BE ABLE TO CHANGE OUR CARE PROCESSES TO CREATE APPROPRIATE ALERTS BASED ON LONGITUDINAL DATA ANALYZED BY MI SYSTEM AT POINT OF CARE. AS PART OF THE PANEL WE WERE ASKED TO THINK ABOUT COMPARATIVE EFFECTIVENESS OF MI SYSTEMS AGAINST USUAL CARE, AS I THINK ABOUT THE CASE WE SHARED, I THINK USUAL CARE WOULD HAVE BEEN UP ENDED IF WE HAD THAT DIAGNOSTIC LONGITUDINAL DATA AT THE POINT OF CARE. SO WHEN WE THINK ABOUT COMPARATIVE EFFECTIVENESS RESEARCH IN THAT SENSE WHAT WE ARE GOING TO NEED TO RETHINK THE OUTCOME MEASURES WE ARE AFTER, IT MAY NOT BE THE OUTCOME MEASURES WE ARE LOOKING AT USUAL CARE. SO I KNOW I HAVE JUST A MINUTE LEFT. SO MY LAST LESSON IS, PAY ATTENTION TO USABILITY SOONER RATHER THAN LATER BUT FOR VARIOUS REASONS IT'S HARD TO DO. PEOPLE SAY TAKES TOO MUCH TIME, WE ARE EXCITE TO GET TO THAT TECHNOLOGY IN AND USE IT AND CLEAN UP THE MESS LATER. WE DON'T HAVE HUMAN FACTORS EXPERTS OR INDUSTRIAL ENGINEERS ON OUR TEAM. SO WITH MI I HOPE WE CAN AVOID THE ADOPTION SAFETY ISSUES WE HAVE EXPERIENCED DUE TO LACK OF EHR USABILITY AND SOME OF THE RESULTING LEGAL AND REGULATORY INTERVENTIONS WE HAVE BEEN SUBJECTED TO. THANK YOU. [APPLAUSE] >> I ERICH HUANG FROM DUKE UNIVERSITY. MY ANGLE TO THE USABILITY ISSUES ACTUALLY EXPERIENCE WE HAVE AT DUKE, IMPLEMENTING MACHINE INTELLIGENCE ARTIFICIAL INTELLIGENCE, WHATEVER YOU WANT TO TERM IT AT DUKE. BEFORE I START I WANT TO LAY TWO THINGS OUT FOR YOU TO THINK ABOUT. I WAS IN THE TEACHING CONFERENCE FOR SURGERY RESIDENTS NOT TOO LONG AGO AND WE WERE TALKING MACHINE LEARNING. I ASKED HIM DO YOU KNOW HOW SERUM SODIUM IS CALCULATED, NONE OF THEM COULD TELL ME HOW WE GENERATE THAT RESULT. SO TO GET AT SOME OF THESE ISSUES WE TALKED ABOUT THROUGHOUT THE DAY, I KNOW YOU WILL HAVE ANSWERS TO THAT QUESTION, IT'S USEFUL TO THINK ABOUT THAT, IT'S LIKE IN THE ACTION OF DELIVERING CLINICAL CARE, WE DON'T SPEND MUCH TIME THINKING ABOUT BLOOD CHEMISTRY VALUES WHEN WE SEE THEM. ARE THEY AS INTERPRETABLE AND EXPLAINABLE AS SOME OF THE THINGS WE ARE TALKING ABOUT DOING WITH MACHINE LEARNING. THE OTHER THING I WANT TO LAY DC COUPLE OF WEEKS AGO TELLING COUPLE OF PEOPLE AT LUNCH, THE DUKE DC OFFICE WE HAD A SESSION WITH A BUNCH OF LEGISLATIVE AIDS AND POLICY MAKERS AND ONE QUESTION ABOUT AI AND HEALTHCARE WAS WHAT ARE THE METHODOLOGICAL THINGS WE NEED, WHAT ARE THE TECHNICAL THINGS THAT WE NEED I ARGUED, YOU KNOW WHAT, TO ME BETTER THAN ALL THOSE IS A VALUE BASED MEDICINE FRAMEWORK FOR US TO DO THESE THINGS BECAUSE THE SOCIO TECHNICAL ECONOMIC COMPLEXITY OF MEDICINE, WE STILL ARE NOT ADDRESSING FUNDAMENTAL ISSUES. AND THIS ANECDOTE AT DUKE WILL SHOW YOU, IF WE CAN GET ALIGNMENT WITH OUR HEALTH SYSTEM ADMINISTRATOR WHOSE ARE FRANKLY -- WHO ARE FRANKLY PAYING ATTENTION TO HEALTH SYSTEM PROFITS, PRIMARY CARE PROVIDER IS PATIENTS. ALIGNMENT AMONG THOSE THINGS IT DOES BECOME EASIER TO THINK HOW WE CAN USE MACHINES TO HELP TAKE CARE OF PATIENTS. SO TO LAUNCH INTO MY SLIDES, I WANT TO START WITH A REAL PATIENT. AND OBVIOUSLY NOT A REAL PICTURE, NOT A REAL NAME, THAT'S AGAINST THE RULES. THIS IS A PATIENT, BETTY C IS WHAT I WILL CALL HER. SHE IS A TYPICAL PATIENT AMONG MEDICARE POPULATION. SHE'S 65 YEARS OLD, SHE HAS A HISTORY OF MI, SHE CONGESTIVE HEART FAILURE AND DRUG DEPENDENCY. SHE ADMITTED TO DUKE EMERGENCY DEPARTMENT WITH CHEST PAIN AND PULMONARY EDEMA THAT RESULTS IN A TEN DAY HOSPITALIZATION, SHE'S DISCHARGED. AND LOSS TO FOLLOW-UPMENT A PATIENT LIKE BETTY WHO IS AMONG OUR MEDICARE PATIENT IS A LOST IN SHUFFLE, SHE DOESN'T SEE PRIMARY CARE PROVIDER BUT DOES COME IN WITH EPISODIC BIG ISSUES. WHEN WE THINK HOW WE HAVE ADMINISTERED HEALTHCARE, WE OFTEN HARKEN BACK TO THIS PICTURE, THIS IMAGE AND I ACTUALLY BORROWED THIS FROM ABRAHAM AT STANFORD WHO LOVE SHOWING THIS IMAGE IN HIS TALKS. THERE'S THIS RELATIONSHIP WITH A PATIENT, THIS INTIMATE RLATIONSHIP WITH A PATIENT, THAT FRANKLY IS DIFFICULT IN THIS DAY AND AGE, HOW DO WE HAVE PATIENTS LIKE BETTY WHO ARE LOST TO FOLLOW-UP COME IN WITH A HIGH INTENSITY CARE AND DISAPPEAR. HOW DO WE START THINKING HOW WE CAN BRING ASPECTS OF THAT BACK INTO THE WAY WE CARE FOR PATIENTS. THAT'S MEDICAL TRADITION. AT DUKE WE HAVE AN MSP ACO, ACCOUNTABLE CARE ORGANIZATION SO MSP IS MEDICARE PROGRAM, CMMI INNOVATION MODELS UNDER OBAMA CARE. DUKE CONNECTED CARE IS A TRACK 3 MSSP PROGRAM. I WILL TELL YOU WHAT THAT MEANS. WE ARE TALKING MACHINE LEARNING OR MACHINE INTELLIGENCE AND WHY IS ERICH SHOWING ME POLICY BUT IT'S IMPORTANT. THE WAY MSP PROGRAM WORKS IS THAT CMS ESTABLISHES A FINANCIAL BENCHMARK FOR OUR PATIENTS. SO I WILL MAKE A NUMBER OUT TOP OF MY HEAD, THEY SAY BASED ON HISTORY AND PATIENT MIX AND THINGS LIKE THAT, WE ARE BENCHMARKINGING YOU AT $8,000 FOR PRIMARY CARE PATIENT MEDICARE PATIENT YOU ARE TAKING CARE OF. WITH THAT MODEL, IF YOU EXCEED WHILE MEETING QUALITY METRICS, IF YOU EXCEED THE COST OF MAINTAINING THAT PERSON'S HEALTH, YOU GET TO ACTUALLY WRITE A CHECK TO CMS. SO YOU ACTUALLY HAVE LOSSES THAT YOU WILL HAVE TO WRITE A CHECK FOR. ON THE OTHER HAND IF YOU MAINTAIN QUALITY AND STAY BELOW THE BENCHMARK YOU GET SAVINGS. UP TO 75% OF SAVINGS SO DUKE CONNECTED CARE LAST YEAR WAS CALCULATED TO HAVE SAVED CMS $22 MILLION AND GOT TO KEEP $9.5 MILLION OF THOSE WHILE MEETING QUALITY METRICS. THIS TWO SIDED RISK MODEL REQUIRES US TO BE ABLE TO QUANTIFY RISK. AT A POPULATION LEVEL. WHICH MOST OF YOU WHO WORK IN A HEALTH SYSTEM KNOW WE DO A TERRIBLE JOB. HOW DO WE DO THIS? IN OUR MSP POPULATION MEDICARE POPULATION, WE HAVE ABOUT 30% ADMISSION RATE. AND THAT WE HAVE ABOUT A 15% READMISSION RATE SO THOSE ARE REALLY GOOD INDICATORS OF HEALTHCARE UTILIZATION AT A PLACE LIKE DUKE. AN AVERAGE HOSPITALIZATION DUKE IS SOMETHING LIKE $18,000. IF THE BENCHMARK WERE $8,000, WE ARE ALREADY WRITING A BIG CHECK PER PATIENT FOR THAT HOSPITALIZATION. IF WE HAVE THAT CONSTRAINT FOR US, HOW DO WE RISK STRATIFY THE $52,000 PATIENTS THAT WE HAVE ATTRIBUTED TO US IN THE MSP PROGRAM. USING HOSPITALIZATION IS OUR METRIC. THE REASON WE ARE USING HOSPITALIZATION BECAUSE SOME PEOPLE ASK WHY DON'T YOU MODEL COST? THE PROBLEM IS COST IS NOT INTERVENABLE. WE CAN'T INTERVENE ON COST. WE CAN'T THINK OF THINGS WE MIGHT DO BUT DIRECTLY COST IS NOT INTERVENABLE. HOSPITALIZATION WE CAN THINK ABOUT HOW WE CAN DO THINGS TO PREVENT THAT HOSPITALIZATION OR READMISSION. SO WHAT WE NEED IS WHAT IS GOING TO KEEP ME SAFELY IN THE AIR WHEN I FLY BACK TO DURHAM TONIGHT. THERE'S AIR TRAFFIC CONTROL, SO AIR TRAFFIC CONTROL KNOWS WHERE EVERY COMMERCIAL PLANE IN THE UNITED STATES IS AT ANY GIVEN MOMENT. IF WE ARE TAKING CARE OF A POPULATION OF 52, 54,000 MEDICARE PATIENTS, DO WE KNOW WHERE THEY ARE? FROM A HEALTH TRAJECTORY STANDPOINT, WE NEED SOMETHING LIKE THAT. CAN WE SEE THEM? CAN WE PREDICT TRAJECTORY BUT MORE IMPORTANTLY CAN WE CHANGE THAT TRAJECTORY? WHAT WE NEED TO DO IS THINK ABOUT IS HOW WE CAN UTILIZE PRECIOUS RESOURCE IN OUR CASE NOW WE DECIDED THE EFFECTER ARM FOR OUR ABILITY TO DO A PREDICTION AND I USE THAT BECAUSE MANY OF YOU SAW THE JAMA PERSPECTIVE BY BOB AND SEQ EMANUEL WHERE THEY TALKED ABOUT AI AND HOW IMPORTANT TO HAVE EFFECTOR ARM BECAUSE WE HAVE THIS ABILITY TO POTENTIALLY DO SOME DIAGNOSTIC. WITH MACHINE LEARNING. UNLESS YOU ACT UPON IT LIKE MED SCHOOL DON'T ORDER A TEST UNLESS YOU'RE GOING TO DO SOMETHING WITH IT YOU HAVE AN EFFECTOR ARM SO OUR EFFECTOR ARM HYPOTHESIS IN DUE CONNECTED CARE IS WE USE COMPLEX CASE MANAGEMENT FOR POPULATION HEALTH MANAGEMENT AMONG PATIENTS. WHAT THAT MEANS IS THIS IS THE DUKE CARE MANAGEMENT GROUP, THEIR ESSENTIAL WAY THEY BREAK THINGS DOWN. YOU HAVE IDENTIFICATION OF PATIENTS, YOU HAVE STRATIFICATION OF PATIENTS, AND THEN YOU HAVE CARE MANAGEMENT INTERVENTION. SO WE NEED TO BE ABLE TO SEE, PREDICT AND CHANGE N THIS CONTEXT. SO WHEN IT COMES TO SEEING TO IS HOW WE USED MACHINE INTELLIGENCE. WE GET A MONTHLY SET OF CLAIMS FILES FOR THESE PATIENTS. THESE CLAIMS FILES ARE FLAT FILES, NOT INHERENTLY ANALYZABLE SO WE HAVE TO DO DATA MONITORING TO USE THEM. WE HAVE EPIC, WHICH Y'ALL KNOW THE CHALLENGES OF WORKING WITH, AND IN OUR CASE SINCE DUKE IS THE COORDINATING CENTER FOR PCORNET WE HAVE THE COMMON DATA MODEL, OTHER PLACES LIKE COLUMBIA OR VANDERBILT USE, IT DOESN'T MATTER YOU HAVE TO HAVE SOMETHING. SO WE ACTUALLY ETL OUT OF EPIC TO THE PCOR NET COMMON DATA MODEL WE MERGE THIS DATA SO THOSE ESSENTIALLY BECOME OUR REPRESENTATIVE OUR REPRESENTATION OF BETTY AND IT'S NOT A COMPLETE REPRESENTATION BECAUSE THERE ARE OTHER THINGS WE ARE NOW WORKING ON INTEGRATING INTO THIS LIKE SOCIAL DETERMINANTS OF HEALTH DATA BUT FOR NOW IN THE FIRST PASS WE HAVE THE CLAIMS FILES, NICE THING ABOUT CLAIMS FILES IS THEY GET GLOBAL SCOPE. IF BETTY GOES TO EMERGENCY ROOM AT OUR FRIENDS ACROSS THE STREET, DOWN TOBACCO ROAD AT UNC THAT -- THOSE DATA ARE NOT IN OUR EHR BUT THE CLAIMS FILES CERTAINLY GIVE INFORMATION ABOUT THAT KIND OF SITUATION SO WE HAVE GLOBAL SCOPE IN CLAIMS FILE AND LOCAL SCOPE IN EHR DATA WE HAVE GRANULAR DATA ABOUT BETTY IN OUR OWN EHR. WHAT THESE DO IS BECOME WAY TO LOCATE BETTY ON THIS RADAR SCOPE. AND FOR US TO DO THAT WE DO PREDICTION. WE HAPPEN TO BE AT A PLACE WHERE WE HAVE AMONG TOP TEN PUBLISHED GROUPS AND MACHINE LEARNING, SO LARRY KAREN WHO IS OUR VICE PROVOST OF RESEARCH HAS FACTOR NETWORK, ANOTHER FLAVOR OF DEEP NEURAL NETWORK. WE RUN THROUGH THE DEEP NETWORK AND GENERATE PREDICTIONS. I PUT THIS UP TOO BECAUSE WHEN I THINK ABOUT IT WE HAVE 200 PRIMARY CARE PROVIDERS IN DUKE CONNECTED CARE. WE HAVE 23 COMPLEX CASE MANAGERS IN DUKE CONNECTED CARE. THERE'S NO WAY THEY CAN LOOK AT 54,000 PATIENTS ON A MONTHLY BASIS. SO AN IMPORTANT COMPONENT OF WHAT WE ARE DOING IS AUTOMATING THE PROCESS OF SCANNING ACROSS A LOT OF PATIENTS. BECAUSE WE ARE DOING POPULATION HEALTH. SO WITH THOSE 52,000 PATIENTS, 32 DIAGNOSTICS WE MAKE PREDICTIONS ACROSS 32 DIAGNOSTIC CATEGORIES SO WE BREAK DOWN STANDARD ICD 10 CODING INTO THOSE CCS CATEGORIES WE HAVE CUSTOMIZED TO SOME DEGREE SO WE HAVE A DIFFERENT PREDICTION FOR CONGESTIVE HEART FAILURE OR COPD, ET CETERA. AND THERE'S SEASONAL VARIABILITY BECAUSE IN THE WINTER MONTHS WE SEE RESPIRATORY DIAGNOSTIC CATEGORIES AS WELL. THESE ARE THESE CHARGED DIAGNOSTIC CATEGORIES WITH THESE PATIENTS. WE TRAIN THE MODEL, DEPLOY THE MODEL, AND THIS IS ESSENTIALLY WHAT THE WORK LOOKS LIKE. SO ON A TIME LINE AT BEGINNING OF THE MONTH WE HAVE A SIX MONTH PREDICTION WINDOW FOR HOSPITALIZATION. SO WE LOOK FOR PROBABILITY OF UNPLANNED HOSPITALIZATION FOR ANY CAUSE OR THESE 31 DIAGNOSTIC CATEGORIES. WE GENERATE ABSOLUTE AND PERCENTILE RANKS, WE USE PAST 12 MONTHS CMS CLAIMS DATA, PAST ACTUALLY WE HAVE BROUGHT BACK TO 12 MONTHS RETROSPECTIVE EHR DATA ABSTRACTED AS PCORNET COMMON DATA MONTH RUN ON A MONTHLY BASIS. WE RETRAIN IT ON A MONTHLY BASIS WITH THE MONTHLY DATA REFRESH, AND THEN ANOTHER IMPORTANT THING THAT TO NOTE IS WE STORE REGISTRY PREVIOUS VERSIONS OF MODEL, PREVIOUS CUTS OF THE DATA WHO LOOKED AT THE -- WHO IS ACTUALLY DID THE COMPLEX CASE MANAGEMENT SERVICE LOOK AT THE MODEL AND ACT UPON THE MODEL. IT'S IMPORTANT TO CONSIDER WHEN WE START TO DEPLOYING THESE INTO HEALTHCARE SETTINGS, ARE WE MAINTAINING A REGISTRY OF HOW WE USE THAT? COLLIN MADE A POINT IF PEOPLE ARE LOOKING AT PREDICTION AND ACTING ON IT AND EFFECT ACTIVELY ACTING ON IT, OVER TIME THE MODEL MAY GET WORSE AND WORSE. IF YOU DON'T KNOW THAT HOW DO YOU ADDRESS THAT ISSUE? YOU NEED THAT REGISTRY. THOUGH THIS IS OUR DEEP CARE MANAGEMENT FRAMEWORK, WE DON'T HAVE A STATIC MODEL, IT CHANGES OVER MONTH TO MONTH. IT HAS HISTORICAL MEMORY BECAUSE WE UTILIZE BACK MEMORY IN OUR SYSTEMS, AND HOPEFULLY ADJUST AS INTERVENTIONS CHANGE AND HOPE ANY OUTCOMES CHANGE AS WELL. AND WE CAN INTEGRATE FEEDBACK FROM THE PROVIDERS TO INFLUENCE THAT LEARNING. BECAUSE WE ALWAYS GOT THAT X MATRIX AND Y VECTOR, WE NEED TO UPDATE THAT Y VECTOR ON A REGULAR BASIS. SOMETIMES ANOTHER INPUT WE NEED FROM PROVIDERS AND PEOPLE MADE THIS POINT, IS THERE A FEEDBACK LOOP, WAY FOR THE PROVIDER, NET FLIX, THUMBS UP FOR THIS PREDICTION OR THUMBS DOWN. IT'S USEFUL TO STORE THOSE KIND OF FEEDBACK IN A REGISTRY AS WELL. THIS IS OUR CARE MANAGEMENT WORK FLOW, OUR CHIEF OFFICER IN DUKE CONNECTED CARE STOOD BACK AND RE-ENGINEERED THE COMPLEX CASE MANAGEMENT WORK FLOW BASED ON THE FACT WHERE THOSE YELLOW DIAMONDS ARE THAT WE WERE USING THIS MACHINE LEARNING FRAMEWORK TO HELP RISK STRATIFY PATIENTS. THEN OBVIOUSLY WE NEED TO CHANGE THINGS. SO FOR SOMEONE LIKE BETTY HERE IS AN EXAMPLE WHAT WE DID. SEND PHARMACY TECHNICIAN TO VISIT HER AT HER HOUSE, SHE DIDN'T UNDERSTAND MEDICATION, SHE WASN'T TAKING THEM, THAT PHARMACY TECHNICIAN ALWAYS A GREAT INTERVENTION BUT IT'S BEGINNING FOR US TO DO THINGS. WE LINKED HER TO RESOURCES AND FINANCIAL SUPPORT BECAUSE SHE COULDN'T AFFORD MEDICATIONS ANDS THAT MY LAST SLIDE. SO THE KEY HERE FOR US WAS WE ALREADY TALKED WITH OUR POPULATION HEALTH MANAGEMENT FOLKS AND SAID HOW CAN WE HELP YOU GUYS TAKE BETTER CARE OF YOUR PATIENTS, WE LET THEM DRIVE US FROM MACHINE LEARNING STANDPOINT, WE DIDN'T SAY DROP BIG DATA SET ON US AND WE'LL GO TO TOWN AND COME UP WITH SOMETHING. THEY SAID LOOK, WE WANT TO TARGET HOSPITALIZATION, WE THINK THAT'S A GREAT SOURCE OF UTILIZATION AND COST FOR US IN TR 3 MSP PROGRAM. THEN HOW CAN YOU ENGINEER THINGS TO HELP US DO A BETTER JOB DOING THAT. I THINK THAT'S IMPORTANT IN TERMS OF USABILITY COMPONENT, LET THEM DICTATE THE CONTEXT WHICH YOU ARE GOING TO DO MACHINE INTELLIGENCE OR MACHINE LEARNING. THANKS. [APPLAUSE] >> HELLO EVERYONE, I'M DINA KATABI, PROFESSOR AT MIT. I WANT TO TELL YOU ABOUT WHOA WE CALL THE MOVE FROM WEARABLES TO INVISIBLES. SO BEFORE I CAN TELL YOU ABOUT THE INVISIBLES I'M SURE YOU GUYS KNOW ABOUT THE WEARABLES. I CAN TELL YOU WHAT INVISIBLES ARE. ACTUALLY IT'S INTERESTING TO LOOK AT MACHINE LEARNING IN THE CONTEXT OF HEALTHCARE AND WHERE THE DATA COMES FROM. SO THE VAST MAJORITY OF -- OF DATA TODAY, MACHINE LEARNING OF COURSE IS ASK IT AS THE DATA, DEPENDS ON THE VAST MAJORITY OF DATA THAT WE HAVE COMES FROM CLAIMS. FROM MEDICAL RECORDS, MEDICAL IMAGING. DATA THAT IS SPARSE, THAT IS DEPENDENT ON THE PATIENT CONNECTING WITH THE HEALTHCARE SYSTEM WHICH DOESN'T HAPPEN OFTEN. IT ALLOWS YOU TO DO RISK ASSESSMENT FOR EXAMPLE. YOU DON'T KNOW HOW WELL MS. SMITH DOING TODAY, YOU DON'T KNOW WHETHER SHE'S SPENDING MORE TIME IN BED THAN YESTERDAY THAN THE LAST WEEK. YOU DON'T KNOW WHETHER SHE HAD BREATHING DEPRESSION FOR EXAMPLE, WHETHER SHE WAS ABLE TO SLEEP OR NOT SLEEP. YOU HAVE NO REAL TIME INFORMATION, YOU HAVE NO RECENT INFORMATION, YOU ARE RELYING ON SPARSE DATA. THAT WAS LIMITING BECAUSE IT MEANS THAT YOU ARE JUST DOING PROBABILISTIC ANALYSIS. WHAT IS THE LIKELIHOOD MS. SMITH IS GOING TO END UP IN THE HOSPITAL IN GENERAL, ACROSS MANY PEOPLE LIKE HER IN THE POPULATION. NOT BASED ON WHAT HAPPENED RECENTLY. BUT IF YOU WANT TO KNOW WHAT HAPPENED RECENTLY, HOW CAN YOU DO THAT? MANY PEOPLE SAY, IT'S -- WE CAN'T HAVE WEARABLES, WEARABLES ARE THE NEW THING THAT WE CAN DEPLOY AND WITH THE PATIENT, WE CAN DEPLOY IT IN HOMES AND THEN WE CAN COLLECT REAL TIME INFORMATION, WE CAN COLLECT THE MOST RECENT INFORMATION ABOUT THEIR STATE. NOW, OF COURSE WEARABLES ARE POWERFUL AND IMPORTANT BUT THEY DON'T ACTUALLY REALLY SOLVE THE PROBLEM. SO WHEN I STARTED TALKING ABOUT COLLECTING THAT DATA ABOUT THE PATIENTS THIS IS WHAT THEY TOLD ME, SOMETHING LIKE THIS. OKAY. GREAT. SO IF YOU ASK WHAT MEASUREMENTS WE CAN COLLECT, IF Y'ALL ARE INTERESTED IN COLLECTING BREATHING SIGNAL FROM THE PATIENT, ARE YOU GOING TO ASK THEM TO WEAR A NASAL PROBE AND THEIR HOME ALL THE TIME OR CHEST ON THEM? IT DOESN'T SEEM EFFECTIVE. YOU CANNOT DO IT CONTINUOUSLY. FOR PATIENTS WHO HAVE PARKINSON'S FOR EXAMPLE OR MULTIPLE SCLEROSIS YOU ARE INTERESTED IN THEIR MOVEMENT. YOU CAN PUT ACCELEROMETERS AND ASK THEM TO KEEP THEM LIKE THAT ALL THE TIME BUT THAT'S NOT DOABLE. FOR YOU ASK PEOPLE TO WE ARE PENDENT AND PUSHED A BUTTON AND FOR SLEEP I'M SURE YOU HAVE BEEN TO A SLEEP LAB WHERE THEY PUT ELECTRODES ON YOUR HEAD AND BODY AND ASK YOU TO SLEEP LIKE THIS. THAT IS NOT WAY TO COLLECT CONTINUOUS DATA AND UNDERSTAND YOUR PATIENT ON A DAILY BASIS. NOT TO MENTION THERE ARE MANY THINGS THAT WE CAN'T MEASURE WITH WEARABLE SO WE RELY ON PATIENT REPORTED OUTCOMES WHICH ARE SUBJECTERS AND YOU HAVE TO ASK PEOPLE THEY HAVE TO REMEMBER, ALL THAT STUFF. WHAT IF SOMEBODY COMES AND SAY I CAN MEASURE FOR YOU ALL OF THIS, AND MANY MORE THINGS WITHOUT ASKING FOR PATIENT TO WEAR SENSORS ON THEIR BODY OR TO CHANGE BEHAVIOR IN ANY WAY. FROM JUST THEY LIVE THEIR LIFE AS USUAL. THIS IS EXACTLY WHAT WE HAVE BEEN DOING WITH MY STUDENTS AT MIT. WHAT WE DID SOMETHING YOU CAN THINK OF ACTS LIKE ADVANCE WIFI LIKE BOX THAT SITS IN THE BACKGROUND OF THE HOME AND USES A WIRELESS SIGNAL IN THE ENVIRONMENT TO MEASURE BREATHING, HEART BEAT, FALLS, INTERACTION WITH CAREGIVER, SLEEP, SLEEP STAGES, APNEA, ALL THAT WITHOUT ASKING THE PATIENT TO WEAR A SENSOR THEIR BODY. WE CALL IT THE EMERALD WIRELES BOX. WHEN I TELL PEOPLE THAT, WHEN I TELL THEM WE CAN MEASURE ALL THESE THINGS, THEY STILL ASK ME WHAT DO I HAVE TO GET THE PATIENT THE WEAR? WRISTBAND, PENDANT? NO, THERE ARE NO WEARABLES NO SINGLE SENSOR ON THE PATIENT BODY THAT IS NECESSARY. IT SHOULDN'T BE VERY CONFUSING OR UNEXPECTED WE LIVE IN A SEA OF WIRELESS SIGNALS SO YOU ARE SWIMMING IN WIFI SIGNAL CELLULAR SIGNAL, MANY MORE YOU DON'T KNOW ABOUT. EVERY SINGLE MOVE THAT YOU DO YOU TAKE A STEP, IT CHANGES THE WIRELESS SIGNAL AROUND YOU. I TOOK A BREATH, THAT CHANGED THE ELECTROMAGNETIC WAVES AROUND ME. EVEN PULSING OF MY BREATH CHANGES ELECTROMAGNETIC WAVE. THIS BOX IS DOING, IT'S A SMART BOX THAT SITS IN THE BACKGROUND, LOOKS AT CHANGES IN THE WAVES AND USES ADVANCED MACHINE LEARNING ALGORITHMS CUSTOMIZED TO OPERATE OVER RADIO SIGNALS AND INFER FROM THAT, OH, HE TOOK A STEP, THIS IS HER BREATHING, THIS IS THE PULSING OF HER BLOOD AND BE ABLE TO GET ALL THIS WITH COMPLETELY CONTACTLESS MANNER. LET ME SHOW YOU SOME -- A VIDEO FIRST TO GET THE CONCEPT. SO THIS IS A HOME, WIRELESS SIGNAL SPREAD INSIDE THE HOME, TRAVERSES WALLS AND OCCLUSIONS AND IT FLEXES OUR BODY, OUR HUMAN BODIES ARE FULL OF WATER AND SOME OF THESE COME BACK TO EARLIER DEVICE WHICH ANALYZES IN THIS CASE EARLY DETECTOR AND CAN ALERT THE CAREGIVER VIA TEXT EMAIL OR MESSAGE. THIS IS AN I WILL TRAYTIVE VIDEO. LET ME SHOW YOU ACTUAL EXPERIMENTS FROM -- THIS IS MY STUDENT IN ONE OF OUR OFFICES AT MIT. OUR DEVICE IS NOT IN THE SAME ROOM, IT IS BEHIND THAT WALL WHERE YOU SEE AN ARROW, SO GOING TO MONITOR HIM FROM ADJACENT OFFICE LIKE SOMEBODY MONITORING US HERE FROM OUTSIDE THIS ROOM. THE IDEA IS TO SHOW YOU WIRELESS SIGNAL TRAVERSE WALS WALLS SO YOU CAN USE ONE DEVICE TO MONITOR MULTIPLE ROOMS. LOOK AT THE RED DOT AND HOW IT IS FOLLOWING HIM. SO YOU CAN SEE THAT IT FOLLOWS HIM ACCURATELY. REMEMBER JUST RELYING HOW HIS BODY IS CHANGING THE ELECTROMAGNETIC WAVES. IT HAS NO SENSORS ON HIS BODY. NO ACCELEROMETERS, NO CELL PHONE, NOTHING. IF YOU HAVE A DEVICE LIKE THIS IN THE HOME, WHETHER CAN YOU TRACK? WHAT ARE -- ONE OF THE THINGS -- I WAS AMUSED TO LEARN THAT DRUGS ARE APPROVED BASED ON SIX MINUTE WALKING TEST WHICH IS LIKE LOOKS A T THE GATE OF THE PERSON OVER SIX MINUTE, FOR EXAMPLE PARKINSON DRUGS IN THAT DOMAIN. AND MY COLLEAGUE PARKINSON DOCTOR SAYS IT DEPENDS, PATIENT COMES ON A BAD DAY, ON A GOOD DAY YOU GET ONE MEASUREMENT, THAT MAY NOT BE REPRESENTATIVE. AND WE KNOW THAT PATIENT WITH THE PERFORMANCE FOR THEIR DOCTORS. SO IF YOU CAN MEASURE THIS CONTINUOUSLY IN THE HOME 24/7. EVEN MORE THAN THAT, ONCE YOU HAVE SOMETHING LIKE THIS IN THE HOME, YOU CAN START ASKING QUESTIONS ABOUT BEHAVIOR, YOU CAN ASK ABOUT THE ETM BEHAVIOR BECAUSE THE DEVICE IS A PERSON AS THEY WALK TO THE KITCHEN HOWEVER THEY TWO THERE, ARE THEY GOING THERE FOR LUNCH, DINNER, BREAKFAST? YOU CAN ASK ABOUT INTERACTION WITH ANOTHER PERSON AND DEPENDENCY ON THEM AND YOU CAN ASK ABOUT THINGS LIKE SLEEP. STARNS OUT, FIRST ALL OF YOU KNOW I'M SURE YOU KNOW WHEN WE SLEEP OUR BRAIN CHANGES WAVES AND WE ENTER LIGHT SLEEP DEEP SLEEP, REM, THE STAGE WE DREAM. AND THESE STAGES ARE IMPORTANT FOR SLEEP DISORDERS BUT THEY ARE ALSO IMPORTANT FOR VARIETY OF DISEASE. FOR EXAMPLE IN DEPRESSION PEOPLE TEND TO HAVE THEM EARLY IN THEIR SLEEP. TODAY IF YOU WANT SLEEP STAGE, YOU SEND YOUR PATIENT, TO THE SLEEP LAB, THEY PUT SENSORS ON HIS HEAD AND BODY, THEY ASK HIM TO SLEEP LIKE THIS. YOU CAN TELL HE'S NOT HAPPY. SURE NOT A SURPRISE TO KNOW THAT. BUT LET ME SHOW YOU HOW WE MEASURE SLEEP STAGES. THIS IS OUR DEVICE, OUR ACTUAL DEVICE TRANSMIT, VERY LOW POWER WIRELESS SIGNAL ANALYZE REFLECTION OF THOSE SIGNAL USING MACHINE LEARNING AN SPITS OUT THE SLEEP STAGE THROUGHOUT THE NIGHT LIKE SLEEP, DEEP SLEEP, REM, IT CAN DISTINGUISH BETWEEN THOSE. WE DID A STUDY WITH MASS GENERAL HOSPITAL, WE HAVE COMPARED IT WITH FDA APPROVED DEVICES FOR SLEEP STAGING. SHOWING THE HIGH ACCURACY. THIS IS A PERSON RIGHT HERE AND WHAT YOU SEE IS NOTHING BUT HIS BREATHING. HIS INHALES, EXHALES. WE CANNED HIM TO HOLD HIS BREATH AND -- ASKED HIM TO HOLD HIS PRETAPED YOU SEE STATE AT STEADY LEVEL, HE EXHALED HE KID -- DID NOT INHALE. WE COMPARED WITH CHEST BAND WITH APPROVED CHEST BAND AND THE COMPATIBILITY IS VERY HIGH, 97%. THIS IS SAME SIGNAL AS BEFORE, THE BREATHING SIGNAL. THESE ARE THE INHALES. THESE ARE EXHALES. YOU SEE SMALL BLIPS ON THE SIGNAL. SO AT THE BEGINNING WE SAW THESE LIKE NOISE. THEN WE DIPPED FURTHER THESE ARE HIS HEART BEATS. SO YOU CAN SEE HIS HEART BEATS, OF COURSE WE HAVE ALGORITHMS THAT SEPARATEED FROM THE HEART BEAT SO WE CAN GET BOTH VITAL SIGNS. SO OVER THE LAST YEAR WE HAVE BEEN WORKING CLOSE WITH OUR COLLEAGUES AND MEDICAL DOCTORS AND PHARMACEUTICAL COMPANIES IN THE BOSTON AREA AND NEW ENGLAND AND DEPLOYED IN MORE THAN 200 HOMES WITH PATIENTS WITH VARIOUS DISEASE AREAS. TYPICALLY WORKING WITH MICHAEL J. FOX FOUNDATION ON PARKINSON WITH HARVARD MEDICAL SCHOOL ON ALZHEIMER DISEASE, ALSO DEPLOYED SOME PATIENTS WITH DEPRESSION AND SOME PATIENT IN CARDIOVASCULAR AND PULMONARY DISEASE COPD. I WILL STOP HERE. AND I WANT TO MENTION ONE LAST THING BEFORE STOPPING. OF COURSE THIS TECHNOLOGY IS QUITE POWERFUL. IT'S COLLECTING PHYSIOLOGICAL SIGNALS WITHOUT TOUCHING THE BODY. WITH POWERFUL TECHNOLOGY OF COURSE VERY IMPORTANT TO BE HIGHLY RESPONSIBLE. WE TAKE THAT VERY SERIOUSLY. SO WE DON'T DO ANYTHING WITHOUT CONSENT OF THE PATIENT AND THE NICE THING IS YOU CAN SEPARATE YOU CAN SAY FOR EXAMPLE YOU ARE MONITORING MOVEMENTS AND YOU DON'T NEED TO MONITOR SLEEP AND USE MACHINE LEARNING MANY TYLE FOR MOVEMENTS, IF SOMEBODY IS MARCHING SLEEP AND NOT MOVEMENT YOU CAN USE THAT MACHINE LEARNING MODULE. SO ALSO OF COURSE LIKE WHO OWN IT IS DATA AND WHO GETS TO DECIDE OUR POLICY IS THAT THE PATIENT HONOR THEIR DATA AND DECIDE WHO GETS ACCESS AND WE USE THE MOST ADVANCED MECHANISM FOR PROTECTING SECURITY AND PRIVACY. THE FUTURE IS VERY, VERY EXCITING, WE HAVE SEEN SO MANY INTERESTING TECHNOLOGIES TODAY BUT I THINK ACTUALLY WITH THESE NEW TECHNOLOGIES, I'M VERY MUCH EXCITED ABOUT THAT WE CAN START THINKING, SO WE HAVE ADVANCE TECHNOLOGY AND ADVANCED MEDICINE BY UNDERSTANDING THE GENOME OVER THE LAST TEN, 15 YEARS. I THINK NOW WE HAVE OPPORTUNITIES TO TO BEHAVIORAL PHONE KNOW TYPING AND UNDERSTAND THE IMPACT OF BEHAVIOR AND ENVIRONMENT AND INTERACTIONS HOW DISEASES CHANGE AND PROGRESS, WITH DIFFERENT PATIENTS. THANK YOU. [APPLAUSE] >> SO WE WILL OPEN IT UP FOR QUESTIONS. STARTING WITH COLLIN. >> GREAT, THANK YOU ALL, FANTASTIC PANEL. SO I HAVE TWO TYPES OF QUESTIONS, THE LAST ONE IS FOR DINA BECAUSE YOUR TALK IS A LITTLE BIT OBVIOUSLY WHAT YOU ARE DESCRIBING PROBABLY A LOT OF US HAVE QUESTIONS ABOUT THAT. MY QUESTION, THERE ARE EXAMPLES FOCUSED ON A SINGLE ACTOR, ONE PERSON WITHIN A SPACE, LOVE TALK HOW YOU HANDLE MULTIPLE BODIES IN A SHARED LIVING SPACE. THE REST OF THE PANEL, REALLY FROM EACH OF YOU, REALLY ENTICING SO SAME THING LIKE FIRST I LOVE THE IDEA OF DELETING EHR. THAT'S SOMETHING WE SHOULD BE THINKING ABOUT AND APPS SHOULD COMPETE WITH RESPECT TO USABILITY AND BRINGING IN WHAT OUR COLLEAGUES SAID AROUND EYE LINING VALUE, HOW DO WE CATALYZE THAT GROWTH AND GET EVERYTHING PACKED SO WE PUSH IN THAT DIRECTION SO THERE'S A CLEAR VALUE WITH RESPECT TO MOVING THE NEEDLE IN USABILITY WHICH IS HARD TO MEASURE. SECOND QUESTION. >> SO I WILL START WITH THAT. IF YOU THE QUESTION ISN'T AN ECOSYSTEM WHERE YOU HAVE INNOVATORS WHO CAN DISTRIBUTE MACHINE LEARNING ALGORITHMS, GENOMIC ALGORITHMS GENOMIC SUPPORT DATA VISUALIZATION IN AN APP STORE TYPE MODEL WHICH WE HAVE SEEN WORK. IF IMPLEMENTED WELL, THE APP STORE MODEL WORKS EXTREMELY& WELL. THEN I THINK YOU ARE ASKING THE RIGHT QUESTION, HOW DO YOU REWARD THE RIGHT APPS AND THE RIGHT ALGORITHMS AND THE RIGHT VALUE COMING OUT OF IT. SO THE -- I THINK THERE'S BOTH AN OPPORTUNITY AND A DANGER. I THINK THAT WE AGREE, VALUE BASED CARE IS THE RIGHT BASIC IDEA. FOLKS IN THIS ROOM IMPLEMENT IT IN THE RIGHT WAY. THERE ARE ALSO GOING TO BE VERY STRONG FORCES THAT TAKE AHOLD OF VARIOUS BIOMARKERS, AND OTHER INDICATORS THAT COME UP FROM THE KINDS OF METHODS WE ARE TALKING ABOUT. AND WILL MARKET THE HECK OUT OF PARTICULAR THINGS. SO I WROTE A PIECE IN JAMA RECENTLY CALLED BIOMARKUP. WHICH HAS TO DO WITH THE FACT THAT EVERY BIOMARKER IS A REVENUE OPPORTUNITY FOR SOMEONE PARTICULARLY WHAT WE EMPHASIZE IN THIS PIECE IS WHILE THE MORE TRADITIONAL BIOMARKERS WHICH ARE POWERFUL ENOUGH IN THIS RESPECT, FOR EXAMPLE, CHOLESTEROL, WHEN THE AMERICAN HEART ASSOCIATION AND COLLEGE OF CARDIOLOGY, CHANGED GUIDELINES OR THRESHOLD OF CHOLESTEROL NEEDED TO BE CONSIDERED PUT ON STATIN, THE NUMBER OF PEOPLE ELIGIBLE WENT FROM 5 MILLION TO 20 MILLION. IN THE CONTEXT OF A $23 BILLION INDUSTRY, YOU GOT TO LOOK AT THAT MOVE AND SAY WELL, THERE'S SOME ECONOMIC IMPLICATIONS HERE. IF YOU LOOK AT THE DIFFERENT GUIDELINESTOR MAMMOGRAPHY FOR EXAMPLE, WHICH HAS A VERY CHECKERED EVIDENCE HISTORY, THE FOLKS ADVOCATEING ARE OFTEN ONES FOR WHOM THERE IS SOME REVENUE IMPLICATION. IF YOU LOOK AT GENOMICS WE ARE MULTIPLYING BIOMARKERS BY THOUSANDS AND THOUSANDS PER TEST, OR IF YOU LOOK AT AICS YOU CAN SEE IN THIS ROOM, THE NUMBER OF AI ALGORITHMS WILL BE OUT THERE. SO TOE ANSWER IN MACHINE LEARNING SPECIFICALLY, I THINK THAT THE DISTRIBUTION OF THESE THINGS WILL BECOME VERY TRIVIAL HOPEFULLY, THAT'S THE ONE IDEA IN THE USABILITY AND THEY WILL COMPETE ON HOW THEY INTERACT WITH THE END USER. BUT IN TERMS OF GETTING VALUE OUT OF IT, THERE'S GOING TO BE -- IT WILL TAKE A LOT OF ATTENTION ON THINKING ABOUT THIS FROM A SYSTEMS PERSPECTIVE. YOU HAD A WHOLE SYSTEMS DIAGRAM, YOU ARE THINKING ABOUT A SYSTEMS PERSPECTIVE BUT HOW DOES THE WHOLE THING COME OUT TO PRODUCE? ONE STRONG ADVOCATE FOR A BIOMARKER AND THEN THAT'S THIS IDEA WITHIN THAT OF REGULATORY CAPTURE, OUR OWN REGULATORY AGENCIES END UP BEING IN REGULATIONS END UP BEING USED TO PROMOTE SOME OF THESE APPS, CMS STAR RATINGS. YOU HAVE TO PUT EVERYONE ON A STAT. SO IN OTHER WORDS, IT GETS COMPLICATED. WE KNOW THE ERR REGULAR RAYINGS IN THE -- EHR REGULATIONS ARE USED IN THE PAST THIS WAY SO IT'S COMPLICATED ANSWER AND SYSTEMS THINKING AND END TO END SORT OF THINKING IN ORDER TO TRY TO SHAPE THAT. BUT BECAUSE I THINK WE ARE SURE THAT IT'S COMING, GOOD TIME TO THINK HOW TO SHAPE IT IN A RESPONSIBLE WAY. >> TO ANSWER YOUR FIRST QUESTION ABOUT MULTIPLE PEOPLE. ALL THE EXAMPLE I SHOWED A SINGLE PERSON BECAUSE IT'S EASIER TO MATCH WHERE THE PERSON IS MOVING BUT IF THERE WERE TWO PEOPLE FOR EXAMPLE, THE GREEN DOT MOVING AND RED DOT AND INTERACT THEY INTERACT TOGETHER. ONE IMPORTANT QUESTION ALSO THAT PEOPLE ASK, HOW DO YOU KNOW WHO IS WHO? SO WE CAN GET RED DOT AND GREEN DOT, WHICH PATIENT IS A CAREGIVER. SO WE USE THE FIRST FEW DAYS OR DEPENDING HOW MUCH DATA WE GET, ARE A WEEK FOR EXAMPLE TO COLLECT DATA AND ASK THE PATIENT TO WELL A SMALL ACCELEROMETER AND MATCH THE WIRELESS DATA WITH ACCELERATION DATA NOT TO GET THE ACCELERATION DATA TO SAY WHEN THIS ACCELERATION IS MOVING AND MATCHING THE WIRE SIGNAL, THIS IS THE SAME PATIENT, THEN TAKE THAT DATA WE CREATE A CLASSIFIER FOR THAT INDIVIDUAL VERSUS THE REST OF THE WORLD. THAT ALLOWS US TO KNOW THE RED DOT ACTUALLY IS BARBARA, THE GREEN DOT IS CHRIS. THAT IS THE ONLY CALIBRATION NEEDED IF YOU WANT TO KNOW WHO IS WHO, BUT FOR THE PHYSIOLOGICAL SIGNAL, IT DOESN'T NEED CALIBRATION. >> THANK YOU. >> AS YOU CAN SEE THERE ARE GOOD REASONS DINA IS A MACK ARTHUR FELLOW. >> FASCINATING TALKS. IN THE MORNING WE WERE TALKING ABOUT THE TRUST, EXPLAINABILITY AND ACTIONIBILITY EMERGEED FROM THAT. THE THOUGHT AFTER HEARING THIS, USABILITY IS POTENTIAL BENEFIT THEN THERE'S UTILITY WHICH IS ACTUALLY REALIZED BENEFIT. AS A COMMUNITY, WHENEVER WE TALK ABOUT ML IT'S MODEL CENTRIC THINGS, TRANSPARENCY, TRUSTWORTHINESS, USABILITY EXPLAINABILITY. BUT THERE'S TWO OTHER THINGS, THERE'S THE POTENTIAL FOR NET BENEFIT, WHICH IS WHAT THE ECONOMIST TYPICALLY FOCUSES ON IN ABSENCE OF THESE RESOURCE CONSTRAINTS, THEN THERE'S REALIZED UTILITY YOU CAN ACCOMPLISH UNDER CONSTRAINTS OF INCENTIVES AND REAL WORLD HEALTH SYSTEM THAT KEN ALLUDED TO. SO WHAT ARE THE PANEL THOUGHTS ON EVALUATION REGIMES BEYOND THE MODEL BUT ALSO POTENTIAL FOR UTILITY, NOT USABLE, UTILITY, AND REALIZING OR CAPTURING IT? >> WE HAVE TO HAVE THAT BE PROBABLY THE DOMINANT CRITERIA FOR JUDGING USE OF MACHINE INTELLIGENCE ARTIFICIAL INTELLIGENCE MACHINE LEARNING IN HEALTHCARE SETTING. WE ARE NOT TRYING TO BUILD TOWARDS GETTING A GOOD AUC IN THE END. WE ARE TRYING TO BUILD TO MAKING PATIENTS BETTER AND CLINICIANS BETTER IN REDUCING WORKLOAD, THINGS LIKE THAT. SO I ABSOLUTELY AGREE WITH YOU, THAT UTILITY FUNCTION, WE DON'T USE THAT IN MACHINE LEARNING, AND PROBABLY WE MAY NOT NECESSARILY DO THAT IN THE ACTUAL MODEL BUILDING BUT IF WE PUT THAT IN CONTEXT OF OOHED MINISTER ORGANIZE HELPING MAINTAIN HEALTH AMONG POPULATIONS, I ABSOLUTELY AGREE THAT THAT'S A FUNCTION WE NEED TO MEASURE. THAT IS WHY I ASSERT BUILD A REGISTRY WITH EVERY ALGORITHM WE DEPLOY IN HEALTHCARE SETTING TO REGISTER THAT SIGNAL. >> I WANTED TO ADVOCATE LOOKING AT UTILITY AS WELL, THE REFERENCE I MADE IN THE ARTICLE IN JAMA ABOUT UTILITY FOLKS SHOULD LOOK AT IT, IT SUGGEST AS FRAMEWORK FOR MEASURING UTILITY, IT'S THE NUMBER NEEDED TO BENEFIT AGAINST A CERTAIN NUMBER OF OTHER FACTORS TOO LIKE COST TO ACTUALLY IMPLEMENT PREDICTIVE ANALYTICS IN THE SYSTEM, ET CETERA. (OFF MIC) >> YOU ARE? MY GOSH. I'M SORRY. I DIDN'T CONNECT YOUR NAME WITH THE LIST ON THAT BUT YES ABSOLUTELY, IT'S IMPORTANT. >> SHOULD HAVE INTERRUPTED (OVERLAPPING SPEAKERS) >> I THOUGHT IT WAS A GREAT ARTICLE. DAVID, THANKS TOO. SO I THINK THE IDEAL COMBINATION IS WE NEED TO LOOK AT USABILITY AND UTILITY. THEY ARE BOTH VERY IMPORTANT. >> ALSO GOING BACK TO SOME OF THE IDEA THAT YOU MENTIONED WHICH IS BASICALLY YOU CAN MEASURE STUFF BUT WHAT -- ARE THEY ACTIONABLE? DO THEY GIVE SOMETHING THAT YOU CAN ACT ON AS IN THE -- GIVEN THE HEALTHCARE SYSTEM AND YOUR UNDERSTANDING OF IT. AND THIS IS ONE OF THE THINGS PEOPLE COME AND SAY CAN YOU MEASURE THIS, THIS IS INTERESTING FOR ME AS PARTICULAR DISEASE AREA, AND THEN WE ASK THEM OKAY IF YOU KNOW THE ANSWER, WHAT WOULD YOU DO ABOUT IT? CAN'T DO ANYTHING, IT DOESN'T BENEFIT YOU IN UNDERSTANDING THE DISEASE, SCHOOL, BUT IT'S PROBABLY NOT THAT INTERESTING. VERY INTERESTING TALKS, THANKS. I HAD A FOLLOW-UP ON YOUR COMMENT ABOUT HOW YOU IDENTIFY FOR DINA HOW YOU IDENTIFY -- GIVE YOU TWOABLE DIFFERENT PATIENTS SHE DID BASELINE MACHINE LEARNING TO CLASSIFY THEM, WHAT IF INTERVENTIONS CONDUCTED, HOW WOULD THAT -- YOU START TO CHANGE SOME OF THE -- HOW WOULD THAT CHANGE THE STUFF? >> BASICALLY THEY ARE MULTIPLE THINGS, ONE THING IS HOW FAST, DOES IT FLIP OVERNIGHT THAT BEHAVIOR. SO MOST LIKELY IT DOESN'T. BUT IF -- SAY FOR EXAMPLE, THE PATIENT -- YOU HAVE A PARKINSON PATIENT AND THEY CHANGE LIKE THEY STAGE ESCALATED AND NOW THEY HAVE MORE TREMOR AND THEY ARE MOVEMENT IS DIFFERENT, BECAUSE THIS CHANGE IS HAPPENING OVER TIME YOU CAN TAKE YOUR MACHINE LEARNING MODELS AND ADAPT THEM. GRADUALLY. WE HAVEN'T DEPLOYED YET BECAUSE MOST STUDIES ARE ACTIVELY SHORT NOW. BUT THIS IS DOABLE. THE OTHER THING WHEN YOU KNOW SOMETHING IS INSTANTANEOUS, WHEN SOMEONE HAS SURGERY AND THINGS CHANGE WE SEND ACCELEROMETER FOR THE NEXT WEEK AGAIN SO THAT WE ADAPT CHANGE IN THE CLASSIFIER. >> FASCINATING APPROACH TO THE MEDICARE ISSUE. AND WHAT I WANTED TO ASK YOU ABOUT IS YOU PRESENTED THIS AS A KIND OF DESIRABLE -- OBVIOUSY VERY DESIRABLE OBJECTIVE WHICH IS REDUCTION IN HOSPITALIZATION. SO IT SOUNDS LIKE A VALUE CREATION. BOTTOM LINE, YOUR BOTTOM LINE IS REALLY ENHANCING YOUR OVERALL REVENUE GENERATION, WHICH IS VALUE EXTRACTS. SO HOW -- EXTRACTION. SO HOW DO YOU ACTUALLY HANDLE, I WOULD ASSUME THE CFO OF YOUR INSTITUTION IS VERY MUCH INTERESTED IN THIS, HOW DO YOU GO ABOUT EVALUATING POTENTIAL OPTIONS THAT ARE OVERALL COST SAVINGS TO THE SYSTEM, SOME OF THOSE COST SAVINGS INCLUDE REDUCTION IN REVENUE TO YOUR INSTITUTION, HOW DO THOSE GET THERE, GET -- HOW DO THOSE SEE THE LIGHT OF DAY BASICALLY? >> YOU ARE DEFINITELY ADDRESSING THE -- ONE OF THE BIFURCATED ISSUES THAT WE HAVE IN ANY ACADEMIC HEALTH SYSTEM. BECAUSE OUR MSB PROGRAM IS A SMALL PORTION OF TOTAL REVENUE. NO DOUBT ABOUT IT, THE POPULATION HEALTH MANAGEMENT OFFICE HAS BEEN LAUDED FOR PERFORMANCE IN THAT NARROW FIELD. BUT IF ANY OF YOU READ -- LAST YEAR DUKE WAS LISTED AMONG TOP 12 HEALTH SYSTEMS AND OPERATING PROFIT WHICH IS PRETTY AMAZING CONSIDERING DURHAM IS A SMALL MARKET. WE HAVE OPTIMIZED FEE FOR SERVICE MEDICINE TO THE INth DEGREE IN SPITE OF PROGRAMS LIKE MSSP. SO PART OF THAT HAVE, WHEN YOU THINK ABOUT IT, THE INCENTIVES ARE PERVERSE, SO I HAVE FOR INSTANCE ASSOCIATED WITH WHERE WE PREDICT SURGICAL COMPLICATIONS. AT A PLACE LIKE DUKE RIGHT NOW WE ARE NOT ALIGNED, IF WE WERE REDUCING SURGICAL COMPLICATIONS AT DUKE WHICH IS A VERY TRADITION, I TRAIN IN THAT TRADITION IN GENERAL SURGERY AND CARDIAC SURGERY, WE REDUCE REVENUE. SO THERE ISN'T I'LL CONCEDE, OUR HEALTH SYSTEM ADMINISTRATOR LEVEL WE ARE NOT UNIFIED MIND IN TERMS OF DOING THAT, WHAT I DO APPRECIATE ABOUT THE MSP PROGRAM IS WITHIN THE CONFINES OF THAT VALUE BASED REIMBURSEMENT PROGRAM, INCENTIVES ARE ALIGNED. THAT'S PART OF THE REASON WE INTENTIONALLY DID THAT FIRST DEPLOYMENT OF MAJOR MACHINE LEARNING INITIATIVE AT DUKE WITHIN THE CONTEXT OF THAT. I KNEW THAT I DIDN'T HAVE TO DEAL WITH SOME OF THIS IMPEDE DENSE MISMATCH BETWEEN MOTIVATIONS OF CFO AND MOTIVATION OF THE PATIENT THAN PROVIDERS. >> I THINK THIS FOLLOWS UP ON THAT QUESTION, WHO IS THE USER? IT'S HEALTHCARE SYSTEM, IT'S PATIENT, DOCTORS, THE CAREGIVER CAREGIVERS, IT'S PUBLISH. SO HOW DO YOU ACTUALLY SEE THINKING ABOUT MACHINE LEARNING APPROACHES AND HOW WE CAN ACTUALLY ADDRESS ALL OF THE DIFFERENT USERS? >> TO ME ALL THE BETTER IF YOU CAN GET PATIENT PROVIDER AND THE PAYER RAY LINED ON THESE THINGS. IT IS HARD BUT THAT'S THE POINT OF VALUE BASED REIMBURSEMENT MODEL. THE REASON THAT ACCOUNTABLE -- AFFORDABLE CARE ACT INCLUDED THESE INNOVATION MODELS FROM CMMI, WAS TO -- CAN WE TEST THIS. BECAUSE RIGHT -- MOST PEOPLE THINK MARKET FORCES WE'RE A FREE MARKET ECONOMY AND MARKET FORCES WORK IN HEALTHCARE, ALL OF US KNOW THAT'S NOT THE CASE. THESE EXPERIMENTAL VALUE BASED FRAMEWORKS IN THEORY PROVIDE THE ALIGNMENT OF FUNDAMENTALLY COMMERCIAL INCENTIVES IN A FREE MARKET ECONOMY LIKE OURS BUT ALLOW HOPEFULLY MARSHALL IN THE INTEREST IN THOSE THREE DIFFERENT Ps. HAVE THEY FIGUREED THAT OUT WELL YET? KEN REFERRED TO ISSUES WHERE THERE ARE PERVERSE THINGS WE CAN TO IN DIFFERENT DIRECTIONS BUT COMPARED TO WHAT WE HAVE BEEN DOING IN A FEE FOR SERVICE WORLD, I THINK AGAIN LIKE I SAID TO POLICY MAKERS A COUPLE OF WEEKS AGO HERE IN D.C., THAT ALIGNMENT IS FAR MORE MOTIVATING FOR US THAN ANY METHODOLOGICAL OR TECHNICAL ADVANCEMENT WE HAVE IN MACHINE LEARNING RIGHT NOW. >> SO I WANT TO ASK A GENERAL QUESTION FOR THE PANEL. YOU SPEAK OF THESE APPS THAT COULD HELP PEOPLE WITH MEDICATION OR THINGS LIKE ASSIST WITH MONITORING THEIR HEALTH. AND YOU ARE TALKING POSSIBLY DOING THIS THROUGH LIKE SOME SORT OF APP SITE LIKE DOWNLOAD APPLE DOES KIND OF THING. MY QUESTION WOULD BE HOW WOULD YOU CONTROL QUOTE UNQUOTE OFF LABEL USE I GUESS? OR ALSO MAYBE SIDE EFFECTS OF INCORRECT USE? >> THINK ABOUT THE APPS IN TWO DIFFERENT MODES INITIALLY. ONE IS PATIENT FACING. ONE IS HOSPITAL AND PHYSICIAN FACING. HOSPITAL AND PHYSICIAN FACING, IT'S AN EXTENSION TO EHR, IN THE HOSPITAL ENVIRONMENT, SO BE CONTROLLED BY THE SAME ENTITIES CONTROLLING THE OTHER APPS. PRESUMABLY IT WOULD BE SOMETHING THAT WAS SELECTED BECAUSE IT'S USEFUL. THERE WE ARE LESS CONCERND WITH THE OFF LABEL USE THOUGH STILL POSSIBLE IT NOT BE USED CORRECTLY BY PHYSICIANS. ON THE PATIENT FACING SIDE THERE'S AN INTERESTING PROPERTY OF THE RULE THAT'S COMING DOWN. WHICH IS THAT BECAUSE THERE A SPECTRUM OF PATERNALISM AND LIBERTARIANISM, WE -- THE THAT HAS A BIG IMPACT WHAT PATIENT COMMUNITIES CAN DO. PATIENT COMMUNITIES WANT APPS TO GET DATA, MOVE AROUND HELP THEMSELVES DRIVE DISCOVERY CONNECT WITH RESEARCHERS WHO WILL UNDERSTAND MECHANISM OF DISEASE, MOVE EXOME AROUND FROM ONE PLACE TO ANOTHER. WITH THE CLINICAL DATA, BLAH, BLAH, BLAH. PATIENTS WANT THIS, NO QUESTION ABOUT IT. BUT THERE IS A RISK. AND IN THE ONC RULE THEY HAVE COME DOWN IN A LIBERTARIAN PERSPECTIVE. A PATIENT ANY APP. SO THERE'S REASONABLE CONCERN OF PREDATORY APPS BECAUSE THERE ARE NOW. IF YOU GO TO APPLE APP STORE, MOST OF THOSE APPS TO NOT CONNECT THROUGH API. THEY ARE SEPARATE. MANY OF THEM HAVE PREDATORY BUSINESS PRACTICE, THIS IS WELL DOCUMENTS. HOW ARE WE GOING TO DO THIS? OFF LABEL IS FOR THE FEW APPS THAT GET PRODUCED ACTUALLY FDA APPROVED, THAT'S ONE ISSUE. THOSE THINGS ALSO WILL BE ABLE TO BE USED OFF LABEL. BUT MANY OTHER SORTS OF APPS WILL NOT BE CONTROLLED FROM PRIVACY OR TECHNOLOGY ANY PERSPECTIVE BY HIPAA. OR BY THE FDA. THEY WILL BE CONTROLLED BY FTC. OUR GROUP IN THIS THE F -- THINKS THE FTC NEEDS MORE FRAMEWORK THAN THEY HAVE BEEN APPLYING TO SAY GENERAL CONSUMER IT SPACE WHICH AS WE KNOW HASN'T GONE PERFECTLY WELL. SO WE THINK THERE IS AN OPPORTUNITY TO HELP JUICE UP WHAT THE FTC IS ALLOWED TO DO OR HELPS OR EXPECT WHAT IS THOSE TERMS OF USE LOOK LIKE WITH THOSE APPS BUT THIS ISSUE YOU BRING UP IS A REAL ONE. >> AT DUKE, I WAS TALKING WITH COLLIN ABOUT THIS AT LUNCH AS WELL. SOME OF THE LEGAL SCHOLARS FOCUSING ON THIS ISSUE HAVE SAID LOOK, WE HAVE SEEN WHAT'S APPROXIMATE HAPPENED WITH FACEBOOK, THE POSSIBILITY OF PREDATORY APPS AND HOW VENDORS DO THINGS IN MALICIOUS WAYS ONCE IT CROSSES OUTSIDE OF HIPAA REGULATION, ONCE OUTSIDE THAT COVERED ENTITY, IT'S THE WILD WEST. AND THE APPROACH THEY SUGGEST IS THINKING ABOUT IT LIKE GINA GENETICS INFORMATION NON-DISCRIMINATORY ACT AND IT'S THE USE OF THOSE DATA THAT ARE REGULATED AS OPPOSED TO TRYING TO REGULATE ALL THE ACCESS POINTS TO GETTING THOSE DATA BECAUSE IN THE END THAT'S A LOSING BATTLE FROM A TECHNICAL STANDPOINT. THEIR SUGGESTION IS WE HAVE EQUIVALENT TO GINA, MAYBE CALL THE DINA. WHICH REGULATE ON THAT END YES YOU CAN PREVENT PREDATORY APPS FROM GETTING THE DATA UNDER ONC RECOMMENDATIONS THAT WILL BE DIFFICULT TO PARSE OUT AND REALLY DO IN A SCALABLE WAY. BUT WHAT YOU COULD REGULATE IS HOW THEY USE THAT DATA ONCE THEY GOT ACCESS TO IT. >> I LIKE THE DINA THING. I THINK THERE IS A DIFFERENT ACCESS ALSO LIKE HOW -- WHAT IS THE ASSOCIATED WITH THE APP IS DOING. FOR EXAMPLE, I CAN IMAGINE LIKE THE DEVICE COULD BE USED IN -- HAS SO MANY USE PATTERNS OFF LABEL, IF YOU WOULD LIKE. THAT COULD BE VERY USEFUL AND VERY INTERESTING. FOR EXAMPLE YOU CAN IMAGINE SOME DEVELOPMENT WIDE DEVELOPMENT APP ON TOP OF THIS, IF GRANDMA HAS ALZHEIMER SAY IN YOU CAN SENSE WHETHER THE STOVE IS ON AND SHE'S LIKE ASLEEP, THEN TURN OFF THE STOVE FOR EXAMPLE. OR YOU ASK FOR SECURITY AND IF SOMEBODY IS OUTSIDE THE HOME AND THERE IS SOMEBODY MOVING IN THE HOME THEN YOU HAVE AN ALERT. SO THERE ARE BECAUSE THE RESPIRATORY DEVICES NO INTERVENTION LIKE WIRELESS SIGNAL TYPICAL SIGNAL YOU HAVE IN THE ENVIRONMENT, THE DANGER IS -- OF COURSE THERE IS A DANGER WITH PRIVACY AND SECURITY WHICH HAS TO BE PROTECTED. SO THERE ARE MULTIPLE AXES FOR THIS, NOT JUST WHAT IS -- HOW DANGEROUS IS THE OFF LABEL USE AND WHAT ARE YOU DEALING WITH. >> IF THESE THINGS FALL UNDER THE FTC JURISDICTION OR AT LEAST OUTSIDE OF FDA AND SOME OTHER THREE OTHER ACRONYM FEDERAL AGENCY, FTC FOR EXAMPLE CAN PROSECUTE COMPANIES FORMALS ADVERTISING. IT'S ONE OF THE STRONGER THINGS THEY CAN USE TO GO AFTER COMPANIES FORMALS CLAIMS. IN HEALTHCARE -- FOR FALSE CLAIMS, CLAIMS ARE HARD TO ESTABLISH. YOUTUBE HAS A HUGE PROBLEM WITH FALSE MEDICAL CLAIMS AND TREATMENTS IN DIABETES PROLIFERATING ON YOUTUBE. I LOVE YOUR THOUGHTS HOW TO COMBINE THAT ASPECT WITH THE STILL FUZZY ASPECT OF UTILITY. I'M GLAD YOU BROUGHT UP TO BENEFIT, I THOUGHT NO ONE NOTICED THAT BUT GLAD SOMEONE DID. BUT IT'S STILL HARD. WHEN WE ARE DOING THAT ANALYSIS TURNS OUT THAT BY PUTTING UTILITY NUMBERS -- YOU CAN MAKE IT LOOK LIKE WHATEVER YOU WANT, AND THE REASON THE PROPOSAL IS SIMPLIFIED IS TO GIVE YOU DIRECTIONAL ESTIMATE BECAUSE THE ABSOLUTE MAGNITUDE YOU CAN SET TO WHATEVER YOU WANT BY RANDOM CHANGES. SO GIVEN THE INHERENT CHALLENGE IN DEFINING UTILITY, WHICH IS DIFFERENT FOR THE INSURER VERSUS THE INDIVIDUAL VERSUS THE SOCIETY AND PROVIDER, HOW DO YOU RECONCILE THOSE? HOW WOULD WE COMBINE VARIOUS NOTIONS OF UTILITY WITH VARIOUS NOTIONS OF REGULATION THAT WE NIGHT NEED FOR PURPOSE OF BENEFITING HEALTHCARE USING ALGORITHMS. >> THAT'S A TOUGH QUESTION. IT'S VERY -- IF YOU LOOK AT THE U.S. HEALTHCARE SYSTEM IT'S VERY HARD TO REGULATE AROUND REVENUE GENERATION. EVERY COMPONENT OF THE HEALTH SYSTEM IS GOING TO TRY TO DO REVENUE GENERATION. SO WE -- THIS BIOMARK UP PAPER WE SIMULATED A CURVE THAT JUST SHOWED IF YOU TOOK THE GOOGLE RETINOPATHY -- DIABETIC RETINOPATHY DETECTION ALGORITHM AND TUNED IT JUST ONE WAY OR THE OTHER, YOU CAN EITHER HAVE A LOT OF OPHTHALMOLOGY KNOWLEDGE REFERRALS OR JUST A FEW. EVERY ONE OF THESE ALGORITHMS WILL HAVE HUGE IMPLICATIONS. IT COULD BE UNCLEAR GENERALLY TO THE END USER WHAT THE DECISIONS WERE MADE IN THAT CON TEXT. I DON'T KNOW THAT'S GOING TO BE AN EITHER AN FDA OR AN FTC OR A OFFICE CIVIL RIGHTS TYPESET OF DECISIONS. THOSE ARE BIGGER, THOSE ARE BIGGER SOCIETAL DECISIONS. IN TERMS OF THIS DIVISION BETWEEN FTC AND HIPAA WORLD, THAT'S WHERE I THINK THERE'S GOING TO HAVE TO BE A LOT OF PUBLIC EDUCATION QUICKLY. IF YOU LOOK AT APPLYING HIPAA TO THESE APPS, IT JUST TOTALLY DOESN'T WORK. THEY ARE NOT PART OF A COVERED ENTITY. THE WHOLE -- THAT'S THE WHOLE POINT. SO SOME PROPOSALS TO ADDRESS IT WITH HIPAA I THINK WHEN YOU LOOK AT IT IN DETAIL, JUS CAN'T BE DONE. I DO LIKE THE DINA IDEA TOO. AND IT'S REALLY ABOUT PLACES LIKE WHAT'S GOING ON AT DUKE IS THINKING VALUE IN THESE WAYS THAT PUT SOME EMPHASIS ON THE SYSTEMS PERSPECTIVE OPPOSED TO THE INDIVIDUAL COMPONENTTRY IN THE FEE FOR SERVICE MODEL WHERE EVERY COMPONENT IS NOT OPTIMIZED BUT MAXIMIZED FOR REVENUE. ARE WE ON TIME? OKAY. NICE. [APPLAUSE] >> WE'LL COME BACK. WE HAVE A BREAK RIGHT NOW, COME SO I WANT TO START OUT BY INTRODUCING OUR CHAIR, WHO IS MAXINE MACKINTOSH. SHE'S A RESEARCHER AT THE ALAN TURNING INSTITUTE AND CO-FOUNDER OF ONE HEALTH TECH. WELL DONE ON THE LAST SESSION. THE NIH WILL SEND YOU 100 POUNDS FOR ME BEING HERE ALL DAY, WHETHER I MAKE IT TO THE LAST SESSION. AMAZING THING HAPPENED AT LUNCH, I FOUND OUT THREE OF FOUR PANELISTS THE NEXT SESSION HAVE AN INTIMATE RELATIONSHIP WITH A HOSPITAL IN OXFORD IN THE UK WHERE I'M FROM. AND I WAS BORN IN THE GIANT CARE HOSPITAL, MATTHEWS TAUGHTER WAS BORN IN THE HOSPITAL AND ALSO BORN IN OXFORD SO WHO WOULD HAVE THOUGHT THREE HAVE NOTHING NO COMMON OTHER THAN THE FACT THEY HAVE CHILDREN THEY THEMSELVES BORN IN OXFORD. SO I'M SPEAKING LAST BECAUSE MY TALK IS ABOUT GIVING CONCRETE EXAMPLES HOW WE LOOK AT TRANSPARENCY IN THE UK. WITHOUT FURTHER ADIEU I WOULD LIKE TO INTRODUCE THE FIRST SPEAKER WHO IS SEZIN PALMER, MISSION AREA EXECUTIVE FOR THE NATIONAL HEALTH JOHNS HOPKIN UNIVERSITY APPLIED PHYSICS LABORATORY. I FIND IT SILENT SO HUGE ROUND OF APPLAUSE FOR SEZIN. [APPLAUSE] >> >> I FEEL LEFT OUT, THE ONLY ONE WITH NO CONNECTION TO THE HOSPITAL IN OXFORD. THEY LET ME STAY PART OF THE PANEL ANYWAY. SO GOOD AFTERNOON. THANKS VERY MUCH FOR THE OPPORTUNITY TO BE HERE. I WILL SHARE WITH YOU A LITTLE BIT OF WHAT WE HAVE BEEN WORKING ON AT JOHNS HOPKINS TO BRING MACHINE INTELLIGENCE TO HEALTHCARE. THE STATE IT WILL OF MY TALK HERE REFLECTS THE TITLE OF OUR PROJECT, PRECISION ANALYTICS PLATFORM, I WILL TELL YOU ABOUT THE PLATFORM BUT NO SURPRISE TO ANYONE IN THIS ROOM WHEN WE STARTED LOOKING WAYS TO BRING DATA SCIENCE AND DATA ANALYTICS AND MACHINE LEARNING TO HEALTHCARE JOHNS HOPKINS ONE THING WE REALIZED IS THE DATA IS SPREAD OUT ACROSS 5 HON DATABASES ACROSS THE INSTITUTION SO THE FIRST THING WE HAD TO DO IS BRING IT TOGETHER IN A PLACE WE COULD ACTUALLY TAKE ADVANTAGE OF ALL THE DIFFERENT DATA SOURCES. SO WE STOOD UP THE PLATFORM, IT WAS DEPLOYED AND WENT LIVE THIS PAST MAY. I WILL TOUCH ON THE ANALYTICS WE HAVE DEVELOPED THAT ARE NOW OPERATIONAL IN THE PLATFORM. IS THE I WILL BE TALKING ABOUT SOME OF THE ISSUES IN TERMS OF BIAS TRANSPARENCY TRUST AS WE GO THROUGH IT. SO AS I -- WE ARE GOOD, RIGHT? EVERYBODY CAN HEAR ME? THE PRECISION MEDICINE ANALYTICS PLATFORM REPRESENTS THE VISION THAT WE WENT INTO WITH THIS PROJECT. AS WAS POINTED OUT EARLIER, EVERYBODY LOVES ELECTRONIC MEDICAL RECORDS AND HOPKINS IS PARTICULARLY FOND OF EPIC WHICH IS THEIR MEDICAL RECORD, DESPITE THAT, WE IDENTIFIED MANY OPPORTUNITIES TO PULL DATA OUT OF EPIC AND COMBINE IT WITH MANY OTHER DATA SOURCES. AND THEN APPLY VISUALIZATION AND DATA ANALYTICS TO MAKE IT MORE USABLE. SO WE HAD A VISION OF COMBINING VARIOUS DIFFERENT DATA SOURCES GOING TO TALK MORE ABOUT THE ONES THAT ARE IN THE PLATFORM. BUT IDEA IS TO BRING THEM TOGETHER IN A WAY THAT ALLOWS US TO CREATE A LEARNING HEALTH SYSTEM. SO BE ABLE TO RAPIDLY TRANSLATE FROM RESEARCH INTO CLINICAL CARE. THAT WAS THE IDEA REPRESENTED IN THE MIDDLE AND THEN OF COURSE THE ULTIMATE GOAL OF THIS EFFORT IS TO IDENTIFY MECHANISTICALLY ANCHORED SUBPOPULATIONS OF PATIENTS ACROSS VARIOUS DISEASE CATEGORIES. THAT WAS THE VISION. THIS REPRESENTS THE DIFFERENT DATA SOURCES, I MENTIONED EPIC AS ONE OF OUR BIGGEST DATA SOURCES, WE HAVE IMAGING DATA ACROSS THE INSTITUTION WE HAVE PHYSIOLOGICAL MONITORING DATA SO THIS IS REAL TIME DATA BEING STREAMED FROM HUNDREDS OF ICU BEDS ACROSS HOPKINS MEDICINE AND WE HAVE GENOMICS DATA OBVIOUSLY LIKE ONE PRETTY EXAMPLE OF A PLUS ONE ALGORITHM BUT THE IDEA IS TO LOOK ACROSS DATA SETS AND BRING IN ADDITIONAL DATA SETS AS WELL AND SEE WHAT WE CAN DO IN TERMS OF APPLYING VARIOUS MACHINE LEARNING APPROACHES. OF COURSE HERE ARE SOME OF THE CHALLENGES WE HAVE BEEN DISCUSSING TODAY, I WILL TOUCH ON WHAT WE ARE WORKING ON IN EACH AREA. SO MACHINE LEARNING TOOLS AMPLIFY HUMAN BIAS, HOW DO WE ACCOUNT FOR THOSE KINDS OF ISSUES. WE HAVE TALKED ABOUT REPRESENTATIVE DATA SETS IN TERMS OF HOW DO WE ENSURE THAT THE DATA THAT WE ARE USING FOR TRAINING FOR PARTICULAR MACHINE LEARNING ALGORITHM IS APPROPRIATE FOR THE POPULATIONS WE ARE TRYING TO APPLY. I HAVE USING APPROPRIATE MACHINE LEARNING APPROACHES HERE BECAUSE WE ARE TALKING EARLIER THIS MORNING ABOUT HOW WELL WE UNDERSTAND DECISION PROCESS THAT THESE DIFFERENT APPROACHES ARE COMING UP WITH AND HOW WE TRUST RESULTS. SO DEPENDING WHETHER YOU'RE TRYING TO USE MACHINE LEARNING TO GENERATE HYPOTHESES, SO WE TALK ABOUT TRADITIONAL HYPOTHESIS DRIVEN RESEARCH BUT ALSO USING MACHINE LEARNING AND DATA SCIENCE TO IDENTIFY NEW HYPOTHESES THAT YOU TEST, PERHAPS YOU DON'T NEED TO UNDERSTAND WHY THE CLUSTERING IS WORKING THE WAY IT IS OR WHY RESULTS ARE WHAT THEY R. BUT IF YOU USE THEM TO INFORM DECISION MAKING, PARTICULARLY CLINICAL DECISION MAKING YOU WANT TO HAVE A BETTER SENSE OF TRACEABILITY OF THAT DATA AND OF THE RESULTS. OF COURSE WHEN TALKING TYING RESEARCH TO CLINICAL CARE, WE HAVE TO HAVE A PROCESS FOR VERIFICATION AND VALIDATION OF THE RESULTS. THIS IS AN AREA THAT WE ARE HEAVILY FOCUSED ON RIGHT NOW. I WILL SHOW YOU A COUPLE OF EXAMPLES WHAT WE HAVE DONE TO DATE. FIRST NATURAL LANGUAGE PROCESSING. I WANT TO SPEND A MOMENT ON THIS ONE T. WHEN WE GOT STARTED, WE HAD A VERY SIMPLE USE CASE THAT WAS PRESENTED TO US. WE WERE WORKING WITH PROSTATE CANCER AND THEY WANTED TO BE ABLE TO EXTRACT PSA SCORES FROM ELECTRONIC MEDICAL RECORD. UNSTRUCTURED TEXT PULL OUT PSA SCORES AND YOU SAID YEAH, NO PROBLEM. APL WHERE ENGINEERS AND DATA SCIENTISTS WORKING CLOSELY WITH THE CLINICS AT HOPKINS. SO YEAH NO PROBLEM WE CAN DO THAT SO WE CREATE IN A FEW WEEKS A TOOL, THAT EXTRACTED PS A AND GLEASON SCORES. WHEN LOOK AT ACCURACY OF THAT, IT WAS THE LOW 9 0s. 90%. WE ARE NOT SURE THAT IS GOOD ENOUGH, WE THINK IT NEEDS TO BE IN THE HIGH 90s. WE DID A COMPARISON OF THE HUMAN PERFORMANCE CLINICIAN TO CLINICIAN. WE SAID MAYBE THAT'S GOOD ENOUGH. TYING BACK TO SOMETHING WE ARE TALKING ABOUT EARLIER, WE HAVE A MUCH HIGHER STANDARDTOR MACHINES THAN WE DO FOR HUMANS. AND I THINK IT'S SOMETHING TO KEEP IN MIND AS WE CONTINUE HAVING THIS DISCUSSION. ANYWAY, WE CREATED THAT TOOL AND THEN WE RECOGNIZE THAT THERE WAS SOME CLAMORING FOR SIMILAR TOOLS FOR OTHER DISEASES, OTHER PARAMETERS. SO WE CREATED A SERIES OF TOOLS REPRESENTED HERE. EXPLORER IS ONE THAT IS JUST A DASHBOARD, IT GIVES CLINICIANS A SENSE OF COHORT, WE CAN LOOK AT DISTRIBUTION OF AGE, GENDER, RACE, ALL THOSE KIND OF THINGS. BEFORE THIS, THAT WAS NOT SOMETHING THAT COULD EASILY BE GLEANED FROM THIS MEDICAL RECORD. THAT'S ONE OF THE THINGS WE THINK IS HELPFUL WHEN IT COMES TO BIAS AN TRANSPARENCY FIRST UNDERSTAND WHAT YOUR COHORT LOOKS LIKE AND MAKE A MORE INFORMED DECISION HOW REPRESENTATIVE IT IS. IS THE SECOND ONE ISN'T JUST A SEARCH TOOL TAKING THE UNSTRUCTURED TEXT PUTTING IT INTO A FORMAT WHERE YOU CAN USE RULE BASED SEARCHES AND HAVE A MUCH EASIER WAY TO INTERACT WITH THE DATA. AND THEN FINALLY THE ONE WE CALL PINE IS ACTUALLY AGAIN CAME UP EARLIER, CLINICIANS ARE NOT NECESSARILY DATA SCIENTISTS NOR DO THEY WANT TO BE. THEY WANTED TO USE THESE TOOLS TO EXTRACT RELEVANT DATA MUCH MORE UBIQUITOUSLY IN VARIOUS DIFFERENT AREAS. SO WE CREATED A GENERALIZEABLE FORM OF THIS TOOL. SO WHAT YOU SEE REPRESENTED HERE IS BASICALLY YOU DON'T HAVE TO BE AN EXPERT, CLINICIAN, ANYONE CAN GO IN AND HIGHLIGHT DIFFERENT TERMS OF INTEREST. SO YOU GO IN AND HIGHLIGHT DIFFERENT RECORDS AND THE TOOL GENERATES ITS OWN ALGORITHM FOR YOU. BASED ON THE NUMBER OF TRAINING SETS YOU USE YOU GET A SENSE HOW ACCURATE IT IS AND TRAIN WITH MORE DATA IF YOU WANT TO INCREASE THE ACCURACY. THAT IS AN EXAMPLE OF SOME OF THE WAYS THAT WE HAVE STARTED INCORPORATING MACHINE LEARNING. AS I SAID, JUST HAVING A SENSE OF THE DISTRIBUTION OF YOUR DATA IS ONE IMPORTANT FACET. I WILL HAVE TO SPEED UP HERE. THIS IS AN AREA OF RESEARCH WE ARE WORKING WITH RIGHT NOW, MARY KATHERINE BEECH AT GERMAN INSTITUTE OF BIOETHICS IDENTIFYING SOURCES OF BIAS. FIRST, THE FIRST STEP IS IDENTIFYING THAT YOU HAVE A PROBLEM, RIGHT? SO AS WE LOOK AT THE ELECTRONIC MEDICAL RECORD THERE'S A HYPOTHESIS SOME CASES YOU MIGHT BE INTRODUCING HUMAN BYE-BYEIAS TO THE ELECTRONIC MEDICAL RECORD. IF YOU LOOK AT EXISTING SENTIMENT BASED TOOLS, THEY DON'T WORK SO WELL WHEN TALKING ABOUT KIND OF MEDICAL DIAGNOSE KNOW SEEDS YOU HAVE IF YOU USE EXISTING TOOLS, THE MOST BIAS SENTENCES THAT COME OUT ARE THINGS LIKE NO RASH, NO SWELLING, NO PAIN. THOSE KINDS OF THINGS. SO THIS IS FOCUSED ON DEVELOPING LINGUISTIC MARKERS OF BIAS TO QUANTIFY THE LEVEL OF BIAS AND ELECTRONIC MEDICAL RECORDS AND OF COURSE LOOK AT FURTHER DOWN THE ROAD IS THAT IMPACTING THE TYPE OF CARE. QUANTITATIVE IMAGING, I'LL GO THROUGH QUICKLY BUT THE POINT IS WE ARE TAKING IMAGING DATA AND TURNING IT TO THINGS LIKE PICTURE ON THE RIGHT HERE WHERE YOU HAVE SEGMENTATION OF THE LESIONS OF INTEREST FOR EXAMPLE IN AN MRI SCAN, THAT WAY LONGITUDINALLY YOU CAN TRACK PATIENTS OVER TIME IN MUCH MORE QUANTITATIVE WAY. THAT'S ALL GREAT. NO REAL ISSUES THERE IN TERMS OF BIAS OR THINGS LIKE THAT IF YOU ARE TALKING INDIVIUAL PATIENT OVER TIME BUT IF YOU WANT TO START LOOKING ACROSS POPULATIONS AND DRAW SOME CONCLUSIONS ABOUT WHAT YOU MIGHT EXPECT TO SEE HERE, YOU NEED MORE DATA. SO HERE IS ANOTHER AREA OF ACTIVE RESEARCH WHERE WE ARE TRYING TO USE GENERATIVE MODELS TO SYNTHESIZE DATA. SO THE EXAMPLE SHOWN HERE IS RETINAL IMAGING ON THE LEFT IS SYNTHETIC DATA OF RETINAL IMAGES WITH AGE RELATED MACULAR DEGENERATION, ON THE RIGHT ARE HEALTHY RETINAS. YOU CAN IMAGINE THE SCALE AT WHICH YOU CAN CREATE SYNTHETIC DATA REPRESENTATIVE OF THE POPULATIONS THAT YOU ARE TRYING TO EXAMINE. IN SUMMARY, HOPE ANY YOU GOT A SENSE OF THE WAY WE ARE APPLYING SOME SPECIFIC MACHINE LEARNING APPROACHES TO RESEARCH AND CLINICAL CARE AT HOPKINS. AS I MENTIONED FOR SOME OF THE WAYS BIAS CAN BE INTRODUCED BY MACHINE LEARNING ALGORITHMS HUMAN SIDE WE LOOK FIRST TO IDENTIFY BIAS AND TRACK AND SEE HOW OR IF IT TRANSLATES TO CARE. ON THE ALGORITHM SIDE I DIDN'T TALK ABOUT IT BUT OUTGOING AREA OF RESEARCH TRYING TO QUANTIFY HOW WELL ALGORITHMS ARE. PROKING SO MAKING IT AUTOMATIC WHERE THE ALGORITHM KNOWS HOW ACCURATE IT IS BASED ON REPRESENTATIVE DATA SETS BEING USED. I'LL LEAVE IT AT THAT. >> THANK YOU VERY MUCH, SUZANNE. NEXT UP WE HAVE MARZYEH GHASSEMI, UNIVERSITY OF TORONTO, SHE LIVED ON THE ROAD PERPENDICULAR TO MY ROAD IN OXFORD. THE WORLD GETS SMALLER AND SMALLER AND SMALLER. >> I COULDN'T THE REMEMBER THE NAME OF THE ROAD WHICH IS EMBARRASSING, GRADUALLY RMEMBERED IT. WHICH MADE ME FEEL MUCH BETTER. I JUST REMEMBERED IT ADS WE WERE TALKING, BUT SOMETIMES YOU READ THESE PAPERS WITH BIOMARK UPS AND THEY MAY SAY PEOPLE WHO HAVE THIS CONDITION CAN'T REMEMBER THIS WORD AT THIS FREQUENCY AND CAN'T REMEMBER THE ROAD THEY GREW UP ON, IT WAS LIKE -- BUT WE'RE FINE, ALL FINE. I'M MARZYEH GHASSEMI, UNIVERSITY OF TORONTO AT VECTOR INSTITUTE. I DID A Ph.D. IN COMPUTER SCIENCE, EMPHASIS ON MACHINE LEARNING FOR HEALTH AT MIT. AND I SORT OF CAME INTO HEALTHCARE THERE SEEM TO BE IMPORTANT INTERESTING PROBLEMS IN IT. ONE OF THE THINGS THAT I HAVE BEEN A LITTLE BIT WORRIED BY IS THE RHETORIC THAT HAPPENS WHEN I START TALKING TO A CLINICAL COLLABORATOR. SO I WILL TELL THEM I DO MACHINE LEARNING FOR HEALTH, THIS IS NAME OF MY LAB, CREATING ACTIONABLE INSIGHTS IN HUMAN HEALTH. SO WE FOCUS ON THREE AREAS, WHAT MODELS ARE HEALTHY, SO CAN YOU MAKE MODELS THAT LEARN THE RIGHT THING AND THERE'S SOME INTERESTING MACHINE LEARNING TO BE DONE THERE. WHAT HEALTHCARE IS HEALTHY BECAUSE I HAVE NEVER HEARD THAT PHRASE, HALF OF THE THINGS WE DO ARE WRONG BUT WE DON'T KNOW WHICH HALF, THAT'S DEPRESSING. DON'T TELL OTHER PEOPLE THAT. I DIDN'T WANT TO HEAR IT, BUT NOW IT'S IN THERE. WHAT BEHAVIORS ARE HEALTHY, SEVERAL PEOPLE HAVE SAID OVER THE COURSE OF THE DAY, THERE ARE A LOT OF DIFFERENT KINDS OF DATA THAT ARE NOT JUST EHR OR HIGH FREQUENCY DATA THAT ARE ABOUT HOW YOU ARE BEHAVING, THAT'S VERY IMPORTANT FOR MANY, MANY CONDITIONS. WHAT WILL HAPPEN IS SOMEBODY WILL SAY FIELD IS STRUGGLING WITH THIS PROBLEM FOR YEARS. SOMEBODY PROBABLY DEEP LEARNING SCIENTIST WILL SAY STRUGGLE NO MORE. HERE ARE THE ALGORITHMS. YOU FIGURE THIS IS LIKE A HARD PROBLEM, NOT GOING TO FIX. SO THIS IS THE REASON I THINK WE NEED TRANSPARENCY IN MODELS. THESE HARD PROBLEMS AND MAKING A REALLY COOL COMPLEX MACHINE LEARNING ALGORITHM THAT I'M EXCITED TO CODE AND PUBLISH, WON'T FIX THE LARGER PROBLEM. SO HERE IS MY FAVORITE EXAMPLE, MY FAVORITE EXAMPLE BECAUSE I DID A PROJECT, IT WAS THE FIRST PAPER OUT OF MY Ph.D. WHERE WE TOOK A BUNCH OF CLINICAL NOTES AND WE SHOWED IF YOU USE THESE TOPIC MODELS TO TRY TO RESTRATIFY RISK PREDICTION OF MORTALITY EVERY 12 HOURS IN ICU YOU DO BETTER IF YOU USE CLINICAL ACUITY SCORES IN THE DATA, SO AGE, GENDER AND SO FORTH, ALL THIS THINGS YOU MIGHT USE IN A STANDARD MODEL. ONE DOCTOR I WORKED WITH SAID DID YOU CHECK HOW FAIR THAT MODEL IS? WHAT DO YOU MEAN CHECK? I PUT ALL THE DATA THERE. I OPTIMIZE FOR THE BEST OUTCOMES. JUST CHECK SO THIS CAME OUT THIS YEAR ACTUALLY AND IT'S SHOWING WHEN YOUN'TMIZE THE BEST PERFORMANCE ACROSS EVERYBODY, YOU DON'T DO BEST FOR SOME PEOPLE. OR IF YOU ARE WATCHING HANDMADE TAIL, FAIR OR WHAT IS IT, BETTER DOESN'T MEAN BETTER FOR EVERYONE. SO THIS IS A CONCERN. IF WE OPTIMIZE A GLOBAL OBJECTIVE FUNCTION WITH A HIGH CAPACITY MODEL OR LOW CAPACITY MODEL LOGISTIC REGRESSION, IT DOESN'T MATTER. WE DEPLOY THAT MODEL AND IT HAS UNFAIR ACCURACY IN PEOPLE WITH PUBLIC INSURANCE IN WOMEN. IN PEOPLE ARE DIFFERENT ETHNICITIES. THAT'S PROBLEMATIC. WE PROBABLY DON'T WANT THAT. BUT THE ISSUE HERE IS THAT ETHICS ISN'T NEW, IT'S NOT LIKE WE ARE SUDDENLY OH ETHICS, IT'S A THING. THE REAL ISSUE IS ETHICAL MACHINE LEARNING IS NEW. WE HAVE KNOWN THAT THERE ARE ETHICAL ISSUES IN MEDICINE FOR A REALLY LONG TIME. AND CLINICAL TRIAL POPULATIONS THERE ARE RETRACTED STUDIES, CONFLICTS OF INTEREST, THESE ARE KNOWN THINGS THAT HAVE BEEN AROUND A WHILE. THE MACHINE LEARNING PART OF THE IT THAT WERE UNCOMFORTABLE WITH, IT'S THIS INTERPLAY, PART OF THE REASON WE ARE UP COMFORTABLE WITH THE INTERPLAY IS HUMANS HAVE ETHICS TRAINING. DOCTORS HAVE GOOD INTENTIONS AND THEY HAVE ETHICS TRAINING. WE KNOW THERE HAVE BEEN MANY, MANY VERY WELL DONE POPULATION LEVEL HEALTH STUDIES THAT HAVE SHOWN THE BIASES OF SOCIETY GET REFLECTED IN THE CARE THAT IS PROVIDED TO PATIENTS. HUMANS ARE BIASED, DOCTORS ARE HUMANSES GUESS WHAT, DOCTORS ARE BIASED CARE, THE DATA WE LEARN FROM AND THE CARE THAT IS PROVIDED OFTEN REFLECTS THOSE BIASES. DO YOU WANT A MODEL TO REFLECT BIASES AGAINST WOMEN, AGAINST OBESE PATIENTS, AGAINST MINORITIES? PROBABLY NOT. SO WHAT ABOUT HUMAN TRANSPARENCY? IF WE ARE TALKING TRANSPARENCY, LET'S TALK ABOUT TRANSPARENCY IN THE WAY YOU MIGHT WANT IT. AS SEVERAL PEOPLE SAID EARLIER TODAY, THIS IS I THINK SOMETHING WE NEED TO REMEMBER WHEN TALKING ABOUT WHAT WE WANT OUT OF MACHINE LEARNING ALGORITHM, WHAT DO WE WANT OUT HUMAN? FORGET MACHINE LEARNING. WHAT DO WE WANT OUT OF HUMAN? WHEN I THINK ABOUT CARE, I THINK HUMAN DECISIONS, DOCTORS ARE EXPERT ADVICE GIVERS SELF-REGULATING PROFESSION. THERE ARE GIVING YOU THIS PROFESSIONAL ADVICE AND YOU CHOOSE WHETHER TO ACCEPT IT AND HOPEFULLY MOST OF THE TIME YOU ARE ACCEPTING IT. SO THESE DECISIONS ARE JUSTIFIED BY RESEARCH OR IN RESEARCH OR IN EXPERIENCE. THEY ARE ALL JUSTIFIED, I ASK WHY ARE YOU GIVING ME THIS SPECIFIC ANTIBIOTIC FOR MY SON? THE JUSTIFICATION IS BECAUSE HE HAS THIS COMPLICATION OR BECAUSE THIS IS BEST FOR HIM OR BECAUSE I CAN'T GIVE IT TO HIM ORALLY. THE ISSUE IS, AS WE HAVE ALL HEARD MANY TIMES TODAY, ROUTINE CLINICAL PRACTICE IS REVERSED SO THIS CAME OUT COUPLE OF WEEKS AGO. IF YOU TAKE THE LAST FIVE OR SIX YEARS OF STUDIES IN JAMA, LANSETT AND NEW ENGLAND JOURNAL, A LOT OF PAPERS, MORE THAN 10% ARE MEDICAL REVERSALS. WE THOUGHT THERE WAS A STANDARD ROUTINE PRACTICE THING THAT WE DO BUT IT'S NOT WHAT WE SHOULD BE DOING. SO HERE IS WHAT I'M GOING TO SAY. I THINK AT A VERY HIGH LEVEL. I THINK TRANSPARENCY IS GOING TO BE REALLY IMPORTANT WHEN THE RECOMMENDATION IS SOMETHING DIFFERENT THAN WHAT A DOCTOR WANTS TO DO IN A ROUTINE WAY. WHEN THERE'S SOME DISAGREEMENT, SOME SORT OF DIFFERENCE BETWEEN A MODEL AND EVALUATOR. HUMAN EVALUATOR. I DON'T THINK IT MEANS MOST CLINICIANS IN MY EXPERIENCE REALLY CARE ABOUT HOW TO MODEL WORKS BECAUSE HONESTLY, LOG ODDS ARE NOT INTERPRETABLE. I HAVE NEVER MET A PERSON, THE COX PROPORTIONAL HAZARD CO-EFFICIENTS ARE VERY EXPLAIN -- NO. WE DON'T THINK THIS WAY. NEITHER ARE NEURAL NETWORK WAVES SO WHEN WE THINK LOGISTIC REGRESSION AND -- THEY'RE NOT INTERPRETABLE EITHER, RIGHT? WHAT MATTERS HERE IS THAT INSTEAD OF BUILDING A SIMPLE MODEL FOR THE SAKE OF TRANSPARENCY AND SAYING A HA, IT IS TRANSPARENT BECAUSE THE MODEL IS MORE SIMPLE, WE SAY IT COULD MEAN WE REQUIRE THERE'S AN UNDERSTANDING OF WHEN WE CAN RELY ON OR REJECT MODEL OUTPUTS BECAUSE THEY ARE IN OR OUT OF BOUNDS, IT COULD MEAN NOTIFICATIONS WHAT POPULATIONS THE MODEL COULD WORK ON OR FAIL ON. IT COULD MEAN WE NEED LIMITATION OF DATA THAT THE MODEL IS TRAINED ON, ONLY TRAINED ON THIS POPULATION. WITH THIS RETROSPECTIVE POLICY THAT WAS APPLIED. SO SOME OF THE TECHNICAL OPTIONS THAT HAVE BEEN DEPLOYED I WILL COVER BRIEFLY. WHICH ARE NUMBER ONE, TRANSPARENCY VIA POST HOCK EXPLANATION. YOU CAN DO THIS FOR ANYTHING. I BUILD A MODEL, SOME KIND OF MODEL, AND I'M GOING TO APPLY A POST DOC INTERPRETATION LIKE LIME OR ANY METHOD WE TALKED ABOUT TODAY BUT POST HOC EXPLANATIONS ARE ONLY USEFUL IF CONSISTENT. AND WHAT I MEAN TECHNICALLY BY CONSISTENCY HERE IS DIFFERENT MODEL BEHAVIORS GIVE YOU DIFFERENT EXPLANATIONS AND THE SAME MOEL BEHAVIOR GIVES YOU THE SAME EXPLANATION. THE ISSUE IS IT'S BEEN ESTABLISHED THE TENSION BASED MECHANISMS AND SALIENCY ARE NOT ALWAYS CONSISTENT. SO THESE ARE METHODS THAT WE'RE GOING TO START USING AND WE ALL USE AS MACHINE LEARNING COMMUNITY. WE NEED TO MAKE SURE WE ARE COMFORTABLE WITH THE GUARANTEES THAT THEY GIVE US OR DON'T GIVE US. IN TERMS OF CONSISTENCY. THERE ARE POSSIBLE FIXES TO SALIENCY AND TO ATTENTION BASED EXPLANATIONS BUT THEY REQUIRE YOU DO MORE COMPETITIVE THINGS. THAT YOU HAVE SOME REGULARIZATION WHERE YOU FORCE ATTENTION MAPS OR SALIENCE MAPS TO COMPETE. ONE EXAMPLE IS WHEN YOU TRY TO PREDICT INTERVENTION IN AN INTENSIVE CARE UNIT USING ALL THE DATA ABOUT A PATIENT, THESE ARE NOTES AND VITALS AND LABS, EVERYTHING ABOUT THEM, NOT JUST PREDICTING ONE INTERVENTION BUT A SET OF INTERVENTIONS AND YOU HAVE TO UPDATE YOUR PREDICTIONS EVERY HOUR OR EVERY FEW HOURS. HERE WE FOUND YOU HAVE TO FORCE THE FEATURE LEVEL ATTENTION MODELS TO COMPETE. BY MAKING THEM CHOOSE WHICH FEATURES ARE GOING TO DAMAGE THE MODEL THE WORST BEFORE THEY ACTUALLY LEARN SOMETHING THAT IS MEANINGFUL. OR YOU CAN SAY I'M GOING TO ENFORCE THAT YOU HAVE TO SHOW ME SOMETHING CONSISTENT WITH RESPECT TO SIDE INFORMATION. THIS IS ANOTHER PAPER WE LOOKED AT CELLS THAT HAVE BEEN GIVEN DIFFERENT AMOUNTS OF A DRUG AND WE SAID WE REQUIRE THE AMOUNT OF THE DRUG THAT WE GIVE YOU AS YOU MODEL THE PATHOLOGY OF THE CELL BE HELD SEPARATE IN THE LATENT SPACE, IT'S IMPORTANT YOU CAN'T TAKE .2 AT TIMES, .3 OF SOMETHING TO .3 SOMETHING ELSE IT HAS TO BE SEPARATE BECAUSE WE KNOW THIS IS IMPORTANT AND YOU HAVE TO SLIDE ALONG THE DIMENSION AND SEE WHAT HAPPENS TO ONE CELL AS IT GETS MORE AND LESS DRUG SO ENFORCING THOSE KIND OF CONSTRAINTS WITHIN THE MODEL. ANOTHER OPTION IS YOU CAN SAY WHY AM I MESSING WITH ALL THESE INTERMEDIATE OUTPUTS? HUMANS USE THESE RECOMMENDATIONS, SO REALLY I SHOULD JUST BE GIVING HUMANS SOMETHING THAT THEY COULD USE AT THE END. SO YOU TALK TO A CLINICIAN. AND BAY SAY I DON'T WANT YOU TO GIVE ME FEATURES OR SALIENCE MAPS AND RADIOLOGY IMAGE, I WANT YOU TO GENERATE MY REPORT FOR ME AND THEN EDIT THE REPORT, IF I THINK THERE'S SOMETHING I DON'T LIKE. HERE WHAT WE DID IS AUTOMATICALLY GENERATE RADIOLOGY REPORTS GIVEN AN IMAGE AND WE HAD TO CREATE TWO DIFFERENT KINDS OF CONSTRAINTS USING REINFORCEMENT LEARNING, AND CLINICAL ACCURACY CLASSIFICATION OBJECTIVE TO MAKE SURE IT'S HUMAN READABLE AND CLINICALLY ACCURATE. THOSE TWO THINGS COMPETE. THOSE DON'T ALIGN. FINALLY TRANSPARENCY REQUIRES THAT YOU CAN AUDIT THIS EMBODY DATA. THESE ARE DATA ABOUT BODIES. HUMAN BODIESES THAT GO THROUGH LIFE, GO TO SCHOOLS, THEY HAVE, EXPERIENCES, AND IF WE DON'T HAVE TRANSPARENT ALGORITHMS THAT OPERATE ON GOOD TRANSPARENT DATA, DATA THAT PEOPLE CAN VERIFY WE RUN THE RISK OF NOT BEING ABLE TO VERIFY END RESULTS BECAUSE WE ONLY HAVE A PIECE OF A PROCESS. TOOLS OTHERS TALKED ABOUT TODAY DATA SHEETS FOR DATA SETS OR MODEL COLOR FOR MODEL REPORTING CAN BE USED AT THE MODEL SIDE. THERE'S ALSO THE BIG PICTURE TOOLS THAT HAVE BEEN PUT OUT BY THE GOOGLE TEAM FOR FAIRNESS TO UNDERSTAND POTENTIAL BIASES INHERENT IN DATA REGARDLESS OF THE MODEL. AND I WANT TO EMPHASIZE WHAT OTHERS HAVE SAID HERE TODAY, WE SHOULD BE WORKING TOWARDS TRANSPARENT PROCESSES NOT MODELS. BECAUSE ONE OF THE THINGS I WAS STRAIGHT OUT OF MY Ph.D. IS YOU MAKE THESE FANCY MODELS AND OUR GOAL IS WE'RE GOING TO SHOW MODEL RECOMMENDATIONS TO CLINICIANS OR SHOW IN THE GROSS EHR FORMAT, AND SHOW THEM IN A PRETTY GOOGLE DESIGNED INTERFACE. SHOWING THIS TO CLINICIANS IN A PRETTY GOOGLE DESIGN INTERFACE MADE THEM FEEL BETTER, THEY SELF-REPORTED, I LIKE IT BETTER. IT'S MUCH BETTER. I LIKE THIS INTERFACE. THEY DID NOT ANY BETTER AT DOING TASKS WE ASK THEM TO DO THAT WERE THE TASK THEY SAID THEY WANTED TO BE BETTER AT. UNDERSTANDING THE ENTIRE PROCESS RATHER THAN JUST THE MODEL IS GOING TO BE CRITICAL MOVING FORWARD. THIS IS MY TEAM UNIVERSITY OF TORONTO AND VECTOR INSTITUTE WHO HAVE YOU INSTRUMENTAL IN ALL THIS WORK. THANK YOU. [APPLAUSE] >> THANK YOU VERY MUCH. ULTIMATELY WE HAVE MATTHEW -- IF YOU ARE CURIOUS WHAT HIS CHINESE NAME IS, HE'S DIRECTOR AT CENTER OF BIOETHICS NEW YORK UNIVERSITY, MATTHEW LIAO AND I HAVE GONE ROGUE AND NOT DONE SLIDES. IT'S MUTUAL BLACK BACKGROUND THAT WE CAN HAVE. SOMETHING JUST -- ROUND OF APPLAUSETOR MATTHEW LIAO. >> THANK YOU. THANKS EVERYONE FOR BEING HERE. THANK YOU FOR INVITING ME, I HAVE LEARNED SO MUCH ALL THE SPEAKERS BEFORE. SO I'M GOING TO LAUNCH RIGHT IN. SO I'M A PHILOSOPHER, VERY DIFFERENT BACKGROUND FROM MOST PEOPLE IN THIS ROOM. ALSO AN ETHICIST. SO I'M GOING TO LOOK AT ALL THESE DIFFERENT ISSUES FROM AN ETHICAL PERSPECTIVE. JUST BY WAY OF BACKGROUND, I'M EDITING A VOLUME RIGHT NOW CALLED THE ETHICS OF ARTIFICIAL INTELLIGENCE THAT WILL COME OUT WITH OXFORD UNIVERSITY PRESS IN JANUARY OR FEBRUARY. AND I ALSO ORGANIZED A CONFERENCE ON THE AI AND HEALTHCARE ABOUT TWO MONTHS AGO AT NYU. SO WHEN I LOOK AT ETHICAL ISSUES IN AI, I FIND IT HELPFUL TO MAKE TWO TYPES OF ETHICAL ISSUES OF RIDING OUT OF AI. ONE IS WHAT I CALL BASICALLY VULNERABILITIES IN AI. OR MACHINE LEARNING GENERALLY. SO THIS IS BASICALLY WHEN AIs ARE NOT WORKING VERY WELL. SO THOSE CAN GIVE RISE TO A BUNCH OF ETHICAL PROBLEMS. THERE IS A DIFFERENT SET OF ETHICAL ISSUES, SETS OF ETHICAL ISSUES WHEN AIs AREN'T WORKING WELL, I CALL THOSE HUMAN VULNERABILITIES. SO JUST I'M NOT GOING TO TALK ABOUT THAT TODAY BUT GIVE YOU AN EXAMPLE OF WHEN AIs ARE WORKING TOO WELL. TAKE FACIAL RECOGNITION. WHEN THEY WORKED WELL GOVERNMENTS CAN USE THAT TO SURVEIL ITS CITIZENS, THAT IS A BIG PROBLEM. SO THOSE RAISE ETHICAL ISSUES. SO -- WITH RESPECT TO THE FIRST TYPE, VULNERABILITIES IN MACHINE LEARNING WE COVERED A BUNCH OF THESE ISSUES, I'M GOING TO GO OVER THEM AND THEN -- BUT I BASICALLY AGREE WE NEED TO BE SORT OF FOCUS ON TRANSPARENCY IN TERMS OF THE PROCESS. BUT I ALSO THINK THAT WE -- I'M ALSO GOING TO TALK WHETHER WE NEED TO BE FOCUSED ON TRANSPARENCY MODELS AS WELL. BUT JUST A COUPLE OF THINGS ARISING OUT OF LIMITATION OF MACHINE LEARNING. AS WE KNOW, MACHINE LEARNING IS VERY DATA HUNGRY, IT NEEDS A LOT OF DATA TO WORK WELL SO SUPERVISE LEARNING ALGORITHMS ARE ABLE TO FINE TUNE THEMSELVES, AND ACHIEVE PREDICTIVE POWER FROM HAVING A VAST AMOUNT OF DATA. GIVEN THAT, WE ARE INCENTIVIZED TO COLLECT INFORMATION AND THAT COULD RAISE ALL SORTS OF PRIVACY ISSUES. SO I LIKE DINA'S LITTLE DEVICE BUT YOU CAN IMAGINE IF LIKE SURREPTITIOUS GOVERNMENT GETS HOLD OF THAT DEVICE SOMETHING COULD -- WE NEED TO THINK ABOUT THAT AS WELL THOUGH OF COURSE SHE'S NOT USING IT IN THOSE WAYS. BUT OTHER PEOPLE COULD. WE ALSO HEARD ABOUT THE GARBAGE IN GARBAGE OUT PROBLEM SO MACHINE LEARNING IS NOT JUST DATA BUT YOU NEED GOOD DATA WHEN YOU HAVE BAD DATA OR INCOMPLETE DATA THAT'S GOING TO GIVE YOU BAD RESULTS. THIRDLY WE TALKED FAULTY AL GOSH RHYTHMS SO YOU CAN HAVE GOOD DATA BUT IF ALGORITHMS ARE BIASED OR NOT VERY GOOD, THEY COULD OVERFIT OR UNDERFIT AND SO FORTH. THEN FOURTHLY, A FOURTH LIMITATION IN AI IS WHEN ALL MACHINE LEARNING SYSTEMS ARE WEAK AI IN THAT THEY ARE PROGRAMMED TO EXCEL AT SPECIFIC TESTS. THEY ARE NOT GENERALIZABLE. NOT STRONG AI SO THEY CAN'T THINK FOR THEMSELVES. IMPORTANTLY THEY LACK A MORAL SENSE. FROM SO THAT'S THE CAPACITY TO ASSESS WHETHER SITUATION IS MORALLY RIGHT OR WRONG. MACHINE LEARNING WE ALREADY -- IT'S ALREADY DEPLOYED IN SITUATIONS WHERE THEY HAVE TO MAKE MORAL DECISIONS, SO TAKE SELF-DRIVING CARS, BUT ALSO AUTONOMOUS WEAPONS,, NOT YET HEALTHCARE BUT MAYBE FUTURE, SO SOMEONE COULD AUTOMATIC SURGERY OR ET CETERA, ET CETERA. THERE'S A LITERATURE ON MACHINE ETHICS HOW TO BUILD ETHICAL DECISIONS INTO AI WHICH I WON'T TALK ABOUT TODAY. HAPPY TO DISCUSS THAT IN SORT OF Q&A. IF YOU LIKE. THE FIFTH IS DEEP LEARNING SPECIFIC FORM OF MACHINE LEARNING AS A BLACK BOX. THAT'S WHERE NOT JUST ABOUT TRANSPARENCY OF THE PROCESS BUT PEOPLE ARE INTERESTED IN TRANSPARENCY IN THE MODEL ITSELF. SO WE HAVE HEARD IT RAISES ISSUES OF INTERPRETABILITY, EXPLAINABILITY, TRUST, AND DEEP LEARNING IS A BLACK BOX BECAUSE IT TYPICALY EMPLOYS THOUSANDS AND POSSIBLY MILLIONS OF CONNECTIONS THAT INTERACT WITH ONE ANOTHER IN A VERY COMPLEX WAY. IT CAN BE DIFFICULT TO INTERPRET HOW THESE CONNECTIONS ARE INTERACTING WITH ONE ANOTHER. AND WHY THEY MAKE CERTAIN PREDICTIONS. ISSUE OF EXPLAINABILITY ARISES BECAUSE WE OFTEN NEED TO KNOW HOW THESE DECISIONS ARE RIGHT AND SOMEONE MENTIONED WE MAY NEED TO JUSTIFY A DECISION, SAY IN CRIMINAL SENTENCING IN CASES. WHAT DEEP LEARNING GIVES US PREDICTION WITHOUT EXPLAINING IN HUMAN TERMS OR OTHERWISE IT ARRIVES AT A PREDICTION. SO THOSE ISSUES ARE PREKNOWN AND IT'S BEEN TALKED ABOUT AND HERE I WANT TO FOCUS ON TWO WAYS BY WHICH PEOPLE ATTEMPTED TO ADDRESS THIS BLACK BOX PROBLEM WHICH I FOUND VERY INTERESTING. FIRST OF ALL, A NUMBER OF AI RESEARCHERS HAVE ATTEMPTED TO DO THROUGH TECHNICAL MEANS SO THEY SOUGHT TO CREATE INTERPRETABLE MACHINE LEARNING BY -- WHAT THAT DOES IS THEY WANT TO ADD AN ADDITIONAL LAYER TO DEEP LEARNING MODELS. BY PLACING ANOTHER LAYER AFTER THE HIDDEN LAYER, LAYERS OF NEURONS AND BEFORE THE OUTPUTS, THIS ADDED LAYER CAN INTERPRET WHAT THE BLACK BOX IS DOING. SO THE AIM HERE IS FOR THIS LAYER, THIS ADDED LAYER TO TELL US THINGS SUCH AS WHICH FEATURE WAS -- WERE THE MOST IMPORTANT FOR ARRIVING AT A PARTICULAR PREDICTION. WHICH FEATURES COULD HAVE HAD AN EVEN GREATER IMPACT ON THE PREDICTION. HOW EACH FEATURE IN THE DATA BEARS ON A PARTICULAR PREDICTION AND HOW EACH FEATURE WOULD EFFECT DIFFERENT POSSIBLE PREDICTIONS. SO I THINK THIS INTERPRETABLE MACHINE LEARNING IS AN INTERESTING NOVEL IDEA. BUT I HAVE TO -- I WANT TO ASK WHETHER IT ADDRESSES THE PROBLEM OF INTERPRETABILITY EXPLAINABILITY AND TRUST. A LOT OF PEOPLE ARE REALLY EXCITED ABOUT THIS AND I'M POURING COLD WATER ON IT. HERE ARE SOME REASONS TO BE CONCERNED. ONE SINCE ADDITIONAL LAYER IS PLACED AFTER THE BLACK BOX SEEMS TO BE OFFERING A POST HOC EXPLANATION WHAT'S GOING ON IN THE BLACK BOX. THAT IS, IT'S -- IT AIMS TO EXPLAIN WHAT THE BLACK BOX HAS DONE AFTER IT HAS MADE ITS PREDICTIONS. GIVEN THIS YOU MIGHT WONDER WHETHER POST HOC EXPLANATION PROVIDE WITH REASONS WHY A BLACK BOX GAVE PREDICTIONS IT DID. SO YOU CAN PUT THIS CONCERN IN A FORM OF A DILEMMA. EITHER THE PREDICTIONS ARE BASED ON THESE POST HOC EXPLANATIONS OR THEY ARE NOT. SO LET'S SUPPOSE THEY ARE NOT BASED ON THESE POST HOC EXPLANATIONS, RIGHT? THEN WHAT IS THE VALUE OF THESE EXPLANATIONS. THESE EXPLANATIONS WOULD BE A POST HOC RATIONALIZATION, THAT DOESN'T COURSE RESPOND TO HOW THE BLACK BOX HAS ARRIVED AT ITS PREDICTIONS. SUPPOSE INSTEAD THAT THE PREDICTIONS ARE BASED ON THE HOST HOC EXMR. -- POST HOC EXPLANATION. YOU SHOULD BE ABLE TO DESIGN A NEW MODEL USING JUST THESE POST HOC EXPLANATIONS. GET RID OF THE BLACK BOX. SO THIS SUGGESTS A WAY TO TEST THESE INTERPRETABLE MACHINE LEARNING SYSTEMS. SO IF THE BLACK BOX REMAINS INDECEMBER PABLE FOR PURPOSE OF MAKING PREDICTIONS THIS WOULD SEEM TO SUGGEST THAT THE POST HOC EXPLANATIONS HAVE NOT GIVEN US THE COMPLETE EXPLANATION WHY BLACK BOX GAVE PREDICTIONS IT DID. SO THAT IS SORT OF ONE ISSUE I HAVE WITH THE INTERPRETABLE MACHINE LEARNING APPROACH. HERE IS A SECOND WAY BY WHICH SOME PEOPLE HAVE ATTEMPTED TO DIRECT THE BLACKS BOX PROBLEM. AND THIS IS BY PROPOSING THE IMPORTANCE OF INTERPRET -- IMPOSING THAT IMPORTANCE OF INTERPRETABILITY AND EXPLAINABILITY MAYBE OVERSTATED. ACCORDING TO THIS LINE OF THOUGHT, THERE'S A TRADE OFF BETWEEN ACCURACY AND EXPLAINABILITY AND TRANSPARENCY IN MACHINE LEARNING. IF THE MACHINE LEARNING SYSTEM CAN MAKE ACCURATE PREDICTIONS, THEN WHAT'S THE PROBLEM. IT DOESN'T SEEM -- IT DOESN'T SEEM TO MATTER IF MACHINE LEARNING SYSTEM IS NOT INTERPRETABLE AND EXPLAINABLE. SO I A FRIEND OF MINE ALEX LONDON ARGUED THAT'S COMING FOR -- COMMON FOR CLINICIANS TO PRESCRIBE MEDICATIONS WITHOUT FULLY UNDERSTANDING WHY THESE MEDICATIONS WORK. HE USES AN EXAMPLE THAT HE MENTIONS, MODERN CLINICIANS PRESCRIBE ASPIRIN AS FOR NEARLY A CENTURY WITHOUT UNDERSTANDING A MECHANISM THROUGH WHICH HE WORKS, LITHIUM HAS BEEN USED AS A MOOD STABILIZER FOR HALF A CENTURY, WHY IT WORKS REMAINS UNCERTAIN. GIVEN THAT MEDICINE IS CURRENT THEORIES OF DISEASE PATHOPHYSIOLOGY OR MECHANISM ARE OFTEN UNKNOWN OR UNCERTAIN VALUE RECOMMENDATION PRIORITIZE EXPLAINABILITY OR INTERPRETABILITY ARE UNWARRANTED. I'M GOING TO EXPRESS SOME RESERVATIONS AND THEN CONCLUDE ABOUT LYNDON'S ARGUMENT THAT ACCURACY MAYBE MORE IMPORTANT THAN IS OFTEN MORE IMPORTANT THAN EXPLAINABILITY. I THINK HE'S PROBABLY RIGHT. WE DON'T FULLY UNDERSTAND HOW MEDICATIONS WORK IN MANY CASES. BUT ARGUABLY WE HAVE SOME IDEAS REGARDING THE CAUSAL MECHANISM THROUGH WHICH THEY WORK. PEOPLE KNEW, TAKE LIKE AN EXAMPLE, PEOPLE KNEW THAT WILLOW CAUSES FEVERS AND PAIN TO BE REDUCED EVEN IF THEY DON'T KNOW SOLICITING -- ACID ACTIVE INGREDIENT IN THE PRODUCTION OF ASPIRIN WAS DOING REDUCTION OF THE PAIN. THIS CONTRASTS WITH MACHINE LEARNING SYSTEM WHICH WORKS THROUGH ASSOCIATIONS AND AT LEAST FOR NOW IS UNABLE TO TRACK CAUSAL RELATIONSHIPS. LIKEWISE IS TRUE THAT FOR A LONG TIME WE DON'T KNOW EXACTLY HOW LITHIUM WORKS, STABILIZES AN INDIVIDUAL'S MOOD, THE CURRENT HYPOTHESIS IS THAT IT MODERATES GLUTAMATE LEVELS IN THE BRAIN BUT WE KNOW THAT LITHIUM IS AT LEAST SOME -- THE DIRECTION WE KNOW THE DIRECTION OF CAUSALITY, IT STABILIZES MOOD IN SOME WAY. AGAIN, WE CANNOT SAY THE SAME THING ABOUT MACHINE LEARNING SYSTEMS, WE CANNOT TRACK SUCH CAUSAL RELATIONSHIPS. I KNOW THAT THERE IS SOME CAUSAL INFERENCE MACHINE LEARNING BUT I'M NOT SURE THOSE WORK. JUST TO SEE WHY IT MATTERS, WHETHER MACHINE LEARNING CAN TRACK CAUSAL RELATIONSHIPS OR NOT IS HELPFUL TO REMEMBER THAT MACHINE LEARNING IS VULNERABLE TO ADVERSARIAL ATTACKS. WE TALKED ABOUT THIS, IN ONE STUDY RESEARCHERS WERE ABLE TO GET DEEP LEARNING ALGORITHMS TO CLASSIFY IMAGE OF CAR OR DOG BY CHANGING ONE PECK SILL, IN ANOTHER STUDY THEY WERE ABLE TO CHANGE ABOUT .04%, 400 PIXELS OUT OF A MILLION, YOU LOOK AT PICTURES THEY ARE INDISTINGUISHABLE BUT YET MACHINES GET THEM WRONG. WHAT THAT SUGGESTS IS THAT DEEP LEARNING MACHINE MODELS CAN BE TRICKED IN THESE WAYS, MAYBE THEY'RE NOT -- WHEN YOU NEED TO WORRY ABOUT RELIABILITY, IT'S NOT JUST ABOUT WHETHER THEY'RE ACCURATE, WHETHER THERE CAN BE RELIABLE OVER TIME IN SORT OF ESPECIALLY IN SOMETHING AS HIGH STAKE AS MEDICINE. SO IN OTHER WORDS THERE ARE DIFFERENT WAYS TO BE WRONG, TRUE HUMANS MAKE WRONG DECISIONS BUT MOST OF THE TIME, AT LEAST WE HOPE, THEY ARE WRONG ON THE BASIS OF SOMETHING WE CAN EVALUATE. IN THE CASE OF THE MACHINE, MACHINE LEARNING WE DON'T HAVE THAT, WE DON'T HAVE THE BASING RELATION. THAT'S SOMETHING IMPORTANT TO KEEP IN MIND. THANK YOU. [APPLAUSE] >> PERFECT. LAST BUT NOT LEAST, INFORMATIVE ALSO QUITE LONG DAY, I APPRECIATE EYES UP AND ENGAGEMENT FOR THE LAST 45 MINUTES BEFORE THE WRAP UP. SO I INTRODUCED MYSELF EARLIER, MAGAZINE MACKINTOSH, BASE IN LONDON. AND THE INSTITUTE IS UK NATIONAL INSTITUTE FOR NINDS SCIENCE, WE PULL ACADEMICS ACROSS THE COUNTRY FROM OXFORD CAMBRIDGE, MANCHESTER, A DOZEN AND WE SERVICE TOGETHER AND WORK ON EVERYTHING FROM BASIC SCIENCE TO HEALTHCARE TO SUPPLY CHAIN TO POLICY ETHICS SO IT'S A LOVELY INTERDISCIPLINARY ENVIRONMENT AND I'M WORKING ON Ph.D. TO -- IN A MONTH SO BE NICE TO ME. I'M FEELING FRAGILE. IT'S EXCITED TO COME TO THIS SESSION, NOT TALKING Ph.D. WHICH IS ABOUT EARLY PREDICTION OF DEMENTIA, I WANT TO GIVE BASICALLY A FEW EXAMPLES OF SOME THINGS BEING ACTION IN THE UK WHEN IT COMES TO TRANSPARENCY AND FAIRNESS. COUPLE OF THINGS I WAS INVOLVED WITH THAT I THINK OBVIOUSLY I FEEL LIKE A U.S. AUDIENCE CAN LEARN A LOT FROM. WE ARE IN A FORTUNATE POSITION TO HAVE A HEALTH SYSTEM BASED ON NEED NOT ABILITY TO PAY, WHICH GIVES FERTILE GROUND TO THINK CAREFULLY ABOUT THINGS LIKE FURNACE SO WHEN THERE ARE THINGS THAT COULD AFFECT OR ATTACK THE EQUALITY OR ACCESS OF ACCESS TO HEALTHCARE WE DO NOT TAKE IT LIGHTLY IN THE UK. SO THE THREE EXAMPLES I WANT TO FIEF YOU FROM A SUPPLY PERSPECTIVE AND WORK FORCE AND FROM PROVIDER SLASH GOVERNMENTAL IN OUR CASE PERSPECTIVE SO THE FIRST SUPPLY PERSPECTIVE, EVERYONE HERE IS FAMILIAR WITH DEEP MIND, AND DEEP MIND HAS SPECIFIC DEED MIND HEALTH TWO ELEMENT LESS SEARCH ARM AND PRODUCT ARM. AND PRODUCT ARM IS ACUTE KIDNEY INJURY DETECTION. AND THIS WAS CONTROVERSIAL PARTNERSHIP ANNOUNCED IN THE UK A FEW YEARS AGO. ON THE BACK THEY ANNOUNCED ESTABLISHMENT OF INDEPENDENT REVIEW BOARD WITHIN DEEP MIND HEALTH. I WAS FORTUNATE TO JOIN AFTER WE STARTED. AND THE INTERNAL HEALTH THAT IS SET UP TO SCRUTINIZE EVERYTHING THAT WAS HAPPENING IN THE ORGANIZATION. IMPORTANTLY WE DID NOT SIGN ANY NDAs SO WE CAN HAVE ACCESS TO ANYTHING IN DEEP MIND, WE CAN TALK TO ANY MEETING, TECHNICALLY ASK FOR EMAILS WE WANT ACCESS TO AND IT WAS OUR LEISURE WHAT WE DID WITH THAT INFORMATION SO AS A GROUP WE WERE MIXTURE OF EXPLOYTITIONS, HEAD OF THINK TANKS, PATIENT ENGAGEMENT PROFESSIONALS, TOKEN MILLENNIAL ON THE BOARD TO OKAY MY OR PRESUME BUT IT WAS AN INTERESTING DIVERSE MIX OF PEOPLE NOT FROM THE RATHER MYOPIC HEALTH TECH SCENE. WE MET FOUR TIMES A YEAR, AND WE JUST CHATTED IN A ROOM OPAQUELY FOR SIX HOURS ABOUT THINGS CONCERNING US IN DEEP MIND HEALTH THAT MANIFESTED AS AN ANNUAL REPORT EVERY YEAR ABOUT THINGS WE THINK DEEP MINE CAN ACTION AND UNFORTUNATELY ABOUT NINE MONTHS AGO THE WHOLE THING WAS DISANDED WHEN DEEP MIND HEALTH WAS ASSIMILATED BACK TO GOOGLE HEALTH AS PART OF THE MASSIVE CONSOLIDATION PROJECT ACROSS GOOGLE HEALTH. AND YOU CAN IMAGINE THAT FOR THE PUBLISH BRITISH PUBLIC THIS WAS AN ENORMOUS SCANDAL BECAUSE THE SINGLE BIG CONCERN PEOPLE HAVE IN UK IS I DON'T WANT THE PROXIMITY OF DEEP MIND AND GOING TOLL GET CLOSER, NOT COMFORTABLE HOW THAT FEELS. THAT'S WRAPPED UP IN A PRIVATIZATION CONCERN THAT WE SEEM TO HAVE BUILT INTO DNA. I WANT TO SHARE WITH YOU THE WHOLE THING IS CLOSED, HOPEFULLY NOT PERMANENTLY, HOW I WOULD POTENTIALLY RETHOUGHT HOW WE DID THE IRB PROCESS GIVEN CIRCUMSTANCES WHAT HAPPENED. THERE'S THREE SPECTRA I WANT TO EXPOSE. ONE IS WE WERE INTERNAL VERSUS EXTERNAL GROUP, IF WE WERE AX EXTERNAL FACING GROUP WE NEED TO HAVE A BIG COMS ENGINE BEHIND US WE WERE NOT GOOD DEALING WITH JOURNALISTS, WE HAD SERIOUS ATTACKS HOW WE BEHAVE, HYGIENIST AT MY HOUSE TO ASK WHAT WAS HAPPENING IN THESE RELATIONSHIPS. IF WE JUST INTERNAL FOCUSED THEN WE COULD POTENTIALLY BE MORE USEFUL WITH OUR TIME AND HOW WE SPEND RESOURCES. THE SECOND THING IS WE ARE PROACTIVE VERSUS REACTIVE, THINGS WE FOUND OUT AFTER THEY HAPPENED BECAUSE WE DID NOT HAVE PROCESS WHICH TO ETHICALLY AUDIT A COMPANY THAT'S NOT PROCESSED THAT REALLY EXISTS SO WE DIDN'T KNOW WHAT WE WERE LOOKING FOR SO OFTEN THINGS WE DISAGREE WITH, WE FIND AFTER IT HAPPENED. THAT WAS A BIT OF A PROBLEM, ALSO CAME FROM LIMITATION WE WERE MEETING FOUR TIMES A YEAR, PEOPLE DID HAVE DAY JOBS, DIDN'T HAVE PROPER JOB BUT EVERYONE ELSE WAS BUSY. THE LAST IS NDA VERSUS NON-NDA. SOMETHING BOLD, ALLOWED US NOT TO SIGN NDA, BUT AT THE SAME TIME THE SINGLE BIGGEST CONCERN BRITISH PUBLIC LED US TO BE CLOSED A DECISION BACK INTO GOOGLE HEALTH AND THAT WAS HAPPENING ABOVE OUR HEADS AND THEREFORE WE DIDN'T HAVE ACCESS TO THAT INFORMATION. SO THERE'S AN IRONY IF YOU DO TRANSPARENCY YOU NEED TO DO IT SO RADICALLY OTHERWISE THERE'S NO POINT DOING IT SO THERE'S SOME INTERESTING LEARNINGS, I HAVE TO SAY THAT I DO COMMEND YOUR POSITION FOR HAVING NOT PUT IT IN THE FIRST PLACE BUT SYSTEM BOLD EXAMPLE OF RADICAL CORPORATE GOVERNANCE. SECOND IS SOMETHING PULLED UP A BIT, THIS IS ABOUT DIVERSITY OF THE WORK FORCE. ONE HEALTH TECH IS AN ORGANIZATION ABOUT THREE YEARS AGO, VOLUNTEER GRASSROOTS COMMUNITY SET UP TO TRY TO GET MORE REPRESENTATIVE INCLUSIVENESS IN THE HEALTH TECH. I HAVE NOTHING AGAINST WHITE MIDDLE CLASS MIDDLE AGE MEN, MAY DAD IS ONE, I'M GETTING MARRIED TO ONE BUT SOMETIMES YOU WANT DIFFERENT PEOPLE IN THE ROOM THAT'S OKAY SO WE SET UP ONE HEALTH TECH TO GET A SLIGHTLY RICHER MIX OF PEOPLE CONTRIBUTING TO THE DISCUSSION AND DIALOGUE. AND VOLUNTEER TEAM AND JUST A LOT OF ENERGY AND KINDNESS, COMMUNITY GRANTED 12,000 MEMBERS IN THE UK AND WE ARE FEDERATED COMMUNITY STRUCTURE THAT HAS NOW GROWN INTO NON-STANDARD INTERESTING AREAS OF HEALTH TECH SO WE'RE NOT IN BOSTON, NOT IN TELL AVIV BUT WE ARE IN RIO AND BRAZIL AND EGYPT. THERE'S A SLIGHTLY MORE DIVERSE VOICE IN HEALTH TECHNOLOGY AND WE HAVE LOTS OF EXAMPLES OF WHERE -- JUST OBVIOUS THINGS IF YOU HAD FOR EXAMPLE WOMEN AT THE TABLE WOULDN'T HAVE COME ACROSS, THE REALLY TROPHY EXAMPLE OF APPLE HEALTH AND NOT HAVING A TRACKER, 50% POPULATION IS -- THAT'S NEED FOR 50% POPULATION A NICHE REQUEST FOR PHYSIOLOGICAL TRACKING BUT WHEN YOU FIRST LAUNCH IT DID NOT HAVE THAT. AND APPLE -- HEAD LOSS OF OTHER EXCITING THINGS TO HAVE PA ENVIRONMENT. THERE'S FEW BASIC THINGS, IF YOU HAD WOMEN AT THE TABLE THAT WOULDN'T HAVE BEEN AN ISSUE. THE THIRD THING I WANTED TO BRING OUT IS WHAT -- THIS IS NOT SOMETHING THAT'S I HAVE BEEN WORKING MYSELF BUT CONTRIBUTING ON THE SIDE BUT WORK OF THE NH HAS A NEW SEXY DIVISION, IT'S CALLED NHSX. BEAN BAGS ARE IMMINENT, IT'S GOING TO BE TERRIBLE. IT WAS SET UP TO BRING TOGETHER ALL THE DIFFERENT DIGITAL DATA BITS ACROSS THE GOVERNMENT AND ACROSS -- IN THE DEPARTMENT OF HEALTH POLICY AND IMPLEMENTATION ARM OF THE DH POLICIES. AS PART OF THAT THEY SET UP SURE PEOPLE LIKE MATTHEW HATE THESE PRINCIPLE BUS LIST OF TEN PRINCIPLES WHICH TO ADHERE FOR DATA DRIVEN TECHNOLOGIES FOR COMPANIES LOOKING TO ENGAGE WITH EHS IF YOU READ PRINCIPLES AND PLETHORA OF PRINCIPLES THE MANY ORGANIZATIONS OUT THERE RELEASE, THEY'RE KIND OF THE SAME THINK ABOUT THE TRANSPARENCY OF THE DATA. I KNOW THE LIST, ALL OF THEM. THE PRINCIPLE FOUR AND SIX IN THE DATA DRIVEN DATA DRIVEN CODE OF CONDUCT FOR DATA DRIVEN TECHNOLOGY AND PRINCIPLE 4 BE FAIR TRANSPARENT AND ACCOUNTABLE ABOUT THE DATA BEING USED AND 6 WAS BE TRANSPARENT ABOUT LIMITATION OF DATA BEING USED. THESE ARE HIGH LEVEL FAIRLY NEBULOUS STATEMENTS BUT WHAT I HAVE TO COMMEND DEPARTMENT OF HEALTH ON ONE HSX NOW IS THEY HAVE GONE THROUGH A REALLY FURTHER DEPARTMENT, NOVEL PROCESS WHICH TO NOW THEY HAVE GOT A SELF-ASSESSMENT TOOL KIT DEVELOPING TO HAVE GUINEA PIG COMPANIES SAY WE'RE GOING TO WORK THROUGH THESE PRINCIPLE TO SEE WHAT IT'S LIKE TO ACTION SOME OF THESE. STHEY BASICALLY HAVE BEEN GIVEN A BLANK SLATE AND REQUIRE FEEDBACK TO THE DEPARTMENT HOW IT'S GOING WHAT IS THAT MANIFESTING AS. I WANT TO GIVE YOU A THING THAT IS EVERYBODY IS EXCITED ABOUT IN THE UK BUT GOOD CONCRETE EXAMPLE HOW FAIRNESS IS A THING TO ACTION. THIS CONCEPT OF VALUE OF DATA UNFORTUNATE TRIGGERING HEADLINE, THE NHS DATA ADDS 10 BILLION POUNDS TO OUR BUDGET, NOT SURE WHERE NAY GOT THE NUMBER BUT A BIG ONE. SO NOW EVERYONE IS OBSESSD WITH THE IDEA THEY CAN PRODUCT AND COMMERCIALIZE THE DATA. I APPRECIATE THIS AUDIENCE H. BUT RADICAL THOUGHT. SOME OF THESE INTERESTING DISCUSSIONS HAVE COME UP IF YOU ARE A COMPANY THAT IS LOOKING TO PARTNER WITH A TRUST OR HOSPITAL OR GQ PRACTICE WHAT IS THAT VALUE EXCHANGE? IS IT A GOFER, A LICENSING -- JOINT VENTURE, AND HOW DO YOU DO THAT WHEN YOU ARE CONFLICT OF INTEREST INTO THE SYSTEM? WHAT POINT DOES THE VALUE ACCRUAL HAPPEN IN THAT RELATIONSHIP? IS IT THE POINT WHERE Y'ALL -- THE FIRST HOSPITAL THAT GIVES THE DATA AND TAKE THE RISK AS A COMPANY OR IS THE VALUE ACCRUED IN THE HOSPITAL TO TEST GENERALIZABILITY OR IS IT THE CCG, A COMMISSIONING GROUP THAT ALLOWS YOU TO SCALE THAT SOLUTION AND OBVIOUSLY THESE ARE REALLY COMPLICATED LEGAL STRUCTURES WE ARE TRYING TO WORK OUT HOW TO NAVIGATE. BUT SOME AMAZING COMPANIES ARE TRYING TO DO THIS LIVE AND FEEDBACK TO NHSX HOW IT'S WORKING HOW IT'S GOING AND MANIFESTING. THAT'S A REALLY BRAVE AND QUITE GREAT COMMUNITY LED WAY TRYING TO DEVELOP SUPPLY FRIENDLY AND PROVIDER FRIENDLY POLICY. OVERALL IT IS A MASSIVE LUXURY TO BE IN THE UK DURING THIS BECAUSE WE DO CARE SO DEEPLY AND FUNDAMENTALLY ABOUT THE FAIRNESS WHAT THESE SOLUTIONS ARE DOING. FOR US IT'S A BIG ISSUE AROUND BIG TOPICS OF CAPITALISM, BASIC WAY THAT -- WE ARE CONCERNED, I THINK IN THE UK ABOUT THE INCREASING POWER AND ENTRANCE OF THE SUPPLY COMMUNITY. WE ALSO ACKNOWLEDGE THAT WE HAVE GOT INCREDIBLE COMPANIES DOORSTEP, DEEP MIND UP THE ROAD AND WE HAVE ACADEMIC ENVIRONMENTS OFTEN AS INSTITUTE, APPROACH WITH CAUTION AND IT'S A GREAT ENVIRONMENT TO BE IN AND YOU ARE WELCOME WHENEVER YOU WANT INTO LONDON. THANK YOU VERY MUCH. SO NOW FOR QUESTIONS. IS EVERYONE HAPPY COMING UP? I FIND IT TERRIBLY AWKWARD. IS THAT LIKE STANDARD? NO, NO, NO ONE PUTS THEIR HANDS UP. UP ASKED SO MANY QUESTIONS. >> I'M OUT OF MY QUOTA. BUT I WANT TO TALK ABOUT THE PANEL, I ENJOYED THE CONVERSATION. MIDWAY THROUGH I WANT TO GIVE HIM A HUG. SERIOUSLY, VERY NICE TALK. I REALLY WAS INTRIGUED ABOUT THE NOTION OF TRANSPARENCY AND FAIRNESS THAT YOU BROUGHT UP AND NOTION OF ML JUST AS MATTER OF BACKGROUND I COME FROM HEALTHCARE SYSTEM AND CANCER CARE ORGANIZATION. WE DEAL WITH THIS ISSUE OF FAIRNESS, EQUATABILITY AND HUMANE CARE AND ADOPTION OF THESE TECHNOLOGIES, POSE IT IS QUESTION REALLY AT THIS CENTER OF OUR BUSINESS MODEL. WE DECIDED TO BASICALLY COMBINE ETHICAL REVIEW AS PART OF DECISION SCIENCE TEAM AS WE ARE DESIGNING BUILDING THESE MODELS, AND EVALUATING BOTH IN VITRO AND IN VIVO TO HAVE ETHICIST WITH THE TEAM. I DON'T BELIEVE THAT WE HAVE A MODEL FOR THAT, THERE WAS A QUESTION OF IT, WHY WE SHOULD BE DOING IT, IT SHOULD BE DONE BY IRB. AS MENTIONED IRB IS NOT PREPARED TO DEAL WITH THESE ISSUE, AT LEAST CURRENT EMBODIMENT AT MEDICAL CENTERS ARE PREPARED TO DO THAT. I'M REALLY INTERESTED IN HEARING FROM THE PANEL WHAT YOUR THOUGHTS ARE ON HOW DO WE REALLY INSTITUTIONALIZE THIS ETHICAL -- IF THIS IS SOMETHING ERRORS THAT WILL DISRUPT THE WAY HEALTHCARE IS DONE. I SEE NHS DOING SOME OR SYSTEMATIC EFFORT TO UNDERSTAND SOME ASPECT OF IT BUT HERE IN THE U.S. I DON'T KNOW WHAT WILL BE THE RECIPE. TELL US MORE WHAT YOU THINK. >> SO I THINK I DO AGREE WITH NAGEM, WE NEED SAYS SPACES TO DO THESE END TO END DEPLOYMENTS. I TRY TO SEPARATE, THIS HAPPENS QUITE A BIT. SOMEBODY WILL -- STUDENTS CAN BE CREEPY, THEY WILL STALK YOU AND COME TO YOUR OFFICE AND SAY I HAVE A STARTUP IDEA AND YOU HAVE TO -- I'M GOING TO AUTOMATICALLY DETERMINE WHO SHOULD GET MENTAL HEALTH TREATMENT. YOU SAY NO, THAT'S A BAD IDEA. THAT'S -- I WOULD NEVER HELP YOU DO THAT. I'M LOOKING AT YOUR MODEL AND IT'S BAD. THEY WILL READ A RESEARCH PAPER WITH LIMITATION SECTIONS AND METHODS AND A CONTEXT AND IS WELL WRITTEN AND THEY WILL SAY I AM GOING TO DO THAT. I'M GOING OPERATIONALIZE IT, PRIVATE ADVERTISE IT, DO -- PRIVATIZE IT, MOVE A STARTUP MAKE MONEY, THOSE THINGS SHOULD BE SEPARATE. AS THEY ARE IN ALMOST ALL RESEARCH COMMUNITIES. WHEN WE TEST SOMETHING NEW IN A CELL LINE AND SHOW THAT IT WORKS IN HELO CELLS WE DON'T AUTOMATICALLY SAY A HA, IT'S CURED, NO, IT'S A RESEARCH PAPER, IT'S CUTTING EDGE, IF THERE'S A WHOLE SERIES OF PEOPLE TO TRY TO GET INTO PRACTICE. SO I THINK THOSE THINGS SHOULD BE KEPT SEPARATE BECAUSE ONE THING THAT I FEAR AS RESEARCHER IN THIS FIELD IS BACKLASH. PEOPLE SAYING WELL, THERE'S DANGER AND YOU HAVE TO CONTROL IT AND WHY ARE YOU DOING -- THERE'S ALWAYS DANGER ALL TOOLS HAVE DANGER. WE NEED TO BE CLEAR ALLOWING RESEARCHERS TO DO RESEARCH OFTEN ON RETROSPECTIVE DATA OR WITH SILENT LEARNING OR THERE'S MANY INTERRUPTED TIME SERIES MODEL THERE'S DIFFERENT WAYS TO TRY TO DO THIS. I THINK THE OTHER THING THAT IS REASON I HAVE SLIDES ON ETHICS IS NOT KNEW RANDOMIZE CONTROL TRIALS MAKE THESE DECISIONS ALL THE TIME, YOU ARE GETTING HALF SICK OF PLACEBO. IT IS NOT AS IF WE DON'T HAVE ETHICAL QUALMS ALREADY. WE ARE NOW DEALING WITH AN AGENT THAT DOES THING FROM A HIGH FUNCTIONAL LEVEL WITHOUT MORALITY ASSOCIATED WITH THAT INTELLIGENCE. IT'S JUST A TOOL, WE JUST NEED TO PUT IT WITHIN OUR ETHICAL TOOL. >> MAYBE I CAN CHIME IN HERE. SO THERE ARE PROBABLY MORE THAN THREE WAYS TO THINK ABOUT THIS BUT THERE'S THREE WAYS TO THINK ABOUT THIS. YOU CAN TAKE A CONSEQUENTIALIST PERSPECTIVE, WE HAVE BENEFITS AND COST. THEY CAN SAY IN THIS CASE IF THE BENEFITS OUTWEIGH THE COST, IT DOESN'T MATTER IF WE COLLECT BUNCH OF DATA AND PEEP DON'T WANT THEIR DATA TO BE COLLECTED. ADS LONG AS WE BENEFIT A LOT OF PEOPLE LIKE INTO THE FUTURE, ET CETERA, ET CETERA. THAT'S WHAT WE SHOULD DO. THAT'S ONE WAY OF THINKING ABOUT IT. THE OPPOSITE WAY IS SORT OF WHERE YOU GIVE -- YOU SAY PATIENT AUTONOMY. YOU SAY WE GOT TO MAKE SURE IT'S TRANSPARENT, WE HAVE TO MAKE SURE THAT PATIENTS ARE INFORMED, AND ET CETERA, THE PROBLEM WITH THAT MODEL IS OFTEN TIMES THE ISSUES ARE TOO COMPLEX FOR INFORMED CONSENT FORMS CLINICAL TRIALS AND ALSO 20 PAGES LONG NOBODY READS IT OR UNDERSTANDS WHAT'S GOING ON. THAT IS ONE ISSUE BUT ANOTHER ISSUE IS IT PUTS TOO MUCH ONUS ON INDIVIDUAL PATIENTS SO THAT'S WHAT I DON'T LIKE ABOUT THAT PARTICULAR MODEL. IT MAKES US EACH OF US RESPONSIBLE FOR FIGURING OUT WHETHER WE SHOULD DO SOMETHING WHEN WE DON'T HAVE TO TIME OR ET CETERA, ET CETERA. THERE'S A THIRD MODEL PEOPLE ARE PUSHING TOWARDS SYMPATHETIC WITH. THAT'S THIS IDEA OF HUMAN RIGHTS MODEL. SO ACCESS NOW AND THERE ARE A BUNCH OF DIFFERENT ORGANIZATIONS THERE, ACTUALLY TALK ABOUT ARTIFICIAL INTELLIGENCE AND HUMAN RIGHTS. SO THE IDEA THERE IS THAT LOOK, WE ALREADY HAVE A FRAMEWORK FOR THINKING ABOUT PEOPLE'S RIGHTS. THEY HAVE A RIGHT TO PRIVACY, THEY HAVE A RIGHT TO ET CETERA, ET CETERA. WHEN WE DO THESE TYPE OF TESTING OR WHEN WE COME UP WITH NEW TECHNOLOGIES WE NEED THE MAKE SURE THAT MINIMALLY WE ARE NOT VIOLATING PEOPLE'S RIGHTS. THAT'S GOING TO INCLUDE SOMETHING LIKE GETTING INFORMED CONSENT, ET CETERA, ET CETERA. NOT ROLL ONLY THAT, THE THING IS, THE WAY HUMAN RIGHTS MODEL WORKS AS I UNDERSTAND, IT SAYS THAT INDIVIDUAL RESEARCHERS, AI RESEARCHERS, COMPANIES, GOVERNMENTS, HAVE A RESPONSIBILITY THEMSELVES, TO MAKE SURE THEY ARE NOT VIOLATING. THEY NEED TO BE VIOLATING HUMAN RIGHTS. THEY NEED TO BE PROACTIVE, NOT TO SAY WELL, IF THE PATIENT SIGNS OFF ON IN. TOED CONSENT OKAY I CAN DO WHATEVER I WANT. WE ALL KNOW HOW THAT WORKS. WE CAN DO GO TO THIRD WORLD, THE WHOLE PROCESS IS FRAUD. THIS OTHER MODEL REALLY SAYS IT'S A COLLECTIVE, OUR SOCIETY: WE NEED TO DO IT TOGETHER. >> SLIGHTLY DIFFERENT PERSPECTIVE. I AGREE, WE HAVE BEEN TO NOT GIVE ONE SIZE FITS ALL SOLUTION OR GENERALIZE TOO MUCH. WE TALKED ABOUT A LOT OF DANGERS IN THE USING BLACK BOX APPROACHES BUT NOT EVERYTHING IS A DEEP NEURAL NET WHERE WE DON'T HAVE A SENSE OF WHERE THE CONCLUSIONS ARE COMING FROM. I COMPETELY AGREE IT'S A BENEFITS FROM RISK EQUATION. IN SOME CASES, THERE ARE MANY, MANY MACHINE LEARNING APPROACHES THAT CAN BE APPLIED TO DATA TO GLEAN BETTER INSIGHT INTO PEOPLE THAT MIGHT BE HAVING A MENTAL HEALTH CRISIS OR A BEHAVIORAL ISSUE. WHAT YOU DO WITH THAT DATA, HOW IT'S PRESENTED BACK TO THE INDIVIDUAL I THINK ARE ALL THINGS THAT NEED TO BE TAKEN INTO ACCOUNT. I JUST THINK WE SHOULD BE CAREFUL NOT TO SAY WE DON'T WANT TO DO ANYTHING UNTIL WE UNDERSTAND THE ETHICAL ISSUES AND UNDERSTAND ALL THE ASPECTS. Q. I WANT TO BRING UP THE TOPIC OF BIAS ALGORITHMIC BIAS. I THINK IT'S BECOMING BETTER ESTABLISHED AMONG EVERYONE YOU HAVE A SENSITIVE VARIABLE AND OPTIMIZE FOR AS YOU NOTED, FOR MAXIMAL PREDICTIVE PERFORMANCE ACROSS AN ENTIRE POPULATION THAT SUBPOPULATIONS WILL HAVE DIFFERENTIALS AND PERFORMANCE. CERTAINLY HAVE IDEAS BUT CURIOUS TO HEAR HOW DO YOU WHAT DO YOU THINK ARE REASONABLE APPROACHES, I WAS TALKING WITH MACHINE LEARNING RESEARCHER AT DUKE, I KNOW THE MATH AND THOSE MACHINE BIAS PEOPLE USE, IN THE END THEY WANT TO REDUCE PREDICTIVE PERFORMANCE AND OF COURSE FOR HIM IT WAS LIKE HE WASN'T ADDRESSING THE ETHICAL ISSUESES HERE. HOW DO WE -- IF WE ARE USING ETHICS ADS OBJECTIVE FUNCTION AND BALANCING THAT AGAINST ACTUAL QUANTITATIVE PERFORMANCE WHAT ARE STRATEGIES WOULD HAVE BEEN THINKING ABOUT FOR ADDRESSING THAT ISSUE? >> I THOUGHT TWO WAYS TO ADDRESS THIS ISSUE IN THIS KIND OF WORK SPECIFICALLY, ONE IS THAT THIS DATA IS -- THERE'S THREE WAYS I HAVE BEEN GENERALLY THINKING AND I THINK ONE IS JUST OFF THE TABLE UNFORTUNATELY. SO THERE'S EQUALITY AND HEALTHCARE. IF YOU WANT TO GO SEE A DOCTOR, THERE'S A QUALITY OF ACCESS, DO WE HAVE ACCESS TO THE SAME HEALTHCARE. I HAVE PRIVATE INSURANCE, SOMEBODY ELSE HAS PUBLIC INSURANCE. WE DON'T HAVE THE SAME ACCESS, WE CAN'T GET THE SAME THINGS. WHEN I LOOK AT A LONGITUDINAL TRAJECTORY OF MY INTERACTIONS. IF I'M HAVING PRIVATE HEALTHCARE, I HAVE BETTER RESOURCES I WILL MAKE GOOD REPREDICTIONS FOR PEOPLE WITH THAT RESOURCING. THERE'S LITTLE I CAN DO WHEN THERE'S INEQUALITY OF ACCESS. AS MACHINE LEARNING ALGORITHM, OR AS HUMANS, IF SOMEBODY PRESENTS TO YOU AND YOU SEE THE PAST TWO DECADES OF LAB VALUES AND TESTS AND NOTES, YOU CAN MAKE SOME INFERENCES, YOU CAN DECIDE WHAT TO WITH THIS PATIENT. IF SOMEBODY HAS NEVER BEEN TO SEE A GENERAL PHYSICIAN AND THEY HAVE JUST BEEN A FREQUENT FLYER AT THE ICU TO DEAL WITH CHRONIC CONDITIONS WITHOUT ACTUAL GOOD HEALTHCARE PRACTICE THERE'S LITTLE YOU CAN DO. MACHINE LEARNING MODELS AREN'T GOING TO CHANGE THAT BUT OTHER TWO THINGS I HAVE BEEN THINKING ABOUT ARE EQUALITY OF TREATMENT AND EQUALITY OF E OF OUTCOME ARE TWO THINGS. LET'S ASSUME WE ARE IN CANADA. PLUG FOR CANADA. CANADIANS ARE VERY PROUD TO BE CANADIAN. MORE THAN AMERICANS. I DIDN'T THINK THAT WAS POSSIBLE. DO YOU KNOW THERE'S A CANADA DAY? NO. THE QUALITY OF ACCESS, LET'S PRETEND THAT'S THE SAME. EVEN HAS SAME ACCESS, QUALITY OF TREATMENT MEANS IF MATTHEW AND I GO INTO CLINICIAN, BECAUSE WE CAN ACCESS THAT, WE CAN GET THE SAME TREATMENT, WE KNOW THAT DOESN'T HAPPEN TODAY. LIKE CARDIOVASCULAR DISEASE IN WOMEN. WE CAN SAY THE SAME THINGS AND HAVE THE SAME LAB VALUES AND THE SAME HISTORY, AND THIS IS AN EPI LEVEL WHAT PEOPLE HAVE SEEN HAPPEN. AND THEN THERE'S A QUALITY OF OUTCOME. LET'S SAY WE GIVE YOU THE SAME DRUG. BUT JUST RESPOND DIFFERENTLY TO IT, WE HAVE EQUALITY OF ACCESS, WE HAVE QUALITY OF TREATMENT WE HAVE QUALITY OF OUTCOME, THOSE ARE NOT THE SAME. IF WE WANT EQUALITY OF OUTCOME IF THAT'S WHAT OUR HEALTHCARE SYSTEM CARES ABOUT, THEN YOU MIGHT HAVE TO GIVE PEOPLE GIVEN KINDS OF TREATMENTS. THOSE TREATMENTS MIGHT HAVE TO BE DIFFERENT BECAUSE THEY DON'T HAVE THE SAME ACCESS. SOME DEPENDS ON WHERE WE PUT OUR ETHICS. I THINK THAT'S A MUCH LARGER QUESTION THAN WHAT DO YOU OPTIMIZE? THOSE ARE THE KIND OF WAYS I HAVE BEEN THINKING ABOUT I. -- ABOUT IT. >> ONE BRIEF COMMENT THEN ONE QUESTION. THE COMMENT IS CAME UP AGAIN ON THIS PANEL, MAKE IT HERE. COME UP SEVERAL TIMES DURING THIS MEETING. MAKE THIS ANALOGY TO MEDS WE POINT TO MEDS, WE HAVE NO IDEA WHAT THEY DO, WHAT EFFECTS WE HAVE. I WOULD POINT OUT THAT IN FACT WE REALLY DON'T UNDERSTAND THE MECHANISM FOR BUNCH OF MEDS WE USE AND WE ALSO RELEASE MEDS INTO THE POPULATION WITH A FAIRLY LIMITED KNOWLEDGE ABOUT THE EFFECTS THEY'LL HAVE ON THE POPULATION, DOES NOT THE MOST REASSURING ANALOGY. I DON'T THINK WE SHOULD BASE OUR FORWARD LOOKING PLANSTOR MACHINE LEARNING ON THAT MODEL. WE ARE STILL TRYING TO FIX THAT PART. ON THE MEDICATION SIDE. THAT'S -- ON THE NHS QUESTION YOU HAD SINCE YOU WERE IN THE MIDDLE, THIS REALLY FASCINATING SITUATION WITH GOOGLE AND DEEP MIND AND WITH DEED MIND AND NHS AND GOOGLING -- GOOGLE DEEP MIND AND NH S. THERE'S MULTIPLE UPS AND DOWNS, FASCINATING PROJECT WITH GREAT TEAMS. GIVEN WHAT YOU SAW WITH REALLY ROBUST EFFORT KIND OF TRANSPARENCY INITIATIVE GIVEN THE WAY ALREADY LARGE TECH WORKS THESE DAYS HERE INTERNATIONALLY, DO YOU THINK THERE IS AN APPROACH TO TRUE TRANSPARENCY AT THAT LEVEL? DOES IT NECESSARY HI INVOLVE REGULATION OR IS THERE A SELF-REGULATED AAPPROPRIATE COULD BE USEFUL? >> I'M NOT SPEAKING -- WE DON'T EXIST. WE HAVEN'T OVERCOME THIS BUT I SIMPLY DON'T BELIEVE A GROUP OF SMALL NUMBER OF INDIVIDUALS SITTING IN DARK ROOM FOUR TIMES A YEAR CHATTING SIX HOURS IS A TRANSPARENT PROCESS. THERE'S A HUGE IRONY THE FACT THAT IF YOU READ ON YOUR REPORTS YOU WOULD HAVE NO IDEA HOW WE CAME ABOUT THE CONCLUSION WE CAME ABOUT HOW WE SOUGHT THE QUESTIONS THAT WE SOUGHT. SO FOR ME THERE'S AN ENORMOUS IRONY IN THE WHOLE IRB STRUCTURES AS IT IS. JUST LAUNCH IN THE U.S., COMPANY CALLED MONZO, VERY BEAUTIFUL, IT'S A CUSTOMER-CENTRIC BANK. THE REASON I GIVE THEM EXAMPLES IS THEY HAVE A RADICAL VIEW HOW THEY LOOK AT CORPORATE GOVERNANCE AND TRANSPARENCY, THAT'S HOW I FEEL MORAL SHOULD BE OPERATING. THEY JUST SAY GOING TO GET NICKED ANYWAY SO WE PUT I ONLINE. SO THEY PUT ALL THEIR PROCESSES FUTURE PIPELINES THEY PUT THEIR TOOL KITS, EVERY SINGLE DOCUMENT THEY HAVE, ON LINE FOR SHOW. THEY RUN WORKSHOPS HOW TO DO EXACTLY HOW YOU DO IT ALONSO AND THEIR ATTITUDE IS BEAR IT ALL, RUN DOWN THE STREET NUDE EQUIVALENT OF. AND AS A RESULT IT'S COMPLETELY SHIFTED WHAT THEY AS A COMPANY. IT'S GIVEN ENORMOUS COMPETITIVE EDGE IN THE UK. YOU CAN'T MISINTERPRET THE INFORMATION. THAT WAS ONE OF THE QUITE IMPORTANT THINGS ABOUT THE DEEP MIND EXAMPLE IS THAT A QUOTE THAT WAS SAID BY ONE FOUNDER IS I FEEL LIKE WE HAVE BEEN GIVEN THE MEDIA THE STICK TO BEAT US. AND SOME DEGREE A LITTLE BIT TRANSPARENT, IT'S ALMOST WORSE BECAUSE YOU ARE NOT GIVING THE WHOLE STORY. AND SO PEOPLE WILL FILL IN THE GAPS WITH THEIR SLIGHTLY APOPLYTIC SCENARIOS, THAT WAS IN TURN VERY DETRIMENTAL. SO I WOULD DO A SLIGHTLY MORE BARREL EXAMPLE, THE SECOND WAY TO ANSWER THAT QUESTION IS WE HAVE SEEN A COMPANY IN UK CALLED SIGN HEALTH WHICH HAS BASICALLY MADE THEIR COMPETITIVE EDGE, THEY HAVE A COMMERCIAL MODEL OF NHS. THEY PRESENT AT CONFERENCES AND THEY SAY WE GIVE BACK TO THE SYSTEM, WE WORK IN PARTNERSHIP WITH THE SYSTEM THAT IS ONE HAND THEY HAVE A REVENUE SHARE AGREEMENT WITH V TRUST, ON THE OTHER HAND THEY ALSO SIGN EXCLUSIVE DATA DEAL WITH THEM SO TAKE ONE HAND GIVE WITH THE OTHER BUT THE WHOLE EDGE IN USP IS NOW THEY ARE ETHICAL COMPANY AND THAT RESONATED INCREDIBLY WELL WITH THE BRITISH PUBLISH SO NOW BEING ETHICAL IS COMMERCIAL BENEFIT BASICALLY. >> MAYBE I CAN COMMENT ON YOUR COMMENT. I WANT TO MAKE MAKE CLEAR I WASN'T SUGGESTING I WAS SKEPTICAL ABOUT THE CURRENT MEDICINE, IN FACT JUST THE OPPOSITE. I THINK LIKE ANTIBIOTIC WE KNOW THAT WE KNOW HOW IT WORKS AND TREATS BACTERIAL INFECTIONS. BETA BLOCKERS REDUCE ANXIETY, ET CETERA. SO THERE ARE A LOT OF THINGS WE DO KNOW. >> IN THE PIPELINE FOR NHSX. THEY ARE DOING STRUCTURE SO EVERYTHING IS GOING TO BE OPEN STANDARDS, MUCH MORE AGGRESSIVELY MANDATED THAN BEFORE AND THERE'S PART OF THE REASON NHS WAS SET UP IS BECAUSE PRE-EXISTING UNIT WHICH IS AN ORGANIZATION CALLED NHS DIGITAL IS NOT PERFORMING AS WELL AS IT COULD HAVE DONE. PART OF THAT IS HOW AGGRESSIVELY THEY WERE MANDATING THESE REQUIREMENTS FROM THE SUPPLY. SO THESE ARE CHANGE A LOT. THE ULTIMATE GOAL O THE CONDUCT IS TO ITERATION ENRICHING PROCESS, THE GOAL IS TO MAKE IT INTO REGULATION AND (INAUDIBLE) IS KEEN TO DO THAT. PEOPLE ARE KEEN TO START TO OPERATIONALIZE THIS BUT APPROPRIATELY WE DO -- LED BY DOING LIVE EXAMPLE WHICH IS QUITE APPROPRIATE GIVEN LOTS HAVEN'T BEEN DONE BEFORE. >> BRITTANY WAS SAYING THANK YOU ALL FOR YOUR PRESENTATIONS, I ENJOY THEM SO I WANT THE MAKE A COMMENT FIRST THEN I HAVE A QUESTION GOING IN A LITTLE DIFFERENT DIRECTION. WE HAVE TOUCHED ON TODAY SO THE COMMENT I THINK I MIGHT BE THE ONLY PERSON FROM ALABAMA. FROM A CITY THAT IS DIVERSE AND WHERE MOST ARE NOT, WE ARE WORKING REALLY HARD TO TRY TO MAKE SURE WE ARE EQUITABLE AND WHEN WE DEVELOP MODELS FOR THINGS LIKE RISK SCORES WE ARE TAKING INTO ACCOUNT OTHER POPULATIONS, BOTH HEALTHCARE ACCESS AS WELL AS DIFFERENT GENETIC MAKE UPS AN THOSE SORTS OF THINGS. I APPRECIATED THE ATTENTION SEVERAL GAVE TO THOSE SORTS OF ISSUES, THAT'S SOMETHING WE ARE WORKING ON HARD THERE WITH ALL OF US PROGRAM AND ALABAMA GENOME HEALTH INITIATIVE. MY QUESTION IS REALLY IN A DIFFERENT DIRECTION THINKING TRANSPARENCY. THAT IS I DON'T THINK ANYBODY MENTIONED TODAY ABOUT HOW A LOT OF THESE WHEN THINKING HOW TO PAIR PATIENTS WITH DRUGS IN THESE UNDIAGNOSED DISEASE FRAMEWORKS OR IN CLINICAL TRIALS WHERE THE GOAL IS TO MATCH A PATIENT WITH A DRUG BASED ON SOME ALGORITHM OR MACHINE LEARNING THE DATA UNDERLYING THOSE IS FROM THINGS LIKE CANCER CELL LINE ENCYCLOPEDIA OR OTHER DATABASES IN MOUSE MODELS AND MODEL ORGANISMS. SO I WONDER IF YOU GUYS FIRST HAD THOUGHTS ON THINGS LIKE HOW WE CAN BE TRANSPARENT WITH CLINICIANS AND PATIENTS ABOUT WHERE THE DATA THE ALGORITHMS ARE WORING ON COMES FROM, GIVEN A LOT OF I DON'T KNOW IF ANYBODY IS ON TWITTER, IN MICE MEMES GOING AROUND AND THOSE THINGS. TWO, THINKING HOW MACHINE LEARNING SUBFIELDS LIKE TRANSFER LEARNING CAN PLAY INTO THIS. THANK YOU. >> I THINK TRANSFER LEARNING HAS ENORMOUS POTENTIAL OR THE NEXT I DON'T KNOW IF YOU KNOW LIKE MAMMAL, MODEL AGNOSTIC META LEARNING OR REPTILE BECAUSE THEY HAD TO TO A FOLLOW-UP TO IT SO THINGS THAT SAY WE NEED TO LEARN IN FEW OR SINGLE SHOTS. YOU HAVE TO BE ABLE TO TAKE A MODEL THAT YOU HAVE AND INCREDIBLY QUICKLY ADJUST IT TO NOVEL INFORMATION TO A NEW SETTING. THAT'S PROMISING. TALKING ABOUT THE INFORMATION MODELS ARE BASED ON AND WHAT'S BAKED IN THAT IS HONESTLY ONE OF THE THINGS I FIND MOST FRIGHTENING T. THIS YOU THINK ABOUT MACHINE LEARNING PERSON, EVEN WITH MODERATE MEDICAL KNOWLEDGE, AND STRONG COLLABORATOR. IN EVERY OTHER DOMAIN I WORK IN, I AM AN EXPERT, I AM EXPERT IN SPEECH, I CAN LISTEN TO THE SAMPLE, AND SEE WHAT'S HAPPENING, I'M AN EXPERT IN TEXT, I CAN READ MY OWN SENTENCE AND SEE IF IT MAKES SENSE. EXPERT IN VISION. I CAN'T TELL IF THE DOGS ARE -- THE CHIHUAHUA'S ARE MUFFINS I FIND THAN EXAMPLE CHALLENGING SO I FEEL SORRY FOR THE ALGORITHM BUT MOST OF THE TIME I'M AN EXPERT. BUT WHEN I'M EVALUATING RECOMMENDATIONS FROM A MODEL EVEN THE CLINICIAN IS NOT ALWAYS AN EXPERT IN THAT, I HAVE DEFINED AN EXPERT ON THAT SPECIFIC THING AND WHEN I HAVE TWO EXPERTS ALLOWED TO DISAGREE. SO DOCTORS ARE EXPERTS BUT THEY ARE ALLOWED TO DISAGREE IN THE RECOMMENDATIONS AND THE DIAGNOSES, EVEN SOME OF WHAT WE TALKED ABOUT THIS MORNING. WHAT IS HYPERTENSION. WHAT ARE THESE THINGS? WHAT ARE MY LABELS? WHAT I'M TRYING TO MAKE A MODEL, I BAKE A LOT OF THESE THINGS IN I DO IT BECAUSE IT'S THERE'S IN THE UMLS DEFINITION, IN THE X NORM, ON THE PUBMED. ABSTRACTS THAT I SCRAPED AND MADE INTO A WARDEN BEDDING THAT I USE TO I HAVE ALL THIS DATA FROM THESE SOURCES THAT EXPERTS GENERATED PUBMED PUBMED FOR ALL THESE THINGS, RIGHT? NOBODY CAN VERIFY THOSE INDIVIDUALS PIECES OF INFORMATION. NOBODY CAN REALLY VERIFY THE ENTIRE THING. THE ENTIRE PROCESS. WE ARE HONESTLY TRYING TO FIGURE THESE THINGS OUT. THAT IS THE THING I AM MOST CONCERNED ABOUT. I TOLD YOU JUST A MOMENT AGO I LOVED YOUR QUESTION. OR YOUR COMMENT EARLIER WHERE YOU SAID HIGHLIGHTING THE BIOARCHIVE PAPER THAT SAID GENOMICS TESTS, THEY HAVE A CRAZY HIGH FALSE POSITIVE RATE FOR WHAT WAS THE NUMBER? 84 PERCENT. THAT'S CRAZY. THAT'S A HIGH FALSE POSITIVE RATE. SO I THINK THAT THAT'S WHAT I WORRY ABOUT. NOT CAN I STRAY TO FIND NEW WAY OF BEING TRANSPARENT OR AUDIT THE DATA OR THE MODEL, I WORRY THAT IN MEDICINE THE DATA IS NOT ENOUGH T. A CHAIR IS A CHAIR AND WE CAN ALL LOOK AT PICTURES OF CHAIRS AND UNLESS IT'S A ESOTERIC CHAIR WE AGREE IT'S CHAIR SO WHEN IT COMES TO MEDICINE SHOWING ME ALL THE DATA, IS NOT ENOUGH. SO WE BUILD ALL THESE CONSTRUCTS ON TOP OF IT, SO SOMEBODY TOLD ME ABOUT THE HAP MAP PROJECT AND DRILL DOWN INTO HOW IT WAS DEFINED. IT WAS DISTURBING, I HAD NO IDEA. I JUST LEARNED THIS RECENTLY AND I DON'T WANT TO LOSE IT BUT ALL THESE THINGS ARE BAKED IN. SO THAT'S WHAT I WORRY YOU HAVE HAD ABOUT. IN RESPONSE TO THAT, I PUT THAT ON YOUR SLIDE, ESPECIALLY IF THEY ARE REGULATORY PEOPLE IN THE ROOM, SHOULD WE STEP BACK AND THINK ABOUT HOW DO WE AUDIT THE PROCESS. OPPOSED TO INDIVIDUAL COMPONENTS OF THE PROCESS BECAUSE PEOPLE WANT TO LOOK INSIDE THE BLACK BOX OR DON'T WANT TO LOOK AT THE DATA WHICH IS HARD TO LOOK AT IN AND OF ITSELF. IS THERE SOMETHING WE CAN LEARN FROM THE SOFTWARE ENGINEERING WORLD WHERE CONCEPT OF CONTINUOUS INTEGRATION AND DEPLOYMENT UNIT AND INTEGRATION TESTING ARE THINGS THAT ARE NOW WELL ACCEPTED AS BEST PRACTICES IN THE SOFTWARE WORLD. BUT ARE THOSE THINGS WE CAN THINK ABOUT ADS PROVIDING US HOOKS INTO UNDERSTANDING THE QUALITY OF PROCESS OPPOSED TO INDIVIDUAL COMPONENTS. >> I THINK WE SHOULD BE O AUDITING PROCESSES, THERE SHOULD BE A TRANSPARENT PROCESS. IF YOU AUDIT THE DATA, IT'S BIASED. I NOPE THAT NOW. FROM IF YOU AUDIT THE MODEL, LOG ODDS ARE NOT -- I CAN SHOW YOU MANY PAPERS. BUT NOBODY THINKS IN LOG ODDS NOBODY EXPO ANYONEIATES A DATA CO-EFFICIENT IN THEIR HEAD. SO BLACKS BOXES, YOU CAN TRY TO EXPLAIN THEM, I WOULD SAY ANY MODEL IS A BLACK BOX WHEN WE ARE THINKING BASIC DECISIONS. BUT THERE'S THE WHOLE HOW DO I DEPLOY THAT? YOU HAVE PERFECT DATA. SOMEBODY MADE COMMENT EARLIER TODAY, YOU HAVE PERFECT DATA PERFECT MODEL RECOMMENDATION BUT FEMALE CLINICIAN DELIVERS THAT RECOMMENDATION >> HOW DO YOU GET AROUND THAT? THAT IS NOT A DATA PROBLEM, THAT IS NOT A MODEL PROBLEM, THAT'S NOT A DELIVERY PROBLEM. THAT IS A PROCESS PROBLEM. IF OUR GOAL IS IMPROVED CARE IT'S PROCESS AUDIT. I THINK WHAT EVERYBODY HAS BEEN SAYING. YOU CAN'T JUST SAY MY STEERING WHEEL WORKS GREAT THAT, SIDE MIRROR IS FANTASTIC. WE NEED THE WHOLE THING. >> RIGHT. THAT SOUND TO ME FOR THE REGULATORS IN THE ROOM AND ONE OF MY COLLEAGUES IS NOW THE DEPUTY PRINCIPLE DEPUTY COMMISSIONER OF THE FDA IS HOW DO WE MOVE FROM THE CONCEPT OF SOFTWARE AS MEDICAL DEVICE BECAUSE THAT DOES REALLY DISPARATIZE A COMPLEX PROCESS. HOW DO WE BREAK THAT OUT? AND BREAK THAT OPEN? SOUNDS LIKE A CHALLENGE. >> I DO THINK THERE ARE BASIC WINS, IF WE'RE GOING TO COMPARTMENTALIZE IT. THERE'S THE HYPOTHESES AND PEOPLE BEFORE THE DATA THE ALGORITHMS AND THE OUTPUT, THINGS WE CAN DO TO MAKE LESS RUBBISH THAN NOW SO MAKE SURE YOU HAVE EXPERTISE AROUND THE TABLE HYPOTHESIZING WHAT YOUR STUDY LOOKS LIKE. HAVING THAT BREADTH OF INPUT IN DISTRESSORTY WHATEVER WAY YOU WANT TO THINK ABOUT DIVERSITY. WHEN Y'ALL ACTIONING THE OUTPUT, LOTS OF PEOPLE LIKE TO SAY THE OUTPUT WAS BIASED AND THAT'S WHAT IS GOING TO PERCOLATE IN CLINICAL PRACTICE, ACTUALLY THAT WOULD NEVER REALLY HAPPEN AND THEREFORE SLIGHTLY AGAIN APOCK LIP TICK SCENARIO DOESN'T REALLY HAPPEN BECAUSE SAFEGUARDS IN PLACE. SO THE WHOLE PROCESS AND ORGANIZATIONS NEED TO BE TRUSTWORTHY AND GOES BACK TO THAT QUESTION, THERE'S BASIC BITS IN THOSE FOUR COMPONENTS THAT COULD BE IMPROVED. >> MAYBE SOMEBODY SAID ONE OF THE WAYS TO MAKE SURE THE WHOLE PROCESS IS WORKING WELL IS TO MAKE SURE EACH COMPONENT IS WORKING WELL. SO YOU NEED -- THAT'S ANOTHER WAY WHERE YOU CAN -- IF YOU -- IF EACH STEP IS RIGHT, THEN YOU MIGHT THINK IT'S -- MAYBE STILL LEADS TO THE WRONG DECISION BUT AT LEAST JUSTIFY IN SOME SENSE. >> I WAS GOING TO ADD I THINK WHEN TALKING ABOUT SOFTWARE ENGINEERING PRACTICES, AGILE DEVELOPMENT IS SOMETHING THAT WE CAN TAKE LESSONS FROM WHEN TALKING ABOUT THIS. SO BUILD A COMPONENT, TEST IT, LOOK HOW IT WORKS ON A PARTICULAR POPULATION AND THEN GO BACK AND REFINE AS NEEDED. I DO THINK THERE ARE LEGITIMATE LESSONS THERE WE CAN APPLY. >> I THINK THERE IS MULTIPLE PEOPLE FROM THE FDA IN THE ROOM AND I'M ONE OF THEM. I CAN SAY WHAT I'M THINKING SO IF THE GOAL IS TO REALLY LOOK AT THE ENTIRE PROCESS, MORE AGENCIES IN ADDITION TO FDA NEED TO GET MORE BECAUSE FDA IS MOSTLY CONCERNED IN TERMS OF IF LOOKING AT DEVICES FOR EXAMPLE, THE SAFETY AND EFFECTIVENESS OF A DEVICE, BUT THEN HOW IT IS BEING ACTUALLY USED BY MEDICAL PROFESSIONALS IN THE FIELD. HOW, WHETHER YOU REALLY FOLLOW IT TO THE END POINT USING ALL THE OTHER VARIABLES THAT THE FDA HAS LITTLE CONTROL WHEN DEVICE IS LET TO THE MARKET IS A BIGGER PROBLEM. BUT IT'S A VERY IMPORTANT PROBLEM WE SHOULD BE LOOKING AT. FOR EXAMPLE, RECENTLY IN THE FDA WHITE PAPER WE SUGGESTED THAT THE CONTINUOUS MONITORING OF THE PERFORMANCE OF THESE DEVICES SHOULD BE ONE OF THE COMPONENTS ESPECIALLY IF THE DEVICE IS CONTINUOUSLY UP THE DATING. I JUST WANTED TO MAKE ANOTHER COMMENT ABOUT DATA AND PRIVACY ISSUES. THEY ARE VERY IMPORTANT. BUT I JUST WANT TO SAY THAT NOT DOING ANYTHING IS ALSO ETHICAL CHOICE. SO IF YOU ARE -- IF THE DATA IS AVAILABLE BUT SOMEHOW SO DIFFICULT TO GET INTO THAT POTENTIAL DEVICES USING THE DATA ARE NOT DEVELOPED, THAT HAS CAUSED A STIR SO WE HAVE TO BALANCE BOTH PRIVACY AND THAT COURSE IN MY OPINION. >> QUICK PLUG FOR THAT. I'M VERY MUCH IN FAVOR OF OPENLY AVAILABLE DATA. HAVING LIKE TESTIMONY DATA SAID WHICH IS WIDELY USED MACHINE LEARNING FOR HEALTH COMMUNITY. >> ME TOO. THE JUST TO MAKE CLEAR. I THINK THAT WE ALSO NEED A DIFFERENT -- THIS REQUIRES A BIGGER CONVERSATION BUT WE NEED A BETTER HEALTHCARE SYSTEM LIKE WHEN PEOPLE ARE WORRIED ABOUT DISCRIMINATED BECAUSE THEY SHARE CERTAIN TYPES OF DATA YOU'RE NOT GOING TO GET FREE FLOWING DATA. SO IF WE HAD SOME OTHER TYPES OF SYSTEM, NOR NATIONALLYIZED WHERE PEOPLE ARE GUARANTEED THEY GET HEALTHCARE NO MATTER WHAT, THEN MAYBE YOU'LL HAVE THAT KIND OF SYSTEM T. >> THERE'S ANOTHER WORD MISSING FROM THE CONVERSATION, I THINK THROUGHOUT THE WHOLE DAY WHICH IS LIABILITY. THE U.S. PRIDES ITSELF BEING A LIT GUS SOCIETY. AND SOME THING IS NOT --LITY YOUS SOCIETY. I DON'T THINK WE HAVE ANY FRAMEWORK HOW LITIGATION AND LIABILITY WOULD WORK. WHEN ALGORITHMIC AGENT IS IN THE MIX. PRESENCE FOR USE OF DEVICES, PRECEDENT FOR USE OF STATIC MODELS, BUT NO PRECEDENTS FOR LIVE MODELS CONTINUOUSLY UPDATING AS FAR AS I KNOW. THAT'S JUST ONE THING THAT I LOVE THE PANEL'S COMMENTS ON. SECOND, ALL THE CONVERSATION ALSO PRE-SUPPOSES THERE ARE DATA TO LEARN FROM. AS A SOCIETY WE DON'T TRUST OR HEALTH SYSTEM WITH OUR DATA, WE DON'T TRUST SURER WITH THE DATA, WE DON'T TRUST OUR GOVERNMENT WITH DATA BUT EVERYONE WANTS TO BENEFIT FROM THE LEARNING HEALTH SYSTEM. SO THAT IS A -- BEYOND THIS ROOM BEYOND NIH PROBLEM BUT AS A SOCIETY AS WE TRUST NO ONE WITH THEIR DATA IT'S HARD TO LEARN A MODEL FROM DATA. OUR OTHER EXPEED YENS THAT MOVE US AS A -- EXPERIENCES THAT MOVE AS A SOCIETY TO GET TO THE LEVEL OF TRUST. WHERE DATA ARE AVAILABLE TO TRAIN AND LEARN TRILLION. I LOVE MIMIC BUT I THINK IT'S BEEN A GREAT DISSERVICE TO THE ML COMMUNITY, IT'S FUNNELED EVERYBODY INTO THINKING HEALTHCARE IS ICU AND NOTHING ELSE. >> BECAUSE WE HAVE FOUR MINUTES LEFT. HOW DO YOU FEEL ABOUT THREE QUESTIONS IN ONE? IT'S 4:30. GO AHEAD. >> BECAUSE THIS -- >> NO, NO, NO. WE'RE DOING ALL IN ONE GO. MORE TIME TO PREPARE AND CONSIDER TO ANSWER YOUR QUESTIONS. BECAUSE THIS SESSION IS FAIRNESS, I WANTED TO BRING UP THE POINT ABOUT FAIRNESS. THROUGHOUT THIS DAY WE HAVE DRAWN IMPLICIT ANALOGY WITH DRUGS. WE HAVE TALKED ABOUT HOW DRUGS THAT ARE COMMONLY USED TODAY, MAY WORK BUT BECAUSE THEY HAVE HISTORY BEHIND THEM, THEY MAY NOT HAVE SORT OF THE MECHANISTIC PROCESSES LIKE FLESHED OUT. WHAT MY QUESTION CONCERNS IS THAT HOW DO WE PROTECT ML FROM BECOMING A REALLY SPECIALIZED TECHNOLOGY THAT'S ONLY AVAILABLE TO THOSE WHO HAVE THE MEANS TO AFFORD IT. BECAUSE IF WE THINK ABOUT IN HEALTHCARE, ONE OF THE ASPECTS THAT IS BROKEN, IN THE DRUG REALM, THERE ARE A LOT OF STUDIES ON EXPENSIVE AND NICHE DRUGS, BECAUSE THEY ARE SPONSORED BY COMPANIES AND PEOPLE WHO HAVE AN INTEREST IN THEM. FOR CONSTIPATION, PRUNE JUICE AN LAXATIVES WORK BUT THERE'S NICHE EXPENSIVE DRUGS THIS WORK ON CHLORIDE CHANNELS SO DO WE REALLY NEED ALL THOSE DRUGS? THAT'S AN OPEN QUESTIONS BUT THERE ARE OTHER AREAS OF HEALTHCARE WHERE THERE ARE OTHER PATIENTS WHO BENEFIT FROM INVESTING THOSE RESOURCES, MONETARY RESOURCE OVER THAT TIME. UNFORTUNATELY THERE ARE MARKET FORCES AT PLAY AND LIKE THE DRUG REALM SO IF YOU DRAW ANALOGY TO MACHINE INTELLIGENCE, HOW DO WE PROTECT I GUESS PATIENTS AND PUBLIC FROM JUST ML BEING DRIVEN BY MARKET FORCES IN HEALTHCARE AND PROPAGATING ON FAIRNESS RATHER THAN ALLEVIATE IT. >> LAST BUT NOT LEAST ST. CXFC >> NO PRESSURE. YOU HAVE TO ANSWER THREE COMPLETELY DIFFERENT QUESTIONS IN ONE GO AROUND. I HAVE BEEN CHEWING ON THE FIGURE WE HAVE AN MH ALGORITHM FOR PREDICTING ML MORTALITY AND PREDICTS DIFFERENTLY WITH INSURANCE STATUS BASED ON RACE. I WORK IN MACHINE LEARNING AND FAIRNESS LITTLE BIT MYSELF AND I ALWAYS THINK ABOUT WHAT CAN WE DO ALGORITHMICALLY, WE SOLVE THE WRONG MATH PROBLEM, BACK TO DRAWING BOARD. AND TWEAK OUR OPTIMIZATION CONSTRAINT THIS WAY OR THAT WAY. BUT NOW THAT I'M FORCED INTO A ROOM THAT HAS WINDOWS WITH SHUTTERS STILL, LITTLE LIGHT IS GETTING TO MY BRAIN AND I'M STARTING TO THINK OUTSIDE THE BOX. SO JUST THINKING ABOUT THE CONTEXT OF COLORECTAL CANCER SCREEN, WE HAVE SCREEN RECOMMENDATIONS AND WE FOUND OUT THAT WE'RE DOING A BAD JOB TREATING AND DIAGNOSING COLORECTAL CANCER IN AFRICAN AMERICANS BUT THE SOLUTION IN THAT LINE OF RESEARCH HASN'T BEEN LET'S GO BACK AND TWEAK THE SCREENING RECOMMENDATIONS NECESSARILY. IT'S BEEN LET'S DO RESEARCH IN HEALTH DISPARITIES TO IDENTIFY WHAT ARE THE ROOT CAUSES. SO I DON'T REALLY KNOW, ARE THERE PEOPLE OUT THERE TREATING THESE INEQUALITY PROBLEMS IN MACHINE LEARNING AS HEALTH DISPARITY RESEARCH? WHY ARE WE PREDICTING WORSE AND PRIVATELY PUBLICLY INSURED VERSUS PRIVATELY ENSURED CAPTURING VERSUS TREATING MORE ALGORITHMIC PERSPECTIVE. >> DIFFERENT QUESTIONS BUT IN THE ESSENCE OF FAIRNESS WE HAVEN'T GOT A CHANCE TO ASK THE QUESTION SO WE HAVE MINUS 30 SECONDS LEFT. THIS TABLE, THE THREE QUESTIONS AROUND -- >> GET TO PICK. >> LIABILITY,, MACHINE LEARNING AND UNDER-REPRESENTED UNDERSERVED GROUPS WHO MAYBE DON'T HAVE ACCESS AND THIRD HEALTH DISPARITY ONE PROBLEM, THAT COULD BE SUM RIDESSED. >> I'LL TAKE A LIGHT ANY ROD LIABILITY QUESTION. SO I GUESS MY THOUGHT ON THAT ONE IS THAT AGAIN, THERE'S A LOT OF DATA YOU MADE A REALLY IMBUED POINT, WHAT ABOUT THE DATA WE ARE NOT USING? SO THERE'S A LOT OF DATA ALREADY THAT EXISTING, THINK ABOUT IN A HEALTHCARE ENVIRONMENT. THINK ABOUT HOW MANY DATA EXISTS IN AN ICU ROOM THAT ISN'T BEING UTILIZED EFFECTIVELY. THERE'S BEEN WORK TO SHOW THAT IF WE COMBINE THAT DATA AND PROVIDE INTEGRATED VISUALIZATION AND DISPLAYS TO CLINICS DO A BETTER JOB UNDERSTANDING WHO SAT RISK FOR CERTAIN HARMS AND BETTER INTERVENE. THAT'S THE SORT OF THING WHERE AGAIN, WE ARE TALKING ABOUT A REALLY WIDE SPECTRUM OF MACHINE INTELLIGENCE KINDS OF TOPICS TODAY. AND I THINK THERE'S A LOT OF LOW HANGING FRUIT. WE ARE APPLYING ALGORITHMS, WE ARE APPLYING MACHINE LEARNING, IN ORDER TO GET BETTER INSIGHT AND PROVIDE BETTER SITUATIONAL AWARENESS IN A HEALTHCARE SET BUG THAT'S VERY DIFFERENT FROM THE KIND OF THINGS WE ARE TALKING ABOUT ON THE FOO DEEP NEURAL NETS AND THINGS WITH BLACK BOX, DON'T UNDERSTAND THE DECISION MAKING. FROM A RELIABILITY PERSPECTIVE MY QUESTION IS WHAT IS OUR LIABILITY FOR NOT USING THE DATA THAT'S AVAILABLE WE SHOULD HAVE USED IN ORDER TO MAKE BETTER DECISIONS. THERE ARE WAYS THAT WE CAN APPLY MACHINE LEARNING TO DO THAT TODAY. WHITE WE CONTINUE THE ADDRESS ETHICAL ISSUES AND TRANSPARENCY ISSUES WITH MORE ADVANCED MACHINE LEARNING TECHNIQUES. >> I HATE CALLING YOU BLUE SHIRT. DO YOU HAVE NAME DAVID. HI, DAVID. SO I THINK THAT THE FAIRNESS COMMUNITY COMMUNITY AS OF THIS YEAR, HONESTLY IT'S BECAUSE FAT STAR CONFERENCE AND THE -- ALL MACHINE LEARNING CONFERENCES HAVE REALLY BAD NAMES. ALL OF THEM. BUT ONE IS FAIRNESS AND TRANSPARENCY CONFERENCE AND THEY AMALGAMATED THINGS AND STAR WAS USED TO, IT'S CALLED FAT STAR, YOU'LL GET THE WRONG THING DON'T GOOGLE BUT WHEN YOU DO, FIND THE ACTUAL CONFERENCE WEBSITE, IT'S A GROUP OF MOSTLY MACHINE LEARNING AND MACHINE LEARNING ADJACENT PEOPLE SO STATISTICIANS, EPIDEMIOLOGISTS, PUBLIC HEALTH PEOPLE WHO DECIDED TO ACTUALLY EMBRACE THIS IDEA IT'S NOT JUST THE MATH, YOU NEED TO FIX THINGS. THERE ARE LIKE 700 DEFINITIONS OF FAIRNESS, CYNTHIA EDWARD'S PAPERS ARE FANTASTIC, LOVE THEM ALL. BUT THOSE ARE NOT GOING TO FIX THE REAL PROBLEM WHICH IS SOMETHING -- THERE ARE MILLION EXAMPLES OF THAT, THE COLORECTAL CANCER SCREENING. I THINK PEOPLE RIGHT CYNTHIA OR MAURICE HART OR JAMIE, THERE ARE COMMUNITIES OF PEOPLE, SPECIFICALLY I THINK WHO FORM FAT STAR AND WORK IN THIS RESEARCH THAT ARE THINKING ABOUT HOW DO YOU UP LEVEL RESEARCH ON IT'S NOT JUST THE MODEL, YOU ARE IDENTIFYING SOMETHING THAT CAN BE FIXED. I THINK WITH A LITTLE BIT OF SALTYNESS, THE PEOPLE IN THE PUBLIC HEALTH RESEARCH COMMUNITY WILL TELL YOU THAT THIS IS WHAT THEY HAVE BEEN DOING FOR YEARS. SO THEY HAVE BEEN TRYING TO NOT LOOK AT THESE FANCY MODELS TO THESE INDIVIDUAL RISK LEVEL CALCULATIONS. AT A PERSON LEVEL. THEY ARE LOOKING AT SOCIETAL RISKS. WE ARE COMING AROUND. >> I WILL BE QUICK AND SAY A BUNCH OF THINGS. BUT SO FIRST IN PUBLIC HEALTH THERE'S A DISTINCTION BETWEEN HEALTH AND SOCIAL DETERMINANTS OF HEALTH. SO A LOT OF PEOPLE ARE REALIZING THAT LOOK WHEN YOU LOOK AT -- YOU CAN'T LOOK AT HEALTH ITSELF, THE SOCIAL INEQUALITIES AND THE POVERTY, ET CETERA, ET CETERA. AND SOMETIMES THOSE FACTOR -- AND EVEN THE POLITICAL SITUATION SOMETIMES THOSE FACTORS AFFECT HOW EVEN MORE THAN SO TAKING THAT INTO ACCOUNT WILL BE REALLY USEFUL. REALLY IMPORTANT AND USEFUL. AND THAT ADDRESSES SOME OF THE ISSUES ABOUT INEQUALITY. JUST A QUICK COMMENT ABOUT THE DATA THAT IF WE ARE NOT USING THE DATA SOMEHOW VIOLATING PEOPLE'S RIGHTS. OR DOING HARM OR SOMETHING LIKE THAT. I THINK WE HAVE TO BE THAT'S A CONSEQUENTIALIST TYPE THINKING AND I WORRY ABOUT THAT. AND HERE IS JUST AN EXAMPLE SO SAY DINA PERFECTS HER TECHNOLOGY, SO NOW WE CAN ACTUALLY PUT THAT EVERYWHERE. AND WE CAN TRACK YOU IN ALL SORTS OF WAYS AND MAKE THE ARGUMENT, IF WE DIDN'T PUT THIS EVERYWHERE, THERE'S SOMEBODY WILL DIE. SOMEBODY WILL SUFFER SOME ILL HEALTH OUTCOME BECAUSE WE DON'T PUT THIS EVERYWHERE. THEREFORE WE HAVE AN OBLIGATION TO PUT THIS EVERYWHERE. SO THAT'S A SLIPPERY SLOPE ARGUMENT. THERE ARE OTHER WAYS TO GET TO THE SAME CONCLUSION, SOMETHING TO THINK ABOUT. I THINK PEOPLE DRAW DISTINCTION IN ETHICS BETWEEN ACT AND OMIT. WHETHER HARM CAUSED VERSUS THINGS WE OMIT AND WE THINK WHEN WE DO THINGS WHEN WE ACT, WHEN WE DO SOMETHING THAT'S -- THAT WOULD BE WORSE THAN IF WE OMIT. IN CASE OF OMISSION. >> JUST RATTLING THROUGH QUICKLY. ON THE MARKET FORCES QUESTION, THERE'S LOTS OF GOOD EXAMPLES I CAN GIVE IN UK BUT ONE THING IS THE WELCOME TRUST ANNOUNCE AD FUND LOOKING AT SPORTING COLLECTING DATA AND DIVERSE DATA SETS AND INDIVIDUALS FOCUSING ON SUB-SAHARAN AFRICA AND INDIA AND THE WHOLE THING IS SLIGHTLY DECOLONIZE THE HEALTH SECTOR INTERESTING APPROACH AND WORTH BEARING IN MIND COUNTRIES LIKEs STONEIA ARE ATTRACTIVE PLACE TO WORK BECAUSE THEY HAVE NO PRE-EXISTING INFRASTRUCTURE. SO THAT WHILE THERE IS A PERIOD OF WHERE MARKETS ARE NOT ATTRACTIVE IT'S WORTH BEARING IN MIND ESTONIA HAS BECOME A PLACE TO DO WORK BECAUSE OF PRE-EXISTING INFRASTRUCTURE. ON THE HEALTH DISPARITIES POINT, A VERY DEPRESSINGLY, I USE NATIONALLY REPRESENTATIVE DATA SETS IN THE UK AND I JUST STANDARD DEMOGRAPHIC FACTORS AND SOCIAL GROUP STATUS IT MAKES PREDICT THINGS JUST AS MUCH AS IF I HAD -- TO PRE-PREVIOUS POINTS, ONE IN ONE PUBLIC HEALTH, UNFORTUNATELY. UNFORTUNATELY IT'S READY RISK FACTORS. JUST LAST ON THE LIABILITY POINTS, AGAIN SOMETHING HAPPENED IN THE UK IS LIABILITY IS A BIG ISSUE FOR PEOPLE, BECAUSE THE DEGREE TO WHICH CLAIMS BY SPECIFIC HEALTH TECHNOLOGY ENERGY STARTUPS ARE MASSIVELY STRETCHED AND WHAT THEIR ROLE IS WITHIN PATHWAY IN EXAMPLES THIS IS PRIMARY CARE SERVICE AND BASICALLY SAYING WE WILL DIAGNOSE YOU AND REPLACE GP AND WHEN THAT HE TURN AROUND AND SAY IT'S ALL ON YOUR FOLDER. SO WHAT THE SUMMARY HAS BEEN IS A HUGE AMOUNT OF COMPLAINTS, TIDING STANDARDIZATION MASSIVELY REDUCE NUMBER OF THOSE CLAIMS MADE BY COMPANIES, SOME THINGS LIKE (INAUDIBLE) BECAUSE WE CAN'T DO DIRECT TO CONSUMER TIDESSENING THE UK SO THIS IS A REALLY INTERESTING NEW SPACE WHERE THESE COMPANIES ARE PUSHING THEIR CLAIMS. IT WAS MENTIONED BEFORE BUT THIS WILL REMAIN THE CASE FOR LONG TIME IN UK BUT LIABILITY ALWAYS FALLS ON HUMAN SHOULDERS AND BECAUSE MOST OF THE MEDICAL DEVICES ARE IN A SPECIFIC CLASS WHICH MEANS THEY ARE NOT DIAGNOSTIC,, INFORMATIVE. MEANS RESPONSIBILITY MASSIVELY SETS CLINICIAN IN THE UK AND I CAN'T SEE THAT CHANGING ANY TIME SOON. SORRY, WE ARE EIGHT MINUTES OVER. BUT CAN WE HAVE A HUGE ROUND OF APPLAUSE TO OUR INCREDIBLE PANELISTS. [APPLAUSE] >> THANK YOU VERY MUCH. WE ARE GOING TO END THE DAY WITH A SHORT WRAP UP FROM EACH OF THESE SESSION CHAIRS. WHAT WE WILL DO IS IF YOU WANT TO TAKE YOUR SEATS BACK IN THE MAIN AREA, I WILL CALL EACH SESSION CHAIR AND GO THROUGH SHORTLY A FEW OF THE TAKE AWAYS FOR THE SESSION. SO THE FIRST SESSION CHAIR I WOULD LIKE TO CALL IS LUCA FOSCHINI, SESSION CHAIR FOR FIRST SESSION WHICH WAS -- SORRY. >> IT'S BEEN A LONG DAY. SO I HAVE PREPARED A SUMMARY OF THE TALKS AND TAKE AWAYS, I'LL SKIP SUMMARIES AND GO TO TAKE AWAYS. MAIN THINGS WERE ML COMPONENT, BLACK BOX IS JUST A SMALL CAR IN A BIGGER MACHINE AND WE NEED TO BROADEN EVALUATION PERSPECTIVE AND BECOME AGE TO END SYSTEM RESEARCH WHICH WE DON'T HAVE NOW AND TO STEAL A QUOTE FROM A DIFFERENT PANEL THAT HAPPENED LATER, HEALTHCARE IS A COMPLEX SOCIAL TECHNICAL SYSTEM OF SYSTEMS. THAT'S A SYSTEMIC VIEW OF THEM. BUT LIKE LOOKING AT THE PARTS THAT WE CAN ONLY GET NECESSARY CONDITION FOR NON-SUFFICIENCY. WE NEED TO BETTER INTEGRATE MONITORING OF SOLUTION OVER TIME FOR INSTANCE TOLL COMPENSATE FOR THE LITTLE EVIDENCE THAT COMES FOR DRUG TRIALS. INCREASE POST MARKET MONITORING AND VALUATION IS TRENDING IN OTHER INDUSTRIES SO WE EXPECT TO SEE THAT IN MORE IN THIS DOMAIN AS WELL. WE TALK TRUST AND TIME TRUST ACCRUALS OVER TIME, THIS IS NAGEM'S QUOTE HOW WE DEAL WITH FEEDBACK LOOPS AS SYSTEMS DEPLOYED REAL WORLD AND MINE CHANGE OUTCOME THERE TO MAKE PREDICTION ABOUT AND THE CONCLUSION IS THERE HASN'T BEEN REALLY ANY GOOD END TO END QUANTIFICATION OF HOW MUCH AND OFTEN HUMANS ARE OVERRIDING MI MODEL DECISION. THAT'S AN OPEN PROBLEM. IT WOULD BE NICE TO SEE OF THE MI MODELS THAT ARE DEPLOY THE, HOW MANY ARE BEING PROVIDED IN HOW DOES THAT AFFECT OUTCOMES WE SHOULDN'T ASSUME THAT MI SYSTEMS TRANSFER ACROSS INSTITUTIONS. THAT'S ALMOST A FALLACY OF USING EXTERNAL VALIDITY AS SURROGATE FOR TRUST. ASSUME INTERNAL VALIDITY IN MY SYSTEM HOW IT WORKS FOR SPECIFIC PROVIDER FOR A SPECIFIC INSTITUTION, WORKS WELL OVER TIME. THAT'S A BETTER WAY TO TRUST, BUILD TRUST OF THE SYSTEM. FINALLY WE ARE CURRENTLY EXAMINE THE MI SYSTEM IN ISOLATION, AND WE NEED TO LOOK BEYOND THE MODEL TO ACTIONIBILITY UTILITY AND ABILITY TO REALIZE THEM WITH OPERATIONAL CONSTRAINTS. THIS IS ALSO FROM NAGEM. THANK YOU VERY MUCH FOR GIVING ME THE OPPORTUNITY TO BE HERE TODAY AND THIS FANTASTIC PANEL. [APPLAUSE] >> ALL RIGHT. NOW I'M GOING THE CALL OUR SESSION 2 EXPLAINABILITY I HAVE THE AGENDA IN FRONT OF ME SO I CAN READ. THE CHAIR FOR THIS SESSION WAS SHINJINI KUNDU. IF YOU WOULD LIKE TO COME UP AND MAKE YOUR REMARKS. >> THANK YOU, CARLY, THANK YOU, EVERYONE, FOR CONTRIBUTING. THE MAIN TAKE AWAYS FOR THIS, WE STARTED OFF BY FLESHING OUT AI EXPLAINABILITY. EXPLORING THE NEW ONES THERE. WE TALK ABOUT APPROACHES FOR EXPLAINABILITY THAT CURRENTLY EXIST, TECHNICAL APPROACHES AND APPROACHES THAT ARE CURRENTLY BEING EXPLORED AT THE FDA. WE LOOKED AT PARTICULARLY DIFFERENT WAYS TO EMBED EXPLAINABILITY, WHETHER AT THE END OF THE MODEL OR INSIDE OF THE MODEL ITSELF. WE ALSO TALKED A LITTLE PIT ABOUT EVALUATING DIFFERENT EXPLANATIONS. DIFFERENT LEVELS OF EXPLANATIONS. AND EXPLANATIONS PARTICULARLY TAYLOR TO THE CONTEXT MR. YOU ARE TALKING TO A PATIENT OR TO ANOTHER AI SYSTEM OR WHETHER YOU ARE TALKING TO SOMEONE WHO IS HIGHLY SKILLED. WHETHER YOU JUST WANT A MATHEMATICAL EXPLANATION. FROM THERE, WE DREW A DISTINCTION BETWEEN INTERPRETABILITY AND EXPLAINABILITY. AND THERE ARE A COUPLE OF EXAMPLES, SO WE EXPLORED SUICIDE RISK STRATIFICATION AS AN EXAMPLE, THE NEED FOR EXPLAINABILITY AND TOUCHED ON THIS ALMOST TRADE OFF BETWEEN COMPLEXITY OF LEARNING MODELS AS -- AND THE EXPLAINABILITY OF THE LEARNING MODEL,. AND THAT WE TALK ABOUT THAT BEING A CHALLENGE MOVING FORWARD. ALTHOUGH LATER IN THE PANEL DISCUSSION, THAT WAS RAISED AS A QUESTION ABOUT WHETHER THAT SHOULD NECESSARILY BE THE CASE. AND WE ALSO TALK ABOUT PRACTICAL ASPECT THAT PREDICTION DOESN'T NECESSARILY MEAN PREVENTION HOW DO WE MAKE THESE EXPLAINABLE MODELS ACTIONABLE DOWN THE LINE. THERE WERE COUPLE REPEATED THEMES, ONE WAS CONTEXT. SO EXPLAINABILITY IS NOT ENOUGH. IT NEEDS TO BE EXPLAINED IN THE CONTEXT OF THE PROBLEM. WE TALKED ABOUT THE SLIPPERY SLOPE WITH CAUSALITY,, EXPLAINABILITY NOT IMPLYING CAUSALITY. WE EXPLORED THE DISTINCTION BETWEEN EXPLAINABILITY AS A SURROGATE FOR TRUST IN MODELS VERSUS EXPLAINABILITY AS SOMETHING THAT ENABLES VALIDATION AND EFFECTIVE DEPLOYMENT OF NEW MI TECHNIQUES. FINALLY WE CLOSED OFF WITH EXPLORING HOW THE DIFFERENT FIELDS THANK YOU VERY MUCH TO EACH OTHER. WE HAVE MATHEMATICIANS, PHYSICIAN WHOSE ARE TRAINED WITH THIS, WHO ARE TRAINED IN THIS TRADITION OF EQUATIONS AND IMPROVING EQUATIONS AND DERIVING THINGS AND PROVING THINGS, RIGOROUSLY MATHEMATICALLY AND WE HAVE A TRADITION OF CLINICIANS WHO RELY ON THE CLINICAL EXPERIENCE. AND MORE PRACTICAL ASPECT. SO HOW TO RECK -- RECONCILE FIELD OF ADVANCEMENT, APPEALS TO BOTH SIDES, SO THAT SUMMARIZE WHAT IS THE EXPLAINABILITY IS ABOUT. [APPLAUSE] ALL RIGHT. OUR NEXT SESSION WAS SESSION 3 USABILITY AND DR. KEN MANDL WILL BE SUMMARIZING FOUR US. >> WITH BREVITY I TRIED TO PICK ONE TAKE AWAY FROM EACH SPEAKER. I WENT FIRST, I WILL GO FIRST HERE. I SAID WE WANT USABILITY TO BECOME EMERGENT PROPERTY OF THE HEALTHCARE INFORMATION ECONOMY ONE WAY TO DO THIS IS TO IMPLEMENT PARSIMONIOUS STANDARDS INCLUDING API SO WE CAN HAVE APP STORE FOR HEALTH SUBSTITUTE ABLE APPLICATIONS COMPETE WITH EACH OTHER. CHRIS MADE THE POWERFUL POINT THAT SUBSTANTIAL AMOUNT OF HARM THAT'S RELATED TO EHRs IS DIRECTLY ATTRIBUTABLE TO ISSUES OF USE IT WILL AND OUTLINED SEVERAL KEY PROGRAMS THAT AHRQ HAS TO ADDRESS THIS. ERICH MADE A VERY GOOD CASE FOR ELEVATING THE IMPORTANCE OF USABILITY BY WRAPPING ML AND VALUE BASED FRAMEWORKS WITH AN END TO END VALUE BASIS AND WORK FLOWS THAT MAKES SENSE. AND DINA SHIFTED THE PARADIGM TO SHOW THAT THE BEST WAY TO IMPROVE THE USABILITY OF WEARABLES IS TO ELIMINATE THE WEARABLES. AND IN FACT TO ELIMINATE THE USER ALL TOGETHER. [APPLAUSE] >> ALL RIGHT. THANK YOU SO MUCH. SO OUR FINAL PANEL TODAY WE JUST CONCLUDED WAS THE TRANSPARENCY AND FAIRNESS PANEL. AND I'M GOING TO INVITE MAGAZINE BACK UP HERE TO GIVE US SOME OF HER -- MAXINE TO GIVE CONCLUSIONS FOR THE PANEL. >> I'M SURE I HAVE FORGOTTEN EVERYTHING. I THOUGHT IT WAS GIVEN 20 MINUTES, BEEN REALLY INTENSE THREE MINUTES ACTUALLY. I'M GOING TO HOW DO I SYSTEM RIDES WHAT HAPPENED? -- SUMMARIZE HAPPENED? I'LL TAKE A BRIEFIUS APPROACH. I WILL TAKE FEW HIGHLIGHTS FROM EACH SPEAKER. FROM SUZANNE WE HAD INTERESTING USE CASE OF THE RISK AMPLIFYING HUMAN BIAS AND LOOKING AT THE FACT THAT WE CAN'T USE STANDARD TOOLS AND -- IN THE CASE OF LOOKING AT NLP IN PRE-TEXT OF JOINT MEDICAL RECORDS. REALLY GREAT TO SEE A LIVE EXAMPLE OF SHOWING THE DISTRIBUTION OF THE DATA IS HELPFUL ACTIONABLE THICK FOR CLINICIANS EVOLVED T. WITH MARZYEH WE HAD PUNS AND STRUCTURES WHAT MODELS ARE HEALTHY, WHAT BEHAVIORS ARE HEALTHY AND TRY TO LOOK AT THE PROBLEM FROM LOTS OF ANGLES AND PRESENTED THREE SOLUTIONS TO WHAT TRANSPARENCY ROOKED LIKE SO TRANSPARENCY BY POST HOC INTERPRETATION, NATURAL OUTPUTS AND AUDIT OF EMBODY DATA. BIG FOCUS TRANSPARENT PROCESSES ARE ARE KEY AND NOT MODEL, NO ONE CAN DESCRIBE NOT FAKE THAT IN PUBLIC WHEN YOU'RE TRYING TO PROVE INFORMATION. THE THIRD SPEAKER WAS MATTHEW, WHO TOOK TWO LENSES TO ETHICAL ISSUES THE VULNERABILITIES IN AI AND THE HUMAN VULNERABLES, AI WORKING TOO WELL. HE TOUCHED ON A COUPLE OF THINGS. LATER ON CAME TO BE QUITE CONTENTIOUS I BELIEVE. SO ONE OF THE THINGS IS ABOUT THE POST HOC ADDING ADDITIONAL LAYER OF EXPLANATION. MENT ON TO THESE BLACK BOXES, THIS HAPPENED AFTER THE EVENT IS THAT AS VALUABLE AS WE LIKE TO THINK. ALSO EXPLORE THE TENSION BETWEEN THE ACCURACY AND EXPLAINABILITY TRADE OFF. AND A LOT OF THIS BOILED DOWN TO SOME OF THE FACT THAT PANELS WE USE ARE ARBITRARY, WE LOOK AT ML TO OTHER HEALTHCARE, BUT ALSO YOU DON'T BELIEVE IN CAUSAL INFERENCE BETWEEN LEARNING SO I'M SURE PEOPLE WILL HAVE A DEBATE ABOUT THAT. ME. THEN I GAVE THREE EXAMPLES OF PRACTICE WE DO IN THE UK AFFECTING THE SUPPLY COMMUNITY SO EXAMPLE OF THE DEEP MIND REVIEW BOARD AND HOW TO BRING TOGETHER MORE DIVERSE REPRESENTATIVE AND INCLUSIVE WORK FORCE TO COME AROUND IN THE BEGINNING BIT OF THE HYPOTHESIS GENERATING AND THE SOLUTION GENERATING LOTS OF HEALTH SOLUTIONS AND LASTLY GIVING INSIGHT HOW NHS ACTS IN THE GOVERNMENT UK TRYING TO THINK TRANSPARENCY AND FAIRNESS OF SCALE AND REALLY INTERESTING QUESTIONS THAT BROUGHT ABOUT A FEW QUESTIONS AROUND LIABILITY, BIG QUESTIONS ABOUT CAPITALISM, MARKET FORCES AND BIG TECH, WHOSE RESPONSIBILITY IS IT TO DO THESE ETHICAL AUDITS. AND APPARENTLY THE FDA HAS SERVED ITS RESPONSIBILITIES. -- SHIRKED THAT RESPONSIBILITY. THAT WAS HEAVY. I DON'T -- SHARED RESPONSIBILITY. THEN DAVID, I LIKE YOUR POINT. THIS IS REALLY SOMETHING FOR ME QUITE -- WE CAN OVERAGONIZE ABOUT THE INTRICACIES IN MACHINE LEARNING BUT IT BOILS DOWN TO VERY WELL NOPE AND UNDERSTOOD PUBLIC HEALTH -- WELL KNOWN ISSUES AROUND SOCIAL DETERMINANTS OF HEALTH AND THE FACT IT HAS ML OVERLAY MEANS WE'RE EXCITED TALKING ABOUT IT BE THE PROBLEMS ARE THE SAME. WE LOOK DOWN THE BARREL HARD ISSUES SOCIAL AND HEALTH INEQUALITYIES. THANKS. [APPLAUSE] >> THANKS AGAIN TO ALL PANELISTS, WE APPRECIATE YOU PUTTING IN THE TIME TO COME HERE AN DISCUSS WITH US. THANK YOU SO MUCH TO EVERYONE WHO IS IN THE ROOM AND EVERYONE WHO CALL IN VIA VIDEOCAST. WE APPRECIATE FEEDBACK AND WE WILL BE INCORPORATING FEEDBACK INTO THE WHITE PAPER. WITH THAT, WE ARE OFFICIALLY ADJOURNED.