>> GOOD GOOD MORNING, EVERYONE. WE'RE GOING TO GET STARTED BECAUSE WE HAVE A TIGHT SCHEDULE, AND I KNOW THE HALL ISN'T FULL YET BUT IT WILL FILL IN DUE TIME. I'M CHRIS KANDARPA FROM THE NIBIB. WELCOME TO THE NIH, AND THE WORKSHOP ON ARTIFICIAL INTELLIGENCE APPLICATIONS IN MEDICAL IMAGING. I'LL START BY THANKING THE SPONSORING INSTITUTES, THE NIH INSTITUTES THAT CO-SPONSORS WITH NIBIB, ARSN, RSNA AND ACADRAD, THE GOAL OF IMPROVING HUMAN HEALTH. THIS GOAL IS SHARED BY THE ENTIRE NIH OF COURSE, AND WITH ALL OF YOU HERE WHO ARE WORKING ON IMPROVING PATIENT CARE AND OUTCOMES BY APPLYING A.I. TO THE MEDICAL IMAGING CHAIN. YOU'RE THE EXPERTS HERE TO SHARE YOUR LATEST KNOWLEDGE AND INSIGHTS TO EXCHANGE NOTES, AND TO HOPEFULLY FORM FUTURE PARTNERSHIPS THAT ACCELERATE ADVANCES IN THIS ARENA. NOW, I'D LIKE YOU TO INDULGE ME WHILE I MAKE A FEW OVERARCHING BUT BY NO MEANS ALL INCLUSIVE OBSERVATIONS ABOUT THE PROMISES, PERILS, PRECAUTIONS AND HOPES OF A.I. IN MEDICAL IMAGING. HERE WE RECOGNIZE THAT A.I. TECHNOLOGIES ARE APPLICABLE THROUGHOUT THE MEDICAL IMAGING CHAIN. OUR FOCUS HERE MAY BE BIASED TOWARDS IMAGING ANALYSIS AND INTERPRETATION, THE OTHER CONTRIBUTIONS THAT A.I. MAKES IN THE DOWNSTREAM AND UPSTREAM INFLUENCES IN THE CHAIN WILL ALSO BE HIGHLIGHTED. WE PROBABLY ALL AGREE TODAY'S MEDICAL IMAGERS ARE NO IMMINENT DANGER OF BEING REPLACED, LIFE MAY CHANGE NOR THE BETTER IN MANY WAYS EVEN IF FOR UNFORESEEN REASONS AND BENEFITS. YES, THERE ARE PROMISES, BUT THERE ARE PERILS TOO. THE DEPARTMENT OF DEFENSE UNDERSTANDABLY INTERESTED IN ADVANCES IN A.I. TECHNOLOGIES WAS ONE OF THE SPONSORS OF THE 2018 JSON STUDIES. IN THE EXECUTIVE SUMMARY, IT'S A PRECAUTIONARY DOUBLE ENTENDRE, DATA IS WHERE TRUTH LIES. MANY ARE FAMILY WITH APPROACHES, WE SEE A DEEP NEURAL NETWORK RECOGNIZING WITH NEARLY 60% CONFIDENCE WHAT WE HUMANS SEE AS A PANDA. WITH THE SLIGHTEST INTENTIONAL INTRODUCTION OF WHAT WE SEE AS NOISE AND DNNCs AS A NEMATODE WITH LOW CONFIDENCES DNN NOW SEES OUR ORIGINAL PANDA AS A GIBBON WITH NEARLY 100% CONFIDENCE. THIS IS NOT ACCEPTABLE TO THE DoD NOR IS IT ACCEPTABLE TO MEDICAL IMAGING. SO THE CAUTION IS THAT A.I. IF NOT APPLIED CAREFULLY AND METICULOUSLY IN THE PRESENT STATE IT IS AS LIKELY TO DOOM US AS HELP US. IN FACT, THE JASON REPORT CONCLUDES MANY APPLICATIONS RELEVANT TO DoD ARE NOT READY FOR PRIME TIME AND MUST AWAIT FURTHER DEVELOPMENT. SIMILARLY, THE MOST IMPORTANT ADVANCES WE WILL SEE IN MEDICAL IMAGING ARE NOT YET READY FOR PRIME TIME EITHER BUT THEN I REMIND YOU THAT A MAJOR PURPOSE OF THIS WORKSHOP IS TO GET US THERE FASTER. WE HAVE UNCHARTED TERRITORIES BUT TO REPHRASE PRESIDENT KENNEDY, WE CLOSE TO GO TO THE MOON NOT BECAUSE IT WAS EASY BUT BECAUSE IT WAS HARD. I WISH I COULD DO HIS BOSTON ACCENT, BUT I CAN'T. THE REALITY IS, GOOD NEWS FORTUNATELY, COMPUTERS ALREADY HELP US WITH MANY COMPLEX DIAGNOSTIC TESTS. BUT THESE MACHINES WILL HELP US EVEN MORE IF WE MAKE THEM INTO SMARTER PARTNERS. FUTURISTS SAY AS MECHANICAL MACHINES OF THE INDUSTRIAL AGE HELPED US ACCOMPLISH PHYSICAL FEATS THAT HUMAN STRENGTH ALONE COULD NOT ACHIEVE, OUR THINKING MACHINES OF TOMORROW SIMILARLY PROMISE UNIMAGINABLE MENTAL STRENGTH UPON US. BUT BEFORE WE GET AHEAD OF OURSELVES AND REJOICE IN THE COMING MEDICAL IMAGING NIRVANA, THERE ARE MANY PRACTICAL REAL WORLD CHALLENGES THAT MUST BE OVERCOME. SOME OF THEM ARE MORE URGENT, MAKE OR BREAK BARRIERS. THE PRESENT STATE OF SOLID DATABASE AND KNOWLEDGE BASE HAS TO IMPROVE, THE FUTURE IS WHERE THEY ARE FUNCTIONALLY INTEGRATED TOOLS UNIVERSALLY USEFUL AND THE THREE CO-EVOLVE TOGETHER. ON THE LEFT ARE DEPARTMENT OF DEFENSE ILITIES RELEVANT TO THE WORK, ON THE RIGHT ILITIES RELEVANT TO MEDICAL IMAGES IN THE CLINIC. JASON REPORTS CONCLUDES DEEP LEARNING IS WEAK ON THE DoD ILITIES. WE'LL HEAR ABOUT THEM IN THE NEXT TWO DAYS. INTEROPERABILITY, OR LACK THEREOF, AND THEN I'LL STOP. ONE OF THE INTERPRETATIONS OF THE MYTH OF THE TOWER OF BABEL IT EXPLAINS THE EXISTENCE OF DIVERSE LANGUAGES. I INTERPRET DIFFERENTLY, A FOREWARNING ABOUT MODERN COMPUTERS SPECIFICALLY OPERATING SYSTEM LANGUAGES AND RESULTING LESS THAN SATISFACTORY INTEROPERABILITY. THIS LACK APPLIES TO THE MANY IN THIS SPACE WHO ARE NOT PROACTIVELY SPEAKING WITH EACH OTHER RIGHT NOW. IN CLOSING, IF WE CAN MAKE THE SLIGHTEST HEADWAY IN CORRECTING THIS COMMUNICATIONS GAP, I WOULD CONSIDER, FOR ONE, THIS WORKSHOP TO HAVE BEEN A MODEST SUCCESS. THANK YOU VERY MUCH, AND I'D LIKE TO THANK ALSO THE SESSION CHAIRS WHO ARE LISTED HERE AND THE SPEAKERS. I WON'T GO THROUGH THEIR NAMES BUT THEY WILL BE INTRODUCED AS THEY COME UP. AND A SPECIAL THANKS TO SHIRLEY CONEY-JOHNSON, WITHOUT HER NONE OF THIS WOULD HAPPEN. A FEW ANNOUNCEMENTS BEFORE THE NEXT INTRODUCTION, THE SESSION CHAIRS, PLEASE MAKE SURE THAT YOUR SESSION GROUP ASSEMBLES ON THE PODIUM AT LEAST FIVE MINUTES BEFORE THE BEGINNING. THIS IS DURING THE BREAKS. ON FRIDAY THE BREAKOUT SESSIONS WILL MEET AT THE REGISTRATION DESK OUTSIDE BY 10:10:00 A.M. AND WILL BE ESCORTED TO THEIR RESPECTIVE ROOMS. THE GROUPS WILL REASSEMBLE IN THIS THEATER TO PRESENT THE REPORTS AT 11:55. WE REMIND YOU AGAIN. COFFEE BREAKS AND THE RECEPTION THIS EVENING ARE AT THE NIH TERRACE. IF YOU NEED A CAB THURSDAY OR FRIDAY LET THE REGISTRATION DESK KNOW. ALSO, THE SPEAKERS MUST GET WIRED THERE BEFORE THEY COME UP HERE. YOU CAN ONLY SPEAK IF YOU'RE ON THE PANEL. SO, NEXT I'D LIKE TO INTRODUCE DOCTORS CURT LANGLOTZ, PROFESSOR AT STANFORD, AND ALSO FROM THE RSNA. CURT? >> THANK YOU, KRIS. GOOD MORNING. PLEASED TO SEE THE HIGH INTEREST AND WE GOT NICE WEATHER FROM THE BIG A.I. IN THE SKY, LOOKING FORWARD TO A GREAT COUPLE DAYS. AS KRIS SAID ON BEHALF -- I'M HERE ON BEHALF OF RADIOLOGIC SOCIETY OF NORTH AMERICA. I SERVE ON THE BOARD THERE. AS THE INFORMATICS LIAISON. AND WE ARE VERY PLEASED TO BE ABLE TO SUPPORT THIS WORKSHOP. ONE OF THE PRIMARY ELEMENTS OF THE RSNA MISSION TO PROMOTE EXCELLENCE IN PATIENT CARE THROUGH EDUCATION, RESEARCH AND TECHNOLOGIC INNOVATION, SO YOU COULD SAY THIS WORKSHOP IS A TRIFECTA BECAUSE IT REALLY HITS ON ALL THREE OF THOSE POINTS. IT'S REALLY PART OF A BROADER A.I. STRATEGY THAT RSNA HAS, THE EDUCATION OPPORTUNITIES INCLUDE ONLINE WEBINARS, IN-PERSON SPOTLIGHT FOCUS COURSES ON A.I. AND EDUCATIONAL OPPORTUNITIES AT THE ANNUAL MEETING INCLUDING A COLLABORATION WITH THE GROUP AT NATIONAL CANCER INSTITUTE, CROWDS CURE CANCER, RADIOLOGISTISTS HAVE THE OPPORTUNITY TO LABEL IMAGES USED AS INPUT TO MACHINE LEARNING ALGORITHMS. AND THE RESEARCH MISSION OF COURSE RSNA FUNDS OVER $4 MILLION A YEAR IN SEED GRANTS AND SCHOLAR GRANTS THAT ARE OFTEN NOW INCREASINGLY HAVE AN A.I. FOCUS, NOT SURPRISINGLY, AND OFFERS AS WELL FOR THE RESEARCH COMMUNITY CHALLENGES AND PUBLIC DATASETS IN CONJUNCTION WITH THE MEETING. THIS YEAR WE HAVE A CHALLENGE OF RECOGNIZING NEW PNEUMONIA ON THE CHEST X-RAY, FITTING IN WITH THE A.I. RSNA STRATEGY. PLEASED TO SEE THE HIGH INTEREST TODAY. IT'S NOT SURPRISING GIVEN THE TOPIC, BUT WE'RE DELIGHTED TO HAVE A GREAT PROGRAM FOR YOU, AND LOOK FORWARD TO THE FOLKS IN THE ROOM AND FOLKS WHO ARE WATCHING AROUND THE WORLD ON VIDEO TO HEAR ABOUT THE LATEST IN A.I. RESEARCH AS IT PERTAINS TO MEDICAL IMAGING. SO I'LL ADD MY WELCOME. THANK YOU FOR BEING HERE. ON A PERSONAL NOTE I WANT TO SAY IT WAS A PLEASURE TO WORK WITH KRIS AND BIBB AND OTHERS IN PUTTING THE PROGRAM TOGETHER. SO I'D LIKE TO NOW INTRODUCE BIBB WHO WANTS TO MAKE A COUPLE COMMENTS. BIBB ALLEN IS THE CHIEF MEDICAL OFFICER OF THE DATA SCIENCE INSTITUTE AT THE AMERICAN COLLEGE OF RADIOLOGY. BIBB? >> AS BIBB IS COMING UP, I JUST WANT TO SAY HE WILL INTRODUCE THE NEXT TWO SPEAKERS, AT LEAST KEITH, AND WE HAVE DR. KALPATHY-CRAMER WHO IS NOT GOING TO SPEAK BUT WILL BE UP HERE. >> GOOD MORNING. THANK YOU. I WANT TO ADD MY WELCOME. I'M BIBB ALLEN, THE CHIEF MEDICAL OFFICER OF THE ACR'S DATA SCIENCE INSTITUTE. WE'RE CERTAINLY PLEASED ON BEHALF OF THE COLLEGE TO BE CO-SPONSOR FOR THIS WORKSHOP. I THINK ALL OF THE THINGS THAT CURT SAID REGARDING THE MISSIONS OF MEDICAL SPECIALTY SOCIETES AND ADVANCING THIS EXCITING FIELD ARE TRUE FROM THE PERSPECTIVE OF THE COLLEGE. I THINK ONE OF THE REASONS THE ACR DECIDED TO CREATE A DATA SCIENCE INSTITUTE WAS TO ACTUALLY INVOLVE RADIOLOGISTS IN MORE IN DATA SCIENCE, HOW WILL DATA SCIENCE APPLICATIONS AND ARTIFICIAL INTELLIGENCE ALGORITHMS BE USED AND INTEGRATED INTO OUR CLINICAL PRACTICES, WHAT ARE GOING TO BE THE -- YOU'RE GOING TO SEE, YOU KNOW, A HUGE AMOUNT OF SCIENCE. THE MISSION OF THE DATA SCIENCE INSTITUTE WAS NOT TO CREATE THE SCIENCE, BUT HOW TO GET THE SCIENCE IN CLINICAL PRACTICE. I THINK THAT THAT'S SORT OF ONE OF OUR MISSIONS SO THE LATTER PART OF THE PROGRAM IN FACT IS GOING TO BE AROUND CLINICAL IMPLEMENTATION INTEGRATION, INTEROPERABILITY, ALL OF THE THINGS THAT ARE GOING TO HAVE TO HAPPEN, SO RADIOLOGISTS WILL BE ABLE TO USE THIS TECHNOLOGY IN OUR PRACTICES. AS YOU CAN SEE, I'M PRETTY -- ONE OF THE GRAY-HAIRED FOLKS, BUT A.I. IS THE FUTURE OF OUR SPECIALTY. I'M A GUY WHO HAS LIVED THROUGH THE DEVELOPMENT OF CT, MRI, PET, ALL OF THESE THINGS THAT I'VE HAD TO SORT OF LEARN ON MY OWN THROUGHOUT THE PRACTICE. AND YOU WOULD THINK, WELL, GOSH, THAT WAS THE GOLDEN AGE OF RADIOLOGY. THIS IS THE GREATEST THING THAT'S GOING TO HAPPEN TO OUR SPECIALTY. BUT I TELL EVERY RESIDENT, EVERY FELLOW, EVERY PERSON IN TRAINING, EVERY MEDICAL STUDENT IS THAT ARTIFICIAL INTELLIGENCE IS GOING TO ADD SO MUCH MORE TO HOW WE'RE ABLE TO PRACTICE, HOW WE'RE ABLE TO HELP OUR PATIENTS SOLVE SOME OF THE PROBLEMS THAT WE DIDN'T HAVE ANY IDEA THAT WE WOULD BE ABLE TO SOLVE WHEN I STARTED IN RADIOLOGY. IF I COULD DO IT ALL OVER AGAIN, I WOULD START RIGHT NOW. ON THAT NOTE, I WANT TO THANK THE NIH, THE ACR SPONSORING ACTUALLY A COUPLE TRAINEES NOW. WE HAVE JUSTIN TAYLOR WHO IS A RESIDENT FROM WALTER REED HOSPITAL THAT'S HERE IS GOING TO BE PART OF THE CONFERENCE TODAY AND WORKSHOPS TOMORROW AND JEFF RUDY IS A FELLOW AT THE HOSPITAL AT THE UNIVERSITY OF PENNSYLVANIA INFORMATICS, WHO IS ALSO GOING TO BE PART OF THE PROGRAM TODAY. THIS IS THEIR FUTURE. HOW THIS WORKS, HOW A.I. EVOLVES, IS THEIR FUTURE. HAVING THAT INPUT I THINK IS IMPORTANT TO THE CONFERENCE. SO, WITH THAT SAID, I THINK WE'RE ABOUT READY TO GET THE SHOW ON THE ROAD. OUR FIRST SPEAKER OF THE DAY IS KEITH DREYER. KEITH IS AT THE MASSACHUSETTS GENERAL HOSPITAL, IN BOSTON, PART OF HARVARD MEDICAL SCHOOL. HE IS ALSO THE CHIEF SCIENCE OFFICER OF THE ACR'S DATA SCIENCE INSTITUTE, SO WE'RE GLAD TO HAVE KEITH GIVE OUR FIRST KEYNOTE OF THE DAY. SO WELCOME, KEITH. >> THANK YOU, BIBB. I THANK KRIS AND THE TEAM FOR PUTTING TOGETHER SUCH A TIMELY AND COMPLICATED SUBJECT TO DISTILL IT DOWN INTO THINGS WE CAN ALL UNDERSTAND. I'M GOING TO START OFF, LET ME ASK THE QUESTION AND I'LL REPEAT THIS TO THE AUDIENCE THAT'S LISTENING OUTSIDE THE ROOM. HOW MANY FOLKS HAVE DIRECT EXPERIENCE WITH A.I. TOOLS IN THE LAST FIVE YEARS OR SO? TREMENDOUS. WOW, THAT'S GREAT. SO I'M GOING TO TAKE YOU THROUGH KIND OF A -- JUST FOR THE RECORD, IT WAS PROBABLY 70% OF THE PEOPLE RAISED THEIR HANDS. I'M GOING TO WALK YOU THROUGH KIND OF A BROAD-BASED VIEW OF THIS TO SEE WHERE WE'RE AT ACROSS THE INDUSTRY, EVEN MORE BROADLY THAN MEDICINE BUT WHERE IT'S AT WITH DATA SCIENCE. I'VE BEEN INVOLVED IN DATA SCIENCE SINCE THE '90s, SO I'VE SEEN A COUPLE SUMMERS AND A COUPLE WINTERS OF DATA SCIENCE. AND THE LAST PASS I THINK IS REALLY KIND OF AN AMALGAMATION OF ALL THE ACTIVITY THAT'S HAPPENED FROM 50 YEARS AGO, ALSO THE RECENT 5 OR 10 YEARS. ARTIFICIAL INTELLIGENCE, MACHINE LEARNING, DEEP LEARNING, PART OF DATA SCIENCE, I USE THE TERM WHEN WE EXTRACT HIGHER ABOUT THE USAGE OF VARIOUS THINGS WE CAN DO. WHEN WE LOOK AT THIS IT'S REALLY EVOLVED BECAUSE WE'RE ABLE TO CAPTURE MORE DATA IN THESE DAYS, SO DEEP LEARNING REPRESENTS THE FACT MORE LAYERS IN THE NEURAL NETS AND MORE DATA TO WORK WITH. THERE HAS BEEN A LOT OF HYPE IN HEALTH CARE AND OUTSIDE OF HEALTH CARE, SO THE REAL QUESTION IS FOR MANY OF THOSE OUT THERE IN INDUSTRY, RESEARCH, AND IN ACADEMICS ARE ASKING THE QUESTION HOW MUCH IS REAL AND HOW MUCH IS HYPE? AND HOPEFULLY I'LL TALK TO SOME OF THAT AND YOU'LL SEE SOME UNCOVER ITSELF, WHY SOME IS HYPE, WHY SOME IS REAL. IF YOU LOOK AT THE TECHNOLOGY ITSELF, APOLOGIZE FOR THE BASIC CONCEPTS FOR THESE THAT EXPERIENCED THIS, BUT IF YOU REALLY WANT TO SAY I WANT TO FIND A SOLUTION TO FIND MOUNTAINS INSIDE OF PICTURES, HOW WOULD YOU DO THAT? THE FIRST THING IS HAVE THE NOTION OF GROUND TRUE. THAT MEANS YOU YOU OR SOMEBODY DECIDES WHAT IS A MOUNTAIN AND WHAT IS NOT A MOUNTAIN IN THIS PICTURE. OTHERS IT'S OBVIOUS, OTHERS IT'S NOT. PILE OF SAND, PLATEAU, YOU HAVE TO DETERMINE THAT TO SET BIAS AND TRUTH AND FALSE ON THE GROUND TRUTHS OF THE NEURAL NETWORK YOU'LL BE TRAINING. IN ANY EVENT YOU TAKE THOSE TRAINED OBJECTS AND PUT THOSE THROUGH NEURAL NETS, YOU HELD BACK A VALIDATION SET AND CHANGE THE WEIGHTS AND MECHANISMS THROUGH BACK PROPAGATION, ET CETERA, GRADIENT DISSENT AND MOVE THE NEEDLE FROM 50-50 TO STATISTICAL NUMBER, 93.2% FOR THE CREATION OF A MOUNTAIN RECOGNIZER IN THIS CASE. AND SHOW A PICTURE NEVER SEEN BEFORE, IT FOLLOWS YOUR BIASES OR DEFINITIONS OF GROUND TRUTH TO GIVE A STATISTICAL PROBABILITY IF THAT IS WHAT YOU IDENTIFIED, IN THIS CASE A MOUNTAIN. NOW YOU HAVE TO APPLY THOSE INTO PRACTICAL USE. IF YOU DON'T, THINK THEY JUST WON'T GET USED. SAME IN CLINICAL PRACTICE, MORE HURDLES, FDA APPROVAL, REGULATORY PROCESS, CLINICAL INTEGRATION, VALIDATION, ET CETERA. IN THIS PARTICULAR CASE LET'S SAY WE TAKE THE MOUNTAIN RECOGNIZER AS APPLE HAS, PUT IT INSIDE OF THE IOS, SO NOW YOU CAN TYPE IN MOUNTAINS AND SEE ALL THE IMAGES INSIDE YOUR PHONE THAT HAVE DEFINITIONS OF WHAT IT SEES AS MOUNTAINS. SO, LET'S GO THROUGH. I GAVE THIS EXAMPLE BECAUSE IT CORRELATES TO MEDICAL IMAGING. FIRST THERE WAS NOTION OF A.I. CONCEPT, USE CASE IT'S IMPORTANT TO FIND MOUNTAINS, NEXT WAS THE DATA ENGINEERING COMPONENT WHICH IS WHERE YOU'RE TAKING MASSIVE AMOUNTS OF DATA, AND PRE-PROCESSING THOSE, LINING THEM UP, MAKING SURE PIXEL SIZE AND DETAILS ARE CORRECT AND ACTUALLY DEFINING THOSE AS MOUNTAINS APPLY DATA SCIENCE THROUGH VARIOUS MECHANISMS OF MACHINE LEARNING, A.I., NEURAL NETS, ET CETERA, WIND UP WITH A.I. ALGORITHM AND TAKE THAT AND PUT THAT INTO APPLICATIONS, CLINICAL APPLICATIONS OR OUTSIDE HEALTH CARE, NON-CLINICAL APPLICATIONS, DEPLOY THAT INTO THE WILD. SO, IF WE LOOK AT THE IMPACT THAT IT'S HAD, IF YOU LOOK AT THE EVOLUTION OF INTELLIGENCE OVER TIME WE'VE SLOWLY PROGRESSED AS A SPECIES BUT LOOK AT COMPUTATION OVER THE LAST 60 OR 70 YEARS THERE'S BEEN HIGH ADVANCE, IF YOU LOOK AND CORRELATE TO NEURONS, FOLLOW FORWARD THERE'S GOING TO BE A CONSIDERABLE GROWTH, MAYBE BEYOND OUR SPECIES. IN ANY EVENT WE HAVE TO FIGURE, KRIS TALKED ABOUT THIS, HOW TO COMBINE THIS HUMAN AND MACHINE INTELLIGENCE TOGETHER. BECAUSE THAT'S REALLY WHERE EVERYONE BELIEVES IS THE FUTURE. A.I. CLEARLY EXISTS IN VARIOUS PLACES, AND AS KIND OF AHEAD OF HEALTH CARE, NOT UNUSUAL FOR TECHNOLOGY TO MOVE FASTER IN NON-REGULATORY AREAS. A.I. IN YOUR CAR IS MOVING QUICKLY BUT FOCUSED ON PARTICULAR ASPECTS, STILL NO SELF DRIVING CAR BEING USED IN THE WILD. A.I. I THOUGHT THIS WAS INTERESTING, IN JUNE -- I'M SORRY, IN JANUARY A.I. IN YOUR OVEN CAME OUT WITH AN INTELLIGENT OVEN CALLED JUNE, CAN YOU LOOK TO SEE WHAT IS IN THERE, SET PARAMETERS FOR BAKING, KNOW IF YOU LIKE IT THIS WAY OR THAT WAY AND DRIVE OUT DIFFERENT RECIPES. I DON'T KNOW IF I WOULD USE ONE BUT SOME PEOPLE MAYBE WOULD. HERE IS ONE MAYBE I WOULD USE, A.I. IN YOUR SUITCASE. THIS WILL GUIDE INDIEGOGO, HOPS ON THE RAMPS, PUT OTHER DEVICES ON TO WALK AROUND WITH YOU. A.I. IN THE PHONE IS ONE OF THE FIRST THINGS WE'LL SEE MORE OF. IT'S INTERESTING BECAUSE WHEN YOU THINK ABOUT A.I. IN YOUR PHONE I DON'T THINK YOU THINK ABOUT IT. YOU THINK YOUR PHONE GOT SMARTER AND BETTER, PROBABLY WHAT WILL HAPPEN IN HEALTH CARE. I TYPE IN CHIHUAHUA, THERE COME THE IMAGES. I HAD ONE, SO IMAGES COME UP. THIS I THINK EXEMPLIFIES SOME REAL WORLD PROBLEMS WE'RE PROBABLY GOING TO FORESEE IN HEALTH CARE AS WELL, KRIS ALLUDED TO SOME OF THOSE IN HIS EXAMPLES. IN THIS PARTICULAR CASE WE'RE LOOKING AT CHIHUAHUA. WELL, THERE'S CLEARLY A CHIHUAHUA, BUT IT WAS TRAINED TO RECOGNIZE UNDER DIFFERENT FEATURES THAT US HUMANS HAVE DIFFERENT FEATURES FOR RECOGNIZING THEM. THERE'S A COMBINATION OF HUMANS AND MACHINES WORK WELL. I DON'T THINK THAT ANY OF US WOULD PROBABLY MISTAKE THESE BLUEBERRY MUFFINS FOR CHIHUAHUAS BUT THE COMPUTER SOFTWARE CLEARLY DOES. IF YOU LOOK, YOU CAN SEE WHERE I'M GOING, IF YOU LOOK AT KOMONDOR, THE COMPUTER THINKS OF THOSE AS MOPS. AND GOLDEN DOODLES APPARENTLY LOOK JUST LIKE FRIED CHICKEN TO A.I. HEALTH CARE OPPORTUNITIES HERE, THIS IS THE QUESTION I THINK WE HAVE TO ASK AS WE GO FROM CREATION TO IMPLEMENTATION, IS WHAT IS IT WE'RE GOING TO DO PARTICULARLY IN MEDICAL IMAGING, WHAT IS IT WE DO DO, HOW DO WE LOOK AT PARTICULAR ASPECTS OF A.I. TO INCORPORATE INTO WHAT IT IS WE DO TO IMPROVE OUR CARE. AND I GO BACK TO THE INFORMATION CYCLE OF RADIOLOGY, FIRST WE HAVE CLINICAL CARE, AND THERE'S A DECISION THAT'S MADE ORDERING OF SOME KIND OF DIAGNOSTIC PROCEDURE, AND THEN PROTOCOL, PRE-ACQUISITION STEPS, THE EXAM GETS ORDERED, ESSENTIALLY PERFORMED, IMAGING PERFORMED, EXTRACTED, NOW THERE'S DATA, INTERPRETATION PROCESS WHICH INFORMS THE CLINICAL CARE TEAM, BACK AGAIN. THAT'S THE CYCLE THAT WE WANT TO APPLY F WE LOOK AT THAT AND LOOK AT SOME OF THE PROGRESSES THAT HAVE BEEN MADE, DEVELOPMENT RESEARCH IN AREAS OF A.I., IT'S AROUND THESE FOUR SECTORS. INTERPRETATION PROCESSES, DETECTION SEGMENTATION, QUANTIFICATION, CLASSIFICATION ARE OBVIOUS ONES, STEP INTO THE IMAGE ACQUISITION ITSELF AND LOOK AT IMAGE ARTIFACT REDUCTION, FINDINGS OPTIMIZATION, DOSE, CONTRAST OPTIMIZATION. AND PROTOCOLS. THERE WE GO. PRE-ACQUISITION, PATIENT POSITIONS, BRINGING UP PRIORS TO MAKE SURE YOU CAN ALIGN CURRENT EXAM WITH PRIOR EXAMS, ET CETERA AND CLINICAL CARE, DECISION SUPPORT ITSELF OR POPULATION HEALTH MANAGEMENT, BEING ABLE TO USE IMAGING IN APPROPRIATE WAY TO MANAGE DIAGNOSTICS, ALONG WITH CLINICAL CARE. SO IF WE LOOK AT THESE TWO TOGETHER, TO TAKE THE RADIOLOGY INFORMATION CYCLE AND A.I. DEVELOPMENT CYCLE THAT'S THE REAL CHALLENGE, HOW DO WE MERGE THE TWO TOGETHER, DIFFERENT SCIENCE AND SUBJECT MATTER EXPERTS. CORE TEAMS THAT YOU NEED TO PULL TOGETHER. IF YOU HAVE THESE A.I. CONCEPTS YOU NEED TO BUILD OUT OF USE CASES, TAKE THESE FROM VARIOUS PLACES, PUT IN DATA ENGINEERING AND DATA SCIENCE APPLICATIONS AND DEPLOY BACK INTO CLINICAL CARE. YOU CAN ALSO DO IT FOR IMAGING, PRE-ACQUISITION AND CLINICAL CARE PROCESS ITSELF BUT YOU NEED SUBJECT MATTER EXPERTS TO OPINE OVER THE USE CASES, THINGS WE NEED TO DO FIRST, LOW-HANGING FRUIT, WHERE SHOULD WE FOCUS RESEARCH AND ATTENTION TO ADVANCE SCIENCES COMBINED. IF WE LOOK AT THIS COMPUTATIONAL GROWTH THAT'S HAPPENING OVER TIME AND SEE WHERE THE FUTURE IS GOING TO BE, IT PROBABLY WILL HEAD TOWARD GENERAL A.I. WHERE WE'RE HAVING ALMOST LIKE HUMAN TYPE OBJECTS BEING ABLE TO RECOGNIZE THINGS AND GENERALIZE AND LEARN AS INFANTS DO TODAY. BUT THAT'S FAR OFF INTO THE FUTURE. IN THE NEAR FUTURE, SOMEWHAT TODAY IS THIS NOTION OF NARROW A.I., FOCUSING THE CHALLENGE OF INTELLIGENCE ON SPECIFIC TASKS. AND WHEN WE LOOK AT THAT I LIKE AT THIS INTELLIGENCE SPACE INCREASING ACCURACY AND DIVERSITY, THE HUMAN SPECIES, WE SIT WITH GENERAL HUMAN INTELLIGENCE, WE'LL JUST GET SMARTER WITH TIME WITH ADDITION OF BIOLOGICAL HARDWARE AND SOFTWARE, WE'LL BE ABLE TO GET SMARTER AND SMARTER. BUT WE'RE GOING TO HAVE TO WAIT HUNDREDS OF THOUSANDS OF YEARS. SO WE FOUND OTHER WAYS TO GET SMARTER, THAT'S BY NARROWING THE HUMAN INTELLIGENCE INTO SPECIFIC TASKS LIKE MEDICINE, GOING TO MEDICAL SCHOOL, DEEPER INTO THAT, GOING INTO RADIOLOGY, RESIDENCY OR FELLOWSHIP, FURTHER I DO NOT THAT. IF YOU LOOK AT GENERAL ARTIFICIAL INTELLIGENCE, IT'S ESSENTIALLY THE SAME THING TO TRY AND GET IT TO GROW IN A BROAD SENSE, YOU REALLY INCREASE THE DIVERSITY BUT IT TAKES A LONG TIME TO IMPROVE THE ACCURACY. AND SO THERE ARE FOLKS THAT ARE CLEARLY WORKING ON THAT. A LOT OF WORK. BUT IT WILL HAVE A LONG TIME BEFORE IT ACTUALLY STARTS TO MOVE THE NEEDLE. SO ALSO YOU CAN USE A.I. TO FOCUS IN NARROW AREAS, AND YOU CAN SPECIFICALLY FOCUS IN THINGS BEING DONE BY HUMANS TO MAKE IT FASTER, MORE EFFICIENT, HIGHER, SMARTER, HIGHER ACCURACY, ET CETERA. THAT IMPROVES YOUR ACCURACY. YOU CAN ALSO WORK IN AREAS WHERE HUMANS AREN'T SUBSPECIALIZED YET AND SO YOU HAVE THIS NOTION OF SUPER HUMAN TASKS OF NARROW A.I. AND SO IT'S THIS AREA THAT REALLY SEE AS THIS INTELLIGENCE SPACE THAT WE NEED TO TRY AND FIND WAYS TO INCORPORATE TOGETHER. IF YOU THINK ABOUT THIS, WE'RE BASICALLY TAKING A.I. AND HUMAN INTELLIGENCE, COMBINING THOSE TOGETHER, THE TRICKS TO DEFINE WHERE DO WE FOCUS ATTENTION IN NARROW A.I., THAT'S THE MAIN FOCUS FOR THE NEXT 20, 30, 40 YEARS AND HOW DO YOU DEFINE THOSE? WELL, THE THOUGHT IS REALLY AROUND THE NOTION OF GETTING THE RIGHT USE CASES, PUTTING THOSE TOGETHER. SHOULD YOU LOOK FOR MOUNTAINS OR SHOULD YOU BE LOOKING FOR CARS OR SHOULD WE BE LOOKING FOR CANCER OR NOR NODULES, ET CETERA, AND HOW DO YOU COMBINE THAT WITH THE HUMAN INTELLIGENCE? HOW DOES ALL THIS COME TOGETHER IS WHAT'S REALLY GOING TO BE NECESSARY FOR MAKING SURE WE GET THIS RIGHT IN THIS COMBINED SPACE OF HUMANS AND MACHINES. IN LOOKING AT THAT, AT THIS INTERPRETATION PHASE SPECIFICALLY, JUST TO GET TO KIND OF SOME BRASS TALKS, WHAT USE CASES WOULD LOOK LIKE, TASKS RADIOLOGISTS DO, SUBSPECIALIZES INTO DIFFERENT AREAS, USE CASES DIVIDED INTO ORGAN SYSTEMS. BEYOND THAT THERE'S ALSO MODALITIES THAT ARE THE FOCUS. THEN YOU HAVE TO LOOK AT ANATOMY SPECIFICALLY, WITHIN THAT MODALITY AND ORGAN SYSTEMS SPECIFIC FINDINGS, NORMAL, ABNORMAL AND COMPONENTS. LET'S LOOK AT MRI OF MUSCULOSKELETAL SYSTEM, WITHIN THAT WE'LL LOOK AT THE KNEE, AND WITHIN THAT ONE ANATOMICAL AREA, WITH ONE FINDING, POSTERIOR CRUCIATE LIGAMENT TEAR. I COULD HAVE PICKED ANY BOX BUT I'LL USE THIS ONE. THERE'S A TEAR OF THE PCL. IF WE'RE TRYING TO BE ABLE TO CREATE ROC CURVES WE HAVE RADIOLOGISTS IN VARIOUS FORMS, 30,000 RADIOLOGISTISTS IN THE UNITED STATES WITH VARIOUS FORMS OF ACCURACY TO DETECT EACH OF THE FINDINGS ON THE PREVIOUS SLIDE. SO WHEN WE HAVE A.I. THE QUESTION IS DO WE HAVE TO HAVE SOMETHING THAT'S SUPER HUMAN ACCURACY OR WHAT IF WE HAVE A.I. IN THE MIDDLE OF THE POPULATION OF RADIOLOGISTISTS, HOW WOULD YOU DEPLOY AND HELP MAKE IT USEFUL? IF YOU LOOK AT THIS CURVE ESSENTIALLY OF RADIOLOGISTS ACROSS THE COUNTRY, PROBABLY A BELL SHAPED CURVE HOPEFULLY SKEWED TO THE RIGHT, BUT PEOPLE ARE BETTER THAN OTHERS DETECTING SPECIFIC FINDINGS. SO WHEN YOU BRING IN A.I. THAT'S BETTER THAN ALL THAT'S WONDERFUL, BUT MORE LIKELY THERE'S GOING TO BE SOMETHING THAT'S KIND OF SOMEWHERE IN THE RANGE OF HUMAN OR UP HIGH AT THE LEVEL OF HIGH ACCURACY HUMANS. WHEN YOU DO, IT'S GOING TO CERTAINLY BENEFIT THOSE BEHIND THE CURVE BUT IT'S PROBABLY GOING TO SKEW THE WHOLE CURVE TO THE RIGHT. WHAT YOU WANT TO DO IS TRY AND OBVIOUSLY INCREASE ACCURACY OF A.I. MORE AND MORE. HOW DO YOU DO THAT, INCREASE ACCURACY? CLEARLY AS A.I. ALGORITHMS IMPROVE THAT IMPROVES INTRINSICALLY. YOU CAN HAVE DEEPER TRAINING SETS AND NARROW THE CLINICAL SCOPE. HOW WOULD YOU NARROW THE CLINICAL SCOPE OF THE PARTICULAR A.I. TASK, MAKE A NARROW A.I. NARROWER? FOCUS ON SPECIFIC PATIENT DEMOGRAPHICS, DEGREE OF CARE, SO LOOK FOR MAJOR, MINOR, AS OPPOSED TO DEGRADATION IN BETWEEN, SPECIFIC PULSE SEQUENCES OR WITH CONTRAST IN MR, CERTAIN ANATOMIC PLANES OR GO DOWN TO THE LEVEL OF CERTAIN EQUIPMENT. MAKE MAJOR DIFFERENCE BETWEEN EQUIPMENT MANUFACTURERS OR FIELD STRENGTH OR SIGNAL TO NOISE AND FOCUS ALGORITHMS IN THOSE AREAS. THAT'S SCIENCE TO DETERMINE HOW NARROW YOU NEED TO GET TO GET THE ACCURACY AND MAKE SURE IT'S NOT BRITTLE WHEN YOU DEPLOY IT. AS I SAY, YOU COULD LOOK BEYOND THAT TO LOOK AT ALL MR OF THE KNEE, OF ALL JOINTS, ALL JOINTS AND MSK OR THE ENTIRE FIELD. WE HAVE TO DO PUT THESE COMPONENTS TOGETHER TO MAKE IT USEFUL. THAT'S JUST ONE AREA, RIGHT? AS WE COME UP WITH THIS IDENTITY OF, SAY, POSTERIOR CRUCIATE LIGAMENT TEAR AS USE CASE, THEY CAN BE LABELED AND DO RESEARCH, DATA SCIENCE, CREATE APPLICATIONS AND DEPLOY THOSE IN THE WILD AND YOU COULD HAVE SOMEBODY THAT HASN'T BEEN INVOLVED WITH THAT DEVELOPMENT PROCESS THAT COULD HAVE THE BENEFIT OF USING THAT WORK AND USING THE ABSTRACTED RESULTING A.I. ALGORITHMS TO DEMONSTRATE ON THEIR DATA THE RESULTS THAT YOU'VE BEEN INTENDED TO BE ABLE TO SHOW FOLKS. THAT'S THE INTEGRATION STEP. THE CHALLENGES HERE, AS I'VE SEEN THEM, JUST MY OPINION, INCREDIBLY PROMISING RESEARCH IN INITIAL PUBLICATIONS AND APPLICATIONS OF HEALTHCARE A.I. THERE'S LIMITED USE OF A.I. IN CLINICAL CARE. WHY IS THIS? THERE'S A NUMBER OF REASONS. SOME OF THE ONES I SEE ARE THE QUESTION I HAVE, USER EXPERIENCE, THE ANSWER IS CLEARLY YES. IT'S ALMOST LIKE PUTTING DRIVERLESS VEHICLE INSIDE OF A 1955 CHEVY, MISSING SO MANY COMPONENTS, BUILDING APPLICATION BEFORE THE A.I. CAN BE INJECTED. THOSE ARE OVERCOMABLE. THERE'S NO SUCCESSFUL ECONOMIC OR BUSINESS MODEL, WE DIDN'T DESIGN HEALTH CARE TO BE ABLE TO HAVE A LOT OF THIS DONE IN AUTOMATED SENSE, ACRs ARE A NUMBER OF MEETINGS WITH INDUSTRY TO TALK ABOUT WAYS TO BE ABLE TO DEPLOY SOLUTIONS INSIDE THE CURRENT BUSINESS MODELS FOR REIMBURSEMENT. LARGE ANNOTATED TRAINING SETS ARE DIFFICULT TO CREATE. THERE'S BEEN A NUMBER THAT HAVE BEEN RELEASED BUT THERE'S MANY MORE THAT NEED TO BE, TO BE ABLE TO SEE THIS KIND OF WORK MOVE FORWARD. THERE'S NO STANDARDS FOR CLINICAL INTEGRATION INTO CARE MANAGEMENT SO ONCE HAVE YOU AN APPLICATION COMPLETE IT'S A ONE-OFF TO DEPLOY THESE INTO THOUSANDS OF HOSPITALS THAT WE HAVE ACROSS THE UNITED STATES ALONE. AND THEN FINALLY, CLINICALLY EFFECTIVE USE CASES FOR A.I. HAVE BEEN POORLY DEFINED BECAUSE WE HAVEN'T HAD A NEED BEFORE A.I. STARTED TO COME INTO THE MAINSTREAM IN THE LAST 5 TO 10 YEARS. I'M FOCUS ON THIS NOTION OF A.I. USE CASES AND TALK MORE ABOUT THAT LATER. IF YOU LOOK AT LUNG CANCER SCREENING, LET'S SAY FOR EXAMPLE, WHICH IS CT, LOW DOSE CT OF THE CHEST, SO YOU'LL SEE PULMONARY NODULES, THAT'S ONE OF THE FINDINGS YOU'RE LOOKING FOR. WELL, WE ANALYZED PULMONARY NODULE SOLUTIONS AT MASS GENERAL AT THE CENTER FOR CLINICAL DATA SCIENCE THERE, AND WHEN WE LOOKED AT THE DIFFERENT ALGORITHMS THAT HAD BEEN FDA APPROVED AND AVAILABLE, THE RESULTING INFERENCE WITH NODULE DESCRIPTIONS OF THREE DIFFERENT PIECES OF SOFTWARE, YOU THINK WOULD BE THE SAME, WELL, THEY ARE NOT. THE CHALLENGE IS NOT ONLY ARE THEY NOT THE SAME, SOMEWHAT UNDERSTANDABLE, BUT EVEN THE WAY THEY DESCRIBE THE NODULES AREN'T THE SAME. SO THE CHALLENGE IS IF YOU'RE PUTTING IN SOFTWARE AND CHANGE FROM YEAR TO YEAR, THE PATIENT LOOKS LIKE THEIR DISEASES ARE CHANGING BUT THEY ARE NOT. IT'S JUST THE SOFTWARE WASN'T DEFINED WELL ENOUGH TO MAKE SURE THERE'S CONSISTENCY ACROSS DEFINITIONS. IF YOU PUT IN NOT JUST ONE EXAMPLE BUT HUNDREDS, THOUSANDS OF USE CASE SOLUTIONS IN PLACE WITHOUT STANDARDS TO BOUNCE BACK BETWEEN THEM, IT REALLY IS A CHALLENGE FOR PATIENT CARE AND MANAGEMENT, ONE THING THAT WILL BE TALKED ABOUT TODAY, I WILL AT THE END, THE FINAL SESSION. WITHOUT THAT NOTION OF THE USE CASE BEING PUT IN PLACE, IT BECOMES CHALLENGING HOW TO TRAIN DATASETS AND AT THE END THE RESULTING ALGORITHMS THAT COME OUT ARE GOING TO BE ABLE TO SHOW DIFFERENT RESULTS FOR THE SAME FINDINGS SO THAT BECOMES A CHALLENGE. IF YOU CAN PUT THAT IN, YOU CAN START TO -- USE CASE UP FRONT, START TO FOCUS A.I. DEVELOPMENT IN A MORE ORGANIZED FASHION, MULTIPLE APPLICATIONS THAT HAVE GONE THROUGH THE SAME DEFINITION OF WHAT THE USE CASES ARE, IN A CONSISTENT WAY HOPEFULLY TO DEPLOY THE SOLUTIONS CLINICALLY. SO JUST TO SUMMARIZE, I THINK IT'S SAFE TO SAY A.I. WILL DRAMATICALLY CHANGE HEALTH CARE, AS KRIS HAD SAID THIS IS CLEARLY COMING FORWARD. THERE'S A LOT OF MOVEMENT IN THE AREA WITH RADIOLOGY AT THE FOREFRONT, PROBABLY BECAUSE A LOT WAS DONE ON IMAGING SO MEDICAL IMAGING IS THE LOGICAL NEXT STEP TO PUT SOLUTIONS IN PLACE, ALSO DIAGNOSTICS LENDS ITSELF WELL BECAUSE OF RICH DATA AND CONSISTENT DEFINITION OF DATA OUT THERE. I HOPE I DESCRIBED A LITTLE BIT THAT STANDARDS ARE GOING TO BE ESSENTIAL FOR CLINICAL ADOPTION OF THE RESEARCH THAT WE ALL DO. I THINK THE FUTURE IS GOING TO LOOK SOMETHING LIKE THIS FOR THE RADIOLOGIST, THAT WITH EHR IN THE PAST YOU'LL GET IMAGE DATA, FINDINGS WILL BE DISPLAYED. NEXT UP THOUGH THERE WILL BE WAYS TO GUIDE THE PATHWAY OF THOSE FINDINGS THROUGH EHR INFORMATION, OTHER PIXEL INFORMATION TO GIVE GUIDANCE TOWARDS RECOMMENDATIONS, CLASSIFICATIONS, NOW THE RADIOLOGIST INTERPRETATION CAN TAKE PLACE BUT ALONG WITH THAT WILL BE STRUCTURED RECOMMENDATIONS AND A.I. QUANTIFIED NOTIONS IN THE DIGITAL PATHWAY FOR MULTIPLE USE CASES SO THE RADIOLOGY SHOULD BE INVOLVED IN THIS PROCESS WITH THE HUMAN AND MACHINE INTEGRATION BUT ALSO BE ABLE TO INTERACT WITH THE HUMANS THAT ARE ALSO ENGAGED IN THIS PROCESS, OTHER CLINICIANS AND THE PATIENT THEMSELVES AND EXPLAIN A LOT OF THE ACTIVITIES THAT ARE TAKING PLACE IN THAT MAN-MACHINE INTERFACE. FINALLY JUST IN CLOSING I'D SAY THE COMBINATION OF HUMANS AND A.I. IS CLEARLY THE FUTURE. AS YOU'RE LOOKING FOR RESEARCH, I THINK THE CHALLENGE IS NOT TO LOOK AT THE REPLACEMENT OF HUMANS. IT'S LAUDABLE IF YOU CAN. BUT THE SHORT STEPS AND WINS WILL BE IN COMBINATION OF PROCESSES TAKING PLACE TODAY. LOVE THIS QUOTE IN 1965, A NASA REPORT ADVOCATING MANNED SPACE FLIGHT WHY I THINK HUMANS WILL BE AROUND A WHILE, MAN IS THE LOWEST-COST 150-POUND COMPUTER SYSTEM WHICH CAN BE MASS PRODUCED BY UNSKILLED LABOR. [LAUGHTER] THANK YOU VERY MUCH. [APPLAUSE] I'M TOLD WE'RE GOING TO DO Q&A AT THE PACKAGE SESSION. NEXT UP CURT CURT LANGLOTZ FROM STANFORD WILL COME UP AND PRESENT. >> GOOD MORNING AGAIN, EVERYONE. SO THANK YOU, KEITH, FOR THAT TREMENDOUS INTRODUCTION. I'M GOING TO BE DESCRIBING SOME OF THE RESEARCH NEEDS, AND I THINK ONE THING THAT'S PRETTY CLEAR IS, AT LEAST THE RESEARCH LABS THAT I KNOW, IS THAT A.I. IS REALLY NOW PERVASIVE THROUGHOUT MOST RESEARCH LABS, NOT JUST MEDICAL IMAGING, BUT OTHER SORTS OF LABS, BIOCHEMISTRY LABS, GENOMICS, SO A TECHNIQUE THAT IS GROWING RAPIDLY. WHAT I'D LIKE TO DO IS GIVE SOME ADDITIONAL CONTEXT, MORE FROM A RESEARCH POINT OF VIEW, AND THEN REVIEW -- I WON'T GIVE TOO MANY EXAMPLES OF A.I. MEDICAL IMAGING TECHNOLOGY BECAUSE YOU'LL BE ABLE TO SEE SOME OF THOSE THROUGHOUT THE PROGRAM. SO I'LL JUST BRIEFLY REVIEW AN EXAMPLE. THEN TALK ABOUT SOME OF THE CHALLENGES AND THEN IN CLOSING REVIEW SOME REAL RESEARCH SO, START WITH JUST A RECAPITULATION OF THE DEFINITIONS THAT KEITH DESCRIBED. SO, A.I. IS REALLY THE BROADEST TERM THAT WE USE. IT HAS THE ADVANTAGE THAT KIND OF LAYPEOPLE KNOW ABOUT IT. IT HAS THE DISADVANTAGE THAT WE TEND TO LAYER A LOT OF HUMAN CHARACTERISTICS ON A.I. THAT PROBABLY DON'T REALLY EXIST, JUST BECAUSE IT CAN DO ONE VERY NARROW TASK. MACHINE LEARNING AS WE KNOW IS A VERY SPECIFIC FORM OF A.I., WHICH IS FEEDING DATA INTO A COMPUTER SYSTEM TO HELP IT ACHIEVE BETTER PERFORMANCE. NEURAL NETWORKS ARE POWERFUL. EVEN REGRESSION WITH HEIGHT AND WEIGHT OF MIDDLE SCHOOL KIDS, YOU CAN PREDICT HEIGHT FROM WEIGHT IS A FORM OF MACHINE LEARNING BUT NEURAL NETWORKS ARE MORE POWERFUL. DEEP LEARNING HAS TO DO WITH THE NUMBER OF LAYERS IN THE NETWORK AS KEITH MENTIONED, BUT I WOULD SAY IN IMAGING ALMOST ALL THE NETWORKS ARE DEEP, SO DEEP LEARNING HAS BECOME A REBRANDING OF NEURAL NETWORKS. SO I'D LIKE TO TAKE YOU THROUGH SOME OF THE HISTORY OF A.I. RESEARCH BECAUSE I THINK IT HELPS EXPLAIN WHERE WE ARE TODAY. SO A.I. WHEN IT WAS ORIGINALLY COINED REALLY DEALT MORE WITH SYMBOLICS SYSTEMS, TRUE AND FALSE, THINGS WE CAN PROVE, RULE-BASED SYSTEMS. IT DIDN'T WORK WELL IN MEDICINE, WORKED WELL FOR A FEW APPLICATION APPLICATIONS, BUT DIDN'T WORK WELL PARTICULARLY FOR IMAGING. NEXT PHASE HAD TO DO WITH MACHINE LEARNING. WHEN I WAS TRAINING IN A.I. IN THE '80s, THEY SAID THEY WERE TRAINING US TO BE KNOWLEDGE ENGINEERS. WHAT THAT MEANT WAS WE WOULD PAIR UP WITH DOMAIN EXPERTS, WHO KNEW HOW TO READ A MAMMOGRAM, FOR EXAMPLE, COULD DISTINGUISH BENIGN FROM MALIGNANT, WHAT FEATURES LEAD YOU TO BELIEVE A MASS MIGHT BE BENIGN OR MALIGNANT AND WE'D FIND WAY TO EXTRACT FROM IMAGES AND FEED INTO A MACHINE LEARNING ALGORITHM. IT WAS KIND OF A MANUAL FEATURE EXTRACTION PROCESS. SO JUST TO GIVE AN EXAMPLE WE MIGHT HAVE THE RED HERE ARE MA MALIGNANT AND BLUE IS BENIGN, IN PRACTICE WE WOULD HAVE A MULTI-DIMENSIONAL FEATURE SPACE. HERE YOU SEE MALIGNANT CASES AROUND THE EDGES, BENIGN CASES ARE IN THE CENTER. SO OUR GOAL IS TO FIND A MACHINE LEARNING ALGORITHM THAT CAN DISTINGUISH THE BENIGN FROM MALIGNANT. HERE ON THE UPPER LEFT YOU SEE A NEAREST NEIGHBOR WHICH HAS A JAGGED EDGE. YOU CAN IMAGINE A SLIGHTLY DIFFERENT CLUSTER WITH THE SAME MORPHOLOGY, THAT MIGHT NOT BE THE RIGHT ALGORITHM. THE SECOND ON THE CENTER TOP, LINEAR SUPPORT VECTOR MACHINE, A SPECIAL WAY TO SUPPORT WITH A LINEAR VECTOR. THIS HIGHLIGHTS THE MODEL SELECTION PROCESS, DECIDING WHICH MACHINING LEARNING ALGORITHM AND DECIDE WHICH FEATURES TO USE TO REPRESENT THE CASES. SO, THAT'S WHERE SORT OF THE THIRD GENERATION OF A.I. COMES IN, AND THAT'S THESE NEURAL NETWORKS THAT KEITH DESCRIBED SO WELL. THEY REALLY DO ALL OF THAT AUTOMATICALLY. THIS SLIDE JUST -- YOU SEE COEFFICIENTS, ILLUSTRATES THESE ARE VERY LARGE MATHEMATICAL MODELS. THE THING THAT'S HAPPENED MOST RECENTLY, COMPUTER SCIENTISTS FIGURED OUT HOW TO ADJUST PARAMETERS OF THESE VERY LARGE MATHEMATICAL MODELS BASED ON WHETHER NEURAL NETWORKS GET THE CASE WRONG OR RIGHT, OVER BILLIONS OF TRIALS THE MODEL GETS BETTER TO ACHIEVE EXPERT PERFORMANCE. ONE OF THE KEY POINTS IN THE HISTORY OF A.I. IN MEDICINE, COMPUTER VISION, IS IMAGE NET, WHICH IS A DATABASE OF OVER 14 MILLION IMAGES THAT WAS ASSEMBLED BY A PROFESSOR OF COMPUTER SCIENCE AT STANFORD. EACH-- YOU SEE ON THE LEFT THE GARDEN SPIDER. EACH IMAGE HAS A DOMINANT OBJECT IN THE CENTER, SOMETIMES VARYING BACKGROUNDS, DIFFERENT ORIENTATION, 21,000 LABELS, 856 TYPES OF BIRDS, 157 MUSICAL INSTRUMENTS AND EVERY YEAR HELD A CONTEST. IN 2011, THE ERROR RATE WAS 25%. IN 2012 WAS THE FIRST TIME THAT THESE NEURAL NET WORKS WERE USED TO ADDRESS THIS CONTEST. REPORT RATE DROPPED DRAMATICALLY, WON BY A TEAM FROM THE UNIVERSITY OF TORONTO, JEFF HINTON, WHO NOW WORKS FOR GOOGLE. 2013 WON BY NYU TEAM, ERROR RATE DROPPED FURTHER. THEN GOOGLE WON IT, MICROSOFT WON IT, THE LAST TWO YEARS WON BY TEAMS FROM CHINA, ERROR RATE CONTINUED TO DROP, NOW AROUND 2.25%, PEOPLE FEEL THAT'S AS MUCH INFORMATION YOU CAN EXTRACT. THEY MOVED TO MORE COMPLICATED TASKS LIKE IMAGES WITH MULTIPLE OBJECTS, CAN WE FIND EACH OBJECT AND CREATE A LABEL, THESE ARE CAPTIONS, A BOTTLE OF WATER, CUP OF COFFEE, PLATE OF FRUIT. WHEN RADIOLOGISTS SEE THAT WE SAY, HEY, THAT'S WHAT WE DO. AND SO EVERYONE I THINK AT THAT POINT RECOGNIZED THE INCREDIBLE POWER OF TECHNIQUES WHEN APPLIED TO MEDICAL IMAGES. SO, WHAT IMPLICATIONS DOES THAT HAVE FOR MACHINE LEARNING RESEARCH IN MEDICAL IMAGING? WELL, IT STARTS WITH A PATIENT ON A SCANNER. LET'S SAY ON AN M.R. SCANNER. THAT SCANNER FROM THE DETECTORS PRODUCES SOURCE DATA OR RAW DATA. THAT DATA NEEDS TO BE RECONSTRUCTED INTO A CARTESIAN, WE COULD USE THE DATA TO DRAW CONCLUSIONS, FOLKS ARE WORKING ON THAT. BUT THERE'S NO QUESTION THERE'S A WEALTH OF RESEARCH GOING ON NOW ON NEW IMAGE RECONSTRUCTION METHODS. SO SAME ISSUE. WE USED TO HAVE HUMAN-GENERATED METHODS. WHAT'S THE BEST WAY TO MIGRATE THROUGH CASE SPACE? NOW WE CAN USE A.I. ALGORITHMS TO DECIDE THE BEST WAY TO RECONSTRUCT AN IMAGE HIGHEST RESOLUTION, LOWEST NOISE. OFTEN INTERPRETED BY A RADIOLOGIST, WHO CREATES A REPORT THAT DESCRIBES FINDINGS IN THOSE IMAGES. AND THERE'S A LOT OF GOOD RESEARCH GOING ON NOW, HOW CAN WE EXTRACT THE INFORMATION FROM THE VAST WEALTH OF REPORTS THAT GO WITH THOSE IMAGES TO CREATE AUTOMATED LABELS FOR THESE IMAGES. AUTOMATED LABELING, EVEN IF IT'S NOISY, CAN BE A POWERFUL CONCEPT. I'LL TALK ABOUT THAT IN A MOMENT. WELL, THE NEURAL NETWORKS DESIGNED TO SOLVE IMAGENET BASED ON COLOR PHOTOGRAPHS, MULTI-MODALITY, TIME-BASED IMAGES. WE NEED NEW NEURAL NETWORK ARCHITECTURE TO SOLVE PROBLEMS UNIQUE TO MEDICAL IMAGES. ALSO PRIVACY ISSUES, MAYBE WE DON'T WANT TO MOVE THE DATA, WE WANT TO MOVE ALGORITHMS AND TRAIN ON DIFFERENT DATA IN DIFFERENT PLACES DUE TO PRIVACY ISSUES. LASTLY AS KEITH DESCRIBED THESE SYSTEMS ARE LIKELY TO BE PARTNERSHIPS BETWEEN THE HUMAN EXPERT AND THE MACHINE, SO RESEARCH ON HOW THE MACHINE CAN BEST EXPLAIN HOW IT CAME TO ITS CONCLUSION CAN BE VERY POWERFUL MAKING SYSTEMS USEFUL IN THE REAL WORLD. THOSE ARE SOME OF THE RESEARCH QUESTIONS, YOU'LL HEAR MORE THROUGHOUT THE MEETING. LASTLY CLINICAL EVALUATION, THESE ARE JUST DIAGNOSTIC TECHNOLOGIES, WE NEED TO BE ASSURED THEY WORK JUST AS OTHER DIAGNOSTIC TECHNOLOGIES HAVE COME ALONG AND ARE NEW. I WANT TO SAY A WORD WILL LABELING, THIS IS IMPORTANT. WE WOULD ALL LOVE IT IF WE HAD HUMAN EXPERTS LABEL ALL OF OUR œLEARNING ALGORITHMS.R MACHINE OFTEN THAT'S NOT THE CASE SO WE NEED TO LOOK FOR WEAK LABELING ALGORITHMS. THIS IS ONE WE USE. IT'S A COMMERCIAL PRODUCT THAT'S KIND OF A GOOGLE SEARCH OF YOUR RADIOLOGY REPORT DATABASE. HERE SEARCHING FOR ALL OF THE CASES THAT CONTAIN THE WORDS TENSION PNEUMOTHORAX, FOR A MACHINING LEARNING ALGORITHM. NOT PERFECT. YOU NEED TO DO NEGATION DETECTION. YOU COULD USE OLDER MACHINE LEARNING, SUPPORT VECTOR MACHINES, CONDITIONAL RANDOM FIELDS TO EXTRACT INFORMATION FROM TEXT. HERE WE HAVE THE BLUE ARE ANATOMY, CHEST, ABDOMEN, PELVIS. GREEN IS OBSERVATION, INFILTRATES, NODULES, FUSION. ORANGE IS UNCERTAINTY AND SO ON. YOU CAN EXTRACT INFORMATION AGAIN NOT WITH PERFECT ACCURACY BUT AUTOMATICALLY. THERE ARE TOOLS LIKE SNORKEL, OPEN-SOURCE TOOL TO DEFINE RULES, SHOW ME ALL THE REPORTS THAT CONTAIN THE WORDS HEMORRHAGE WITHIN FIVE WORDS OF THE WORD "BRAIN" AND PERMUTATIONS, A SINGLE BEST RULE TO LABEL THE CASES. LASTLY YOU CAN USE THESE NEURAL NETWORKS AS WELL. THERE'S THE GLOVE TECHNIQUES TO DEVELOP WORD EMBEDDING, VECTORIZATION OF THE WORDS, APPLY DEEP LEARNING METHODS TYPICALLY BY DIRECTIONAL LSTM NETWORKS TO EXTRACT INFORMATION FROM THE TEXT. SO, VERY IMPORTANT CONCEPT, WEAK AUTOMATED LABELING. WHY IS THIS SO IMPORTANT? SO HERE ARE TWO PAPERS, BOTH ANALYTICAL PAPERS THAT DESCRIBE, SO HERE WE HAVE A PLOT, ON THE Y-AXIS THE LEVEL OF NOISE IN THE DATA. SO THIS IS THE AMOUNT OF ERROR THAT OUR LABELING TECHNIQUE MIGHT HAVE. ON THE Y-AXIS, WE HAVE THE DATA SET SIZE, HOW MUCH ADDITIONAL DATA WOULD WE NEED IF WE HAVE NOISY DATA TO TRAIN TO THE SAME LEVEL OF ACCURACY. SO, FOR EXAMPLE, HERE IF WE HAD 10% NOISE, SO 90% ACCURATE LABELING METHOD, 60% MORE DATA NEEDED. IF WE HAD 15% NEED WE WOULD NEED TWICE THE AMOUNT OF DATA. 20% NOISE WE NEED THREE TIMES THE AMOUNT OF DATA, ASSUMING NORMALLY DISTRIBUTED NOISE, NOT STRICTLY TRUE IN THE WORLD WORLD BUT IMPORTANT POINT TO ACHIEVE SIMILAR ACCURACY WITH NOISY LABELS IF YOU HAVE ENOUGH LABELS. IN CLINICAL MEDICINE, TYPICALLY IT'S IN THE RANGE OF 85 TO 90% ACCURATE. SO WE'RE IN THIS BAND OF ACCURACY, AND OFTEN IN HEALTH CARE WE DO HAVE PLENTY OF IMAGES. IT'S JUST WE DON'T HAVE LABELS. YOU CAN TAKE THESE AUTOMATED NOISY LABEL TECHNIQUES, RUN THEM AND ACHIEVE THE SAME ACCURACY WITH NOISY LABEL. YOU NEED GREAT HUMAN LABELS TO MEASURE ACCURACY TO EVALUATE YOUR MODEL ONCE YOU BUILT IT BUT IN ORDER TO TRAIN THE MODEL THEY CAN BE EXTREMELY POWERFUL. I WANT TO TAKE ONE EXAMPLE OF A DEEP LEARNING MODEL JUST TO MAKE A POINT ABOUT PRECISION MEDICINE. THIS IS THE WORK OF A TEAM LED BY DAVID LARSON WHO YOU'LL HEAR FROM LATER IN THIS MEETING. THIS IS THE ISSUE OF BONE AGE. MANY LABORATORIES ADDRESSED THIS, IT'S A RIFE PROBLEM. CHILDREN WHO ARE SUSPECTED, MAY HAVE DEVELOPMENTAL DELAY, COMPARE CHRONOLOGIC AGE TO PHYSIOLOGIC AGE, USE A HAND X-RAY TO ASSESS PHYSIOLOGIC AGE, LOOK AT THE DEVELOPMENT OF BONES AND DETERMINE THAT. THE STATE OF THE ART FOR DETERMINING THAT IS A BOOK, GRULICH AND PYLE, TRYING TO FIND THE ONE THAT MATCHES, 7 1/2, MUST BE 7 1/2 YEARS OLD. OBVIOUSLY RIPE FOR APPLICATION OF A.I. THE REFERENCE DATA IN THAT BOOK CAME FROM 300 CAUCASIAN CHILDREN WHO GREW UP IN THE 1950S. SO RIPE FOR A.I. APPLICATIONS. THIS IS A HEAT MAP WHICH SHOWS WHICH PIXELS CONTRIBUTED MOST. NOT THAT YOU CAN ACHIEVE EXPERT PERFORMANCE BUT IN THE OLD CASE WITH FEATURE ANALYSIS AND HUMANS BEING KNOWLEDGE ENGINEERS, TWO OR FOUR YEARS TO BUILD THE A.I. MODEL, MASTER'S OR Ph.D. TO DETECT MALIGNANT FROM BENIGN BREAST MASSES, NOW IN WEEKS, TO FIT WITH DIFFERENT GROUPS AND CASE MIXES AND BUILD A MODEL THAT GETS THE RIGHT ANSWER FOR THE RIGHT GROUP OF PATIENTS, AN IMPORTANT CONCEPT. KEITH ALLUDED TO THIS, WHERE DO WE SEE A.I. HAVING A BENEFIT THROUGHOUT THE IMAGE LIFE CYCLE? KRIS MENTIONED THIS AS WELL. TEST SELECTION, SOMEONE'S ABOUT TO REQUEST A STUDY, CAN WE PROVIDE INFORMATION HOW LIKELY IS IT THAT STUDY WILL BE POSITIVE IN PATIENTS JUST LIKE THE ONE ON WHOM WE'RE ORDERING THAT STUDY. IMAGE RECONSTRUCTION WE TALKED ABOUT THAT. NICE WORK ESTIMATING A VERY LONG M.R. SEQUENCE WOULD LOOK LIKE FROM A SHORTER, ESTIMATE THE FULL PET DOSE FROM PARTIAL PET DOSE, WHAT A FULL CONTRAST DOSE MIGHT LOOK LIKE FROM PARTIAL, SAVING RADIATION TIME AND DOSE, POWERFUL CONCEPTS. IMAGE QUALITY CONTROL, YOU HAVE PATIENTS WHO MAY BE MOVING ON THE SCANNER, MAYBE BREATHING, IMAGE QUALITY IS NOT GREAT. THAT MAY NOT BE DISCOVERED UNTIL THE RADIOLOGIST LOOKS AN HOUR OR TWO LATER. NOW THE PATIENT IS HOME, CONTRAST IS EXCRETED, THEY HAVE TO COME BACK, TAKE UP ANOTHER SLOT, NO ONE IS HAPPY ABOUT THAT. YOU CAN IMAGINE EMBEDDING IN THE SCANNER TECHNOLOGY THAT WOULD IDENTIFY THOSE CASES AUTOMATICALLY AND ALERT THE TECHNOLOGISTS. IMAGING TRIAGE, I'M A CHEST RADIOLOGIST, WE GET DOZENS EVERY MORNING, AROUND FIVE OR SIX IN THE MORNING, THOSE ARE TAKEN BECAUSE THEY ARE DESIGNED TO IDENTIFY THE FEW CASES WHERE THERE ARE URGENT PROBLEMS, TUBES OUT OF PLACE, PNEUMONIA, THAT SORT OF THING. WOULDN'T IT BE WONDERFUL TO HAVE AN ALGORITHM THAT WOULD BRING THOSE CASES TO THE TOP? SO THAT I COULD LOOK AT THEM FIRST INSTEAD OF WAITING UNTIL 9:30 IN THE MORNING WHEN I FIND THE END OF THE STACK. BY THE WAY, THAT ALGORITHM DOESN'T NEED TO BE PARTICULARLY ACCURATE. IF IT TOOK TEN OUT OF THE HUNDRED AND ONLY FIVE OF THE TEN HAD ABNORMALITIES, THAT'S STILL A POSITIVE BENEFIT TO THE PATIENT. COMPUTER-AIDED DETECTION SOMETHING WE'VE BEEN DOING FOR MANY YEARS, MACHINE LEARNING TECHNOLOGIES WILL PLAY A ROLE. THERE DISEASE CLASSIFICATION, VERY POPULAR TO RADIOLOGISTS BECAUSE OFTEN WE'RE ON CALL AT NIGHT OR WE'RE IN A SMALL HOSPITAL, WE'RE LOOKING AT AN IMAGE OUTSIDE OF OUR TYPICAL ZONE OF COMFORT, MAYBE I'M A CHEST RADIOLOGIST, I SEE SOMETHING IN THE BONES I'M NOT FAMILIAR WITH, CAN I PUT A CIRCLE AROUND IT, HAVE THE ALGORITHM TELL ME THE MOST LIKELY DIAGNOSTIC POSSIBILITIES OR SHOW ME A REPORT ON A SIMILAR CASE. LASTLY THIS ISN'T LOOKING AT PIXELS OF THE IMAGE BUT ALL OF THE IMAGE REPORTING FUNCTIONS, SO, YOU KNOW, IS THERE SOMETHING MENTIONED IN THE RECORD THAT NEEDS TO BE COMMUNICATED? IS THERE SOMETHING WHERE WE OUGHT TO ELABORATE A LITTLE MORE, IS THERE MORE INFORMATION NEDED THAT COULD BE REQUESTED AT THAT TIME AS PART OF THE REPORT? OKAY. SO LET'S TAKE A MINUTE TO TALK ABOUT WEAKNESSES. THIS IS A CROSS TABLE LATERAL OF THE KNEE, COMPLEX TIBIAL FRACTURE. WHAT HAPPENS IS THE FAT FROM BONE MARROW WILL COME INTO THE JOINT IF IT'S INTRAARTICULAR, THERE'S A FAT FLUID LEVEL IN THE KNEE JOINT. COMMON FINDING. YOU CAN IMAGINE, FIRST WE TRAIN OUR RESIDENTS, TRAIN HUMANS, YOU CAN ALSO TRAIN MACHINES, TRAIN ONE OF THESE ALGORITHMS, FEED A THOUSAND POSITIVES AND NEGATIVES, NO DOUBT YOU COULD TRAIN AN A.I. ALGORITHM TO RECOGNIZE FAT FLUID LEVEL. RESIDENT. RESIDENT WOULD SAY AHA, WELL, I KNOW THIS IS A T1 WEIGHTED IMAGE, FAT IS BRIGHT, THIS LOOKS LIKE FAT, I SEE THE HORIZONTAL LINE, CLEARLY FAT FLUID LEVEL ON MR, THEY MAY HAVE NEVER SEEN IT BEFORE. FOR AN A.I. ALGORITHM START WITH A THOUSAND CASES THAT SHOW FAT FLUID LEVEL AND START FROM SCRATCH, HOW CAN WE HELP THEM USE COMMON-SENSE AND OTHER FIRST PRINCIPLES THAT HUMANS USE MAKING INFERENCES, RELIEVING US OF NEEDS FOR DATA TO FEED INTO THE ALGORITHMS. BROAD CONCEPTS, SO WE'VE TRIED TO TRAIN AN ALGORITHM, HAD SOME SUCCESS, IT WAS A CHALLENGE TO TRAIN TO RECOGNIZE FRACTURES ACROSS BODY PARTS BECAUSE A FRACTURE OF THE FINGER, FEMUR VERSUS THE SCAPHOID LOOK DIFFERENT, ABSTRACT CONCEPTS WOULD BE USEFUL TO INFER. A.I. 1.0, HOW DO YOU COMBINE THE TWO METHODS? AND KRIS MENTIONED SECURITY. THIS IS CLEARLY AN ISSUE. YOU CAN BUILD NEURAL NETWORKS TO TRY TO DEFEAT OTHER NEURAL NETWORKS, HERE ONE BUILT TO DEFEAT A STOP SIGN RECOGNIZER, VERY IMPORTANT FOR SELF-DRIVING CARS BUT IF YOU PUT BLACK AND WHITE STICKERS THE ALGORITHM NO LONGER RECOGNIZES IT AS A STOP SIGN. THIS MEDALLION, SOMETHING THAT GOOGLE DEVISED, PUT IT ON ANY IMAGENET TYPE IMAGE, IT SAYS THAT IMAGE IS A TOASTER. AUTOMATICALLY. TRUE. MEDICAL, SO HERE'S A DERMATOLOGIC IMAGE. ALGORITHM CHARACTERIZES IT ALMOST CERTAINLY BENIGN, ADD THIS ADVERSARIAL NOISE, IMPERCEPTIBLE TO THE HUMAN EYE. NOW ADD THE NOISE, YOU CAN'T SEE THE NOISE, 100% CHANCE OF PNEUMOTHORAX. SO VERY IMPORTANT THAT WE CONSIDER THE BRITTLENESS OF IN CLOSING, WHAT ARE SOME RESEARCH OPPORTUNITIES? NUMBER ONE, CURRENTLY NON-NEURAL, HOW CAN WE TAILOR FOR CLINICAL IMAGES? COSTLY CUSTOM DEVELOPED HUMAN CREATED LABELS, WE SPEND TIME PAYING HUMAN BEINGS TO LABEL CASES, CAN WE DEVELOP NEW METHODS THAT MORE ACCURATELY CAN AUTOMATE THE LABELING OF CASES FOR TRAINING? WE DEVELOP VERY LARGE NOISILY LABELED DATASETS WHICH ARE GREAT FOR TRAINING THESE MODELS. WE HAVE TODAY STATIC TRAINING DATASETS AT HIGH COST, ONE DATASET, ONE DATASET, ONE FOR EACH RESEARCH PROGRAM. THINGS LIKE STRUCTURED REPORTING AND CAPTURING DISCRETE DATA CAN CREATE A CYCLE WHERE THE DATA IS PART OF CLINICAL CARE CAN BE USED AS LABEL AND FEED INTO A MACHINE LEARNING ALGORITHM. WE NEED MORE PUBLIC CHALLENGES AND PUBLIC RELEASE OF DATA, ESPECIALLY MULTI-INSTITUTIONAL AND ALGORITHMS THAT ALLOW US TO USE MULTI-INSTITUTIONAL DATA WITHOUT MOVING IT. ONE SIZE FITS ALL A.I. ALGORITHMS, A LOT OF COMPANIES SAY I HAVE AN ALGORITHM, I'LL SELL TO YOU AND YOU AND YOU. WE MAY NEED MORE TAILORED FOR PREVALENCE OF DISEASE, PARTICULAR TYPES OF SCANNERS, DEMOGRAPHIC MIX, INDIVIDUALOS O ORGANIZATIONS MAY WANT TO THEY WILL EVEN OTHER. RATHER THAN SINGLE APPLICATIONS WE HAVE MULTIPLE PRECISION APPLICATIONS BUILT FOR SPECIFIC PATIENT GROUPS. AND THEN AS I SAID RATHER THAN SYSTEMS THAT RELY SOLELY ON MACHINE LEARNING, WE WANT SYSTEMS THAT COMBINE MACHINE LEARNING AND SYMBOLIC SYSTEMS. I THINK WE'RE IN FOR A PROFOUND CHANGE, NO QUESTION THESE TECHNIQUES WILL PROFOUNDLY AFFECT THE PRACTICE OF RADIOLOGY. I THINK WE'RE JUST AT THE BEGINNING, SO SOME RESEARCH THAT WE'LL HEAR ABOUT TODAY IS GOING TO REALLY CHANGE HOW A.I. IS GOING TO AFFECT THE CARE OF CLINICAL PATIENTS OVER TIME, A LOT OF THAT IS STILL IN THE RESEARCH LAB BUT CERTAINLY COMING. I'M REALLY EXCITED TO HEAR WHAT ALL OF THE OTHER SPEAKERS HAVE TO SAY OVER THE NEXT DAY AND A HALF AND LOOK FORWARD TO THAT WITH YOU. THANK YOU VERY MUCH. [APPLAUSE] >> IF KEITH AND BIBB AND JAYASHREE CAN COME UP WE'LL GET STARTED ON QUESTIONS AND ANSWERS. WHILE WAITING I WANT TO THANK CURT AND BIBB FOR SETTING THIS PROGRAM UP. FAILED TO DO THAT EARLIER. IT'S BEEN A PLEASURE WORKING WITH THEM. THEY HAVE DONE A WONDERFUL JOB. IF YOU CAN SIT, THAT'S GREAT. ONE MORE THING FOR SPEAKERS, WE'RE ON VIDEOCAST SO PLEASE USE THE MOUSE HERE TO POINT TO YOUR SLIDES RATHER THAN THE ACTUAL POINTER. SO, ARE THERE ANY QUESTIONS FROM THE AUDIENCE? ANYBODY? PLEASE COME UP TO THE MICROPHONE. >> I'M ONE OF THE RADIOLOGISTS HERE IN THE CLINICAL CENTER. AND IT'S VERY FASCINATING TO SEE ARTIFICIAL INTELLIGENCE BEING USED FOR MEDICINE BECAUSE 25 YEARS AGO I DECIDED TO DO THIS, PREPARED TO DO THIS, I GUESS I WAS 25 YEARS TOO EARLY. BUT YOU MENTIONED, THE LAST SPEAKER MENTIONED A COUPLE OF THINGS. HE SAID THAT WE NEED A LOT OF DATA, AND -- BUT HE MENTIONED TWO THINGS AT THE END THAT PEOPLE ARE I THINK NYU IS DOING SOME RESEARCH WHERE THEY ARE LOOKING AT UNDERSAMPLING OF DATA TO REDUCE THE IMAGING TIME. SO HOW DOES THAT FIT IN WITH THE CONCEPT THAT WE NEED LOTS OF DATA TO INCREASE OUR MODELS? THAT'S NUMBER ONE. NUMBER TWO, YOU ALSO MENTION THE ISSUE ABOUT STRUCTURED REPORTING. RADIOLOGY REPORTS HAVE VERY DIFFERENT STYLES, TEXTS, AND SO THERE'S BEEN THIS INTEREST IN TRYING TO LOOK AT STRUCTURED REPORTING, HOW DO YOU THINK THE NEED FOR STRUCTURED REPORTING FOR IMPROVED HEALTH POLICY DATA MINING AND THINGS LIKE THAT USING A STRUCTURED REPORT, HOW DOES THAT CONTRAST WITH HAVING ENOUGH CLINICAL INFORMATION TO PROVIDE APPROPRIATE CLINICAL CARE? >> THANK YOU FOR THOSE QUESTIONS. I'VE OFTEN FELT A COUPLE DECADES EARLY AS WELL. I AGREE, I THINK THAT THE USE OF THESE A.I. ALGORITHMS FOR IMAGE RECONSTRUCTION, IF YOU WANT TO CALL IT THAT BROADLY, IS LIKELY TO BE ONE OF THE FIRST AREAS WHETHER WE SEE APPLICATION OF THESE TECHNIQUES. THERE'S BEEN SO MUCH MANUAL THOUGHT PUT IN BY RESEARCH AS TO HOW TO SAMPLE CASE SPACE, HOW TO MANAGE SONOGRAM DATA, SOME TECHNIQUES ARE EXTREMELY POWERFUL. AGAIN, WITH THE RIGHT DATASETS, YOU CAN REALLY GET SOME EXCELLENT PERFORMANCE REDUCING IMAGING TIMES, REDUCING CONTRAST DOSE, REDUCING RADIATION. SO THAT'S VERY POWERFUL. REGARDING YOUR SECOND QUESTION, I THINK THAT THE -- THERE ARE TRADEOFFS WITH RESPECT TO STRUCTURED REPORTING BECAUSE IF IT'S NOT DONE PROPERLY CAN SLOW THE RADIOLOGIST DOWN, A SERIOUS CONCERN. I DO THINK YOU COULD FIND SOME OF OUR -- AT LEAST MORE COMMON, MORE IMPACTFUL EXAMINATIONS, AND FIND MAYBE ONE OR TWO THINGS ON THAT EXAMINATION THAT OUGHT TO BE CAPTURED IN STRUCTURED FORM. I KNOW AT STANFORD WE WERE JUST FORTUNATE, ONE OF OUR NOW RETIRED RADIOLOGISTS DECIDED AT ONE POINT WE SHOULD HAVE A SINGLE CODE AT THE END OF EVERY REPORT THAT SAYS IS THIS NORMAL, IS IT BENIGN, ABNORMAL, TRULY ABNORMAL, CRITICALLY ABNORMAL, DONE FOR A PERIOD OF TEN YEARS AND JUST THAT DATA, KNOWING NORMAL VERSUS ABNORMAL IS INCREDIBLY POWERFUL FOR CREATING THESE MODELS. SO I THINK SOMETHING SIMPLE THAT COULD BE DONE COULD BE QUITE POWERFUL IN PRODUCING DATA FOR MACHINE LEARNING. >> I WANT TO ADD ONE THING ON THE STRUCTURED REPORTING PIECE, AND CERTAINLY THERE ARE MANY OF US WHO SEE STRUCTURED REPORTING AS SORT OF FILL IN THE BLANKS, I'M PERSONALLY, YOU KNOW, OPPOSED TO STRUCTURED REPORT FOR CT ABDOMEN THAT GOES DOWN THE LIVER NORMAL, PELVIS ACUTE APPENDICITIS, OVARIES NORMAL. YOU'VE GOT THE IMPORTANT FINDING BURIED SOMEWHERE IN THERE. BUT ON THE OTHER HAND, GETTING TO STRUCTURED REPORTING WHERE THE OUTPUT OF ALGORITHM WILL BE STRUCTURED AND HOW IT FITS INTO OUR REPORTING SYSTEM, SO I THINK ONE OF THE CHALLENGES FOR CLINICAL INFORMATION IS AS WE DEVELOP USE CASES FOR A.I. AND WE KNOW THE INFORMATION WE NEED, SO WHERE IS THAT INFORMATION GOING TO LIVE? AND IT MAY BE THAT AS WE'RE DEVELOPING A.I. ALGORITHMS WE OUGHT TO CREATE MEANINGFUL STRUCTURED REPORTS WITH A BASIN, IF YOU WILL, FOR THE OUTPUT OF ALGORITHM TO LIVE IN SO I THINK STRUCTURED REPORTING WILL BE MORE UPON AS WE LOOK TOWARD CLINICAL INTEGRATION. >> ONE MORE COMMENT, EXACTLY I THINK IF YOU NEED THE STRUCTURED DATA ON THE INPUT, AS CURT HAD SAID, YOU KNOW, NEED LESS DATA IF YOU HAVE HIGHLY ACCURATE DATA. A BIG ADVANTAGE TO THAT. THE QUESTION IS DO YOU SLOW DOWN THE RADIOLOGIST TO CAPTURE THAT DATA? THE OTHER THING THAT'S GOING TO HAPPEN OR IS NEW IN A.I. WITH STRUCTURED REPORTING NOT JUST PIXEL BUT DETERMINED BY ALGORITHM, A SENSE OF STRUCTURE, IF YOU MAKE MODIFICATIONS TO THAT YOU'RE ESSENTIALLY CHANGING THAT STRUCTURE, IDENTIFYING THE AREAS WITHIN THE REPORT THAT YOU WANT TO MODIFY. NOW YOU HAVE THIS KIND OF INHERENT STRUCTURE THAT HAS BEEN SET UP BY A.I. PROBABLY FOR VALIDATION OR IMPROVEMENT OF ALGORITHMS IN THE WILD, IF YOU WILL. WE'RE GOING TO SEE ANOTHER OPPORTUNITY TO CREATE STRUCTURE FROM SEMI STRUCTURED REPORTS THAT WE HAVE FROM A.I. AS OPPOSED TO TODAY TRYING TO START FROM JUST NOTHING, AND THEN FORCING THE RADIOLOGIST TO CREATE STRUCTURE. >> THANK YOU. WHO IS NEXT? GO AHEAD. >> CAN I GO AHEAD? >> SURE, PLEASE. >> OKAY. I WANT TO RAISE THE QUESTION ABOUT HARMONIZATION. THE LAST SPEAKER TALKING ABOUT MR CASE, YOU CAN DO THE DEEP MENDING IN CASE SPACE, ALSO IMAGING SPACE, SO EVEN IN CASE BASE YOU COULD HAVE A NUMBER OF WAYS, CARTESIAN OR A NUMBER OF WAYS. SO THEN FROM DIFFERENT MECHANISMS HOW AM I GOING TO GET CONSISTENT RESULTS, THAT'S ONE THING. ALSO, THERE'S SO MANY -- THIS ALSO GENERATES SO MANY WAYS TO GENERATE DATA, GENERATE PROCESS, HOW TO HARMONIZE. WE NEED QUANTITATIVE BIOMARKERS FOR IMAGING, RIGHT? >> YEAH, SO GOOD QUESTION. SO I THINK THAT WE -- FIRST OF ALL, WE NEED BETTER WAYS TO MEASURE IMAGE QUALITY, BEFORE WE GET TO, YOU KNOW, IS THERE IS A NODULE OR IS THERE NOT A NODULE, WE WANT TO GET TO IS THIS M.R. IMAGE OF APPROPRIATE QUALITY SIGNAL TO NOISE AND THE LIKE, PARTICULARLY AS WE GET MORE ADVANCED ALGORITHMS WE'LL NEED METHODS TO EXCHANGE QUALITY MEASUREMENT ALGORITHMS AND EXCHANGE IMAGES OF THE TYPE THAT YOU'RE DESCRIBING TO HELP KIND OF LEVEL THE PLAYING FIELD ACROSS ALL THE ALGORITHMS BEING DEVELOPED. WITH RESPECT TO GROUND TRUTH, I'M NOT SURE IF THIS WAS PART OF YOUR QUESTION BUT WE THINK OF TWO DIFFERENT CATEGORIES. THINK OF A LUNG NODULE. IS THE LUNG NODULE PRESENT? THAT'S SOMETHING THAT THE RADIOLOGIST WOULD KNOW AND WOULD HAVE IN THE REPORT SO THE RADIOLOGY REPORT IS A GOOD SOURCE OF TRUTH FOR THAT. AND WE'D LIKE AN ALGORITHM THAT PERFORMS OF THE LEVEL OF RADIOLOGIST IN DETECTING NODULES, MAY WANT MULTIPLE RADIOLOGISTS. THEN IS THE NODULE CANCER? RADIOLOGIST DOESN'T KNOW THE ANSWER TO THAT QUESTION TYPICALLY AT THE TIME OF THE SCAN. THAT'S WHERE YOU NEED TO GO TO THE CHART AND THERE ARE A LOT OF GOOD A.I. ALGORITHMS THAT INGEST THE CHART AND LOOK AT WHETHER OR NOT A PARTICULAR DIAGNOSIS CAN BE ASSIGNED, DIGITAL PHENOTYPING, ANOTHER FORM OF TRUTH WE WOULD USE TO VALIDATE DIFFERENT IMAGING ALGORITHMS. >> OKAY. >> GOING BACK TO YOUR QUESTION ABOUT GROUND TRUTH AND NEED FOR DIFFERENT SAMPLING, THAT'S DEFINITELY THE CASE IF YOU HAVE PARTICULAR SAMPLING SET, CASE BASE, YOU NEED GROUND TRUTH FROM THAT. THERE'S A LOT OF TECHNIQUES BEING DEVELOPED TO MAP BETWEEN DIFFERENT DOMAINS, FOR INSTANCE, DIFFERENT ACQUISITION TYPES. SO YOU CAN ACTUALLY LEARN HOW THE DATA MIGHT LOOK LIKE IF IT WERE ACQUIRED IN DIFFERENT CONDITIONS. GIVEN SUFFICIENT AMOUNT OF DATA YOU CAN LEARN SOME OF THESE MAPPINGS USING NEWER TECHNIQUES SO YOU CAN PUT SOME AMOUNT OF PAIRED DATA, LEARN THAT AND REDUCE AMOUNT OF ACQUISITION SPECIFIC DATA YOU MIGHT NEED. >> I JUST WANT TO MAKE ONE MORE COMMENT, AS PEOPLE ARE ASKING. THAT BASICALLY MEANS WE NEED TO DO BETTER MEASUREMENT, THAT MEANS IN THE PHYSICAL MEASUREMENT ONE THING, ANOTHER THING ALSO BUILD UP LIKE DIGITAL SOMETHING SO TWO WAYS TO GET A CONSISTENT RESULT, RIGHT? >> BUT THAT'S TRUE, ALSO IF YOU HAVE GOOD SIMULATORS, SO FOR VARIOUS THINGS, SOME OF THAT CAN BE USED FOR DATA AUGMENTATION. IF YOUR MODEL IS STRONG, NOT USUALLY THE CASE IN REALITY BUT YOU CAN ALWAYS USE PHYSICS BASED MODELS TO AUGMENT DATA TO TRY TO UNCOVER SOME OF THAT. BUT DIGITAL PHANTOMS, THINGS LIKE THAT ARE VERY USEFUL. >> THANK YOU. GO AHEAD. >> DAVID McMULLEN, NATIONAL INSTITUTE OF MENTAL HEALTH. THE EXAMPLES YOU GIVE FALL INTO I GUESS WHAT THE FDA WOULD TERM CLINICAL DECISION SUPPORT THAT IS PROVIDING RECOMMENDATIONS BUT YOU STILL HAVE A DOMAIN EXPERT THAT CAN USE HUERISTICS. WHAT ABOUT WHETHER THEY WON'T BE ABLE TO INTERPRET THE DATA, MACHINE LEARNING, DATA FROM SCANS, GET THIS TREATMENT OR THAT, BUT YOU CAN SHOW IT TO IT TO A PSYCHIATRIST OR RADIOLOGY NOBODY WOULD BE ABLE TO POINT TO ANYTHING, HOW WILL THAT BE USED IN THE FIELD? THAT >> THAT GETS TO SOMETHING WE DIDN'T TALK ABOUT, EVEN IN RADIOLOGY WE HAVE ISSUES LIKE THAT. RADIOGENOMICS WHERE WE'RE TRYING TO MAKE A CORRELATION BETWEEN THE IMAGE AND SOME GENOMIC SIGNATURE, AND THAT INFORMATION ISN'T NORMALLY SOMETHING THE RADIOLOGIST EVEN HAS AT HAND AND WOULDN'T HAVE A CHANCE TO MAKE THAT CORRELATION OVER TIME. YEAH, THOSE ARE VERY INTRIGUING, I WOULD SAY KIND OF USING A BASEBALL ANALOGY, SWING FOR THE FENCES LIKE WE'RE NOT SURE OF THE MOST OF THE A.I. EXPERTS WE TALKED TO IN OTHER INDUSTRIES SAY, FOR EXAMPLE, ANDREW ING LIKES TO SAY A.I. IS GOOD FOR THINGS THAT HUMANS ALREADY KNOW HOW TO DO AND CAN MAKE A DECISION IN TWO SECONDS. SO IF YOU'RE HOPING AN A.I. ALGORITHM IS GOING TO FIND SOMETHING WE HAVEN'T THOUGHT OF THAT WOULD BE COOL AND MENTAL HEALTH IS ONE AREA WE MAY SEE THAT FIRST. INTERESTING AREA. >> I WOULD ADD, THAT'S WHAT I WAS TRYING TO ILLUSTRATE WHERE THE HAD THE ARROWS GOING WITH HUMANS AND THEN GOING AGAINST HUMANS. I DO THINK OF IT THAT WAY IF WE ALREADY HAVE PROVEN SCIENCE THAT SAYS THAT IF YOU MAKE THESE FINDINGS THAT HERE'S THE STATISTICS THAT SHOW THERE'S CORRELATIVE DISEASE, SO WHAT YOU'RE DOING IS REPLACING REPLACING ABILITY TO DETECT OR AUGMENT OR ENHANCE, THERE'S THINGS WE CAN DO THAT WE COULDN'T DO BEFORE, NEW SCIENCE THAT REQUIRES PROSPECTIVE STUDIES, I THINK A LOT OF PEOPLE FORGET WHEN WE COME UP WITH NEW DEVICE THERE'S PROCESSES THAT TAKE PLACE AFTER THE DEVELOPMENT OF THOSE DEVICES. >> THANK YOU. >> I MIGHT JUST ADD, ONE THINK WE CAN DISCUSS AS WE GO ALONG, WHAT IS GOING TO BE THE ROLE OF TRANSPARENCY IN THE ALGORITHMS TO NOT JUST RADIOLOGISTS BUT TO THE WHOLE MEDICAL COMMUNITY, HOW ARE WE GOING TO BE ABLE TO UNDERSTAND THAT ALGORITHM JUST DOESN'T TAKE AN IMAGE AND SAYS 85% OF LUNG CANCER WITHOUT NECESSARILY IDENTIFYING A NODULE AND CHARACTERIZING A NODULE AND SO FORTH. SO I THINK FOR US TO GET -- AND THIS IS, YOU KNOW, FOR US TO BE ABLE TO GET A.I. INTO CLINICAL PRACTICE, QUICKLY, OR MORE QUICKLY, I THINK WE'RE GOING TO HAVE TO BE ABLE TO HAVE THE ALGORITHM DEVELOPERS BE ABLE TO SHOW THAT TRANSPARENCY AND EXPLICABILITY, TO TELL US THINGS WE WOULDN'T OTHERWISE SEE OR PERCEIVE FROM IMAGES OR DATA THAT GOES TO MAKE UP THOSE IMAGES, AND SO I THINK IT'S BOTH, BUT SHORT-TERM EXPLICABILTY AND TRANSPARENCY IS NECESSARY TO BE ACCEPTED IN THE MEDICAL COMMUNITY. >> ONE LAST QUESTION. >> I WAS THINKING ABOUT OTHER DIMENSION, ARTIFACTS. SO WHEN YOU HAVE NEW MACHINE LEARNING RECONSTRUCTION ALGORITHMS, THROWING ARTIFACTS YOU HAVE NOT SEEN BEFORE, I BELIEVE RADIOLOGISTS ARE TRAINED TO SEE USUAL ARTIFACTS BUT WE SEE NEW ARTIFACTS NOT SEEN BEFORE HOW DO YOU GET ALONG THAT AND WHAT IS SOLUTION FOR THE END USER OF IMAGES? >> YEAH, SO THAT GETS TO THE NEED FOR TRAINING. SO HEARKENING BACK TO THE DAYS OF M.R. WHEN IT FIRST CAME OUT PEOPLE SAID, YOU KNOW, THE RADIOLOGIST ISN'T GOING TO BE NECESSARY, ABNORMALITIES ARE SO CLEAR. THERE WAS ONE VENDOR AT THE TIME SAID WE'RE NOT GOING TO SELL TO RADIOLOGISTS, WE'LL SELL TO GENERAL PRACTITIONERS, PATIENTS WILL SLIDE THROUGH AND BOOM. WELL, DIDN'T TURN OUT THAT WAY, RIGHT? DEVICES ARE COMPLICATED WITH PLUSES AND MINUSES, AND RADIOLOGISTS OVER TIME HAVE REALLY OWNED THE TRAINING ABOUT M.R. PHYSICS. RADIOLOGISTS CAN'T BUILD AN M.R. SCANNER BUT KNOW WHAT ARTIFACTS MIGHT OCCUR TO BE CLINICAL SIGNIFICANT, THE SAME PROCESS NEEDS TO HAPPEN FOR A.I., IT'S GOING TO BE OUR TOOL, WHEN IT'S APPROPRIATE TO USE AND WHEN NOT TO USE IT, AND THAT'S ALL ABOUT TRAINING AND EDUCATION OF THE CLINICIANS WHO USE THESE ALGORITHMS. >> THANK YOU. >> I WOULD ADD THAT THAT'S WHAT I WAS TRYING TO SHOW WITH THE FRIED CHICKEN ANALOGY IS THAT IF YOU'RE LOOKING AT THAT, YOU KNOW THAT IT'S NOT A GOLDEN DOODLE. YOU KNOW IT'S FRIED CHICKEN. IT'S GOING TO BE THE SAME THING THAT SOME OF THESE ARE CLEARLY SUBTLE. YOU CAN PUT NOISE IN HUMANS CAN'T SEE BUT MOST TIMES IT'S THE CASE WHERE YOU GO THIS ALGORITHM NEVER WORKS IN THE CORNER OF THE LUNG OR IN THE BASE OR IN THIS QUADRANT, BECAUSE OF THE HUERISTICS YOU BUILD WITH TIME WITH THESE ALGORITHMS THAT YOU SEE. THE CHALLENGE HERE, I THINK OF THESE AS MORE PERCEPTUAL ARTIFACT AS OPPOSED TO PHYSICS ARTIFACTS, A HIGHER ORDER OF MAGNITUDE IN THE DATA AND KNOWLEDGE WE'RE CREATING. THE CHALLENGE WILL BE THAT M.R. COMES OUT AFTER CT, AFTER ULTRASOUND, DECADES TO CREATE. THESE ALGORITHMS WILL TAKE WEEKS TO CREATE SO YOU HAVE TO BE COMFORTABLE WITH WHAT ARTIFACTS ARE BEING CREATED BUT IT WILL ALWAYS BE THE CASE A CLINICIAN OF SOME SORT, I THINK RADIOLOGISTS, THAT'S GOING TO SEE THESE ALL THE TIME, PROBABLY GOING TO BE MORE KNOWLEDGEABLE OF THOSE PER CEPTIVE ARTIFACTS TO MAKE SURE YOUR INSTITUTION POPULATES, IDENTIFIES THOSE AND THOSE ARE ARTICULATED ACROSS THE DISCIPLINE. >> THANK YOU. WE'RE GOING TO STOP NOW AND START THE NEXT SESSION AT EXACTLY 10:00. THANK YOU ALL FOR YOUR COMMENTS. [APPLAUSE] ALL RIGHT. AS YOU MAKE YOUR WAY FORWARD, I'LL INTRODUCE MYSELF, I'M BRAD ERICKSON, RADIOLOGIST AT MAYO CLINIC, AND MY CO-CHAIR IS JAYASHREE KALPATHY-CRAMER, AT HARVARD, YOU'RE ALREADY MET HER. WE'LL TALK ABOUT GAPS IN FOUNDATIONAL RESEARCH IN MACHINE LEARNING, AND I'M ACTUALLY ALSO GOING TO BE THE FIRST SPEAKER IN THIS SESSION, AND I'M GOING TO TALK ABOUT CORE INFRASTRUCTURE NEEDS FOR MACHINE LEARNING RESEARCH. AND I'M ACTUALLY GOING TO HAVE MY TALK GO RELATIVELY SHORT, JUST TO ALLOW MORE PANEL TIME. AS I THOUGHT ABOUT THIS TOPIC MORE AND MORE, I REALIZED THIS IS ALMOST A RELIGIOUS DISCUSSION, AND I THINK I'M GOING TO PRESENT JUST A VERY HIGH LEVEL OF BASIC CONCEPTS, AS YOU DRILL DOWN TO HOW YOU DO IT IN PRACTICE, EVERYBODY IS DOING THIS, I THINK THERE HAS PROBABLY NOT BEEN A LOT OF CONSENSUS OR COMMON TOOLING FOR CORE INFRASTRUCTURE OF MACHINE LEARNING, SO AS A CONSEQUENCE WE BUILT OUR OWN TOOLS TO REFLECT PRACTICES, DATA SOURCES, BIASES, THE BASIC INFRASTRUCTURE YOU THAT WE HAVE AT OUR INSTITUTION, SO I'M ACTUALLY GOING TO ENCOURAGE A LOT OF DISCUSSION AT THE END AND SAY ERICKSON, YOU'RE FULL OF IT, WE NEED IT BECAUSE OF DISTANCE BUT I'M GOING TO PRESENT BASIC CONCEPTS AND SOME THINKING WE HAD THAT WE USED WHEN WE BUILD SOME OF THE TOOLING. SO, FROM MY PERSPECTIVE, YOU'VE SEEN LOTS OF CYCLES OF CLINICAL PRACTICE. I THINK THERE IS AN A.I. PROJECT LIFE CYCLE. AT SOME POINT YOU START WITH PROJECT SELECTION, SOMEWHERE YOU HAVE TO HAVE AN IDEA OF WE'RE GOING TO FOCUS ON DOING THIS ONE THING. YOU PERHAPS HAVE SOME PRIORITIZATION PROCESS, I'M NOT SURE, BUT AT SOME POINT YOU HAVE TO SAY THIS IS THE PROJECT WE'RE GOING TO DO. AND AT THAT POINT THEN SOMEBODY TYPICALLY WOULD WRITE A PROPOSAL UP, SOME SORT OF RESEARCH DOCUMENT, IN MOST CASES YOU HAVE TO GO TRUE YOUR INSTITUTIONAL REVIEW BOARD, AND THEN TYPICALLY THAT IS YOUR STARTING CHARTER OR JUST DESCRIPTION OF WHAT YOU'RE GOING TO FOCUS ON. AT THAT POINT THEN YOU NEED TO START COLLECTING THE DATA. AND THAT MAY BE TEXTUAL DATA. WE'LL HEARD ABOUT NLP, WE'RE HEAR MORE. THIS IS THE IN THE NIBIB SO WE'RE GOING TO TALK ABOUT IDENTIFICATION OR PSEUDOIDENTIFICATION TO CONNECT IT BACK BUT WE WANT TO MAKE IT REALLY HARD FOR ANYBODY WORKING ON THE RESEARCH DATA TO ACTUALLY FIGURE OUT WHO THAT PERSON IS. AND YOU'LL SEE AT THE CENTER HERE I HAVE A CONTENT MANAGEMENT SYSTEM, AND I USE THAT IN A VERY GENERIC SENSE. WE ACTUALLY IN OUR SITUATION DO USE A TRUE CONTENT MANAGEMENT SYSTEM, BUT I'M USING THAT MORE IN THE GENERAL SENSE OF SAYING WE NEED TO HAVE SOMETHING THAT MANAGES ALL THE INFORMATION THAT COMES IN FROM ALL THESE VARIOUS POINTS AND CAN ALSO FEED BACK OUT. DATA GOES INTO THE MANAGEMENT SYSTEM OR DATABASE IF YOU WANT TO THINK OF IT THAT WAY. THEN TYPICALLY YOU NEED TO DO A Q.C. STEP ON THE DATA THAT COMES IN. YOU WANT TO MAKE SURE IMAGES ARE OF GOOD QUALITY. IT'S FAIRLY COMMON IN CLINICAL PRACTICE BUT IF THE PATIENT MOVES A LOT, THAT YOU REPEAT THAT SCAN. SO DO YOU WANT BOTH OF THE, FOR INSTANCE, LET'S SAY THEY MOVED A LOT ON THE T2 AXIAL, DO YOU WANT THE MOTION OR THE GOOD ONE THAT CAME AFTERWARDS? IN OUR CASE WE TYPICALLY FLAG THE MOTION SEQUENCE AS DO NOT USE, THEY ARE STILL AVAILABLE IN CASE WE SHOULD EVER WANT THEM, BUT TYPICALLY FOR TRAINING ALGORITHMS WE WANT THE BETTER QUALITY IMAGES AVAILABLE. SO THERE'S SOME SORT OF Q.C. STEP. YOU WANT TO MAKE SURE YOU REALLY HAVE PROPERLY CONNECTED THE TEXTUAL COMPONENT WITH THE IMAGE COMPONENT. NEXT STEP IS ANNOTATION, IF YOU'RE WORKING ON SEGMENTATION SOMEBODY HAS TO SAY THIS IS THE LIVER, THIS IS BRAIN, THIS IS THE TUMOR, SOMETHING LIKE THAT. AND SO THAT ANNOTATION STEP IS, IN MY EXPERIENCE, PROBABLY THE MOST LABOR INTENSIVE PIECE. WE TALKED ABOUT HOW HARD IT IS TO GET THE DATA IN, BUT FOR A LOT OF PROJECTS I SEE ANNOTATION AS BEING THE MOST CHALLENGING THING TO DO SO YOU'LL HEAR A LITTLE BIT MORE ABOUT THAT. ONCE ANNOTATION IS DONE THAT'S WHERE WE GET TO THE THING WE TYPICALLY THINK OF WITH MACHINE LEARNING, WHICH IS THE STEP OF TRAINING THE ALGORITHM, AND FOR THAT YOU TYPICALLY, ESPECIALLY NOW WITH DEEP LEARNING, YOU NEED A GOOD COMPUTE ENGINE BEHIND IT. AND THEN AFTER YOU'VE DONE THAT TRAINING STEP, YOU NEED TO DO SOME SORT OF REVIEW AND VALIDATION. YOU NEED TO FIGURE OUT AM I REALLY GETTING THE RESULTS THAT I EXPECT TO SEE? AND THEN YOU WOULD WANT TO THINK ABOUT CLINICAL DEPLOYMENT, HOW AM I ACTUALLY GOING TO PUT THIS INTO PRACTICE? AND YOU KNOW, OBVIOUSLY IF YOU HAVE A GREAT BONE AGE ALGORITHM AND FEED CHEST X-RAYS, IF YOU FEED A BONE AGE ALGORITHM A CHEST X-RAY IT WILL COME OUT WITH AN AGE OF THAT CHILD. AND SO THAT'S A PROBLEM IF YOU DON'T HAVE YOUR WORK FLOW SET UP PROPERLY. OKAY. SO IT'S IMPORTANT TO THINK ABOUT ALL THESE STEPS AS YOU BUILD YOUR ALGORITHMS AND FROM THE VERY START OF THE PROJECT SELECTION IF YOU DON'T HAVE A WAY TO DEPLOY THAT INTO THE CLINIC THAT'S GOING TO BE A PROBLEM. IF YOU DON'T HAVE A WAY TO ACCURATELY ANNOTATE THE DATA, IF YOU DON'T HAVE A WAY TO GET ENOUGH IMAGES OR TEXT ANNOTATIONS, THAT'S A PROBLEM. SO YOU HAVE TO THINK ABOUT ALL OF THESE STEPS WAY BACK AT THE PROJECT SELECTION STEP. SO LET'S TALK ABOUT THE DATA IN STEP. THERE'S DICOM, WE'RE FORTUNATE IN RADIOLOGY TO HAVE LARGE STORES OF DICOM IMAGES, TYPICALLY THAT IS A RECEIVER, REMOVAL OF PATIENT ID, PUT A MAPPING TO ANOTHER ID SO IF YOU EVER WANT TO LOOK BACK OR CONNECT IT TO CLINICAL DATA YOU NEED TO HAVE THAT MAPPING. AND THEN SOMETHING THAT'S A LITTLE BIT MORE CONTROVERSIAL IS DO YOU KEEP IT AS DICOM OR DO YOU CONVERT TO SOME OTHER FORMAT? A LOT OF LABS INCLUDING MINE WE CONVERT IT TO NIFTI, WE KEEP THE DICOM AS TAGS BUT WE FIND IT MORE FACILE TO USE NIFTI, TOOLS OUT THERE A LOT ACCEPT NIFTI, NOT DICOM, THAT'S A RELIGIOUS QUESTION HOW YOU WANT TO HANDLE YOUR DATA. IN TERMS OF TEXT RESULTS AND LAB RESULTS, WE HAVE A CLINICAL WAREHOUSE FACILITY, CAN YOU DO SEARCHES AND BRING THAT BACK TO CSV FILES OR EXCEL OR STRUCTURED TEXT AND WE HAVE AN APPLICATION THAT MARRIES THAT BASED ON THE PSEUDONYMIZER WE USE WITH DICOM. THE DATA MANAGER, THE CENTRAL PIECE, NOT TO GET TOO RELIGIOUS AND SAY WE USE VENDOR X, Y OR Z, I THINK THERE'S SOME BASIC FUNCTIONAL REQUIREMENTS THAT EXIST. FOR IMAGE ANNOTATIONS YOU NEED TO KNOWED STRUCTURE, IF YOU'RE INTERESTED IN PATHOLOGY WHICH I SUSPECT 100% OF US ARE, WE NEED SOME WAY OF IDENTIFYING THE PATHOLOGY AND OF COURSE YOU NEED TO KNOW THAT PATHOLOGY IS SITTING IN THE LIVER BUT IF IT'S METASTASES IT COULD BE SITTING IN ANY ORGAN, SO SOME WAY TO REPRESENT THE FACT THAT YOU CAN HAVE A PIXEL THAT IS BOTH A LIVER AND A TUMOR, AND A NECROTIC PART OF THE TUMOR, THAT RICHNESS OF THE INFORMATION REPRESENTATION IS CRITICAL AND SO HAVING AN INFORMATION SYSTEM THAT SUPPORTS THAT RICHNESS OF LABELING IS CRITICAL. YOU NEED TO HAVE A WAY TO ASSOCIATE THE CLINICAL DATA AND, AGAIN, I TOLD YOU HOW WE DO THAT UP FRONT BUT POTENTIALLY YOU MAY FIND MORE INFORMATION LATER ON. IF YOU'RE FOR INSTANCE DOING CANCER STUDIES YOU WANT TO KNOW ABOUT SURVIVAL AND MAY COLLECT IMAGES TODAY AND YOU MAY KNOW ABOUT TREATMENT RESPONSE FOR INSTANCE BUT OFTENTIMES YOU WANT TO KNOW ABOUT OVERALL SURVIVAL SO SOME WAY TO UPDATE THE SYSTEM AND BE AWARE EVENTS HAPPEN AFTER SOME INITIAL INGEST PROCESS IS ALSO IMPORTANT. THE THIRD REQUIREMENT THAT I THINK IS PRETTY MUCH UNIVERSAL IS THAT, AGAIN, THERE'S SOME NEED FOR COMPUTE, AND THAT COMPUTE ENGINE SHOULD INTEGRATE WELL WITH YOUR MANAGEMENT SYSTEM. IN OUR CASE, YOU KNOW, IF WE'RE DOING GENOMICS STUDY FOR INSTANCE WE WANT TO IDENTIFY THE PTIENTS THAT HAVE DISEASE X, GLIOMAS, WE WANT TO GET THEIR PREOPERATIVE MRI SCANS, AND SPECIFICALLY WE WANT LET'S SAY OF DIFFUSION IMAGES. WE USE PYTHON AND CAN CREATE A QUERY AND ALL THE RESULTS THAT MATCH THAT INPUT REQUIREMENT COME OUT AS A LARGE PYTHON LIST, AND THAT PYTHON LIST GETS FED TO OUR PROCESSING PIPELINE AND DOWN THE PIPELINE, RELATIVELY STRAIGHTFORWARD BUT FUNCTIONAL INTERFACE BETWEEN THE DATA SYSTEM AND PROCESSING PIPELINE. DATA CURATION, AGAIN, THIS IS SOMETHING THAT YOU HAVE TO DO WELL BECAUSE WHILE, YEAH, YOU CAN GET BY WITH NOISY DATA IF YOU HAVE ENOUGH OF IT, I'M ONE WHO AT LEAST THE DISEASES WE TEND TO WORK WITH WE'RE FAIRLY DATA LIMITED, JUST YOU CAN'T CREATE CANCER JUST BY HAVING NLP SO WE ONLY HAVE SO MANY CASES AND SO I'M A STICKLER FOR DOING HIGH QUALITY DATA CURATION TO MAKE SURE WE REALLY UNDERSTAND WHAT'S GOING ON. IT'S AMAZING HOW MUCH GARBAGE THERE IS IN NOTES OR HOW TEST RESULTS SEEM TO GO BACK AND FORTH OR PEOPLE MISUNDERSTAND EACH OTHER, AND SO UNDERSTANDING VERY WELL WHAT DATA YOU HAVE AND THE CORRECT ANNOTATION IS REALLY CRITICAL. AND SO WE HAVE A VIEWING TOOL TO MAKE SURE THAT THE IMAGES THAT THE RIGHT IMAGES ARE THERE, THAT WE DON'T WORK ON DATA THAT SHOULD BE THROWN OUT, THAT WE HAVE THE CORRECT ASSOCIATION BETWEEN IMAGES AND THE TEXT, AND WE DO FULL LOGGING OF WHO DID THAT, WHAT THEY DID IN TERMS OF SAYING, WELL, REALLY THIS GOES WITH THIS PATIENT AND WHEN THEY DID IT. IT'S AMAZING HOW MUCH ALMOST MIGRATION THERE IS OF TEST RESULTS AND HOW THEY DID IT. MGMT METHYLATION, THRESHOLDS FOR DECIDING IT WAS POSITIVE OR NEGATIVE, FOR INSTANCE, IS ONE THAT HAS MIGRATED AT MY INSTITUTION. SO KNOWING WHEN SOMEBODY LABELED SOMETHING WAS VALUABLE BECAUSE THEN WE COULD GO BACK AND SAY, WELL, FROM THIS DAY BACK, WE NEED TO READJUST SOME OF OUR DATA LABELING. THE PROCESSING PIPELINE IS THE THING THAT GETS ALL THE APPROPRIATE IMAGE DATA, ASSOCIATED METADATA, THIS IS FOR THOSE OF US WHO ARE COMPUTER GEEKS. WE REALLY GET TO GEEK OUT AND SAY, YOU KNOW, OH, I'VE GOT SO MANY CORES OF GPU SERVER AND ALL THAT BUT, YOU KNOW, THERE'S A LOT MORE PRACTICAL COMPONENT HERE. ONE IS THAT I THINK THE MORE YOU CAN BUILD MODULAR TOOLS SO THIS IS THE WHOLE CONCEPT OF A PIPELINE THAT YOU DON'T HAVE ONE MONOLITHTIC APPLICATION THAT DOES EVERYTHING BUT TYPICALLY YOU HAVE MULTIPLE STEPS. SO FOR INSTANCE IF YOU'RE DEALING WITH MULTIPLE IMAGES YOU MAY NEED TO DO IMAGE REGISTRATION. IF YOU'RE DEALING WITH MRI, AT LEAST IN MY EXPERIENCE, YOU ALMOST ALWAYS HAVE TO DO SOME SORT OF A CORRECT UNIFORMITY CORRECTION, AS WELL AS INTENSITY NORMALIZATION, SO LIKE WE USE N-4 FOR THAT, AND SO I THINK THE MORE THAT YOU CAN USE TOOLS THAT ARE WELL ESTABLISHED FOR SOME OF THOSE PROCESSING COMPONENTS BEFORE YOU GET INTO YOUR GBU CLUSTER STUFF I THINK THAT'S A VALUABLE THING AND MOST PEOPLE ARE THERE. IT'S JUST A QUESTION OF HOW YOU IMPLEMENT, WHETHER IT'S A BASH SCRIPT, THE CONTENT MANAGEMENT SYSTEM HAS A WORKFLOW BUILD INTO IT MANAGING THE PIPELINE AND CAN RECOVER FROM ERRORS AND THAT SORT OF THING. SCALABILITY, THIS IS SOMETHING WHERE YOU HAVE TO FIGURE OUT ARE WE GOING TO RUN THIS ON PREMISES OR ARE WE GOING TO RUN IT AGAINST AMAZON OR GOOGLE CLOUD OR SOMETHING LIKE THAT, AND WHAT ARE THE ISSUES OF MOVING DATA UP TO THE CLOUD, STORING IT THERE, AND THAT PROBABLY REFLECTS A LITTLE BIT HOW YOUR INSTITUTION THINKS ABOUT SECURITY AND ALSO HOW THEY THINK ABOUT FINANCING SOME OF THESE RESOURCES. AND THEN IT ALSO NEEDS TO BE REASONABLY SECURE. A LOT OF US USE DOCKER FOR DOING THE COMPUTE. YOU PROBABLY KNOW THAT DOCKER TENDS TO RUN AS ROOT, YOU HAVE TO HAVE FULL ACCESS AND ANYBODY WITH A DOCKER HAS FULL ACCESS TO THE DATA. WE DID HAVE ONE CATASTROPHIC ACCIDENT WHERE SOMEBODY PUT AN RM-RF AND DIDN'T PUT THE PROPER PATH IN, DELETED ALL OUR DATA. WE HAD AN UNINTENDED TEST OF OUR BACKUP SYSTEM, FORTUNATELY IT WORKED. BUT THAT IS A PROBLEM. SO WHEN I'M TALKING ABOUT THE SECURITY, YOU KNOW, WE OFTEN THINK ABOUT PRIVACY AND CONFIDENTIALITY OF PATIENTS, BUT IN THE RESEARCH REALM I THINK ESPECIALLY WHERE WE TEND TO HAVE NEW PEOPLE COMING INTO THE LAB AND ROTATING THROUGH, THEY ARE THE ONES WHO CAN MAKE THE LOGICAL ERRORS THAT CAN SOMETIMES PRODUCE BIG PROBLEMS. SO HAVING AN ARCHITECTURE WITHIN YOUR PIPELINE THAT CAN PARTITION PEOPLE OFF I THINK IS VALUABLE. RESULTS VISUALIZATION, YOU NEED SOME WAY TO EXPLORE RESULTS. PARTICULARLY UNDERSTANDING THE INTERMEDIATE STEPS TO SEE DID I REALLY GET THE PIPELINE STEP, DID I GET THE RESULT FROM THAT STEP IN THE PIPELINE THAT I THOUGHT WAS GOING TO BE THERE. IF YOU'RE GOING TO DO MULTI-SITE COLLABORATION HOW ARE YOU GOING TO PROVIDE A WAY FOR AN EXTERNAL SITE TO SEE INTO THE RESULTS THAT YOU'RE GETTING. AND THEN OF COURSE FINAL RESULTS ARE REALLY IMPORTANT TOO. AND HOW DO YOU EXPORT THE RESULTS? DOCUMENTATION, WE USED A LOT OF PYTHON, NOW CALLED JUPITER, A GOOD WAY TO DOCUMENT THE CODE, THE INPUT AND RESULTS AND I THINK THAT'S BECOMING PRETTY POPULAR AS WELL. PUBLICATION TOOLS, AGAIN, THERE ARE A LOT OF GOOD LIBRARIES NOW THAT PRODUCE PUBLICATION-QUALITY DIAGRAMS AND I THINK THAT'S VALUABLE IF YOU CAN BUILD THAT IN BECAUSE THAT WAY ONCE THE FIRST GRAD STUDENT LEARNS HOW TO DO IT THEY CAN THEN SHARE WITH EVERYBODY ELSE IN YOUR LAB RATHER THAN EVERYBODY ELSE HAVING TO LEARN IT ON THEIR OWN. SO THAT'S THE CORE INFRASTRUCTURE. IF YOU LOOK AT THE PROGRAM ALMOST ALL THE OTHER SECTIONS HAVE FOUR SPEAKERS. OUR SECTION HAD THREE. THAT'S PARTLY BECAUSE WE HAVE JAYASHREE, WHO IS EXTRA SPECIAL BUT SOMEBODY CANCELED AT THE LAST MINUTE, THE TOPIC WAS EXPLAINABLE A.I. RATHER THAN TRYING TO FIND SOMEBODY WE DECIDED EACH WOULD PRODUCE A LITTLE BIT OF CONTENT ON EXPLAINABLE A.I. SO EXPLAINABLE A.I. IS WHAT YOU HAVE HEARD A LITTLE BIT ABOUT, YOU KNOW, DEEP LEARNING HAS THIS BLACK BOX, WE CAN'T EVER UNDERSTAND WHAT THIS THING IS SEEING. AND THAT HAS BEEN RECOGNIZED AS AN IMPORTANT PROBLEM. THIS PARTICULAR THREE-LETTER ACRONYM IS FROM THE DoD. THEY SEEM TO BE ONE OF THE GROUPS THAT IS REALLY PUSHING UNDERSTANDING WHAT A.I. IS LEARNING ABOUT. AND SO I'M GOING TO PRODUCE A LITTLE BIT OF -- OR TALK A LITTLE ABOUT WHAT WE'VE WORKED ON TRYING TO UNDERSTAND WHAT THE ALGORITHMS ARE SEEING. I THINK INCREASINGLY TOOLS WILL MAKE IT SO DEEP LEARNING IS NOT A BLACK BOX, PROBABLY MORE CORRECTLY CALLED AN OPAQUE BOX AND WE HAVE TO LEARN HOW TO SEE INSIDE OF IT. AND ONCE WE DO THAT, THE REASON FOR THIS IS THAT IT PROVIDES CONFIDENCE IN THE PREDICTIONS, BUT I THINK ALSO CAN PROVIDE INSIGHT INTO THE DISEASE OR THE MAGES. IF YOU LEARN THAT A CERTAIN FEATURE IS A TEXTURE THAT'S REALLY IMPORTANT, PROBABLY THAT MEANS YOU NEED TO BE VERY CONSISTENT IN HOW YOU ACQUIRE IMAGES AND MIGHT TWEAK YOUR ACQUISITION PROTOCOL TO FURTHER IMPROVE THE TEXTURE EXTRACTION CAPABILITIES OF THAT SEQUENCE. OR FOR INSTANCE IF YOU FIND THAT ON CT IT'S A CERTAIN ENERGY OR CERTAIN STRUCTURE, MAYBE YOU WANT TO DO THINNER RECON THROUGH THE AREA AND GET A BETTER SIGNAL TO NOISE AND BETTER PERFORMANCE FOR YOUR NETWORK. IT'S NOT JUST TO UNDERSTAND WHAT YOU'RE SEEING BUT CAN IMPROVE THE QUALITY OF IMAGES FROM THE COMPUTER'S PERSPECTIVE AND THAT MAY IMPROVE OUR ABILITY TO DO MEDICINE. SO, NOW, THIS IS KIND OF A NASCENT FIELD. SO I DON'T THINK WE REALLY HAVE A GOOD ONTOLOGY FOR DESCRIBING THE WAYS TO DO EXPLAINABLE A.I. ONE OF THE GENERAL CATEGORIES I THOUGHT MADE SENSE, TRIGGER BASIN-BASED METHODS WHERE YOU ALTER THE VALUE OF PIXELS TO SOME DEGREE. SO ONE SIMPLE THING, OCCLUSION. YOU CAN MARCH IT ACROSS THE IMAGE AND SEE WHAT IT DOES. IF YOU BLACK OUT PARTS IF PERFORMANCE DROPS THAT PART IS IMPORTANT TO THE ALGORITHM. IF DO YOU THAT ACROSS LARGE POPULATIONS, YOU CAN START TO FIGURE OUT, AH, THAT'S SOMETHING THAT'S REALLY IMPORTANT TO THE ALGORITHM. AS OPPOSED TO OTHER AREAS IF IT DOESN'T AFFECT PERFORMANCE IT'S NOT TELLING YOU ANYTHING. THAT'S ONE WAY IT HAS A COMMON SENSE APPROACH AND CAN PRODUCE WHAT SEEMED LIKE SENSIBLE ATTENTION MAPS. THAT'S DONE AT THE SHALL WE SAY THE DUMB PIXEL LEVEL, RIGHT? DOING A BLACK BOX, NOT THINKING ABOUT STRUCTURE WITHIN THE IMAGE. THE NEXT LEVEL WOULD BE WHAT SOMETIMES IS CALLED A SUPERPIXEL IMPLYING A WAY TO DEFINE SOME WAY THAT THESE PIXELS ARE ALL ALIKE AND THEN YOU'RE GOING TO BLACK THOSE OUT. SO IT MEANS YOU'RE NOT USING A SIMPLE GEOMETRICAL SHAPE, YOU'RE INSTEAD USING SOMETHING WHERE YOU HAVE AN INTUITION THAT A CERTAIN THING IS IMPORTANT. FOR INSTANCE, THIS IS ONE EXAMPLE WHERE THEY SAY, YOU KNOW, HOW CAN I TELL THE DIFFERENCE BETWEEN A WOLF AND HUSKY? WELL, IN THIS CASE, IT'S BECAUSE IT WAS ACTUALLY LOOKING AT THE SNOW. YOU CAN SEE THAT'S NOT A NICE RECTANGULAR SHAPE, BUT IT WAS ALL SIMILAR INTENSITY STRUCTURE. AND THIS WAS ACTUALLY TAKEN FROM THIS PAPER, AND IF YOU KNOW THIS REFERENCE YOU KNOW THIS REFERENCE SAYS DON'T DO THIS BUT I THOUGHT IT WAS A PRETTY PICTURE THAT SHOWED THE EXAMPLE. AND SO THE CHALLENGE WITH THIS TECHNIQUE OF COURSE IS YOU NEED TO UNDERSTAND SIMILARITIES. WE UNDERSTAND SIMILARITY IF SIMILARITY MEANS PIXEL INTENSITY, RIGHT? OR HOW BRIGHT IT IS. BUT FOR INSTANCE IN RADIOGENOMICS, YOU KNOW, I THINK WE'RE PROBABLY FINDING IT'S MORE OF A TEXTURAL SORT OF THING, YOU NEED TO FIGURE OUT TEXTURES SEEN AND COMPUTE TEXTURE AND SEE WHETHER THOSE ARE THE SIGNAL THE SYSTEM IS SEEING. NOW, THERE ARE OTHER TYPES OF METHODS, SO ANOTHER GENERAL CATEGORY IS BACK PROPAGATION-BASED METHODS, SALIENTCY MAPS, TRYING TO LOOK AT THE TRAINED NETWORK, AND YOU'RE TRYING TO LOOK WHERE THE GRADE CENTS ARE, GRADIENTS ARE, LARGE VALUES CONNECTING NODES, CRITIAL INFORMATION THAT'S WHAT'S BEING FOCUSED ON. THIS IS SOME OF THE WORK THAT WE'VE BEEN DOING. IN THIS CASE WE DID A FAIRLY DUMMY TASK, BASICALLY TRYING TO FIGURE OUT FROM THE CONTRAST ENHANCED ABDOMINAL CT FIGURE OUT THE PHASE OF CONTRAST. THERE'S PRE-CONTRAST, ARTERIAL, CORTICAL, MEDULLARY AND LATE PHASE. SO WE TRAIN THE SYSTEM IN ON THAT. THEN WE STARTED TO FIGURE OUT, OKAY, SO WHAT PART IS IT LOOKING AT? SO FOR AORTA YOU MIGHT GUESS IT'S LOOKING AT AORTA BUT THEN OUGHT TO BE LOOKING AT THE KIDNEYS THE REST OF THE TIME. THIS IS A COMPARISON OF SOME OF THE DIFFERENT TYPES OF BACK PROPAGATION METHODS OUT THERE. AND THEY EACH HAVE DIFFERENT STRENGTHS AND WEAKNESSES. SALIENCY MAPS MAY NOT BE INFORMATIVE, SOME ARE DIFFICULT TO COMPUTE, SUSCEPTIBLE TO NOISE. I TEND TO USE MULTIPLE METHODS AND SEE IF THEY ALL ALIGN, AND IF THEY ALL ALIGN THAT PROBABLY MEANS THAT YOU'RE GETTING REAL INFORMATION. ANOTHER TECHNIQUE IS LAYER-WISE RELEVANCE METHOD, DEEPLIFT IS ANOTHER ONE. I FORGOT TO GIVE CREDIT TO THAT FIRST PAPER, THE ROBERO PAPER IN ARCHIVE. THEY DEVELOPED ANOTHER TECHNIQUE THAT IS A -- NOW I'M BLANKING ON THE NAME. LIME. LOCAL INTERPRETATION OF MODEL AGNOSTIC EXPLANATION. WHAT THEY DO IS PICK OUT A NUMBER OF EXAMPLES AND SEE WHICH PIECES OF THOSE EXAMPLES, WHICH FEATURES ARE IMPORTANT TO THE OUTPUT OF THE IMAGE. AND THE -- ONE OF THE MANY DELIVER CLEVER THINGS IS LOOK FOR FEATURES WITHIN THAT WHOLE TRAINING SET, AND THEY FIND DIFFERENT EXAMPLES THAT EXERCISE DIFFERENT PARTS OF THE FEATURE SPACE. SO THAT YOU GET A BETTER SAMPLING OF EVERYTHING THAT'S OUT THERE. AND SO I THINK WE'RE STARTING NOW TO SEE TOOLS THAT WILL HELP US UNDERSTAND WHAT THESE DEEP LEARNING ALGORITHMS ARE REALLY LOOKING AT, NOT ONLY SPATIALLY, THAT'S SOMETHING WE'RE USED TO SEEING AS RADIOLOGISTS, BUT ALSO IN TERMS OF THINGS LIKE TEXTURES OR FREQUENCIES AND WHAT NOT THAT ARE MUCH MORE CHALLENGING FOR US TO REPRESENT. AND SO I THINK THAT THAT'S AN INTERESTING WAY TO START TO EXPAND WHAT WE UNDERSTAND AS RADIOLOGISTS. I THINK KEITH IS RIGHT THAT, YOU KNOW, WHAT WE LEARN TO DO AS RADIOLOGISTS IS ALL SPATIAL AND WE LOOK AT STRUCTURES. I THINK THE CAPABILITY OF THESE TOOLS IS GOING TO RAPIDLY EXPAND WHAT THE ROLE OF THE RADIOLOGIST IS AND I THINK THAT'S REALLY EXCITING FOR US TO DO. ANOTHER WAY THAT YOU CAN UNDERSTAND IS THAT USUALLY AT THE END OF THE A DEEP LEARNING NETWORK YOU HAVE THE FULLY CONNECTED NETWORK OR FCN COMPONENT AND THERE ARE CLEVER TECHNOLOGIES THAT MAP TO DECISION TREES. IF THE FIRST PART IS CONVOLUTION TO FIGURES, IT CAN BE CONVERTED TO DECISION TREES. HERE IS A PAPER DESCRIBING THAT. OKAY. SO JUST TO BRING IT TO A CLOSE HERE, I THINK THAT THERE'S A DEFINABLE WORKFLOW TO THE A.I. TOOL CREATION, THINGS LIKE DATA COLLECTION, CURATION, ANNOTATION, VERIFICATION, TRAINING, IMPLEMENTATION, AND HAVING A SYSTEM THAT SUPPORTS THAT FLOW IS CRITICAL TO SUCCESS. WE'LL HAVE EXTRA TIME DURING THE PANEL DISCUSSION BECAUSE I'M REALLY INTERESTED TO HEAR FROM THE AUDIENCE WHAT SORT OF TOOLS YOU ALL ARE USING AND WHAT STEPS ARE THE BIGGEST CHALLENGE THAT YOU HAVE. THE OTHER THING IS I THINK THIS WHOLE AREA OF EXPLAINABLE A.I. IS A VIBRANT AREA THAT'S JUST STARTING TO BLOSSOM NOW, AND I THINK IT'S CRITICAL, PARTICULARLY FOR US IN MEDICINE AND IMAGING TO BE RIGHT AT THE FOREFRONT AND EMBRACING THESE TECHNIQUES SO WE CAN EXTRACT MAXIMUM AMOUNT OF INFORMATION AND UNDERSTANDABILITY FROM DEEP LEARNING NETWORKS. SO THANK YOU FOR YOUR ATTENTION. [APPLAUSE] THE NEXT SPEAKER HERE IS JAYASHREE KALPATHY-CRAMER WHO WILL ALSO SPEAK ABOUT THE GAPS WE HAVE IN DEEP LEARNING. >> THANKS FOR THE OPPORTUNITY TO SPEAK HERE AGAIN. THIS IS A REALLY WONDERFULLY ORGANIZED MEETING, I'M PRETTY EXCITED TO BE HERE. I THINK AS HAS BEEN MENTIONED BEFORE, A LOT OF WORK WE'VE SEEN THUS FAR AND A LOT OF WHAT THE MEETING IS TALKING ABOUT HAS BEEN FOCUSED ON THE IMAGE INTERPRETATION PART PRIMARILY WITH CNNs, IF YOU LOOK AT THE LITERATURE, ESPECIALLY IN RADIOLOGY LITERATURE, MEDICAL IMAGING LITERATURE SPECIFICALLY, MOST OF WHAT IS PUBLISHED THUS FAR HAS BEEN USING SUPERVISED MACHINE LEARNING, TYPICALLY WITH CNNs. AS HAS BEEN MENTIONED BY CURT AND OTHERS BEFORE, IMAGENET FOR INSTANCE HAS HELPED US TO TAKE SOME ADVANCES THAT CAME FROM THE COMPUTER VISION COMMUNITY AND DIRECTLY APPLY THEM TO MEDICAL IMAGES. THERE WAS A NICE WHITE PAPER PUT OUT BY THE CANADIAN ASSOCIATION OF RADIOLOGISTS TALKING ABOUT THE WORKFLOW AND WHERE WE'VE SEEN THE APPLICATIONS, YOU'VE SEEN SIMILAR THINGS FROM KEITH AND CURT OVERALL WHERE A.I. CAN BE HELPFUL POTENTIALLY SO I THOUGHT -- THIS IS OUR VERSION OF THAT. AGAIN, NOT COMPREHENSIVE BUT A LOT OF WHERE WE SEE THE ROLE OF A.I. IS IN THE ENTIRE WORKFLOW, NOT JUST THE INTERPRETATION PART OF IT. SO FOR INSTANCE A LOT OF PROTOCOLING AND SELECTING WHAT NEEDS TO BE DONE IS OFTEN DONE BY EXPERT CONSENSUS. WE DO A LOT OF BRAIN IMAGES, THERE'S A PROTOCOL THAT'S BEEN DECIDED BY THE BRAIN IMAGING, BRAIN TUMOR IMAGING PROTOCOL USED FOR CLINICAL TRIALS AND A& LOT OF THESE TYPES OF THINGS HAVE TYPICALLY BEEN DONE BY GETTING EXPERT IN THE ROOM THAT WILL SAY THIS IS WHAT WE NEED BUT WE CAN ENVISION A TIME WHERE WE CAN LOOK AT ALL OF THE DIFFERENT SEQUENCES THAT ARE ACQUIRED, LOOK AT THE VARIOUS KINDS OF USE CASES AND SEE THE MOST USEFUL ONE, HOW CAN WE GET THE MAXIMUM INFORMATION WITH MINIMUM AMOUNT OF TIME, NOT EVERYTHING IS USEFUL SO ARE THERE OPPORTUNITIES IN TERMS OF A.I. GIVEN A PRESENTATION HOW CAN WE FIGURE OUT WHAT THE BEST PROTOCOL MIGHT BE. AGAIN, ONCE WE DECIDE WHAT SET OF SEQUENCES ARE, THERE'S A VARIETY OF DIFFERENT WAYS THAT ACQUISITION HAPPENS AND MORE ADVANCED TECHNIQUES, MR FINGERPRINTING. ALL OF THESE, AGAIN, TRY TO GET MORE INFORMATION, WHICH SHORTER ACQUISITION TIMES, BETTER SCAN QUALITY AND SO ON, BUT THERE'S A LOT OF ROOM FOR A.I. IN THESE AREAS AS WELL. SO THERE'S THE NOTION OF SMART SCANNING OR AUTOMATED QA, CAN YOU USE A.I. TO FIGURE OUT WHAT SEQUENCES TO ACQUIRE AND HOW BEST TO ACQUIRE THEM. THEN ONCE WE HAVE THE RAW DATA -- ACTUALLY GOING BACK TO THE PREVIOUS ONE THERE'S INTERESTING WORK THAT CAME OUT OF COLLEAGUES AT THE MARTINOS CENTER TO COME UP WITH PULSE SEQUENCING. PRETEND THE IDEA OF GETTING OPTIMAL SEQUENCE IS THE GAME, HAVE THE SYSTEM PLAY AGAINST ITSELF LIKE WE SAW WITH OTHER SITUATIONS LIKE THAT, AND THESE A.I. COMES UP WITH A PULSE SEQUENCE THAT SEEKS TO ANSWER THE QUESTION THAT YOU'RE TRYING TO ANSWER, WHAT IS THE PULSE SEQUENCE THAT WILL HELP ME GET THE MOST USEFUL IMAGE FOR THE KIND OF THINGS I'M DOING. SO I'M SUSPECTING WE'RE GOING TO START TO SEE A LOT MORE OF THOSE SORTS OF APPLICATIONS GOING FORWARD. RECONSTRUCTION SPACE TYPICALLY WHAT WE'VE USED IS PHYSICS BASED APPROACHES, WE KNOW HOW THE ACQUISITION HAPPENS, THIS IS WHAT WE NEED FOR RECONSTRUCTION, BUT, AGAIN, DEPENDING ON THE SAMPLING PATTERNS YOU CAN AGAIN FIGURE OUT WHAT NEEDS TO BE DONE. BUT RECENT WORK AGAIN USING THE IDEA OF USING A.I. FOR RECONSTRUCTION SEEMS TO BE VERY PROMISING. SO THIS IS WORK THAT CAME OUT AGAIN AT THE MARTINOS CENTER WITH THE IDEA OF USING A.I. AS THIS BEING A SUPERVISED -- RECONSTRUCTION AS A SUPERVISED LEARNING TASK. SO GIVEN A BUNCH OF DATA, HOW DO WE LEARN PERFORMANCE BETWEEN THE RAW SENSOR DATA AND ACQUISITION DATA. THEY FOUND USING A NEURAL NETWORK THEY ARE ABLE TO RECONSTRUCT DATA VERY, VERY EFFICIENTLY, AND THESE RECONSTRUCTION METHODS ARE VERY GOOD FOR UNDERSAMPLED DATA IN A VARIETY OF DIFFERENT WAYS. ESSENTIALLY WHAT IT'S LEARNING IN SOME CASES IS A FOURIER TRANSFER, YOU CAN START WITH IMAGE NETWORK AND CREATE THE FOURIER TRANSFER OR START WITH MR IMAGES OF THE BRAIN AND LEARN THAT. WHAT THE SYSTEM ENDS UP LEARNING IS A SPARSE REPRESENTATION OF THE MAPPING FACE, HELPING TO ENCODE A LOT OF INFORMATION ABOUT THE VERY SPECIFIC DOMAIN THEY ARE USING. AND THERE BY ALLOWING YOU TO DO THINGS LIKE THIS WHERE YOU CAN START WITH ESSENTIALLY DATA THAT WOULD NORMALLY BE RECONSTRUCTED ON THE LAST TRADITIONAL METHOD, USING THIS CAN GET A HIGH QUALITY RECONSTRUCTION. AND THIS CAN BE TRUE FOR A VARIETY OF ENCODINGS, FOR THE MR, SO YOU CAN USE A VARIETY OF DIFFERENT UNDERSAMPLING SCHEMES, SPIRALING AND OTHERS, GET GOOD RECONSTRUCTION PERFORMANCE. MR FINGERPRINTING, DEEP LEARNING HAS BEEN APPLIED USING FINGERPRINTING. AGAIN, THIS METHOD ALLOWS YOU TO LEARN THE VARIOUS TRAJECTORIES AND LEARN FROM THAT HOW TO RECONSTRUCT THE IMAGE MOST EXPEDIENTLY. THE LAST PART IS WHAT WE'VE SEEN A LOT OF THE WORK IN THE DIAGNOSIS, QUANTIFICATION, SEGMENTATION, CLASSIFICATION AND SO ON. SO, EXAMPLE OF THIS IS THINGS LIKE BRAIN TUMOR SEGMENTATION, SO THIS HAS BEEN THE LAST FEW YEARS WE'VE SEEN A LOT OF DIFFERENT METHODS FOR VARIOUS KINDS OF TUMOR AND OTHER ORGAN SEGMENTATION. ONE THING THAT HAS ENABLED A LOT OF THIS IS MAKING AVAILABLE PUBLIC DATASETS, WITH ANNOTATIONS, BRAIN TUMOR FOR INSTANCE THERE WAS A CHALLENGE THAT USED DATA FROM THE TCI AND HAD EXPERT ANNOTATIONS, THE GRASS CHALLENGE. THE OUTPUT OF THAT CHALLENGE HAS BEEN A RANGE OF VERY GOOD 3D SEGMENTATION ALGORITHMS FOR BRAIN TUMORS. THE NEXT STEP FROM THAT, DISEASE CLASSIFICATION, SO THIS IS AN EXAMPLE OF SOME WORK IN TERMS OF DISEASE CALLED ADENOPATHY THAT WE'VE BEEN WORKING ON FOR A NUMBER OF YEARS, FUNDUS PHOTOGRAPHY, CLASSIFYING DISEASE TO PRE-PLUS TO PLUS. THE METHOD WE USE THAT IS FAIRLY COMMON IS OFTEN A TWO-STEP PROCESS OR THREE-STEP PROCESS, SO SOMETIMES YOU HAVE A SEGMENTATION NETWORK FOLLOWED BY CLASSIFICATION NETWORK, SOMETIMES YOU HAVE A CANDIDATE IDENTIFICATION NETWORK THAT TRIES TO IDENTIFY POSSIBLE LESIONS AND THEN YOU HAVE ANOTHER NETWORK THAT DOES THE CLASSIFICATION. SO THE ARCHITECTURES WE HAVE TO START TO EXPLORE FURTHER ARE MORE -- A LONGER TYPE LINE OF VARIOUS THINGS. YOU MIGHT HAVE SPECIFIC TASKS, SPECIFIC NETWORKS THAT ARE LINKED TOGETHER IN A CHAIN. THIS WAS, AGAIN, KIND OF INTERESTING WORK IN TERMS OF THE FACT THAT WE HAD A NUMBER OF HUMAN RATERS AS WELL AS EXPERT CON CONSENSUS AND GROUND TRUTH AND WANTED TO SEE PERFORMANCE OF ALGORITHM. SIMILAR TO OTHER GROUPS, IF YOU PLOT ROC CURVE THE HUMANS OFTEN LIE ON VARIOUS PARTS OF THE ROC CURVE THAT THE ALGORITHM'S ROC CURVE, YOU COULD TUNE THE ALGORITHM TO BEHAVE LIKE ANY ONE USER. THESE THINGS WE'VE SEEN ARE USERS TEND TO BE PRETTY BIASED. SO IF WE'VE DONE THIS OVER AND OVER AGAIN, PEOPLE TEND TO RELIABLY UNDERCALL OR OVERCALL, SO YOU CAN ACTUALLY HAVE THE ALGORITHM BEHAVE LIKE AN UNDERCALLER OR OVERCALLER IF YOU REALLY WANTED TO. I THINK BRAD AND OTHERS WILL TALK ABOUT IT MORE SO YOU CAN HAVE THE NETWORKS DO THINGS LIKE PREDICTION, LOOKING AT IMAGE, COMBINE IMAGING WITH OTHER CLINICAL PARAMETERS LIKE AGE AND SO ON, TO GET PRETTY GOOD PERFORMANCE. THE SAME NETWORKS CAN BE USING A NETWORK TRAINED FOR ONE PARTICULAR DISEASE, WITH MINOR MODIFICATIONS OF THE FINAL LAYERS OR POTENTIALLY FINE TUNING OF THE WHOLE NETWORK CAN BE USED TO EXPLORE OTHER DISEASE CONDITIONS, SO THE SAME NETWORK THAT POTENTIALLY WE STUDIED FOR BRAIN TUMORS CAN NOW USE A SIMILAR NETWORK ARCHITECTURE, SAME AMOUNT OF TRAINING TO DO STROKE SEGMENTATION, PREDICTION OF OCCURRENCE, EVEN DOWNSTREAM PREDICTIONS SUCH AS LIKELIHOOD THIS PERSON IS GOING TO HAVE THIS KIND OF DISABILITY AT THIS POINT OF TIME. WE START OFF WITH SEGMENTATION, WE GO TO THINGS LIKE DISEASE CLASSIFICATION, AT THAT TIME POINT, BUT THEN WE ALSO CAN GO ON TO THINGS LIKE PREDICTING OUTCOMES MUCH FURTHER DOWNSTREAM. THIS IS A REALLY INTERESTING NEW WORK THAT CAME OUT OUT IN TERMS OF LOOKING AT OCT AND FUNDUS PHOTOGRAPHY FOR THE IDEA OF REFERRAL, SO TRIAGING. THIS HAD BEEN MENTIONED BEFORE, CAN WE USE A.I. FOR TRIAGING. IN THIS PARTICULAR CASE THEY SHOWED THAT THEY GOT EXCEEDINGLY GOOD PERFORMANCE IN USING A DEEP LEARNING ALGORITHM LOOKING AT FUNDUS PHOTOGRAPHY AS WELL AS OCTs, TO IDENTIFY THE MOST URGENT CASES. THIS IS A PERFORMANCE OF THE ALGORITHM, RETINA SPECIALIST, OPTOMETRIST, AND ERROR BARS. YOU CAN SEE THE ALGORITHM DOES A REALLY GOOD JOB COMPARED TO THE -- CERTAINLY COMPARED TO A LOT OF HUMANS THAT THEY HAVE BEEN COMPARING TO. AN INTERESTING FEATURE OF THIS ALGORITHM WAS THIS INITIAL PART OF THE NETWORK, WHICH IS A DEVICE ADAPTATION. YOU CAN ALMOST THINK OF IT LIKE A PARTICULAR LENS ON A MICROSCOPE, SWAP OUT THAT FOR A DIFFERENT SYSTEM, SO DEPENDING ON THE SYSTEM THAT HAD BEEN USED TO ACQUIRE THE IMAGES THAT PARTICULAR MODULE WAS REPLACED. THE VERY FIRST STEP WAS THE VERY DEVICE-SPECIFIC MODULE. ONE CAN THING ABOUT THINGS LIKE THAT, THINGS FOR ACQUISITION ANALYSIS. PERHAPS WE DON'T NEED TO TRAIN THE NETWORK FOR EVERY SINGLE ACQUISITION SYSTEM. WE CAN ONLY TRAIN A PARTICULAR SUBMODULE AND THEN SWAP THAT OUT WHEN YOU KNOW WHAT THE ACQUISITION SYSTEM IS. A LOT OF THE -- IT'S A VERY EXCITING TIME FOR US. A LOT OF INTERESTING THINGS ARE HAPPENING BUT THERE ARE MANY, MANY CHALLENGES. THESE AGAIN HAVE BEEN MENTIONED BY OTHERS, A LOT OF THESE METHODS NEED LOTS OF DATA, PATIENT IMAGES ARE HARD TO SHARE, ANNOTATIONS ARE DIFFICULT TO ACQUIRE, THEY ARE NOISY AND BIASED, THEY DON'T PROVIDE UNCERTAINTY, BLACK BOXES CAN BE FOOLED. SO THERE'S A SUBSTANTIAL LIST OF SOME CHALLENGES BUT FOR THOSE, EACH ONE OF THOSE IS WITH Ph.D. STUDENTS, EACH IS A GOOD Ph.D. TOPIC. WE NEED LOTS OF DATA. WHAT ARE SOME WAYS TO MITIGATE AGAINST THAT? TRANSFER LEARNING HAS BEEN MENTIONED MANY TIMES, ALMOST EVERYTHING YOU SEE USES MODEL TRAINED ON IMAGENET. NOT EVERYTHING BUT A LOT OF THING USE MODELS TRAINED ON IMAGENET, THE QUESTION IS HOW BEST TO DO THAT FOR OUR PARTICULAR DOMAINS. WE TALKED ABOUT THIS IN THE PREVIOUS SESSION, ABOUT THE CONCEPT OF MORE SENSIBLE AUGMENTATION, IN TERMS OF -- YEAH, GOING TO TRANSFER LEARNING, THIS IS WHAT EVEN FOR ROP SYSTEM USE TRANSFER LEARNING FROM IMAGENET AND IT PERFORMS WELL BY FINE TUNING THE FINAL LAYERS. THE AUGMENTATION TECHNIQUES THAT ARE TYPICALLY USED ARE USUALLY STANDARD AND THEY INVOLVE ROTATION AND FLIPS AND ZOOMS AND THINGS LIKE THAT, A LOT DOESN'T MAKE SENSE FOR OUR DATA. MOST OF OUR DATA ACTUALLY HAS VERY SPECIFIC ANATOMICAL AND STRUCTURAL SHAPES, AND BY DOING SOME OF THESE MOST COMMON USE AUGMENTATIONS WE'RE ACTUALLY DISREGARDING A LOT OF INHERENT STRUCTURE IN OUR DATA. THE SAME METHODS THAT ARE REALLY USEFUL FOR CATS AND DOGS MAY NOT BE THE BEST METHODS FOR THE DATA WE DEAL WITH. ON THE OTHER HAND, WE ACTUALLY HAVE A REALLY GOOD UNDERSTANDING OF THE PHYSICS OF THE ACQUISITION, AND WHAT WE'RE NOT DOING ENOUGH OF IS APPLYING PHYSICS IN A SENSIBLE MANNER TO DO THE THINGS THAT WE NEED TO DO IN TERMS OF AUGMENTATION. SO FOR INSTANCE IF YOU HAD A GOOD SIMULATORS AND KNEW WHAT THAT IMAGE WOULD LOOK LIKE UNDER DIFFERENT ACQUISITION CONDITIONS, WITH DIFFERENT TRs, IT MIGHT BE USEFUL TO DO THINGS LIKE THAT IN TERMS OF AUGMENTATION AS OPPOSED TO CUES, MAKING YOUR HEAD LOOK LIKE A SHAPE IT WOULD NEVER BE. BEING MORE SENSIBLE ABOUT HOW WE CAN COMBINE THE PHYSICS THAT WE KNOW SO WELL WITH THE KINDS OF METHODS THAT WE NEED FOR AUGMENTATION CAN BE USEFUL. YOU CAN HAVE MOTION SIMULATORS, KNOW THE EFFECT OF MOTION ON THE IMAGE, IF YOU, AGAIN, HAD A LOT OF THESE MORE FEASIBLE BIOLOGICALLY SENSIBLE STIMULATORS, PHYSICS AND BIOLOGICALLY SENSIBLE SIMULATORS THAT CAN BE USEFUL. PATIENT DATA IS HARD TO SHARE. SO ONE OF THE THINGS THAT WE'VE BEEN LOOKING AT IS DISTRIBUTED MODELS. SCENARIOS WHERE THE TRAINING OCCURS IN A DISTRIBUTED FASHION BUT NO DATA LEAVES EACH INSTITUTION. AND ONE OF THE FIRST QUESTIONS THAT PEOPLE ASK, HOW ABOUT PATIENT PRIVACY, CAN YOU NOT RECONSTRUCT THE IMAGE FROM THE MODELS? HOW DO WE DEAL WITH THINGS LIKE THAT? THERE'S A LOT OF WORK, AGAIN, HAPPENING IN THE AREA OF ENCRYPTION AND WORKING WITH ENCRYPTED MODELS. STARTING TO GET EXCITING THERE. IN TERMS OF MODEL SHARING, EXPERIMENTS THAT WE DID WITH COLLABORATORS AT STANFORD, LOOKING AT THIS IDEA OF DISTRIBUTED LEARNING FOR MODELS. THE CONCEPT HERE, YOU HAVE DATA AT MULTIPLE INSTITUTIONS, YOU'RE ALL FRIENDS, ALL WILLING TO WORK TOGETHER AND SHARE THE MODEL BUT THE DATA CANNOT LEAVE YOUR INSTITUTIONAL FIREWALL. HOW CAN YOU LEARN BY SHARING MODEL PARAMETERS? WE FOUND IF YOU'RE SMART HOW WE DO THIS WE CAN GET PRETTY GOOD PERFORMANCE THAT STARTS TO APPROACH HAVING ALL DETAIL CENTRALLY LOCATED, BY MOVING MODEL PARAMETERS WE CAN GET REASONABLE PERFORMANCE. AND THIS IS VERY EARLY WORK BUT THERE'S A LOT OF PROMISE AND A LOT OF INTEREST IN MAKING THIS HAPPEN, ESPECIALLY IN CASES WHERE THE DATA CANNOT LEAVE. WE'VE HEARD A LOT ABOUT HOW DIFFICULT ANNOTATIONS ARE AND TALKED ABOUT GETTING THE DATA FROM MEDICAL RECORDS. ONE THING WE'VE BEEN EXPLORING IS USE OF CROWD SOURCING, SO AT LAST YEAR AT THE RSNA WE HAD A CROWDS CURE CANCER BOOTH. WE'LL HAVE THAT AGAIN THIS YEAR. THE IDEA AGAIN IF YOU HAVE A SET EVERYONE MIGHT BE AN EXPERT, BY GETTING THE POWER OF THE CROWD YOU CAN START TO APPROXIMATE SOMETHING THAT RESEMBLES THE BEST EXPERTS. SO HERE IS THE GROUND TRUTH DATA FROM A COUPLE EXPERTS, VERSUS THE DATA FROM THE CROWDS. WE CAN START TO SEE THE ANNOTATIONS FROM THE CROWDS CAN APPROXIMATE THAT OF EXPERTS. A LOT OF EXCITING WORK HAPPENING IN THE AREA OF SEMI SUPERVISED AND UNSUPERVISED LEARNING, HOW DO WE BEST COMBINE SMALL AMOUNT OF WELL-CURATED DATA WITH LOTS OF ANNOTATED OR WEAKLY ANNOTATED DATA TO GET GOOD RESULTS? WE'RE STARTING TO SEE PRETTY GOOD ADVANCES THERE ALSO. ANNOTATIONS ARE NOISY. THIS IS SOMETHING WE KEEP FINDING OVER AND OVER AGAIN. THIS IS AN EXAMPLE AGAIN FROM THE OPHTHALMOLOGY FIELD WHERE WE HAD MULTIPLE EXPERTS ANNOTATE THE SAME DATA SET AND COLOR CODING IS 1:3 DISEASE SEVERITY. PERSON 1 HAD VERY FEW DARK BLUES, A LOT OF LIGHT BLUES. VERSUS PERSON 8 HAD ALMOST NO LIGHT -- THEY HAD NONE OF THE LIGHTEST AND HAD MOST OF THE DARK BLUES. THEY WERE VERY CONSISTENTLY BUT SYSTEMATICALLY BIASED ACROSS -- IN HOW THEY TAKE SOMETHING ON THE SEVERITY SCALE, BEND TO MILD AND MODERATE. WE DID THIS EXPERIMENT OVER EIGHT YEARS, THEY HAVEN'T CHANGED. PEOPLE WHO ARE MORE OR LESS UNDERCOLORED TEND TO BE THAT WAY. WE DO CHANGE -- SEE SOME CHANGES WITH EXPERIENCE, AND THINGS OF THAT, SO MORE EXPERIENCED PEOPLE OFTN HAVE SLIGHTLY DIFFERENT THRESHOLDS FOR WHEN THEY START TO GET WORRIED COMPARED TO PEOPLE WHO ARE STARTING OUT. BUT THE PROBLEM HAS BEEN USING DATA FROM YOUR RETROSPECTIVE DATA THAT'S BEEN ANNOTATED ALREADY, YOU WILL SEE THAT IT'S VERY NOISY. AND IF YOU -- WHAT WE FOUND IS IF YOU ACTUALLY TAKE DATA FROM ANY KIND OF EHR, AND JUST LOOK AT THE HISTOGRAMS OF THE RATINGS BY PERSON, THEY LOOK VERY DIFFERENT. AND THOSE DIFFERENCES PERSIST ACROSS YEARS. SOME PEOPLE TEND TO HAVE 1s AND 2s, OTHERS HAVE 2s AND 3s. IF YOU CAN LEARN FROM THE FACT THESE DATA ARE VERY NOISY, HOW DO WE LEARN FROM THAT, WE CAN ACTUALLY IMPROVE THE PERFORMANCE OF THESE VERY NOISY DATA. ONE THING WE ASKED, GIVEN A PAIR OF IMAGES, WHICH ONE IS WORSE, PEOPLE WERE VERY GOOD AT THAT TASK, SO PEOPLE ALWAYS AGREE WHICH ONE IS MORE SEVERE BUT THEY DIDN'T AGREE ON WHETHER IT'S MILD OR MODERATE, OR MODERATE OR MODERATE-PLUS, THINGS THAT MAY BE USEFUL IN DEVELOPING MODELS MORE RESILIENT TO THE NOISE WE SEE. I THINK WE TALKED ABOUT THE EXPLAINABILITY, SO -- AND THEN SO I THINK I'M GOING TO STOP HERE BECAUSE WE'RE DEFINITELY RUNNING OUT OF TIME. THANK YOU. [APPLAUSE] >> THE LAST SPEAKER TODAY AS HE GETS THE MICROPHONE ON IS MATT LUNGREN FROM STANFORD TALKING ABOUT ANNOTATING MEDICAL IMAGING DATA FOR DEEP LEARNING. WELCOME, MATT. >> THANK YOU, BRAD. THANKS FOR HAVING ME HERE. I'M GOING TO TALK ABOUT ONE OF THE MOST EXCITING PARTS OF MEDICAL DEEP LEARNING IMAGING. WORK IS ON LABELING. THAT'S SOMETHING THAT I'M ASKED TO DO ALL THE TIME. SOMETHING I CAN'T SAY I LOOK FORWARD TO DOING. [LAUGHTER] BUT UNFORTUNATELY IT'S NECESSARY TO SOME EXTENT SO I WORK REALLY HARD IN MY LAB TO FIND WAYS SO I DON'T HAVE TO DO ANNOTATING MYSELF AND WE'RE GOING TO TALK ABOUT STRATEGIES FOR THAT. WHEN I LOOK AT A NEW PROJECT FOR DEEP LEARNING IN MEDICAL IMAGING I THINK OF THE VALUE OF THE LABEL THAT WE HAVE. AT THE VERY TOP IS GROUND TRUTH, LEAST ABUNDANT BUT CERTAINLY THE BEST, LIKE PATIENT OUTCOME OR PATHOLOGY. SOMEWHERE IN THE MIDDLE PROSPECTIVE ANNOTATIONS INCLUDING FREE TEXT REPORT, STRUCTURED LABELS CURT MENTIONED AND RECORDING. AT THE BOTTOM IS FINDING A WAY THAT ANNOTATE IMAGES COLLECTING BUT DON'T HAVE ANY OTHER INFORMATION. HERE IS AN EXAMPLE OF A GROUND TRUTH PROJECT FROM A STANFORD PEDIATRIC NEURORADIOLOGIST, DATA ON BLASTOMAS TIED TO MOLECULAR SUBTYPES, TO DO RADIOMICS OR DEEP LEARNING WORK TO PREDICT NON-INVASIVELY THE MOLECULAR SUBTYPE OF TUMORS. WHY IS THAT IMPORTANT? THE MOLECULAR SUBTYPES TIE WELL TO PROGNOSIS AND OUTCOMES FOR THESE PATIENTS, AN OPPORTUNITY TO POTENTIALLY HELP THE TREATMENT OUTCOME FOR THESE PATIENTS SO A GREAT LABEL AND CERTAINLY AN IMMEDIATE IMPACT. WE HAVE A LOT OF PROSPECTIVE STRUCTURE LABELS AT STANFORD, USED FOR ALL KINDS OF PROJECTS,& VERY NICE TO HAVE IT. I LIKE THE SAYING THE BEST TIME TO PLANT A TREE IS 20 YEARS AGO, SECOND BEST TIME IS NOW. THAT'S HOW WE LOOK AT THE PROGRAM. WE ARE SO GLAD IT HAPPENED 15 YEARS AGO, ALL THE DATA IS LABELED ALREADY USED FOR DEEP LEARNING WITH TRIAGE PROJECTS, RELEASED AS DATASET IN PARTNERSHIP WITH THE ANDREW NG LAB, YOU CAN USE THAT FOR YOUR PROJECTS. THE THING WE DEAL WITH THE MOST UNFORTUNATELY IS TRYING TO ASSIGN A LABEL TO AN IMAGE THAT HAS ANY NUMBER OF POTENTIAL LABELS, RIGHT? THIS OPACITY COULD BE PNEUMONIA, CANCER, WE DON'T HAVE ENOUGH INFORMATION. AND TURNS OUT RADIOLOGISTS AREN'T REAL CONSISTENT. THIS IS A STUDY 8 OR 9 YEARS AGO OUT OF BRIGHAM, ABDOMINAL CTs, A SMALL GROUP READ, SHOWED A FEW MONTHS LATER, THEY DISAGREED WITH COLLEAGUES AND WITH THEMSELVES A THIRD OF THE TIME, SCARY WHEN WE HAVE PEOPLE ANNOTATING IMAGES FOR US. ANOTHER THING IS YOU CAN HAVE TWO IMAGING EXAMINATION WITH TWO LABELS, THESE ARE ESSENTIALLY IDENTICAL, ONE IS NORMAL, ONE HAS A SEVERE NEUROLOGIC DISEASE, WHICH CAUSES A DELAY IN MYELINATION AND BRAIN OF A 3-YEAR-OLD TO LOOK LIKE THE BRAIN OF A 3-MONTH-OLD. DIFFICULT TO DO THESE ANNOTATIONS WITHOUT THE CONTEXT. COMING BACK TO RETROSPECTIVE LABELS WE COULD USE FREE TEXT REPORT, IT'S NOISY AND MESSY, THERE'S LOTS OF LANGUAGE VARIATIONS WHEN WE DICTATE BUT IF WE CAN PULL OUT SALIENT FINDINGS AND ASCRIBE THE RIGHT LABEL WE HAVE AN OPPORTUNITY TO SCALE AND GET ME OFF THE HOOK FOR HAVING TO LABEL SO MUCH. LET'S TALK ABOUT LABELING METHODS. AGAIN I LOOKED TO WORLD IN PYRAMIDS, I'M NOT SURE WHAT THAT SAYS ABOUT ME. TOWARDS THE TOP OF THE ANNOTATION METHODS YOU HAVE INCREASING NOISE. THE MOST SCALABLE BUT MOST NOISE WITH SIMPLE SEARCH TOOLS. IN THE MIDDLE ARE NLP TECHNIQUES, AT THE BOTTOM IS HUMAN ANNOTATION, NOT SCALABLE BUT ONE OF THE MOST ACCURATE WAYS TO DO THINGS. HERE IS AN ADVERTISEMENT LOOKING FOR RADIOLOGISTS TO ANNOTATE MEDICAL IMAGES, FROM ENLETTIC. WE HAVE TO LOOK FOR OTHER OPPORTUNITIES TO DO OUR LABEL, WE DON'T HAVE THE BUDGET. SEARCH TOOLS IS ONE WAY. CURT MENTIONED THIS EARLIER. YOU CAN SEARCH WITH CERTAIN TERMS BUT WITH EVERY TERM YOU ADD ON YOU'LL ADD MORE NOISE. LOOKING FOR PULMONARY EMBOLISM WITH EACH TURN YOU'LL ADD MORE DIAGNOSES, YOU'LL HAVE A SECOND STEP TO CURATE THAT. THIS IS WORK A WHILE AGO, MACHINE LEARNING. WE'VE SINCE MOVED AWAY. WE SUPPORT VECTOR MACHINES TO CLASSIFY FREE TEXT REPORTS, A BINARY CLASSIFICATION TASK LOOKING FOR SURGICALLY IMPORTANT LESIONS. SVMS ARE UNWIELDILY. RESULTS WERE OKAY. THIS IS A SNIPPET FROM OUR PAPER TO GIVE A SENSE THEY CAN BE ACCURATE BUT CARRY SOME NOISE AND THEY AREN'T AS GENERALIZABLE AS WE HOPED. THEY ARE ALSO -- WE HAVE TO CHOOSE THE KERNEL, DEAL WITH REGULARIZATION PENALTIES, SLACK VARIABLES, ALL KINDS OF PROBLEMSES THAT SWITCH TO DEEP LEARNING FOR NLP. THEY TEND TO GENERALIZE WELL ON EXTERNAL REPORT DATASETS, I'LL SHOW AN EXAMPLE OF THAT. HERE'S A FIGURE FROM OUR PAPER IN RADIOLOGY WHEN WE SORT OF TALKED AGAIN AS BRAD MENTIONED WE WERE GOING TO SHOW SOME VISUALIZATION EXAMPLES. THIS IS A RELATIVELY SHALLOW CNN, WE USED PRE-TRAINED WORD VECTORS TO CREATE A CLASSIFIER TO PULL POSITIVE P.E. FROM FREE TEXT CTA REPORTS. THIS IS ALL DOCUMENT LEVEL. YET THE CNN MODEL WAS ABLE TO IDENTIFY WORDS YOU WOULD EXPECT IT TO, IN ORDER TO COME UM WITH DIAGNOSIS OF P.E. THAT WAS REASSURING. ON THE RIGHT-HAND PANEL THOSE ARE AREA INTO THE CURVE FOR P POSITIVE AND P ACUTE LABELS. WE CALL THIS PROJECT OUR KITCHEN SINKER, THE AVENGERS PROJECT, TAKING EVERY CLASSIFIER METHOD INTO ONE PAPER, USING HALF A MILLION REPORTS FROM FOUR INTUITIONS, DONE IN COLLABORATION WITH PHILLIPS HEALTH CARE, COULD WE BUILD A DEEP LEARNING CLASSIFIER FOR NLP THAT MATCHES WHAT THE REPORT STRUCTURE LOOKS LIKE FOR RADIOLOGY REPORTS? WE DID END TO END LSTM LAYERS TO HAVE SOME WORD REPRESENTATION BUILT IN AS WELL AS SENTENCE REPRESENTATION BUILT IN. PASSING THAT THROUGH A SOFT MAX LAYER. LEFT-HAND IS ANOTHER VISUALIZATION MAP WHERE THE CNN MAP GOT IT WRONG BUT THE NO METHOD GOT IT RIGHT. WORD LEVEL DEPENDENCY IN GREEN, SENTENCE LEVEL DEPENDENCY IN BLUE, YOU CAN SEE IN SENTENCE 3 THE DISTANCE BETWEEN NO AND EMBOLISM IS LONG, SO YOU CAN SEE WHY MAYBE THE CNN MODEL MAY NOT HAVE PICKET THAT UP BUT MAINTAINING THAT SENTENCE LEVEL WEIGHTING THAT SENTENCE ENDED UP BEING -- OR THOSE WORDS ENDED UP BEING SIGNIFICANT FOR THE SECOND CLASSIFIER. RESULTS ARE ON THE RIGHT, TRAINED ON 4,000 DOCUMENT LEVEL LABELED PE STUDIES THAT WE HAD HUMANS DO AND WE CAN SEE RESULTS WERE GOOD COMPARED TO OTHER CLASSIFIERS. WHEN WE TRIED TO GENERALIZE TO DUKE DATASET THEIR REPORTS ARE NOT AS STRUCTURED AS HOURS BUT IF WE CAN DO THIS AT SCALE MAYBE WE CAN GET AWAY WITH IT. LET'S TALK ABOUT QUANTITY AND QUALITY. WHEN WE TALK ABOUT HOW MUCH TRAINING DATA WE REALLY NEED AND HOW IMPORTANT NOISE IS WE LOOK TO THE IMAGE IN THAT COMMUNITY, MACHINE VISION COMMUNITY,& BECAUSE THEY HAVE DONE A LOT OF THIS WORK. THIS IS A GREAT PAPER OUT OF THE GROUP IN M.I.T. AND CORNELL LOOKING AT PURPOSELY PUTTING NEGOTIATION -- NOISE IN TO SEE HOW IT AFFECTED. EVEN AT 5:1 NOISE PERFORMANCE DOESN'T DIP THAT MUCH. IT'S DOWN BUT NOT AS MUCH AS YOU COULD EXPECT WITH 5:1 NOISY TO ACCURATE LABELS, ENCOURAGING US. CURT SHOWED THIS EARLIER, WORK FROM A GROUP OUT OF STANFORD WITH MANY EMR CLASSIFIERS TO PREDICT DIABETES OR HEART ATTACK, HOW MUCH NOISE COULD HE PUT IN HIS DATASET BEFORE THE CLASSIFIER PERFORMANCE WENT DOWN AND IF IT DID GO DOWN HOW COULD HE BRING IT BACK UP TO 100% ACCURACY LABEL? THERE'S A RELATIONSHIP BETWEEN NOISY DATA AND THE AMOUNT OF DATA YOU HAVE TO OVERCOME THAT. THIS IS SOME VERY RECENT WORK FROM US. WE HAVE 200,000 CHEST X-RAYS LABELED AS NORMAL OR ABNORMAL IN OUR ARCHIVE USED TO TRAIN CLASSIFIERS, THIS IS A CLASSIFIER THAT WAS TRAINED, BOTH REPRESENTING THE SAME TRAINING DATA BUT TWO TEST SETS. SO THE DOTTEDs LINE, LOWER PERFORMANCE, HOLDOUT TRAINING LABELS, THERE'S NOISE IN THERE, RIGHT? 10 TO 15% NOISE. WE TOOK THE SAME TEST SET AND RELABELED AND TOOK OUT THE NOISE, BY DOING JUST THAT, A .89 TO .96, SIGNIFICANT. SO THAT WAS OUR EXPERIENCE WHEN WE DID THIS WITH ANDREW NG WHEN WE USED THE NIH DATASET WHICH HAS BEEN CRITICIZED AS BEING NOISY WE DIDN'T FIND IT BEING THAT BAD. WE HAD AN EXPERT LABELED TEST SET TO DEMONSTRATE OUR PERFORMANCE. AND WHEN WE COMPARED AGAINST 12 PRACTICING RADIOLOGISTS WE FOUND OUR CLASSIFIER AGAIN TRAINED ON THIS NOISY DATA DID AS WELL AS THEY DID. AND SO WE KNOW WE CAN USE NOISY LABELING DATA TO CREATE ACCURATE CLASSIFIERS IF WE HAVE THE RIGHT PRINCIPLES IN PLACE. ALL RIGHT. SO IN CONCLUSION MEDICAL DATA IS MESSY AS I MENTIONED. I TALKED ABOUT LABEL VALUE HIERARCHIES. YOU CAN USE DATA VOLUME TO OVERCOME NOISE, IMPORTANT TO KNOW. OF COURSE ACCURATE TEST SET LABELS ARE VERY IMPORTANT. THAT'S ALL I HAVE. THANK YOU. [APPLAUSE] >> I'LL ASK JAYASHREE TO COME AND HUGH HARVEY TO COME UP TO THE PANEL. FOR THOSE WITH QUESTIONS, PLEASE GO TO THE MICROPHONE. LOOKS LIKE WE HAVE A FEW QUESTIONS. >> I HAVE A QUESTION FOR JAYASHREE. I WAS INTRIGUED WITH YOUR COMMENTS ABOUT DATA SHARING AND PARTICULARLY THE NOTION OF KEEPING DATA AT A HOME INSTITUTIONAL LOWING -- ALLOWING DEVELOPERS TO ACCESS. HOW DO WE ENSURE DATASETS ARE WHAT WE THINK THEY ARE, MEANING WITH THE EVER PUSH TO EXPAND TO GET MORE DATA MY FEAR IS A DATASET ACQUIRED SAY CT IMAGES WITH A CERTAIN NOISE INDEX WILL GET POLLUTED WITH DATA BEING MUSHED FROM OTHER SOURCES TO CREATE EVEN LARGER DATASETS. IS THERE AN EFFORT TO WATERMARK TO SAY THIS IS WHAT'S IN THE BOX AND WE'RE NOT GOING TO TINKER WITH IT? >> SO I'M NOT SURE -- I DO KNOW THERE ARE WAYS TO PUT -- SORT OF PUT A BOW ON THE DATASET AND HAVE PROMINENCE AND HAVE IT CURATED SO YOU KNOW A MODEL WAS TRAINED ON THE DATASET AND THAT'S NOT BEEN CHANGED SINCE IT WAS ACQUIRED. THE SECOND PART OF THE QUESTION ABOUT THE FACT DIFFERENT DATASETS MIGHT AFFECT THE MODEL DIFFERENTLY IS AN IMPORTANT QUESTION. METHODS DEVELOPED TO LEARN DISTRIBUTION OF DEFINITION DATASETS SO YOU KNOW THE NEW DATASET COMPARED TO THE BUILT-IN MODEL FOR INSTANCE, YOU CAN GET A SENSE OF HOW WELL THE MODEL IS GOING TO WORK OR HOW MUCH THAT OTHER DATA IS GOING TO POLLUTE YOUR EXISTING MODEL BY LOOKING AT METRICS OF DISTRIBUTION BETWEEN THE TWO DATASETS BUT I DON'T KNOW OF A SPECIFIC EFFORT TO SAY HERE IS THE DATA, OTHER THAN JUST DOING IT. I DON'T KNOW IF ANYBODY ELSE HAS THAT. >> YES, HELLO, MIKE GARRIA, NATIONAL INSTITUTE OF STANDARDS AND TECH FOLLOWING UP FOR JAYASHREE. I'M WITH YOU WITH DISTRIBUTED LEARNING BEING ABLE TO SHARE THE MODEL TO KEEP THE DATA PROTECTED. I'M WONDERING, CURIOUS ABOUT YOUR RESEARCH, YOUR COLLABORATIONS AND YOUR PRACTICE, IS THERE CONSIDERATION FOR PRIVACY CONCERNS WHEN YOU TAKE THE MODEL BACK OUT THAT IT MIGHT CONTAIN SENSITIVE SALIENT INFORMATION THAT IN FACT SOME DEEP LEARNING MODELS, JUST TALKING WITH MY COLLEAGUE FROM NIH, YOU KNOW, WE GET CLOSE TO MEMORIZATION ON SOME OF THIS, AND SO WHAT CONSIDERATIONS HAVE YOU MADE AND WHAT ARE YOUR THOUGHTS ABOUT WHAT WE CAN DO TO SOLVE THAT TECHNICAL ISSUE TO REALLY MAKE DISTRIBUTED LEARNING A REALITY? >> SO THAT COMES UP IN EVERY CONVERSATION WE'VE HAD, THAT'S VERY MUCH A CONCERN YOU MIGHT BE ABLE TO RECONSTRUCT SOMETHING THAT IS CLOSE ENOUGH THAT YOU LOSE PRIVACY. THERE ARE EFFORTS THAT I KNOW OF IN TERMS OF ENCRYPTING THE MODEL AND ACTUALLY DOING A VARIETY OF THINGS TO SORT OF PREVENT THAT FROM HAPPENING BUT IT'S A LITTLE MORE NOT PURE YET BUT THERE'S RESEARCH THAT I KNOW OF HAPPENING IN THAT AREA. >> THANK YOU. >> SEEMS LIKE ALSO DATA AUGMENTATION TECHNIQUE IS GOING TO HELP GET RID OF THAT, THAT'S WHY YOU DO DATA AUGMENTATION TO KEEP THE ALGORITHM FROM LEARNING THIS SPECIFIC EXAMPLE. SEEMS LIKE THAT SHOULD BE ONE THING THAT CAN HELP ADDRESS THAT, WHETHER IN STANDARD, BUT ALSO IN THIS CASE. >> THAT'S A REALLY GOOD POINT. ONE OF THE THINGS PEOPLE ARE LOOKING AT USING THINGS AGAIN TO SIMULATE, TO GET DATA THAT AUGMENT YOUR DATASET THAT ARE NOT REAL SO YOU CAN ALMOST GET SORT OF DATASETS BETWEEN PEOPLE, FOR INSTANCE, AND BY DOING THAT THEN ADD SOME MORE BUFFER AGAINST BEING ABLE TO RECONSTRUCT THE SAME PERSON SO -- >> HI. MY NAME IS BRENDAN GALLIS FROM THE FDA. MY QUESTION IS RELATED TO THE TOPIC THAT YOU MENTIONED WITH MULTIPLE READERS EVALUATING THE SAME IMAGES FOR THOSE ANNOTATIONS AND VALUE THAT COMES WITH THAT, EITHER IN THE TRAINING OR TESTING SIDE. I'M CURIOUS TO KNOW IF THERE'S ACTUAL TRAINING METHODS THAT WOULD HONOR THE INDIVIDUAL OBSERVATIONS. I MEAN ONE TYPICAL WAY IS AVERAGE OVER ANNOTATIONS BUT WHEN YOU TALK ABOUT CATEGORIES THAT'S NOT R EALLY -- THERE'S NO SUCH THING AS ACTUAL CATEGORYS THERE RESEARCH IN THAT SPACE ON THE TRAINING SIDE AND POINTING TO METRICS THAT WOULD USE THAT KIND OF DATA ON THE TESTING SIDE. >> YES, ONE OF THE THINGS WE'VE BEEN LOOKING AT IS PRECISELY THAT, IF THIS PERSON IS TWO STEPS BEHIND THE OTHER PERSON YOU CAN BUILD THAT IN THE MODEL ON THE TRAINING SITE. IF YOU HAVE DATA THAT'S BEEN LABELED FOR MULTIPLE PEOPLE THAT MAKES LIFE EASIER BECAUSE THEN YOU KNOW THAT THIS IS A BIAS, IF IT'S A CONSISTENT BIAS. IF YOU DON'T HAVE THAT BUT HAVE HISTORIC DATA, WHERE CAN YOU LOOK AT SORT OF THE HISTOGRAM OF PERSON A'S RATING AS OPPOSED TO PERSON B, YOU CAN DO THIS ITERATIVE PROCESS LEARNING THE MAPPING BETWEEN THE PEOPLE LEARNING THE BIAS IN THE PEOPLE WHILE DEVELOPING A MODEL THAT CAN LEARN THE -- LOOK LIKE EITHER PERSON, FOR INSTANCE, AND YOU CAN DO THAT AT THE TRAINING SITE AS WELL AS TEST SITE. BEGIN, IT'S EARLY RESEARCH BUT IT'S LOOKING QUITE PROMISING IN TERMS OF LEARNING THE FACT THAT DIFFERENT PEOPLE ARE DIFFERENT. IN OUR PARTICULAR CASE PEOPLE WERE INTERESTED MORE THEY HAD BIASES BUT THERE ARE OTHER TYPES OF INTEGRATOR DIFFERENCES IF THEY DON'T AGREE WITH THEMSELVES 30% OF THE TIME I'M NOT SURE YOU CAN LEARN THAT, WHAT THEY HAD FOR BREAKFAST THAT CAUSED THAT DIFFERENCE BUT I DON'T KNOW. >> WE DEAL WITH THAT A LOT AS YOU CAN IMAGINE. OUR TEST SETS ARE WHERE IT'S MOST IMPORTANT BECAUSE IF WE TEND TO GET MULTIPLE READERS FOR OUR TEST SETS WE HAVE PANELS OF EXPERTS BUT STILL DISAGREE. MAJORITY VOTE ISN'T CAPTURING WHERE THE TRUTH IS VERY WELL. ESPECIALLY IF WE DON'T HAVE GROUND TRUTH ALREADY. SO WE'VE BEEN WORKING WITH OUR Ph.D. FOLKS TO FIGURE OUT THE BEST POSSIBLE WAY TO WEIGHT INDIVIDUAL RATERS IN A CERTAIN WAY BASED ON DISTANCE FROM THE MEAN CONSISTENTLY TO SEE IF WE IT FIND A COMMON GROUND. I'M SORRY, MAJORITY VOTE IS WHAT WE'RE TRYING TO DO NOW, I DON'T LIKE MAJORITY VOTE VERY MUCH, THORACIC RADIOLOGIST I SHOULD WEIGHT THAT HIGHER THAN A RESIDENT, IT'S SOMETHING WE'RE ACTIVELY WORKING ON IN OUR LAB. >> THANK YOU. >> CAN I ADD SOMETHING? OTHER. I'M DR. HOWIE FROM THE U.K., HENCE THE ACCENT. I WORK IN A.I. AT ROYAL COLLEGE, YOU TOUCHED ON GANS, SYNTHESIZING NEW DATA, THERE'S LOTS OF WORK ON DISTRICTED LEARNING SIDE OF THINGS WHERE YOU CAN TAKE ALGORITHMS INTO SOMEONE'S INSTITUTE, LEARN THE FEATURE SPACE OF THE DATA THAT YOU'RE WILLING TO LEARN ON. TAKE THE FEATURE SPACE AWAY AND SYNTHESIZE NEW DATASET WHICH LOOKS KIND OF LIKE THE ORIGINAL BUT ISN'T EXACTLY THE SAME. >> (INAUDIBLE). >> SO THAT HELPS ANSWER DISTRIBUTED LEARNING, ALSO USEFUL IN TERMS OF CREATING BILLIONS OF WEAKLY LABELED DATA POINTS. WE'VE SEEN A COUPLE GRAPHS TODAY ON HOW MUCH ACCURATE DATA LABEL YOU NEED, ACTUALLY YOU TALK ABOUT ELOQUENTLY ON IF YOU HAVE LOTS AND LOTS OF WEAKLY LABELED DATA, YOU GET THE SAME PERFORMANCE. SO IF WE CAN CREATE 10 BILLION SYNTHETIC CHEST X-RAYS WITH WEAK LABELS COULD WE GET (INAUDIBLE), NO ONE KNOWS YET. >> THEY HAVE DONE EXAMPLES WITH SYNTHETIC DATA, UNBELIEVABLY GOOD, SURPRISING. IF WE ADD THE PHYSICS OF ACQUISITION YOU MIGHT GET FURTHER ALONG THAT PATH. >> WE HAVE TIME FOR ONE MORE QUESTION. >> GO AHEAD. >> WELL, THANKS. ELLIOTT SIEGLE, UNIVERSITY OF BALTIMORE, HENCE THE ACCENT. [LAUGHTER] I'D LIKE TO THANK BRAD FOR TALKING ABOUT EXPLAINABLE A.I. IN THE BLACK BOX. I'D LIKE TO EMPHASIZE NOT JUST EXPLAINABLE A.I. TO EXPLAIN THE ALGORITHM AND WHAT THE ALGORITHM IN GENERAL IS DOING, BUT FOR A PARTICULAR CASE BEING ABLE TO UNDERSTAND WHY IT CAME UP WITH THAT SOLUTION, FOR EXAMPLE MAMMOGRAPHY CAD THERE'S BEEN A TREMENDOUS AMOUNT OF RESEARCH, 25 YEARS AGO IT WAS SAID IT WAS GOING TO REPLACE RADIOLOGISTS AND MAMMO GRAPHERS, IT'S BEING WIDELY USED, NOT CAPTURED. BECAUSE OF THE UNEXPLAINED ASPECT, WHAT WOULD BE GREAT WOULD BE ON A CLINICAL BASIS, ONE IS INTERPRETING SAY MAMMOGRAMS JUST FOR EXAMPLE NOT ONLY MARK A SPOT AND SAY THIS MEETS A THRESHOLD FOR MALIGNANCY BUT WOULD SAY WHY, IS IT MICROCALCIFICATION, LESION, ASYMMETRY OR WHAT IS IT ABOUT IT? I THINK REINVENTING A.I. APPLICATIONS, HOW WE USE THEM CLINICALLY MAY BE TO A LARGE EXTENT THEIR SUCCESS OR FAILURE MAY BE RELATED TO WHAT EXTENT WE CAN LOOK IN THE BOX FOR PARTICULAR CASE, WHEN I'M TALKING WITH A RESIDENT, THE RESIDENT WILL TELL ME WHY SHE OR HE BELIEVES THAT THIS CASE IS POSITIVE OR SUSPICIOUS, IT WOULD BE GREAT TO LOOK INSIDE THE BOX FOR EACH CLINICAL CASE AND DO THAT. DO YOU THINK THAT'S PRACTICAL AND HOW COMPUTATIONAL? SO MUCH IN THE LAYERS, SO MUCH INFORMATION WE DON'T SEE WHEN IT ESSENTIALLY GIVES US A YES OR NO ANSWER OR PUTS AN X ON A SPOT IT WOULD BE GREAT TO ELUCIDATE THAT. >> THERE'S A METHOD OF VECTORS, IF YOU HAVE A CONCEPT, SPECIFICATION OR CALCIFICATION. >> ASYMMETRY OR WHATEVER. >> YOU CAN TAKE THE FEATURES FROM THE NEURAL NETWORK AND REGRESS AGAINST THE CONCEPTS AND PROVIDE A SORT OF SCORE FOR THE THING, THEY HAVE DONE IT WITH ZEBRAS, LOOKING AT STRIPES, YOU CAN SEE THE STRIPES HELP THE CLASSIFICATION OF SOMETHING AS A ZEBRA VERSUS A HORSE, DOING THE SAME THINGS WITH MEDICAL IMAGES. IF YOU HAVE USER-DEFINED CONCEPTS YOU KNOW FOR ALL THE CASES YOU CAN REGRESS THE FEATURES IN THE NETWORK FOR DOING THAT. >> THAT COULD HAPPEN REAL TIME AS A CLINICAL READER IS INTERPRETING STUDIES. >> JUST A VERY QUICK COMMENT, DISCLOSURE, I WORK AT A COMPANY DOING A.I. AND WE HAVE ALL THE FEATURES YOU'VE MENTIONED, WE'RE WORKING ON USER TESTING TO SEE IF THEY DO HELP BUT I THINK YOU'RE RIGHT, THE PREVIOUS SESSION OF CAD WORK IS EXPENDABLE, AS ACCURATE (INDISCERNIBLE) CAN BE. WE CAN TAKE IT OFF LINE. >> I MENTIONED LIME. THERE'S A WEBSITE EXPLAIN-AI.ORG WITH SOFTWARE, TENSOR FLOW HAS SOMETHING YOU CAN ADD AT THE END WHERE IT WILL LET YOU SEE SOME OF THE VISUALIZATIONS THAT WE'VE SEEN. WE'VE RUN A LITTLE BIT OVER TIME BUT KRIS SAID THIS IS THE BEST SESSION OF ALL. [LAUGHTER] SO THANK YOU FOR YOUR ATTENTION. [APPLAUSE] >>> I'M ADAM FLANDERS FROM THOMAS JEFFERSON UNIVERSITY HOSPITAL CO-HOSTING WITH TESSA COOK FROM THE UNIVERSITY OF PENNSYLVANIA, SHE WILL ALSO BE SPEAKING. BRAD SAID SESSION D WAS THE BEST, IT'S GOING TO BE THE SECOND BEST BECAUSE SESSION C IS COMING. DATA NEEDS FOR MACHINE IMAGING. I'M PROUD TO PRESENT A GREAT LINEUP OF SPEAKERS TO CREATE PROVOCATIVE THINKING AND GET A LOT OF DISCUSSION. WE'LL HAVE GE WANG FROM THE RSI, DAVID MENDELSON FROM MOUNT SINAI, TESSA COOK FROM UNIVERSITY OF PENNSYLVANIA, KEITH BIGELOW FROM G.E. MEDICAL SYSTEMS AND MARK COLIFROM UNIVERSITY OF CALIFORNIA, SAN FRANCISCO AND ELLIOTT SEGAL FROM UNIVERSITY OF MARYLAND. LET'S GET GE TO THE STAGE TO TALK ABOUT REPOSITORIES OF -- DATA REPOSITORIES. >> THANK YOU VERY MUCH. I'M GOING TO TALK ABOUT A MODEL IMAGE DATASET, BASICALLY WE KNOWN THERE ARE TWO MAIN APPROACHES, FROM THE TOP DOWN, YOU HAVE A RULE BASED SYSTEM, YOU HEARD IN THE FIRST SESSION FROM CURT HOW (INDISCERNIBLE) THE REASON COMPUTER ASSURES VERY HOT. NOW WE HAVE MACHINE LEARNING BASIC METHOD, THE MAIN APPROACH OF ARTIFICIAL INTELLIGENCE NOWADAYS IS INDUCTIVE, SO THAT GOES FROM DATASET. SO DATASET IS PARAMOUNT IMPORTANCE. WE ALREADY SEE MANY MEDICAL IMAGE DATASETS AVAILABLE, HERE THIS IS A FEW EXAMPLES. WE HAVE ADNI, SO FOR OUR TEAM WORK DISEASES NEUROIMAGING, AND WE KNOW THIS IS A LONGITUDINAL STUDY, THE SECOND EXAMPLE FOR MULTI-MODAL BRAIN TUMOR SEGMENTATION CHALLENGE, DATASET HAS IMPROVED THIS YEAR WITH MORE IMAGES AND HUMAN LABELED RESULTS. ALSO WE HAVE THE CANCER IMAGING ARCHIVE AND BASICALLY CANCER IMAGES, MRI IMAGES AND T1, T2 PROTON WEIGHTED MRI, AND CHESTX-RAY8 PROVIDED DATASET VERY USEFUL, RECENTLY WE SEE MURA FROM STANFORD GROUP, INTENDED FOR MACHINE LEARNING TO IMPROVE PERFORMANCE. THIS YEAR WE ADDED TRANSACTION MEDICAL IMAGING, MACHINE LEARNING FOR IMAGE RECONSTRUCTION, HERE WE SEE DIFFERENT. THE MOST IMAGE ANALYSIS APPLICATION REALLY TALK ABOUT IMAGE ANALYSIS, EXTRACT FEATURES FROM IMAGES. SO GO FROM IMAGES TO FEATURES. ON THE OTHER HAND TOMOGRAPHIC RECONSTRUCTION, CASE-BASED DATA, LINE INTEGRALS, INDIRECTLY MEASURE THE DATA FEATURED. HERE WE WANT TO GO FROM FEATURES TO IMAGES, SO WE TALK ABOUT TOMOGRAPHIC IMAGES AND MEDICAL BIG DATA, A LOT OF KINDS, SO WE FOCUS ON TOMOGRAPHIC RECONSTRUCTION, IN THIS SPECIAL ISSUE INCLUDED HIGH-QUALITY PAPERS, THE DATASETS WERE USEFUL FOR MRI, CT, ULTRASOUND, OTHER MODALITIES. AGAIN, THIS SPECIAL ISSUE WE SEE MULTIPLE KINDS OF DATASETS. THE DATASETS WERE GENERATED BASICALLY FROM REAL MRI SCANS, NUMERICAL SIMULATION, PHYSICAL EMULATION AND DATASET, THE REAL KNOWN DATASET SIMULATED BRAIN DATABASE FROM IMAGING CENTER IN CANADA. SO THIS IS REALLY BASED ON FIRST PRINCIPLE, CODING, YOU REQUEST GENERATE MRI DATASET AS YOU WISH. AS YOU SEE HERE YOU HAVE T1 IN THE TOP ROW, T2 IMAGE IN THE BOTTOM, FROM LEFT TO RIGHT YOU SEE IDEAL IMAGE, AND TYPICAL, LARGE NOISE, LARGE INTENSITY, UNIFORMITY, VERY THICK SLICE, YOU CAN GENERATE AS MANY AS YOU WANT. SECOND ONE MRI, IXI EXTRACTION FROM IMAGES, FROM THREE HOSPITALS, THE GROUP IN U.K., SO YOU HAVE KINDS OF IMAGES SHOWN HERE. AND NOW WE TALK ABOUT CT DATASET, EXAMPLE IN OUR PAPER. WE UTILIZE THE CT DATASET, ESTABLISHED BY MAYO CLINIC, VERY USEFUL AND WE COLLECTED FROM THE MAYO CLINIC DATASET ABOUT 6,001-MILLIMETER THICKNESS FOR CT IMAGES FROM 10 PATIENTS, AND ALSO LOW-DOSE COUNTERPARTS WERE SIMULATED BY ARTIFICIAL NOISE INSERTION. WE RECEIVE THE PRETTY GOOD RESULTS AND SHOW THE REFERENCE IMAGES AND LEARNING BASED NEURONETWORK BASED RECONSTRUCTION, A LOT OF DETAILS IN THE PAPER. AND ON THE OTHER HAND WE CAN ALSO RELY ON NUMERICAL SIMULATION MORE. OUR GROUP APPROACH, IMAGES CT SCAN OF INDIVIDUAL PATIENTS, THEN WE SEE TO VISIBLE HUMAN PROJECT, SO THIS WAY WE CAN REALLY COLORIZE VISIBLE HUMAN. ALSO WE CAN USE THE FORMABLE ATLAS TECHNIQUE, INTRODUCE DEFORMATION TO GENERATE BIG DATA SET THAT WAY, MORE COST EFFECTIVE, ALSO LABEL CAN BE BROUGHT OVER. PATHOLOGICAL FEATURES COULD BE INTRODUCED IN NUMERICAL SIMULATION. FURTHERMORE WE HAVE A UNIVERSAL CT BENCHTOP DONATED FOR OUR LABORATORY, AND THIS IS INDUSTRIAL PLATFORM, SOLVES THE PHANTOM STAGES, THE NINE DEGREES OF FREEDOM, EMULATING ANY -- FOR EXAMPLE WE COULD GENERATE FISCALLY REALISTIC DATA, SO THERE ARE MULTIPLE POSSIBILITIES. WE SEE NEW OPPORTUNITY, WE GENERATE DATA NOT ONLY FROM SIMULATE AND EMULATE, COULDN'T COMBINE BOTH, THIS IS COLLABORATION, IN THE ROOM, ALSO STONY BROOK UNIVERSITY, IN THE CLOUD, WE'RE WORKING ON MACHINE LEARNING MEASURES TO GENERATE BIG DATA SO INVOLVING WE (INDISCERNIBLE) AND TRANSFER LEARNING, A NEW WAY BIG DATA NEEDS COULD BE ADDRESSED BASED ON MACHINE LEARNING NEURONETWORK APPROACH, ON THE LEARNING CURVE HOPE TO PRODUCE GOOD RESULTS IN THE NEAR FUTURE. AND THEN NOT ONLY THAT WE SAY MULTI-MODAL DATASET, MULTI-MODAL DATASET COULD BE OBTAINED ON HYBRID SCANNER, ALSO COULD USE REGISTRATION TO FUSE (INDISCERNIBLE) IMAGE MODALITY DATASET SO THIS SLIDE WAS KINDLY OFFERED BY MY COLLEAGUE, ALSO IN THE ROOM, HE USED 700 DATASETS, CROSS DATA MRI, TRACKED ULTRASOUND IMAGES, USED MACHINE LEARNING TO DO REGISTRATION AND WAS ABLE -- THE RESULT SHOWS THEY COULD REDUCE REGISTRATION ERROR FROM 16-MILLIMETER TO ABOUT A FULL MILLIMETER SO THIS DATASET AS SHOWN HERE. BASICALLY WE SAY FOR DATA-DRIVEN MACHINE LEARNING, MULTI-MODALITY DATASETS ARE NEEDED, PUBLICLY AVAILABLE FOR TOMOGRAPHIC RECONSTRUCTION AND SEVERAL TYPICAL DATASETS WERE SUMMARIZED, AND THEREFORE I WOULD LIKE TO UNDERLINE WE CAN'T OBTAIN BIG DATA NOT ONLY THROUGH SIMULATE AND EMULATE REAL SCANS BUT ALSO MACHINE LEARNING, TRANSFER LEARNING WILL PLAY A BIG ROLE, NOT ONLY INDIVIDUAL DATASET, MULTI-MODAL DATASET, COULD BE OBTAINED HYBRID SCANNER THROUGH DEEP REGISTRATION FOR UNIFIED ANALYSIS AND RADIOMICS. THANK YOU VERY MUCH. [APPLAUSE] >> ALL RIGHT. NEXT IS DAVID DAVID MENDELSON FROM MOUNT SINAI TALKING ABOUT PATIENT-MEDIATED DATA SHARING. >> THANK YOU VERY MUCH. SO I WAS ASKED TO ADDRESS A VERY SPECIFIC TOPIC, HOW PATIENTS CAN CONTRIBUTE TO SHARING, AND I'LL TOUCH ON THAT IN A MOMENT. I THINK WE HAVE TO ASK ABOUT DATA QUALITY, PEOPLE SPOKE ABOUT GROUND TRUTH TODAY, IN FACT ONE OF MY GREAT INTERESTS IS HOW WE APPROACH REALLY GETTING DATASETS THAT APPROACH GROUND TRUTH BECAUSE REALLY WE'RE AT A POINT WITH A STARTER SET RIGHT NOW A LOT OF WORK OVER THE LAST TWO YEARS ON IDENTIFYING BLOBS ON X-RAY EXAMS, WE CAN GO FURTHER WITH THE RIGHT DATASETS AVAILABLE. ONE OF THE QUESTIONS HOW CAN WE SHARE DATA, CONTRIBUTE DATA TO DIFFERENT RESEARCH ENDEAVORS AND CERTAINLY TODAY WE ALL DO THIS. THE ENTERPRISE SENDS THE DATA, SOMETIMES TO A PATIENT CONSENT AT THE PATIENT REQUEST, DO SEND STUDIES INTO DIFFERENT RESEARCH ENDEAVORS AT PATIENT REQUESTS. I'M GOING TO TOUCH ON OTHER WAYS THAT THIS CAN BE ACCOMPLISHED, ONE OF THE WAYS IS TO PUT THE IMAGES IN THE PATIENT'S POSSESSION, AND LET THE PATIENT DECIDE TO DIRECTLY CONTRIBUTE TO DATA. AND THE IMAGE, ONE WAY OF DELIVERING EXAMS INTO THE PATIENTS HANDS AND TALK ABOUT THE FEDERAL PROGRAM, NIH PROGRAM THAT STARTED TWO YEARS AGO, INTENDED TO MAKE IT EASY FOR PATIENTS TO CONTRIBUTE THEIR DATA TO RESEARCH ENDEAVORS. I JUST WANT TO FLAG EVERYBODY, THAT THERE ARE MULTIPLE WAYS OF EXCHANGING DATA. THERE ARE LOTS OF ARCHITECTURES THAT WE CAN TAKE ADVANTAGE OF OUT THERE, HEALTH INFORMATION EXCHANGES, COLLECT DATA CERTAINLY REGIONALLY AND THAT MAY BE ONE PLACE WE CAN BROADCAST DATA FROM. THERE ARE MANY PROPRIETARY SOLUTIONS OUT THERE BUILT INTO THE RSNA CLINICAL TRIALS PROCESSOR, EASILY ABILITY TO EXCHANGE AND CONTRIBUTE DATA. I'VE ALREADY MENTIONED RSNA IMAGE SHARE, SENT IMAGES INTO PATIENT-CONTROLLED PERSONAL HEALTH RECORD ENVIRONMENTS, AND FROM THAT POINT ON THE PATIENT COULD CONTRIBUTE THAT DATA WHEREVER THEY SO WISHED AND WILL COME OUT TO SYNC FOR SCIENCE BUILT ON MODERN WEB-BASED STANDARDS. YOU MAY HAVE HEARD ME SPEAK ABOUT THIS IN THE PAST BUILT ON A BANKING MODEL, MADE TO BE EASY IF YOU CAN GET CASH OUT OF AN ATM MACHINE FROM A BANK REPOSITORY SITTING BACK HERE, THROUGH A CLEARINGHOUSE INTERMEDIARY, WHY COULDN'T WE DO THE SAME WITH IMAGES? SO IN FACT WE BUILT AN ARCHITECTURE WHERE THE PATIENT IS STANDING ANYWHERE IN THE WORLD WITH A BROWSER AVAILABLE, AND THEY CAN IN FACT GET THEIR EXAMS FROM AN ARCHIVE NO MATTER WHERE IT SITS. AND THIS IS THE AC TECH TIRE THAT WAS USED -- ARCHITECTURE USED TO DO THAT. WE BUILD A CLEARINGHOUSE AND BUILT PERSONAL HEALTH RECORDS IMAGE ENABLED FOR WHICH THE PATIENT COULD HAVE AN ACCOUNT LIKE AN AMAZON ACCOUNT, COMMERCIALLY AVAILABLE RETAIL ACCOUNT, EASY ACCESS TO IMAGES ANYTIME, ANYWHERE THEY ARE. AND FROM THAT ACCOUNT THE PATIENT COULD SIMPLY SEND EXAMS ANYWHERE THEY WANT, A DESTINATION CAN BE RESEARCH ENDEAVOR. AND JUST TO LET YOU KNOW THERE WERE 35,000+ PATIENTS INVOLVED IN THE RSNA IMAGE SHARE WHICH CONCLUDED THIS YEAR, 145,000 EXAMS DISTRIBUTED, TWO VENDORS CONTINUED WITH THAT PROJECT. NOW COMES ALONG THE INH WITH A VERY AMBITIOUS PROJECT CALLED "ALL OF US" OR SYNC FOR SCIENCE TO MAKE IT EASY TO CONTRIBUTE ANY KIND OF DATA INTO ANY RESEARCH ENDEAVOR, THE GOAL IS ONE MILLION PATIENTS CONTRIBUTING DATA. SO THE PATIENTS WE APPROACH A RESEARCH APPLICATION TO THEIR PHONE THAT SAYS WOULD YOU LIKE TO CONTRIBUTE? ASK WHERE IS YOUR DATA, AUTHENTICATE YOU, WITH THE CLICK OF A BUTTON GO THROUGH AN EHR PORTAL AND SENT TO RESEARCH DESTINATION, EXTREMELY SIMP SIMPLE. THIS COMES FROM A TECHNICAL LEADER, JOSH MANDEL, SHOWING HOW WE MIGHT ADD IMAGING INTO THE CYCLE OF GETTING DATA. YOU'LL NOTICE IT HAS DICOM WEB AND FHIR TRANSACTIONS SO A PATIENT WITH A SIMPLE APPLICATION ON THEIR PHONE CAN TALK TO THEIR EHR, AND CONTRIBUTE ALL THE DATA IN THAT ENVIRONMENT TO A RESEARCH ENDEAVOR. A YEAR-AND-A-HALF AGO, A GROUP OF PEOPLE FROM THE RSNA IMAGE SHARE PROJECT, KRIS CARR IN THE AUDIENCE, WYATT FROM UCSF WITH MY ASSISTANCE WORKED OUT HOW TO TAKE THE RSNA IMAGE SHARE ARCHITECTURE, PUTTING IN BROKERS, GETTING IT TO WORK ALONGSIDE THE SYNC FOR SCIENCE APPLICATION. SO THEY HAVE DELIVERED A REFERENCE IMPLEMENTATION, WHICH PUTS A DICOM WEB BROKER AND FHIR BROKER IN FRONT OF THE RSNA IMAGE SHARE APPLICATION, AND THEY HAVE DEMONSTRATED NOW IN PUBLIC DEMONSTRATIONS HOW THIS CAN WORK AND SET A RESEARCH REPOSITORY AS DESTINATION FOR IMAGES, ADDED TO THE SYNC FOR SCIENCE RESEARCH PROGRAM. IT TAKES ADVANTAGE OF ALL THE COMMON WEB STANDARDS TODAY, AN ATTEMPT TO MODERNIZE THE RSNA IMAGE SHARE TO MODERN WEB-BASED STANDARDS. HERE ARE SOME REFERENCE SITES WHERE YOU CAN FIND OUT ABOUT THE SYNC FOR SCIENCE AND ARCHITECTURE USED FOR THIS. I WANT TO BRANCH FOR A MOMENT BECAUSE I THINK THE SYNC FOR SCIENCE PATIENT CONTRIBUTION IS VERY IMPORTANT BUT THINK WE NEED HUGE RESOURCES OF DATA TO REALLY MAKE ALL THE A.I. TECHNICAL WORK REALLY VERY USEFUL DOWN THE ROAD AND MUCH MORE COMPATIBLE WITH A VISION OF GROUND TRUTH AND REALLY DOING MORE THAN JUST FINDING BIG BLOBS ON X-RAY EXAMS. THE NIH IS SPONSORING A FAIR CONCEPT, FINE FINDABLE, ACCESSIBLE, INTEROPERABLE BIG DATA. I WANT TO THINK ABOUT WHAT WE DEVELOPED AT MOUNT SINAI WHICH IS A NOTION I WAS SURPRISED THAT IS NOT VERY COMMON, WHAT WE DO NOW IS WE BUILT AN IMAGING RESEARCH WAREHOUSE WHICH IS DE-IDENTIFIED, PSEUDOANONYMIZED DATA, EVERY EXAM COMING BOO THE LOCAL PACS SYSTEM IS PUSHED THROUGH AN ENGINE, DE-IDENTIFIED, PSEUDOANONYMIZED INTO AN IMAGE WAREHOUSE, IDENTIFYING VENDORS, DICOM TAGS, RIGOROUS PROCESS. LAST SPRING STARTED EXPORTING EXAMS AND PACS, A THOUSAND A DAY, WE RIGHT NOW HAVE OVER 200,000 EXAMS, THAT GROWS BY A THOUSAND A DAY. MOUNT SINAI MERGED WITH ANOTHER ENTERPRISE, TOGETHER WE HAVE ABOUT A MILLION EXAMS GENERATED A YEAR. OUR INTENT IS TO ACCOMPLISH DE-IDENTIFYING THE FULL MILLION EXAMS A YEAR OVER THE NEXT FEW YEARS. WE DID THIS WITH A COMMERCIAL VNA TO MORPH DATA, MASK PHI, AN EXPORT RULE LOOKS FOR THINGS WE KNOW ARE TROUBLESOME THAT CONTAIN PIXEL DATA TO WHICH WE APPLY OTHER TECHNIQUES TO REMOVE, THE BURN INDEX IN THE IMAGES AND FILL 200,000, BUILT FOR MULTI-SITE COLLABORATIVE RESEARCH, SO TOGETHER THE NOTION THAT PATIENTS CAN CONTRIBUTE DATA THAT'S ESPECIALLY ATTRACTIVE WHERE YOU THINK ABOUT PATIENTS WITH UNIQUE DISEASES, FROM USER GROUPS, THINGS LIKE INFLAMMATORY BOWEL DISEASE, SARCOIDOSIS, THOSE PATIENTS CAN PROVIDE US A LARGE COHORT OF DATA WITH SPECIFIC DISEASES IN MIND, AND THEN ON THE OTHER END OF THIS USING THINGS LIKE AN IMAGING ESEARCH WAREHOUSE, TO CONTRIBUTE HUNDREDS OF THOUSANDS IF NOT MILLIONS OF EXAMS TO MULTI-SITE COLLABORATIVE DATA AIMED AT GROUND TRUTH. SO I THANK YOU, AND I'LL STOP THERE. [APPLAUSE] >> THANK YOU, DAVID. NEXT SPEAKER IS TESSA COOK FROM UNIVERSITY OF PENNSYVANIA TO TALK ABOUT STRUCTURED EMR DATA. >> THANK YOU, ADAM. GOOD MORNING, EVERYONE. SO, WE FACE A NUMBER OF CHALLENGES AS WE TRY TO EXTRACT MEANINGFUL INFORMATION FROM HEALTH DATA. ONE OF THE BIGGEST THAT I ENCOUNTER, I SUSPECT DOES THE SAME FOR MANY OF YOU, THE PLETHORA OF UNSTRUCTURED DATA WE HAVE TO MANAGE INCLUDING RADIOLOGY REPORTS, BUT ALSO VARIETY OF TEXT-BASED DOCUMENTS IN THE ELECTRONIC MEDICAL RECORD. OFTENTIMES DATA COME FROM DISPARATE SOURCES, YOU HAVE TO FIGURE OUT HOW TO MAKE THINGS TALK TO EACH OTHER, CLEAN THE DATA, MAKE SURE THEY ARE IN COMPATIBLE FORMAT AND MAKE SURE MULTIPLE PATIENT IDENTIFIERS ARE RECONCILED SO THAT YOU KNOW WHAT DATA BELONGS TO WHICH INDIVIDUAL, AND THAT CAN BE TRUE EVEN WITHIN A PARTICULAR PRACTICE OR HEALTH SYSTEM. SO, SOME OF THE APPROACHES TO DEALING WITH THE CHALLENGE OF UNSTRUCTURED DATA TO INTRODUCE STRUCTURE WHERE THERE PREVIOUSLY HASN'T BEEN. AND THAT HAS BEEN TRUE AS FAR AS STRUCTURED REPORTING IN RADIOLOGY WHICH WE'VE TALKED ABOUT HAS BEEN ADOPTED AT DIFFERENT PRACTICES, ALSO USE OF TEMPLATES IN THE EMR SINCE ELECTRONIC MEDICAL REPORTS ARE PERVASIVE NOW. THE RELATED INITIATIVE TO STRUCTURED REPORTING INCLUDE MRRT, MANAGEMENT OF RADIOLOGY REPORT TEMPLATES, AN IHE PROFILE, TO MAKE INDIVIDUAL STRUCTURED REPORTING TEMPLATES MORE EASILY DISTRIBUTED,ENED INCORPORATING INTO SYSTEMS SO WE CAN STANDARDIZE SOME TEMPLATES. RELATED IS CDE EFFORT, COMMON DATA ELEMENTS EFFORT, TO START TO MEANINGFULLY DEFINE DISCRETE DATA ELEMENTS INCLUDEDDED IN A DATA REPORT AND EXTRACT FOR ANALYSIS. ONE OBJECT TO INTRODUCE STRUCTURE, THE OTHER HAS BEEN TO AUTOMATE DATA EXTRACTION WITH NATURAL LANGUAGE PROCESSING AND NOW MORE RECENTLY WITH MACHINE LEARNING TECHNIQUES AS WELL. SO, WE HAVE USED STRUCTURED REPORTING TO TRY TO EMBED DISCRETE DATA IN REPORTS, I'D LIKE TO SHARE WITH YOU TWO EXAMPLES FROM OUR INSTITUTION. THE FIRST HAS TO DO WITH ABDOMINAL IMAGING, AND THE GOAL WITH THIS EFFORT WAS TO TRY TO IDENTIFY PATIENTS WHO HAD FINDINGS OF POSSIBLE MALIGNANCY ON ABDOMINAL IMAGING AND MONITOR TO COMPLETION OF RECOMMENDED FOLLOW-UP. IN ORDER TO DO IS THIS WE DEVELOPED AN INTERNAL RECOLLECTION CON, CODE ABDOMEN, LOOSELY BASED ON THE BIRADS LEXICON TO IDENTIFY AND MAP TO ONE OF THREE CATEGORIES ESSENTIALLY, BENIGN INDETERMINATE OR SUSPICIOUS LESION. THE WAY WE DID THIS IS WE HAD RADIOLOGISTS EMBED A SMALL TEMPLATE AT THE BOTTOM OF THE LOOSELY OR UNSTRUCTURED REPORTS TO INDICATE CLASSIFICATION FOR A SPECIFIC SUBSET OF ORGANS IN THE ABDOMEN. WE BUILT ON TOP OF THAT AN AUTOMATED RADIOLOGY RECOMMENDATION TRACKING ENGINE THAT WOULD MINE THE DATA FROM THE REPORTS AND IDENTIFY PATIENTS WITH FOLLOW-UP RECOMMENDATIONS AND THEN BE ABLE TO LOOK THROUGH TIME TO FIGURE OUT WHICH PATIENTS COMPLETED FOLLOW-UP, WHO WAS STILL SCHEDULED FOR FOLLOW-UP AND THOSE PATIENTS THAT WE DID NOT HAVE ANY RECORD OF FOLLOW-UP BEING COMPLETED FOR ONE REASON OR ANOTHER. AND SO FOR THE PATIENTS THAT HAD NO DOCUMENTED FOLLOW-UP, WE'VE BEEN ABLE TO BUILD A SYSTEM THAT SENDS NOTIFICATIONS TO THE PHYSICIAN WITH THE GOAL A PATIENT WHO CLINICALLY NEEDS FOLLOW-UP ACTUALLY GETS THEIR FOLLOW-UP, AND WE'RE ABLE TO DECREASE THE NUMBER OF ADVERSE OUTCOMES, UNDIAGNOSED CANCERS AND ADVANCED DISEASE THAT MAY BE DIAGNOSED LATER ON. IN THE PROCESS WE ALSO CREATED A LABELED DATASET OF REPORTS WE CAN USE FOR OTHER APPLICATIONS. ABOUT TWO YEARS INTO THAT INITIAL EFFORT WITH CODE ABDOMEN WE GOT FEEDBACK FROM ONCOLOGY COLLEAGUES THAT IT WASN'T REALLY RELEVANT IN THE ONCOLOGY PATIENT POPULATION BECAUSE THOSE PATIENTS WERE VERY CLOSELY FOLLOWED AND RARELY LOST TO FOLLOW-UP. AND SO WE ADAPTED THIS SPECIFICALLY FOR PATIENTS WITH EXISTING CANCER DIAGNOSIS AND CREATED CODE ONCOLOGY. VERY LOOSELY BASED ON RESIST CRITERIA BECAUSE OUR RESIST REPORTING IS NOT ACTUALLY DONE BY THE RADIOLOGISTS THEMSELVES. AND SO TWO CATEGORIES HERE, ONE WITH QUALITATIVE DESCRIPTORS OF EXISTING LESIONS, ONE WITH QUALITATIVE DESCRIPTORS OF NEW LESIONS AS YOU CAN SEE HERE. AND SO AGAIN USING A STRUCTURED TEMPLATE RADIOLOGISTS WOULD CHOOSE ONE OF THESE TWO VALUES FOR ONE OF THESE TWO -- FOR EACH OF THE TWO CATEGORIES TO PUT INTO THEIR REPORT. USING THIS LABELED DATA WE TRIED TREATMENT RESPONSE, TOOK NINE CATEGORIES FOR EXISTING LESIONS AND THREE FOR NEW LESIONS, AND BROADLY MAPPED THEM INTO FOUR SMALLER CATEGORIES, SUPER-SETS OF PROGRESSION, IMPROVEMENT, STABLE DISEASE AND RESOLUTION OR NO CANCER. AND WE USED A COMBINATION OF NLP PRE-PROCESSING, APPLIED WITH THE TEMPLATE REPORT, TEMPLATE AS GROUND TRUTH, JUST THE IMPRESSION OF THE REPORT AND WE NOTICED WHEN WE USED THE IMPRESSION ALONE WE GOT BETTER PERFORMANCE. AND ALSO AS WE WENT THROUGH ALL OF THE OTHER ASPECTS OF THIS REALIZED HOW SENSITIVE SOME OF THESE METHODS WERE TO THE HETEROGENEITY AND REPORTS THEMSELVES. DIFFERENCES IN PHRASING, IN STYLE. AND SO WE HAVE CERTAINLY MORE WORK THAT NEEDS TO BE DONE. SO THERE ARE MULTIPLE CHALLENGES AND GAPS AND OPPORTUNITIES AS WELL. ONE OF THE THINGS THAT WOULD HELP US IN GENERATING THIS DATA IS THE AVAILABILITY OF SMART TEMPLATES. IN OTHER WORDS, RESPONSIVE TEMPLATES THAT AS THE RADIOLOGIST IS GENERATING THE REPORT, IDENTIFIES IMPORTANT CONCEPTS THAT ARE RELATED TO THE FINAL CONCLUSION TO ANY POTENTIAL RECOMMENDATIONS AND AUTOMATICALLY POPULATES THEM ELSEWHERE, RATHER THAN POTENTIALLY CREATING AN OPPORTUNITY WHERE THERE MAY BE CONTRADICTIONS IN THE REPORT. WE ALSO NEED A WAY TO PRESERVE AND EXTRACT DISCRETE DATA ELEMENTS, AGAIN CDE EFFORT WILL BE VALUABLE FOR THIS. NOT ONLY IN THE RADIOLOGY REPORT BUT ALSO IN ALL OF THE VARIOUS TEXT-BASED DOCUMENTS IN THE ELECTRONIC MEDICAL RECORDS. ONE OF THE INTERESTING THINGS THAT WE HAVE COME ACROSS IN DOING THE RESEARCH WITH THESE TWO LABELED REPORT DATASETS IS THE DISCORDANCE BETWEEN FREE TEXT AND CONTENT OF THE TEMPLATE THAT HAS THE RECOMMENDATION OR THAT HAS THE INDICATION OF DISEASE STATE. SO, AGAIN, GETTING BACK TO THE RESPONSIVE TEMPLATES WE NEED A WAY TO MINIMIZE DISCORDANCES THAT WE'RE GENERATING. A CHALLENGE AND GAP REALLY IS THE NARROW ADOPTION OF MRRT AND CDE, TO HELP IN STANDARIZING THIS DATA. IN SUMMARY THERE ARE MANY POTENTIAL OPPORTUNITIES, NOT ONLY IN THE DEVELOPMENT OF RESPONSIVE TEMPLATES, THE ABILITY TO REALLY DISCREETIZE DATA WE NEED TO CAPTURE OVER THE UNSTRUCTURED TEXT AND PRESERVE IN THE REPORT, RATHER THAN THE CURRENT SITUATION WHERE YOU HAVE FIELDS DEFINED IN A REPORT BUT IT'S POSSIBLE FOR THE PERSON THAT'S GENERATING THE REPORT TO DELETE THAT FIELD AND REPLACE IT WITH FREE TEXT. IMPROVED METHODS FOR AUTOMATED EXTRACTION AND APPLY EXISTING ONTOLOGY, HARMONIZED ONTOLOGY TO HELP INDEX AND PARSE AND EXTRACT MEANINGFUL INFORMATION FROM THE UNSTRUCTURED DATA WITH WHICH WE WORK NOW. THANK YOU VERY MUCH. [APPLAUSE] >> WONDERFUL, TESSA. FINAL SPEAKER BEFORE THE DISCUSSION AND Q&A IS KEITH BIGELOW FROM GENERAL ELECTRIC WHO IS GOING TO TALK ABOUT DATA AGGREGATION AND SHARING ON THE COMMERCIAL SECTOR. >> GOOD MORNING. KEITH BIGELOW. I LEAD UP THE ANALYTICS OF A.I. PLATFORM FOR G.E. I THOUGHT IT WOULD BE INTERESTING FROM A COMMERCIAL PERSPECTIVE TO SHARE SOME OF THE CHALLENGES THAT WE RUN INTO AS WE TRY TO PRODUCTIZE, AS KEITH DREYER SAID, THE INFINITE PRODUCT SPACE OF MODALITY, ALL THE PATHOLOGY'S THAT WE -- PATHOLOGIES THAT WE MIGHT BE ABLE TO SOLVE, TO SOLVE THE INSOLVABLE PRECISION HEALTH SAY CROSS A QUARTER -- RATHER SEVERAL BUSINESSES, ONE IS AROUND DIAGNOSTICS, WHICH IS WHY WE'RE HERE TODAY, OUR THERAPIES GROUP AND FINALLY OUR MONITORING GROUP. FOR EACH OF THESE THEY ARE ALL GOING THROUGH A VERY SIMILAR TRANSFORMATION, AS WE SAW FROM SOME OF OUR PREVIOUS SPEAKERS, FROM MACHINE LEARNING TO DEEP LEARNING. AND I WANTED TO SHOW YOU A LITTLE BIT OF OUR JOURNEY SO FAR. THE FIRST PIECE IS DEEP LEARNING AND DATA VOLUME, THE PURPOSE OF THE SLIDE TO CONVEY HOW VORACIOUS IN TERMS OF DATA FOR A SPECIFIC CONDITION DEEP LEARNING IS. AND SO YOU MAY HAVE A PETABYTE OF DATA IN YOUR INSTITUTION OR LIKE THE V.A. MAYBE 30 PETABYTES OF INFORMATION BUT HOW MUCH DATA DO YOU HAVE THAT'S EQUALLY BIASED FOR A SPECIFIC MEDICAL CONDITION FOR A SPECIFIC MODALITY, LIKE KEITH DREYER SAID. RAPIDLY IT SHRINKS. AS YOU LOOK ACROSS, AS WE ADD DATA, AND WE LOOK AT UNBIASED DATA, AS WE DID HERE, WE CAN RAPIDLY SEE THAT AS WE APPROACH A THOUSAND WE CAN IMPROVE OUR ACCURACY, 5,000 POSITIVE INSTANCES BECOMES A REALLY GOOD LOCATION FOR US IN TERMS OF IMPROVING THAT YET MORE. AND THEN AS WE HEARD FROM MATT, HEY, YOU CAN GET REALLY GREAT ACCURACY IF YOU HAVE 100,000 LABELED IMAGES, SO THIS IS ALL COMING BACK TO YOU, AND TO US, HOW MUCH DATA DO YOU HAVE, AND IT REALLY WILL DRIVE THE ACCURACY OF WHAT YOU CAN RECEIVE. NOW, WHAT'S REALLY INTERESTING IS IT'S NOT JUST ABOUT THE VOLUME. IT'S ABOUT THE DIVERSITY OF THE DATA THAT YOU HAVE. SO IF YOU ARE A SINGLE INSTITUTION, YOU PROBABLY HAVE A COMMON SET OF PROTOCOLS, BEST PRACTICES THAT YOUR TEAMS USE, YOU PROBABLY ALSO HAVE A CERTAIN SET OF VENDOR EQUIPMENT THAT YOU USE. AND AS WE HAVE BEGUN OUR JOURNEY OVER THE LAST, SAY, FIVE YEARS OR SO IN DEEP LEARNING, WE'VE BEEN DRAWING DATA FROM MANY DIFFERENT SITES, BECAUSE AS YOU CAN SEE HERE IN DIFFERENT COUNTRIES THE FIELD OF VIEW IS DIFFERENT. AND IN THE CASE OF A CHEST X-RAY IF WE'RE LOOKING FOR A PNEUMOTHORAX, MY FIELD OF VIEW INCLUDES THE ARM, THIS IS A PNEUMOTHORAX AS FAR AS A DEEP LEARNING ALGORITHM IS CONCERNED. IF THAT FIELD OF VIEW IS EVEN LOWER, ANYTHING THAT'S IN MY DIGESTIVE TRACT THAT IS AN AIR POCKET IS A PNEUMOTHORAX. WE NEED TO THINK IN TERMS OF CULTURAL APPLICATIONS OF THE USAGE OF THE MODALITY FOR THE PURPOSE BECAUSE OTHERWISE WE'RE GOING TO OPTIMIZE FOR OUR NARROW POPULATION, I THOUGHT IT WAS BRILLIANT TO GIVE THE ATLAS EXAMPLE EARLIER THIS MORNING, AWESOME. WE HAVE AN ATLAS OF THE SMALL SET OF YOUNG WHITE KIDS. THAT DOESN'T WORK FOR COMMERCIAL ENTITY THAT'S TRYING TO PROVIDE HEALTH CARE, PRECISION HEALTH, IN 120 COUNTRIES, AND THOSE GROWING COUNTRIES, BE IT INDIA, CHINA OR AFRICA, THEY AREN'T REPRESENTED IN THE DATASETS, SO AS A RESULT WE'RE BIASING IF WE DON'T REACH OUT TO BUILD THOSE. I LOVED DAVID SESSION, WE NEED TO THINK IS DATA CURRENCY? OR IS DATA LIKE BLOOD AND WE SHOULD BE DONATING IT? I WOULD ARGUE THAT IF WE ACTUALLY WANT TO BRING ABOUT THE IMPACT THAT ARTIFICIAL INTELLIGENCE GIVES US, IF WE TRADE IT LIKE CURRENCY WE'LL HOARD AND INTEREST THE TRAGEDY OF THE COMMONS. IF WE MOVE FORWARD AND LOOK AT ANNOTATION, THIS COMES BACK TO YOUR VOLUME AND SOURCES, SO NOW AS WE LOOK AT ANNOTATION, HEY, WE CAN DO WEAK LABELS, IF YOU HAVE 100,000 INSTANCES YOU CAN IN FACT USE WEAK LABELS, AND GET 90% OR THEREABOUTS, BUT WE'VE BEEN ABLE TO SHOW THAT ON SMALLER DATA SETS, ESPECIALLY AS YOU THINK OF THE LONG TAIL OF THAT INFINITE MATRIX THAT KEITH DREYER DISCUSSED YOU'LL HAVE TO WORRY ABOUT SMALLER POSITIVE ENDS AND IN THAT CASE YOUR ANNOTATION HAS TO BE MUCH, MUCH STRONGER. HERE WE CAN SHOW WITH SMALL DATASETS BUT WITH VERY GRANULAR PIXEL LEVEL ANNOTATION WE CAN BRING UP THE QUALITY OF OUR MODELS DRAMATICALLY. AND YOU CAN MIX THESE. SO YOU DON'T NEED TO THINK OF THIS AS BINARY ONE OTHER THE AREA, YOU CAN HAVE A LARGE N WHERE YOU DELICATELY AND PRECISELY DELEGATE A SMALL PORTION, THE ALGORITHMS CAN LEARN FROM THE COMBO, AN AREA OF RESEARCH THAT'S FASCINATING. BUT AS YOU SAW FROM TESSA AND FROM A NUMBER OF THE COLLEAGUES TODAY, RETROSPECTIVE CURATION NOT ONLY IS EXPENSIVE AS YOU CAN SEE HERE, NOT ONLY IS IT KIND OF MUNDANE FOR THE OPERATORS DON'T WANT TO DO IT, BUT IT'S EXPENSIVE. IT'S REALLY EXPENSIVE. AND SO WE WILL SEND OUT DATASETS ACROSS THE WORLD, THE EXACT SAME ONES AND LOOK AT VARIANCE IN CURATION BECAUSE AS PRIOR SPEAKERS MENTIONED ALMOST YOUR CULTURE IS REFLECTED IN HOW YOU CURATE. AND THEN WE CAN SEE BY INFERENCING ON CURATED IMAGES WHERE IS THE VARIANTS, BACK TO A COMMENT FROM MATT TODAY. WE CAN START TO GUIDE OUR CURATORS TO COMMON NORMS BECAUSE NORMALIZATION DATA IS EVERYTHING FOR A.I., AND THEN WE CAN GET CONSTANT QUALITY FROM ALL OUR CURATORS, REGARDLESS OF LOCATION, REGARDLESS OF PRICE. AND THIS IS SOMETHING WE FIND COMPELLING TO WORK ON. I AGREE WITH TESSA'S COMMENTS, TOOLS WE HAVE, DEVICES WE BUILD, THE PACS, VNA, THESE NUMBER ANNOTATION DEVICES. THERE SHOULD NEVER BE, AS CURT WAS SAYING, AN ACTION OF A CARE PATHWAY WHERE THE DATA IS NOT CONSTANTLY ANNOTATED, YOU THE SHOULD BE PART OF THE WORK FLOW, NOT RETROSPECTIVE TO THE WORK FLOW, WHICH BRINGS US TO THE QUESTION IF THIS IS A BRILLIANT ROBUST AUDIENCE OF INDIVIDUALS IN THE ROOM, AND IT IS, AND YOU'RE ALL CREATING ALGORITHMS, WHICH YOU ARE, 70% OF YOU RAISED YOUR HANDS, AWESOME, HOW DO WE MAKE THOSE MODELS PORTABLE? SO IF YOU'RE WORKING WITH PHILLIPS EQUIPMENT OR WORKING WITH G.E. EQUIPMENT OR YOU'RE WORKING WITH AN AGFA OR G.E. PAX, HOW DO WE DEPLOY TO TARGET END POINTS AND STILL HAVE FIDELITY? THIS GOES BACK TO DATA VOLUME AND DIVERSITY. IF I TRAIN ON ONE VENDOR I HAVE NO PORTABLE ENDPOINT TO A SOFTWARE APPLICATION. LET THAT SINK IN. IF YOU'RE BUILDING AN ALGORITHM AND YOU HAVE A FIXED VENDOR YOU'RE USING YOU CANNOT DEPLOY TO A COMMERCIAL MARKETPLACE AND THINK YOUR ALGORITHM IS VALID BECAUSE IT'S BLIND TO THE OTHER VENDORS OUT THERE, A PROBLEM WHEN IT COMES TO COMMERCIALIZING SO WE'RE TRYING TO MAKE THIS PIPELINE POSSIBLE FOR THE CREATION OF THE BEAUTIFULLY CURATED DATASET WITH A GREAT CATALOG THAT THEN WE CAN ALLOW OTHERS TO COME INTO AND BRING THEIR OWN I.T. INTO SO THAT WE CAN HELP THEM TARGET ALL THE ENDPOINTS ON THE FAR RIGHT OF THE SLIDE. AND THIS TO US IS THE NIRVANA, HOW DO WE CREATE ENOUGH DATA, CURATED CONSISTENTLY SO WE HAVE GROUND TRUTH, WITH ENOUGH DIVERSITY SO WE CAN TARGET ALL THESE ENDPOINTS. BUT BEFORE WE TARGET ANY OF THOSE ENDPOINTS FROM A COMMERCIAL SPECTOR WE NEED TO WORRY ABOUT VALIDATION, COMING BACK TO AN ALGORITHM FOR A MODALITY FOR A PATHOLOGY, THAT'S ONLY AS GOOD AS INPUT DATA AND WHERE YOU VALIDATE IT. SO THERE WILL BE ALGORITHMS FOR WHICH WE DO NOT NEED TO HAVE REPRESENTATION, FROM EVERY COUNTRY IN THE WORLD. WONDERFUL. SO THEN WE CAN VALIDATE THOSE IN A BROADER WAY AND MAY VALIDATE THE SAME ALGORITHM NEEDING DATA FROM EACH OF THOSE POPULATIONS, WHICH IS GOING TO MAKE THIS VERY COSTLY OVER TIME. WE TALKED ABOUT TRANSFER LEARNING AND CHALLENGES OR BENEFITS FROM THAT BUT ALSO THINK THROUGH THE LIFE CYCLE, SO FROM A VALIDATION PERSPECTIVE IF I SHIP VERSION 1 WITH TEN COUNTRIES, VERSION 1.1 ADDS TEN MORE, I HAVE TO GO BACK TO INSTITUTIONS OR MIGHT RISK CATASTROPHIC FORGETTING, A PROBLEM. THIS IS WHERE SOME COMMENTS WE HEARD FROM THE MARTINOS CENTER WERE INTRIGUING, CAN WE TAKE DATA, GE MENTIONED THIS AS WELL, SUFFICIENTLY TRANSFORM IT SO WE CAN TAKE THAT VIRTUAL COPY AND MAINTAIN THAT IN PERPETUITY AS WE KEEP ITERATING THROUGH THE ALGORITHM, IT MUST BE CONSTANTLY VALIDATED. THIS IS GOING TO BE A REAL CHALLENGE COMMERCIALLY FOR EVERY VENDOR AND EVERY INSTITUTION THAT WANTS BROAD DEPLOYMENT, CAN I REPRODUCE AND SHOW PROVENANCE, THAT IT'S STILL VALID AS I ADD NEW DATA THAT I HAVEN'T LOST FIDELITY? AND FINALLY FOR US WE WORRY ABOUT COUNTRIES, SO, AGAIN, ETHNICITY, SOCIOECONOMIC CAPABILITIES, AGE AS WE ALREADY DISCUSSED, ALL OF THIS DIVERSITY IS CRUCIAL TO BEING ABLE TO DEPLOY AT SCALE. SO, SUMMARY, YES, THERE'S A HUGE GAP IN DATA VOLUMES, AND QUALITY FOR THAT DATA, FIELD OF VIEW, ALL THOSE THINGS ARE RADICALLY DIFFERENT. DIVERSITY FOR COMMERCIAL IS A MASSIVE ISSUE. WE WANT TO CREATE GLOBAL SOLUTIONS THAT ADDRESS EVERYONE ON THE ECONOMIC HIERARCHY. WE DO SEE AS OTHER SPEAKERS HAVE MENTIONED THOSE ANNOTATION GAPS, AND IF YOU HAVE DELTAS IN ANNOTATION AS WE'VE SEEN ACROSS COUNTRIES AND CULTURES, YOU HAVE TO TRAIN THROUGH THAT OR YOU'RE GOING TO HAVE GARBAGE IN, GARBAGE OUT, AT SCALE. STANDARDS FOR PORTABILITY OF MODELS ARE CRUCIAL SO WE'RE WORKING WITH THE BIGGEST VENDORS AS YOU SAW IN THE CLOUD, CHIP SETS, TO CREATE INPUTS AND OUTPUTS THAT ARE CONSTANT SO THAT WE CAN PORT THOSE MODELS REGARDLESS OF WHO CREATES THEM THROUGHOUT THE ECOSYSTEM AND FINALLY THERE'S NOTHING MORE IMPORTANT THAN THIS VALIDATION STEP, AS I MENTIONED. I THINK THIS IS SOMETHING WHERE WE NEED EVERYONE'S HELP BECAUSE WE HAVE TO MAKE SURE THAT WE ARE EQUALLY TREATING WITHOUT BIAS ALL OF OUR PATIENTS. AND THAT IS OUR DUTY. IT IS ALSO OUR COMMERCIAL OBLIGATION FROM A CAPABILITY PERSPECTIVE. AND SO I WOULD JUST ASK FOR YOUR HELP THERE BECAUSE THIS IS -- THERE'S NO SINGLE PLAYER THAT'S GOING TO WIN THIS. THIS IS GOING TO BE A GROUP OF US THAT DO IT TOGETHER. SO THANK YOU. [APPLAUSE] >> SO LET'S GET ALL OF OUR PANELISTS UP HERE ON STAGE, INCLUDING MARCOLI FROM UCSF AND FROM THE UNIVERITY OF MARYLAND. THERE'S AN EXTRA CHAIR. SOMEONE CAN SIT ON MY LAP. >> I HAVE A QUESTION FOR KEITH. DO WE NEED DICOM HEADERS FOR TRAUNCHS OF DATA, CLOTHES SAYS 65% POLYESTER, 25% WOOL, SO FORTH. WITH TRAUNCHES OF DATA WE DON'T SPECIFY WHAT'S IN THE BOX, G.E., OR 85% ASIAN-AMERICAN POPULATION, WE HAVE KNOW WAY OF DEFINING WHAT'S IN OUR TRAUNCHES OF DATA. >> AWESOME POINT. ACR DSI IS LEADING WITH THE DICOM STANDARDS GROUP, NEW METADATA WE SHOULD BE TAGGING AND ARGUING WHETHER IT NEEDS TO BE OR NEW STANDARDS OR EXPAND. INDUSTRY AS WELL AS ACADEMICS ARE WORKING TOGETHER RIGHT NOW TO TRY AND DRIVE FORWARD HOW DO WE EXPAND THE TAGS IN DICOM TO ENSURETHAT WE HAVE THIS TYPE OF INFORMATION, LIKE EVEN THE MODEL VERSION AS WELL AS IF WE'RE GOING TO START DOING INFERENCING AND DROPPING THAT BACK INTO THE DICOM, SO THAT TYPE OF METADATA WE WANT AS WELL AS LIKE YOU SAID GIVE ME SOME NON-RETRACEABLE ATTRIBUTES OF MY POPULATION SO THAT I KNOW WHAT MY MODEL LOOKS LIKE. SO WE'RE WORKING ON THAT. >> WELL, I'M REALLY HAPPY TO SEE FOLKS FROM THE NATIONAL LIBRARY OF MEDICINE HERE BECAUSE THE FOLKS FROM THE NATIONAL LIBRARY OF MEDICINE HAVE DONE SUCH A GREAT JOB WITH TAGGING AND ALLOWING US TO SEARCH AND FIND A VARIETY OF PUBLICATIONS. IT WOULD BE GREAT TO SEE THE NATIONAL LIBRARY OF MEDICINE TACKLE THE WHOLE IDEA OF BEING ABLE TO FIND WAYS TO LABEL DATA SETS, GREAT TO HAVE AN INDEX AND REPOSITORY, SORT OF EQUIVALENT OF PubMed FOR PUBLICATIONS, FOR THESE TYPE OF DATASETS. IT WOULDN'T ONLY INCLUDE DICOM DATA, BUT COULD INCLUDE A VARIETY OF OTHER TYPES OF DATA THAT ONE MIGHT USE TO CATEGORIZE PUBLICATIONS ALSO. SO I'D LOVE THE QUESTION AND WOULD VERY MUCH -- THERE'S LOTS OF DATA OUT THERE, LOTS OF US ARE LOOKING FOR SOURCES MUCH DATA, COMMERCIAL FOLKS LOOKING AT SELLING DATA, BUYING DATA, ET CETERA, AND MECHANISMS TO DISCOVER THAT AND INDEX THAT WOULD BE REALLY TREMENDOUS. >> DAVID. >> HI, DAVID LARSON, STANFORD. THANK YOU VERY MUCH FOR A REALLY PROVOCATIVE DISCUSSION. ONE CONCEPT THAT I REALLY LIKED WAS THIS CONCEPT OF TREATING DATA AS SOMETHING YOU DONATE LIKE BLOOD RATHER THAN PROFIT FROM LIKE CURRENCY, COUNTS -- SOUNDS GREAT BUT INSTITUTIONS ARE INVOLVED IN THIS ARE FOR-PROFIT ENTITIES AND BEHOLDEN TO STAKEHOLDERS THAT HAVE A COMMERCIAL INTEREST. SO HOW DO WE RECONCILE THAT? HOW DO WE -- ESPECIALLY WHEN IT'S FROM A FOR-PROFIT COMPANY WHO IS SAYING YES, WE WOULD LOVE FOR YOU TO DONATE, NOW PLEASE DONATE TO US THAT WE'LL PROFIT OFF OF SO HOW DO WE MAINTAIN THIS? THAT'S TO BE EXPECTED, THAT'S WHAT THEY ARE THERE TO DO BUT HOW DO WE DO IT IN A WAY THAT'S FAIR? >> I THINK YOU'RE GOING TO SEE FUNDING IS GOING BECOME DEPENDENT ON WILLINGNESS TO CONTRIBUTE ANONYMIZED DATA, IF YOU WILL, TO MULTI-COLLABORATIVE TRIALS, MULTI-SITE. INSTITUTIONS WILL BE MOTIVATED WHEN CLINICAL TRIALS COME DOWN THE ROAD WHERE YOU HAVE TO CONTRIBUTE DATA IN ORDER TO BE FUNDED. >> MAYBE JUST ONE BRIEF ADDITION TO THAT. WE HAVE A VERY NICE MODEL IN THE SOFTWARE COMMUNITY OF OPEN SOURCE, AND THEN WE ALL HARVEST IT AND USE THAT INSIDE OUR EQUIPMENT, G.E USES A LOT OF OPEN SOURCE SOFTWARE WITH THE BENEFIT OF MANY EYES AND TRANSPARENCY WE TALKED ABOUT, SO WE KNOW OF SECURITY AND KNOW HOW TO MAINTAIN SAID OPEN DATA OR OPEN SOFTWARE SO I THINK THERE ARE OTHER MODELS WE CAN LOOK AT BE IT A PATCHY LICENSING OF DATA OR OTHER OPEN SOURCE TYPE MODELS THAT AREN'T SIMPLY COPYRIGHT MODELS LIKE WE HAVE TODAY OR FOR RESEARCH ONLY BUT I'M WITH YOU. I GUARANTEE YOU YOU'RE CORRECT. G.E. IS NOT GOING TO GIVE UP ALL OF THE DATA THAT IT'S AMASSED TO ITS COMPETITORS WITHOUT SOME OTHER LARGER COMMUNITY EFFORT THAT COMES FORWARD, IF THAT MAKES SENSE SO YOU'RE SPOT ON. >> I THINK THE OTHER THING THAT'S IMPORTANT IN THE SOFTWARE ANALOGY IS THE IDEA OF IDENTIFICATION WITH REGARDS TO THAT SOFTWARE. A LOT OF LICENSES HAVE VERBIAGE THAT SAY IF YOU USE THE SOFTWARE YOU'RE USING IT AT YOUR OWN RISK. I THINK THAT'S ONE OF THE STUMBLING BLOCKS WHEN IT COMES TO THE APPLICATION OF DATASETS THAT WERE CREATED BECAUSE OF UNINTENTIONAL BIAS AND/OR POTENTIAL PRIVACY-RELATED CONCERNS, WERE WHEN YOU HAVE REALLY LARGE DATASETS ARE THEY TRULY DE-IDENTIFIED IN THE APPROPRIATE MANNER SO I'D LIKE TO SEE SOME, YOU KNOW, HIPAA 2.0 OR SOME SORT OF ADDRESSING THE PATIENT PRIVACY CONCERNS THAT REALLY BREAK DOWN BARRIERS TO LARGE DATA SHAKER. >> I REALLY THINK NUMERICAL SIMULATION TO HELP ADDRESS THE ISSUE. I READ ARTICLE, RON KING, GENERATEED DIVERSITY, HEART ATTACK ANALOGY IN MACHINE LEARNING FIELD AND READ (INDISCERNIBLE) GENERATED REALISTIC DATA IMAGES SO I HOPE WITH MACHINE LEARNING AND MEDICAL IMAGES COULD BE GENERATED, SIMULATED AS RULE MACHINE LEARNING, AVOID MANY PRIVACY ISSUES. >> THIS GENTLEMAN. >> YEAH, HI. I JUST WANT TO MAKE A COMMENT ABOUT DEEP NEURAL NETWORK AND CONVOLUTION, MY NAME IS (INDISCERNIBLE), MY COMPANY IS (INDISCERNIBLE), ALL OF THE CONVOLUTION, ENFORCEMENT LEARNING, BASICALLY STATISTICAL. SO IN MY OPINION THEY CANNOT MAKE ANY REASONING OR, YOU KNOW, FIGURE OUT THE CONFLICTS AND THOSE KINDS OF THINGS SO IT CAN ACTUALLY -- I CALL IT SPECIFIC A.I. OR DUMB A.I., A DUMB BOX. LAST SEVERAL YEARS, SMALL FUNDING FROM PLACES, MY TEAM AND PROFESSORS OUT THERE, FATHER OF PHYSIOLOGYICS, WE WORK ON ALGORITHM THAT OVERCOMES ALL OF THE DEEP NEURAL NETWORK PROBLEMS, SO MANY LAYERS FOR COGNITION AND REASONING AND THINGS THAT NEURAL NETWORKS CANNOT DO. WE PROVE YOU DON'T NEED NEURAL NETWORK, DEEP NEURAL NETWORK TO DO THIS KIND OF THING, BASICALLY USE ALGORITHM BASED ON GENERAL A.I., AND SO WHAT WE DO IS WE DON'T NEED THAT MANY TRAINING SAMPLE, CONTINUOUS LEARNING, YOU CAN FIND FEATURES AND SOME ONE-OFFS, A COUPLE SAMPLES OUT OF MILLIONS. IT DOESN'T HAVE ANY OF THESE THINGS, EVEN PROFESSOR HINTON LAST YEAR SAID DEEP NEURAL NETWORK CANNOT GENERALIZE WELL FOR DATA, YOU HAVE TO THROW IT AWAY AND START FROM SCRATCH, A COUPLE INTERVIEWS BEFORE. SO BASICALLY MOST COMPANIES LIKE IBM AND INTEL ARE SIMILAR THINGS, THE USE OF NEURAL NETWORKS, SO NEURAL NETWORKS HAVE USAGE IN A.I. BUT CANNOT BE USEFUL FOR ALL PURPOSES AND HAS A CEILING THAT ALREADY REACHED, YOU KNOW. >> SO LET ME MAKE SOME COMMENTS. I AGREE WITH SEVERAL SPEAKERS, RIGHT NOW THE MACHINE LEARNING OR DEEP LEARNING IS A NEW AREA AND A LOT OF RESEARCH OPPORTUNITIES AS OUTLINED ALREADY TALKING ABOUT PHYSIOLOGIC SYSTEM, ACTUALLY SOME WORK GOING ON IN MY GROUP. WE BASICALLY REPRODUCE THE CONVENTIONAL NEURON, A LOT OF (INDISCERNIBLE) NEURON, CURRENT NEURON, WE THINK THAT LINEAR OPERATION SO WE (INDISCERNIBLE) SO CORDOTIC NEURON IS A PHYSIOLOGIC (INDISCERNIBLE) THEN WE IMMEDIATELY HAVE DEEPER PHYSIOLOGIC SYSTEM, THAT OFFERS ENGINEERING INTERPRETATION OF NEURAL NETWORK WE HOPE THAT WILL OPEN THE DOOR AND ALSO LINKAGE TO PHYSIOLOGIC SYSTEM THEORY AND TECHNOLOGY SO THAT WAY WE THINK PHYSIOLOGIC AND NEURAL NETWORK COULD BE COMBINED, SYNERGIZED. >> THANKS FOR YOUR COMMENTS. DR. DREYER. >> THANK YOU, DR. FLANDERS. IN LOOKING AT THIS FIELD AND WATCHING IT A LITTLE BEHIND OTHER AREAS LIKE AUTOMOTIVE, I SEE IN TALKING TO JENSON AND INVIDIA REALIZED DATA IS THE SOURCE CODE, HUMANS CREATING WAS THE GROUND TRUTH OF DIVERSION, NOW WE'RE BACK TO DATA SO MY QUESTION IS, IN YOUR EXPERIENCES AND AS YOU SEE GOING FORWARD WHETHER YOU BORROW DATA, COMBINE DATA OR HAVE WEAK ACCESS TO DATA GETTING SMARTER ANNOTATION, SMARTER ANNOTATIONS BUT AS SOON AS SOMEBODY CORRECTS ONE OR THE SOFTWARE CORRECTS IT THE ORIGINAL ALGORITHM ALL THE WAY BACK TO THE DATA HAS CHANGED, HAVE YOU THOUGHT ABOUT VERSIONING BACK TO THE DATA AND WHAT THOUGHTS TO DO THAT OR WHERE DO YOU THINK THE FIELD IS GOING TO GO IN THAT RESPECT? >> I THINK THAT THAT WOULD PROBABLY BE A GREAT QUESTION, I'D LOVE TO HEAR WHAT BRAD ERICKSON HAS TO SAY, SOUNDS LIKE THEY HAVE BUILT QUITE A BIT OF INFRASTRUCTURE AT MAYO WITH MANAGEMENT SYSTEM TO DO THE CURATION PROCESS AND I THINK DOING THAT CURATION PROCESS IN A WAY THAT'S REPRODUCIBLE LIKE GIT IS REPRODUCIBLE ACROSS SOFTWARE IS CRITICALLY IMPORTANT TO GO FORWARD. I DON'T KNOW THAT ANY OF US HAVE A SOLUTION FOR THAT TODAY. >> WELL, ON THE COMMERCIAL SIDE WE DO, YEAH. G.E. IS WORKING ON THAT PRECISELY SO WE CAN VERSION AND SHOW PROVENANCE OF EXACTLY WHAT DATA, WHICH CURATORS, FROM WHICH SITES. AND THEN BE ABLE TO VERSION THAT ARE CONSISTENTLY, THIS IS WHY I WAS TALKING ABOUT THE CHALLENGES INHERENT IN TRANSFER LEARNING IF WE DON'T HAVE PERSIST ACCESS, HOW DO YOU VERSION THAT? ABSOLUTELY THAT LEVEL OF PROVENANCE JUST LIKE IN A DATA WAREHOUSE WHEN YOU DO A FINANCIAL FILING FOR YOUR 10K, WE FEEL THAT YOU NEED IT FOR YOUR 510(K). >> THE ONLY POINT I WOULD MAKE AS FAR AS NOT HAVING IT IS THAT, YOU KNOW, GIT HAS BECOME A DE FACTO STANDARD ACROSS THE OPEN SOURCE COMMUNITY BUT YOU'VE GOT OTHER THINGS LIKE BIT BUCKET AND MERCURIAL AND I DON'T KNOW THAT WE HAVE GOOD GENERALIZABLE TOOLS THAT ARE AVAILABLE ACROSS INSTITUTIONS TODAY, WHILE INDIVIDUAL INSTITUTIONS ARE STILL DEVELOPING EACH OF THESE TOOLS AS THEIR OWN KIND OF COTTAGE INDUSTRY, I THINK WE NEED TO TAKE THAT AND ELEVATE IT SO WE CAN DO THIS IN A REPRODUCIBLE WAY AND I'M NOT SURE WHETHER OUR INSTITUTIONS SUCH AS OUR SOCIETES, WHO IS RESPONSIBLE FOR DOING THAT AND PULLING THAT TOGETHER IS A GOOD QUESTION. >> AND I WOULD ADD ONE MORE THING. THERE'S THE TOOLS ASPECT, AND THE METHODOLOGY AND BEST PRACTICES, I THINK WE NEED TO ACTUALLY THINK ABOUT THOSE AND DEFINE THEM AS WELL BECAUSE HAVING THE TOOLS TO DO THIS AND TO CURATE THE DATA IS IMPORTANT BUT TO MAKE SURE THAT THERE'S SOME SET OF GUIDELINES OR BEST PRACTICES THAT EVERYONE CAN FOLLOW AS THEY GET INTO THE SPACE IS GOING TO BE JUST AS IMPORTANT. >> GOOD POINT. >> BILL THOR, AMERICAN COLLEGE OF COOLING. FROM A PATIENT-CENTRIC STANDPOINT, SHIFT OF POPULATION SECURITY CONCERNS, WHETHER IT BE THE GDPR IN EUROPE OR AFTER MULTIPLE HUGE HACKS OF EVERYBODY DOWN TO THE SECURITY ORGANIZATIONS OF THE UNITED STATES, ARE WE MISSING AN OPPORTUNITY WITH THIS REPRESENTATION OF ORGANIZATIONS IN THE ROOM, JUST LIKE WE HAVE ON OUR DRIVER'S LICENSE, YOU KNOW, WASN'T TO BE AN ORGAN DONOR, GOING BACK TO THE CONCEPT OF BEING A BLOOD DONOR, SHOULD WE BE LOOKING AT PUBLIC PRESSURE TO HAVE FACILITIES DISTINGUISH THEMSELVES SAYING WE'RE SUCH AND SUCH FACILITY BECAUSE WE CONTRIBUTE DATA TO LARGER ENTITY, TO CONTRIBUTE TO THIS HEALTH CARE PROVISION? SHOULD WE LOOK FOR PUBLIC SUPPORT FOR THIS EFFORT? >> IT'S INTERESTING THAT THE ALL OF US, SYNC FOR SCIENCE, I'VE STOOD AND WATCHED. THEY HAVE HAD TROUBLE GETTING PARTNERS TO PILOT THIS ORIGINALLY, I THINK ORIGINALLY THEY HAD ABOUT 15 OR 16 DIFFERENT ENTERPRISES WITH MAJOR VENDORS, DROPPING DOWN TO 8, SO YOU'RE RIGHT IT DOES NEED SOME PUBLIC ENGAGEMENT, IF YOU WILL, TO TRY TO GET THE MEDICAL ENTERPRISES, EHR VENDORS, TO ALL WANT TO PARTICIPATE IN PATIENT ENGAGEMENT WITH THE RIGHT SECURITY MODEL AROUND THAT DATA. >> THIS WILL HAVE TO BE THE LAST QUESTION. >> OKAY, YEAH, NICK PETRICK, FDA. IT'S IMPORTANT TO UNDERSTAND DATA IS THE WHOLE KEY HERE AND THE DATA IS WHERE THE MONEY IS. I THINK IT'S VERY DIFFICULT TO MAKE EVERYTHING INTO A DONOR -- YOU KNOW, THE PATIENT OWNS -- IN SOME SENSE IT'S THEIR DATA, IT'S FINANCIAL -- THERE'S MONEY ASSOCIATED WITH THAT AND THE PEOPLE THAT CAN AGGREGATE CAN MANGE MONEY BECAUSE THEY CONTROL THE ALGORITHMS. THAT WASN'T MY QUESTION. THE QUESTION I HAVE, HOW MUCH ARE WE WORKING TOWARDS AGGREGATING ACROSS NOT JUST IMAGING BUT ACROSS ALL TYPES OF DATA? IN REALITY THIS IS AN IMAGING ISSUE BUT REALLY A GENERAL MEDICAL ISSUE AND TRYING TO AGGREGATE NOT JUST IMAGING DATA OR DIFFERENT DATABASES BUT TRYING TO AGGREGATE INTO ON A UNIFIED MODEL ACCESSIBLE TO ADDRESS MEDICAL ISSUES, I JUST WONDER IF YOU COULD COMMENT ON THAT. >> JUST THE COMMENT ABOUT THE GENERALIZABILITY ACROSS ALL OF MEDICINE, I THINK THAT GETS DOWN TO HOW DIFFICULT IT IS TO DO NORMALIZATION, THAT PROBLEM IS EXPONENTIAL WHEN YOU INCREASE THE DOMAIN AREA. WITH COMMON DATA ELEMENT, THE DISCUSSION TESSA MENTIONED EARLIER, THERE'S A LOT OF WORK THAT'S GOING ON IN THE HL7 STRUCTURE DATA CAPTURE AREA AND THINGS LIKE THAT. IT'S REALLY DIFFICULT TO IDENTIFY LIKE HOW MUCH DO YOU SLOW DOWN PROGRESS IN ORDER TO WORK ON THE NORMALIZATION VERSUS HOW MUCH DO YOU DO NORMALIZATION IN A SMALL AREA AND MAKE IT PUBLICLY AVAILABLE SUCH THAT YOU CAN JOIN IN WITH THOSE LARGER PIECES AS THEY BECOME AVAILABLE. THAT PART IS VERY DIFFICULT TO ANSWER. >> AND, NICK, I THINK YOUR QUESTION IS A REALLY IMPORTANT ONE, THE METADATA ASSOCIATED WITH IMAGE. ONE OF THE CHALLENGES IS A.I. ALGORITHMS, A LOT WORK IN ISOLATION, RADIOLOGISTS CAN GO BACK TO PRIOR ONES, HAVE ACCESS TO THE PATIENT PROBLEM LIST, THINGS ARE ARTIFICIAL WITH A SUBSET AND BRINGING IN ADDITIONAL INFORMATION WILL BE CRITICALLY IMPORTANT TO SUCCESS, EVEN DATABASES AND ARCHIVES BRINGING THAT INFORMATION IN IS CRITICAL. THAT GETS BACK TO SOMETHING RELATED TO WHAT KEITH MENTIONED, ONE OF THE THINGS EXCITING FOR ME ABOUT MACHINE LEARNING IS ITS DYNAMIC ASPECT, THE FACT THAT EVERY NEW CASE POTENTIALLY FOR ME AS A RADIOLOGIST AND MY RESIDENTS IS A LEARNING OPPORTUNITY. BUT THE FDA ENDS UP CLEARING AN ALGORITHM IN A STATIC PLACE AND THEN THAT'S SHRINK WRAPPED, AS I ACQUIRE ADDITIONAL DATA IT DOESN'T LEARN THE SAME WAY THAT HUMANS MIGHT BE ABLE TO. WHAT WOULD BE GREAT IS A MECHANISM TO REFER BACK TO ORIGINAL ALGORITHM, FDA TESTED AND CLEARED, BUT AT THE UNIVERSITY OF MARYLAND BASED ON YOUR EXPERIENCE HERE WOULD BE INFORMATION FROM OUR AGGREGATED ADDITIONAL DATA THAT YOU HAVE AND MAYBE HAVE A MECHANISM TO BE ABLE TO DO BOTH. I KNOW THAT'S A TRICKY ISSUE AS FAR AS CLEARANCE BUT THAT WOULD BE SUPER HELPFUL TO DO THAT AND WOULD ALLOW CONTINUOUS LEARNING ENVIRONMENT AS YOU SUGGESTED FROM OTHER TIMES OF METADATA TOO. >> ONE OTHER POINT THAT I WANTED TO MAKE, YOU MADE THE STATEMENT THE PATIENT OWNS THE DATA. AND THAT I THINK IS NOT A COMPLETELY GENERALLY ACCEPTED APPROACH, I WOULD SAY THE PATIENT OWNS A COPY OF THE DATA, BUT IT'S BASED ON -- THERE'S HETEROGENEOUS STATE LAWS ABOUT WHO ACTUALLY OWNS AND IS RESPONSIBLE FOR MAINTAINING A COPY OF THAT DATA. AND WHAT CAN BE DONE WITH IT. SO I THINK THERE'S SOME SERIOUS ETHICAL QUESTIONS THAT WE NEED TO ANSWER ALONG THE WAY, THAT'S ONE OF THE REASONS WHY I THINK THAT THE MORE OPEN YOU CAN MAKE THE DATA AND LESS TIED TO A PARTICULAR COMMERCIAL INTEREST, THE BETTER OFF WE'RE GOING TO BE BECAUSE I THINK THE ETHICS ARE CLEANER FOR OPEN AND PUBLIC DATA THAN THEY ARE FOR DATA THAT'S REALLY KIND OF HELD AS CURRENCY. >> FROM A STATE AND FEDERAL STATUTORY PERSPECTIVE IT'S THE FACILITY THAT OWNS THE DATA FROM A LEGAL SENSE ANYWAY. >> (INAUDIBLE). >> CONTROL. >> CONTROL OF THE DATA. (INAUDIBLE) I THINK I MISSPOKE. I WAS TRYING TO SAY THERE'S ENTITIES THAT CONTROL THE DATA, WHETHER THE PATIENT OR INSTITUTION, AND THE POINT IS THERE'S IS A LOT OF MONEY ASSOCIATED WITH THAT SO TO SAY IT'S JUST GOING TO BE DONATED, THIS IS A PUBLIC HEALTH, THAT WOULD BE GREAT. I DON'T SEE THAT AS BEING REALISTIC. COMPANIES ARE LARGER THAT CAN HAVE MORE DATA ARE GOING TO TRY TO CONTROL IT TO TAKE ADVANTAGE FINANCIALLY. THAT'S DIFFICULT TO SEE HOW IT WORKS BUT I'M ALL FOR MAKING THIS WORK IN A UNIFIED WAY. IT'S JUST HARD TO SEE THAT COMING, I GUESS. >> THAT'S GOING TO HAVE TO BE THE LAST COMMENT. WE'RE AT THE BOTTOM OF THE HOUR. GREAT DISCUSSION. A LOT OF ENTHUSIASM. AWESOME SESSION. [APPLAUSE] SO EVERYONE HAS AN HOUR FOR LUNCH. SHIRLEY TOLD ME TO REMIND THE ORGANIZERS AND INVITED SPEAKERS TO MEET WITH HER ON THE TERRACE DURING THE LUNCH HOUR. SO NOW IT'S TIME FOR SESSION D, THIS IS MY FAVORITE TOPIC, ABOUT MACHINE LEARNING IN THE IMAGING LIFE CYCLE. MY CO-CHAIR MATTHEW LUNGREN FROM STANFORD UNIVERSITY WILL TAKE NOTES. ANYWAY, I HAPPEN TO BE THE FIRST SPEAKER UNDER THE TITLE TOMO GLASS GRAPHIC MACHINE LEARNING, MY FAVORITE TOPIC, WORKING ON TOMOGRAPHIC RECONSTRUCTION FOR QUITE A WHILE. IN 2016 I WROTE A PERSPECTIVE ARTICLE, ON DEEPER IMAGING, THIS ARTICLE HAS RECEIVED OVER 6,000 FULL TAGS OF VIEW, I FEEL VERY CONFIDENT ON THIS PERSPECTIVE. IN THE PERSPECTIVE ARTICLE SHOWN HERE, A BIG PICTURE ON SMART PRECISION IMAGING OR MEDICINE. THE RIGHT ARROWS FROM IMAGES TO FEATURES, IMAGE ANALYSIS, AND IN THE MORNING I MENTIONED TOMOGRAPHIC RECONSTRUCTION THROUGH MACHINE LEARNING, GOES ALL THE WAY FROM RAW DATA TO FINAL TOMOGRAPHIC CT OR IMAGES. THE GREEN ARROWS AND RIGHT ARROWS YOU CAN COMBINE THEM TO FORM THE WHOLE CHAIN, AND NOW WE'LL TALK ABOUT END TO END, FROM TOMOGRAPHIC TO USE IMAGE GUIDED THERAPY, ROBOTIC SURGERY, PERSPECTIVE IS RATHER EXCITING. I WOULD LIKE TO SAY A FEW THINGS ABOUT GENERAL BACKGROUND, NEW APPROACH, NEW TECHNOLOGIES APPEAR, PEOPLE MAY REASONABLY HAVE CONCERNS, RETROSPECTIVE WHEN ANALYTICAL RECONSTRUCTION FIRST CAME INTO PLAY AND PEOPLE HAD SOME THOUGHTS LIKE SOME MAY SAY MEDICINE, FINITE NUMBER OF PROJECTION MATHEMATICALLY CAN IMPROVE RECONSTRUCTION IS NOT UNIQUE, MEASURED IN THE GIVEN SITE OF DATA. LATER ON WE OVERCOME THE HURDLE. THEN RECONSTRUCTION BECOMES POPULAR TO ADDRESS LIKE SLOW MRI SIDE, HIGH RELEASE DOSE, YOU PERFORM RADIORECONSTRUCTION USE SOME TERMS, DEPENDING ON THE COEFFICIENT, DEPENDING ON THE PENALTY TERM, SOME SUBJECTIVE NATURE, LATER ON WE WORKED ON THE PROBLEM AND YOU CAN OPTIMIZE BE, YOU STILL GET A DIAGNOSTIC QUALITY. THEN COMPRESSED SENSING BECOME POPULAR. THERE IS MORE CHALLENGES. THE SOLUTION IS NOT THE TRUTH SO WE SHOW EXAMPLE WITH COMPRESSED TUMOR-LIKE STRUCTURES COULD BE INTRODUCED OR PATHOLOGICAL PLAQUES MAY BE INCORRECTLY REMOVED BUT THE PROBLEM DOESN'T STOP, COMPRESSED SENSING APPLIED IN PRACTICE, AS AN ART YOU DO GOOD WORK, GOOD ALGORITHM DESIGN, AND VALIDATION SCHEME, YOU CAN PRODUCE USEFUL RESULTS. NOW COMES TO MACHINE LEARNING, WE HEARD A LOT ABOUT THE BLACK BOX NATURE, MAXWELL EQUATION FROM MACHINE LEARNING, YES, YOU DO TRAINING AND PARAMETER ASSESSMENT, HOW MUCH (INDISCERNIBLE) THAT'S THE ISSUE BUT NOT SOMETHING UNSOLVABLE. WE PRODUCE SOME GOOD RESULTS ON THE BASIS X-RAY CT AS PARTICULAR MODALITY OF OUR INTEREST. THIS IS LOW-HANGING FRUIT AND MACHINE LEARNING ALGORITHM, NEURON NETWORK IN PARTICULAR CAN MAP QUARTER DOSE CT SLIDES ALL THE WAY TO THE NOISE RESULT, INCLUDE AGREEMENT WITH FULL DOSE COUNTERPART, THE ARTICLE OUR GROUP HAS TWO ARTICLES IN IEEE SPECIAL ISSUE SO YOU CAN SEE DETAIL ABOUT HOW MACHINE LEARNING ALGORITHM WORKS. BASICALLY WE USE VERY CHEAP COST EFFECTIVE FILTER PROJECTION AS STARTING POINT, THEN MACHINE LEARNING OFFERS SMART (INDISCERNIBLE) CAPABILITY, THAT OFFERS STATE-OF-THE-ART PERFORMANCE. AND NEXT EXAMPLE THAT WE ARE WORKING ON IN COLLABORATION WITH SEVERAL COLLABORATORS MONOCHROMATIC CT RECONSTRUCTION, NORMALLY PERFORM AGAIN, DO MONOCHROMATIC IMAGE DECOMPOSITION, THAT OF PERFORMING (INDISCERNIBLE) AND WE TRY ON THE NEURO NETWORK, GOT SOME INTERESTING RESULTS, FROM SINGLE CURRENT-INTEGRATING DATASET THROUGH MACHINE LEARNING, THE IMAGE CONTAINS NATURAL ARTIFACTS, WE USE THE MACHINE LEARNING PARTICULAR PROFILE INTO MONOCHROMATIC COUNTERPART. IT TURNED OUT THE RESULT IS PRETTY GOOD. THE SPARSE DATA CT, LEARN SEGMENT BASED RECONSTRUCTION NETWORK. (INDISCERNIBLE) EACH DAY YOU STILL HAVE A REGULAR ITERATIVE FORMULA HERE. IN PARALLEL TO THAT ARM YOU HAVE A NEURAL NETWORK ARM, AND THE MODEL NOT EXPLAINED BY REGULAR FORMULATION, IT WILL BE LEARNED THROUGH TRAINING. SO THIS IS SOMETHING VERY ADAPTIVE, GOOD RECONSTRUCTION RESULT, BUT SIMILAR IDEA COULD BE APPLIED FOR LIKE TOMOGRAPHY, LIMITED ANGLES, TRYING DIFFERENT THINGS. AN ARTICLE TALKED ABOUT SUPER-RESOLUTION CT, AND WE GAVE THE NETWORK A NAME, GAN-CIRCLE. THE CIRCLE REALLY MEANS CONCENTRATED BY IDENTICAL (INDISCERNIBLE) CIRCLE LEARNING ASSEMBLE PUTTING TOGETHER CAN GET LOW RESOLUTION IMAGE MICRO CT IMAGE IMPROVED THROUGH SUPERVISED LEARNING VERY CLOSE TO FULL RESOLUTION, HIGH RESOLUTION COUNTERPART. AND WE SEE ABOUT 100% RESOLUTION IMPROVEMENT. TWO OR ONE DECADE AGO, MY GROUP PUBLISHED SEVERAL PAPERS WE USED TRADITIONAL RECONSTRUCTION TO DO SUPER-RESOLUTION DEEP LEARNING WORK. AT THAT TIME WE CLAIM ABOUT 30% RESOLUTION IMPROVEMENT. NOW WITH MACHINE LEARNING WE SEE ABOUT 100 PERCENTAGE GAIN, SO THAT'S VERY EXCITING PROGRESS. AS FAR AS METAL ARTIFACT REDUCTION IS CONCERNED, PROFESSOR HENGYONG YU'S GROUP, THEIR IDEA TO USE MULTIPLE COPIES, (INDISCERNIBLE), AND PUT INTO NEURAL NETWORK AND THEY CAN COMBINE BENEFIT OF MULTIPLE ALGORITHMS INTO THE DATA OF ARTIFACTS REDUCTION IMPROVEMENT AS SHOWN IN THE RIGHT BOX, SO YOU SEE THE SHADOW IN BASICALLY REMOVED, STREAKING ARTIFACT DISAPPEARED IN THE MACHINE LEARNING ALGORITHM. FURTHERMORE, ALTHOUGH THE TRAINING TIME IS VERY LONG, AND THE GROUP SHOWS THAT BASIC MACHINE LEARNING AND AFTER TRAINING THE RECONSTRUCTION TIME IS VERY (INDISCERNIBLE) AND ORDERS OF MAGNITUDE IMPROVED OVER THE STATE-OF-THE-ART ARTIFACT WE DEDUCTION ALGORITHM AND NAME, ADMIRE, SHOWING ADVANTAGES. SO ALL THESE EXAMPLES GAVE US IDEA, WITH MACHINE LEARNING WE CAN MAKE IMPROVEMENT OVER STATE OF THE ART. SO ANY PROVEN ALGORITHM LIKE METAL ARTIFACT REDUCTION THERE ARE SOME GOOD ALGORITHMS BUT AS A RESULT STILL NOT PERFECT. THEN YOU TAKE THAT IMAGE AS YOUR STARTING POINT AND PUT THE RESULT INTO NEURAL NETWORK, YOU ONLY GET EVEN BETTER RESULTS, SO WE CAN TRAIN NEURAL NETWORK TO MAP IMPERFECT OR BAD RESULTS TO A STANDARD, IF YOU LIKE YOU CAN CHOLATE B2B, IMPROVING STATE-OF-THE-ART ALGORITHM WITH MACHINE LEARNING. FURTHERMORE, THE MACHINE LEARNING ALGORITHM, MACHINE LEARNING BASED IMAGE RECONSTRUCTION CAN BE INTERPRETED IN A BROADER SENSE, A CONCEPT I SUGGEST IS THAT OF RAWDIOMICS, A NEW WORD, SO WE GO ALL THE WAY FROM RAW DATA, FROM RAW DATA YOU DO ANY RECONSTRUCTION, YOU CANNOT GO BACK, THAT MEANS SOME INFORMATION ALREADY LOST IN THE PROCESS. SO RECONSTRUCTION ALGORITHM SERVES AS A FEATURE, CHANNEL OF FEATURE, RIGHT NOW WE DO MULTIPLE IMAGE RECONSTRUCTION, USING COMPLEMENTARY ALGORITHMS, THEN YOU EFFECTIVELY ENHANCE, BROADEN THE BASE OF FEATURE, BASED ON BROADEN BASE OF FEATURE THEN YOU PERFORM IMAGE ANALYSIS AND YOU DO RADIOMICS FOR BETTER PERFORMANCE, UNIFIED AS YOU MAY OR MAY NOT INVOLVE INTERMEDIATE RECONSTRUCTION IMAGES, SO THE WHOLE PROCESS GAVE BETTER OPPORTUNITY AND YOU MAY INTERPRET THIS AS OTHER INFORMATION LIKE GENOMIC INFORMATION, PUT IT TOGETHER IN BIG MACHINE LEARNING NETWORK, ACTIVELY PURSUING RESEARCH IN THIS DIRECTION IN COLLABORATION WITH AMBER SIMPSON FROM MEMORIAL SLOAN-KETTERING AND BRUNO DE MAN, MY COLLEAGUE PINGKUN AND NAMMUDEEP, WE THINK IT'S A PROMISING DIRECTION, THIS IS A SUMMARY. OVERALL WE BELIEVE IN PRINCIPLE MACHINE LEARNING CAN OUTPERFORM ITERATIVE RECONSTRUCTION PROCESS. WHY? I ALREADY MENTIONED ITERATIVE RECONSTRUCTION OR ANALYTIC COULD BE USED AS A COMPONENT IN THE NEURAL NETWORK, OR BASELINE IN THE NEURAL NETWORK, AND THE TRAINING CAPABILITY AND DRIVEN NETWORK WILL PERFORM SO YOU WILL HAVE BETTER PERFORMANCE THAT IS VERY POWERFUL IN TERMS OF EXTENSIVE PRIOR AND NON-LINEAR LEARNING CAPABILITY DUE TO BIG DATA. THANK YOU. [APPLAUSE] >> SO OUR NEXT SPEAKER IS DAVID LARSON FROM STANFORD, HIS TITLE IS IMAGING QUALITY CONTROL. SO PLEASE. >> THANK YOU, GE. ALSO A PLEASURE TO BE WITH YOU THIS AFTERNOON. I APPRECIATE THE OPPORTUNITY. I DO HAVE ONE DISCLOSURE. MY BACKGROUND IS IN QUALITY IMPROVEMENT IN IMAGING ESPECIALLY IN APPLIED CLINICAL SETTING, THIS IS A SUPER-EXCITING TIME FOR THOSE IN MY FIELD AS WELL AS MORE IN THE TECHNICAL SIDE BECAUSE WHAT I SEE IS WE'RE NOW SEEING APPLICATIONS COMING WITH GREATER AND GREATER POWER TO DO THINGS LIKE IMPROVE IMAGE QUALITY, IMPROVE REPORT ACCURACY, DECREASE RADIATION DOSE AND SO FORTH, MANY EXAMPLES YOU'VE SEEN FROM THIS MORNING. I THINK ONE OF THE MAJOR APPLICATIONS OF MACHINE LEARNING IN A WAY IT WILL IMPACT THE FIELD IS THROUGH QUALITY. BUT I'M NOT GOING TO SPEND A LOT OF TIME ON THE SPECIFIC USE CASES BECAUSE HAVING SPENT SOME TIME IN THIS AREA, I'VE ALSO COME TO LEARN SOME OF THE CAVEATS ASSOCIATED WITH IT, AND I SEE SOME WARNING SIGNS ON THE HORIZON THAT IF WE'RE NOT CAREFULLY THINK THERE'S ACTUALLY A CHANCE THAT THESE TOOLS THAT ARE DESIGNED TO IMPROVE QUALITY COULD ACTUALLY SERVE TO DO THE OPPOSITE AND MAKE IT EVEN HARDER TO ACHIEVE RESULTS. I'M GOING TO FOCUS ON THAT AND TELL MY TALE OF WOE AND USE CASE I SPENT A FAIR AMOUNT OF TIME ON, CT RADIATION DOSE OPTIMIZATION, IMMINENTLY SOLVABLE. OUR CT SCANNERS ARE COMPUTER NUMERIC CONTROLLED MACHINES, IN MANY WAYS YOU CAN TAKE THE HUMAN OUT OF THE LOOP. IT'S SOMETHING WE SHOULD HAVE GOTTEN RIGHT BY NOW. SOMETHING THAT WE'VE WANTED TO DO FOR MANY YEARS. WE'VE GOT MORE THAN ALMOST TWO DECADES NOW OF EFFORT THAT HAVE GONE INTO THIS. YET HERE WE ARE, WE SEE PUBLICATION AFTER PUBLICATION THAT SHOW THERE'S STILL A TREMENDOUS AMOUNT OF VARIABILITY IN RADIATION DOSE. SO WHY IS THAT STILL THE CASE? WELL, GOING INTO THE HISTORY JUST A LITTLE BIT, IT STARTED IN THE BEGINNING THAT CT SCANNERS WERE MANUALLY CONTROLLED BY THE OPERATIVES WHO WOULD MANUALLY ADJUST THE TWO CURRENT AND KVP ON THE SLIDE, QUICKLY IT WAS RECOGNIZEDS THAT PROBLEMATIC, OVERDOSING SMALLER PATIENTS, UNDERGOING LARGER PATIENTS, MANUFACTURERS DEVELOPED AUTOMATED TUBE CURRENT MODULATION WHICH ALLOWED THE MACHINE TO AUTOMATICALLY ADJUST BASED ON PATIENT SIZE AND OPERATOR WOULD PRESS A BUTTON AND THE MACHINE WOULD TAKE CARE OF IT. SO THE THOUGHT WAS THIS WOULD SOLVE THE PROBLEM BUT IT DID NOT SOLVE THE PROBLEM. SO WHY IS THAT? I WOULD SAY THERE ARE A FEW REASONS. FIRST OF ALL, TURNS OUT THE ALGORITHMS THEY DEVELOPED AT THE BEGINNING WERE BASED ON FAULTY ASSUMPTIONS THAT RADIOLOGISTS WANTED IMAGE NOISE TO BE MAINTAINED ACROSS PATIENT SIZE, THAT WAS NOT THE CASE BUT THAT'S OKAY. THERE SHOULD HAVE BEEN A PROCESS TO SELF CORRECT THAT. THE PROBLEM WAS THE VENDORS KEPT THE ALGORITHMS SECRET AND PROPRIETARY, STANDARD PRACTICE, BUT VENDORS -- NO ONE PROVIDED A MEANS TO EASILY PROVIDE A MEASURE ESPECIALLY OF IMAGE QUALITY TO VALIDATE CLAIMS OF PERFORMANCE. IF WE STEP BACK THERE'S A PROBLEM EVEN THAT THE OPTIMIZATION PROBLEM WAS NOT REALLY FRAMED IN A SYSTEMATIC WAY SO IN GENERAL WE SHOULD SAY IT'S A SIMPLE OPTIMIZATION PROBLEM WHERE WE WANT TO REDUCE IMAGE -- REDUCE DOSE TO THE EXTENT WE CAN WHILE PRESERVING IMAGE QUALITY. IN ABSENCE OF A GOOD OPTIMIZATION FRAMEWORK MANY PEOPLE HAVE GONE AFTER JUST REDUCING IMAGE QUALITY -- REDUCING DOSE WHICH IS A BIT SILLY BECAUSE IT'S EASY TO REDUCE DOSE. TURN DOWN DOSE PARAMETERS. HOW DO YOU REDUCE TO A PRECISE POINT WHERE YOU HAVE ADEQUATE IMAGE QUALITY? AND THEN FINALLY, WE DID NOT HAVE GOOD MONITORING MECHANISMS TO MAKE SURE WE WERE MONITORING DOSE OUTCOMES AND IMAGE QUALITY IN A MEANINGFUL WAY. IN FACT TO A LARGE DEGREE WHEN WE WERE MONITORING THEM WE OFTEN DON'T TAKE IMPORTANT FACTORS INTO CONSIDERATION LIKE PATIENT SIZE. THAT LED TO AN OPPORTUNITY ESSENTIALLY THAT WE SAW AT CHILDREN'S HOSPITAL IN OUR LAB, WE SAID WE THINK WE CAN TACKLE THIS. SO WE CREATED OPTIMIZATION FRAMEWORK, DEVELOPED QUANTIFIABLE TARGET CURVES, REVERSE ENGINEERED AUTOMATED TUBE CURRENT MODULATION ALGORITHM, CREATED PREDICTION AND CONTROL MODEL, MONITORING APPLICATION, WE USED RELATIVELY STRAIGHTFORWARD MACHINE LEARNING APPLICATIONS TO DO THIS, AND THE RESULTS WERE WE REDUCED DOSE AND REDUCED VARIATION. SO HERE WE CAN SEE BEFORE AND AFTER OUR IMAGE QUALITY AND OUR DOSE. WE SEE NOT ONLY REDUCED IT BUT IMPROVED THE CONSISTENCY. AND SO WE WERE REALLY EXCITED, AND WE RUSHED TO PUBLISH THIS AND WE LICENSED TECHNOLOGY AND THOUGHT IT WOULD MAKE A BIG IMPACT ON THE FIELD. WHAT ACTUALLY HAPPENED WAS IT LANDED WITH THE RESOUNDING THUD. IT DIDN'T GO ANYWHERE, DIDN'T MAKE ANY IMPACT. IT HASN'T EVEN -- IT'S NOT EVEN LICENSED, THE LICENSEE GAVE IT BACK. I SPENT TIME REFLECTING ON THIS, WHAT HAPPENED? I DON'T THINK IT WAS THE MODELS. EVEN DEEP LEARNING MODELS I DON'T THINK WILL OUTPERFORM THESE BY TOO MUCH. I DON'T THINK THAT WAS THE PROBLEM. BUT I THINK A LOT WAS FUNDAMENTAL MISUNDERSTANDING OF THE CONCEPT OF QUALITY CONTROL IN RADIOLOGY COMMUNITY. SO THAT'S WHAT I WANT TO TALK MORE ABOUT TODAY. I WANT TO START FROM THE BASICS. HOW DO YOU EVEN DEFINE QUALITY? WE COULD SPEND A LONG TIME ON THAT, MANY PEOPLE HAVE. AT STANFORD WE HAVE REDUCED TO SAYING QUALITY IS ESSENTIALLY CONSISTENT, EXCELLENT. EX--- CONSISTENT EXCELLENCE, TRANSLATED INTO SOMETHING MORE TANGIBLE, MIGHT SAY RADIOLOGY REPORT TURNAROUND TIME IN ONE HOUR OR LESS IN 90% OF CASES, EXAMPLE OF A DEFINITION OF QUALITY. AND YOU NOTICE THAT HAS TWO ELEMENTS, IT HAS A DEFINITION OF EXCELLENCE OR PERFORMANCE AND DEFINITION OF CONSISTENCY. NOW THE OPPOSITE OF CONSISTENCY IS UNNECESSARY OR UNDESIRED VARIATION. SO TO A LARGE DEGREE, QUALITY CONTROL IS THE MINIMIZATION OF THAT UNNECESSARY VARIATION, AN IMPORTANT POINT. OFTEN WE RUSH TO REDUCE DOSE OR IMPROVE IMAGE QUALITY BUT THE TARGET IS MINIMIZE THE VARIATION TO MAKE IT PREDICTABLE AND CONSISTENT. AND FINALLY THERE ARE MANY ELEMENTS THAT HAVE TO BE IN PLACE TO ENSURE CONSISTENT EXCELLENCE HAPPENS IN THE CLINICAL ENVIRONMENT, AND THOSE ELEMENTS WHEN COMBINED CONSTITUTE A PROCESS CONTROL SYSTEM. WE'LL TALK ABOUT PROCESS CONTROL SYSTEM. WHAT I WOULD PROPOSE, KEY ELEMENTS, IT STARTS WITH A FRAMEWORK. IF YOU'RE GOING TO PUT TOGETHER SOMETHING YOU THINK CAN IMPROVE QUALITY YOU SHOULD ARTICULATE YOUR OPTIMIZATION MODEL. WHAT IS YOUR OBJECTIVE, YOUR VARIABLES AND CONSTRAINTS, CAN YOU STATE SO OTHER PEOPLE CAN EVALUATE WHAT YOUR APPROACH IS. SECOND, PERFORMANCE MEASURES. SO IF YOU HAVE A DIMENSION THAT YOU WANT TO IMPROVE, UNTIL YOU CAN DEFINE IT IN A WAY OR MEASURE IT IN A WAY THAT CAN BE SHARED AND DISCUSSED, THEN IT'S NOT SOMETHING THAT WE CAN REALLY USE AS A COMMUNITY RESOURCE, PERFORMANCE MEASURES SHOULD BE QUANTITATIVE AND REPRODUCIBLE. PERFORMANCE STANDARDS AND TARGETS, NOW YOU CAN MEASURE SOMETHING, WHAT DO YOU WANT IT TO BE? QUALITY IS A NORMATIVE CONCEPT, WE HAVE TO HAVE A STANDARD TO BUILD ON OUR PERFORMANCE MEASURES. AND THEN MONITORING APPLICATION, SO YOU KNOW WHAT YOU WANT, YOU CAN MEASURE IT, KNOW WHAT YOU WANT IT TO BE, YOU NEED TO LOOK AT WHAT IS ACTUALLY HAPPENING AND COMPARE TO WHAT SHOULD BE HAPPENING. WITHOUT A MONITORING APPLICATION, THERE'S NO WAY WE CAN DO THIS. THIS IS THE BLOCK AND TACKLING. NOW THAT GETS US TO THE INTERESTING AND HARD PART, THAT IS NOW HOW DO YOU MAKE IT BE WHAT YOU WANT IT TO BE ON A CONSISTENT BASIS? AND TO THINK ABOUT THAT, WHAT A PROCESS IS, IN GENERAL WE DEFINE A PROCESS AS SIMPLY A CONVERSION OF THE SET OF INPUTS TO OUTPUTS. SO TO FIGURE OUT HOW TO MAKE THE OUTPUTS YOU WANT BASED ON INPUTS WE USE MODELS, THERE ARE MANY TIMES. IN GENERAL I REFER TO THREE, MAYBE FOUR MODELS. ONE IS A MEASUREMENT MODEL. CAN YOU TAKE A SET OF DATA AND QUANTIFY IT IN STRUCTURE IN SOME WAY? SO I THINK THIS IS AN AREA RIPE WITH OPPORTUNITY IN MACHINE LEARNING. WE'RE GOING START TO SEE THAT THESE ALGORITHMS WILL MAKE WHAT WAS PREVIOUSLY UNMEASURABLE TO BE MEASURABLE. SECOND IS PREDICTION MODEL, SO CAN YOU PREDICT BASED ON WHAT YOU MIGHT DO HOW THAT WILL INFLUENCE OUTCOME. NEXT IS OPTIMIZATION MODEL, CAN YOU IDENTIFY A TARGET AND PREDICT TAKING DIFFERENT STRATEGIES PREDICT WHICH STRATEGY WOULD ALLOW YOU TO REACH YOUR TARGET IN THE MOST DESIRABLE WAY. FOR EXAMPLE, ONE MODEL OR SET OF MODELS WE'RE USED TO IS A MAPPING MODEL. THERE'S A MEASUREMENT MODEL THAT TELLS YOU WHERE YOU ARE, HOW FAR YOU ARE FROM YOUR DESTINATION. THERE'S A PREDICTION MODEL, WHAT WILL HAPPEN IF YOU TAKE CERTAIN ACTIONS. OPTIMIZATION MODEL WILL TELL YOU HOW TO GET THERE IN THE MOST -- THE FASTEST WAY POSSIBLE. AND THEN ANOTHER WAY WOULD BE A CONTROL MODEL, WHERE YOU BYPASS THE HUMAN, TAKE THIS OPTIMIZATION AND EMBED IT INTO THE MACHINERY, INTO THE PROCESS. PREDICTION MODELS AND OPTIMIZATION MODELS ARE GOING TO BE ELEMENTS WE SEE INCREASINGLY FROM THESE DEEP LEARNING MODELS BUT NEED TO BE SURROUNDED BY THIS OTHER INFRASTRUCTURE. ONCE YOU HAVE THE ABILITY TO ACHIEVE WHAT YOU WANT TO ACHIEVE, THEN YOU HAVE TO RECOGNIZE THAT THAT'S NOT ALWAYS GOING TO BE THE CASE, YOU HAVE TO HAVE ABILITY TO CONTINUE TO MAINTAIN YOUR PROCESS OR CHANGE YOUR PROCESS SO WE NEED FEEDBACK MECHANISMS ALLOWING OPERATORS TO CORRECT THE INPUT VARIABLES BASED ON OUTCOMES TO MAKE THIS A SELF-CORRECTING PROCESS. AND FINALLY BECAUSE THIS IS A COMPLEX SYSTEM AND REQUIRES A LOT OF MANAGEMENT, THERE NEEDS TO BE ACCOUNTABILITY MECHANISMS. ALL THIS IS DEPENDENT ON TRANSPARENCY. SO IT'S REALLY CRITICAL. EVEN IF WE HAVE SO-CALLED BLACK BOX TECHNOLOGIES, EVEN IF MODELS AREN'T PUBLISHED, IF WE HAVE PERFORMANCE STANDARDS AND TARGETS, IF THEY PUBLISH FRAMEWORK, ESPECIALLY WITH CONTROL MODELS ACTING WITHOUT HUMANS, THAT CAN BECOME SCARY. SO MY SUMMARY WOULD BE THAT TO THE TAKEHOME POINT, QUALITY REQUIRES ALL ELEMENTS OF A PROCESS CONTROL SYSTEM, SO THOSE WHO ARE DEVELOPING THIS MODEL, DESIGNED TO IMPROVE QUALITY, I WOULD SAY IT'S NOT ENOUGH TO DEVELOP A PIECE OF THAT. IF YOU WANT TO IMPACT QUALITY AND PERFORMANCE, YOU GOT TO DEVELOP THE WHOLE THING. WHAT I FORESEE IS LIKELY TO HAPPEN IS OUR PERFORMANCE -- OUR MEASUREMENT MODELS WILL START TO MAKE -- AGAIN MAKE WHAT WAS PREVIOUSLY UNMEASURABLE MEASURABLE. WE'LL RECOGNIZE WHAT'S ALREADY THERE, A TREMENDOUS AMOUNT OF VARIATION. YOU'LL BE SHOCKED. I HOPE YOU WON'T BUT YOU WILL BE SHOCKED BECAUSE IT'S ALREADY THERE, IT'S GOING TO MAKE IT VISIBLE. WE'RE GOING TO RUSH TO FIX THE MESS SO TO SPEAK BEHIND THE SCENE, SO I WOULD ADVOCATE WE ALL INSTEAD OF JUST GOING AND FIXING THE MESS BEHIND THE SCENE THAT WE CREATE A COMPREHENSIVE PROCESS CONTROL SYSTEM. THANK YOU. [APPLAUSE] >> THIRD TALK WILL BE GIVEN BY DR. LAKHANI FROM THOMAS JEFFERSON UNIVERSITY. >> GREAT. THANK YOU FOR HAVING ME HERE. SO, WE'VE TALKED A LOT ABOUT A.I. SYSTEMS. THIS IS SORT OF A NUTS AND BOLTS TALK ABOUT HOW DO WE GET THESE SYSTEMS INTO THE CLINICAL PRACTICE, HOW DO WE INCORPORATE THEM ESPECIALLY AS THEY BECOME MORE ROBUST, AND COMMERCIALLY AVAILABLE, EVEN AVAILABLE ON A RESEARCH BASIS. A COUPLE THINGS WE'LL TALK ABOUT IS IMAGE TRIAGE, CRITICAL FINDINGS. THERE ARE MANY TYPES OF WORKFLOWS FOR A.I. SYSTEMS, WHETHER THEY ARE INVOLVED IN IMAGE PROCESSING OR IMAGE ANALYSIS OR IMAGE CLASSIFICATION BUT WE'LL BE FOCUSING MORE ON LIKE IMAGE TRIAGE AS WELL AS CRITICAL RESULTS. SO, THERE ARE A FEW DIFFERENT OPTIONS ONE CAN THINK ABOUT BUT THIS IS JUST ONE BASIC ONE. IF YOU'RE A RADIOLOGIST, ONE POTENTIAL WORKFLOW IS A A.I. ON DEMAND. I WILL QUERY A.I. FOR ANALYSIS ON DEMAND, THIS COULD SUPPORT PATH OR ER-DRIVEN WORKFLOW, COULD BE FOR A PARTICULAR IMAGE, FOR A SERIES OF IMAGES FOR THE PATIENT LIKE A CT SCAN OR EVEN A PART OF IMAGE LIKE A LESION, LIVER LESION ON A CT. LET'S SAY I START WITH PACS, THIS IS A CHEST SPRAY, I SEE ABNORMALITY. I MIGHT HAVE MY OWN INTERPRETATION BUT WANT TO SOLICIT AN A.I. INTERPRETATION MAYBE AS A SECOND READ OR HELP ME MAKE MY DECISION. I MAKE A REQUEST, IMAGES ARE SENT TO A SERVER, FOR EXAMPLE, AN A.I. SERVER, RESULT CAN BE SENT TO THE PACS OR ME DIRECTLY, I RENDER A FINAL REPORT, USING A.I. TO DELIVER A FINAL RECORD. - - FIND REPORT. A.I. RESULTS COULD BE SENT DIRECTLY IF YOU WANT TO YOUR ELECTRONIC HEALTH RECORD OR TO A RIS. THIS IS AN EXAMPLE. THIS IS A RADIOLOGY WORK LIST THAT WE DEAL ARE EVERY DAY. IF I WANT, I CAN SAY RIGHT CLICK ON A STUDY, THIS IS A CT OF THE CHEST. I CAN SEND IT TO AN A.I. SYSTEM. THIS WOULD BE SENDING THE ENTIRE STUDY BUT GENERATED I CAN SEND A PARTICULAR IMAGE OR PORTION OF IMAGE WITH OTHER TYPES OF WORKFLOW OPTIONS. WHAT ARE SOME PROS AND CONS OF THIS TYPE OF WORKFLOW? WELL, I THINK ONE OF THE ADVANTAGES IS THAT I'M IN CONTROL OF SOLICITING A RELEVANT A.I. INTERPRETATION THAT MAY REDUCE FALSE POSITIVES BY SENDING SELECT CASES. BUT THE CON, IT DOES REQUIRE MANUAL STEPS, NOT ALL CASES WILL BE ANALYZED BY THE A.I. SYSTEM OR IF YOU HAVE A NUMBER OF A.I. SYSTEMS, THERE'S A POTENTIAL TO MISS FINDINGS. ANOTHER WORKFLOW OPTION WOULD BE TO SEND ALL STUDIES TO YOUR A.I. SYSTEMS OF A CERTAIN MODALITY OR EXAM TYPE, SO YOUR A.I. SERVER MIGHT AUTO-RECEIVE STUDIES, LIKE ALL CT CHEST STUDIES FOR RULING OUT P.E. FOR EXAMPLE. AS A RADIOLOGIST, I'LL SEE A WORK LIST LIKE THIS OR COULD SEE A WORK LIST LIKE THIS, IF I HAVE SUCH A SYSTEM I MIGHT WANT TO BE œABNORMALITY, WE COULD USE A FLAG OR ICON, THIS IS AN EXAMPLE WHAT ONE COULD LOOK LIKE. THIS IS AN A.I. CRITICAL RESULT. SO IN THIS CASE, A.I. SITUATION ITEM HAD RECEIVED ALL CT CHESTS FOR RULING OUT P.E. AND IT DETECTED ABNORMALITY HERE, AND SO I MIGHT WANT TO OPEN THIS FIRST. IF I WORK WITH A WORK LIST LIKE THIS MIGHT WANT TO OPEN ALL THE STUDIES WITH A POTENTIAL CRITICAL RESULT FLAGGED BY THE A.I. SYSTEM, A FORM OF READING PRIORITIZATION. I THIS COULD APPLY TO NORMAL VERSUS ABNORMAL. IF I HAD AN A.I. SYSTEM SCREENING ALL CHEST X-RAYS FOR NORMAL VERSUS ABNORMAL ONE WAY RADIOLOGISTS CAN OPEN ABNORMAL OR POTENTIALLY ABNORMAL RESULTS FLAGGED BY A.I. WHAT IF YOU HAVE MULTIPLE WORKLISTS? YOU CAN HAVE A SYSTEM IF YOU'RE READING OFF ONE WORKLIST BUT THERE'S A CRITICAL RESULT ON A ANOTHER YOU COULD HAVE A POP-UP WITH THE PATIENT'S NAME, THE FINDINGS, IN THIS CASE POSSIBLE PNEUMONIA AND LAUNCH THE STUDY WHICH SHOWS THE IMAGE YOU CAN DIRECTLY VIEW AND REPORT THAT. FOR THIS SITUATION AUTO-RECEIVING STUDIES FROM YOUR PACS OR MODALITY DIRECTLY, THERE ARE TWO WORKFLOW OPTIONS. ONE IS THAT YOU HAVE A RADIOLOGIST MANAGING THOSE PRELIM FINDINGS BEFORE ROUTING THE FINAL REPORT. HERE YOU HAVE YOUR A.I. SERVER GENERATING PRELIM INTERPRETATION, A RADIOLOGIST WILL LOOK AT THAT PRELIM, AND USE THAT TO GENERATE A FINAL REPORT. AND THAT GOES TO YOUR ELECTRONIC HEALTH CARE RECORD OR YOUR RIS. A RADIOLOGIST WILL ENSURE -- CAN ENSURE ACCURACY OF FINAL REPORT, YOU WON'T NEED IS A DISCREPANCY MANAGEMENT SYSTEM WHICH I'LL COVER IN THE NEXT SLIDE. COMPARED TO A SITUATION WITH A.I. GOING DIRECTLY TO THE EHR OR RIS LIKE A FINAL READ ALMOST OR PRELIM TO YOUR EHR THAT'S THE FASTEST TURNAROUND TIME. A RADIOLOGIST IS STILL IN THE LOOP TO LOOK BEFORE SIGNING OFF. THERE'S A RELATIVE DECREASE IN TURNAROUND TIME BUT IT'S STILL FASTER THAN WHAT WE HAVE TODAY WHERE WE'RE REVIEWING STUDIES FIRST IN, FIRST OUT. THIS IS A MISNOMER BUT -- WHAT IF YOU HAVE A SECOND SITUATION WITH A.I. PRELIMINARY RESULTS THAT ARE INITIALLY ROUTED TO YOUR EHR OR RIS? IN THAT CASE YOU'LL NEED TO MANAGE DISCREPANCY. YOU HAVE IMAGES FROM YOUR MODALITY, CT SCANNER OR X-RAY MACHINE, SPENT TO YOUR SERVER OR PUT FROM YOUR PAC. AND THEN THE A.I. GENERATES A PRELIM THAT GOES RIGHT TO YOUR EHR. SO, CLINICIANS CAN VIEW THAT, PULMONOLOGISTS, NEUROSURGEONS, AS RELATIVELY AS RADIOLOGISTS, BUT WHAT IF YOU HAVE A DISCREPANCY WHERE A RADIOLOGIST HAS A DISAGREEMENT WITH THE A.I. SYSTEM, YOU NEED A SYSTEM TO MANAGE THAT SIMILAR TO TRAINEES, OVERNIGHT TRAINEES, RESIDENTS, FELLOWS, WHO DO REPORT PRELIMINARY FINDINGS AND THE NEXT DAY AN ATTENDING STAFF RADIOLOGIST SOMETIMES HAS DISCREPANCIES AND WE HAVE SYSTEMS IN PLACE TO MANAGE THAT TO CALL IN THOSE DISCREPANCIES AND RECONCILE IT. IF YOU HAVE THIS SORT OF SYSTEM, WHERE YOU'RE CONFIDENT IN A.I. AND THEY ARE SENDING REPORTS TO YOUR EHR RIS, YOU WOULD NEED A DISCREPANCY MANAGEMENT SYSTEM TO RECONCILE THOSE. WHAT ARE THE PROS? THIS IS THE FASTER TURNAROUND TIME WITH PRELIMINARY RESULTS, IMPORTANT FOR CRITICAL RESULTS, INTRACRANIAL HEMORRHAGE, PNEUMOTHORAX WHERE TIME IS OF THE ESSENCE, BUT THE CON IS THAT YOU WOULD NEED A DISCREPANCY MANAGEMENT SYSTEM FOR A.I. PRELIMINARY FALSE POSITIVES AND FALSE NEGATIVES, I THINK YOU WOULD NEED A PRETTY ACCURATE SYSTEM IF YOU'RE GOING THIS ROUTE THAT'S HIGHLY SENSITIVE AND SPECIFIC. CLINICIANS COULD ACT ON INACCURATE PRELIMINARY RESULTS, AND THEIR POTENTIAL NEGATIVE DOWNSTREAM EFFECTS. I THINK THIS MAY PROBABLY WILL INCREASE PHONE CALLS TO THE READING ROOM BECAUSE YOU CAN IMAGINE CLINICIANS WILL CALL TO DOUBLE CHECK A LOT OF THESE A.I. PRELIMINARY READS. THERE ARE GOING TO BE MEDICOLEGAL RAMIFICATIONS, IF AN A.I. SYSTEM THINKS THERE'S A PNEUMOTHORAX, THERE'S A PROBLEM. IN CONCLUSION, I THINK YOU'LL SEE A COMBINATION OF WORKFLOWS, YOU'LL SEE ON DEMAND WHERE RADIOLOGISTS OR CLINICIANS ILLICIT A RELEVANT A.I. INTERPRETATION. YOUR A.I. SYSTEM WILL AUTO-RECEIVE STUDIES DIRECTLY FROM MODALITIES OR PAC AND I CAN HAVE RADIOLOGIES IN THE LOOP PRIOR TO THE FINAL REPORT OR HAVE IN CERTAIN SITUATIONS MAYBE WITH VERY HIGHLY ACCURATE ALGORITHMS FOR CRITICAL RESULTS, FOR EXAMPLE, A SYSTEM WHERE A RADIOLOGIST MANAGES DISCREPANCIES FOR THESE A.I. PRELIMINARY RESULTS INITIALLY ROUTED TO YOUR EHR/RIS. EITHER WAY, RADIOLOGIST ALSO WILL HAVE AN IMPORTANT ROLE TO ENSURE ACCURACY, SAFETY AND QUALITY. THAT'S IT. THANK YOU. [APPLAUSE] >> NEXT IS A TALK ON DETECTION BY DR. PREVEDELLO FROM OHIO STATE. PLEASE. >> THANK YOU VERY MUCH FOR THE OPPORTUNITY. I'LL BE TALKING HERE TODAY ABOUT DEFECTION. TO START, I WOULD LIKE TO PRESENT THE DEFINITION OF DETECTION, THERE'S A LOT OF CONFUSION, WHAT DETECTION REALLY MEANS. IF YOU LOOK IN THE DICTIONARY IDENTIFYING PROCESS OF SOMETHING CONCEALED. WHAT THIS MEANS IS THAT WE'RE IDENTIFYING AN OBJECT THAT IS INSIDE OF ANOTHER -- OF AN IMAGE. BASICALLY FROM A STANDPOINT OF MACHINE LEARNING WHAT WE'RE TALKING ABOUT HERE IS LOCALIZATION AND CLASSIFICATION ALL TOGETHER. IF YOU ASK A FIRST-YEAR RESIDENT WHAT DID THEY FEEL THAT THE STEPS THAT ARE NEEDED IN ORDER TO GET THE DETECTION THEY MAY SAY I HAVE THIS IMAGE, THIS IS A RIGHT FRONTAL LOBE INTRAPARENCHYMAL HEMORRHAGE BUT BEFORE WE GET THERE I'M LOCALIZING THE LESION AND THEN LOOKING AT THE LESION IN MORE DETAIL TO TRY TO DISCERN WHETHER THIS IS AN INTRAPARENCHYMAL, SUBARACHNOID, AND ONCE I HAVE THE LOCALIZATION AND CLASSIFICATION TOGETHER, I HAVE DETECTION, FOR THE FIRST-YEAR RESIDENT IT MAY LOOK LIKE A SERIAL PROCESS OF LOCALIZATION, AND DETECTION. BUT IF YOU LOOK FROM A MACHINE LEARNING STANDPOINT, THERE'S SEVERAL DIFFERENT PATHS YOU CAN GET TO THE DETECTION. WHAT I WANT TO PRESENT TODAY IS THAT INTERSECT BETWEEN LOCALIZATION, CLASSIFICATION, SEGMENTATION, THEY ALL CAN LEAD TO DETECTION. THERE ARE WAYS FOR US TO LOCALIZE A LESION BASED ON THE SEGMENTATION, THERE ARE WAYS FOR THE CLASSIFICATION TO TELL US WHERE THE LOCALIZATION OF A LESION IS. AND SO THEY ALL INTERSECT AND DETECTION CAN BE DONE MULTIPLE WAYS. IT'S IMPORTANT TO UNDERSTAND THESE POSSIBILITIES. SO FROM A MACHINE LEARNING STANDPOINT, I THINK THAT DEEP LEARNING INVIGORATED THE FIELD MANY FOLDS. MAINLY THERE'S SEVERAL WAYS FOR US TO GET TO DETECTION. ONE IS TO BUILT A NEURAL NETWORK THAT CAN CLASSIFY WITH A BOUNDING BOX THE AREAS OF INTEREST AND LOCALIZE THE PROCESS IN QUESTION. WE CAN ALSO CREATE A CLASSIFICATION ALGORITHM USING CONVOLUTION NETWORKS AS MENTIONED PREVIOUSLY TO TELL US BASED ON -- (INAUDIBLE) -- THE BOUNDARIES OF THE LESION, AND CAN ALSO GET TO THE DETECTION BY CREATING THIS MASS AROUND THE OBJECT OF INTEREST. SO AS WE CAN SEE, MANY WAYS TO GET TO THAT. STARTING WITH CLASSIFICATION, WHICH WAS WHAT WAS MENTIONED ALREADY, THE CONVOLUTION OF NEURAL NETWORKS, LET'S IMAGINE THAT YOU HAVE A SLIDING WINDOW THAT CAN -- THAT HAS THE ABILITY TO DETECT THAT HEMORRHAGE WITHIN THAT WINDOW. IF YOU GO ACROSS THAT IMAGE, WITH THAT DETECTION ENGINE, AND YOU KNOW WHERE THE AREAS OF THE IMAGE THAT HAVE THAT POSITIVE CASE, YOU CAN THEN CREATE A BOUNDING BOX BASED ON THAT APPROACH, SLIDING WINDOW APPROACH, TO DETECT THE LESION USING A CLASSIFICATION ENGINE. THE PROBLEM OF THAT APPROACH IS VERY INEFFICIENT, COMPUTATIONALLY EXPENSIVE, YOU'RE GOING TO HAVE TO DO THAT CLASSIFICATION MULTIPLE TIMES ACROSS THE ENTIRE IMAGE. THE OTHER POSSIBILITY HERE IS TO THEN INSTEAD USING A CONVOLUTIONAL NEURAL NETWORK, WE CLASSIFY THIS IMAGE AS EITHER A POSITIVE OR NEGATIVE CASE FOR HEMORRHAGE, AND IF IT IS POSITIVE THEN WE TAP INTO THE NEURAL NETWORK WITH THOSE SALIENCY MAPS OR ATTENTION MAPS DESCRIBED PREVIOUSLY BEING DONE EITHER WITH OCCLUSION OR OTHER MECHANISMS. WE CAN THEN DETECT WHAT AREA IS THE AREA OF INTEREST FOR THE NETWORK. AND THEN BASED ON THAT, THE NETWORK WILL THEN TELL US THAT THIS IS THE AREA OF INTEREST OR THIS IS THE AREA THAT MADE ME THINK THAT THIS IS A POSITIVE CASE FOR INTERPARENCHYMAL HEMORRHAGE. ALLS SALIENCY MAPS IS ANOTHER STRATEGY. MORE RECENTLY THE CONCEPT OF PROPOSED REGIONS HAVE ALSO BEEN PRESENTED. AND THE CONCEPT HERE IS THAT YOU SELECT WITHIN THAT IMAGE SEVERAL OPTIONS FOR YOU TO LOOK AT, WITH A CONVOLUTION NEURAL NETWORK. ONCE AREAS ARE PROPOSED YOU RUN A CONVOLUTION NEURAL NETWORK AND DETECT WHICH HAS THE HIGHEST PROBABILITY OF REPRESENTING THAT DISEASE OR LESION. AND IF WE GO THROUGH THESE PROPOSED REGIONS, INTERPARENCHYMAL HEMORRHAGE WILL BE SELECTED THERE. THERE WERE SEVERAL OF THESE PROPOSED REGIONS ALGORITHMS THAT USED THAT PROPOSED REGION SO R-CNN, PROPOSED REGIONS WITH CLASSIFICATION, FAST R-CNN WITH SLIDING WINDOW APPROACH AND FASTER R-CNN, WAITING FOR AN ALGORITHM CALLED "EVEN FASTER" BUT THAT NEVER HAPPENED. WHAT ACTUALLY HAPPENED THE EVOLUTION OF THE FASTER R-CNN WAS MASK R-CNN WHICH ON TOP OF THE CLASSIFICATION, IDENTIFICATION AND CLASSIFICATION OF THE LESION PROVIDED A MASK. I WILL SHOW AN EXAMPLE OF THAT A LITTLE LATER. WITH THAT, IN ONE SINGLE IMAGE WE CAN HAVE SEVERAL REPRESENTATIONS, SEVERAL IDENTIFICATIONS OR DETECTIONS, WITHIN ONE SINGLE IMAGE IN GETTING INTO SEMANTIC SEGMENTATION. NOW, WHAT IS THE CONTRIBUTION OF SEGMENTATION TO GET TO LESION DETECTION? THERE'S SEVERAL APPROACHES TO THIS. ONE OF THEM IS CALLED U-NET, ENCODER/DECODER ALGORITHM THAT ONCE YOU SELECT AN AREA OF INTEREST, THE NETWORK THEN LEARNS HOW TO SEGMENT THE ENTIRE 2D DATASET IN THE CASE OF U-NET, YOU BUT YOU CAN DO THE SAME THING FROM IF YOU HAVE THE ENTIRE DATASET USING A V-NET VOLUMETRIC, YOU CAN SEGMENT THE ENTIRE LESION. SO WHAT I'M GOING TO TRY TO PRESENT TODAY NOW WITH THESE CLINICAL EXAMPLES IS THE IMPORTANCE OF DOING THIS LESION DETECTION THROUGH SEGMENTATION. AND BUT BEFORE WE GO THERE, I'M JUST GOING TO PRESENT, SO THIS IS A CASE OF MULTIPLE HEMORRHAGIC LESIONS, INTERPARENCHYMAL HEMORRHAGE WITH SURROUNDING EDEMA. WHAT I ALLUDED FIRST IN THE PROPOSED REGIONS, IN SOME ALGORITHMS, THIS IS THE EXAMPLE OF A MASK R-CNN, SELECTING THE AREAS OF INTERPARENCHYMAL HEMORRHAGE DONE IN MILLISECONDS, THE COMPUTER CAN DETECT THOSE AREAS QUICKLY. THIS IS AN EXAMPLE OF A 3D USING A VOLUMETRIC SEGMENTATION STRATEGY USING A V-NET AT OUR LAB, SEGMENTS LEFT AND RIGHT LUNG, AND THE HEART, HALF OF A SECOND. THE FULL DATASET CAN BE PRESENTED IN A VERY FAST FASHION. AND THEN YOU CAN USE THAT SAME SEGMENTATION STRATEGY SO YOU SELECTED, NOW YOU KNOW ANATOMY, WHAT IS THE LUNG IN THE ENTIRE DATASET, AND THEN YOU CAN USE THE SAME SEGMENTATION STRATEGY TO GET INTO THE NODULE ITSELF, SO YOU'RE BUILDING PRETTY MUCH A NODULE DETECTION ALGORITHM BY DOING SEGMENTATION OF THE NODULE. AND THEN THIS IS A -- THE MASK THAT IS PRODUCED AFTER -- SO THIS IS THE PREDICTED MASK FOR THIS NODULE, AND THEN WE HAVE THE SEGMENTATION OF THE NODULE HERE. ONE OF THE THINGS WITH SEGMENTATION THAT IS NOT EMPHASIZED ENOUGH I THINK IS THAT WE KEEP SEEING DEEP LEARNING REQUIRES LOTS AND LOTS OF DATA. THE VOLUMETRIC EVALUATION OF THE LUNGS AND THE HEART, WE DID WITH 30 PATIENTS ONLY. AND IT HAS COEFFICIENT OF 98, WHICH IS ALMOST PERFECT. SO VERY SMALL NUMBER OF PATIENTS. AND SEGMENTATION DOES NOT REQUIRE A LOT OF PATIENTS IN A TYPICAL IMPLEMENTATION. WHEN IT GETS TO MORE SOPHISTICATED LESIONS, SMALLER LESIONS THAT GETS MORE COMPLICATED, IF YOU HAVE THAT STRATEGY OF GOING FROM ONE ANATOMICAL DISTRIBUTION IT CAN GET MANAGEABLE. THIS NODULE DETECTION HERE ALSO USED A SMALL NUMBER OF PATIENTS AND PERFORMED AT A 90.90, DONE IN MILLISECONDS, LESS THAN A SECOND TO SEGMENT SOLID LESIONS AND SOLID GLASS LESIONS. FROM A RADIOLOGIST STANDPOINT WE ATTRIBUTE SEGMENTATION TO SEEING THE LESION IN 3D, BUT FROM A COMPUTER STANDPOINT ONCE YOU SEGMENT THE NODULE YOU CAN CREATE BOUNDING BLOCKS, MEASURE IN THE 2D SPACE THE SAME NODULE, YOU CAN MEASURE THE VOLUMETRIC CHARACTERISTICS OF THAT NODULE AND YOU CAN SEND THOSE EXTRACTED FEATURES OF THE NODULE TO ANOTHER CLASSIFICATION ENGINE WHICH WILL THEN FURTHER CHARACTERIZE THIS NODULE, FOR EXAMPLE, INTO A BENIGN OR MALIGNANT. IN SUMMARY, DEEP LEARNING HAS CREATED VERY NEW OPPORTUNITIES IN TERMS OF LESION DETECTION, SEVERAL WAYS TO DO THAT BEING FROM A LOCALIZATION SEGMENTATION OR CLASSIFICATION STANDPOINT, AND SEGMENTATION SEEMS TO BE IMPORTANT WAY IN MEDICINE BECAUSE IT DOESN'T REQUIRE AS MANY IMAGES OR PATIENTS AS THE OTHER PROCESSES REQUIRE. AND THERE ARE SOME ADVANTAGES IN TERMS OF QUANTIFICATION AS WELL. THANK YOU VERY MUCH. [APPLAUSE] >> THE FIFTH PRESENTATION IS ON CLASSIFICATION, TO BE GIVEN BY OUR CO-CHAIR, MATTHEW LUNGREN, STANFORD. >> ALL RIGHT. GOOD AFTERNOON, I'M BACK. SO LET'S TALK ABOUT DEEP LEARNING CLASSIFICATION PROCESS. OUR LAB DOES DIFFERENT PROJECTS LIKE YOU DO. THIS IS SOMETHING THAT HAPPENS IN THE RADIOLOGY ENTERPRISE. I DO WORK IN DEEP LEARNING FOR NLP AND WORK FOR DETECTION AND CLASSIFICATION PROJECTS. I'LL TAKE US THROUGH ONE THAT WILL TIE CAN KEITH DREYER'S TALK. WE DIDN'T COORDINATE. A CLASSIFIER BASED ON 1400 KNEE MRIS, LABELED FOR NORMAL, ACL AND MENISCUS. WE HAD A HOLDOUT TEST SET OF 120 TO READ AND LABEL. THEY HAD USE OF EMR AND PACS, A NICE GROUND TRUTH TEST SET. ARCHITECTURE BASED ON 2D, CON NETS TO TRANSFORM EACH IMAGE IN PARALLEL TO FEATURE MAPS. WE TRANSFORMED AVERAGE FEATURE MAPS INTO MULTI-DIMENSIONAL VECTORS PASSED THROUGH THE CONNECTED LAYER TO CREATE PROBABILITY. THAT WAS GREAT BUT THE PROBLEM IS WE HAVE THREE SERIES AND THREE LABELS, NINE CON NETS TO DEAL WITH, USING LOGISTIC REGRESSION TO FORM INTO THREE BEST WEIGHTED DECISIONS BASED ON TRAINING DATA. THIS IS OUR RESULT ON THE TEST SET, EXCEEDINGLY WELL FOR ABNORMAL, ACL AND LESS FOR MENISCAL. WE DIDN'T DO ANNOTATION OR BOUNDING BOXES ON OUR IMAGES, SIMPLY EXAMINATION LEVEL LABELS. YOU CAN IMAGINE PERFORMANCE COULD IMPROVE ON THE MENISCAL TEAR HAD WE DONE THAT. WE ASKED RADIOLOGISTIST TO DO THE TEST SET TO COMPARE. WE DID PRETTY WELL AND WERE PROUD BUT WE WANTED TO FIGURE OUT HOW WOULD WE USE THIS IN PRACTICE? THERE'S BEEN TALK ABOUT AUGMENTED RADIOLOGY, HOW WOULD WE USE THE OUTPUT TO HELP A RADIOLOGIST MAKE A DECISION? I'M AN MPH, I CAN DESIGN A DRUG TRIAL, I'LL DESIGN A CROSSOVER TRIAL. WE TOOK TWO GROUPS OF RADIOLOGISTS, RANDOMIZED THEM, IN THE FIRST GROUP FOR NON-MSK RADS AND ONE ORTHOPEDIC SURGEON, IN THIS THREE NON-MSK AND ONE ORTHOPEDIC. THE IDEA IS THAT MAYBE THERE'S AN OPPORTUNITY TO USE A.I. TO HELP THE CLINICIANS READ THE IMAGES IN THEIR CLINIC AND POTENTIALLY MANAGE THEIR OWN IMAGING. IT'S POSSIBLE. WE GAVE GROUP A A.I. SYSTEMS, OUTPUT GIVEN TO THE READER, THE LABEL AND PROBABILITY OF THE MODEL THOUGHT IN TERMS OF CONFIDENCE. WE HAD A WASHOUT PERIOD. AND THEN WE SWITCHED THE GROUPS. RANDOMIZED ORDER AND SWITCHED GROUPS. HERE WERE OUR RESULTS. WE, AGAIN, SMALL SAMPLE SIZE, TREND IN IMPROVEMENT ON EVERY CATEGORY, EVEN MENISCAL TEAR, INTERESTING BECAUSE IT WASN'T HIGHEST PERFORMANCE, STATISTICALLY SIGNIFICANT IMPROVEMENT IN ACL TEAR, MORE STATISTICAL ANALYSIS WE FOUND THE SPECIFICITY IMPROVED FOR ACL TEAR, THERE WOULD BE LESS FALSE POSITIVE TAKING FEWER PEOPLE TO THE O.R. FOR SUSPECTED ACL TEARS SPEAKING BACK TO WHAT KEITH SAID AT THE BEGINNING OF THE CONFERENCE THAT THERE ARE OPPORTUNITIES TO USE CLASSIFIERS EVEN IF NOT PERFECT TO HELP RADIOLOGISTS AND OTHERS MAKE BETTER DECISIONS. BEFORE WE GOT TOO CARRIED AWAY I DID WANT TO TAKE THE NEXT FEW MINUTES TO TALK ABOUT BIAS BECAUSE THIS IS REALLY CRITICAL AND SOMETHING TO THINK ABOUT. THERE'S BOTH A.I. SYSTEM AND HUMAN BIAS, LET'S TALK ABOUT A.I. SYSTEM BIAS, A FIGURE FROM A BLOCK POST, AN AUSTRALIAN RADIOLOGIST WHO SAID LABELS WENT USEFUL. THESE ARE THE EXAMPLES FOR POSITIVE FOR PNEUMOTHORAX AND CIRCLED IN GREEN ALL THE POSITIVE CASES THAT HAD CHEST TUBES. IF YOU TRAIN A CLASSIFIER, THE CONCERN WOULD BE YOU'RE TRAINING A CLASSIFIER TO LEARN WHERE A CHEST TUBE IS AND NOT WHAT A PNEUMOTHORAX IS. BETTER EXAMPLE FROM THE GOOGLE GROUP IN COLLABORATION WITH MOUNT SINAI, THEY WANTED TO CREATE MULTI-INSTITUTIONAL DATASET TO PREDICT PNEUMONIA ON CHEST X-RAY. THEY TRAIN CLASSIFIER AND LOOKED AT DECISIONS WITH SALIENCY MAPS, LOOKING AT THE LATERALITY MARKER, TWO HOSPITALS USED TWO MARKERS, WITH 1% AND 34% PREF PREVALENCE, NOT CLINICALLY USEFUL,ANOTHER EXAMPLE OF BIAS. BEFORE WE BLAME BIAS ON A.I. MODELS IT'S IMPORTANT TO CONSIDER OUR OWN BIASES, FOR ME AND MOST OF US GET AROUND WITH AID OF A HELPER HERE, GPS, RIGHT? WE TRUST THIS ALMOST TOO MUCH IN CERTAIN CASES, AND THINGS LIKE THIS CAN HAPPEN. THIS IS ACTUALLY A PHOTOGRAPH FROM SOMEONE WHO DID FOLLOW THEIR GPS ON A TRAIN TRACK BECAUSE THE GPS TOLD THEM IT WAS A ROAD. LUCK THIS THEY SURVIVED. NOW THERE'S A TERM CALLED DEATH BY GPS. THE REASON THIS IS IMPORTANT IS WE HAVE AN INHERENT AUTOMATION BIAS TRUSTING THE COMPUTERS MORE THAN COMMON SENSE. THIS CAN HAPPEN IN AIRPLANE COCKPITS, ICUs, NUCLEAR POWER PLANTS SO IT'S IMPORTANT AS WE BEGIN TO THINK ABOUT HOW WE'RE GOING TO IMPLEMENT SYSTEMS IT'S A REAL SIGN. WE HAVE TO THINK ABOUT WARNINGS LIKE THIS, WAYS TO INFORM THE PEOPLE USING THESE MODELS ON THE BIASES THAT MIGHT BE PRESENT IN THE MODEL ITSELF WHERE IT COULD FAIL BUT ALSO OUR OWN BIASES, HEY, MAYBE YOU SHOULD CONSIDER CLINICAL INFORMATION WHEN THIS SAYS PNEUMONIA FOR SOMETHING LIKE THAT. HUMAN CLASSIFICATION IS DOABLE. AUGMENT THE IS EXCITING. CONTROLLING FOR BIAS IS GOING TO BE CRITICAL, PARTICULARLY AS WE START TO MOVE MODELS FROM ONE INSTITUTION TO THE OTHER BEFORE WE UNDERSTAND HOW WELL THEY ARE GOING TO WORK. AND THANKS AGAIN. [APPLAUSE] >> THE LAST TALK FOR THIS SESSION, THE TITLE IS ON A.I. AND THE RADIOGENOMICS GIVEN BY DR. BRADLEY ERICKSON, MAYO CLINIC. >> ALL RIGHT. THANK YOU. ACTUALLY BEFORE I GET STARTED PEOPLE HAVE ASKED ABOUT THE SOFTWARE I MENTIONED WE USED IN OUR LAB, OPEN SOURCE ON GITHUB, SEARCH FOR MAYO AND QIN YOU'LL SEE IT, CALLED MERMAID. LET'S SEE. THERE I GO. ALL RIGHT. SO ACTUALLY WE HAVE TO BE A LITTLE BIT CAREFUL ABOUT HOW WE DEFINE RADIOGENOMICS BECAUSE THE RADIATION, OUR COLLEAGUES IN RADIATION ONCOLOGY GOT THERE FIRST AND DESCRIBE IT AS STUDY OF GENERIC VARIATION, RESPONSE THROUGH RADIATION, I DON'T KNOW ANYTHING SO I HOPE YEAR HERE TO LISTEN TO THE SECOND, CORRELATION BETWEEN CANCER IMAGING FEATURES AND GENE EXPRESSION. SO, THIS IS MY VIEW OF THE WORLD, AT LEAST AS IT EXISTS IN MY LAB. WE START WITH AN IMAGE, THAT'S WHAT ALL TRUTH AND KNOWLEDGE STARTS WITH, RIGHT? AND WE FEED THAT IMAGE INTO A COMPUTER, AND OUT OF IT IS GOING TO BE A MOLECULAR MARKER OF SOME TYPE. AND THAT WILL REFLECT WHAT IS GOING ON AT THE GENETIC OR CHROMOSOMAL OR SOME MOLECULAR LEVEL, AND IN A WILL INFORM US HOW TO BEST TREAT THE PATIENT TO GIVE APPROPRIATE TARGETED AGENT AND THEN HOPEFULLY THERE'S A RESPONSE, FEED THE IMAGES INTO THE SAME COMPUTER, YOU CAN SEE THIS VIRTUOUS CYCLE WHERE I WHERE WHERE I WHERE THE IMAGING IS GOING TO SOLVE THE PROBLEMS OF THE WORLD. HOW MUCH INFORMATION CLEARLY IS GOING PAST MY EYEBALLS THAT EXISTS IN THESE IMAGES. SO THIS IS ONE THING THAT WE'VE STARTED TO FOCUS ON A LOT, LOOKING AT GENOMIC MARKERS IN BRAIN TUMORS. TWO CAREERS AGO THE WORLD HEALTH ORGANIZATION CHANGED THE -- TWO YEARS AGO THE W.H.O. CHANGED THE WAY TUMORS ARE RECORDED, USED TO SAY GRADE 4-GLIOBLASTOMA. THE IMPORTANT INFORMATION NOW THREE MOLECULAR MARKERS, IDH1 MUTATION, TERT AMPLIFICATION AND A HYBRID MARKER ON TOP OF ALL OF THOSE. SO THOSE ARE REALLY THE THINGS NOW THAT THE ONCOLOGISTS AND NEUROSURGEONS CARE ABOUT WHEN THEY THINK ABOUT TREATING A PATIENT. AND WE STARTED TO PREDICT THESE PRETTY ACCURATELY SO IN THIS CASE WE HAVE 500 SUBJECTS, USING T2 WEIGHTED IMAGES. WE DID EXAM LEVEL ANNOTATION, DIDN'T SAY HERE'S THE HUMAN. WE JUST SAID THIS EXAM HAS IDH1 MUTATION OR SOMETHING LIKE THAT. WE KEPT OUT 100 FOR TESTING, TRAINED ON 400, USED 50-LAYER RESNET TO COMPARE ACCURACY, VGGNET. MORE IDH1 THIS IS FOR THE TEST SET, NOT TRAINING VALIDATION PHASE, 90% ACCURACY FOR PREDICTING IDH1, FOR 1P19Q 90%, ATRX 90%, AND FOR MGMT METHYLATION 90%. WE'RE NOT PERFECT. YOU KNOW WHAT? PATHOLOGY ISN'T EITHER. 5 TO 10% OF THE TIME THEY DON'T GET ENOUGH TISSUE TO BE VIABLE. SO WHILE I DON'T THINK WE'RE GOING TO REPLACE PATHOLOGY, I THINK WE'RE COMPLEMENTARY TO WHAT PATHOLOGY DOES, AND THAT ANYTIME WE GET A DISCREPANCY THAT'S AN INTERESTING CASE WE NEED TO LOOK AT CLOSELY, PERHAPS THERE'S SOME VARIABILITY WITHIN THE TUMOR AND THEY NEED TO LOOK AT THAT PATIENT MORE CAREFULLY THAN THEY OTHERWISE MIGHT. OF COURSE, SOMETIMES THEY JUST DON'T GET A RESULT. I THINK THERE'S SOME USE. I DON'T SEE WE'RE GOING TO STOP HAVING SURGEONS DO BIOPSIES OR RESECTIONS OR STOP DOING PATHOLOGY. THE OTHER THING THAT'S INTERESTING IS WHY IS IT T2-WEIGHTED IMAGES? IDH1 ESSENTIALLY SAYS GLIOBLASTOMA OR NOT. ESSENTIALLY THAT MEANS IS IT ENHANCING OR NOT? WHY DIDN'T T1 COME OUT AS THE IMAGE BEST FOR PREDICTING IDH1? THIS GETS TO THE CHALLENGE OF VARIABILITY IN ACQUISITION PROTOCOLS. IF WE ONLY USE MAYO, IT'S T1, WE USE MULTIPLE SITES, SOME ARE 15+ YEARS OLD, AND THINK ABOUT THE VARIABILITY IN T1 POST CONTRAST IMAGING, RIGHT? THERE'S FPGR, CUBE, SPACE, ALL WAYS TO DO T1s. ABOUT HOW MUCH INNOVATION IN T2? WELL, WE BASICALLY WENT FROM SPIN ECHO TO FAST SPIN ECHO, RIGHT? NOT MUCH CHANGE. THAT'S WHY WE DO SO MUCH BETTER ON T2-WEIGHTED IMAGES OVER THIS LARGE COLLECTION OF IMAGES BECAUSE OF THE HIGHER CONSISTENCY IN ACQUISITION COMPARED TO WHAT WE SEE WITH T1 WEIGHTED. QUITE FRANKLY I THINK THAT A COUPLE YEARS AGO IF YOU WOULD HAVE SET TO NEURORADIOLOGISTISTS WE CAN PREDICT IDT 1 FROM T2 WEIGHTED IMAGES NONE WOULD HAVE BELIEVED THAT. I WOULDN'T HAVE BELIEVED THAT. THIS IS ONE OF THE THINGS THAT GIVES ME HOPE THERE'S A LOT MORE INFORMATION. SO IT ISN'T ABOUT A.I. REPLACING THE RADIOLOGISTS, THIS WILL MAKE US BETTER AND DRIVE MORE IMAGING, GETTING SO MUCH MORE INFORMATION FOR TAKING CARE OF PATIENTS. OBVIOUSLY I KNOW THE MOST ABOUT THAT. IT'S NOT ONLY HAPPENING IN BRAIN. THIS IS WHERE THEY HAD RADIOLOGISTS REPORT ON FEATURES, AND THEN USE THAT TO PREDICT RADIO GENOMIC FEATURES IN LUNG CANCER. THIS IS THE SAME GROUP LOOKING AT THE PROGNOSTIC IMPLICATIONS. THIS IS ANOTHER ONE LOOKING AT MOLECULAR CHARACTERISTICS FOR GLIOBLASTOMA, AND MORE THE SEMANTIC FEATURES. THIS IS ONE LOOKING AT NEUROBLASTOMAS, AND NEUROBLASTOMAS HAVE VARIOUS CHROMOSOMAL ABNORMALITIES, THE TYPES CAN BE PREDICTED BY IMAGING. NOW, SEEING THERE'S A LOT OF POTENTIAL HERE WE HAVE TO REMEMBER FIRST OF ALL CORRELATION DOES NOT PROVE CAUSATION. AND SO WE HAVE TO THINK ABOUT WHY IS THAT IMAGING ABLE TO IDENTIFY WHAT THE GENOMIC MARKER IS, AND I THINK THAT MAY LEAD TO INSIGHT IN HOW TO DO IMAGING BETTER AND MAY LEAD TO SOME INSIGHT INTO THE BIOLOGY. I THINK WHILE CORRELATION ISN'T AS GOOD AS CAUSATION, A LOT OF MEDICINE IS BASED ON CORRELATION AND WE SIMPLY HAVE TO VALIDATE AND MAKE SURE OUR VALIDATION METHODS ARE GOOD. AND QUITE FRANKLY METHODS AND DATA TO VALIDATE CORRELATIONS ARE OFTEN DIFFICULT TO OBTAIN. I'M ONE THAT I'M NOT THAT BIG A BELIEVER IN USING RADIOLOGY REPORTS AS GOLD STANDARDS. I THINK THE MORE THAT WE CAN RELY ON OTHER THINGS OBVIOUSLY I LIKE TO SEE GENOMIC TESTS AS MY GOLD STANDARD, SURVIVAL DATA, BUT SOMETHING OUTSIDE OF IMAGING THAT'S NOT BASED ON IMAGING EVEN IF IT DOES PASS THROUGH RADIOLOGISTS, IT'S ALWAYS GOING TO BE A BETTER PROOF OR INDICATOR THAT THE FINDINGS ARE RELIABLE. AND THEN WE NEED METHODS TO GAIN UNDERSTANDING FROM THE CORRELATION AND THAT'S THAT XPLAINABLE A.I. PIECE THAT I TALKED ABOUT. I THINK, YOU KNOW, THE COOL THING IS THAT A.I., YOU'VE HEARD SEVERAL TIMES ALREADY TODAY, THIS IS GOING TO CAUSE SUBSTANTIAL CHANGES IN HOW WE PROVIDE HEALTH CARE TO PATIENTS. IT'S GOING TO MEAN THAT WE WILL BE MUCH MORE CAREFUL IN HOW WE COLLECT DATA AND TRY TO COLLECT EVERY BIT OF IT THAT WE CAN, BOTH INPUTS AND OUTPUTS. I THINK THAT WE WILL HAVE MUCH MORE ROBUST EVALUAION OF THE RELATIONSHIP BETWEEN INPUTS AND OUTCOMES, AND MEANS WE WHO ARE HEALTHCARE PROVIDERS HAVE TO BECOME DATA SCIENTISTS OR AT LEAST DATA AWARE TO ENSURE DATA ARE PROPERLY HANDLED. YOU HEARD ABOUT BIASES BEING INTRODUCED IN DATA, WE HAVE TO BE CAREFUL AND THOUGHTFUL, AND CAREFUL WITH DATA AND THAT MEANS WE HAVE TO BE THE ADVOCATES FOR OUR PATIENTS AS WELL TO PROTECT HOW THEIR DATA IS USED. I USE THAT, THEIR, IN THE SENSE I BELIEVE IT BELONGS TO THEM REGARDLESS OF WHAT THE LAW SAYS AND FINALLY HAVE TO PREPARE US AND OUR PATIENTS FOR THE CHANGES THAT A.I. WILL DRIVE. THANK YOU. [APPLAUSE] >> JIM FROM MGH. YOU'VE BEEN A MASTER IMAGE RECONSTRUCTOR FOR DECADES, I WAS INTRIGUED BY YOUR BAD TO BEST APPLICATION FOR MACHINE LEARNING. MY QUESTION IS, IS THE BAD TO BEST POTENTIAL SO GREAT AS TO BE THE NEXT BIG DISRUPTION IN CT? I.E. DO YOU THINK IT'S BIG ENOUGH TO TAKE US BACK TO FLAT PANEL CT TO REALLY IMPROVE CHROME BEAM RECONSTRUCTION TO ALLOW US TO REPLACE THE ROTATING GANTTRY? >> THANKS VERY MUCH, JIM. I DO HAVE A HIGH LEVEL EXCITEMENT AND CONFIDENCE. MACHINE LEARNING IS AT EARLY STAGES, A LOT OF THINGS CAN BE DONE, POTENTIAL IS GREAT. FURTHER DEVELOPMENT DEPENDS ON PARTNERSHIP WITH MEDICAL SCHOOLS AND CT VENDORS AND I FEEL VERY EXCITED. SPECIFIC ANSWER DEPENDS ON AVAILABILITY OF BIG DATA, RIGOROUS EVALUATION, AND OPTIMIZATION. MANY KINDS OF NEURAL NETWORKS, A LOT OF WORK NEEDS TO BE DONE. >> THIS IS (INDISCERNIBLE) FROM RPI. MANY CASES WE DON'T HAVE THE GROUND, COULD HAVE GOLD STANDARD FOR EXAMPLE DOCTORS (INDISCERNIBLE) ONE STEP FURTHER EVEN IMAGING RECONSTRUCTION PROCESS THAT'S OPTIMIZED FOR HUMANS, THAT PROCESS ITSELF, RAW DATA IS ALMOST IMPOSSIBLE TO GET. MY QUESTION FOR OUR COMMUNITY, AS A COMMUNITY, WHAT WE CAN DO TO ACHIEVE THAT. >> SOUNDS LIKE YOU'RE ASKING ABOUT MORE LABELED RAW DATA, OR WERE YOU SAYING LABEL DATA IN GENERAL? >> IN GENERAL. >> YEAH, THAT'S SOMETHING THAT HAS BEEN A MISSION OF OUR CENTER. WE FEEL IT'S PART OF THE ACADEMIC POINTS OF BEING AN ACADEMIC TO HELP THE COMMUNITY, SO OUR GOAL IS TO TRY TO DO WHATEVER POSSIBLE TO RELEASE AS MUCH LABELED MEDICAL IMAGING DATA AS WE CAN. IT'S NOT GOING TO SOLVE ALL THE PROBLEMS BUT WE WOULD LIKE TO ENCOURAGE OTHERS WORKING ON THESE PROJECTS TO BEGIN THE CONVERSATION WITH PRIVACY OFFICERS, TO CONSIDER RELEASING DATA. WE HAVE A PROCESS IN PLACE AT STANFORD, NOT NECESSARILY PERFECT BUT ONE WE PLAN TO USE BECAUSE WE DO THINK IT'S IMPORTANT TO HAVE THIS DATA OUT THERE. AND IF MORE PEOPLE DID THAT YOU CAN HAVE BETTER MODELS. WHAT WE FOUND WITH THE RSNA CHALLENGE, A GREAT EXAMPLE OF THIS, THE TEAM THAT WON THE RSNA CHALLENGE FOR BONE AGE, IT WASN'T GOOGLE OR MICROSOFT. IT WAS GRAD STUDENTS IN TORONTO BECAUSE WE RELEASED THE DATA. THAT'S WHAT WE PLAN TO DO. >> I AGREE. I THINK THERE'S AN IMPORTANT CONTRIBUTION THAT CHALLENGES LIKE THIS, LIKE WE'RE HOSTING AT RSNA, I CHAIR THE MACHINE LEARNING STEERING COMMITTEE FOR RSNA, ONE OF THE GOALS WE HAVE IS TO FOSTER MACHINE LEARNING TO ADVANCE MACHINE LEARNING THROUGH COMPETITIONS, AND THIS IS ONE WAY TO PROVIDE LABELED DATA FROM MANY ORGANIZATIONS TO THE COMMUNITY, THE NEXT CHALLENGE AS CURT HAD MENTIONED PREVIOUSLY IS GOING TO BE ON PNEUMONIA DETECTION, SO KEEP AN EYE AND EAR FOR NEXT WEEK, WHEN WE'RE GOING TO LAUNCH IT. >> AMBER SIMPSON, MEMORIAL SLOAN-KETTERING. THE IDEA WAS A GENERAL SEGMENTATION ALGORITHM, TO MAKE AN IMAGE NET, WE'RE PUTTING THAT OUT THERE TO KEEP GOING WITH THAT CHALLENGE YEAR AFTER YEAR TO BUILD IT OUT. AS AN INDIVIDUAL INVESTIGATOR, JUNIOR INVESTIGATOR, I SIT ON ALL LABELED DATA, MY PRIVACY OFFICE WILL MET ME SHARE IT. THAT'S NOT A PROBABLY. THE PROBLEM IS GETTING THE IMAGES, GETTING THE ANNOTATIONS BURIED IN THE IMAGES IN A WAY THE COMMUNITY CAN WORK WITH THEM SO WHETHER YOU DO THAT THROUGH DICOM OR NIFTI, THAT'S DIFFERENT ACROSS INSTITUTIONS, NO UNIFORM WAY TO DO THAT. I'M JUST A JUNIOR INVESTIGATOR TRYING TO MAKE THIS HAPPEN AND HAVE TO BUILD THIS IN BY GRANTS. I'M WRITING SCRIPTS, THERE'S GOT TO BE A BETTER WAY. WHETHER THAT COMES FROM NCI OR NIH, I DON'T KNOW. BUT WE NEED SOME SUPPORT FOR THAT TO MAKE THESE THINGS INTEROPERABLE TO GET THE DATA OUT THERE. I DON'T REALLY HAVE A QUESTION. IT WAS REALLY JUST -- I WANT TO GIVE YOU GUYS DATA. I HAVE A LOT. PLEASE TAKE IT. [LAUGHTER] [APPLAUSE] >> I THINK AMBER MENTIONED A VERY IMPORTANT ISSUE. I WOULD ADD TO THAT TOMOGRAPHIC RECONSTRUCTION FOR CT RAW DATA FORMAT TURN OUT TO BE VERY IMPORANT. IN OUR FIELD, DATA FORMAT, NOT UNIVERSALLY OPEN RELY ON GENERAL ELECTRIC BUT IN THE FUTURE SOME MECHANISM, I DON'T KNOW IF WE CAN MAKE OTHER VENDORS' RAW DATA SOMEHOW MAYBE AFTER CERTAIN TYPE OF CONVERSION AVAILABLE TO RESTRICTED COMMUNITY THINGS LIKE TO FINAL DIAGNOSIS, NOT ONLY MENTIONED CT AND MRI, MULTIPLE MODALITIES, RADIOMICS, LIKE UNIFIED RADIOMICS, SO THIS IS KIND OF COMMUNITY NEEDS FOR DATA INFRASTRUCTURE. I WOULD AGREE. >> YEAH, I AGREE. WE HAVEN'T OPENED THE CAN OF WORMS YET HOW WE'RE GOING TO DE-IDENTIFY OR MAKE SURE PRIVACY OFFICE IS HAPPY THE RAW DATA IS DE-IDENTIFIED. IT DOESN'T ACTUALLY CONTAIN ANY PHI BUT WE HAVE TO CONVINCE PEOPLE IT DOESN'T, RIGHT? I HAVEN'T TACKLED THAT ISSUE YET. THAT'S A CAN OF WORMS FOR ANOTHER DAY. THESE ARE ALL THINGS IN OUR WORLD THAT ARE JUST STOPPING US FROM MOVING FORWARD. >> VERY EXCITING TALKS. THANK YOU VERY MUCH. STARTING WITH THE BIAS ISSUES YOU MENTIONED WITH THE CHEST X-RAYS, LABELING PNEUMONIA, YOU START LABELING HOSPITALS, FROM THERE WE SEE IMAGE CONSTRUCTION, DESPITE THE FACT THAT THE MEDICAL IMAGING DEVICES HAVE GOOD MATHEMATICAL MODELS TO DEFINE HOW THE RECONSTRUCTION IS DONE. I'M A LITTLE BIT CONCERNED THAT WHEN YOU TRY TO PUT A.I. IN FOR RECONSTRUCTION, THE TRAINING TEST WILL BE THE OUTCOME, AND WHAT ARE YOUR THOUGHTS? >> I DON'T DO A LOT OF RECONSTRUCTION WORK BUT I AGREE THERE'S POTENTIAL FOR BIAS IN THE SETTING. ONE OF THE THINGS WE'VE SEEN WITH REGARD TO VENDOR DATA, IF YOU TRAIN ONLY ON ONE TYPE OF SCANNER, THE MODEL DOESN'T WORK SO WELL ON OTHER TINES OF SCANNERS. THAT'S EVEN TRUE ACROSS THE YEARS. ONCE G.E. COMES OUT WITH VERSION 6.5 MY MODEL TAPED -- TRAINED ON 6.0 DON'T WORK AS WELL. THAT'S SOMETHING WE HAVE TO HAVE A CONVERSATION WITH VENDORS ON A REGULAR BASIS. IN TERMS OF RAW DATA SIDE I'M CERTAIN THERE ARE GOING TO BE ISSUES IN TERMS OF TRANSFORMATION AS WELL. COMMENTS? >> I WAS ALSO MENTIONING AT THE LOWER LEVEL OF IMAGE CONSTRUCTION IN TERMS OF PHYSICAL MODELS CHANGING THE PHYSICAL MODELS WITH THE TRAINING SET, YOUR POINT IS WELL TAKEN, IT IS AT A HIGHER LEVEL, BUT IMAGE CONSTRUCTION IS SITTING AT THE BOTTOM OF EVERYTHING THAT WE'RE DOING, SO -- SORRY. >> I WOULD ADD TO DISCUSSION, I THINK MACHINE LEARNING CAPABILITY HAS TO BE VERY IMPRESSIVE. I READ SOME ARTICLES TALKING ABOUT CONVERGING CONVERGING -- CONVERTING, LIKE MRI TO CT, CT TO MRI. IF WE GAVE MORE GENERAL INTERPRETATION, THE SCANNER ALGORITHM, EVEN IMAGING MODEL ALGORITHM, THEY CAN BE CONVERTED SOMEHOW SO THAT LAYER MAY BE THE VENDOR DIFFERENCE, MODEL MISMATCH, COULD BE SOMEHOW SMOOTHED OUT, THIS IS CLEARLY OPEN TOPIC FOR FURTHER INVESTIGATION. >> ONE LAST THING YOU COULD EVEN TAKE THAT FURTHER AND RECONSTRUCT THE WHOLE BRAIN TISSUE, FOR INSTANCE, YOU KNOW, IF YOU WANT TO GO THAT FAR. >> RIGHT. THAT'S POSSIBLE. >> OKAY, THANK YOU. >> CAN I COMMENT? MAYBE THIS DOESN'T DIRECTLY ADDRESS YOUR QUESTION, BUT I THINKWE'RE HEARING OVER AND OVER DOES EVERY MODEL NEED TO BE VALIDATED AND TRAINED ON EVERY SINGLE MACHINE AND EVERY SINGLE SETTING? IF THAT'S THE CASE WE'RE GOING TO HAVE A MAJOR PROBLEM WITH COMBINATORIAL EXPLOSION QUICKLY, AND SO WE'LL HAVE TO FIGURE OUT WAYS AROUND THAT. ONE WAY YOU CAN LOOK AT ACTUALLY THE BONE AGE APPROACH IS PROVIDED A NICE EXAMPLE WHERE THE STANDARD IS A NICE STANDARD OUT THERE EVEN THOUGH IT WAS ORIGINALLY DEVELOPED ON A SMALL SUBSET OF CAUCASIAN MALES, YOU CAN TRAIN A MODEL FOR THE PILE STANDARD AND THEN TAKE LOCAL VARIATION AND CONNECT IT OR CHANGE THE WEIGHTS RELATIVE TO THE STANDARD. SO YOU MIGHT HAVE SOME INTERMEDIARY. EITHER WAY WE'LL HAVE TO FIND A WAY WE CAN OVERCOME THIS COMBINATORIAL PROBLEM. >> HI, RON SUMMERS FROM THE NIH CLINICAL CENTER. I HAVE A COMMENT AND QUESTION ABOUT CHALLENGES, AND THEIR UTILITY IN ADVANCING SCIENCE IN OUR FIELD. IN THE IMAGENET LARGE SCALE VISUAL RECOGNITION CHALLENGE WE LEARNED SOMETHING WHEN DEEP LEARNING CAME ALONG AND THE ERROR RATE DROPPED. COMMENT ON WHAT WE LEARN FROM RADIOLOGY-FOCUSED IMAGING CHALLENGES. >> I THINK AT LEAST FROM OUR EXPERIENCE WHERE THE CHALLENGES IN RSNA, AT THIS POINT THEY HAVE BEEN FOCUSED ON VERY SPECIFIC CLINICAL SCENARIOS, THAT THE EARLY PRESENTATION FROM KEITH THIS MORNING SHOWING THAT GRID OF ALL THE CLINICAL SITUATIONS POSSIBLE IN THAT IT'S ONLY A DROP IN THAT OCEAN POSSIBILITY, BUT WE HAVE BEEN ADVANCING THE FIELD FORWARD IN TERMS OF UNDERSTANDING THAT VARIETY, THE DATA COMPLEXITY, UNDERSTANDING THE PRIVACY CONCERNS, UNDERSTANDING SEVERAL PIECES RELATED TO ALGORITHM DEVELOPMENT AND VALIDATION. THIS IS A NEW FIELD FOR EVERYBODY. I MEAN, UNLESS YOU ARE ONE OF THOSE EARLY CREATORS OF THE DEEP LEARNING, YOU ARE IN THE BOAT AS EVERYBODY ELSE WHICH IS WE'RE ALL TRYING TO BE -- TRYING TO FIGURE OUT AS WE GO. AND I THINK THAT BY EXPOSING THESE DATA SETS TO THE VARIETY OF DIFFERENT DOMAINS AND PEOPLE WITH DIFFERENT BACKGROUNDS, WE ARE SEEING A MULTIPLICATION OF SOLUTIONS IN A FASHION THAT IS VERY DIFFICULT TO REPLICATE WITH THE STANDARD PUBLICATIONS IN THE PREVIOUS CHALLENGE, FOR EXAMPLE, THE 2017 BONE AGE CHALLENGE, THE ALGORITHMS, THE SOLUTIONS PRESENTED, I THINK THE PREVIOUS STATE OF THE ART, FOR EXAMPLE, FOR THAT SPECIFIC CLINICAL SCENARIO, AGAIN, IT'S ONE SINGLE DROP IN THIS OCEAN BUT AT LEAST THERE ARE SOME NON-TANGIBLE THINGS THAT ARE BEING LEARNED WITH THOSE CHALLENGES THAT ARE ALSO VERY HELPFUL. >> ONE OTHER THING THAT I GUESS I WOULD MENTION IS THAT THERE'S PROBABLY BEEN A GREATER LEVEL OF HUMILITY AMONG PHYSICIANS. YOU KNOW, I THINK RADIOLOGY, WE'VE ALWAYS HAD OUR MISTAKES POINTED OUT TO US, YOU KNOW, SOMEBODY SAYS WE TAKE PICTURES OF EVERY ONE OF OUR MISTAKES. BUT I ALSO WORK WITH PATHOLOGY AND CARDIOLOGY, AND I THINK THEY WERE PROBABLY NOT AS AWARE OF VARIABILITY STARTING TO BECOME MORE AWARE AS WE BUILD OUT THESE CHALLENGES AND BUILD THESE DATASETS, I THINK THAT THEY WERE MUCH MORE INSULAR THAN WE WERE IN RADIOLOGY AND I DON'T THINK THEY WERE AWARE OF HOW MUCH VARIABILITY THERE WAS. I THINK THAT'S PROBABLY ONE OF THE BIGGER BENEFITS OF THIS WHOLE CHALLENGE MENTALITY AND BUILDING LARGE DATASETS IS THE ART OF MEDICINE THAT SOMETIMES GOT IN THE WAY OF SCIENCE MEDICINE. >> COULD I COMMENT FROM THE PERSPECTIVE OF THE GROUP WHO DONATED THE IMAGES AND SPEAKING OF HUMILITY, WE HAD ALREADY PUT TOGETHER OUR OWN ALGORITHM AND KNEW WE WERE GOING TO GET DEFEATED SOUNDLY BUT THAT'S OKAY. ACTUALLY IT PROVIDED A NICE -- WE TALKED ABOUT WHAT INCENTIVE FOR ORGANIZATIONS TO SHARE DATA. I THINK THIS PROVIDES, NUMBER ONE, NOTORIETY AND HIGHER PROFILE BUT ALSO REALLY SETS THE TONE FOR THE WHOLE FIELD, FOR HOW WE SHOULD APPROACH IMAGE SHARING. SO I THINK ESPECIALLY GIVEN HOW THE LEVEL OF ENGAGEMENT AND JUST OVERALL CULTURE, AS DIFFICULT TO QUANTIFY AS THAT IS, IT'S REALLY CRITICAL ESPECIALLY AT THE OUTSET OF THIS WHOLE THING. >> COULD I ADD SOMETHING TO THAT? >> QUESTION, GO AHEAD. >> ONE OF THE PEOPLE PARTICIPATING ON THE CHEST X-RAY CHALLENGE, I'D SAY WE HAVE ACTUALLY LEARNED A LOT ON THE COMMITTEE ABOUT WHAT DOES IT TAKE TO TRY TO CREATE A VERY LARGE SCALE DATASET AND HOW DO YOU DO THAT WELL. AND I THINK THAT KNOWLEDGE IS SOMETHING THAT YOU ALSO WORKED IN AND WE'RE PLANNING ON SHARING THAT WITH A LARGER COMMUNITY BECAUSE WE'RE TRYING TO TAKE PEOPLE WHO ARE USED TO CLINICAL RESEARCH AND REALLY HELP THEM UNDERSTAND THAT GAP BETWEEN CLINICAL RESEARCH AND GOOD DATASETS THAT WORK FOR MACHINE LEARNING. >> I HAD ANOTHER QUESTION. >> YOU WERE STANDING FIRST. >> OKAY. CAN YOU TELL US AT ALL WHAT -- WE HEARD A TON ABOUT IMAGE DATA AND WE'VE ALSO HEARD THAT WEAK LABELS ON A LARGE NUMBER OF IMAGES COULD BE VERY BENEFICIAL FOR TRAINING DEEP LEARNING ALGORITHMS. ARE YOU AWARE OF ANY EFFORTS TO CREATE LARGE SCALE TEXT DATASETS THAT ARE RELATED TO RADIOLOGY STUDIES, AND WHEN WE MIGHT SEE THOSE? BECAUSE THE mLP THINGS ARE SO IMPORTANT. >> YOU READ MY MIND, TRYING TO GET PRIVACY OFFICE TO ALLOW US ON RELEASE REPORTS BECAUSE WE WANT THOSE AS PART OF THE DATASET, BETTER LABELS THAN OUR NOISY PROSPECTIVE LABELS, FOR THE X-RAY CHALLENGE BONE X-RAY. IT'S HARDER BECAUSE A TINY SLIPUP AND YOU'RE IN TROUBLE WITH HIPAA, RIGHT? AND SO WE'RE WORKING ON A DEAL RIGHT NOW, I DON'T KNOW IF THIS WILL WORK FOR YOUR INSTITUTION, BUT SEEM TO BE CLOSE TO AN AGREEMENT THAT DOUBLE MANUAL REVIEW MEANING HAVING A HUMAN TRAINED KNOWING REQUIREMENTS OF HIPAA WITH VISUAL EYES ON THAT REPORT TWICE, AND SO HOWEVER FAST YOU CAN SCALE THAT WITH WHATEVER BUDGET YOU'RE WORKING WITH, AT LEAST THAT'S HOW WE'VE BEEN APPROACHING, WHAT SEEMS LIKE IT MAY RARE. WHEREAS MAYBE RON COULD TELL US HOW HE WAS ABLE TO RELEASE AT LEAST THE CHEST X-RAYS, FOR OUR SIDE IT REQUIRES MANUAL REVIEW, NO OTHER WAY AROUND IT, YEAH. >> I'LL JUST COMMENT, WE ALSO LOOKED DOUBLE READ EVERYIVE -- EVERY IMAGE. >> MY NAME IS JOE. CONSIDERING NOISE AND DATA IN IMAGES THAT CAN'T BE SEEN LIKE PUT UP EARLIER WITH STUFF THAT'S NOT VISIBLE TO THE HUMAN EYE, DO YOU THINK THERE'S ROOM FOR A NEW TYPE OF IMAGING THAT RELIES ONLY ON A.I. RATHER THAN HUMAN EYES, ONCE WE TRUST A.I. ENOUGH? >> I'M NOT READY TO GO TO A NEW TYPE BUT I DO THINK THERE ARE NEW WAYS TO OPTIMIZE IT. A NEW TYPE COULD BE VIEWED BROADLY, I'LL PUT THE FLASHLIGHT UP TO YOU, THAT'S MY NEW TYPE. IN TERMS OF M.R. IS REMARKABLY FLEXIBLE, HOW IT ACQUIRES, FINGERPRINTING TECHNOLOGY WHERE YOU CAN -- THAT'S SORT OF AN ACQUISITION, MAY HAVE A LOT MORE INFORMATION THAN WHAT A HUMAN CAN PROCESS OR APPRECIATE. I THINK IT'S VERY LIKELY THAT WILL OCCUR. >> THANK YOU. >> I WOULD SAY I THINK THE HUMAN BRAIN IS AN EXCELLENT DEEP NEURAL NETWORK MODEL, AND IF YOU CAN CREATE IT IN A WAY THAT'S VISUAL AND WE CAN LEARN FROM IT, IT'S HARD TO BEAT THAT. >> REGARDING THE RECONSTRUCTION AND BIAS, SO THE TASK WHEN WE DO RECONSTRUCTION IS ALWAYS A BALANCE BETWEEN THE BIAS AND VARIANCE. ITERATIVE CONSTRUCTION TRIES TO FIGURE OUT HOW MUCH TO REGULARIZE WITHOUT INCREASING THE BIAS. I LIKE THE APPROACH BUILDING ON TOP OF THE REGULARIZATION FROM ITERATIVE METHODS FOR A.I. BUT I - THERE'S A WORD OF WARNING FOR HAVING A.I. FROM SONOGRAM SPACE TO RECONSTRUCTION, THERE'S A LOT OF PHYSICAL PROPERTIES YOU'RE MODELING IN THE IMAGE PROPAGATION, INHERENT TO EVERY SYSTEM. THAT IS WHY THERE'S HOPE IF YOU HAD AN ITERATIVE APPROACH AND ON TOP OF IT A.I., I DON'T THINK THAT WOULD WORK IF YOU START FROM THE SONOGRAM AND PREDICT FROM CT AND MR BECAUSE THAT -- ALTHOUGH YOU'RE RIGHT, THERE HAVE BEEN PAPERS THAT SHOWED THAT, MAYBE ON AVERAGE THAT'S CORRECT BUT EVERY TIME THERE'S DEVIATIONS NORMAL DEVIATIONS YOU'RE NOT GOING IT PICK THEM UP BECAUSE THAT WAS NEVER IN YOUR DATA. COULD YOU COMMENT ABOUT THAT? >> I WOULD AGREE. HOW TO OPTIMIZE IMAGING PERFORMANCE FROM RAW DATA, ALL THE WAY TO THE END, IS NOT A TRIVIAL ISSUE. SO REALLY THINK IF WE NEED BIG DATA, SINCE WE'RE TALKING ABOUT DATA-DRIVEN, WE NEED DATASET LARGE ENOUGH REPRESENTATIVE, MANY VARIABILITIES, SO THIS IS SOMETHING I BELIEVE CAN BE DONE BUT SYSTEM CANNOT BE GENERALLY CLAIMED, IF YOU HAVE BIG DATA SET YOU POWER OF MOST OF OUR ABILITIES THE RESULT SHOULD BE OKAY, AND THE FINAL CLINICAL PURPOSE SHOULD BE TAKEN INTO ACCOUNT. >> I'LL ASK A QUESTION FOR PARAS, PUTTING THESE THINGS IN PRACTICE CAN BE A CHALLENGE. YOU MENTIONED AUTOMATED -- ONE OF THE CHALLENGES, YOU HAVE TO FIGURE OUT, WELL, THIS IS A CTA CAN CONTRAST, P.E., HOW HAVE YOU THOUGHT ABOUT MAKING SURE THE RIGHT SUBSET OR EXAM GOES? >> I WAS THINKING MORE HIGH LEVEL BUT HOW TO DO IT IS THE CHALLENGE. YOU COULD LOOK AT METADATA IF YOU HAVE METADATA OR EXAM CODES, LIKE WE HAVE GRANULAR EXAM CODES FOR A LOT OF THESE THINGS, ONE WAY TO ACCOMPLISH IT. BUT MAYBE A MORE ROBUST METHOD THAT CAN WORK ACROSS MULTIPLE SITES IS FIRST DEVELOPING WHAT DR. ERICKSON HAD SUGGESTED, MAYBE CNNs TO DETECT IF THERE'S A CONTRAST ENHANCED STUDY AND IF THERE IS, IS IT OPTIMALLY TIME FOR EVALUATING P.E., SO THERE'S SO MANY THINGS THAT NEED TO BE DEVELOPED JUST TO EVEN GET TO THAT WORKFLOW. MAYBE TO SIMPLIFY THINGS IF YOU HAVE AN INFORMATICIST THAT WITH WORK WITH YOUR VENDORS YOU MIGHT HAVE TO HAVE CUSTOM SOLUTIONS AT VENDOR HOSPITALS TO THINK ABOUT THESE THINGS. >> I THINK WE HAVE A GREAT DISCUSSION, AND GIVEN THE TIME LIMITATION WE CAN CONTINUE INTERACTIONS AFTER THE PANEL. THANK YOU. [APPLAUSE] GOOD AFTERNOON, EVERYBODY. AND TIME FOR THE LAST SESSION, APTLY PUT, THE LAST MILE OF THE PROGRAM TODAY. I'M BIBB ALLEN. AGAIN, IT'S BEEN MY PLEASURE TO HAVE BEEN ONE OF THE COORDINATORS FOR THE WORKSHOP. I'M REALLY, REALLY PLEASED WITH HOW IT'S GOING. WE REALLY DO APPRECIATE THE FACT THAT OUR AUDIENCE HAS BEEN ENGAGING, ASKING QUESTIONS, BECAUSE THE ULTIMATE GOAL OF THE PROJECT IS FOR US TO PUT TOGETHER A WHITE PAPER THAT SORT OF SUMS UP ALL OF THIS. SO HAVING THIS ENGAGEMENT FROM EVERYBODY IN THE AUDIENCE IS REALLY, REALLY HELPFUL. IN THIS SESSION, THE LAST MILE, TEEING UP TOMORROW'S FUTURE APPLICATIONS, WE'RE CHANGING A LITTLE BIT OF THE FOCUS FROM THE FOUNDATIONAL RESEARCH, THE OBSTACLES AND CHALLENGES IN THE FOUNDATIONAL RESEARCH SIDE TO MORE WHAT ARE THE CHALLENGES IN MOVING DATA SCIENCE TO CLINICAL PRACTICE. I THINK ALL OF US WHO HAVE BEEN AROUND RESEARCH HAVE THOUGHT ABOUT, YOU KNOW, THE BENCH TO BEDSIDE KIND OF CONCEPT. AND I THINK THIS IS MUCH THE SAME WAY OF HOW WE'RE GOING TO TAKE THESE PRINCIPLES OF DATA SCIENCE AND MOVE THEM INTO THIS. SO IN OUR GROUP TODAY WE HAVE FOLKS FROM THE GOVERNMENT TO TALK TO US ABOUT REGULATORY CHALLENGES, HOW THEY ARE ABLE TO DO THEIR JOB OF ENSURING THAT NEW MEDICAL DEVICES ARE SAFE AND EFFECTIVE FOR THE PUBLIC, HOW WE'RE GOING TO MONITOR THOSE IN CLINICAL PRACTICE, TALK ABOUT REGISTRIES AND TALK ABOUT FROM INDUSTRY'S PERSPECTIVE WHAT DO DEVELOPERS NEED FOR THIS ECOSYSTEM. AGAIN, I DO NOT HAVE PERSONAL CONFLICTS OF INTEREST. I'M THE CHIEF MEDICAL CENTER FOR THE ACR DATA SCIENCE INSTITUTE. SO IF YOU LEARNED FROM TODAY, I THINK WHAT WE ALL SEE IS THAT THE RESEARCH AT A.I. IS PROBABLY AHEAD OF THE PATHWAYS FOR CLINICAL IMPLEMENTATION. A LOT OF THINGS ARE BEING DONE AT SINGLE INSTITUTIONS WITH SINGLE INSTITUTIONS DATA FOR TRAINING, TESTING, VALIDATION, INTEGRATING INTO THEIR OWN CLINICAL WORKFLOWS, AND SO FAR THERE'S REALLY BEEN LIMITED RESEARCH SUPPORTING THE GENERALIZABILITY OF THE OUTPUT OF A.I. ALGORITHMS FROM THE SINGLE INSTITUTIONS INTO WIDESPREAD CLINICAL PRACTICE. SO MY BACKGROUND IS REALLY I'M NOT A RESEARCHER, I'M NOT A DATA SCIENTIST BUT I'M A GUY WHO PRACTICES GENERAL RADIOLOGY IN A COMMUNITY HOSPITAL IN A MEDIUM-SIZED SUBURBAN HOSPITAL IN BIRMINGHAM, ALABAMA. SO MY THOUGHTS FROM THE COLLEGE PERSPECTIVE IS THAT I THINK I REPRESENT A LARGE SEGMENT OF THE RADIOLOGIST POPULATION OUT THERE AND HOW ARE WE GOING TO USE THESE TOOLS IN OUR PRACTICES. SO, SOME OF THE REQUISITES WE THINK ARE IMPORTANT IS THAT THE A.I. USE CASES NEED TO BE VALUABLE FOR PATIENTS AND PHYSICIANS. A LOT OF TIMES LEFT TO THEIR OWN DEVICES, PEOPLE WILL GO OUT AND SAY, GEE, I CAN DO THIS AS GOOD AS A RADIOLOGIST CAN, AND SO, OH GREAT. BUT I'M NOT SURE THAT'S SOMETHING THAT'S ALWAYS NECESSARILY USEFUL FOR RADIOLOGISTS BECAUSE, FOR INSTANCE, FRACTURE DETECTION AND THOSE SORTS OF THINGS, WE'RE PRETTY GOOD AT DOING THOSE THINGS. AND SO REALLY WHAT ARE THE USE CASES FOR A.I. WHERE THEY WILL HELP US DO SOMETHING WE ALREADY DO BETTER, OR HELP US DO THINGS THAT WE CAN'T ALREADY DO NOW IN THE FUTURE. AND HOW DO WE PROMOTE INTEROPERABILITY AND COMMON PATHWAYS FOR IMPLEMENTATION IN WAYS THAT REDUCE UNINTENDED BIAS? WE WANT TO CONSIDER PATIENT SAFETY AND HOW DO WE VALIDATE ALGORITHMS BEFORE THEY ARE DEPLOYED BUT UNDERSTANDING THAT THERE ARE HUGE VARIANCES IN THE WAY RADIOLOGISTS PRACTICE, HOW ARE WE ABLE TO MONITOR THEM IN CLINICAL PRACTICE SO WE CAN KNOW THE INSTANCES WHERE AN ALGORITHM MAY BREAK IN CLINICAL PRACTICE WHERE IT DIDN'T BREAK DURING DEVELOPMENT OR IN VALIDATION. AGAIN, WE TALKED AT THE BEGINNING ABOUT TRANSPARENCY, AND THEN BUSINESS MODELS. WHEN I SAY BUSINESS MODELS THAT PROMOTE SOCIAL JUSTICE, ONE OF THE THINGS I FEAR IS IF WE DON'T INTEGRATE A.I. INTO THE WORKFLOW OF STANDARDIZED WAYS THAT WE PRACTICE, STANDARDIZED WAYS THAT CMS AND OTHER PAYERS CAN REIMBURSE A.I. AND THAT SORT OF THING WE'RE GOING TO END UP WITH SORT OF HAVES AND HAVE NOT, THE BIG INSTITUTIONS WILL HAVE THESE TOOLS AND SMALLER HOSPITALS WON'T HAVE THEM AVAILABLE SO PATIENTS WILL SUFFER BY NOT HAVING IT AVAILABLE. SO, AGAIN JUST LIKE WE SAW TODAY, AT THE MACHINE LEARNING SHOWCASE AT RSA 2017 HEALTHCARE AI COMPANIES HAVE BEEN ABLE TO DEMONSTRATE ABILITY TO A.I. TO BE TRANSFORMATIVE FOR OUR SPECIALTY. KEITH SHOWED THIS TOP FIVE LIST. LIST OF THINGS. ONE THING WE'LL FOCUS ON IN THIS SESSION SOME OF THESE WAYS OF HOW TO LOOK AT DATA, HOW TO GET CLINICAL INTEGRATION ACCOMPLISHED AND TALK A LITTLE BIT ABOUT CLINICALLY EFFECTIVE USE CASES AND HOW WE CAN DEFINE THOSE IN WAYS TO HELP US. ONE OF THE PROJECTS THAT I THINK EVERYONE SHOULD BE AWARE OF IS SORT OF A COMBINED EFFORT BETWEEN THE ACR AND DATA SCIENCE INSTITUTE AND RSNA FOR COMMON DATA ELEMENTS CALLED RADELEMENT.ORG FOR USE CASES IN ALGORITHM OUTPUT, OPPORTUNITY FOR STANDARDIZATION OF THE WAYS WE DO ANNOTATION TRAINING AND TESTING AND VALIDATION OF ALGORITHMS, AND STANDARDIZED WAYS FOR CLINICALLY INTEGRATING A.I. MODELS INTO THE PACS, TRANSCRIPTION INTO EHRs, CREATING STANDARD APIs. TOMORROW WE'LL HAVE MORE ABOUT COMMON DATA ELEMENTS. BUT THAT'S IMPORTANT BECAUSE AS WE DEVELOP A CLINICAL USE CASE, THE IDEA IS TO TAKE A NARRATIVE DESCRIPTION OF AN IDEA THAT WILL SOLVE A CLINICAL PROBLEM FOR RADIOLOGY INTO MACHINE READABLE LANGUAGE THAT CAN BE USED BY A.I. DEVELOPERS, SO IN THIS INSTANCE FOR COLON POLYP DETECTION WOULD INCLUDE ELEMENTS FOR VALIDATION OF ALGORITHMS, INTEGRATION INTO CLINICAL WORKFLOW AND POPULATING REGISTRIES FOR USE TO MONITOR ALGORITHM PERFORMANCE IN THE REAL WORLD AND PROVIDE FEEDBACK FOR THE DEVELOPERS AND FOR THE FDA AND OTHER GOVERNMENTAL AGENCIES. SO WHEN YOU LOOK AT THIS SORT OF POTENTIAL A.I. ECOSYSTEM, IT ALL BEGINS WITH THE USE CASES. ECOSYSTEM BEING RADIOLOGY PROFESSIONALS, RESEARCHERS, INDUSTRY DEVELOPERS, GOVERNMENT AGENCIES, EVEN PATIENTS TO WORK IN THIS. I WANT TO MENTION ONE THING ABOUT PATIENTS AND GOING BACK TO DAVE'S TALK ABOUT PATIENT-CENTRIC DATA AND USING PATIENT DATA AND PATIENTS IN CONTROL OF THEIR DATA. ONE OF THE THINGS THAT CAME OUT OF ONE OF THE JASON PROJECTS LOOKING AT WHERE THIS BREAKS DOWN AND PEOPLE GET ENERGIZED, LET'S SAY, ABOUT THEIR DATA BEING USED WITHOUT THEIR KNOWLEDGE OR THAT SORT OF THING, IS IT'S IMPORTANT. WHEN THEY LEARN THAT, THAT SOMETHING'S BEEN USED WITHOUT THEIR KNOWLEDGE, IT'S A LOT WORSE THAN IF WE FIGURED OUT A WAY UP FRONT TO SAY, HEY, WOULD YOU ALLOW US TO USE YOUR DATA TO ADVANCE MEDICAL KNOWLEDGE? AND THEN ALL OF A SUDDEN PATIENTS ARE ENGAGED AND PLEASED TO DO IT. SO I THINK AS WE GO THROUGH THIS, FINDING WAYS TO EMPOWER PATIENTS TO HELP US BY USING THEIR DATA ARE IMPORTANT AND WILL ALLEVIATE ALL OF SOME OF THE PROBLEMS THAT WE'VE SEEN ABOUT MISUSE OF DATA. OUR VALUE PROPOSITION IS TO DEVELOP TRUSTED PARTNERSHIPS WITH INDUSTRY AND REGULATORS, ENSURE PATIENT SAFETY, AND MINIMIZE DISPARITIES. INCREASED RADIOLOGY PROFESSIONALS, RADIOLOGISTS VALUE IN OUR HEALTH CARE SYSTEM. IF WE CAN USE STANDARDIZED SPECIFICATIONS IN USE CASES FOR DATASET TRAINING, MULTIPLE SITES CAN CREATE DATASETS AROUND THESE SPECIFIC USE CASES. AND WHAT THAT DOES IS DEVELOPRS IF ACCESS THEY CAN ENSURE DIVERSITY HELPS PREVENT UNINTENDED BIAS, MAKE SURE ALGORITHMS WORK ACROSS A BROAD SELECTION OF VENDORS AND TECHNICAL PARAMETERS AND SO FORTH. ALLOWS MORE INDIVIDUALS AND INSTITUTIONS TO PARTICIPATE IN A.I. DEVELOPMENT SO EVEN A PRACTICE LIKE MINE COULD, IF I HAD A SPECIFIED USE CASE AND SPECIFIED THING TO DO, COULD PROVIDE SOME DATA, YOU KNOW, FROM OUR CASES TO THIS WAY. AS FAR AS ALGORITHM VALIDATION GOES, I THINK ONE OF THE THINGS THAT WE NEED TO SEE IS THAT THE DATASETS ARE NOVEL. AND THAT IS NOT TO SAY THAT WHEN A SITE WITHHOLDS DATA THAT THEY DIDN'T USE IN TRAINING AND TESTING TO EVENTUALLY USE FOR ALGORITHM VALIDATION, IT'S STILL IN A LOT OF WAYS THE SAME DATA, THE SAME SCANNER, SAME PARAMETERS, ALL THOSE THINGS. IF WE COULD FIGURE OUT A WAY TO USE MULTIPLE INSTITUTION DATA TO ENSURE GEOGRAPHIC TECHNICAL AND PATIENT DIVERSITY AND VALIDATION SET, WE WOULD HAVE A VERY GOOD WAY OF ENSURING THAT THAT ALGORITHM WAS LIKELY VALID AND READY FOR CLINICAL DEPLOYMENT. AND NICK LATER FROM THE FDA IS GOING TO TALK ABOUT ALL THE DIFFERENT WAYS THE FDA IS LOOKING AT SORT OF STREAMLINING SOME OF THESE PROCESSES IN THE BEGINNING. SPECIFICATIONS FOR CLINICAL INTEGRATION, HOW IS THE ALGORITHM OUTPUT GOING TO GET BACK INTEGRATED, WE TALKED A LITTLE BIT ABOUT EARLIER THE ABILITY TO HAVE A STRUCTURED REPORT DESIGNED TO CAPTURE SOME OF THE OUTPUT OF ALGORITHM OUTPUTS INTO CLINICAL WORKFLOW, AGAIN ALL THIS NEEDS TO BE DONE IN A TOTALLY VENDOR NEUTRAL ENVIRONMENT SO THAT ALL OF PACS SYSTEMS, TRANSCRIPTION SYSTEMS CAN INGEST THIS MATERIAL EQUALLY. AND FINALLY FOR MONITORING ALGORITHM PERFORMANCE AND CLINICAL PRACTICE, LOOKING AT RADIOLOGIST INPUT AND METADATA ABOUT THE EXAM CAN BE CAPTURED IN THE BACKGROUND. THAT IS, IF I'M DICTATING A CASE, AND REPORTING SOFTWARE SUCH AS POWER SCRIBE, AND POWER SCRIBE OFFERS UP AS PARTS OF THE REPORTING TEMPLATE WHAT AN A.I. ALGORITHM SAID ABOUT THE CASE, THEN I AS A RADIOLOGIST CAN EITHER AGREE OR DISAGREE, AND IF I -- THAT DATA THEN CAN BE CAPTURED WITHIN A REGISTRY, AND IN THE BACKGROUND THE REGISTRY CAN BE CAPTURED METADATA ABOUT THE EXAMINATION AND AGAIN THE USE CASES WOULD DEFINE WHAT ARE THE PARAMETERS THAT WE WANT TO LOOK AT TO SEE WHERE AN ALGORITHM MIGHT SYSTEMATICALLY BREAK, AND THIS IS A TOOL THAT CAN BE USED IN PRACTICE CAPTURING THESE DATA IN THE BACKGROUND. SO BEING ABLE TO ASSESS REPORTS, AGAIN, PROVIDE METRICS BACK TO THE DEVELOPERS FOR UPDATING THE ALGORITHMS AND EVEN REPORTS BACK TO GOVERNMENTAL AGENCIES IN THE POST-MARKET SURVEILLANCE PROCESS OF THESE TOOLS. AND THIS IDEA CAME TO US BY GREG NIST GROUP AND FDA ABOUT HOW WE COULD SHORTEN THE TIME FOR PRE-MARKET REVIEW BY INCREASING THE VERACITY OF POST-MARKET SURVEILLANCE, AND SO WE'LL TALK A LITTLE BIT ABOUT SOME OF THOSE TOOLS THAT ARE AVAILABLE. AND THEN FINALLY, JUST IN SUMMARY, HOW IS A.I. GOING TO BE USED IN CLINICAL PRACTICE? SO IF YOU GET THE IDEA THAT THE OUTPUT OF AN A.I. ALGORITHM LUNG CANCER SCREENING WILL LOOK AT VISUALIZATION AND REPORTER INTERFACE, MAY BE IN THE CLOUD, ON PREM, MODALITY PACS OR TRANSCRIPTION, A CLINICAL USE CASES WILL BE THE SUM OF MULTIPLE USE CASES, AN ALGORITHM WILL HAVE TO DETECT AND LOCALIZE, QUANTIFY AND CHARACTERIZE, AND THEN FEED ALL THAT DATA BACK INTO A REPORT. NOW AN A.I. ALGORITHM COULD JUST GO FROM THE CASE AND SAY THIS IS A 85% CHANCE OF LUNG CANCER BUT TO ME WITHOUT USE CASES FOR DETECTING AND LOCALIZING AND QUANTIFYING AND CHARACTERIZING THAT THE ALGORITHM WILL BE NOT AS USEFUL CLINICALLY BECAUSE AS RADIOLOGISTS WE'RE GOING TO HAVE TO REPORT ALL THIS INFORMATION BACK TO THE CLINICAL TEAM FOR SURGERY OR WHATEVER. AFTER ITS CLASSIFIED AS LUNG RADS 3, DATA THEN GOING INTO OUR REPORT AND THAT RECORD IS NOT ONLY INGESTED BY THE ELECTRONIC HEALTH RECORD. SO IN QUICK SUMMARY, AGAIN, THIS WHOLE ECOSYSTEM FROM USE CASE TO SPECS FOR TRAINING DATASETS, FOR NOVEL VALIDATION SETS, INTEGRATION INTO CLINICAL PRACTICE, MONITORING THE PERFORMANCE AFTER ITS OUT IN PRACTICE, IS WHAT WE HAVE BEEN THINKING ABOUT THE MOST AT THE ACR'S DATA SCIENCE INSTITUTE, TO SOME DEGREE SOME SPEAKERS WILL TALK ABOUT THAT. SO WITHOUT FURTHER ADO I'D LIKE TO INTRODUCE NICK PETRICK FROM THE FDA TO TALK A LITTLE BIT ABOUT BOTH THE PRE-MARKET CLEARANCE PROCESS AND THEN SOME TOOLS FOR POST-MARKET SURVEILLANCE. SO THANK YOU VERY MUCH. [APPLAUSE] >> SO, I WILL TALK ABOUT THE REVIEW AND EVALUATION OF A.I. ALGORITHM FOR CLINICAL PRACTICE AND PUT THIS IN THE CONTEXT OF WHAT WE'RE DOING CURRENTLY FOR A.I. TOOLS THAT ARE COMING TO THE AGENCY NOW AND THEN I'LL HAVE A SECOND TALK FOR ONE OF THE OTHER COLLEAGUES THAT WASN'T ABLE TO BE HERE AND TALK ABOUT SORT OF WHERE WE MIGHT BE GOING WITH A.I. SO FIRST, I'LL GIVE AN OVERVIEW OF CLINICAL ASSESSMENT, AND I'LL CONCENTRATE ON IMAGING AND A.I. DEVICES. AGAIN I'LL TALK ABOUT MOSTLY RADIOLOGICAL DEVICES AND INFORMATION THAT COMES FROM THAT AREA. SO FIRST I WANTED TO GIVE A BACKGROUND ON WHAT WE HAVE AVAILABLE. THE AGENCY HAS A NUMBER OF GUIDANCES AND SPECIAL CONTROLS AVAILABLE FOR PEOPLE DEVELOPING A.I. THAT WOULD COME INTO THE AGENCY FOR REVIEW. THE FIRST ONE THIS CAME OUT IN 2012, A PAIR OF GUIDANCES THAT'S COMPUTER AIDED DETECTION, A GUIDANCE THAT LAYS OUT FRAMEWORK OF WHAT WE LIKE TO SEE IN SUBMISSION FOR A CLASS 2 OR 510(K) SUBMISSION, AND CLINICAL PERFORMANCE ASSESSMENT, HOW A STUDY WOULD ASSESS. THESE ARE AVAILABLE, UTILIZED BY A NUMBER OF COMPANIES. THAT BEING SAID, THERE'S BEEN MORE RECENT SUBMISSIONS AND DEVICES APPROVED FOR COMPUTER AIDED DIAGNOSIS, PARTICULARLY THIS IS A CADX FOR BREAST LESION IN MR IMAGING, AND THESE WOULD THEN BE A DIAGNOSTIC TOOL THAT TELL WHETHER SOMETHING IS BENIGN OR MALIGNANT POTENTIALLY, IN THE PROCESS OF APPROVING THIS DEVICE OR GETTING IT TO MARKET WE DEVELOPED SPECIAL CONTROLS AND PARALLELLING WHAT WE SEE FOR CADX DEVICES FOR ASSESSING COMPUTER AIDED DIAGNOSTIC TOOL CLAIMS. THERE'S ANOTHER DEVICE THAT CAME OUT THIS YEAR, OSTEOTECH DEVICE, COMPUTER AID DETECTION AND DIAGNOSIS FOR WRIST FRACTURES, NOVEL DEVICE THAT CAME TO MARKET. WE'VE DEVELOPED SPECIAL CONTROLS FOR THAT. CONTROLS PARALLEL WHAT WE'RE ASKING FOR FOR COMPUTER AIDED& DETECTION, I'LL GO THROUGH THAT IN A GENERAL OVERVIEW. TRIAGE DEVICE, TO AID PRIORITIZING TRIAGING RADIOLOGIC IMAGES, WE SAW EXAMPLES EARLIER TODAY, THIS CAME ON THE MARKET THIS YEAR, NOTIFY OF LARGE VESSEL OCCLUSION IN CT ANGIOGRAPHY, SPECIAL CONTROLS IN A FRAMEWORK FOR HOW WE'RE GOING TO REVIEW THIS ACKNOWLEDGE DEVICE CLAIMS. YOU CAN SEE THERE'S A SYNERGY BETWEEN THE CAD DEVICES AND TRIAGE DEVICES, A LOT OF ALGORITHM UNDERNEATH ARE THE SAME BUT IMPLEMENTATION UTILIZATION IS DIFFERENT. FINALLY I'LL TALK ABOUT THIS, WE HAVE A NEW DEVICE REALLY COMPUTER AID DETECTION FOR RETINAL DIAGNOSTIC SOFTWARE TOOL, SPECIAL CONTROLS DEVELOPED FOR THIS IMPLICATION, IMPLEMENTATION, THAT ARE ON THE MARKET FOR AN OPTICAL RETINAL IMAGING APPROACH FOR DIABETIC RETINOPATHY. JUST SOME BASICS FOR THE ASSESSMENT FRAMEWORK, I'LL TALK ABOUT THIS IN THE DEVICE DESCRIPTION, GIVE AN IDEA WHAT IT IS AND CLINICAL ASSESSMENT CONSISTS OF STAND-ALONE PERFORMANCE ASSESSMENT AND READER PERFORMANCE WITH OTHER FACTORS ASSOCIATED WITH SOFTWARE AND QUALITY OF SOFTWARE AND DEVELOPMENT THAT COMES IN THAT I WON'T TALK ABOUT HERE BUT IS PART OF ANY SUBMISSION THROUGH THE AGENCIES. TALKING ABOUT DEVICE DESCRIPTION IT'S NOT JUST TO SAY OUR DEVICE DOES X OR Y BUT IT'S SPECIFIC UTILIZATION OF WHAT'S THE MODE OF OPERATION, WHAT'S THE PATIENT POPULATION, SPECIFIC INDICATION, THIS INCORPORATION STRUCTURE FOR TRADITIONAL ALGORITHMS, TRADITIONAL MACHINE LEARNING OR TO DEEP NETWORKS, STRUCTURE OF THAT NETWORK. WE WANT TO UNDERSTAND THAT FIRST BECAUSE THERE MAY BE IMPLICATIONS FOR DIFFERENT STRUCTURES IMPORTANT TO UNDERSTAND THAT MAY HAVE DIFFERENT REGULATORY IMPLICATIONS, SECOND A BENCHMARK FOR OTHER CHANGES TO THE ALGORITHM TO UNDERSTAND HOW LARGE CHANGES ARE AND WHEN THEY MIGHT NEED NEW DATA TO COME BACK IN. WE'RE ALSO INTERESTED OBVIOUSLY IN INPUT AND OUTPUT, WHAT IS THE TYPE OF INPUT, NOT JUST INTERESTED IN WHERE IT'S A CT BUT LIMITATIONS OF CT, IN THE SLICE THICKNESS, RECONSTRUCTIONS, SOME DOSE RESTRICTIONS, RESTRICTIONS OF CERTAIN MANUFACTURERS THAT GO WITH THAT. WE'RE INTERESTED OBVIOUSLY IN TRAINING PROCESS AND HOW THE ALGORITHM IS DEVELOPED. WE'RE ALSO INTERESTED IN DETAILS OF THE TRAINING AND TEST DATASET. LOOKING AT GENERALIZABILITY, HOW ROBUST AND DIRECT EVALUATION OF THE SYSTEM. THERE ARE OTHER FACTORS, REFERENCE STANDARDS, HOW THINGS ARE SCORED AND OTHER FACTORS. THE CLINICAL PERFORMANCE TESTING IS BROKEN UP INTO TWO PARTS, STAND-ALONE PERFORMANCE, OBVIOUSLY THE ALGORITHM PERFORMANCE BY ITSELF, INTRINSIC FUNCTIONALLY, APPLICATION OF A.I. MACHINE LEARNING AND SOME SORT OF SCORING APPLIED AND STATISTICAL ANALYSIS ASSOCIATED WITH THAT. THAT REALLY IS SOMETIMES A BENCHMARK, JUST A DEVICE BY ITSELF, NOT NECESSARILY COMPARED TO OTHER DEVICES OR DATABASES, DATA UTILIZED MAY BE DIFFERENT, AN OPPORTUNITY TO LOOK AT PERFORMANCE ON LARGER DATAET IS, COHORTS OF PATIENTS OR DIFFERENT IMAGES, THAT MAY BE IMPORTANT IN DETERMINING WHETHER THIS DEVICE IS APPROPRIATE FOR EVERY APPLICATION, THERE MAY BE LIMITATIONS ASSOCIATED WITH IT. FOR THOSE DEVICES THAT ARE AIDS, CLINICAL PERFORMANCE ASSESSMENT, OF CLINICAL PERFORMANCE UTILIZING THE DEVICE, THERE ARE MANY POSSIBLE DESIGNS, MOSTLY RETROSPECTIVE BASED ON MULTI-READER MULTI-CASE DESIGN, MAY READ ALL THE CASES, MAY BE SUBSETS. OF BASIC FLOW, ESTABLISHMENT OF GROUND TRUTH, THE CLINICAL PERFORMANCE WITHOUT THE AID OR NORMAL CLINICAL WORKFLOW, THEN A.I. TOOLS APPLIED IN CLINICAL ASSESSMENT WITH CLINICIAN, COMPARISON BETWEEN PERFORMANCE WITH AND WITHOUT THE AID TO DETERMINE HOW WELL THAT DEVICE IS SUPPORTING CLINICAL DECISION MAKING. THERE'S AN ORGANIZATION CALLED INTERNATIONAL MEDICAL DEVICE REGULATORS FORUM, VOLUNTARY GROUP, THERE'S A WORK GROUP SPECIFICALLY RELATED TO SOFTWARE AS A MEDICAL DEVICE, CMD, WITH OUTPUTS FROM THIS DEVICE, ONE IS THIS CLINICAL EVALUATION UNDERLINED THEY BOTTOM. THIS IS PARTICULARLY IMPORTANT OUTPUT OF THAT GROUP BECAUSE IT'S GOING TO ADOPT FDA GUIDANCE IN LATE 2017. TO CONSIDER GUIDANCE IN DEVELOPING FOR TECHNOLOGY, AN IMPORTANT CONSIDERATION. I JUST WANT TO LAY OUT THE THREE BASIC PILLARS WHICH MATCH WHAT WE'RE DOING ALREADY, WHICH IS DEVELOPING CLINICAL ASSOCIATION, IS THE SOFTWARE OUTPUT RELATED TO CLINICAL CONDITION, CAN YOU ESTABLISH THAT? A LOT OF TIMES THE EVIDENCE GENERATION IS LITERATURE OR PROFESSIONAL SOCIETY GUIDE LINES, SOME CASES MAY REQUIRE SOME SECONDARY DATA ANALYSIS OR CLINICAL TRIALS. THE SECOND COMPONENT IS ANALYTICAL VALIDATION DONE FOR ALL SOFTWARE TOOLS BUT CERTAINLY FOR SOFTWARE AS A MEDICAL DEVICE, A.I. AND MACHINE LEARNING ALGORITHMS. DOES THE SOFTWARE MEET TECHNICAL REQUIREMENTS, DOES IT CONFORM TO SOFTWARE -- IS IT CONFIRMED THE SOFTWARE IS CORRECTLY CONSTRUCTED, MEET SPECIFICATIONS AND MEET THE NEEDS OF THE USER AND INTENDED POPULATION? AGAIN, ANALYTICAL VARIATION IS TYPICALLY ASSOCIATED WITH ALL SOFTWARE, ESPECIALLY FOR MEDICAL DEVICE. FINALLY A CLINICAL VALIDATION, DOES THE EVIDENCE SHOW IT'S BEEN TESTED IN THE CORRECT TARGET POPULATION FOR INTENDED USE AND CAN ACHIEVE CLINICAL MEANINGFUL OUTCOME, AGAIN THAT MATCHES WHAT WE'RE DOING ALREADY AND OUR ASSESSMENT FOR SOFTWARE. THERE'S A RISK-BASED APPROACH THAT'S ASSOCIATED WITH THIS, IT'S VERY COMPLICATED, BUT JUST THE LEVEL OF CLINICAL EVALUATION, IMPORTANCE OF INDEPENDENCE REVIEW SHOULD BE COMMENCE RATE WITH RISK ASSOCIATED WITH DEVICE. SO THAT'S AN OPPORTUNITY FOR US TO ADJUST HOW WE'RE APPROACHING SOME OF OUR A.I. AND MACHINE LEARNING EVALUATION. SO THE ANALYTICAL VALIDATION DEMONSTRATES SOFTWARE FEATURES AND FUNCTIONS MEET CONDITIONS, GENERALLY HAS COMPONENTS OF IMAGING A.I. ASSESSMENT PARADIGM FOR CAD AND CADX. THE A.I. HAS HUGE POTENTIAL, WE SEE THIS IN DIAGNOSTIC AND DETECTION TOOLS THAT WE'VE SEEN ON THE MARKET, WE'RE STARTING TO SEE TRIAGE AND WORKFLOW TOOLS ON THE MARKET AS WELL. WE HAVE A REGULATORY FRAMEWORK AND SUBSTANTIAL GUIDANCE AND SUPPORT FOR INDUSTRY AND DEVELOPERS, WITH CAD-E IN 1998, BUT TRYING TO MODIFY THAT BASED ON WHAT WE'LL SEE AT A.I. AS IT EVOLVES, TRYING TO ADJUST BASED ON LEVEL OF CLINICAL EVALUATION AND EXTENT OF INDEPENDENT REVIEW BASED ON RISK-BASED MODEL. MAIN COMPONENTS ARE CLINICAL ASSOCIATION, ANALYTICAL VALIDATION AND CLINICAL VALIDATION. THANK YOU. [APPLAUSE] >> NOW I'LL GIVE A TALK FOR COLLEAGUES THAT COULDN'T ATTEND, EXTENDING HOW WE'RE GOING TO EXTRAPOLATE MOVING FORWARD INTO THE FUTURE. AGAIN, THESE ARE FROM MY COLLEAGUES AT FDA WHO WEREN'T ABLE TO ATTEND TODAY, I'LL HOPEFULLY DO THIS JUSTICE. THIS IS TALKING ABOUT THE USE OF REAL WORLD EVIDENCE, AGAIN SOME TOPICS WE'VE DISCUSSED TODAY HOW TO DEVELOP THAT DATA AND USE IT IN A REGULATORY FRAMEWORK. I'LL TALK ABOUT THE CONTENT AND DRIVERS FOR REAL WORLD EVIDENCE AND TRY TO GIVE EXAMPLES FOR THE RELEVANCE WITHIN CDRH AND TALK ABOUT THE SOFTWARE AS A MEDICAL DEVICE AND PRE-SOFTWARE MEDICAL DEVICE PRE-CERTIFICATION PROGRAM. IT'S PART OF OUR REGULATIONS SO THE AGENCY IS COMMITTED TO USE REAL WORLD EVIDENCE, DEVELOPED A NATIONAL EVALUATION SYSTEM FOR HEALTHCARE TECHNOLOGIES, THE NEST PROGRAM. THE NEST PROGRAM MISSION TO ACCELERATE DEVELOPMENT AND TRANSLATION OF NEW AND SAFE HEALTH TECHNOLOGIES, LEVERAGING REAL WORLD EVIDENCE. THIS IS REALLY -- THE ECOSYSTEM INCORPORATION CLINICAL GROUPS, PAYERS, REGULATORS, HEALTH SYSTEMS, PATIENT GROUPS AND INDUSTRY, SO THIS IS A LARGE ENVELOPE OF PARTICIPANTS IN THE NEST PROGRAM. AND THIS IS A VERY BUSY SLIDE BUT I WANT TO HIGHLIGHT A COUPLE THINGS. THE NEST HAS SELECTED TEAMS FOR VALUE INITIATIVES, 11 TEAMS SELECTED SO FAR, TWO ASSOCIATED WITH SURVEILLANCE, SIX WITH POST-MARKET FOR REAL WORLD EVIDENCE, THREE WITH PRE-MARKET USE OF REAL WORLD EVIDENCE. WE WANT TO INCORPORATE REAL WORLD EVIDENCE INTO THE WHOLE ECOSYSTEM OF DEVICE REGULATION AND LIFE CYCLE OF THE DEVICE. AGAIN, THE GOVERNANCE COMMITTEE IS BROAD, MULTI-ORGANIZATIONAL AND WE CAN SEE INPUT FROM, SAY, BUSINESS PLAN COMES FROM 14 ORGANIZATIONS ASSOCIATED WITH DATA COLLECTORS, INDUSTRY AND DEVELOPERS OF TOOLS AS WELL AS OTHER STAKEHOLDERS. SO LET'S LOOK AT THE EVALUATION PIPELINE. THIS IS WHAT I LAID OUT IN THE FIRST EXAMPLE. NOW THERE'S SORT OF THIS PRE-CLINICAL TESTING THAT HELPS GIVE YOU PRELIMINARY DATA FOR DEVICE. THAT MOVES TO SOME SORT OF CLINICAL TYPES OF STUDY OR ASSESSMENT OF THAT DEVICE, THAT IS SUBMITTED TO THE AGENCY IN PRE-MARKET APPLICATION, DEVICE HOPEFULLY PROVED, -- APPROVED AND GETS ON THE MARKET, IF THERE ARE RISKS WE DIDN'T SEE ORIGINALLY. WE WANT TO TALK ABOUT SHIFTING THE PARADIGM INCORPORATING REAL WORLD EVIDENCE. WE KNOW REAL WORLD EVIDENCE COMES OUT IN INFORMED CLINICAL DECISION MAKING HOW TOOLS OR DEVICES ARE UTILIZED, UTILIZING THIS PIPEWAY TO LOOK AT DEVICES THAT COME ON THE MARKET AND READAPT AND MORE IMPORTANTLY EVEN IF UTILIZE THOSE AND USING THEM IN PRE-MARKET DATA TO SUPPORT A SUBMISSION OR POTENTIALLY THE POST-MARKET ASPECTS AS WELL. IN ORDER TO CHARACTERIZE REAL WORLD EVIDENCE THERE'S CONCEPT OF RELEVANCE, AND SOME EXAMPLES OF FACTORS EVALUATED, IS ASSESSMENT SCHEDULE CAPTURING ENDPOINTS OF INTEREST, POPULATION APPROPRIATE AND REPRESENTATIVE, AND ANALYSIS PLAN APPROPRIATE TO DEVELOP THE QUESTION OF INTEREST. ANOTHER PART IS RELIABILITY. SO THIS REALLY GETS THE OVERALL DATA QUALITY, WHAT IS THE QUALITY OF THE DATA BEING ACQUIRED. AND THAT'S ASSOCIATED WITH CHARACTERISTICS ASSOCIATED WITH DATA ACCRUAL PROCESS, WHAT IS THE PROCESS USED FOR ACCRUAL OF DATA, WHAT IS DATA ASSURANCE AND QUALITY CONTROL WHEN YOU REQUIRE THAT DATA. SO THAT'S 20,000-FOOT PICTURE OF REAL WORLD EVIDENCE. ONE AREA THAT CAN BECOME IMPORTANT, THE SOFTWARE PRE-CERTIFICATION PROGRAM, A NEW APPROACH TO HOW WE EVALUAT SOFTWARE AS A MEDICAL DEVICE, INCORPORATION A.I. AND MACHINE LEARNING FOR IMAGING PARTICULARLY. THIS IS REALLY A PROGRAM THAT'S GOING TO TRY TO NOT JUST LOOK AT THE DEVICE BY ITSELF BUT THE ORGANIZATION AND TRY TO ASSESS THE ORGANIZATION TO ESTABLISH TRUST THAT THEY HAVE A CULTURE OF QUALITY AND ORGANIZATIONAL EXCELLENCE, SUCH THAT THEY CAN DEVELOP HIGH QUALITY SOFTWARE AS A MEDICAL DEVICE, SHIFTING TO LOOK AT NOT JUST DEVICE ALONE BUT THE ORGANIZATION HOPING THIS WILL STREAMLINE THE REGULATORY PROCESS. FDA RECOGNIZES INCREASE IN DIGITIZATION AND THIS WILL INCORPORATE A LARGE POTENTIAL INCREASE IN NUMBER OF SUBMISSIONS IN THIS AREA AND PARADIGM SHIFT IN HOW WE MIGHT APPROACH THIS. OUR CURRENT REGULATORY PARADIGM IS WE TEND TO HAVE A LENGTHY PRE-MARKET DEVELOPMENT PROCESS WHERE THE PRODUCT COMES IN, UNTIL THEY COME WITH A NEW SUBMISSION OR MODIFICATION. GENERALLY A PASSIVE SURVEILLANCE WITH STABLE NUMBER OF SUBMISSIONS INTO THIS AGENCY, ROUGHLY 3500 SUBMISSIONS, YOU CAN SEE THE NUMBER OF PRE-SUBMISSIONS. WITH THE ADVENT OF DIG -- DIGITAL HEALTH PARADIGM, FREQUENT ITERATIONS, THIS CHANGES THE FRAMEWORK FOR REVIEWING NUMBERS, SOFTWARE MAY ANALYZE FROM ALL USERS, THERE'S A LARGE DATA COHORT THAT CAN BECOME AVAILABLE, AN OPPORTUNITY TO UTILIZE IN THE DEVELOPMENT PHASE AND ASSESSMENT AND LONG-TERM EVALUATION PHASE, THERE'S A POTENTIAL FOR INCREASE IN VOLUME OF SUBMISSION. LOOK AT RISK-BASED FRAMEWORK, RISK WITH A PARTICULAR SOFTWARE TOOL, THAT'S ASSESSED AS WELL AS PRE-CERTIFICATION LEVEL, WHAT'S THE QUALITY OF THE ORGANIZATION AND BASED ON THAT THERE MAY BE DETERMINATIONS FOR LOW RISK SOFTWARE THE ORGANIZATION COULD HANDLE THE EVALUATION BY THEMSELVES, NO NEED FOR FORMAL SUBMISSION, NOT IN THE SAME EXTENT AS BEFORE. FOR THOSE AT HIGHER RISK OR LOWER PRE-CERTIFICATION LEVEL THERE MAY BE A STREAMLINED REVIEW PROCESS THAT ALLOWS US TO ASSESS SAFETY AND EFFECTIVENESS AT PRE-MARKET STREAMWORK. THAT NEEDS TO BE DATA TO ASSESS AND SUPPORT PRE-MARKET SUBMISSIONS FROM DATA THAT COMES IN, THAT'S IN THE FRAMEWORK USED BEFORE, WHETHER DATA TO THE AGENCY TO REVIEW OR USED BY THE ORGANIZATION BUT ALSO TO ASSESS QUALITY OF PRE-CERTIFICATION PROCESS, HOW ORGANIZATIONS ARE OPERATING, WHETHER IT'S WORKING IN THE PRACTICAL SENSE. THE FOUNDATIONS FOR EXCELLENCE IN THE PRE-CERTIFICATION ARE PATIENT SAFETY, QUALITY, RESPONSIBILITY, CYBER SECURITY RESPONSIBILITY AND PRO-ACTIVE CULTURE. THIS IS THE DAYTIME LINE. COMING TO THE END OF 2018, WHERE THEY WERE TRYING TO LAUNCH PRE-CERT DEVELOPMENT. I'LL END QUICKLY BY GOING THROUGH OPPORTUNITIES FOR A.I. COMMUNITY AND TALK ABOUT PARTICULAR PARTNERSHIPS IN THE NEST PROGRAM, ONE IS MEDICAL DEVICE EPIDEMIOLOGY NETWORK, A GLOBAL PUBLIC/PRIVATE PARTNERSHIP AND INCORPORATION OVER 130 PARTNERS, OVER 100 NATIONAL AND REGIONAL REGISTRIES FROM 37 COUNTRIES. IT HAS ACCESS TO 30 MILLION PATIENT DEVICE ENCOUNTERS IN THE REGISTRIES, HUNDREDS OF MILLIONS OF CLAIM RECORDS, AN OPPORTUNITY TO UTILIZE EVIDENCE IN REGULATORY DECISION MAKING. GLOBAL NETWORK FROM ACROSS THE WORLD. AND THE COMMUNITIES OF PRACTICE ARE BEING LAUNCHED TO CREATE A LEARNING COMMUNITY WHERE COORDINATED REGISTRY NETWORKS CAN SHARE LESSONS LEARNED AND BEST PRACTICES AND FACILITATE CONTRIBUTING REAL WORLD EVIDENCE TO REGULATORY DECISION MAKING. PART IS TO HELP THESE CRNs MATURE TO SHARE LESSONS LEARNED, DEVELOPED MATURITY MODELS, CORE DATASETS AND INFORMATICS UNDERPINNINGS TO FACILITATE AVAILABILITY OF THIS DATA. THIS IS A PARTNERSHIP BETWEEN MDEPINET AND NEST PROGRAM, NEST IS LEVERAGING THE EXPERIENCE AND STRENGTH THROUGH PARTNERING WITH THE MDEPINET AND LEADERSHIP AND NEST PROGRAM AND THEN UTILIZING MDEPINET IN THE DEMONSTRATION FRAMEWORK. A.I. TOOLS ESPECIALLY THOSE ASSOCIATED WITH RADIOLOGY ARE GENERALLY REGULATED DEVICES, THEY ARE OPPORTUNITIES FOR A.I. DEVELOPERS AND VENDORS TO PARTICIPATE IN FDA PRE-CERTIFICATION PROGRAM THROUGH DIRECTLY PARTICIPATING AS A VENDOR OR PROVIDING FEEDBACK TO THE AGENCY WHAT IMPLEMENTATIONS COULD BE WORKABLE, AND REAL WORLD EVIDENCE FOR GENERATION, A BROAD OPPORTUNITY HERE FOR EXPERTISE IN THIS AUDIENCE AND ACROSS THE A.I. ECOSYSTEM TO HELP SUPPORT THAT APPLICATION. THANK YOU. [APPLAUSE] >> NICK, THANK YOU VERY MUCH FOR THAT SUMMARY OF WHAT'S GOING ON AT THE FDA. THE NEXT SPEAKER IN THE PROGRAM IS JUDY BURLESON, FROM THE AMERICAN COLLEGE OF RADIOLOGY AND SHE'S OUR SENIOR STAFF THAT WORKED A LOT WITH CLINICAL DATA REGISTRIES AND IS GOING TO GIVE US A LITTLE BIT OF INSIGHT ABOUT WHAT THE ACR EXPERIENCE IS WITH REGISTRIES AND HOW THAT CAN BE ADAPTED TO THE DEVELOPMENT OF A.I. REGISTRIES. SO JUDY, THANK YOU. >> GOOD AFTERNOON, THANK YOU, BIBB. THANK YOU FOR INVITING ME HERE. I APPRECIATE BEING INCLUDED IN SUCH A LEARNED GROUP OF PRESENTERS WHO SHARED WITH US TODAY SOME REMARKABLE WORK AND DATA AND INFORMATION, REALLY A GREAT DEAL OF FOOD FOR THOUGHT. WHAT I'D LIKE TO DO IS OFFER ANOTHER TOPIC AREA FOR THOUGHT, THINKING ABOUT HOW TO BEST USE REGISTRIES, WHAT THEY POTENTIALLY COULD OFFER AND BRING TO THE TABLE IN CONSIDERING IMPLEMENTING IN THE REAL WORLD A.I. ALGORITHMS. I'D LIKE TO START BY JUST GIVING SOME SENSE OF HOW WE GOT HERE TODAY, WHERE WE ARE WITH CLINICAL DATA REGISTRIES, TALKING ABOUT CLINICAL PRACTICE REGISTRIES, QUALITY REGISTRIES, AND HOW THEY REALLY MOVED FORWARD OVER THE LAST FEW DECADES. IN THE EARLY 21ST CENTURY, ACTUALLY THE TURN OF THE CENTURY, THERE WERE REPORTS FROM THE THEN-IOM THAT EMPHASIZED THE VARIATION AND THE ISSUES WITH HEALTH CARE IN THE UNITED STATES. AND SO THAT BROUGHT A GREAT DEAL OF FORETHOUGHT AND EXAMINATION OF HEALTHCARE SYSTEM TODAY AND WHAT WE CAN DO TO IMPROVE IT. PART OF THAT WAS DEVELOPMENT OF CLINICAL DATA REGISTRIES BY SEVERAL SPECIALTY SOCIETIES, SUCH AS THE SOCIETY FOR THORACIC SURGEONS AND THE AMERICAN COLLEGE OF SURGEONS, MANY DECADES AGO. IN THE LAST FEW YEARS, THERE'S BEEN A FOCUS ON THE USE OF REGISTRIES AND PAYMENT REFORM, AND LOOKING AT THE DATA THAT'S COLLECTED IN THE REGISTRIES TO BRING -- IDENTIFY WAYS WE CAN MORE FOCUS CARE ON THE PATIENT, AND INVOLVE PATIENTS IN THE CARE. AND WITH THAT PROVIDING PUBLIC REPORTING OF PERFORMANCE DATA TO ALLOW PATIENTS AND CONSUMERS CHOICE IN HOW THEIR HEALTH CARE IS DELIVERED TO THEM. JUST TO GIVE YOU JUST SORT OF THE PLAYING FIELD OF THE ACR'S EXPERIENCE WITH REGISTRIES, WHICH IS SIMILAR TO A NUMBER OF OTHER REGISTRIES DEVELOPED BY SPECIALTY SOCIETIES, WE'RE NOW IN OUR TENTH YEAR OF REGISTRY MONITORING AND STEWARDSHIP. WE'VE BEEN COLLECTING DATA FOR THAT PERIOD OF TIME, AND CURRENTLY THERE ARE SIX DISTINCT REGISTRIES, THREE OF WHICH ARE FOCUSED ON CANCER SCREENING, A COUPLE THAT ARE MODALITY SPECIFIC, AND ALSO LOOKING AT DOSE MONITORING AND OPTIMIZATION. WE HAVE CLOSE TO 4500 SITES PARTICIPATING IN OUR REGISTRY, AND TO DATE 89 MILLION EXAMS HAVE BEEN COLLECTED WITH HUNDREDS OF THOUSANDS BEING COLLECTED EVERY MONTH. WE'VE INTEGRATED VARIOUS SUBMISSION METHODS TO RECEIVE DATA FROM SITES USING TECHNOLOGY SUCH AS WEB-BASED APIs, HL7 MESSAGING, WE'RE LOOKING AT NOW USING FHIR, TRADITIONAL METHODS OF FILE UPLOADS AND THROUGH OUR PROPRIETARY SOFTWARE TRIAD. WE HAVE DOZENS OF PARTNERSHIPS WITH VENDORS, VERY COLLABORATIVE, OF VARIOUS TYPES THAT SUPPORT OUR SITES GETTING DATA TO US, AND INFORM US IN WAYS THAT WE CAN EXPAND AND IMPROVE OUR REGISTRY PROCESSES. MOST RECENTLY WE'VE INTEGRATED THE USE OF STRUCTURED REPORTS. WE'VE WORKED COLLABORATIVELY WITH THE SOCIETY FOR INTERVENTIONAL RADIOLOGY TO IMPLEMENT THEIR STRUCTURED REPORTS INTO QUALITY REGISTRY AND TO CONSUME DATA FROM THOSE STRUCTURED REPORTS THAT HAVE DISCRETE DATA ELEMENT FIELDS TO USE FOR CALCULATING PERFORMANCE MEASURES AND PROVIDING FEEDBACK TO PARTICIPANTS. WITH ALL THAT, WE'VE DEVELOPED A PRETTY ROBUST INFRASTRUCTURE TO DATE, AND HAVE BEGUN COORDINATING MORE AND MORE WITH ACR INFORMATICS AND DATA SCIENCE INSTITUTE. JUST TO GIVE A SENSE OF SOME OF THE USES OF THIS TYPE OF REGISTRY DATA, FIRST AND FOREMOST IT IS FOR QUALITY IMPROVEMENT PROVIDING PARTICIPANTS WITH COMPARATIVE FEEDBACK ON THEIR MEASURES COMPARED WITH OTHERS PARTICIPATING IN THE REGISTRY. IN THE LAST FOUR YEARS IN A HAS ALSO BEEN USED FOR VALUE-BASED PAYMENT REPORTING FOR THE MEDICARE PAYMENT PROGRAM. WITH THE GROWING VOLUME OF DATA THAT'S BEEN COLLECTED IN THE REGISTRY OVER THE PAST 10 YEARS, WE'RE NOW ABLE TO USE IT TO DEVELOP BENCHMARKS SUCH AS WE DID FOR USING DATA FROM THE DOSE INDEX REGISTRY DEVELOPED DIAGNOSTIC REFERENCE LEVELS FOR CERTAIN CT EXAMS, AND WITH THAT INCREASING VOLUME OF DATA WE'RE RECEIVING AND USING THAT IN CONJUNCTION WITH IMAGING DATA, REPORTING DATA, FINDINGS AND RECOMMENDATIONS, INTEGRATED WITH BIOPSY AND PATHOLOGY OUTCOME DATA, THAT SORT OF INFORMATION MAY ENABLE US TO INFORM AND MODIFY IMAGING PARADIGMS OR SUCH AS SCREENING INTERVAL RECOMMENDATIONS OR MOST VIABLE MODALITIES FOR DETECTION, AND OTHER PRACTICE BENCHMARKS AND MODIFICATIONS. TO LOOK AT HOW THE A.I. ALGORITHMS MIGHT BE INTEGRATED INTO THE REGISTRY PROCESS, AS I MENTIONED, WE'RE CURRENTLY USING STRUCTURED REPORTS ON A SMALL SCALE, WITH STANDARDIZED DATA ELEMENTS AND VALUES, AND SO THE IDEA IS TO THEN DEVELOP THESE STRUCTURED TEMPLATES THAT HAVE INTEGRATED A.I. ALGORITHMS AND HAVE BEEN IMPLEMENTED IN REPORTING SOFTWARE THAT IS THEN EXTRACTING DATA FROM THE ACTUAL POINT OF CARE WHEN THE RADIOLOGIST IS REPORTING THE SUBMITTING TO THE REGISTRY THROUGH THE MEANS OF TRANSMISSIONS THAT WE MAY ENABLE. THIS IS REALLY USE CASE FOR QUALITY MEASUREMENT AND PERFORMANCE FEEDBACK, PERFORMANCE REPORTING THAT SPURRED AND ENCOURAGED A NUMBER OF PRACTICES TO USE THAT SORT OF PROCESS, BECOMING INTERESTED IN USING THAT FOR THAT PURPOSE. SO, WHILE THE DATA THAT'S BEING COLLECTED MAY HAVE ASSOCIATED METADATA FROM THE A.I. ALGORITHM WHICH COULD BE INTEGRATED INTO THE REPORTING TOOL, WE'RE ALSO COLLECTING THE PERFORMANCE DATA, SO THIS IS MULTI-USE PROCESS WHERE THE DATA IS AVAILABLE AND USED BY DIFFERENT TYPES OF DEVELOPERS, REGULATORY AGENCIES, AND MOST IMPORTANTLY THE PHYSICIANS USING IT. JUST TO GO QUICKLY THROUGH WHAT BIBB BROUGHT ABOUT IN LOOKING AT THE SPECIFICATIONS FOR MONITORING ALGORITHMS IN PRACTICE, FIRST STEP IS OBVIOUSLY AS I MENTIONED GETTING THE DATA FROM RADIOLOGISTS INPUT ON THE EXAM ITSELF, THE FINDINGS OF THE EXAMINATION, RECOMMENDATIONS, AND ASSOCIATING THAT METADATA AT THE POINT OF THE EXAM, GATHERING THAT AT THE SAME POINT AND CONTRIBUTING IT TO THE REGISTRY. THAT METADATA AND THE ACTUAL DISCRETE DATA ELEMENTS FROM THE REPORT IS CAPTURED IN THE BACKGROUND, WHERE IT IS THEN TRANSMITTED TO THE REGISTRY, THE ALGORITHMS ARE ASSESSED AT THE SAME TIME AS THE PERFORMANCE METRICS FOR THE PHYSICIANS, AND THAT IS THE METADATA ASSOCIATED WITH EXAM PARAMETERS, USED TO DEVELOP ALGORITHMS BY THE DEVELOPERS AND TO PRESENT INFORMATION TO FDA FOR EVALUATION OF THE ALGORITHMS. THE PREVIOUS SPEAKER, DR. PETRICK, TALKED ABOUT THE NEST PROGRAMS THAT ARE OCCURRING. I WANT TO JUST GO OVER VERY QUICKLY ONE OF THOSE DEMONSTRATION PROJECTS THAT HAS BEEN APPROVED UNDER NEST, WHICH IS BEING VIEWED FOR PRE AND POST MARKET SURVEILLANCE OF AN ALGORITHM. THIS IS THE MODEL THAT IS BEING ASSESSED, USING LUNG RADS INTEGRATED INTO ASSIST MODULE. LUNG RADS IS A TOOL THAT IS USED, DEVELOPED BY ACR, USED TO STANDARDIZE CT REPORTING AND RECOMMENDATION FOR LUNG CANCER SCREENING USING CT, LOW-DOSE CT, SO THE -- IN THE DEMONSTRATION PROJECT, THE GUIDELINE THAT IS INTEGRATED INTO THE REPORTING MODULE IS ALSO -- LAST AN ASSOCIATED ALGORITHM, SO THAT AT THE SAME TIME THAT WE'RE COLLECTING DATA FOR VALIDATION OF THE ALGORITHM AT A LOCAL LEVEL, AND TESTING IT PRIOR TO MARKET APPROVAL, THE DEMONSTRATION PROJECT LOOKS TO FACILITATE INTEROPERABILITY BETWEEN THE REPORTING AND A.I. DEVELOPERS, AND THE VENDORS, TO GENERATE A STANDARDIZED DATA IN A REAL-WORLD SETTING. SO AS THE DATA IS BEING CAPTURED, AND VALIDATED, THE IDEA IS TO BEGIN COLLECTING IT IN A REAL WORLD SETTING USING PARTICIPANTS THAT MAY BE INVOLVED IN THE LUNG CANCER SCREENING REGISTRY. SO THE IDEA IS TO BUILD ON THE EXISTING REGISTRY INFRASTRUCTURE, FOR POST-MARKET SURVEILLANCE. TO SUMMARIZE, AGAIN, THE FLOW OF THAT PROTOTYPE, INITIALLY CAPTURING THE DATA AT THE POINT OF CARE WITH THE RADIOLOGIST INPUT AND METADATA BEHIND THE ALGORITHM, AND ONE WHAT IS IN THE STRUCTURED REPORT, SUBMITTING THAT, TRANSMITTING IT TO A REGISTRY, AND THEN LOOKING AT THE DATA IN MULTIPLE WAYS TO PROVIDE FEEDBACK TO ALL THOSE PARTIES INVOLVED, WHETHER IT'S THE DEVELOPER, REGULATORY AGENCIES, OR THE PHYSICIAN IN PRACTICE. SO MY QUESTION FOR YOU, HOW BEST CAN THIS IDEA OF USING A PRACTICE REGISTRY BE INTEGRATED INTO THE IMPLEMENTATION OF A.I., IN THE REAL WORLD SETTING, HOW CAN REGISTRIES BEST SUPPORT DETERMINING WHETHER THE ALGORITHMS ARE ACCURATE, HOW BEST CAN THEY VALIDATE AND MONITOR THIS AND BUILD TRUST IN HOW THE ALGORITHM WORKS. WE CAN BUILD ON EXISTING REGISTRY INFRASTRUCTURE, BUT THERE, AS WE'VE HEARD TODAY, THERE'S SO MUCH TO CONSIDER, SO MANY POINTS OF DATA COMING FROM ONE PLACE OR ANOTHER, WHAT SHOULD BE LOOKED, AT WHAT POINT IT SHOULD BE LOOKED AT, WHEN IT SHOULD BE IDENTIFIED OR ANONYMIZED, ISSUES WITH PHI AND SECURITY AND PRIVACY, THERE'S SO MANY ASPECT OF USING IT, BUT TO USE A DEMONSTRATION TO INTEGRATE AND BRING DATA INTO THE REGISTRY, DEVELOP RELATIONSHIPS WITH THOSE IN ATTENDANCE, LEARN HOW BEST THIS CAN BE SUPPORTED. THANK YOU. [APPLAUSE] >> THANK YOU. JUDY, THANK YOU VERY MUCH. OUR NEXT SPEAKER IS KEVIN LYMAN, KEVIN IS THE CEO OF ENLITIC, GIVING US PERSPECTIVE FROM THE LAST MILE REGARDING AGAIN SCIENCE TO PRACTICE FROM DEVELOPER INDUSTRY PERSPECTIVE. KEVIN, WELCOME. THANK YOU VERY MUCH. >> I'M KEVIN, CEO AND FORMER LEAD SCIENTIST AT ENLITIC IN SAN FRANCISCO DEVELOPING ARTIFICIAL INTELLIGENCE FOR RADIOLOGY. THANK YOU FOR HAVING ME, IT'S AN HONOR TO BE HERE AMONG THE OTHER FANTASTIC SPEAKERS. AS SOMEBODY WHO HAS HAD THE PLEASURE OF WORKING EVERY ROLE THERE IS AT A COMPANY THAT'S ACTIVELY DEPLOYING CLINICAL A.I. AROUND THE WORLD I'M HERE TO SHARE INSIGHT AND SPEND TEN MINUTES FOCUSED ON TEN SPECIFIC CHALLENGES TOUCHING ON EACH OF THEM BRIEFLY ENOUGH TO AT LEAST PUT THEM ON EVERYONE'S RADAR FOR THE SAKE OF SPARKING INTERESTING DISCUSSION. TO START, RADIOLOGY DATA IS HIGHLY NUANCED, UNINTENDED BIAS IS HARD TO AVOID WITHOUT PROPER EDUCATION. IT'S CRITICAL TO UNDERSTAND DIFFERENCE BETWEEN DETECTING VISUAL PATTERNS SUGGESTIVE OF DIAGNOSIS AND PROPERLY GOING THE FULL MILE OF DIAGNOSING SOMETHING. I CAN SAY WITH CERTAIN DEGREE OF CONFIDENCE IN MY ABILITY TO DETECT CALCIFIED LUNG NODULES, PLEURAL EFFUSION, CAVITATION, BUT JUMPING TO THE CLAIM OF FULLY DIAGNOSES TUBERCULOSIS WITH A J PEG IS ANOTHER STORY. AND FULLY APPRECIATING THAT KIND OF NUANCE IS DIFFICULT WHEN YOU DON'T HAVE A PROPER MEDICAL EDUCATION BACKGROUND. HOWEVER, IT'S CRITICAL TO DO SO WHEN YOU'RE BUILDING TOOLS BASED ON THESE ASSUMPTIONS. A QUICK FIX FOR THIS WOULD BE SOME KIND OF PROPER EDUCATIONAL BACKGROUND IN RADIOLOGY FOR NON-RADIOLOGISTS. WEB SERIES OR EVEN SOMETHING THAT SIMPLE COULD SURPRISINGLY HELP MOVE THE NEEDLE FOR US NOT AS VERSED IN THE MULTI-DISCIPLINARY FIELD. DATA SCIENTISTS AND ENGINEERS NEED SUPPORT, WE HIRE RADIOGRAPHERS, PARAMETERS DICTATE REASONS AND PARAMETERS WITH WHICH THAT IMAGE SHOULD BE READ. IN CT, FOR EXAMPLE, DIFFERENT SLICE THICKNESS, RECONSTRUCTION KERNELS, WINDOWING LEVELS WILL DICTATE HOW THE IMAGE SHOULD BE READ. IN THE MRI THERE ARE KNOBS AND DIALS BEFORE THE RADIOLOGIST SEES THE IMAGE THAT DICTATES WHAT THEY CAN GET OUT OF IT. IF YOU'RE BUILDING A TOOL THAT READS DICOM AND NOT RAW SENSOR DATA YOU NEED TO BE ON TOP OF THE SAME PROBLEMS. HAVING A PROPER EDUCATION ON THE FULL DIAGNOSTIC LIFE CYCLE NOT JUST ON IMAGE ACQUISITION AND INTERPRETATION BUT EVERYTHING ELSE IS PART OF THAT IS JUST AS CRITICAL. MORE TIME SPENT BUILDING TOOLS THAT ENABLE US TO BUILD CLINICAL A.I. THAN ACTUALLY BUILDING CLINICAL A.I., DIFFICULT TO TACKLE WITHOUT THE PROPER INFRASTRUCTURE. OVER FOUR YEARS WE'VE HAD TO BUILD STATE-OF-THE-ART TOOLS FOR DATA EXTRACTION, ANONYMIZATION, LABELING, INTEGRATION INTO A WORKFLOW, EACH HAS BEEN EXPENSIVE TO BUILD, REQUIRED A GREAT DEAL OF PRODUCT DISCOVERY AND ULTIMATELY HAS MOVING TARGETS'S A AS FAR AS WHAT REQUIREMENT SHOULD BE, EDUCATIONAL BUT A DISTRACTION FROM BUILDING AND DEPLOYING CLINICAL A.I. PUBLICLY AVAILABLE OPEN-SOURCE TOOLS THAT EVERYONE CAN USE FOR THEIR INFRASTRUCTURE WOULD BE A MASSIVE BOON FOR ALL AND DIFFICULT TO MINE FOR GOLD WHEN YOU NEED TO START BY BUILDING A MINING PEG. PRIVACY AND SECURITY ARE SUBJECTIVE MEASURES MOVING TARGETS, ESPECIALLY ON A GLOBAL SCALE. HIPAA HAS SAFETY HARBOR GUIDELINES MEANT TO BE SAFE ENOUGH, BUT IN REALITY THE REAL BAR YOU NEED TO PASS IS THE SUBJECTIVE INTERPRETATION OF PRIVACY TEAM AT EACH SITE YOU WORK WITH. IN MOST CASES RELY ON EXPERT OPINION BUT NONE OF THEM HAPPEN TO KNOW WHO THE EXPERT IS AND THEIR OPINION SEEMS TO CHANGE A NEAR DAILY BASES, WE HAD TO BUILD CUSTOMIZED TOOLS AND TACKLING ON A GLOBAL SCALE IS ANOTHER STORY. I SPENT A FEW DAYS IN CHINA, FOR EXAMPLE, RUNNING BLIND TESTS ON THE SPOT WITH SOME PARTNERS TO ASSUAGE CONCERNS ABOUT DATA CROSSING BORDERS. OTHER SIDES ARE HAPPY TO ATTACH YESTERDAY'S SCANS ON AN E-MAIL. ULTIMATELY ALL OF US SHOULD BE CRITICALLY CONCERNED ABOUT PATIENT PRIVACY AND SECURITY AS A TOP CONCERN BUT THAT MEANS WE NEED TO RELY ON WELL DOCUMENTED GUIDELINES AND NOT ON OPINIONS. MOST HOSPITALS DO NOT DESIGN SOFTWARE WITH EASE OF A.I. IN MIND. MY FIRST JOB WAS TO TRAVEL ON SITE AND FIGURE OUT HOW TO EXTRACT AND MAKE USE OF HUNDREDS OF TERABYTES OF DATA, THIS BECAME AN EXERCISE IN TRYING TO REVERSE ENGINEER AND WRESTLE WITH ANTIQUATED SYSTEMS WITHOUT TAKING THEM OFFLINE. AS YOU CAN IMAGINE THAT'S QUITE A CHALLENGE. EVEN MORE OF ONE WHEN YOU'RE TRYING TO DO THE REVERSE AND INTEGRATE A SYSTEM BACK ON TOP OF THOSE EXISTING TOOLS WITHOUT CREATING OVERBEARING BURDEN. THE SOONER WE CAN ARRIVE AT OPEN STANDARDS AND APIs THE SOONER YOU'LL SEE TOOLS OBTAIN UBIQUITY. WE NEED TO PUSH PROVIDERS TO ADOPT INFRASTRUCTURE. MODELS DON'T GET REGULATORY APPROVALS, CLAIMS GET REGULATORY APPROVALS. WE TRAINED CHEST X-RAY MODELINGS TO HAVE NEED FOR FOLLOW-UP BUT THAT'S A BOLD CLAIM AND ONE THAT'S VERY UNLIKELY TO GET PROPER REGULATORY APPROVAL UNLESS WE HAD A GIGANTIC VALIDATION SET WITH EXAMPLES OF EACH THING UNDER THE UMBRELLA OF EVERYTHING, THIS BIASED EARLY COMMERCIALIZATION MODEL TO FOCUS ON SCREENING PROGRAM WAS LIMITED NUMBERS OF DETECTIONS OF INTEREST AS A STEPPING STONE TOWARD PROPER VISION AND LONG-TERM GOAL OF FULL BODY DIAGNOSTIC COVERAGE. NOW, ULTIMATELY THIS COMES ALONG WITH A NUMBER OF ADDITIONAL CHALLENGES WHEN YOU RECOGNIZE MANY CLAIMS WE HOPE TO MAKE ARE UNKNOWN UNKNOWNS. HOW DOES ONE DESIGN A TRIAL TO VALIDATE A TOOL THAT IDENTIFIES NORMAL STUDIES UP FRONT? HOW ABOUT A TOOL THAT HELPS A DOCTOR FOLLOW A FINDING UP YEARS IN ADVANCE OF WHEN THEY OTHERWISE MAY HAVE? IT'S VERY DIFFICULT FOR INDUSTRIES TO PUSH FORWARD ON THESE CLAIMS WHEN REGULATORY BODIES DON'T NECESSARILY HAVE THESE ANSWERS TO BEGIN WITH. ON A SIMILAR NOTE, VALIDATION IS VERY EXPENSIVE. AND IT'S IMPORTANT TO REPEAT FOR EACH TARGET POPULATION. TURNS OUT IT'S NOT VERY CHEAP TO HIRE 17 RADIOLOGISTS EACH TO READ THREE TO FIVE HUNDRED CT SCANS TWICE, ONCE WITH AND ONCE WITHOUT ASSISTANCE FROM YOUR MODEL. IT BECOMES MORE EXPENSIVE WHEN YOU REALIZE WHEN THE FDA MIGHT BE HAPPY WITH YOUR VALIDATION ON AN AMERICAN POPULATION, THE PMDA LIKELY WANTS TO SEE VALIDATION ON A JAPANESE POPULATION. ULTIMATELY, IT WOULD BE MASSIVELY BENEFICIAL TO SEE GOVERNING BODIES WORK TOGETHER TO ADOPT INTERNATIONAL STANDARDS AND COLLABORATE ON DATASETS FOR TRAINING AND VALIDATION. EACH DIFFERENT POPULATION HAS DIFFERENT INCIDENCES OF ABNORMALITY, ANATOMIC VARIATION, IMAGING PROTOCOLS AND DEFINITIONS OF WHAT NEEDS TO BE FOLLOWED UP, AND WHETHER OR NOT IT'S BEING DONE FOR REGULATORY APPROVAL IT'S CRITICAL TO DO VALIDATION IN EVERY SETTING WHERE YOU INTEND TO DEPLOY. WORKFLOW INTEGRATION NEEDS TO BE AS CONVENIENT AS NON-THREATENING AS POSSIBLE. AND WE OFTEN SEE THIS AS SOMETHING THAT'S HEAVILY OVERLOOKED. NO RADIOLOGIST WANTS TO LEARN A NEW SOFTWARE SYSTEM TO SWITCH BETWEEN FOR EACH STUDY, USER EXPERIENCE NEEDS TO BE A CRITICAL FOCUS EARLY ON IN THE PIPELINE. HOWEVER, IN MANY CASES IT'S VERY CHALLENGING TO EVEN ORGANIZE THIS. IN REALITY, DATA SCIENTISTS AND ENGINEERS SHOULD SIT BY BY SIDE WHERE THEY WANT TO INTEGRATE, HOWEVER YOU CAN IMAGINE IT'S OFTEN DIFFERENT CULT TO GET -- DIFFICULT TO GET A DATA SCIENCE FROM A PROPER HOSPITAL. IT WOULD BE HELPFUL TO HAVE MORE CASE STUDIES ON THIS AREA, ACCESS TO MORE VIDEOS OF THIS KIND OF LIVE INTERPRETATION. ADOPTION REQUIRES BUYIN FROM EVERY STAKEHOLDERS, PUSHING INTO A LIVE CLINICAL SETTING REQUIRES BUY-IN FROM THE PATIENT, RADIOLOGISTIST AND ULTIMATELY FINANCIER WHO IS GOING TO SIGN THE CHECK IN THE FIRST PLACE. AND IN MANY CASES THE VALUE PROPOSITION YOU MAKE WILL BE AT ODDS WITH ONE ANOTHER. THEORETICALLY IT SHOULD BE EASY TO PROVE REDUCTIONS IN READ TIMES AND FALSE POSITIVE RATES ALONGSIDE IMPROVEMENT IN TRUE POSITIVE RATES BUT TRANSLATING INTO ECONOMIC VALUE IS OFTEN A CHALLENGE WITHOUT GREATER ACCESS TO DATA AND UNDERSTANDING OF THE DATA ON THE ECONOMICS OF RADIOLOGY, HOW DO YOU PLACE A PRICE, FOR EXAMPLE, ON HELPING SOMEONE FIND A CRITICAL FINDING THEY MAY OTHERWISE HAVE MISSED? AND FINALLY, NOT ALL PROBLEMS IN RADIOLOGY WITH CREATED EQUAL. SOME ARE MORE IMPACTFUL AND ATTAINABLE THAN OWES. OTHERS, TO A RADIOLOGIST IT OPENS TO A WORLD OF QUESTIONS, NODULE DETECTION, DIAGNOSTIC, CHARACTERIZATION, MEASUREMENT? THESE ARE ALL CRITICALLY DIFFERENT PROBLEMS WITH DIFFERENT VALUE PROPOSITION AND DIFFERENT COSTS WITH BUILDING SYSTEMS IN THE FIRST PLACE. AND ULTIMATELY, FOR THOSE THAT ARE NOT RADIOLOGISTS, TO KEEP US FOCUSED ON SOLVING IMPORTANT ON WHAT THESE IMPORTANT PROBLEMS- ARE. IN MANY CASES WE OFTEN BIAS TOWARD DETECTION AND DIAGNOSIS BUT AS WE'VE HEARD EARLIER THERE ARE PROBLEMS IN RECONSTRUCTION, BEYOND EDUCATING ON RADIOLOGY AS A WHOLE I THINK I WOULD LOVE TO SEE MORE FOCUS ON SPECIFICALLY WHAT PROBLEMS NEED TO BE TACKLED SO WE'RE NOT JUST WORKING ON WHAT'S EASY BUT WHAT'S MEANINGFUL. THANK YOU. [APPLAUSE] >> KEVIN, THANK YOU VERY MUCH. CHALLENGING FOR THOSE WHO WANT TO IMPLEMENT A.I. OUR LAST SPEAKER IS OUR FIRST SPEAKER, KEITH DREYER WILL SUM UP OUR SESSION TALKING A LITTLE BIT ABOUT HOW IF WE THINK OF THE END GAME AND THE FRONT GAME AND HOW USE CASES THAT ARE WELL DEFINED, WELL SPECIFIED, CAN HELP US GET TO SOLVE SOME OF THESE CHALLENGES. SO KEITH, THANK YOU. >> THANKS, BIBB. WELL, THANK YOU. I THOUGHT THAT WAS AN EXCELLENT SET OF PRESENTATIONS, YOU HAVE TO SPEND TIME TO PUT THE COMPONENTS TOGETHER BUT THEY TIE IN AND MAKE SENSE. I'LL TRY TO PICK UP FROM THE PRESENTATION I GAVE EARLIER THIS MORNING AND WRAP IN SOME CONCEPTS AND CARRY OUT SOME ANSWERS TO THE QUESTIONS. I TALKED ABOUT, AS OTHERS DURING THIS SESSION TALKED ABOUT, AREAS OF INTEREST WERE A.I. WORK THAT'S BEING DONE TODAY. I ASKED ABOUT HOW MANY FOLKS HAVE USED A.I. ALGORITHMS OR WORKED IN THE SPACE, 70, 75%. HOW MANY PEOPLE HAVE USED THEM IN CLINICAL PRACTICE? SO MAYBE 5%. THAT'S NOT UNCOMMON. WHY IS THAT? IS IT TOO EARLY? ARE WE IN AN EARLY PHASE OF RAMP UP? KEVIN DESCRIBED THEM FROM A DEVELOPER'S PERSPECTIVE AND YOU'VE SEEN SOME OTHER EFFORTS THAT ARE BEING DONE TO TRY AND SOLVE SOME OF THESE THINGS. I'LL GO BACK TO THIS SLIDE I TALKED ABOUT EARLIER, OTHERS HAVE SHOWN, THAT IS WHAT ARE THE THINGS SLOWING US DOWN? THE ONE I WANT TO FOCUS ON IS THE TOP ONE, USE CASES. I WOULD GO BACK TO KEVIN, IF YOU'RE A RADIOLOGIST OR IF YOU'RE A GROUP OF RADIOLOGISTS, A SOCIETY RSNA, ASNR, TRY TO ANSWER THOSE QUESTIONS, HOW DO YOU PROPAGATE THAT INFORMATION? SO I THINK ONE OF THE WAYS YOU CAN DO THAT IS BY CONCEPT OF USE CASES. IF I CAN DESCRIBE WHAT IS IMPORTANT TO RADIOLOGY AND WHAT COULD BE LOW-HANGING FRUIT MAYBE THAT'S A WAY TO SAY HERE'S THE ALGORITHMS THAT'S GOING TO COME TO YOU WITH THE OUTPUTS THAT YOU NEED TO WORRY ABOUT INPUT INTO THOSE CLASSIC I.T. SYSTEMS IN PLACE INSIDE RADIOLOGY TODAY. SO LET'S LOOK AT THE LUNG CANCER EXAMPLE MANY HAVE GIVEN AND TALKED ABOUT. LOOK AT PULMONARY KNOWLEDGE TO SCREEN FOR LUNG CANCER, I TALKED ABOUT THE CHALLENGE OF HAVING MULTIPLE ALGORITHMS GIVING DIFFERENT OUTPUTS, HOW WOULD YOU SOLVE THAT PROBLEM OF THE DISCORD WHEN THE DATA COMES BACK? WELL, BIBB TALKED ABOUT INTRODUCTION OF AMERICAN COLLEGEEL RADIOLOGY DATA SCIENCE INSTITUTE, RSNA AS OTHER INITIATIVES. WE REALIZE THERE'S SOMETHING WE NEED TO DO, IT HASN'T BEEN DONE BEFORE, WE DON'T HAVE MANY PLACES TO LOOK FOR PRECEDENTS AND HAVE TO START IN THE NEW SPACE TO LISTEN TO PRESENTATIONS AND DELIVERS ANSWERS AS BEST WE CAN. WITH THE DATA SCIENCE INSTITUTE, THE GOAL TO TAKE EXPERTS AND DERIVE SOME ANSWERS. THE FOCUS IS IN MANY, MANY AREAS, ETHICS, LEGAL, COMMERCIALIZATION, EDUCATION STANDARDS, I WANT TO FOCUS ON USE CASES BECAUSE IT ADDRESSES MANY ISSUES AND ALLOWS SOMETHING TO PRESENT IN TEN MINUTES AND NOT A LOT MORE. JUDY MENTIONED, BIBB AS WELL, THE CONCEPT OF LUNG RADS, A WAY TO DEFINE HOW TO READ FOR LUNG CANCER SCREENING LOW DOES CT, MANY WAYS TO DO IT, ONE OF THE STANDARDS APPROVED IS TO USE LUNG RADS. IT HAS A SET OF STRUCTURED GUIDELINES. A COUPLE ADVANTAGES, WHEN YOU LOOK FOR DATA THAT'S TRUE DATA AT LEAST FROM THE RADIOLOGIST PERSPECTIVE NOW YOU HAVE STRUCTURED REPORTING AND DATA. MODELS ARE TRAINED TO GIVE THE OUTPUT, YOU CAN LOOK AT THE ALGORITHMS TO POPULATE THAT AND THEN ALSO HAVE THE RADIOLOGIST CORRECT THAT AND MAKE CHANGES, YOU CAN FEED THAT INFORMATION BACK INTO THE ALGORITHMS. IF YOU FOLLOW ONE OF THESE PATHWAYS YOU CAN SEE THE ADVANTAGES YOU HAVE. IF WE LOOK AT THAT INTERPRETER AND LET THEM BECOME EXPERTS, WE SAY YOU DEFINE WHAT IT IS AS WHAT IS IMPORTANT TO YOU, WHAT KEVIN TALKED ABOUT THEY NEED TO KNOW, WHAT'S GOING TO MAKE A DIFFERENCE AND DEFINE IN A WAY THAT MIGHT BE EASY FOR DATA SCIENTISTS TO USE AND CREATE. SO IF THIS IS THE USE CASE THAT YOU WOULD USE, YOU COULD SAY ANYBODY CAN TAKE DATA THEN, MAYBE THEY HAVE ALREADY USED LUNG RADS FOR, WHICH MANY PEOPLE ARE USING, AND CREATE A CORPUS OF TRAINING DATA AND VALIDATION DATA. SO NOW YOU HAVE MULTIPLE SITES CREATING INDEPENDENTLY AND DISTRIBUTEDLY DATA THAT LOOKS SIMILAR. YOU CAN THEN HAVE DATA SCIENCE ALGORITHMS THAT ALLOW A THOUSAND FLOWERS TO BLOOM BY USING TECHNIQUES AND MECHANISMS BUT THE OUTPUT IS THE SAME AS TRAINED INPUT DATA, YOU CAN HAVE MULTIPLE APPLICATIONS THAT COULD INTEGRATE BECAUSE YOU'VE DEFINED WHAT THAT INTEGRATION LOOKS LIKE AT LEAST FROM A DATA PASSAGE STANDPOINT, AND ALLOW PEOPLE TO KIND OF FOCUS IN THE AREAS OF MAKING THE U I LOOK DIFFERENT AND MAKE WITH TIME WE'LL SEE UIs LOOK BETTER THAN THESE, RADIOLOGISTS WANT THE DATA BEFOREHAND, AFTERWARDS, INTERACT AND SEE SALIENCY MAPS, ET CETERA, ALLOWING FOR THE NOTION TO INTEGRATE INTO DESKTOPS, REPORTING SYSTEMS, RISK SYSTEMS AND EHR SYSTEMS. STRUCTURED USING LUNG RADS TO POPULATE, OUTPUT ALGORITHM THAT COULD BE DIFFERENT BUT OUTPUTS WOULD BE SIMILAR. A NEW PATIENT APPEARS, LOW DOSE CT SCREENING PROCEDURE, YOU HAVE THE ALGORITHM THAT CAN RUN, OUTPUT THE FINDINGS DETECTED AND QUANTIFIED IN PREPARATION FOR SCORING AND PREPARATION, THE RADIOLOGISTS ARE FAMILIAR WITH THIS DATA SO THEY DO THE WORK OR TRAINEES DO THE WORK OF FINDING FINDINGS AND IT WON'T BE BRAND NEW NET NEW INFORMATION FOR THEM, THINGS THEY HAVE ALREADY SEEN. THEY CAN INTERACT, MAKE MODIFICATIONS AND CHANGES AND DISCUSS THIS INFORMATION FOR MEDICAL MANAGEMENT WITH CLINICAL COLLEAGUES THEY ARE FAMILIAR WITH WHY THIS SAYS WAIT SIX MONTHS, DO A BIOPSY, WAIT A YEAR, ET CETERA. INFORMATION CONSISTENT WITH MANUALLY GENERATED INFORMATION. CHANGE THEM ON THE BACK END, DIFFERENT INTERACTION WHEN THE NUMBERS ARE CHANGED, RESULTS ARE CHANGING, NOW YOU HAVE THE SAME PATHWAY INDEPENDENT OF THE ALGORITHM THAT WOULD BE USED. THE OTHER ADVANTAGE, BECAUSE YOU HAVE THIS COMMON MECHANISM FOR DATA OUTPUT, YOU CAN TAKE A LOOK AT DIFFERENT WAYS TO INPUT DATA, DOESN'T MATTER IF IT'S ON THE SCANNER, BUILT IN, ON PREMISES OR CLOUD SOLUTIONS, STILL PROVIDES THE SAME ALGORITHM AND ALSO THE MECHANISM BY WHICH THE RADIOLOGIST INTERACTS HAS FEEDBACK INTERACTION, THEY AGREE, DISAGREE, AGGREGATE THAT INSIDE OF THESE REGISTRIES THAT EXIST LIKE THIS ONE. YOU CAN EXTEND TO CAPTURE THAT REAL WORLD EVIDENCE THAT NICK TALKED ABOUT, THAT WOULD HELP TO SUPPORT THE NEST PROGRAM FROM FDA. YOU CAN INFORM PANELS THAT CREATED GUIDELINES TO MAKE MODIFICATIONS OR ENHANCEMENT VARIANCES FROM HUMAN TO MACHINE OR MACHINE TO MACHINE, INFORM HEALTH CARE MANUFACTURERS TO SEE A.I. ALGORITHMS IN THE WILD IN A WIDE DISTRIBUTED SOLUTION, POPULATE INFORMATION TO SAY IT WORKS ON THIS PATIENT POPULATION OR THIS LOCATION OR THESE DEVICES, NOT WORKING ON THESE, ET CETERA. FURTHERMORE YOU COULD INFORM REGULATORY AGENCIES OF THIS REAL WORLD INFORMATION AND EVIDENCE AS IT'S GOING ON IN THE WILD. SO, THIS IS JUST ONE EXAMPLE. YOU CAN IMAGINE THERE'S MANY, MANY ALGORITHMS BEING WORKED ON BY THE DATA SCIENCE INSTITUTE BUT BEYOND THAT ENCOURAGING OTHER SOCIETIES TO CREATE THESE, THE ACR -- OTHER SUBSPECIALITIES ARE INTERESTED. ANOTHER AREA OF INTEREST TO FIT INSIDE THE 10-MINUTE WINDOW, LUMBAR SPINE MRI, WEARING MY OTHER HAT, CLINICAL DATA SCIENCE, INITIATIVE THAT JIM BEGAN TWO YEARS AGO, WHICH IS INTERESTING, YOU FEEL THE SAME PAIN WHEN YOU GO FROM DIFFERENT ANGLES. KEVIN MENTIONED A CHALLENGE OF CREATING PICKS WHILE LOOKING FOR GOLD, SAME THING THAT HAPPENED, HAD TO CREATE THE PIPELINES TO MOVE THE PROJECTS AT A RAPID ACCELERATED PACE. IF YOU DON'T HAVE THE PICKS, YOU START FROM SCRATCH, THE FIRST THING IS BUILD THE PICKS. A LOT OF PEOPLE HAVE BEEN DOING THIS, THIS WOULD BE AN AREA THAT WOULD BE RIPE FOR COMMON OPEN SOURCE INFORMATION, A WAY TO PASS AND EXPEDITE DEVELOPMENT. SEVERAL DOZENS OF ALGORITHMS HAVE BEEN MADE, I WANT TO HIGHLIGHT MRI LUMBAR SPINE, A PROBLEM THAT COULD BE SOLVED BY DATA SCIENCE TO DETECT FINDINGS OFTENTIMES DIFFICULT OR HAVE HIGH ERROR BETWEEN HUMAN RADIOLOGISTS MAKING THE DIAGNOSIS, TO BE ABLE TO CREATE A TOOL THAT CAN GUIDE RADIOLOGIST OR GIVE MORE INFORMATION TO THE RADIOLOGIST AT THE TIME OF INTERPRETATION. SAME KIND OF THING YOU HEARD EARLIER TODAY, TRUTH COMES OUT OF NLP FROM REPORTS, THOUSANDS OF STUDIES TO LOOK AT SPINAL CANAL, FORAMINAL STENOSIS, WHEN YOU FOCUS AND NARROW THE A.I. CHALLENGE, 90 TO 99% TO CREATE CLINICAL INTEGRATION, ANOTHER STEP OF ADDITIONAL DIFFICULTY BEYOND THE NOTION OF DOING DATA SCIENCE WHICH IS NOT SIMPLE IN AND OF ITSELF. I'LL SHOW YOU WHAT THE LOOKS LIKE IF YOU DIDN'T INTEGRATE, LOOK IN THIS AREA AND SEE AS IT FINDS EACH LEVEL, IT WILL GO THROUGH AND TELL YOU RIGHT AND LEFT FORAMEN, SPINE, LEVEL OF SEVERITY, STILL YOU CAN APPRECIATE THAT'S NOT INTEGRATION ENOUGH, WHAT YOU NEED TO DO BEYOND THAT -- FIRST IS ADVANCE THE SLIDE. LET'S SEE IF I CAN DO IT THIS WAY. YOU NEED TO INTEGRATE IT INTO THE RADIOLOGIST DESKTOP, SAID SEVERAL TIMES, IT'S HETEROGENEOUS ENVIRONMENT, LOOK IN THE UNITED STATES ALONE PAC SYSTEMS, THE MOST POPULATED TODAY DEPLOYED AT 19%. SO THERE'S VERY HETEROGENEOUS MARKET, A DOZEN OR SO SYSTEMS YOU WOULD NEED TO INTEGRATE TO BE ABLE TO SOLVE THE PROBLEM OF JUST INTEGRATING TO PAC SYSTEMS ACROSS THE UNITED STATES ALONE. IF YOU LOOK AT EFFORTS TO DO THIS YOU CAN SEE USES CASE DEFINITION WOULD AID IN THE PROCESS AND ALLOW THE ADDITIONAL INFORMATION OF USER INTERFACE TO GO THROUGH THE PROCESS. THIS IS ONE INTEGRATION, ALSO THE PROCESS THAT WE'RE USING TO INFORM THE DATA SCIENCE INSTITUTE OF HOW WE SEE THE INTEGRATION THAT COULD AND SHOULD OCCUR. HERE IT IS, RADIOLOGISTS THAT'S LOOKING AT THE WORK LIST, SELECT A CASE, WHEN THEY SELECT A LUMBAR SPINE CASE IN THE TOP CORNER YOU'LL SEE UP HERE, YEAH, THEY ARE GOING TO CLICK THAT BUTTON WHICH WILL PROVOKE THE ASSOCIATED A.I. TOOLS, IT'S THE ONE I SHOWED YOU SO IT'S GOING TO GO THROUGH, BRING UP THE, THE RADIOLOGIST IS CONTROL, AS THEY READ AND LOOK AT LEVELS THEN THE WAY THIS USER INTERFACE WAS CONFIGURED IT WILL GIVE THEM READOUT OF SOFTWARE INTERPRETATION OF THE LEVEL THEY ARE LOOKING AT. THEY CAN REVIEW AND ANALYZE THAT AREA, IMAGINE SALIENCY MAPS COMING UP, DISAGREED LEVEL OF CONFIDENCE BUT THEY CAN SAY I WANT TO SAY YES I AGREE, WHEN THEY DO THE SAME INTERFACE WOULD POPULATE INTO THE REPORTING SYSTEM SO THAT STANDARD REPORTING INFORMATION IS THERE. IF THEY DISAGREE THEY COULD CHANGE LEVELS, REPORTED BACK INTO THE A.I. SYSTEM AND TRANSFERRED TO THE REPORTING SYSTEM MOVED THROUGH APIs. I SHOW HOW EARLY IT IS TO DO THE INTEGRATIONS, 75% HAND RAISE TO 5% OF PEOPLE THAT USE IT IN THE WILD HAS CHALLENGES ASSOCIATED WITH IT. FOR US TO GET TO THE POINT WHETHER IT'S FIVE YEARS, TEN YEARS FROM NOW, WHEN YOU DEVELOP SOMETHING YOU CAN SEE IT IMPLEMENTED, I HOPE I CAN DEMONSTRATE THE CHALLENGES, THE CORE THE FINAL MILE AND DISCUSSIONS FROM THE PANEL. THANK YOU VERY MUCH. [APPLAUSE] >> NOW I'D LIKE OUR SPEAKERS TO COME AND I'D ALSO LIKE TO INTRODUCE GREG PAPPAS AND GREG SORENSON TO BE ON OUR PANEL, I MENTIONED GREG PAPPAS BEFORE, FROM THE FDA, WHO WAS INVOLVED WITH THE NEST PROGRAM AND IS NOW DOING OTHER WORK FOR THE FDA. GREG SORENSON PROBABLY NEEDS NO INTRODUCTION TO ANY OF US RADIOLOGISTS WITH HARVARD MGH ROOTS AND ON TO MANAGING SIEMENS IMAGING DIVISION AT SOME POINT AND NOW INVOLVED IN A.I. VENTURE, AND SO WHAT I'D LIKE TO DO IS IN THE BEGINNING, LET EACH OF THEM TALK A LITTLE BIT, AND WHAT I'D LIKE FIRST IS TO GET GREG PAPPAS TO TALK A LITTLE BIT ABOUT WHAT NICK HAD TALKED ABOUT WITH REAL WORLD DATA AND SOME OF THE THINGS THAT YOU SEE AND SORT OF YOUR TAKE ON SOME POTENTIAL PARTNERSHIPS BETWEEN THE REGULATORY AGENCIES AND SPECIALTY SOCIETIES AND THE OTHER STAKEHOLDERS, THAT WOULD HELP MOVE THE PROCESS ALONG. >> THANKS SO MUCH. I WANT TO BE MODEST. I COME TO THIS FROM A PUBLIC HEALTH BIG DATA PERSPECTIVE AND MOVED TO BIOLOGICALS, WE'VE GOT NICHOLAS HERE THE EXPERT FROM DEVICES. I WANTED TO SAY THAT MOST OF ALL WHAT I WAS DOING DURING THIS MEETING WAS REALIZING CONNECTIONS, SENDING E-MAILS AND CONNECTING PEOPLE AT THE FDA AND WITH OTHER PARTNERS. THERE ARE A LOT OF MOVING PIECES ON THIS. I REALLY CONGRATULATE ACR AND BIBB AND KEITH FOR SEEING THE OPPORTUNITIES FOR COLLABORATION, AND I'LL JUST NOTE A COUPLE OF THE COLLABORATIONS THAT I HAD IN MIND TODAY. ONE IS THE DISTRIBUTED ANALYTICS ASPECT OF THIS WORK. THE RADIOLOGICAL SPACE PROVIDES A UNIQUE OPPORTUNITY TO TEST THAT. RIGHT NOW WHAT'S GOING ON, THE MILITARY IS SUPPORTING MITRE TO DEVELOP IEEE STAND ORDERS FOR ANALYTICS. WHEN JAYASHREE HEARD ABOUT THAT SHE IMMEDIATELY WANTED TO BE IN CONTACT WITH THOSE FOLKS, A GOOD THING MOVING FORWARD. THIS IS A VERY EXCITING AREA THAT IS GOING TO FACILITATE MORE DATA SHARING IN SECURE WAYS AND OF COURSE YOU ALL ARE DEALING WITH THE HUGE PIECES OF DATA THAT DON'T DO WELL WITH REPOSITORIES. THE SECOND PIECE IS MDEPINET. I WANT TO THANK JUDY FOR HER VERY PRECISE AND DETAILED UNDERSTANDING OF PRESENTATION ON WHAT ACR IS DOING AROUND REGISTRIES, WITH A COMMUNITY OF OTHERS IN DEVICE SPACES THAT THIS COLLABORATION SHARING IS SHARING LESSONS LEARNED AND BEST PRACTICES TO FACILITATE LEARNING HERE. THANK YOU AND I'D LIKE TO CONGRATULATE YOU. >> GREG, FROM THE PERSPECTIVE OF A BIG GIANT IN THE HEALTHCARE INDUSTRY DOWN TO NOW A START-UP IN THE A.I. INDUSTRY, WHAT CAN YOU ADD TO WHAT KEVIN TOLD US ABOUT INDUSTRY CHALLENGES AND WHERE WE'RE GETTING IT RIGHT AND WRONG. >> THANKS FOR THE OPPORTUNITY. I LEFT ACADEMIA TO TRY AND SPEND SOME TIME IN INDUSTRY, I FELT LIKE AS AN ACADEMIC I HAD BRILLIANT IDEAS, GETTING GRANTS AND WRITING PAPERS AND YET NOBODY WAS USING THEM. AND I REALLY IDENTIFY WITH DAVID'S COMMENT EARLIER ABOUT, YOU KNOW, FEELING LIKE YOU INVENTED THE BEST A.I. ALGORITHM EVER, IT'S NOT GETTING INTO CLINICAL PRACTICE. I COULD BARELY GET MY OWN COLLEAGUES TO USE IT, IT SEEMED LIKE. I WAS CURIOUS WHY THAT IS AND THINK THE LAST MILE METAPHOR RESONATES WELL WITH ME. I OFTEN USE THE METAPHOR OF FARMING, GETTING WATER TO THE END OF THE ROW OF THE IRRIGATION DITCH. I DID TOO MUCH OF THAT WHEN I WAS A TEENAGER. AS I LOOKED AT THAT FROM THE OUTSIDE ACADEMIA INDUSTRY PERSPECTIVE WHAT I REALIZED IS AS WE'VE ALL COME TO RECOGNIZE THERE ARE MANY, MANY PLAYERS TO DLIVER CARE. THERE'S HOSPITALS, PHYSICIANS, THERE'S LOTS OF CAREGIVERS AND TECHNOLOGISTS, ET CETERA. AND WHAT I FOUND, PROBABLY NOW OBVIOUS IN RETROSPECT, IF EVERY PERSON ALONG THE ROW DOESN'T HAVE INCENTIVE TO HAVE THE WATER KEEP MOVING, THE WATER STOPS RIGHT THERE. AND SO EVERY SINGLE PERSON ALONG THE LINE HAS TO WIN BY USING THAT TECHNOLOGY, IN NON-ACADEMIC CENTERS I CAN BE CANDID, THAT'S USUALLY A FINANCIAL SITUATION, BECAUSE THEY DON'T HAVE INTELLECTUAL GOALS FOR THE INSTITUTION, THEY ARE TRYING TO SURVIVE. AND MARGINS IN MOST PLACES ARE NOT GREAT ENOUGH TO DO THINGS IF IT DOESN'T ACTUALLY MAKE FINANCIAL SENSE. BUT THERE ARE NOT ALL FINANCIAL REASONS TO DO THINGS. PATIENT CARE CAN PULL THINGS ALONG EVEN IF IT'S NOT FINANCIALLY INCENTIVIZED. I THINK THAT, TO ME, AS I LOOK AT THE A.I. PROBLEM, WE HAVE TO BE THINKING IF WE'RE GOING TO GET TO THE LAST MILE, WE HAVE TO HAVE SOME REASON FOR EVERYBODY TO USE IT ALL THE WAY. 19% MARKET SHARE FOR THE BIGGEST PACS INSTALLATION, TRYING TO SOLVE THAT FOR THE 50, THERE NEEDS TO BE A LOT OF WIN FOR SOMEBODY TO SPEND ALL THAT MONEY. AND SO I WOULD SAY MAYBE ONE OF THE BIG TAKEAWAYS FOR ME FOR THIS MEETING I HEARD A LOT OF A.I. FOR SCIENTIFICALLY INTERESTING PROBLEMS, FOR IMAGE VOX BUILDER PROBLEMS, RECON AND THINGS LIKE THAT, BUT THE PROBLEM IS WHAT BIBB AND YOU STARTED, EVERYBODY TALKED ABOUT, KIND OF THIS IDEA IT COULD HELP DELIVER BETTER MEDICAL CARE, PRIORITIZING THAT SEEMS LIKE THE REAL GOAL FOR US, WHAT IS THAT BIGGEST OPPORTUNITY TO REALLY HELP HUMAN HEALTH AND THOSE WERE THE AREAS I THINK WHERE WE'RE GOING TO FIND ENOUGH FINANCIAL WIN FOR THE INDUSTRY TO DEVOTE RESOURCES AND HOPEFULLY GET A RETURN ON INVESTMENT. I'LL PAUSE THERE. THANKS. >> WELL, THANK YOU. I WOULD PUT TOGETHER A PANEL THAT DIDN'T HAVE ANY INTEREST, I HAD LIKE 100 QUESTIONS I WAS GOING TO ASK BUT WE GOT A GREAT LINEUP OF QUESTIONS. I'M GOING TO GO STRAIGHT TO THE QUESTIONS FROM THE GROUP, OR COMMENTS. PLEASE. >> HI. MANY SPEECH HAVE SPOKEN ABOUT THE NEED FOR INTEROPERABILITY AT MANY LEVELS ALONG THE DEVELOPMENT CHAIN, SHARING DATA TO HAVING API ACCESS AND BEING INTERRUPTIBLE FOR PACS SYSTEMS, EVERY SINGLE LAYER IS SCREAMING WE NEED TO WORK MORE WITH EACH OTHER BUT IT SEEMS AS AN OUTSIDER TO THIS COUNTRY OF LIKE YOUR BASEBALL WORLD SERIES WHICH IS ONLY OPEN TO AMERICANS REGULATORY SYSTEM IS ALSO ONLY OPEN TO PEOPLE WHO HAVE DONE THAT TESTING ON AMERICN DATA, CAN ONLY USE Y USE YOUR DEVICE IN AMERICAN. I WANT YOUR THOUGHTS ON GLOBALIZATION, ESPECIALLY THE MEDICAL DEVICE SINGLE AUDIT PROGRAM TO GET APPROVAL IN JAPAN, BRAZIL AND CANADA AND MANY OTHER COUNTRIES IN ONE AUDIT. >> YEAH, I'M NOT AN EXPERT ON THE INTERNATIONAL REGULATORY, BUT THAT'S SOMETHING THE AGENCY IS WORKING ON. I THINK IT'S A CHALLENGE BECAUSE SOME OF THE REGULATIONS ARE IN LAWS IN DIFFERENT COUNTRIES, COME INTO SYNERGY FOR THIS TO WORK. WITHIN THE SOFTWARE AND A.I. FRAMEWORK WE'VE BEEN WORKING WITH THE INTERNATIONAL PARTNERS AND REGULATORY AGENCIES TO COME UP WITH A SYNERGISTIC APPROACH, SO I THINK THERE'S HOPE THAT THERE WILL BE SOME MORE CONSISTENCY AND POTENTIALLY HOPEFULLY A SINGLE PROCESS FOR EVALUATING THAT WOULD PROPAGATE TO MULTIPLE COUNTRIES. UTILIZING DATA, THE OTHER ISSUE, IT DOESN'T HAVE TO BE UTILIZED FOR THE FDA PERSPECTIVE, WE DO USE A LOT OF INTERNATIONAL DATA AND OTHER COUNTRY DATA. THE ISSUE IS IT HAS TO BE RELEVANT, SHOWN TO BE RELEVANT TO THE U.S. POPULATION SO THERE'S, AGAIN, IDEAS THAT CAN BE ASSOCIATED WITH THAT BUT I THINK THERE'S THIS PROCESS THAT WE'RE WORKING TOWARDS, I'M NOT SURE IT'S THERE YET BUT HOPEFULLY IN DIGITAL HEALTH PERSPECTIVE ONE PLACE WE DO BETTER AT COORDINATING ACROSS THE WORLD TO GET MORE HETEROGENEOUS SYSTEM FOR HOW WE EVALUATE A.I. >> TWO PANELISTS MENTIONED MY FAVORITE TWO WORDS, OPEN SOURCE. AND BOTH SAID IT WOULD BE GREAT IF WE HAD MORE OPEN SOURCE TOOLS AND ALSO TALKED ABOUT THE TOOLING THEY INVENTED AND DEVELOPED IN THEIR OWN SHOPS. SO CAN YOU TALK ABOUT THE BARRIERS TO RELEASING THE THINGS THAT YOU PROVIDED OR THE DECISION PROCESS THAT GOES INTO DECIDING NOT TO RELEASE THOSE AS YOU'RE ASKING FOR MORE OPEN SOURCE IN THE WILD? THANKS. >> YEAH, I THINK I'LL START THAT OFF BY SAYING I HAVE A BIT OF INTERESTING PERSPECTIVE ON THAT AS SOMEBODY THAT, AGAIN, IS VERY MUCH AN ENGINEER AT HEART AND HAS MOVED FROM THE MORE LEAD SCIENTIST POSITION TO CEO POSITION, AND HAVE TO STRUGGLE WITH THE DIFFERENT WAYS OF LOOKING AT EXACTLY THAT KIND OF PROBLEM. SPEAKING ON BEHALF OF MYSELF, NOT ON BEHALF OF THE COMPANY, I'M INCREDIBLY IN FAVOR OF OPEN SOURCE AND HISTORICALLY HAVE CONTRIBUTED TO A LOT OF OPEN SOURCE AND WOULD LIKE TO CONTINUE THAT KIND OF TREND. I THINK THAT IN AN INDUSTRY LIKE THIS, IT'S INHERENTLY BENEFICIAL TO SPARK EVEN THINGS THAT ADD MORE COMPETITION BECAUSE COMPETITION BREEDS INNOVATION, AS SOMEBODY WHO PROBABLY WILL ALSO BE A PATIENT I WOULD FEEL BETTER KNOWING THAT CLINICAL A.I. USED IN DIAGNOSING MY OWN SCANS IS BUILT THE RIGHT WAY, AND SO I THINK THAT THERE ARE A LOT OF ARGUMENTS IN FAVOR OF OPEN SOURCING TOOLS LIKE THIS, AND THE LARGE ARGUMENTS AGAINST IT DON'T REALLY TAKE INTO ACCOUNT THE FULL PICTURE. I MEAN, BUILDING ANY KIND OF CLINICAL A.I. IS AN ENORMOUS AMOUNT OF WORK, HAVING THE PROPER TOOLS MEANS YOU HAVE A FAIR SHOT TO THEN PUT IN THAT ENORMOUS AMOUNT OF WORK. SO THERE'S STILL HUGE BARRIERS TO ENTRY THAT I THINK CAN STILL JUSTIFY A COMPANY LIKE OURS PROVIDING OPEN SOURCE MATERIALS, THAT SAID IT'S A CONSTANT UPHILL BATTLE BECAUSE NOT EVERYBODY THAT'S IN THE POSITION OF SIGNING OFF ON THOSE KINDS OF DECISIONS HAS THE SAME KIND OF BACKGROUND IN SOFTWARE DEVELOPMENT OR SAME KIND OF APPRECIATION FOR THE CHALLENGES THAT WILL STILL EXIST FOR THE MARKET EVEN IF THEY HAVE ACCESS TO THOSE TOOLS. SO I WOULD SAY THAT THE REAL CHALLENGES ARE FAR MORE POLITICAL THAN ANYTHING ELSE. >> I THINK THE OTHER ONE WAS ME, IS THAT RIGHT? SO, IN LOOKING AT SAY ACADEMIC INSTITUTION TO CREATE OPEN SOURCE TOOLS THE LIMIT IS HERE IS WHAT WE CREATED FOR OURSELVES AND HERE IS WHAT'S AVAILABLE TO THE WORLD. I DON'T KNOW THAT THAT'S ENOUGH. IN THE PROCESS OF THEM LOOKING AT THE OPEN SOURCE TOOLS AVAILABLE THEY THOUGHT THEY WEREN'T OPEN ENOUGH TO MAKE MODIFICATIONS TO USE SO I THINK EVERYONE IS EAGER TO MAKE TOOLS SHAREABLE BUT THERE NEEDS TO BE SOMEBODY WITH A LARGE AS YOU SAY POLITICAL WIN, AMOUNT OF CASH, THAT WOULD SAY WE'RE GOING TO CREATE AN OPEN SOURCE TOOL TO CURATE, GO FORWARD WITH, AND ALSO BE MINDFUL OF THE FACT THAT HOPEFULLY EVERYONE GOT THE IDEA THAT IT'S NOT JUST THE SOFTWARE THAT'S MISSING, IT'S EVEN THE DEFINITION OF THE SOFTWARE. I SPENT LUNCH TIME TODAY USING THE TERMS THAT SIT IN THE TOOL, I THINK THERE NEEDS TO BE A WHOLE LEVEL GRASS ROOTS FROM THE BOTTOM UP FROM A CENTRALLY RESPECTED ORGANIZATION, NIH, THAT COULD MAYBE, OR SOMEBODY LIKE A GOVERNMENT AGENCY TO LOOK AND SAY HERE IS WHAT NEEDS TO BE DONE AND WILL HELP SUPPORT THAT ACTIVITY. MAYBE IT'S MY BIAS BECAUSE I'M LOOKING FROM THE OTHER WAY, BUT CLEARLY WITHOUT A DOUBT OPEN SOURCE SOLUTIONS NEED TO BE THERE AS WELL AS STANDARDS NEED TO BE THERE TO MOVE US FORWARD FASTER. >> CAN I RESPOND TO THAT OPEN SOURCE REALLY QUICK? NICK FORTUNATELY IS NOT READING HIS TEXT RIGHT NOW BUT I SENT HIM ONE TO REMIND HIM ABOUT A PROGRAM AT THE FDA, MEDICAL DEVICE DEVELOPMENT TOOL PROGRAM. IT IS A PROGRAM THAT IN THE END YOU DID GET A STAMP OF FDA QUALIFICATION FOR A TOOL AND THEN DISSEMINATE IT, WHEN IT'S USED IN FDA SUBMISSION THERE WOULDN'T NEED TO BE DISCUSSION BECAUSE THAT WOULD BE TAKEN CARE OF IN THE QUALIFICATION PROCESS SIMILAR TO THE DRUG BIOMARKER QUALIFICATION, TO DO SEGMENTATION OR WHATEVER, THAT'S A VIABLE TOOL THAT COULD FIT INTO THIS PROGRAM. LIKEWISE, A PROJECT THAT I'M TRYING TO MOVE FORWARD IN THE PATHOLOGY SPACE IS DATA AS A TOOL, IMAGES PLUS ANNOTATIONS COULD BE QUALIFIED, AND DISCUSSED WITH THE PROGRAM PEOPLE, LIAISON WITH THAT GROUP, SOME KIND OF TOOL THAT THEY WOULD BE INTERESTED IN. PLEASE CONTACT NICK AND I, MDDT, MEDICAL DEVICE DEVELOPMENT TOOL PROGRAM. >> NO, I THINK THAT -- I'M SORRY. I WAS GOING TO SAY THE MDDT PROGRAM IS CERTAINLY VERY INTERESTING TO INCORPORATE INTO USE CASE DEVELOPMENTS TO HAVE A STANDARDIZED WAY OF LOOKING AT ALGORITHMS TO DO DETECTION, CLASSIFICATION, SEGMENTATION AND CREATE STANDARDS. ONE THING ABOUT THE OPEN SOURCE PIECE, NUMBER ONE, I THINK SAY YOU WANT AN OPEN SOURCE ANNOTATION TOOL, AND THERE'S SOME ACTUALLY THAT ARE OUT THERE, ONE OF THE PROBLEMS OF AN INSTITUTION OR DEVELOPER IS GIVING AWAY WHAT THEY USE IN HOUSE, WHO IS GOING TO SUPPORT IT? WHO IS GOING TO HAVE THE HELP DESK, THE STUFF? MAKING IT AVAILABLE, YOU KNOW, IS A LITTLE BIT DIFFICULT FROM THOSE THINGS SO WE RECOGNIZE THAT. I THINK HAVING THE STANDARDS OF HOW YOU USE THE TOOLS SO THAT ANY TOOL THAT YOU WOULD USE FOR ANNOTATION WOULD DO IT TO CERTAIN SET OF STANDARDS I THINK IS SOMETHING THAT OUGHT TO BE OUT THERE THAT EVERYBODY SHOULD SEE THAT WOULD BE PART OF THESE COMMON DATA ELEMENTS THAT WE'RE DEVELOPING. SO PLEASE. >> QUICK QUESTION FOR NICK. ARE ALL THESE A.I. ALGORITHMS BEING DEVELOPED BY DIFFERENT COMPANIES, ARE THEY CLASS 2 DEVICES FOR FDA APPROVAL OR DIFFERENT? >> I MEAN, I'M NOT CLEAR. ALL OF A.I. IN GENERAL I'M NOT SURE. THERE'S SOME THAT GOES IN CLASS 2 AND 3 BUT MOST RADIOLOGICAL TOOLS, ALMOST ALL ARE CLASS 2 WITH THE ONLY EXCEPTION BEING BREAST CAD-E DEVICES WHICH IS IN THE PROCESS OF LOOKING FOR RECLASSIFICATION DOWN TO CLASS 2, MORE OF A HISTORICAL REASON. THERE MAY BE SOME HIGH RISK OR HAVE UNIQUE PROPERTIES THAT MAY BE CLASS 3 BUT FOR THE MOST PART CLASS 2 OR FALLING BELOW WHERE IT'S NOT NECESSARILY A STRONG RISK. >> QUICK COMMENT, KEITH, I FEEL BAD BECAUSE OUR PRACTICE ALL THE RADIOLOGISTS DICTATE LUMBAR SPINES EXACTLY THE SAME AND THE SPINE SURGEONS AGREE EVERY TIME. [LAUGHTER] >> WHAT A COINCIDENCE, SAME AS OURS. >> PLEASE. >> MY QUESTION RELATES TO THAT EXACTLY THAT POINT, IN BEING FROM AN ORGANIZATION THAT WE HAVE BEEN DEVELOPING A.I. TOOLS, AND IMPLEMENTING THEM IN THE CLINICAL SETTING, ONE THING I SEE IS A HUGE IMPORTANCE OF HAVING THE RADIOLOGIST ANALYZING THE OUTPUT OF THAT NETWORK OR OF THAT ALGORITHM. HOWEVER, THERE'S SITUATIONS THAT I'M NOT SURE THE RADIOLOGIST KNOWS WHAT TO DO WITH THAT INFORMATION. ONE OF THE THINGS THAT WE'RE DOING, FOR EXAMPLE, IS EXTRACTING THE VOLUMETRY OF THE BRAIN. I'M NOT SURE I'M EMPOWERED TO ARGUE AGAINST THE A.I. TOOL TELLING ME THERE'S A 16% DIFFERENCE IN THE VOLUME OF THE BASAL GANGLIA OR, YOU KNOW, THAT'S VERY HARD FOR A HUMAN TO EVALUATE VOLUMETRIC ANALYSIS, SIMILARLY LUMBAR SPINE ANALOGY, THERE ARE PAPERS OUT THERE, THAT SAYS THAT ABOUT .26 KAPPA AGREEMENT BETWEEN THE READERS FOR ANALYSIS OF STENOSIS OF LUMBAR SPINE STENOSIS, TO WHAT AGREE WE ADD BIAS INHERENT OR VERSUS IMPROVED THE ABILITY TO DETECT MORE AND MORE, IS IT GOING TO BE DISEASE SPECIFIC OR IS IT GOING TO BE A GENERIC APPROACH? I THINK THOSE HAVEN'T BEEN ANSWERED, BUT I WANT TO SEE YOUR OPINION. >> I'LL GIVE YOU MINE. I THINK YOU'RE EXACTLY RIGHT. I DON'T THINK THERE'S A SINGLE ONE SIZE FITS ALL ANSWER, TO HOW MUCH DATA YOU NEED, COMPONENTS OR STRUCTURES AMENABLE, VIABLE PRODUCTS. AND ALSO THERE'S NO STANDARD WAY AS TO SAY WE NEED SEVEN RADIOLOGIST SUBSPECIALIZED OR TWO. EVEN BEYOND THAT WHEN YOU GET RICH DATA BACK, SHOULD YOU TRUST IT AS GROUND TRUTH OR NOT. THE ADVANTAGE WE'RE GOING TO HAVE LOOKING YEARS FORWARD IF WE DO COLLECT THIS DATA PROSPECTIVELY VERBS US RETROSPECTIVELY WE'LL HAVE BENEFIT OF SAYING HERE'S THE LARGE DATA THAT'S COMING IN, SO OF THE 100 RADIOLOGISTS BRINGING IN DATA, THESE ARE THE ONES THAT SEEM TO BE IN SYNC WITH THE ALGORITHMS, THESE FOLKS ARE NOT. THAT MEANS THERE'S SOMETHING OF INTEREST THERE THAT YOU NEED TO LOOK AT FURTHER. RIGHT NOW THERE ISN'T ENOUGH RICH DATA BECAUSE AS WAS MENTIONED BEFORE BY KEVIN AND OTHERS THAT YOU ONLY HAVE SO MUCH MONEY TO SPEND ON SO MANY RADIOLOGISTS TO GET SO MUCH DATA SO YOU MAKE DECISIONS THAT ARE SPARSE OF ACTUAL KNOWLEDGE YOU NEED. SO I DON'T THINK THERE'S AN ANSWER THAT YOU CAN PREDICT TO SAY ADD ALL THIS NEW DATA BECAUSE IT'S PERFECT. WE'LL HAVE TO LOOK AT THE DATA AS IT COMES IN WITH DEEPER DATA SETS THAN WE HAVE TODAY. >> I'M GOING TO ADD ONE THING TO THE IDEA ABOUT WHAT RADIOLOGISTS CAN'T SEE. WE IN PARTICULAR, WHEN PUTTING THE CONFERENCE TOGETHER, DECIDED TO STICK WITH THE IMAGE ANALYSIS, IMAGE INTERPRETATION, AND WE DIDN'T REALLY GO MUCH INTO RADIOMICS, POPULATION HEALTH MANAGEMENT AND OTHER THINGS BUT IMAGINE THE DAY WHEN A PATIENT COMES FOR TRAUMA AND THEY HAVE AN EXAMINATION OF THE CHEST AND ABDOMEN, IN THE BACKGROUND A.I. IS CALCULATING THE FAT VOLUME IN THE LIVER, THE DEGREE OF PULMONARY EMPHYSEMA AND WE KNOW THERE'S METRICS WHERE INTERVENTION CAN ABSOLUTELY HELP THESE PATIENTS. WELL, WHILE THEY ARE THERE, I COULD SAY THE PATIENT HAS PULMONARY EMPHYSEMA OR FATTY LIVER, BUT IMAGINE THE DAY A QUANTIFIABLE MESSAGE COULD BE SENT BY FIRE TO THE EHR, DELIVERED TO THE PROBLEM LIST OF THE PATIENT FOR IMPORTANT THINGS WE FOUND INCIDENTALLY FROM IMAGING THAT BECOME IMPORTANT TO PATIENT CARE SO I THINK TO ANSWER YOUR QUESTION ABOUT WHAT ARE WE GOING TO DO WITH THESE THINGS THAT RADIOLOGISTS HAVE TROUBLE WITH, I THINK IT GOES BEYOND JUST THAT, CAN GO TO CLINICAL CIRCUMSTANCES AND STUFF AND HOW WE CAN MAKE ALL THIS STUFF INTEROPEARABLE SEEMS A LAUDABLE GOAL FOR THE FUTURE. ALL WE HAVE TO DO IS GO FROM HERE TO THE TERRACE FOR THE RECEPTION. WE'VE GOT A COUPLE MORE QUESTIONS THAT I WOULD HOPE ASKERS AND ANSWERS WOULD BE QUICK AND WE'LL DO BOTH AND ADJOURN. >> I'M HORRIBLE AT BEING QUICK, I'M SORRY. I'M NOT A RADIOLOGIST, SO I'LL APOLOGIZE TO THE RADIOLOGISTS, BUT WHEN WE'RE TALKING ABOUT GROUND TRUTH I WONDER EVEN IF WE HAVE 20 RADIOLOGISTS WHETHER THAT'S THE WAY TO ANNOTATE THE DATA. WE TALKED EARLIER HOW YOU CAN TRAIN YOUR A.I. TO MATCH ANY GIVEN RADIOLOGIST BIAS, IF I'M THE FATHER OF THE PREEMIE PART OF THE DATASET WITH ROP, I DON'T HONESTLY CARE WHICH RADIOLOGIST THE A.I. IS ABLE TO APE, WHAT I WANT TO KNOW IS DOES MY BABY HAVE ROP, THAT'S WHAT MATTERS TO ME NOW. ALL DUE RESPECT TO RUDY GUILIANI, TRUTH IS TRUTH. MY BABY DOES OR DOESN'T HAVE ROP, RIGHT? SO HOPEFULLY THERE'S SOME WAY AS WE'RE GOING FORWARD TO INCREASINGLY APPEND PATIENT OUTCOMES, TO APPEND REAL DIAGNOSES AND WHAT'S THERE SO I'M A RESEARCH CHEMIST AT THE NATIONAL INSTITUTE OF STANDARDS AND TECHNOLOGY, SO I KEEP THINKING A LOT OF TIMES WHAT WE'RE MEASURING IS ACTUALLY SOMETHING REAL AND PHYSICAL. AND WE CAN GIVE THAT TO THE A.I. IN SOME CASES WHERE WE'RE ABLE TO GET IT. MAYBE THAT'S THROUGH PATHOLOGY. OR WHETHER THAT'S THROUGH CALIBRATION DATA, PHANTOM IMAGES, SOMETHING ELSE, THAT CAN BE APPENDED TO THE CLINICAL IMAGES THAT ARE BEING FED INTO THE A.I. SO I WANTED TO THROW THAT OUT THERE AS ANOTHER VERSION OF GROUND TRUTH AS IT WERE. >> I'LL JUST ANSWER BRIEFLY THAT I THINK THAT'S THE PURPOSE OF THESE REGISTRIES, EXPANDED NOTION TO PROVIDE ADDITIONAL SCIENCE TO UNDERPIN OUR DECISIONS. SO YOU'RE EXACTLY RIGHT. >> AND I WOULD TOUCH ON THIS TOO, A NON-RADIOLOGIST WHO SPENDS A LOT OF TIME STRUCK LING WITH THIS EXACT PROBLEM HOW TO ORCHESTRATE LARGE TEAMS OF RADIOLOGISTS TO ARRIVE AT SOMETHING RESEMBLING GROUND TRUTH IN MANY CASES, CASE-BY-CASE BASIS, SOMETIMES HAVING A LOT OF OPINIONS, IS THE PROPER WAY TO LOOK AT IT. OTHER TIMES THERE'S HIGHER ORDER OF FOLLOW-UP THAT CAN SERVE AN EVEN BETTER PURPOSE. IN INTERPRETING CHEST X-RAYS IN A SCREENING SETTING, A LOT OF TIMES THE BEST YOU CAN HOPE FOR IS A LOT OF DIFFERENT OPINIONS AND PANEL CONSENSUS IS THE BEST AVAILABLE GROUND TRUTH BUT IF YOU'RE TRYING TO DIAGNOSE LUNG CANCER FROM A CHEST CT YOU CAN TRAIN YOUR MODEL ON PATIENTS WITH FOLLOW-UP SCANS AND ULTIMATELY A BIOPSY INDICATING WITH A MUCH HIGHER ACCURACY WHETHER BENIGN OR MALIGNANT. WE'VE HAD TO STRUGGLE TO ANSWER QUESTIONS WITH EXPERIMENTATION, TO THE POINT WHERE EACH SPECIFIC TASK REQUIRES 100 PAGES OF GUIDELINES WITH VERY SPECIFIC EXAMPLES TO TRAIN OUR RADIOLOGISTS ON HOW TO DO EXACTLY THAT. >> I JUST WANT TO RESPOND TO THE QUESTION ABOUT THE ROP. ONE OF THE CONCERNS -- WE WANTED TO DO EXACTLY THAT, THE TRUTH IS SOMETHING OF -- MUCH BEYOND WHAT SOMEBODY CALLS IT. WE WOULDN'T KNOW WHAT HAD HAPPENED TO THAT BABY HAD THEY NOT BEEN TREATED BY THAT PHYSICIAN AT THAT TIME SO WE DON'T KNOW, THIS PERSON CALLED IT HERE, TREATED IT HERE, THIS CALLED IT HERE AND TREATED HERE, WE HAVE NO WAY OF KNOWING WHAT HAPPENED IF THIS PERSON TREATED THERE OR THIS PERSON TREATED HERE, WE WANT TO DO THAT BUT THAT'S -- >> SO BIBB I WOULD SAY AS A SPECIALTY WE AS RADIOLOGISTS NEED SOME EARLY SUCCESSES FOR A.I., AND THESE KINDS OF QUESTIONS SORT OF GUIDE US AS TO WHAT KINDS OF QUESTIONS WE HAVE HOPE OF ANSWERING. >> THE ABILITY TO EVENTUALLY INCORPORATE PATHOLOGIC DATA INTO OUR UNDERSTANDING OF GOING BACK AND LETTING THAT AFFECT A.I. ALGORITHMS WILL TEACH MORE ABOUT OUR ABILITY TO MAKE A PATHOLOGY DIAGNOSIS WHEN WE SEE A SPECIFIC LESION RATHER THAN A DIFFERENTIAL DIAGNOSIS. >> I WOULD ADD ONE ISSUE THERE IS VARIABILITY IN THIS REFERENCE OR TRUTH, MAYBE THERE'S AN UNDERLYING TRUTH THAT EVERYONE CAN KNOW AND GET THROUGH PATHOLOGY OR SOME OTHER METHOD BUT EVEN PATHOLOGY HAS VARIABILITY. IF THE A.I. NOT JUST SAY CANCER, NON-CANCER, BUT PUT AIR BARS OR -- ERROR BARS OR METRICS, THAT COULD BE USEFUL AND MAY BE AN APPROACH. >> I THINK I SAW ON AN EARLIER SLIDE THERE WAS SOME LARGE AVAILABLE THROUGH ACR COLLECTED IN THE PAST. HOW MUCH OF THAT DATA COULD BE RELEASED TO OTHER SOCIETIES FOR TESTING? I SAW NUMBERS LIKE 45 MILLION, I DON'T KNOW IF I MISREAD IT, WOULD THERE BE PLANS FOR RELEASING THAT DATA? EVERYONE IS ASKING FOR DATA. I'M SURE IT'S WELL TREATED BECAUSE YOU'RE IN CHARGE OF IT. LIKE WOULD YOU HAVE PLANS TO RELEASE BY RSNA AND NIH, ARCHIVES AVAILABLE, IF THAT WAS MADE AVAILABLE BY YOUR INSTITUTION, LIKE DATA SCIENCE INSTITUTE, WOULD SERVE OR CAN BE DONE LIKE IN BATCHES LIKE, OKAY, WE'RE TRAINING FOR CARDIAC, TRAINING FOR BRAIN, AND IT'S WELL TREATED. WE STAND BEHIND IT. AND IT MAY TURN INTO A GREAT RESOURCE FOR ALMOST TRYING TO DEVELOP IN THIS SPACE. >> I'LL START. THE 45 MILLION NUMBER THAT YOU SAW WERE RELATED TO OUR WHOLE REGISTRY PROGRAM, 99% OF THE REGISTRY INFORMATION THAT WE HAVE RIGHT NOW IS ABOUT FOR CLINICAL DATA REGISTRIES, SO IT'S LIKE DOSE PARAMETERS ON A CT SCAN OR OTHER CLINICAL PERFORMANCE TYPE MEASURES THAT AREN'T ASSOCIATED WITH IMAGES. NOW, THE COLLEGE DOES HAVE AN ACCESS TO A LIBRARY OF IMAGES THROUGH THE CLINICAL TRIALS PROGRAM THAT WE HAD, AND IN FACT SOME OF THOSE DATA FROM THE LUNG CANCER SCREENING TRIALS HAD BEEN RELEASED TO THE PUBLIC FOR TRAINING AND TESTING AND USE LIKE THAT, AND WE'RE ALWAYS EXAMINING WHAT OUR DATA USE AGREEMENTS ARE WITH THE NIH, THE INSTITUTIONS, TO SEE WHETHER OR NOT WE MIGHT BE ABLE TO PROVIDE SOME OTHER DATA, AGAIN IN A FAIRLY LOOSELY ANNOTATED FORM BUT POTENTIALLY. >> THE OTHER THING I YOU WANT TO MENTION WE'RE FORTUNATE TO HAVE RON SUMMERS HERE, YOU CAN TALK ABOUT THE DATA NIH RELEASED, EFFORTS IN VARIOUS AREAS. >> SO WE RELEASED TWO DATASETS, CHEST X-RAY DATASET AND DEEP LESION CT DATASET. EACH DATASETS ARE QUITE LARGE, CHEST IS 110,000, DEEP LESION SET IS SOMETHING LIKE OVER 4,000 PATIENTS AND 10,000 CTs AND 32,000 ANNOTATED LESIONS. ONE OF THE CHALLENGES IN RELEASING TWO DATASETS WAS I WANTED TO MAKE SURE THAT WE MINIMIZED THE RISK OF RELEASING ANY PROTECTED HEALTH INFORMATION AND SO I SPENT A LOT OF TIME TRYING TO FIGURE OUT HOW TO DO THAT BECAUSE WE SOMETIMES FOUND INFORMATION IN THE PIXEL DATA ITSELF, ONE OF THE PRESENTERS MENTIONED THAT, THAT THEY HAVE A WAY I GUESS TO FIND INFORMATION IN THE PIXEL DATA WHICH I'D LOVE TO TALK ABOUT HOW THEY DID IT. WE ACTUALLY USE OPTICAL CHARACTER RECOGNITION ON IMAGES TO TRY TO FIND TEXT, IN THE PROCESS FOUND THINGS LIKE EMBEDDED DATE AND TIME INFORMATION IN THE PIXELS, CASSETTE NUMBERS, AND EVEN PATIENTS WHO ARE WEARING JEWELRY THAT HAD THEIR NAME ON IT AND IT WAS IN THE IMAGE. SO THEN THE POSTDOCTORAL FELLOW TOOK THE LEAD, HAD TO CUT OUT ALL THIS INFORMATION FROM THE IMAGES. FOR THE CT SCANS IT WAS A DIFFERENT SITUATION. THESE IMAGES TYPICALLY DON'T HAVE INFORMATION BURNED INTO THE PIXELS. BUT ON THE OTHER HAND WE STILL HAD THE JEWELRY PROBLEM. MANY PATIENCE WERE WEARING NIH ID BADGE DURING THE CT SCAN ITSELF, WHICH WE HAVE A QA ISSUE. I DON'T KNOW FANTASTIC POSSIBLE TO RECONSTRUCT A NAME FROM AN NIH BADGE BUT IT WASN'T GOING TO RISK THAT. IN BOTH CASES, I HAD MY TEAM, IN SOME CASES WITH THE HELP OF ANOTHER LAB REVIEW THE IMAGES MANUALLY SO SOMEBODY LOOKED AT EACH OF THE IMAGES MANUALLY, IN THE CASE OF THE CHEST X-RAY IMAGES WE HAD TWO PEOPLE LOOK AT EACH IMAGE, MORE DIFFICULT WITH THE CT SCANS BECAUSE THEY ARE VOLUMES, AND IN THAT CASE WHAT WE DID WAS TOOK CORONAL REPROJECTIONS OF THE CT SCANS AND REVIEWED THOSE AND THE OTHER THING WE DID WAS WE LOOKED AT SAGITTAL REFORMATTED MIDLINE SAGITTAL IMAGES. OH, I REMEMBER WHY WE DID THAT. THE REASON WAS I DIDN'T WANT TO HAVE ANY FACES IN THE DATASET BECAUSE THEY COULD BE 3D RECONSTRUCTED SO WE REVIEWED THE SAGITTAL MIDLINE IMAGES AND IF THERE WAS EITHER THE MOUTH AND NOSE OR NOSE AND EYES, BECAUSE THESE WERE CHUNKS, THEY DIDN'T TYPICAL INCLUDE ALL THREE FACIAL FEATURES, WE WOULD CUT OFF THE FRONT OF THE FACE. FROM THE IMAGES. THOSE ARE THE PAIN POINTS. I SHOULD ADD THIS PROJECT WAS DONE THROUGH MY LAB BUDGET, THERE WAS NO ADDITIONAL FUNDING FOR THIS EFFORT, I THANK THE NIH CLINICAL CENTER FOR PROVIDING MY LAB BUDGET, BUT I THINK EVERYBODY HERE CAN APPRECIATE HOW WHEN YOU HAVE A MULTI-MILLION DOLLAR FUNDED STUDY WHERE YOU HAVE STATISTICIANS AND OTHER SUPPORT PERSONNEL COLLECTING DATA, ACCORDING TO STANDARDIZED PROTOCOLS AND COLLECTING IMAGES, STANDARDIZED PROTOCOLS, THIS IS A TREATMENT OPPORTUNITY FOR CREATING DATASETS THAT DEAL WITH LOTS OF IMAGES AND DEAL WITH SOME OF THE ISSUES THAT CAN COME UP LIKE ANONYMIZATION AND METADATA AND ANNOTATIONS AND LABELS, THINGS LIKES THAT. IT'S REALLY, REALLY DIFFICULT FOR ANY ONE LAB TO CHECK ALL THE BOXES ON CREATING A DATASET THAT'S USEFUL FOR ALL PURPOSES. WE RELEASED THESE DATASETS FROM THE PERSPECTIVE OF AN A.I. DEVELOPMENT LAB THAT WAS TRYING TO TEACH COMPUTERS HOW TO IDENTIFY DISEASES ON THE IMAGES. WE DID NOT RELEASE THE DATASET WITH THE PERSPECTIVE OF SOMEBODY DEVELOPING ALL TIMES OF IMAGING, VENDORS EQUIPMENT, SO WE DIDN'T INCLUDE INFORMATION LIKE WHICH EQUIPMENT WAS BEING USED, WE TRIED TO LIMIT ALL KINDS OF INFORMATION THAT COULD BE USED RETRO RETROSPECTIVELY TO FIGURE OUT WHO PATIENTS WERE. I DIDN'T WANT TO HAPPEN. WE DID PROVIDE SERIAL IMAGING FOR PATIENTS WHO HAD SERIAL IMAGING AND PROVIDED THE CARDINAL ORDER OF THE IMAGES SO YOU KNEW WHICH ONE CAME FIRST, SECOND, THIRD, FOURTH AND SO ON. WE DID INCLUDE TIME DELAY, THINGS LIKE THAT, BETWEEN IMAGES. SO THE LIMITATIONS IN THE DATASET ARE THERE FOR A REASON, AND THE PRIMARY REASON IS TO LIMIT THE RISK. SO THERE ARE LOTS OF -- IT'S VERY INTERESTING. I FOUND THAT WHEN YOU RELEASE A BIG DATA SAID LIKE THAT ALL THE CRITICS COME OUT OF THE WOOD WORK, AND THERE'S ABOUT A MILLION AND ONE CRITICISMS THAT HAVE BEEN APPLIED, BUT WHAT'S INTERESTING TO ME IS THAT DESPITE THE CRITICISM SO MANY RESEARCHERS HAVE FOUND LOTS OF GREAT APPLICATIONS FOR THIS DATASET, AND, FOR EXAMPLE, I WAS TALKING TO MARK COLLIER EARLIER, TELLING ME ABOUT AN RSNA CHALLENGE TO USE THE DATASET, IT'S HARD TO BE WHAT EVERYBODY WANTS YOU TO BE, GIVEN THE AMOUNT OF MONEY. EVERY DATA RELEASE WILL BE LIMITED FINANCIALLY IN SOME WAY OR ANOTHER, BUT IF THERE WERE A WAY FOR THE COMMUNITY TO COME TOGETHER, SOME SPEAKERS SAID STANDARDIZE PRIVACY ISSUES, I THINK THE GENTLEMAN FROM ENLITIC MENTIONED STANDARDIZE SO THEY DON'T VARY, STANDARDIZE IRB REQUIREMENT ISSUES SO IT WAS EASY TO SHARE ANONYMIZED DATA, THE IDEA OF STANDARDS WITH MANY ASPECTS OF MEDICINE IS CRITICALLY IMPORTANT. >> THANK YOU VERY MUCH. AND SO ON BEHALF OF OUR PANEL, FOR THE LAST MILE PANEL, AND THEN ON BEHALF OF CURT, KRIS AND ME, WE WANT TO THANK ALL OF YOU FOR YOUR ATTENDANCE TODAY AND ALL OF YOUR GREAT INPUT AND FEEDBACK, AND LOOK FORWARD TO CONTINUING TOMORROW AND IMMEDIATELY NOW FOLLOWING THERE'S A RECEPTION ON THE TERRACE, AND SO WE HOPE YOU ALL WILL ENJOY THAT AND CAN NETWORK AND FEEL FREE TO COME UP TO US DURING THE COURSE OF THAT OR TOMORROW AND SHARE MORE OF YOUR OPINIONS AND FOR THOSE OF YOU WHO WATCHED VIA THE INTERNET, THANK YOU VERY MUCH FOR YOUR ATTENTION TODAY AS WELL. WE APPRECIATE IT. CHRIS, DO YOU HAVE ANY LAST COMMENTS OR ANYTHING? WE'RE ADJOURNED TILL TOMORROW MORNING. THANK YOU.