Welcome to the Clinical Center Grand Rounds, a weekly series of educational lectures for physicians and health care professionals broadcast from the Clinical Center at the National Institutes of Health in Bethesda, MD. The NIH Clinical Center is the world's largest hospital totally dedicated to investigational research and leads the global effort in training today's investigators and discovering tomorrow's cures. Learn more by visiting us online at http://clinicalcenter.nih.gov TODAY, WE WELCOME TWO COLLEAGUES TO SPEAK ON ARTIFICIAL INTELLIGENCE AND THE USE IN PROSTATE CANCER DIAGNOSIS. THE FIRST SPEAKER IS DR. RONALD SUMMERS IN THE IMAGING BIOMARKERS AND COMPUTER-AIDED DIAGNOSIS LABORATORY IN THE DEPARTMENT AT THE NIH CLINICAL CENTER. HIS RESEARCH INTERESTS INCLUDE DEEP LEARNING, VIRTUAL COLONOSCOPY, COMPUTER-AIDED DIAGNOSIS AND THE DEVELOPMENT OF RADIOLOGIC DATABASES. HIS EXPERTISE ARE THROWER -- THORACIC RADIOLOGY. HE EARNED HIS DEGREES FROM THE UNIVERSITY OF PENNSYLVANIA AND COMPLETED AN MEDICAL INTERNSHIP AT THE PRESBYTERIAN OF PENNSYLVANIA HOSPITAL IN PHILADELPHIA FOLLOWED BY A RADIOLOGY RESIDENCY AT THE UNIVERSITY OF MICHIGAN HOSPITAL IN ANN ARBOR AND MRI FELLOWSHIP AT DUKE UNIVERSITY MEDICAL CENTER. IN 1994 HE JOINED THE RADIOLOGY AND IMAGING DEPARTMENT AS A STAFF RADIOLOGIST WHERE HE DIRECTS THE IMAGING BIOMARKERS AND COMPUTER-AIDED DIAGNOSIS LABORATORY AND THE FOUNDING CHIEF OF THE NIH CLINICAL IMAGE PROCESSING SERVICE. A POSITION HE HELD UNTIL 2018. DR. SUMMERS' MEMBERSHIPS INCLUDE THE RADIOLOGICAL SOCIETY OF NORTH AMERICA, THE SOCIETY OF ABDOMINAL RADIOLOGISTS AND THE MEDICAL IMAGE COMPUTING AND INTERVENTION SOCIETY AND PART OF THE JOURNAL OF MEDICAL IMAGINING, RADIOLOGY AND ARTIFICIAL INTELLIGENCE AND ACADEMIC RADIOLOGY AND SERVED PREVIOUSLY ON THE EDITORIAL BOARD OF RADIOLOGY. IN ADDITION TO PUBBISHING -- PUBLISHING ORIGINAL MANUSCRIPTS HE HAS 14 PATENTS RELATED TO COLONOSCOPY AND OUR SECOND SPEAKER IS AN ASSOCIATION RESEARCH PHYSICIAN WITH THE MOLECULAR IMAGING PROGRAM AT THE NATIONAL CANCER INSTITUTE CENTER FOR CANCER RESEARCH. DR. TURKBEY EARNED HIS DEGREE IN ANKARA, TURKEY WHERE HE SPECIALIZED IN RADIOLOGY AND JOINED THE MOLECULAR IMAGING PROGRAM IN 2007. FIRST AS A RESEARCH FELLOW AND THEN A STAFF CLINICIAN. DR. TURKBEY'S INTERESTS INCLUDE MULTI-MODAL PROS CANCER IMAGING AND CANCER BIOPSY TECHNIQUES AND PROSTATE CANCER AND IMAGE PROCESSING INCLUDING ARTIFICIAL INTELLIGENCE AND DECISION-SUPPORT SYSTEMS. HIS AREAS OF CLINICAL EXPERTISE INCLUDE PROSTATE CANCER AND LYMPHATIC IMAGING AND EXPERIMENTAL MRI AND PART OF THE PROSTATE IMAGINING AND REPORTING SYSTEM AND THE COMMITTEE AND THE CO-CHAIR OF THE SOCIETY OF ABDOMINAL RADIOLOGIES PROSTATE DISEASE FOCUSSED PANEL AND SERVED ON THE AMERICAN COLLEGE OF URINARY APPROPRIATENESS COMMITTEE AND HAS AUTHORED 300 PUBLICATIONS IN THE BIOMEDICAL LITERATURE. LET'S BEGIN BY WELCOMING DR. SUMMERS. >> HELLO AND THANK YOU ALL FOR ATTENDING. IT'S A GREAT PLEASURE TO SPEAK TO YOU ON THE IMPACT OF ARTIFICIAL LEARNING ON RADIOLOGY. HERE'S MY DISCLOSURES. CAN EVERYBODY HEAR ME OKAY? CAN YOU HEAR ME IN THE BACK? OKAY. ALL THE WORK THAT I'M GOING TO SHOW TODAY WAS DONE AT THE NIH MEDICAL CENTER. I'D LIKE TO START BY TELLING YOU A LITTLE BIT ABOUT WHAT MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE ARE AND HOW ARE THEY DEFINED. MACHINE LEARNING USES STATISTICAL TECHNIQUE TO GIVE COMPUTER SYSTEMS THE ABILITY TO LEARN, IN OTHER WORDS, PROGRESSIVELY IMPROVE PERFORMANCE ON A SPECIFIC TANK FROM DATA WITHOUT BE PROGRAMMED AND ARTIFICIAL INTELLIGENCE IS WHEN THEY IMITATE LEARNING AND PROBLEM SOLVING. WHY ARE WE HEARING SO MUCH ABOUT ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING IN THE LITERATURE. THERE'S A NUMBER OF REASONS FOR THAT. THERE'S BEEN A SERIES OF COMPUTER SCIENCE DEVELOPMENTS THAT HAVE ENABLED NEURAL NETWORKS WITH MANY LAYERS TO BE TRAINED BY MANY LAYERS IT USED TO BE THEY ONLY HAD TWO OR THREE LAYERS AND NOW THEY CAN HAVE HUNDREDS OR THOUSANDS OF LAYERS AND THAT LEADS TO A BETTER ABILITY FOR THE SYSTEMS TO MAKE PREDICTIONS MORE ACCURATE PREDICTIONS FROM DATA. ANOTHER REASON IS THAT INEXPENSIVE PROCESSING UNITS HAVE BEEN DEVELOPED AND DEEP NEURAL TRAINING BY AT LEAST ONE AND IN SOME CASES TWO ORDERS OF MAGNITUDE. A THIRD REASON IS THAT THESE MACHINE LEARNING ALGORITHMS WITH THE NEW DEVELOPMENTS HAVE HAD SUCCESS AT SOLVING HEART PROBLEMS SUCH AS OBJECT RECOGNITION AND NATURAL WORLD IMAGES AND THE COMPUTER SCIENCE COMMITTEES HAD A SERIES OF COMPETITIONS IN WHICH DIFFERENT RESEARCHERS HAVE TRIED TO COMPETE TO PROVIDE THE MOST ACCURATE PREDICTIONS OF NATURAL WORLD IMAGES. THOSE ARE PICTURES YOU TAKE WITH YOUR CELL PHONE OR OTHER CAMERA AND THESE COMPUTER SYSTEM USING DEEP NEURAL NETWORK CAN LOOK AT THE IMAGES AND DESCRIBE, WITHOUT HUMAN INTERVENTION WHAT'S GOING ON IN THE PICTURE. FOR EXAMPLE, THERE ARE TWO CHILDREN PLAYING FRISBEE IN A PARK. THE COMPUTER CAN COME UP WITH A CORRECT DESCRIPTION OF THE PAPER. IN RADIOLOGY WE ALSO USE PICTURES, OF COURSE. AND RADIOLOGY ENCOMPASSES MANY TYPES OF USEFUL PICTURES THAT HELP US MAKE DIAGNOSES AND GUIDE INTERVENTIONS AND I HAVE SHOWN CHEST RADIOGRAPHS AN CT SCANS AND PET CT SCANS SUCH AS THIS ONE AND THERE'S OTHER IMAGES SUCH AS ULTRASOUND AND INTERVENTIONAL RADIOLOGY WHERE GUIDANCE IS USED. IN EACH MODALITIES ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING HAS PLAYED A MAJOR ROLE AT LEAST IN THE RESEARCH WORLD. NOW, BEFORE THE RECENT BOOM IN DEEP LEARNING FOR RADIOLOGY OR THE ADVANCES IN MACHINE LEARNING, HERE AT THE NIH WE WORKED ON A SERIES OF PROJECTS TO DEVELOP ADVANCED TECHNIQUE TO TEACH A COMPUTER HOW TO READ THE IMAGES. I SHOW A PROJECT TO DETECT POLYP AND THIS ANALYZES THE SPINE AND THIS MEASURES BONE MINERAL DENSITY USEFUL FOR DETECTING OSTEOPOROSIS IN THE ELDERLY. THIS SHOWS A CLOSE UP OF THE SPINE AND SPINAL CANAL AND SHOWED METASTATIC DEPOSIT FROM PROSTATE CANCER WHICH CAN IMPINGE ON THE SPINAL CORD OR NERVE ROOTS. THIS FINDS A SCAN OF THE ABDOMEN. THE SMALL BOWEL IS SHOWN IN BLUE AND THE VESSELS THAT FEED THE MESSENTARRY ARE IN YELLOW AND THIS IS THE SPLEEN, KIDNEY AND AORTAS ON CT AND THIS SHOWED THE DETECTION OF KIDNEY TUMORS. THIS IS THE KIDNEY AND THE RED BLOGS ARE TUMORS IN A PATIENT WITH A HEREDITARY PREDISPOSITION TO KIDNEY CANCER AND THIS SHOWS AN ONCOLOGY APPLICATION OF LYMPH NODES ON BODY CT AND TRIED TO AUTOMATE WORK FOR TREATMENT EVALUATION. NOW, THE AMAZING PROGRESS IN ARTIFICIAL INTELLIGENCE AND MACHINE LEARNING HAPPENED IN THE COMPUTER SCIENCE AROUND 2012. WITHIN A COUPLE YEARS I HAD DIRECTED MY LAB TO FOCUS ALL OF ITS EFFORTS ON APPLYING THESE NEW TECHNOLOGIES TO THE PROBLEMS OF IDENTIFYING DISEASE IN RADIOLOGY IMAGES. THIS EXAMPLE SHOWS WORK DONE IN MY LAB BY SOME MEMBERS IN THE AUDIENCE TODAY AND THIS DETECTS INFLAMED BOWEL COLITIS AND THIS IS ANOTHER ONE BY DR. LU DETECTING ARTHURO SCLEROTIC PLAQUE AND THIS IS DETECTION OF PROSTATE CANCER ON MULTI-PERIMETRIC MRI. BEFORE THE BOON IN RATIONAL INTELLIGENCE AND MACHINE LEARNING THIS IS HOW WE WOULD DO IT AND TAKE DATA AND CALCULATE FEATURES. INTO QUANTITIES INTO A CLASSIFIER AND MAKE A PREDICTION. TO DETECT A POLYP IN THE COLON WE WOULD TAKE A CT SCAN OF THE ABDOMEN AND CALCULATE FEATURES ALONG THE FEATURES FOR EXAMPLE THE CURVATURE OF THE SURFACE OF THE COLON AND MAYBE THE INTENSITY OF THE PIXELS INSIDE THE SURFACE OF THE COLON AND PASS THE NUMBERS INTO THIS CLASSIFIER. THE PROBLEM IS THAT THIS PARADIGM IS VERY TIME CONSUMING. IT WOULD TAKE US A YEAR JUST TO IMPROVE THESE FEATURES IN ORDER TO EKE OUT A 5% IN IMPROVEMENT IN SENSITIVITY AND NEW TECHNIQUES CAME ALONG AND DROPPED THIS TIME-CONSUMING STEP AND IN THIS PARADIGM WE TAKE THE DATA AND PUT IT IN A NEURAL NETWORK AND TAKES THE FEATURES AND MAKES A PLEA -- PREDICTION. BY SKIPPING THE SKIP WE CAN TAKE INSTEAD OF A YEAR TO GET A 5% IMPROVEMENT, WITHIN A WEEK OR TWO WE CAN GET A 5% IMPROVEMENT. SO HOW WELL DO THE TECHNIQUES PERFORM? FOR POLYPS IN A COLON THE SENSITIVITIES HAVE IMPROVED FROM AN EARLIER PERFORMANCE OF 58% FOR POLYPS 6 MILLIMETERS AND FOR THE LARGER POLYPS THE ONES REALLY CONCERNING SENSITIVITY IMPROVED FROM 62% TO 98%. FOR METASTATIC DEPOSITS IN THE SPINE SHOWN HERE IN THIS CASE IN THIS VERTEBRAL BODY. THIS IS BREAST CANCER ME TASTIS AND FOR THIS THE PERFORMANCE SOARED FROM 43% TO 77%. SO YOU CAN SEE FOR SOME APPLICATIONS WE'RE GETTING OUTSTANDING PERFORMANCE SOMETHING COMPARABLE TO RADIOLOGISTS AND OTHER AREAS NEED IMPROVEMENT AND WHAT HELPS US MOVE FORWARD VERY QUICKLY IS THE PUBLIC ACCESSIBILITY OF DATA, GOOD DATA SETS. WE'VE TRIED TO CONTRIBUTE TO THE WORLDWIDE EFFORTS TO DEVELOP ARTIFICIAL INTELLIGENCE AND MEASURE LEARNING ACCURATE FOR RADIOLOGY BY UPLOADING OUR DATA SETS TO THE INTERNET FOR OTHERS TO USE. THIS IS THE FIRST OF FOUR DATA SETS I'LL MENTION IN WHICH WE'VE DONE JUST THAT. THE LYMPH NODE DATA SET WAS UPLOADED TO THE CANCER DATA SET ARCHIVE RAN BY NIH AND CAN BE DOWNLOADED BY ENTERING THE INTERNET ADDRESS OR TYPING INTO A SEARCH ENGINE AND THERE'S 106 ANONYMIZED CT SCANS AND ALL THE ANNOTATIONS OF THE LYMPH NODES AND MASS. THIS IS THE APPLICATION I MENTIONED EARLIER ABOUT COLITIS DETECTION. THE INFLAMED COLON IS SHOWN HERE THIS IS WITHIN THE ASCENDING COLON AND THIS THICK STRUCTURE IS THE INFLAMED COLON. A NORMAL BOWEL SHOULD HAVE A THIN OR IMPERCEPTIBLE WALL LIKE THIS BOWEL HERE AND THIS ONE HERE. THE COLORS REPRESENT A PROBABILITY MAP SHOWING WHERE THE INFLAMMATION IS LIKELY TO BE LOCATED. THERE'S NEARLY 94% SENSITIVITY AND A METRIC CALLED THE R.O.C. CURVE OF .986. THIS SAY -- IS A HANDFUL OF EXAMPLES WHERE THE PERFORMANCE IS SO HIGH OF A RADIOLOGIST. MOST THE SYSTEMS TO DATE DO NOT YET PERFORM AT THIS HIGH LEVEL. AND THERE'S LESIONS AND ORGANS IN 3-D. THIS IS THE PANCREAS ON THE CT SCAN. THE RED COLOR IS THE MANUALLY DRAWN REFERENCE STANDARDS AND THE GREEN IS THE AUTOMATED SCAN AND YOU CAN SEE THE TRACING ON THE RIGHT SIDE. THIS HAS DRAWN INTEREST FROM THE SCIENTIFIC COMMUNITY AND THERE'S A NUMBER OF INVESTIGATORS AROUND THE WORLD ARE USING THIS DATA SET TO IMPROVE PANCREAS SEGMENTATION. IT'S USEFUL FOR MEASURING THE PANCREAS DENSITY IN DIABETES AND FOR ALL THE -- AUTOMATED DETECTION OF CANS -- CANCEROUS TUMORS. THIS IS WORK I'VE BEEN DOING WITH DR. BARIS TURKBEY AND HAVE BEEN WORKING ON THIS OVER FIVE YEARS AND YOU'LL HEAR HOW WE DEVELOPED THE SYSTEM TO DETECT CANCERS LIKE THIS ONE USING ADVANCED MACHINE LEARNING TECHNIQUES. AND BARIS WILL TALK A LITTLE BIT MORE HOW TECHNOLOGIES LIKE MACHINE LEARNING CAN LEAD TO BETTER DIAGNOSIS WHEN THE SOFTWARE IS USED BY PRACTITIONERS. SOME OTHER THING WE CAN DO WITH MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE INCLUDE AUTOMATED BODY PART RECOGNITION AND NORMAL NORMALIZATION SO THE COMPUTER CAN KNOW WHICH PART OF THE BODY THIS IS AND THAT HELPS AS A PRE-PROCESSING STEP FOR OTHER TYPES OF ANALYSES. WE TALKED ABOUT DETECTING ATHEROSCLEROSIS AND DETECTING PLAQUE AND THIS COULD BE USEFUL FOR GI AND OTHER SECCERS AND THIS IS AN EXAMPLE OF LUG -- LUNG SEGMENTATION AND THERE'S A TECHNIQUE IN THE LITERATURE AND A TECHNIQUE DEVELOPED IN A LAB AND THE TECHNIQUE CAN IDENTIFY THE BOUNDARIES OF THE LUNGS. THE LUNGS ARE THESE DARK STRUCTURES HERE AND HE'S ABLE TO HANDLE DIFFICULT ANATOMY THE PRIOR TECHNIQUES HAD TROUBLE WITH LIKE THE AREA WHERE THE ESOPHAGUS IS LOCATED. CONFUSION OF BOWEL AND LUNG AND WHEN IT'S ABNORMAL HIS TECHNIQUES CAN FIND THE EDGES OF THE LUNG. THIS ENABLES MORE AUTOMATED PROCESSING FOR ALL TYPES OF LUNG DISEASES. THIS IS A PROCESS ON WHICH I COLLABORATED ALSO FROM DAN MALUR'S LAB LOOKING AT INTERSTITIAL LUNG DISEASE WHERE RADIOLOGISTS IDENTIFY PORTIONS OF THE DISEASE ABNORMALITIES IN THE LUNG AND THE COMPUTER CAN PROPAGATE THOSE LABELS TO THE SIMILAR-APPEARING ONES ALONG THE SCAN AND THESE ARE CRITICAL FOR DEVELOPING THESE SYSTEMS BUT EXTREMELY TIME CONSUMING. SOMETIMES THESE CASCANS HAVE HUNDREDS OF IMAGING AND MANUAL LABELLING OF HUNDREDS OF IMAGES IS JUST NOT FEASIBLE. IN THE MRI, WE'VE ALSO HAD APPLICATIONS FOR MUSCLE AND THE EXTREMITIES. THIS IS A PROJECT FROM JACKIE HOWE IN THE DEPARTMENT IN WHICH I COLLABORATED. THERE WERE PATIENTS WITH AND WITHOUT SEVERE ATROPHY AND THE THIGH MUSCULATURE AND THIS IS FATTY REPLACEMENT AND IT'S BEEN REPLACED BY FAT AND THIS IDENTIFIED THE FASCIA TO QUANT QUANTITATE HOW MUCH HAS BEEN LOST. AND THIS IS INTOLERANCE TO TREATMENT BE IT CHEAP -- CHEMOTHERAPY AND OTHERS AND THIS MEASURES THE MUSCLE GROUPS AND THEIR VOLUMES AND THE ATTENUATION CHARACTERISTICS TO ASSESS FOR ATROPHY AND WE'RE LOOKING INTO THIS FOR A FA VARIETY OF POPULATIONS INCLUDING SCREENING. AND THIS IS A COLLABORATION WITH THE UNIVERSITY OF WISCONSIN LOOKING AT MUSCLE FAT AND BONE DENSITY AND WE LOOKED AT OVER 12,000 SUBJECTS AN MEASURED THEIR VISCERAL FATS AND CORRELATED THAT WITH OTHER FACTORS SUCH AS CARDIOVASCULAR DISEASE THAT UNFORTUNATELY I DON'T HAVE TIME TO GO IN TO. TUMOR GROWTH MODELLING IS ANOTHER APPLICATION WE'VE BEEN INTERESTED IN. THIS IS AN EXAMPLE OF A PANCREATIC ENDCRINE TUMOR ON CT AND THIS TUMOR GROWTH MODELLING USING MACHINE LEARNING IS ABLE TO PREDICT THE SIZE OF THE TUMOR AT ANY POINT IN THE FUTURE WITH CONSIDERABLE ACCURACY. IN THE LAST PART OF THE PRESENTATION I'D LIKE TO GIVE A VIGNETTE ABOUT HOW WE ARE TRYING TO LEVERAGE THE MASSIVE AMOUNTS OF RADIOLOGY DATA THAT WE HAVE IN OUR HOSPITAL INFORMATION SYSTEM. IT'S NOT POSSIBLE TO MANUALLY ANNOTATE HUNDREDS OF THOUSANDS OF CT SCANS AND CHEST X-RAYS AND SO FORTH. SO SINCE EVERY RADIOLOGY STUDY pWHAT'S FOUND IN THAT STUDY, WHAT WE'RE TRYING TO DO IS TEACH THE COMPUTER HOW TO READ THE REPORT AND USE THAT TO LEARN HOW TO READ THE IMAGES. ONCE THAT'S DONE, WE CAN THEN APPLY THE COMPUTER TO READ THE IMAGES WOULD THE WRITTEN REPORT. THAT'S THE IDEA. IN PRACTICE THE WAY IT WORKS IS WE FEED THE IMAGES AND WRITTEN REPORT INTO A DEEP-LEARNING SYSTEM CALLED A CONVOLUTIONAL NEURAL NETWORK AND THE SYSTEM LEARNING -- LEARNS HOW TO READ THE IMAGES. IT CAN IDENTIFY BODY REGIONS LIKE THE HEAD, THE BREAST, THE SPINE OR THE ARMS AND LEGS. AND CAN IDENTIFY AREAS WITH MA TAS TA CEASE AND YOU CAN TYPE IN A PHRASE AND GET BACK CAPTIONS OR SITES THAT YOU CAN VISIT TO ANSWER WHATEVER QUESTION YOU'VE SEARCHED FOR. IN THIS THE QUESTION IS WHAT'S GOING ON IN THE IMAGE. THIS IS AN MRI IMAGE OF THE ABDOMEN SHOWING CANCER ON THE LEFT KIDNEY, THE LEFT SIDE OF THE PATIENT IS ON THE RIGHT SIDE OF THE IMAGE AND THE COMPUTER CAME UP WITH THESE RESULTS, THIS SEARCH ENGINE, CAME UP WITH DIAMETER MASS AND KIDNEY. IT'S A MASS IN KIDNEY AND LEARNED DIAMETER BECAUSE THE SENTENCE DESCRIBING THE LESION SAID THE LESION WAS 1.6 CENTIMETERS IN DIAMETER. SOMETIMES THE RESULTS ARE NOT USEFUL BUT WITH FURTHER DEVELOPMENT THE TECHNIQUES ARE GETTING BETTER AND PROVIDING MORE USEFUL LABEL. NOR THE MRI AND CT OF THE ABDOMEN SHOWING LIVER MA TAS TOO CEASE THEY WERE OUTPUT BY THE SYSTEM AND ALL ACCURATE LABELS THAT DESCRIBE THE IMAGES. AND THE YELLOW BOX SHOW THE TRUE ANNOTATION AND THE COMPUTER BOXES SHOW THE ANNOTATIONS AN THE COMPUTER'S ABLE TO IDENTIFY THE NORMAL IMAGES AND CHANGES IN THE SPINE AND IN SOME CASES A DISEASE LIKE IN THIS CASE, GRAN GRANULOMAS AND THING LIKE THAT AND ONE THING IS THE MACHINE LEARNING TOOLS ARE BLACK BOXES WHERE IT'S NOT ENTIRELY CLEAR HOW THE MACHINE CAME TO ITS CONCLUSION. ONE WAY WE DEAL WITH THAT IS WE DEVELOPED SOMETHING CALLED SAIL EN SI MAPS SHOWING WHERE THE COMPUTER FOCUSSED ITS ATTENTION IN THE HOPE THAT INFORMATION MAY BE HELPFUL TO RAISE THE CLINICIAN'S CONFIDENCE ON THE ACCURACY OF THE COMPUTER-GENERATED DIAGNOSIS. WE BRANCHED OUT INTO IDENTIFYING MULTIP MULTIPLE DISEASES. THIS CHEST X-RAY WAS ABLE FOR EIGHT ABNORMALITIES AND COLLAPSED LUNG AND ENLARGED HEART OR COLLAPSED LUNG. DEF THE ACCURACIES OF THE SYSTEMS RANGE WIDELY FROM 16% FOR KNOWLEDGE -- NOD YOU'LLS TO 99% FOR MEGGALY. THE DATA SET WAS ALSO RELEASED TO THE PUBLIC. IT CONSISTS OF 112,000 FRONTAL CHEST RADIOGRAPHS FROM 13,000 UNIQUE PATIENTS AN INCLUDES METADATA AND BOUNDING BOXES FOR THE DISEASE AND 1,000 OF THE IMAGES. AND THIS DATA SET IS BEING USED FOR AN INTERNATIONAL COMPETITION AT THE RADIOLOGIC SOCIETY OF NORTH AMERICA MEETING HAPPENING NEXT MONTH. AND THERE WAS A TEXT EMBEDDING NETWORK DEVELOPED AND THE INPUT IS THE CHEST X-RAY AND THE OUTPUT IS A FULLY RENDERED TEXT REPORT DESCRIBING THE IMAGE. AND SOMETIMES THE REPORT IS ACCURATE AND OTHER TIMES IT MAY MAKE GRAMMATICAL SENSE BUT IT'S COMPLETELY CLINICALLY WRONG. HOWEVER, IN MANY CASES IT IS CORRECT AND WE'RE WORKING ON DEVELOPING THESE ANNOTATIONS. THE LAST DATA SET I'LL TELL YOU ABOUT IS CALLED THE DEEP LESION DATA SET. THIS WAS RELEASED A COUPLE MONTHS AGO AND HAS BEEN ALREADY DOWNLOADED THOUSANDS OF TIME. IT CONSISTS OF LESIONS THROUGHOUT THE BODY INCLUDING LUNG LESIONS, SOFT TISSUE LESIONS LIKE AXILLARY LESIONS AN THESE ARE LIVE -- LYMPH NODES AND MEDIA SPINAL LESIONS AN BONE LESIONS IN THESE EXAMPLES HERE AND THERE ARE 32,000 LESIONS IN THE DATA SET FROM 4,000 UNIQUE PATIENTS. THE IDEA IS THE DATA SET WILL ALLOW RESEARCHERS INCLUDING OUR GROUP IS TEACH A COMPUTER TO DETECT ANY LESION ANYWHERE IN THE BODY IN A CT SCAN AND THESE ARE RECENT CT SCANS WITH LESIONS IN THE 1 TO 3 CENTIMETER SIZE RANGE. THIS IS EXAMPLES OF WORK IN MY GROUP FROM A POST-DOCTORAL FELLOW AND THIS IS A NODE THE SYSTEM CAN PULL OUT ALL THE SUB CORINAL NODE AND THIS CAN PULL OUT THE RIGHT LOWER LOBE MASSES. LIVER LESION. THE SYSTEM CAN PULL OUT ALL THE LIVER AND KIDNEY LESIONS AND SOFT TISSUE LESIONS. YOU CAN SEE THE SYSTEMS ARE BECOMING MORE KNOWLEDGEABLE ABOUT HUMAN ANATOMY. CAN ADDRESS A WIRED -- A WIDER RANGE OF PROBLEMS AND IT LENDS SOME CONFIDENCE THE SYSTEMS CAN BE GENERALIZABLE TO LOTS OF DISEASES. I'LL GIVE YOU AN EXAMPLE ONE TIME I'VE USED IT FOR PANCREATIC LESIONS AN THE SYSTEM WAS ABLE TO PULL OUT QUICKLY HALF A DOZEN OR MORE SUCH CASES NEARLY INSTANTLY. AND ULTIMATELY, WHAT THEY TRIED TO DO WAS DEVELOP A UNIVERSAL LESION DETECTER TO DETECT THEM ANYWHERE IN THE BODY, FOR EXAMPLE IN THE NECK, THORACIC AND DELIVER AND SPLEEN AND SO FORTH. THAT CONCLUDES MY REMARKS. I INVITE YOU TO LOOK AT MY RECENT REVIEWS AND ARTICLES OB MEDICAL IMAGING AND RADIOLOGY. THIS IS THE WORK OF MANY TRAINEES AND INVESTIGATORS AND COLLABORATORS NOT ONLY AT THE CC AND INSTITUTIONS BUT ALSO AT FDA AND DEPARTMENT OF DEFENSE. IF YOU'D LIKE TO LEARN MORE I INVITE YOU TO GO TO MY WEBSITE AND I'D LIKE TO BRING OUT MY COLLEAGUE, DR. TURKBEY. [APPLAUSE] I'M HONORED TO SPEAK HERE ABOUT OUR RESEARCH ON ARTIFICIAL INTELLIGENCE AND PROSTATE CANCER I HAVE NO DISCLOSURE. PROSTATE MRI IS THE MOST ESTABLISHED IMAGING MODALITY FOR LOCALIZED PROSTATE CANCER AND HAS A HISTORY OF OVER TWO DECADE AND I'D LIKE TO SUMMARIZE THIS IN THIS SLIDE. SO IN THE EARLIER TIMES OF THE PROSTATE MRI IN THE EARLY 2000s THE MAIN CHALLENGE PROVIDED TO ANSWER WAS LOCAL STAGING. WHAT IS THE EXTENT TO DETECT THE CANCER. AND RELATED WITH THAT WE WERE DOING OUR SCANS BY USING FOUR THAT'S WHY PROSTATE MRI IS CALLED [INDISCERNIBLE]. IN 2007 WE INTRODUCED A NEW PROSTATE MRI TO CANCER CLINICAL CARE FOR THE FIRST TIME IN THE WORLD WE TOOK OUT THE PREVIOUSLY IMAGES AND FUSED THEM WITH REAL-TIME ULTRA SOUND AND WE GOT VISIBLE LESIONS BY LEADERSHIP OF DR. PETER PINTO AND DR. BRAD ABOUT A -- BOOTE AND THE COMMUNITY AND RESEARCH ORGANIZATIONS TOOK UP THIS IMMEDIATELY AND BY THE TIME WE CAME TO 2013 THERE WERE EITHER PLATFORMS UTILIZING MRI ARE YOU -- ROUTINELY AND THERE'S LOCAL STAGING TO BIOPSY GUIDANCE. EMERGENCE OF THE PLATFORMS AN WIDESPREAD MRI BROUGHT HETEROGENEITY BECAUSE PEOPLE WERE READING THE IMAGES. IT WAS DIFFICULT TO HANDLE. SO TO COVER THAT, AMERICAN COLLEGE OF RADIOLOGY IN COLLABORATION WITH THE COLLEAGUES FROM EUROPE INTRODUCED A PROSTATE IMAGING AND REPORTING DATA SYSTEM. THIS SYSTEM BASICALLY GIVES THE GUIDELINES ON HOW TO ACQUIRE AND READ PROSTATE MRIs. FOLLOWING THIS WHEN WE COME TO THE 2016, 2017 TIME FRAME, MRI WAS MORE OFTEN ASKED FROM RADIOLOGIS RADIOLOGISTS AND OFTEN ACQUIRED AND USED AND WE START TO SEE CENTER MRIs AND SOME WERE DOING TE BIPARAMETRIC IMAGING AND THIS IS FROM 2018 AND THIS IS HOW WE USED TO DO THE MRIs 10 YEARS AGO. THERE'S A RESOLUTION DIFFERENCE AND LESIONS WE WERE DEALING WITH WERE LARGER IN THE EARLY TIMES. NOWADAYS WITH THE CHEAPER MRI TECHNIQUES WE CAN DETECT THOSE SMALLER LESIONS INSTEAD OF [INDISCERNIBLE] WE'RE GETTING B VALUE IMAGES SO THE SCAN WAS TAKEN IN AN HOUR AND 10 MINUTES AND THIS SCAN WAS DONE IN 35 MINUTES. THIS SHOWS THE BIG DIFFERENCE OVER TWO DECADE. HAVING COHORT STUDIES IN 2018 WE START TO SEE THE EVIDENCE PAPERS SUPPORTING THE PROSTATE MRI AND PROSTATE CANCER CARE. THIS IS THE MOST FAMOUS OF THEM PUBLISHED IN THE NEW ENGLAND JOURNAL OF MEDICINE AND YOU'LL SEE THIS STUDY A LOT IN YOUR DAILY LIFE IF YOU'RE INTERESTED IN PROSTATE MRI. THE TWO KEY FINDINGS OF THE 500-PATIENT PROSPECTIVE RANDOMIZED STUDY WAS IN THE BIOPSY ARM THE RESEARCHERS WERE ABLE TO DETECT MORE SIGNIFICANT CANCERS COMPARED TO THE STANDARD OF CARE ARM. IN THE MRI TARGETED BIOPSY ARM THEY DETECTED LESS INDOLENT CANCERS AND THERE'S OVER TREATMENT OF AVERSIVE DISEASE AND THIS SHOWS AN MRI CAN HELP ANSWER THE QUESTION. WHILE EVERYTHING I EXPLAINED SO FAR IS GREAT ABOUT PROSTATE MRI BUT NOT WITHOUT LIMITATION. ESPECIALLY IF YOU ARE A PATIENT OR A PHYSICIAN WHO USED PROSTATE MRI YOU'RE MORE FAMILIAR WITH THOSE BUT THE BIGGEST LIMITATION IS REPRODUCIBILITY. OBSERVER AGREEMENT. AFTER EMERGENCE OF PIREX WE HAVE DONE SEVERAL STUDIES INCLUDING COLLABORATIONS WITH NYU AND HAVE SHOWN PROSTATE MRI IS ONLY MODERATELY REPRODUCIBLE BETWEEN DIFFERENT RADIOLOGISTS INVOLVING SEVERAL HUNDREDS OF PATIENTS FROM DIFFERENT CULTURES AND THE CAPPA VALUES ARE NOT VERY PROMISING. RECENTLY WE WANTED TO STUDY THE REPRODUCIBILITY OF RADIOLOGISTS IN OUR OWN COHORT. ONE OF OUR FORMER FELLOWS CONDUCTED THE STUDY FOR READERS AND THE RESULT INTERESTING. WE FOUND THE IT WAS QUITE LOW. SO WHAT WE DID IN THE STUDY IS WE TOOK ABOUT 100 PATIENT AN GAVE THE SCREEN CAPTURES TO THE RADIOLOGISTS AN TOLD THEM TO SCORE THE VISIONS BY USING THE FIVE CATEGORY SYSTEM. INTERESTINGLY, THIS GRAPH SHOWS THE INTERNAL OBSERVER AND SOME HAD POOR AMOUNTS AND THEY WERE NOT ABLE TO REPLICATE THEIR SCORING PERFORMANCE. AND AS EXPECTED, THE INTRAOBSERVER AGREEMENT WAS NOT VERY PROMISING AS WELL. IT WAS MODERATE BUT IN THE REST IT WAS LOW TO MODERATE. SO ANOTHER PROBLEM IS TODAY AND WE KNOW THAT MRI MISSES OR UNDER ESTIMATES THE CANCERS. YOU MOST OFTEN SEE POSITIVE RESULT PAPERS. THIS IS ONE OF THE FEW NEGATIVE RESULT PAPERS ABOUT PROSTATE MRI AND WHAT WE DID WAS WE TOOK A HUNDRED CONSECUTIVE PATIENTS AN TOOK OUR PROSPECTIVE READOUTS AND WENT TO THE PATHOLOGY MAPS AND CALCULATED HOW MANY OF THE LESIONS OF THE PATIENTS WE MISSED CANCERS. WE OBSERVED TWO MAIN PATTERNS. IN 5% TO 10% OF OUR PATIENTS WE SCANNED AND MANAGED PROSPECTIVE HERE, YOU CAN SEE A LESION ON THE LEFT SIDE AND THE MRI SHOWS NOTHING OVER THERE. SO MRI CLEARLY MISSED CANCERS IN THIS PATIENT AND WHEN YOU LOOK AT THE FINAL PATHOLOGY THERE'S A MUCH BIGGER LESION AND IF YOU RELY ON THE MRI YOU CAN UNDER TREAT THE PATIENT OR PUT THEM UNDER ACTIVE SURVEILLANCE BUT WHERE THE PATIENT HAS A BIGGER TURMER THAN WHAT YOU EXPECT. THIS -- TUMOR THAN WHAT YOU EXPECT. THIS IS THE TABULATIONS OF THE LESIONS WE MISSED. INTERESTINGLY, THE MISSED LESIONS ARE BARELY VISIBLE OR LOW-SCORE LESIONS BUT THEY'RE SIGNIFICANT LESIONS AND IN THE FOUR CATEGORY. THIS IS ANOTHER LIMITATION OF THE IMAGING WE HAVE TO BE AWARE OF. SO REGARDING THIS LIMITATIONS, WE FELT LIKE ARTIFICIAL INTELLIGENCE OR COMPUTER-AIDED DIAGNOSIS CAN ASSIST US IN SOLVING THE PROBLEMS. WE START TO WORK ON CAD OR A.I. IN 2011-2012 TIME FRAME. OUR FORMER SCIENTIST AND HIS FELLOW HAVE DEVELOPED A SYSTEM FOR US. THE SYSTEM UTILIZING THE KINETIC MAPS OF MRI NO LONGER USED AND A.D.C. MAPS AND ONLY WORKED IN THE PERIPHERAL ZONE BUT THE LARGE LESION ON THE RIGHT WAS SUCCESSFULLY MAPPED BY THE ARTIFICIAL INTELLIGENCE SYSTEM AND IT BUSINESS ABOUT 90% WHICH IS A HIGH NUMBER AS DR. SUMMERS MENTIONED IN HIS OWN SAMPLES. THE LIMITATIONS OF THE SYSTEM IT WAS VERY OVER ENGINEERED AND THE SYSTEM WAS ONLY WORKING IN THE PERIPHERAL ZONE. WE GAVE UP ON IT AFTER THE EXPERIMENT. AFTER THREE TO FOUR YEARS, WE WERE OBSERVING THE LIMITATIONS AND WENT TO DR. SUMMERS AND EXPLAINED WE NEEDED A SYSTEM TO HELP US TO MAKE US LOOK BETTER. ONE OF HIS FORMER EMPLOYEES CAME UP WITH THIS SYSTEM CALLED A.I. 2.0 AND WAS TAKING THE FIXTURES AND BY USING FEATURE ENGINEERING AND MAPPING THE CANCER-SUSPICION LESIONS ON THE PROSTATE. I TESTED THE SYSTEM IN A PILOT MANNER AND LIKED THE SYSTEM AND WANTED TO STUDY THE SYSTEM IN A MULTI-CENTER FASHION. I TOOK 160 PATIENTS WITH A FORMER MRSB AND GAVE IT TO NINE DIFFERENT READERS FROM ALL OVER THE WORLD AND ASKED THEM TO EVALUATE THE IMAGES FIRST AND GAVE THE IMAGES WITH THE CAD MAPS AND THEN WE WANTED TO SEE HOW HELPFUL THE CAD IS. SO THIS IS A SAMPLE PATIENT FROM THAT COHORT. IN THE FIRST ROUND EACH COLUMNS REPRESENT A READER AND AS YOU SEE THE FIRST ROUND, FOUR OF THE READERS COULDN'T DETECT ANYTHING ON THE IMAGE WHEREAS FIVE OF THEM DETECTED THIS LESION. THEN WE PROVIDED THE CAD MAP TO THEM AND AS YOU SEE, THE UNSUCCESSFUL FOUR READERS WERE ABLE TO CORRECTLY DETECT THE LESION WITH THE HELP OF CAD AND THE FINAL ONE DETECTED THE ONE THERE. THIS WAS QUITE HELPFUL EXAMPLE FOR US TO SEE THE INTERACTION OF THE A.I. WITH THE RADIOLOGISTS. IF YOU LOOK AT THE BIGGER RESULTS OVERALL ARTIFICIAL INTELLIGENCE IMPROVED IT 8.4% IN DETECTING SIGNIFICANT CANCERS AN THE CAD, THE A.I. HELPED UP LOWER PIREX AREAS. THE POINTER DOESN'T WORK BUT THE CAD HELPED THE RADIOLOGISTS IN LOWER PIREX SCORE LESIONS. THIS HELPS IN LOWER PI-RAD SCORES. AND THEY ARE STARTING TO SPEAK THE SAME LANGUAGE BY USING THIS ARTIFICIAL INTELLIGENCE. THIS SYSTEM WAS TRAINED AND TESTED WITH THESE IMAGES WITH THAT TYPE OF IMAGES. IT IS A VERY HIGH RESOLUTION AND OBTAINEDTOWN -- OBTAINED BETWEEN THE NCI AND THEY'LL UNDERSTAND THIS IS FROM NIH IF YOU TAKE IT ANYWHERE ELSE IN THE WORLD BUT THE REST OF THE WORLD IS ACQUIRING THE IMAGES IN A DIFFERENT WAY. I DIDN'T MEAN TO PUT THIS TO SCARE YOU BUT THESE ARE 23 SAMPLES FROM DIFFERENT CENTERS AND AS YOU SEE, THE IMAGE POSITION IS HETROGENOUS AND THIS COMES TO US EVERY DAY AND WE HAVE TO UNDERSTAND WHAT'S GOING ON IN THE PATIENT. SO A TOOL THAT YOU USE IN TERM OF A.I. SHOULD BE WORKING ON THESE TYPES OF IMAGES, NOT THE GOOD QUALITY NIH IMAGES ONLY. SO TO ANSWER THAT QUESTION, WE TOOK THE SAME CAD SYSTEM AND ONE OF OUR FELLOWS, SONYA GAR HAS DESIGNED A STUDY WITH OUR STATISTICIAN AND TOOK IMAGES FROM FIVE DIFFERENT CENTERS OUTSIDE OF NIH WITH THE CONTROLS AND GIVE IT TO NINE DIFFERENT READERS FROM SIX DIFFERENT COUNTRIES AND WE TESTED THE SAME. LIKE TWO ROUNDS READING WITH FOUR-WEEK WASHOUT PERIOD. THIS IS AN EXAMPLE FROM THAT COHORT AND YOU CAN SEE ONLY ONE PERSON WAS ABLE TO DETECT THE LESION OVER HERE. BUT IN THE CAT ROUND FOUR ADDITIONAL READERS WERE ABLE TO DETECT THE LESION. IN THIS STUDY, THE SENSITIVITY RESULT NOT AS PROMISING AS OUR NIH STUDY BECAUSE OF THE HETEROGENEITY AND WE SHOWED A POSITION FOR INEXPERIENCED PEOPLE. SO THE SYSTEM KIND OF HELPED THE RADIOLOGISTS. SO THE TWO EXPERIMENTS I SHOWED YOU WORKED VERY WELL BUT WE FOUND SOME CHALLENGES. BASICALLY IN THE PATIENTS WE FAILED TO DETECT CANCER IN TWO ROUNDS. THE RADIOLOGIST DIDN'T KNOW HOW TO INTERACT WITH THE ARTIFICIAL INTELLIGENCE SYSTEM. THIS IS A CLEAR EXAMPLE OF THAT. SO IN THE FIRST ROUND, WE GAVE THE IMAGES TO NINE PEOPLE AND ONLY THREE WERE ABLE TO FIND SOME LESION HERE. AND THEN WE PROVIDED THEM THIS CAD MAP LIKE FIVE ADDITIONAL PERSONS OR RADIOLOGIST ABLE TO DETECT THE LESION. HOWEVER, LIKE ALL OF THEM THEY DIDN'T PUT IT INTO THE REPORTS. BUT IF YOU -- AND ONLY ONE OF THIS PARTICULAR PATIENT. IF YOU LOOK AT THE REALITY MAP PROVIDED BY DR. MARINO, THERE ARE THREE LESIONS IN THE SAME SLIDE AND ALL THREE LESIONS -- SO ONLY ONE RADIOLOGIST WAS ABLE TO IDENTIFY THIS EVEN USING THE CAD SYSTEM AND SO THE COLOR MAPS WERE TAKING AWAY THE OBJECTIVITY OF THE RADIOLOGIST THEY WERE NOT TO HANDLE THE IMAGES IN SUCH CHALLENGING PATIENTS. SO TO ANSWER THAT, WE DEVELOPED ANOTHER CAD SYSTEM AGAIN WITH D. SUMMERS GROUP AND THIS TIME INSTEAD OF THE COLOR MAPS, WE TOLD THE CAD SYSTEM TO DELIVER US ANY SUSPICIOUS FOUR AREAS IN A PARTICULAR MRI AND WE WANTED TO SEE THE INTERACTION OF RADIOLOGISTS WITH THAT METHOD INSTEAD OF THE COLORFUL MAPS. THIS IS AN EXAMPLE FROM THAT COHORT. THE A.I. SHOWS THIS BOXED AREA AND TELLS THE RADIOLOGIST LOOK HERE, THERE MAY BE SOMETHING HERE AND THE RADIOLOGISTS LOOKS THERE AND THEN WE USE THE OTHER MODALITIES LIKE A.D.C. MAP AND HIGH DWI AND ACCEPTS OR REJECTS IT. THIS IS A CASE FROM BRAZIL. A LOW-QUALITY IMAGE BUT THE SYSTEM WAS ABLE TO LOCATE THIS AGGRESSIVE CANCER. SO WITH THE SAME DESIGN, WE ARE IN PROCESS OF TESTING THIS SYSTEM IN NINE READERS WITH DIVERSE IMAGE QUALITY. THE RESULTS ARE STILL PENDING, BUT AS I MENTIONED EVERY SYSTEM DR. SUMMERS BRINGS TO US I DO A PILOT STUDY ON MY OWN. THIS PILOT STUDY RESULTS WERE INTERESTING. WE TOOK 40 PATIENTS AND 105 LESIONS WERE PRESENT IN 40 PATIENTS. YOU SEE THIS IS MY PERFORMANCE AND THIS IS THE PERFORMANCE OF THE MACHINE. FOR THE FIRST TIME IN MY LIFE I LOST A CASE AGAINST A MACHINE. I DON'T KNOW. I DON'T KNOW BUT I WAS HEAD OF THE SOFTWARE WE DEVELOPED. SO THE MACHINE IS CONSISTENT AND I'M TRYING TO BE CONSISTENT AS WELL. IF YOU LOOK AT THIS INTERESTINGLY 89 WERE DETECT HUMAN AND MACHINE BUT 28. DETECTED BY THE CAD ONLY. SO THE MRI SO PLEASE REMEMBER MY SLIDE ON THE RADIOLOGY PAPER WHERE WE SHOWED LESION ON THE LEFT AND NOTHING ON THE MRI. THESE WERE THIS TYPE OF PATIENTS AN ONLY NINE WERE POSITIVE ON THE MRI. IF YOU KNOW HOW TO TRUST THE MACHINE, IT CAN TAKE YOU TO THE CORRECT ANSWER. THE ONLY TRADE-OFF IS THE HIGHER FALSE POSITIVE RATE. THE SYSTEM GIVES YOU FOUR GENERIC LESIONS BUT IF YOU CAN UNDERSTAND THE PROBABILITY OF THE PATIENT, YOU CAN TUNE THAT NUMBER TO LIKE MORE OR LESS THAN FOUR. SO I HAD SHOWN YOU SEVERAL MULTI-READER STORIES. SOME PEOPLE SENT IMAGES TO US. HOW DOES THIS HAPPEN? WELL, THIS IS THE UNITED STATES MAP AND THEN THE RED STARS ARE THE PLACES WHERE WE TRAIN SOMEONE HERE AND SHADOWED US AND THEY TAUGHT FROM US AND THEY WENT AND STARTED TO PRACTICE AND THEY BECOME OUR READERS ULTIMATELY. THIS IS A CLUB. AND THIS IS THE MAP AND THE BLUE STARS ARE SHOWING THE PLACES WHERE WE HELPED SOME PEOPLE AND THEY DIDN'T LEAVE US. THEY ARE STILL WORKING WITH US. THEY ARE STILL HELPING US. WHENEVER WE DEVELOP SOMETHING WE GO TO THESE PEOPLE AND THEY'LL RELATE IT FOR US AND THIS IS A GOOD CHANCE TO BE ACADEMICALLY EFFICIENT. NONE OF THE ALGORITHMS ARE BASED ON DEEP LEARNING. IT'S VERY NEW FOR US AND WE DON'T CURRENTLY HAVE AN ACTIVE DEEP LEARNING ALGORITHM FOR MRI BUT ARE INTERESTED IN UNDERSTANDING THE BASICS AN BUILD CORRECT DATABASE FOR THE PRODUCT AND WE HAVE A GROUP AT MIT AND OUR GOAL IS TO FIND 10,000 PATIENTS WITH CORRECT ANNOTATION AND OUR PORTFOLIO WILL BE PROSTATE MRI AND CT IMAGES AND INCLUDING RADIOLOGISTS, YOU'ROLOGISTS -- UROLOGISTS AN ENGINEERS. THE MOST IMPORTANT SEAT IS THIS SEAT. THE ON-SITE RADIOLOGIST WHO COMES HERE ON THEIR OWN TO LEARN HOW TO READ THESE IMAGES FROM US AND THEN THE PERSON UNDERSTANDS HOW TO READ THEM. WE TRAIN THEM HOW TO ANNOTATE THEM AND THEY'RE ANNOTATING LESION FOR US BECAUSE IT'S TAKING TOO MUCH TIME. THERE'S A DOCTOR FROM CHILE AND ONE FROM TURKEY AND ARGENTINA AFTER THIS. SO THIS IS AN IMPORTANT SEAT AS DR. SUMMERS MENTIONED THE GOOD QUALITATIVE DATA IS CRUCIAL FOR THE ARTIFICIAL INTELLIGENCE BUSINESS. YOU'LL REMEMBER THE GRAPH AND TRIED TO EXPLAIN WHERE IS THE PROSTATE MRI COMING FROM AND WHERE'S IT GOING. WE START TO WORK IN ARTIFICIAL INTELLIGENCE IN 2012 BUT WE HAVE HIGH INTEREST SINCE 2014, 2015. AND THE FEELING IS ARTIFICIAL INTELLIGENCE WILL BE HERE WITH US FOREVER IN OUR IMAGING INTERPRETATION AND READ-OUT ROOMS. HOPEFULLY WE WILL NOT LOSE OUR JOBS TO ROBOTS OR COMPUTERS. TO CONCLUDE, PROSTATE MRI IS A POWERFUL IMAGING TECHNIQUES BUT NOT WITHOUT LIMITATIONS. THESE ARE IMPORTANT LIMITATIONS. WE HAVE A DETECTION PROBLEM. WE HAVE IMAGE ACQUISITION AND IMAGE INTERPRETATION. WE HAVE TO COVER THOSE AND THEY CAN'T POTENTIALLY SOLVE THE LIMITATIONS AND TO COVER THAT WE NEED MORE DATA, MORE TRAINING AND MORE RESEARCH AND THIS DATA SHOULD BE DIVERSE AND MULTI-CENTERED. BEFORE I END MY WORK THIS IS THE MOST IMPORTANT SLIDE OF THE TALK. WE WORK WITH LOTS OF PEOPLE FROM NIH AND ALL OVER THE WORLD AND THESE ARE THE PEOPLE WHO CONTRIBUTED TO THIS. IN THE MEANTIME, I WOULD LIKE TO THANK OUR PATIENTS AND THEIR FAMILIES BECAUSE THIS IS THE RESULTS OF THEIR DEDICATION. AND THAT'S ALL I WANT TO SAY. THANK YOU. >> OKAY. SO NOW WE ARE READY FOR QUESTIONS IF YOU CAN COME UP TO THE MICROPHONES, PLEASE. LET'S START ON THIS SIDE. >> THE GLEESON CLASSIFICATION IN AND OF ITSELF DOESN'T HAVE A PERFECT CAPA. YOU'RE APPLYING ALL THIS TO THE IMAGING PORTION BUT HOW ABOUT THE PATHOLOGY PORTION? FROM EXPERIENCE. I CAN TELL YOU WON'T GET 100% AGREEMENT IN TERMS OF THE GLEESON CLASSIFICATION ITSELF. DO YOU HAVE ANY PROGRAMS WORKING ON EXPERT SYSTEMS OR ARTIFICIAL INTELLIGENCE FOR THE HISTOLOGY OF PROSTATE CANCER? >> THAT'S A GOOD QUESTION. DIGITAL PATHOLOGY AND AUTOMATED POLICING SCORING FIRST OF ALL THE CANCER DETECTION AND GLEESON SCORING IS IN OUR PORTFOLIO BUT TO ACHIEVE THAT WE NEED LOTS OF GOOD QUALITY DATA AND WE'RE IN THE PROCESS OF ACQUIRING THAT. >> OKAY. THANK YOU. >> HI, DO YOU MIND COMMENTING ON THE TRANSLATIONAL PATHWAY FOR THESE TOOLS. I BELIEVE THEY'D BE REGULATED AS CLASS TWO DEVICES FROM THE FDA BUT DO YOU WORK WITH LARGE COMPANIES AND MRI MANUFACTURERS AND TRY TO SPIN OUT LICENSE TO SMALL COMPANIES? HOW DO YOU GET THAT TO PATIENT CARE? >> WE HAVE LICENSED SOME OF OUR TECHNOLOGIES TO DIFFERENT COMPANIES AND THEN THEY TAKE THE TECHNOLOGY TO GET FDA APPROVAL. I HAVEN'T BEEN INVOLVED WITH THE ACTUAL FDA PROCESS BUT I DO KNOW THE FDA IS IMPROVING ARTIFICIAL INTELLIGENCE-BASED SOFTWARE PACKAGES FOR USE IN MEDICINE. I'M AWARE, FOR EXAMPLE, THERE WAS A TOOL I JUST HEARD ABOUT THAT HAS MACHINE LEARNING FOR ASSESSI ASSESSING DIABETIC RETINOPATHY AND THEY'RE EXTREMELY KNOWLEDGEABLE ABOUT WHAT'S GOING ON WITH THE TECHNOLOGIES. >> WHEN YOU SEE A DISCREPANCY BETWEEN WHAT THE SYSTEM RECOGNIZES AND THE EXPERT HUMAN RECOGNIZES, IF THE DISCREPANCY IS A FALSE POSITIVE OR A MISS, DOES IT TELL YOU ANYTHING INTERESTING IF YOU GO BACK AND EXAMPLE THE DATA THAT LED TO THE DISCREPANCY. DOES IT TELL YOU ANYTHING INTERESTING BEFORE ABLE UNUSUAL CASE OR IS IT AN ARTIFACT OF THE IMAGING OR DATA COLLECTION? SOMETHING LIKE THAT. >> THAT'S A VERY GOOD QUESTION. SO WE OBSERVE IT AND IN MOST OF THE CASES IT IS NON-TUMOR PATHOLOGIES OR PROSTATES AND WE REALLY WANT TO FURTHER TRAIN OUR SYSTEMS WITH THAT INFORMATION. THE DIFFERENCE IN THE VERSION 2 AND 3 WAS BASICALLY WE GOT MORE ANSWER TO FALSE POSITIVES TO DR. SUMMER'S CAD SCIENTISTS AN THE SYSTEM STARTED WORKING BETTER. >> MY QUESTION IS THIS, IN TERMS OF ADVANCING WHAT YOU ULTIMATELY WANT TO DO, AND I ASSUME YOUR SURGICAL COLLEAGUES WANT TO DO IS YOU DON'T WANT TO MISS A LESION. THE SURGICAL COLONIES DON'T WANT TO TAKE A PATIENT TO THE O.R. AND STICK NEEDLES AN BIOPSIES AND THING THAT -- THINGS THAT ARE NORMAL. WHERE IS THE GAIN GOING TO BE, IN THE ALGORITHMS YOU'RE TRYING TO CREATE, IN OTHER WORDS, THE MACHINE LEARNING OR IS IT REALLY THING THERE BASED ON WHATEVER PHYSICAL PROPERTIES ARE RELATED TO THE TUMOR? WHAT WILL GET YOU TO WHERE YOU NEED TO BE? >> THE ANSWER IS SORT OF ALL OF THE ABOVE. THE IMAGING MODALITY HAS TO BE OUTSTANDING. OUR UNDERSTANDING OF THE VARIABLE APPEARANCE OF PATHOLOGY ON THE IMAGES HAS TO BE UNDERSTOOD IN DETAIL. THE ALGORITHM NEEDS TO BE TRAINED ON A DIVERSE GROUP OF DATA SETS OBTAINED USING DIFFERENT DATA POPULATIONS TO BE GENERALIZABLE AGAINST THE VARIATION THAT CAN BE ENCOMPASSED BY THE VARIABLES. UNFORTUNATELY THERE'S NO SIMPLE ANSWER. YOU HAVE TO DO ALL THOSE THING. THINGS. >> WELL, THANK YOU ALL FOR COMING AND NEXT WEEK WE'LL HAVE OUR SPECIAL HALLOWEEN GRAND ROUNDS SO WE'D LIKE TO SEE YOU IN THE AUDIENCE. [APPLAUSE] .