WE'RE ALL SET TO GET GOING HERE WE HAVE A MORNING SESSION THAT'S FULL. WE'RE EXCITED TO TALK ABOUT INTERACTION FOR THE QIBA LATER THIS MORNING AND I'M HOPING A FEW OF YOU WHO ARE BOTH INVOLVED IN QIN AND QIBA WILL COME UP HERE FOR THE PANEL DISCUSSION AT THAT POINT, BUT BEFORE WE DO THAT I'M GOING TO TURN IT OVER TO MY COLLEAGUE, DR. ZHANG, AND HE WILL OPEN THE MORNING SESSION. >> BY THE WAY, WELCOME BACK, ON THE SECOND DAY. SESSION A 5 IS ABOUT CCP, CHALLENGES AND COLLABORATIVE PROMISE. ONE TALK FROM MY FRIEND AND COLLEAGUE KEYVAN FARAHANI, AND THE NEXT IS JUSTIN KIRBY, KEYVAN WILL BE THE FIRST. I'M HAPPY TO TALK ABOUT QIN BENCHMARKS AND APPRECIATE SOME OF THE MENTIONS, SO HOPEFULLY YOU'D LIKE TO COMMENT ON. THIS IS ESSENTIALLY A REPORT OF THE CHALLENGE TASK FORCE THAT WE PUT TOGETHER LATE LAST SUMMER, AND WITH ESSENTIALLY COMPOSED OF MYSELF, AS A TASK GROUP CHAIR, AND EC CHAIR DR. MARYELLEN GIGER, REPRESENTATIVES, WEI HUANG, CATHY KRAMER, DR. NORDSTROM AND DR. TATA, DR. WAHL FROM WASH U. QIN IS 20 TEAMS, AS WE ALL KNOW, THEY ARE INVOLVED IN A VARIETY OF TECHNIQUES AND MODALITIES AND DISEASE SITES, ET CETERA.& OVER THE LAST FEW YEARS HAVE BEEN CONDUCTING MUCH COLLABORATIVE ACTIVITIES THROUGH THIS MECHANISM WE REFER TO AS CHALLENGES AND COLLABORATIVE PROJECTS THROUGH CCPs, CONCEPTUALLY CCPs COVER FROM VALIDATION OF TOOLS SUCH AS SEGMENTATION TO CLINICAL TRANSLATION. MOST OF THE ACTIVITY HAS BEEN ON THE LEFT SIDE OF THE SPECTRUM. JUST TO REFRESH AND FOR THOSE NEW TO QIN, CHALLENGES REFER TO MULTI-SITE TEST OF COMPUTATIONAL ALGORITHMS DESIGNED TO PERFORM IMAGE PROCESSING TASK OR ANALYSIS, AND WE ELEMENTS TO QIN PROJECTS OR MISSION, USING TRAINING AND TEST DATASETS THAT OFTEN COME FROM TCIA, AND USING SOME CLINICAL REFERENCE STANDARDS, AND OF COURSE EVALUATION METRICS AND HOPEFULLY STATISTICAL DESIGN. THEY COULD BE IN THE TECHNICAL OR CLINICAL DOMAIN. AND HOPEFULLY THEIR PERFORMANCE WOULD BE IN THE GREEN ZONE OF THIS DYAD. THE COLLABORATIVE PROJECTS ARE MORE PROJECTS DONE ON EVALUATING PERFORMANCE OF DIFFERENT MODALITIES, OR DIFFERENT APPROACHES SUCH AS MRI OR PET CALIBRATION, OFTENTIMES WITH PHANTOMS, STATISTICAL FALL FAN -- FAN COMES. IT'S AN OPPORTUNITY TO BRAND AND HIGHLIGHT QIN PRODUCTS. THERE'S A LOT OF ACTIVITY GOING ON IN THE NETWORK, AND FORTUNATELY IT'S GETTING ATTENTION OF THE OUTSIDE WORLD BUT IT WOULD BE NICE TO HAVE LABELS THAT DESIGNATE WHAT STAGE OF DEVELOPMENT THESE TOOLS ARE. FURTHERMORE, THEY SHOULD HELP INCENTIVIZE SHARING OF DATA, TOOLS AND METHODS THROUGH VARIOUS MECHANISMS WE HAVE. STANDARD LABELS FOR DEVELOPMENT AND TRANSLATIONAL TOOLS, WE THINK CONSENSUS ON PRODUCTS AND GUIDE CATALOGING OF TOOLS AS WELL AS GAUGING TOOLS FOR CLINICAL TRIALS. IN OTHER WORDS, CLINICAL TRIAL LEADS COULD LOOK AT LIST OF QIN TOOLS AND SEE WHAT BENCHMARKS THEY HAVE ACHIEVED AND WHICH ARE MORE READY FOR PRIME TIME. SO BEFORE WE GET INTO DISCUSSION OF BENCHMARKS, I WANT TO OFFER A DEFINITION FOR QUANTITATIVE IMAGING PRODUCTS, QI PRODUCTS. IT REFERS TO SOFTWARE TOOLS OR PACKAGE, PHYSICAL OR DIGITAL REFERENCE OBJECT USED FOR PRODUCTION OR ANALYSIS OF CANDIDATE QI BIOMARKER, FOR DIAGNOSIS OR STAGING OF CANCER OR MEASUREMENT OR EVALUATION OF RESPONSE TO THERAPY. OF COURSE, PRIMARY MISSION OF QIN IS MEASUREMENT OR EVALUATION OF RESPONSE TO THERAPY BUT IT'S REALIZED MANY TIMES THESE TOOLS ARE USED IN DIAGNOSIS OR STAGING OF CANCER IN THOSE SETTINGS. SO WE STARTED, I THINK WE STARTED ONE LEVEL BENCHMARK AND THREE AND THEN FIVE. THERE WERE SUGGESTIONS TO CONSIDER LARGER NUMBER BUT DECIDED TO KEEP IT TO FINITE NUMBER OF BENCHMARKS THAT REALLY HIGHLIGHT KEY STAGES OR MILESTONES IN DEVELOPMENT OF TOOLS OR TECHNIQUE FROM BENCH TO BEDSIDE ESSENTIALLY. SO THERE'S PRE-BENCHMARK, BASIC, TECHNICAL TEST BENCHMARK, CLINICAL TRIAL BENCHMARK AND CLINICAL USE BENCHMARK. AND BEFORE I GET INTO DISCUSSION OF EACH OF THESE, I WANT TO MENTION THAT ONE IS NOT REQUIRED TO ACHIEVE EVERY SINGLE BENCH MARK. IN OTHER WORDS, YOU DON'T NEED TO HAVE BENCHMARKS ONE THROUGH FOUR TO GET BENCHMARK FIVE, BUT THE REQUIREMENT IS -- ONE OF THE REQUIREMENTS IS THAT THE CANDIDATE BENCHMARK SHOULD SATISFY THE PRIOR LEVEL BENCHMARK, NOT NECESSARILY HAVE OBTAINED IT BUT SATISFIED REQUIREMENTS OF THE PROJECT LEVEL. SO THE PRE-BENCHMARK IS REALLY INCENTIVE TO GET INTO THIS PROCESS, ESSENTIALLY A IMAGING PROJECT FROM SINGLE OR MORE CLINICAL SITES WOULD QUALIFY FOR PRE-BENCHMARK LEVEL, AND DECIDE TO KEEP THESE DESIGNATIONS, THIS GETS ONE STAR. NEXT LEVEL BASIC BENCHMARK A PRODUCT THAT FULFILLS PRIOR LEVEL BENCH MARK, PRE-BENCHMARK, AND OTHER TOOLS PARTICIPATED IN CHALLENGE OF COLLABORATIVE PROJECT TOOLS, SOFTWARE TOOL OR PHANTOM. AND PROJECT COULD BE QIN OR OUTSIDE, MADE PUBLICLY AVAILABLE THROUGH A VARIETY OF AVENUES, GITHUB, DOCKER, HUB OR STORE, OR THROUGH MATERIAL TRANSFER AGREEMENT. REQUIREMENT AGAIN FOR ALL THE BENCHMARKS IS PEER-REVIEWED PUBLICATION. THE REASON FOR THAT IS TO PROVIDE INDEPENDENT ASSESSMENT OR ENDORSEMENT, IF YOU WILL, OF THE CLAIM. TECHNICAL TEST BENCHMARK IS MID-LEVEL BENCH MARK, AND THAT'S FOR PRODUCTS THAT PASSED OR SATISFIED REQUIREMENTS FOR THE BASIC BENCHMARK LEVEL, AND THE TOOL HAS BEEN TESTED BY ONE OR MORE PEER LABORATORY OR INDUSTRY, AND THIS ESSENTIALLY ADDRESSES THE TOOL PATHWAY THAT WE DISCUSSED BEFORE, ESSENTIALLY EXCHANGING TOOLS FOR EVALUATION AT PEER SITES FOR NOT SO MUCH FOR SCIENTIFIC EVALUATION OF THE TOOL BUT FOR PRESUMABLY THAT'S BEEN ALREADY DONE THROUGH PRIOR RESEARCH AND PUBLICATION, BUT FOR PERFORMANCE, EVALUATION, CRASH RATES, HANDLING OF NON-NATIVE DATA, WHAT I MEAN BY THAT IS DATA THAT'S ACQUIRED ON DIFFERENT PLATFORMS. REQUIREMENTS IS PUBLICATION LED BY TEST SITE TO INCENTIVIZE THE TEST SITE USING THE SPONSOR TOOL TO DESCRIBE FUNCTIONAL, PERFORMANCE OF THE TOOL ON THEIR PLATFORM AND THE PLATFORM THAT TOOL WAS INITIALLY DEVELOPED FOR. AND THEN USING REFERENCE OR COMMON DATASETS AND INDEPENDENT DATASETS, ACQUIRED AT TEST SITE OR EVALUATION SITE. AND THIS HAS A THREE-STAR DESIGNATION. WE GET TO THE CLINICAL DOMAIN, AND THIS IS WHERE WE LIKE TO SEE MANY OF THE TOOLS GET TO. SO IT'S A PRODUCT THAT HAS PRESUMABLY PASSED TECHNICAL TEST BENCH MARK REQUIREMENTS, CAN BE USED IN A CLINICAL SETTING AS LEADING OR SECONDARY TOOL, THAT'S WHAT WE MEAN BY CROSS-CHECK IN A CLINICAL TRIAL TO NATIONAL NETWORKS THAT WERE DESCRIBED YESTERDAY, OR OTHER INDEPENDENT CLINICAL TRIALS MECHANISM. PEER REVIEW PUBLICATION, DESCRIBING PERFORMANCE AND USE OF TOOL IN CLINICAL TRIAL SETTING. AND FINALLY THE LAST BENCHMARK, CLINICAL USE BENCHMARK, WHICH IS FOR QUANTITATIVE IMAGING TOOL THAT'S BEEN THROUGH A CLINICAL TRIAL, AND USED TO ACQUIRE CLINICAL TRIAL DATA OR EVALUATE QUANTITATIVE METRIC USING THE TRIAL. REQUIREMENT IS PEER-REVIEWED PUBLICATION, DESCRIBED UTILITY AND PERFORMANCE, FIVE STARS, HIGH PRIZE. THE APPLICATION AND REVIEW OF BENCHMARKS CERTIFICATION, PROCESS IS PUT IN PLACE NOW. WE DON'T HAVE APPLICATION FORM BUT IT WILL BE READY SOON AND PRESUMABLY WOULD BE VERY STRAIGHTFORWARD, BASICALLY DESCRIBING THE CLAIM AND PROVIDING SUPPLEMENTARY MATERIALS, OBVIOUSLY PEER REVIEWED PUBLICATION IS AMONG THEM. WE WANT TO BE FAIR AND TRANSPARENT, LED BY THE COORDINATING COMMITTEE, SO FOR THOSE NOT FAMILIAR THE COORDINATING COMMITTEE IS COMPOSED OF CHAIRS OF EACH OF THE WORK GROUPS AND PROGRAM STAFF. AND WE THINK WE'LL BRING IN ADDITIONAL EXPERTISE FROM NIH, ET CETERA, AD HOC BASIS. IF APPLICATION IS REJECTED THERE WILL BE OPPORTUNITY TO AMEND AND BRING IT BACK. BENCHMARKS WOULD BE ISSUED CERTIFICATE OF RECOGNITION BY OUR PROGRAM AND IT WILL BE AWARDED TO EACH PRODUCT TEAM. WE ALSO PLAN TO ISSUE ADDITIONAL OBJECTIVITIERS TO DOCUMENT THAT CONTAINS CERTIFICATE, QUALIFICATION DEDESCRIPTOR AND REFERENCES BROUGHT IN, CITED ON QIN WEBSITE AND REGISTRY, TOOLS SPREADSHEET. SO THERE ARE A NUMBER OF ADVANTAGES TO THIS PROCESS. I MENTION IT COULD PROVIDE INCENTIVE FOR MEMBER PARTICIPATION IN CCPs, ESTABLISH A FAMILY OF QIN PRODUCTS, ATTAIN HIGHER IMPACT FOR PROJECTS THROUGH ENHANCED RECOGNITION AND UTILIZATION BY THE COMMITTEE, AND POTENTIALLY THESE COULD PROVIDE SOME LEVEL OF GUIDANCE FOR INDUSTRY IN DEVELOPING QUANTITATIVE IMAGING TOOLS. AND FURTHERMORE, IT'S LIKELY THAT THIS PROCESS COULD FURTHER FACILITATE INTERACTION OF QIN PRODUCTS WITH QIBA PROFILES AND QIBA ACTIVITIES, THERE WILL BE SOME DISCUSSION OF THAT LATER THIS MORNING. SO, OBVIOUSLY DIFFERENT TOOLS LAND IN DIFFERENT AREAS OF THIS GRAPH, BUT PRESUMABLY THIS IS PROBABLY WHAT THE GRAPH WOULD LOOK LIKE NOW. MOST TOOLS ARE LIKELY TO QUALIFY FOR PRE-BENCHMARK, THEY CAME WITH PEER-REVIEWED APPLICATIONS, TOOLS WERE DESCRIBED, PERHAPS IN DIFFERENT SETTING BUT THEY HAVE ACHIEVED THAT LEVEL. AND HOPEFULLY OVER TIME THEY WILL FLATTEN OUT THIS GRAPH AND MAYBE EVEN REVERSE IT. SO, THERE ARE A NUMBER OF TOOLS THAT OBVIOUSLY QUALIFY FOR THESE DIFFERENT LEVELS. TOOLS THAT ENTER WITH GRANTS FOR PRE-BENCHMARK LEVEL, COLLABORATOR PROJECTS AND CHALLENGES WOULD QUALIFY FOR BASIC BENCHMARK, TECHNICAL TEST BENCHMARK FOR TOOLS SHARED, TESTED BY OTHER SITES, SOME EXAMPLES WERE DISCUSSED YESTERDAY. CLINICAL TRIALS, CLINICAL TEST BENCHMARKS ARE MORE ADVANCED TOOLS THAT WERE DEPLOYED IN MULTI-SITE CLINICAL TRIALS AND SOME IN CLINICAL SETTING. YESTERDAY I WAS TAKING NOTES OF TOOLS THAT WERE INTRODUCED, AND I THINK THESE CLEARLY WOULD QUALIFY FOR ONE LEVEL OF BENCHMARK, AND THESE INCLUDE BRICKS, SPECTRO SCOPEY TOOL FOR BRAIN, MR DIFFUSION, QIBA, DROs, THE MICHIGAN THAT WAS INTRODUCED, AND THE TOOLS BY YANKALOF GROUP, MULTI-B VALUE, DWMRI, FOR HEAD AND NECK CANCER, MRI CAMPER, CIA MICROSCOPE, ePAD, ET CETERA. THIS IS A PATHWAY TO TAKE TOOLS FROM BENCH TO BEDSIDE, IN SOME CASES SCALE AND GET THEM CLOSER TO CLINIC, RELATING TO BRIDGE TO MAKE THIS TRANSLATION. I HAVE TWO QUICK SLIDES ABOUT CURRENT COLLABORATOR PROJECTS ONGOING, AND IF YOU'RE NOT PARTICIPATING CAN YOU CONSIDER JOINING THESE TEAMS. FIRST STARTED LAST YEAR, CHAD QUARLES, ASSOCIATE MEMBER OF QIN, AND THIS IS A TWO-PHASE STUDY, THEY HAVE COMPLETED THE FIRST PHASE ANALYZING RESULTS FOR CEREBRAL BLOOD VOLUME, D SC-MRI, AND THE PROJECT STARTED WITH PETER LEADING THAT, HYPOXIA PHANTOM STUDY APROVED BY THE COORDINTING COMMITTEE, IVAN AT TORONTO IS LEADING THAT. THAT WAS APPROVED LAST MONTH. A SET OF OUTSIDE CHALLENGES, THE ONES UPCOMING MICCAI, YOU MAY HAVE COLLABORATED IN THE PAST, AND THESE INCLUDES BRATS SINCE 2012, NOW IT'S LED BY SPIRE, BACCHUS AND THE GROUP AT PENN, THAT'S A POPULAR CHALLENGE WHICH NOW HAS ADDITIONAL TASK OF RADIOMICS IN ADDITION. WE'RE IN THE FOURTH TIER COMPUTATIONAL PRECISION MEDICINE WORKSHOP AND CHALLENGES, MORNING WORKSHOP, AFTERNOON CHALLENGES, TUESDAYING TCGA BRAIN DATASETS, MRI PATHOLOGY, DIGITAL PATHOLOGY SEGMENTATION, NUCLEI AND IMAGES, SPATIAL PATHOLOGY IMAGES BY JULES GROUP AND RADIO MIX STRATIFIERS LED BY DAVE FULLER FROM M.D. ANDERSON, THOSE ARE THE URLs FOR THESE CHALLENGES, YOU CAN ALSO FIND THE WIKI PAGE ON CHALLENGES AT TCIA TO LEAD YOU TO THIS AND COLLABORATION WITH TCIA IS APPRECIATED. THIS MEETING IS IN SPAIN. I HOPE I GET TO GO. BUT IF YOU'RE GOING, BON VOYAGE. THANK YOU. >> THANK YOU. [APPLAUSE] QUESTIONS FOR KEYVAN? TOO EARLY IN THE MORNING? WELL, SO WELL, MAYBE I'LL THROW OUT ONE TO GET THE CONVERSATION GOING. TO ME, IT SOUNDS LIKE THAT PROCESS HAS SORT OF LIKE A PERSPECTIVE FROM THE DEVELOPERS, PRODUCT DEVELOPER SIDE, EVEN THOUGH THE LAST TWO STEPS CROSS OVER INTO THE CLINICAL TRIAL AND CLINICAL USE TERRITORY. I WAS WONDERING, I MEAN, THINKING ABOUT DISCUSSIONS WE HAD YESTERDAY. IS IT TOO EARLY TO START THE PROCESS FROM THE USER SIDE TO AT LEAST EXPLORE AND MAYBE ORGANIZE THOUGHTS ABOUT STEPS TO ACTUALLY REACH OVER, YOU KNOW, THINGS LIKE COST CONSIDERATIONS, COMPLEXITY OF ADOPTION, THINGS LIKE THAT, I MEAN, ANYBODY, KEYVAN, WOULD YOU LIKE TO EXPLORE THAT? >> THOSE ARE INTRIGUING QUESTIONS. WE HADN'T THOUGHT ABOUT EXACTLY THOSE. COST CONSIDERATION WOULD JUST KILL THE WHOLE DISCUSSION BECAUSE THERE'S OFTEN NEED TO SPONSOR TOOLS TO GET INTO CLINICAL TRIALS. SO BUT THOSE ARE REALISTIC, YOU KNOW, PARAMETERS TO CONSIDER. AND WE'RE HOPING THAT THE PIONEERS IN THIS PATHWAY WOULD SORT OF OPEN THE WAY FOR OTHERS TO FOLLOW AND I EXPECT INITIALLY MIGHT BE SOME WRINKLES BUT WE'LL IRON THAT OUT. >> BASICALLY FOLLOWING UP ON THAT QUESTION, BETWEEN THE FOURTH AND FIFTH, BETWEEN CLINICAL TRIALS AND CLINICAL YIELDS, MOST OF THE THINGS TRYING TO DO WITH POST-PROCESS ANALYSIS, IF YOU'RE DOING PROSPECTIVE DATA ACQUISITION IT'S ALWAYS CHALLENGING TO GET INTO CLINICAL TRIAL OR ONGOING CLINICAL TRIAL, SO HOW MUCH YOU KIND OF SEE BETWEEN IT CAN BE EASIER TO GET INTO CLINICAL USE ANALYSIS WHERE WE CAN EVALUATE THE TOOL AND PROSPECT, AND ONGOING STUDY IN A CLINICAL USE WE CAN TRY TO HAVE ADDITIONAL SEQUENCE TO GET -- IT'S ALWAYS MORE CHALLENGING GETTING INTO THE CLINICAL TRIALS SO HOW MUCH DO YOU SEE IT HAS TO HAVE A CLINICAL TRIAL BENCHMARK BEFORE MOVING TO THE CLINICAL USE, JUST GETTING YOUR COMMENTS. >> THANKS. SO THAT WAS ONE OF THE MAJOR MOTIVATIONS FOR THIS WHOLE THING WAS BECAUSE LAST YEAR OR LAST COUPLE YEARS WE'VE BEEN EMPHASIZING MOVING TOWARDS CLINICAL TRIALS AND SOME CHALLENGES WERE DISCUSSED BY PANELISTS YESTERDAY TO CLINICAL TRIAL NETWORKS, AND THAT'S THE REALITY. BUT I THINK WE WANT THIS PROCESS TO BE RIGOROUS ENOUGH BUT NOT TOO RIGOROUS THAT PEOPLE WOULD NOT HAVE TIME TO FOLLOW, IT'S NOT FDA REGULATORY PATHWAY BUT ON THE OTHER HAND SHOULD BE, YOU KNOW, SHOULD HAVE ENOUGH RIGOR AND I THINK CLINICAL TRIAL REQUIREMENT IS AN IMPORTANT ONE. SO IT'S NOT EXPECTED THAT EVERY TOOL GOES THROUGH THAT STAGE BUT THERE ARE TOOLS THAT ARE CLINICAL TRIALS LIKE R PERSIST AND A FEW OTHERS THAT WERE MENTIONED. >> YEAH, I SAW THE QUESTION AND DEFINITION THERE, IN THE FIRST STAGE IT INCLUDE PATENT APPLICATION AND ALL THAT, AND THE SECOND STAGE SAID PUBLICLY AVAILABLE, SO WHAT DO YOU MEAN BY PUBLICLY AVAILABLE, DOES THAT MEAN REALLY AVAILABLE OR WHAT DID -- PATENT RELATES TO CONSIDERATION? >> YEAH, THAT'S A GOOD QUESTION. THANK YOU. I WAS THINKING ABOUT THAT LAST NIGHT AS I WAS UPDATING MY NOTES. I THOUGHT THEY WERE GOING TO BRING UP THAT SLIDE. IDEALLY WOULD BE PUBLIC AVAILABLE. I SHOULD ALSO -- YOU SAID THE QUALIFIER THESE REQUIREMENTS AND ARE SUBJECT TO CHANGE, BUT THE IDEA WAS THAT IT WOULD BE MADE -- WOULD NOT BE JUST KEPT, YOU KNOW, UNDER ARMS IN ONE INSTITUTION. EITHER PUBLICIZED THROUGH A AT PATENT BECAUSE THEY ARE ACCESSIBLE IN TERMS OF INFORMATION, BUT IDEALLY PUBLICLY AVAILABLE THROUGH GITHUB OR OTHER MECHANISMS. YES? >> THE DESCRIPTION IS VERY GOOD, AND IT'S VERY NICE THAT THERE ARE NOT STRONG REQUIREMENTS ON ACHIEVING EACH OF THE LEVELS. THAT IS APPROPRIATE FOR FLEXIBILITY. I THINK ONE THING WE MIGHT THINK ABOUT IS NOT HAVING A REQUIREMENT THAT YOU ACHIEVE A CERTAIN LEVEL OF, YOU KNOW, VARIABILITY OR THINGS LIKE THAT, LIKE NANCY TALKED ABOUT YESTERDAY, BUT HAVING A STATEMENT OF WHAT THE TOOL HAS ACHIEVED MIGHT BE A GOOD REQUIREMENT. SO NOT TO SAY YOU HAVE TO MAKE 5% BUT YOU HAVE TO TELL US AT LEAST WHAT -- >> WITHIN SOME RANGE. >> AT LEAST WHAT THE TOOL WAS ABLE TO DO OR DEMONSTRATE ITS FUNCTIONALITY OR SOME STATEMENT OF HOW IT CAN CONTRIBUTE TO THAT WHOLE PROCESS. DOES THAT MAKE SENSE? >> OH, SURE, IT MAKES A LOT OF SENSE. GREAT SUGGESTION, THANK YOU. YES, DR. WAHL? >> I THINK ONE OF THE INTERESTING THINGS IS MOVING TO LEVEL 5, THE QUESTION OF CLINICAL USE, I MEAN, I THINK IT MIGHT BE INTERESTING TO HEAR THE PERSPECTIVE OF OTHERS IN THE ROOM. IF YOU HAVE A TOOL THAT'S ADVANCED, THE WAY WE'VE DONE ANYTIME ST. LOUIS WE CAN USE IT IN OUR PRACTICE AS ADJUNCT BUT DISSEMINATING TO OTHERS FOR CLINICAL USE IS SOMEWHAT OF A RISK I GUESS IN TERMS OF THE REQUIREMENT FOR FDA CLEARANCE OR 510(K), AND I JUST -- THAT STEP 5 MAY NEED SOME CLARIFICATION. IN TERMS OF YOU DON'T OBVIOUSLY REQUIRE FDA APPROVAL BUT TO USE IT IN LARGER SCALE CLINICAL APPLICATION YOU WOULD PROBABLY WANT TO HAVE IT CLEARED. >> RIGHT, BUT IT WOULD REQUIRE IND. >> PRESUMABLY 510(K) DEPENDING HOW DIFFERENT IT IS FROM EXISTING, PREEXISTING STUFF AND HOW YOU STRUCTURE IT. >> YEAH. >> BUT THAT STEP TO GET TO SORT OF WIDER CLINICAL UTILIZATION IS ONE WHERE IT MIGHT BE INTERESTING TO GET REGULATORY INPUT. >> YES. >> I'VE HEARD DIFFERING THINGS IN THIS SPACE BUT HARDENED ONE THAT'S FDA APPROVED PROBABLY WOULD BE HELPFUL. THE OTHER THING, ONE OF THE CHALLENGES I THINK, SO NOT IN THE SENSE OF YOUR CHALLENGES, BUT HOW DO WE REALLY MOVE THESE OUT TO MULTIPLE CENTERS? IN OTHER WORDS IF YOU TRY TO SUPPORT MULTIPLE CENTERS WITH SOFTWARE THERE'S A LOT OF TIME INVOLVED AND, YOU KNOW, A CLOUD DEPLOYMENT MAY BE THE BEST WAY, BUT MAYBE WE CAN COME UP WITH SOME BEST PRACTICES GOING FORWARD AS TO HOW YOU DO THAT EXTENSION, WHETHER IT'S CLOUD-BASED, ONE SITE EXPERT CENTRAL PROCESSING. >> YEAH. >> THAT MAY BE AN IMPORTANT -- >> YEAH, GOOD POINT. QUICKLY, PRESUMABLY 2X WHICH COMES BEFORE CLINICAL DEPLOYMENT, WOULD FACILITATE THAT, MAKE THIS AVAILABLE TO SITES, TECHNICAL EVALUATION, THAT SATISFIES CLINICAL TEST, OR CLINICAL TRIALS USE. WITH REGARDS TO FDA INITIALLY WE -- WE CONSIDERED FDA APPROVAL FOR THAT STAGE, WHEN I TALKED TO OUR COLLEAGUES AT FDA, NICK PETRICK, AND BRET, THEY THOUGHT IT WAS APPROPRIATE FOR QIN, QIBA AND QIN, BUT NOTED FDA HAS NOT TO DATE APPROVED ANY TOOL FOR QUANTITATIVE IMAGING. PEOPLE ARE USES THEM OFF LABEL FOR QUANTITATIVE PURPOSES BUT THEY DIDN'T THINK IT WAS REALISTIC. REOPTION TO IMPROVE BY CONSIDERING WHAT FDA PATHWAYS AND GUIDANCE ARE SO AS TOOLS GO THROUGH THIS PROCESS THEY ALSO COLLECT DATA THAT SATISFIES FDA APPROVAL BUT THAT WILL BE DONE OVER TIME. THANK YOU. >> JUSTIN YOU'LL BE NEXT. WE'RE ONE MINUTE AHEAD OF SCHEDULE SO YOU HAVE THE FULL 20 MINUTES. >> THANKS. SO DR. PRIOR AND DR. BENNETT DID A NICE JOB TALKING ABOUT THE UNDERLYING INFRASTRUCTURE WITH TCIA YESTERDAY. I'LL FOCUS ON DATA AND SERVICE SIDE OF THINGS AND JUST GIVE A GENERAL UPDATE ON WHAT'S AVAILABLE. SO I THINK MOST EVERYBODY IS FAMILIAR WITH WHAT THE CANCER IMAGING ARCHIVE IS ALREADY. I ALWAYS LIKE TO BRING THIS SLIDE UP JUST TO HIGHLIGHT THE NUMBER OF DATASETS AND SO WE'RE UP THE TO 84 NOW, 41,000 SUBJECTS, AND LARGE CHUNK OF THAT IS STILL MADE UP OF NLST BUT THINGS ARE EVENING. WE ENCOURAGE PEOPLE TO SUBMIT MORE THAN IMAGING DATA. NEW DATA APPLICATIONS COMING IN HAVE REFLECTED THAT, OVER THE PAST YEAR, WHICH HAS BEEN GREAT. SO KINDS OF DATA COMING IN, THE DATA SETS ARE RICHER IN TERMS OF HAVING CLINICAL PATHOLOGY, GENOMICS, LOTS OF DIFFERENT KINDS OF ANALYSIS DATA, SO THAT'S BEEN GOOD TO SEE. A REMINDER IN TERMS OF HOW THINGS ARE BROKEN OUT CURRENTLY WE REFER TO THE DATA COLLECTION SIDE OF THINGS AS ONE ARM OF WHAT TCIA DOES, AND THEN DATA ACCESS WITH THE DATA PORTAL AND ALL THE OTHER DIFFERENT COMPONENTS FRED WAS TALKING ABOUT YESTERDAY, AND THEN THE THIRD ARM DATA ANALYSIS CENTERS WHERE A LOT OF QIN TOOLS COULD COME INTO PLAY AS THINGS THAT EXTEND THE CAPABILITIES OF TCIA IN WAYS THAT SORT OF FALL OUTSIDE WHAT WE HAVE THE CAPACITY TO DO. AND REALLY ONE OF THE BIGGER THINGS THAT, YOU KNOW, THE TCIA DOES IS SERVICE COMPONENT, SO TAKING THAT RESPONSIBILITY OFF OF THE P.I. IN TERMS OF DOING THE DE-IDENTIFICATION AND HAVING TO WORRY WHETHER YOU'VE GOT EVERYTHING OUT WHILE KEEPING ALL OF THE SCIENTIFICALLY USEFUL INFORMATION IN. I JUST WANTED TO BRING THIS UP AGAIN IN TERMS OF DIFFERENT TYPES OF DATA, SO WE HAVE TWO SORT OF SEPARATE PATHWAYS TO PUTTING DATA INTO THE ARCHIVE, WHAT WE REFER TO AS PRIMARY DATA, YOU KNOW, THE ACTUAL DICOM IMAGES, PATHOLOGY, CLINICAL DATA GOES THROUGH A FORMAL APPLICATION PROCESS. BUT THEN WE ALSO ENCOURAGE PEOPLE TO PUT IN ANALYSIS RESULTS FROM THINGS ALREADY IN THE ARCHIVE SO WE STARTED TO SEE THAT PICK UP A LITTLE BIT. I'LL GET MORE INTO THAT IN A SECOND. ONE OF THE SORT OF NEW THINGS THAT WE'RE TRYING TO PUSH, AND I BROUGHT THIS SLIDE UP FOR MAYBE A YEAR OR TWO NOW, IS YOU SHOULD BE TRYING TO PUBLISH -- WHEN YOU SHARE YOUR DATA YOU GET A DATA CITATION, PRIMARY DATASET. SIMILARLY YOU CAN DO THE SAME THING FOR ANALYSIS DATA THAT COMES IN. OBVIOUSLY YOU'VE GOT YOUR TRADITIONAL PUBLICATION CITATION, BUT ONE OF THE THINGS THAT'S REALLY STARTED TO PICK UP IN THE LAST YEAR IS SORT OF THE FORMATION OF THESE NEW DATA DESCRIPTOR JOURNALS, AND SO THERE'S A HANDFUL OF THEM THAT ARE NOW WHEN YOU GO TO THE LAST PAGE, AND YOU CLICK PUBLISHER DATA AND YOU'RE GOING INTO THE PRIMARY DATA ANALYSIS RESULTS YOU'LL SEE A LIST OF THESE DIFFERENT JOURNALS HERE NOW. AND THE IDEA WITH THESE, IT'S A PEER-REVIEWED PROCESS THAT YOU GO THROUGH, AND YOU'RE BASICALLY GETTING ANOTHER PUBLICATION OUT OF IT, BUT THE POINT IS NOT TO DESCRIBE THE SCIENTIFIC FINDINGS, IT'S TO DESCRIBE YOUR DATASET IN DETAIL IN A WAY OTHER PEOPLE WHO COME ALONG AND FIND IT IN TCIA WILL BE ABLE TO MAKE BEST USE OF IT. WE ENCOURAGE PEOPLE TO TAKE ADVANTAGE AS WAY TO GET CREDIT FOR SHARING DATA BECAUSE FRED AND BILL TALKED A LOT ABOUT YESTERDAY, IT'S A LOT OF WORK AND YOU SHOULD BE REWARDED FOR THAT WORK. WE HAVE EXPERIENCED REALLY SIGNIFICANT GROWTH IN THE PAST YEAR. WE HAVE 38 INCOMING DATA SETS IN SOME STAGE OF CURATION, WE'VE GONE FROM 3,000 TO 10,000 A MONTH, AND DOWNLOADS HAVE DONE THE SAME THING, NOW PUBLICATIONS ABOVE 613. THE 38 NOW SOUNDS DAUNTING BUT I WANTED TO POINT OUT BUT PART OF THE REASON THE MUST BE IS SO BIG THERE'S A NUMBER OF MULTI-YEAR, MULTI-TRIAL ACTIVITIES ONGOING. SO EVEN IF WE HAD ALL THE RESOURCES IN THE WORLD, THE DATA IS STILL BEING ACCRUED, SO PART OF THAT NUMBER IS JUST THE PROJECTS ARE ONGOING. SO THERE'S SORT OF THE TAIL END OF THE CANCER GENOME PROJECT, BASICALLY PROBIOTIC AND PRE-BIOTICKING UP, WE HAVE SOME NEW ONES LIKE CPTAC AND A NUMBER OF DIFFERENT OTHER NIH DATA COLLECTION ACTIVITIES, AND THEN ABOUT 12 DIFFERENT COMMUNITY PROPOSED DATASETS IN THE PIPELINE RIGHT NOW. SO JUST IN TERMS OF WHAT TCIA'S ROLE IN THE QUANTITATIVE IMAGING NETWORK, WOULD YOU DIVIDE USE CASES INTO THREE CAT CLOSER, COLLABORATIVE, CHALLENGE COMPETITIONS AND DATA SHARING REQUIREMENTS, WHETHER RELATED TO GRANT OR PUBLICATION REQUIRED BY A JOURNAL YOU'RE PUBLISHING IN. WE'RE HAPPY TO SUPPORT ANY OF THOSE THINGS. SO RIGHT NOW A NUMBER OF YOU HAVE TAKEN ADVANTAGE OF THAT. THERE'S 15 DIFFERENT DATA SETS CONTRIBUTED BY QIN SITES THAT REPRESENTS 11 OUT OF THE 19 ACTIVE SITES RIGHT NOW. AND I WON'T GO THROUGH THESE IN DETAIL, BUT IF ANYBODY'S INTERESTED IN SLIDES, YOU CAN SEE THE DIFFERENT REASONS THEY ARE BEING USED IN THE FAR RIGHT COLUMN THERE. JUST A COUPLE SUGGESTIONS AFTER TALKING TO PEOPLE AT THE MEETING HERE AND SORT OF SEEING THE OTHER PRESENTATIONS, BEING IN THE WORKING GROUPS, AT THIS POINT WITH 83 DATA SETS IN THE ARCHIVE IT WOULD BE GOOD TO TRY TO FOCUS, ALSO WITH 38 MORE IN THE QUEUE, TO TRY TO EXTEND THE FOCUS OF THE WORKING GROUPS MORE IN THE DIRECTION OF REUSE OF STUFF THAT'S ALREADY IN THERE AS MUCH AS POSSIBLE, SO WE'VE -- I RECOGNIZE THAT'S NOT ALWAYS EASY TO DO, DATA SETS HAVE THEIR OWN NUANCE, AND YOU HAVE TO MAKE SURE THAT THEY ALIGN WITH WHATEVER THE PROJECT OR CHALLENGE IS, BUT, YOU KNOW, JUST ANYWHERE WE CAN GET COST SAVINGS FROM NOT SENDING IN ANOTHER DATA SET THAT'S SIMILAR TO SOMETHING IN THERE WE SHOULD BE TRYING TO MAXIMIE THAT AS MUCH AS POSSIBLE. ANOTHER THING QIN DOES A LOT OF STUFF WITH TAKING DATA AND THEN YOU ALL DO THE GROUP PROJECTS, AND CREATE AN ANALYSIS SET OF DATA THAT COMES FROM THAT. BUT IT HASN'T BEEN THAT COMMON THAT RESULTING DATA HAS BEEN PUT BACK INTO THE ARCHIVE SO THAT OTHER PEOPLE COULD USE IT DOWNSTREAM. SO I THINK KEEPING AN EYE ON THAT GOING FORWARD COULD BE ANOTHER PLACE WHETHER WE COULD GET MORE VALUE OUT OF THE ARCHIVE. MORE LEAD TIME, THIS IS MAKING THE POINT ABOUT HOW MUCH DATA IS IN THE QUEUE SO IF YOU HAVE A PROJECT THAT'S COMING DOWN THE LINE AND YOU KNOW THAT YOU'RE GOING TO NEED TO PUT A NEW DATASET IN THE ARCHIVE TRITE TO GET IN FRONT OF THAT, PUT YOUR APPLICATION IN AS SOON AS YOU CAN BECAUSE WE HAVE A LOT OF STUFF IN THE PIPELINE AND DON'T WANT TO DELAY YOUR PROGENY MORE PROJECT, ANY MORE THAN WE HAVE TO. RICH DATASETS, A LOT OF STUFF THAT'S COME FROM QIN SITES LATELY HAS A HUGE VARIETY OF STUFF, SO KEEPING THAT GOING IN TERMS OF PROVIDING CLINICAL DATA, GENOMICS, EVEN IN SOME CASE, THE OTHER THING WITH COLLABORATIVE AND PROJECTS, TEMPTATION IS STRONG TO GO WITH QUICK AN DIRTY SOLUTION A LOT OF TIMES, BUT HUGELY SEEMS AFTER THE FACT WE FIND OUT THAT NOT STANDARDIZING THE FORMAT OF THE DATA AND NOT DOING ALL THAT HARMONIZATION UP FRONT RESULTS IN THINGS BEING A DISASTER FOR THE PERSON WHO IS RUNNING THE PROJECT AT THE END, SO SAVE THAT PERSON'S LIFE AS MUCH AS POSSIBLE, KEEP THEM WANTING TO DO ANOTHER PROJECT BY STANDARDIZING AT THE BEGINNING ALTHOUGH IT SEEMS LIKE MORE WORK. THE LAST THING, GIVE YOU A HEADS UP ABOUT OTHER DATASETS, THINGS OF INTEREST OR COMING ALONG IN THE NEXT 6-12 MONTHS. KEYVAN MENTIONED THE BRATS CHALLENGE. ONE THING THAT'S GREAT IS THAT THEY IN ADDITION TO REUSING DATA IN THE ARCHIVE FROM THE CANCER GENOME ATLAS HAVE CONTRIBUTED BACK THE 3D SEGMENTATIONS THAT THEY ARE USING AS GROUND TRUTH AND RADIOMIC FEATURE, TOOK FULL ADVANTAGE OF SCIENTIFIC DATA PUBLICATION PROCESS SO THIS IS A REALLY GREAT MODEL FOR OTHER PEOPLE TO LOOK TO. ON THE SOFTWARE SIDE OF THINGS, FRED COVERED A LOT OF THINGS IN THE PRISM PROJECT, THERE'S DEVELOPMENT EFFORTS AT NCI, SPONSORED BY CENTER FOR BIOINFORMATICS, AND THE NBIA SOFTWARE IS A COMPONENT THAT UNDERLIES THE DICOM SEARCHING QUERY, AND THEY HAVE BEEN WORKING FOR PROBABLY SIX MONTHS ON THIS NEW BETA INTERFACE TO BE ABLE TO SORT OF BRING THE LOOK AND FEEL INTO THE 21ST CENTURY, IT LOOKED LIKE SOMETHING FROM 1990 UNTIL NOW. AND WE'VE ADDED A LOT OF NEW COOL FUNCTIONALITY IN TERMS OF BEING ABLE TO SHARE DATASETS AND QUERIES AND I THINK A LOT OF THAT WILL BE REALLY HELPFUL FOR THE DIFFERENT KINDS OF PROJECTS YOU RUN HERE, SO WE'LL LET YOU KNOW ON THE WORKING GROUPS BUT THE BETA SHOULD BE STARTING REALLY SOON. WE ALSO RECOGNIZE THERE'S LOTS OF DIFFERENT KINDS OF USERS OUT THERE WHO WANT TO SUBMIT DATA, AND SOME OF THEM HAVE REALLY IN-DEPTH DICOM KNOWLEDGE, AND WOULD BE HAPPY TO WORK WITH SOMEONE LIKE BILL WHO COULD, YOU KNOW, KNOWS ALL THE NUANCES OF EVERY TENTH LEVEL OF SEQUENCE OF EVERY DICOM TAG, BUT THERE ARE A LOT OF OTHER PEOPLE WHO, YOU KNOW, WE'RE WORKING WITH PEOPLE ON THE CLINICAL SIDE, AND THEY JUST -- THIS IS SOMETHING THEY ARE DOING ON THE SIDE BETWEEN THEIR OTHER DUTIES AND THEY DON'T WANT TO MESS WITH SOMETHING AS COMPLICATED AT CPT, SO WE'VE BEEN DEVELOPING A FRONT-END WEB INTERFACE THAT TAKES A WIZARD-LIKE APPROACH STEPPING THROUGH THE PROCESS, COMPLEX EDITING, CONFIG FILES, SO THEY ARE STEPPING THROUGH SOMETHING MORE STRAIGHTFORWARD, AND PROBABLY LOOKING AT A FEW MONTHS BEFORE THE TESTING IS COMPLETE ON THAT. BUT THE FIRST RELEASE IS PRETTY MUCH READY TO GO. AND IF ANYBODY IN THIS GROUP WANTS TO HELP US PILOT IT, WE'D BE HAPPY TO HAVE FRIENDLY USERS TO TRY IT OUT. KEN TALKED ABOUT THIS YESTERDAY, REFRESHER ON THE ECOG-ACRIN, ISPY1 AND FLT-BREAST AND 6688 YOURS FOR THE TAKING UNTIL THE END OF THE YEAR, AFTER THAT IT WILL BE PUBLIC SO HIGHLY ENCOURAGE PEOPLE TO TRY TO COME UP WITH IDEAS TO TAKE ADVANTAGE OF THAT. THERE'S TWO ADDITIONAL ONES NEXT IN LINE, 6684/BRAIN FMISO AND 6668/NSCLC FDG-PET AND BRAIN FMISO SHOULD BE COMPLETED NEXT MONTH AND WE'RE AT THE HALFWAY POINT ON THE 6668. ONE OTHER THING IN TERMS OF CLINICAL TRIALS, WE'RE IN THE EARLY STAGES OF A PILOT ACTIVITY COLLABORATING WITH CTEP. THEY HAVE A NEW WEBSITE CALLED THE NCTN N CORP. DATA ARCHIVE, A DATABASE FOCUSED ON CLINICAL DATA ASSOCIATED WITH NCTN TRIALS, OBVIOUSLY A LOT OF THOSE TRIALS HAVE IMAGING DATA THAT HAVEN'T BEEN PROVIDED TO TCIA, BUT WE'RE USING THE RTO GTO 617 AS A PILOT, THE IDEA IS THE TWO SITES WOULD BE ABLE TO CROSS-LINK TO EACH OTHER SO WE WOULDN'T HOST OR DUPLICATE THE CLINICAL DATA IN THE ARCHIVE ON THEIR SIDE, AND NOR WOULD THEY BE TRYING TO COPY OUR SERVICES. WE'D JUST HAVE CROSS-LINKS TO EACH OTHER. ANOTHER MAJOR PROJECT THAT'S JUST STARTING TO SPIN UP IS THE CLINICAL PROTEOMICS TUMOR ANALYSIS CONSORTIUM, FOR ANYBODY IN THE CANCER GENOME ATLAS SETS, IT WILL INCLUDE EXTENSIVE PROTEOMIC ANALYSIS TO ADD ANOTHER LAY YOU ARE TO YOUR PRECISION MEDICINE QUESTIONS, ONE THING THAT IS A VALID CRITIQUE, IMAGES ARE DATED, A LOT OF THAT IS BECAUSE THE CALL FOR TISSUE INVOLVED A LOT OF PEOPLE JUST RAIDING THEIR TISSUE BANKS AND, YOU KNOW, A LOT OF THOSE SAMPLES WERE PRETTY OLD SO THE IMAGING THAT BELONGED TO THE PATIENTS WAS ALSO OLD. WITH CPTAC IT'S PROSPECTIVE, A TON OF SITES ARE CONTRIBUTING BUT TECHNOLOGY SHOULD BE NEWER AND STATE OF THE ART. THERE'S I BELIEVE TEN TOTAL CANCER TYPES PART OF THIS PROJECT BUT WE'RE ONLY COLLECTING FROM NINE OF THEM. I THINK WE HAVE DATA FROM IT'S LIKE SEVEN OR EIGHT OF THEM SO FAR, THERE WILL BE NEW DATA COMING ONLINE IN QUARTERLY RELEASES, EVERY FEW MONTHS SO YOU CAN KEEP AN EYE OUT, MORE PATIENTS ACCRUING UP TO A TOTAL OF 200 PER TISSUE I THINK. ANOTHER MAJOR NIH PROJECT OR NCI PROJECT IS THIS APOLLO, AND FOR THOSE THAT WENT TO THE SEMANTICS WORKSHOP, EARLIER THIS WEEK, YOU GOT A LITTLE BIT OF INTRODUCTION TO THIS. THIS IS SORT OF ALONG THE SAME LINES OF CPTAC, MULTI-DISCIPLINARY, YOU CAN SEE ALL THE KINDS OF DATA HERE, AND THE IDEA IS THAT THE IMAGING COMPONENTS OF THIS WOULD COME TO TCIA, AGAIN BE GENOMIC DATA GOING TO THE GENOMIC DATA COMMONS, PROTEOMIC DATA GOING TO PROTEOME IT DATA COMMONS. THIS IS RUN IN A FEW PHASES. THERE ARE A COUPLE PILOTS SORT OF ALREADY UNDERWAY OR COMPLETED. THE APOLLO 1 IS THIS LUNG DATASET. AND THEN APOLLO 2 IS AN OVARIAN DATASET. SMALL SUBSET, SEVEN CASES FROM APOLLO 1, I'M NOT EVEN SURE, THAT MIGHT BE MULTIPLE CANCER SITES. A SUBSET OF APOLLO 1 IS AVAILABLE, WE'LL BE EXTENDING THAT SO THERE WILL BE THE COMPLETE DATASET IN THE COMING MONTHS. APOLLO 2 IS IN THE SYSTEM ALREADY, BUT IS UNDER A DATA EMBARGO FOR A PERIOD OF TIME. ONCE THE TEAM GETS THEIR INITIAL PUBLICATION OUT YOU'LL BE ABLE TO GET ACCESS TO THAT. AND THEN THE BIG PROJECT COMING DOWN THE PIKE WOULD BE THE APOLLO 5, ALL PROSPECTIVE DATA COLLECTION, A NUMBER OF CANCER TYPES COLLECTED, MANY-YEARS-LONG PROJECT. THIS IS JUST YEAR 1. ONE OTHER THING I WANTED TO MENTION BRIEFLY IS THE CROWDS SECURE CANCER THAT WE DID AT RSNA LAST YEAR, JOSH REID LED THIS IN TERMS OF GETTING SET UP. WE TOOK DATA FROM THE CANCER GENOME ATLAS IN OVARIAN, RENAL, LUNG AND LIVER, AND MADE THAT AVAILABLE TO ATTENDEES AT THE MEETING BASICALLY SAID THIS IS -- WE HAD A FEW RADIOLOGISTS GIVE US TWO DIMENSIONAL MEASUREMENT OF WHERE THE TUMOR WAS IN ADVANCE OF THE MEETING, AND THEN AT THE MEETING WE ASKED PARTICIPANTS WANDERING BY TO GO AHEAD AND TRY TO FIND THE TUMOR AND DRAW THAT SAME MEASUREMENT. AND SO WE'LL BE ABLE TO DO A COMPARISON TO SEE FOR PEOPLE THAT WERE MOTIVATED TO WRITE A RESEARCH PAPER ABOUT THIS, HOW DOES THAT COMPARE TO THE PEOPLE THAT WERE JUST RANDOM PASSERS-BY AT A CONFERENCE, ALSO USED AS SEED DATA FOR ANYONE THAT WANTED TO USE ALGORITHMS, THAT BECAME PART OF THE BIDS CONVERSATION YESTERDAY, WE MIGHT BE ABLE TO USE THAT IN THE PIPELINE PROJECT, FIRST STEP IN THE PIPELINE PROCESS WITH ADDITIONAL STEPS BEING TO COMPUTE RADIOMIC FEATURES FROM SEGMENTATIONS AND BECAUSE OF CCGA THERE'S ALSO CLINICAL DATA SO WE COULD ADD A PREDICTION OF OUTCOMES STEP IN THERE, ALL OF THIS IS VERY EARLY DISCUSSION, YOU SHOULD JOIN US IN THE BREAKOUT THIS MORNING IF YOU WANT TO -- IF YOU'RE INTERESTED IN THAT. THE DATA SET ITSELF IS 99.999% DONE. I WAS HOPING TO FLIP THE SWITCH TODAY BUT I'M WAITING ON ONE MORE THING. HOPEFULLY IT WILL BE OUT LATER TODAY OR TOMORROW IF YOU WANT TO DOWNLOAD AND TAKE A LOOK AT IT. ONE OTHER THING THAT CAME UP IN THE PET CT MEETING YESTERDAY BREAKOUT WAS THE ONGOING DISCUSSION OF THE ONTOLOGY STANDARDIZATION FOR RADIOMIC FEATURES, I JUST WANTED TO POINT OUT ONE OF THE IDEAS FLOATED WAS TO USE THIS I THINK CREDENCE CLEARWATER RADIOMICS PHANTOM, SO THAT'S ANOTHER EXAMPLE OF SOMETHING WHERE THE WORKING GROUPS ARE TAKING ADVANTAGE OF THINGS THAT ARE IN THE ARCHIVE AND HOPEFULLY WHATEVER THE BY-PRODUCT IS AT THE END WE'LL BE ABLE TO PUBLISH SOMETHING ON THERE TO LINK BACK TO THIS DATASET. I THINK THAT'S IT. JUST A REMINDER WE DON'T EXPECT YOU GO TO THE SITE WHEN YOU DRINK YOUR MORNING COFFEE, BETTER TO SIGN UP FOR FACEBOOK OR LINKED IN SO YOU GET NOTIFICATION WHEN WE ADD MORE DATASETS. THANKS. [APPLAUSE] >> OKAY. QUESTIONS FOR JUSTIN? THERE. SANDY, YOU'RE NEXT. >> ONE OF THE SHORT COMING OF TCIA DATA HAS BEEN LACK OF CLINICAL OUTCOMES DATA. CPTAC IS DIFFERENT? >> WELL, SO -- YEAH. I MEAN THERE IS WITH THE CANCER GENOME ATLAS THERE IS SOME CLINICAL DATA OUTCOMES, BUT YEAH, I THINK THE LONGER TERM FOLLOW-UP WAS SOMETHING THAT THEY DIDN'T THINK THROUGH COMPLETELY AND SO, YEAH, THERE'S SOME OF THAT IS MISSING. WITH CPTAC IT'S PROSPECTIVE DATA COLLECTION SO YOU'RE WAITING TO SEE WHAT HAPPENS. I DON'T KNOW CONTRACTUALLY, YOU KNOW, IF THEY HAVE BETTER POLICIES IN PLACE TO REQUIRE PEOPLE OR IF THERE'S FUNDING TO MAKE SURE THAT GETS COVERED. >> THE BASELINE DATA ARE RICHER THAN TCGA? >> THEY ARE VERY, VERY BEGINNING. THERE'S THE DATA COLLECTION CENTERS WORKING ON CPTAC ARE DEBATING HOW TO MAP CLINICAL TERMS AND HOW TO FIGURE OUT -- WE'VE SEEN A COUPLE OF LIKE EARLY SPREADSHEETS BASICALLY THAT SAY THIS IS WHAT WE THINK WE'RE GOING TO PUT OUT, BUT THEY HAVEN'T EVEN AGREED FOR SURE SO IT'S A LITTLE BIT TOO EARLY TO SAY. BUT I'M HOPEFUL ALL THE THINGS THEY LEARNED IN THE CANCER GENOME ATLAS WOULD BE IMPROVED AS THEY DO THIS AGAIN. >> SANDY, YOU'RE NEXT. >> THAT WAS A GREAT PRESENTATION. TWO QUESTIONS. SO, ONE IS AS YOU KNOW WE'RE TRYING TO GET ONE OF OUR DATASETS INTO NATURE OF SCIENTIFIC DATA PUBLICATION, IN THE PROCESS OF DOING THAT REVIEWERS SEE THINGS THAT THEY WISH WERE INCLUDE AND ASK US TO INCLUDE AND THAT SORT OF THING SO WE'RE ITERATING ON THAT. I'M WONDERING WHAT HAPPENS AFTER EVERYTHING'S IN AND PUBLISHED, AND THEN YOU GET A QUESTION FROM SOMEBODY WHO IS TRYING TO USE YOUR DATA, SAY, BY THE WAY, DID YOU KNOW WHEN SOMETHING HAPPENED? DO YOU KNOW WHEN THE PATIENT GOT CHEMO, IT'S NOT IN YOUR DATA TABLE. AND THEN YOU GO AND UPDATE IT. IT'S NOW OUT OF SYNC WITH THE PUBLICATION THAT POINTS TO IT. IS THERE GOING TO BE A WAY TO ON THE TCIA SITE FOR INSTANCE PUT ADENDA TO THE PUBLICATION? YEAH, THIS MIGHT BE RISKY, BUT THERE'S -- OOPS. WE HAVE THE BEGINNINGS OF SOMETHING FOR THAT IN PLACE RIGHT NOW WHICH IS JUST WHEN YOU GO TO A DATASET, AND WE HAVE THIS TEMPLATE OF ALL THE INFORMATION ABOUT THE DATASET, ONE OF THE THINGS WE'RE DOING IS TRACKING VERSIONS. SO WHEN THERE'S SOME KIND OF CHANGE TO THE DATASET, WE CAN KEEP TRACK OF THIS DOWNLOAD LINK WILL ONLY GRAB YOU THIS .132 GIGABYTES HERE DURING THE FIRST VERSION, AND WHEN NEW SUBJECTS WERE ADDED THIS DOWNLOAD LINK WILL GRAB THE NEWEST VERSION. IN A CIRCUMSTANCE LIKE YOU DESCRIBED, IN FACT THIS JUST HAPPENED WITH DANIEL AND REBECCA FROM STANFORD WHO, YOU KNOW, WANTED TO CHANGE -- THEY FOUND AFTER THE DATA WAS OUT THERE,& THEY GOT A PEOPLE WHO REPORTED THINGS AND THEN THEY CAME BACK AND SAID WE'D LIKE TO FIX THIS. AND SO I ASKED THEM TO WRITE UP A NOTE BASICALLY THAT EXPLAINS WHAT THE CHANGES WERE, SO THAT SOMEBODY COULD SEE IT. THAT DOESN'T HELP WITH THE CONNECTION TO THE PUBLICATION. >> RIGHT. >> BUT AT LEAST SOMEBODY WHO GOES FROM THE PUBLICATION TO THE DATA SET WILL SEAT IN OUR SITE THERE WAS A CHANGE. >> SO THAT'S GOOD. AND THEN I LIKE THE IDEA OF PUTTING -- SHIP YOUR DATA TOOLS INTO TCIA SO WE CAN AVOID HAVING TO INSTALL A SEPARATE COPY OF CTP AND GO THROUGH THE WHOLE PROCESS. >> I MAY HAVE LED YOU ASTRAY A LITTLE BIT THERE. CTP IS STILL WORKING BEHIND THE SCENES. >> NO, BUT IT'S BEHIND THE SCENES. I MEAN, YOU KNOW, THAT'S PERFECTLY FINE. AND I THINK THAT WILL STREAMLINE THE PROCESS IN THE FUTURE. >> I HOPE SO. EVEN AS SOMEONE WHO USES CTP A LOT IT CAN BE DAUNTING SOMETIMES. >> WILL YOU BE ABLE TO SUPPLY TABLES, FOR INSTANCE, TO TRANSLATE SOME IDENTIFIERS INTO OTHER ONES LIKE FOR INSTANCE IF YOU HAVE A PATIENT NAME AND YOU WANT EACH PATIENT NAME TO BE CHANGED INTO A SPECIFIC DE-IDENTIFIED PATIENT NAME. >> YEAH, THAT WAS DEFINITELY PART OF THE PROCESS THAT WE WERE LOOKING AT. AND WHAT WE WANTED TO DO, I WAS GOING TO GO ONE MORE THAN THAT, USUALLY PEOPLE HAVE BASICALLY A SPREADSHEET ANYWAY THAT SAYS PATIENT A NEEDS TO NOW BE PATIENT 1. >> RIGHT. >> SAME THING WITH THE DATES. WE'RE FINDING IT'S MORE COMMON THAT PEOPLE ARE INTERESTED IN SETTING SOME KIND OF BASELINE DATE LIKE DATE OF DIAGNOSIS OR DATE OF ENROLLMENT IN A TRIAL. AND SO IN THIS NEW TOOL ONE OF THE THINGS YOU'LL ABLE TO DO WITH CONFIGURE STEP ON THE SCREEN HERE IS BASICALLY WE GIVE YOU A SPREADSHEET TEMPLATES THAT SCANS THE STUFF YOU PUT IN, HERE IS YOUR ORIGINAL PATIENT IDs, FILL IN WHAT YOU WANT THE NEW PATIENT ID TO BE IN THE SPREADSHEET AND FILL IN DATE OF DIAGNOSIS OR WHATEVER, DATE OF ENROLLMENT. SAVE THE SPREADSHEET, UPLOAD IT BACK INTO THIS WEB INTERFACE AND IT MAGICALLY DOES ALL THE RIGHT TRANSFORMATIONS FOR YOU. >> SHIFTING THE DATES. >> YEAH. >> THAT'S GREAT. >> YEAH. >> BY THE WAY, THIS IS ALL CAPTURED BY THE GOOGLE DATABASE, BECOMING PART OF THE ANSWER TO, OK, GOOGLE, WHAT IS TCIA. TAKAGAGA? >> NICE PROGRESS. THANKS. I HAVE A QUESTION. FOR ALL THE NCI-SPONSORED CLINICAL TRIALS, IS IT AUTOMATIC THAT THE IMAGES GO TO TCIA OR DOES THERE NEED TO BE A PROPOSAL EACH TIME, AND COULD WE CONCERT TO AUTOMATION? >> DO YOU WANT TO SAY ANYTHING ABOUT THAT? >> NO, I DON'T. IF THE IMAGING TEST IS ESSENTIAL TO DOING THE TRIAL THEY ARE COLLECTED INTO IROC. THEY DO NOT GO TO TCIA. WE CAN, IF WE CAN GET APPROPRIATE FUNDING, WITH DE-IDENTIFICATION AND ALL THAT, WE CAN DO THAT AND WOULD LIKE TO DO PROSPECTIVELY UNTIL THE TRIAL IS OVER. IF THERE'S RADIATION THERAPY THOSE ARE PROBABLY AVAILABLE BUT FOR THE OTHER STANDARD OF CARE IMAGING AT EVERY SITE, THAT'S A MIGHT FAIR. IT WAS AWFUL WITH TCGA TO GET IT. SO THE ANSWER IS NO. >> LARRY, MIC. >> YEAH, SPEAK INTO THE MIC. >> UNFORTUNATELY THE STANDARD OF CARE IMAGING FOR, YOU KNOW, RESPONSE ASSESSMENT INCLUDING SOMETIMES ADVANCED IMAGING ISN'T EVEN COLLECTED CENTRALLY. MOST OF THE TIME. SO MOST OF TIME IT'S NOT EVEN COLLECTED IN IROC, I'D LOVE THIS GROUP TO TAKE THAT ON. THAT ABSOLUTELY MAKES NO SENSE. WITH ELECTRONIC CAPTURE YOU'RE SHOWING THE COST OF DOING THAT RELATIVE TO THE VALUE THAT CAN POTENTIALLY HAVE IS TREMENDOUS. I DON'T THINK WE YET HAVE BUY-IN FROM, YOU KNOW, THE NCI ONCOLOGISTS THAT THAT'S NECESSARILY THE CASE. WE STILL PROBABLY HAVE SOME CONVINCING TO DO. I MEAN, WHAT WE TRY TO COLLECT ANYTHING THAT WE POSSIBLY CAN, IF THERE'S EVEN A SHADOW OF INTEREST, YOU KNOW, WE MAKE SOMETHING UP. YOU KNOW, TO COLLECT IT. AND WE'VE ALWAYS USED THE DATA, RIGHT? BUT, YOU KNOW, A STANDARD PHASE 3 TRIAL COMPARATOR ARMS MIGHT NOT ACTUALLY BE COLLECTED. >> ANY MORE QUESTIONS? OKAY. ONE MORE. >> SO I'D SAY ANOTHER THING WITH THE CPTAC GOING FORWARD ONE OF THE PROBLEMS WE'VE RUN INTO SIGNIFICANTLY WITH TCGA IT'S NOT JUST THAT YOU HAVE VARIABILITY IN CLINICAL DATA BUT YOU HAVE BLANKS. AND SO IF YOU HAVE YOUR THING, YOU DON'T KNOW WHAT A BLANK MEANS. BLANK MEANS A PERSON DIDN'T FILL IT OUT, DIDN'T GET CHEMOTHERAPY, IT'S NOT JUST WHEN/WHERE. SO I WOULD ENCOURAGE PEOPLE IF YOU'RE GOING TO HAVE A TEMPLATE AND PUT IT IN, MAKE IT A REQUIREMENT ALL THE FIELDS GET FILLED. BECAUSE IF YOU DON'T FILL THE FIELDS, YOU JUST DON'T KNOW. AND THAT MAKES YOUR ANALYSIS OF THE DATA ESSENTIALLY GUESSING, AND, YOU KNOW, WE GO INTO THESE THINGS, WE'RE GOING TO DO SPAR SAMPLING TO SOLVE THE PROBLEM BUT I DON'T BELIEVE THAT. I THINK IT'S A REAL PROBLEM. >> DEFINITELY GOOD POINT. >> OKAY. THANK YOU. THAT CONCLUDES THE CCP SESSION. >> VERY GOOD. WE'RE GOING TO START THE NEXT SESSION, WHICH IS GOING TO BE AN EXCITING ONE. WE'RE GOING TO TALK ABOUT QIBA AND QIN. BEFORE WE DO THAT, IS DR. KAHN IN THE ROOM? RIGHT IN FRONT OF ME. COME FORWARD. I WANT TO INTRODUCE A VERY IMPORTANT PERSON TO THE QIN RIGHT NOW. THIS IS THE SRO THAT'S GOING TO BE DOING REVIEWS OF YOUR APPLICATIONS. I WANT YOU TO MEET HIM. DO YOU HAVE A FEW WORDS TO SAY? >> WELL, I DON'T HAVE A MAGIC SOLUTION BUT I GUESS I'LL PROVIDE YOU SOME TIPS THAT CAN MAKE A DIFFERENCE IN YOUR APPLICATIONS HOW IT IS RECEIVED BY THE PANEL. FEEL FREE TO TALK TO ME ANYTIME DURING THIS MEETING. I MAY BE AVAILABLE OUTSIDE THE POSTER ROOM BETWEEN 11:00 TO NOON, SO STOP BY AND WE CAN DISCUSS ANYTHING THAT YOU HAVE. THANKS. >> HE TAKES TENS AND TWENTIES, I SHOULD SAY, BUT I WON'T SAY THAT. YOU DIDN'T HEAR THAT FROM ME. [LAUGHTER] OKAY. LET'S START THE NEXT SESSION. IT'S REALLY A PLEASURE FOR ME TO HAVE ED JACKSON HERE. WE'VE CROSSED PATHS MANY TIMES BETWEEN QIN AND QIBA AND LAST YEAR WE GOT TOGETHER AND SAID, HEY, WE REALLY OUGHT TO START SPEAKING FROM A COMMON POINT OF VIEW. AND WE HAD SOME TIME TOGETHER AT THE RSNA WHERE WE COULD GET SOME IDEAS GOING AND HE'S INVITED ME TO BE PART OF HIS STEERING COMMITTEE AND HE'S NOW PART OF OUR EC, SO WE'RE BUILDING THIS BRIDGE VERY CLOSELY, I'D LIKE FOR ED TO COME UP AND TALK ABOUT HIS PERSPECTIVES ON THAT. AND WE'LL HAVE SOME PEOPLE COME UP FOR A PANEL DISCUSSION AND KICK AROUND SOME IDEAS HOW QIN AND QIBA COULD GET CLOSER TOGETHER. ED? BY THE WAY, HE'S BEEN IN CALIFORNIA FOR HIS SON'S GRADUATION, HE HOPPED OVER TO CHICAGO FOR THE SECOND PART OF THE QIBA MEETING AND HOPPED ON A PLANE TO COME HERE FOR YESTERDAY. HE DOESN'T KNOW WHICH CITY HE'S IN. >> ACTUALLY SOMETIMES IT'S JUST A COMMON OCCURRENCE. THANK YOU FOR THE INVITATION, BOB. IT'S AN HONOR TO TALK ABOUT THE QIBA BUT I'M LOOKING FORWARD ABOUT TALKING ABOUT POTENTIAL INTERACTIONS. I'VE BEEN ASKED TO GIVE AN IDEA ON QIBA ACTIVITIES AND THOUGHTS ON POTENTIAL AREAS OF INTERACTION BETWEEN THE QIN AND QIBA. SO FOR THOSE WHO DON'T KNOW, QIBA WAS INITIATED BY THE RADIOLOGICAL SOCIETY OF NORTH AMERICA NORTH IN 2007, IT WOULD BE ONE APPROACH TO REDUCING VARIABILITY IN RADIOLOGY, TO BE EXTRACTION OF OBJECTIVE QUANTITATIVE RESULTS FROM IMAGING STUDIES. I SAY ONE APPROACH BECAUSE THERE ARE MANY OTHER WAYS TO REDUCE VARIABILITY, INCLUDING LEXICONS, ET CETERA. THIS IS THE MISSION STATEMENT, TO IMPROVE VALUE AND PRACTICALITY BY REDUCING VARIABILITY, SINCE THAT'S A MOUTHFUL WE REDUCE THAT TO SAYING WE'RE SEEKING TO INDUSTRIALIZE IMAGING BIOMARKERS. AND I THINK WE ALL AGREE, WE'RE MORE COMMON THAN WE ARE DIFFERENT BETWEEN OUR INITIATIVES HERE, THAT BIOMARKERS HAVE POTENTIAL FOR PRECISION MEDICINE AND SHOULD BE ENABLING OF PRECISION MEDICINE STARTING FROM THE BEGINNING OF PATIENT STRATIFICATION TO DETERMINE THE ALTERNATIVE STRATEGIES, ASSESSING PHENOTYPIC HETEROGENEITY LIKE NO OTHER IMAGING OR TECH TECHNIQUE. THIS IS A CRITICAL APPLICATION AND DOES REALLY REQUIRE QUANTITATIVE PART OF THAT. AND THEN MONITORING TREATMENT EFFICACY SHORT AND LONG-TERM SURVEILLANCE MONITORING. ALL OF THOSE WOULD BE ENABLED BY QUANTITATIVE IMAGING. THE BAR IS GETTING HIGHER FOR THE NEED FOR QUANTITATIVE IMAGING UNDERSTANDING BIAS AND VARIANCE OF OUR MEASUREMENTS, ILLUSTRATED WITH THE 2017 GUIDELINES. AND NOTICE IN THE POPULATIONS AT LOW RISK WHAT THE SIDES CRITERIA ARE FOR DIFFERENT TYPES OF MANAGEMENT OF THESE PATIENTS. SO IF A NODULE IS LESS THAN 6 MILLIMETERS, NO ROUTINE FOLLOW UP, 6-8, STEP 2. GREATER THAN 8 CONSIDER CT. THERE ARE DIFFERENT PATIENT MANAGEMENTS BUT BEING ABLE TO HIT SIZE CRITERIA REQUIRES PLUS OR MINUS 1 MILLIMETER IN VARIANCE OR AREA OF MEASUREMENTS. THAT'S THE BAR, WE CAN'T GET THERE IF WE DON'T UNDERSTAND, IN MY VIEW, BIAS AND VARIANTS OF OUR MEASUREMENTS, HOW COULD WE SAY WE'RE PROVIDING THIS LEVEL OF SERVICE? THIS IS THE PROBLEM OF COURSE. WE HAVE A MEASUREMENT, IT MIGHT BE 7, WE'RE TRYING TO PERHAPS PROVIDE THE FLEISCHNER CRITERIA. THERE ARE MANY CAUSES, PATIENT HANDLING, SEGMENTATION, ET CETERA. DIAGNOSTIC IMAGING EQUIPMENT IS NOT MANUFACTURED AT THIS POINT BEING MEASUREMENT DEVICES. IT'S REALLY BUILT TO PROVIDE THE BEST CONTRAST TO NOISE RATIO IN THE SHORTEST PERIOD OF TIME, IT'S A QUALITATIVE ASSESSMENT, NOT QUANTITATIVE. WE KNOW MEASUREMENT DEVICES, WHAT THEY ARE BASED ON. THERE'S A SPECIFIC MEASURING THE DEVICE CAN PRODUCE, WE KNOW THE CONFIDENCE INTERVALS OF MEASUREMENTS. HERE'S SPECIFIC REQUIREMENTS FOR THOSE BEING REPRODUCIBLE, VENDOR A, B, C, D, TIME POINT 1 THROUGH 50. MANUFACTURING CRITERIA AND COMPETITIVE ADVANTAGE IN THAT SPACE IS BEST IMAGE QUALITY IN THE SHORTEST TIME. TYPICALLY QUALITATIVE ASSESSMENT. THERE HAVE BEEN VERY FEW SPECIFIC REQUIREMENTS WITH LACK OF DETAILED ASSESSMENT OF SOURCES OF BIAS AND RARANCE -- VARIANCE, LEADING TO LACK OF STANDARDS. THERE'S BEEN LITTLE SUPPORT HISTORICALLY FROM IMAGING EQUIPMENT VENDORS. I DON'T SAY ANY A NEGATIVE PEJORATIVE WAY. THERE'S NOTHING TO CAUSE THEM TO DO SO FRANKLY, NO COMPETITIVE ADVANTAGE DOCUMENTED, NO CARROT NOR STICK IN MOST CASES. AND ALL OF THIS LEADS TO VARYING MEASUREMENT RESULTS ACROSS VENDOR CENTERS AND/OR TIME, HENCE THE MISSION STATEMENT OF QIBA. SO QIBA'S APPROACH HAS BEEN FROM THE VERY BEGINNING ADOPTING METROLOGY PRINCIPLES. WITH ANY BASIS WE LOOK AT SOURCES OF BIAS AND VARIANCE. ACCURACY IS VERY DIFFICULT TO ASSESS FOR QUANTITATIVE IMAGING TOOL OR ANY OTHER TECHNIQUE BECAUSE THERE'S ABSENCE OF REFERENCE STANDARDS, FEW GROUND TRUTH MEASURES FOR MANY QUANTITATIVE IMAGING BIOMARKERS WE USE. AND THIS IS OF COURSE POTENTIAL ROLE FOR APPLICATION-SPECIFIC PHANTOMS WHICH I'LL COME TO TOWARDS THE END. THE OTHER PART OF THIS OF COURSE IS WHAT'S THE VARIANCE OR PRECISION OF OUR MEASUREMENT, REPEATABILITY, BUT ALSO REPRODUCIBILITY LOOKING AT DIFFERENT OPERATORS DOING THIS ON DIFFERENT DAYS, SEPARATED BY LARGER PERIODS OF TIME. SO IN THE FORMER REPEATABILITY WE ASSESS REPEATABILITY COEFFICIENTS AND OTHER MEASURES REPRODUCIBILITY. ONCE YOU ESTABLISHED LEVELS OF BIAS AND VARIANCE THAT REMAIN AFTER MITIGATION STEPS YOU HAVE THE CONFIDENCE INTERVALS AND KNOWING THESE INTERVALS AS NANCY SUMMARIZED AND ERIC YESTERDAY, ALLOWS US TO DEVELOP STATISTICAL VALID STUDY DESIGNS WITH ADEQUATE POWER AND FEWERS NUMBER OF PATIENTS, THIS IS IMPORTANT BECAUSE IN THIS SIMULATION WHERE WE'RE LOOKING AT VARIOUS EFFECT SIZE ON THE X-AXIS AND REQUIRED SAMPLE SIZE ON Y-AXIS FOR 80% POWER IF WE HAVE A DESIRE TO SEE A 20% EFFECT SIZE CHANGE, THEN DEPENDING ON THE LEVEL OF OUR VARIANCE FROM 10 TO 40 POSITIVE MAKES PROFOUND IMPACT ON NUMBER OF PATIENTS REQUIRED TO HIT THE LEVEL. BY REDUCING THIS WE CAN POWER OUR TRIALS WITH THE FEWERS NUMBER OF PATIENTS BUT WE HAVE TO UNDERSTAND AND CHARACTERIZE THE UNDERLYING CONFIDENCE INTERVALS, BIAS AND VARIANCE OF OUR MEASUREMENTS. I THINK EVERYONE AGREES, THERE'S BEEN MANY PEOPLE SAYING THIS IN THIS MEETING AND OTHERS THERE'S NEED FOR DATA SHARING. THE DATA THAT WE'RE ACQUIRING IS EXPENSIVE. INDIVIDUAL TRIALS TYPICALLY HAVE SMALL NUMBERS OF PATIENTS, TYPICALLY IN PHASE 1, PHASE 2 SETTING. DATA COMING IN, WE WOULD AGREE, LEADS TO POOLED HIGHER QUALITY DATASET THAT BE USEFUL FOR ALL OF THESE REASONS AND MANY MORE SO META-ANALYSIS STUDIES, APPLICATIONS OF IMAGING, RADIOMICS, AND MOVING FORWARD EVIDENCE-BASED MEDICINE AS WELL AS COMPARATIVE EFFECTIVENESS STUDIES WOULD BENEFIT FROM THIS TYPE OF POOLED DATA. THIS IS FROM PAUL KINAHAN, BY ASSESSING YOU CAN HARMONIZE, THIS IS A PET EXAMPLE, X-AXIS IS UPDATES, Y-AXIS IS SMOOTHING APPLIED FOR VARIOUS VENDORS, 68 DIFFERENT IMAGING CENTERS. AND IF YOU USE THOSE ACQUISITION PROTOCOLS AND PARAMETERS, AND THEN MEASURE THE RC, RATIO OF OBSERVED ACTIVITY TO ACTUAL ACTIVITY CONCENTRATION FOR A RANGE OF DIAMETER OBJECTS BECAUSE OF VARIATION IN TIMES OF RECONSTRUCTION PARAMETERS USED IN THE PET RECONSTRUCTION YOU GET A WIDE RANGE OF CHARACTERISTICS OF THAT. IF YOU GO THROUGH THE PROCESS OF THE IMAGING METROLOGY, YOU CAN USE THAT DATA TO HARMONIZE RESULTS TO SAY IT DOESN'T MATTER WHAT THE VENDORY IS OR IS, IF WE HAVE DATA WE CAN HARMONIZE THE MEASURES FOR CLINICAL TRIAL ET CETERA. THERE'S A LOT OF REASONS FOR GOING THROUGH THESE STEPS. QIBA WAS FORMED IN 2007, I KNOW YOU CAN'T READ THIS. IT'S NOT MEANT TO BE AN EYE CHART. I WANT TO HIGHLIGHT THIS IS THE CURRENT ORGANIZATIONAL STRUCTURE BUT WHAT'S IMPORTANT ABOUT THE ORGANIZATIONAL STRUCTURE? SEVERAL THINGS. ONE, QIBA HAS OVER THE YEARS BEEN JOINED BY OVER 1,050 INDIVIDUALS FROM EVERYTHING SHOWN ON THE RIGHT, ACADEMIC RADIOLOGY, IMAGING SCIENCE, EQUIPMENT VENDORS, ET CETERA. THERE ARE 18 REPRESENTATIVES FROM THE FDA THAT ARE INVOLVED. THERE ARE 22 OTHER GOVERNMENT REGULATORY AGENCY INVOLVEMENT, WE HAVE NUMEROUS INDIVIDUALS FROM NIST AND NIDA, NANCY SPOKE YESTERDAY IS HEAVILY INVOLVED, IN FACT IS THE LEAD STATISTICIAN FOR THE QIBA EFFORTS. SO, THAT I THINK IS REALLY IMPORTANT TO HAVE A VERY BROAD STAKEHOLDER BASIS, AND I WILL ALSO SAY THAT 98% OF ALL THESE EFFORTS ARE VOLUNTARY FROM THESE INDIVIDUALS. SO IT'S REALLY AMAZING SHOWING THE LEVEL OF COMMITMENT THAT ALL OF THESE INDIVIDUALS ACROSS STAKEHOLDER GROUPS HAVE TO QUANTITATIVE IMAGING AND APPLICATIONS IN THE FUTURE OF MEDICINE. SO THAT REPRESENTATION IS TRUE AT THE STEERING COMMITTEE LEVEL THAT BOB MENTIONED. IT'S ALSO TRUE AT THE LEADERSHIP OF EACH AND EVERY ONE OF THE MODALITY COMMITTEES WHICH ARE SHOWN FOR CT, MR AND ULTRASOUND AND FOR ALL THE UNDERLYING BIOMARKER COMMITTEES AND TASK FORCES THAT ACTUALLY DO THE WORK OF QIBA THAT I'LL DESCRIBE IN JUST A MOMENT. IT'S ALL ORGANIZED BY MODALITY. THERE'S A PROCESS COMMITTEE TO MAKE SURE THESE EFFORTS ARE HARMONIZED ACROSS THE VARIOUS EFFORTS OF QIBA. WE DO HAVE A QUANTITATIVE IMAGING DATA WAREHOUSE I'LL TALK ABOUT TOWARD THE END AND THEN OF COURSE THERE'S A SUSTAINABILITY, YOU CAN'T RUN FOREVER WITH 98% VOLUNTARY EFFORTS. THERE ARE FUNDING REQUIREMENTS THAT ARE NECESSARY AND SO THAT'S THE AREA THERE. THERE ARE SCIENTIFIC LIAISONS INCLUDING TOM, PAUL FOR NUCLEAR MEDICINE. NAMES ARE COMMON TO QUANTITATIVE IMAGING NETWORK. THE GOAL OF QIBA TO TAKE THAT MEASUREMENT PROBLEM, REALLY UNDERSTAND THE SOURCES OF BIAS AND VARIANCE AND DEVELOP WHAT WE CALL PROFILES WHICH ARE REALLY STANDARDS DOCUMENTS, TECHNICAL PERFORMANCE DOCUMENTS THAT OUTLINE WHAT MUST BE DONE FOR REQUIREMENTS FOR ACQUISITION PARAMETERS, RECONSTRUCTION, RESOLUTION, ET CETERA. WHAT ARE THE IMAGING PROCESS REQUIREMENTS, IF SEGMENTATION IS INVOLVED, WHAT ARE THE STEPS FOR THAT. WHAT ARE THE REQUIREMENTS FOR EACH OF THESE SUCH THAT IF ALL OF WHAT WE CALL ACTORS IN THE IMAGING CHAIN FROM START TO FINISH FOLLOW THE QIBA PROFILE, THAT YOU WILL LEAD TO A MEASUREMENT THAT HAS MUCH MORE ESTABLISHED, MUCH TIGHTER CONFIDENCE INTERVALS, THAT'S THE OVERALL GOAL. WE DON'T TAKE IN QIBA ENVELOPE OR PORTFOLIO, ONE, WE'RE NOT DEVELOPING NEW TOOLS. THAT'S THE DOMAIN OF THE QIN. WE TAKE EXISTING TOOLS THAT ARE SHOWN TO BE -- OR SATISFY THESE FIVE PRIMARY CRITERIA, NOT THE MOST CUTTING EDGE TOOLS THAT ARE OUT THERE. BUT THEY ARE TOOLS SHOWN TO HAVE POTENTIAL FOR BEING TRANSFORMATIONAL, TRANSLATIONAL, FEASIBLE, PRACTICAL AND COLLABORATIVE. THE GOAL IS THESE ARE TOOLS THAT SHOULD BE ABLE TO BE TRANSLATED IN THE CLINICAL TRIAL ENVIRONMENT IN A FEW YEARS OF HORIZON, NOT 10-YEAR HORIZON OR MORE. THERE'S A COORDINATION OF GROUND WORK WHERE IT SAYS WHAT IS LITERATURE THAT'S AVAILABLE, META-ANALYSIS OF LITERATURE WHEN POSSIBLE, NOT OFTEN POSSIBLE, THAT'S A WHOLE DIFFERENT TOPIC OF DISCUSSION. BUT LOOKING FOR THE LEVELS AND BIAS AND VARIANCE, WHAT ARE THE HOLES THAT NEED TO BE FILLED IN. THE PROFILE DRAFTING BEGINS. THIS IS THE BEGINNING OF WHAT I WAS SAYING IS TECHNICAL PERFORMANCE STANDARD. IT IS PUBLISHED, AND IN THAT PROCESS IT GOES TO A VERY WIDE STAKEHOLDER BASIS, I'M SURE MANY OF YOU IF NOT ALL HAVE SEEN THE INVITATIONS TO REVIEW THE PROFILES IN YOUR VARIOUS HATS WE ALL WEAR. THOSE WITH BE USED TO VALIDATE EQUIPMENT AND SITES FOR SITE QUALIFICATION, ET CETERA. I'VE LISTED THE DRAFT PART OF ONE HERE THERE'S A 95% PROBABILITY THAT A MEASURED CHANGE OF THIS RANGE ENCOMPASSES TRUE VOLUME CHANGE FOR SOLID TUMORS GREATER THAN X. IT TELLS THE USER IS THIS SOMETHING THAT WOULD WORK IN MY CLINICAL TRIAL SETTING, ET CETERA. IF YOU'RE AN EQUIPMENT VENDOR WHY DO YOU WANT ME TO DO THIS? THERE'S A SERIES OF PROFILE ACTIVITIES, TELLS THE USER WHAT I NEED TO DO, HOW DO I ACHIEVE THE CLAIMS STATED. TELLS THE VENDORS WHICH PRODUCTS ARE AFFECTED, WHAT I HAVE TO DO TO BE COMPLIANT. LASTLY THERE'S ASSESSMENT PROCEDURES. IN THIS CASE USER AND EQUIPMENT ARE THE SAME. HOW DO I DEMONSTRATE I CONFORMED WITH THIS PROFILE? WE HAVE TWO MAIN CLAIM LANGUAGES, NANCY ALLUDED TO THIS YESTERDAY. CROSS-SECTIONAL CLAIMS, THIS IS A SINGLE TIME POINT SO MEASURED VOLUME OF X. 95% CONFIDENCE INTERVAL TRUE VOLUME IS X PLUS OR MINUS SOME PERCENTAGE. LONGITUDINAL SIMILARLY FOR A MEASURED CHANGE IN SOME PARAMETER X, THERE'S A 95% CONFIDENCE INTERVAL FOR THAT. AND THEN WE'RE VERY SPECIFIC TO SAY THERE IS FINE PRINT, CLAIM WILL HOLD WHEN, YOU KNOW, THE RELATIVE CLINICAL INFORMATION IS THERE. SO THESE ARE VERY MUCH THE WAY THAT EACH OF THESE PROFILES ARE DEVELOPED. NOW, I DON'T EXPECT YOU TO READ THIS BUT THIS IS BASICALLY THE WHOLE AREA WHEN WE STARTED THIS IN 2007 ABOUT THREE YEARS LATER IT WAS PRETTY CLEAR THAT WE NEEDED TO HAVE A COMMON METROLOGY SET OF PRINCIPLES TO FOLLOW ACROSS MODALITIES. MARY ELLEN WAS INVOLVED, NANCY WAS ONE OF THE LEADERS, ERIC WAS INVOLVED IN THIS, IN DEVELOPING WHAT WE CALL THE IMAGING METROLOGY WORK GROUP THAT LED TO A SERIES OF PUBLICATIONS THAT I'LL SHOW YOU IN JUST A MOMENT THAT MIGHT BE OF INTEREST. BUT ALSO UNDERLIES EVERYTHING IN TERMS OF DEVELOPMENT OF THESE CLAIMS, IF THEY ARE CROSS-SECTIONAL HOW DO WE CHARACTERIZE BIAS, YOU CAN FOLLOW ONE SET IF NEGLIGIBLE, IF UNKNOWN A DIFFERENT SET. IF LONGITUDINAL, SYSTEMS ARE THE SAME SYSTEM AT ALL TIME POINTS SCENARIOS ALLOWS DEVELOPMENT OF ROBUST CLAIMS, IF NOT CHARACTERIZE THE BIAS. THIS IS THE CLAIM GUIDANCE THAT HAS BEEN DEVELOPED OUT OF THE METROLOGY WORKING GROUPS LED PRIMARILY BY NANCY. WE ALSO HAVE A SERIES OF PROFILE STAGES, STARTING WITH DRAFT FOR PUBLIC COMMENT, THIS LITERALLY IS JUST WHAT THAT IS. THAT DRAFT IS THEN RELEASED BROADLY TO A LARGE GROUP OF STAKEHOLDERS CENTRALLY MAINTAINED BY RSNA STAFF. ALSO IF THERE'S A SPECIFIC PROFILE BEING RELEASED, AND THERE'S A TARGET AUDIENCE THAT PEOPLE WANT TO MAKE SURE THAT SEES THAT, THOSE ARE INCLUDED IN THAT DATABASE AS WELL SO IT'S VERY BROADLY DISTRIBUTED, THOSE COMMENTS COME BACK AND ARE RESOLVED BY WRITING GROUP AND THAT LEADS TO CONSENSUS OR TAKE 2 TECHNICAL STANDARD, MOVES TO TECHNICALLY CONFIRMED STAGE. WHAT WE MEAN BY THAT IS PROCESS BY WAY CONFORMANCE TO THE PROFILE HAVE BEEN DEFINED AND ALSO THAT THERE'S BEEN FEASIBILITY TEST IN THE FIELD. THAT THESE PROFILES HAVE BEEN IMPLEMENTED FROM START TO FINISH, AND AT LEAST TWO CENTERS, USING EQUIPMENT FROM AT LEAST TWO DIFFERENT VENDORS, AND SHOWN TO BE PRACTICAL AND FEASIBLE. IT DOESN'T SHOW THE CLAIM WAS CONFIRMED IN THOSE SITES BUT YOU CAN REALLY IMPLEMENT THIS IN A REAL WORLD SETTING. SO THAT LEADS TO TECHNICALLY CONFIRMED STAGE, CLAIM CONFIRMED MEANS IMPLEMENT AND SHOWN THE CLAIM HAS HELD TRUE IN THOSE SITES AND THEN CLINICALLY CONFIRMED WHERE WE ALL WISH TO BE. AND THAT IS THE BROAD BASED DISSEMINATION OF PROFILES AND IMPLEMENTATION SHOWING CLAIM IS BROADLY CON CONFIRMED. ANOTHER ILLUSTRATION WHERE WE'RE MORE ALIKE THAN DIFFERENT, I NOTICE BENCHMARKS, HAVE A REALLY SIMILAR PATHWAY AND I THINK THIS IS AN AREA OF OPPORTUNITY FOR COLLABORATION. UNDOUBTEDLY. SO FROM BASIC BENCHMARK TO TECHNICAL TO CLINICAL, TO CLINICAL USE, SO THIS IS A NICE AREA OF ALIGNMENT WHICH I THINK, AGAIN, ITERATES ON OUR ABILITY TO HAVE STRONGER COLLABORATIONS. SO WHAT'S THE CURRENT STATUS? WE HAVE 20 PROFILES IN DEVELOPMENT BROKEN DOWN BY MODALITY. MOST ADVANCED IS FTG PET, RICH AND PAUL AND OTHERS M THIS GROUP WERE INSTRUMENTAL IN THIS, THIS IS TECHNICALLY CONFIRMED STAGES, THEY ARE LOOKING AT CLINICAL TRIALS TO MOVE THIS TO THE CLAIMED, CONFIRMED, EVEN CLINICAL CONFIRMED STAGE SOON. CONSENSUS STAGE, TWO PROFILES IN CT DOMAIN FOCUSED ON LONG NODULE VOLUMES, TUMOR VOLUMES, PERIOD. WE HAVE OUR FIRST SPECT PROFILE AND DCE PROFILE, WE HAVE FOUR THAT ARE IN PUBLIC COMMENT, CLOSING PHASES OF PUBLIC COMMENT. YOU CAN SEE THREE ARE MR, ONE IS ANOTHER NUCLEAR MEDICINE FOR PET AMYLOID FOR ALZHEIMER'S DISEASE, THREE IN STAGES OF BEING RELEASED FOR PUBLIC COMMENT, YOU CAN SEE THOSE THERE. WE HAVE A WHOLE HOST OF OTHERS THAT IN EARLIER STAGES OF DEVELOPMENT. I WILL MENTION THE LAST TWO ILLUSTRATIVE THAT QIBA IS VERY MUCH NOW AN INTERNATIONAL GROUP. WE COLLABORATE BOTH DOMESTICALLY, FOR EXAMPLE, WITH AMERICAN INSTITUTE OF ULTRASOUND AND MEDICINE, ON CERTAIN ULTRASOUND BIOMARKER DEVELOPMENTS, AND WE HAVE OUR FIRST OFFICIAL COLLABOATION WITH THE EUROPEAN SOCIETY OF RADIOLOGIES, EIBALL, FOCUSING ON ARTERIAL SPIN LABELING. THERE'S A QIBA JAPAN GROUP, CLOSE COLLABORATIONS WITH THAT GROUP AS WELL. I MENTION THE METROLOGY WORKING GROUP PUBLICATIONS, FIVE, STATISTICAL METHODS FOR RESEARCH IN 2014-15, AND THEN IN RADIOLOGY FOLLOWING YEAR DAN SULLIVAN, NANCY AND OTHERS HAVE A PAPER, METROLOGY STANDARDS, OVERVIEW AND REFERENCE TO MORE DETAILED MANUSCRIPTS THAT CAME OUT BEFORE THAT. SO IN ADDITION TO THE PROFILES, FREQUENTLY PEOPLE HEAR QIBA AND THAT'S ALL WE THINK OF, IS PROFILES, THE DELIVERABLE, THAT'S IT BUT WE DON'T THINK OF IT THAT WAY. OBVIOUSLY IF YOU'RE GOING TO HAVE THESE PROFILES OF TECHNICAL PERFORMANCE STANDARDS, TELLING PEOPLE THIS IS HOW YOU DEMONSTRATE CONFORMANCE YOU HAVE TO HAVE TOOLS FOR THEM TO DO THAT WITH. WE HAVE A SERIES OF PHYSICAL PHANTOMS AS WELL AS DIGITAL REFERENCE OBJECTS DEVELOPED, THE FORMER IS FOCUSED ON ACQUISITION SIDE OF DATA, AND LATTER OF THOSE IS FOCUSED ON THE VARIOUS IMAGING PACKAGES OR RECONSTRUCTION PACKAGES THAT ARE USED TO ANALYZE THE DATA, TO ALLOW US TO BE VERY MUCH IN A CHALLENGE SITUATION, IN FACT ONE OF THESE DIGITAL REFERENCE OBJECTS WAS USED IN A QIN CHALLENGE, MAYBE MORE BUT I KNOW OF AT LEAST ONE. SO I THINK ANOTHER PLACE WHERE WE HAVE NATURAL INTERACTIONS. BUT THIS IS REALLY IMPORTANT TO BE ABLE TO COMPARE TOOLS OR COMMERCIAL PACKAGES ACROSS VENDORS TO SAY DO I GET THE SAME RESULT EVEN IF I THROW THE SAME INPUT DATA TO IT. AND I'LL COME BACK TO THAT AS AN EXAMPLE IN JUST ONE MOMENT. THERE'S A HOST OF PHYSICAL PHANTOMS LISTED HERE, HOST OF DIGITAL REFERENCE OBJECTS. THIS IS A PHYSICAL PHANTOM, TOM WAS HEAVILY INVOLVED, AND MIKE FROM NIST BOULDER. INCLUDES VIALS FILLED WITH PVT TO MIMIC ICE WATER TEMPERATURES, IN A NIST TRACEABLE WAY. TOM'S GROUP DEVELOPED THE SOFTWARE THAT WOULD ANALYZE DATA FROM THIS, SO IF WE WERE USING THIS IN A CLINICAL TRIAL SETTING FOR EXAMPLE THE PHANTOM WOULD BE AVAILABLE TO TEST ACQUISITION, SOFTWARE AVAILABLE TO DO ANALYSIS, SO WE DON'T HAVE MULTIPLE SETS OF SOFTWARE ANALYZING DATA AND JUST INTRODUCING VARIANTS INTO THE MEASUREMENT DUE TO MULTIPLE SOFTWARE PACKAGES. THIS IS A SET THAT'S AVAILABLE. DEVELOPMENT OF PHANTOMS, THIS IS COMING FROM COLUMBIA GROUP, ZHAO WAS INVOLVED AND NICK PETRICK, REALISTIC LESIONS FOR ARTERIAL AND PORTAL VENOUS FEYS PHASE, LIVER. THIS IS WORK OF PAUL KINAHAN, THE OBJECT HAS WELL KNOWN GROUND TRUTH FOR SUV, DIFFERENT REGIONS OF INTEREST ANALYSIS, THIS WAS DONE AS A CHALLENGE. I THINK IT WAS 20-SOME-ODD VENDORS PROVIDED RESULTS, THIS IS THE FIRST PASS RESULT OF SUV. 25% NOT ONLY DIDN'T GET CLOSE, JUST HAD WRONG VALUES OF SUV. 30% WERE MARGINAL. AND SO THIS LED TO MANY SOFTWARE COMPANIES REVISING THEIR CODE, AND ACTUALLY RE-RELEASING THE SOFTWARE. SO THIS TYPE OF DIGITAL REFERENCE OBJECT IS IMPORTANT. TO GO FROM THAT EXTREME TO THE OTHER EXTREME OF MORE ANATOMICALLY APPROPRIATE DIGITAL REFERENCE OBJECTS, THIS IS WORK OF SAMEI AT DUKE, PATIENT SCAN DATA BACK TO RAW DATA, SONOGRAM, INJECT INJECT BACK INTO SONOGRAM SPACE, ACTUALLY GETTING A SERIES OF LESIONS THAT I THINK YOU WILL AGREE LOOK VERY REAL IN THE SETTING OF LIVER, LUNG AND REGION. BY NOW YOU'VE DECIDED WHICH OF THE PANELS IS REAL, WHICH IS SYNTHETIC, BUT THOSE ARE THE ANSWERS. WE CAREFULLY KNOW THE EXACT VOLUME OF THESE LESIONS, YOU CAN USE THIS TO COMPARE DIFFERENT IMAGING TOOLS, OR DIFFERENT SOFTWARE PACKAGES. SIMILARLY WE'VE DONE WORK FOR DCE TAKEN SYNTHETIC DATA FOR THE HUMAN BRAIN, GRAY AND WHITE MATTER, INJECTED LESIONS FOR WHAT WE KNOW, DCE PARAMETERS. SO COMING TO THE CONCLUSION OF THIS, THIS IS JUST SOME SAMPLES OF WHAT WE CONSIDER DELIVERABLES QIBA BEING YES TO PROFILES, UNDOUBTEDLY, BUT A SUITE OF IMAGING METROLOGY TOOLS, PROCESSES DEVELOPED ALONG THE WAY, ALL THESE DIGITAL REFERENCE OBJECTS AND PHYSICAL PHANTOMS. QIBA, QIN INTERACTIONS OVERLAP IN MISSION, BOTH SEEK THE SAME THING, TRANSLATING TO BIOMARKERS AND PATIENT CARE, UNDOUBTEDLY THAT'S THE COMMON THING. WE BOTH HAVE DELIVERABLES APPLICABLE TO CO-CLINICAL TRIALS. WE HAVE NO OVERLAP IN SPECIFIC DELIVERABLES, QIN TOOLS DATA PIPELINES, CHALLENGES HAVE BEEN LEADING FROM THIS, QIBA REALLY STANDARDS, IMAGING METROLOGY PROCESS, PHANTOMS AND SUPPORTER STANDARDIZATION AND TOOL INTER-COMPARISON. WE'VE HAD NATURAL INTERACTIONS, MANY COMMON INDIVIDUALS, I'VE NAMED SOME, THERE ARE OTHERS, LARRY SCHWARTZ, RICH WALL, THAT'S BEEN NATURAL. AND AS BOB MENTIONED WE HAVE RECIPROCAL APPOINTMENTS ON LEADERSHIP COMMITTEES. IDEAS FOR FUTURE, HOPEFULLY EXPANDED SYNERGISTIC COLLABORATIONS, I STRESS THESE THINGS BECAUSE IT SEEMS TO ME THAT WOULD BE A VERY GOOD NATURAL FIT FOR THE QUANTITATIVE IMAGING NETWORK BENCHMARKING AND HOW MIGHT WE COLLABORATE ON THOSE, ADVANCING TOOLS THROUGH THE BENCHMARK LEVELS. I THINK SOME OF THE METROLOGY PROCESSES AND MIKE ALLUDED TO THIS IN HIS COMMENT AND QUESTION EARLIER, I THINK OBVIOUSLY SUPPORTING TRANSLATION OF TOOLS TO CLINICAL TRIAL APPLICATION, METROLOGY, BIAS AND VARIANTS. ONE GROUP, IT'S A NICE TOOL, WHAT'S THE CONFIDENCE INTERVAL SO I CAN POWER APPROPRIATELY. SUPPORTING CO-CLINICAL TRIAL EFFORTS IS ANOTHER NATURAL FIT. ENGAGEMENT OF QIBA STAKEHOLDER DOMAIN, I MENTIONED A BROAD BASIS, INDUSTRY, FDA AND OTHERS, I THINK IS ANOTHER AREA OF NATURAL COLLABORATION. AND I THINK THE REAL INTERESTING THING PERSONALLY PAIRING OF NETWORK TOOLS WITH QIBA PROFILES, WE'VE GOT STANDARDIZATION, EMBEDDED TOOLS COMING FROM THE QUANTITATIVE NETWORK. I WOULD LOVE TO SEE THIS HAPPEN. WE'VE HAD PRELIMINARY DISCUSSIONS OF HOW THAT MIGHT HAPPEN. AND THAT MIGHT NATURALLY LEAD TO SOME QUANTITATIVE IMAGING NETWORK APPLICATIONS THROUGH TEAMS THAT DEVELOP BETWEEN QIBA AND QUANTITATIVE IMAGING NETWORK. I'LL STOP WITH THAT WITH ACKNOWLEDGMENTS. SORRY I WENT OVER. HOPEFULLY THAT HAS INITIAL FOOD FOR THOUGHT FOR DISCUSSIONS. I THANK ALL THE PEOPLE THAT LET ME SHOW THEIR DATA AS IF I DID THAT, BUT IT'S UNDERLINED BY A HUGE EFFORT OF VOLUNTEERS, PRIMARILY, THREE CONSECUTIVE TWO-YEAR AWARDS OF CONTRACTS FROM NIBIB THAT ALLOWED US TO DEVELOP GROUND WORK PROJECTS, PHANTOMS, DROs, THE OTHER THINGS I MENTIONED. SO I WANT TO ACKNOWLEDGE THAT FROM THE NIBIB. I'LL STOP THERE. THANK YOU VERY MUCH. [APPLAUSE] >> RATHER THAN ENTERTAIN QUESTIONS, SPECIFICALLY HERE, I'D LIKE TO HAVE A COUPLE OF THE FOLKS THAT ARE CROSSING THE PATHWAY, MARYELLEN YOU'RE PART OF IT, IS RICH STILL HERE? MIKE, YOU'RE PART OF IT. LARRY. AMITA IS STILL HERE. SOME FOLKS, SOME OF THESE GUYS BOTH QIBA AND QIN CENTRIC, I'D LIKE YOU TO COME UP HERE AND WE'LL THROW SOME QUESTIONS OUT TO YOU TO SEE WHAT YOUR THOUGHTS ARE ON HOW WHAT ED SAID COULD ACTUALLY BE A LEVEL OF INTERACTION. WHILE RICH IS COMING UP, THE FIRST QUESTION I THOUGHT TO ASK, ED WAS TALKING ABOUT NO OVERLAP, NO INTERACTION WITH QIN AND QIBA, AS A POSITIVE THING. IN OTHER WORDS, WE EACH HAVE OUR OWN PART OF THE VENN DIAGRAM. BUT IS THAT REALLY A GOOD THING? OR SHOULD WE BE LOOKING FOR AREAS WHERE THERE IS OVERLAP? >> SO LET ME CLARIFY WHAT I MEANT. QIN HAS BEEN FOCUSED ON TOOLS, DATA PIPELINES, ET CETERA, TO MOVE INTO CLINICAL TRIALS. WE HAVE -- QIBA HAS FOCUSED MORE ON THE PROCESSES, STANDARDIZATION, AND DEVELOPMENT TOOLS TO DEMONSTRATE CONFORMANCE WITH THOSE STANDARDS. AND SO, YES, THERE'S NATURAL OVERLAP, AND I HOPE I HIGHLIGHTED A FEW OF THOSE. BUT WHAT I MEANT TO SAY BY THAT I GUESS, THIS IS NOT A COMPETITIVE SPACE. THIS IS REALLY A COLLABORATIVE VENTURE. >> I WAS HOPING YOU'D SAY THAT. IT WAS A TRICK QUESTION. NO. OKAY. SO IF I THINK OF -- WHEN I THINK OF QIBA THE FIRST THING THAT COMES TO MIND ARE PROFILES. WHEN I GO AND LOOK UP A PROFILE, THEY ARE NOT SIMPLE DOCUMENTS. THEY ARE DIFFICULT THINGS TO INTERPRET AND THEN TO PUT INTO ACTION. HOW DO YOU SEE QIN -- REMEMBER YOU GOT OTHER PEOPLE HERE. I'M NOT GOING TO ASK YOU. HOW DO YOU PEOPLE IN YOUR QIBA HATS THINK OF THE PROFILES AS BEING FUNCTIONAL IN THE CLINICAL ENVIRONMENT? >> WELL, PROBABLY THE MOST MATURE PROFILE IS THE PET PROFILE, IT'S A DECADE TO MATURE, MAYBE IT'S POST--MATURE, 85 PAGES OR SOMETHING. IT'S SO COMPLICATED IF YOU LOOK AT ALL 85 PINGS 85 PAGES IT'S HARD TO FIGURE OUT. ONE STEP TAKING OUT OF CLINICS TO DEVELOP A CHECKLIST OF THE KEY STEPS WHICH ARE NECESSARY AND FEASIBLE, AND I THINK THAT WAS REDUCED TO SOMETHING LIKE 3 PAGES. AND THEN BEING ABLE TO COMPLY WITH THE CHECK LIST REALLY MAKES IT EASIER. AND I THINK THAT SEVERAL SITES, CERTAINLY OUR PRACTICE, THE PLACES I PRACTICE OF LATE, BALTIMORE AND ALSO IN ST. LOUIS, WE BASICALLY ARE FOLLOWING THE QIBA PROFILE. YOU KNOW, SO IT IS -- TRYING TO MAKE IT FEASIBLE AND PRACTICAL REQUIRES SOME DISTILLATION OF THAT INFORMATION, AND SOME OF WHAT -- I THINK WE DON'T WANT TO HAVE THE PERFECT PROTOCOL, PROFILE, RATHER. WE WANT SOMETHING THAT'S GOOD AND WORKABLE THAT WE CAN ACTUALLY DEPLOY. >> ALSO I THINK IT SHOWS THE POTENTIAL OR ADDITIONAL COLLABORATION BETWEEN QIN AND QIBA BECAUSE IN THE QIN HERE WE HAVE A LOT OF PEOPLE WHO ARE VERY ADVANCED IN COMPUTERIZING AND AUTOMATING RADIOMICS, AND IF WE CAN TAKE THAT KNOWLEDGE BASE AND APPLY TO AUTOMATING QIBA PROFILES SO THAT YOU HAVE LIKE A CLOSED LOOP WHERE THIS SYSTEM FOLLOWING THIS PROFILE WITH THIS PHANTOM INPUT GENERATES THE QUANTITATIVE OUTPUT, IF THAT WAS DONE WITH MINIMAL HUMAN INTERACTION THEN IT WOULD MAYBE SIMPLY FILE THE PROFILE AND GET IT USED MORE. SO WE HAVE EXPERTISE TO MERGE. >> SO THE OTHER AREA OF SYNERGY, THIS IS ONE THAT WE HAVEN'T REALIZED BUT DEFINITELY WILL REALIZE AND HEARD ABOUT THE TCGA, QUALITY OF DATA, HETEROGENEITY OF QUALITY OF DATA, AS WE IN QIN TRY TO APPLY OUR TOOLS, I THINK WE'RE GOING TO QUICKLY FIND AS MARYELLEN SAID, WE REALLY WILL NEED SPECS OF THESE PROFILES IN ORDER TO GET ROBUST REPRODUCIBLE RESULTS. WE BOTH FACE THE SAME CHALLENGE OF INTRODUCING KIND OF AHEAD OF THEIR TIME INTO GROUPS BUT I THINK, YOU KNOW, DOING ONE INDIVIDUALLY VERSUS THE OTHER ACTUALLY COULD BE SOMEWHAT DANGEROUS, AND POTENTIALLY DOING THEM TOGETHER MAKES A LOT MORE SENSE. >> I'D SAY THAT GIVEN WHAT WE'VE HEARD ABOUT THE NATIONAL CLINICAL TRIALS NETWORK SINCE SO MANY STUDIES ARE DONE IN COMMUNITY HOSPITALS IF WE CAN EXPORT BEST PRACTICES WHICH ARE FEASIBLE WHICH ANALYTICAL TOOLS, AND DEPLOY THOSE MORE BROADLY, AND HAVE CONTINUOUS QUANTITATIVE PHENOTYPIC DATA VOLUME AND GLYCOLYSIS AS EXAMPLE OR MR CHARACTERISTICS WE CAN WE LIE ON THIS WOULD BE A RICH SOURCE OF DATA THAT WE'RE JUST BASICALLY -- WHICH EXISTS AND WE'RE JUST NOT COLLECTING. TO THINK WE CAN DO BIG DATA ONLY ON THE GENOME AND NOT REALLY -- WE DON'T THINK THAT BUT SOME PEOPLE THINK THAT MAYBE MUCH OF THE ANSWER, BUT THIS PHENOME, PHENOTYPE THAT WE CAN ASSESS WILL BE SO IMPORTANT AND EVERY YEAR WE DON'T COLLECT IT SYSTEMATICALLY I THINK WE'RE DISADVANTAGING OUR SELF. >> AND JUST TO GO BACK TO THE COMMENTS OR OBSERVATION, BOB, ABOUT THE COMPLEXITY AND RICH'S COMMENT WHEN THESE CHECK LISTS ARE DEVELOPED, THIS IS REALLY THE WAY BY WHICH THE VARIOUS ACTORS IN INIMAGE CHANGE HAVE PROFILES. TESTING DISTILLATION OF COMPONENTS, THREE ACADEMIC CENTERS TO BEGIN WITH, THAT LED TO A REFINEMENT OF THAT LIST AND FURTHER DISTILLING DOWN TO THE MOST KEY CRITICAL THINGS AND THAT WENT OUT TO 11 OTHER SITES THAT WERE NOT ACADEMIC SITES BUT INCLUDED COMMUNITY GROUPS TO MAKE SURE THIS REALLY IS SOMETHING THAT CAN TRULY BE PRACTICALLY AND FEASIBLY DONE BECAUSE WE HEAR THIS ALL THE TIME WE CAN'T IMPACT ACCRUAL, CLINICAL TRIALS, ET CETERA, SO HOW DO WE MAKE MAKE SURE THIS CANS FEASIBLY, YOU KNOW, THE LAST ITEM ON THE CZECH LIST -- CHECK LIST, I NEED TO DO THE FOLLOWING STEPS, IF I'M THE SOFTWARE VENDOR. KEY POINT, YOU CAN'T HAND SOMEBODY 86 PAGES AND SAY LET ME GO THROUGH AND FIND WHAT I WANT. >> WHAT I'M HEARING, BEING QIN CENTRIC FOR A MINUTE, OUR FOCUS IN QIN IS GETTING THE TOOLS THAT WE HAVE UNDER DEVELOPMENT AND THROUGH INITIAL CLINICAL VALIDATION INTO NCTN CLINICAL TRIALS, WE'VE BEEN PUSHING ON THAT VERY HARD FOR THE PAST YEAR. AM I HEARING THAT COLLABORATION OR INCLUSION OF THE FACT THAT WE ARE PROFILE -- THAT WE'RE WILLING TO PUT OUR TOOLS TOGETHER WITH QIBA PROFILES DOES THAT GIVE AN ADVANTAGE IN THIS TACK, DOES THAT GIVING SOUSE GIVE US SOMETHING WE DIDN'T HAVE BEFORE QIN AND QIBA GOT TOGETHER? >> WORKING TOGETHER YOU HAVE MORE MUSCLE. ARE RSNA YOU'RE TALKING TO IMAGING, THE RADIOLOGISTISTS OUT THERE. RSNA HAS A GOOD LINK WITH INDUSTRY, IN ADDITION. ALL THE FOLKS WHO GO TO THE RSNA MEETING. AND I THINK THEN WE MIGHT GET MORE INDUSTRY SUPPORT FOR THE QIN IMPLEMENTATION. >> SO I THINK IT WILL GIVE A LOT OF SYNERGY TO BOTH GROUPS, AT RELATIVELY THE SAME COST, WHICH ISN'T LOW. AS WE'VE DISCUSSED YESTERDAY. BUT I DON'T THINK THAT A LOT OF THE CLINICAL TRIAL SITES ARE ACTUALLY RECOGNIZING DIFFERENCE BETWEEN THE TOOL AND PROFILE. THEY DON'T HAVE TO. WE JUST WANT THEM TO IMPLEMENT BOTH, RIGHT? THE SYNERGY IS GOING TO BE THIS. WHY ARE YOU DOING A QIBA PROFILE? WHY ARE WE DOING IT? WHAT YOU SHOWED, IT'S ALSO THE ABILITY TO USE A TOOL WHICH WILL GIVE BETTER DATA, AND WHY ARE YOU DOING -- OR WHY DO YOU NEED THE PROFILE FOR THE TOOL, BECAUSE WITHOUT THE PROFILE YOU MAY NOT HAVE ADEQUATE DATA TO GET THOSE BETTER RESULTS SO I THINK THAT THEY REALLY BOTH DO GO HAND IN HAND AND POTENTIALLY COULD BE, YOU KNOW, SYNERGISTIC AT THE SAME COST. >> SO WHAT I THINK I'M HEARING IS THAT THE TOOLS ARE A METHOD TO GET TO THE BIOMARKERS. IN OTHER WORDS, THEY ARE THE MECHANISM TO GET TO THE BIOMARKERS, AND THE PROFILES ARE THE PROTOCOL FOR DOING THAT. IS THAT KIND OF WHAT I'M THINKING? >> YEAH, I THINK THE PROFILE SORT OF ASSURES YOU HAVE HOPEFULLY -- HOPEFULLY ASSURES YOU HAVE HIGH QUALITY IMAGING DATA IN A RELATIVELY CONSISTENT MATTER OF ACQUISITION WHICH IS THEN SUITABLE FOR QUANTIFICATION BY A QIBA TOOL. AND THEN THE TOOLS CAN ALSO BE THE PHANTOMS CAN ALSO BE VERY IMPORTANT FOR VALIDATING YOUR ACQUISITION METHOD AND THE ANALYTICAL, SO IN COLLECTION, THEY HAVE ELEMENTS OF QUALITY CONTROL AND CONSISTENCY, SO THAT THE END PRODUCT QUANTITATIVE NUMBER, YOU'RE CONFIDENT IS GOING TO BE REPRODUCIBLE ANYWAY AND HOPEFULLY REPRESENTING TRUTH IN A CONSISTENT FASHION. BEFORE LARRY GOES ON, I WANT TO MENTION ONE THING. HE KEEPS SAYING THE SAME PRICE. ONE OPTION, YOU CAN'T GET HIGH QUALITY CLINICAL IMAGES FROM PRIVATE PRACTICES OR FROM COMMUNITY PRACTICES, THE PRACTICES ARE GETTING BIGGER AND BIGGER ACROSS THE COUNTRY SO SMALL PRACTICES ARE MERGING, MANUFACTURERS ARE EMBRACING QUANTITATIVE IMAGING AND IT MIGHT BE POSSIBLE, YOU PAY A THOUSAND DOLLARS A CASE, YOU COULD PAY 900 IF IMAGING DATA IS NOT INCLUDED AND DOESN'T FOLLOW PROFILE OR QUANTITATION, THEN YOU COULD PAY 1100 IF IT DID, IT COULD BE REVENUE NEUTRAL BUT YOU COULD INCENTIVIZE INCLUDING THIS IMPORTANT QUANTITATIVE PHENOTYPIC DATA WITH BEING NET NEUTRAL THROUGH A SMALL CHANGE IN INCENTIVES. IN CLINICAL PRACTICE OF RADIOLOGY THERE ARE QUALITY PAYMENTS, IF YOU MEET A QUALITY STANDARD YOU GET A BONUS, IF NOT YOU GET A SMALL POLICY, IT MIGHT BE STRUCTURE THINGS TO ENCOURAGE IF IT'S VIEWED AS A PRIVATE. ALL DATA ARE AVAILABLE, WE'RE NOT CAPTURING ALL RIGHT NOW. IT'S A GREAT OPPORTUNITY. >> SO, YOU KNOW, I DON'T WANT TO PLAY UP TOO MUCH THE DIFFICULTY IN DOING THIS. EVERY ONE OF THESE COMMUNITY HOSPITALS HAVE LAB EQUIPMENT, THAT THEY SUBMIT BOTH FOR CLINICAL PRACTICE AS WELL AS CLINICAL TRIALS, THAT'S STANDARDIZED, OPTIMIZED, EVEN COMPLEX. LIQUID TUMOR ASSAYS WHICH THEY ARE FACING, WE WERE IN THIS VERY ROOM, WHAT, THREE MONTHS AGO TALK BECOME THAT AND SOME OF THE VARIABILITY THERE. YOU KNOW, IT'S ALL EMINENTLY DOABLE, JUST PUTTING IT TOGETHER AND FINDING, YOU KNOW, A FEW BEST USE CASES WHICH WILL PROVE ITS VALUE. >> JUST TO GO ON, TO THE LAB, I MEAN I THINK A LOT OF LABS ARE CLIA COMPLIANT. I THOUGHT WE NEED A CILIA IMAGING ASSESSMENT OR ACCREDITATION BUT WE MIGHT NEED TO MOVE TO THE SAME LEVEL AS YOU'RE SEEING BUT IT'S NOT WITHOUT PRECEDENT IN THE PATH LABS. >> RADIOLOGY IS BACKWARDS IN THAT REGARD, THERE'S A HISTORICAL REASON. I LIKE TO SAY WE WERE BORN BREECH. >> WE'RE COMING OUT OF THIS. >> IN MAMMO EVERY PRACTICE DOES THAT. THAT WAS DRIVEN BY RADIOLOGY ALONE. THIS WILL BE DRIVEN BY A COMBINATION OF PURPOSES, AND, YOU KNOW, BOTH OF THEM ARE VERY SYNERGISTIC. >> ANY THOUGHTS FROM THE GROUP? THIS THOUGHTS? >> THE CO-CLINICAL TRIAL IS ALL NETWORK, INDEPENDENT FROM QIN, AND ALSO ANOTHER THING WE DO HAVE TEAM, ONE OF TEAM, ONE WORKING ON THE PRE-CLINICAL PROFILE LIKE QIBA BUT THAT'S NOT A GENERAL INTEREST, SO THAT MIGHT BE A LITTLE BIT DIFFERENT FROM QIN BECAUSE SOME OF THE METHOD, WE'RE ASKING THEM YOU HAVE USE IDENTICAL IMAGING METHOD IN CLINICAL SETTING, COULD HAVE SOMETHING NEW, IN ESSENCE THE CLINICAL METHOD MAY NOT EXACTLY FROM THOSE QIBA PROFILES, THAT'S ONE CHALLENGE FOR THAT. >> YEAH, SO I DIDN'T MEAN TO IMPLY THE PROFILES ARE SPECIFICALLY RELEVANT TO THE CO-CLINICAL TRIALS, IT WAS MORE THE IMAGING METROLOGY BASIS, DEVELOPMENT OF THINGS LIKE DROs AND OTHER THINGS FOR TESTING APPLICATIONS BEING USED IN THE PRE-CLINICAL SPACE AND CLINICAL SPACE, I THINK UNDERSTANDING THE MEASUREMENT BIAS AND VARIANCE IS IMPORTANT REGARDLESS OF WHICH OF THOSE IN ORDER TO DO -- EMPOWER THOSE TRIALS APPROPRIATELY. I DID NOT MEAN TO IMPLY PROFILES ARE DIRECTLY MAPPABLE TO CO-CLINICAL TRIALS. MAINLY IMAGING METROLOGY BASIS AND UNDERLYING STATISTICAL NATURE AND TOOL SETS DEVELOPED TO ASSESS THOSE. THAT'S WHAT I WAS TRYING TO -- >> THE PROFILES, WHAT'S THE VARIATIONS ACROSS THE SITE, WHEN DO YOU DECIDE WITH DIFFERENT SCANNER OR DIFFERENT DATA, DIFFERENT MACHINE, DIFFERENT SOFTWARE TO PROCESS DATE, WHAT'S THE VARIATION FOR THOSE? >> I THINK IT'S VERY MODALITY DEPENDENT BUT ALSO APPLICATION DEPENDENT RIGHT NOW. SO WHEN WE LOOK AT THOSE LEVELS OF VARIANCE AND BIAS WE SEE SOMETHING DIFFERENT IF YOU'RE LOOKING AT PET SUV THAN DCMRI FOR EXAMPLE. BUT IT'S REALLY THE METHOD BY WHICH WE ESTABLISH WHAT THOSE CONFIDENCE INTERVALS ON THE MEASUREMENTS ARE, AND THEN DEMONSTRATE IF ANOTHER TECHNIQUE IS DEVELOPED HOW DO THEY COMPARE. AND IF YOU WISH TO HARMONIZE THAT DATA, WHAT INFORMATION IS NEEDED TO BE ABLE TO HARMONIZE THE DATA ACROSS A COLLECTION OF INSTRUMENTS, FROM DIFFERENT VENDORS, OVER SOFTWARE UPGRADES, THINGS LIKE THAT. SO IN THAT SETTING THINK THE UNDERLYING IMAGING METROLOGY AND PRINCIPLES, CONCEPTS OF DEVELOPMENT OF VROs AND ACQUISITION PHANTOMS TO TEST ACQUISITION SITE AND IMAGE PROCESSING SIDE IS IMPORTANT. USING THOSE TO DETERMINE IN APPLICATION-SPECIFIC MANNER WHAT IS CONFIDENCE INTERVAL IN THIS APPLICATION AND USE THAT TO HARMONIZE RESULTS IF I NEED TO DO SO ACROSS VENDORS. >> SO FOR THE SAME METHOD SAY DCMI FOR EXAMPLE DOING CROSS-SIDE WITH PROFILE PROVIDED BY QIBA THEN WHAT KIND VARIATION WOULD EXPECT, WOULD BE ACCEPTABLE FROM -- >> RIGHT, THAT'S STATED SPECIFICALLY IN THE CLAIM. THE WAY CLAIMS ARE DEVELOPED IS IT STARTS WITH METANALYSIS OF EXISTING LITERATURE, WHAT TEST, RETEST DATA HAS BEEN DONE. HOW DO WE ESTABLISH THAT. WHAT ARE THE BEST GUESSES OF WHAT, HONESTLY, GIVEN EXISTENCE OF HIGH QUALITY METADATA THAT'S AVAILABLE, HOW DO WE INFORM WHAT THE ERROR BARS THAT ARE STATED OR CONFIDENCE INTERVALS STATED IN CLAIM WILL BE, IF THERE'S HOLES IN THAT INFORMATION HOW DO WE DEVELOP THE GROUND WORK PROJECTS TO GO OUT AND MEASURE THIS DATA IN A MULTI-CENTER SITE OR GET SOME OF THIS INFORMATION TO ESTABLISH THOSE CONFIDENCE INTERVALS. THAT COMPLICATED FLOW DIAGRAM I SHOWED OF CLAIM DEVELOPMENT REALLY ADDRESSES EXACTLY THE QUESTION YOU'RE ASKING. HOW DO YOU DEVELOP THOSE CONFIDENCE INTERVALS FOR A GIVEN APPLICATION, DIFFERENT MODALITY, APPLICATION ON THAT MODALITY, HOW DO YOU ESTABLISH THAT IN A STATISTICALLY ROBUST WAY, AND THAT'S REALLY THE UNDERLYING IMAGING METROLOGY THAT CUTS ACROSS ANY APPLICATIONS. >> I SUGGEST YOU READ SOME OF THE PROFILES TO GET AN IDEA WHAT THEY ARE TALKING ABOUT. >> EXCELLENT TALK ABOUT THE QIBA. I JUST HAVE ONE QUESTION, YOU KIND OF ALLUDED ABOUT NOT HAVING TOO MUCH OF INCENTIVE FROM VENDORS TO MAKE QUANTITATIVE ASPECTS, AS WELL AS EARLIER QIBA MENTION ABOUT FDA, HAVEN'T REALLY UPGRADED ANY OF THOSE THINGS. I'M PRIMARILY TALKING FROM MR MODALITY, DISCUSSIONS WITH VENDORS DOING QUANTITATIVE ASPECTS, WE'RE NOT REALLY PUSHING INTO MAKING INTO A PRODUCT BECAUSE FDA DOES NOT APPROVE IT, JUST WANT TO GET YOUR THOUGHTS ABOUT HOW MUCH -- THERE'S A LOT OF INVOLVEMENT IN THE INDUSTRY IN THE QIBA SO JUST GET YOUR THOUGHTS. >> YES, I THINK WE'VE SEEN AN EVOLUTION THERE. I THINK IT'S IN SOME CASES MORE SLOWLY DEVELOPING THAN OTHERS. WHEN WE STARTED THE PROCESS WE WENT TO MAJOR VENDORS, HERE IS OUR CONCEPT, THOUGHTS, WHAT DO YOU THINK. WE WERE MET BY, OKAY, GO AHEAD. BUT WHAT WOULD BE THE MOTIVATION FOR US TO DEVELOP THESE TIMES OF QUANTITATIVE IMAGING PLATFORMS? THERE IS STILL ELEMENTS OF THAT OF COURSE BECAUSE I MENTIONED THERE'S NOT A CARROT OR STICK THAT'S REALLY FORCING OR REWARDING PER SE, BUT IN LIGHT OF THAT I THINK WE HAVE SEEN CONTINUED EVOLUTION, IF YOU GO TO THE RSNA SHOWS AND ON THE FLOOR YOU SEE PARTICULARLY IN PET DOMAIN A STRESSING ON QUANTITATIVE IMAGING ACCURACY. THEY MAY NOT DEFINE EXACTLY WHAT THEY MEAN. WE'VE SEEN THAT ALSO IN THE MR DOMAINS, CT. MIKE CAN SPEAK TO THIS MORE. I DON'T THINK I'VE SEEN IT AS MUCH IN THE CT DOMAINS. WE'RE SEEING THE WORD QUANTITATION COME INTO THE MARKETING OF DEVICE MANUFACTURERS, MARY ELLEN STATED WHEN RSNA PUTS A SUBSTANTIAL AMOUNT OF SUPPORT BEHIND IN THIS CASE QIBA, OUR PRIMARY FUNDER AT THIS POINT, I THINK THAT CARRIES WEIGHT WITH VENDORS AND SO I THINK WE'RE SLOWLY BEGINNING TO SEE THAT MIGRATION. WE HAVE A LOT OF VENDOR INVOLVEMENT. THE HEAD OF OUR PROCESS COMMITTEE IS TOSHIBA. I MEAN, WE HAVE ON EACH OF OUR PROGRAMS -- SORRY, NOW CANON, THANK YOU FOR THE CORRECTION. ONE REASON I STRESS THE WAY THE STEERING COMMITTEE AND BIO, MAKER COMMITTEE LEADERSHIPS ARE STRUCTURED WE SEEK SOMEONE FROM INDUSTRY ON EACH OF THOSE, BOTH DEVICE MANUFACTURERS AS WELL AS SOFTWARE MANUFACTURERS TO KEEP THAT LEVEL OF COMMUNICATION GOING. HAVING SAID THAT, ALMOST ALL THOSE PEOPLE ARE IN THE RESEARCH SIDE OF THOSE DEVICE MANUFACTURERS, NOT CERTAINLY MARKET ENGINEERING SIDE. >> IS THERE SOMETHING QIN IT DO TO HELP, I DON'T THINK THAT'S TOO MUCH INVOLVEMENT ON THE QIN ASPECT, IS THERE ANYTHING MORE QIN CAN HELP IN THAT ASPECT? >> I WAS LOOKING FOR THE OTHER WAY AROUND MYSELF. >> BOTH WAYS. IF THESE TOOLS WHICH THEY ARE USEFUL, AS WE PROVED THOSE TOOLS ARE USEFUL THAT'S GOING TO NATURALLY HELP. YOU KNOW, DEVELOP AND EXPAND THE USE OF PROFILES. >> I THINK FROM AN ECONOMIC STANDPOINT THE ISSUE OF REIMBURSEMENT IS ONE THAT COMES UP, AND THERE HAVE BEEN SOME EFFORTS TO PUSH CERTAIN QUANTITATIVE METRICS APPEARING IN RADIOLOGY REPORTS AS AN INDEX OF QUALITY THAT, YOU KNOW, THERE ARE SOME PROCESSES WHERE YOU COULD HAVE THIS SLIGHT BONUS, AND REDUCTION. IF PHYSICIANS ARE PAID MORE TO PROVIDE QUANTITATIVE DATA, YOU KNOW, THAT'S IT'S MORE INDICATIVE OF A QUALITY REPORT THAT THAT COULD BE A BIG INCENTIVE MOVING FORWARD. IT WOULD ALSO BE INCENTIVE FOR COMPANIES TO PROVIDE RELATIVELY QUICK AUTOMATED METHODS TO EXTRACT QUANTITATIVE DATA. THAT'S IN DISCUSSION. >> MIKE? >> I REALLY APPRECIATE THE DISCUSSION. AND WE WON'T HAVE TIME TO ANSWER MY QUESTION, BUT I WOULD LIKE TO THROW IT BACK TO THE EXECUTIVE COMMITTEE BECAUSE I THINK ED, QIBA IS NOW AN ASSOCIATE MEMBER OF QIN. >> YES. >> I THINK YOU'RE ALSO ON THE -- >> CORRECT. >> AGAIN, I'D LIKE TO HAVE THE EXECUTIVE COMMITTEE AT LEAST ON QIN ADDRESS AND IDENTIFY SOME REALLY KEY AREAS BECAUSE I HEAR THE DISCUSSION, AND IT'S REALLY VERY VALUABLE, AND I'M REFLECTING BACK ON OUR DISCUSSION WITH NCTN LAST FALL, THINKING QIN IS FOCUSED ON CLINICAL TRIALS, QIBA IS A LITTLE BROADER THAN THAT, IF I CAN ALLOW THAT, YOU DON'T NEED TO BOTHER TO ANSWER. IT'S EITHER YES OR NO. WE NEED SOME REALLY STRATEGIC -- THERE'S A THOUSAND THINGS WE COULD DO JOINTLY, I THINK I'D LIKE TO HEAR SOME DISCUSSION ABOUT WHAT ARE KEY THINGS KEEPING US OUT OF CLINICAL TRIALS AND HOW CAN WE WORK TOGETHER TO IDENTIFY A COUPLE KEY AREAS WHERE WE COULD MAKE A SUBSTANTIAL CONTRIBUTION TO THAT, SORT OF A GAP ANALYSIS, AND SAY HERE ARE SOME THINGS WE COULD ATTACK JOINTLY TOGETHER. I DON'T KNOW THAT POST BIGGEST LIMITATION. I'M A PHYSICIST. I'D LIKE TO HAVE ONGOING DISCUSSION. >> THE GOOD WORD IS ONGOING. THAT'S NOT A SIMPLE QUESTION TO ANSWER IN THE MIDDLE. GEORGE, IN THE MIDDLE. ONE ASPECT THAT IS IMPLICIT TO WHAT YOU'VE BEEN DISCUSSING IS SITE QUALIFICATION. AND ASSURANCE THAT THE SITE IS PRODUCING ADEQUATE DATA AND I WONDER IF MAJOR RESPONSIBILITY FALLS ON QIN OR QIBA SIDE, BUT IT'S ABSOLUTELY ESSENTIAL BECAUSE IF YOU GET GARBAGE ACQUIRED, YOU'RE GOING TO GET GARBAGE OUT THE OTHER END. YOU'RE ASSUMING THEY ARE GOING TO DO IT THE RIGHT WAY. THAT'S CERTAINLY NOT THE CASE. >> SO WE'RE NOT ASSUMING THAT. I'LL ANSWER THE GARBAGE QUESTION AFTER. JOHN, DO YOU WANT TO -- >> YEAH, ACTUALLY, AS DISCUSSIONS ARE GOING ON ONE OF THE THINGS THAT OCCURRED TO ME, I'M NOT AWARE OF THIS AT ALL, BUT, YOU KNOW, WITH THE VENDORS, PARTICULARLY, TRYING TO SOLVE SOME OF THESE PROBLEMS, HAS THERE BEEN ANY OVERLAP OR DISCUSSION WITH THE IHE EFFORTS, BECAUSE A LOT OF IHE PROFILES IN TERMS OF HOW THEY ARE TRYING TO ADDRESS THIS ARE DIRECTLY RELEVANT AND OVERLAPPING BUT SOMEWHAT DIFFERENT. I'D BE CURIOUS AS TO -- IF THERE'S BEEN ANY AND HOW MUCH. >> DEFINITELY THERE'S INTERACTIONS WITH IHE, NOT NECESSARILY IN SAYING QIBA PROFILES NEED TO BE PART OF IHE BUT THE HEAD OF PROCESS COMMITTEE IS INTEGRATED IN IHE, INVOLVED IN IHE AND DICOM STANDARDS, USING VERY SIMILAR PRINCIPLES, HOW DO WE DEVELOP THAT THEY ARE CONSISTENT BETWEEN IHE PROCESS AND THE WAY THAT'S HANDLED, AND QIBA AND THE WAY THE PROCESSES ARE DONE WITHIN QIBA. TO THAT DEGREE, BUT THERE'S NO MORE DIRECT INTERACTION OF SAYING, YOU KNOW, PROFILES WOULD BE ADOPTED IN IHE SETTING. DOESN'T GET TO THAT LEVEL. AND IF I COULD FOLLOW UP ON MIKE'S COMMENT BRIEFLY, LARRY HAD A COMMENT TOO, SO THE PROFILES, THE WAY THEY ARE STRUCTURED, THERE'S VERY SPECIFIC GUIDANCE FOR THE SITES THEMSELVES, WHAT ARE REQUIREMENTS FOR THE SITES IN TERMS OF THE WAY THE DATA IS ACQUIRED, IF THERE'S A PHANTOM THAT'S FELT TO BE IMPORTANT FOR THAT APPLICATION-SPECIFIC PROFILE BIOMARKER THAT CAN BE USED, WE DID THAT AT ONE OF THE FIRST TESTS OF ONE OF THE PROFILES IN A CLINICAL TRIAL SETTING, WHICH WAS WITH ECOG-ACRIN, 6701 WITH PROFILES WERE USED, TEST-RERETEST SETTING USING PHANTOMS AS PART OF REQUALIFY SIX EVERY SIX MONTHS. IT'S IMPORTANT IN OUR CASE THE PROFILES ARE SET UP SO THERE'S A VERY SPECIFIC SET OF REQUIREMENTS TO BE CONFORMANT WITH PROFILE FOR THE SITES THAT INCLUDES QUALIFICATION SIDE, AS WELL AS IMAGE ANALYSIS, THAT'S WHERE THE DROs COME IN. SO I TOTALLY AGREE WITH YOUR POINT. >> HERE IS THE CHALLENGE, I'LL RESTATE IN A DIFFERENT WAY. IF A CLINICAL TRIALIST WERE HERE AND LISTENING TO THIS CONVERSATION, THEY WOULD SAY SOMETHING LIKE THIS. THEY WOULD SAY, LOOK, THE GARBAGE IS RANDOMIZED PHASE 3 TRIAL WITH 750 PATIENTS IN BOTH ARMS, YOU KNOW, THERE'S NO REASON TO THINK THERE'S GOING TO BE DIFFERENT AMOUNTS OF GARBAGE BETWEEN THE ARMS. SO THE BIAS IS EVENLY DISTRIBUTED, I'VE POWERED MY STUDY TO TAKE INTO ACCOUNT ALL THAT. RIGHT. NOW WE SAY MAYBE IF WE COULD DO IT WITH FEWER PATIENTS AND THAT'S ACTUALLY A LOT OF SMART PEOPLE HAVE REALLY TRIED TO DO THAT AND THAT'S VERY, VERY DIFFICULT TO TELL AS WELL. FOR A NUMBER OF DIFFERENT REASONS. BUT-- SO THAT'S WHERE WE WERE. THE GOOD NEWS IS WE MAY BE AHEAD OF OURSELVES. AND HAVE BEEN AHEAD OF OURSELVES BUT NOW THE FIELD IS CATCHING UP DAYS OF 750-PATIENT RANDOMIZED PHASE 3 TRIALS ARE GOING AWAY. IT'S GOING TO BE -- WE'RE DOING CONTINUOUS PHASE 1, 2, 3 TRIALS WITH NO CONTROL SO WE HAVE TO START UNDERSTANDING THE BIAS. WE HAVE EXAMPLES IN LITERATURE RIGHT NOW OF IDENTICAL PHASE 2 TRIALS WITH RESPONSE RATE, ONE POSITIVE, ONE NEGATIVE. IDENTICAL TRIALS. ONE RUN BY NCI, WHICH TURNED OUT TO BE THE POSITIVE ONE AND THE INDUSTRY ONE WAS ACTUALLY NEGATIVE. SAME POPULATION. WHY-- THERE'S A NUMBER OF FACTORS. I'M SURE IMAGING CONTRIBUTES TO IT. I'M POSITIVE IMAGING CONTRIBUTES TO IT. SO NOW, YOU KNOW, THAT WE'RE CHANGING THIS PARADIGM, NOW IT'S GOING TO BE IMPORTANT BUT WE HAVE TO THINK BACK, YOU KNOW, TO WHAT THE HISTORICAL CLINICAL TRIALIST IS GOING TO SAY. WELL, IT'S IN BOTH ARMS, AND, YOU KNOW, I SIMPLY HAVE HEARD THEM SAY THIS, I SIT ON THE RESIST COMMITTEE. IT'S BOTH ARMS. THAT'S RAPIDLY CHANGING. THAT'S PART OF THE VALUE PROPOSITION THAT WE'RE GOING TO NOW BRING AND THAT'S PART OF THE ISSUE ABOUT, YOU KNOW, THERE ARE MANY PHASE 2 TRIALS OR SINGLE ARM TRIALS THAT DEPENDING UPON HOW THEY ARE SET UP, DEPENDING WHAT SITES THEY RESIDE IN, THERE'S GOING TO BE A LOT OF DIFFERENCES IN IMAGING WHICH WILL IMPACT THE RESULTS. >> LET'S GO TO MIKE AND FRED AS THE LAST CONTRIBUTOR. >> SO I HOPE I'M NOT OPENING A CAN OF WORMS WITH THIS QUESTION OR MAYBE BEATING A DEAD HORSE FOR ME. QIBA APPROACH, RETROSPECTIVE, WE GO BACK AND DESIGN OUR PROFILES, IN PARTICULAR CLAIM STATEMENTS BY LOOKING AT WHAT OTHER PEOPLE HAVE DONE SO WE'RE RELIANT ON THE BODY OF LITERATURE THAT'S OUT THERE. MAYBE I'M JUST SENSITIVE BECAUSE THE PROFILE IS LIMITED INP A INDICATION TO ORGANS WHERE THE LITERATURE WAS ADEQUATE. TO COMPLEMENT AND ADDRESS THE HOLES, THE GAPS IN THE PROFILES APPLICATIONS, IF I LOOK AT DWI WE DON'T SAY ANYTHING ABOUT BREAST BECAUSE LITERATURE WASN'T SUFFICIENT. IS THERE ROOM FOR QIN TO HELP QIBA OUT AND VICE VERSA, LOOKING AT DTI WORK THIS YEAR, PRESENTED BY TOM AND AVITA? >> THANKS FOR THE QUESTION. WE'RE WRITING A PAPER WITH LARRY, ED, MIKE, YOURSELF AND THE OTHER CO-AUTHORS, AND THAT WILL FOCUS ON THE POSITION NANCY WAS TALKING ABOUT YESTERDAY AND TWO TECHNIQUES, DIFFUSION RATE MRI, IT IS TAKING IN THE TABLES THAT ARE IN THE PROFILES AS WELL AS ALSO PUTTING IN SOME ORGANS NOT IN THE PRO FIGHTS AND I WAS JUST TALKING TO ED YESTERDAY, LIKE A LOT OF WORK HAS GONE INTO PAPERS READ BY VOLUNTEERS AT QIBA AND MR COMMITTEES, SO WE ARE -- AND WE DO EMPHASIZE IN THAT PAPER WHICH WE'RE RECOMMITTING SHORTLY TO GRI, A DEARTH OF TEST, RETEST, THERE HAS TO BE MORE TEST, RETEST. >> FRED, CLOSE US OFF. >> A QUICK ONE. LOOKING AT -- IT GOES BACK TO SOMETHING JOHN ASKED. LOOKING AT THE STRUCTURE OF QIBA, LOOKING AT THE WAY YOU CARRY OUT YOUR DUTIES, HAVE YOU THOUGHT OF BECOMING A STANDARD-SETTING BODY? NISO CERTIFIED, HAVING GREATER IMPACT IN THE VENDOR COMMUNITY THERE BY? >> THERE WAS A COUPLE WORKSHOPS AT NIST BOULDER. THE FIRST, MIKE, YOU REMEMBER BETTER THAN I, 2014, 2015? NOT SPECIFICALLY QIBA, IT WAS THE NEED FOR STANDARDS AND DEVELOPMENT OF THAT AND WHAT'S NEEDED IN TERMS OF STANDARDS DEVELOPMENT. SO THERE WAS A LOT OF DISCUSSION THERE AS TO, YOU KNOW, IS THERE A NEED FOR ANOTHER STANDARDS GROUP, AND WHAT WOULD IT TAKE TO DO THAT. I DON'T THINK THAT'S THE APPROACH QIBA HAS REALLY TRIED TO TAKE. IT WANTS TO INFORM THIS AND DEVELOP THESE TECHNICAL PERFORMANCE STANDARDS, WE DO USE THAT WORD BUT CALL THEM PROFILES TO AVOID THE STANDARDS WORD. AS OF RIGHT NOW, I WOULD SAY THERE'S -- THAT'S NOT THE HIGH PRIORITY FOR QIBA TO TRY TO DO THAT, IT'S NOT IN OUR PATHWAY AT THE MOMENT FOR TRYING TO DEVELOP YET ANOTHER STANDARDS. WE WOULD RATHER PARTNER WITH STANDARDS ORGANIZATIONS THAT EXIST SUCH AS NIST AND MITA IN PARTICULAR TO BE ABLE TO HANDLE ANY OF THAT TYPE RATHER THAN TRY TO RECREATE A NEW STANDARDS BODY. >> I WILL SAY WE'RE ALSO -- WE HAVE ACTIVE DISCUSSION WITH ACR TOO, AND OTHER ENTITIES FOR LOOKING AT HOW MIGHT SOME OF THESE ASPECTS BE INTEGRATED INTO ACCREDITATION PROGRAMS, NOT NECESSARILY EXISTING ONES, BUT POTENTIALLY RESEARCH TRACK OF THOSE THROUGH THE RESEARCH SIDE OF ACR, SO ALL THOSE ARE IN DISCUSSION BUT THEY ARE VERY MUCH AT THIS POINT STILL IN DISCUSSION. >> I'M GOING TO CUT THIS OFF. WE'LL TAKE OUR BREAK NOW, WE'LL MEET BACK HERE AT 11:00, AN HOUR FOR THIS, SO THE BIDS GROUP IS GOING TO BE MEETING ON SPECIAL ATTACK AND WE ALSO HAVE THE HUB OPEN HOUSE UPSTAIRS IF YOU WANT TO GO THERE. SO WE'LL MEET BACK AT 11:00. >> THANKS FOR GIVING ME THE CHANCE TO DESCRIBE OUR VISION. WE WERE GIVEN ASSOCIATE MEMBERSHIP LESS THAN FOUR WEEKS AGO, WE'RE THE NEWEST ONES. WE'RE BUILDING A UPON THE EXISTING PROSTATE CANCER TRANSATLANTIC CONSORTIUM, DEVELOPED BY FOLAKEMI ODEDINA AT THE UNIVERSITY OF FLORIDA, THIS IS OUR COLLABORATION DESIGN TO FURTHER GLOBAL COLLABORATIONS IN PROSTATE CANCER. PRIMARILY WE HAVE COLLABORATIONS IN NIGERIA AND CAMEROON. NIGERIA IS LABELED THERE, THERE'S A GLOBAL HEALTH PROBLEM ESPECIALLY IN THOSE SITES. BUT A LIMITATION OF THE PROSTATE CANCER CONSORTIUM DESIGNED AS A BIOREPOSITORY COLLECTION OF BLOOD SAMPLE, THERE'S NO MECHANISM OR INFRASTRUCTURE FOR IMAGE DATA COLLECTION. WEST AFRICAN CANCER DATABASE ALLIANCE, WACANDA. IT TURNS OUT THEY LIKE THAT VERY MUCH. WE'VE IDENTIFIED EIGHT SIDES FROM THE CONSORTIUM WITH MORE INTEREST IN INFRASTRUCTURE AND IMAGING SPREAD AROUND NIGERIA, ONE IN CAMEROON. SOME OF THESE SITES ARE IN THE MIDDLEEL OF NOWHERE, INFRASTRUCTURE IS LIMITED. SOME SITES HAVE AN ESTABLISHED IMAGING SYSTEM AND DATABASE USED AS A LOCAL HUB FOR SMALLER ENTITIES. WE HAVE THE UNIVERSITY OF FLORIDA, I'M TAKING THE LEAD AS FAR AS THE PROJECT AND GRANTS THAT WE HAVE APPLIED FOR TO TIE THIS TOGETHER, WE'RE GOING TO LEVERAGE THE RESOURCES OF IROC OR QARC TO HELP US DEVELOP INFRASTRUCTURE AND DEVELOP SECURE NETWORKING CONNECTIONS TO TIE THEM TOGETHER AND TIE THEM WITH US AND IROC AND QIN. SO WE'RE TRYING TO SHOEHORN IN THIS IDEA INTO THE PAR-28-248. THIS FUNDING ANNOUNCEMENT IS WORDED SUCH A WAY TO ENHANCE DEVELOPMENT OF IMAGING SOFTWARE TOOLS AND APPLICATIONS. THE FIRST PART SUPPOSED TO BE DEVELOPMENT AND VALIDATION, SECOND PART CLINICAL APPLICATION. OUR PRIMARY PURPOSE, DEVELOP INFRASTRUCTURE THAT'S REALLY NOT WHAT THIS GRANT WAS SUPPOSED TO BE ABOUT. SO WE'VE MIXED IN A COUPLE DIFFERENT THINGS TO TRY TO MAKE IT ADDRESS THE NEEDS OF THE GRANTEE AGENCY. SO THE TITLE OF IS IDENTIFYING PATTERNS OF EARLY SPREAD OF BREAST AND PROSTATE CANCER. THIS WILL HELP US TO CONDUCT A CLINICAL STUDY, PART OF THE LATER AIMS, BUT BASICALLY BY DEVELOPING THE INFRASTRUCTURE WE'RE GOING TO AID THE AFRICAN COMMUNITIES. THERE'S A LOTS OF SITES, 195 MILLION IN NIGERIA, 31 MEDICAL PHYSICISTS FOR THE ENTIRE COUNTRY, SO THE POSSIBILITY OF DOING TELEMEDICINE AND TELERADIOGRAPHY IN THESE OTHERS IS SOMETHING THEY ARE VERY MUCH INTERESTED IN DOING. IF QIN AND QARC CAN DO THAT, WE'LL CONDUCT A PROSPECTIVE STUDY, HIGH RISK BREAST AND PROSTATE CANCERS, FULL BODY SCANS, SEND SCANS TO MY LAB, UNIVERSITY OF FLORIDA, WE'LL DO OUR WONDERFUL TUMOR DETECTION AND SIZING APPROACHES TO DO EARLY DIAGNOSIS OF METASTATIC DISEASE IN HIGH RISK BREAST AND PROSTATE CANCER PATIENTS. THAT'S VALIDATION STUFF. THEN WE'LL GO AND EXPRESS MODELS FROM YESTERDAY, COME UP WITH A MODEL THAT DESCRIBES PHENOMENON PATTERNS OF EARLY METASTATIC SPREAD SO EVENTUALLY CAN DEVELOP A PATIENT-SPECIFIC PROTOCOL THAT INVOLVES IMAGING AND MAYBE BLOOD BIOPSIES TO INDICATE HOW WE SHOULD GO ABOUT DOING PROSPECTIVE SURVEILLANCE FOR THESE HIGH RISK PATIENTS. THE IDEA IS IF YOU CAN DETECT METASTATIC DISEASE EARLIER ON, YOU HAVE A BETTER CHANCE OF HAVING DEFINITIVE THERAPY FOR THE PATIENTS. THAT WAS IT. ANY QUICK QUESTIONS ON THAT? I SHOEHORNED IN SO I'M APPRECIATIVE. THANK YOU VERY MUCH. [APPLAUSE] >> THANK YOU. >> LESS THAN FIVE? >> YES, PERFECT. THANK YOU FOR PROVIDING AN OVERVIEW. SO THE NEXT SPEAKER IS DR. BRENNER, ALSO ONE OF OUR NEW ASSOCIATE MEMBERS, AND HE WILL BE TALKING ABOUT HIS PROJECT. >> OKAY. THANK YOU VERY MUCH FOR GIVING ME THE OPPORTUNITY HERE FOR THAT. DOES IT WORK NOW? YES, OKAY. SORRY. OKAY. THANKS FOR GIVING ME THE OPPORTUNITY HERE TO TALK ABOUT OUR IDEAS AND TO PRESENT OUR WORKING GROUP. I'M WINFRIED BRENNER, HEAD OF THE DEPARTMENT OF NUCLEAR MEDICINE AT CHARITE IN GERMAN, DIRECTOR OF BERLIN EXPERIMENT RADIOIMAGING CENTER, WHERE WE DO THE SAME IN MICE AND RATS, WHAT WE'RE TRYING DO IN CLINICAL SETUP. YESTERDAY I SAW THERE'S ALSO THE CO-CLINICAL IMAGING APPROACH HERE. I'M ALSO A MEMBER OF THE FACULTY FOR TWO GRADUATE SCHOOLS FOCUSING ON IMAGE, QUANTITATIVE IMAGING AND ONCOLOGY AND BERLIN SCHOOL, MOSTLY PRE-CLINICAL AND BASIC RESEARCH, AND ALSO BIOPHYSICAL QUANTITATIVE IMAGING TO WHAT CLINICAL DIAGNOSIS GRADUATE SCHOOL, WHERE IT'S ALL ABOUT QUANTITATIVE IMAGING IN DIFFERENT SETTINGS, NOT ONLY NUCLEAR AND RADIOLOGY BUT OPTICAL IMAGING, PHOTOAACOUSTICS AND ULTRASOUND. CLINICAL AND PRE-CLINICAL SIDE WE HAVE SPEC SPECT CT, PET CT AND PET/MRI. MY INTEREST IN QUANTITATIVE IMAGING WAS NOT FOCUSING ONLY ON TRACER UPTAKE BUT ESPECIALLY WHAT DOES IT MEAN BIOLOGICALLY. SO IT'S NOT ABOUT DATA, BUT I ALWAYS WAS INTERESTED, WHAT DOES IT MEAN IN DAILY LIFE FOR DEALING WITH THE PATIENTS. AND ONE OF THE PAPERS WHICH I WAS REALLY INTRIGUED ON THE ONE SIDE, ON THE OTHER SIDE I WASN'T QUITE SURE WHETHER I LIKED IT, THIS PAPER BY A GROUP FROM FRENCH PEOPLE, PUBLISHED ALREADY IN 2011, NOBODY REALLY TALKED ABOUT RADIOMICS OR BIG DATA AS WE DO NOWADAYS. AND THEY CAME UP WITH MAX AND PEAK, DERIVED 38 FEATURES FROM THESE PET IMAGES. AND OF COURSE THE FEATURES WERE EXACTLY MATHEMATICALLY DEFINED, AND THEY LOOKED FOR CORRELATION WITH OUTCOME AND ALL OF THIS, AND OF COURSE IT'S WHAT I THOUGHT, IT'S EASY TO DO THIS, ON AN EXTRACTION BASIS, BUT ON THE OTHER SIDE WHAT DO THESE 38 FEATURES REALLY MEAN, HOW CAN I USE IT IN MY DAILY LIFE, AND THE OTHER THING, THAT IS EVEN MORE IMPORTANT NOWADAYS, IF WE LOOK INTO RADIOMICS, WE OFTEN LOOK FOR CORRELATION, BUT WHAT DOES THAT MEAN IS DEFINITELY NOT ABOUT CAUSALITY. THAT'S WHERE I THOUGHT I'M NOT QUITE SURE THAT I REALLY LIKE THIS APPROACH, I LIKED IT A LOT WE'RE GOING INTO TEXTURAL ANALYSIS. CAME TO WORK WITH TEXTURAL ANALYSIS IN SEATTLE, I LEARNED FROM EERIE AND O'SULLIVAN, AND FINBARR IS GIVING THE NEXT TALK. WE MUST ALSO LOOK AT SPATIAL DISTRIBUTION, AND HETEROGENEITY NOW IS COMMONPLACE IN ALL OF ONCOLOGY, CHARACTERISTIC FOR ALL THE PHYSIOLOGICAL BUT ALSO PATHOLOGICAL PROCESSES, AND THEY ARE DIFFERENCES BETWEEN INDIVIDUALS, YOU NEVER WILL FIND THE SAME CANCER IN ANY TWO PATIENTS. IT'S ALSO BETWEEN THE LESIONS WITHIN ONE INDIVIDUAL, WE ALL KNOW THAT LESIONS IN BONE, FOR EXAMPLE, CAN BEHAVE COMPLETELY DIFFERENT THAN LESIONS IN THE LIVER OR LUNGS. AND EVEN WITHIN THE LESION THERE CAN BE DIFFERENCES. THESE HETEROGENEITIES USUALLY TRANSLATE INTO TREATMENT FAILURE, THE MORE WE GO INTO SPECIFIC DRUGS WE USE NOWADAYS, AND BECAUSE THEY ALWAYS WORK IN A SPECIFIC PATHWAY OR WHATEVER, AND IF THEY ARE DIFFERENT IT'S WITHIN THE LESION OR BETWEEN LESIONS, AND OUTCOME WILL BE NOT VERY GOOD BECAUSE THERE WILL BE TREATMENT FAILURE. IT'S VERY EASY TO THINK THAT HETEROGENEITY WILL TRANSLATE INTO POOR SURVIVAL, AND THAT IS WHAT THE ORIGINAL IDEA WAS BACK THEN IN 2000, THAT PET DATA COMPRISE QUANTITATIVE INFORMATION ON TRACE UPTAKE, HYPOTHESIS, I THINK THAT'S WHERE WE HAVE TO LOOK FOR MORE FOR HYPOTHESIS-DRIVEN APPROACHES THAN JUST EXTRACTING THOUSANDS OF DATA OUT OF AN IMAGE. THEY CAME UP WITH IDEA HETEROGENEITY CONSTITUTES A RISK FACTOR. FIRST PUBLICATION IN 2002, ANOTHER IN 2003, I DON'T WANT TO GO INTO THE MODEL, THEY JUST CAME UP WITH IDEA HOW TO MEASURE HETEROGENEITY, AND THE LOWER THE HETEROGENEITY INDEX WAS THAT MEANS THE MORE HOMOGENOUS A TUMOR WAS, COMPARED TO THIS TUMOR, WHICH HAS A HIGHER GRADE OF HETEROGENEITY WITH NECROTIC AREA IN BETWEEN. BETTER CORRELATION WITH OUTCOME. WHAT WAS INTERESTING FOR ME, THRILLING AT THAT TIME, THAT THIS PARAMETER, SIMPLE FTD-BASED PARAMETER WAS DOING GOOD PROPOSING OUTCOME RISK. YOU CAN SEE IN THIS CASE SUV DID NOT WORK SO WELL, IN MANY CASES IT DOES. IT'S INTERESTING TO HAVE A NEW PARAMETER WHICH TALKS ABOUT HETEROGENEITY, AND THAT IT WORKS FOR PROGNOSIS OF PATIENTS, THAT'S NOW MORE THAN 15 YEARS AGO. AND IN THE MEANTIME TRIED TO SET UP A VERY EASY PARAMETER TO PLOT IN OUR OWN DEPARTMENT BACK IN BERLIN WITH MY PHYSICIST AND CAME UP WITH PARAMETER WHICH WE TERMED ASPHERICITY, MEASURE FOR DEVIATION OF TUMOR SHAPE FROM SYMMETRY, LOWEST VOLUME WHEN YOU HAVE A PERFECT SPHERE, EASY TO DEFINE BY MEASURING THE SURFACE AND METABOLIC TUMOR VOLUME. YOU CAN SEE IF YOU USE IT IN PATIENTS WITH HEAD AND NECK CANCER WHERE WE DID IT FIRST, ROUND SHAPED, VOLUME 43, ASP 7.5. BELOW A TUMOR IN SIZE SIMILAR, 52 mLs, BUT QUITE IRREGULAR, NECROTIC CENTER AND ASP VALUE MUCH HIGHER. OF COURSE AS YOU CAN GUESS THIS ASPHERICITY WORKS QUITE NOW, PREDICTED FOR PROGRESSION FREE SURVIVAL FOR OVERALL SURVIVAL AND CORRELATED WITH METABOLIC TUMOR VOLUME BECAUSE WE LOOK AT VOLUME IN THIS, AND ALSO IN THIS CASE SUV MAX WAS PREDICTED. WHAT TO DO WITH THE PARAMETER NOW? IN THE MEANTIME, I DID IT LAST WEEK FOR THE LAST TIME, 11 PAPERS ON ASPHERICITY IF YOU GO INTO PubMed. THREE PAPERS FROM OTHER GROUPS WHICH HAS NOTHING DO WITH US, THEY DIDN'T EVEN CONTACT US BEFORE, SO FROM FRANCE, KOREA, TAIWAN USING THIS PARAMETER. AND METHOD SO FAR WHICH HAS BEEN USED FOR ASPHERICITY IN MOST CASES PET IMAGING AS WE STARTED OUT, BUT WE ALSO STARTED TO DO IT IN SPECT IMAGING, IN NEUROBLASTOMA, AND ENDOCRINE. NOW HEAD AND NECK, BREAST, PROSTATE, NEUROBLASTOMA. IT WORKED IN PSMA, OCREOTIDE AND MIBG, POTENTIALLY A TEXTRO IMAGING PARAMETER INDEPENDENT OF TRACER, TUMOR ENTITY, SPECT OR PET. WHAT WE DO NOW IS THE IMPORTANT STEP. I LEARNED YESTERDAY YOU HAVE 62 PARAMETER TOOLS IN THE QIN BUT YOU USE ONLY VERY FEW, I THINK THAT'S EXACTLY THE PROBLEM WE HAVE BECAUSE THERE ARE THOUSANDS MORE HETEROGENEITY PARAMETERS OUT THERE. BUT HARDLY ANYONE USED DAILY ROUTINE PRACTICE, WHERE I WOULD LIKE TO CONCENTRATE HOW TO BRING THIS PARAMETER FORWARD. DOESN'T HAVE TO BE ASPHERICITY, IT COULD BE ANY ONE, BUT WE HAVE TO BRING IT INTO THE TOOLS WE CAN USE ON A DAILY BASIS. THE DEFINITION OF ASP IS DONE, OF COURSE. WHAT I WANT TO DO IN THE NEXT STEP IS STEP FOR INTRODUCIBILITY, THIS I WOULD LIKE TO GET IN CONTACT WITH OTHER GROUPS USING OUR PARAMETERS, SEND THEM OUR DATA AND SEE WHAT THEY COME UP WITH, VALUES, I WOULD LIKE TO ADD THEIR DATA TO SEE WHETHER WE ALL TALK ABOUT THE SAME ASPHERICITY DOES THE TERM IS THE SAME, I DON'T KNOW WHETHER WE CAN REPRODUCE THE RESULTS. IF THAT WORKS I WOULD LIKE TO STANDARDIZE IMPLEMENTATION USING ROVER SOFTWARE AND I WOULD BE HAPPY TO SHARE WITH QIN. AIMING FOR VALIDATION AND MULTI-CENTER STUDIES, THAT'S VERY IMPORTANT AND WOULD LIKE TO BRING IT INTO THE GERMAN CANCER CONSORTIUM. IT'S EIGHT NATIONALLY FUNDED CENTERS FOR ONCOLOGY RESEARCH, HIGHLY INTERESTED TO BRING IT TO WHAT WE ALSO DO ON THE OTHER SIDE, NOT ONLY FOCUSING ON THIS PARAMETER, IT'S ALSO AN INTEGRATION INTO MACHINE LEARNING RISK MODELS, THAT IS AN APPROACH WITH THE UNIVERSITY OF VIENNA, WHERE WE WANT TO INCLUDE IN MODEL APPROACHES WHERE THEY USE IMAGING PARAMETERS, PATIENT CHARACTERISTICS, AS WE ALWAYS DID HISTOPATHOLOGY BUT ALWAYS MOLECULAR AND GENETIC TUMOR-SPECIFIC MARKERS TO SEE WHETHER THERE ARE CORRELATIONS. I TOLD YOU CORRELATION IS CAUSALITY, MIGHT BE INTERESTING TO SEE WHETHER IT'S HELPFUL FOR DAILY PRACTICE, WHAT I'D LOVE TO DO IN THE FUTURE. I THINK I CAN STOP HERE OR I CAN GO ON, THERE'S ANOTHER PARAMETER I'M INTERESTED IN. IT'S UP TO YOU. THE SECOND THING WHICH I THINK IS QUITE INTERESTING, ALSO A BASIC PARAMETER, THAT WAS BROUGHT UP BY DAVE MENIKOFF, BLOOD FLOW AND GLUCOSE ARE TIGHTLY COUPLED IN MOST ORGANS, ESPECIALLY THE BRAIN. IN TUMORS THERE'S A MISMATCH BETWEEN FLOW AND METABOLISM, WE SEE POOR FLOW AND HIGH GLUCOSE TUMOR UPTAKE. IT'S INDICATIVE FOR POOR TREATMENT RESPONSE AND POOR SURVIVAL. WE ALSO WANTED TO TEST THIS BECAUSE I THOUGHT IT'S VERY INTERESTING TO HAVE TWO DRIVING PARAMETERS FOR TUMOR LESIONS, AND WE DID WHAT DAVE DID, ANOTHER TUMOR SETTING IN CERVICAL CANCER, TUMOR GLUCOSE METABOLISM FDG-PET AND BLOOD FLOW, AND DID IT IN A FEW PATIENTS, WE CALCULATED MALIGNANT MISMATCH, TUMOR SEGMENTATION AND LOOKED WHAT IS MEAN SUV AND ALL PIXELS WERE TURNED ABOVE OR BELOW MEAN, THAT WE DID FOR FDG, SUV, K1 BY MODELING OF WATER FLOW. WITH THIS PARAMETER WE TOOK IT INTO CERVICAL CANCER, CALCULATED ABSOLUTE TUMOR VOLUME, VOXELS, VOLUME VOXEL, TIMES THE VOXEL, INDICATED FOR POOROUT COME IN THESE WOMEN. I THOUGHT I WILL DELAY THIS BECAUSE WATER PET IS NOT WHAT YOU REALLY CAN DO IN DAILY LIFE, BUT THE RAIDORY THERAPY GUYS COME UP WITH THIS AGAIN AND WANT TO HAVE IT. SO AT THE MOMENT WITH GRADUATE SCHOOL, WE'RE TRYING TO SUPPLEMENT PERFUSION MEASUREMENT WITH WATER BY CT OR MRI, IT'S MORE THE RELATIONSHIP BETWEEN FLOW AND CHANGES DURING TREATMENT. THAT'S WHAT WE WANT TO DO TO SEE WHETHER WE CAN SUPPLEMENT WITH CT OR ONE OF THE DIFFERENT MRI TECHNIQUES, AND THE OTHER THING IS I ALSO WANT TO GO INTO DIFFERENT TRACERS TO SEE WHETHER IT ONLY WORKS WITH FDT BECAUSE SURE THERE'S DIRECT COUPLING BETWEEN PERFUSION AND METABOLISM, GLUCOSE DISTRIBUTION, MIGHT BE DIFFERENT IN PSMA WHERE WE'RE LOOKING AT LIGANDS SAME WITH DOTOTATE, DISTRIBUTION TO TUMOR IS DEPENDING ON PERFUSION SO I'M INTERESTED HOW THAT WORKS OUT. MAYBE WE CAN COME OUT WITH DERIVED PARAMETERS, FOR EXAMPLE FLOW-RELATED UPTAKE IN GENERAL. THAT'S THE IDEA WHAT WE WANT TO DO, I WOULD BE HAPPY TO SPREAD THIS WITHIN THE QIN AND IF ANYBODY IS INTERESTED FEEL FREE TO CONTACT ME. THANK U [APPLAUSE] >> ANY QUESTIONS? PLEASE GO AHEAD. >> THAT'S VERY INTERESTING. I'M JUST WONDERING WHY TUMOR HAS LEAST FLOW IN THE METABOLISM MISMATCH, AND IF YOU QUALITY INTO IMAGES, HOW ABOUT MISALIGNMENT BETWEEN THE TWO IMAGES? >> THE MISALIGNMENT, YEAH, THAT'S A PROBLEM THAT'S THE REASON WE THOUGHT WE SHOULD LOOK INTO CT, MANY PATIENTS ARE GOING PET CT, AT LEAST IN ONE SESSION, ALTHOUGH IT'S NOT AT THE SAME TIME AND HAVE A PET MR SCANNER. I MEANING THE MISALIGNMENT, YES THAT MIGHT BE A PROBLEM, BUT BECAUSE WE'RE MAPPING ALWAYS THE WHOLE TUMOR THOUGH I HOPE WE CAN COME UP WITH AN IDEA FOR THIS. THE PHYSICAL PART OF THE QUESTION IS EVEN MORE INTERESTING, WHY THERE IS THIS MISMATCH, AND I THINK THAT'S EXACTLY WHAT WE HAVE TO TALK ABOUT BIOLOGY BECAUSE DRIVERS CAN BE DIFFERENT. WE SEE FOR EXAMPLE IN FAST-GROWING TUMORS, HIGH UPTAKE, MORE THE METABOLIC TURNOVER BECAUSE THERE'S FAST REPLICATION BUT ON THE OTHER SIDE WE SEE HYPOXIA CAN BE A STRONG DRIVER, IF PERFUSION GOES DOWN WE SEE INCREASE, AND IF WE'RE LOOKING IN RESPONSE TO TREATMENT THERE ARE DIFFERENT PATTERNS. IT'S NOT SO EASY WITH FDT ALONE, DIFFERENT DRIVERS BRING TUMOR METABOLISM UP. THAT'S WHY I'M INTERESTED TO SEE HOW PERFUSION IS IN CORRELATION TO THIS. >> VERY INTERESTING TO SEE PARTICULARLY SOME OF THOSE FEATURES. THE FIRST TWO THAT YOU TALKED ABOUT, ASPHERICITY AND TEXTURE MEASURE, HAVE YOU LOOKED AT HOW SENSITIVE EACH ONE OF THOSE IS TO THE SEGMENTATION, BECAUSE IT SEEMS THE ASPHERICITY WILL BE VERY SENSITIVE, HETEROGENEITY MAY NOT BE. >> YEAH, THAT'S ONE OF THE POINTS BECAUSE WHERE I'M NOT SURE BECAUSE OF COURSE THE WAY WE CALCULATE IT THERE'S ALSO A STRONG CORRELATION TO METABOLIC TUMOR VOLUMES, THE QUESTION FOR US WAS DOES IT MAKE A DIFFERENCE OR JUST CAN WE TAKE ONLY MTE BUT WE SAW SITUATIONS WHERE IT MADE A DIFFERENCE AND I'M NOT QUITE SURE WHY SO FAR. BUT THAT HAS TO DO WITH SEGMENTATION, DEFINITELY. >> IS THE PET ACTIVITY -- ONCE YOU GET BEYOND ANY OBVIOUS HOT SPOTS, IT WILL DECAY GRADUALLY. IF THAT'S SEGMENTATION THRESHOLD YOU USE EVERYTHING WILL BE A SPHERE PRETTY MUCH. >> COULD BE. YEAH, THAT'S TRUE. >> DO YOU HAVE A QUESTION? SO IF NO MORE QUESTIONS WE'LL HAVE THE NEXT SPEAKER, HE'S FROM IRELAND, AND ANOTHER ONE OF OUR NEW ASSOCIATE MEMBERS. >> OKAY. I'D LIKE TO THANK DR. TANDON AND NORDSTROM FROM THE INVITATION AND WINFRIED FOR DESCRIBING HETEROGENEITY, AND FLOWNESS MATCH, BOTH OF WHICH FEATURE IN THE SUMMARY I'M GOING TO GIVE. SO I'M FROM THIS PLACE, UNIVERSITY COLLEGE CORK, AND WHICH PROBABLY BOTH THE -- (INDISCERNIBLE) A FAMOUS MATHEMATICIAN, I'M IN THE SCHOOL OF MATHEMATICAL SCIENCES, HEAD OF IT, WE HAVE A GROUP THAT IS ASSOCIATED WITH QIN INVESTIGATORS, AND I'LL DESCRIBE THAT. I GUESS OUR PHILOSOPHY IS THIS -- SOME OF YOU MAY RECOGNIZE A SLIDE THAT COMES BOTH FROM DAVID MENIKOFF AND (INDISCERNIBLE) SUBSTANTIALLY, WHICH IS THE CONCEPT, THAT CANCER IS A MULTIVARIATE PROPERTYES, THERE ARE -- PROCESS, A NUMBER OF FACTORS CONTROL DEVELOPMENT AND PET IS SHORT OF A REMARKABLE TECHNOLOGY BECAUSE IT HAS THE ABILITY TO POTENTIALLY EXPLORE THESE DIFFERENT METABOLIC ASPECTS OF CANCER, AND POTENTIALLY MAY ALLOW THOSE KINDS OF -- THAT TYPE OF IMAGING TO BE TRANSLATED AND ULTIMATELY HELP TO GUIDE STRATIFICATION OR MAYBE EVEN TREATMENT DECISIONS FOR INDIVIDUAL PATIENTS. THE QI ACTIVITY FOR UCC SEC ANY CAL IN NATURE -- TECHNICAL IN NATURE, IN LARGE PART WE HAVE SOME INTEREST IN MODELING SCANNERS USING PHANTOMS TRYING TO UNDERSTAND WHAT THE EMPIRICALLY WHAT NOT ONLY BIOVARIANTS CHARACTERISTICS ARE IN THE SCANNER ALSO NOISE PROPERTIES IN THE SCANNER ESPECIALLY IF DATA ARE RECONSTRUCTED BY ITERATIVE PROCESS, THEN THOSE ARE PROPERTIES MORE COMPLICATED TO UNDERSTAND, AND WE THINK THE PHANTOM DATA MIGHT PROVIDE A USEFUL VEHICLE FOR UNDERSTANDING STOCHASTIC CHARACTERISTICS OF VARIANTS OR DISTRIBUTIONAL CHARACTERISTICS OF DATA BUT ALSO THE CO-VARIANCE CHARACTERISTICS, 3D AND HOW THAT MIGHT VARY SPATIALLY WITHIN THE FIELD OF VIEW, AND ULTIMATELY THAT INFORMATION MIGHT POTENTIALLY BE USED AS AN INTEGRATED INTO CONSTRUCTING UNCERTAINTY MAPS FOR METABOLIC CHARACTERISTICS THAT YOU WOULD DERIVE OR RECOVER FROM METABOLIC IMAGES. SECOND SORT OF THEME IS HETEROGENEITY MEASUREMENTS OF THE TYPE THAT WINFRIED DESCRIBED, OKAY? WE HAVE SPENT A BIT OF -- I WOULD PUT THESE AS SORT OF MORE MODEL-BASED, I THINK THAT WAS THE TERMINOLOGY THAT WINFRIED USED, AND CERTAINLY THE KIND OF WORK THAT DR. JARUS DESCRIBED YESTERDAY, CERTAINLY MOTIVATED THE TYPE OF HETEROGENEITY MEASURES, THOSE TUMOR SPHEROIDS, CONCEPTS, AND HOW THOSE THINGS EVOLVE OVER TIME TO CREATE A PARTICULAR DISTRIBUTION OR UPTAKE PATTERN IS SOMETHING WE'VE BEEN TRYING TO USE IN OUR CONSTRUCTION OF HETEROGENEITY MEASUREMENTS. AND THESE MEASURES ARE, AGAIN, THINGS WE TRY TO SORT OF NOT ONLY, I THINK, I MEAN, IS IT IMPORTANT TO HAVE A MEASUREMENT, BUT YOU ALSO NEED TO HAVE A SORT OF AN UNDERSTANDING ABOUT WHAT FRACTION OF THE INFORMATION AND THE DATA IS ACTUALLY DESCRIBED BY THAT MEASUREMENT, JUST BECAUSE YOU CALCULATE A NUMBER, AN SUV, HETEROGENEITY VALUE, IT'S NOT CLEAR WHETHER THAT NUMBER ACTUALLY IS LEADING BEHIND IN THE IMAGE SET, IN INFORMATION, OR WHETHER IN FACT IT DESCRIBES ANYTHING MEANINGFUL IN THE PARTICULAR IMAGE YOU HAVE IN HAND. AND SO THERE NEEDS TO BE SOME SORT OF CONCEPT OF DIAGNOSTICS, ASSOCIATED WITH FEATURES, THAT WOULD ALSO HELP TO UNDERSTAND THEIR USABILITY OR UTILITY IN THE CONTEXT OF ANY KIND OF PREDICTIVE MODELING. THE THING I'LL PROBABLY ELABORATE THE MOST IS ON KINETIC ANALYSIS AND WHERE KINETIC ANALYSIS IN MOST PET SETTINGS SORT OF RELATES TO SORT OF COMPARTMENT MODELS, USUALLY THIS ONE, THE TWO COMPARTMENT FTG MODEL, SOME MORE COMPLICATED THINGS ARE VERY WELL ESTABLISHED, QUITE OLD CONCEPTS, THEY CAN HAVE A ROLE OR PLAY A ROLE IN CONSTRUCTING MECHANISMS FOR MAPPING METABOLIC PARAMETERS, PARAMETERS SUCH AS THE FLOW MISMATCH WHICH AGAIN I'M AMAZED AT WINFRIED, WE DID NOT COORDINATE, I CAN GUARANTEE, BUT JUST LOOK WHAT I'M GOING TO TALK ABOUT. ONE THING, NANCY GAVE A VERY NICE PRESENTATION YESTERDAY, AND JANICE MENTIONED THIS THING ABOUT THERE BEING A DIVIDE BETWEEN IMAGERS AND CLINICAL SCIENTISTS, AND I THINK ONE OF THE -- WE'RE TRYING TO BRIDGE THAT GAP, I THINK IT PROBABLY IS AS MUCH ABOUT EDUCATION AS JUST SORT OF LIKE THROWING A BUNCH OF TOOLS ACROSS THE GAP AND ASKING PEOPLE ON THE OTHER SIDE TO USE THEM. I THINK THERE IS A SORT OF POTENTIALLY A ROLE FOR QIN IN TRYING TO EDUCATE THE COMMUNITY OF CLINICAL INVESTIGATORS, AND AMONG THAT COMMUNITY ARE CERTAINLY STATISTICIANS. THEY WORM THEIR WAY. I'M A STATISTICIAN. INTO A NUMBER OF CONVERSATIONS AND SORT OF AN EASY MECHANISM OR, YOU KNOW, THAT COULD HELP TO SORT OF TRANSLATE SOME OF THE QIN METHODOLOGIES WOULD BE IN FACT TO TRY AND CONNECT WITH THE THINGS THAT STATISTICIANS USE, I'M NOT SURE IF YOU NOTICED BUT I NOTICED THAT IN NANCY'S TALK SHE SAID I DEVELOPED SOME CODE FOR THIS TO DO THE POWER CALCULATIONS FOR THIS, THAT OR THE OTHER. WELL, IF YOU WANT STATISTICIANS WHO ARE INVOLVED IN CLINICAL TRIALS TO START USING QIN TOOLS, IT MIGHT BE SORT OF A GOOD IDEA TO ULTIMATELY TRY AND BRING THOSE TOOLS INTO CONTACT WITH STATISTICAL GROUPS, AND OR IS A GOOD ENVIRONMENT TO DO THIS BECAUSE WHETHER WE LIKE IT OR NOT, I SUPPOSE, THE TRAINING OF STATISTICIANS NOWADAYS IS ALMOST CERTAINLY TO INVOLVE EXPOSURE AND EDUCATION IN THE USE OF O.R AND O.R. IS NOT SOMETHING YOU'D WRITE HOME TO YOUR COMPUTER SCIENCE PROFESSORS ABOUT BEING A WONDERFUL ENVIRONMENT FOR DEVELOPING THIS OR THAT, IT'S A FUNCTIONAL LANGUAGE, OF THE TYPE OF PYTHON-LIKE IN ITS NATURE, BUT IT'S GOT A HUGE AMOUNT OF TRADITIONAL STATISTICAL STUFF SET IN CONFIDENCE INTERVALS, ALL THAT BUSINESS, A LOT OF FAIRLY SOPHISTICATED MACHINE LEARNING TOOLS, NEURAL NETWORKS, RANDOM FOREST, VASU, THERE'S GOOD COLLECTION, A WHOLE BUNCH OF STATISTICIANS WORKING TO PUT TOOLS IN THAT. IF THERE WERE A BUNCH OF QIN TOOLS THEY MAY END UP USING THEM AS WELL AND MAY END UP GETTING USED AT LEAST IN RETROSPECTIVE ANALYSIS BY STATISTICAL COORDINATING CENTERS FOR MAJOR CLINICAL TRIAL ORGANIZATIONS LIKE MAYBE ECOG-ACRIN AND THE LIKE. THAT'S BEEN A THEME IN OUR GROUP TO DEVELOP TOOLS THAT CAN BE INTEGRATED INTO O.R. FOR THAT REASON. SO HERE IS SORT OF AN OLDER WAY OF THINKING ABOUT KINETICS WHICH GOES BACK TO GUYS THAT ACTUALLY WORKED HERE. ZIERLER AND MEIER, ORIGINALLY WORKED ON VASCULAR FLOW NETWORKS, AND DEVELOPED UNDERSTANDING HOW TO ANALYZE TIME COURSE DATA AND THAT WOULD RESULT FROM INJECTION OF A TRACER, OR INTRODUCTION OF, AND IT FOCUSES LIGHT ON THE IMPULSE RESPONSE TO THE INJECTED DOSE OR RESIDUE FUNCTION. AND RESIDUE FUNCTION ACTUALLY IN STATISTICAL TERMS IS A LIFE TABLE FOR RADIO TRACER ATOMS INTRODUCED. THOSE TRACER ATOMS HAVE A CERTAIN LIFETIME IN THE REGION OF INTEREST THAT YOU HAPPEN TO BE LOOKING AT, AND THEN YOU LOOK AT THOSE OVER TIME AND, OVER TIME MORE AND MORE OF THEM LEAVE THE REGION OF INTEREST, AND THAT'S EXACTLY THE STRUCTURE OF LIFE TABLE. IN FACT, ZIERLER YOU KNOW, MEIER YOU DON'T. HE'S THE SECOND HALF OF THE KAPLAN-MEIER SURVIVAL CURVE, AND SO HE DID THIS FIRST. OKAY. AND SO THE WAY IN WHICH YOU CAN ANALYZE OR UNDERSTAND KINETICS IS ALSO ABOUT TRYING TO UNDERSTAND THIS RESIDUE FUNCTION. AND THE RESIDUE FUNCTION CAPTURES VARIOUS QUANTITIES THAT YOU'VE SEEN PRESENTED YESTERDAY AND MOST RECENTLY BY WINFRIED, AND WHICH ARE FLOW, FLUX, AND SO IF YOU HAVE A SORT OF WAY TO EXTRACT THE RESIDUE, THEN YOU HAVE A MECHANISM TO WORK OUT PARAMETERS OR QUANTITIES RELATED TO THOSE, RELATED TO THAT, RELATED TO THAT STRUCTURE. AND SO VOLUMES OF DISTRIBUTION, AND VARIOUS TYPES, EITHER VASCULAR OR EXTRAVASCULAR AND RETENTION OF TRACER CAN BE READ OFF THIS QUANTITY. THE OTHER THING ABOUT IT, THAT QUANTITY ALLOWS YOU, YOU SEE THE WAY IT'S ENTERING IN HERE, IT'S ENTERING IN IN A SIMPLE LINEAR WAY, AND THAT CAN BE TAKEN ADVANTAGE OF TO ANALYZE DATA, USING ESSENTIALLY LINEAR PROCESS OR AT LEAST CONSTRAINED SQUARE TYPE PROCESSES THAT ARE ONE STEP, THAT ARE ONE STEP, SCHEMES FOR RECOVERING THE IMPULSE RESPONSE. SO WE'VE BEEN USING THIS TO TRY AND IMPLEMENT PARAMETRIC IMAGING TOOLS FOR PET, BASED ON APPLYING THOSE KINDS OF -- THAT SORT OF RESIDUE ANALYSIS, AT A VOXEL BY VOXEL BASIS. LIKE A LOT OF IMAGE ANALYSIS STUFF THAT GOES ON, YOU KNOW, IF YOU SEE 4D DATA THERE'S OFTEN A POINT SOMEBODY WILL SAY WE SHOULD DO PRINCIPAL COMPONENTS OR SINGLE VALUE COMPETITION, MIXTURE MODELING IS A WAY TO DESCRIBE THE VOXEL LEVEL CHARACTERISTICS AS LINEAR COMBINATION OF TIME COURSE INFORMATION AND THAT'S WHAT THESE MU 1 TO MU K ARE TIME COURSE OBSERVATION. MIXTURE MODELING, IF YOU GO ABOUT, I GUESS THIS IS IMPORTANT, IF YOU GO ABOUT TRYING TO RECOVER THESE TIME COURSES, PROPERLY, THAT IS PHYSIOLOGICALLY CONNECTED TO ANATOMY YOU'RE IMAGING, YOU HAVE A POSSIBILITY TO MAP RESIDUES ASSOCIATED WITH THESE TIME COURSES AND THEN EVENTUALLY MAP THE QUANTITIES ON A VOXEL BY VOXEL BASIS. AND FROM THOSE RECOVER PARAMETERS. I'LL SORT OF NOW -- WINFRIED. WINFRIED WAS JUST TALKING ABOUT FLOW MISMATCH STUDY, AND I'D LIKE TO LOOK AT A COLLECTION OF FLOW MISMATCH STUDIES, WHICH WERE THE ONES WINFRIED MENTIONED, ONES THAT CAME FROM A STUDY IN BREAST CANCER BY DAVID MANKOFF AT UNIVERSITY OF WASHINGTON. THIS DATA IS COHORT OF 53 PATIENTS, STUDIED, WITH WATER AND FTG, BEFORE AND AFTER CHEMOTHERAPY. AND SO WE HAVE OUTCOME AS WELL AS THE IMAGING DATA IN THE SETTING. THE STUDIES WERE RUN IN SEQUENCE, THAT IS WATER FIRST, FOLLOWED BY FDG, THERE ISN'T A REGISTRATION ISSUE IN THIS CASE, AND THERE'S A WHOLE BUNCH OF TIMEFRAMES THAT THE DATA -- THAT ARE ASSOCIATED. YOU RECONSTRUCT THE DATA, QUALITY SUFFERS FROM NEGATIVE ARTIFACTS YOU MIGHT BE UPSET. I IMPLEMENTED AN APPROACH TO DOING THIS THAT INVOLVED SORT OF SPLIT AND MERGE SEGMENTATION PROCESS, SPLIT AND MERGE AND MIXTURE MODELING. I THINK IN THIS PROCESS THESE ARE SNAPSHOTS OF ANALYSIS, BUT MAINLY TO HIGHLIGHT CERTAIN ASPECTS OF OPTIMIZATION, WE HAVE CERTAIN SORT OF UNIQUE WAYS OF CALCULATING OR IMPOSING SPATIAL CONSTRAINTS ON MIXING FRACTIONS AND IN ORDER THAT THE RECOVERED MIXING COEFFICIENTS HAVE SOME SPATIAL COHERENCE, AS WELL AS TIME COURSES, AND YOU SEE IN THIS THING WE HAVE TIME COURSES LINKED FOR THE WATER PLUS THE FDG. THIS IS MY THING. I THINK IT IS IMPORTANT TO HAVE INFORMATION OF THIS TYPE AS WELL TO SATISFY YOURSELF THAT THE CALCULATION IS ACTUALLY SQUEEZING, IS WHAT'S IN THE DATA, OUT OF THIS, AND WHAT'S LEFT IS NOISE. SO DIAGNOSTICS I THINK ARE KIND OF IMPORTANT, IF YOU'RE DOING QUANTITATION. THEN YOU CAN FOLLOW ON, THE ANALYSIS LIKE THIS, WITH RESIDUE CALCULATION, TO ESSENTIALLY MODEL OR ANALYZE TIME COURSES THAT YOU'VE RECOVERED FROM THE ALGORITHM, BY -- YOU SEE THESE CURVES HERE ARE LIKE SURVIVAL ANALYSIS CURVES. THAT'S BECAUSE THEY ARE SURVIVAL CURVES. THOSE ARE LIFE TABLES, CHARACTERISTICS. AND SO THERE ISN'T A SORT OF PARAMETRIC KINETIC MODEL THAT'S INVOLVED IN EVALUATING THESE. AND YOU COULD ARGUE FROM A NUMBER OF POINTS OF VIEW THAT IT WOULDN'T MAKE SENSE TO CONSIDER THOSE QUANTITIES AS REPRESENTING ANYTHING THAT IS HOMOGENOUS IN TERMS OF THE KINETICS EITHER. AND SO THOSE THINGS ARE SORT OF PUT TOGETHER TO OBTAIN RESIDUE FUNCTIONS PER VOXEL, AND CREATE MAPS LIKE THIS. OKAY. MAPS-- SO THIS IS A STUDY, THIS IS SHOWING CORONAL AND SAGITTAL VIEWS SUPERIMPOSED, THIS IS CT TRANSMISSION DATA, THIS IS NOT A PET CT SCANNER, THIS CT INFORMATION IS OBTAINED THROUGH TRANSMISSION SCANS, EARLY -- OR LATE 1990s. AND BUT ANYWAY, YOU CAN SEE, YOU KNOW, THE ABILITY, THIS IS LIKE AN UPTAKE IMAGE, THE FIRST PANEL. THE SECOND PANEL IS SORT OF SHOWING THE EXTRACTED FLUX OF FDG, HIGH IN THE MYOCARDIUM, AND IN THE TUMOR. AND THEN IT'S BLOOD FLOW IMAGE, THIS ONE. THIS ONE IS POSSIBLY INTERESTING. THIS ONE IS SHOWING RETENTION OF WATER. SO MAYBE POSSIBLY DISRUPTED VASCULAR CHARACTERISTICS, IN THE NEIGHBORHOOD OF THE TUMOR. BUT ALSO, YOU'LL NOTICE THIS IN THE AREA OF THE MYOCARDIUM, THERE'S ALSO SOMEWHAT DISRUPTED PICTURE OF KINETICS IN THAT REGION. THAT'S CONSISTENT WITH IDEAS ABOUT HOW CHEMOTHERAPY CAN ACTUALLY IMPACT MICRO VASCULATURE IN PATIENTS AS WELL. AND THEN THERE IS CALCULATION WHICH IS SOME SENSE LIKE A TOUR DE FORCE, BECAUSE CALCULATION OF RATIO, ESPECIALLY RATIO INVOLVING WATER WHICH IS VERY NOISY TYPE OF QUANTITY TO CALCULATE AND FDG, CAN BE AFFECTED IN THIS WAY. SO TO PRODUCE SOMETHING THAT ADDS A REASONABLE PATTERN WITHIN FIELD OF VIEW. I CAN SEE BOB LOOKING AT HIS THING. YOU CAN TAKE THOSE IMAGES AND YOU CAN ANALYZE THOSE IMAGES, AS FEATURES, IF YOU LIKE, TO EVALUATE WHAT SORT OF RELATIONSHIP THOSE THINGS HAVE BETWEEN NORMAL TISSUE AND TUMOR, BUT PROBABLY MORE IMPORTANTLY TO EVALUATE IN THE CONTEXT OF OUTCOME ANALYSIS, AND NOT AS A SINGLE VARIABLE, THE PET VARIABLE ON ITS OWN VERSUS OUTCOME, BUT IN A MULTIVARIATE WAY WHERE YOU ACTUALLY TAKE ACCOUNT OF OTHER FACTORS THAT ARE PRESENT, AND THEN EVALUATE IN THAT SETTING THE ADDITIONAL BENEFIT OF THE PET MEASURE. AND SO THAT KIND OF STUFF IS ONE -- THERE'S A LINE HERE WHICH IS SORT OF THE LINE OF SIGNIFICANCE, STATISTICAL SIGNIFICANCE, AND THERE'S TUMOR TO CONTRALATERAL BREAST MEASURES, THAT SUGGESTS VARIOUS FEATURES MAY HAVE A POTENTIAL ROLE OR RELATIONSHIP TO OUTCOME TO THESE THREE OR OVERALL SURVIVAL, THAT'S SORT OF OUR THING TO DEMONSTRATE THE 4D ANALYSIS. AT THE MOMENT WE'RE TRYING TO DEVELOP A BETTER UNDERSTANDING OF THE EFFICIENCY OF ALGORITHMS LIKE THIS, IMPROVE THE LINKING OF MULTIPLE INJECTIONS, SO WE CAN ANALYZE STUDIES WITH MANY INJECTIONS, AND POTENTIALLY MANY TYPES OF DATA, MR AS WELL AS PET, IN A SCHEME LIKE THIS, AND PROBABLY NUMBER THREE IS PROBABLY THE MOST IMPORTANT ONE FROM THE PART OF WHAT WAS BEING TALKED ABOUT AT LEAST YESTERDAY, SOME OF TODAY, TRYING TO DEVELOP UNCERTAINTY ESTIMATES AS A ROUTINE FOR DERIVED METABOLIC QUANTITIES. I'D LIKE TO THANK YOU FOR YOUR ATTENTION. I THINK I'M STANDING BETWEEN YOU AND YOUR LUNCH AT THIS POINT. SO THANK YOU. [APPLAUSE] >> QUESTIONS? I GUESS -- OH, YEAH, WE DO. OKAY. ONE AND THEN WE CAN BREAK FOR LUNCH. EVERYBODY'S LOOKING. >> IT SEEMS TO ME, IN THIS APPROACH WITH MIXTURE ANALYSIS, THAT YOU'RE PUSHING THE LIMITS IN TERMS OF NUMBER OF PARAMETERS, THAT YOU HAVE TO ESTIMATE IN THIS MODELING, BECAUSE THAT'S WHAT YOU'RE DOING. AND YOU TEND TO -- WHEN YOU HAVE TOO MANY PARAMETERS YOU TEND TO GET TERRIBLE COVARIATES BETWEEN PARAMETER ESTIMATES THAT CAN BE VERY CLOSE TO 1. AND IF YOU LOOK CAREFULLY, AT THAT END OF THINGS, TO SEE WHAT'S ACTUALLY IDENTIFIABLE, VERSUS WHAT ISN'T, I THINK -- SCIENTIFIC -- IT'S A REAL POTENTIAL PROBLEM. >> IN THE END I GAVE A DISPLAY, THERE WERE 17 METABOLIC PARAMETERS RECOVERED FROM THIS DATASET, IN WHICH THERE ARE 82 TIMEFRAMES OF DATA. SO, THAT'S SORT OF THE -- AND THEN YOU CAN -- I THINK WHAT YOU'RE BASICALLY DESCRIBING THERE CAN BE ANALYZED FORMALY, YOU CAN SORT OF LOOK AT WHAT ARE THE COVARIATE CHARACTERISTICS, 17 DIMENSIONAL SPACE, AND IF YOU'RE ASKING WHETHER THERE ARE RELATIONSHIPS BETWEEN THOSE PARAMETERS, ABSOLUTELY. SO IF YOU DO A PRINCIPAL COMPONENT ANALYSIS ON THOSE VARIABLES, 90% OF THE VARIANCE IN DATA IS EXPLAINED BY FOUR OF THOSE PARAMETERS. CERTAIN LINEAR COMBINATIONS THAT CAN SORT OF COME TOGETHER TO EXPLAIN OR REPRESENT THAT METABOLIC SPACE. IF YOU WERE DOING NEURAL NETWORKS, THAT WOULD BE GOOD NEWS. YOU WOULD THEN SAY, OKAY, WELL, THAT'S THE NEURAL NETWORK, IT'S GOING TO FIND MAGICALLY THE WEIGHTING OF THOSE PARAMETERS TO IDENTIFY THE BITS THAT ACTUALLY SORT OF SEPARATELY CONTRIBUTE TO OUTCOME, AND THAT WOULD BE SORT OF ONE WAY OF SORT OF GOING AT THAT. I DON'T THINK THE ANSWER IS TO SORT OF UP FRONT -- LET'S ONLY EXTRACT FOUR PARAMETERS, I THINK THAT PROBABLY MAY LIMIT THE RICHNESS OF THE ANALYSIS YOU CAN DO AT THE OTHER SIDE. A BETTER APPROACH, OKAY, IS TO MILK DATA AS MUCH AS YOU CAN, GET TO SETS DESCRIBING WHAT'S GOING ON, IF YOU LIKE, AND THEN GO FROM THERE TO THE OUTCOME ANALYSIS. >> OKAY. I THINK IF WE DON'T HAVE ANY OTHER QUESTIONS, WE'LL BREAK FOR LUNCH. I'M GOING TO INVOKE THE DIRECTOR'S PREROGATIVE AND CHANGE THE SCHEDULE THIS AFTERNOON A LITTLE BIT. WE'RE GOING TO -- INSTEAD OF TALKING ABOUT THE TOOL CATALOG IN MORE DETAIL, I THOUGHT WE WERE GOING TO HAVE THE BENCHMARKS MORE IN PLACE AND WE COULD GO DOWN THAT LIST OF BENCHMARKS, BUT WE'LL WAIT AND DO THAT AT SOME OTHER TIME. INSTEAD OF GOING INTO DETAIL ON THE TOOLS, I'M GOING TO MOVE MY DISCUSSION OF FUTURE IN QIN UP TO BEFORE THE BREAK, AND THEN TOM HAS SOME INFORMATION ON THE BIDS GROUP, WE PURPOSELY DIDN'T PUT IN REPORTS FROM THE DIFFERENT WORKING GROUPS BECAUSE WE DIDN'T WANT TO PRESSURE YOU GUYS INTO MAKING A LOT OF SLIDES AND EVERYTHING. BUT TOM SAID HE WOULD DO THAT ANYWAY. SO HE MADE A LOT OF SLIDES FOR THE BIDS GROUP. THERE'S INTERESTING THINGS GOING ON. HE WANTS TO CAPTURE PEOPLE'S INTEREST IN THIS. SO TOM, GO AHEAD. >> THANKS. THE MAIN REASON I WANTED TO GET IN FRONT OF YOU AND DO THIS, WITHIN THE BIDS GROUP WE HAVE A GOOD VIBRANT ACTIVE BUNCH OF PEOPLE BUT THEY ARE NOT ALL TOOL DEVELOPERS. WE DON'T HAVE A WAY TO GET THE INTERSECTION OF ALL OF THESE GROUPS HERE AT THIS MEETING. SO BASICALLY THIS IS A CALLOUT TO TOOL DEVELOPERS TO DO SEGMENTATION AND FEATURE EXTRACTION AND PREDICTION IN THIS INITIAL PIPELINE WE'RE DOING, AND WE ACTUALLY HAVE A PRETTY GOOD LIST OF TOOLS WE'RE GOING TO DO IN THIS ROUND OF THIS DEMONSTRATION PROJECT, BUT WE DEFINITELY WOULD LIKE TO HAVE SOME OTHERS. I'LL SHOW YOU IN THE DIAGRAM EXACTLY WHAT WE DO HAVE. >> (INAUDIBLE). >> I LOVE IT. EVERY SEMESTER I GET TO DO THIS WITH ONE STUDENT. WHAT A STUPID QUESTION! I ALWAYS LIKE TO DO THAT. THERE'S NO SUCH THING AS A STUPID QUESTION, BUT IN ENGINEERING THERE ARE SOME STUPID QUESTIONS. NO. IT'S NOT STUPID. I DON'T KNOW WHAT IT STANDS FOR. HOW'S THAT? [LAUGHTER] >> BIOINFORMATICS I.T. AND DATA SHARING. >> THERE. >> WHEN WE -- >> I ALMOST BLUFFED MY WAY OUT OF THAT. >> WHEN WE STARTED THESE THINGS MANY YEARS AGO, TEN YEARS AGO, THE GROUP HAD 13 WORKING GROUPS. WE DECIDED THAT WAS TOO MANY SO WE CUT IT DOWN. THERE'S A LOT OF "ANDS." >> BUT NO As. >> WE PUSHED EVERYTHING TOGETHER. >> THIS IS THE DESCRIPTION, I CUT AND PASTED FROM THE DESCRIPTION WE SENT TO THE COMMITTEE TO APPROVE. IT'S A COLLABORATIVE PROJECT, IT'S NOT A CHALLENGE OR COMPETITION THING. THE GOAL IS TO COME UP WITH DECISION SUPPORT TOOLS, THAT'S THE LONG TERM GOAL THAT WE REALLY WANT TO FACILITATE IN BIDS, AND WE WANT TO IDENTIFY THE BEST PRACTICES FOR CREATING AND SHARING MODULES SO THERE'S A LOT OF MODULES. THE SPREAD SHE'S 60, 70-SOME TOOLS, QIN DESPERATELY TRYING TO GET PEOPLE TO SHARE THESE TOOLS, EVALUATE AND TRY ON DIFFERENT DATASETS AND SO ON. WE'RE TRYING TO COME UP WITH A WAY TO DO THAT. AS A BY-PRODUCT TO SPUR EVEN MORE INTERINSTITUTIONAL COLLABORATION. SO THE INITIAL AND LONG-TERM GOALS TO TEST OUT MULTIPLE INFRASTRUCTURES, SO IT'S REALLY IMPORTANT TO SAY WE'RE NOT GOING TO COME UP AT THE END OF THIS THING, AT THE END OF A YEAR WHEN THIS IS GOING TO TAKE, WITH LIKE THE SOLUTION TO THIS THING. I MEAN, IN THE LONG TERM WE REALLY DO HOPE WE COME UP WITH SOMETHING THAT IS SO STANDARDIZED THAT PEOPLE REALLY WILL BE ABLE TO JUST DEVELOP ONCE, PACKAGE ONCE AND PLUG MODULES INTO SOMETHING. RIGHT NOW WE'RE NOT INTO PICKING WINNERS. THERE'S AT LEAST FOUR INSTITUTIONS DEVELOPED, THAT'S GREAT, WE WANT TO REPLICATE THAT EXPERIENCE IN A COMMUNITY ENVIRONMENT WITHIN BIDS, TO MAKE SURE PEOPLE OUTSIDE KNOW TO YOU TO PLUG THINGS IN. IT'S AN EXERCISE WE'LL DO THIS NEXT YEAR. HERE ARE THE FOUR PIPELINE INFRASTRUCTURES, QIFP SAND AND DANIEL DEVELOPED AT STANFORD, YUNPIPE AT EMORY BY ASHISH, C-B,POP AND JAYASHREE AND THIS. THIS WILL ALLOW PEOPLE TO PLUG IN TOOLS TO DO THE INDIVIDUAL TASKS THAT PEOPLE IN IMAGE ANALYSIS AND IMAGING PROCESS HAVE BEEN DOING FOR DAYS AND LET THEM WORK TOGETHER. HARDWARE BY FRED AT ARKANSAS, A LOT OF TOOLS WE'LL TEST OUT. I DON'T WANT ANYBODY TO FEEL LEFT OUT, IF THEIR NAME IS NOT ON THE NEXT SLIDE THAT THEY ARE NOT INCLUDED. YOU'RE JUST NOT THERE YET. CONTACT ME IF YOU WANT TO. WE WENT THROUGH YESTERDAY AT OUR ORGANIZING MEETING SOME VARIABLE DATASETS WHERE THERE'S NOT ONLY IMAGING DATA BUT ALSO CLINICAL DATA WITH SOME OUTCOMES, AND FROM MY SELFISH INTEREST I WOULD LIKE MOLECULAR DATA. THAT'S IN OUR FUTURE, IT'S ACTUALLY IN OUR PRESENT, BUT NOT AS MUCH AS JUST THE IMAGING DATA. SO HERE'S THE DRAFT THAT WE LITERALLY CAME UP WITH DURING OUR MEETING YESTERDAY ON A WHITE BOARD, AND THEN IT WAS PUT TOGETHER ON OUR GOOGLE DOC. AND IT BASICALLY UP AT THE TOP UP HERE WE'VE GOT OUR DATASET, TO USE 50 CASES FROM RSNA C3, LUNG AND CT IMAGES, ANNOTATED, AND JUSTIN IS GOING TO CONVERT THESE TO AIM FORMAT THAT DANIEL DEVELOPED IN THE FIRST OR SECOND YEAR OF QIN AT STANFORD. AND THEN WE'RE GOING TO TAKE IT THROUGH THREE PROCESSING PHASES, SEGMENTATION, FEATURE EXTRACTION, AND THEN PREDICTION, WHICH WILL PROBABLY INCLUDING SOME FEATURE PRE-PROCESSING AS WELL. AND WE DEFINED THE FORMATS IN BETWEEN THESE, OKAY, NOTHING ELSE IS SPECIFIED ABOUT HOW EACH ONE OF THESE DOES WHAT -- HOW EACH STAGE DOES WHAT IT'S DOING. THIS IS AIMED, THIS IS DICOM SEG, WE HAVE CANDIDATES FOR HOW TO SPECIFY THE FEATURES THEMSELVES, BUT WE'RE A LITTLE BIT OPEN ON THAT RIGHT NOW. AS YOU CAN SEE INSIDE THESE BOXES WE'VE GOT A COLLECTION OF TOOLS, EACH ONE OF THESE TOOLS WILL PERFORM THIS SEGMENTATION TASK. YOU CAN SEE WE'VE GOT MULTIPLE INSTITUTIONS INVOLVED HERE, NOW, AND WE'D LIKE TO HAVE EVEN MORE INSTITUTIONS INVOLVED. SAME THING WITH FEATURE EXTRACTION. WE HAVE A NUMBER OF POSSIBLE TOOLS HERE. AND THEN FOR PREDICTION, THERE'S STILL SOME DEBATE AS TO WHETHER THIS MODULE SHOULD ONLY BE A CLASSIFICATION, A BUILT MODEL OR WHETHER IT COULD BE SOMETHING WHERE THE PIPELINE RESULTS IN BUILDING A MODEL. EITHER ONE OF THEM COULD REALLY BE OKAY. I THINK THE ORIGINAL THING WE HAD IN MIND THEY WOULD BE BUILT MODELS, DO PREDICTION, TAKEOUT COMES DATA FROM DATASET, LONG DATA SET AND TRY TO TAKE IMAGES, PROCESS THEM THROUGH THE PIPELINE, MAKE A PREDICTION ON EACH OF THE 50 CASES AND SEE HOW THEY DO. I WANT TO EMPHASIZE THIS IS NOT A COMPETITION. THAT'S WHY WE DIDN'T MAKE THIS A CHALLENGE. THIS IS AN EXERCISE WHERE, YOU KNOW, THE GEAR HEADS IN THE BIDS GROUP WHO, YOU KNOW, SPEAK DOCKER AND, YOU KNOW, LINUX AND THINGS LIKE THIS, WE'RE TRYING TO PUT THE PIECES TOGETHER, ARE TRYING TO TRAIN UP OR GET PEOPLE TO HAVE MORE EXPERIENCE WITH USING THOSE TECHNOLOGIES, SO THEY ARE NOT JUST A BUNCH EVER -- OF ACRONYMS AND NAMES AND HOPEFULLY IN A YEAR WE'LL HAVE PEOPLE EXCHANGING DATASETS, TRYING DATASETS, THE GOAL OF QIN FROM THE BEGINNING. THAT'S ALL I HAVE. ANY QUESTIONS? >> (INAUDIBLE). >> YEAH, WE'RE HOPING TO INVENT A FEW MORE, JUST FOR YOU, MIKE. >> GREAT. IT'S INTERESTING. IT WAS ONLY A FEW YEARS AGO WE WERE JUST STARTING TO TALK ABOUT THINGS LIKE THESE PIPELINES, AND NOW WE'RE BUSY INTO THEM. SO, WE'RE INTO THE LAST PORTION OF WHAT'S GOING ON HERE, AND WHERE'S MY -- OKAY. IT ISN'T HERE FROM YESTERDAY. IT SHOULD BE ON THE L DRIVE. WHO LOGGED IN TO THIS? THIS IS NOT OUR L DRIVE. OKAY. SPEAK QUIETLY AMONG YOURSELVES. [OFF-MIC DISCUSSION] >> OKAY. HERE WE GO. SO I WANT TO SHARE WITH YOU SOME OF THE THINGS THAT HAVE BEEN GOING ON HERE AT NCI, BEHIND THE SCENES WHILE YOU'VE ALL BEEN BUSY MAKING TOOLS AND MAKING ALL THESE GOOD THINGS WORK. WE HAVE A NUMBER OF OTHER ISSUES THAT WE DEAL WITH, WITH RESPECT TO THE WHOLE GRANTS PROCESS, AND HOW THINGS MOVE FORWARD. AS YOU KNOW, EVERY THEY YEARS WE HAVE TO PUT IN A CONCEPT FOR THESE KINDS OF APPLICATIONS FOR THE -- YEAH, THE PARs, AND SO WE'VE DONE THAT. WE'VE PUT IT ALL IN FOR THIS ROUND AND EVERYTHING IS GOING FINE. I WANT TO SHARE WITH YOU SOME OF THE THINGS THAT ARE GOING ON AS THAT GOES FORWARD. LOOKING BACKWARDS JUST FOR A SECOND, THIS IS HOW THE QIN HAS GROWN OVER THE LAST 10 YEARS. WE STARTED OFF BACK IN 2008, AND IT'S GROWN STEADILY EVER SINCE, THROUGH 2014, AND AS YOU CAN SEE WE STARTED TO PEAK OVER A LITTLE BIT. THESE ARE THE TEAMS, THE DATA HERE ARE ONLY THE TEAMS THAT ARE FUNDED, NOT THOSE THAT ARE IN NO-COST EXTENSION OR ANYTHING OF THAT SORT. SOME PEOPLE TELL YOU THERE'S 20 TEAMS, SOME PEOPLE TELL YOU THERE'S 16 TEAMS. I COUNT ONLY THE ONES THAT ARE CURRENTLY FOUNDED, NOT NO IN NO-COST EXTENSION, OTHER PEOPLE USE NO-COST EXTENSION. WE'VE HAD INTERESTING STATISTICS, 256 RECEIVED APPLICATIONS, THAT'S VERY GOOD OVER 10 YEARS. WE'VE HAD 35 AWARDS. NOW, I KNOW YOU'RE ALL FRANTICALLY GETTING OUT YOUR CALCULATORS, THAT'S A SUCCESS RATE OF 12.7%. AND THAT'S WELL ABOVE WHAT THE PAYLINE IS RIGHT NOW FOR R01s. IT'S RUNNING AROUND 8%, IN THE EARLY DAYS IT WAS 9%, SO QIN IN GENERAL LOOKS LIKE IT'S DOING A LOT BETTER THAN JUST PUTTING IN FOR NORMAL R01s. I WANTED TO LOOK AT THAT A LITTLE MORE CLOSELY. TO DO THAT I WANT TO TALK TO YOU A LITTLE BIT ABOUT HOW THE U01 SUPPORT TAKES PLACE. FIRST OF ALL, U01s ARE PART OF THE OVERALL RPG POOL, RESEARCH PROJECT GRANTS POOL OF MONEY NIH HAS, BROKEN DOWN INTO INSTITUTES. UNFORTUNATELY THE U01 POOL IS A FRACTION OF WHAT THE TOTAL RPG POOL IS. AND BY ALL ACCOUNTS IT'S NOT A CONSTANT FRACTION, ROUND TO ROUND, YEAR TO YEAR. IT SEEMS TO VARY QUITE A BIT. IT CERTAINLY DOESN'T TAKE INTO ACCOUNT THE NUMBER OF U01s THAT ARE COMING UP FOR APPROVAL. NOW, IN FAIRNESS TO OUR DIRECTORS AT DCTD, IT'S NOT THEIR FAULT. THIS SEEMS TO BE COMING FROM SOMEBODY OUTSIDE THE DIRECT SYSTEM, PROBABLY FROM THE FINANCIAL PEOPLE OF ONE KIND OR ANOTHER, THAT DECIDE HOW MUCH MONEY GOES INTO THIS POOL. AS A RESULT, THE PROGRAM MANAGERS, MYSELF AND OTHERS, ARE JUST SORT OF LEFT TO DO WHATEVER IT IS WE CAN DO, TO MAKE APPLICATIONS THAT SCORE WELL FUNDED. AND WE DO THAT BY SELECTING APPLICATIONS FROM EACH PROGRAM, EACH OF THE U01 APPLICATIONS FROM EACH PROGRAM, ONE AT A TIME UNTIL THAT U01 MONEY RUNS OUT. AND WHEN THAT'S GONE, THEN THAT'S THE FUNDING THAT TOOK PLACE FOR THAT ROUND. SO WE HAVE TO WRITE JUSTIFICATIONS FOR THIS. OF COURSE THEY HAVE TO BE VERY WELL FUNDED. VERY WELL WRITTEN. AND THEY CAN ONLY BE ONE PAGE. OKAY. SO THIS IS A PROCESS IN COMPLETENESS, YOU'RE FAMILIAR WITH THE PEER REVIEW PROCESS, YOU'VE ALL BEEN THROUGH THAT. YOU'VE PROBABLY ALL SAT ON STUDY SECTIONS WHERE THAT'S DONE. THOSE GRANTS WITH SCORED AND COME TO ME, PROGRAMMED, WE LOOK AT THE SCORES, WE DECIDE IF THERE'S SOME SORT OF CUTOFF, SOME SORT OF PLACE THAT WE CAN SAY, OKAY, THESE ARE WORTHY OF GOING FORWARD. THESE ARE NOT GOING TO MAKE IT. GENERALLY THAT'S DONE BY SCORE. BUT NOT ALWAYS. SO IT DEPENDS. THERE'S NO PERCENTILING. AFTER THAT PROGRAM, NAMELY ME AND DARRELL HAS BEEN DOING THAT PREPARE A ONE-PAGE JUSTIFICATION SHEET WHICH GOES THROUGH THE EEPs, EXCITEMENT YOUR PROGRAM HAS WITHIN IT, AND POTENTIAL THAT YOUR PROGRAM WILL HAVE IF IT'S SUCCESSFUL. THAT'S ALL YOU GET TO WRITE ON THIS PAGE BEFORE YOU'RE DONE. ALL THAT WORK YOU DID PUTTING YOUR GRANT TOGETHER COMES DOWN TO ONE PAGE, THAT MOVES FORWARD. WE THEN MAKE THAT PRESENTATION TO THE DIRECTORS OF OUR DIVISION, AND THERE'S A PRIORITIZATION ON THOSE GRANTS,& AND WE'LL SEE WHO MOVES FORWARD. NOW, BECAUSE PERCENTILING IS NOT DONE, THAT MEANS EVEN IF YOU GET A PERFECT 10, WE'VE GOT TO WRITE THE JUSTIFICATION FOR IT. EVEN IF YOU GET A NEARLY PERFECT 10, NEARLY PERFECT SCORE, WE HAVE TO WRITE THE JUSTIFICATION FOR IT. THE OTHER PROBLEM IS UNDER PRIORITIZATION WE'RE NOT PRIORITIZING JUST THE QIN AWARDS. WE'RE PRIORITIZING ALL THE U01 AWARDS THAT GO FORWARD. >> I WANT TO ADD A LITTLE BIT OF BACKGROUND ABOUT THAT. WHAT I'VE LEARNED FROM WHAT YOU'VE TOLD ME THE SUCCESS OF THIS PARTICULAR QIN, U01, LED TO A LOT OF COPY-CAT ACTIVITY HERE, AND THAT'S AN INFLUENCE IN THE COMPETITION FOR THIS DESIGNATION OR THIS TYPE OF MECHANISM. >> GOOD POINT. >> IT'S NOT COMPLETELY OUT OF OUR CONTROL BUT PART OF IT IS OUR OWN SUCCESS. >> WE'VE BEEN TOLD MANY TIMES BY OUR DIRECTOR WE'RE VICTIMS OF OUR OWN SUCCESS. YOU'VE DONE SO WELL WITH THIS U01 EFFORT AND ALL THAT, THAT IT'S GETTING HARDER AND HARDER TO PUSH SOMETHING THROUGH BECAUSE YOU'VE BEEN SO SUCCESSFUL. THAT'S A VERY GOOD POINT. I'LL SHOW YOU ANOTHER STATISTIC ALONG WITH THAT. IF WE LOOK AT THE YEARS OF RECEIPT AND FUNDING FOR QIN, THIS IS WHAT YOU SEE. ASIDE FROM THE VERY FIRST YEAR WHERE WE GOT ONE AWARD FUNDED OUT OF 12, WHICH IS 8.3% RATE, YOU CAN SEE FOR THE FOLLOWING MANY YEARS WE WERE WELL, WELL ABOVE ANY PAY LINE NUMBERS. 20, 16, EVEN UP TO 22% SUCCESS RATE FOR QIN, AND THEN FREQUENTLY RIGHT NOW IN THE LATER YEARS IT'S DROPPED PRECIPITOUSLY, PART OF WHAT JANET JUST SAID, VICTIMS OF OUR OWN SUCCESS FOR THE MOST PART. I WANT TO SHOW YOU HOW THAT REALLY FACTORS IN. IF YOU LOOK AT THE NUMBER OF U01 AWARDS RECEIVED BY DCTD, BY OUR DIVISION YOU SEE IN THE LOWER YEARS, 2008 TO 2011, THE QIN WHICH IS IN THE ORANGE IS A SIZEABLE FRACTION OF TOTAL NUMBER THAT IT RECEIVED. IN FACT IN 2010 I THINK WE WERE 88%, ALMOST 90% OF THE U01s RECEIVED BY OUR WHOLE DIVISION WERE QIN GRANTS. YOU CAN ALSO SEE THAT THE GRANTS BEING RECEIVED BY -- APPLICATIONS RECEIVED BY DCTD ARE GROWING EXPONENTIALLY. FISCAL 2018 HASN'T BEEN COMPLETED, IT'S A PROJECTION, BUT YOU CAN SEE THE CURVE IS GROWING VERY RAPIDLY. AND WE'VE GONE FROM WHAT LOOKS LIKE 80, 90% TO THE SINGLE DIGIT PERCENT FOR LAST YEAR AND THIS YEAR. THE QIN GRANTS HAVE STAYED CONSTANT, WE GET 26 TO 30 GRANTS PER YEAR IN THE QIN. NOW, IF WE GO PLOT THE PERCENT OF DCTD U01s THAT ARE QINs, AND PLOT AGAINST THAT THE PERCENTILE OF QIN APPLICATION AWARDED YOU SEE YOU GET A CURVE LIKE THIS. I MEAN, IT'S OBVIOUS. THE MORE YOU GOT IN THERE, THE MORE CHANCES YOU ARE OF GETTING FUNDED. AND THE DAYS WHERE THIS WAS THE SIDE OF THE FENCE WE WERE WORKING ON, THOSE ARE LONG GONE. WE'RE WORKING DOWN HERE NOW. WE'RE WORKING WHERE WE'RE LOW FRACTION OF THE TOTAL NUMBER OF U01 GRANTS COMING INTO THE PROGRAM, AND SO WE CAN EXPECT TO BE FAIRLY LOW FRACTION OF THOSE THAT ARE QIN BEING FUNDED. SO, THAT'S THE DILEMMA. WE AS PROGRAM HAVE TO FACE THAT AND DECIDE WHAT TO DO. SPEAKING OF PROGRAM -- >> I HAVE ANOTHER COMMENT. THE OTHER FRAMEWORK IS THE FUNDING VICISSITUDES EXPERIENCED IN RECENT TIME PERIOD AS WELL. >> WELL, YES. >> THERE HASN'T BEEN ANY BUDGET. >> THERE'S NOT BEEN ANY GROWTH IN THE AMOUNT OF MONEY AVAILABLE. >> IT HASN'T COME IN. WE'VE HAD APPLICATIONS, THEY SAY WE'RE NOT FUNDING ROUNDS UNTIL WE HAVE SOMETHING OTHER THAN RESOLUTION. >> WE JUST NOW GOT FUNDING AND WE'LL SEE WHAT HAPPENS. FOR THE MOST PART IF WE TAKE THE YEAR BEFORE, A COUPLE YEARS BEFORE, IT'S NOT GOING TO GET REAL ROSY REAL QUICK, OKAY? WE HAVE A CHOICE. WE COULD CERTAINLY SIT HERE ON OUR HANDS AND SAY IT'S GOING TO BE BUSINESS AS USUAL. BUT I GET THE IMPRESSION THAT'S GOING TO BE SORT OF LIKE THIS. YOU KNOW, I'M AFRAID THAT'S GOING TO BE A TOUGH PUSH. SO, THE QUESTION IS WHAT CAN WE DO, WHERE CAN WE GET CREATIVE AND CLEVER AND DO SOMETHING DIFFERENT TO MAKE SURE THE GOOD APPLICATIONS IN THE QIN ARE INDEED FUNDED? ONE OF THE THINGS WE'RE TALKING ABOUT IS DOING AN END RUN. AND THAT END RUN CONSISTS OF MOVING THE QIN FROM THE U01 FUNDING CYCLE TO R01 FUNDING CYCLE. NOW, THAT HAS SEVERAL DISADVANTAGES, SEVERAL ADVANTAGES. I WANT TO SHARE THOSE SO YOU KNOW OUR TRAIN OF THOUGHT IN THIS PROCESS. THE DISADVANTAGES ARE OBVIOUS. NO LONGER CAN WE REQUIRE PARTICIPATION IN A NETWORK. NO LONGER CAN WE REQUIRE PARTICIPATION IN AN ANNUAL MEETING SUCH AS THIS. WE CAN'T REQUIRE INCLUSION OF YOUR WORK INTO AN ANNUAL REPORT THAT WE'VE BEEN SO PROUD OF DISPLAYING TO A LOT OF OUR PEOPLE. R01 DOESN'T PERMIT THOSE REQUIREMENTS. U01 MECHANISM DOES. THE ADVANTAGE TO ALL OF THIS IS WE CAN GET TO A STUDY SECTION WHERE THE SCORES ARE PERCENTILED, AND WE SUDDENLY RUN INTO THE OPPORTUNITY RIGHT NOW THE PERCENTILE IS AT 9% I BELIEVE. AND IT MAY GO UP EVEN HIGHER, IF THE FUNDING IMPROVES. AND OF COURSE WE'RE ALSO PARTICIPATING ON THE OTHER SIDE OF THAT MAGIC FENCE, RPG POOL, NOW PARTICIPATING IN A BIGGER SIDE OF THE FUNDING POOL. SO THERE'S SOME ADVANTAGES THERE. >> THE NEW NCI CHIEF IS COMMITTED TO KEEPING THAT ENLARGED. >> GONE ONRECORD SAYING HE'S PUTTING $100 MILLION IN THE RPG POOL THIS YEAR, SO THAT'S RIGHT. SO THERE ARE A LOT OF GOOD REASONS FOR THAT. WE HAVE TO BE VERY CAUTIOUS THOUGH. WE HAVE TO BE VERY CAUTIOUS. THE MAJOR CONCERN THAT I HAVE IS WHETHER OR NOT THE PERCENTILING OF THE U01s WILL BE DONE IN A WAY THAT EMPHASIZES THE QIN MISSION. IF WE GET PUT INTO A STUDY SECTION WHERE IT'S A FEW QIN APPLICATIONS ARE A PART OF A MUCH BIGGER STUDY SECTION, I'M AFRAID IT'S NOT GOING TO GO WELL BECAUSE THOSE STUDY SECTION REVIEWERS ARE USED TO HYPOTHESIS-DRIVEN RESEARCH AND THAT SORT OF THING. AND THIS IS NOT WHAT WE HAVE HERE. AND PREACHING TO THOSE REVIEWERS IS GOING TO BE TOUGH. SO WE WANT TO MAKE SURE WE GET INTO A STUDY SECTION WHERE THE QIN GRANTS ARE REVIEWED ONLY IN THAT STUDY SECTION, AND THAT THEY ARE PERCENTILES AGAINST SOME PERCENTILING WHERE YOUR SCORES WILL GET CONVERTED INTO A PERCENTILE. SO, HAVING SAID THAT, I'M NOT SUGGESTING IN ANY WAY, SHAPE OR FORM THE NETWORK GOES AWAY. WE STILL HAVE PLENTY OF U01s IN THE POT. WE'RE GOING TO CONTINUE TO HAVE A NUMBER OF -- YOU'RE HOLDING U01s NOW, SO THAT WILL CONTINUE TO MOVE ON. THE UG3/UH3 WE'VE INSTITUTED STAYS, NOTHING IS HAPPENING WITH THAT AT ALL. IF YOU LOOK AT THE U01 MECHANISM RIGHT NOW, THAT IS REALLY MEANT TO BE FOR YOU PEOPLE TO PUT IN YOUR TYPE 2s. THAT'S WHAT IT WAS INTENDED FOR. SO THE PEOPLE APPLYING TO THAT ARE FOR THE MOST PART GOING TO BE YOU PEOPLE THAT ARE ALREADY A PART OF A NETWORK AND YOU PROBABLY STAY PART OF THE NETWORK, EVEN IF YOU WERE RUNNING ON R01 RATHER THAN U01. I DO HAVE TO SAY THAT PROGRAM ANNOUNCEMENTS DON'T LAST FOREVER. THEY ARE INTENDED TO BE SHOTS IN THE ARM FOR NEW TECHNOLOGIES, NEW AREAS OF INTEREST, AND THEN ONCE THOSE GET STARTED WE BACK AWAY FROM PROGRAM ANNOUNCEMENTS, AND LET YOU GUYS RUN YOUR RESEARCH THROUGH INVESTIGATOR-INITIATED R01s. THIS IS THE FIRST STEP FOR DOING THAT. SO, IT'S NOT LIKE THIS ISN'T GOING TO HAPPEN EVER, BECAUSE WE CAN'T KEEP GOING ON THIS FOREVER, BUT IT IS SOMETHING THAT WE COULD PHASE OUT IF WE DO THIS CORRECTLY, WE COULD BE PHASING OUT THE PROGRAM PARTICIPATION IN THIS EFFORT, AND PHASING YOU IN MORE FOR YOUR INVESTIGATOR-INITIATED RESEARCH. SO IT HAS SOME STRENGTHS IN THAT REGARD. THE SCHEDULE FOR THIS SORT OF THING, I HAVE NO IDEA. NOTHING HAS BEEN DECIDED YET. I'M STILL TESTING THE WATERS TO SEE WHAT'S GOING TO HAPPEN. IF IT DOES HAPPEN, A NOTICE WILL BE WRITTEN, TO TERMINATE THE CURRENT PROGRAM ANNOUNCEMENT, CURRENT U01 PROGRAM ANNOUNCEMENT, AND A NEW PROGRAM ANNOUNCEMENT WILL THEN BE PUBLISHED. I WILL SAY THAT I'VE WRITTEN A DRAFT, JUST TO BE AHEAD OF THE SCHEDULE HERE TO MAKE SURE THAT WHEN PEOPLE SAY "PUT IT IN THE MAIL" I HAVE IT READY TO GO. BUT I'M STILL WORKING ON IT. PUSHPA IS GIVING ME NEW IDEAS THIS MORNING WE MIGHT INCORPORATE TO IT TO MAKE THINGS MORE INTERESTING. AND WE HAVE TO WORK TO MAKE SURE THAT THE APPLICATIONS ARE REVIEWED, LIKE I SAID, IN A PARTICULAR CSR SPECIAL EMPHASIS PANEL, SO THAT THE PERCENTILING GOES THE RIGHT WAY FOR ALL YOU PEOPLE. SO I WANTED TO SHARE THAT WITH YOU, IN CASE YOU'RE THINKING OF WRITING -- IF YOU'RE THINKING OF WRITING AN APPLICATION FOR THE NEXT ROUND, GO RIGHT AHEAD. IT'S GOING TO BE FINE. AS WE MOVE FORWARD THOUGH AND YOU START THINKING ABOUT A YEAR DOWN THE ROAD, IF YOUR GRANT IS COMING TO AN END IN A YEAR, YOU'RE THINKING ABOUT WHEN SHOULD I BE APPLYING, THIS MAY AFFECT YOU A LITTLE BIT ON HOW YOU FORM A STRATEGY. DO YOU HAVE ANY QUESTIONS ON THAT BESIDES TOM? >> SO THE LAST BULLET, I WANT TO MAKE SURE I UNDERSTAND. SO THAT'S ASSUMING THAT YOU GO TO THE R01 MECHANISM? >> YES. >> OKAY. >> YES. YES, THAT WOULD ONLY OCCUR THEN. JANET? >> I THINK I WOULD LIKE TO EMPHASIZE THAT IF YOU OR YOU KNOW OF PEOPLE WHO ARE THINKING OF APPLYING TO U01 OR THE UG3 OR NEW R01 IF WE HAVE IT, IT'S REALLY WORTH EVERYBODY'S WHILE, OURS ESPECIALLY, AND YOURS, TO CALL UP. >> OH BOY. >> THE FOLKS AND TALK TO US ABOUT THAT STRATEGY BECAUSE WE CAN TELL YOU THE WAY THAT WE THINK THE WIND IS BLOWING RIGHT THEN AT THE TIME YOU'RE READY FOR SUBMISSION SO I WOULD ENCOURAGE YOU TO SPREAD THE WORD WE WANT TO TALK TO YOU AND HELP YOU PLAN. >> EXCELLENT POINT. EXCELLENT POINT, RIGHT, RIGHT. AND THAT'S GOOD ANYTIME, BUT PARTICULARLY IN THESE TIMES WHERE WE'RE TRYING TO FIND A PORT IN THE STORM, SO TO SPEAK, MORE IMPORTANT THAN EVER. SO LIKE I SAY, UG3/UH3 IS GOING TO BE OKAY, BUT DOES SOMEBODY HAVE A MICRO PHONE? >> UNDER THE RPG WHAT POOL? >> FOLLOWS IN THE U POOL. ALL THE THINGS THAT YOU MENTIONED. >> IT HAS SOME ADVANTAGES, WE DID THIS SORT OF ON PURPOSE. WHEN WE GO TO PRIORITIZE THESE, REMEMBER YOU'RE BEING PRIORITIZED IN UG3 POOL AGAINST PROGRAMS THAT HAVE PAY LIMITS AT 500K, UG3 IS 300K. >> YOU MENTIONED U01 STARTED, QIN BEING A BIG FRACTION, SIGNIFICANT INCREASE, HOW ABOUT UG3 MECHANISM? >> UG3 JUST STARTED. THIS IS OUR FIRST YEAR IN THAT WHOLE PROCESS. THE WHOLE CHART THAT I GAVE YOU WAS THE U AWARDS IN PARTICULAR. NOT JUST U01s. THERE'S A FEW U24s, UG3s, JUST A SMALL FRACTION OF THE TOTAL, OF THE TOTAL DCTD GRANTS. ANY-- I MEAN, RIGHT NOW THAT'S THE BEST STRATEGY WE CAN COME UP WITH. I CAN'T THINK OF ANOTHER ONE THAT WOULD -- BUTTON, BUTTON. >> WE'RE GOING TO TRY TO MAINTAIN THE NETWORK. WE'RE COMMITTED TO MAINTAINING THE NETWORK NO MATTER HOW THIS SHAKES OUT OVER THE NEXT THREE, FOUR, FIVE YEARS. THERE ARE A NUMBER OF U GRANTS THAT ARE STILL IN PROCESS, THE NEW UG3s, ET CETERA, BUT EVEN INVITING R01 MEMBERS TO PARTICIPATE IN THE NETWORK IS AN IMPORTANT ADMINISTRATIVE FUNCTION THAT WE'RE COMMITTED TO. >> RIGHT. LET ME SHARE SOMETHING THAT PUSHPA AND I TALKED ABOUT THIS MORNING. YESTERDAY? NO. A COUPLE DAYS AGO WE HEARD -- I HADN'T HEARD THIS BEFORE. THIS IS AN R01 MECHANISM, TWO TEAMS, EACH APPLY TO A PARTICULAR PROGRAM ANNOUNCEMENT. BUT THEY APPLY AS A TEAM, WE'RE GOING TO COORDINATE WITH THIS TEAM AND THAT TEAM. NOW, BOTH OF THEM HAVE TO BE AWARDED BEFORE THAT COLLABORATION TAKES PLACE. THAT'S THE DOWN SIDE OF THAT. HER IDEA IS IF WE HAVE AN R01 TEAM APPLY AS A PARTNER TO AN EXISTING U01 TEAM, OKAY, THAT COULD BE AN INCENTIVE FOR THE R01 INDIVIDUAL TO JOIN THE QIN, TO JOIN THE NETWORK, BE PART OF THE NETWORK, AND WE COULD PROBABLY WRITE INTO THAT THE GRANT APPLICATION, IF YOU DO THAT YOU WOULD HAVE TO JOIN THE NETWORK, KEEPING THE NETWORK INTACT WITH THAT STRATEGY. SOMETHING MAY BE WORTH DOING. WE'RE ALSO CONSIDERING WHETHER OR NOT WE CAN SEE THE 10% SET ASIDE FOR THE U01 CURRENTLY. THAT MAY BE DIFFICULT TO PUT IN AN R01 MECHANISM BUT MAY BE OKAY. THESE ARE ALL HURDLES LEFT FOR US TO COVER. I WANTED YOU TO KNOW WE DO SOMETHING OTHER THAN JUST SITTING AROUND HERE ALL DAY. WE DO TRY TO WORK ON YOUR BEHALF, AND MAKE THINGS HAPPEN. SO HAVING SAID THAT, I THINK THAT'S ALL I HAVE FOR THAT PARTICULAR STRATEGY. WE CAN MOVE FORWARD RIGHT NOW WITH THE ONLY PART OF THE BUSINESS MEETING THAT I REALLY WANT TO TALK ABOUT IS TO MAKE SURE THAT WE HAVE APPROPRIATE CHAIRS FOR THE WORKING GROUPS. YES. >> (INAUDIBLE). >> I KNOW. THAT'S WHY I LOOKED AT YOU WHEN I SAID IT. SO WHAT I HAVE RIGHT NOW, THE IMAGE ANALYSIS AND PERFORMANCE METRICS, THIS IS THE GROUP THAT MEETS ONCE A QUARTER. IT WAS THE PARENT OF THE MRI AND PET CT WORKING GROUPS, THESE TWO GROUPS SPLIT OFF AND MEET MONTHLY. IMAGE ANALYSIS AND PERFORMANCE METRICS WORKING GROUP ONLY MEETS QUARTERLY LORI ARLINGHOUSE WILL CHAIR THAT. SHE'S NOT HERE, WE VOTE FOR THAT. PET-CT, SANDY IS DOING PET-CT THIS YEAR. SOLO? >> (INAUDIBLE). >> OKAY, GREAT. AND THEN THE MRI, AMITA, OKAY. >> (INAUDIBLE). >> YEAH, WITH LAURA BELL, RIGHT, GOT THAT. AND BIOINFORMATICS GROUP, TOM, YOU'RE LEADING THAT. AND CLINICAL TRIALS AND DESIGN AND DEVELOPMENT, I WAS A BIT CONFUSED ABOUT THAT BECAUSE HE SAID HE WOULD CONTINUE DOING IT, BUT IF SOMEONE ELSE WOULD DO IT HE WOULDN'T DO IT, SO -- >> (INAUDIBLE). >> OKAY. OH, OKAY, FINE. THAT'S GREAT. ALSO JOHN, WERE YOU PART OF THAT? >> (INAUDIBLE). >> WELL, I ALSO HAD ELIZABETH'S NAME ON THAT. >> (INAUDIBLE). >> YEAH, I'M NOT SURE IF -- I MEAN -- >> SUSAN TOO. >> SHE HAD THIS CULTURE THIS LAST YEAR, SO -- BUT I MEAN I THINK THAT -- SHE WASN'T SURE IF SHE COULD COMMIT. >> OKAY. LET'S GET TOGETHER. I'VE GOT TO GET THE RIGHT NAMES. >> I'LL MAKE SURE. >> OKAY. I WANT TO MAKE SURE. OF COURSE WE KNOW THE DAVE I'LL CONTINUE TO MAKE SURE THAT THE COORDINATING COMMITTEE MEETS ON A MONTHLY BASIS. COORDINATING COMMITTEE, THE GUYS WE JUST TALKED ABOUT, YOU'RE NOW ON THE COORDINATING COMMITTEE, YOU KNOW THAT? OKAY. AND WE'LL SET UP A SCHEDULE FOR THAT. ANYTHING ELSE IN THE WAY OF BUSINESS, NEW BUSINESS, OLD BUSINESS, ANY KIND OF BUSINESS? YOU WANT TO GO TO PLANES AND GET HOME. DO WE HAVE ANYTHING MORE? >> YES, BOB. IT'S FRIDAY AFTERNOON. >> IT IS? >> YEAH. >> IT'S RAINING. >> YEAH. >> EVERYBODY HERE KNOWS THAT EXCEPT ED. HE STILL THINKS IT'S LAST WEEK. [LAUGHTER] >> WELL, THAT'S COOL, ED. THANK YOU VERY MUCH. I GUESS I'D LIKE TO SAY THANK YOU ALL VERY MUCH FOR COMING. ALSO WONDERFUL TO HAVE YOU ALL HERE, THANK YOU FOR STICKING AROUND TODAY. WE LOVE SEEING YOU. PLEASE KEEP UP THE CONVERSATIONS. >> YEAH, WE HAVE A LOT OF THINGS TO TALK ABOUT, EC, A LOT OF STANDING ISSUES NOW THAT WE'LL BE PUTTING ON EVERY MONTH, SO IT WILL BE INTERESTING. GREAT. THANK YOU ALL FOR BEING HERE. SAFE TRIPS HOME.