WONDERFUL TO SEE YOU HERE ON FRIDAY MORNING. THANK YOU FOR A VERY INTENSE PRODUCTIVE DAY YESTERDAY. AND GLAD WE WERE ABLE TO HAVE A LITTLE UNWINDING YESTERDAY EVENING. AS FAR AS I CAN TELL NOBODY LEFT ARTICLES OF CLOTHING AT OUR HOUSE SO I THINK EVEN MANAGED TO GET BACK HOPEFULLY TO THE HOTEL WHERE YOU WERE SUPPOSED TO BE. THAT WAS VERY PLEASANT EVENING. SOFT TODAY WE HAVE A BUNCH OF REALLY IMPORTANT THINGS TO TALK ABOUT BEGINNING WITH LARRY'S GOING TO WALK YOU THROUGH THIS OUTSIDE AWARD ISSUE THEN WE'LL HEAR FROM SUSAN AND ANDREA ABOUT OUR NIH IT ECOSYSTEM, WE'LL TAKE A BREAK, THEN HAVE THE PRESENTATION OF THE FINAL REPORT OF THE ACD WORKING GROUP ON ARTIFICIAL INTELLIGENCE FROM DAVID AND LARRY. TEN WE'LL HAVE AN UPDATE ON WHAT HAPPENED WITH FOREIGN INFLUENCES FROM MIC LAUER AND CAPPING IT OFF, JIM ANDERSON THERE WILL TALK ABOUT THE NIH WIDE STRATEGIC PLAN IN THE PROCESS OF BEING PUT TOGETHER FOR 21 TO 25. LOTS OF INTERESTING THINGS TO DISCUSS HERE. THANK YOU, EVERYBODY FOR THE WAY IN WHICH YOU HAVE APPROACHED THOSE TOPICS. WE ASK YOU TO DEAL WITH YESTERDAY AND I'M SURE YOU WILL TODAY. A COUPLE OF NOTES FROM THINGS THAT HAPPENED IN TOWN YESTERDAY, I DID MENTION YESTERDAY AFTERNOON THAT THE FDA COMMISSIONER WAS IN FACT CONFIRMED BY THE SENATE SO STEVEN HAHN IS THE NEW FDA COMMISSIONER. WE WILL LOOK FORWARD TO WORKING CLOSELY WITH HIM. ALSO ENCOURAGING NEWS ABOUT THE BUDGET THOUGH KNOW DETAILS BUT A BUDGET DEAL IS ACHIEVED BETWEEN HOUSE AND SENATE AND TREASURY SECRETARY STEVEN MINUCIAN AND THERE'S MORE INFORMATION NEXT WEEK. I CAN'T TELL YOU WHERE WE ENDED UP IN THAT BUDGET DEAL BUT WE ARE INTENSELY INTEREST IN THE INFORMATION THAT IS FORT COMING BUT IT SOUNDS THERE IS A GOOD CHANCE THIS WILL ALL GET WRAPPED UP AND PASSED AND SIGNED BY THE CURRENT DEADLINE OF DECEMBER 20th WHICH WAS GOING TO BE IMPORTANT IN ORDER NOT TO HAVE SHUT DOWN OR ANOTHER CR SO WE ARE FEELING ENCOURAGED AT THIS POINT, MORE DETAILS TO COME. SHALL WE PLUNGE IN TO THE FIRST ITEM OF BUSINESS WHICH REVIEW OF OUTSIDE AWARDS FOR ACD APPROVAL? LARRY, I THINK THAT WOULD BE YOU. >> THAT'S ME. >> WITH HELP FROM LINDA. >> ABSOLUTELY. THANK YOU, GOOD MORNING, EVERYBODY. FIRST I ALWAYS SAY THIS IS THE ABSOLUTE HIGHLIGHT OF THE MEETING FOR EVERYBODY. >> MORE THAN GROUP PHOTO? [LAUGHTER] >> ABSOLUTELY GROUP PHOTO BUT I'M TOLD IT ERODES MY CREDIBILITY SO I WON'T DO THAT. TAB 13 LISTS ALL THE AWARDS THAT ARE BEING BESTOWED UPON NIH STAFF FOR ACD REVIEW AND APPROVAL. THE REASON WE HAVE TO DO THIS AS A REMINDER, FEDERAL LAW PROHIBITS FEDERAL EMPLOYEES FOR RECEIVING ADDITIONAL MONEY DURING THE PERFORMANCE OF THEIR JOBS. SO WE HAVE TO REVIEW EVERY AWARD AND THIS IS DONE BY THE NIH ETHICS OFFICE WITH A VERY STRINGENT SET OF CRITERIA. WE THEN SUBSEQUENTLY SEND IT TO IN THIS CASE A COMMITTEE OF ONE THERE LINDA GRIFFITH WHO HELPS US. THANK YOU, LINDA. THANK YOU. SEE? AS AN ASIDE, LINDA WILL SPEND THE REST OF THE MEETING RYING TO CONVINCE ONE OF YOU TO TAKE ON HER ROLE GOING FORWARD AS SHE ROTATE OFF BUT THESE HAVE BEEN PRE-SCREENED AND DEEMED TO BE APPROPRIATE. LINDA, I DON'T KNOW IF YOU HAVE ANY SPECIFIC COMMENTS >> NO SPECIFIC COMMENTS, LARRY. THESE ARE ALL WONDERFUL AWARDS. TOTALLY MEET THE CRITERIA. >> THANK YOU. WHAT I WOULD LIKE TO BE EFFICIENT IS A MOTION TO APPROVE SECOND. ALL THOSE IN FAVOR? ANYBODY OPPOSED? >> AYE. >> OKAY. ANY ABSTENTIONS? YOU HAD NO SAY, YOU ARE GOING TO ABSTAIN? DR. FLORES HAS ABSTAINED. EVERYBODY ELSE THINKS IT'S OKAY. THANK YOU VERY MUCH. NOW I HAVE LEAVING BACK TEN MINUTES. >> OFF TO A GREAT START. LET US HEN MOVE TO THE NEXT ITEM TO HEAR INTERESTING INFORMATION ABOUT OUR IT ECOSYSTEM. ANDREA IS NOT ABLE TO JOIN US BUT SUSAN GREGURICK IS AND WILL WALK US THROUGH IMPORTANT RELEVANT INFORMATION. THIS IS OBVIOUSLY CONNECTED A BIT WITH OUR MORNING FOCUS ON WHERE WE ARE WITH IT BOTH IN TERMS OF HARDWARE AND SOFTWARE AND HOW WE WILL MAKE THE BEST OF THIS REMARKABLE MOMENT WHERE LIFE SCIENCES AND COMPUTATIONAL CAPABILITIES ARE FINDING THEIR WAY TOWARDS EACH OTHER AND VERY PRODUCTIVE PARTNERSHIPS. SUSAN, THANK YOU FOR BEING HERE. I INTRODUCE SUSAN YESTERDAY WHEN I WAS GOING THROUGH NEW APPOINTMENTS AS ASSOCIATE DIRECTOR FOR DATA SCIENCE OTHERWISE KNOWN AS THE ADS. WHICH MEANS SHE OVERSEES THE OFFICE OF DATA SCIENCE STRATEGY AND WE ARE DELIGHT TO HAVE HER IN THAT LOWELL. SHE HAS TAKEN THIS ON WITH GREAT ENERGY AND SKILL. THANK YOU FOR BEING HERE. >> THANK YOU SO MUCH, FRANCIS, IT'S A DELIGHT TO BE HERE TODAY AND TO SOME TOGETHER TO DISCUSS THE NIH DATA SCIENCE IT ECOSYSTEM. ANDREA SENDS REGARDS FROM WEEKEND SO I WILL DO DOUBLE DUTY AND PRESENT THE CLOUD SERVICE PROVIDER STRIDES PROGRAM AS WELL AS WHAT WE ARE DOING IN OUR DATA ECOSYSTEM. SO THIS IS JUST -- YES. SORRY. CAN YOU HEAR ME A LITTLE BETTER? JUST A LITTLE TOO TALL. SO THIS IS JUST ONE OF THE FIVE GOALS OF THE DATA SCIENCE STRATEGIC PLAN FOR NIH. THERE ARE FOUR OTHER GOALS. THIS STRATEGIC PLAN IS REALLY A ROADMAP AND GUIDE FOR HOW WE CAN ADVANCE DATA SCIENCE AT NIH. SO I'M DELIGHTED TO TELL YOU ABOUT OUR NIH DATA SCIENCE ECOSYSTEM. I WANT TO FOCUS OUR TALK IN THREE MAIN AREAS, THE FIRST IS I WANT TO TELL YOU WHAT YOU TOLD US FOR SOME OF YOUR GREATEST CHALLENGES FACING THIS AREA IN TERMS OF CONNECTING OUR DATA RESOURCES. THEN I WANT TO TELL YOU WHAT WE ARE DOING NOW TO MAKE IT EASIER FOR YOU TO ACCESS OUR NIH DATA RESOURCES. THAT'S GOING TO FOCUS AROUND STRIDES. THEN I WANT TO TELL YOU WHAT ELSE WE CAN DO TO GET ACCESS TO DATA AND TOOLS YOU NEED FOR YOUR RESEARCH. THAT'S GOING TO BE A VERY SMALL SLIVER OF A MUCH LARGER PROGRAM. SO FIRST HERE ARE SOME OF THE ISSUES THAT YOU TOLD US YOU HAD WHEN YOU WANTED TO DO RESEARCH THAT SPANNED MORE THAN ONE DATA RESOURCE. THIS IS THE STORY OF CHRONIC OBSTRUCTIVE PULMONARY DISEASE, WHICH IS A SIGNIFICANT CAUSE OF DEATH IN THE US. THERE ARE GENETIC DIETARY DATA AVAILABLE USED TO FURTHER UNDERSTAND THE EFFECTS OF THIS DISEASE. THERE ARE SEPARATE STUDIES CONDUCTED IN GENOMICS AND DIETARY SUBJECTS FOR CPOD. RESEARCHERS KNOW THE SAME PARTICIPANTS PARTICIPATE IN MULTIPLE STUDIES. YET IT IS INCREDIBLY DIFFICULT FOR RESEARCHERS TO GET ACCESS TO THIS SAME PATIENT DATA ACROSS MULTIPLE RESOURCES. THEY KNOW IF THEY WERE ABLE TO GET ACCESS TO THE SAME PATIENT DATA THEY WOULD HAVE MUCH BROADER BREADTH OF DATA AND UNDERSTANDING TO MAKE THOSE CONNECTIONS BETWEEN GENETIC AND DIETARY EFFECTS. SO IF WE CAN JUST HAVE RESEARCHERS BE ABLE TO FIND DATA ACROSS STUDIES WE WOULD GREATLY APPROVE YOUR CAPABILITIES. THE SECOND STORY IS OF RARE DISEASES LIKE PEDIATRIC CANCER, THERE ARE PARTICULARLY CHALLENGING BECAUSE NO ONE RESOURCE HAS ENOUGH DATA TO ALLOW FOR THE IDENTIFICATION OF CAUSAL VARIANTS ON THEIR OWN. BEING ABLE TO AGGREGATE AS MUCH DATA AS POSSIBLE IN RARE DISEASES WOULD PROVIDE A MUCH RICHER AND RESOURCE -- RESEARCHERS WHO WANT TO STUDY PATIENTS ACROSS DIFFERENT RARE DISEASES. SO CLEARLY ACCESSING DATA ACROSS MULTIPLE RESOURCES WILL ALLOW RESEARCHERS TO PULL IN THIS DATA IN RARE DISEASE YET IT IS VERY DIFFICULT FOR A RESEARCHER TO DO THIS NOW SO WE WANT TO SOLVE THAT PROBLEM. THIS IS PARTICULARLY IMPORTANT BECAUSE TIME IS OF THE ESSENCE IN RARE DISEASES PARTICULARLY FOR PEDIATRIC ONCOLOGY. MOREOVER THERE'S A LOT OF GROUPS ADVOCACY GROUPS THAT HAVE DATA RESOURCES THEY ARE STANDING UP AS WELL AND BEING ABLE TO INTEGRATE THOSE PATIENT ADVOCACY RESOURCES WITH OUR DATA WILL ENHANCE CAPABILITY FOR RESEARCH IN THIS AREA SO THIS IS DEFINITELY A GOAL. THE LAST STORY I WANT TO TELL YOU ABOUT IS CARDIOVASCULAR GENOMICS. RESEARCHERS INVESTIGATE IN GENETIC COMPONENTS OF CARDIOVASCULAR DISEASE NEED TO ENTRY GATE DATA FROM ACROSS MULTIPLE REPOSITORIES AND PLATFORMS AND HERE THIS IS THE INTEGRATION OF NHLBI TOP NET PROGRAM WITH EXTENSIVE DATA RELATED TO HEART LUNG BLOOD AND SLEEP WITH 144,000 PARTICIPANTS AND 80 STUDIES. BEING ABLE TO INTEGRATE THAT DATA WITH THE GENOMICS PLATFORM NHGRI ALLOW RESEARCHERS TO UNDERSTAND THE GENOMIC EFFECTS OF MANY DISEASES LIKE CARDIOVASCULAR DISEASE. YET IS VERY DIFFICULT RIGHT NOW IF NOT IMPOSSIBLE TO DO SO. WHAT WOULD WE ACCOMPLISH IF WE WERE TO ADJUST TO THREE USERS STORIES WHICH ARE EXEMPLIFIED ACROSS MANY RESEARCH FIELDS AT NIH? HERE ARE SOME COMMON THEMES WE HAVE HEARD FROM RESEARCHERS, RESEARCHERS KNOW THAT THE SAME SUBJECTS PARTICIPATE IN MANY STUDIES ACROSS NIH BUT IT IS EXTREMELY DIFFICULT AND TIME CONSUMING TO LINK THAN DATA. SECURELY QUERYING AND ACCESSING MULTIPLE RESOURCES IS A PREREQUISITE ALLOWING DIFFERENT DATA SETS TO BE IDENTIFIED AND USED COMBINING THOSE DATA TYPES HOUSING REPOSITORIES INCREASES POWER OF DATA ANALYSIS. THE FIRST THREE BULLETS WE CAN WORK ON NOW, THE LAST BULLET HARMONIZING DATA IS ACTIVE FIELD OF RESEARCH. THERE ARE A LOT OF THOUGHTS HOW TO DO THIS YET IF WE COULD BE ABLE TO INTEGRATE AND ANALYZE DATA TOGETHER IT WOULD BE AMAZING POWER FOR RESEARCH ONERS. -- RESEARCHERS. WHAT WOULD WE ACCOMPLISH IF WE WERE ABLE TO DO THIS? IMAGINE TAKING DATA FROM FRAMINGHAM HART STUDY AND COMBINE WITH DATA FROM ALZHEIMER AND AGING TO UNDERSTAND THE CORE LAY ACTIVE EFFECTS OF CARDIO VASCULAR HEALTH WITH AGING AND DEMENTIA. THIS IS THE HOLY GRAIL THAT WE ARE GOING FOR. NOW I WANT TO TELL YOU STEPS WE ARE TAKING ALONG THE WAY TO AGGREGATE AND INTEGRATE ESOURCES AND DATA ACROSS OUR PLATFORMS. THE FIRST IS LEVERAGING POWER OF CLOUD FOR BIOMEDICAL RESEARCH. AS YOU KNOW, CLOUD COMPUTING OFFERS MULTIPLE OPPORTUNITIES THAT NIH CAN LEVERAGE INCLUDING THE ABILITY TO COMPUTE ON DATA UNPRECEDENTED SCALE. BEING ABLE TO ACCESS CUTTING EDGE TECHNOLOGIES FOR EXAMPLE LEVERAGING SECURITY TOOLS LIKE THE CLOUD SERVICE PROVIDERS CAN PROVIDE FOR US REALLY UP OUR GAME IN DATA SCIENCE. CLOUD STORAGE FOR LARGE DATA ENABLES US TO EASILY ACCESS SHARE AND REUSE DATA FROM OTHER RESEARCHERS, CLOUD COMPUTING PROVIDES A COMMUNITY DRIVEN APPROACH WHERE DATA SCIENCE BRINGS DOWN SILO OF DISCIPLINARY BOUNDARIES AND ALLOW RESEARCHERS TO WORK TOGETHER TO DEVELOP AND ADOPT TOOLS FROM INDUSTRY OR ACADEMIA UTILIZING GET HUB OR GALAXY. SO HARNESSING THE POWER OF CLOUD FOR DATA STORAGE, DATA COMPUTING AND DATA ACCESS AND SHARING. SO THIS IS WHAT WE ARE DOING IN STRIDES. STRIDES IS THE SCIENCE AND TECHNOLOGY RESEARCH INFRASTRUCTURE FOR DISCOVERY, EXPERIMENTATION AND SUSTAINABILITY. ENABLES INNOVATIVE PARTNERSHIPS WITH CLOUD SERVICE PROVIDERS AND OFFERS NIH FUNDED INSTITUTIONS DISCOUNTS ON CLOUD STORAGE AND COMPUTE AND RELATED CLOUD SERVICES SO FAR WE HAVE SEEN A SAVINGS OF ALMOST $5 MILLION BY LEVERAGING OUR CLOUD SERVICE PROVIDERS GOOGLE P. BY WORKING WITH CLOUD SERVICE PROVIDERS WE ARE ABLE TO ACCESS PROFESSIONAL TECHNICAL SERVICES. FOR EXAMPLE ONE OF OUR HIGH THROUGH PUT PIPELINES WAS ABLE TO SEE A TENFOLD INCREASE IN PRODUCTIVITY JUST BY HARDENING THEIR CODE AND OPTIMIZING IT FOR CLOUD COMPUTING. WE ALSO SEE DISCOUNTS IN PERSONAL AND ONLINE TRAINING, WE HAVE HA A NUMBER OF TRAININGS HERE AT NIH AND NOW EXPANDING THOSE TRAINING OPPORTUNITIES TO BE AT INSTITUTIONS, REGIONAL AND AT NIH BECAUSE WE ARE ABLE TO TRAIN AND WORK WITH RESEARCHERS DATA OWNERS AND OTHERS INTERESTED IN CLOUD COMPUTING. THERE ARE NEW OPPORTUNITIES TO EXPLORE NEW METHODS AND APPROACHES THAT MIGHT ADVANCE BIOMEDICAL RESEARCH THROUGH DIFFERENT COLLABORATIONS. FINALLY WE HAVE ACCESS TO NEW EMERGING TECHNOLOGIES ANDS MACHINE LEARNING AND ARTIFICIAL INTELLIGENCE FOR AGGREGATING WORKING AND PREPARING DATA. TODAY WE ARE WORKING WITH A NUMBER OF TARGETED INSTITUTIONS, IN ORDER TO PILOT OUR APPROACH OF ON BOARDING INSTITUTIONS AND NIH FUNDED RESEARCHERS TO WORK IN THE CLOUD. WE ARE SEEKING TO STREAMLINE AUTOMATE THE ON BOARDING PROCESS. NEW INSTITUTIONS AND RESEARCHERS WILL CONTINUE TO HANDLE YOUR DAY TO DAY OWNERSHIP AND MANAGEMENT OF RESEARCH DATA. THIS PAST YEAR STRIDES STARTED IN 2018. THIS PAST YEAR WE HAVE MOVED OVER 30 PETABYTES OF DATA TO GOOGLE AND AMAZON INCLUDING THE NIH FRAMINGHAM HART STUDY I MENTIONED EARLIER, THE ALL OF US RESEARCH PROGRAM, THE NCI CANCER RESEARCH DATA COMMONS, WE HAVE MOVED 12 PEDABYTES OF SEQUENCE DATA FROM NCBI TO GOOGLE AND AMAZON, THAT IS HUGE. WE HAVE THE TOP MED PRECISION MEDICINE PROGRAM AND THE GABRIELLE MULLER KIDS FIRST PROGRAM IN GOOGLE AMAZON AND NHGRI. THIS COULD POSSIBLY REPRESENT THE LARGEST AMOUNT OF BIOMEDICAL DATA AVAILABLE FOR RESEARCH. YET STILL HARD TO AGGREGATE ACCESS ACROSS PLATFORMS. NEXT YEAR WE ANTICIPATE 50 PETABYTES OF DATA AND I SUSPECT WHEN THE ALL OF US PROGRAM IS REACHING THE FULL STRIDE WE COULD ACTUALLY SEE ZETA BYTES OF DATA IN THE CLOUD PARTICULARLY IF WE START GEE INTEGRATING THINGS FROM SENSOR DATA. THAT'S UNPRECEDENTED SCALE OF DATA, THERE AREN'T COMPUTATIONAL ALGORITHMS TODAY THAT CAN WORK AT THIS SCALE SO WE HAVE GRAND CHALLENGES AHEAD OF US. I REALLY LOOK FORWARD TO THAT. WHAT CAN WE DO NEXT IN STRIDES TO REALLY EXPAND THE IMPACT AND ACCESS OF CLOUD SERVICE PROVIDER COMPUTING. WE STREAMLESSLY WORK WITH OUR INSTITUTIONS AND OUR UNIVERSITIES, TO LOWER THE COST FOR NIH FUNDED RESEARCHERS. WE ARE INCREASING TRAINING AND CLOUD COMPUTING SO WE CAN INCREASE YOUR SKILL SET. WE ARE ADDING ADDITIONAL STRIDES PARTNERS TO GREATLY ENHANCE THE ANALYTIC AND DATA MANAGEMENT CAPABILITIES. I WILL TALK MORE ABOUT THIS AT THE END. AND WE ARE INSTANCEIATING COMMON PROTECTIONS AND STANDARDS O TO HAVE GREAT ASSURANCE IN CONFIDENTIALITY, INTEGRITY AND AVAILABILITY OF THE DATA. SO AS I MENTIONED WE HAVE MOVED A NUMBER OF OUR NIH DATA PLATFORMS TO THE CLOUD. THIS INCLUDES THE CANCER COMMONS RESEARCH FRAMEWORK, THE NATIONAL DATA ARCHIVE, THE GENOMIC ANALYSIS FOR VISUALIZATION AND LAB SPACE ANVIL, ALL OF US PROGRAM, THIS IS ACTUALLY MY DASHBOARD, I'M ALSO A PARTICIPANT. THE KIDS FIRST GABRIEL MILLER PLATFORM AND NEW DATA BIOCATALYST. ALL THESE PLATFORMS ARE RICH AND PROVIDE RESEARCHERS WITH A NUMBER OF GREAT OPPORTUNITIES INCLUDING PORTALS THAT ARE INTUITIVE FOR DATA COMPUTATION AND ANALYTICAL RESOURCES. THEY PROVIDE ACCESS TO HIGH VALUE BIOMEDICAL DATA SETS THAT SPAN MULTIPLE DOMAINS IN MANY DISEASE AREAS. THEY PROVIDE A RICH SET OF COMPUTATIONAL RESOURCES ANDS TOOLS TO EXPLORE, TO ANALYZE AND VISUALIZE DATA. MANY PLATFORMS PROVIDE INDIVIDUAL AND GROUP WORK SPACES THAT ENABLE RESEARCHERS TO WORK TOGETHER TO ACCESS DATA CREATE NEW EXPERIMENTS CONDUCT ANALYSIS AND STORE RESULTS. THEY HAVE APPLICATIONS THAT REMAIN SAFE SECURE AND PRIVATE SO THE DATA PLATFORMS ARE RICH YET I CAN'T GET TO THAT END GOAL OF CONNECTING CARDIOVASCULAR DATA WITH AGING DATA. RIGHT NOW THAT'S THE PROBLEM THT I REALLY WANT TO SOLVE, I WANT TO PROVIDE FOR YOU OUR RESEARCH COMMUNITY, A GREATER DATA PLATFORM THAT LEVERAGES ALL THE EXCELLENT CAPABILITIES THAT WERE ALREADY STANDING UP AND ENHANCES THOSE. SO HOW CAN WE DO THAT? HERE IS JUST SOME IDEAS. THAT WERE WORKING WITH TODAY, AT LEAST THE FIRST COUPLE OF BULLETS. TO CREATE GREATER INTEROPERABILITY BETWEEN OUR CLOUD SUPPORTED HIGH VALUE DATA PLATFORMS. WE CAN STANDARDIZE THE USER EXPERIENCE SO THE LOOK AND FEEL OF EACH PLATFORM IS SIMILAR SO THAT YOU ARE NOT CONFUSED. WE KNOW THAT YOU LIKE THE PLATFORMS THAT YOU USE, AND THAT YOU WANT TO HAVE GREATER CAPABILITIES ACROSS OUR PLATFORMS. WE CAN PROVIDE UNIFORM EFFICIENT DATA ACCESS TO GET THE DATA QUICKER AND EASIER. THERE ARE CHALLENGES. WE NEED TO RESOLVE PATIENT AND DATA IDENTIFIERS SO THAT WHEN YOU SAY YOU WANT PATIENT X ACROSS THE PLATFORMS THAT'S THE DATA THAT YOU GET. SO FINDING ALL DATA RELEVANT TO A PATIENT STUDY DATA COLLECTION IS VERY IMPORTANT. BEING ABLE TO SEARCH AND EXPLORE DATA ACROSS OUR PLATFORMS AGGREGATING THAT IS A CHALLENGE. IT MEANS METADATA HARMONIZATION, THERE'S NEW WAYS MAPPING THAT AND MAPPING ONTOLOGIES METADATA WILL BE VERY IMPORTANT. FINALLY, WORK FLOWS ACROSS OUR PLATFORMS AND ACROSS OUR SERVICE PROVIDERS IS SOMEWHAT A HOLY GRAIL CERTAINLY FOR ME AS A COMPUTATIONAL PERSON, AND I JUST WANT TO GIVE YOU ONE HIGHLIGHT THAT THIS IS POSSIBLE AND HAPPENING. LAST MONTH THE NSF FUNDED ICE CUBE PROJECT WHICH STUDIES MASSIVE DATA ANALYSIS AND NUTRINOS WAS BASICALLY ABLE TO COMMANDEER AZURE MICROSOFT GOOGLE AND AWS FOR TWO HOUR COMPUTE THAT TIED UP ALL THEIR SYSTEMS. IT WAS ESSENTIALLY A DATA COMPUTE ACROSS ALL CLOUD SERVICE PROVIDERS, FOR TWO HOURS THAT RESULTED IN ABOUT 40 YEARS OF DATA ANALYTICS. I THINK IF WE CAN HARNESS THAT KIND OF POWER, WE WOULD REALLY REVOLUTIONIZE DATA SCIENCE AND THE WORK YOU ARE DOING. SO THE FIRST STEP, THERE ALWAYS A FIRST STEP AND BEGINNING, THIS IS IT. THIS IS THE IT PART OF THE TALK SO HOPEFULLY INTERESTING. IT'S REALLY PROVIDING YOU AN EASY WAY TO SIGN ON AND GET ACT TO OUR PLATFORMS, BASICALLY SINGLE SIGN ON. AD STREAMLINE LOG IN AND AUTHORIZATION FOR CONTROL ACCESS DATA. WE WILL BE MAKING USE OF INDUSTRY STANDARD TECHNOLOGY IN THIS CASE AN AFICIONADO, WE ARE REQUIRING MULTI-FACTORIAL AUTHENTICATION FOR SECURITY. HERE IS THE MANDATE FROM MY OFFICE. WE HAVE A NUMBER OF RICH DATA PLATFORMS AND DATA STAGE CANCER COMMON RESEARCH FRAMEWORK. THEY HAVE FULL STACK, THEY HAVE BEEN ENGINEERED. WE DON'T WANT TO HAVE PEOPLE SPENDING MANY HOURS AND TIME RE-ENGINEERING THEIR ALREADY EXISTING PLATFORMS, THEY HAVE USERS THEY HAVE RESEARCH E SO WHAT WE ARE DOING IS DEVELOPING A VERY FLEXIBLE LAYER ON TOP OF THESE PLATFORMS TO ENABLE SINGLE SIGN ON AND OTHER CAPABILITIES. WE ALSO KNOW THERE IS PLATFORMS HETEROGENEOUS, THEY ARE NOT THE SAME, THEY HAVE DIFFERENT RESEARCHERS DIFFERENT DEVELOPERS SO WE NEED TO BE FLEXIBLE TO ALSO ADD NEW PLATFORMS IN THE FUTURE SO THIS IS THE FIRST STEP. HERE IS HIGH LEVEL WORK FLOW. I WANT TO CALL OUT ONE THING IN PARTICULAR. THAT IS THAT WE HAVE DIFFERENT COMMUNITIES. NIH INVESTIGATORS. WE HAVE AN INTRAMURAL RESEARCH INVESTIGATOR COMMUNITY. EXTRAMURAL RESEARCHERS AND WE ARE SERVING BOTH. THROUGH OUR INTRAMURAL RESEARCHERS YOU WILL BE ABLE TO LOG IN WITH YOUR NIH CREDENTIALS. TO THE EXTRAMURAL RESEARCHERS THERE IS A NUMBER OF WAYS WHICH YOU CAN LOG TO OUR SYSTEMS. ALL WHICH WILL REQUIRE MULTI-FACTOR AUTHENTICATION. HERE WHAT WE ARE DOING IS AGGREGATING YOUR INFORMATION ACROSS MULTIPLE PLATFORMS SO IF YOU HAVE ACCOUNTS THEY ARE LINKED TOGETHER. THIS IS A FEDERATED BROKER SYSTEM. THAT WILL PROVIDE A UNIFIED EFFICIENT AND SECURE AUTHENTICATION AND AUDITING SYSTEM THAT IS IMPORTANT SO YOU HAVE ACCESS TO THE DATA SETS AND TOOLS ACROSS THE SYSTEMS THAT YOU ALREADY ARE ALLOWED TO SEE. THIS IS CALLED RESEARCH AUTHENTICATION SERVICE. WE HAVE A NUMBER OF LEADING PARTNERS. I WANT TO TELL YOU ABOUT THE FOUR CAMPAIGNS THAT WE ARE DEVELOPING, FIRST IS IN PEDIATRIC ONCOLOGY, REALLY INTEGRATING ACROSS THE NHLBI DATA STAGE INCLUDES PROJECT AND THE KIDS FIRST GABRIELLE MILLER PLATFORM SO YOU CAN GET ACCESS TO THE DATA IN PEDIATRIC CANCER ACROSS OUR DIFFERENT PLATFORMS. THE SECOND IS INTEGRATING ACROSS CANCER COMMONS RESEARCH PROTEOMICS AND GENOMICS FRAMEWORK WITH NHGRI G TEXT FRAMEWORK AS WELL AS BIODATA CATALYST TO GET ACCESS TO DIFFERENT DATA TYPES SIMILAR ACROSS THESE SYSTEMS. THE THIRD ONE IS REALLY ONE ABOUT INTEGRATING PLATFORMS IN A CONSORTIUM SO WE STAND UP A NUMBER OF HIGH VALUE PLATFORMS, INCLUDING BRAIN AND HEAL, ALL THOSE INITIATIVES HAVE DIFFERENT DATA COORDINATING CENTERS. WHAT WE WANT TO DO IS PROVIDE CAPABILITY TO INTEGRATE ACROSS COORDINATING CENTERS AND TO DO SO WE NEED TO HAVE A PILOT. SO WHAT WE WILL BE DOING IS INTEGRATE ALL DIFFERENT NODES AND CANCER COMMONS RESEARCH FRAMEWORK SO THAT WE CAN PROVIDE THAT CAPABILITY TO OTHER INITIATIVES. FINALLY WE WILL BE WORKING WITH DB GAP TO IMPROVE EFFICIENCY ACROSS OUR SYSTEMS WITH DB GAP, THOSE ARE THE FIRST PARTNERS FIRST USE CASES FOR WHAT WE WILL BE DEVELOPING. SO AGAIN TO RECAP LOGGING INTO THE ANALYTICAL PLATFORM IN A SEAMLESS WAY TO ANALYZE DATA STORED ACROSS MULTIPLE RESOURCES. LOGGING ON WITH ID CREDENTIALS OR ERA COMMONS AND VERY IMPORTANTLY, DEVELOPING STREAMLINE PROCESS FOR AUDITS OF LOGGING FOR DATA WITH ANY GIVEN DATA SET SO WE CAN RAPIDLY RESPOND TO ANY DATA MANAGEMENT INSTANCES SO LOG ON CREDENTIALING AND SECURITY AND PROMINENCE OF WHAT IS HAPPENING IN CASE THERE IS A DATA MANAGEMENT RESPONSE NEEDED. I WOULD LIKE TO TELL YOU ABOUT OUR FIRST MILESTONE. SO THIS IS THE INTEGRATION WITHIN OUR SYSTEM. IT ALLOWS RESEARCHERS TO NOW LOG IN WITH OUR ERA COMMONS ACROSS THEIR DATA RAINS FER SERVICES BY USING OPEN ID CONNECT, IT'S A MODERN WAY TO EXCHANGE INFORMATION BETWEEN SYSTEMS. THIS PRESENTS A FUNDAMENTAL PART OF WHAT WE WILL BE DOING BECAUSE WE ARE ABLE TO EXTEND OUR LOG INS ACROSS GOOGLE, ACROSS YOUR UNIVERSITY IDs, ACROSS ERA COMMONS SO BASICALLY AS LONG AS YOU ARE USING MULTI-FACTORAL AUTHENTICATION, YOU SHOULD BE ABLE TO GET ACCESS ACROSS OUR SYSTEMS THAT'S THE GOAL. SOL FOR 2020 WE WILL BE ADDING ADDITIONAL PARTNERS TO THE STRIDES PROGRAM. THESE PARTNERS WILL BE DATA PLATFORM SERVICE PROVIDERS TO ALLOW YOU TO HAVE ADDITIONAL ANALYTICAL CAPABILITIES AND DATA SERVICES. WE WILL BE ENHANCING THE CLOUD BASED WORK FLOWS ANALYSIS SO RESEARCH ERGS ARE ABLE TO DEVELOP TOOLS AND SOFTWARE ON THE CLOUD. WE WILL BE IMPROVING OUR RESEARCHER EXPERIENCE WITH EASIER ACCESS TO NIH DATA RESOURCE, IMPROVING DATA ACCESS BY STREAMLINING SOME OF THE ACTIVITIES IN DB GAP AND IMPROVING FEDERATED DATA SEARCH. WE WILL BE MAINTAINING SECURITY AND ASSURANCES THROUGH STANDARDIZE AUDITS AND LOG TRACES AND OF COURSE WE WORK WITH COMMUNITIES TO ADOPT AND ADHERE TO COMMUNITY STANDARDS LIKE (INDISCERNIBLE) SO THESE ARE SOME OF THE GOALS FOR 2020. FINALLY I JUST WANT TO GIVE THANKS TO A NUMBER OF MY COLLEAGUES WHO WORKED ON THIS ANDREA NORRIS IN THIS CASE WEBER AND TODD RILEY ALL STRIDES TEAM WORKING HARD THE ON BOARD NEW INSTITUTIONS FOR CLOUD SERVICE PROVIDERS. OUR RESEARCH AWE HENTCATION SERVICES IN PARTICULAR JEFF ERICSON AND REBECCA ROSEN LEADING OUR TEAM BUT THERE'S ADDITIONAL PARTNERS FOR DATA RESOURCE INTEROPERABILITY. ALICE THOMPSON, OTHERS WORKING ON SECURITY. THANKS TO YOU FOR THE TIME, THERE'S PROBABLY PLENTY OF TIME FOR QUESTION AND ANSWER, HAPPY TO ADDRESS ANY QUESTIONS YOU MIGHT HAVE. >> THANKS VERY MUCH SUSAN. LARRY, SOME OTHER CONTEXT, COMMENTS. >> AS I THINK BACK TO MY DAYS IN THE UNIVERSITY AS RESEARCH DEAN, EVERY TIME I WENT TO A BOARD PRESENTATION I WANT TO TALK ABOUT INFRASTRUCTURE, EVERYBODY'S HEADS WENT DOWN ON TABLE BECAUSE INFRASTRUCTURE IS BORING. BUT IT'S WHAT MAKES THE WORLD GO ROUND. SO I HOPE YOU HAVE GAINED AN APPRECIATION FOR ONE SMALL PIECE BECAUSE IT'S NOT THE WHOLE PICTURE. OF SOME OF THE ATTEMPTS BEING MADE TO AGRADE MODERNIZE AND ENHANCE THE FUNDAMENTAL INFRASTRUCTURE OF THE ORGANIZATION SO ALL OF YOU CAN DO YOUR JOBS BETTER. THIS IS REALLY AN OPPORTUNITY, SOME OF YOU ARE POWER USERS. SOME TELL YOU THERE ARE ALGORITHMS TO DEAL WITH A BILLION PEDABYTE BUT OTHERS ARE LIKE ME, YOU JUST WANT THE DAMN THING TO WORK AND DON'T CARE HOW IT WORKS. THIS IS AN OPPORTUNITY TO GIVE NIH FEEDBACK ABOUT WHEREVER YOU ARE IN THE FOOD CHAIN. THE SUPER POWER USERS, TO THE -- FOR THE REST OF US WHO THINK IT'S A MIRACLE THE COMPUTER WORKS EVERY DAY. AND EVERYTHING IN BETWEEN. IN OTHER WORDS FOR US TO DO THE BEST JOB POSSIBLE AND FOR US TO ACTUALIZE USE CASES WHICH ARE I ILLUSTRATIVE BUT THERE ARE OTHERS YOU CAN -- UPON, WE NEED TO KNOW WHAT ARE THE CHALLENGES YOU ARE ALL FACING AND WHATEVER SPACE YOU WORK IN. THERE IS ENOUGH DIVERSITY OF EFFORT AROUND THE TABLE THAT UNIVERSITY CHANCELLORS OR PRESIDENTS YOU HAVE ONE LEVEL OF CONCERN AND PEOPLE WHO ARE STILL AT THE BENCH HAVE ANOTHER LEVEL OF CONCERN. ET CETERA. SO I HOPE WE TAKE ADVANTAGE OF THIS TIME FRAME TO GIVE US THE FEEDBACK WE NEED AND IF NECESSARY CHALLENGE US ABOUT WHAT WE ARE NOT DOING OPTIMALLY. ALL COMPLAINTS TO ME, ALL CREDIT TO SUSAN THAT. TOES WAY THIS WORKS LET'S START THERE AND WORK AROUND. >> GREAT TALK, REALLY EXCITING TO THINK ABOUT POSSIBLY PEOPLE BEING ABLE TO ACCESS THE DATA MUCH MORE FREELY. I HAD TWO QUESTIONS. FIRST, I WAS HOPING YOU CAN COMMENT ON METADATA GENERALLY AND WHERE YOU THINK WE ARE IN TERMS OF ENCOURAGING RESEARCHERS TO COLLECT APPROPRIATE METADATA AND STORE IN A WAY OTHERS CAN ACCESS. MY OTHER QUESTION WAS A TECHNICAL ONE ABOUT GLOBIS. WONDERING IF YOU CAN TALK MORE ABOUT THAT AND I THINK YOUR SLIDE SAID MAYBE CURRENTLY PEOPLE CAN LOG ON WITH THEIR COMMONS ACCOUNT. YOU CAN SAY IF THAT'S CORRECT AND HOW PEOPLE ARE USING THAT AT THE MOMENT. >> THE SECOND ONE FIRST. YES. WE HAVE INTEGRATED THAT SO YOU CAN LOG IN TO GLOBIS WITH YOUR ERA COMMONS ACCOUNT. THAT IS POSSIBLE. WE HAVE DO HARD WORK CONNECTING OUR RESOURCES THROUGH RAS BUT YOU CAN LOG INTO WITH YOUR ERA COMMONS ACCOUNT THAT'S ACCOMPLISHED SO THAT IS NOW A PART OF OUR INFRASTRUCTURE. SO THE SECOND ONE ABOUT METADATA EACH COMMUNITY HAS ITS OWN IDEA ABOUT METADATA AND THE STANDARDS THEY USE AND THE WAY THEY COMMUNICATE THE INFORMATION ABOUT THEIR DATA. THERE IS NO ONE SIZE FITS ALL FOR SURE BUT WHAT I WOULD SAY IS IF WE WANT TO THINK ABOUT DATA INTEGRATION AND DATA HARMONIZATION WE HAVE TO THINK ABOUT COMPUTABLE METADATA. A GOOD EXAMPLE IS WHAT'S HAPPENING IN THE FIRE COMMUNITY. FAST HEALTHCARE INTEROPERABLE RESOURCES WHERE WE ARE LOOKING AT INTEGRATING ACROSS HEALTH IT SYSTEMS AND PAYER PROVIDERS. BY MAKING THAT SORT OF METADATA COMMUTABLE AND RESOURCES. SO IT'S JUST LIKE IF YOU ARE DOING A GOOGLE SEARCH AND YOU SEE THE LET ME FIND FIRE, YOU WILL SEE THE TAG ON TOP THAT HAS INFORMATION AND CHARACTERS. IT'S LIKE THAT. WHERE COMPUTABLE INFORMATION ABOUT THE HEALTH IT DATA. THERE IS GOOD LESSONS TO BE LEARNED ABOUT COMPUTABLE INFORMATION METADATA SO WE CAN SEARCH ACROSS METADATA, COMPUTE ON IT AND UNDERSTAND WHERE WE CAN INTEGRATE AND WHERE WE CAN. THERE IS INTERESTING WORK GOING ON NOW IN SEMANTIC TECHNOLOGIES AND AI INTEGRATING ACROSS DIFFERENT RESOURCES, THIS IS HAPPENING HERE AT NIH, THE DATA TRANSLATOR PROGRAM. ALSO HAPPENING IN THE EU WHERE THEY ARE ALSO LOOKING AT THIS STANDING OF DIFFERENT INTEGRATED RESOURCES IMAGING AS WELL AS OTHER RESOURCES MORE IN THE GENOMICS AND PROTEIN DATABASE SCIENCE. THAT'S MY QUICK RESPONSE METADATA COMPUTABLE AND ABLE TO MAP ONTOLOGIES ACROSS METADATA TYPES. MAYBE THERE'S A LOT TO BE LEARNED IN THE FIRE COMMUNITY THINKING ABOUT THIS IN DIFFERENT WAYS. DOES THAT ADDRESS YOUR QUESTION? GREAT. >> DINA. >> THANKS SUSAN, THIS IS AMAZING. I'M HAPPY TO SEE THAT NIA HAS TAKEN THE STEP OF WHERE THEY USE THE CLOUD SYSTEM IN THE MOST EFFICIENT WAY. TWO QUESTIONS. ONE ABOUT THE WORK PLEA AND OTHER ABOUT THE DATA ITSELF. IN TERMS OF WORK FLOW BASED ON WHAT YOU SAID, S SAY FOR EXAMPLE I'M A RESEARCHER I WANT TO USE THE NIH CLOUD SYSTEM. CAN I CHOOSE -- SAY I WANT TO WORK ON THE COPD DATA IT IS MY CHOICE TO USE AWS VERSUS AINSURE VERSUS GOOGLE CLOUD? I CAN USE ANY ONE OF THESE PLATFORMS OR I HAVE TO SOMETHING SPECIFIC? >> GREAT QUESTION. STRIDE IS CLOUD AGNOSTIC, PARTNERS ARE GOOGLE AND AWS AND YOU CAN CHOOSE WHICHEVER YOU WANT, WE HAVE TO BUILD CAPABILITIES TO COMPUTE ACROSS THE DATA AND IF YOU CHOOSE GOOGLE IN YOUR DATA SET HAPPENS TO BE AWS OR VICE VERSA. SOME CASES WE DUPLICATE DATA FOR BOTH THAT IS THE CASE FOR SRA DATA IN GOOGLE AND AWS SO SIX PEDABYTES IN GOOGLE AND AWS. OTHER CASES PLATFORMS CHOSEN A PARTICULAR CLOUD SERVICE PROVIDER SO WE WILL BE NEEDING TO WORK IN A LITTLE BIT MORE -- YOU WILL SEE IT INFRASTRUCTURE TALKS I'M SURE HOW WE CAN ACTUALLY ENABLE THAT YOU CHOOSE GOOGLE Z THE PRIMARY SOURCE YET INTERESTING DATA IS AWS SO WE DO HAVE SOME WORK TO DO. >> IN TERMS OF THE MODELS, WHAT KIND OF RESOURCES, SAY MY MODELS ARE REALLY BIG AND I WANT HUNDRED GPUs AND DO YOU HAVE LIMITATIONS, CAN THE INSTITUTE JUMP IN AND PAY FOR ADDITIONAL COSTS IF THE MODELS ARE COMPLEX AND REQUIRES RESOURCE? >> YOU CAN CHOOSE AS MANY RESOURCES YOU FEEL IMPORTANT FOR YOUR RESEARCH. THE ONUS IS THROUGH THE GRANTING SYSTEM. TO PROVIDE THE FUNDS. IN SOME CASES, PLATFORMS ARE PROVIDING RESOURCES TO THEIR USERS AND I KNOW THAT'S THE CASE PARTICULARLY FOR -- THAT'S ALSO HAPPENING IN THE DATA STATE BUT THERE'S PROBABLY SOME PRIORITY CAPS AND OTHER ACTIVITIES. >> THE RESOURCES ARE UNBOUNDED? >> AS FAR AS I KNOW THEY ARE OPEN. SOMEBODY -- Q. I TALK ABOUT THE ICE CUBE PROJECT. HE TOLD ME A LOT BUT NOT THE COST WHICH WOULD BE INTERESTING. Q. MY OTHER QUESTION IS ABOUT THE DATA ITSELF. SO YOU CHOOSE SOME DATA SETS TO START WITH. I'M INTERESTED IN HEARING WHAT GUIDED THAT CHOICE, IN PARTICULAR THE REASON I'M ASKING THIS QUESTION IS THE CHOICE OF DATA IS VERY IMPORTANT BECAUSE LIKE IF YOU LOOK AT HOW DATA HAS FUELED DEVELOPMENT OF MACHINE LEARNING MODELS IN FIELDS LIKE COMPUTER VISION, OR NLP, IT'S REALLY IMPORTANT TO HAVE DATA SET THAT ALLOWS FOR DEVELOPMENT OF MANY POTENTIAL MODEL ADDRESSING MULTIPLE DIFFERENT THE DATA SET YOU MENTIONED SEEMS MORE LIKE FOCUS ON VERY SPECIFIC APPLICATIONS? IF THE DATA HUNDRED THOUSAND OF MEDICAL RECORDS, THERE'S SO MANY THINGS, MANY MEDICAL -- SO MANY THINGS PEOPLE CAN THINK ABOUT HOW TO USE THAT DATA. HERE THEY ARE FOCUSED ON SPECIFIC PROBLEMS. CAN YOU ELABORATE ABOUT CHOICE OF DATA IN WHAT TO PUT THERE? >> ABSOLUTELY. WHAT I CAN SAY IS THAT THIS IS LIKE AN ONION WHERE THERE'S DIFFERENT LAYERS AND WE ARE AT THE TOP LAYER, WE CHOSE DATA STS AND DATA PLATFORMS THAT WERE ALREADY ON THE CLOUD VERY SAVVY CLOUD RESEARCHERS WITH A FAIRLY ROBUST COMMUNITY WORKED ON THESE PLATFORMS SO WE COULD DO WOULD BE TO ENSURE THE WORK WE ARE DEVELOPING IN RESEARCH AUTHENTICATION WOULD NOT CAUSE ANY HARM TO THOSE PLATFORMS, FIRST OF ALL BECAUSE IT IS A PILOT AND ENABLE RESEARCH TO THE PLATFORMS, THEY ARE ALSO PROVIDING. TO YOUR POINT THERE ARE INTERESTING PROGRAMS AND YOU WILL SEE ALL THIS IS UP HERE ON THE SLIDE BUT I HAVEN'T MENTIONED IT, THAT'S CERTAINLY GOING TO BE INCREDIBLY RICH DATA SET FOR MY PARTICIPANTS AND RESEARCHERS THAT WILL PROBABLY HAVE ATTRIBUTES YOU TALKED ABOUT. THAT'S PHASE 2 INTEGRATION. THEY HAVE TO REACH THEIR GOAL OF DEVELOPING THEIR WORKBENCH. WE WILL INTEGRATE IN PHASE 2 WHICH IS PROBABLY SOMETIME IN EITHER LATE 2020 OR EARLY 2021. ONCE THEY HAVE ESTABLISHED ROBUST PLATFORM THEY ARE ABLE TO WORK WITH US FOR THE AUTHENTICATION SERVICES. I WANT TO SAY EXTENDING OUT TO SOME OF THOSE DATA SETS AND PLATFORMS THAT DO ADD VALUE WOULD BE AN IMPORTANT COME POINTED FOR THE FUTURE. ABSOLUTELY. >> I UNDERSTAND DR. BRENTON HAS A QUESTION ON THE FOB. ARE YOU THERE? >> I AM, THANK YOU VERY MUCH, DR. COLLINS. SUSAN, PHENOMENAL PRESENTATION. THANK YOU SO MUCH AND HANKS TO YOUR TEAM FOR THE VERY HARD WORK THAT YOU ALL HAVE DONE. A QUICK QUESTION THERE IS A SUBSTANTIAL AMOUNT OF DATA GENERATED THROUGH ACCELERATED MEDICINES PARTNERSHIP SPEAKING SPECIFICALLY ABOUT ALZHEIMER'S SPACE. IS THERE A PLAN TO INTEGRATE OF THESE DEEP DATA SETS BENEFIT GENERATED AT THIS POINT THROUGH THAT PLATFORM? >> ABSOLUTELY. THAT'S PHASE 2. YOU WILL SEE SOME THINGS ARE LISTED HERE LIKE THE NIMH MENTAL DATA HEALTH ARCHIVE AND OTHER PLATFORMS THAT WE NEED TO INSTANCEIATE PLATFORMS ON THE CLOUD AND DEVELOP MORE SOFTWARE ON THEIR SIDE. SO SOME OF THESE PLATFORMS LIKE THE ALZHEIMER PLATFORM, WILL BE MORE PHASE 2. AGAIN, THIS IS JUST A FIRST STEP BUT NOT THE END. THERE'S QUITE A BIT MORE WORK THAT WE NEED TO DO IN ORDER TO REALIZE THAT VISION OF TRULY BEING ABLE TO ALLOW RESEARCHERS TO WORK ACROSS OUR MANY DATA PLATFORMS AND MANY DATA SETS BECAUSE WE BELIEVE AND YOU WILL TELL US IF WE ARE WRONG, IF YOU HAVE THE ABILITY TO HAVE ACCESS TO A VERY LARGE AMOUNT OF DATA THAT IS DIVERSE, YOU CAN ADVANCE RESEARCH GOALS. >> THANKS FOR RAISING THAT. THAT IS NOT JUST CASE OF THE ALZHEIMER'S PART OF AMPLE BUT ALSO TYPE 2 DIABETES WHICH HAS A KNOWLEDGE PORTAL WHICH CONTAINS A LARGE AMOUNT OF VARIOUS TYPES OF OMIC DATA AS WELL AS PHENOTYPE DATA. THE PARKINSON PROJECT PUT UP THEIR KNOWLEDGE PORTAL THE LAST TWO OR THREE WEEKS WITH A LOT OF DATA HEAVILY TRAVELED BUT AT THE MOMENT NOT CONNECTED TO THIS. THE RHEUMATOID ARTHRITIS LUPUS WITH SINGLE CELL BIOLOGY DONE THERE IS TRANSFORMATIVE WITH PEOPLE WORKING ON AUTOIMMUNE DISEASE SO GREAT EXAMPLE OF THE THINGS THAT FIT NICELY INTO THIS IN PHASE 2. >> WHICH IS GREAT. WONDERFUL. >> JEFF GINSBERG. >> THANKS FOR GREAT PROGRESS. THREE QUESTIONS. FIRST ONE, THIS IS RELEVANT TO LARRY'S POINT ABOUT USER FEEDBACK AND I THINK IT'S CRITICAL TO GET ONGOING FEEDBACK. I WONDER IF THERE EXISTS A USER COMMUNITY THAT'S COMMITTED TO THIS. OF EXTRAMURAL AND INTRAMURAL RESEARCHERS THAT KNOW THEY ARE EXPECT TO PROVIDE THAT KIND OF FEEDBACK. >> GREAT QUESTION, THE ANSWER IS YES. WE DO HAVE FOR EACH OF THESE CAMPAIGNS WE DO HAVE USER INTRAMURAL AND EXTRAMURAL WHO ARE WORKING WITH US BECAUSE WE SIT ON THEIR BACKS AND SHOULDERS TO MAKE SURE WHAT WE WERE DEVELOPING WORKS IS A PROTOTYPE BEFORE WE INSTANTIATE IT. YES. AND WE WOULD LOVE TO REACH OUT IN DIFFERENT WAYS TO GET ADDITIONAL USE CASES BECAUSE EVERYTHING IS BUILT UPON USER STORY OR RESEARCH TRAJECTORY. SO WE UNDERSTAND EXACTLY WHAT THE NEEDS ARE OF THE RESEARCH COMMUNITY AND WE CAN TAKE THAT CO-LESS IT AND SYNTHESIZE IT UP TO A PATH FORWARD. YES. >> GREAT TO HEAR. MY SECOND QUESTION IS HOW DOES A MORE DATA NAIVE RESEARCHER LIKE MYSELF KNOW WHAT'S IN ALL THESE DATABASES. CREATE A DATA SET UNIQUE ACROSS THESE TO ANSWER A PARTICULAR QUESTION HOW DO I KNOW WHAT IS IN THEM? IS THERE A WAY TO SEARCH CERTAIN KINDS OF DATA? DO YOU ANTICIPATE THAT'S THE CASE? >> THAT WOULD BE ONE OF OUR EARLY NEXT GOALS TO SEARCH ACROSS THE DATA RESOURCES SO YOU CAN FIND DATA OF INTEREST FOR YOU. RIGHT NOW YOU HAVE TO GO TO EACH PLATFORM AND SEARCH ACROSS THEM. THERE'S SOME CAPABILITIES DB GAP TO DO THIS AS WELL SO YOU CAN GO THERE AND GET INTERESTING DATA OF INTEREST FOR GENOTYPE AND PHENOTYPE BUT TO GET DIFFERENT TYPES OF DATA YOU HAVE TO GO TO THESE RESOURCES. THAT IS THE CURRENT SITUATION NOW BUT IN THE FUTURE WHAT WE LOVE IS FOR YOU TO SIGN ON AND SEARCH PLATFORMS AND PULL THE DATA OF INTEREST TO YOU AND YOUR RESEARCH TOGETHER. >> >> WHAT KIND OF TIME HORIZON DO YOU ANTICIPATE THAT HAPPENING ON? >> THE FIRST THREE OR FOUR USERS CASES SHOULD BE COMPLETED BY BEFORE FY 21, BEFORE OCTOBER. GO OUT WITH PHASE 2. AT LEAST FOR BIODATA CATALYST ADVIL AND CANCER COMMONS RESEARCH FRAMEWORK AND DB GAP SHOULD BE INTEGRATED WITHIN LESS THAN A YEAR. SINGLE SIGN ON, WE HAVE TO BUILD SEARCH CAPABILITIES AND SOME OF THE ISSUES WITH I D RESOLUTION. >> THANKS. MY LAST QUESTION MAYBE LESS MAYBE MORE FOR FRANCIS, HOW DO YOU INCENTIVIZE VARIOUS INSTITUTE DIRECTORS AND RESEARCHERS HERE TO ADOPT STANDARD GOING FORWARD TO BE CONSISTENT AND ALLOW FOR THIS VISION TO BE REALIZED? >> THAT IS A CRUCIAL QUESTION. I THINK WE HAVE TRIED TO ORGANIZE OURSELVES REALLY TO PROVIDE THAT KIND OF CORPORATE COMMITMENT TO MAKING THESE CHANGES HAPPEN. PART IS FORMATION OF SCIENTIFIC DATA COMMITTEE. JOHN LORSH IS HERE WHO CO-CHAIRS THAT WITH BRUCE TROMBERG AND IT INVOLVES REPRESENTATION FROM ALL THE INSTITUTES THAT HAVE MAJOR DATA CONTRIBUTING FEATURES. THAT HAS BEEN PRETTY SUCCESSFUL, THEY PUT TOGETHER A DATA STRATEGIC PLAN WHICH IS WELL RECEIVED AND GUIDES ALL THIS. WE TALK ABOUT THIS A LOT AT OUR LEADERSHIP FORUM RECENTLY AND INSTITUTE DIRECTOR MEETINGS. I THINK EVERYBODY IS ON BOARD THAT TRANSLATION OF THAT INTO ACTION IS NOT OBVIOUS. IN TERMS OF EXACTLY WHAT THAT SHOULD MEET BUT I DON'T THINK THERE'S MUCH OF A POCKET OF RESISTANCE, JOHN DO YOU WANT TO SAY ANYTHING ABOUT THAT IN TERMS OF HOW WE ARE MAKING SURE THAT WE ARE ALL ROWING IN THE SAME DIRECTION? >> I THINK EVERYTHING IS RIGHT, FRANCIS. I THINK A LOT OF CREDIT TO SUSAN AND HER OFFICE WHICH IS EXTREMELY COLLABORATIVE AND IS CREATING THE RIGHT ENVIRONMENT FOR THE ICs TO WORK TOGETHER AND BRINGING TREMENDOUS TEAM PLAYERS LIKE JESS WHO IS OVER THERE ON BOARD. THE SDC, THE DATA SCIENCE POLICY COMMITTEE, AND THE OFFICE OF DATA SCIENCE STRATEGY WORKING HAND IN HAND TO MAKE THIS HAPPEN. >> BRENDAN? >> I LOOK AROUND THE TABLE AND SEE ALL MACS I'M THE ONLY ONE WITH A PC. THAT SAYS SOMETHING I THINK. THERE IS ONE. FRANCIS AND I ARE ALIGNED. IN TERMS OF USERS, RELEVANT TOUCHTONE LARRY'S QUESTION, MOST EXTRAMURAL AND INTRAMURAL POTENTIAL USERS ARE PROBABLY NOT USERS OF THIS ENORMOUS -- IN TERMS OF THIS UNCOUPLING BETWEEN AMOUNT OF DATA AND WHO IS USING IT. AND I THINK WE HAVE THIS GROWING DISPARITY. JUST LIKE WITH HEALTHCARE DISPARITY, THERE'S DATA USE AND ACCESS DISPARITY MANY THE SCIENTIFIC COMMUNITY. THAT'S DRIVEN AT THIS COMMENT I MADE ABOUT MACS AND PC. HOW DO YOU CALIBRATE AND PUT RESOURCES INTO THE SOPHISTICATIONAL LEVEL OF TOOLS THAT WILL ENABLE YOU TO OVERCOME THIS DISPARITY? THERE IS A SUPER USER I'M SURE ONCE YOU GET THIS INFRASTRUCTURE IN PLACE THEY WILL BE GREAT STUFF GENERATED BUT THAT MAY ACCOUNT FOR I DON'T KNOW, 5%, 1%, .1%. HOW DO YOU WORK TOWARDS MOVING THAT THRESHOLD DOWN TO MAXIMIZING THE INVESTMENT WE HAVE MADE GENERATING THESE DATA? >> GREAT QUESTION. I HAVE HAD A LOT OF TALK WITH ANDREA ABOUT EXACTLY THIS ISSUE. SO AT THE SAME TIME THAT WE ARE DEVELOPING THIS, REMEMBER THIS IS JUST ONE OF THE FIVE GOALS OF THE STRATEGIC PLAN. WE ARE INSTANTIATING, A LARGE NUMBER OF TRAINING OPPORTUNITIES AT INSTITUTIONS REGIONALLY AND AT NIH AND WE ARE OUTREACHING TO RESEARCH COMMUNITIES THAT ARE MORE DIVERSE SO GOING TO SMALLER COLLEGES, COLLEGES IMPACTED BY THE TRIBAL COMMUNITIES AS WELL BECAUSE THERE IS A LARGE AMOUNT OF RESEARCH COMMUNITY THAT WOULD BENEFIT FROM WHAT WE ARE DOING AND WE WANT TO TRAIN THEM HOW TO USE CLOUD COMPUTING HOW TO CRUISE THE PLATFORMS HOW TO INTEGRATE THIS WORK TOGETHER. SO THAT IS ONE THING WE ARE DOING. SECOND, WE WILL BE HELPING RESEARCHERS WORKING IN COMPUTER AND INFORMATICS FOR A WHILE WITH COMPUTING CLUSTERS AT UNIVERSITIES TO TRANSLOCATE SOME OF THAT WORK TO THE CLOUD TAKE ADVANTAGE OF PLATFORMS AND DATA TO ENHANCE BIOINFORMATICS AND COMPUTATIONAL SCIENCE COMMUNITY TO BRING TOOLS AND ABILITIES TO THESE PLATFORMS AND TO THE CLOUD. SO WE DO UNDERSTAND THAT TRAINING AND OUTREACH IS NECESSARY, WILL ALSO BE DOING QUITE A FEW CODETHONS AS WELL AS COMMUNITY ENGAGEMENT ACTIVITIES THAT BROADEN THE NUMBER OF PEOPLE WHO CAN PARTICIPATE IN DATA SCIENCE RESEARCH. USING OUR CLOUD DATA SETS. >> ONE EXAMPLE AROUND WHAT THE ALL OF US RESEARCH PROGRAM IS DOING TO START FIGURING HOW TO ANSWER THAT QUESTION, ALL OF US WITH THE INITIAL LAUNCH RESEARCHER TOOLS IS PROVIDING LEVELS OF RESEARCHER TOOLS INCREASING SOPHISTICATION AND SMALLER AUDIENCE FOR EACH SO THE FIRST IS A PUBLIC DATA BROWSER WHICH IS A WEBSITE THAT ANYONE IN THE WORLD, ANY OF US COULD GO TO RIGHT NOW AND BROWSE AGGREGATES STATISTICS AND GET GENERAL CHARACTERISTICS OF THE DATA SETS AIMED AT A LARGE AUDIENCE SEXED IS REGISTERED TIER DATA BROWSER FOR PEOPLE WHO HAVE GONE THROUGH APPROVAL AS RESEARCHERS BUT A POINT AND CLICK INTERFACE THAT DRILLS THROUGH INTO THE COHORT AND BUILD SUBCOHORTS. HOW MANY PEOPLE WITH THESE DEMOGRAPHICS IS POWERED, CAN I FIND A SUBSET THAT HAS CERTAIN DEMOGRAPHICS I WANT TO DO FURTHER RESEARCH ON. STILL ALL POINT AND CLICK. THE THIRD TIER IS A RESEARCH EARLY WORKBENCH TO GET INTO CODING ENVIRONMENT WITH PYTHON AND JUPITER NOTEBOOKS AND SOMEBOD ON YOUR TEAM WITH THOSE SKILLS CAN TAKE THE COHORT AND GO FURTHER. I DON'T KNOW IF WE HAVE THE DETAILS RIGHT BUT THAT TIERED APPROACH OF PROVIDING MULTIPLE LAYERS OF TOOLS YOU ARE TRADING OFF ACCESSIBILITY AND POWER,S AND LET RESEARCHERS FIND THE RIGHT LEVELS, AND FILL IN THE GAPS AS WE HEAR FROM PEOPLE, I'M MISSING -- YOU MISSED ME. LET'S ADD A TOOL SO THAT'S HOW ALL OF US IS STARTING TO ANSWER THAT. >> A RELATED QUESTION, WHEN YOU ARE PUTTING UP ALL THESE LARGE DATA SETS THIS 30 PEDABYTE AND MORE TO COME AND PEOPLE ARE INTERESTED IN BEING ABLE TO DO INTERESTING ANALYSES, IT IS NOT SUFFICIENT TO HAVE THE DATA IN THE CLOUD, THE ANALYTICS HAVE TO BE THERE TOO. YOU ARE NOT GOING TO DOWNLOAD THIS TO YOUR LAPTOP AND PLAY WITH IT. THAT MUST MEAN THEN THAT YOU HAVE TO HAVE HAVE UP FRONT A GOOD IDEA WHAT ARE THE KINDS OF ANALYTICS PEOPLE ARE GOING TO WANT TO HAVE ALREADY THERE FOR THEM SO THAT IF YOU ARE NOT A POWER USER, YOU HAVE TO POWER TO DO SOMETHING AND LOOK AT DATA AND ANALYZE IT. HOW WILL THAT WORK IN TERMS OF ASSESSING WHAT ANALYTIC ARE LIKELY OF INTEREST TO USERS AND MAKING SURE THEY HAVE IN FACT BE PLACED ALONGSIDE THE DATA IN THE CLOUD AND MAYBE EVEN TESTED OUT IN DEBUGGED TO BE SURE THEY WORK AS MUCH >> ABSOLUTELY. THAT IS CERTAINLY SOMETHING WE WANT TO THINK ABOUT WHEN THINKING ABOUT TRUE INTEROPERABILITY OF OUR RESOURCES IS TAKING ADVANTAGE OF TOOLS THAT EACH OF THOSE PLATFORMS LIKE A INHVIL OR CATALYST ARE DEVELOPING AND WORKING WITH THOSE TOOLS ACROSS OUR PLATFORMS. SO ASSESSING WHAT THE RESEARCHERS NEEDS IN TERMS OF SAME USER STORIES, WHAT IS WORKING RIGHT NOW, HOW ARE THEY DOING IT, WHAT WOULD THEY LIKE TO DO IN THE FUTURE. IS ABSOLUTELY A GOAL. THERE IS A LITTLE NASCENT PROJECT IN DATA RESOURCE INTEROPERABILITY THAT WE WILL BE STUDYING AND LEVERAGING IN THE FUTURE. IT IS A GOAL FOR ME AND MANY AGENCIES TO CREATE JUST LIKE A DATA ECOSYSTEM, A COMPUTER ECOSYSTEM BECAUSE AS YOU KNOW, THE HETEROGENEITY OF COMPUTATIONAL RESOURCES IS EVER INCREASING. WE HAVE NOW THE ABILITY TO ON CHIPS FOR THINGS AND NEW CHIPS DEVELOPED FOR AI AND TRAINING AND INFERENCE THAT ARE COMPLETELY DIFFERENT. THEY ARE NEW ARCHITECTURES ARE COMING OUT IN WAIVERS WHICH IS COMPLETELY DIFFERENT FROM MICROELECTRONICS AND NOW WE ARE SEEING THE RISE OF INFORMATION SYSTEMS SO IN THE FUTURE WE WILL HAVE A COMPUTING ECOSYSTEM JUST LIKE WE HAVE A DATA ECOSYSTEM, SO MAKING SURE THAT WE HAVE THE TOOLS THAT CAN WORK ACROSS THOSE VERY DIFFERENT PLATFORMS IS ABSOLUTELY SOMEWHAT A LONGER TERM GOAL, PERHAPS STRATEGIC PLAN 2.0. >> FRANCIS. >> VERY EXCITING. I WAS HAVE YOU BEEN WHEN DAVID WAS SAYING THE THREE LEVELS OF USER, SAY I'M AT THE -- HOW WOULD I GET -- I CAN GENERATE QUOTE STUFF I THINK IS REALLY INTERESTING, HOW WOULD I GET ANYONE TO LOOK AND SEE I AM MESSED UP OR AT EVERY LEVEL WHERE IS THE PEER REVIEW OF THIS? IS THERE SORT OF A ALMOST LIKE A CUSTOMER SERVICE -- YOU CAN SPEND A LOT OF TIME FOLLOWING A RA RATHER IGNORANT PATH. LEAST I COULD. >>NA IS PROBABLY A GOOD MESSAGE TO HEAR. THAT IS IMPORTANT. EACH OF OUR RESEARCH COMMUNITIES MAY BE HAVING SOMETHING LIKE THAT. BUT THINKING ABOUT THAT FROM A DATA SCIENCE PERSPECTIVE, I THINK THAT'S AN IMPORTANT ELEMENT THAT COULD BE ADDED IN THE FUTURE. MORE OR LESS DATA SCIENCE HELP DESK. THAT IS SOMETHING WE CAN CONSIDER, THANK YOU FOR THAT IMPORTANT THOUGHT. >> I WANT TO GO TO THE OTHER EXTREME OF USERSES LIKE PEOPLE WHO ARE EXPERTS IN MACHINE LEARNING. I THINK I ENCOURAGE YOU TO THINK SLIGHTLY DIFFERENT WAY PRESENTING DATA. IT COULD BE THE SAME DATA BUT HAS TWO DIFFERENT INTERFACES. FROM IF MACHINE LEARNING PERSPECTIVE IS ACTUALLY BETTER TO THINK ABOUT DATA FORMAT AS OPPOSED TO PARTICULAR APPLICATION. SO LET ME MAKE THIS MORE CONCRETE. YOU MIGHT HAVE SLEEP STUDIES LIKE PSG DATA WHICH HAS VERY SPECIFIC FORM THAT COMES FROM SLEEP LABS SO YOU MIGHT HAVE THAT IN APPLICATIONS OR IN SOME STUDIES THAT ARE RELATED TO COPD. YOU MIGHT HAVE SOME OTHER THAT ARE RELATED TO CARDIOVASCULAR DISEASE. YOU MIGHT HAVE OTHERS THAT ARE COMING FROM JUST UNDERSTANDING POPULATION AND UNDERSTANDING THE SLEEP AND APNEA AND ALL THAT STUFF. SO HAVING ALL PSG DATA AND BECAUSE IT HAS SPECIFIC FORMAT AND YOU CAN HAVE METADATA DESCRIBING THAT IT COMES FROM THIS STUDY AND THIS PATIENT IN LIKE OF COURSE NOT THE NAME OF THE PATIENT BUT UNIQUE IDEA THAT ALLOWS THE PEOPLE TO REFERENCE, THAT PARTICULAR INDIVIDUAL. THINKING ABOUT DATA IN TERMS OF HERE IS ALL THE PSG DATA THAT WE HAVE. HERE IS A -- OF ALL OF THE GENOMIC DATA THAT WE HAVE. HERE IS ACCELERATION DATA OF DIFFERENT ACCELEROMETERS CELL PHONE ACCELEROMETER. FIT BIT APPLE WATCH WHATEVER, ALLOWS PEOPLE TO THINK MORE GLOBALLY ABOUT DOING TRANSFORMATIVE APPLICATIONS P POSED TO THINKING ABOUT THE DATA I KNOW THE DATA COMES FROM PARTICULAR STUDY BUT TAKING A STEP PACK AND COLLECTIVELY COMBINING DATA THAT HAS SIMILAR FORMAT WHICH ALLOWS PEOPLE TO ACCESS IT MORE EFFECTIVELY. >> ABSOLUTELY. I THINK THAT IN THIS SYSTEM, YOU CAN ORGANIZE DATA IN ANY WAY THAT RESEARCHER FEELS IS IMPORTANT. IT IS CERTAINLY POSSIBLE. THERE'S A LOT OF INTERESTING THOUGHT AND LOOKING FORWARD TO THE NEXT SECTION, WE TALK ABOUT THIS A LITTLE BIT MORE. THERE IS NO A PRIORI ORGANIZATION IS RESEARCHER WHO HAS TO PULL THE DATA THROUGH THIS SYSTEM FOR THE WAY THEY WANT TO DO THEIR RESEARCH. >> LET ME ADD ONE THING. SO I UNDERSTAND WHAT YOU ARE SAYING RESEARCHER WE CAN ORGANIZE DATA INTO THAT STUFF AND OF COURSE BUT I THINK NIH, IF YOU GUYS WANT TO SEE LIKE BIG PROJECT, IT'S IMPORTANT TO THINK ABOUT FEW BIG DATA SETS THAT ARE VERY TRANSFORMATIVE. DEPENDING WHAT DATA YOU HAVE, BECAUSE YOU HAVE TO HAVE SO MANY RECORDS OF THAT DATA, THE RECORDS HAVE TO BE TO LARGE EXTENT SIMILAR, EBB IF THEY COME FROM DIFFERENT STUDIES. SO THE RESEARCH CAN COMBINE THE DATA FROM DIFFERENT STUDIES TO CREATE THEIR OWN POOL OF DATA THAT THEY ACCESS, BUT FOR YOU TO ENABLE THESE TRANSFORMATIVE PROJECTS, YOU HAVE TO THINK WHERE DO YOU HAVE THAT DATA TO PULL TOGETHER CREATE MANY, MANY RECORDS THAT LARGE EXTENT SYSTEMATIC. >> ILLUSTRATIVE EXAMPLES. >> THAT WILL BE THE NEXT PRESENTATION WILL LEAD INTO THAT. SO HOLD THAT THOUGHT >> IMPORTANT THOUGHT. >> P KRISTINA. THEN HANNAH. >> LET ME ADD TO CHORUS OF THIS IS PHENOMENAL. THANK YOU. NAIVE QUESTIONS. MAYBE ALREADY DOING THIS BUT I WAS THINKING COULD YOU ENVISION A TIME THIS RESOURCE WOULD BE OPEN TO SAY CONNECTING THE PRIVATE SECTOR SAY THERE'S ABOUT 15 MILLION FOLKS ON ANCESTRY.COM THAT HAVE DNA AND DNA IS 1.5 GIGABYTES SO THINK THAT ABOUT THAT, THAT'S TEN PEDABYTE, DO YOU SEW A TIME THIS MIGHT BE OPEN TO PEOPLE BEING ABLE TO YOU LOAD RECORDS AND BE PART OF THE CLOUD? TWO, 30 PEDABYTE SOUND LIKE A LOT BUT ON THE OTHER HAND I DON'T KNOW HOW MUCH YOU REALLY NEED TO MAKE SOME SORT OF DEFINITIVE -- HOW BIG DOES THE DATA SET HAVE TO BE? MACHINE LEARNING SENSE THOSE WITH MOST DATA WIN. GENERALLY BUT I DON'T KNOW IS 30 PEDABYTE ENOUGH TO DO RESEARCH, JUST CURIOUS. >> THE FIRST QUESTION, ABSOLUTELY, YES, AND RESEARCHERS CAN CHOWED SERVICE PROVIDERS INTEGRATE AND UPLOAD DATA. WE WOULD CERTAINLY LOVE TO PARTNER PARTICULARLY WITH AGENCIES, INTERNATIONAL AND PATIENT ADVOCACY GROUPS WHO HAVE DATA RESOURCES THEMSELVES, TO INTEGRATE THOSE RESOURCES WITH WHAT WE ARE DOING. WE HAVE A LOT OF SIMILARITY AND WE SHARE A LOT OF THE SAME TECHNOLOGIES THAT THE ELIXER PROJECT IN THE UK IS USING. SO I ANTICIPATE IT'S NOT THAT FAR IN THE FUTURE WE WILL BE ABLE TO PARTNER WITH OUR COLLEAGUES IN THE EU WITH PROJECTS LIKE ELIXIR. TO YOUR SECOND QUESTION, I MIGHT HAVE TO DEFER TO THE AI EXPERTS HOW MUCH DATA IS ENOUGH. I HEAR WHAT PEOPLE TELL ME IS PEOPLE WITH THE MOST DATA WIN IN AI BUT I THINK THAT'S ALSO A DOUBLE EDGE SWORD BECAUSE SOMETIMES IT'S HARDER TO DODD AI ON DATA THAT'S QUITE MESSY. >> SUSAN, THANK YOU VERY MUCH AGAIN, THAT WAS JUST SO EXCITING TO ME. WHAT ISN'T CLEAR TO ME IS AT THE MOMENT WHO IS MAKING THE DECISION AS TO WHETHER OR NOT AN INDIVIDUAL INVESTIGATOR CAN LOAD THE DATA? AND MANY OF US HAVE LARGE DATA SETS. WHO IS THE GATEKEEPERS? THE LEVEL OF THE IC DIRECTORS? I SORT OF TRIED TO DO THIS AND SEEMS TO ME THERE'S SOME HOOP OR BARRIER THAT I DON'T UNDERSTAND THAT I NEED TO UNDERSTAND BETTER TO GET MY DATA THERE. >> GOOD QUESTION. IF IT'S CONTROLLED ACCESS DATA WE HAVE A PROCESS THROUGH DB GAP AND OTHERS TO UPLOAD THAT DATA. IF IT'S OPEN DATA THAT YOU CAN BE SHARED, THERE SHOULD BE NO BARRIER OTHER THAN THE TECHNICAL BARRIER OF BRINGING IT TO THE CLOUD AND WE ARE CERTAINLY WORKING WITH ANDREA, VERY CLOSELY, TO ENABLE AS MANY OF OUR DATA PLATFORMS TO MIGRATE TO THE CLOUD SO THAT MIGHT BE SOMETHING THAT WE CAN PROVIDE ADDITIONAL TRAINING AND RESOURCE ON. >> VERY INTERESTING DISCUSSION. DAVID, ONE MORE QUESTION. >> SUSAN, JUST ON THE LAST COMMENT ABOUT EUROPE AND OTHERS, DO YOU WANT THE SAY ANYTHING ABOUT THE GA 4 GH? >> THANK YOU VERY MUCH. ABSOLUTELY. I DON'T KNOW IF YOU CAN GO BACK A SLIDE OR TWO. SO QUITE A LOT OF THE WORK THAT WE ARE DOING IS LEVERAGING WHAT THE COMMUNITY IS ALSO DOING BECAUSE WHEN WE WORK TOGETHER AS A COMMUNITY WE ACTUALLY CAN INTEGRATE MUCH FASTER AND GO FURTHER. SO GA 4G 8 IS PROVIDING STANDARDS AROUND THE JASON WEB TOKENS FOR THE RESEARCH AUTHENTICATION AND ALSO ON THE TOOLS THAT WILL ALLOW US TO DO AUDITING AND ASSURANCE FOR SECURITY. SO WE WORK CLOSELY WITH OUR STANDARDS COMMUNITY AND GA 4 GH IS ONE OF THE LEADERS IN THIS AREA. NOT ONLY WE, BUT ALSO THE ELIXIR AND OTHER PROJECTS ARE, WHOING WITH GA 4 GH, THAT'S HOW I HAVE CONFIDENCE THAT WE CAN START TO INTEGRATE ACROSS RESOURCES FROM DIFFERENT FUNDING AGENCIES WHEN WE WORK TOGETHER. I HOPE THAT ADDRESSES YOUR QUESTION. THAT'S A GOOD ONE. >> FOR THOSE NOT FAMILIAR WITH THE ABBREVIATION, GA 4 GH IS THE GLOBAL ALLIANCE FOR GENOMICS AND HEALTH WHICH HAS BEEN FUNCTIONING NOW FOR FIVE YEARS I GUESS, MAYBE MORE. AS A COMMUNITY BASED EFFORT INTERNATIONAL TO TRY TO DEVELOP COMMUNITY BASED STANDARDS FOR OPERATING THESE KIND OF COMPLEX DATA SCIENCE QUESTIONS SO IT IS A WONDERFUL PLATFORM THAT WE CAN TAP INTO. AND KNOW WE ARE UTILIZING AN APPROACH THAT IS ALREADY VETTED BROADLY AMONGST WIDE RANGE OF SCIENTISTS ETHICISTS INFORMATICS EXPERTS AND SO ON SO GOOD POINT TO FOLD THAT IN. I THINK THAT'S A GOOD STOPPING POINT FOR THIS USEFUL DISCUSSION. SUSAN, THANK YOU VERY MUCH FOR PRESENTING THE INFORMATION AND ANSWERING ALL THESE MANY QUESTIONS. I THINK THIS IS SOMETHING THE ACD WILL WANT TO KEEP TABS ON BECAUSE IT'S ONE OF THE MORE IMPORTANT AND CHALLENGING NEW AREAS THAT WE HAVE TO PUSH IN TERMS OF USE OF CLOUD FOR ALL THINGS WE WANT TO DO. IT WILL CONNECT QUITE OBVIOUSLY WITH THE NEXT SESSION WHICH IS ON ARTIFICIAL INTELLIGENCE AND WHAT NIH SHOULD BE DOING IN TERMS OF INVESTING IN THAT SPACE. WE ARE A LITTLE AHEAD OF SCHEDULE, I'M LOATHE HOW FAR TO START EARLY WITH THE NEXT SESSION BECAUSE PEOPLE HAVE SEEN THE SCHEDULE AND THERE'S A LOT OF INTEREST IN THIS AND PEOPLE WILL BE SIGNING ON TO WEBEX AND I DON'T WANT TO JUMP THE GUN SO WE WILL NOW GIVE YOU A REALLY NICE BREAK UNTIL 10:30 AND THEN PROMPTLY AT 10:30 WE WILL START WITH THAT PRESENTATION FROM THE WORKING GROUP ON AI. ENJOY YOUR BREAK. WE ARE NOW GOING TO HEAR THE FINAL REPORT OF THE ACD WORKING GROUP ON ARTIFICIAL INTELLIGENCE. THIS WORK GROUP WAS ASSEMBLED JUST ABOUT A YEAR AGO AND WAS CHARGED IN FEBRUARY OF 2019. THEY GAVE YOU AN INTERIM REPORT IN JUNE INCLUDING SEVERAL DO NOW RECOMMENDATIONS, THAT THEY CONTINUE TO BUILD AND DEVELOP OVER PAST SIX MONTHS. ONE OF THOSE WAS TO HOLD WORKSHOP TO BRING TOGETHER THE COMPUTATIONAL AND BIOMEDICAL COMMUNITIES. THE TIMING TURNED OUT TO BE INTERESTING BECAUSE THAT WORKSHOP IS HAPPENING TODAY. BUT NOT HERE. WITH BIG THANKS TO PLANNING GROUP THAT INCLUDES MANY OF THE WORKING GROUP MEMBERS ALONG WITH DEDICATED FOLKS FROM THE BROAD. I WAS UNABLE TO ATTEND BUT I DID SEND A VIDEO AND HOPE IT WAS WELL RECEIVED. OF COURSE I'M HERE AS ALL OF YOU ARE LISTENING CAREFULLY NOW TO HEAR WHAT THIS WORKING GROUP HAS COME UP WITH. I THINK FROM THE VIEW OF MANY OF US IF YOU COULD PICK AN AREA THAT HAS EMERGED IN THE LAST HALF DECADE THAT IS GOING TO HAVE A MASSIVE IMPACT ON PIE MEDICAL RESEARCH THAT WE NEED TO -- BIOMEDICAL RESEARCH THAT WE NEED THE PAY ATTENTION TO, THIS IS IT. SO WE ARE WAITING WITH BAITED BREADTH. TO HEAR FROM DAVID GLAZER AND LARRY TABAK EXACTLY WHAT THE RECOMMENDATIONS ARE GOING TO BE. I SHOULD SAY WORKING GROUP MEMBERS ON THE PHONE, IF IT'S ACTUALLY HAPPENED, MAYBE I WILL CHECK AND SEE, IS READY ON THE PHONE? MICHAEL MCMAHANUSS. >> YES, I'M HERE. >> WITH THAT, WITH MANY THANKS FOR THE HARD WORK THAT'S GONE INTO THIS, LET ME TURN THIS OVER TO ACD MEMBER AND CO-CHAIR, OF THE AI WORKING GROUP DAVID GLAZER. >> THANK YOU, FRANCIS. THANK YOU TO EVERYONE ON THE WORKING GROUP INCLUDING REDIA AND MICHAEL ON THE PHONE AND DINA IN THE ROOM WHO WAS NOT ON ACD WHEN WE STARTED BUD GOT TO WORK TOGETHER IN BOTH ROLES. T LARRY THANK YOU AND THANK YOU TO JESS IN THE BACK WHO HELPED KEEP US ON A PATH AS WE WENT FORWARD. SO I'M GOING TO QUICKLY REVIEW THE HOW WE GOT HERE CHARGE AND START TO TRY TO LAY OUT THE BIG PICTURE OF WHERE WE STARTED HOW WITH A SET OF EIGHT RECOMMENDATIONS WE WILL WALK THROUGH IN SOME OF THEM MORE DETAIL THAN OTHERS. SO THE BACKGROUND WHERE DID WE START? PREMISE, THESE WERE THE SLIDES& THAT LARRY AND FRANCIS PUT TOGETHER AS WE ARE KICKING OFF. WE KNOW THAT WE ARE ALL AS HUMANS GENERATING LOTS OF DATA ALL THE TIME. WE KNOW THAT AI IS MAKING A DIFFERENCE IN ALL ASPECT OF LIFE THESE DAYS AND THEREFORE OUR CHARGE WAS HOW DO THESE THINGS COME TOGETHER? WHAT CAN THE NIH DO TO MAXIMIZE THE OPPORTUNITY FOR AI TO MAKE A DIFFERENCE IN BIOMEDICINE. IN ONE SLIDE THIS WAS THE PREMISE OF WHAT WE STARTED THERE ARE TWO REVOLUTIONS HAPPENING. HAPPENING RIGHT NOW, THEY ARE -- THEY HAVE ALREADY CHANGING THE WORLD. ONE IS AROUND GENERATING DATA, ONE IS AROUND ANALYZING DATA. THESE TWO REVOLUTIONS ARE OPENING THE DOOR THE NEW WORLD WE COIN TO HAVE A WAY TO TALK ABOUT IT THE PHRASE ML BIOMED BUT THEY COINED THIS NEW OPPORTUNITY TO NOT JUST DO THESE THINGS NEXT TO EACH OTHER BUT HAVE THEM INFORM AND CHANGE EACH OTHER SO THE REAL AND BIGGEST BIOMEDICINE ADVANCES THE STATE OF MACHINE LEARNING AND MACHINE LEARNING HELPS ADVANCE THE STATE OF BIOMEDICINE WITH EXPERIMENTS AND ADVANCES DESIGNED FOR EACH OTHER. I WAS THINKING THIS MORNING ABOUT TIMING AND BACKGROUND ABOUT THE FACT THIS IS A REVOLUTION AND THINGS ARE MOVING FAST. I REMEMBER MY INTERACTION WITH THE NIH SOMEONE WHO HAS DONE SOFTWARE MY WHOLE LIFE IN 2014, WE WERE GOING TO WORK GENOMICS AND MY FIRST INTERACTION WAS TO GET ACCESS TO DB GAP DATA, GCDA, I DON'T HAVE TO SAY ANY MORE. MANAGED TO GET ACCESS TO TCGA DATA, MANAGED TO -- THIS WAS IN COLLABORATION WITH CANCER RESEARCHER AT BROAD, MANAGE TO TRANSFER SEVERAL HUNDRED TERABYTES OF DATA INTO THE CLOUD AND START DOING ANALYSIS HE WANTED TO DO, USING SCALABLE MODERN DATA ANALYSIS TECHNIQUES, WE WERE VERY EXCITED. THEN IN JANUARY 2014 I GOT A C SEE CEASE AND DESIST EMAIL SAYING WE DON'T LET PEOPLE USE THE CLOUD BECAUSE IT'S DOCUMENT& TO OUR ATTENTION THE CLOUD IS CONNECTED TO THE INTERNET. THAT'S NOT ALLOWED. I HAVE THE EMAIL. WE DELETED SEVERAL HUNDRED TERABYTE OF DATA FROM THE CLOUD OF TCGA AND STOPPED: IT'S LIKE THE POLICY WAS THIS IS TOO SCARY TOO SOON. THAT WAS ONLY FIVE YEARS AGO. IF YOU THINK ABOUT THE PRESENTATION SUSAN GAVE US AND THE COMPLETE TRANSFORMATION, THAT IS THE WORLD WE LIVE IN, EXCITING, THE RIGHT TIME, THE DATA GENERATION CONTINUES TO GROW AND EXPAND WITH NEW KIND AND SCALES OF DATA AND DATA ANALYSIS IS EXPANDING AND COMING TOGETHER. THAT'S WHY WE ARE HERE. THAT'S WHY THIS IS THE RIGHT TIME FOR EXACTLY THE KINDS OF OPPORTUNITIES SUSAN TALKED ABOUT AND THAT WE ARE TALKING ABOUT NOW. AS WE THOUGHT ABOUT THIS NEW WORLD, WE REALIZE THERE WERE THREE THEMES WE HAD TO WORK ON INEXTRICABLY CONNECTED, YOU WILL SEE US TALK REGULARLY AND ORGANIZE OUR RECOMMENDATIONS INTO THESE THEMES OF DATA THAT IS HELPING MOVE TO A WORLD WHERE MACHINE LEARNING INFORMS BIOMEDICINE, THE ETHICS TO KEEP US ON THE RIGHT TRACK AND PEOPLE NEEDED WHO CAN WORK ACROSS THESE DIFFERENT WORLDS ACROSS THESE DISCIPLINES AND COMING TOGETHER. WHERE WE STARTED, WHERE WE -- WHEN WE AS A WORKING GROUP FIRST CAME TOGETHER, AND SAID WHAT ARE THE THINGS WE ARE IN AGREEMENT ON AND THINGS WE WILL WORK AND LEARN TOGETHER AND REFINE OUR OPINIONS. THESE ARE PROBABLY THE FIRST FOUR POINTS AS I WORKING GROUP WE QUICKLY SAID WE ARE ALL ON THE SAME PAGE HERE. FIRST WAS TO SCALE THE OPPORTUNITY. THAT WAS OBVIOUS TO EVERYONE. FRANCIS WHEN HE HAD YOU MADE YOUR CALLS TO PEOPLE ASKING IF THEY ARE WILLING TO SERVE, NUMBER ONE IS WHY YOU GOT YES AND FAST. SECOND, WE QUICKLY FELT LIKE WE NEED NEW DATA GENERATION PROJECTS, WE WILL TALK MORE ABOUT THAT OVER THE NEXT FEW SLIDES. THIRD IS WE SPENT A LOT OF TIME TALKING ABOUT HOW DO WE GET PEOPLE TO PAY ATTENTION, HOW TO WE GET MACHINE LEARNING PEOPLE WHO ARE AND THE ANALOGY IS IT'S HARD TO HEARD CATS AND REJOINER IS NO, IT'S NOT YOU MOVE CAT FOOD. CAT FOOD FOR COMPUTATIONAL EXPERTS FROM MACHINE LEARNING EXPERTS IS DATA. SO THE RIGHT WAY THE ATTRACT THE RIGHT EXPERTS IS TO CREATE RIGHT DATA. THAT WAS ONE O OUR STARTING POINTS, A LARGE PART OF HOW WE GET THE RIGHT BRAINS ENGAGED IN SOLVING THESE PROBLEMS. FINALLY THEY ARE INEXTRICABLY TIED, THE TIME TO INVEST IN ETHICS IS NOW BEFORE DIGGING A DEEPER HOLE. WE WILL TALK WAYS WHICH NEW TOOLS SHARP ONLIES CUT BOTH WAYS AND TALK ABOUT SOMETHING WE NEED FOR FURTHER INVESTMENT. THAT'S WHERE WE STARTED. I WILL TALK OPPORTUNITIES WE SAW HIGH LEVEL AND CHALLENGES AND THAT'S HOW WE SET UP THE ACTUAL ACTIONS AND RECOMMENDATIONS. THAT WE WANTED TO TAKE. ANYONE KNOW WHAT THIS IS A PICTURE OF? I THOUGHT THAT WAS RECOGNIZABLE. I WASN'T SURE. I WAS TRYING TO EXPLAIN WHAT IS ACTIVE LEARNING TO MY WIFE? THIS WAS THE METAPHOR I CAME UP WITH. THE IDEA OF ACTIVE LEARNING IS FEEDBACK LOOPS BETWEEN RUNNING EXPERIMENTS AND GET RESULTS OF THE EXPERIMENTS. IF YOU PICTURE PLAYING A GAME OF BAT SHIP YOU RUN AN EXPERIMENT YOU SAY SEE SEVEN. YOU GET BACK RESULT FROM THAT EXPERIMENT. IF YOU WERE BAD AT BATTLESHIP, YOU WOULD PREDEFINE UP FRONT SET EXPERIMENTS AND SAY I'M GOING TO ASK THESE QUESTIONS IN THIS ORDER TO SEE IF I CAN FIND THE OTHER SHIPS. THAT WOULDN'T BE A SMART WAY TO PLAY THE GAME BATTIN SHIP. AN ACTIVE LEARNING WAY TO PLAY BATTLESHIP WOULD BE TO BASED ON THE RESULTS OF THE FIRST FEW EXPERIMENTS, INFORM FUTURE EXPERIMENTS AND CONTINUE TO REFINE WHAT EXPETER TO RUN NEXT. WHAT DATA SHOULD I GATHER NEXT. IN ORDER TO MAXIMIZE THE VALUE OF EACH NEW QUESTION SO I LEARN MORE ABOUT OCEAN. PICTURE BATTLESHIP INSTEAD OF TEN BY TEN TWO DIMENSIONAL GRID MILLION BY MILLION THOUSAND DIMENSIONAL GRID. BY THE WAY WE ARE NOT SURE WHAT THE SHIPS ARE SHAPED LIKE OR HOW MANY THERE ARE BUT WE ARE SURE THEY ARE OUT THERE. THOSE ARE BIOLOGICAL SEARCH SPACES ARE LIKE. IF YOU ARE DOING PROTEIN DESIGN, THAT'S MORE LIKE THAT KIND OF SEARCH SPACE. IF YOU TRY TO FINE TUNE DRUG REGIMENS COMBINING EVERY POSSIBLE SEQUENCE OF INTERVENTIONS ON A PARTICULAR CHARACTERISTIC SET OF CELLS, IF YOU ARE TRYING TO DISSECT HOW CELLS WORK, THOSE SORTS OF HUGE SEARCH SPACES THERE IS A WHOLE AREA OF MACHINE LEARNING ACTIVE LEARNING THAT IS DESIGNED TO HELP THOSE FEEDBACK LOOPS, HELP THAT PROCESS OF MEASURE, USE WHAT YOU LEARNED, TO REFINE THE NEXT MEASUREMENTS TO MAXIMIZE THE VALUE OF WHAT YOU LEARN NEXT. THIS IS ONE OF THE OPPORTUNITIES WE THINK THAT THE NEW MEASUREMENT AND PERTURBATION TOOLS IN BIOMEDICINE ARE WELL MATCH TO THE NEW CAPABILITIES AND OBVIOUSLY LEADS TO NEW LEADS FOR EXPERIMENTAL DESIGN. THIS IS NOT -- DOESN'T WORK IF YOU GATHER ALL THE DATA UP FRONT. THAT'S LIKE PLAYING BATTLESHIP WHERE YOU SAY FIRST I'M GOING TO DECIDE WHERE I'M GOING TO LOOK. THAT'S ACTIVE LEARNING. WE TALKED ABOUT THE OPPORTUNITIES TO DO MORE THAN JUST MAKE PREDICTIONS FROM EXISTING DATA BUT FILL IN GAPS TO SAY I HAVE PARTIAL DATA AND I WANT TO USE MACHINE LEARNING TO TELL ME WHAT I DIDN'T SEE IN THE DATA TO IMPUTE AND FILL IN SPACES BETWEEN THINGS. WE HAVE EXAMPLES HERE BUT WE THINK THERE ARE OPPORTUNITIES,& THESE ARE OBVIOUSLY MORE THAN JUST DO WHAT WE ARE DOING TODAY BUT BETTER FASTER WHICH YOU CAN DO WITH MACHINE LEARN,, THIS IS TRYING TO DO TRANSFORMATIVE THINGS WITH MACHINE LEARNING. MOVING FROM RESEARCH TO CARE, WE TALKED OPPORTUNITIES FOR WHAT CAN MACHINE LEARNING DO TO MAKE A DIFFERENCE IN CLINICAL CARE AND THE FIRST BULLET SUMMARIZES THE PARADIGM, MEASURE INPUTS OUTPUTS, MAKE PREDICTIONS. WE NOW ARE ABLE TO MEASURE BOTH INPUTS TO HEALTH AND RESULTS, OUTCOMES OF HEALTH IN GOOD SCALABLE WAYS AND THEREFORE WE EXPECT THAT THERE WILL BE GROWING NUMBER OF MODELS POSSIBLE TO TRAIN THAT LET YOU DO PERSONALIZED PREDICTION, ONE IMPORTANT THING YOU CAN DO WITH MACHINE LEARNING IS YOU DON'T NEED TO KNOW WHICH OF THE INPUT SIGNALS ARE THE ONES THAT ARE MOST RELEVANT FOR OUTCOME. YOU DON'T NEED TO KNOW UP FRONT WHICH PARTICULAR VARIANTS, WHETHER IT'S VARIANTS OR PROTEOME, WHETHER IT'S ACTIVITY OR DIET OR AN ASSAY, YOU CAN TRAIN THE MODEL BY GIVING IT ALL OF THE INPUTS YOU HAVE AND THEN ONCE THE MODEL IS REFINED YOU CAN FIGURE WHICH ONES MATTERED WHICH THEN LETS YOU SCALE DEPLOYING IT BECAUSE YOU SAY NOW I KNOW BECAUSE I HAVE TRAINED THE MODEL ON HUGE DATA SET THAT GOING FORWARD IF I MEASURE THESE 20 THINGS I CAN USE THOSE 20 THINGS TO INFORM THE RESULTS. SO THE PREDICTIONS TO THE LAST BULLET HERE, PREDICTIONS CAN -- THESE KINDS OF PREDICTIONS CAN BE SCALED BROADLY, CHEAP SENSORS AND UBIQUITOUS COMPUTING MACHINE LEARNING MODELS HAVE BEEN BUILT THERE'S LOTS OF WORK IN OPTIMIZING THEM TO RUN ON SMALL DEVICES. SO AS THESE NEW MARKERS AND PREDICTORS COME TOGETHER WE SEE A HUGE OPPORTUNITY TO NOT ONLY FIND THE NEW PREDICTIONS BUT DEPLOY THEM AT SCALE. I DID A SHALLOW DIVE CASE STUDY WE HAVE A REPORT WITH LINKS IF PEOPLE WANT TO READ MORE. I FIND INTERESTING THE SCIENCE IS INTERESTING BUT MORE THE OPPORTUNITY TO LOOK BEYOND THE OBVIOUS THAT WE ARE STARTING TO SEE HINTS OF. THESE TWO PAPERS REPRESENTED HERE ONE ON THE LEFT WAS A RESEARCH GROUP THAT WAS SAYING IF WE TAKE A BUNCH OF FUNDUS IMAGES, BUNCH OF PICTURES OF BACK OF PEOPLE'S EYES AND TAKE KNOWN DIAGNOSES WHICH OF THOSE PEOPLE ARE SUFFERING FROM DIABETIC RETINOPATHY, CAN WE TRAIN A MACHINE LEARNING MODEL TO MAKE THOSE SAME PREDICTIONS AS ACCURATELY AS HUMAN OPHTHALMOLOGISTS? EVERYONE EXPECTED THE ANSWER TO BE YES, THAT'S AREAS MACHINE LEARNING HAS GOTTEN GOOD AT FIRST. AS EXPECTED YOU CAN SEE THE CURVES WERE VERY GOOD. YES WE WERE ABLE TO USE MACHINE LEARNING. VERY BROADLY HERE THIS IS NOT ME. WE USE MACHINE LEARNING TO DETECT KNOWN SIGNALS AND KNOWN INFORMATION. THIS IS SOMETHING PEOPLE CAN DO, WE THOUGHT MACHINES SHOULD BE ABLE TO DO IT ALSO. THEY WERE ABLE TO. THAT TEAM WENT ON AND YEAR OR TWO LATER COUPLE OF YEARS LATER PUBLISHED A SECOND PAPER THAT SAID WHILE LOOKING AT THE DATA SEE WHAT ELSE IS HIDDEN IN PEOPLE'S EYES. CAN WE TRAIN A MODEL TO LOOK AT BACK OF SOMEONE'S EYE AND FIGURE OUT HOW HOLD THEY ARE? TURNS OUT YES YOU CAN. CAN PRETRAIN TO TO PREDICT OTHER CARDIOVASCULAR RISK FACTOR? YOU CAN. NOT PERFECTLY, THAT SECOND AROUND BLOOD PRESSURE IS NOT GREAT BUT IT'S -- THERE'S A SIGNAL. THAT'S SOMETHING THERE. AMEND THERE IS A SET OF OTHER THINGS I LOOK AT. SURPRISING. THE FIRST TIME THAT RESULT POPPED UP IN TEAM DOING THE WORK, THEY SAID YOU MUST HAVE DONE IT WRONG. I THINK IT WAS -- THE FIRST AGE PREDICTION JUST AS A TEST OR MAYBE IT WAS THE GENDER PREDICTION. YOU CAN TELL WHETHER SOMEONE IS MALE OR FEMALE BY BACK OF EYE. NO ONE EXPECTED THAT BUT THE DATA SPOKE. THAT WAS STEP 2. STEP 3 IS WHERE THE LOOP COMES BACK TO BASIC RESEARCH. WHAT YOU SEE THERE IN STEP 3 DOESN'T COME THROUGH WELL BUT YOU SEE A SINGLE IMAGE OF SINGLE EYE, TOP LEFT IS REAL COLOR, THE OTHERS ARE FAKE COLOR WHERE THEY ARE HIGHLIGHTED BASED ON WHAT DID THE MACHINE LEARNING MODEL FIND MOST USEFUL IN THAT IMAGE TO PREDICT AGE? WHAT DID IT FIND USEFUL IN THAT IMAGE TO PREDICT BLOOD PRESSURE? YOU CAN START TO BRING THE HUMANS BACK IN THE LOOP AND SAY OKAY, YOU FOUND A SURPRISING SIGNAL, HOW DID YOU FIND IT? LET'S START TO EXPLORE AND INVESTIGATE. SO THIS IS A HINT OF WHAT'S POSSIBLE MORE THAN BUSINESS AS USUAL, TO DO WHAT HUMANS ALREADY DO, FIND NEW THINGS AND GO DEEPER AND BACK AND FORTH BETWEEN CARE, WHICH IS WHERE THE STARTED, AND RESEARCH, AND INSIGHT. ED FROM THOSE ARE A LOT OF THE SCIENTIFIC OPPORTUNITIES WE SAW, WE ALSO IN DISCUSSION SAID LET'S NOTLY. OURSELVES, WE HAD DISCUSSIONS UP FRONT, THERE'S TWO KINDS OF PEOPLE IN THE WORLD, THERE'S BIOLOGISTS AND COMPUTER SCIENTISTS. OUR WORKING GROUP FORTUNATELY WAS WELL CONSTITUTED TO CORRECT US AND SAY NO, THOSE ARE NOT THE ONLY TWO KINDS OF PEOPLE IN THE WORLD. THERE ARE GREAT OPPORTUNITIES TO BUY INCLUDING PEOPLE FROM THE OTHER DISCIPLINES, SOCIAL SCIENCES THE HUMANISTIC SCIENCE TO INCLUDE THAT EXPERTISE AND THAT PERSPECTIVE, EARLY IN BRINGING MU DATA IN TO HELP NEW PLACES AND FIND NEW APPLICATIONS, WHAT ARE SOME OF THE NON-TECHNICAL DETERMINANTS OF HEALTH THAT SHOULD BE INCORPORATED AS WE ARE WORKING IN A WORLD OF MULTIPLE SIGNALS, MULTIPLE INTERVENTIONS. WHERE DO SOCIAL FACTORS LIKE PATIENT DOCTOR COMMUNICATION, HOW SHOULD WE FACTOR THAT INTO BOTH EXPERIMENTAL DESIGN AND DEPLOYMENT OF MODELS. THEN AROUND ETHICS WE SAID BECAUSE MACHINE LEARNING IS CAUSING US TO MOVE FORWARD WITH NEW TECHNIQUES, NEW PRACTICES AND ETHICS THERE IS AN OPPORTUNITY TO TAKE SOME OF THOSE NEW IDEAS THAT WILL BE DEVELOPED, AS WE WORK WITH MACHINE LEARNING, AND USE THOSE IDEAS TO REALLY EXPAND PEST PRACTICES ACROSS ALL BIOMEDICINE. YOU SEE EXAMPLES HERE AROUND CONSENT, ACCOUNTABILITY, AROUND CROSS EVALUATION OF MODELS. SO THAT WAS -- WE ARE EXCITED. THAT WAS HERE IS ALL THE OPPORTUNITIES WE SEE, HOW DO WE MOVE FORWARD. WE ALSO WANT TO TALK CHALLENGES THAT MIGHT LOCK US FROM GETTING TO THAT WORLD OR THAT MIGHT BE UNINTENDED CONSEQUENCES THAT WOULD MAKE US UNHAPPY. WITH ASPECTS OF MOVING INTO THAT NEW WORLD. THE FIRST CHALLENGE IS THE DATA ITSELF. SUSAN WENT THROUGH A FEW GOOD EXAMPLES WHY EXISTING DATA SETS ARE NOT IDEAL FOR WORKING IN THE WORLD OF MACHINE LEARNING AND SCALABLE ANALYSIS AND SOME OF THE QUESTIONS AND CONVERSATION THERE GOT TO THAT. THESE ARE SOME OF THE KINDS OF LIMITATIONS, OBVIOUSLY NOT ALL OF THESE LIMITATIONS APPLY TO ALL DATA SETS BUT THERE ARE MANY CASES IN WHICH EXISTING DATA, I THINK PROBABLY THE ROOT CAUSE IS MOST EXISTING DATA WAS ORIGINALLY COLLECTED FOR A NARROW MIGHT BE THE WRONG WORD BUT RESTRICTED PURPOSE. THE QUESTIONS EARLIER ABOUT ORGANIZEMIZING BY DATA TYPE INSTEAD OF WHAT QUESTION WE ARE TRYING TO ANSWER, THAT IS AN EXAMPLE. OR HOW COME WE HAVE -- EARLIER EXAMPLES OF THIS DATA SET HAS THIS INFORMATION OR THAT DATA SET HAS THAT INFORMATION, IT MADE SENSE FOR THE ORIGINAL PURPOSE BUT WHEN DOING THE BROADER PURPOSE LET'S UNLOCK THE VALUE OF THIS INFORMATION WHICH IS WHAT THE NEW TOOLS LET US DO, WHEN YOU GET TO THAT POINT, THAT O ORIGINAL LIMITATION IS LIMITING. IF YOU TRAIN A MODEL FROM A NARROW DATA SET YOU WILL LEARN ABOUT THAT MARROW DATA SET. AND AS I THINK WE TALKED ABOUT IN OUR INTERIM REPORT ALSO, THERE'S A RISK THAT MACHINE LEARNING CAN MAKE IT WORSE. BY ADDING RESPECTABILITY ON TOP OF SAME LIMITATIONS AND SAY IT'S NOT ME BEING BIASED, THE MACHINE SAID. SO THERE'S AN EXTRA CHALLENGE AROUND THAT. AND THE LAST BULLET THERE BACK TO MY DB GAP STORY FROM FIVE YEARS AGO ACCESS POLICIES MATTER. A LOT OF THE ANECDOTES WE HAVE FROM PEOPLE ON THE WORKING GROUP AND PEOPLE THEY WORK WITH ABOUT HAIR CLINICAL HEDGES AND GETTING PERMISSION TO WORK WITH DATA, EVEN WITH A DATA SET PROMISING THE PROCESS OF BEING ALLOWED TO WORK WITH IT SLOWED DOWN IN MANY CASES BLOCKED THEIR ABILITY TO GO FORWARD. SO A LOT OF CHALLENGES AROUND DATA. CHALLENGES AROUND CONCEPT, THESE TIE -- CONSENT. THESE TIE. BUT ONE CHALLENGE OF CONSENT IS THE WORLD OF MACHINE LEARNING AND BIOMEDICINE ARE COMING FROM DIFFERENT STARTING PLACES, NOT SURPRISINGLY, GIVEN THE DIFFERENT DOMAINS OF DATA. SOME MACHINE LEARNS PRACTICES ARE WELL SUITED HOW CAN WE ACCESS AND USE DATA BUT NOT NECESSARY HI THINKING ABOUT THE -- NECESSARILY THINKING PANT THE KINDS OF INFORMED CONSENT AND UNDERSTANDING OF WIDE USE AND REUSE REQUIRED IN BIOMEDICINE. AND THERE'S A LACK OF THE GUIDELINES TO REALLY SAY HOW SHOULD THIS BE DONE IN A WAY THAT MAXIMIZES THE VALUE TO ALL POPULATIONS BY MAKING SURE THE DATA ON ALL POPULATIONS IS PART OF BUILDING THESE NEW INCITES AND INTERVENTIONS. WHILE AT THE SAME TIME PROTECTING ALL POPULATIONS. THAT IS A CHALLENGE TO GET THAT RIGHT, TO GET THAT BALANCE RIGHT. FROMTHEN FINALLY WE TALK ABOUT CHALLENGES AROUND ETHICS AND WITHOUT NEW COORDINATED EFFORTS AS WE TALKED ABOUT, MACHINE LEARNING CAN ACTUALLY MAKE THINGS WORSE. MANY OF YOU HAVE SEEN THE PAPER LINKED TO THERE, I WON'T GO THROUGH IT IN DETAIL BUT THE GIST IS THIS IS AN ALGORITHM BY LOOKING AT PAST INEQUITIES PERPETUATED THOSE INEQUITIES AND THE AUTHORS OF THE PAPER SAY HEY I SEE WHAT HAPPENED HERE. I UNDERSTAND HOW IT WORKED. IF NOT CAREFUL MACHINE LEARNING ALGORITHMS WILL BE HARDER TO SEE WHAT HAPPENED THERE AND LESS OBVIOUS SOMETHING IS GOING ON WHERE THEY WERE LOOKING AT PAST OUTCOMES INSTEAD OF NEEDS. SO LOT OF CHALLENGES WE WANT THE ADDRESS. THE ONE I TALKED ABOUT HERDING CATS IS ALSO CHALLENGE OF GETTING THE PEOPLE IN THE ROOM. I'M NOT GOING TO DO IT BUT IF I ASK SHOW OF HANDS FOR PEOPLE IN THE ROOM FOR PEOPLE WHO FEEL COMFORTABLE WITH EXPERTISE WITH SOME ASPECT OF BIOMEDICINE AND PEOPLE FEEL COMFORTABLE WITH EXPERTISE ON SOME DEEP ASPECT OF MODERN COMPUTATION, MOST HANDS WOULD GO UP, FEW PEOPLE WOULD HAVE RAISED HAND TWICE. BUT IN ORDER FOR TEAMS TO ACTUALLY MAXIMIZE THIS OPPORTUNITY,, WE NEED TEAMS BILINGUAL MULTI-LINGUAL TO DO THAT AND WE NEED TEAMS THAT CAN WORK TOGETHER THOSE COLLABORATIVE TEAMS, NONE OF OUR SYSTEMS TRAIN PEOPLE TO BE ABLE TO WORK ACROSS THESE DISCIPLINES. THAT'S A RADAR UNUSUAL SET OF PEOPLE WHO ARE LITERATE LET ALONE FLUENT IN MULTIPLE DOMAINS. THERE IS A BUNCH OF CHALLENGES TO GET THERE. WHAT DO WE THINK THE NIH CAN DO ABOUT THIS? WE ORGANIZE OUR RECOMMENDATIONS, I WILL START AT TEN O'CLOCK AND GO CLOCKWISE FROM ONE TO EIGHT AND YOU WILL SEE RECOMMENDATIONS REALLY DO SPAN ALL ASPECTS OF DATA AND ETHICS AND PEOPLE. HERE THEY ARE IN READABLE FORM. NOT GOING TO READ THEM NOW BECAUSE WE WILL TOUCH ON EACH IN ORDER. BUT NUMBER ONE IS NUMBER ONE FOR A REASON. IT WILL DRIVE ALL ASPECTS OF THE PROCESS, IT WILL ATTRACT THE RIGHT PEOPLE, GENERATE THE DATA TO ENABLE INCITES AND IT WILL BE -- GIVE US TEST BEDS TO GET ETHICAL GUIDELINES AND CRITERIA RIGHT. BEFORE JUMPING INTO THAT I WILL TALK THREE THINGS IN PARTICULAR WE TALKED ABOUT ADS A GROUP WE THINK ARE IMPORTANT AND WE DECIDED FOR DIFFERENT REASONS NOT TO INCLUDE IN OUR RECOMMENDATIONS. THE FIRST ONE IS HEY WHAT ABOUT IMPROVING STATE OF THE ART MACHINE LEARNING? SHOULD THE NIH INVEST IN FUNDING OR ACCELERATING THAT? NO. THAT'S HAPPENING. THERE'S OTHER VENUES FOR GENERAL PURPOSE MACHINE LEARNING. WE DON'T THINK THE NIH HAS A UNIQUE NEED OR CAPABILITY THERE AND LET'S LEAVE THAT TO THE EXISTING COMMUNITY. WE TALKED ABOUT THE CONTINUED USE OF EXISTING MACHINE LEARNING TOOLS ON EXISTING DATA. WE SAID THAT'S AWESOME. IT'S GOING TO KEEP HAPPENING. THERE'S ENOUGH GOING ON THERE THAT WE DON'T NEED NEW ACTIONS NEW RECOMMENDATIONS. WE THINK THE EXISTING DATA SETS AND THE EXISTING TOOLS ARE ALREADY BEING USED WELL, THAT WILL CONTINUE TO HAPPEN. AS MORE PEOPLE BECOME MULTI-LINGUAL IN DIFFERENT DOMAINS, IT WILL HAPPEN MORE. IT DOESN'T NEED A SPECIAL EXTRA FOCUS THAT WILL HAPPEN. FINALLY, AS WE HEARD FROM SUSAN THIS MORNING WE ARE NOT TALKING INVESTMENT AND SCALABLE SKEWER CLOUD INFRASTRUCTURE, IT'S ESSENTIAL. NOTHING ELSE WE ARE TALKING ABOUT HAPPENS WITHOUT THE DATA INFRASTRUCTURE ON WHICH WE CAN BUILD THESE HIGHER LEVEL KINDS OF ANALYSES. BUT IT'S NOT MACHINE LEARNING SPECIFIC. SO WE ARE TALKING WITH SUSAN AT THE BREAK HOW ALL DATA INFRASTRUCTURE WORK IS NECESSARY BUT NOT SUFFICIENT. BECAUSE WE KNOW THAT'S BEING HANDLED WE ARE TAKING THAT AS A PRE-REQ BEING ADDRESSED SO RECOMMENDATIONS. FIRST RECOMMENDATION IS TO SUPPORT FLAGSHIP DATA GENERATION EFFORTS TO PROPEL CONGRESS BY THE COMMUNITY. WE LISTED IN THE GREEN BOX WHAT WE THINK ARE THE KEY ATTRIBUTES OF WHAT THESE EFFORTS SHOULD BE. RATHER THAN GOING INTO ALL DETAILS HERE, I WILL HIT THE TWO BOLD POINTS BECAUSE IF WE GET THOSE RIGHT, ALL OTHER THINGS NATURALLY FALL OFF, THAT'S HOW THE WORKING GROUP STARTED WITH WHAT DO WE THINK THE FLAGSHIP DATA EFFORTS SHOULD LOOK LIKE. IF THE PROJECTS ARE DESIGNED AND PROJECT TEAMS ARE CONSTITUTED TO INVOLVE STRONG ENGAGEMENT FROM LEADING MACHINE LEARNING RESEARCHERS RIGHT AS PART OF THE PROJECT TEAMS WHO ARE DESIGNING AND PLANNING TO ANALYZE THESE NEW DATA SETS AND IF THE PROJECT REVIEW PROCESS INCORPORATES EXPERTISE IN MACHINE LEARNING AS WELL AS IN THE TRADITIONAL BIOMEDICAL DOMAINS THEN A LOT OF THE OTHER ATTRIBUTES OF WHAT SHOULD THESE NEW DATA GENERATION EFFORTS DO, WHAT SHOULD THEY FOCUS ON, THAT WILL FALL NATURALLY. AND WILL LEAD TO THE DATA THAT WILL FUEL MULTIPLE ANALYSES AND TO THE TEST BEDS TO DRIVE GENERAL UPGRADES AND TOOLS AND STANDARDS AND PROCESSES MANY OF OUR SUBSEQUENT RECOMMENDATIONS ARE ABOUT THE NEW CRITERIA AND NEW MECHANISMS WE THINK WILL BE THESE DATA GENERATION EFFORTS WILL BE THE TEST BEDS IN WHICH THAT HAPPENS. NOT IN A PRESCRIPTIVE SENSE BUT IN A OPENING SENSE TO SAY WHAT ARE EXAMPLES DATA GENERATION EFFORTS. NO DETAIL BUT THIS IS ONE OF THE EASIEST SECTIONS OF REPORT AND ONE OF THE EASIEST SLIDES BECAUSE SOON AS WE TOSSED OUT TO THE GROUP, EVERYONE HAD TWO OR THREE WOULDN'T IT BE GREAT TO HAVE A DATA SET THAT LET ME DO THIS. AND I WOULD BE DOING IT TODAY BUT KNOWING THE DATA SETS OUT THERE TODAY ARE LETTING ME DO WHAT I WANT TO DO. ON MY AREA OF RESEARCH. FOR THOSE PEOPLE SAY WHAT IT WAS AND WE CAN SUMMARIZE THOSE GAPS IN CRITERIA, BUT THESE ARE PLACES YOU CAN GET A START WITH DATA PEOPLE HAVE SEEN OUT THERE BUT ONLY A START. THIS IS WHY THIS IS OUR RECOMMENDATION. THE NEXT SLIDE IS ONE THAT I ADDED LARRY'S REQUEST. WHEN THESE RECOMMENDATIONS WERE PREVIEWED THIS WAS A VERY LOUD QUESTIONS THAT PEOPLE ASKED AND CORRECTLY. IT'S AN IMPORTANT QUESTION. IT IS NOT AN EITHER OR, IT'S ABOUT WHAT ARE THE NEW CAPABILITIES THAT YOU CAN'T GET TO WITH TODAY'S DATA AND CAN WITH NEW DATA. AND HOW FAST AND TO WHAT EXTENT DO WE WANT TO INVEST IN THAT. NOT TO SAY THERE IS A IS NOT HUGE VALUE IN DATA AND WON'T CONTINUE TO BE, OF COURSE THERE WILL, WE TALK ABOUT THE EFFORTS UNDERWAY THE TRY THE TAKE THE EXISTING DATA AND ADDRESS SOME OF THE GAPS BETWEEN IT. BUT AS SUSAN POINTED OUT, THERE'S CHALLENGES IN THAT AND THE AMOUNT OF EFFORT TO TAKE PRE-EXISTING DATA SETS ONE PERSON IN THE WORKING GROUP SAID, I OFTEN FEEL LIKE I GO TO BIOLOGIST AND I WANT DATA. I HAD GREAT DINNER ON MY DATA PROJECT AND PUT ALL THE SCRAPS IN THE COMPOST HEAP HERE. KNOCK YOURSELF OUT. I HAVE LOTS OF DATA. THAT IS SLIGHTLY OVERSTATING IT BUT JUST SLIGHTLY. IT WAS THE DATA DESIGNED FOR SET OF PURPOSES MAY NOT HAVE BEEN WELL DESIGNED OR ANTICIPATE THE NEEDS THAT WEREN'T OBVIOUS AT THE TIME. WHEN SUSAN PRESENTED WE HAVE TWO DIFFERENT DATA SETS AT MINI WE PROBABLY KNOW THE SAME PEOPLE BUT NOT JOINABLE. WHAT WERE WE THINKING? WE WEREN'T BECAUSE THERE WERE TWO INDEPENDENT PROJECTS DOING GREAT WORK TO SOLVE INDEPENDENT PROBLEMS. OBVIOUSLY FROM THE MIND SET THIS PRESENTATION HOPEFULLY LEAVES YOU WITH, YOU WOULD NEVER DO THAT, YOU WOULD ONLY SAY I WANT TO GATHER INFORMATION IN A WAY THAT SUPPORTS THE MOST POSSIBLE DISCOVERIES AND USES. TAKE MY -- OSCOPY EXAMPLE EARLIER, IF YOU ARE STUDYING EYE DISEASE WHY BOTHER RECORDING THE CARDIOVASCULAR HEALTH OF THE PARTICIPANTS? BECAUSE MAYBE THERE'S SOMETHING YOU WEREN'T EXPECTING. THOSE ARE KINDS OF THINGS THAT ARE HARD TO RETROFIT. I WILL ALSO ADD I FORGET WHO I SHOULD CREDIT BUT SOMEBODY ON THE ACD SUGGESTED THIS YESTERDAY, ALSO THAT THERE'S ALSO THE OPPORTUNITY AFTER YOU HAVE WITH NEW DATA SETS DESIGNED FOR INFERENCE YOU HAVE BUILT MODELS AND YOU CAN USE MACHINE LEARNING TO FIND MODELS IN THAT DATA DESIGNED FOR PURPOSE, YOU CAN TAKE THOSE MODELS AND APPLY THEM TO OLDER MESSIER DATA AND FIND THINGS. YOU CAN GO BACK AND IMPUTE THINGS. FROM THE CONSUMER SPACE, WHEN YOU TAKE A PICTURE OF SOMEBODY IN THE BACKGROUND IS BLURRY AND THEY POP IN THE FOREGROUND, MODERN PHONES THE GOOGLE PIXEL HAD FOR LITTLE WHILE HAD PORTRAIT MODE, IF I TAKE A PICTURE IN PORTRAIT MODE IT DOES A COUPLE OF IMAGES AND GIVES ME THAT NICE WHICH IS BEAUTIFUL AND LIKE EVERYTHING LOOKS BETTER IN PORTRAI MODE. SO THAT IS A NICE FEATURE. ONCE THAT HAD BEEN OUT THERE FOR A YEAR, THE PEOPLE ON CAMERA TEAM TOOK BUNCH OF EXISTING PICTURES AND TRAIN THE MACHINE LEARNING MODEL AND NOW A FEATURE RELEASED THIS MONTH, RELEASED WHERE IN THE -- AFTER THE FACT YOU CAN TAKE A PICTURE THAT YOU TOOK WITH OUT ANTICIPATING THAT YOU WERE GOING TO DO THIS AND THE MACHINE LEARNING MODE FIGURES WHICH PIXELS WERE FOREGROUND AND BACKGROUND AND THAT MODEL CAN ADD THAT NICE ENHANCEMENT AFTER THE FACT. YOU COULDN'T HAVE DONE THAT WITH THE EXISTING PICTURES BUT SOMEONE IS GOING TO TAKE A PICTURE OF ME AND SEE IF THIS IS TRUE. THAT S AN EXAMPLE HOW SOMETIMES MODELS BUILT ON NEW DATA CAN BE USED TO SQUEEZE MORE VALUE OUT OF EXISTING DATA. THE DETAILS OF DATA SETS SPECIFIC. ENOUGH ELEPHANTS. OUR SECOND RECOMMENDATION IS TO FORMALIZE THINGS I HAVE BEEN TALKING ABOUT CHALLENGES AND OPPORTUNITIES WHAT MAKES A DATA SET, WHAT ARE THE ATTRIBUTES OF DATA SET THAT MAXIMIZE ITS VALUE TO FUTURE DOWNSTREAM USE. WE LAID OUT WHAT WE THINK ARE SOME AREAS THE CRITERIA ARE LIKELY TO FALL BUT WE THINK WOULD BE A GOOD TASK TO REFINE AND PUBLISH THESE CRITERIA, TO EVALUATE EXISTING DATA SETS AGAINST CRITERIA TO SAY WHICH ARE ALREADY CHECKING THREE BOXES SIX BOXES NONE OF THESE BOXES AND TO USE THAT ONCE CRITERIA REFINED WITHIN A COUPLE OF YEARS START TO MAKE REQUIREMENTS FOR ALL NEW DATA GENERATION, START TO SAY WHEREVER YOU ARE GENERATING DATA YOU SHOULD ALWAYS DO THIS AND THIS AND THIS AS PART OF THE SEPARATING DATA. IT'S A LITTLE PREMATURE TO MAKE MANDATES BUT IT IS NOT PREMATURE TO ARTICULATE THEM IN A WAY THAT PEOPLE WHO AREN'T MACHINE LEARNING EXPERTS LOOK AT CRITERIA, AND SAY I UNDERSTAND WHAT I SHOULD BE DOING IN DESIGNING MY NEXT DATA GENERATION PROJECT TO MAXIMIZE THE LIKELIHOOD MACHINE LEARNING COMMUNITY HELPS WHAT I WANT TO DO TO GET VALUE FROM MY DATA. THIS ONE WE TALKED ABOUT IN THE INTERIM REPORT, I WILL BE BRIEF BUT THERE ARE EARLY STANDARDS AROUND DATA SHEETS LABELS FOR SETS OF DATA AND MODEL CARDS LABELS FOR TRAIN MODELS BUILT BY MACHINE LEARNING ALGORITHMS ON TOP OF DATA. WE THINK'S AN OPPORTUNITY TO REALLY REFINE THOSE STANDARD AND EXTENTND APPLY TO BIOMEDICAL DATA SETS SO ONE OF THE MOST OBVIOUS USES IS AROUND UNINTONINGSAL BIAS IN THE POPULATIONS REMIT IN THE DATA SET, LET'S HAVE A STANDARD WAY TO DESCRIBE POPULATIONS REPRESENTED IN THE DATA SET SO USING THAT DATA SET FOR DOWNSTREAM PREDICTION WILL KNOW WHAT THEY ARE AND AREN'T GETTING. DO THE SAME FOR MODELS AND THEN TEST THIS OUT IN THE REAL WORLD. FOURTH RECOMMENDATION AROUND CONSENT AND DATA ACCESS STANDARDS, THIS COMES DIRECTLY FROM THE CHALLENGES WE TALKED ABOUT EARLY AND WE THINK THERE'S REALLY A NEED TO RECONCILE THESE THREE DIFFERENT STREAMS OF THINKING AROUND HOW CONSENT AND DATA ACCESS WORKS. WHAT IS COMMON IN THE ML COMMUNITY GOOD AT SOME THINGS BUT DOESN'T ANTICIPATE, TWO EXISTING BIOMEDICAL BEST PRACTICES AND THREE, AND THIS AGAIN TAPS INTO SOME OF THE POINTS SUSAN MADE EARLIER AROUND RESEARCHER ACCESS, THERE'S ONGOING WORK TO HARMONIZE CONSENT AND DATA ACCESS AND DATA USE TO MAKE IT EASIER FOR RESEARCHER ONCE THEY HAVE SHOWN I'M A BONA FIDE RESEARCHER, I AM TRUSTABLE TO WORK WITH DATA OF THIS TYPE. TO HAVE THAT BE A ONE AND DONE ACROSS MULTIPLE DATA SETS SO YOU œDON'T HAVE TO -- SAME WAY YOU DON'T WANT TO LOG IN OPERATE SEPARATELY OR EACH DATA SET, FOR LARGE SETS OF DATA SHOULD BE POSSIBLE TO FIND STANDARDS BUT IT HAS TO BE DONE CAREFULLY TO BALANCE THAT'S WHY THREE DIFFERENT INPUTS TO BALANCE TO EASE OF RESEARCHER ACCESS WIMAXMIZES THE VALUE OF THE DATA WITH THE PROTECTION OF THE DATA, TO MAKE SURE WE ARE HONORING THE PARTICIPANTS AND THEIR DONATION OF THEIR INFORMATION. FIFTH RECOMMENDATION IS TO TAKE A STEP FORWAR FORMALIZING CAPTURE THE ETHICAL PRINCIPLES THAT ARE NEW AND BEYOND EXISTING PRINCIPLES FOR ETHICS IN BIOMEDICINE, WHAT ARE THE ETHICAL PRINCIPLES THAT ARE ADDED BY THE CHALLENGES OF WORKING WITH MACHINE LEARNING IN BIOMEDICINE AND WE WENT THROUGH SUMMARIZE ON THE SLIDE IN MORE DETAIL IN THE REPORT. SEVERAL AREAS WHERE WE THINK IT'S MOST LIKELY TO NEED ENHANCEMENTS AND REFINEMENTS TO THE EXISTING PRINCIPLES AROUND BIOMEDICAL ETHICS TO ADDRESS SPECIAL NEEDS OF MACHINE LEARNING. SIX BACK TO THE PEOPLE. WE STILL BELIEVE THE BEST WAY TO ATTRACT THE RIGHT PEOPLE IS TO GENERATE THE RIGHT DATA. BUT THAT'S NOT ENOUGH. I BOLDED THE KEY GOAL OF THIS RECOMMENDATION, HEALTH EXPERTS FROM ALL FIELDS SUCCESSFULLY COLLABORATE ACROSS DISCIPLINES. THAT REQUIRES A VARIETY OF DIFFERENT CURRICULA AND TRAINING MATERIALS. THE TRAINING MATERIALS FOR PEOPLE ON COMPUTATIONAL TRACKS IS FIRST RAISE AWARENESS WHAT IS NEEDED, WHAT IS USEFUL AND POSSIBLE, IN BIOMEDICINE, SECOND, GIVE THEM HANDS ON OPPORTUNITIES TO SAY HERE IS WHAT IT'S LIKE TO USE SOME OF THE COMPUTATIONAL TOOLS YOU ARE LEARNING AND APPLY THEM TO BIOMEDICAL DATA. HERE ARE THINGS TO THINK ABOUT, HERE ARE THINGS THAT MIGHT BE DISCOVERED. CONVERSELY, PEOPLE WHO ARE IN BIOMEDICAL TRACKS EITHER ALL THE WAY FROM UNDERGRADUATE UP THROUGH GRADUATE AND ONGOING PROFESSIONAL EDUCATION, HOW CAN WE HELP THEM UNDERSTAND WHAT'S POSSIBLE, WHAT IS EASY, WHAT IS HARD, WHAT SORTS OF QUESTIONS THEY SHOULD BE ASKING OR COMPUTATIONAL TEAM, HOW THEY CAN WORK TOGETHER TO SAY HEY I WOULD LIKE TO DO THIS ON MY DATA, IS THAT A REASONABLE QUESTION? WHAT IS THE RIGHT WAY TO FACILITATE THOSE CONVERSATIONS? I HAD A CONVERSATION AT THE BREAK ABOUT -- I WON'T GET THE NAME OF THE INSTITUTE RIGHT BUT NURSING INSTITUTE IS PUTTING TOGETHER AN AI BOOTED CAMP FOR LEADERS IN THEIR FIELD, THAT IS THE KIND OF EFFORT WE THIRD THE NIH SHOULD MAKE EASY AND FACILITATE MATERIALS TO MAKE IT EASY TO INFORM BIOMEDICAL EXPERTS WHAT THEY NEED TO KNOW ABOUT MACHINE LEARNING AND VICE VERSA. SO COMPLIMENTARY BUT DIFFERENT MATERIALS TO HELP EXPERTS TALK TO EACH OTHER. SEVEN IS A QUICK ONE, THIS IS ONE OF THE DO NOWS OVER SUMMER THERE ARE SET OF FELLOWSHIP TRAINEE PROGRAMS THAT WORK ON MACHINE LEARNING, A QUICK SUMMARY OF PROJECTS IN THE REPORT, IT WENT WELL, KEEP IT UP, STANDARD PART ONGOING FOCUS ON MACHINE LEARNING AS ONE OF THE DOMAINS WORKED ON BY THESE PROGRAMS, FRANCIS TALKED ABOUT THIS WHICH ARE PIE HOTTING NOW THE IDEA OF INCLUDING BIOMEDICAL TRACKS, EXCUSE ME COMPUTATIONAL CONFERENCES. WE THINK THAT SHOULD BE EXPANDED AND THE CONVERSE SHOULD HAPPEN TO INCLUDE SOME MACHINE LEARNING TRACKS ATOPY MEDICAL CONFERENCES. I WILL READ THE LAST TWO PARAGRAPHS OF THE REPORT THAT SUMMARIZE WHAT -- WHY OUR INITIAL EXCITEMENT WAS REINFORCED OVER THE COURSE OF THE LAST YEAR. RECENT ADVANCES IN DATA GENERATION AND ANALYSIS BROUGHT BIOMEDICINE TO THE CUSP OF A NEW WORLD OF ML BIOMED. THE COMPUTATIONAL AND BIOMEDICAL COMMUNITIES ARE POISED TO DRIVE TRANSFORMATIVE PROCESS AND PIE MEDICAL RESEARCH LEADING TO NEW INCITES HOW EXISTING SYSTEMS WORK AND IN CARE DELIVERY LEADING TO IMPROVEMENTS IN HEALTH OF ALL HUMANS AND COMMUNITIES. THE NIH IS WELL-POSITIONED TO ACCELERATE THAT PROGRESS BY SUPPORTING THE THREE COMPLIMENTARY AREAS OF DATA, TO FUEL THE ANALYSIS ENGINES, ETHICS, TO ALWAYS BE STEERING IN ACCORDANCE WITH HIGHEST VALUES, AND PEOPLE, TO DRIVE PROJECTS FORWARD. FROM THE EIGHT RECOMMENDATIONS IN THIS REPORT SUGGESTS SPECIFIC WAYS TO PROPEL CONGRESS, WE LOOK FORWARD TO SEEING THE RESULTS UNFOLD. WITH THAT I WOULD LIKE THANK THE WORKING GROUP FOR ALL THE EFFORTS, VARIOUS GUESTS ATTENDED, THE PROCESS AND I'M HAPPY TO TAKE QUESTIONS ON ANYTHING. PROBABLY BEFORE DOING THAT, IF ANY MEMBERS OF THE WORKING GROUP START WITH DINA SHE'S IN THE ROOM BUT DINA AND OTHERS ON THE PHONE WANT TO ADD ANYTHING BEFORE WE OPEN IT UP. >> THANKS FOR THE GREAT PRESENTATION AND FOR CONVEYING ALL THE IDEA THAT THE WORKING GROUP HAS WORKED ON. I WANT TO ADD ONE THING. I THINK THE DATA GENERATION WAS LIKE THE BIGGEST MOST IMPORTANT TOPIC, THE ML FRIENDLY DATA GENERATION THIS ALLOWS NOT JUST SOLVING NICHE APPLICATIONS BUT ALLOWS FOR TRANSFORMATIVE CHANGES. I ENCOURAGE NIH TRADITIONALLY IS KNOWN FOR GENERATING THINGS LIKE BIG PROJECT, BIG TRANSFORMATIVE THINGS, THINKING BIG. THIS IS ONE DOMAIN WHERE THERE'S A NEED FOR DOING SOMETHING SIMILAR WHERE THE LEADERSHIP IN THE NIH WITH THE COMMUNITY CAN COME UP MAYBE WITH SOME INITIATIVE AROUND BRINGING MACHINE LEARNING AS A FIRST CLASS CITIZEN, WITH BIOLOGISTS, MEDICAL DOCTORS, PEOPLE WHO UNDERSTAND ETHICS CREATING INITIATIVES IN A DOMAIN THAT MAKE A DIFFERENCE IN BIOMEDICAL APPLICATION. SO JUST TO GIVE A CONCRETE EXAMPLE, MAYBE LIKE CREATING INITIATIVES AROUND SOLVING CAR MANAGEMENT IN MULTIPLE DISEASE AREAS. FOR EXAMPLE, IF YOU THINK AUTOIMMUNE DISEASE, AND BRING IN THE MACHINE LEARNING TO UNDERSTAND PATIENT THEIR JOURNEY, THEIR PHYSIOLOGICAL DATA AND CLINICAL DATA WITH GENOMIC DATA AND FROM THAT COMBINING THE MODELS AND THE DATA TOLL CREEIATE INSIGHTS THAT ADVANCE THE CARE, MEDICAL CARE AS THE PATIENT COULD BE IN DIFFERENT AREA SUCH AS ALZHEIMERS FOR EXAMPLE OR SOME OTHER. >> ANYONE ON THE PHONE FROM THE WORKING GROUP WANT TO ADD A COMMENT? >> THIS IS MICHAEL. CAN YOU HEAR ME? >> HI, MICHAEL. >> DR. I WANT TO ADD THAT ALL WHAT YOU PRESENTED, TODAY IS REFLECTION OF WHAT THIS TEAM, AI WORKING GROUP WAS ABLE TO PULL TOGETHER BECAUSE WE HAD DIVERSE REPRESENTATION AND LOTS OF OVERLAP BECAUSE OF THAT TEAM WE PULLED THIS REPORT TOGETHER TO HAVE ALL THIS OUTPUT. IT REPRESENTS TO ME WHAT WE ARE TRYING TO FIND OUT IN THE MARKET. IN THE REAL WORLD TALKING EXPERTS, THIS COMMITTEE OR WORKING GROUP SHOWED THAT THE POWER OF THAT COMBINATION. IT IS AN HONOR TO BE PART OF THIS, THAT'S ALL I HAD TO ADD. >> THANK YOU. NOT SURE IF YOU JOINED IF YOU WANTED TO ADD ANYTHING? WILL THE'S OPEN IT UP TO THE ACD. LINDA. >> FROM CONGRATULATIONS, THIS IS AN AMAZING STUDY, FABULOUS TO SEE NIH IS DOING THIS. I APPLAUD THESE DIRECTIONS. I ALSO WANT TO PROVIDE REFLECTION ADS A PERSON WHO IS TRILINGUAL, I WOULD HAVE RAISED MY HAND BUT THERE IS A THIRD LANGUAGE I KNOW, IT'S CALCULUS. OTHER KIND OF MATH THAT REPRESENT PHENOMENA THAT CAN BE SOMEWHAT AT LEAST DESCRIBED IN DETERMINISTIC MANNER. WHEN I LOOK AT WHAT WE ARE DOING WITH THE BIG DATA, IT IS TRANSFORMED MY LIFE. I WAS TRAINED AS A CHEMICAL ENGINEER AND DID A LOT OF MODELING, BIOLOGICAL SYSTEMS, EVEN AS A GRADUATE STUDENT TRANSPORT AND REACTION OF NEW GROWTH FACTORS AND THINGS IN CELL CULTURES. THE KIND OF COMPUTING POWER THAT CAME ABOUT IN THE PAST YEAR SINCE I GRADUATED, UNDERGRADUATE STUDENTS NOW DO THE KIND OF COMPUTING, ON A DIFFERENT MODELER I SPENT HALF MY Ph.D. DOING BY NUMERICAL METHODS AS A GRADUATE STUDENT. I LEFT CHEMICAL ENGINEERING AFTER I GOT TEN YOUR BECAUSE MY RESEARCH MOVE TO A REALM WHERE I NEEDED MUCH MORE MULTIPLEX KIND OF DATA, IT WASN'T DETERMINISTIC SO I AS A CHEMICAL ENGINEER WHO REALLY COULD TEACH ABOUT THE INTEGRAL MINIMUM APPROXIMATION BOUNDARY LAYER HAD TO LEARN LOT IN MATH BECAUSE I'M CONSTANTLY LEARNING NEW KINDS OF MATH. WE USE NON-NEGATIVE MATRIX FACTORIZATION TO GAIN INSIGHTS FROM SAMPLES FROM ENDOMETRIOSIS PATIENTS AND PUBLISHED TWO PAPERS. WE ALWAYS LEARN NEW KINDS OF MATH. BUT WE ARE NOT THREING AWAY THE KIND OF -- THROWING AWAY THE KINDS OF MATH WE USE SO I USE CALICULUS ALMOST EVERY DAY, THIS AFTERNOON I WILL DO A MODEL THAT'S A CALCULUS MODEL TO INTERPRET DATA FROM ONE OF OUR TISSUE CHIP MODEL SYSTEMS ON LIVER INSULIN RESISTANCE MODEL. AND WHAT CONCERNS ME ABOUT WHAT THE WAY THIS IS PRESENTED IN THE REPORT IS THAT PERSON WHO UNDERSTANDS THERER FISCAL BIOCHEMICAL PHENOMENA THAT MUST BE INTEGRATED IN WAYS ACROSS MULTIPLE LINKS AND TIME SCALES INVOLVING OMIC DATA SETS OF THE SORT WE ARE TALKING THIS MORNING, THAT KIND OF PERSON IS NOT EXPLICITLY INCLUDED AND IS SO NECESSARY. I'M WORRIED NIH IS NOT CULTIVATING THE PERSON WHO MIGHT BE CALLED MULTI-SCALE MODELING. PEOPLE WHO REALLY ARE CONVERSING IN CALCULUS BUT ALSO MANY THE NEW MATH BECAUSE ENGINEERS DON'T CARE WHAT KIND OF MATH, MACHINE LEARNING OR CALCULUS. I AM VERY CONCERNED ABOUT THAT. I'M EXCITED ABOUT THIS EFFORT BECAUSE I NEED MACHINE LEARNING, I NEED THESE KIND OF DATA SETS AND EVERYTHING I DO. I AM SO EXCITED FOR THE RESOURCES BUT I ALSO FEEL THE PROGRESS IS SLOWER AND WEAK AND THE CAVEATS YOU PUT ABOUT MACHINE LEARNING CAN MAKE IT WORSE. THE RELIANCE WE CORRELATE IS UNLESS WE NURTURE THE TRILINGUAL PEOPLE WHO HAVE BIOPHYSICAL INSIGHTS THAT COME WITH KNOWING HOW TO PUT TOGETHER MULTIPLE INSTICTIVE INSIGHTS FROM PHYSICS AND CHEMISTRY THEY WERE TRAINED ON THAT INVOLVE CALCULUS ON LINK AND TIME. >> INTERESTING POINT. DINA WANTS TO COMMENT. >> THANKS, LINDA. SO IF THIS CAME ACROSS THAT IT'S IGNORING THAT EXPERTISE, IT'S NOT. BY SAYING YOU WANT TO BUILD PROGRAMS FROM THE GROUND UP, INTEGRATING MACHINE LEARNING AND EXPERTISE IN THE DOMAIN AND MODELING OF THIS SYSTEMS, BY BIOLOGICAL SYSTEMS, THAT IS THE INTENTION. JUST TO COMMENT ON WHAT YOU SAID ABOUT MODELS ON MATH MATH AND CALCULUS, ALL THAT STUFF. WHEN THIS SIMPLE MACHINE LEARNING MODEL, THE MODELS ARE NOT DONE BY EXPERTS ARE SIMPLE JUST LIKE YOU HAVE DATA YOU TRAIN TO ADDITIONAL MODEL AND YOU GET -- BUT REALLY THERE ARE ADVANCE MODEL THAT CAN ENTRY GRATE INFORMATION ABOUT THE DOMAIN ABOUT THE INTERACTION AND THE UNDERSTANDING OF SOMEBODY WHO UNDERSTAND THE COMPLEXITY INTO THE MODEL. FOR THAT YOU NEED TO WORK DIRECTLY DEVELOPING MACHINE LEARNING MODEL SO HE CAN TAKE MATHEMATICAL EQUATION AND EXPRESS THEM IN THE MODEL IN A COHESIVE WAY. >> SO MY SUGGESTION IS TO BE EXPLICIT. SO THERE IS A CONTINUUM FROM PEOPLE WHOLE ARE GREAT LIKE YOU AND REGINA AND DAVID AND THEN BIOLOGISTS AND CLINICS AND THEN THERE'S ME WHO BOUNCE BACK, NOT GOING TO DO NEW ALGORITHMTOR MACHINE LEARNING BUT I WILL IMPLEMENT THINGS AND PROVIDE THAT KIND OF INSIGHT LIKEWISE THE CLINIC OR BIOLOGIST. THIS IS WHY I SPENT A LOT -- I HAVE A HUGE LAB BUT SPENT TIME DEVELOPING UNDERSTOOD GRADUATE CURRICULUM THAT DOES WHAT I DESCRIBED BECAUSE WE DO NEED THIS MIDDLE TRANSLATOR BETWEEN REALMS OF DATA, IT GOES THROUGH PHYSICAL BIOPHYSICAL SYSTEMS, I GUESS YOU COULD CALL IT SYSTEMS ENGINEERS. SO AGAIN, ENGINEERS ARE PROBLEM DRIVEN OFTEN. SO WE WILL TAKE COMPLICATED PROBLEMS AND THINK HOW DO I PULL IN AND AGAIN, THIS IS AMAZING BUT I REALLY WORRY WE HAVE FORGOTTEN THAT MIDDLE TRANSLATOR AND IT WILL BE HARDER UNLESS WE NURTURE THOSE PEOPLE. >> BRENDAN AND JOSE AND JEFF. >> CONGRATULATIONS TO YOUR WORK GROUP I LEARNED A LOT. CLEARLY YOU MAKE THE CASE OF TRANSFORMATIVE. I WANTED TO ASK ABOUT THE CLINICAL LOOK PART YOU RAISED ABOUT BIAS. AND THAT TOUCHED A CORD ESPECIALLY WITH THE EXAMPLE YOU USE OF EHR DATA AND DIAGNOSTIC CODING. IN PRACTICE EVERY DAY,S THAT GETS MANIPULATED BECAUSE OF BILLING. YOU GIVE A DIAGNOSIS SOMETIMES TO A PATIENT WHO MAY OR MAY NOT HAVE THAT SO YOU CAN ACHIEVE A CERTAIN GOAL. IT UNDERSCORES THIS ISSUE OF INHERENT NATURE OF THE DATA AND I THINK AN EXAMPLE THAT'S WE ARE AWARE OF IN THE GENOMICS WORLD WE THINK ABOUT HISTORY OF GENOME WIDE ASSOCIATION STUDY AND INHERENT STRATIFICATION POPULATIONS WHICH OCCURS NATURALLY, THE DATA FROM THAT OF COURSE IS SPECIFIC FOR THAT POPULATION. THIS IS WHY REPLICATION AND OTHER DIVERSE POPULATIONS IS SO IMPORTANT IN THAT AREA SO I SEE SIMILARITIES IN THAT SAME ISSUE. I DIDN'T SEE IN RECOMMENDATIONS HOW WE OVERCOME THAT, ADDRESS THAT. ARE THERE STRATEGIES FOR VALIDATION, ORTHOGONAL IN TERMS OF APPROACH? HOW DO WE ENSURE OR MINIMIZE THAT RISK IN TERMS OF THE -- BECAUSE THERE'S BIAS IN ALL DATA SETS OUT THERE. >> I DON'T THINK THERE'S A SINGLE SILVER BULLET. I THINK THE EASY ANSWER NO ONE WILL BE SURPRISED COMPUTATION IS MORE DATA. BECAUSE THE MORE DATA THE MORE THE NOISE CAN START TO WASH OUT. SO THAT'S PART OF THE THE ANSWER. PART OF THE ANSWER IS THE NOTION OF DATA SET LABELING. SO YOU AT LEAST KNOW WHAT THE RESTRICTIONS ARE WITH WHAT YOU ARE GETTING. PART OF THE ANSWER IS MULTI-MODAL. IT'S LIKE YEAH, MAYBE ACTUAL INFORMATION IN THE ICD 10 CODES WAS DISTORTED BY MULTIPLE SIGNALS BUT THERE'S ALSO NOTES IN THERE. THOSE NOTES ARE DISTORTED IN A DIFFERENT WAY. MAYBE TOGETHER. AND THAT'S ALSO ACTUAL LAB READINGS. AND SO MAYBE THAT. IF YOU ADD IN THE OPPORTUNITY TO SAY FOR SOME POPULATION I'M ALSO GOING TO TAKE THEIR BEDSIDE MONITOR AND WILL FEED ALL THAT DATA. SO I >> PET RAW INFORMATION WHAT IS HAPPENING IN ADDITION TO THE -- RIGHT? SO THE MORE DIFFERENT SIDE YOU COME FROM, THE MORE TRIANGULATION IS POSSIBLE THAT CAN EVENTUALLY, I HAVE SEEN STUDIES, WHERE PEOPLE TAKE EHR RECORDS SANDS SAY FIGURE OUT WHICH OF THESE PEEP HAD A PARTICULAR HEART CONDITION. YOU START BY OF COURSE LOOKING FOR THE CODE BUT THEN THE IDEA IS CAN WE BUILD A MODEL THAT WILL PREDICT ACCURATELY WHICH PEOPLE DO. YOU GET SOME FALSE POSITIVES AND FALSE NEGATIVES FROM THE CODE. THE ANSWER IS YES YOU CAN MAKE HEADWAY ON THAT, SO THAT CHART REVIEW WILL LARGELY MATCH WHAT THE MODEL SAYS IS IN THE EHR BECAUSE YOU LOOK AT LOTS OF ASPECTS, NOT JUST AT THE CODE, IN THEORY IS THE ANSWER. SO THOSE ARE ALL PARTS OF ANSWER. >> THANKS. >> THANKS FOR A GREAT PRESENTATION. I WILL ASK THIS QUESTION TO DINA BECAUSE I CAN SEE A LITTLE MORE CLEARLY. I THINK YOUR POINT ABOUT INVOLVING STRONG ENGAGEMENT FROM LEADING ML RESEARCHERS WAS RIGHT ON POINT. I WONDER IF YOU TALKED ABOUT THE LOGISTICS HOW TO DO THAT, DID YOU ENVISION GRANT MECHANISMS THAT INCLUDED FULL TIME EMPLOYMENT FOR SOMEBODY IN THAT INDUSTRY OR THEY WOULD BE HIRED BY UNIVERSITY. I STRUGGLE WITH THIS MYSELF. WONDER HOW TO MAKE IT HAPPEN. >> THIS IS A KEY PROBLEM. ONE OF THE BIG ISSUES IS -- DIFFERENTLY WHAT ATTRACTS PEOPLE IS BASICALLY A GLOWING CAREER. THE STRONGEST ARE ATTRACTED IF THIS IS APPROXIMATE OPPORTUNITY TO MAKE -- WONDERFUL CAREER. TO DO THAT, TWO THINGS, ONE IS RESOURCES. IF THERE IS MONEY AND RESOURCES, I PUT ON A BIG PROBLEM, THAT IS GOING TO ATTRACT PEOPLE BECAUSE THEY THINK THEY CAN DO SOMETHING. THE OTHER THING IS REALLY COMMUNITY. THERE IS INTEREST IN THE MACHINE LEARNING COMMUNITY AND COMPUTATIONAL COMMUNITY IN HEALTH RELATED PROBLEMS. NOW SO THAT INTEREST IS STRUGGLING BECAUSE IF DATA BECAUSE IF THERE IS NOT ENOUGH RESOURCES, NIH DOES NOT TRADITIONALLY FUND MACHINE LEARNING PEOPLE, ALL THESE THINGS. BUT ONCE YOU BRING A COMMUNITY STRONG RESEARCHERS, IS LIKE YOU START WITH FEW THINGS BUT THEN THAT CREATED COMMUNITY ALL STRONG WORK ON THOSE PROBLEMS. IF THINGS ARE GOING TO HAPPEN IN THIS DOMAIN, THEY NEED RESOURCES. AND THEY NEED BRINGING A FEW KEY LEADERS TO GRAB THE PEOPLE AROUND THEM AND CREATE A COMMUNITY AROUND FUNDAMENTAL HEALTH PROBLEMS THAT CAN BE SOLVED TOGETHER. THAT I THINK TO BE SUCCESSFUL CANNOT BE DONE ALONE WITH MACHINE LEARNING PEOPLE. THE HOPE IS CREATE A SUBFIELD EVENTUALLY WHERE PEOPLE LIKE AS DAVE WAS SAYING IF I ASK PEOPLE WHO ARE THE EXPERT ON MACHINE LEARNING, AND BIOMEDICAL OR BIOENGINEERING, THEN THERE ARE PEOPLE WHO ARE GOING TO HAVE THEIR HANDS UP FOR MULTIPLE OF THESE THINGS. THAT IS A PROCESS. AT THE BEGINNING, THE NIH IS MANY THE POSITION OR MAYBE ONLY ENTITY THAT HAVE RESOURCES POSITION AND THE DATA AND THE WEIGHT TO PULL THOSE THINGS TOGETHER. >> DO YOU THINK SHORT TERM SOMEBODY WOULD COME LEAVE INDUSTRY JOB AND BECOME A FULL TIME EMPLOYEE OF THE NIH? OR OTHER INSTITUTION? MAYBE THEY TAKE LEAVE OF ABSENCE. >> WE HAVE -- NOT REALLY BECAUSE IF THAT PERSON IS REALLY STRONG MACHINE LEARNING PERSON OR COMPUTATIONAL PEOPLE, GETTING WAY MORE SALARY AND COMPENSATION AND ALSO THEY GET PUBLISED AND RECOGNIZE MUCH BETTER GIVEN WHAT ULTIMATELY THEY CAN DO. SO UNTIL LIKE GROWING THAT CAPABILITY INSIDE NIH SO THAT PERSON CAN SAY MONEY IS A SECONDARY THING, MANY PEOPLE ARE NOT INTERESTED OR LESS INTERESTED IN MAKING PERSONAL MONEY BUT THEY NEED THE MONEY FOR RESOURCES FOR REALLY MAKING GREAT IMPACT. IF THEY FEEL IF THE PERSON STARTS FEELING THEY MAKE REALLY GREAT IMPACT, THEY ARE GOING TO& BE RECOGNIZED TO DO TRANS FORMATIVE THINGS BY WORKING ON THE NIH DATA, THE MEDICAL APPLICATION, THEN PEOPLE ARE GOING TO DO THIS. THE PROBLEM NOW IS THAT THERE ARE MANY REASONS WHY A PERSON OUR BEST GRADUATE DON'T FEEL THEY ARE GOING TO HAVE REALLY THE BIGGEST IMPACT AND THE MOST GLOWING CAREER BY WORKING ON THESE PROBLEMS BECAUSE THESE LIKE SMALL FUNDAMENTAL ISSUES OR CHALLENGES. >> I WANT TO FOLLOW UP ON THIS BY ASKING WHETHER IN ADDITION TO THE BARRIERS THAT YOU HAVE DESCRIBED IN TERMS OF THINGS LIKE SALARY AND THINGS LIKE HAVING DATA SETS THAT ARE NOT VERY WELL DESIGNED AND THIS PROBLEM OF BILINGUAL NEED WHICH IS OFTEN NOT PRESENT, YOU IS THERELING A GEOGRAPHIC ISSUE? SHOULD WE THINK ABOUT ESTABLISHING AN OUTPOST OF NIH THAT CHALLENGE AS MUCH? >> I PERSONALLY DON'T THINK THAT IS NEEDED. AGAIN, LET ME RESTATE, GRADUATE STUDENTS GET THE WORST POSSIBLE SALARY STULL THEY WORK ON THAT BECAUSE THEY CAN SEE A GLOWING CAREER AND IMPACT. IT'S LESS LIKE FOR SOME PEOPLE IT'S VERY IMPORTANT TO MAKE PERSONAL MONEY BUT THERE ARE STILL HUGE NUMBER OF PEOPLE WHO ARE REALLY STRONG AND THE SMARTEST PEOPLE ALSO THAT WHO NOT THERE FOR MONEY BUT THERE ARE DIFFERENT FOR CAREER AND IMPACT. >> I'M NOT THINKING SO MUCH SALARY, I'M THINKING PROXIMITY WHERE THIS CRITICAL MASS OF PEOPLE WHO ARE THINKING IN VERY CREATIVE WAYS ABOUT MACHINE LEARNING. WE ARE NOT PERCEIVED RIGHT NOW AS NECESSARILY PART. >> ASKING THE WRONG PERSON, I'M FROM MIT. CENTER IN BOSTON. >> FRANCIS, I THINK IT WOULD BE MATTER, IT MIGHT NOT MATTER DIRECTLY AROUND NOW WE GET TO ATTRACT SOMEONE WHO ACCEPTS AN OFFER HERE THAT WOULDN'T ACCEPT IT THERE. I'M NOT SURE THAT WOULD BE THE IMPACT. I THINK THE CIRCLES THAT I TRAVEL IN, FIRST PEOPLE DON'T KNOW WHAT THE NIH IS. THEY DON'T UNDERSTAND THE SORTS OF PROBLEMS AND OPPORTUNITIES MOST AMENABLE THAN NOT. THEY DON'T TRAVEL IN THE SAME WORLDINGS, THEY PROBABLY -- THEY REASON SYMMETRICALLY I DON'T KNOW WHAT THE NIH DOES AND I'M CLUELESS ABOUT THEM THEY MUST BE CLUELESS ABOUT ME, WHICH ISN'T A GOOD REASON REACH OUT. I THINK THERE'S AN OPPORTUNITY TO HAVE AN EMBASSY. I DON'T KNOW WHAT THE SET OF FUNCTIONS WOULD BE THERE. THAT CREATE THE CRITICAL MASS. BUT IT WOULD OPEN THE DOOR TO HOSTING VARIOUS EVENTS WITH AN NIH LOGO ON THE DOOR, MIGHT MAKE A DIFFERENCE. >> AMBASSADOR FLORES, YOU ARE COMING OFF THE ACD. >> TO ANSWER YOUR QUESTION WHAT MIGHT IT LOOK LIKE, WE DIDN'T GET DEEP INTO MECHANISMS PARTLY BOSON NONE OF US ARE EXPERTS ON MECHANISMS BUT I THINK THAT THERE ARE A LOT OF THESE PUBLIC PRIVATE EXAMPLES OF THINGS THAT THE NIH CAN WORK WELL WITH AND IF I LOOK AT ALL OF US THERE ARE PEOPLE WHO ARE CORE PARTS OF ALL OF US WHO ARE COMMERCIAL COMPANIES AS WELL AS PEOPLE WHO ARE TRADITIONAL RESEARCHERS. AND MECHANISMS AND SOCIAL PROCESSES ARE STARTING TO WORK. SO I THINK THAT IT WOULD BE VERY POSSIBLE TO DO THE SAME THING. NOT LIKE YOU NEED A 20 PERSON MACHINE LEARNING LAB. A LITTLE BIT GOES A LONG WAY TO MOVE THINGS FORWARD. >> I HAVE JOSE, JEFF, KRISTINA AND LINDA AND WE HAVE TEN MINUTES ON THE SCHEDULE BUT AND HANNAH HAD HER HAND UP TOO. SO WE WILL GET THERE. JOSE. >> ON THIS TOPIC, LEARNING WHAT WORKED IS HELPFUL SO ANECDOTAL EXPERIENCE FROM THE BROAD INSTITUTE IS PEOPLE WHO COME FROM THIS WORLD ARE HUNGRY FOR MEANINGFUL PROBLEMS TO TACKLE, I THINK BIOMEDICINE HAS THOSE PROBLEMS AND I HAVE SEEN PEOPLE WITH MATH BACKGROUNDS COMPUTATIONAL BACKGROUNDS WHO WALK THROUGH THE DOORS AND SO EXCITED DESPITE SALARIES WE OFFER GETTING INVESTED. YOU CAN IMAGINE THIS MODEL REPLICATING, THAT IS NOT NECESSARY HI NIH PACED INTRAMURAL PROGRAM SUPPORTING CENTERS WHERE THIS ACTION CAN TAKE PLACE. WE HAVE THE CHALLENGE HOW WE PAY THESE PEOPLE COMPETITIVE SALARIES ONCE ESTABLISHED. SO IT IS A BIG BURDEN ON TRADITIONAL RO1 GRANTS TO HAVE SOFTWARE ENGINEERS AND BIOINFORMATICS PEOPLE, BUT IT'S PALPABLE. THE OTHER EXPERIENCE WE HAVE IS THE NIDDK DID HAVE A T 32 FOR BIOINFORMATICS AND METABOLISM. SO THE PROBLEM I WANT TO TRACK PEOPLE WITH THOSE BACKGROUNDS INTO THEIR PARTICULAR PHENOTYPE IS GONE THREE OR FOUR YEARS, WE HAVE ONE OF THOSE IN BOSTON, INTERESTING TO KNOW HOW THE OTHERS WORK OUT, ICs WITH TRAINING GRANTS, WHAT ARE THEY PUTTING OUT AS A PRODUCT PEOPLE WHO ARE TRAINED, WHO HAVE THOSE BACKGROUNDS AND BECOME INTEREST IN A BIOLOGICAL PROBLEM. MAYBE INFORM CORP RATING CURRICULA WHICH IS WHAT YOU HAVE IN YOUR THING AND BULLETS ENTICING MEANS MECHANISMS WHICH THESE PEOPLE CAN BE TRAINED. >> I'M GUESSING THE CONCEPTS OF REPRODUCIBILITY OF RESEARCH CAME UM IN DISCUSSIONS IN YOUR WORKING GROUP. I'M WONDERING I DIDN'T SEE IT EXPLICITLY STATED ANYWHERE IN YOUR RECOMMENDATIONS, AND I CAN IMAGINE THERE MIGHT BE SOME VERY UNIQUE ASPECTS OF THIS PARTICULAR TYPE OF APPROACH CHALLENGING REPRODUCIBILITY. CAN YOU MAKE COMMENTS ON THAT? >> IT DIDN'T COME UP EXPLICITLY. MY TAKE ON THAT IS THAT IS IS IN THAT LAYER OF NECESSARY BUT NOT SUFFICIENT. THAT I TAKE FOR GRANTED THAT WE WILL CREATE AND ENABLE DATA ENVIRONMENTS THAT MAKE IT SO THAT WHEN I PUSH SAME BUTTON THE SECOND TIME I GET SAME ANSWER, AND DINA CAN PUSH THE BUTTON EVEN IF I BUILT IT. THERE'S WORK TO MAKE THAT HAPPEN. THE IDEA THAT FOR EXAMPLE, IF THE MODEL THAT I BUILD I HAVE LABELED IT WITH HERE ARE THE TRAINING SETS THAT I USE TO TRAIN THE MODEL THAT IS A HUGE STEP FORWARD. FOR TESTING MODEL, HERE IS HOW I APPLIED THIS MODEL TO THIS DATA. HERE IS HOW I DREW THE CURVE AND HOW I CALCULATE THE AUC, THAT IS EASY TO SAY MY MODEL IS AVAILABLE FOR YOU TO TRY ON YOUR DATA AND TELL ME WHAT YOU GOT. SO I THINK IT'S ESSENTIAL. I ACTUALLY THINK DRY LAB REPRODUCIBILITY IS FUNDAMENTALLY EASIER THAN WET LAB REPRODUCIBILITY. I DON'T HAVE TO WORRY WHAT KIND OF BEDDING MY MACHINE LEARNING MODEL WAS USING. THERE IS A WHOLE SET OF VARIABLES I DON'T HAVE TO. STILL HAS TO BE DONE BUT I THINK IT'S TRACTABLE. >> THANK YOU VERY MUCH, ENJOYED AS EVERYONE ELSE SAID MORE A COMMENT MAYBE RECOMMENDATION. ABOUT 25 YEARS AGO MY RESEARCH BOULDER, ANOTHER BASTION FOR AI FOR THE OUTPOST, PRODUCED THE FIRST ANNOTATED DATABASE OF CERVICAL CANCER. FROM SMEARS, CERVICAL SMEARS. SO IT DIDN'T EXIST BEFORE 1994 OR 5, IT WAS CRAZY. WHAT WE LEARNED FROM THAT BY HAVING EXPERTS TOGETHER, WE HAD MATHEMATICIAN, PATHOLOGISTS, ML, WE WERE USING POPULAR ML PROGRAM OF THE DAY WHICH IS BACK PROPAGATION, 25 YEARS AGO. WHICH LEARNED THE EXPERTS WERE IMPORTANT IN SOLVING THE PROBLEM. SO YOU FIND EVERY CANCER CELL ON THE SLIDE. THAT WASN'T TRUE, WHAT YOU WANT TO DO IS NOT HAVE FALSE POSITIVES. PATHOLOGISTS WHO ULTIMATELY MAKE THE FINAL RECOMMENDATION NEEDED TO SEE THE TOP 100 IMAGES A FEW CANCER CELLS. ON THIS SLIDE WHERE A WOMAN NEEDED TO BE TREATED, YOU MIGHT HAVE 200 CANCER CELLS, YOU JUST HAVE TO FIND 50%, IF THAT'S WHERE YOUR FALSE NEGATIVES ARE THE FALSE POSITIVES ARE REALLY SMALL SO THE RECOMMENDATION PERHAPS IS MAYBE THERE'S A MECHANISM FOR NIH TO SUPPORT ML LEADERS WHERE THEY ARE AND TAKE SABBATICAL INTO UNIVERSITIES AND WHO UNDERSTAND THE PROBLEM, BUT DON'T HAVE THOSE PARTICULAR TOOLS TO SOLVE THE PROBLEM. THERE MIGHT BE A WAY OF DRAWING THEM INTO VARIOUS GROUPS AND GETTING BENEFITS AND THEN THEY GO BACK COMPANY IS BETTER BECAUSE THEY UNDERSTAND REAL WORLD PROBLEMS, MAYBE THAT TWEAKS YOU WERE SAYING DURING YOUR PRESENTATION THE ALGORITHMS YOU USE AND OF COURSE THE INVESTIGATORS GAIN FROM HAVING PEOPLE ON CUTTING EDGE OF ML. JUST A THOUGHT. >> COMMENT, HANNAH. >> THIS IS VERY EXCITING. THANK YOU TO YOUR COMMITTEE. A COMMENT AND QUESTION, STRUCK ME HOW OFTEN YOU REFERRED TO BIAS IN THE CONTEXT OF DEVELOPING THESE SYSTEMS THAT MIGHT BE AMPLIFIED THE EFFECT OF BIAS. I WONDER WHETHER THE COMMITTEE DISCUSSED THE CONVERSE, IS IT POSSIBLE IN THE FUTURE TO ACTUALLY DEVELOP SYSTEMS OF MACHINE LEARNING TO MITIGATE THE EFFECTS OF BIAS? >> WE DID TALK ABOUT IT A LITTLE BIT. I THINK THERE'S A COUPLE OF SENTENCES MENTIONING THAT AS ONE OF THE OPPORTUNITIES. WE DIDN'T GO DEEP INTO IT BUT I THINK OBVIOUSLY THE EASY OPPORTUNITY FOR THAT, IS THE SAME REASON PEOPLE DO BLIND AUDITIONS FOR CONCERTS. IF YOU DON'T GIVE THE MACHINE THE THING YOU WANT TO IGNORE INPUT IT WON'T PAY ATTENTION TO INPUT. THAT POTENTIAL IS VERY MUCH THERE. IT HAS PROMISE AS LONG ADS YOU ARE CAREFUL THAT YOU ARE NOT HAVING UNINTENDED CONSEQUENCE ALSO. >> I'M WONDERING IF YOU CAN FOR SEE A SYSTEM WHEREBY DURING THE PROCESS OF MAKING DECISION, THERE MIGHT BE SOME KIND OF PROMPT, NEUROLOGICAL PROMPT TO AVOID THE IMPLICIT BIAS WITS IS DIFFICULT TO ALLEVIATE. >> MAYBE. INTERESTING, I DON'T KNOW. >> PATTY BRENNON. >> GOOD MORNING. DIRECTOR OF THE NATIONAL LIBRARY OF MEDICINE. CONGRATULATIONS, I'M VERY, VERY EXCITED AN'T -- ABOUT THIS REPORT. THE ISSUES WE WORK ON NATIONAL LIBRARY OF MEDICINE RELEASE AD FUNDING ANNOUNCEMENT TO, CAN RISING BIAS IN DATA SETS WHICH WILL TAKE US A DIFFERENT DIRECTION. I'M HERE TO ASK YOU A DIFFERENT QUESTION FOR MEMBERS OF THE COMMITTEE, PIPELINE AND TRAINING. WE NIH GENERAL FOCUS ON GRADUATE LEVEL TRAINING WE NEED THE STIMULATE THE K-12 COMMUNITY, STIMULATING THE HIGH SCHOOL AND COLLEGE COMMUNITY, DID YOUR GROUP HAVE ANY THOUGHTS ABOUT HOW WE AND WHO WE SHOULD PARTNER WITH TO STIMULATE A LITERATE POPULATION TO BE READY FOR THIS? >> ONLY THAT WE SHOULD. NOT A RECOMMENDATION WAS A RECOMMENDATION WAS THE NIH SHOULD. I THINK CHECKING ONE PIECE OF YOUR QUESTION, WE FELT THAT THE BIGGEST LEVER WAS NOT CREATING MORE NUMERIC PEOPLE. THE BIGGEST LEVER INITIALLY WAS TAKING THE EXISTING NUMERATE PEOPLE AND GETTING THEM INTERESTED AND LITERATE ENOUGH IN BIOLOGY THAT WE THINK THAT CERTAINLY THE EXPERIENCE ITCH OTHER PEOPLE IN THE WORKING GROUP HAD IS WHEN I MENTION TO SOMEONE I'M WORKING ON SOME BIOLOGY THING, THEY ARE EXCITED AND THEY WANT TO FIND OUT HOW THEY CAN HELP. THEN USUALLY THEY SAY WHERE IS THE DATA. I SAY COME BACK IN A FEW YEARS. THE PROBLEM IS NOT FINDING THE INTEREST. THE PROBLEM IS UNLOCKING THE INTEREST CONNECTING WITH PEOPLE WHO MOW WHERE THE DATA AND PROBLEMS WHERE WHERE THEY CAN BE HELPFUL NOW. >> BACK TO JOSE'S POINT. I THINK A GENERAL MISCONCEPTION THAT THE FIELD OF MEDICINE AND BIOLOGY AND YOU CAN HAVE THESE BIG PROBLEMS YOU CAN SAVE PEOPLE'S LIVES AND THE COMPUTER SCIENTIST AND THE ML PEOPLE WHO ARE EXCITED ABOUT THAT BECAUSE THAT'S A BIG PROBLEM. BUT YOU ARE FUNDAMENTALLY FORGETTING SOMETHING IMPORTANT. THOSE ARE PEOPLE WHO CHOSE TO NOT TO GO TO MEDICINE, NOT TO GO TO BIOLOGY, NOT TO GO TO NURSING SCHOOL BUT TO GO TO ENGINEERING OR TO GO TO COMPUTER SCIENCE. THERE BELIEVED WHAT IS BIG AND IMPORTANT IS SOLVE A PROBLEM IN MATH OR COMPUTER SCIENCE OR BUILD THIS NEW MODEL CAN HAVE MUCH MORE POWERFUL FUNCTIONALITY. OF COURSE, THERE ARE SOME PEOPLE WHO HAVE SOME BECAUSE NAY SAW SOMETHING IN THAT I SHALL PERSONAL LIFE ATTRACTED THEM TO SOLVE LIKE WE WANT TO SAVE PEOPLE, EVERYONE WANTS TO SAVE PEOPLE, ENGINEERS AND COMPUTER SCIENTISTS ARE NOT BAD PEOPLE BUT THEY FEEL THEIR CONTRIBUTION TO HUMANITY COMES ALONG A DIFFERENT PATH. IF YOU WANT TO ATTRACT THE BEST COMPUTER SCIENTISTS, BEST MACHINE LEARNING PEOPLE, THE BEST ENGINEERS, YOU HAVE TO THINK LIKE THEM. PUT YOUR PROBLEM IN THE -- WHAT THEY CARED ABOUT BEST MODELS BEST SOLVING THE DEEP MATHEMATICAL PROBLEM, THAT WHAT ATRACK THEM, YOU GET SOME WHO ARE SO MUCH MOTIVATED TO SAVE PEOPLE'S LIVES, BUT AT THE END OF THE DAY THEY WENT TO A DIFFERENT SCHOOL BY CHOICE. >> WE ARE GOING TO RUN OUT OF TIME HERE. THIS IS FUN BUT JOSE, IN TEN SECONDS OR LESS. >> I WILL ADMIT THAT MINE WAS A GENERALIZATIONS AND MY SAMPLE IS BIASED BECAUSE OF PEOPLE I SEE AT BROAD PEOPLE WHO MAKE THE DECISION THAT AFTER GOING TO SCHOOL TO BE TRAINING IN THIS MANNER AND MATHEMATICAL PROBLEMS THEY WERE DISILLUSIONED OR LOOK FOR A DIFFERENT DIRECTION SO THE PEOPLE I DO SEE MADE THAT DISCOVERY AND THE QUESTION IS HOW DO WE TAKE THAT INCREDIBLY ATTRACTIVE APPEALING CONTRIBUTION AND MAKE IT MORE ACCESSIBLE TO OTHERS. POINT TAKEN. >> THAT WAS FUN. VERY IMPORTANT CONVERSATION TO HAVE. WE PROBABLY NEED TO WIND THIS UP. BUT I JUST WANT TO SAY A SINCERE THANK YOU TO DAVID, DINA, ALL MEMBERS OF THIS WORKING GROUP, TO LARRY, TO JESS FOR THE WAYS WHICH THIS CHALLENGING TASK HAS BEEN HANDLED WITH SUCH THOUGHTFUL EXPERTISE.& BECAUSE THIS IS THE FINAL REPORT OF THE WORKING GROUP WE NEED TO HAVE A SENSE OF THE ACD ABOUT YOUR WILLINGNESS TO ACCEPT IT. THEY HAVE PRESENTED WITH SOME VERY INTERESTING RECOMMENDATIONS AND USUAL IS FOR ACD TO DECIDE WHETHER TO EMBRACE. SO I WOULD ENTERTAIN A MOTION. Q. MOVE. >> >> SECOND. >> I HEAR A SECOND. ALL IN FAVOR OF ACCEPTING THE RECOMMENDATION FROM THE WORKING GROUP. ARE THERE ANY ABSTENTIONS AS MUCH ARE THERE ANY KNOWS? -- NO? DR. BRENTON ARE YOU STILL ON THE PHONE? >> I AM. I APPROVE. >> >> GREAT. SEEMS TO BE UNANIMOUS. WE WILL HAVE A LOT OF WORK TO DO TO TAKE THESE ON BOARD AND TRY TO TURN THEM INTO ACTIONS. I DON'T KNOW IN THIS INSTANCE WE HAVEN'T SUGGESTED THAT THIS WORKING GROUP MAINTAIN THE IN EXISTING FORM, WE BEGIN TO UNDERSTAND THE IMPLEMENTATION AND HAVE INFORMAL OPPORTUNITIES ESPECIALLY WITH YOU BUT THE REST OF THE WORKING GROUP AS THE NEED ARIDESES AND WITH DINA AS PART OF THE ACD TO BE SURE THAT WE ARE ON TRACK. >> WE MIGHT CONSIDER BROADENING IT OUT PARTICULARLY SOME OF OUR ACD MEMBERS WHO ARE CYCLING OFF SO WE KNOW WHERE YOU ARE. WE MIGHT BE ABLE TO TAP INTO SOME OF -- >> I AM GETTING EMAIL PEOPLE LISTENING TO THE COMMENTS, THE MULTI-SCALE MODELING PROBLEM WAS CUT AND I REALLY THINK WE WILL TALK ABOUT IT LATER. BUT THERE IS COMMUNITY IS LISTENING AND REALLY WANTS TO -- >> GOOD. >> ALL RIGHT. THANK YOU. SIGNIFICANT MOMENT, THIS IS AN OPPORTUNITY I THINK FOR US ACROSS ALL NIH TO THINK ABOUT WAYS THAT WE COULD CREATIVELY INVEST IN THIS SPACE, AND ALREADY THERE HAVE BEEN SIGNIFICANT CONVERSATIONS ABOUT THE POSSIBLY ROLE OF THE COMMON FUND IN THAT AREA. THIS CONVERSATION WILL HELP A LOT OF FOR US TO THINK THAT THROUGH. ON TO A DIFFERENT TOPIC, ONE I THINK IS REALLY IMPORTANT AND ALSO AT TIMES THIS IS NOT AN AREA THAT WOULD HAVE BEEN ON AGENDA TWO OR THREE YEARS AGO, HERE WE ARE. MIKE LAUER PUT INCREDIBLE POSITION INTO FOLLOW-UP OF CHALLENGES WE HAVE HAD IN THIS AREA FOREIGN INFLUENCES AND I WANT TO SAY HOW GRATEFUL WE ARE TO HIS LEADERSHIP AND A TOPIC HE WOULDN'T HAVE CHOSEN EITHER TO BE DOMINANT IN EXPERIENCE BUT WHICH HE HANDLED WITH PROFESSIONALISM, PLEASE PROCEED. >> THANK YOU VERY MUCH, FRANCIS, THANK YOU FOR THE OPPORTUNITY TO PROVIDE A BRIEF UPDATE WHAT IS GOING ON WITH FOREIGN INFLUENCES. AS A QUICK REMINDER ABOUT A YEAR AGO, THE WORKING GROUP PRESENTED RECOMMENDATIONS, HANKS TO ROY AND LARRY FOR APPROACHING THAT GROUP, ROY AND I HAD A LITTLE GET TOGETHER LESS THAN A MONTH AGO AT AMC AS EVIDENCE THIS IS CONTINUING TO BE QUITE DYNAMIC. ALSO REMINDER BACK IN AUGUST OF 2018 FRANCIS SENT OUT A LETTER TO THE COMMUNITY EXPRESSING SOME CONCERNS WHICH WERE THEN DESCRIBED IN MORE DETAIL IN THE ACD WORKING REPORT. IT IS IMPORTANT TO KEEP IN MIND THAT WE ROUTINELY COLLABORATE PRODUCTIVELY WITH INVESTIGATORS IN FOREIGN COUNTRIES WE RELY ON PRODUCTIVE RESEARCH COLLABORATIONS, THE INDIVIDUALS WHO WE HAVE IDENTIFIED IS VIOLATING LAWS AND POLICIES REPRESENT A SMALL PROPORTION OF SCIENTISTS WORKING IN AND WITH YOU AS INSTITUTIONS, WE MUST NOT REJECT BRILLIANT MINDS WORKING COLLABORATIVELY TO PROVIDE HOPE AND HEALING. KEY CONCERNS ONE IS FAILURE TO DISCLOSE SUBSTANTIVE FOREIGN RESOURCES WHAT'S HAPPENED OVER THE LAST SIX MONTHS SINCES WE TALKED ABOUT THIS TOPIC IS THAT IT'S GETTING WORSE. WE ARE SEEING MORE AND SOME OF THE TYPES OF PROBLEMS WE ARE SEEING ARE EVEN MORE SEVEREMENT WE ARE SEEING UNDISCLOSED FOREIGN EMPLOYMENT ARRANGEMENTS, UNDISCLOSED GRANT SUPPORT INCLUDING SUBSTANTIVE OVERLAP DUPLICATION AND OVERCOMMITMENT. WE HAVE HAD CASES WHERE WE ARE FUNDING THE EXACT SAME GALLANT CHUTE iABSOLUTELY IDENTICAL AS A GRANT FUNDED BY A FOREIGN COUNTRY. IN THOSE CASES WE HAD TO GET GRANTS REIMBURSED. WE HAVE SEEN CASES WHERE PEOPLE HAVE FULL TIME JOBS IN A FOREIGN COUNTRY WHILE THEY ALSO HAVE A FULL TIME JOB HERM IN THE UNITED STATES. THAT WOULD BE 24 MONTHS COMMITMENT. IN MANY CASES NEARLY ALL CASES AMERICAN INSTITUTIONS ARE EITHER UNAWARE OF WHAT THEIR EMPLOYEES ARE DOING OR HAVE A MISLEADING IMPRESSION WHAT THEIR EMPLOYEES& ARE DOING. WE HAVE SEEN EMAILS INSTITUTIONAL LEADERS WRITE TO FACULTY UNTIL WE WERE CONTACTED BY THE NIH WE DID NOT KNOW YOU HAD THESE OTHER FOREIGN RESOURCES. WE HAVE SEEN FAILURE TO DISCLOSE SIGNIFICANT FOREIGN COMPLEX FINANCIAL CONFLICTS OF INTEREST AND PATENTS, AND WE HAVE SEEN SIGNIFICANT PEER REVIEW VIOLATIONS, YOU HEARD ABOUT THE ONES FROM MD ANDERSON, WHERE CONFIDENTIAL APPLICATIONS WERE EMAILED TO US IN ANOTHER COUNTRY WE HAVE SEEN MORE FROM OTHER INSTITUTIONS. THE ACD RECOMMENDATIONS ARE BEING IMPLEMENTED ONE WAS TO HAVE A BROAD AWARENESS CAMPAIGN AND WORK WITH OUTSIDE GROUPS. THIS IS HAPPENING ON MULTIPLE FRONTS ESPECIALLY OSTP JOINT COMMITTEE ON RESEARCH ENVIRONMENTS WHICH CARRIE MENTIONED YESTERDAY, THERE IS A SUBCOMMITTEE ON SCIENCE AND SECURITY WHICH I AM THE CO-CHAIR, I WILL TELL YOU MORE IN A BIT. THERE'S EXTENSIVE COORDINATION WITH OTHER PARTS OF GOVERNMENT, PARTICULARLY CALL OUT NSF, DEPARTMENT OF ENERGY, AND DEPARTMENT OF STATE, MY COLLEAGUE JODY BLACK IS SPEARHEADING THE INTERNATIONAL EFFORT THE DEPARTMENT OF STATE RECOGNIZES THIS IS A PROBLEM WHICH IS NOT UNIQUE TO THE UNITED STATES. THIS JASONS SOME OF YOU FAMILIAR WITH THEM, THIS IS A GROUP THAT WORKS WITH THE DEFENSE DEPARTMENT ISSUED AN NSF COMMISSION REPORT A FEW DAYS AGO ABOUT THE THREATS OF FOREIGN TOWNS PROGRAMS, MY COLLEAGUE DR. PATRICIA VALDEZ AND I HAD AN OPPORTUNITY TO INTERACT WITH THEM, THERE'S DISCUSSION ABOUT NIH EFFORTS IN THEIR REPORT. AND WE HAD MULTIPLE INTERACTIONS WITH OTHER VARIOUS GROUPS, ROY AND I DID A SESSION AT AAMC MEETING IN NOVEMBER, JODY DID A SESSION WITH AAU AND APLU, I DID A DISCUSSION A WHILE BACK WITH VARIOUS HEADS OF SCIENCE TECHNOLOGY SOCIETIES THROUGH THE AAAS. WE PUT OUT JULY A CLARIFICATION OF OTHER SUPPORT, WE ARE WORKING WITH THE NATIONAL SCIENC FOUNDATION TO MAKE IT EASIER MORE USER FRIENDLY FOR PEOPLE TO INFORM US OF SUPPORT. WE ARE DOING EXTENSIVE TRAINING AMONG STAFF ON PEER REVIEW. PERHAPS MOST IMPORTANT, WHAT IS TAKING UP MOST AMOUNT OF TIME IS CONTACT WE HAVE HAD NOW WITH OVER 70 INSTITUTIONS, WE HAVE DONE IN DEPTH OVER 140 SCIENTISTS AND WE HAVE MANY MORE TO GO. SO I MENTIONED OSTP BACK IN SEPTEMBER, KEVIN THE DIRECTOR OF O OSTP PUT OUT A LETTER TO THE SCIENTIFIC U.S. RESEARCH COMMUNITY AND IN THIS LETTER HE SAYS SOME NATIONS EXHIBITED INCREASINGLY SOPHISTICATED EFFORTS TO EXPLOIT INFLUENCE AND UNDERMINE OUR RESEARCH ACTIVITIES AND ENVIRONMENTS, RESEARCH -- BREECHES OF RESEARCH ETHICS INCLUDE FAILURE TO DISCLOSE REQUIRED INFORMATION SUCH AS FOREIGN FUNDING, UNAPPROVED PARALLEL FOREIGN LABS OR SHADOW LABS, UNAPPROVED AFFILIATIONS AND APPOINTMENTS AND CONFLICTING FINANCIAL INTERESTS. CONDUCTING UNDISCLOSED RESEARCH FOR FOREIGN GOVERNMENTS OR COMPANIES, US AGENCY TIME, OR WITH U.S. AGENCY FUNDING. WE HAVE BEEN REIMBURSED BY SOME UNIVERSITIES BECAUSE PEOPLE WERE SPENDING SUBSTANTIAL AMOUNTS OF TIME WORKING IN A FOREIGN COUNTRY COLLECTING SALARY THAT WAS PAID FOR BY THE NIH. WE ALSO -- HE GOES ON TO TALK DIVERSE INTELLECTUAL PROPERTY AND BREACHES OF CONFIDENT AND CONFIDENTIALITY IN SURREPTITIOUS GAINING OF PEER REVIEW PROCESS. IN THE WILL EFFORT DR. -- DESCRIBE AS NUMBER OF GOALS OF THE NSTC, THIS INCLUDE COORDINATING OUTREACH AND ENGAGEMENTS PART OF THAT IS PRESENTING AN ARRAY OF EXAMPLES WHERE WE WERE EXPLOITED OR COMPROMISED. I WILL GET TO THATNA A MOMENT. ESTABLISHING WHAT DISCLOSURE REQUIREMENTS ARE. AND DOING THAT IN A HARMMIZED WAY ACROSS THE U.S. -- HARMONIZED WAY ACROSS THE GOVERNMENT UNDERSTANDING DISCLOSURE IS A CENTRAL TENANT OF RESEARCH INTEGRITY. DEVELOPING BEST PRACTICES FOR RESEARCH INSTITUTIONS, WHEN I TALKED TO YOU LAST JUNE I MENTIONED THE OUTSTANDING WORK, WHICH ISING WITH DONE BY AAU APLU AND OTHERS. AND DEVELOPING METHODS FOR IDENTIFICATION ASSESSMENT AND MANAGEMENT OF RISK. AND I WOULD SAID OVER TIME WORKING IN CLOSE COORDINATION WITH OUR COLLEAGUES AND INTELLIGENCE AND IN LAW ENFORCEMENT WE ARE GETTING BETTER. SO I MENTIONED ARRAY OF EXAMPLE EXAMPLES, PART FULFILLED BY THE UNITED STATES SENATE, THIS IS THE HOME LAND SECURITY AND GOVERNMENT AFFAIRS SUBCOMMITTEE ON INVESTIGATIONS. THEY DID AN INVESTIGATION WHICH WAS RELEASED BACK ON NOVEMBER 19th, WE HAD ADD HEARING WE PARTICIPATED IN THAT HEARING, IT IS AN INTERESTING REPORT WORTH READING AND THERE IS AN APPENDIX WHICH THERE ARE EXAMPLES OF CONTRACTS AMERICAN SCIENTISTS SIGNED WITH FOREIGN COUNTRIES. SO IN THIS REPORT THEY SAY THOUSAND TALENT PLAN MEMBERS SIGN LEGALLY BINDING CONTRACTS WITH CHINESE INSTITUTIONS LIKE UNIVERSITIES AND RESEARCH INSTITUTIONS. THE CONTRACTS CAN INCENTIVIZE MEMBERS THE LIE ON U.S. GRANT APPLICATIONS, SET UP SHADOW LABS IN CHINA AND TRANSFER SCIENTISTS HARD EARNED INTELLECTUAL CAPITAL. SOME OF THE CONTRACTS ALSO AND REQUIRE CHINESE GOVERNMENT POSITIONS TO TERMINATE GOVERNMENT. THESE ARE IN STARK CONTRAST TO RESEARCH COMMUNITIES BASIC NORMS VALUES AN PRINCIPLE. WE HAVE HAD THE OPPORTUNITY TO SEE DOZENS OF CLINICAL TRIALS, ONE QUESTION THAT WE SOMETIMES ASK DID YOU SEE THIS BEFORE YOUR COMPANY SIGN ED IT. NO HAD YOU SEEN IT BEFORE YOUR EMPLOYEE SIGNED IT WOULD YOU HAVE SAID YES THIS IS FINE PLEASE GO AHEAD AND THE ANSWER ALMOST ALWAYS IS NO. LET ME CONCLUDE WITH A QUOTE I SHARED LAST TIME FROM PENN STATE UNIVERSITY FROM THERE OUTSTANDING WEBSITE, WHICH THEY SAY THAT MOST INTERNATIONAL COLLABORATIONS ARE ACCEPTABLE AND ENCOURAGED. WE URGE RESEARCHERS TO ERR ON TRANSPARENCY, IT PROTECTS EVERYONE'S INTEREST, FEDERAL GOVERNMENT, PENN STATE, FILL IN YOUR INSTITUTION, INDIVIDUAL RESEARCHERS AND THIRD INTERNATIONAL COLLABORATORS TO HAVE INTERNATIONAL RELATIONSHIPS DISCLOSED AND VETTED. CRITICAL PART IS NOT JUST DISCLOSE BUT ALSO VET TO DETERMINE IF THERE ARE POTENTIAL CONFLICTS OF COMMITMENT, DUPLICATIONS OF RESEARCH AND DIVERSION OF INTELLECTUAL PROPERTY AND PERFORMANCE OF FEDERALLY FUNDED RESEARCH. WE HAVE NOW SEEN A FEW EXAMPLES OF EVERY ONE OF THESE CONFLICTS OF COMMITMENT, DUPLICATION, DIVERSION OF INTELLECTUAL PROPERTY. SO THIS IS A NEW TYPE OF THREAT NOR NIH THOUGH NOW WE HAVE BEEN WORKING ON THIS A NUMBER OF YEARS SO WE ARE LEARNING MORE ABOUT I. WE APPRECIATE YOUR WORK AND YOUR RECOMMENDATIONS WHICH WE ARE IMPLEMENTING. THE EXTENSIVE INSTITUTIONAL OUTREACH YIELDED RESULTS, ONE INTERESTING RESULT IS THAT A NUMBER OF INSTITUTIONS NOW ARE SELF-DISCLOSING. THEY ARE DISCOVERING ARE PROBLEMS ON THEIR OWN AND COMING TO US AND TELLING US PROBLEMSSTHEY DICOVER AMONG SCIENTISTS, EAT GREAT TO SEE THIS. WE ARE WORKING CLOSELY WITH OTHER AGENCIES AND STAKEHOLDERS AND BIG PART COORDINATED TO THE OSTP AND WE REITERATE IMPORTANCE OF CONTRIBUTION OF FOREIGN SCIENTISTS BIOMEDICAL RESEARCH, WE MUST NOT CREATE A CLIMATE UNWELCOMING TO THEM. THIS SLIDE ILLUSTRATES THIS IS A TEAM SPORT, WE ARE WORKING CLOSE WITH PEOPLE FROM OTHER PARTS OF GOVERNMENT FROM NON-FEDERAL ORGANIZATIONS. I WANT TO GIVE PARTICULAR THANKS TO THE DOZENS OF THE VICE PRESIDENTS FOR RESEARCH, COMPLIANCE AND RESEARCH INTEGRITY OFFICER, AND MANY UNIVERSITY AROUND THE COUNTRY. THEY HAVE TAKEN THIS EXTRAORDINARILY SERIOUSLY, THEY HAVE WORKED HARD WITH US TO LEARN AND UNDERSTAND THE NATURE OF THE THREAT. AND TO APPROPRIATELY DEAL WITH IT. THANK YOU. [APPLAUSE] JUDITH. >> AMAZING TAGSK YOU HAVE TAKEN ON, EVERYTHING YOU HAVE BEEN DOING IS DIFFICULT. WON CONCERN IN THE SCIENTIFIC COMMUNITY IS THIS IS TARGETING ETHNIC CHINESE. I UNDERSTAND A LOT OF PROBLEMS HAVE COME UP HAVE THAT SOURCE. BUT THERE ARE LOTS OF PEER REVIEW VIOLATIONS THAT HAVE NOTHING TO DO WITH ETHNIC CHINESE AND SHADOW LABS WITH NOTHING TO DO WITH ETHNIC CHINESE AS WELL. SO IN TERMS, THE TRANSPARENCY AND THAT YOU ARE TALKING ABOUT IN DOING THIS AS SOON ADS POSSIBLE, WOULD BE GREAT IF YOU COULD HAVE SOME EXAMPLES IF YOU ARE PUTTING SAMPLES UP OF PEOPLE NOT ETHNIC CHINESE WHO ARE ALSO SUBJECT TO THE SAME POLICIES. AND I GUESS THAT IS THE PRIMARY THING, THE OTHER THING I SAW THIS LETTER UNDER TAB 19,, THAT HAS VERY IMPORTANT INTERESTING QUESTIONS. I DON'T KNOW IF YOU HAVE -- SEEN INFORMATION. >> MICHAEL FISHER. >> TO ME THOSE QUESTIONS WERE ON TARGET AND HOPE YOU ARE PAYING ATTENTION TO THEM. >> >> THANK YOU. I WOULD POINT OUT THAT THESE ARE CRITICALLY IMPORTANT POINTS. WE ARE TARGETING BEHAVIORS, UNDISCLOSED EMPLOYMENT, UNDISCLOSED FOREIGN GRANTS, PEER REVIEW BREECHES ARE ALL BEHAVIORS. SOME OF THE TARGETING IS FOREIGN GOVERNMENTS TARGETING EXPATS, I WOULD POINT OUT THEY ARE NOT ETHNICALLY CHINESE. THERE IS A SIGNIFICANT PORTION WHO ARE NOT AND SOME OF THE MOST EGREGIOUS CASES WE HAVE SEEN INVOLVE PEEPED WHO ARE NOT ETHNICALLY CHINESE. YOU ARE ALSO RIGHT, PEER REVIEW VIOLATIONS TRANSCEND THIS INVOLVE OTHER PEOPLE AS WELL. WE HAVE RESPONDED TO DR. FIB FISHER. ONE THING HE ASKED FOR IS REDACTED COPIES OF CLINICAL TRIALS AND THE SENATE COMMITTEE RELEASED THAT. HE ALSO ASKED FOR SOME DATA WHICH WE ARE WORKING ON. FROM PRESIDENT >> I THINK THERE'S THREE, FOUR MAYBE FIVE MEETINGS HOSTED BY FBI OR WHITE HOUSE EVER SINCE OUR REPORT CAME OUT AND SOME OF THEM, LARRY HAS BEEN THERE AND SOME OF THEM MICHAEL HAS BEEN THERE. I HAVE NOTICED, AND MIKE INTERESTED IN YOUR THOUGHTS ON THIS, A DISTINCT DIFFERENCE IN TONE FROM THE INITIAL MEETINGS TO NOW. THE INITIAL MEETINGS EMPHASIS WAS MORE PUNITIVE STUFF AND SCARE TACTICS. AND THERE WAS BACKLASH FROM THE UNIVERSITY COMMUNITY. I THINK THE THEY HAVE PROBABLY MODIFIED THEIR APPROACH AS RESULT OF THAT. THE TONE NOW IS MUCH MORE WHILE ACKNOWLEDGING WHILE STILL MAINTAINING THERE IS HUGE PROBLEM NOT BACKING FROM THAT. THE TONE IS MORE PARTNERSHIP WITH THE ACADEMIC COMMUNITY. IT IS A MARKET CHANGE. AT ONE POINT I WAS GETTING CONCERNED WE WERE TALKING NOT ALLOWING CHINESE STUDENTS TO COME AND ALL KIND OF STUFF, I DON'T HEAR THAT NEARLY AS MUCH ANY MORE, IN THE JASON REPORT THAT CAME OUT BASICALLY THE BOTTOM LINE MESSAGE WAS NEW RESTRICTIONS ON FOREIGN SCIENTISTS, FUNDAMENTAL RESEARCH WAS NOT NEEDED AND INSTEAD BROADER DISCLOSURE REQUIREMENTS AND EDUCATION EFFORTS ARE WHAT IS NEEDED TO PROTECT U.S. SCIENCE FROM FOREIGN INFLUENCES. THAT'S KIND OF THE DIRECTION THAT THE WORKING GROUP WENT IN. >> CAN I ADD, BECAUSE THIS IS BEEN IN THE PAPERS, SO PEOPLE READ THOSE EARLY THINGS CIRCULATED SO THIS NEEDS TO BE SOMETHING VISIBLE THAT IS BRINGING THIS NEW TONE. >> CARRIE, BENEFIT DAN, JEFF NOT SURE IF YOU ARE UP. >> BECAUSE I LIKE TO CONNECT DOTS, WE HEARD YESTERDAY FROM THE ANTI-HARASSMENT WORKING GROUP AN EMPHASIS ON CODES OF CONDUCT AND STANDARDS FOR PROFESSIONAL CONDUCT. THAT HAS PROVED USEFUL IN THIS ARENA AS WELL CAN YOU SAY MORE ABOUT THE EXPERIENCE HOW THAT'S USEFUL FOR UNIVERSITY IN THIS SPACE? >> THANKS, CARRIE. SOME UNIVERSITY AMONG -- IN THEIR CODE OF CONDUCT THEY STATE CEARLY THAT YOU CANNOT DO RESEARCH THROUGH ANY OTHER INSTITUTION THAN OURS. IF YOU WANT TO DO RESEARCH FROM INSTITUTION OTHER THAN OURS YOU HAVE TO TALK TO US AND IN GENERAL EVERYTHING HAS TO COME THROUGH OUR OFFICE SPONSORED RESEARCH. THEY HAVE PROVISIONS LIKE YOU CANNOT HAVE EMPLOYMENT IN ANY OTHER INSTITUTION OTHER THAN OTHERS UNLESS WE VET AND APPROVE IT IN ADVANCE. SO IN SOME CASES WHAT WE HAVE HEARD BACK FROM INSTITUTIONS IS, MERELY BY FACT THAT SCIENTISTS HAVE SOUGHT FUNDING THROUGH RESEARCH -- INSTITUTIONS OTHER THAN THEIRS, THEY HAVE VIOLATED THEIR EMPLOYMENT ARRANGEMENTS AND THEREFORE NIH YOU DON'T NEED TO WORRY. WE HAVE A PROBLEM WE NEED TO DEAL WITH. >> BRENDAN. >> I WAS WONDERING, THERE ARE THESE CLEAR EXAMPLES OF MISCONDUCT, SNOW QUESTION ABOUT THAT. WHAT IS THE MAGNITUDE? HOW MANY ARE WE TALKING ABOUT WHERE THERE'S DEAF METALLY CLEAR MISCONDUCT TRY TO GET A SENSE OF NUMBERS COMPARED TO GENERAL MISCONDUCT WE ARE AWARE OF? THE HONEST QUESTION IS WE DON'T YET KNOW EXTENT OF THE PROBLEM. WE HAVE LOOKED IN DEPTH AT 140 SCIENTISTS OF THOSE, 75% OR SO HAD A REAL PROBLEM. THERE IS SUBSTANTIAL COMPLIANCE ISSUE. WE HAVE DOZENS MORE TO DO GO. THE JASON REPORT AS AN EXAMPLE SAID THE SCOPE OF THE PROBLEM IS AS YET UNCLEAR AND I WOULD AGREE WITH THAT. WE DON'T FULLY HAVE A HANDLE ON THIS. I WAS GOING TO ADD I TOTALLY AGREE THE TONE CHANGED. WE HAD A RECENT MEETING WITH ALL ACADEMIC INSTITUTIONS IN ILLINOIS AND CHICAGO AREA. THAT INVOLVED INTELLIGENCE, LAW ENFORCEMENT, NIH NSF AND VARIOUS ACADEMIC INSTITUTIONS. IT WAS MORE A PARTNERSHIP COLLABORATIVE FEELING THAN WHAT WE HAVE SEEN BEFORE. >> DINA. >> OF COURSE COMING FROM ACADEMIA, WE VERY MUCH DEPEND EVERY DAY WORK WITH INTERNATIONAL STUDENTS AND POST DOC, PARTICULARLY IN CHINESE TALKING MACHINE LEARNING. JUST FROM THE PREVIOUS SESSION. SO IT IS VERY IMPORTANT TO ALSO LIKE SOME OF THIS -- SAY IT DIFFERENT. THE CULTURE MIGHT BE DIFFERENT. IN MANY CASES, NOT EVERY CASE OF COURSE, BUT IN SOME CASES SOME OF THOSE INDIVIDUALS MAY DO THINGS COMPLETELY INNOCENT TO A NOT REALIZING THAN THEY ARE BREAKING ANY LAW OR ANY RULE OF ANY FORM. BECAUSE THEY DIDN'T KNOW, THEY DIDN'T UNDERSTAND OR COME FROM A DIFFERENT CULTURE WHERE THAT WAS NOT AN ISSUE. I DON'T KNOW OUR ORIENTATION FOR STUDENTS TELL THEM ABOUT THAT KIND OF MISCONDUCT AND MAYBE LIKE WILLER SOME EFFORT SHOULD BE DONE BY INSTITUTIONS, NIH TELLING INSTITUTION THEY FUND TO ADD THAT TO THE ORIENTATION OF POST DOC AND STUDENTS. SO THAT THEY ARE AWARE. MANY SMALLER ISSUES WOULD DISAPPEAR BY DOING THAT. >> IMPORTANT POINT. I HAVE HEARD ONE PERSON WHO ARTICULATED THIS WELL. DR. ZUBER. AT MIT. ABOUT IMPORTANCE OF EDUCATING STUDENTS AND POST DOCS IN ALL VISITING SCIENTISTS ABOUT WHAT MICS T RULES ARE. ONE RULE IS MIT PROPERTY IS MIT PROPERTY. YOU CAN'T TAKE IT SOMEPLACE ELSE WITHOUT GOING THROUGH APPROPRIATE PROPER PROCEDURES. SO THIS IS INCREDILY IMPORTANT POINT. I WOULD ALSO POINT OUT, OUR INTERACTIONS WITH INSTITUTIONS HAVE BEEN ESSENTIALLY ENTIRELY ON SENIOR SCIENTISTS. AND INITIALLY, HE WERE WITH WONDERING, MAYBE A BIG MISUNDERSTANDING BUT WE HAVE SEEN SO MUCH DECEPTION AND SO MANY LIES IT IS CLEAR THIS IS WILLFUL ACTIVITY. >> LAST QUESTION, SHELLEY. >> I WANT TO SAY AT LEAST AT PENN, GRADUATE STUDENTS HAVE A VERY CLEAR INSTRUCTION THEY CANNOT THEY CAN'T DO ANYTHING IN INDUSTRY WITHOUT PERMISSION SO JUST AN EXTENSION OF THIS INFORMATION TO THEM WHEN THEY COME IN ALL GRADUATE STUDENTS, THAT I ARE AWARE THAT THEY CAN DO THAT. TO YOUR POINT IT WOULDN'T BE TOO DIFFICULT TO JUST EXTEND THIS TO -- >> MORE INFORMATIVE ORIENTATION. >> YES. >> OKAY. MIKE, THANK YOU VERY MUCH FOR THAT UPDATE. AND FOR ALL THE HARD WORK YOU AND YOUR TEAM HAVE BEEN DOING. WE WILL PROBABLY CONTINUE TO INFORM ACD WHEN WE GET TOGETHER ABOUT STATUS BECAUSE THIS IS AN EVOLVING SITUATION. TO AHUGHES US TO HEAD TOWARD THE EXITS WITH ALL KINDS OF HAPPY VISIONARY THOUGHTS WE THOUGHT IT WOULD BE USEFUL FOR YOU TO HEAR WHERE WE ARE IN THIS EFFORT TO DEVELOP A NEW FIVE YEAR NIH WIDE STRATEGIC PLAN WHICH IS REQUIRED BY CONGRESS, THE ACD WAS HELPFUL IN GETTING THE FIRST ONE PUT TOGETHER, NOW IT'S TIME FOR REFRESH. AND JIM ANDERSON HAS BEEN PUTTING HIS WISE PERSON INTO THIS PROCESS AND WILL TELL YOU WHERE WE ARE. >> >> THANK YOU, FRANCIS. THANK YOU FOR INTRODUCING THIS AS BY INTRODUCING WE ARE TRYING TO FINISH. SO I WILL BE BRIEF. SO THE TIME IS HERE TO -- IT'S TIME TO UPDATE THE NIH WIDE VENAL MAN. JUST AS REMINDER, THE FIRST PLAN WHICH WAS REQUIRED BY CONGRESS COVERED THE PERIOD 2016 TO 2020. THIS WAS MANDATED IN THE 15 APPROPRIATIONS ACT. AND THERE WERE ADDITIONING FROM THE CURES ACT I WILL GET TO. AND THE ACD BECAUSE IT WAS FIRST PUT THE ACD WAS ASKED TO HELP US, SOME PARTICIPATED IN PUBLIC EVENTS AND WE RAN THIS BY AS IT WAS DEVELOPED SO I'M HEAR TO LET YOU KNOW HOW WE ARE BEGINNING TO DEVELOP THE NEXT PLAN. AND I WILL KEEP IT AT HIGH LEVEL BECAUSE WE DON'T HAVE CONTENT YET AND I WANT YOU TO REMEMBER THIS PLAN IS FOR THE PUBLIC IT'S FOR CONGRESS, FOR STAKEHOLDERS, NOT JUST RESEARCHERS E IT'S NOT A STRATEGIC PLAN TO CURE DISEASE, IT IS AN EXPLANATION WHAT NIH DOES. SO I WILL TODAY TALK ABOUT THE OUTLINE, WHY IT'S ORGANIZED THE WAY IT IS, WHAT WE ARE TRYING TO ACCOMPLISH WITH IT. IN ADDITION TO WHAT WE PUT OUT BEFORE, THE 21st SENT ARE CURES ACT ADDED SPECIFICITY HOW CONGRESS WANTED THIS TO BE DONE. INTERESTINGLY WE PUT OUT THE PLAN THREE DAYS AFTER THIS ACT WAS PASSED SO WE WILL INCORPORATE IN THIS PLAN WHAT THEY ADDED. THEY HAD A LONG LIST. BUT I WILL KEY ON A FEW THINGS. ONE IS THE STRATEGIC PLAN FOR NIH HAD TO BE DEVELOPED AT LEAST EVERY SIX YEARS, WE ARE GOING TO DO IT EVERY FIVE YEARS, NOT TO TORTURE OURSELVES BUT BECAUSE MOST INSTITUTES AND CENTERS HAVE FIVE YEAR PLANS. CONGRESS ALSO ASKS THAT THE IC STRATEGIC PLAN BE INFORMED BY NIH WIDE PLAN MANY TERMS OF DIRECTIONS OR THEMES. SO MAKES IT EASIER TO SYNC. WE WERE ALSO ASKED TO DEVELOP A COMMON TEMPLATE. IF YOU LOOKED IN THE PAST, EVERYONE HAD A DIFFERENT ORGANIZATION FOR HOW THEY PUT TOGETHER A PLAN SO WE HAVE COME UP WITH A COMMON TEMPLATE, WHICH WAS AGREED ON BY LEADERSHIP ABOUT A YEAR AGO, IT'S BEEN USED FOR A PLAN ANYONE THAT'S DOCUMENT OUT SINCE JANUARY OF THIS CALENDAR YEAR. IT HAS COMMON ORGANIZATION WHAT THE INTENT IS, CERTAIN ELEMENTS, IT CAN BE INTERPRETED. IT'S THERE. SOME OF THE OTHER THINGS CONGRESS REQUIRED WERE NOT DISEASE SPECIFIC ISSUES BUT TELL US -- TELL THE WORLD HOW WE IDENTIFY GAPS AND OPPORTUNITIES JUST GENERAL APPROACHESES TO DOING YOU YOU ARE SCIENCE, ALSO HOW YOU ADDRESS HEALTH DISPARITIES HAVE TO BE SPECIFICALLY ADDRESSED. THE THERE IS A LIST OF THINGS EASY TO ACCOMMODATE. THINKING HOW TO ORGANIZE THIS OR HOW -- WHAT IS DRIVING OUR THINKING WE DECIDED WHAT THIS PLAN BE AND WHAT WILL IT NOT BE. THIS WAS INFORMED BY THE FOLKS WHO PUT TOGETHER THE FIRST PLAN AND THEIR SENSE WHAT HAPPENED AFTER THEY PUT IT OUT AND THINKING SINCE THEN. WE DECIDED THIS PLAN IS GOING TO CLEARLY ARTICULATE THE HIGHEST PRIORITIES FOR NIH OVERALL. IT WILL EXPLAIN HOW WE ACHIEVE THOSE HIGH PRIORITIES. IT'S ALSO SINCE SECOND PLAN NOW GOING TO REPRESENT UPDATE ON THE LAST PLAN SO WE WILL INCLUDE ACCOMPLISHMENTS OF WHAT WAS DONE UNDER THE LAST PLAN, AND ALSO NEW INITIATIVES THAT KICKED OFF OR STARTING SINCE THAT PLAN WAS PUT OUT. ALSO WHAT WILL IT NOT BE, IT'S NOT GOING TO DESCRIBE EVERYTHING NIH DOES. IT WILL HAVE SPECIFIC EXAMPLES OF PROGRAMS INITIATIVES, BUT THEY WILL BE USED TO AS EXAMPLES OF NIH WIDE INTENTS. IT'S NOT GOING TO ADDRESS THE INDIVIDUAL PRIORITIES OF THE INSTITUTES CENTERS AND OFFICES, SINCE THEY ALL HAVE THEIR OWN STRATEGIC PLANS. IT IS NOT A COMPLETE OVERHAUL OF THE LAST PLAN. SO HOW ARE WE APPROACHES THIS? Z FROM WE STARTED IN SEPTEMBER BY IDENTIFYING FOLKS FROM ALL OVER NIH WHO PARTICIPATE IN DEVELOPING THE OUTLINE FOR THE PLAN AND EVENTUALLY CONTENT FOR THE PLAN. I AM HERE TO TELL YOU HOW THAT PROCESS IS GOING. WHEN WE FLESHED OUT A DRAFT PUT IT OUT FOR PUBLIC COMMENT AND THEN I HAVE ALREADY PUT THIS ON THE COUNCIL OF COUNCILS MEETING FOR MAY, WE WILL BRING IT TO LOOK AT IT IN SOME DETAIL. WE WILL PUT TOGETHER A LATE DRAFT, BRING IT BACK TO YOU, AND EVENTUALLY WE WILL ASK FRANCIS TO ENDORSE IT AND HOPE TO GET IT OUT BY NOVEMBER NEXT YEAR. WE WILL GET IT OUT BY NOVEMBER NEXT YEAR. AT VERY HIGH LEVEL IT WILL BE A HALF PAGE SHORTER FRANCIS, I PROMISE. HOW IS IT ORGANIZED LAST TIME? THERE ARE DIFFERENT LABEL AN HEADINGS LEFT OUT BUT THIS IS THE GENERAL WAY IT WAS PUT TOGETHER WITH AN OVERVIEW THAT STARTS WITH MESSAGE FROM THE NIH DIRECTOR, THE MISSION OF NIH, ORGANIZATION, STATUTORY AUTHORITIES. AND THEN REALLY SIGNIFICANT PART IS THE OBJECTIVES, TO ACHIEVE STRATEGIES EXPLAINED ABOVE. THEY FELL INTO THESE CATEGORIES WHICH WE CHANGED A LITTLE BIT SO I WILL EXPLAIN WHY. THE FIRST WAS THE SCIENCE. WHAT IS IT WE DO? ADVANCE OPPORTUNITIES IN BIOMEDICAL RESEARCH. WE BROKE IT INTO FUNDAMENTAL SCIENCE, AND THEN TREATMENTS AND CURES AND HEALTH PROMOTION AND DISEASE PREVENTION. OTHER THREE IS OVERSEE THE SCIENCE AND OBJECTIVES FOR THAT. SETTING PRIORITIES HOW WE DO THAT WHERE AND WHY. SCIENTIFIC STEWARDSHIP LIKE ANTI-HARASSMENT ISSUES WE TALKED ABOUT YESTERDAY. MORE RIGOR AND REPRODUCIBILITY AND THIRD IS MANAGING FOR RESULTS. SO TURNS OUT THINKING WAS OR THE OBSERVATION WHEN THEY PUT THIS TOGETHER SCYTHIA HAVE WOVEN THINGS AROUND THAT. WE REORGANIZED THAT. ONE INTERESTING THING I DON'T REMEMBER BUT CERTAIN THIS WAS FRANCIS IDEA WAS TO INCLUDE A SET OF BOLD PREDICTIONS ABOUT AMERICA'S FUTURE OR WHAT WE'LL ACHIEVE DURING THE PERIOD OF THAT PLAN. IT TURNED OUT TO BE 14 BOLD GOALS. I WILL REPORT OUTCOME OF SOME. SO THIS IS GOING TO BE THE HIGHEST LEVEL ORGANIZATION OF THE UPCOMING PLAN. AGAIN OVERVIEW THAT STARTS WITH THE MESSAGE FROM THE DIRECTOR HOW ORGANIZED SO THE PUBLIC SEES THAT, THE MISSION, WHAT CONGRESS HAS GIVEN US IN TERMS OF AUTHORITIES WHAT IS BOUNDARIES. AND THEN THIS IS WAY THE OBJECTIVES WILL BE ORGANIZED. AGAIN, WHAT IS THE SCIENCE WE DO, NUMBER ONE, ADVANCING BIOMEDICAL AND BEHAVIORAL SCIENCES, AND THEN WHAT DO WE NEED TO DO TO MAKE SURE THE SCIENCE CAN WORK. AND IS FACILITATED. THIS WAS HAS IN THE PLAN BEFORE BUT PULLED OUT SEPARATELY NOW. DEVELOPING MAINTAINING RENEWING SCIENCE RESEARCH CAPACITY WHICH IS GENERALLY GOING TO BE WORK FORCE AND INFRASTRUCTURE. THE THIRD IS HOW DO WE OVERSEE AND MANAGE THIS PROCESS. SO IT COMES OUT HOW WE WANT AND FULFILLS EXPECTATIONS SO THIS IS PROMOTING SCIENTIFIC INTEGRITY, PUBLIC ACCOUNTABILITY, SOCIAL RESPONSIBILITY. WE WILL HAVE BOWLED PREDICTIONS. COPPING DID HAVE REQUIREMENTS LIKE PUTTING THE COMMON FUND STRATEGIC PRIORITIES AND PLAN WITHIN THIS WHICH WE NORMALLY HAVE SEPARATE PLAN SO TO MAKE IT MORE GRACEFUL FOR READING, WE WILL PULL OUT AN APPENDIX REQUIRE THINGS COMMON FUND STRATEGIC PRIORITIES. OKAY. I WANT TO QUICKLY BREAK DOWN WHAT WE WOULD DO UNDER THESE OBJECTIVES. ONE IS THE SCIENCE THAT WE DO. YOU CAN THINK ABOUT THIS, WE SOMETIMES EXPLAIN IT TO THE PUBLIC AS BASIC SCIENCE LEADS TO PRE-CLINICAL, LEADS TO CLINICAL. TO IMPLEMENTATION SCIENCE. WHAT WE DOING HERE IS TALKING ABOUT FUNDAMENTAL SCIENCE WE DO. IN AND THAT CAN BE BASIC RESEARCH TO POPULATION AND EPIDEMIOLOGIC FOUNDATIONAL SCIENCE. THEN HOW WE PREVENT DISEASE AND PROMOTE HEALTH. HA IS EXPLAINED MORE IN THE TRADITIONAL PATHWAY FROM TRANSLATION CLINICAL IMPLEMENTATION AND POPULATION. AS IS HOW WE TREAT DISEASE, HAVE INTERVENTIONS FOR DISEASE, AND CREATE CURES. THAT IS THE WAY THAT'S DONE. THAT WILL BE USED ONE OF THE OPPORTUNITIES WHAT ARE CHALLENGES, AND WE'LL USE EXAMPLES FROM THE INSTITUTES. THE SECOND IS WHAT DO WE NEED TO TO TO MAKE SURE THIS PROCESS WORKS. THAT'S AN INVESTMENT IN WORK FORCE AND INFRASTRUCTURE. WHAT ARE GOALS FOR TRAINING WHY DO WE DO IT HOW DO WE DO IT. WE USE SERIES OF EXAMPLES WE WILL EMPHASIZE DIVERSITY PROGRAMS AND SPECIAL TRAINIG NEEDS SUCH AS FOR CLINICIAN SCIENTISTS, NURSES VETERINARIANS, SO ON. IN INFRASTRUCTURE THIS IS VERY IMPORTANT SOMETIMES WE HAVE TO CREATE PLATFORMS TO DO THE WORK. THAT CAN BE ANYTHING FROM SHARED INSTRUMENTS, TO COMPREHENSIVE CANCER CENTERS, WE HEARD FROM SUSAN TODAY. SOME FROM DAVID'S TALK ABOUT THE NEED TO ENHANCE DATA. DATA SCIENCE NOT JUST PLATFORMS BUT RESOURCES TO BE ABLE TO DO DATA SCIENCE. THAT IS THE SECOND. THEN THIRD, HOW DO WE OVERSEE THIS AND WHAT ARE PRINCIPLES FOR DOING THE SCIENCE AND SUPPORTING IT? SO AGAIN, THIS IS SCIENTIFIC INTEGRITY PUBLIC ACCOUNTABILITY AND SOCIAL RESPONSIBILITY. WE WILL BREAK IT INTO FOUR AREA. SCIENTIFIC SEWEREDSHIP, HOW WE SET PRIORITIES AND WHY. HOW DO WE MONITOR PROGRESS AND HOLD OURSELVES ACCOUNTABLE AND OTHER THINGS IN THAT CATEGORY. WE HAD DONE THIS BEFORE BUT CONGRESS ACTUALLY ASKED US TO POINT OUT THE PARTNERSHIPS, HOW DO WE LEVERAGE PARTNERSHIPS. THE MAJOR PARTNER FOR NIH IS UNIVERSITIES OF RESEARCH INSTITUTES. WE ALSO SEEK PUBLIC INPUT KEEP ON TRACK WHETHER TRIBAL INTENTIONS OR SPECIFIC COMMUNITIES INTERESTS. WE HAVE ENGAGED WITH OTHER COUNTRIES AND FOUNDATIONS, THAT'S THE PARTNERSHIPS AND ENSURING ACCOUNTABILITY AND CONFIDENCE IN OUR SCIENCE. SO THESE ARE THE ETHICAL ISSUES ENHANCING THE WORK FORCE. THIS IS WHERE WE COVER ANTI-HARASSMENT ISSUES THAT CAME UP YESTERDAY. THE LAST MIGHT SEEM INWARD FOCUSED BUT EXTREMELY IMPORTANT. THAT IS OPTIMIZING OPERATIONS. AND MIKE WILL RESONATE WITH THIS. THIS IS HOW DO WE DO OUR BUSINESS? HOW DO WE INCREASE EFFICIENCY AND EFFECTIVENESS? YEARS AGO EACH INSTITUTE HAD THEIR OWN EMAIL SYSTEM. NOW WE HAVE PRETTY MUCH ENTERPRISE WIDE SYSTEM, THINGS LIKE THAT. HOW DO WE BECOME MORE EFFICIENT AND EFFECTIVE AND SECURE. BUSINESS PROCESS AND OTHER THINGS WE DO TO MANAGE FOR AS A RESULT. THAT LOOKS LIKE FOUR SILOS BUT IT'S NOT GOING TO BE. THERE ARE STRONG THEMES WOVEN THROUGHOUT THE WHOLE REPORT. ONE WILL BE IN ALL OF THESE DOMAINS WHAT ARE WE DOING TO INCREASE AND ENHANCE DIVERSITY. AS WE HEARD AGAIN TODAY VERY IMPORTANT RECOGNITION WE HAVE TO WEAVE DATA SCIENCE ADVANCES THROUGH EVERYTHING THAT WE DO. PROMOTING COLLABORATION AND WE WILL TAKE ON PUBLIC HEALTH CHALLENGES THAT ARE VERY œTHESE DOMAINS.UCH ON MANY OF SUCH AS THE HEAL PROGRAM WITH THE OPIOID CRISIS, HIV AND OTHERS. I'M GOING TO FINISH. THAT'S IT. I DIDN'T SHOW YOU THE FULL HEADINGS FOR THE SPIRE THING. YOU WILL SEE THEM LATER WHEN WE BRING IT BACK TO YOU IN THE FALL. I WANT TO END WHERE WE WHERE WITH THE BOLD PREDICTIONS SO THERE WERE 14. I PUT SIX ON THE SLIDES AND I JUST WANT TO HIGHLIGHT TWO, ONE OF YOU EXAMPLE WHERE WE HIT IT OUT OF THE PARK AND ANOTHER ONE WHERE WE GOT THE SECOND BASE. FIRST IS CANDIDATE VACCINE THAT INDUCES BROAD AND BODY RESPONSE TO MULTIPLE INFLUENZA AS WE STEP FORWARD TO FLU VACCINE. THERE'S FOUR CANDIDATE VACCINES NOW IN DIFFERENT STAGES OF CLINICAL TRIALS. MAP EWE SCRIPT IS BEING PREPARED THAT REPORTS ON TWO TRIAL RESULTS IN 020. THAT'S UT OF THE PARK. ON THE BOTTOM IS EXAMPLE OF FIRST OR SECOND BASE. WE PREDICTED THAT THERE WOULD BE A DOZEN NEW FDA APPROVED THERAPY FOR RARE DISEASE. IT'S MORE LIKE EIGHT. THOUGH SOME WENT OUT OF THE PARK. SOME LIKE CF THERAPIES REPORTED IN OCTOBER ARE OUT OF THE PARK. BUT THEN THERE'S OTHERS. >> THERE WAS ONE MORE APPROVED TODAY. SO UP TO NINE. >> WE CAN STILL MAKE IT. ANOTHER MIGHT BE ANTISENSE THERAPIES FOR DOES DUCHENNE MUSCULAR DYSTROPHY. WHAT WE WILL DO, WE WOULD LIKE TO PRODUCE REPORTS ON THE PROGRESS HERE BUT TO KEEP THE DOCUMENT GRACEFUL WE ARE PULL IT OUT APPENDIX OR PUT IT ON THE THE WEBSITE NEXT TO THE REPORT. SO I WILL LEAVE IT THERE. THAT'S WHERE WE ARE, WE WILL. COBACK TO YOU LATER NEXT YEAR. >> THANK YOU, JIM. COMMENTS, QUESTIONS. JUDITH. >> OKAY. >> TURN YOUR MIC ON PLEASE. >> YOUR MAIN OBJECTIVES I DIDN'T NOTICE RIGOR REPRODUCIBILITY IN THERE AND I THINK THAT THAT IS SOMETHING THAT SHOULD BE WOVEN IN THERE SOMEHOW. >> IT IS. VERY STRONGLY YES. I TRY TO KEEP THIS FROM BEING INVENTORY. >> >> THANKS, JIM LOOKING FORWARD TO SEEING THE MORE FLECKERED OUT VERSION. COUPLE OF SMALL ONES, ONE JUST OBSERVATION ON YOUR BOLD ONES I REALLY LIKE THE LAST ONE, THE LAST SLIDE THAT DIDN'T GET TO OF THE NIH WILL SHOW THE SIX THINGS DON'T WORK. I THINK THAT IS A WONDERFUL CATEGORY. AND I HOPE THERE ARE MORE OF THOSE COMING. I DON'T KNOW WHERE TO LOOK FOR THEM BUT I THIS I THAT'S LESS GLAMOROUS AND AT LEAST AS VALUABLE AS SHOWING NEW THINGS THAT DO WORK. >> GOOD SUREDSHIP. >> THANK YOU. THAT WAS NOT THE MOST POPULAR ONE WHEN WE PUT IT FORWARD ON THIS LIST SOME PEOPLE ARE LIKE YOU'RE GOING TO DO WHAT? >> I HAVE ONE QUESTION WHICH I THINK IS PROBABLY JUST -- PROBABLY A BORING ANSWER BUT I NOTICE TWO OBJECTIVES ARE THE FIRST PLAN KIND OF DROPPED OFF AND ONE WAS FOSTER INNOVATION BY SETTING PRIORITIES AND OTHER IS MANAGED FOR RESULTS. CURIOUS DEEP THOUGHT THERE OR JUST REARRANGE TABLE OF CONTENTS? >> STRONG ELEMENTS STILL EXPLAIN IN DIFFERENT PARTS. >> ABSOLUTELY. THESE ARE VERY HEALTH ORIENTED, THAT'S GREAT BUT WHAT ABOUT A COUPLE OF PREDICTIONS ON BASIC RESEARCH? FUNDAMENTAL RESEARCH. HOW ABOUT THAT? >> MY OWN SENSE WERE THAT THESE WERE BASIC. I HAVE ONLY SHOWN SIX OF 14. EXAMPLES ACROSS THE SPECTRUM INCLUDING HEALTH DISPARITIES RESEARCH. THERE WERE FOR EXAMPLE, SO THE TOP ONE HERE, WAS ACTUALLY MINE. >> WHICH WAS ANTICIPATING THE APPLICATION OF CRYO-EM. AND HOW TO TRANSFORM NOT ONLY WORK FORCE STRUCTURAL BIOLOGIST BUT SEEING LIGANDS BOUND TO RECEPTORS AND SHAPE CHANGES IN THEM AND WE DO HAVE THERE'S MULTIPLE EXAMPLES OF THAT. >> I GATHER THE ACD THINKS THOSE WERE FUN AND THEREFORE IT WOULD BE GOOD IDEA TO HAVE A LIST FOR THIS NEXT ROUND TWO AM I SEEING NODDING HEADS, I'M ENTHUSIASTIC ABOUT THIS. THOUGH YOU MAY PUT FORWARD THINGS THAT WILL FAIL TO HIT, THE IDEA OF PUTTING OUT BOLD IDEAS PEOPLE WOULD GO WOW, YOU MIGHT BE ABLE TO DO THAT, WITH RISKS OF NOT ACHIEVING THE GOAL, IT'S WORTH IT. >> DAVID, WE WILL REPORT ON THE THINGS THAT ARE AFFECTED. >> THE I HAVE ROY, BACK TO JUDITH. >> THE MIC IS NOT ON. >> >> FROM IS ONE LIGHT THAT IS ON AND IT CONFUSES ME BECAUSE I FEL LIKE IT'S ON. I THINK TWO LIGHTS ON NOW SO I SHOULD BE GOOD. INDIVIDUAL ICs DO THEIR OWN STRATEGIC PLANS. IS IT THE EXPECTATION THAT STRATEGIC PLANS WILL FOLLOW IN THE SAME KIND OF GENERAL FORMAT AND IDEAS? OBVIOUSLY ONE WOULD BE DIFFERENT BECAUSE THEY HAVE DIFFERENT DISEASES. BUT IN THE GENERAL FOCUS AREAS, WHOLE APPROACH TO THIS -- >> CONGRESS ASKED FOR A COMMON TEMPLATE SO WE DID DEVELOP IT AND IT HAS COMMON ELEMENTS THEY NEED TO ADDRESS INCLUDING AGAIN WOMEN'S HEALTH, WHATRY THEY GOING TO DO TO ADDRESS NEEDS IN WOMEN HEALTH AND HEALTH DISPARITIES THERE'S A REQUIREMENT FOR THEM NOW. I HAVE TO SAY LOOKING BACK OVER THE YEARS, SOME MORE LIKE STRATEGIC PLANS FOR CURING DISEASE AS OPPOSED TO EXPLANATION OF WHY THAT DISEASE AND WHY THOSE MECHANISMS TO DO IT AND WHY WE NEED TO IMPROVE THE WORK FORCE TO DO IT. SO THIS IS A -- WE ARE HOPING THAT EXPECTING THEY WILL ADDRESS A BROADER PERSPECTIVE ON HOW TO DO RESEARCH. IT DOESN'T HAVE TO LINE BY LINE SAME AS NIH PLAN BUT CONGRESS DID ASK FOR A LITTLE MORE UNIFORMITY. >> JUDITH, LAST COMMENT. >> OVER THE LAST DAY AND A HALF I HAVEN'T HEARD ANYTHING ABOUT CLIMATE CHANGE. WHEN WE ARE TALKING ABOUT STRATEGIC PLAN AND BOLD PREDICTIONS AND WHERE WE WANT TO GO, WE SHOULD BE THINKING HOW WE WANT TO INCORPORATE CLIMATE CHANGE AND HOW THAT WILL IMPACT HUMAN HEALTH AND WHAT THE SCIENCE IS WITH -- >> YOU PROBABLY KNOW THE NATIONAL INSTITUTE OF ENVIRONMENTAL HEALTH SCIENCES IS IN FACT THE LEAD FOR CLIMATE CANGE RESEARCH AT NIH AND HAD BEEN IN THAT ZONE FOR QUITE SOME TIME. AND HAVE A FAIRLY VIGOROUS PROGRAM IN THAT SPACE, THIS IS MORE POLITICALLY CONTENTIOUS THAN IT MIGHT HAVE BEEN BUT IT IS WHAT IT IS AND WE CERTAINLY ARE CONTRIBUTORS TO THAT, TRY TO GET THE DATA. AND DERIVE AS MUCH EVIDENCE AS POSSIBLE ABOUT WHAT CONSEQUENCES ARE OF THE WARMING OF THE PLANET. SO POINT TAKEN. JIM, THANK YOU VERY MUCH FOR PUTTING THIS PLAN IN FRONT OF EVERYBODY. I'M SURE WE WILL HAVE A CHANCE TO LOOK AT MORE DETAILED VERSION COME NEXT JUNE. SO I THINK THAT BRINGS US AFTER A PRETTY INTENSE DAY AND A HALF TO THE END OF THIS MEETING, JOSE ASKED FOR A MINUTE OR TWO TO SAY SOMETHING BY THE WAY OF FAIR WELL TO THE TROOPS OR SOMETHING LIKE THAT. THE FLOOR IS YOURS. >> INDULGE ME. IF YOU HAVE TO GO YOU CAN GO BUT SO I WANTED TO SAY ON MY LAST DAY LAST ROOM FRANCIS WHEN YOU INVITED ME FIVE YEARS AGO TO TAKE PART OF THE ACD I DIDN'T HAVE ANY GRAY HAIR. AND REFLECTING THOSE FIVE YEARS, JUST TO SIGH THREE THINGS. THE FIRST ONE IS IT HAS BEEN AN ABSOLUTE DELIGHT. TO BE ABLE TO PARTICIPATE IN DISCUSSIONS THAT REALLY HAVE FUNDAMENTAL IMPACT ON HOW SCIENCE AND BIOMEDICINE IS CONDUCTED IN THIS COUNTRY. AND TO INTERACT WITH YOU GUYS AND THE NIH PERSONNEL, BRILLIANT AND THOUGHTFUL AND ACCOMPLISHED AND SUCH NICE PEOPLE. SO THAT HAS BEEN AN AMAZING EXPERIENCE FROM POINT OF VIEW OF INTERACTION. THE NIH STAFF DEFINES WHAT I BELIEVE IS A CIVIL SERVANT AND A SCIENTIST SO HAS BEEN GREAT TO LEARN FROM ALL OF YOU. SECOND ON A PERSONAL NOTE I WOULD SAY HAVING BORN IN NEW HAMPSHIRE BUT GREW UP IN SPAIN THEN CAME BACK AT 18 WITH THIS ACCEPT. THAT I ALWAYS FELT A LITTLE BIT OUTSIDER IN SOME LEVELS AND THE SAME TIME I AM PROFESSIONALLY I OWE A LOT TO HAVING BEEN IN THIS COUNTRY, I LOVE THIS COUNTRY AND WHAT I WANT REPRESENTSES TO IS OPPORTUNITY TO BE ABLE TO SERVE THE NATION WAS INC ABLY VALIDATING FROM THAT STANDPOINT. TO BE ABLE TO SAY HERE I AM GIVING SOMETHING BACK FOR EVERYTHING I RECEIVED. A THIRD POINT FRANCIS, I WOULD SAY JUST WORKING WITH YOU AND BEING SCIENTIFICALLY ON TYPES OF DIKES GENETICS WE CAN SHARE BUT ALSO IN TERMS OF LEADERSHIP, IN TERMS OF THOUGHTFULNESS, HUMANITY AS A PERSON, SCIENTIST, A LEADER, HAS BEEN A PRIVILEGE TO BE ABLE TO -- SO FOR THE DELIGHT FOR THE HONOR, AND FOR THE PRIVILEGE, I WANTED TO EXPRESS THANK YOU PUBLICLY. [APPLAUSE] >> JOSE, THANK YOU FOR SUCH KIND WORDS. THIS IS GOING TO BE HARD TO HAVE THIS GATHERING WITHOUT YOU AND JEFF AND LINDA. AND WE WILL HAVE TO FIGURE OUT HOW TO FIND OTHER PEOPLE WITH SIMILAR WISE VISIONARY PERSPECTIVES. THIS HAS BEEN A REMARKABLE GROUP CURRENTLY GATHERED HERE. SERVING US IN THIS WAY, PROVIDING THIS ADVICE. I WANT TO THANK ALL OF YOU FOR THE WAY IN WHICH YOU HAVE HELPED US IN THE LAST DAY AND A HALF. WHEN YOU THINK ABOUT THE NUMBER OF TOPICS THAT WE HAVE ASKED YOU TO WRESTLE WITH, IT IS BREATHTAKING, WE OF COURSE RANGE EVERYTHING FROM DEEP QUESTIONS HAVING BEEN AUTHOR OF THE BRAIN INITIATIVE, HAVING BEEN AUTHOR OF PRECISION MEDICINE AND ALL OF US, NOW HELPING US AUTHOR A WHOLE NEW PLAN IN ARTIFICIAL INTELLIGENCE BUT THEN ON TOP OF THAT WE HAND YOU REALLY HARD QUESTIONS ABOUT OUR WORK FORCE IN TERMS OF DIVERSITY, HOW TO BE SURE WE ARE SUPPORTING THE NEXT GENERATION OF RESEARCHERS AND BEST POSSIBLE WAY, AND THEN SOME REALLY THORNY POLICY ISSUES LIKE CONVERSATION AND RECOMMENDATIONS WE WRESTLED WITH YESTERDAY AFTERNOON ON SEXUAL HARASSMENT. YOU SEEM CAPABLE WRAPPING YOUR ARMS AROUND THOSE THINGS AND PROVIDING US WITH THE INSIGHT WE NEED. I DIDN'T HAVE WHITE HAIR WHEN I STARTED AS NIH DIRECTOR EITHER BUT THIS IS MY 11th ACD MEETING AND PROBABLY ONE OF THE MORE INTENSE ONES I CAN RECALL BUT ALSO GRATIFYING TO HAVE THE CHANCE NOT TO SHY AWAY FROM VERY TOUGH DIFFICULT ISSUES AND SEEING WHAT WE CAN DO, THAT'S OUR RESPONSIBILITY. SO I WANT TO THANK YOU AGAIN FOR YOUR CONTRIBUTIONS FOR YOUR COMMITMENT OF TIME BUT PARTICULARLY COMMITMENT OF YOUR PERSON HOOD,INGS YOUR MIND, YOUR INTELLIGENCE YOUR WISDOM TO HELP US MAKE THE DECISIONS AS BEST ADS WE CAN GIVEN THE CIRCUMSTANCE. FINALLY I WANT TO SAY A COUPLE OF WORDS OF THANKS OF STAFF BEHIND ME. DEGREE SHEN AND CINDY, MANY OTHERS WHO HELPED MAKE THIS HAPPEN PARTICULARLY TO LARRY TABAK WHO HAD OPPORTUNITY AS IT WERE TO MAKE SURE WE SET UP A SCHEDULE GOING TO WORK AND VET IT ALL PRESENTATIONS, EXCEPT FOR DAVID'S BECAUSE WE ONLY SAW THAT ONE THIS MORNING. AWFUL LOT OF WORK GOES IN TO MAKING THIS THING HAPPEN. MANY THANKS TO THOSE FOLKS. WITH TEN MINUTES TO DO WHATEVER YOU WANT TO DO, I THINK GET READY MIKE, HOLD YOUR EARS. WE ARE ADJOURNED.