>> WELCOME TO THE SECOND DAY OF THE 106th MEETING OF THE ACD. HOPE YOU ALL WERE ABLE TO BE THERE, ENJOYED LAST NIGHT'S SHIN DIG AT THE SMITHSONIAN AND UNVEILING OF THE GENOME EXHIBIT WHICH REALLY QUITE AMAZINGLY PUT TOGETHER. I MUST SAY I WANT TO GO BACK AND SPEND A FEW HOURS WHEN NOT TRYING TO TALK TO THESE PEOPLE THAT I HAVEN'T SEEN IN A LONG TIME BECAUSE THERE'S A LOT OF INTERESTING REUNION EXPERIENCES GOING ON IN THE HALLWAYS DOWN THERE AT THE NATIONAL MUSEUM OF NATURAL HISTORY. BUT OUR HATS ARE OFF TO THOSE WHO WORKED CLOSELY BETWEEN SMITHSONIAN AND THE GENOME INSTITUTE TO CREATE THIS JUST AT THE RIGHT MOMENT HERE THE 10TH ANNIVERSARY OF THE COMPLETION OF THE GENOME PROJECT. SO THAT WAS A LOT OF FUN. WE HAVE STUFF TO DO THIS MORNING. WE HAVE LOST A COUPLE OF MEMBERS WHO WERE OFF GIVING COMMENCEMENT ADDRESSES TODAY. IT IS THAT SEASON. THAT'S WHY WE DON'T HAVE MARY SUE AND REED, I THINK THOSE ARE THE TWO I KNEW ABOUT BUT WE HAVE A QUORUM AND STUFF TO DO. SO LET'S BEGIN WITH SERIOUS LOGISTICAL AND HOUSEKEEPING THINGS THAT WE NEED TO TAKE CARE OF. FIRST I NEED TO ASK FOR ANY COMMENTS OR CORRECTIONS IN THE MINUTES FROM THE DECEMBER 6, 7 MEETING WHICH IS IN YOUR BOOK UNDER TAB 11. ARE THERE ANY COMMENTS OR CORRECTIONS? WE TRY TO BE FAIRLY CAREFUL ABOUT THIS BUT IF YOU FOUND ANYTHING YOU DIDN'T LIKE, YOU CAN SAY SO. HEARING NONE MAY I HAVE A MOTION FOR ACCEPTANCE. >> SO MOVED. >> SECOND. >> SECOND. >> ANY OTHER COMMENTS OR DISCUSSION? ALL IN FAVOR PLEASE SIGNIFY. THANK YOU. NOW WE ALSO HAVE A REGULAR TASK WE TAKE CARE OF REWE VIEW OUTSIDE OF FOR ACD APPROVAL. DR. TABAK GETS ALL THE FUN JOBS. >> NOW I'M WEARING WEARING THE OTHER HAT THAT I WEAR WHICH IS DEPUTY ETHICS COUNSELOR FOR THE AGENCY. IF I CAN DIRECT YOUR ATTENTION TO TAB 12 IN YOUR BINDERS YOU'LL FIND THE LIST OF AWARDS FOR ARCCD REVIEW AND APPROVAL. FEDERAL LAW PROHIBITS FEDERAL EMPLOYEES FROM RECEIVING ADDITIONAL MONEY DURING THE PERFORMANCE OF THEIR JOBS, THE AWARDS THAT ARE PRESENTED TO YOU TODAY WERE PRE-SCREENED BY DR.S JENKINS AND WILSON AND HAVE ALSO BEEN REVIEWED AND DEEMED ELIGIBLE UNDER OURETHICS REGULATION BY THE NIHETHICS OFFICE. RENEE, I DON'T KNOW IF YOU HAVE ANY COMMENTS TO MAKE. (OFF MIC) >> THANK YOU. SO DO ANY MEMBERS OF THE ACD HAVE ANY QUESTIONS OR COMMENTS? OKAY. SO IF I COULD ASK ALL THOSE WHO ARE IN FAVOR OF APPROVING THESE AWARDS, IF YOU COULD SIGNIFY BY RAISING YOUR HANDS, PLEASE. THANK YOU. ANY OPPOSED? ALL RIGHT. THANK YOU. >> WELL, WE CAN NOW MOVE ON TO COUPLE OF VERY SUBSTANTIVE TOPICS. WHICH ARE GOING TO OCCUPY US FOR THE REST OF THE MORNING. I MENTIONED A MINUTE AGO WHAT A NICE EVENING WE HAD LAST NIGHT AND I DO WANT TO RECOGNIZE ERIC GREEN FOR THE REMARKSABLE EFFORT HE AND HIS TEAM VINCE BONNHAM LAURA RODRIGUEZ AND OTHERS PUT TOGETHER CREATING THEIR PART OF THIS EXHIBIT THAT WE UNVEILED. I'M SURE CONSUMED MORE HOURS THAN YOU WOULD HAVE IMAGINED WHEN THE IDEA FIRST POPPED UP TWO YEARS AGO. BUT THOSE MILLIONS OF PEOPLE WHO WILL SEE THE EXHIBIT ARE GOING TO BE WELL SERVED BY IT. ONE OTHER THING, I'M PASSING AROUND TO Y'ALL A GRAPHIC WHICH I MENTIONED YESTERDAY, WHICH IS THE THING THAT I SHOWED IN THE SENATE APPROPRIATIONS HEARING THAT SEEMED TO HAVE CAPTURED THE ATTENTION OF BOTH PARTIES SHOWING IN THAT INSTANCE ON A PERCENTAGE BASIS WHAT'S HAPPENING TO SUPPORT FOR BIOMEDICAL RESEARCH IN QUITE A FEW COUNTRIES IN THIS CASE BETWEEN 2011, 2012, THIS DOESN'T INCLUDE THE SEQUESTER. THE GRAPHIC SPEAKS FOR ITSELF IN TERMS OF WHAT'S HAPPENING ELSEWHERE AND WHAT'S NOT HAPPENING HERE. SO FEEL FREE TO WAVE THAT AT PEOPLE IF YOU THINK IT WOULD BE HELPFUL FOR A CAUSE. THIS IS THAT'S GOING 11 TO 12, THAT DOESN'T INCLUDE 13. 13 WOULD BE WORTH. (OFF MIC) >> SURE. THIS COMES, DATA COMES FROM A PAPER PUBLISHED IN CELL SO THE REFERENCE IS AT THE BOTTOM IN CASE ANYBODY QUIZZES YOU ABOUT HOW DO YOU KNOW THAT'S RIGHT? WE CAN SEND AROUND THE POWERPOINT IMAGE OF IT. ERIC IS NOW GOING TO TALK TO US ABOUT BIG DATA KNOWLEDGE. ERIC IS WEARING TWO HATS NOW. AS DIRECTOR OF NATIONAL GENOME RESEARCH INSTITUTE BUT ALSO ACTING ASSOCIATE DIRECTOR FOR DATA SCIENCE, A TASK HE'S TAKEN ON WITH GREAT ENERGY AND CONVENED ACROSS NIH A LARGE NUMBER OF OTHER FOLKS WITH INTEREST IN BIG DATA TO PARTICIPATE SO YOU WILL BE SPEAKING ABOUT A PRETTY LARGE ENTERPRISE, ONE THAT IS TAKING SHAPE RATHER QUICKLY AND MOVING IN AN INTERESTING DIRECTION. HE'S ALSO I THINK GOING TO TELL YOU SOMETHING ABOUT WHAT WAS ANNOUNCEDDED JUST A WEEK AGO THIS GLOBAL ALLIANCE FOR HOW TO MANAGE DATA STANDARDS FOR GENOME INFORMATION IN THE CLOUD WHICH HAS BEEN A MULTI-PARTY EFFORT BUT I THINK YOU WILL ALSO FIND HIGHLY RELEVANT. ERIC, THANK YOU FOR EVERYTHING YOU'RE DOING HERE. YOU'RE HAVING QUITE A WEEK. >> GOOD MORNING, EVERYBODY. GREAT TO SEE Y'ALL AGAIN. THOSE OF YOU ABLE TO MAKE IT LAST NIGHT THANKS SO MUCH FOR JOINING. I JUST TELL YOU A COUPLE OF THINGS YOU MAY NOT HAVE REALIZED ABOUT YESTERDAY, YOU COULDN'T HAVE ASKED FOR A CRAZIER TIME. PLANNING THAT EXHIBITION HAS BEEN OBVIOUSLY MONTHS AND MONTHS, YEARS OF EFFORT, PLANNING LAST NIGHT WAS LIKE PLANNING A WEDDING, IT WAS UNBELIEVABLY COMPLICATED. WHAT WE DID DURING THE DAY, NOT SURE PEOPLE APPRECIATED, IT WAS ALL DAY BECAUSE WE HAD A PRESS EVENT TO UNVEIL THE EXHIBITION AND THEN IN THE MORNING. IN THE AFTERNOON WE HAD BASICALLY A SCIENCE WRITERS WORKSHOP FILLED, A ROOM FILLED WITH SCIENCE WRITERS COVERING THE GENOMICS AND GENETICS BEAT AND WE LINED UP SPEAKERS. JIM EVANS SPOKE, LAURA SPOKE AND SO FORTH WOMEN HAD NO IDEA THE SUPREME COURT THAT DAY WAS GOING TO SEND -- HAND THIS DECISION WHICH WAS GREAT BECAUSE WE HAD THIS CAPTIVE AUDIENCE WHICH ALL WANT TO GO TO WRITING STORIES, ONE ABOUT THE EXHIBITION SO THE TIMING WAS REMARKABLE. AND THEN A LITTLE SECRET I COULDN'T SAY LAST NIGHT BUT I CAN TELL YOU NOW, IF IT WASN'T CRAZY ENOUGH, I LEARNED YESTERDAY MORNING AND SWORN TO SECRECY, YOU DIDN'T REALIZE BUT IN THE EXAM LAST NIGHT WHILE WE WERE IN BARE ROT AUDITORIUM, MICHELLE OBAMA HAD A SMALL GROUP SHE CAME OVER WITH TO SEE A COUPLE OF EXHIBITIONS, MOSTLY IN THE HUMAN ORIGIN EXHIBITION, THERE WAS AN ATTEMPT TO HAVE HER COME LOOK AT THE GENOME ONE BUT I DON'T THINK SEQUENCERS WOULD ALLOW OR THINGS THEY SCRIPTED, SECRETARY (INAUDIBLE) RAN OFF AND HAD MET WITH HER. APPARENTLY TOLD HER ABOUT THE EXHIBITION SHE PROMISED TO COME BACK TO SEE IT. I DON'T KNOW WHEN THAT WILL BE BUT THAT'S VERY EXCITING BUT IT WAS EXHAUSTING AND IT WAS LIKE PLANNING A WEDDING. I WENT HOME LAST NIGHT AND TOLD TWO TEENAGE KIDS THEY HAVE TO ELOPE BECAUSE I CAN'T GO THROUGH THIS EVER AGAIN. IT'S EVEN MEANER THAT I'M COMING TO YOU GUYS TO TALK ABOUT -- I COULDN'T COME YESTERDAY BECAUSE I WAS BUSY ALL DAY, BUT I'M HAPPY TO BE HERE. AND TO TELL YOU CHANGING HATS FROM THE SMITHSONIAN TO THE OTHER THING YOU'RE DOING TO GIVE AN UPDATE. BEFORE I START LET ME INTRODUCE THOUGH MANY OF YOU KNOW MARK GUYER, IN THE BACK CORNER. HE IS DEPUTY DIRECTOR OF NHGRI AND PREVIOUSLY DIRECTOR OF EXTRAMURAL RESEARCH PROGRAM AND FOR MANY MAJOR THINGS WE'RE GOING TO TALK ABOUT HE AND I PARTNER TO MOVE THIS FORWARD. I WILL TELL YOU ABOUT HIS ROLE AND I WOULDN'T BE SURPRISED AT THE QUESTIONS YOU HAVE I WILL ASK MARK TO HELP ANSWER. THIS IS A TOPIC OF A SLIDE YOU HAVE SEEN BEFORE, YOU HAVE HELPED US DEAL WITH THIS A COUPLE OF YEARS. THE MAIN MESSAGE OF COURSE IS THAT WE ARE IN A MONO PLACE IN BIOMEDICAL RESEARCH, WE ARE AT THE GROWN UPS TABLE WITH WITH RESPECT TO BIG DATA, WE GET TO JOIN PARTICLE PHYSICISTS AND CLIMATOLOGISTS AND ASTRONOMERS, IT'S A NEW REALITY FOR US. I'M HERE SPEAKING, HAVE THIS NEW ROLE BECAUSE GENOMICS IN MANY WAYS IS A POSTER CHILD FOR THIS BUT I WANT TO EMPHASIZE WE'RE TALKING ABOUT MANY DATA TYPES IN BIOMEDICAL RESEARCH, JUST THAT GENOMICS HAVE DEFINED IT IN A WAY THAT MADE IT CLEAR. BUT THE NO MEANS THE ONLY DATA WE'RE TALKING ABOUT WHETHER PROTEOMICS, THERE ARE OTHER LARGE DATE SETS BEING GENERATED AND DEALT WITH. SIMILAR TO SURGEONS IN TECHNOLOGIES FOR ANALYZING DNA, I GUESS YOU HEARD FROM STORY LANDIS YESTERDAY YOU HER ABOUT SURGES IN TECHNOLOGY AND IMAGING AND WHETHER THAT'S CELLULAR IMAGEING OR ORGANIST MALL TISSUE BEING GENERATED. WE'RE GETTING PHENOTYPIC MEASUREMENTS AND DATA, INCREASE LEG CERTAINLY FUTURE WANT TO HAVE MORE AND BETTER TECHNOLOGIES FOR MEASURING EXPOSURES. AND OF COURSE WE ARE ALL SAL INVESTIGATING AT PROSPECT OF BEING ABLE TO HAVE BETTER ELECTRONIC HEALTH RECORD AND MINE THOSE AS CLINICAL DATA AS PART OF VERY LARGE STUDIES. SO MANY WAYS WE'RE VICTIM ZS OF OUR OWN SUCCESS, WE WERE SUCCESSFUL TECHNOLOGY DEVELOPMENT PUSHED THIS TO A NEW AREA, WE ARE NOW IN A SITUATION WHERE DATA GENERATION IS NOT THE RATE LIMITING STEP. IT'S ANALYZING THAT DATA. THE OTHER KEY MESSAGE I THINK YOU AGREE WITH, THIS IS NOT A TWO OR THREE YEAR PROBLEM CH THIS IS THE NEW REALITY FOR BIOMEDICAL RESEARCH. WE ARE A DATA SENSESIVE ENTERPRISE AND WE NEED TO READY OURSELVESES TO FACE THAT REALITY. SO EARLY IN 2011 FRANCIS ASKED H THIS GROUP TO HELP APPOINT A WORKING GROUP OF THE ACD CO-CHAIRED WHICH LARRY TABAK AND ASKED THEM TO DELIVER A REPORT SOMETHING LIKE A YEAR AND A HALF AND REALLY ALMOST TO THE DAY I GUESS IT IS ALMOST TO THE DAY, TOMORROW IS THE ONE YEAR ANNIVERSARY OF WHERE THIS REPORT WAS DELIVERED, PROBABLY IN THIS ROOM TO THIS GROUP. AND OFF WE WENT TO TAKE THESE RECOMMENDATIONS AND TO TURN THEM INTO SOMETHING THAT COULD BE IMPLEMENTED. WE HAD BEEN SELF-ORGANIZING HERE AT NIH TO BE READY FOR THESE RECOMMENDATIONS AND ACTUALLY INTERACTED WITH THE WORKING GROUP ALONG THE WAY SO WE WOULD HIT THE GROUND RUNNING RATHER THAN HAVING A HEAD START TO TAKE ON WHAT NEEDED TO BE DONE. THERE WAS AN UPDATE ABOUT WHAT WAS HAPPENING IN THE DECEMBER MEETING LARRY TABAK GAVE A SUMMARY THAT SCRIPTED SOME OF THE THINGS HAPPENING THAT WE WERE OFF AND RUNNING ABOUT AND WHAT HAD BEEN DECIDED. I'LL BE ABLE TO UPDATE YOU ABOUT THAT. I WOULD POINT OUT AS A REMINDER THAT DATA INFORMATICS WORKING GROUP TACKLED TWO MAJOR AREAS OF DATA. ON THE ONE HAND THE SCIENTIFIC DATA WHICH IS WHAT I'M GOING TO TALK ABOUT AND TELL YOU WHAT'S GO ON THERE AND THE OTHER COMPONENT WAS ADMINISTRATIVE DATA THAT DEALT WITH ON CAMPUS ISSUES WITH RESPECTl9 ALSO THE HUGE AMOUNTS OF ADMINISTRATIVE DAYTIME ASSOCIATED -- DATA WITH THE NIH ENTERPRISE. THERE ARE COMPLETELY SEPARATE SOLUTIONS TO TACKLE EACH, YOU TALK THE INFRASTRUCTURE PLUS INITIATIVE. I WON'T TALK ABOUT ANY OF THAT, THERE IS COORDINATION BETWEEN THE ACTIVITIES WHERE THEY TOUCH, BY AND LARGE NEED SEPARATE SOLUTIONS ON SEPARATE PATHS TOWARDS BEING ON. IN TERMS OF BIOMEDICAL BIG DATA WE KNEW AND THE WORKING GROUP SORT OF ABSOLUTELY HELPED POINT OUT THAT WE WERE AT A PIVOTAL POINT IN THE BUY MEDICAL RESEARCH ARENA, WHEREBY RISK, CAPITALIZE ON THESE WONDERFUL TECHNOLOGICAL ADVANCES WHETHER DNA SEQUENCING PHENOTYPING AND SO FORTH. AND RESOLVE -- RUSS AT MAP POINTED OUT WE HAVE TO DO SOMETHING IF NOT IT WILL BE INSTITUTIONAL MALPRACTICE BUT IT E NOT A MATTER OF THROWING MUCH TO HAVE GRANTS AND CONTRACTS, NOT JUST BUILDING NEW SOFTWARE TOOLS OR PUTTING THINGS UP IN THE CLOUD OR SIMPLY HAVING BETTER APPROACHES FOR DATA ANALYSIS. THERE NEEDS TO BE CULTURAL CHANGES, YOU CAN SEE HOW THIS WILL COME FULL CIRCLE AT THE VERY END.'o VARIOUS AREAS OF CULTURAL CHANGE ARE NEEDED ALONG THE WAY. WE DEVELOP NEW OPPORTUNITIES FOR SHARING DATA FOR ANALYZING DATA AND VERY IMPORTANTLY FOR INTEGRATING DATA. IT REALLY WAS RECOGNIZED THAT IT'S NOT JUST A MATTER OF SOLVING A PROBLEM EACH DIFFERENT DATA DOMAIN YOU NEED TO WORK THROUGH IN AN INTEGRATIVE WAY TO CAPITAL SIZE OPT WEALTH OF DATA OUT THERE. AS I SAID, WE'RE NOT TALKING SHORT TERM SOLUTION, I OOHs A LONG TERM SOLUTION. I LIKE THIS QUOTE. I THOUGHT IT WAS WELL PUT BY DATA INFORMATICS WORKING GROUP BUT WITHIN THIS SPIRIT WE WERE EFFECTIVE OVER THE LAST YEAR TO REALLY GET CONSENSUS ACROSS THE HIPPOTO REALLY HAVE A SOLUTION BE A CORPORATE SOLUTION, WHERE THEY TALK A FINAL KEY STRATEGIC CHALLENGES TO ENSURE THE NIH CULTURE CHANGES OR WITH RECOGNITION OF KEY ROLE INFORMATIC COMMUTATION FOR EVERY IC MISSION AN INFORMATICS AND COMPUTATION SNOT CHAMPIONED BY A FEW ICs BASED ON PERSONAL VISION OF PARTICULAR LEADERS BUT THE LEADERSHIP MUST ACCEPT ADVANCED COMPUTATION SUPPORTING SUPPORTING THE RESEARCH PORTFOLIO OF EVERY IC. I WOULD LIKE THE THANK MY INSTITUTE AND CENTER DIRECTOR COLLEAGUES AND BUILDING ONE LEADERSHIP BECAUSE EVERYBODY REALLY CAMING TO OVER THE LAST YEAR. AS FAR AS I KNOW ALMOST WHAT WE'RE PROPOSING IS GOING TO MOVE FORWARD HAPPENING AT TIME OF PHYSICAL CONDITION STRAINT. EVEN BOUGHT INTO THIS AND THAT MADE THE JOB A HECK OF A LOT EASIER. SO PROBLEMS THAT NEED TO BE SOLVED, LET'S REVIEW YOU HAVE TO FIND THE DATA OUT THERE, WE NEED TO BE ABLE TO ACCESS THE DATA OUT THERE. WE NEED BETTER POLICIES AN PRACTICES FOR STRUCTURING THAT DATA, ORGANIZING, MANAGING AND PROCESSING THAT DATA. BETTER METHODS FOR ANALYZING THAT DATA. INTEGRATING, LOTS OF QUESTIONS AND LOTS OF CHALLENGES AROUND TRAINING RESEARCHERS, UP AND COMING RESEARCHERS AND SCIENTISTS ABOUT INDIVIDUALS WHO WANT TO CROSS TRAIN TO FACILITATE THEIR USE OF DATA IN DIFFERENT DOMAINS. SO WHAT'S THE SOLUTION? SOLUTION WHICH VERY MUCH WAS SCRIPTED BY THE RECOMMENDATIONS OF THE WORKING GROUP AND WE ARE FOLLOWING VERY, VERY CLOSELY. WE ARE TACKLING THIS DATA PROBLEM FIRST RELATED TO OVERSIGHT AND LONG-TERM ABILITY TO GOVERN WHAT IS NEEDED AND PROGRAM MATICALLY LEAD IN THIS AREA HERE WITHIN NIH AND WE NEED NEW INITIATIVES THAT ARE GOING TO HELP LAUNCH THINGS THAT NEED TO BE DONE. SO THERE'S THREE PRONGS ASSOCIATED WITH THIS. ONE IS THE CREATION OF A NEW POSITION, LEADERSHIP POSITION NIH CALLED ASSOCIATE DIRECTOR FOR DATA SCIENCE. THERE IS ALSO A NEW GROUP AT HIPPOCALLED THE SCIENTIFIC DATA COUNSEL RECOMMENDED BY THE WORKING GROUP TO TAKE -- HAVE A LONG TERM RESPONSIBILITY FOR OVERSEEING DATA SCIENCE ACTIVITIES HERE AT NIH. THEN BIG DATA KNOWLEDGE, NOTICE THE OVERLAPPING VIN DIAGRAM, THEY'RE EXTENSIVELY OVERLAPPING AND ENTERRELATEDDED IN VERY IMPORTANT WAYS. LET ME TELL YOU ABOUT EACH. FIRST LET'S START WITH ASSOCIATE DIRECTOR FOR DATA SCIENCE AN DOWN TO THE LAST HOUR BEFORE THIS WAS ANNOUNCED, WHAT WORDING SHOULD WE USE FOR THIS TITLE? I THINK IT'S LOGOS. WE KEPT TALKING ABOUT DATA THAT'S DONE SOPHISTICATED WE HAVE A LOT OF NAMES ALL WHICH ARE RELEVANT WHETHER BIOINFORMATICS OR COMPUTATIONAL BIOLOGY. BIOMEDICAL INFORMATICS IS IMPORTANT. INFORMATION SCIENCE, BIOSTATISTICS, IT INCLUDES QUANTITATIVE BIOLOGY. WHAT WE DECIDE TO GO WITH WAS DATA SCIENCE, UP AND COMING TERM IF YOU WILL, RATHER GENERIC BUT IN MANY WAYS IT ENCOMPASSED WORDS NEUTRAL ENOUGH THAT EVERYBODY COULD L BUY INTO IT SO THAT'S THE NAME WE WENCH WITH. ADMITTEDLY ONE REASON WE WENT WITH THAT NAME IS WHEN YOU'RE IN DOUBT ABOUT WHAT NAME TO DO WITH, YOU SHOULD ALWAYS GO WITH THAT. WE HAD JUST SEEN HARVARD BUSINESS REVIEW ARTICLE TALKING DATA SCIENTIST BEING THE SEXIST JOB OF THE 21st CENTURY THE SHORTAGE OF DATA SCIENCE IS BEING BECOMING A SERIOUS CONSTRAIN IN SOME SECTORS INCLUDING OURS WITH THAT O DEEMED APPROPRIATE, HE WAS EVEN A GRAPHIC HARVARD BUSINESS REVIEW PUT OUT OF IT. IF YOU LOOK AT CIRCLE MOST STATISTICS MATH, SCIENCE DATA, THAT'S WHAT WE'RE TALKING ABOUT. SO WE WENT WITH DATA SCIENCE. BY THE WAY, WE WENT WITH THAT. AND SURE ENOUGH THIS YEAR IN NATURE CAME UP THIS ARTICLE BY THIS INDIVIDUAL, A VISION FOR DATA SCIENCE AND TALKING ABOUT HOW TO SOLVE BIG DATA CHALLENGES RESEARCHERS NEED SKILLS IN SCIENCE AND COMBINATION TOO RARE NEW BREED OF DATA SCIENTIST. SO I THINK FRANCIS MADE THE RIGHT DECISION, GOING WITH THE PHRASE DATA SCIENCE. WHAT WILL ASSOCIATE DIRECTOR DO, WHAT ARE THE RESPONSIBILITIES HERE? SIMPLY PUT THIS IS THE PROGRAMMATIC. THIS IS THE POINT PERSON. THIS IS THE STRATEGIC LEADER. THIS IS THE LOOK AT NIH, THIS IS THE INDIVIDUAL, WE CAN TALK TO THAT WILL BE ABLE TO GET US THE INFORMATION WE WAN OR BE ABLE TO INTERACT WITH IN A WAY THAT REPRESENTS NIH AT A HIGH LEVEL LEADERSHIP POSITION TO BE HIGH LEVEL LEADERSHIP POSITION IT WAS IMPORTANT THAT I THINK IT REPORT IT TO THE NIH DIRECTOR AND FRANCIS AGREED, POSITIONINGS POINT TO THE NIH DIRECTOR DONE GET CREATED EVERY DAY OR WEEK. THEY RECOGNIZE THIS WAS IMPORTANT. (OFF MIC) >> 46. YOU DON'T LIKE THAT NUMBER TO GROW BUT YOU'RE WILLING TO LET IT GROW. HE ASKED ME TO MANY PEOPLE WORKED CLOSELY WITH FRANCIS WE DONE KNOW HOW TO SAY NO O I WAS WILLING TO DO IT ON AN INTERIM BASIS. I ALSO CO-CHAIR THE SEARCH COMMITTEE WITH JIM ANDERSON AND WE WORK VERY HARD BECAUSE I WANT TO GET RID OF THIS JOB AND PUT IN THE HANDS OF SOMEONE WHO REALLY HAS EXPERTISE THIS THIS AREA. SOME MAY HAVE SEEN THIS AD BEFORE THIS POSITION, THE SEARCH COMMITTEE IS VERY ACTIVE, THE INITIAL ROUND OF APPLICATIONS WERE DUE BY MAY 13th. IN FACT WE ALREADY VERY MUCH IN THE MODE OF EVALUATING THOSE œ INDIVIDUALS BUT IT IS AN OPEN UNTIL FILLED POSITION ANY TIME A CANDIDATE COMES IN WE CAN CONSIDER AN WORK THEM INTO THE PROCESS OF BEING EVALUATED. SO FEW IF YOU KNOW PEOPLE, HAVE THEM CONTACT ME, WITH WE'RE STILL OPEN TO TALKING TO PERSPECTIVE P CANDIDATES. WHAT ABOUT THE SCIENTIFIC DATA COUNCIL? THIS IS A HIGH LEVEL INTERNAL NIH GROUP TO PROVIDE PROGRAMMATIC LEADERSHIP AND COORDINATION OF DATA SCIENCE ACTIVITIES. IT WILL BE CHAIRED BY ASSOCIATE DIRECTOR FOR DATA SCIENCE, INTERIM BASIS BUT EVENTUALLY THE PERMANENT AD. IT HAS VERY STRONG TRANS-NIH REPRESENTATION AND I ALSO HAVE HAND OUTS SO SOME OF THESE SLIDES GO QUICKLY, WE'RE INTERESTED TO SEE WHO FRANCIS AND LARRY AND I DECIDED TO HAVE SERVE ON THESE, THESE INDIVIDUALS YOU CAN SEE IT'S INSTITUTE DIRECTORS AND DEPUTY DIRECTORS AND SCIENTIFIC DIRECTORS, SCIENTIFIC OFFICERS HIGH LEVEL INDIVIDUALS SERVING THIS ROLE. INAUGURAL SCIENTIFIC DATA COUNSEL NOW MET TWICE, WE'RE STARTING TO GET OUR LEGS UNDER US AND WE CLEARLY HAVE LOTS TO DO. THE EASE CREST WAY TO SUM PRIZE -- EASIEST WAY TO SUMMARIZE, THISES THE AD LEADING THIS GROUP SOY PUT TOGETHER THE JOINT RESPONSIBLE. THOSE ARE TRIVIAL SO FOR SIMPLICITY I WOULD SAY JOINTLY. THIS LEADERSHIP POSITION WORKING CLOSELY WITH THE HIGH LEVEL INTERNAL NIH GROUP, FIRST OF ALL WILL PROVIDE OVERSIGHT OF THE THIRD COMPONENT OF THIS SOLUTION THEY DATED KNOWLEDGE INITIATIVE I'LL GET TO IN A MINUTE BUT THE INTERNAL GROUP TO OVERSEE IT. THIS IS A GROUP AND LEADERSHIP POSITION TO PROVIDE TRANS-NIH INTELLECTUAL AND PROGRAMMATIC HUB FOR DATA SCIENCE. HELP COORDINATE AND CONVENE. THERE IS A LOT OF DATA SCIENCE GOING ON IN THE INSTITUTES. NO QUESTION. BUT A LOT IS NOT VERY WELL COORDINATE AND A LOT PEOPLE, I THINK MARK AND I MORE THAN ANYTHING TALKING VERY SIMILAR PROGRAM HOW THINGS ARE GOING ON, WE NEED TO DO IT IN A MORE ROBUST FASHION, OTHER GROUP CONTACT DOES AN DEVELOP A COMMUNITY OF DATA SCIENCE ACROSS THE NIH. AS A RESULT BECAUSE THEY CAN COORDINATE SCIENCE ACTIVITIES BEYOND THE NIH WHETHER OTHER GOVERNMENT AGENCY, OTHER FUNDING AGENCIES INCLUDING ABROAD OR THE PRIVATE SECTOR. ONE REASON FRANCIS AN LARRY WANTED TO APPOINT SOMEBODY ACTING SO THEY CAN HIT FORWARD KEY ON EMAIL, IT'S EFFECTIVE BECAUSE I GET EMAILS ALL THE TIME BECAUSE WE'RE DEVELOPING RELATIONSHIPS WITH OTHER GOVERNMENT AGENCIES WITH OTHER FUNDING AGENCIES INTERNATIONAL GROUPS AND PRIVATE SECTOR, SINGLE POINT PERSON TO TALK TO INTERACT WITH AND I'M DOING MY BEST TO KEEP UP WITH IT AND MAC AND OTHERS THAT WE ASSEMBLED TO HELP ARE HELPING TO MOVE THOSE INTERACTIONS TO MORE FRUITFUL DIRECTIONS. LONG TERM STRATEGIC PLANNING, WE CAN JUST DO THIS A YEAR AND MOVE ON, WE NEED TO THINK HOW THE SCIENCE OF DATA SCIENCE WILL GROW AND CHANGE IN THE COMING YEARS AN NOW HAVE A STANDING BODY THAT CAN HELP DO THIS. ALSO PLAY A KEY ROLE IN DATA SHARING POLICY DEVELOPMENT AND OVERSIGHT. WE HAVE GOTTEN ADVANCED IN OUR SOPHISTICATION WITH RESPECT TO THE SHARING OF GENOMIC DATA BUT THERE IS VERY AMBITIOUS THINGS DESCRIBED BY THE DATA INFORMATICS WORK GROUP TO HAVE A BROADER DATA SHARING APPROACH FOR OTHER DATA TYPES AND WE NEED A WAY TO HAVE THAT BE BETTER COORDINATED AND MORE ROBUST APPROACH FOR TACKLING EACH OF THE NEW AREAS SO I THINK THIS GROUP WILL HELP WORK CLOSELY WITH OTHER PARTS OF NIH THAT ARE DOING THAT TO MAKE IT A MUCH MORE COORDINATED RESPONSIBILITY. THIS GROUP NEEDS TO COORDINATE WITH PARALLEL EFFORTS TO ADVANCE ADMINISTRATIVE DATA EFFORTS INCLUDING INTERACTIONS INTERACTIONS WITH CHIEF INFORMATION OFFICER OF THE NIH. SO THAT'S THE AD. THAT'S THE SCIENTIFIC DATA COUNSEL, A STRUCTURAL PART OF THE SOLUTION. NOW LET ME TELL YOU ABOUT THE INITIAL PROGRAMMATIC SOLUTION. BIG DATA KNOWLEDGE OR BD-2K. THIS IS A MAJOR TRANS-NIH INITIATIVE, KEY ROADBLOCK THAT IS HIGHLY AMBITIOUS AND NEED TO BE CATALYTIC AND SYNERGISTIC. WHEN MARK AND I SAT DOWN FOR ONE SENTENCE OVERRER A CHING GOALING WE THOUGHT BY THE END OF DECADE A QUANTUM LEAP IN BIOMEDICAL RESEARCH ENTERPRISE TO MAXIMIZE THE VALUE OF GROWING VOLUME AN COMPLEXITY OF BIOMEDICAL DATA. LET ME STRESS FROM THE BEGINNING, THIS IS NOT A PROGRAM THAT AIMEDED TO REPLACE WHAT INDIVIDUAL INSTITUTES ARE DOING, THIS IS A PROGRAM TO AUGMENT WHAT THEY'RE DOING BETTER COORDINATE WHAT THEY'RE DOING AND WE WILL NOT BE SUCCESSFUL IF INSTITUTES SAY OH, WE DON'T HAVE TO DO THAT ANY MORE, THAT'S NOT THE INTENTION, THAT WOULD BE A BAD OUTCOME, THIS IS BETTER SYNERGIZEED TO FILL GAPS INDIVIDUALS COULDN'T TAKE ON IN A MORE TRANS-NIH WAY. SO WHAT ARE THE MAJOR ELEMENTS FOR PROGRAM TURNS OUT FOUR OF THEM, BRIEFLY INTRODUCE, IF YOU WANT TO DRILL DOWN THOSE THINGS THAT MARK WILL BE ABLE TO ANSWERx PARTICULARLY BETTER THAN I CAN. AND SCRIPTED BY THE DATA INFORMATICS WORKING GROUP. ONE AREA OF BD-2K FACILITATING BROAD USE OF BIOMEDICAL BIG DATA THINKING THE DATA OUT THERE, WE DOPE KNOW ABOUT IT, WE DON'T HAVE IT ORGANIZED, WE NEED A CATALOG TO POINT PEEP WHERE IT IS, WE NEED TO PEOPLE THAT USE THAT DATA SITABLE, PEOPLE GET CREDIT FORIOUS OF DATA AND ORGANIZED IN A MUCH BETTER WAY THAN NOW. THERE'S A HOST OF THINGS THAT CAN BE DONE TO DEAL WITH THAT AND SOME WHICH AGAIN RELATE TO WHAT I'M GOING THE TALK ABOUT AT THE END. WE NEED BETTER METHODS FOR ANALYZING DATA DISSEMINATING IT WHOLE AREA AROUND P SOFTWARE DEVELOPMENT AND METHODS THAT NEED TO BE ACCELERATED AND AREA NUMBER 2 NEEDS TO DO THAT. TRAINING OF COURSE THE WORKING GROUP MADE A NUMBER OF RECOMMENDATIONS NOT JUSTIATE MORE DATA SCIENCETISES BUT HOW PEOPLE GETTING DOCTORAL DEGREES IN AMERICA HAVE LITTLE MORE DATA SCIENCE ASSOCIATED WITH WHAT THEY'RE LEARNING IN GRADUATE DOCTORAL PROGRAMS. THINKING APT OTHER AREAS THAT . AND MAYBE THE FLAGSHIP IN MANY WAYS OF BD-2K ARE CENTER OF EXCELLENCE PROGRAM CAPITALIZING ON WHAT WE LEARNED OVER THE YEARS AN BEST WAY TO PUT CENTERING TO THE TACKLE BIG PROBLEMS. THERE ARE BIG PROBLEMS AN BIG DATA AND HAVING SOME OF THESE THINGS WE ALREADY KNOW WE WAN THAT BUBBLED UP FROM STRATEGIC PLANS AND FROM WORKSHOPS AND IDEAS ON TABLES LIKE THIS, CENTERS OVERSLEEPS TO SOLVE BUT A GOOD NUMBER OF THESE INVESTIGATOR INITIATED. HAVING THEM TELL US WHAT NEED TO BE SOLVED IN THIS ARENA AN CENTER OF EXCELLENCE TACKLING THOSE AS INVESTIGATOR INITIATED IDEAS. HOW WE ACCOMPLISH THIS? WE'RE BEING AMBITIOUS IN A TIME WHERE MONEY IS TIGHT AND BUT WE ALSO NEED TO MAKE SURE AT LEAST SIGNAL TO THE COMMUNITY AND BE SERIOUS, IT'S ONE ROUND, WE NEED A LONG TERM COMMITMENT, THE WAY WE ORGANIZE WHICH IS REASONABLE IN THE INITIAL SEVEN YEAR FUNDING PLAN TO THE END OF THE DECADE, WE NEED MONEY FOR THAT. AND THE REAL MONEY FOR THIS WILL BEGIN NEXT FISCAL YEAR, WE HAVE SEED MONEY THIS YEAR TO PLAN THINGS WORKSHOPS AND GET OURSELVES READY AND GET RFAs OUT, THE REAL FUNDING BEGINS NEXT FISCAL YEAR. THE SCALE OF THIS AT THE END OF THE DAY, IT WILL RAMP TO SLIGHTLY OVER $100 MILLION, AT LEAST INITIALLY ENVISIONED. BY FISCAL 2017. HOW DO WE DO THIS, COME UP WITH THE MONEY? HERE I GIVE CREDIT AGAIN TO THE HIPPOLEADERSHIP BOTH THE PEOPLE IN BUILDING ONE, FRANZY LARRY, CATHY AND JIM ANDERSON, WE SAW AN OPPORTUNITY TO TRY -- IT'S A MODEL THAT NOT PRECISELY TRIED THIS WAY TO TAKE ADVANTAGE OF THE OPPORTUNITIES T COMMON FUN PROVIDES BUT RECOGNIZE IN THE LONG RUN THIS SOLUTION CAN'T BE THE COMMON FUND. IT HAS TO BE INSTITUTES AN CENTERS BANNING TOGETHER TO COME ONE A SOLUTION BUT PARTNERSHIP IN A CREATIVE WAY SO THIS IS A NOVEL FUNDING MODEL, NEVER TRY THIS ONE, THE COMMON FUND FRONT LOADS TO GET US -- TO START BUILDING UP ESPECIALLY A TIME WHEN INDIVIDUAL INSTITUTES AN CENTER ARE STRAPPED FOR MONEY AND WE DOPE KNOW IF WE'LL DIG OUT ANOTHER YEAR. BUT EARLY GOING, MAJOR COMMON FUND CONTRIBUTIONS, THAT WILL BE HELPFUL BUT AT THE SAME TIME EVERYBODY SKIN IS GETTING IN THE GAME SO FROM THE BEGINNING NEXT YEAR THERE WILL BE CONTRIBUTIONS FROM EVERY INSTITUTE AND CENTER PROPORTIONAL TWO THIRD SIZE AND WE HE WILL HAVE POOLING OF THIS MONEY, AND GO FORWARD ON THAT. THEN A PLAN SO SLOWLY THE COMMON FUND INVESTMENT WILL GO DOWN, INSTITUTE CENTERS INVESTMENT WILL GO UP SO BY THE YEAR 2020 OR SO INSTITUTES AND CENTER WILL COMPLETELY OWN IT. THIS IS THEIR BABY AND AT THAT POINT WE'LL FIGURE WHAT'S NEXT BUT I THINK THAT'S FINE. WHAT NEXT IMAGINE ONE OF MANY THINGS BUT CLEARLY THE INSTITUTES AND CENTERS WILL BE RESPONSIBLE FOR OWNING IT AT THAT POINT. WE NEED LOTS OF INFORMATION LOTS OF GOOD IDEAS AND WE GIVE THOSE KINDS OF INPUT REQUEST FOR APPLICATIONS AND WORKSHOPS AND ALL THE THINGS NIH DOES. HERE THE REQUEST FOR INFORMATION WE PUT OUT EARLIER THIS YEAR, FEBRUARY OR SO TO GET INFORMATION ABOUT TRAINING NEEDS TO FIND THAT THIRD AREA AROUND TRAINING, BIOMEDICAL BIG DATA AND LAST WEEK, WE LEARN MORE WHAT PEOPLE IDEA HAVE -- THE DATA IDEA CATALOG, DATA CATLING TO SITES TO HAVE ORGANIZED WAY KNOWING WHAT DATA IS OUT THERE AND THOUSAND SITE AN ORGANIZE IT SO THAT'S A LIVE RFA NOW. THE OTHER WAY WE GET INFORMATION OF COURSE IS THROUGH WORKSHOPS AND MARK AND TEAM HAVE BEEN BUSY PLANNING WORKSHOPS, WON'T GO THROUGH ALL INDIVIDUALLY BUT YOU CAN SEE IN THE AREA OF BROAD USE OF BIG DATA THREE WORKSHOPS PLANNED IN THE COMING MONTH IN SOFTWARE AND ANOTHER ONE DEFINED THE DATA, THE OTHER TWO TO BE DETERMINED. ACTUALLY OUR FIRST WORKSHOP WILL BE AROUND TRAINING WHICH IS WHY WE HAVE THAT FIRST RFI EARLIER. AND THEN WE ALREADY KNOW AT LEAST ONE WORKSHOP TO HELP DESIGN FINE BETTER SOME OF THE ISSUES AROUND DATA INTEGRATION AND NEED FOR CENTERS OF EXCELLENCE IN THAT AREA. SO YOU CAN SEE WE HAVE A VERY BUSY AGENDA WITH WORKSHOPS BUT WE WANT TO MAKE SURE WE GET AS MUCH INPUT Z POSSIBLE THAT WILL LEAD TO ROBUST REQUEST FOR APPLICATION AND OTHER PROGRAMMATIC DEVELOPMENT. SO OTHER DETAILS OF BD-2K, FIRST THING I WILL TELL YOU, I ALLUDED EARLIER, WONDERFULLY STRONG SUPPORT ACROSS NIH. WHEN ALL THIS STARTS TO GET IMPLEMENTED I TOOK OUT RESPONSIBILITY OF ADDING AD AND HAD A LOT OF HIGH LEVEL THINGS TO DEAL WITH. SO MARK DIRE WAS WILLING TO TAKE THE OVER THE RESPONSIBILITY OF LEADING THE BD-2K PLANNING EFFORT BY ASSEMBLING A TRANS-NIH WORKING GROUP TO WORK WITH THEM. SO WE ASKED INSTITUTE AND CENTER DIRECTOR THE SEND A NAME OR TWO TO WORK WITH MARK ON THIS AND STEP 5 AND 6. SO MARK NOW HAS A WORKING GROUP OF 125 MEMBERS WHICH IS A LITTLE UNRULY BUT ON THE OTHER HAND IS WONDERFULXCC THAT PEOPLE ARE ENGAGED ENGAGED IN. THIS YOU CAN SEE 24 INSTITUTES AND CENTERS AND SEVERAL OTHER OFFICES THAT ARE INVOLVED, IN ADDITION HE HAS AN EXECUTIVE GROUP THAT DOES SOME OF THE MORE EXECUTIVE LEVEL THINGSES TO KEEP HIS GROUP MOVING FORWARD, SO FORTH. GOOD NEWS IS THERE IS VERY STRONG SUPPORT FOR THIS ACROSS THE NIH. THEN WE HAD TO DEAL WITH SPECIFIC FUNDING PLAN IF YOU'RE INTERESTED SEQUESTER DID EFFECTIVE CAN'T RAMP QUITE AS QUICKLY THE COMMON FUND P HAS CONSTRAINTS TO FRONT LOAD AS QUICKLY AS WE HAD HOPED SO WITH WHITE WHAT WE ORIGINALLY PLANNED WAS THE ORIGINAL BLUE PRINT BUT IN LIGHT OF THE BUDGETARY CONSTRAINTS YOU CAN JUST SEE WE'RE SLOWED DOWN LALE BIT, YOU CAN SEE THE NEXT INITIAL THREE YEARS, BD-2K IT'S A SLIGHTLY LOW ERACH BUT WE'LL BE OFF AND RUNNING. FINALLY LET ME POINT OUT THAT THE IDEA OF ALL OF THESE THINGS BD-2K, SCIENTIFIC DATA COUNSEL IS NOT TO PUSH THEM IN SOME CORNER, LOOK AT WHAT YOU'RE SAYING AND LET US KNOW HOW IT'S GOING BUT TO?7V MAKE SURE IT'S PART OF THE GOVERNANCE STRUCTURE WITHIN NIH TO MAKE SURE IT HAS PROPER CHECKS AN BALANCE BUS ALSO PROPER ACCESS TO REPORT DECISION MAKING P BODIES SO LET ME PIPE OUT, THE WAY WE MADE THE DECISION, HERE IS FRANCIS AND HERE IS HIS STEERING COMMITTEE. HERE IS THE SCIENTIFIC DATA COUNSEL, STEERING COMMITTEE, ASSOCIATE DIRECTOR INPUT FOR DIRECTOR BUT ALSO HAVE INPUT TO THE STEERING COMMITTEE AN IT WILL SHARE SCIENTIFIC DATA COUNCIL AND IF NECESSARY SPEND WORKING GROUPS SCIENTIFIC DATA CARE, MARK DIRE CHAIR IT IS BD-2K WORKING GROUP, CERTAINLY DOING ARENA OF DATA SHARING AND OTHER THINGS LONG TERM STRATEGIC PLANNING IS ANOTHER ONE AND THAT'S THE WAY WE'LL DISTRIBUTE THE WORK. I REALLY THINK THIS ACTIVITY IS VITAL FOR SUCCESS OF ALL THIS ENTERPRISE. SO IN CLOSING, THIS PART OF THE TALK BIOMEDICAL RESEARCH ENTERPRISE UNDERGOING A PHASE CHANGE WITH RESPECT TO BIG DATA AND DATA SCIENCE. WE'RE IN A NEW WORLD HERE AND WE WILL BE HERE FOREVER. THIS IS A TRANS-NIH PROBLEM, THAT NEEDS A TRANS-NIH SOLUTION. INDEED WE HAVE ONE. I THINK THESE SOLUTIONS INCLUDE MULTI-FACETTED CULTURAL CHANGES NOT JUST THROWING MONEY AT THE PROBLEM. BUT I THINK THE PLANS I HAVE DESCRIBED TO YOU, THEIR MISSION CRITICAL. I THINK THEY WILL BE TRANSFORMATIONALMENT AND MY PERSONAL VIEW IS EVERYTHING I TOLD YOU TODAY IS TRANSITIONAL, I THINK THIS IS AROUND TO MUCH LONGER TERM AN LARGER COMMITMENT, I DON'T KNOW WHAT THAT LOOKS LIKE YET. WE SHOULD REVISIT THAT IN THREE OR FOUR YEARS. SO THAT'S WHAT I WANT TO TELL YOU ABOUT BD-2K. BUT THERE WAS DEVELOPMENT LAST WEEK THAT I THINK IS WORTH YOU HEARING ABOUT. SO I TELL YOU ABOUT THAT THEN QUESTIONS. IT WAS INTERESTING WHAT THE WORKING GROUP FOUND AND CERTAINLY BD-2K PROGRAM FOLKS WE HAVE BEEN WORKING WITH RECOGNIZE, EYE AMONG THE CULTURE CHANGES NEEDED WERE HINGES LIKE AMONG THE NITTY GRITTY THAT WAS REALLY CREATING THE BOLT NECK SOME THINGS EPIVISIONED THINGS LIKE STANDARDS AND METADATA AND DEALING WITH A WHOLE HOST OF ISSUES AROUND DATA SHARING AN PRIVACY AND SECURITY AND SOME THINGS SPECIALLY DATA STANDARDS YOUR EYES WANT TO ROLL UP, LIKE NOT THE SEXIST PART OF DATA SCIENCE BUT IT WAS VERY, VERY CLEAR FROM THE BEGINNING AND EVEN DATA FROM THE WORKING GROUP SAYING THIS WAS THE FUNDAMENTAL BARRIER IF YOU DON'T SOLVE, EVERYTHING ELSE L YOU DO WILL FAIL. SO WE DID IT ALL. I WOULD SAY 15 THINGS WE NEED TO TAKE CARE OF, PROBABLY THE FIRST THREE WERE IN THIS ARENA, DATA STANDARDS AND ISSUES PART OF THE CULTURAL CHANGES AROUND DATA SHARING. SO WE WERE PREPARED TO DO IT ON OUR OWN BECAUSE BD-2K HAS THE TO DO IT BULLA WEEK THERE WAS BREAKING NEWS, AND WE KNEW THIS WAS COMING AN INTERACTING WITH WITH THEM A LITTLE BIT BUT IT IS WE THINK THE TIMING IS FANTASTIC. WE THINK IT'S GOING TO BE VERY VALUABLE FOR EVERYTHING I JUST DESCRIBED AND CERTAINLY BD-2K SO LET ME TELL YOU ABOUT THIS. AND IF YOU REALIZE THERE WERE ANNOUNCEMENTS, PRESS RELEASES ABOUT GLOBAL ALLIANCE, I THINK ITS NAME WILL CHANGE TO ENABLE RESPONSIBLE SHARING OF GENOMIC AND CLINICAL DATA, IT WAS ANNOUNCED LAST WEEK HERE TWO PRESS RELEASES, ONE FROM BROAD INSTITUTE, SAENGER AND OTHER GROUPS PUT OUT SIMILAR PRESS RELEASES. AND WHAT THE PRESS RELEASE IS RELATES TO A WHITE PAPER, FRANCIS, WHAT I DON'T KNOW IS WHETHER THE ACD DO YOU HAVE LEAVE THE THIS OR WHETHER YOU HAVE THIS WHITE PAPER, I REALIZE WE COULD ABSOLUTELY PUT THAT TOGETHER FOR THEM. WE COULD CONSIDER ALL THIS STUFF A SERIES OF WHAT SIZE THE POINT THEM TO, A LOVELY WHITE PAPER THAT CAME OUT OF A MEETING IN NEW YORK IN JANUARY OF MANY PEOPLE FROM MANY COUNTRIES. I COULDN'T GO FRANCIS WAS THERE, HAROLD VARMUS WAS THERE, MARK GUY WAS THERE, THAT ATTEMPTED TO TACKLE SOME OF THESE PROBLEMS THAT REALLY ARE FACING THE SESSION FOR SHARING DATA IN AN EFFECTIVE WAY IN PARTICULAR TO TAKE ADVANTAGE OF ACTIVITIES IN GENOMICS AND CLINICAL STUDIES. THIS GOT QUITE A BIT OF PRESS COVERAGE, GOOD THING, IT WAS YOU WILL GOOD PRESS COVERAGE. YOU MIGHT HAVE SEEN A PIECE IN NEW YORK TIMES TALKING ABOUT THIS NEW GLOBAL ALLIANCE. NATURE AND SCIENCE BOTH PIECE, DAVID AL SHORE IS A KEY FIGURE IN THIS AND HE WAS Q&A IN SCIENCE INSIDERS FEATURING DAVID AND GUARDIAN ALSO GOT INTERNATIONAL PRESS COVERAGE AS WELL. SO I DON'T WANT TO WALK YOU THROUGH THE WHITE PAPER BUT FRANCIS DID WANT TV YOU GET FAMILIAR SO WE ASKED DAVID TO SEND US SLIDES SO HE WAS KIND ENOUGH TO DO THAT TO SHARE AND THOSE ARE PREPPED OUT. I WON'T GO LINE BY LINE BUT THESE ARE THE NEXT STEP OF SLIDES ARE FROM DAVID AL SHORE TO SHARE WITH YOU, A VERY IMPORTANT TO KNOW GENERAL FLAVOR OF WHAT THIS IS ABOUT. IT REALLY IS AGAIN TALKING ABOUT THIS EXPLOSION OF INFORMATION, ESPECIALLY AROUND GENOMIC DATA. AND ASSOCIATED CLINICAL DATA. AND THAT IN ORDER TO REALLY BE SUCCESSFUL IN AREAS LIKE CANCER, PEDIATRIC DISEASE, COMMON DISEASE, INFECTIOUS DISEASE WE NEED CLINICAL INTERPRETATION OF INDIVIDUAL GENOME SEQUENCES WITH WITH A VERY STRONG EVIDENCE BASE. THE ONLY WAY TO GET IT VERY STRONG EVIDENCE BASE IS TO GET VERY LARGE NUMBERS OF INDIVIDUALS AND THEIR DATA AN GET IT IN A WAY THAT CAN BE SHARED TO GET THE STATISTICAL POWER TO MAKE IT MEANINGFUL. OF COURSE THE CHALLENGE, BESIDES BEING THAT YOU NEED VERY LARGE COMPARATIVE DATA SETS, LITERALLY MILLIONS IS PEOPLE AREN'T ORGANIZED. WE KNEW THIS. WE HEARD THIS. WHAT BD-2K PLANNING TOLD US WE ORGANIZED DATA IN SILOS. REGULATION SCIENTISTS NOT HARMONIZED IN A FASHION TO ALLOW THE SHARING AND EVERYTHING IS NON-STANDARDIZED AND AS A RESULT WE DON'T ACT WE HAVE A HODGEPODGE OF BALKANIZED SYSTEM AND IF WE DON'T ACT WE'LL HAVE UNCERTAINTY AN PRIVACY AND ETHICS ISSUES. IN THE U.S. IT'S PROBLEM BUT BIGGER PROBLEM INTERNATIONALLY SO THE PROCESS WAS GETTING PEOPLING TO TO TART TALKING ABOUT THIS. RECOGNIZING THAT WHAT WE'RE DOING HERE, SOME ARE FOCUSED ON CERTAIN DISEASES LIKE CANCER, WE'RE GUILTY OF THAT. SOME ARE LIMBED TO A SINGLE COUNTRY LIKE EVERYTHING FROM BD-2K. WE'RE GUILTY OF THAT. SO THE VISION OF PEOPLE GETTING TOGETHER LIKE MIKE STRATTON FROM THE SANGER AND SO FORTH, A BUNCH OF PEOPLING TO, PARTICIPANTS IN 8 COUNTRIES SPANNING MULTIPLE DISEASE AREAS AND DISCIPLINES AND GOT THEM IN A ROOM TOGETHER AND PUT TOGETHER A WHITE PAPER TO SOLVE THIS. THAT'S WHAT THE WHITE PAPER IS ABOUT THIS PROPOSES A SOLUTION. THIS IS THE VISION AND THIS IS AGAIN ARTICULATING DAVID'S REPRESENTATION OF THIS, WHICH I DO LIKE, IT TALKS ABOUT LAYERED ECOSYSTEM WHERE THE FOUNDATION FOR THIS ARE SET OF STANDARDS FOR BEING ABLE TO SHARE DATA AND ETHICAL FRAMEWORK TO THE SHARE DATA WITH RESPECT TO CONCEPT AN PRIVACY AN DATA SECURITY. IF YOU GET THE FOUNDATION RIGHT YOU CAN BUILD UP ON IT. IF YOU GET THE FOUNDATION WRONG OR HAVE IT BE BALL CANNIZED YOU NEVER HAVE AN EFFECTIVE ECOSYSTEM FOR BEING ABLE TO ACCOMPLISH WHAT YOU NEED TO ACCOMPLISH. SITTING ON TOP OF THIS FOUNDATION, THAT THE GLOBAL ALIGN AIMS TO CREATE, WILL BE THINGS CALLED OPERATING ENTITIES, THESE ARE DATA RESOURCES, THESE CAN BE DATABASES, THESE ARE TOOLS THAT I ALLOW YOU TO AGGREGATE DATA AND SHARE DATA WHETHER CLINICAL DATA AND RESEARCH DATA AND MULTIPLE OPERATING ENTITIES BUT ALL OPERATING ON THE SAME FOUNDATION OF COMMON STANDARDS AND FRAMEWORK FOR DEALING WITH. THIS SO YOU HAVE THIS MIDDLE LAYER RESOURCE APPROPRIATELY AGGREGATING HARMON RECOGNIZING DATA AND THEN YOU HAVE THE SCIENCE GO ON IN UPPER LAYER, TRANSFORMATIVE RESEARCH PROJECT ANALYZING THE DATA, DISEASE SPECIFIC PORTALS. THIS IS WHERE THE SCIENCE IS GOING ON. AND THIS IS WHERE THE INFRASTRUCTURE O FOR SHARING DATA AND TAKING ADVANTAGE OF THE DATA AND ALL THAT RESTS ON THE VERY EFFECTIVE FOUNDATION OF STANDARDS. NOT GOING THE READ YOU THIS SLIDE, TALK ABOUT IT FROM THE WHITE PAPER. IN ORDER TO ACCOMPLISH THE GLOBAL ALLIANCE YOU HAVE TO HAVE CORE PRINCIPLES SURE ALL OF YOU WOULD AGREE TO. TO ACCOMPLISH THIS YOU NEED SHARED AND P OPEN TECHNICAL STANDARDS AND IN PARTICULAR I WOULD POINT YOU TO THIS BULLET BECAUSE THIS MAKES SENSE TO ME AND DAVID IS VERY ARTICULATE, HOW IS THE WORLD WIDE WEB CAMING TO? WHY EVERY COUNTRY DIDN'T HAVE THEIR OWN WAY? HOW CAN WE JUMP TO THE WEB AND IMMEDIATELY ACCESS THE WEBSITE AND THE OTHER SIDE OF THE WORLD? INTERNATIONALLY A GROUP CAMING TO WE NEED A CONSORTIUM TO SET STANDARDS HOW THE TALK TO ONE ANOTHER IN THE WORLD WIDE WEB. THEY GOT IT RIGHT AND THE WEB TOOK OFF. THIS SHOULD BE OUR GOAL. THIS HOW ABOUT OUR MODEL AND WE HOW ABOUT IN A SITUATION WHERE EVERYBODY AGREES ON FUNDAMENTAL STANDARDS AND ALL OF A SUDDEN DATA CARE SHARED. WE CAN'T FORCE DATA TO BE SHARED BUT AT LEAST WE HAVE AN OPPORTUNITY TO GET EVERYBODY TO AGREE IF THEY'RE GOING TO H SHARE WE'LL DO IT IN A WAY THAT'S EFFECTIVE BECAUSE WE HAVE COMMON STANDARDS AND APPROACHES. SO IN THINKING ABOUT -- THERE WILL BE HARD ISSUES TO DEAL WITH IN HARMONIZING ETHICS, PRIVACY CONCEPT ISSUES ONE KEY MESSAGE THEY PUT IN THE WHITE PAPER, NOT LOOKING FOR SINGLE SOLUTION BUT A MANUAL OF OPTIONS THAT EVERYBODY HAS CHOICE BUT AGREES TO THAT MENU AN PEOPLE WILL DECIDE HOW THEY'RE GOING THE SHARE WHAT THEY SHARE, SO FOR WITHIN A LIMITED SET OF PARAMETERS THAT EVERYBODY AGREES TO AND EVERYBODY ABIDES BY. THIS HAS BEEN ENTHUSIASTICALLY SUPPORTED. AT LEAST SO FAR, PRIOR TO ANNOUNCING THEY WEAPON AROUND WITH A LETTER OF INTENT, WOULD YOU AGREE TO THIS SORT OF THING. SO FAR THEY HAVE GOTTEN 73 INSTITUTIONS. 40 COUNTRIES TO SIGN ON. FRANCIS SIGNED ON ON BEHALF OF NIH, HAROLD SIGNED ON, VARMUS SIGNED ON ON BEHALF OF CANCER INSTITUTE I SIGNED ON AND BEHALF OF NHGRI. THERE'S OTHER FUNDING AGENCIES NOT JUST NIH BUT OTHER FUNDING AGENCIES SIGNING US ON. THIS IS BEFORE IT WAS ANNOUNCED, SURE MANY MORE WILL SIGN ON. SO FOR THE GLOBAL ALLIANCE, THIS IS JUST AN ANNOUNCEMENT, STILL A LOT TO DO, CREATE AN ORGANIZATION BUT NIGH NEED TO GET SET UP, STRUCTURE FUNDING MEMBERSHIP, WORKING GROUP AND THEY NEED TO GET GOING TO DO ALL THESE THINGS. SO THAT'S WHAT'S GOING TO HAPPEN NOW BUT I THINK SO FAR SO GOOD THE ANNOUNCEMENT WENT WELL, ALL POSITIVE FEEDBACK AND NOW IN PARTICULAR WE'RE INTERESTED TO FIGURE HOW WE CAN HELP. SO MY SUMMARY OF THE GLOBAL ALLIANCE IN MY LAST SLIDE, INTERNATIONAL ALLIANCE ENABLES SECURE SHARING OF CLINICAL AND GENOMIC DATA BY ESTABLISH PING INTEROPERABLE STANDARDS FOR GENOMIC AND CLINICAL DATA INITIALLY. THAT'S WHAT WE TALK INITIALLY, IF IT'S SUCCESSFUL YOU CAN GO TO OTHER DATA DOMAINS AS WELL DOING SOMETHING SIMILAR. BUT DEVELOPING A FRAME TORQUE FOR HARMONIZING DATA SHARING PRACTICES TO ADDRESS ISSUES TOETHICS, PRIVACY AND CONSENT. 73 INSTITUTIONS, 14 COUNTRIES HAVE SIGNED ON BUT IT IS JUST GETTING OFF THE GROUND. MUCH TO BE DONE, THEY ME NEED THE HIRE PEOPLE, DO ALL SORTS OF H THINGS SO WHAT WE THINK IS FANTASTIC, AIMS TO TACKLE SEVERAL MAJOR PROBLEMS NIH ALREADY IDENTIFIED THAT ARE COMPONENTS OF BD-2K. SO ALREADY WE'RE EXPLORING WITH WITH THEM HOW WE CAN HELP THEM THROUGH BD-2K TO DO THE JOB THAT WE KNOW NEEDS TO BE DONE AND NOW DONE AT INTERNATIONAL STAGE. SO THAT'S WHAT I PREPARED TO TELL YOU ABOUT WHAT WE'RE DOING HERE AT NIH AND NOW TIMELY AMAZINGLY TIMELY INTERSECTS SO EFFECTIVELY WITH THIS GLOBAL ALLIANCE. FRANCIS WANTED TO HEAR ABOUT THAT TOO. I WILL STOP THERE. I KNOW FRANCIS CAN HELP ANSWER QUESTIONS ABOUT THE GLOBAL ALLIANCE AND THEN MARK AND I ASK ANSWER QUESTIONS ABOUT BD-2K. >> GRETCHEN JUST SEND AROUND THE WHITE PAPER THAT DESCRIBE IT IS GLOBAL ALLIANCE IN MORE DETAIL. THANK YOU FOR THE TOUR DEFORCE GOING THROUGH A LOT OF ISSUES THAT ARE ALL TIED TOGETHER BY THIS THEME OF DATA SCIENCE. QUESTIONS, COMMENTS, ON VAGUES? -- OBSERVATIONS? >> I HAVE SOME PRACTICAL QUESTIONS. WE PRESUME THAT THE COST OF PROCESSING DATA STORING DATA IS GOING TO DROP AS DRAMATICALLY AS SEQUENCING HAS, RIGHT? SO IN TERMS OF HOW TO STAGE STORAGE ANALYSIS OF DATA SOME IS PROBABLY EASIER TO PROCESS, HAVE YOU THOUGHT IN TERMS OF WHETHER OR NOT YOU'RE GOING TO DO GENETIC DATA FIRST AND SAVE IMAGING UNTIL THOSE COSTS BEGIN TO DECREASE? WHAT IS YOUR APPROACH? >> MARK, COME TO THE MICROPHONE BECAUSE I HAVE GIVEN APPROXIMATE ANSWER BUT HE'S DEALING WITH 150 PROGRAM DIRECTORS SO I SUSPECT HAVE STRONG OPINIONS ON A LOT OF THINGS CH FIRST I COMPLETELY AGREE WITH YOU AND ABOUT STORAGE COSTS, ALMOST FOR CERTAIN GOING DOWN AN OBVIOUSLY CLOUD COMPUTING BECOMES A BIG PART OF THIS, THAT EVEN IN THE GLOBAL ALLIANCE THE WHOLE NOTION OF CLOUD COMPUTING AND ISSUES WITH THAT, NOT GOING TO BE THE SOLUTION. SO OBVIOUSLY THERE ARE PEOPLE WHO ARE STORING DATA PROCESSING DATA DIFFERENT TYPES. FINANCIAL DATA DATA, CONSUMER DATA, PROBABLY HAVE INSIGHTS THAT WE DON'T HAVE IN BIOLOGY. ARE THEY INVOLVED WITH CRYSTAL BALL TELLING YOU WHERE THINGS WILL BE IN TWO YEARS? >> PART OF BEAUTY SETTING THIS UP, THERE'S A STRUCTURAL PLACE THE INTERACT WITH GROUPS. THEY ARE FINDING US, WE'RE FIND THEM. I KNOW WORKSHOPS THE GOAL WAS ABSOLUTELY TO INVITE THESE PEOPLE, EVEN OUTSIDE OF SCIENCE WHO REALLY HAVE DEALT WITH THESE DATA CRUNCH ISSUES TO LEARN BEST PRACTICES AN THINK ABOUT WHAT THEY HAVE DONE HOW IT MIGHT BE APPLIED TO US. IN TERMS OF YOUR QUESTION ABOUT WHICH DATA AREA IMAGING DATA GENOMIC DATA FIRST I DON'T THINK IT'S AN ABSOLUTE BUT WHAT IS THE EXPERIENCE SO FAR? >> SO FAR DISCUSSIONS HAVE BEEN ABOUT ALL DATA TYPES AND NOT STAGING BECAUSE WE'RE STILL AT THE STAGE OF DEALING WITH THE INTERESTS OF ALL THE ICs. THERE ARE CURRENT DATA STORAGE SOLUTIONS FOR A LOT. THERE ARE A NUMBER OF DATA TYPES WHICH THERE IS NO CURRENT DATA STORAGE. THAT IS SOMETHING THAT WE NEED TO DEAL WITH. >> SO A YEAR AGO THERE WAS A SUBCOMITTEE OF THE ACD THAT MET TO DISCUSS DAVID LIT MAP'S BUDGET. WE TALKED ABOUT THAT POINT ABOUT CHALLENGES WITH DATA SHORTAGE. AND HE TOLD US AT THAT TIME HE THOUGHT THAT IMAGING WAS GOING TO BE A BUDGET BUSTER. IS THAT STILL THE CASE? MICHAEL AND I ARE INVOLVED WITH NCBI BUDGET NEEDS AND WE HAVEN HEARD THAT FROM DAVID. HE IS NOT A SHY ONE SO WE'RE NOT HEARING -- THAT'S NOT DRIVING THEM CRAZY. BUT BELINDA, I DON'T KNOW IF YOU WANT TO ADD IN THIS CASE TO THAT AS WELL. >> MANY IMAGING RESEARCHERS ARE MOVING TO THE CLOUD TECHNOLOGY. AND NOT ONLY TO STORE THE IMAGE BUT THE TOOLS FOR ANALYZING IMAGES ALSO IN THE CLOUD SO AS YOU INDICATED, THAT AREA THAT CALLED IS DECREASING. >> EXPLANATION. >> BELINDA IS DEPUTY DIRECTOR OF NATIONAL INSTITUTE OF BIOIMAGING AND BIOENGINEERING. DAVID LIT MAP IS DIRECTOR OF THE NATIONAL CENTER FOR BIOTECHNOLOGY INFORMATION. >> I THINK THE DECISION FASCINATING THE TOPIC IS ENORMOUS IMPORTANCE. A FEW COMMENTS. ONE, YOUR FIRST SLIDE YOU TALK ABOUT (INAUDIBLE). ALL THOSE GENOMIC IMAGING PHENOTYPIC EXPOSURE, ET CETERA, IS ABOUT DATA WE EXTRACT FROM PEOPLE. THERE'S A WHOLE NEW EMERGING WHICH IS THE DATA THAT'S GENERATED BY PEOPLE. FOR EXAMPLE NOW GETTING THE FOUNDATION FOR GLOBAL RISK. PERVASIVE PRESENCE OF SMART PHONES FOR PEOPLE WHO DIDN'T REPORT GENERATED THERE IS THIS UNIVERSE, DATA WE TAKE OUT OF PEOPLE, PEOPLE GENERATE, AND THESE ARE MILLIONS AN MILLIONS OF PEOPLE SO JUST TO ADD TO YOUR LIST OF COMPLEXITY. THE SECOND POINT, I LIKE THE NOTION OF DATA SCIENCE. THERE'S A LARGE DEPARTMENT OF BIOSTATISTICS, THIS WHOLE WHAT IS IT, THE WHAT'S THE NAME BUT IT'S IMPORTANT TO -- I MEAN, IT ONE A GREAT CONTRIBUTION TO TRY TO DEVELOP A RIGOROUS DEFINITION OF THIS EMERGING DISCIPLINE, WHAT IS IT, HOW DOES IT RELATE TO OTHERS PARTICULARLY BECAUSE YOU'RE USING DATA SCIENCE SCIENTIFIC DATA FOR THE OTHER WHERE THE QUALIFIED SWITCH PLATE COULD BE IMPORTANT. I UNDERSTAND DATA IS ALMOST LIKE A META SCIENCE, ABOUT THE WAY TO DEAL WITH PRODUCTS OF SCIENCE. AND I THINK IT WOULD BE A GREAT CONTRIBUTION JUST TO IF THAT'S THE CHOSEN CONCEPT TO BE RIGOROUS. BECAUSE QUALIFY AS A SCIENCE IT SHOULDN'T ONLY BE ABOUT GENERATING A BODY OF KNOWLEDGE. THE FAMOUS TS ELLIOT WISDOM TO KNOWLEDGE INFORMATION YOU CAN SAY%/G THAT DATA IS AT THE BOTTOM SO HOW DO YOU MOVE FROM DATA TO FORMATION TO KNOWLEDGE TO WISDOM, HOW DID YOU GENERATE A SCIENCE ABOUT THE DATA IS A VERY INTERESTING PART OF THIS QUESTION. THEN FINALLY, THE IDEA OF THE CONSORTIUM OR THE APIPELINE, THE OTHER ISSUE, IN ADDITION TO WHAT WOULD BE TO UNDERSTAND WHAT SOME OTHER OF THOSE FIELDS THAT YOU MENTION. ASTROPHYSICISTS. WHAT HAVE THEY DONE TO DEAL WITH THIS VERY SIMILAR SET OF ISSUES? THAT INFORM IT IS WORK MOVING FORWARD. >> ALL GREAT POINTS. >> ELABORATE? >> FIRST I KNOW THAT PEOPLE LIKE DAVID (INAUDIBLE) ARE TALKING TO OTHER COMMUNITIES AND SCIENTIFIC DISCIPLINES MAYBE SENSE. THE ISSUE AROUND DATA SCIENCE, IN SOME WAYS THE ONLY THING THAT I WOULD THINK ABOUT LONGER IS THAT YOU CREATED TOO TIDY AS A SCIENTIFIC DISCIPLINE, OVER HERE, AND P I WANT SOMETHING SO GENERIC THAT IT TOUCHES EVERYTHING ELSE. I THINK IT DOES TOUCH EVERYTHING WE DO. SO THAT'S THE -- THE OVERLY DEFINED I WORRY IT'S OVERLY CONSTRAINED BUT I GUESS I THINK ABOUT THIS MORE. >> I WOULD ADD TO THE FIRST POINT DATA GENERATED BY PEOPLE AND ABOUT PEOPLE COMING FROM SOURCES TRADITIONAL IN BIOMEDICAL RESEARCH. DEFINITELY SOMETHING WE HAVE INCLUDED AS ONE OF THE OBJECTIVE S. >> SO I WANT A CLARIFICATION ON THINGS HOW THE BD-2K THING WOULD WORK. THE INSTITUTES CONTRIBUTE PROPORTIONAL TO THEIR SIZE. IS THAT BASICALLY SAYING YOU'RE TAKING A TAX ON INSTITUTES AND MOVING IT INTO -- OR ARE THEY STILL -- >> YEAH, LARRY SAYS NO. [LAUGHTER] >> BAD WORD. A CONTRIBUTION. >> SO IT'S MOVING INTO A SEPARATE ENTITY WHERE THE INSTITUTES ARE NOT REALLY MAKING FUNDING DECISIONS BUT THEY'RE BEING TOLD WHAT TO CONTRIBUTE. IS THAT -- >> WELL, YOU WANT TO TAKE -- I MEAN, HERE IS WHAT IT'S VERY MUCH LIKE. IT IS VERY MUCH LIKE THE WAY WE DO COMMON FUND PROJECTS WHEREBY THIS IS THIS CENTRAL POOL OF MONEY, SOMETIMES, I CAN TELL YOU THIS, INSTITUTES DONATE EXTRA INTO IT, WE LIKE AN INITIATIVE WE PUT EXTRA MONEY IN, THEN THERE'S TRANS-NIH WORKING GROUP OF EXTRAMURAL PROGRAM DIRECTORS THAT OVERSEE. ONE INSTITUTE IS RESPONSIBLE FOR AN RFA AND THEN IT GOES TO THAT INSTITUTE COUNCIL TYPICALLY. AND THOSE FUNDING DECISIONS ARE MADE IN A COORDINATED WAY SO IT WILL BE MODELED AFTER THAT. >> BUT IT'S THE WHOLE ESSENCE OF THE COMMON FUND IS THAT IT CAN BE VERY FLUID AND CAN RESPOND TO WHATEVER NEEDS EXIST AT A PARTICULAR TIME. IF YOU PARTITION THE PIECES AND SAY THIS IS NOT THE COMMON FUND, THIS IS THE BIG DATA FUND IT SORT OF DEVELOPS ITS OWN IDENTITY, WHETHER OR NOT IT'S SIZE IS APPROPRIATE, RELATIVE TO OTHER THINGS IN THE FUTURE. WOULDN'T IT MAKE MORE SENSE TO INCLUDE THAT WITHIN THE COMMON FUND? Q. LONG TERM? >> I ACTUALLY THINK IT'S A GREAT EXAMPLE OF THE COMMON FUND SEEING INCREDIBLE NEED JUMPING TO HEALTH BUT DOING IT ON A TERM LIMITED BASIS. BUZZ LONG RUN I THINK DATA SCIENCE PROBLEM SHOULDN'T BE THE COMMON FUND PROBLEM IN PERPETUITY. >> BUT THEY CONTRIBUTE RELATIVE TO BUDGET EFFECTIVELY COMMON FUND THEY'RE JUST NOT CALLING IT THAT. >> THE COMMON FUND IS IN STATUTE. IT IS AN LCATION PROVIDED BY THE CONGRESS OF A CERTAIN AMOUNT OF DOLLARS THAT NEED TO BE SPENT IN VENTURE CAPITAL SPACE HOW WE THINK OF COMMON FUND AND PROJECT INTO THE COMMON FUND ARE SUPPOSED TO LIVE THERE TEMPORARILY FOR OCCASIONALLY A LONG AS TEN YEARS BUT NEVER LONGER, THAT MEANS THAT SOMETHING OF THIS SORT WHERE YOU BELIEVE THAT THIS IS AN AREA OF SCIENCE THAT WILL LIVE INDEFINITELY WOULD NOT BE A GOOD THING TO BASICALLY SAY WE'RE GOING TO PUT IT MANY THE COMMON FUND AND LEAVE IT THERE. BUT IT IS A GOOD OPPORTUNITY FOR THE COMMON FUN TO MAKE A CONTRIBUTION TO GETTING THE THING OFF THE GROUND B D-2K BECAUSE IT IS A NEW HIGH RISK HIGH REWARD FUNDABLE PROJECT ULTIMATELY, THIS NEED TO BE AN ENTERPRISE THE INSTITUTES REALLY OWN AND THE IDEA OF CONTRIBUTING ACCORDING TO THE SIZE OF THE BUDGET IS PRETTY MUCH JUST THE WAY PRACTICALLY YOU HAVE TO DO THIS TO GET EXPECT A SMALL INSTITUTE THOUGH THEY HAVE A LOT OF INTEREST IN DATA SCIENCE MAKE A HUGE CONTRIBUTION ESPECIALLYžŽ THINGS SO FIGHT RIGHT NOW. ANOTHER ANALOGY IS THE NEUROSCIENCE BLUEPRINT WHERE THEY WILL CONTRIBUTE PART OF THEIR FUNDS TO A COMMON POOL AND THEN ALL THE INSTITUTES INVOLVED IN NEUROSCIENCE WITH WHICH IS WHAT, 14 OF THEM, MAKE DECISIONS ABOUT OKAY, WHERE ARE THE MOST IMPORTANT NEEDS FOR NEUROSCIENCE THAT COULD BE CONTRIBUTED INCLUDING AT THIS POINT THE BRAIN PROJECT. IT IS A VOLUNTARY GATHERING O OF PEOPLE WITH SHARED CONFIDENCE, THIS IS THE RIGHT THING TO DO. VOLUNTARY IN THE SENSE THE INSTITUTE DIRECTORS GOT TOGETHER, HEARD ABOUT THIS, AND SUGGEST WE NEED -- SAID YES WE NEED TO DO THIS THOUGH NOT ALL OF THEM HAVE THE SAME STAKE IN DATA SCIENCE, THEY ALL HAVE SOME STAKE. AN THIS IS GOING TO HAVE TO BE SUCCESSFUL FOR IT TO BE SUCCESSFUL. >> I WOULD SUPPORT THAT THIS IS CLEARLY AN EFFORT OF THE MOMENT THAT IS IMPORTANT OF THE MOMENT. I WOULD NOT SUPPORT THAT THIS IS SOMETHING THAT THIS HAS TO EXIST FOREVER, IN ANY LARGE ORGANIZATION IT'S EASY TO BUILD SOMETHING LIKE A DEPARTMENT OR AN INSTITUTE OR PROGRAM. IT'S MUCH HARDER TO DISASSEMBLE IT. FIVE OR TEN FROM NOW THE COMMON FUND FRAME OF MIND YOU LOOK AND SAY DOES THIS NEED TO EXIST IN THE FORM THAT EXISTS NOW, HAS IT MOVED TO SOMETHING THAT LARGE NUMBERS OF PEOPLE ARE DOING IN A DIFFERENT WAY. >> I THINK THAT'S OVERDOING. TO BE CLEAR COMMON FUND CONTRIBUTION ONLY FOR LIKE THREE OR FOUR YEARS. >> THERE'S A -- THEN YOU'RE SAYING AFTER THAT IT'S SOMETHING WHERE EVERYONE IS MAKING CONTRIBUTIONS PROPORTIONAL TO THE INSTITUTE -- [LAUGHTER] >> AFTER 2020 THERE IS NO PLAN. YOU HEARD ME EDITORIALIZE AN REASONABLE PEOPLE CAN CHOOSE WHETHER IT'S A LONG TERM SOLUTION OR NOT I'M QUITE SURE I KNOW THE WAY THIS WORKS. PROBABLY 2017 OR 2018 WE WILL -- I WOULDN'T BE SURPRISED IF THERE WAS ANOTHER WORKING GROUP OF THE ACD TO LOCK AND SAY WHAT IS THE NEXT PHASE? BECAUSE LIKE 2018 WE SAY THIS IS ALL SOLVED OR GOING TO BE SOLVED, WE HAVE TO WIPE IT DOWN AND NOT REISSUE RFAs OR RENEWALS. IT ENDS 2020 OR WE COULD SAY THIS IS A TEMPORARY SOLUTION NOW WE NEED SOMETHING BIGGER OR SOMETHING SMALLER AN GUESS AWAY. I THINK REASONABLE PEOPLE FIGURE OUT WHAT'S GOING ON BEYOND THAT. >> I WOULD ALSO SAY THAT THERE ARE ELEMENTS TO THE WAY IN WHICH THIS IS STARTING TO GET OFF THE GROUND IN TERMS OF ORGANIZATION THAT RESEMBLE THE WAY THE COMMON FUND IS DOING THINGS BUT IT'S NOT WITHIN THE PROVINCE. AND THAT -- POINT NUMBER 1. POINT 2, THIS IS AN ATTEMPT TO APPROACH THE ISSUE ON A GENERALIZED BASIS WITH THE UNDERSTANDING THAT SOLUTIONS FOR ANY INDIVIDUAL INSTITUTE OR AREA ARE NOT GOING TO HAPPEN AT BD-2K IT MIGHT EPIABLE IT BUT THERE'S STILL INSTITUTE HEAVILY INVOLVEV BD-2K WHEN IT REACHES A MAX OF 100 MILLION IS ONLY 20% OF WHAT THE NIH IS INVESTING IN WHAT YOU CAN CALL DATA SCIENCE. NUMBER THROUGH, AS ERIC SAYS IF WE COME TO 2017, 2018, RIGHT NOW ALL THE INSTITUTES ARE WILLING TO PARTICIPATE. THIS ISN'T WORKING IN 2017, 2018 THEY WON'T BE. >> SO I WOULD LIKE THE ASK REGARD TO BRAIN INITIATIVE AND BIG DATA, WE HEARD THERE'S A WORKSHOP COMING UP FAIRLY SOON ABOUT THE DATA ISSUES THAT ARE GOING TO BE GENERATED BY THIS VERY AMBITIOUS PROJECT TO COLLECT LOTS OF THIS WAS ABOUT HOW NEURAL CIRCUITS WORK. THAT POINT OF CONNECTION NOW BEEN THOUGHT ABOUT? >> NOT SURE YET. I DON'T KNOW. OTHER PROJECTS INCLUDING COMMON FUND, THERE'S A MAJOR BIG DATA COMPONENT SUCH AS LINKS, THIS IS A GREAT AREA OF SYNERGY. THIS WILL DEVELOP AND WE'RE CERTAINLY (INAUDIBLE) AT ANY TIME. >> ONE INSTITUTE I FORGOT THE MENTION YESTERDAY IN THE BRAIN PROJECT IS NEBIB. DR. MEDIGREW AND I SPOKE LAST NIGHT. THERE'S CERTAINLY AN INSTITUTE WITH A LOT OF EXPERIENCE WITH VERY LARGE IMAGING DATA SETS AND HAS BEEN DOING A SPECTACULAR JOB OF MAKING THOSE AVAILABLE ON THE CLOUD. THOUSANDS OF BRAIN IMAGING DATA SETS ARE AVAILABLE IN THE CLOUD. SO THERE'S A CLEAR INTERSECTION THERE. >> YOU MAY OR MAY NOT HAVE NOTICED, BELINDA CETO IS MEMBER OF THE SIGN ACTIVE INC. DATA COUNSEL. -- SCIENTIFIC DATA COUNCIL. (OFF MIC) >> I WANTED TO FOLLOW-UP ON THE QUESTION EARLIER BY IAN THAT BELINDA COMMENTED ON. REGARDING THE BURDEN THAT IS IMPOSED BY IMAGING DATA AND CONCERN ABOUT THE SIZE OF IMAGING DATA AND SO FORTH. SO I THINK THAT'S A FAIR QUESTION. BUT WHEN YOU TALK TO PEOPLE IN IMAGING COMMUNITY THE GREATER CONCERN IS NOT DATA STORAGE. THE GREATER CONCERN REALLY IS ON EASE OF DATA SHARING. ALTHOUGH WE'RE MANY DECADES DOWN THE ROAD WITH ADVANCED IMAGING TECHNIQUES AN TECHNOLOGIES, WE STILL DON'T HAVE A FACILE SYSTEM THAT IS UNIVERSEAL FOR EASE OF DATA SHARING. IF YOU HAVE A CT SCAN IN BOSTON NOW BECAUSE YOU HAVE A HEADACHE AND YOU'RE ON VACATION, THAT'S YOUR HOME AND YOU'RE ON VACATION AT SAN DIEGO, AND YOU FALL AND HIT YOUR HEAD AND YOU NEED THE CT SCAN FOR COMPARISON YOU MAY HAVE A CHALLENGE IN GETTING IT. YOU THINK YOU COULD GO OPT INTERNET AND PULL IT UP AND WE SHOULD BE ABLE TO DO THAT AND ONE DAY WE WILL. BUT RIGHT NOW THAT'S A DIFFICULTY. >> MY QUESTION IS A FOLLOW-UP ABOUT DATA SHARING BOTH IN TERMS OF WHAT ARE THE CHALLENGES TECHNICALLY SOUNDS LIKE THEY'RE SIGNIFICANT, BUT THE OTHER IS, WHAT ARE THE L HA LENGS CULTURALLY? AS YOU PUT THIS ORGANIZATION TOGETHER, THE TRANS-NIH ORGANIZATION TOGETHER PRESUMABLY YOU'RE FINING SCIENTIFIC NICHES WHERE DATA SHARING IS NOT THE NORM. I'M CARIOUS HOW EXTENSIVE THAT IS AND DO YOU THINK THIS IS GOING TO BE A SOLVABLE ISSUE. >> ACROSS NIH ARE YOU (INAUDIBLE)? >> I'M ASSUMING THAT SCIENTIFIC CULTURES ARE WELL REPRESENTED. I UNDERSTAND THAT'S ANOTHER LEVEL OF COMPLEXITY BUT I'M CURIOUS JUST ACROSS THE NIH HOW VIRRABLE IS THE ATTITUDE TOWARD DATA SHARE SOMETHING >> I -- GREAT QUESTION, I THINK IN THE DOE THE MAIN OF GENOMIC DATA THE THINGS WE HAVE CONVERTED, I THINK THOSE AN EXHAUSTING PROCESS, PROBABLY NOT FAR ENOUGH ALONG ON SOME OF THESE OTHER DATA DOMAINS HOW WE WANT TO GET THERE. TO KNOW WHAT RESISTANCE THERE'S GOING TO BE. I DON'T KNOW. I'M LOOKING TO CATHY OR FRANCIS OR MARK WHAT THEIR IMPRESSIONS ARE. >> THERE'S A GENERAL ACKNOWLEDGEMENT TO WORK HARDER ON OTHER TYPES OF DATA. GRAM STAFF ARE GETTING TUNED IN AND ANYTHING OVER $500,000 IS A REQUIREMENT THAT THERE HAS TO BE SOME PRE-AGREED UPON UNDERSTANDING ABOUT DATA SHARING. SOMETIMES THOSE DON'T HAVE A LOT OF TEETH AND THEY OTHER NOT ENFORCED AND THERE'S A NATURAL RESISTANCE PARTLY BECAUSE IT'S A PAIN TO FIGURE OUT HOW TO PUT YOUR DATA WHERE SOMEBODY ELSE CAN SEE AND PARTLY BECAUSE FRANKLY PEOPLE AREN'T THAT EXCITED ABOUT HAVING SOMEONE ELSE PLAY WITH THE DATA THEY HAVE GENERATED UNTIL THEY EXHAUST THEIR ABILITY TO PLAY WITH IT. SO WE'RE WORKING ON IT. >> IT'S INTERESTING THING, WHEN IT COMES TO BACK TO COMMON FUND, BIG COMMON FUND PROJECTS, IT'S STARTS AT THAT'S THE DEFAULT. AND IT'S -- I DON'T REMEMBER ANY COMMON FUN PROJECTS WE HAVE BEEN INVOLVES HAVING RESISTANCE BECAUSE IT WASN'T AN ISSUE YOU CAN RAISE. AS YOU GET TO SMALL REGRANTS AND OUT IN THE -- GRANT AND DIFFERENT INSTITUTES IT'S MORE HETERO JEEPIOUS BUT IN OTHER CASES MAYBE WE HAVEN'T GOT TON THE POINT OF TALKING OR ENFORCING THE ISSUE, I KNOW THERE'S MANY ISSUES ABOUT DATA SHARING PLANS INDIVIDUAL GRANTS, WHAT DO WE DO WITH THEM AND FOLLOW-UP AN OVERSIGHT, THOSE ARE THINGS WE NEED TO ADDRESS. >> WE'RE ACTUALLY JUST HAVING THIS CONVERSATION THE OTHER DAY. WHAT GOOD IS IT FOR TO HAVE A REQUIREMENT OR REGULATION IF;kv YOU DON'T HAVE ANY WAY TO ENFORCE IT? BUT AT THE SAME TIME, I THINK ENFORCEMENT IS NOT GOING TO BE THE SOLUTION. IT'S CREATING INCENTIVES TO DO IT. SO THE ISSUE OF INCENTIVES FOR GENERATING DATA AND SHARING DATA IS ONE OF THE WAYS WE WAN THE APPROACH THIS. >> THAT GOES BACK TO DATA CATALOG, STANDARDS, ALL THESE THINGS, THERE'S CERTAIN CIRCUMSTANCE LAYERTY WITH WITH THIS. CIRCUMSTANCE QUEUE LAYERTY WITH THIS. >> I SAW A KID DRINKING OUT OF A FIRE HOSE, THAT'S HOW IT FEELS FOR SOME OF THE PEOPLE, IF THEY'RE NOT REALLY PART OF IT. SO MY QUESTION HAS TO DO WITH THE COULD VELOCITY AND WHETHER IN FACT THE WORKSHOPS ARE THINK INVITATIONAL? ARE THEY OPEN TO OTHER INVESTIGATORS? HOW DO YOU SEE THAT CULTURE FLOW OCCUR SOMETHING >> YOU -- OCCURRING? >> YOU MEAN FOR PLANNING INITIATIVES OR DISSEMINATING THE KNOWLEDGE? BECAUSE I KNOW THIS THOSE ARE TWO DIFFERENT THINGS. I CERTAINLY THINK DISSEMINATING THE KNOWLEDGE TO MAKE IT SO THE KID FEELS LIKE INSTEAD OF FIRE HOSE IT FEELS LIKE A WATER FOUNTAIN, THAT'S RAISING THE GAME THIS, FALLS UNDER TRAINING WHETHER IT'S THE GRADUATE STUDENTS OR WHETHER IT'S THE MIDTERM INVESTIGATORS. I MEAN, I THINK THAT'S PART OF THE GOAL OF ELEMENT NUMBER 3. THE WORKSHOP THAT I WAS TALKING ABOUT ARE REALLY ABOUT PROGRAMMATIC DEVELOPMENT, GET FUNDING ANNOUNCEMENTS OUT AND OUR APPROACH FOR THAT. >> IT'S INTERNAL. >> NO, IT'S NOT INTERNAL. >> PEOPLE FROM THE OUTSIDE, IT'S ALWAYS ATTENTION BETWEEN HAVING A RELATIVELY SMALL GROUP TO ALLOW YOU TO HAVE -- USEFUL DISCUSSION AND OPENING UP WHAT WE'RE GOING TO DO WITH WORKSHOPS IS PUT THEM OUT ON THE WEB. THEY'RE GOING TO BE WEBCAST. >> YES, I WANTED TO THANK YOU FOR A EXCITING, THOUGHTFUL AND COMPREHENSIVE PRESENTATION. I JUST HAVE TWO COMMENTS TESTIMONY FIRST IS I THINK THIS PROVIDES A TREMENDOUS OPPORTUNITY FOR BROADENING PARTICIPATION AND NIH RESEARCH PARTICULARLY IN STATES AND INSTITUTIONS THAT ARE NOT AS WELL REPRESENTED IN THE NIH PORTFOLIO PRESENTLY AS THEY COULD BE. SECOND, FOR THIS REASON I THINK IT'S IMPORTANT THAT THE TRAINING OF INVESTIGATORS AT ALL STAGE OF CAREERS BE STARTING RELATIVELY QUICKLY OTHERWISE MAXIMIZE POTENTIAL OF THIS PROJECT. WHAT ARE YOUR THOUGHTS OF THE TRAINING COMPONENTS? >> >> THOUGHTS VERY MUCH ECHO WHAT YOU JUST SAID, THAT -- THERE WAS EXTENSIVE DISCUSSION AT SEVERAL ACD MEETINGS THAT I CAN REMEMBER ABOUT THIS. FIRST WE SHOULDN'T AWE NEED MORE DATA SCIENCE TRAINING PROGRAM, WE NODE TO RECOGNIZE THAT IT'S THE SPECIALIST, GETTING EVERYBODY A LITTLE BIT STRONGER WHO IS COMING THROUGH GRADUATE SCHOOL BUT ALSO THINKING ABOUT THE INVESTIGATORS OUT THERE WHERE THIS IS -- THEY'RE LEFT BEHIND. MY GRADUATE SCHOOL MEDICAL SCHOOL L COLLEAGUE, I KNOW SOME WHO ARE OVERWHELMED. THEY WORK IN ONE AREA HOW DO I GET, I DON'T HAVE TIME TO LEARN IT I. WE HAVE TO HELP THEM, DEVELOP WHATEVER IT TAKES. FAST AND FURIOUS. SO THAT'S GENERAL FRAMEWORK BUT BY NO MEANS DO WE HAVE ALL THE ANSWERS. THAT'S WHY WE PUT AN RFI. MANY RESPONSES CAME IN AN THAT'S WHY IT LEADS TO THE WORKSHOP TAKING PLACE NEXT MONTH SO WE'RE REFINING THAT. >> THE CO-CHAIRS FOR THE TRAINING WORKSHOP ARE (INDISCERNIBLE) BIOSTATISTICIAN AT HOPKINS AND (INDISCERNIBLE) A TRANSLATIONAL INVESTIGATOR IN BOSTON. THE -- WITH THEIR ADVICE WHAT WE'RE GOING TO TRY TO DO AT THE WORKSHOP IS ADDRESS BOTH HOW THIS KIND OF AREA, THIS INFORMATION CAN BE INCORPORATED INTO EXISTING TRAINING PROGRAMS. ALSO, ARE THERE NEW TRAINING MECHANISMS THAT ARE GOING TO BE NEEDED IN ORDER TO TARGET MANY SPECIFIC COMMUNITIES. >> THANKS FOR THE PRESENTATION. I LIKE THE FACT THAT THERE IS THE GLOBAL ALLIANCE IS GOING ON. I GUESS MY QUESTION IS, WHAT ARE WE DOING IN TERMS OF COORDINATING OUR NATIONAL PUBLIC SECTOR ACTIVITIES IN THIS AIR ARE? WE MENTIONED LAST MEETING THE SAME SORTS OF ACTIVITIES ARE TAKING PLACE AT THE OFFICE OF NATIONAL COORDINATOR BRANCH FOR HIT. I WAS IN A STAFF MEETING WHERE BIG DATA WAS THE TOPIC APPROXIMATE PEOPLE WERE TALKING ABOUT THE WORK GOING ON THERE. CMS IS FUNDING A LOT OF STATES AND DOING BIG DATA PROJECTS IN TERMS OF INTERPRETING DATA. SO HOW IS THIS BEING COORDINATED? IS THERE A SLOT, HAVE ALL THE NATIONAL ORGANIZATIONS THAT ARE WORKING TOGETHER ON THIS AT ALL, ESPECIALLY HEN WE THINK ABOUT SEQUESTER AND P BUDGET CHALLENGE. FOR NUMBER OF REASONS I DON'T KNOW THAT WE CAN AFFORD NOT TO HAVE THIS AS A COORDINATED ACTION, BECAUSE IT'S THE RIGHT THING TO DO BUT ALSO BUDGET WISE I DON'T THINK WE HAVE THE LUXURY OF HAVING A THUMB OF ORGANIZATIONS WITH THEIR OWN SEPARATE INITIATIVES TAKING PLACE. YOU SHOW ME A SLIDE WHERE NATIONAL ORGANIZATION ROUND TABLE, ET CETERA. BUT WHAT'S HAPPENING IN THIS AREA? WHAT'S THE PLAN MOVING FORWARD? >> SO I DON'T HAVE -- I ABSOLUTELY AGREE WITH THIS NOTHING AND I WAS AWARE OF IT SIX MONTHS AGO AND THE SECOND I GOT TO BECOME THE ACTING I BECAME HYPERAWARE BECAUSE IT'S ACCOMPLISHING WHAT WE WAPPED, NOW THERE'S A SINGLE L POINT PERSON, THESE OTHER GROUPS CAN CONTACT AND I GET INVITED TO LEFT AND RIGHT, VARIOUS THINGS TO GET MY HEAD AROUND BUT GUESS GET AN EXPERT IN HERE TO DO THIS JOB FOR REAL WHO DOESN'T HAVE TO REASON -- SO PART OF THE ANSWER HERE IS EXACTLY I THINK THE SYSTEM THIS IS WE'RE DOING THE RIGHT THIN, BUT WE DON'T HAVE ALL THE ANS IN SIX -- ANSWERS IN SIX MONTHS. FRANCIS FEELS IT'S EXTENSIVELY WITH WITH ALL THESE GROUPS BUT WE'RE BETTER COORDINATED WITH THE GROUPS MORE THAN SIX MONTHS AGO BUT WE HAVE A LONG WAY TO GO. >> WE HAVE A EXTENSIVE SERIES OF BILATERAL ARRANGEMENTS WITH THE AGENCIES THAT YOU MENTIONED. WE'RE MEETING QUARTERLY WITH CMS AND THEIR TOP LEADERSHIP. TO TRY TO BE SURE WE TAKE ADVANTAGE OF THEIR DATA SCIENCE NEEDS BUT ALSO RESEARCH INTEREST. THAT HAS GONE WELL, RESEN TURN OVER IN STAFF CREATED A LITTLE BIT OF HICCUP IN THAT. WE NEED VERY -- MEET REGULARLY WITH ONC OFFICE OF NATIONAL COORDINATOR, PART OF THAT (INDISCERNIBLE) A BIG FAN OF WHAT WE'RE TRYING TO DO WITH DATA HAS BEEN UP TO NIH SEVERAL TIMES AND WE HAVE REGULAR ALMOST EVERY WEEK INTERACTIONS AT SOME LEVEL WITH ONC THROUGH COORDINATING COMMITTEE ON COMPARATIVE EFFECTIVENESS RESEARCH. WITHIN SF THERE'S ANOTHER POTENTIAL HERE OF DOING A JOINT BIG DATA INITIATIVE BETWEEN NSF APPROXIMATE NIH WHICH WE OTHER JUST STARTING THE DISCUSS. WE DID A SMALL ONE A YEAR AGO WHICH COULD HAVE BEEN BETTER COORDINATED IF WE HAD OUR OWN ACT TOGETHER AS WE'RE TRYING TO DO. BUT I THINK, YEAH, YOUR POINT IS EXTREMELY WELL TAKEN. DO WE NEED SOME SORT OF MULTI-LATERAL CONVENING ENTERPRISE? HHS HAS SUCH A CONVENING BODY BUT NOT SURE THEY ARE AS HIGH LEVEL AS THEY NEED TO BE IN ORDER TO IMPROVE FUNCTION SO WE FOCUS ON DOING THESE IN ONE AGENCY AT A TIME. >> >> I WOULD SAY THE FRANK ANSWER TO YOUR QUESTION WITH RESPECT TO THE BD-2K IS THAT WE ARE AWARE OF THE NEED FOR THIS BUT FOCUS IS GETTING INTERNALLY BD-2K ORGANIZED. WE HAVE HIRED A AAA FELLOW WHO IS STARTING AT THE END OF THE SUMMER TO CREATE THE SLIDE THAT YOU ASKED FOR, TO GO OUT AND FIND OUT ALL THE ACTIVITIES IN -- THAT ARE RELEVANT BOTH IN THE U.S. AND ABROAD AND PUT TOGETHER A LIST SO THAT WE KNOW WHAT -- WHO WE HAVE TO COORDINATE WITH. >> I SHOULD HAVE SAID ANOTHER MAJOR CROW GROWING PLAYER IS PCORI THE PATIENT CENTERED OUTCOMES RESEARCH INSTITUTE WITH ANNOUNCEMENT OF INITIATING A NATIONAL CLINICAL RESEARCH NETWORK WITH PERHAPS TENS OF MILLIONS OF PATIENTS IN THE REAL WORLD THAT'S GOING TO BE AN ENORMOUS OPPORTUNITY AND A CHALLENGE THERE DEALING WITH THE DATA THAT. COMES OUT OF THAT ENTERPRISE. CLOSELY CONNECTED WITH THAT BY THE FACT THAT I SERVE ON THEIR BOARD AND NUMBER OF OTHER FOLKS ARE WORKING WITH WITH THEM QUITEO! CLOSELY. WE DON'T WANT TO MISS THAT DONE. I'M EXCITED ABOUT THIS NETWORK ANNOUNCED IN THE FORM OF RFAs TO CREATE THESE CLINICAL DATA RESEARCH NETWORK NODES WHICH ONCE THEY'RE ASSEMBLED TOGETHER WILL PUT US IN A POSITION TO HAVE ACCESS TO VERY LARGE NUMBERS OF PATIENTS WITH EXISTING ELECTRONIC HEALTH RECORDS OPPORTUNITIES TO HAVE TAKING CARE OF EFFICIENTLY AND A RESEARCH STAFF THAT WILL BE STABLE SO THAT YOU CAN MOUNT CLINICAL RESEARCH STUDIES ON THIS PLATFORM RATHER QUICKLY INSTEAD OF GOING THROUGH THE VERY LONG PAINFUL EXPENSIVE PROCESS OF STARTUP AN CLOSE DOWN. >> FIRST, I REALLY LIKE THE NAME. I DO. IT'S REALLY IMPORTANT YOU PICKED A GOOD ONE. BIG DATA, OBVIOUSLY CAN'T MAKE UP FOR REALLY PRECISE GOOD DATA. ONE OF THE PROBLEMS WITH CLINICAL DATA WHEN SOMEONE SAYS HYPERTENSION, IT MEANS DIFFERENT THINGS TO DIFFERENT PEOPLE. SO LIKE ANYTHING, COMMUNICATION IS GOING TO BE EVERYTHING. WHAT I THINK IS GOING TO BE A REALLY POSITIVE EFFECT OF WHAT YOU'RE DOING AS YOU CREATE YOUR DATA DICTIONARY AND YOUR TERMS AND EVERYTHINGINGS IS THE PRECISION WITH WHICH PEOPLE CATEGORIZE WHAT THEY'VE SEEING, AS CLINICIAN, SCIENTIST, WHATEVER, THERE'S THIS FEEDBACK THAT IS GOING TO HAVE A VERY POSITIVE EFFECT BUT YOU HAVE TO DO IT RIGHT ON THE FRONT END SO THAT THE -- MAKING DECISIONS HOW ARE YOU GOING TO DEFINE, DIABETES, IF YOU LOOK, THERE ARE MULTIPLE WAYS PEOPLE DEFINE SUCH A TERM, IT WILL BE CRITICAL THAT THE TERMS BE SO PRECISE. THAT'S SO CHALLENGING. IT WILL HAVE A REALLY GOOD EFFECT. THERE WILL BE A POSITIVE EFFECT ON P HOW PEOPLE CATEGORIZE PEOPLE. IN THE HEALTHCARE WORLD. >> IOM REPORT ABOUT TAXONOMY OF DISEASE WHICH BASICALLY SAYS THE TERMS WE USE LIKE DIABETES ARE INHERENTLY LESS IDEAL THAN WHAT WE MIGHT ACHIEVE WITH MOLECULAR DATA AND WE SHOULD FILL THAT KNOWLEDGE BASE AN TRY TO TRANSFORM THAT INTO A MORE PRECISE DIAGNOSTIC SET OF CRITERIA. >> COUPLE OF YEARS AGO THERE WAS A NATIONAL BIOSURVEILLANCE ADVISORY SUBCOMITTEE WHICH IS NO LONGER ACTIVE. BUT WE DID A LARGE REVIEW OF WHAT WAS BEING DONE VIS-A-VIS BIOMEDICAL RESEARCH, SURVEILLANCE, WE WERE STRUCK BY THE FACT THAT IT WAS VERY LITTLE CONVERSATION BETWEEN DEPARTMENT OF DEFENSE, HOMELAND SECURITY, EVEN INDIVIDUAL GROUPS WITHIN THOSE ORGANIZATIONS. IT MIGHT BE INTERESTING FOR YOU THE REACH OUT TO THOSE PEOPLE AND FIND OUT WHAT THEY DID IN TERMS OF RECONCILING DIFFERENCES IN DIAGNOSES, THERE WAS NO COLLECTION OF DEATH RECORDS, TOXICOLOGY REPORTS, ALL THESE THINGS. SO THERE WAS A LARGE GROUP, HUNDRED PEOPLE THAT STRUGGLED WITH THIS. THE REPORT IS ON LINE. I CAN GET IT THE YOU. BUT IN ADDITION, IF YOU'RE TALKING WITH HIM HE COULD ALSO BE HELPFUL. >> IT'S BEEN A VERY HELPFUL DISCUSSION. ERIC, THANK YOU FOR SHOULDERING THIS RESPONSIBILITY EVEN THOUGH YOU'RE JUST DOING IT AS ACTING YOU'RE OBVIOUSLY PUTTING A LOT OF TIME AND CONSIDERABLE ENERGY INTO IT AS IS MARK AND P A LOT OF OTHER PEOPLE. IT'S IMPRESSIVE HOW FAR YOU HAVE COME IN JUST A YEAR MINUS ONE SINCE THE REPORT WAS PUT FORWARD BY THE ACD. AND OFF WE WENT TO THE RACES. I'M SURE THIS GROUP WILL WANT TO HEAR REGULARLY ABOUT HOW THIS IS TAKING SHAPE BECAUSE IT IS FUNDAMENTAL TO A LOT OF THINGS WE WANT TO ACCOMPLISH. SO I THINK WE'LL TAKE A BRIEF BREAK FOR TEN MINUTES OR SO AND RECONVENE AT 10:30 TO HEAR A VERY INTERESTING DISCUSSION ABOUT PEER REVIEW SO WE HAVE ONE MORE TOPIC FOR YOU ALL TO CHEW ON. AND THIS ONE IS EXTREMELY IMPORTANT, IT IS COMPLICATED, IT IS FASCINATING. AND AT SOME POINT MAYBE AT THE NEXT ACD MEETING WE MIGHT ARRIVE AT THE POINT OF SETTING UP A POSSIBLE WORKING GROUP TO WORK WITH US. I THINK RIGHT NOW WE WANT TO GET A LITTLE FURTHER WITH THIS INTERNAL EFFORT WHICH HAS INVOLVED A LOT OF PEOPLE AND LARRY TABAK IS GOING TO TELL YOU ABOUT. THE GOAL SIMPLY PUT IS TO BE SURE THAT THE WAY IN WHICH WE HAVE OUR IRGs AND STUDY SECTIONS ORGANIZED TO DO PEER REVIEW, ACCURATELY REFLECTS THE WAY IN WHICH SCIENCE IS MOVING. AND SCIENCE MOVES VERY FAST. SO LARRY HAS THE TASK HERE OF WALKING YOU THROUGH THE WORK OF FOLKSK AND P THERE'S GOING TO BE SOME INTERESTING ANALYSES HERE. I'M GOING TO SUGGEST IF THERE'S SOMETHING HE SHOWS WHERE IT'S NOT QUITE CLEAR HOW THE ANALYSIS WAS DONE AND YOU WANT CLARIFICATION ON THE METHOD, THAT WE STOP AND MAKE SURE THAT COMES ACROSS BUT HOLD OFF BIG LARGE DISCUSSION UNTIL HE'S GOTTEN THROUGH THE PRESENTATION WHICH HAS A LOT OF DATA IN IT. LARRY, PLEASE PROCEED. >> THANK YOU. GOOD MORNING, EVERYONE. SO WHAT WE'RE DOING HERE IS WELCOMING YOU TO OUR NIH LAB MEETING. I WANT YOU TO KEEP IT IN THAT CONTEXT. AND AS FRANCIS HAS INDICATED, THE TOPIC AREAS ABOUT PEER REVIEW I NEED NOT TELL THIS GROUP HOW IMPORTANT PEER REVIEW IS TO THE NIH MISSION. TWO TIERED SYSTEM IS FOUNDATION UPON WHICH THE FUNDING EXTRAMURAL RESEARCH IS BASED. WHILE THIS SYSTEM IS HIGHLY REGARDED THROUGHOUT THE WORLD WE FEEL THAT IS VITAL FOR NIH TO CONTINUE TO INNOVATE AND OPTIMIZE THE PROCESS GRANT APPLICATIONS ARE REVIEWED. THERE'S ACTIVITY ALREADY ONGOING IN THIS SPACE SO MANY OF YOU MAY REMEMBER ENHANCING PEER REVIEW PROJECT THAT WAS BEGUN A NUMBER OF YEARS AGO, AND IN THAT THE COMMITMENT WAS MADE TO CONTINUOUSLY SURVEY RESULTS OF CHANGES MADE AT THAT TIME. RECENTLY THE SECOND OF THESE SURVEYS WAS COMPLETED. THE INFORMATION IS AVAILABLE AT THIS LINK, IF PEOPLE ARE INTERESTED IN THE DETAILED REPORT. AND THE SURVEY SHOWS IN GENERAL FOLKS ARE SATISFIED AND/OR HAVE ACULT RATEED TO THE CHANGES, BUT THERE ARE CERTAINLY DIFFERENCES IN PEOPLE'S PERCEPTIONS FUNDED YOU LOOK AT THE SYSTEM IN A MORE KINDER WAY, IF YOU ARE NOT FUNDED YOU THINK LESS OF THE SYSTEM. THAT'S ALWAYS THE CONUNDRUM AND CHALLENGE WHEN ASSESSING PEER REVIEW TO PARSE OUT THE OBVIOUS RESPONSE THAT ONE HAS IF AN APPLICATION IS NOT ULTIMATELY SELECTED TO SUPPORT VERSUS IF YOUR APPLICATION WAS FUNDED. NEVERTHELESS WE HAVE CONDUCTED THESE SURVEYS, CAN'T TO DO SO AND I URGE YOU THE TAKE A LOOK AT THIS REPORT FOR THE MOST RECENT. YOU HEARD A LOT YESTERDAY FROM ROBERT MEDEGREW AND REED TUXON ABOUT EFFORTS RELATED TO PEER REVIEW AS RELATES TO DIVERSITY EFFORT AND RICHARD NAKAMORA DETAILED SOME OF THE ACTIVITIES IN THIS SPACE SO I WILL NOT GO THROUGH IT AGAIN. THE SUMMARY DOCUMENT IS REFERRED TO HERE. BUT THERE ARE ALWAYS ADDITIONAL ISSUES TO CONSIDER. SO LET ME FRAME THESE BY SAYING WE ARE CONSTANTLY BEING ASKED HOW DO YOU KNOW THAT WHAT YOU ARE SUPPORTING IN NIH IS THE BEST? AND THAT BECOMES INCREASINGLY DIFFICULT AS THE BUDGET CONSTRAINTS BECOME GREATER AND GREATER. THE REFLEX ANSWER OF COURSE IS PEER REVIEW. BUT INCREASEINGLY PEOPLE ARE ASKING FOR THE NEXT STEP. THEY'RE ASKING FINE WE UNDERSTAND PEER REVIEW AND WONDERFUL IT IS BUT HOW DO WE KNOW PEER REVIEW IS DOING WHAT YOU SAY IT DOES? IT'S IN THAT CONTEXT THAT WE'RE BEGINNING TO CONSIDER ADDITIONAL ISSUES. SO SPECIFIC HI THERE'S CONCERNS RAISED OVER THE YEARS THAT THE STRUCTURE OF THE CENTER FOR SCIENTIFIC REVIEW INTEGRATED REVIEW GROUPS, SO CALLED IRGs TAKEN TOGETHER WITH OUR DEPENDENCE ON NORMALIZED PERCENTILEING ACROSS IRGs MIGHT LEAD TO FUNDING OF APPLICATIONS THAT ARE NOT OF THE HIGHEST PRIORITY. WHEN I SAY PRIORITY, THAT'S A COMPLY PILATION OF MANY THINGS. IT'S THE SCIENTIFIC QUALITY OF NOVELTY, THE ALIGNMENT OF CORE MISSION OF INSTITUTE CENTER OR AGENCY. IN THEORY, THINGS LIKE SELECT PAY OR HIGH LOW PROGRAM RELEVANCE COULD BE USED TO ADDRESS THIS ISSUE, FOR THOSE WHO SERVED ON INSTITUTE OR CENTER ADVISORY COUNCILS NOW IN GENERAL ICs ADHERE TO PERCENTILE SCORE RANKINGS. SO TO CONTINUE THIS, SHOULD A PORTION OF NIH RESOURCES BE REDIRECTED IN A MORE SYSTEMATIC WAY TO ENSURE WE SUPPORT THE QUOTE BEST OPPORTUNITIES. I PURPOSEFULLY PUT THAT IN QUOTE BECAUSE BEST MEANS MANY THINGS TO DIFFERENT PEOPLE SO WE HAVE TO ACKNOWLEDGE THAT BUT IF WE WANT TO TRY AND APPROACH THAT, SHOULD WE TRY AND SYSTEMATICALLY EVALUATE THE CHARACTERISTICS OF STUDY SECTION, QUOTE, PERFORMANCE. IN QUOTATION. NOW, PROPONENTS FOR THE CURRENT SYSTEM WOULD ARGUE THAT NO, THE CURRENT SYSTEM IS GREAT. WE NEED HIGHLY SPECIALIZED EXPERTS AT ALL LEVELS BECAUSE THEY APPRECIATE NUANCES OF A HIGHLY SPECIALIZED FOCUSED FIELD. AND SIMILARLY, PROPONENTS WILL ARGUE WHO IS TO SAY WHAT FIELD IS MORE IMPORTANT THAN ANOTHER. AND THAT HAS SOME VALIDITY. NEVERTHELESS, WE HAVE AN IRG ORGANIZATION DRIVEN BY NATURE AND NUMBER OF APPLICATIONS THAT ARE BEING SUBMITTED. IT BEGS THE QUESTION SHOULD WE BE MORE PROACTIVE IN ATTEMPTING TO IDENTIFY EMERGING FIELDS OF SCIENCE, GET A LITTLE AHEAD OF THE CURVE TO ENSURE AN OPTIMAL REVIEW OF THE FRESHEST IDEAS. SO ATTAIN THAT CONSISTS OF LEADERSHIP AND STAFF FROM THE CENTER FOR SCIENTIFIC REVIEW, THE DIVISION OF COORDINATION PLANNING AND STRATEGIC INITIATIVES AND THE OFFICE OF EXTRAMURAL RESEARCH WAS CONVENED ON BEHALF OF THE NIH DIRECTOR IN JANUARY AND IT WAS GIVEN A VERY MODEST CHARGE. FIRST TO OVERSEE DEVELOPMENT OF METHODS THAT COULD POTENTIALLY IDENTIFY EMERGENT HIGHLY ACTIVE AND AREAS OF SCIENCE THAT MAY HAVE BECOME STAGNATED. SECOND, TO RECOMMEND APPROACHES TO COUPLE THE STATE OF SCIENTIFIC FIELD TO HOW WE DO STUDY SECTION ORGANIZATION. THERE BY YIELDING A MORE OPTIMIZED DYNAMIC SYSTEM THAT IS RESPONSIVE TO CHANGES IN SCIENTIFIC TRENDS. SO TWO MODEST OPPORTUNITIES FOR THIS GROUP. SO I WOULD LIKE TO SHARE WITH YOU SOME OF THE IDEAS THAT WE HAVE BEEN TESTING. AT LEAST AMONG OURSELVES. THE GOAL HERE THIS MORNING IS TO TEST THESE IDEAS WITH YOU TO GET YOUR FEEDBACK. AND I'M WEARING THE APPROPRIATE UNDERGARMENT FOR YOU OF FEEDBACK SO IT WON'T GET TOO HOT IN HERE. WHAT ARE THE QUANTITATIVE APPROACHES THAT WE COULD USE? WE COULD ANALYZE WHAT WE CALL STUDY SECTION INPUTS. IN OTHER WORDS, WHAT IS BEING PROVIDED TO THE STUDY SECTIONS. WE COULD FOR EXAMPLE, EXAMINE THE NUMBER OF NEW APPLICATIONS NUMBER OF NEW AWARDS AND THE RELATIONSHIP BETWEEN THE TWO FOR DIFFERENT STUDY SECTIONS, CONTROLLING FOR THEIR DIFFERENT SIZES. SO THIS IS THE FIRST RATHER DATA INTENSE PLOT YOU WILL NOTICE THAT I HAVE DEIDENTIFIED EVERYTHING HERE. THIS IS A COMPLETELY ANONYMOUS PRESENTATION. THIS CHART SHOWS STUDY SECTIONS, EACH DOT REPRESENTS A STUDY SECTION, DISTRIBUTED BY NUMBER OF NEW TYPE 1 UNSOLICITED APPLICATIONS ON THE X AXIS RECEIVED VERSUS THE NUMBER AWARDED ON THE Y AXIS, EXPRESSED AS RATES PER PERSON YEAR. FOR EXAMPLE, THREEVL APPLICANTS APPLYING OVER TWO YEARS IS SIX PERSON YEARS. THE DATA ARE COLLECTED OVER A PERIOD OF TIME FROM 2008 TO 2012. SO JUST LOOK AT THE DIFFERENT QUADRANTS, IF THERE WAS -- YOU CAN IMAGINE HIGH RATES OF NEW APPLICATIONS WITH HIGH RATE OF REWARDS, THAT IS THIS UPPER RIGHT HAND QUADRANT, COULD, AGAIN, I EMPHASIZE COULD SUGGEST THESE AREAS REPRESENTED IN THESE AIR VARIOUS STUDY SECTIONS MIGHT BE MORE VIBRANT WHERE NEW SCIENCE IS BEING PROPOSED. CONVERSELY LOOK AT THE LOWER LEFT QUADRANT, WHERE YOU HAVE LOWER RATES OF NEW APPLICATIONS. THIS COULD POTENTIALLY MEAN SOME OF THESE AREAS ARE POTENTIALLY STAGNATING. WHAT WE DON'T KNOW AND WHAT WOULD BE A REASONABLE QUESTION TO ADDRESS, IS WHETHER THERE INHERENT DIFFERENCES IN APPLICATION RATES AMONG DIFFERENT TYPES OF SCIENCE. FOR EXAMPLE, IF OFF COMPLEX RESEARCH QUESTION THE APPLICATION RATE MAYBE SLOWED AS YOU HAVE TO BUILD YOUR TEAM AND SO FORTH. TO THE PERCENT PERCENTILE SCORES ARE USED TO MAKE FUNDING DECISIONS YOU WOULD EXPECT THERE WOULD BE HIGH CORRELATION BETWEEN APPLICATION AND FUNDING RATES, IT WOULD BE, YOU EXPECT A STRAIGHT LINE CORRELATION. SO RELATIVELY HOE AWARD RATES WHERE WE ARE DOWN BELOW THIS LINE. THIS COULD MEAN A NUMBER OF THINGS. FOR EXAMPLE, THE LOW AWARD RATE MAY MEAN THESE PARTICULAR STUDY SECTIONS FOR WHATEVER REASON GIVING FOR SCORES TO THE A ZERO APPLICATION. THERE BY CREATING THE QUEUE WE TALK SO MUCH ABOUT DURING THE ORIGINAL ENHANCING PEER REVIEW EFFORT. IT MAY BE THAT THEY ARE FAVORING MORE ESTABLISHED INVESTIGATORS. ON THE OTHER HAND, UP HERE, WHERE YOU HAVE HIGH RATES OF AWARD IT COULD MEAN THE STUDY SECTION ARE MORE OPEN TO NEW IDEAS OR PERHAPS GIVING PREFERENCE TO NEW INVESTIGATORS. SO IF THERE IS NOT A CARING BIAS, RELATIVELY LOW AWARD RATE, AGAIN, DOWN ON THIS PORTION OF THIS FIGURE, COULD MEAN THE AREAS OF NEW SCIENCE THAT ARE BEING PROPOSED JUST SIMPLY AREN'T AS MERITORIOUS AS THE THINGS ABOVE THIS LINE. IT'S CONVERSELY RELATIVELY HIGH AWARD RATES, MAY MEAN YOU HAVE AREAS THAT REVIEWERS ARE PARTICULARLY ENTHUSIASTIC ABOUT AND THAT'S FINE, ALSO, IF AWARD RATES ARE NOT ACCOUNTED BY THE PERCENTILE SCORES, THAT'S ANY DEVIATION FROM THE STRAIGHT LINE OF COURSE, YOU COULD SAY THERE ARE AREAS THAT MAY BE SCIENTIFICALLY SATURATED. IN OTHER WORDS, MORE OF THE SAME AND THEREFORE THE AWARD RATE FALLS OFF THE LINE. THEN THERE ARE DIFFERENCES THAT ARE DRIVEN BY VARIATION THAT ONE OBSERVES IN INDIVIDUAL INSTITUTES AND CENTERS. A STUDY SECTION MAYBE PROVIDING THEIR OUTPUT TO AN INSTITUTE OR CENTER THAT HAS A PARTICULARLY POOR PAY RATE FOR THAT FISCAL YEAR. SO ANY AND ALL OF THESE ARE POSSIBLE BUT YET IT IS A SOURCE OF INFORMATION THAT WITH ADDITIONAL EXAMINATION, ADDITIONAL STUDIES MAY BEGIN TO GIVE US SOME INSIGHT PARTICULARLY TAKENNING TO WITH WITH SOME OF THE OTHER MATERIAL AND INFORMATION THAT I WILL SHARE WITH YOU. >> LARRY A METHODS QUESTION. >> AS A NEW SUBMISSION, NEW INVESTIGATOR OR COULD IT BE NEW SUBMISSION FROM INVESTIGATOR FUNNED BEFORE? BASICALLY A NEW SUBMISSION TYPE 1 FROM ANYONE VERSUS A CONTINUATION, A TYPE 2 OF PREVIOUS PROJECT. MUCK MIC >> RIGHT. >> THAT'S CORRECT. (OFF MIC) Q. WE HAD NOT YET DONE THE SUBANALYSES BUT YOU'RE QUITE RIGHT. THAT WOULD BE A NEXT LOGICAL SET OF QUESTIONS. >> SORRY. >> THIS IS MAKING ME VERY UNCOMFORTABLE BECAUSE THERE'S SO MANY VARIABLES. I DON'T KNOW WHETHER I'M LOOK AT CALIBRATION OR BIAS OR AMOUNT OF MONEY IN AN INSTITUTE. >> AND THE ANSWER IS YES. THIS IS A SNAP SHOT OF THE MANY VARIABLES THAT I OUTLINE, AND YOU HIGHLIGHTED SEVERAL OF THEM. AND IF YOU WILL, HOLD IT QUIET FOR A MOMENT AND LET ME CONTINUE AND -- PARDON? (OFF MIC) >> I DON'T KNOW IF GLUCOSE LEVEL CORRELATES WITH THIS OR NOT BUT THERE IS OPPORTUNITY FOR CLINICAL STUDY. YES. >> THIS GRAPH CONFUSEDDED ME SLIGHTLY BECAUSE IT'S APPLICATIONS PER PERSON PER YEAR. THAT SEEMS TO SUGGEST THAT WHAT YOU'RE LOOKING AT IS FOR PEOPLE IN THAT STUDY SECTION ARE THEY SENNING YOU ONE A-0 APPLICATION FOR THAT INVESTIGATOR, IS THAT RIGHT? IS THAT WHAT YOU'RE LOOKING AT? >> JIM GENERATED IT. THE REASON HAIR TRYING TO EQUILIBRATE THE DIFFERENCES IN THE RATE INDIVIDUALS SUBMIT BECAUSE YOU HAVE SOME INVESTIGATORS THAT SUBMIT FIVE APPLICATIONS A YEAR VERSUS SOME INVESTIGATORS THAT SUBMIT ONE. WHICH IS MAJORITY. SO THAT'S WHAT THEY WERE TRYING TO DO HERE. >> LOOKS LIKE THE AVERAGE IS ONE EVERY TWO YEARS, THAT .5 THERE ON THE X AXIS. BUT THAT MAYBE PARTLY WHY YOU DON'T SEE TERRIBLY STRONG CORRELATION. I ASSUME IF U ACTUALLY LOOKEDDED AT TYPE 1 AO APPLICATIONS PER YEAR, NOT PER PERSON BUT WHOLE STUDY SECTION AND PLOTTED THAT AGAINST AWARDS, THAT LOOKS LIKE A MUCH MORE STRAIGHT FORWARD GRAPH BECAUSE OF THE WAY WE DO PERCENTILING. OTHERWISE SOMETHING IS REALLY WEIRD. SO I DON'T KNOW THAT WE HAVE THAT DISPLAY OR NOT BUT I TAKE YOUR POINT. ANOTHER WAY WE CAN LOOK AT THIS, THIS IS INPUT INTO STUDY SEC, QUANTITATIVE DATA WE CAN CAPTURE IS PERCENT OF AWARDEES WHO SUBMIT COMPETING RENEWALS BY IRGs. SO THIS IS NEXT DATA SLIDE. THESE ARE DATA COLLECTED FROM 2004 THROUGH 2007 SO IT TURNS OUT, EACH DOT REPRESENTS A STUDY SECTION, EACH STUDY SECTION VARIES IN THE PROBABILITY THAT FUNDED INVESTIGATOR WILL APPLY FOR AND SUCCESSFULLY RECEIVE A RENEWAL. SO IT WOULD BE VERY INTERESTING TO UNDERSTAND WHY INDIVIDUALS IN DISPLAYED IN THIS -- AMONG THESE STUDY SECTIONS ARE SO HIGH, WHEREAS INDIVIDUALS IN THESE STUDY SECTIONS ARE LOW. THERE ARE MANY REASONS WHY THIS COULD BE. SO AREAS OF ONGOING WORK THAT ARE SHOWING GREAT PROMISE, A STUDY SECTION MAYBE FAVORING ESTABLISHED INVESTIGATORS. AN AREA OF SCIENCE REQUIRES LONG TERM CONTINUOUS EFFORTS. OTHER TYPES OF WORK MAY REQUIRE A MUCH MORE SHORT TERM. SO THINK OF A CLINICAL TRIAL WHERE THE ANSWER IS ACHIEVED IN FOUR TO FIVE YEARS, THAT THE GRANT IS FUNDED FOR, THAT'S IT AND YOU NEED NOT EXTEN BEYOND SO IT CAN BE ANY OR ALL OF THOSE THINGS BUT LESSENNER OF RENEWAL RAY, THINGS FALLING IN THIS AREA, INDICATE AREAS OF FUNNED RESULTS HAS NOT GENERATED RESULTS WORTH PURR SIGHING BY THE INVESTIGATOR. HOW CAN YOU TRY AND FIND EMERGING FIELD? WE'RE ALWAYS ASKED THIS. HOW DO YOU SCAN THE HORIZON TO FIGURE OUT WHERE THE NEXT RNAi OR PCR OR WHATEVER YOUR FAVORITE EMERGING FIELD IS? SO THERE ARE A WHOLE SERIES OF APPROACHES NOW BEING TESTED SO AMONG THEM ARE ANALYSES OF WORK LITERATURE OR APPLICATIONS WHICH CAN PROCEED WIDESPREAD ADOPTION, THAT COULD INDICATE THE EMERGENCE OF A NEW AREA. IF YOU LOOK AT APPLICATIONS AND YOU SEE PEOPLE WHO HAVE NEVER BEEN SUPPORTED BY THE NIH BEFORE, THAT MIGHT REPRESENT EMERGENCE OF A NEW FIELD. TUSHES OUT NEW EMERGING FIELDS TEND TO SITE INTERDISCIPLINARY REFERENCES MOREt SO ANALYSIS OF THE APPLICATION MAY PROVIDE YOU SOME INSIGHT. THEN THERE'S A WHOLE ALTERNATIVE UNIVERSE OUT THERE WHICH I CONFESS TO BEING LARGELY IGNORANT OF BUT THERE ARE FOLKS WHO SPEND TIME WITH THIS, QUOTE, OFF METRICS, AND SO HERE IS ONE POTENTIAL WEBSITE THAT YOU MIGHT WANT P TO INTERROGATE TO LOOK AT ALL THE METRICS AND WHAT THIS IS IS MINING THE SOCIAL MEDIA TO GET AN UNDERSTANDING OF WHAT PEOPLE ARE, QUOTE, TALKING ABOUT. IN TERMS OF DOWNLOADS OF ELECTRONIC LITERATURE AND IN TERMS OF WHAT IS BEING TALKED ABOUT, WRITTEN ABOUT AND BLOGGED. AND WHAT IS BEING TALKED ABOUT IN THINGS LIKE TWITTER. SO IF YOU THINK THIS IS JUST ALL SILLY AND LAST WEEK'S CHRONICLE OF HIGHER EDUCATION, THERE WAS THIS ARTICLE ABOUT THIS GENTLEMAN, STEPHEN ROBERTS, WHO IS PUTTING TOGETHER HIS TENURE AND PROMOTION PACKAGE INCLUDED A WHOLE LIST OF THESE OLD METRICS TO ENHANCE HIS CASE FOR TENURE DECISION. SO THOSE OF YOU LEADERSHIP POSITIONS AND UNIVERSITIES, BE INTERESTING TO HEAR IF THIS IS REACHED YOUR SHORES YET. BUT IT IS OUT THERE AND IT IS BECOMING INCREASINGINGLY MORE USED. SO, THE NEXT THING WE CAN DO IS WE CAN LOOK AT STUDY SECTION OUTPUTS. SO OBVIOUSLY A SET OF OF APPLICATIONS COMES TO STUDY SECTION, THERE ARE ASSIGNED A SCORE, TO BE ABSOLUTELY CLEAR, STUDY SECTIONS DO NOT FUND APPLICATIONS. DID I GET THAT RIGHT, RICHARD? >> YES. >> SCORE PERCENTILES (INAUDIBLE) AS TO WHAT ULTIMATELY GETS SUPPORT. ONE COULD IN THEORY LOOK AT THE BIBLIOMETRIC HISTORY OF PUBLICATIONS OR PATENTS, NORMALIZED BY THE FIELD OF THE SCIENCE ATTRIBUTED TO FUNDED APPLICATIONS THAT WERE REVIEWED BY AN IRG. SO LET'S STOP FOR A MOMENT. I COULD GO AROUND THE TABLE AND EACH ONE OF YOU CAN GIVE ME TEN REASONS WHY CITATION ANALYSIS HAS LIMITATIONS. IT WOULD ALL BE INDEPENDENT. SO BY TIME WE GOT AROUND THE TABLE THERE WOULD BE 100 OR SO REASONS WHY THERE ARE LIMITATIONS TO CITATION ANALYSIS. BUT IF DONE WITH CONTROL IT'S POSSIBLE THAT WE MIGHT BE ABLE TO DERIVE SOME INTERESTING INFORMATION. ED LIKE TO SHARE WITH YOU A POTENTIAL APPROACH TO GET YOUR REACTION. SO AGAIN JUST TO PRELUDE THIS, CITATIONS PER YEAR VERSUS JOURNAL IMPACT FACTOR AS A FUNCTION OF TIME, YOU MIGHT BE ABLE TO REVEAL THE, QUOTE, PERFORMANCE OF A STUDY SECTION. NOW, HERE IS WHAT I'M NOT SAYING, BECAUSE IT INTRODUCES AN ANAPHYLACTIC RESPONSE TO PEOPLE, I'M NOT TALKING ABOUT ABSOLUTE CITATIONS AS A NUMBER. I'M NOT TALKING ABOUT JOURNAL IMPACT FACTOR PER SE, WHAT I'M TALKING ABOUT IS AN APPROACH THAT ALLOWS ONE TO SELF-CONTROL FOR THESE TYPES OF MEASURES THAT MAY PROVIDE US SOME INFORMATION ABOUT THE PERFORMANCE OF A STUDY SECTION AS FUNCTION OF TIME. I WOULD LIKE TO SHARE WITH WITH YOU HOW THAT MAY BE POSSIBLE. SO WE REALLY ARE NOW ENTERING A LAB MEETING AND WE ARE SHARING WITH YOU PRELIMINARY DATA, I WON'T TELL YOU WHICH OF THESE FOLKS I AM, I'LL LET YOU -- THE GUY AT THE END IS FRANCIS OF COURSE. ALREADY. SO JUST TO GIVE YOU A SENSE OF WHAT IS BEING LOOKED AT, ITÖ1fc IS THE BIBLIOMETRIC OUTPUT OF RO-1s IN ALL CSR STANING STUDY SECS FROM THE YEARS 2007 TO 2011. THAT RESULTS IN OVER 149,000 PUBLICATIONS. IT IS A FAIRLY ROBUST DATA SET. BY YEAR, THIS IS THE NUMBER OF STUDY SECTIONS, RO-1s, THE MIRACLE CALLED ARRA. NUMBER OF JOURNALS THAT THESE PUBLICATIONS APPEAR, THE NUMBER OF PUBLICATIONS THAT CONTINUE TO GO UP OR DOWN, THINGS HAVE LEVELED OFF A BIT. AND THEN VARIOUS ARITHMETIC MEASURES THAT ARE LISTED. SO I SHOW YOU THIS TO GIVE A SENSE O MAGNITUDE OF WHAT'S BEING LOOKED AT. THE AVERAGE IMPACT FACTOR OF ALL PUBLICATIONS WITHIN AN IRG, THERE ARE SEVERAL THINGS TO TAKE FROM THIS FIGURE. THE FIRST, THE OVERALL IMPACT FACTOR THAT IS THE AVERAGE IMPACT FACTOR IS RELATIVELY STABLE OVER THIS PERIOD OF TIME PER IRG. THESE ARA BARS ARE FAIRLY MODEST. IF THINGS WERE JUMPING ALL OVER THE PLACE YOU EXPECT TEN PLUS OR MINUS TEN. THAT'S NOT THE CASE. SO THIS IS THE -- THE X AXIS ARE THE DIFFERENT IRGs AND THEY'RE NOT ALPHABETICAL -- JUST THE WAY THE DATA SHOWEDDED UP. NOW, NEXT CURVE OR FIGURE SIMPLY DISPLAYS THE AVERAGE CITATIONS PER YEAR PER PUBLICATION OVER THAT SAME PERIOD OF TIME AGAIN, IT'S STILL QUITE STABLE, THE ERROR BARS ARE PETTY TIGHT. BUT TO OBTAIN A MEANINGFUL READ-OUT OF THIS SO-CALLED BIBLIOMETRIC OUTPUT, YOU HAVE TO COMPARE THE ACTUAL CITATION RATES SHOWN HERE TO WHAT WOULD BE EXPECT PED FOR COLLECTION OF JOURNALS EACH GROUP OF P PIs PUBLISH THEIR PAPERS IN. IN OTHER WORDS, AT PRESENT IT DOESN'T SEEM REASONABLE TO COMPARE FIELD X TO FIELD Y. I DON'T KNOW IF ANY OF US KNOW HOW TO DO THAT COMPARISON. WHEN YOU COMPARE THE OUTPUT OF A GIVEN FIELD WITH ITSELF, PERHAPS THAT'S A VALID WAY OF APPROACHING THIS. YOU KNOW THE JOURNALS THAT THE WORK WAS PUBLISHED IN. YOU KNOW WHAT THE AVERAGE CITATION HISTORY OF ALL THE PAPERS PUBLISHED IN THAT JOURNAL ARE. YOU KNOW WHAT THAT IS FOR THE SUBSET OF PUBLICATIONS DERIVED FROM THESE DIFFERENT IRGs, TO ASK A QUESTION, IS IT THE SAME AS THE AVERAGE, IS IT MORE THAN THE AVERAGE OR LESS THAN THE AVERAGE. AND THAT IS WHAT THE NEXT FIGURE DISPLAYS. LET ME WALK YOU THROUGH THIS. THIS IS A PLOT OF OBSERVED VERSUS EXPECTED CITATION RATE FOR ALL IRGs, 2007 THROUGH 2011. AND THEY ARE STRATIFIED BY THE AVERAGE IMPACT FACTOR OF ALL THE JOURNAL ARTICLES THAT ARE PARTICULAR IRG GRANTS PRODUCED. PER AVERAGE PER YEAR PUBLICATION. THE LINE IN THE MIDDLE THE BLACK LOIN, THAT IS THE AVERAGE. SO YOU SEE SOME IRGs, PERFORM TO THE AVERAGE THAT THE CITATIONS ARE EQUIVALENT TO EVERYTHING ELSE THAT'S BEING PUB ACCOMPLISHED IN THOSE PARTICULAR JOURNALS. AND THEN YOU SEE THERE ARE OTHER IRGs THAT APPEAR TO HAVE A PUBLICATION GROUP THAT QUOTE PERFORM BETTER AND CONVERSELY YOU HAVE OTHERS TO APPEAR TO PERFORM LESS SO. SO I SHOULD HAVE ALSO SAID THAT THE SIZE OF EACH CIRCLE IS PROPORTIONAL TO THE AVERAGE NUMBER OF PUBLICATIONS PER RO-1, SO THERE'S SOME VARIATION BUT MOST PART THEY'RE CLOSE. WE ARE NOT COMPARING ONE FIELD TO ANOTHER FIELD, WE ARE NOT USING ABSOLUTE NUMBER OF ANYTHING. WE'RE SIMPLY ASKING THE QUESTION ARE THE PUBLICATIONS AVERAGE, RELATIVE TO ALL OTHERS PUBLISH IN THE SAME JOURNALS? OR ARE THEY RECEIVING MORE CITATIONS OR LESS CITATIONS? YOU CAN GET MORE GRANULARITY THAN THIS OFFERALL PLOT AND I'M NOW GOING TO SHOW YOU BY STUDY SECTION SYSTEM OF THESE DATA. AND I'M GOING TO SHOW IT TO YOU AS A FUNCTION OF TIME SO YOU GET A SENSE OF THE VIRIOUS STUDY SECTIONS RIGHT IN THE MIDDLE. IN WE'RE USING A HOT AND COLD LATER ON. THESE WERE INTERMEDIATE. AND HERE ARE THE SO-CALLED HOT AND COLD ONES, THOSE THAT FELL BELOW OR ABOVE THE LINE. IF YOU LOOK AT THIS, AS A FUNCTION OF TIME, I CAN NOW RUN THROUGH THIS. SO HERE IS 2007, 2008, 2009, 2010. 20 # 1. WHAT -- 2011. WHAT YOU OBSERVE IS THAT OVER TIME THE GROUPS THAT ARE ABOVE THE LINE REMAIN SO. THE GROUPS THAT WERE BELOW THE LINE REMAIN SO. AND SO THERE APPEARS TO BE STABILITY TO THIS MEASURE SOME WHAT DOES THIS PROVIDE YOU IN WAY OF INSIGHT? SO LET ME WALK YOU THROUGH THIS. WELL, SO ALL KIDDING ASIDE, YOU'RE RIGHT. THIS BECOMES A BIG DATA PROBLEM. HERE WE'RE MAKING USE OF ADMINISTRATIVE DATA BUT IN FACT, SOME OF THESE ANALYSES WERE CONDUCTED USING PROGRAMS THAT HAVE BEEN USED TO LOOK AT FOR EXAMPLE MICROARRAY ANALYSIS. SO EACH NODE, THAT IS, EACH DOT, REPRESENTS ONE OF THE 159 STUDY SECTIONS THAT EXISTED BETWEEN 2007 AND 2011 THAT STILL EXIST TODAY. CIRCLES REPRESENT THE IRGs, AND THE NODES -- THE EDGES, I WANT TO MAKE SURE I GET -- THE EDGES, THAT IS THE LINES BETWEEN THINGS, THOSE REPRESENT THE RELATEDNESS BETWEEN THE DIFFERENT STUDY SECTIONS. SO AS YOU WOULD EXPECT, STUDY SECTIONS THAT WERE -- THAT ARE IN -- IDENTICAL OR RELATED IRGs, ARE HIGHLY RELATED, SO YOU SEE MANY, MANY, MANY LINES CONNECTING. BUT NOT SURPRISINGLY, BECAUSE OF THE WAY SCIENCE IS CONDUCTED, THERE ARE MANY IRGs THAT HAVE -- MANY STUDY SECTIONS THAT HAVE RELATEDNESS TO OTHER STUDY SECTIONS. IN VERY DIFFERENT IRGs, THAT'S NOT A BAD THING, IT'S JUST INHERENT TO THE INTERDISCIPLINARITY OF SCIENCE. NOW, WHAT WE HAVE HERE IN THE WAY OF THE CODE IS FOR THOSE STUDY SECTIONS THAT WERE BELOW 90% OF EXPECTED CITATIONS PER YEAR FOR PUBLICATION WHICH IS 95% CONFIDENCE LEVEL, THOSE ARE DISPLAYED IN GREEN. SO YOU SEE A BUNCH OF THESE IN GREEN. AND THOSE THAT WERE ABOVE 110% OF THE EXPECTED CITATIONS, THEY ARE DISPLAYED IN RED. AND THE ONES THAT ARE BETWEEN 90 AND 110%, THEY'RE DISPLAYED IN BLUE. AND I'M JUST GOING TO TAKE AN EXAMPLE, THIS GROUPS HERE. YOU HAVE MORE OF EVERY FLAVOR, INTERESTINGLY ENOUGH THESE TWO STUDY SECTIONS APPEAR TO BE LINKED IN TERMS OF WHAT THEY ARE LOOKING AT. AND FOR WHATEVER REASON ONE APPEARS TO BE DISPLAYING CITATIONS THAT ARE UNDER AND OTHER IS AT THIS PLAYING CITATIONS THAT -- THE QUESTION BECOMES WHY IS THAT? -- THAT IS VALUABLE INFORMATION TO KNOW. YOU CAN THINK OF REASONS WHY THAT IS. BUT GIVEN THE, QUOTE, PERFORMANCE OF THOSE TWO STUDY SECTIONS IS THERE VALUE IN US UNDERSTANDING IN US UNDERSTANDING THAT MORE FULLY IN ANY WAY SHAPE OR AS WAY OF ONMIZING SIMPLEST CASE, MAYBE THERE IS EXPERTISE IN THIS STUDY SECTION THAT DOESN'T EXIST IN THAT STUDY SECTION AND MAYBE THESE TWO STUDY SECTIONS COULD BENEFIT FROM AN ADD MIXTURE, I'M MAKING THAT UP BECAUSE I DON'T KNOW THE SPECIFICS OF THOSE STUDY SECTIONS BUT THAT COULD BE ONE POTENTIAL THING. SO THAT LEADS TO THE FOURTH ARM OF THIS WHERE YOU CAN LOOK AT THE ANALYSIS OF STUDY SECTION UNIQUENESS, THE PRECEDING FIGURE, AT LEAST TO ME REVEALED HOW INCREDIBLY INTERCONNECTED ALL THE SCIENCE THAT WE'RE DOING IS. DISPLAYED IN A WAY THAT I NEVER QUITE FULLY APPRECIATED. SO THERE CAN BE A FORMAL ANALYSIS OF HOW UNIQUE THINGS ARE. AND THERE ARE DIFFERENCE TOOLS THAT CAN BE USED TO HELP INFORM THIS USING FINGERPRINTS OF APPLICATIONS, YOU CAN LOOK AT REVIEWERS CITATION PATTERNS TO SEE HOW DIFFERENT THE REVIEWERS ARE. YOU CAN ANALYZE ASSIGNMENT REQUESTS ACTUAL ASSIGNMENTS. YOU CAN LOOK AT THE APPLICANT PUBLICATION, THERE ARE MANY WAYS THAT YOU CAN DO THIS. THIS IS A HEAT MAP. OF STUDY SECTION CONNECTIONS. AND THE ROWS AND COLUMNS REPRESENT EACH OF THE STANDING STUDY SECTIONS AT CSR THAT ARE GROUPED BY IRG. IT SHOW IT IS PROBABILITY THAT APPLICANTS TO ONE STUDY SECTION ALSO APPLY TO ANOTHER STUDY SECTION OVER A FIVE YEAR PERIOD SO HIGHLY CONNECTED STUDY SECTIONS LIKE THIS GROUPING DOWN HERE MEAN A VERY LARGE NUMBER OF APPLICANTS ARE IN COMMON TO ALL THESE STUDY SECTIONS. AGAIN, THIS IS NOT SURPRISING. AND SOME INSTANCES BECAUSE OF PRACTICAL CONCERNS, A SINGLE STUDY SECTION HAD TO BE SPLIT. ONE OR MORE STUDY SECTIONS TO HANDLE VOLUME. SO IF THAT WERE THE VENICESIS OF IT YOU EXPECT IT TO BE BRIGHT RED AND I CAN TELL YOU WITHOUT REVEALING WHO IT IS THAT THAT'S THE CASE HERE. BUT REGARDLESS OF THAT SORT OF THING, YOU CAN SEE THERE ARE SERIES OF HOT SPOTS WHICH DOES PROVIDE SOME SORT OF INSIGHT INTO THE SCIENTIFIC TREND. IF THE CONNECTION IS NOT BIDIRECTIONAL, THAT IS, APPLICANTS FROM ONE STUDY SECTION APPLY TO ANOTHER BUT NOT VICE VERSA, THAT MAY GIVE YOU SORT OF A HEADS UP ABOUT HOW CERTAIN SCIENCE IS FOCUSING OR SHIFTING TO ANOTHER AREAS AND ALSO THIS MAY REVEAL MISMATCHES WE SHOULD KNOW ABOUT SO WE CAN RECESS IT. ADDITIONALLY, THIS MAY SHOW YOU SOME ISOLATED STUDY SECTIONS. IT COULD EITHER INDICATE UNIQUE AREAS OF SCIENCE AND THERE MAYBE GOOD REASON WHY THOSE AREAS ARE UNIQUE. IT CAN ALSO MEAN AN AREA OF SCIENCE THAT IS NOT AS WELL INTEGRATED WITH REST OF BIOMEDICAL RESEARCH AND MAYBE THAT IS A BIT OF A PROBLEM. HERE IS A SYSTEM LEVEL GRAPH REPRESENTATION OF ALL THE IRGs, WE IDENTIFY THE IRGs BECAUSE IT IS WHAT IT IS. AND REALLY, THE EDGES THAT IS THE LINES ARE JUST DRAWN BASED ON SCIENTIFIC SIMILARITY BETWEEN APPLICATIONS REVIEWED AND THIS DISPLAY GENERATED APPLICATION FINGER PRINTING AND SO FORTH. BUT THERE ARE OTHER WAYS OF DOING THIS AND THEN YOU WANT TO SUPERIMPOSE TO SEE WHAT THE GOODNESS THEN IF YOU ZOOMED IN BIOLOGICAL CHEMISTRY, ONLY BECAUSE SO AGAIN, YOU CAN ASK HOW THIS GROUP SHOULD BE BEST OPTIMIZED TO FLUCTUATING WORKLOADS, TO MAKE SURE THAT INVESTIGATORS HAVE AT LEAST TWO PLACES THAT THEY CAN SEND THEIR APPLICATIONS AND SO FORTH. NOW, ALL THAT IRSHARE WITH YOU, ALL THESE SO CALLED QUANTITATIVE APPROACHES THAT COB GENERATED WITH HELP OF COMPUTERS INDOLE GOSH RHYTHMS, OBVIOUSLY HAVE LIMITATIONS. SO WE CERTAINLY ACKNOWLEDGE THE ANALYSIS OF REVIEW GROUP OUTPUTS IN PARTICULAR, THAT CAN BE SUGGESTIVE BUT CLEARLY THAT'S NOT GOING TO PROVIDE YOU THE FULL INSIGHT THAT YOU REQUIRE AND TO THE NATURE AND QUALITY OF THE SCIENCE WE'RE SUPPORTING. I'M SURE ALL OF YOU HAVE EXPERIENCED A PUBLICATION OF YOUR OWN OR A COLLEAGUE, THAT IS CITED MUCH LESS FREQUENTLY THAN THE AVERAGE PAPER IN A PARTICULAR JOURNAL, BUT 20 YEARS LATER REVOLUTIONIZES A FIELD. SO ANY QUANTITATIVE ANALYSIS THAT WE CONSIDER HAS TO BE SUPPLEMENTED WITH EXPERT QUALITATIVE INPUT SO LET ME GIVE YOU SOME EXAMPLES. THESE ARE THINGS THAT WE'RE NOT DOING AT THE MOMENT BUT THEY ARE THINGS THAT WE COULD DO. THESE ARE PROPONENTS IF YOU WILL. WE CAN AGGREGATE THE TOP 15% OF ALL PROPOSALS WITHIN AN IRG AND P THEN SUBJECT THAT GROUP OF APPLICATIONS TO A SECONDARY RANKING BY AN INDEPENDENT GROUP OF EXTERM EXPERTS. WE CAN DO THAT AND LINK THE SECONDARY RANKING WITH THE STUDY SECTION TO SEE IF THAT DISTRIBUTION IS RANDOM. SO IF IT IS THEN WE'RE DONE. BUT IF IT'S NOT RAN DOCUMENT IS THAT INFORMATION THAT AN INSTITUTE OR CENTER WOULD LIKE TO KNOW ABOUT SO CONCRETE, TWO STUDY SECTIONS, VERY RELATED, THE INDEPENDENT RANKING SAYS THE FIRST 30 APPLICATIONS HERE, TOP 30, UNTIL YOU GET TO 31, THE NUMBER ONE APPLICATION FROM THIS GROUP DOESN'T MAKE THE LIST, DOING THIS IN AN EXTREME WAY TO MAKE THE POINT. BUT IF THAT WERE A CASE SHOULDN'T THAT INFORMATION BE AVAILABLE TO AN INSTITUTE OR CENTER TO FACTOR INTO THEIR DECISION MAKING. THEN IF YOU TAKE THIS TO A GREAT COMPLETENESS, SHOULD YOU BE CONDUCTING A SERIES OF HIPPOWIDE IRG REVIEWS TO ANALYZE THE STRUCTURE AN COMPARE HIGHLY SCORED APPLICATIONS ACROSS ALL THE SCIENTIFIC REVIEWS. IT TURNS OUT CSR IS ALREADY, THIS IS SOMETHING THEY LAUNCH AD YEAR AGO. ABOUT A YEAR AGO, RICHARD? A YEAR AGO SO THIS IS KNOWN TO THE COMMUNITY, SO FORTH. EVALUATION SCHEDULE LOOK AT THEIR IRGs, THESE HAPPEN TO BE ONES THAT WERE SELECTED AND THESE ARE QUESTIONS THEY ASK OF THEIRszl EVALUATION PANEL. AND IS THE ORGANIZATION CONSISTENT WITH THE SCIENTIFIC DOMAIN, OVERLAPS AN GAPS, EMERGING TRENDS BEING ACCOUNTED FOR, ARE ROSTERS APPROPRIATE, BEST APPLICANTS IN EACH COMPARABLE IN TERMS OF EXPECTED QUALITY OF RESEARCH. ALL REASONABLE QUESTIONS. VERY GOOD CHECK IN AND REFRESH IF YOU WILL OF THE IRGs. NOW, TO CONTINUE OTHER TYPES OF QUALITATIVE ANALYSES ONE CAN DOING, YOU CAN CONDUCT AN NIH WIDE PORTFOLIO REVIEW TO COMPARE QUALITATIVE MEASURES TO QUANTITATIVE ASSESSMENTS BY EXPERTS. AGAIN, I DONE MEAN TO IMPLY THAT THE QUANTITATIVE SUGGESTIONS THAT I MAKE ARE THE ONLY QUANTITATIVE MEASURES YOU CAN MAKE. THERE ARE OTHERS BUT WE USE THESE AS EXAMPLES. CLEARLY IT'S MUCH EASIER TO COMPARE PERFORMANCE WITHIN A SINGLE FIELD NONE HAVE BEEN ABLE TO FIGURE HOW TO COMPARE DIFFERENT FEELS TO ONE ANOTHER BECAUSE IT IMMEDIATELY IS CO-FOUNDED BY VALUE JUDGMENT ABOUT RELATIVE IMPORTANCE AND ALIGNMENT OF MILLS ONE FIELD VERSUS THE OTHER AND THE QUESTION THIS IS A PROBLEM NO MATTER WHICH FIELD IS SELECTED BEGS THE QUESTION WHY WAS THAT FEEL CHOSEN PEOPLE BECOME VERY UPSET, NERVOUS, HISTORICAL, ET CETERA, BEFORE OPENING UP FOR COMMENTS OR QUESTIONS. THESE ARE PEOPLE THAT DID THE WORK. THEY WANTED A BIGGER TARGET UP HERE SO THAT'S WHY I'M HERE. JIM ANDERSON FROM DPCSPI, OFFICE OF EXTRAMURAL RESEARCH, CATHY HUDSON FROM OFFICE OF THE DIRECTOR. RICHARD NAKAMURA WE HAVE MET. OFFICE OF EXTRAMURAL RESEARCH, SALLY YOU MET AND GEORGE SANTANGELO FROM PCIPSE. WITH THAT I'LL STOP AND ENTERTAIN COMMENTS OR QUESTIONS. >> THANK YOU, LARRY. >> THAT WILL G ME SOMETHING TO THINK ABOUT GOING HOME. >> IN TERMS OF HOT SPOTS YOU MAY FIND HOT SPOTS CORRELATE TO YEAR IN WHICH THE STUDY SEC WAS STARTED BECAUSE WE SEE A LOT OF STUDY SECS THAT ARE -- THAT GET A LOT OF APPLICATION APPLICATIONS IN A CERTAIN AREA, A REALLY HOT WHO AREA SO START A NEW STUDY SECTION FOR THAT AREA TO DEVELOP IT. >> YOU'RE RIGHT IN THESE ANALYSES OVER THE FIVE, SIX YEAR PERIOD, QUITE RIGHT. SEVERAL STUDY SECTIONS HAVE BEEN IN EXISTENCE THREE YEARS OR FOR YEARS. BUT THAT IS PLACE WHERE YOU GET A BURST OF ACTIVITY BUT OVER TIME YOU EXPECT UNLESS L IT'S A HIGHLY EMERGING FIELD, EXAMPLES WE DETECTED YOU EXPECT THAT TO COME BACK TO SOME NORM. THERE ARE SOME FIELDS THAT STAY AT THAT HIGH. >> >> NUMBERS OF PUBLICATIONS SELF-REPORTED NUMBER OR PUBLICATIONS? >> THAT'S DONE AN P LITICALLY. >> BY -- >> WE HAVE A SYSTEM CALLED SPIRES WHICH PULL IT IS GRANT NUMBER AND IT ASSOCIATES WITH ALL PUBLICATIONS THAT ARE CITED AT THE SITE DECK. IT'S DONE AUTOMATICALLY. >> ONE THING THAT EVERYBODY NEEDS TO BE AWARE OF, I'M PRETTY GOOD AT BODY LANGUAGE AND I SENSE SKEPTICISM BECAUSE IT'S QUIET IN THE ROOM. UNDERSTAND THAT EVERYTHING WE HAVE DONE HERE CAN BE DONE BY ANY ONE OF YOU IF YOU WERE SO INCLINED SITTING AT YOUR DESK. PEOPLE ARE ACTUALLY BEGINNING TO DO THIS SORT OF STUFF. SO FOR NO OTHER REASON WE HAVE TO HAVE INTERNALLY A VERY DEEP UNDERSTANDING OF LIMITATIONS OF THESE TYPES OF ANALYSES, AS WELL AS THEIR POTENTIAL VALUE. SO ONE LEVEL WE HOPE THIS APPROACH INFORMS THINGS GOING FORWARD. BUT CONVERSELY, WE ALSO NEED TO DO THIS PROACTIVELY BECAUSE SOMEBODY ELSE IS DOING THIS IN THE EXTRAMURAL WORLD ANDSYME SOMETIMES THEY FRANKLY WILL USE AN ANALYSIS, TO MAKE A POINT THAT MAY OR MAY NOT BE SUPPORTED BY THEIR OWN DATA. >> I HAD A FOLLOW-UP QUESTION HOW DID YOU HANDLE PUBLICATIONS PROVIDING MORE THAN: ONE GRANT? Q. A PUBLICATION IS ASSOCIATED WITH WHATEVER GRANT THEY ARE CITING. IF IT CITES MORE THAN ONE GRANT THAT WILL SHOW, I WILL HAVE TO ASK, AGAIN I WISH I HAD THE EXPERTS IN THE ROOM BUT I WOULD ASSUME THAT A GRANT WOULD -- LET ME GO BACKWARDS. IN YOUR -- WHEN YOU CITE YOUR PUBLICATIONS AND PULLED FROM SPIRES, THAT GRANT WILL SITE EVERY PUBLICATION THAT MENTIONS THAT GRANT NUMBER SO THE ASSOCIATION WITH THE GRANT WOULD SHOW THAT. SO A PUBLICATION CAN BE CITED ON MORE THAN ONE GRANT. >> DOUBLE COUNT. >> THAT'S CORRECT. DOUBLE COUNTED. WE DON'T HAVE FOR ANY STRATIFICATION OF HOW MUCH SUPPORT THEIR PARTICULAR (INAUDIBLE). Q. SO HIGHER IMPACT (INAUDIBLE) HAS MORE (INAUDIBLE). >> NO DOUBT ABOUT IT. BUT IF YOU LOOK AT THE ANALYSIS, THE SO CALLED ONE WORD JOURNALS, FORGIVE MY BLASPHEMY BUT I DON'T PUBLISH ONE WORD JURY ROOMS SO I CAN BE BLASPHEMOUS. THEY DO NOT RELATE REPRESENT FIELDS OF SCIENCE, THEY TEND TO BE MORE GENERAL IN SCOPE, PUB ESTABLISHING FROM A WIDE RANGE OF FIELDS. ONCE INTO THE MIDDLE AND LOWER IMPACT FACTOR JOURNALS, NOW YOU ARE SEEING A WINDOW INTO FIELDS OF SCIENCE, THEY TEND TO BE MUCH MORE SPECIFIC TO MORE FOCUSED FEELS. SO IN SOME WAYS THE ANALYSIS OF THOSE CAN BE MORE INFORMATIVE THAN HIGH IMPACT. >> ONE OF YOUR FIST SLIDES YOU PUT IN QUOTATIONS OPPORTUNITIES, I THINK THE PURPOSE OF ALL OF THIS IS DEPENDENT ON THE USUAL CRITICISM TO THIS EXTENSIVE SYSTEM OF PEER REVIEW BOTH FOR FUNDING AND PUBLICATION, IT'S A VERY POWERFUL WAY REPRODUCING THE -- NORMAL SCIENCE, RIGHT? AND IT DOESN'T MAKE IT EASY WHEN THERE ARE MAJOR PARADIGM SHIFTS WHETHER WITHIN A DISCIPLINE OR BECAUSE YOU'RE MOVING TO (INAUDIBLE) NOT YET BEEN. SO IF HOW DO YOU MACK YOUR YOU DOPE MISON THOSE SHIFTS THAT ARE OUT THERE, GIVEN THAT THE WHOLE MACHINERY OF PEER RERUE IS A VERY POWERFUL SOCIAL REPRODUCTION MECHANISM, IT'S ONE THING. IF THAT MEANS YOU'RE NOT GETTING THE REVIEW IS A FUNDING LOW QUALITY RESEARCH BY WHATEVER DEFINITION, OR LOWp TERMS OF MODULATION INTO SOLUTIONS, HOW -- YOU SAID YOU'RE -- I'M PUTTING THIS IN QUOTATIONS, ISN'T THAT THE QUESTION? >> IT IS CERTAINLY A SEMINOLE QUESTION BUT I THINK ALL THE MORE REASON WHY WE SHOULD MINE OUR OWN BIG DATA ADMINISTRATIVE LY. TO UNDERSTAND CHARACTERISTICS WE'RE USING THE FRAME IRG AND STUDY SECTIONS, IT IS A CONVENIENCE FRAME BEING USED BECAUSE OUR SYSTEM IS BUILT UPON THOSE IRGs AND STUDY SECTIONS. BUT THINK BACK TO THE FIRST DIAGRAM WHERE WE HAD APPLICATION S MANY RENEWAL APPLICATIONS FUNNED HIGH RATE VERSUS REFERRAL APPLICATIONS AT LOW RATE, THERE ARE REASONS WHY THEY MAYBE A DIFFERENT QUADRANT OF THAT FIGURE. BUT I CAN TELL YOU THE ENTITIES IN THE LOWER QUADRANT CORRESPOND TO SOME OF THE GROUPS JUDGED ABOVE OR BELOW THE LINE. NO ONE MEASURE WORKS HERE AND NONE OF THE QUANTITATIVE MEASURES NO MATTER HOW MANY YOU USE IN ISOLATION IS VALUABLE SUCH AS YOURSELVES BUT WE NEED TO UNDERSTAND BETTER, THE TRENDS THAT MAXIMIZE THE POSSIBILITY OF BEING OPEN TO AN EMERGENT IDEA THE TRENDS THAT MAXIMIZE THE DONE TO FUND THE SCIENCE THAT CONTRIBUTES TO MOST, IT WOULD BE WONDERFUL IF WE CAN HAVE DONE THIS TYPE OF ANALYSIS BUT THESE TYPES OF DATA JUST DON'T EXIST FOR THAT TIME PERIOD. SO WE'RE BEGINNING NOW IN HOPES OF CREATING A BASELINE FIVE YEARS FROM NOW, TEN YEARS FROM NOW IT WILL BE MUCH CLEARER WITH THE BENEFIT OF HINDSIGHT AS TO WHETHER THIS IS REALLY WORTHWHILE OR NOT. >> LARRY, IT'S WONDERFUL THAT YOU'RE ASKING THIS QUESTION. I WAS WITH A FORMER MEMBER OF THE ACD, ONLY LAST WEEKEND, FACETIOUSLY SUGGESTING THAT EVERYONE IN HIS FIELD SHOULD SUBMIT TWO GRANTS EVERY TIME YOU SUBMIT A GRANT. ONE THE GOOD ONE AND ONE LOUSY GRANT. S THAT IS THE WAY TO INCREASE THE LIKELIHOOD THAT THE GOOD GRANT WILL GET FUNDED. SO IN OTHER WORDS, I THINK THERE'S DEEP SKEPTICISM IN THIS SCIENTIFIC COMMUNITY THAT THE CURRENT SYSTEM IS OPTIMIZING THE FUNDING OF THE VERY BEST SCIENCE, THAT'S GREAT. SO WHAT I WAS CURIOUS ABOUT, THE PIECE OF DATA THAT CAUGHT MY EYE THE MOST LOOKING AT THE CLUSTERS OF IRGs, AND WITHIN THEM THE STUDY SECTIONS THAT WERE UNDERPERFORMING RELATIVE TO AVERAGE OR OVERPERFORMING. AND THERE WERE TWO THAT REALLY STOOD O. ONE ON THE TOP QUADRANT THAT HAD ALL GREEN EXCEPT ONE BLUE AND THERE WAS A LITTLE ONE IN THE MIDDLE THAT HAD FOUR REDS AND NO OTHER ONES. I KNOW YOU DON'T WANT TO TELL WHICH IRGs ARE AND I RESPECT THAT BUT YOU KNOW WHICH THOSE ARE. WHAT THE DATA MAYBE SAYING AND IT WILL TELL YOU THIS ANALYSIS IS THE RIGHT WAY TO GO OR IT MAY SAY NO THERE'S FLAW IN THE LOGIC THE WAY WE'RE DOING THAT. >> THAT'S CERTAINLY A FAIR QUESTION AND A GOOD OBSERVATION. NOT COMPLETELY SURPRISED IF WE HAD BEEN MAYBE WE WOULD HAVE BEEN TALKING ABOUT THIS TODAY BUT WE ARE A LONG WAY FROM SAYING WE HAVE CORROBORATED THIS USING EXPERT OPINION. WE HAVE NOT YET DONE THAT. I DON'T REGARD MYSELF OR MY COLLEAGUES WHO LOOKED AS THESE AS NECESSARILY EXPERTS FOR THOSE PARTICULAR THINGS))œ THAT YOU MENTION. IF WE WERE TO GO TO NEXT STEP AND ASK THEM FOR EXPERT OPINION, AND THEY LOOKED AT THIS AND SAID WELL, INDEED, AGAIN, HOW WOULD YOU DO THAT? SO YOU TAKE MUCH PUBLICATIONS, YOU DON'T TELL EVERYONE WHERE THEY CAME FROM, JUST OKAY LIST OF PUBLICATIONS, QUALITY, ONE, ONE TO A THOUSAND. IF ALL CLUSTER IN ONE PLACE, ONES NOT SO IN ANOTHER PLACE, THAT MAY HELP US CORROBORATE. CONVERSELY IF THEY'RE SCATTERED RANDOMLY, THAN WE'RE DONE AND WE NEED A QUALITATIVE MEASURE FOR SURE. >> EXPAND ON THAT. I SEE HOW YOU CAN DO THAT WITHIN AN IRG. BASICALLY UX GET EXPERTS WHO KNOW ABOUT THAT DISCIPLINARY AREA, THEY CAN LOOK AT THIS COLLECTION PUBLICATIONS AND DECIDE HOW TO RANK THEM AND YOU CAN SEE WHETHER THAT MAPS RANDOMLY TO THE STUDY SECTIONS WITHIN THAT IRG. REALLY HARD QUESTIONS, HOW DO YOU COMPARE ACROSS IRGs, RECOGNIZING THAT WHATEVER THAT CIRCLE IS WITH THE GREEN DOTS IN IT, IS A DISCIPLINE PROBABLY VERY DIFFERENT THAN THE RED DOTS. HOW ARE YOU GOING TO FIND SOMEBODY TO EVALUATE PUBLICATIONS WHO DOESN'T HAVE A INTRINSIC BIAS THEMSELVES ABOUT WHAT HOT SCIENCE AND WHAT ISN'T. >> >> INITIALLY I THOUGHT THAT WAS A HOPELESS THING TO ADDRESS. THIS STY GRAM MAKES IT PERHAPS A LITTLE LESS SO. AGAIN, I'M JUST TAKING THIS RANDOMLY. SO THIS POOR SOUL HERE, FOR SOME REASON IS RELATED TO THEY'RE HERE. AND THEY HAVE DIFFERENT CHARACTERISTICS. NOW I TRULY DON'T REMEMBER WHO IS WHO BUT TO ME THAT'S A NATURAL EXPERIMENT AND BECAUSE OF THE RELATEDNESS ASSUMING THAT WHATEVER ALGORITHM WAS USED HERE IS ROBUST ENOUGH TO REALLY PROVE THESE THINGS ARE RELATED, LET'S MAKE THAT ASSUMPTION, MAYBE A -- A BLINDED COMPARISON OF THIS, THAT, ALL OF THIS AND THAT AND THAT, WOULD BE USEFUL. EVEN THOUGH THIS IS A VERY DIFFERENT THING, YET, THEY ARE RELATED IN CERTAIN WAYS SO IT MIGHT BE YOU REQUIRE SUBSET OF INDIVIDUAL WHOSE ARE FACILE WITH SO WHEN I LOOKED I SAID MY GOD, ANOTHER ONE OF THESE HAIR BALLS. TELL ME WHAT IT MEANS. SO VERY PATIENTLY PEOPLE HAVE BEEN TRYING TO DO THAT. AND SO I THINK THESE TYPES OF STUDIES SAY THAT IT MIGHT BE A TRACTABLE PROBLEM. NOW, HOW DO YOU COMPARE THIS TO THAT? I DON'T THINK YOU CAN. NOT BY THE WAYS WE HAVE AVAILABLE TO DATE. >> I WOULD LOVE TO SEE A DIAGRAM OF THE PEOPLE THAT ARE IN THE STUDY SECTION. USING THESE SAME METRICS OF IMPACT FACTOR AND PUBLICATION. I WOULD LIKE TO SEE THAT FIRST. REVIEWERS. BECAUSE I PERSONALLY MY VIEW IS THE PROBLEM WITH PEER REVIEW IS PEER. HOW ARE PEOPLE IN STUDY SECTION IN RELATIONSHIP ARE PEERS GETTING A PEER REVIEW. >> SO YOU HAVE RAISED A CRUCIAL POINT. IT'S SAYING IN A SLIGHTLY PERHAPS CLEARER WAY THE QUESTION I ORIGINAL LILY POSED, TWO STUDY SECTIONS, A AND B, THIS STUDY SECTION HAS THE SENSE THAT THE BEST STUFF GOES THERE, AND THE LESSER STUFF GOES HERE. IF YOU WEREN'T FUNDED YOU'RE GOING TO THINK THAT PERHAPS THOUGH WHAT IF IT'S TRUE? WHAT IF THIS STUDY SECTION FOR WHATEVER REASON, COALESCED THE BEST PEER REVIEWERS AND AS A RESULT THE BEST SCIENCE EMERGES. THAT'S ONCE OF THE QUESTIONS YOU COULD ANSWER RICHARD NOKAMORA HAS CLEVER Qx=P‡IVE WAYS TO APPROACH THAT, I SHARE ONE EXAMPLE OF THAT WITH YOU. BUT ALSO IF YOU DID THE SAME TYPE OF ANALYSIS FOR PEER REVIEW US IT MAY ALSO IF YOU SUPERIMPOSE IT MAY ALSO GIVE US SOME INSIGHT SO THAT'S VERY GOOD SUGGESTION. THANK YOU. >> SO I WOULD LIKE TO AMPLIFY THE COMMENTARY BECAUSE I THINK THERE'S NOTHING MORE DISHEARTENING THAN HAVE SOMEONE SUBMIT A GRANT AND FEEL AS IF THE REVIEW WAS NOT GIVEN A FAIR DUE DILIGENCE BY TRUE PEER. AND EVEN THOUGH I HAD WRITTEN DOWN SOME THINGS I THOUGHT IMPORTANT ATTRIBUTES OF REVIEWERS, IT ALSO RELATES TO ADMINISTRATIVE OFFICER, PROJECT OFFICER. AND LOOKING AT BEHAVIOR IN STUDY SECTION OVER TIME REFERABLE TO THE DISPERSION OF A WARDS AND UNDERSTAND IF THERE IS A CULTURE WITHIN CERTAIN STUDY SECTIONS DRIVEN BY ADMINISTRATIVELY, EITHER INCLINING TOWARDS OPTIMAL PERFORMANCE OR NOT AS GOOD PERFORMANCE. >> I KNOW RICHARD AND COLLEAGUES AT CSR HAVE BEEN THINKING ABOUT THAT SET OF QUESTIONS VERY DEEPLY. OF COURSE THIS ALL COALESCES THE NEW SUBCOMITTEE OF THE ACD WORKING GROUP DIVERSITY, WILL PROVIDE INSIGHTS THAT ARE NOT GERMANE TO DIVERSITY PER SE BUT THE ENTIRE FABRIC OF HOW STUDY SECTION SOCIOLOGY OF HOW STUDY SECTIONS WORK. WE CAN OPTIMIZE THINGS AS WE GET A BETTER UNDERSTANDING OF THE TYPES OF THINGS THAT YOU REFER TO. >> WHEN I LOOK AT THIS, THIS WAS REALLY GOOD WORK. BUT I GET THE SENSE -- THIS IS LIKE FIRST AND SECOND AND THIRD ORDER ANALYSES ARE TAKING PLACE. IF WE LOOK AT THE ZERO ORDER ANALYSIS. THAT IS IF WE REFER TO WHAT HAPPENED LAST YEAR NIH IS NOT FUNDING THE BEST SCIENCE IN CONTROVERSY THE ARTICLE CAME OUT LAST YEAR OR THE YEAR BEFORE. SO THE QUESTION, ZERO ANALYSIS IS NIH IS FUNNING THE BEST SCIENCE AND HOW DO WE MAKE -- AND THEN AFTER THAT, WE TALK HOW WE OPTIMIZE THE PROCESS HOW WE DO THAT. OPTIMIZING PEER REVIEW I THOUGHT THE FIRST QUESTION WOULD BE ADDRESSING UP HAPPY WHEN IT CAME OUT AND PEOPLE DID REBUTTALS HERE AND THERE BUT IN TERMS OF REALLY SAYING THAT THE BEST SCIENCE IS BEING FUNNED, SUPPORTED HERE, AND BEING ABLE TO MAKE THAT STATEMENT LOOK AT THE TOP THOUSAND, NOT SURE HOW IT WAS DONE MAYBE TOP THOUSAND CITED PAPERS BETWEEN 2005 AND 2009, WHICH OF THOSE PAPERS WERE FUNDED THROUGH THE NIH AND WHICH ONES WEREN'T. THAT ANALYSIS TO LOOK AT MAYBE ZERO ORDER ANALYSIS OF NIH FUNDING THE BEST SCIENCE AND HOW DO WE TALK ABOUT THAT, OPTIMIZING BUT NOT SURE, COMPLETELY ADDRESSED BUT NOT SURE THAT WOULD BE DISCUSSED OR WHAT. >> THE PARTICULAR PAPER YOU'RE REFERRING TO IS ONE EXAMPLE I WAS REFERRING TO EARLIER INDIVIDUALS OUTSIDE OF NIH USING SIMILAR TOOLS ARE GOING TO MINE DATA AND FRANKLY COME UP WITH IN THAT PARTICULAR CASE INCREDIBLY SPURIOUS CONCLUSIONS. SO MUCH SO THAT ONE WONDERS ABOUT THE EDITORIAL PROCESS THAT RESULTED IN PUBLICATION OF THAT. BEAUTY IS IN THE EYE OF THE BEHOLDER SO TO SAY ALL OF NIH SCIENCE AS YOU ALL KNOW DIFFERENT FIELDS ARE CITED IN DIFFERENT WAYS, SO FORTH. WE'RE LOOKING AT AN ACTIONABLE PLAN TO ADDRESS WHAT A NUMBER OF PEOPLE IN THE COMMUNITY HAVE TOLD US. SO WHEREAS YOU REFER TO FIRST ARE YOU GUY FUNDING THE BEST STUFF, VERSUS WHAT ARE YOU DOING TO ENSURE THAT THERE'S FAIRNESS AND THAT THINGS ARE NORMALIZED AND EQUALIZED, WE HAVE OPTED FOR THE LATTER APPROACH. WE WANT TO DO SOMETHING ABOUT THIS. AND WHERE YOU DO SOMETHING WHERE THE RUBBER MEETS THE ROAD ARE REVIEWS CH I SHOULD SAY, WE ARE FOCUSING ON CSR BECAUSE ROUGHLY 70% REVIEWS ARE DONE THERE. BUT TO TAKE THIS TO ITS LOGICAL CONCLUSION YOU HAVE TO LOOK AT STUDY SECTION IN INSTITUTES AND CENTERS AS WELL. >> SO ONE THING THAT WE CAN LOOK AT WHICH OF THE CRITERION SCORES DRIVES IMPACT. >> SALLY I THINK CAN SPEAK. >> WE HAVE LOOKED AT THAT. WE LOOKED AT OF COURSE APPROACH. AND SIGNIFICANT ARE THE TWO THAT DRIVE THE OVERALL IMPACT SCORE. MORE OFTEN THAN NOT. PARTICULARLY APPROVE. >> THAT'S THE BIT OF A PROBLEM IF APPROACH IS THE THING DRIVING, IT MEANS PEOPLE ARE PLAYING IT SAFE AND MAKING MISTAKES, NOT BEING PARTICULARLY INNOVATIVE. >> THAT IS WHY WE HAVE TO CONTINUE TO DO THESE ANALYSES AND WE HAVE TO CONTINUE TO REFINE THE INSTRUCTIONS GIVEN TO PEER REVIEWERS. I KNOW CSR RECENTLY DID THIS. VIS-A-VIS COMPACTION OF SCORING, CHANGE IN INSTRUCTION APPEARS TO HAVE ALLOWED A SPREADING OF THE SCORES BUT AGAIN, IT IS A CONSTANT ITERATION. >> BRINGING BACK WHAT STORY TALKED ABOUT YESTERDAY, IN BRINGING RIGOR TO REQUIREMENTS, THAT IS A PARTICULAR ITEM SHE'S TALKING ABOUT RIGOR ARE PART IN THE APPROACH SECTION. SO WE HAVE THE YIN AND YANG WHEN THINKING HOW TO DO THIS IN REVIEW WITHOUT TOO MUCH ONLY FA SAY ON APPROACH. >> (INAUDIBLE) ON APPROACH. I DISAGREE. IT'S ABSOLUTELY CRITICAL. THAT'S MEAT OF THE GRANT. >> HAVING IT IT IS IMPORTANT BUT THE INNOVATION SHOULDN'T TAKE SECOND PLACE. >> I REALLY STRONGLY DISAGREE. >> WE CAN HAVE A LONG DEBATE ABOUT THIS. [LAUGHTER] >> I SENSE THAT THE ACD IS REACHING CLOSURE, ITS PEOPLE ARE CLOSING THEIR BOOKS. AND LOOKING AT THEIR WATCHES. LARRY, THANK YOU FOR WALK US THROUGH WHAT IS BEGINNING AND WHICH WILL DEFINITELY EXPAND INTO OTHER ANALYSES. IF P YOU HAVE OTHER THOUGHTS ABOUT THINK DEEPLY ABOUT THIS. THIS IS A REALLY IMPORTANT ISSUE. ESPECIALLY IN A TIME OF CONSTRAINED RESOURCES. WE CANNOT AFFORD TO JUST LOOK THE OTHER WAY IF WE'RE NOT GETTING THE RIGHT BALANCE IN OUR PORTFOLIO. WHATEVER METRICS WE CAN COME UP WITH THAT ARE NOT INHERENTLY BIASED IN THEIR OWN WAY ARE WORTH LOOKING AT TO SEE IF WE CAN DO A CORRECTION HERE ON THE WAY WHICH THAT PORTFOLIO BALANCE IS ACHIEVED. >> THIS SEEMS TO BE AT THE HEART OF KIND OF WHAT THE OVERSIGHT IS, THIS COMMITTEE IS ABOUT. AND EVEN THOUGH I COULDN'T FOLLOW ALL EXACTLY, I DID APPRECIATE IT. WE WOULD BE HAPPY TO CONTINUE TO HAVE THESE PRESENTATIONS. BUT ARE THEY ALSO GOING TO COUNCIL? I THINK THIS IS AN IMPORTANT MESSAGE TO THE DIFFERENT COUNCILS. >> THEY'RE CERTAINLY BEING DISCUSSED AT THE CSR ADVISORY COUNCIL, THEY'RE INTENSELY INTERESTED IN THIS. THE INDIVIDUAL COUNCILS WE WILL GET THERE, I THINK PROBABLY WE NEED SOMETHING A BIT MORE DISTILLED IN ORDER TO HAVE THAT BE PRODUCTIVE CONVERSATION. THAT'S PART OF THE GOAL IS TO GET THERE. SO I PROMISE YOU WE WILL BE TALKING ABOUT THIS ISSUE AGAIN IN DECEMBER. WELL, THANK YOU ALL REALLY, FOR WONDERFUL MEETING OF A DAY AND A HALF, ALL YOUR INPUTS DULY NOTED. WE WILL BE FOLLOWING UP ON A VARIETY OF ISSUES BY ELECTRONIC MEANS AND PLAN TO SEE YOU AGAIN IN DECEMBER. I THINK THOSE DATES HAVE BEEN IN YOUR BOOK SOMEWHERE AND I HOPE THEY'RE IN YOUR DIARY SOMEWHERE SO THAT WE HAVE A FULL ATTENDANCE AT THIS TIME. MEANWHILE, STAY AWAY FROM TORNADOES, HAVE A LOVELY SUMMER, ALL OF YOU. PLEASE CALL ON US AT ANY TIME IF WE CAN ANSWER QUESTIONS OR HELP WITH MANY OF THESE ISSUES. THIS IS A REALLY PERFORM TIME FOR US TO ALL WORK TOGETHER. THANK YOU. WE'RE ADJOURNED.