GOOD MORNING, EVERYONE. LET ME BEGIN BY WELCOMING EVERYBODY. WE HAVE A GREAT DAY PLANNED, I BELIEVE, AND I'M NORM AUGUSTINE, I'M CHAIR OF THE SCIENTIFIC MANAGEMENT REVIEW BOARD, AND BELIEVE IT OR NOT, ACCORDING TO MY NOTES HERE, THIS IS OUR 16TH MEETING OF THE BOARD AS A WHOLE, AND INNUMERABLE MEETINGS OF OUR VARIOUS WORK GROUPS, AND I PARTICULARLY WANTED TO WELCOME THE MEMBERS OF THE BOARD BUT ALSO THE SPEAKERS WHO ARE GOING TO BE JOINING US, THE STAFF FROM THE INSTITUTES AND ALSO OUR PUBLIC GUESTS. THE MEETING TODAY IS FOCUSED ON THE OBVR WORK GROUP, WHICH IS, ORVETION THE VALUE BIOMEDICAL RESEARCH GROUP. IT'S BEEN ADDED FOR SOME MONTHS NOW, HAS A GOOD DEAL TO REPORT AND ALSO TODAY WILL BE A DAY FOCUSED ON COLLECTING INFORMATION FOR THE GROUP AS A HOLE. AS UH-UH KNOW OUR RULES SET UP BY THE CONGRESS, WE AS A TOTAL GROUP NEED TO MEET AT LEAST FIVE TIMES ON EACH TOPIC, AND TODAY IS ONE OF THOSE SESSIONS THAT WILL RELATE TO THAT. AND BEFORE THE MEETING BEGINS, I'D LIKE TO AS WE ALWAYS DO GO AROUND THE TABLE AND LET EVERYBODY BRIEFLY INTRODUCE THEMSELVES. I THINK I'VE ALREADY INTRODUCED MYSELF SO GAIL, HOW ABOUT IF YOU WERE TO GO NEXT. >> GAIL CASSELL, LECTURER AT HARVARD IN THE DEPARTMENT OF GLOBAL HEALTH AND SOCIAL MEDICINE, AND RECENTLY RETIRED FROM ELI LILLY AS VICE PRESIDENT FOR SCIENTIFIC AFFAIRS. AND I'M A MEMBER OF THE BOARD. >> I'M ALLEN GUTTMACHER, DIRECTOR OF THE EUNICE KENNEDY SHRIVER -- >> I'LL SPEAK IN SINGLE WORDS AND TAKE A BREADTH IN BETWEEN. I'M DAN GOLDIN, CHAIRMAN OF THE -- AND CEO OF THE INTELLSIS -- INFORMATION TECHNOLOGY BUSINESS, BUT PRIOR TO THAT, I WAS THE ADMINISTRATOR OF NASA FOR A NUMBER OF YEARS. I DEVELOPED A DEEP RESPECT AND PASSION FOR THE NIH AND WHEN ASKED TO SERVE ON THIS COMMITTEE, I DID SO WITH GREAT ENTHUSIASM. THANK YOU. >> I'M GILL OMENN FROM UNIVERSITY OF MICHIGAN. INTERNAL MEDICINE, MEDICAL GENETICS, PUBLIC HEALTH, AND NOW A NEW DEPARTMENT WE CALL COMPUTATIONAL MEDICINE AND BIOINFORMATICS. >> GOOD MORNING, LARRY TABAK, PRINCIPAL DEPUTY DIRECTOR OF NIH, HERE PINCH HITTING FOR FRANCIS. >> GOOD MORNING. MARTHA SUMMERMAN, DIRECTOR OF NATIONAL INSTITUTE OF DENTAL AND CRANIOFACIAL RESEARCH. >> I'M LA LAW REHAAK. >> JIM ANDERSON, DIRECTOR OF DIVISION OF PROGRAM COORDINATION PLANNING AND STRATEGIC INITIATIVES IN THE OFFICE OF THE DIRECTOR. >> GOOD MORNING. I'M DELLA HANN, DEPUTY DIRECTOR FOR THE OFFICE OF EXTRAMURAL RESEARCH. >> MORNING, GRIFFIN RODGERS, DIRECTOR OF THE NATIONAL INSTITUTE OF DIABETES, DIGESTIVE AND KIDNEY DISEASES. >> I'M ARTHUR -- PROFESSOR OF MEDICINE, UNIVERSITY OF PENNSYLVANIA, ENDOCRINOLOGIST BY TRAINING. >> AMY PATTERSON, STAFF TO THE SMRB. THANK YOU. >> WELL AGAIN, THANK YOU AND WELCOME, EVERYONE, AND AS YOU NOTED, DR. COLLINS WASN'T ABLE TO BE HERE TODAY, BUT HE HAD SOMETHING COME UP THAT PREVENTED HIM FROM JOINING US, BUT FORTUNATELY LARRY HAS AGREED AS PRINCIPAL DEPUTY TO REPRESENT NIH DURING OUR DELIBERATIONS TODAY, AND SO LARRY, THANK YOU FOR DOING THAT. ALSO ONE OF OUR MEMBERS IS JOINING US TODAY IN ANED AD HOC CAPACITY. THAT IS GIL. WHILE THE OFFICIAL PAPERWORK IS BEING CONFIRMED AND THEY'RE CHECKING WITH THE IRS AND ALL THESE THINGS THAT HAVE TO BE DONE. WE HAVE GREAT OPTIMISM THAT THIS IS GOING TO WORK OUT. SO GIL, WE'RE GLAD TO HAVE YOU WITH US. ALSO I SHOULD NOTE THAT WE'RE USING A PUSH TO TALK SYSTEM HERE TODA SO IF YOU COULD REMEMBER TO TURN ON YOUR MICS WHEN YOU'RE SPEAKING, THAT WOULD BE VERY HELPFUL. WE DO HAVE A LOT TO ACCOMPLISH TODAY, SO I'M GOING TO BE FAIRLY BRIEF. WE'RE GOING TO START OUT WITH A -- SORT OF AN UPDATE ON SOME OF THE THINGS THAT ARE GOING ON AT NIH. LARRY WILL PRESENT THAT. THEN WE -- LATER, HE WILL PRESENT A NEW CHARGE TO THE GROUP THAT THE NIH WOULD LIKE US TO UNDERTAKE. IT'S ONE WE'VE TALKED ABOUT FROM TIME TO TIME IN THE PAST, AND I THINK IT'S A GREAT OPPORTUNITY. THE REST OF THE DAY, WE'LL DEVOTE TO THE TOPICS RELATED TO THE SO-CALLED WORKING GROUP, THE VALUE OF BIOMEDICAL RESEARCH, AND GAIL WILL BE LEADING THAT DISCUSSION AND THOSE PRESENTATIONS IN HER CAPACITY AS THE CHAIR OF THE OVER WORKING GROUP. FOLLOWING HER OPENING PRESENTATION, WE'LL HAVE A SERIES OF PANELS WITH A NUMBER OF TERRIFIC SPEAKERS THAT WILL HAVE PRESENTATIONS. TWO OF THE SPEAKERS AT THE LAST MINUTE HAD ISSUES COME UP IN THEIR PERSONAL LIVES THAT PREVENTED THEM FROM BEING HERE, BUT THEY'VE AGREED TO BE HERE IN THE FALL, AND WE HAVE A VERY FULL AGENDA IN ANY EVENT. AT THE END OF THE DAY, WE WILL TALK A BIT ABOUT THE -- SORT OF THE OVERALL STATUS OF THE SMRB, NEXT STEPS, AND WHERE WE'RE HEADED IN THE FUTURE. AS ALWAYS THE CUSTOM, I WANT TO CALL IN THIS CASE LARRY, UPON YOU, I'LL ASK YOU A LITTLE LATER TO TALK ABOUT THE PROPOSED TASK, BUT JUST IF THERE'S ANYTHING YOU'D LIKE TO SHARE WITH US, WE'D LOVE TO HEAR. >> IN JUST A MOMENT, DR. TABAK IS GOING TO MAKE A FORMAL PRESENTATION, BUT I THINK JUST IF YOU HAVE ANY REFLECTIONS, MUSINGS YOU WANT TO SHARE. >> REFLECTIONS OR MUSIC. ALL RIGHT. A MAN OF FEW WORDS. SO A COUPLE OF THE ADMINISTRATIVE ITEMS THAT WE ALWAYS COVER, FIRST OF ALL MEMBERS OF THE PUBLIC ARE WELCOME TO MAKE COMMENTS DURING THE APPROPRIATE TIMES THAT HAVE BEEN DESIGNATED, AND IF YOU WOULD BE SO KIND, IF YOU WOULD LIKE TO MAKE COMMENTS, FIRST OF ALL SIGN UP AT ONE OF THE DESKS. I'M NOT SURE WHICH ONE. WHERE DO YOU SIGN UP? OUT IN THE HALL. OKAY. INDICATED YOU WOULD LIKE TO SPEAK. AS OUR TRADITION, WE'D ASK YOU HOLD YOUR COMMENTS TO FIVE MINUTES SO THAT EVERYONE WHO WISHES TO MAKE A COMMENT HAS AN OPPORTUNITY TO DO THAT. IF YOU'D LIKE TO MAKE A LONGER STATEMENT OR AN ADDITIONAL STATEMENT IN WRITING, WE CERTAINLY WELCOME THOSE, AND WE POST THOSE ON THE SMRB WEBSITE AND INCLUDE THEM IN THE BRIEFING MATERIALS AS THEY PERTAIN TO THE TOPIC WE'RE DISCUSSING. ONE OTHER ADMINISTRATIVE ITEM, ANOTHER IMPORTANT ONE, THOUGH, HAS TO DO WITH THE CONFLICT OF INTEREST RULES GOVERN OUR MEETING, SO AMY, WOULD YOU DO THAT. >> SURE THING. AND THIS IS A RITUAL PART OF OUR MEETING AND IT'S ONE OF THE MOST EXCITING PARTS SO I ASK YOU TO FASTEN YOUR SEAT BELTS. I WOULD LIKE TO READ INTO THE RECORD THE RULES OF CONDUCT AND THE CONFLICT OF INTEREST STATEMENT. AS MEMBERS OF THE SMRB, YOU ARE SPECIAL GOVERNMENT EMPLOYEES AND ARE, THEREFORE, SUBJECT TO THE RULES OF CONDUCT THAT APPLY TO GOVERNMENT EMPLOYEES. THESE RULES AND REGULATIONS ARE EXPLAINED IN THE REPORT ENTITLED "STANDARDS OF ETHICAL CONDUCT FOR EMPLOYEES OF THE EXECUTIVE BRANCH." AND YOU EACH RECEIVED A COPY OF THIS DOCUMENT WHEN UH-UH YOU WERE APPOINTED TO THE COMMITTEE. AT EVERY MEETING IN ADDITION TO REMINDING YOU ABOUT THE IMPORTANCE OF FOLLOWING THE ETHICS RULES, WE ALWAYS LIKE TO REVIEW THE STEPS WE TAKE AND ASK YOU TO TAKE TO ENSURE THAT ANY CONFLICT OF INTEREST BETWEEN YOUR PUBLIC RESPONSIBILITIES AND YOUR PRIVATE INTERESTS AND ACTIVITIES ARE IDENTIFIED I AND ADDRESSED. AS YOU KNOW BEFORE EVERY MEETING, YOU PROVIDE US WITH A GREAT DEAL OF INFORMATION ABOUT YOUR PERSONAL, PROFESSIONAL AND FINANCIAL INTERESTS. WE IN TURN USE THIS INFORMATION AS THE BASIS FOR ASSESSING WHETHER YOU HAVE ANY REAL, POTENTIAL OR APPARENT CONFLICTS OF INTEREST THAT WOULD COMPROMISE YOUR ABILITY TO BE OBJECTIVE IN GIVING ADVICE ON THE MATTERS BROUGHT BEFORE THIS COMMITTEE. IF SUCH CONFLICTS ARE IDENTIFIED, WE EITHER ISSUE A WAIVER OR RECUSE YOU FROM A PARTICULAR PORTION OF THE MEETING. NOW WE USUALLY WAIVE CONFLICTS OF INTEREST FOR GENERAL MATTERS BECAUSE WE BELIEVE YOUR ABILITY TO BE OBJECTIVE WILL MOT BE AFFECTED BY YOUR INTEREST IN SUCH MATTERS. BUT IT SHOULD BE NOTED THAT WE RELY TO A GREAT DEGREE ON YOU TO BE ATTENTIVE DURING THE COURSE OF THE MEETINGS TO THE POSSIBILITY THAT AN ISSUE MAY CROP UP THAT COULD AFFECT OR AT LEAST APPEAR TO AFFECT YOUR INTERESTS IN A VERY SPECIFIC WAY. AND IF THIS HAPPENS, WE ASK THAT YOU RECUSE YOURSELF FROM THE DISCUSSION, AND I'D LIKE TO BE NOTIFIED AND I WILL ASK YOU TO PHYSICALLY LEAVE THE ROOM. IF YOU HAVE ANY QUESTIONS ABOUT THE RULES OF CONDUCT OR CONFLICT OF INTEREST, OUR COMMITTEE MANAGEMENT OFFICER, LISA RUSSTIN, WILL BE HAPPY TO ADDRESS THEM. THANK YOU. >> THANK YOU. DOES ANYBODY HAVE ANY QUESTIONS ON THIS TOPIC? IT IS IMPORTANT TO OUR CREDIBILITY. OKAY. NOW LARRY, I'LL TURN TO YOU FOR YOUR REMARKS AT THIS POINT. >> OKAY. THANK YOU, NORM, AND AGAIN, GOOD MORNING, EVERYBODY. I'M VERY SENSITIVE TO THE FULL AGENDA SO I'M GOING TO TRY AND JUST BRIEFLY HIGHLIGHT SOME OF THE CURRENT ACTIVITIES AND INITIATIVES, INCLUDING A NEW CHARGE THAT NORM HAS ALREADY ALLUDED TO THAT WE'D LIKE THE SMRB TO TAKE ON FOR THE NIH. SO THIS JUST OVERLIES WHAT I'M GOING TO BE SPEAKING ABOUT. FIRST TO GIVE YOU SOME UPDATES ON CONCLUSIONS OF IMPLEMENTATION OF A SERIES OF WORKING GROUP REPORTS TO THE ADVISORY COMMITTEE TO THE DIRECTOR RELATED TO DATA AND INFORMATICS, BIOMEDICAL RESEARCH WORKFORCE, AND THEN IN PARTICULAR, THE EXPANDING THE DIVERSITY OF THE NIH-FUNDED WORKFORCE, ARTICULATE A NEW CHARGE TO THE SMRB, AND THEN SAY A FEW WORDS ABOUT VOBR. I DIDN'T KNOW HOW THE ACRONYM WAS PRONOUNCED UNTIL THIS MORNING. AND THEN IF THERE ARE ANY QUESTIONS, OF COURSE, WE'LL TAKE THOSE. SO THE ACD WORKING GROUPS REPORTED OUT ON A SERIES OF VERY IMPORTANT TOPICS, THE FIRST OF WHICH RELATES TO SO CALLED BIG DATA, AND AROUND THIS TABLE, MANY OF YOU ARE EXPERT IN THIS FIELD, AND SO THE OVERARCHING THEMES OF THIS REPORT, AND IF YOU WANT TO READ THE FULL REPORT, THE LINK IS LOCATED ON YOUR BOTTOM RIGHT, BUT THE COMMITTEE VIEWED NIH TO BE AT A PIVOTAL POINT IN ITS HISTORY, AND THEY ARGUE THAT UNLESS WE CAPITALIZE ON ALL OF THE EMERGENT TECHNOLOGICAL ADVANCES, WE REALLY RUN THE RISK OF FALLING SHORT OF ACHIEVING OUR MISSION. THEY POINT TO A WHOLE -- A SERIES OF INADEQUACIES, PARTICULARLY WITH REGARD TO THE NEXT GENERATION OF TOOLS THAT ARE REQUIRED TO BOTH MANIPULATE AND INTERROGATE THE EVER-GROWING SETS OF DATA, PARTICULARLY IN THE AREAS OF GENOMICS IMAGING AND ELECTRONIC HEALTH RECORDS. ONE MEMBER OF THE COMMITTEE POETICALLY CALLED OUR LACK OF ACTION BORDERING ON INSTITUTIONAL MALPRACTICE. I'M NOT SURE IF I FEEL THAT STRONGLY ABOUT IT, BUT NEVERTHELESS, TO GO FORWARD, CULTURAL CHANGES AT NIH ARE CERTAINLY NEEDED AND ARE ESSENTIAL, AND WE HAVE TO DEVELOP A SET OF OPPORTUNITIES TO ENHANCE DATA SHARING, TO ENHANCE OUR ABILITY TO INTEGRATE DATA FROM DISPARATIVE SETS OF DATA, AND THEN, OF COURSE, THE NEXT GENERATION OF ANALYTICAL TOOLS, AND MOST IMPORTANTLY, THE COMMITTEE EMPHASIZED THAT THIS CANNOT BE EPISODIC, WE CAN'T DO THIS AS A ONE OFF. THIS NEEDS TO BE A LONG-TERM COMMITMENT. AND SO AMONG THE MAJOR PROBLEMS THEY SEE US NEEDING TO BE ABLE TO SOLVE, FIRST OF ALL WE NEED TO FIGURE OUT WHERE ALL OF OUR DATA IS. WE SUPPORT THE GENERATION OF ENORMOUS QUANTITIES OF DATA AND THE TRUTH IS, WE DO NOT HAVE A SYSTEMATIC WAY OF LOCATING AND REACHING OUT TO TOUCH THOSE VARIOUS DATA SETS. KNOWING WHERE THEY ARE ISN'T SUFFICIENT, WE ALSO HAVE TO BE ABLE TO ACCESS THOSE DATA SETS, AND THIS WILL IN PART REQUIRE NIH EXTENDING POLICIES AND PRACTICES FOR DATA SHARING. NOW SOME FIELDS OF SCIENCE DO THIS VERY WELL. I THINK GENOMICS HAS LED THE WAY IN MANY INSTANCES. BUT THERE ARE OTHER FIELDS OF SCIENCE WHERE YOU ARE LITERALLY BURIED WITH YOUR DATASET. AND THAT IS PROBABLY NOT THE BEST USE OF DATA. AND SO TO ORGANIZE, TO MA MANAGE, TO PROCESS BIOMEDICAL BIG DATA IS, OF COURSE, THE BIG CHALLENGE THAT EVERY ACADEMIC HEALTH CENTER IN THE NATION IS NOW FACING, AND THEN HOW YOU THEN ANALYZE THOSE DATA SETS. AND THEN FINALLY, TRAINING, AND THIS IS NOT ONLY TRAINING EXPERTS, AND GOODNESS KNOWS WE NEED MORE OF THOSE, BUT IF SOME OF YOU THINK BACK SOME 35 YEARS AGO, WHEN SCIENCE CAME TO THE REALIZATION THAT WE NEEDED TO EMBED MOLECULAR BIOLOGY INTO THE CURRICULA OF ALL GRADUATE STUDENT TRAINING REGARDLESS OF WHAT THE PROGRAM WAS, WE ARE NOW AT THE POINT WHERE QUANTITATIVE METHODS TO BE ABLE TO USE BIG DATA EFFECTIVELY MUST ALSO BE INTEGRATED INTO CURRICULA AROUND THE COUNTRY, REGARDLESS OF THE COURSE OF STUDY. SO IT'S BOTH TYPES OF TRAINING THAT THE COMMITTEE SPOKE TO. SO HOW ARE WE TACKLING THIS? WE ARE IN THE MIDST OF RECRUITING A NEW LEADERSHIP GROUP AT NIH, THE ASSOCIATE DIRECTOR FOR DATA SCIENCE, THE ACRONYM IS ADDS. I'LL LEAVE YOU TO DECIDE IF THAT'S CLEVER OR NOT. WE WILL BE USING TWO NEW OVERSIGHT BODIES TO GOVERN DATA, A SCIENTIFIC DATA COUNCIL AND ADMINISTRATIVE DATA COUNCIL. BOTH OF THESE WILL BE TIED DIRECTLY TO THE STEERING COMMITTEE OF NIH, WHICH AS MANY OF YOU KNOW, IS THE GROUP THAT PROVIDES THE GOVERNANCE STRUCTURE INTERNALLY AT THE AGENCY AND THEN THERE IS A NEW TRANSNIH INITIATIVE OF BD2K, BIG DATA TO KNOWLEDGE. THIS WILL BE STARTING IN FY14, AND THE VARIOUS GOALS ARE ENUMERATED HERE TO FACILITATE THE BROAD USE AND SHARING OF COMPLEX DATA SETS THROUGH DEVELOPMENT OF POLICIES, RESOURCES AND STANDARDS, TO DEVELOP AND DISSEMINATE NEW ANALYTICAL METHODS AND SOFTWARE, TO ENHANCE TRAINING OF DATA SCIENTISTS, COMPUTER ENGINEERS AND INFORMATICIANS AND LAST TO ESTABLISH CENTERS OF EXCELLENCE TO ADDRESS BIOMEDICAL ANALYTICS, COMPUTATIONAL BIOLOGY AND MEDICAL INFORMATICS. THE SECOND REPORT THAT I'D LIKE TO DRAW YOUR ATTENTION TO WAS ON THE BIOMEDICAL RESEARCH WORKFORCE. THE COMMITTEE, AND THEN THE REPORT LINK IS LOCATED ON YOUR BOTTOM RIGHT. IF YOU WANT MORE OF THE DETAIL. POINTED OUT THAT IT IS INCREASINGLY DIFFICULT FOR OUR PH.D.s TO LAUNCH TRADITIONAL INDEPENDENT ACADEMIC RESEARCH CAREERS, IN PART BECAUSE THERE'S A RISING NUMBER OF PH.D.s, AND IN PART BECAUSE THE NUMBER OF ESTABLISHED INVESTIGATORS TEND TO STAY IN FIELDS LONGER. THE ECONOMISTS WHO WERE ENGAGED IN THIS EFFORT POINTED OUT THAT THE LONG TRAINING TIME RELATIVELY LOW EARLY CAREER SALARIES MAKE BIOMEDICAL RESEARCH CAREERS LESS ATTRACTIVE FROM AN ECONOMIC VIEWPOINT THAN OTHER PROFESSIONS, AND SO YOUNG PEOPLE DO THE CALCULUS, AND THEY DECIDE THAT IT WOULD TAKE TOO MANY YEARS TO CATCH UP FINANCIALLY, SHOULD THEY EMBARK ON A CAREER IN BIOMEDICAL RESEARCH. FINALLY, IT WAS NOTED THAT OUR TRADITIONAL TRAINING PROGRAMS OFFER LITTLE PREPARATION FOR CAREERS OUTSIDE OF THE TRADITIONAL ACADEMIC PIPELINE DESPITE A DECREASING LIKELIHOOD OF FINDING THAT TRADITIONAL TYPE OF PROGRAM. SO THE RECOMMENDATIONS BASED ON THOSE OBSERVATIONS AND FINDINGS THAT GRADUATE STUDENTS BE PROVIDED ADDITIONAL OR ENHANCED TRAINING TO PREPARE THEM FOR THE MULTIPLE CAREER OUTCOMES, THAT WE SHORTEN THE PATHWAY FOR POSTDOCTORAL RESEARCHERS TO AN INDEPENDENT CAREER AND PRESUPPORT FOR THEIR TRAINING, IMPROVED PAY AND BENEFITS. WE NEED TO DO A MUCH BETTER JOB OF TRACKING CAREER OUTCOMES, WE NEED TO DO A MUCH BETTER JOB OF BEING ABLE TO DO WORKFORCE ANALYSIS IN A DYNAMIC WAY. THIS GROUP PRODUCED A STATIC PICTURE OF WHERE THE PH.D. WORKFORCE IS TODAY, BUT WITH BETTER TRACKING, WE SHOULD BE ABLE TO DO THIS DI DYNAMICALLY. CURRENTLY AN ANCILLARY STUDY IS BEING LAUNCHED TO LOOK AT PHYSICIAN AND OTHER CLINICIAN SCIENTISTS BECAUSE THIS INITIAL REPORT ONLY FOCUSED ON PH.D.s. THERE WAS A PARTICULAR PLEA TO LOOK AT BETTER WAYS OF PROMOTING THE VALUE AND STATURE OF WHAT WE CALL INTERNALLY AT NIH STAFF SCIENTISTS. YOU PROBABLY CALL THEM RESEARCH ASSISTANT PROFESSORS OR ASSOCIATE SCIENTISTS IN YOUR ORGANIZATIONS AROUND THE COUNTRY, AND THEN FINALLY, THE CALL FOR A GRADUAL REDUCTION IN THE PERCENTAGE OF FUNDS FROM NIH THAT IS USED TO SUPPORT FACULTY SALARY. UNDERSTANDING THAT THIS CANNOT HAPPEN OVERNIGHT, THAT DIFFERENT ORGANIZATIONS HAVE DIFFERENT BUSINESS MODELS, BUT THAT LONG TERM STEPS NEED TO BE TAKEN MOW TO BEGINOWTO BEGIN THAT CONVERSATION AND ULTIMATE REDUCTION. THE THIRD ACD WORKING GROUP REPORT RELATED TO EXPANDING THE DIVERSITY OF THE NIH-FUNDED WORKFORCE, THE REPORT NOT SURPRISINGLY FOCUSED ON FOUR ELEMENTS: THE NEED FOR NIH TO TAKE STEPS TO ENHANCE THE PIPELINE IN MENTORING, TO FACILITATE SHORING UP INFRASTRUCTURE IN KEY INSTITUTIONS AROUND THE COUNTRY AND THEN TO DEAL WITH CERTAIN ELEMENTS OF THE PEER REVIEW SYSTEM TO ENSURE ITS FAIRNESS. THE CHALLENGES THAT WE FACE IN THIS SPACE, FIRST THE REALIZATION THAT NO ONE SET OF INITIATIVES IS GOING TO DIVERSIFY THE NIH-FUNDED WORKFORCE OVERNIGHT. SECOND, THE VERY COMMUNITIES THAT WE MUST ENGAGE ARE THOSE WHERE WE HAVE LOST TRUST, WHERE WE HAVE LOST CREDIBILITY AND WE HAVE TO WORK HARD TO REGAIN THAT TRUST. NO MATTER WHAT NIH DOES, WE CAN'T DO ANYTHING IN ISOLATION. OUR SUCCESS REQUIRES COLLABORATION AND COOPERATION WITH ALL OF OUR EXTRAMURAL PARTNERS AND STAKEHOLDERS. DIVERSIFYING THE NIH-FUNDED WORKFORCE AND ENSURING THE FAIRNESS OF THE PEER REVIEW SYSTEM ARE COLLECTIVE RESPONSIBILITIES FOR EVERYBODY AT NIH, AND I WOULD OFFER FOR EVERYBODY IN THE EXTRAMURAL COMMUNITY AS WELL. THERE ARE FOUR INTERRELATED APPROACHES THAT WILL BE IMPLEMENTED TO TRY AND REACH THESE GOALS. THE FIRST, THE NIH BUILDING INFRASTRUCTURE LEADING TO DIVERSITY PROGRAM, SO CALLED BUILD, WHICH WILL BE LAUNCHED IN 2014. THIS FOCUSES ON UNDERGRADUATE STUDENTS, PROVIDING THEM WITH THE RESOURCES, MENTORING AND INTENSE HAND-ON MEANINGFUL RESEARCH EXPERIENCES TO GET THEM TO GO INTO PH.D. PROGRAMS RELEVANT TO BIOMEDICAL RESEARCH. NATIONAL RESEARCH MENTORING NETWORK TO AUGMENT AND STRENGTHEN WHAT IS ALREADY GOING ON LOCALLY AT INSTITUTIONS, THAT ALONE DOES NOT SEEM TO BE SUFFICIENT, AND SO THIS IS A NATION-WIDE EFFORT TO PROVIDE MENTORING EITHER VIRTUALLY OR FACE TO FACE THROUGH MANY WHO WOULD BENEFIT FROM ADDITIONAL MENTORING A SUBGROUP OF THE ACD WORKING GROUP ON DIVERSITY TO HELP WORK THROUGH APPROACHES TO ENSURING FAIRNESS IN PEER REVIEW, AND FINALLY INCREASED ENGAGEMENT BY ALL NIH LEADERSHIP, A STEERING COMMITTEE WORKING GROUP ON DIVERSITY HAS BEEN CREATED, AND WE ARE CURRENTLY RECRUITING A CHIEF OFFICER FOR SCIENTIFIC WORKFORCE DIVERSITY. SO NOW WE COME TO WHAT WE WOULD LIKE TO HAVE AS A NEW CHARGE FOR THE SMRB. THERE IS NO REST FOR YOU, THE WEARY. WHAT WE ARE ASKING THE SMRB TO DO IS TO EXAMINE THE NIH REVIEW AND AWARD PROCESS. AND I WILL ELABORATE ON THE DIMENSION OF THIS. AS YOU ALL KNOW, NIH REVIEW AND AWARD PROCESSES ARE FUNDAMENTAL TO OUR MISSION. WE HAVE LONG USED A TWO-TIERED REVIEW SYSTEM. IT IS THE FOUNDATION UPON WHICH THE AGENCY'S FUNDING OF EXTRAMURAL RESEARCH IS BASED, AND AGAIN, YOU ALL KNOW THAT OVERWHELMINGLY OUR RESOURCE GOES TO EXTRAMURAL RESEARCH GRANTS. WE CAN'T REST ON OUR LAURELS, IT'S VIE IT WILL THAT WE VITAL WE CONTINUE TO INNOVATE AND OPTIMIZE THE PROCESS BY WHICH GRANT APPLICATIONS ARE SUBMITTED, REVIEWED, PROCESSED, AND AWARDED. NOW JUST IN TERMS OF ORIGINS, AGAIN, I KNOW MANY OF YOU ARE VERY AWARE OF THIS, THE NATIONAL CANCER ACT AMENDMENTS OF 1974 REQUIRED PEER REVIEW OF BOTH NIH AND THEN THE ADAMHA GRANT APPLICATIONS AND CONTRACT PROJECTS. TODAY, HOWEVER, IF YOU FAST FORWARD TO MODERN TIME, THE RESEARCH ENTERPRISE IS FACING A WHOLE HOST OF ADDITIONAL CHALLENGES. IN NO SHORT PART DUE TO THE ECONOMIC CONSTRAINTS THAT HAVE RESULTED IN DECREASES IN APPLICATION SUCCESS RATES, AND AT THE SAME TIME, THERE HAVE BEEN A WHOLE PLETHORA OF TECHNOLOGICAL ADVANCES THAT WE NEED TO PERHAPS CAPITALIZE ON TO IMPROVE THE OVERALL EFFICIENCY AND EFFECTIVENESS IN OUR GRANT AWARDING PROCESS. SO THE SPECIFIC CHARGE, WE REQUEST THAT THE SMRB RECOMMEND WAYS TO FURTHER OPTIMIZE THE PROCESS OF REVIEWING AND AWARDING GRANTS. IN ADDRESSING THIS CHARGE, THE SMRB SHOULD CONSIDER SPECIFICALLY HOW NIH CAN FIRST STREAMLINE THE GRANT-MAKING PROCESS AND SHORTEN THE LENGTH OF TIME FROM APPLICATION TO ALLOCATION OF FUNDS, AND SECONDLY TO ADDRESS THE ADMINISTRATIVE BURDEN ON APPLICANTS, IN THEIR INSTITUTIONS, SCIENTIFIC REVIEWERS, COUNCILMEMBERS, AND NIH STAFF WHILE STILL MAINTAINING A HIGH QUALITY REVIEW PROCESS. SO THIS JUST UNDERSCORES WHEN YOU DO A GOOD JOB, BE CAREFUL BECAUSE WE COME BACK AND WE ASK YOU TO DO THINGS MORE EXPANSIVELY. SMRB OBVIOUSLY BEGAN TO APPROACH THIS GENERAL QUESTION IN LOOKING AT THE SBIR PROGRAM. SO WE REQUEST THE FUNDAMENTAL PRINCIPLES THAT YOU RECOMMEND ANY UNRECOMMENDED CHANGES TO THE PROCESS, A COMPREHENSIVE ANALYSIS OF EACH STEP IN THE PROCESS, AND FINALLY RECOMMENDED STRATEGIES AND OPTIONS FOR IMPROVING THE PROCESS AS WELL AS THE RATIONAL FOR THE RECOMMENDATIONS. OKAY. THEN I JUST WANTED TO SAY A FEW WORDS ABOUT THE VALUE OF BIOMEDICAL RESEARCH CHARGE TO SMRB. FIRST I WANT TO PERSONALLY THANK YOU ALL FOR TAKING THIS ON. THIS WAS A NON-TRIVIAL INCREDIBLY IMPORTANT SUBJECT AREA COMING AT A CRUCIAL TIME IN THE AGENCY'S HISTORY. IN THISCHARGE, DR. COLLINS ASKED FOR ADVICE ON ASSESSING THE VALUE OF BIOMEDICAL RESEARCH. THE IMPETUS, OF COURSE, IS THAT AS THE LARGEST BIOMEDICAL RESEARCH AGENCY, THIS COUNTRY WE ARE ENTRUSTED WITH MUCH OF THE PUBLIC'S INVESTMENT IN BIOMEDICAL RESEARCH, AND AS THE STEWARDS OF THIS INVESTMENT, WE ARE, OF COURSE, RESPONSIBLE FOR ENSURING THAT THESE FUNDS ARE USED IN WAYS THAT PROVIDE VALUE TO THE PUBLIC AND THE CURRENT ECONOMIC CLIMATE SIMPLY UNDERSCORES THE NEED TO DEMONSTRATE TO ALL UNAMBIGUOUSLY THE VALUE OF INVESTING IN NIH. YOU KNOW THE MISSION OF THE NIH LISTED HERE. THE BIOMEDICAL RESEARCH FUNDED BY NIH PRODUCES THE FOUNDATIONAL KNOWLEDGE THAT IS USED BY RESEARCHERS FROM MANY, MANY SECTORS TO DEVELOP TOOLS AND AN UNDERSTANDING TO IMPROVE HUMAN HEALTH. SO IN JULY OF LAST YEAR, DR. COLLINS ISSUED THE CHARGE TO THE BOARD, NIH REQUESTED THE SMRB IDENTIFY APPROPRIATE PARAMETERS AND APPROACHES FOR ASSESSING AND COMMUNICATING THE VALUE OF BIOMEDICAL RESEARCH SUPPORTED BY NIH. SO WITH THAT, I WILL STOP. I THINK ON TIME AND ON BUDGET. AND IF THERE ARE ANY QUICK QUESTIONS, I'M HAPPY TO ENTERTAIN THEM. >> LARRY, THANK YOU VERY MUCH. WE DO HAVE TIME FOR QUESTIONS. PLEASE, IF YOU WANT TO GO AHEAD, ARTHUR. >> I JUST WANT TO COMPLIMENT YOU, LARRY. IT DOES SEEM TO ME THAT THIS TOPIC THAT THE NIH HAS ASKED THE SMRB TO LOOK AT, WHICH IS THE STATE OF PEER REVIEW, IS A REALLY TIMELY ONE, BECAUSE IN ALL THE YEARS THAT I'VE BEEN INVOLVED IN THIS, TOO LONG TO MENTION BUT MORE THAN 30 YEARS, THE DISSATISFACTION AMONGST PEOPLE ASKED TO PEER REVIEW GRANTS AND THE DIFFICULTY THEY HAVE IN TERMS OF IDENTIFYING DIFFERENCES IN GRANTS AT THE TENTH PERCENTILE OR LOWER, AND FOR THE FRS TIME I'VE HEARD MORE SENIOR INVESTIGATORS TURN DOWN BECAUSE THEY DON'T THINK IT'S WORTHWHILE, SO IT'S A REALLY TIMELY THING TO UNDERSTAND THE IMPORTANCE OF PEER REVIEW AND THE CREDIBILITY IT'S HAD FOR SOME YEARS, I THINK LOOKING AT IT OBJECTIVELY AND SEEING IF SOME CHANGES CAN BE MADE IS REALLY IMPORTANT, SO MY BOTTOM LINE IS JUST TO THINK IT'S A VERY, VERY IMPORTANT AND TIMELY TOPIC, AND ONE THAT WE SHOULDN'T TAKE FOR GRANTED THAT IT'S WORKING PERFECTLY IN A CHANGED ENVIRONMENT. >> GAIL. >> YES, I WOULD DEFINITELY SECOND WHAT ARTHUR HAS JUST SAID. IN FACT, I TOLD NORM I'VE BEEN WAITING FOR THIS ONE. IT'S NOT JUST A MATTER OF, I THINK, THE FACT THAT SOME OF THE MORE SENIOR PEOPLE, THE REASON THEY'RE TURNING IT DOWN IS BECAUSE THEY'RE TOO BUSY WRITING THEIR OWN GRANTS AND LOSING THEIR FUNDING THEMSELVES, SO THE PAIN THAT I HAVE SEEN IN TERMS OF THE QUALITY OF THE PEER REVIEW, THE FEEDBACK, IF YOU WILL, I THINK IS FAR LESS USEFUL THAN IT FORMERLY WAS IN TERMS OF BEING INSTRUCTIVE SO THAT IT MAYBE IS NOT AS EFFICIENT AS IT COULD OR SHOULD BE, ESPECIALLY FOR MENTORING THE NEW AND YOUNGER INVESTIGATORS. >> CERTAINLY THIS IS A MULTIDIMENSIONAL SET OF ISSUES. IN NO SMALL PART, BEING WORSENED BY THE CURRENT BUDGETARY SITUATION. WE FEEL THAT THERE IS ALWAYS ROOM FOR IMPROVEMENT. WE FEEL THAT GIVEN THE GOOD WORK THAT YOU WERE ABLE TO DO WITH THE SBIR STPR SET OF PROGRAMS, THAT IT'S NOW FINE TO LOOK TO SEE IF WE CAN TRANSFER SOME OF THOSE EFFICIENCIES AND BEST PRACTICES TO THE LARGER SET OF PROGRAMS. THIS WILL COMPLEMENT SOME ONGOING THINGS THAT ARE ALREADY GOING ON WITHIN CENTER FOR SCIENTIFIC REVIEW, AND WITHIN THE OFFICE OF THE DIRECTOR, WITHIN OER. THE POUGHKEEPSIE -- WE'RE ALWAYS TRYING TO ENHANCE HOW PEER REVIEW IS DONE FROM ITS VARIOUS DIMENSIONS, BUT FUNDAMENTALLY, THE OBSERVATIONS THAT YOU'VE EACH MADE ABOUT SENIOR PERSONS INCREASINGLY EXPRESSING CONCERN AS TO WHETHER THEY SHOULD EVEN PARTICIPATE IN THE PROCESS IS SOMETHING THAT WE HOPE CAN BE OVERCOME IN PART BY MAKING THE PROCESS MORE EFFICIENT, BECAUSE WE ONLY GO TO BUSY PEOPLE AS YOU ALL KNOW, AND I THINK BUSY PEOPLE ARE PARTICULAR LISTENS ADVERTISED TO PROCESSES AND PROCEDURES THAT THEY PERCEIVE AS BEING LESS THAN OPTIMALIZED WITH REGARD TO THE EFFICIENCY. >> OTHER COMMENTS, PLEASE. >> AGAIN, I'D LIKE TO COMPLEMENT THE LEADERSHIP ON LOOKING AT THE PEER REVIEW PROCESS BECAUSE IT RELATES TO SOME OF THE OTHER ISSUES THAT YOU HAD, AND AS THE SMRB LOOKS AT IT, I THINK IT SHOULD LOOK TO THE FOR A ADJUSTMENT OR SOLUTION IN HAVING A BIG BANG, IF YOU WILL, BUT TO LOOK AT THE PEER REVIEW MAY HAVE TO BE CHANGED DUE TO ITS VERY NATURE, NOT TO DISRUPT ITS INTEGRITY OVER TIME. SO ONE MIGHT LOOK AT MILESTONES, WHAT CAN WE DO IN THE FIRST YEAR, THIRD YEAR, AND THE FIFTH YEAR, BECAUSE CHANGING THAT SYSTEM IS GOING TO BE VERY DIFFICULT AND IF IT GOES TOO FAST, IT COULD BE DISRUPTIVE AND IF IT'S NOT CHALLENGING ENOUGH, IT'S NOT GOING TO BE EFFECTIVE, SO THERE'S AN INTERESTING BALANCE, SO LOOKING AT MILESTONES OVER MAYBE A FIVE-YEAR PERIOD MAY BE VERY APPROPRIATE. >> SO CERTAINLY ONE OF THE GREAT STRENGTHS OF THE SMRB IS THE PROCESS ENGINEERING THAT MANY OF YOU WITH ENGINEERING BACKGROUNDS IS BROUGHT TO THE TABLE, AND I THINK THAT'S PRECISELY SORT OF THING THAT WE'RE HOPING TO GAIN FROM THIS IMPORTANT CHARGE. >> ONE THING I THINK WOULD BE VERY HELPFUL FOR THE SMRB TO ADDRESS IS THE TWO-TIERED PROCESS, AND ADVICE AND INPUT INTO HOW TO MORE EFFECTIVELY REALLY UTILIZE BOTH TIERS, I THINK WOULD BE HELPFUL TO US WHO ARE INVOLVED IN IMPLEMENTING THESE COMPLEX ISSUES. >> ARTHUR, YOU HAVE SOMETHING YOU WANTED TO ADD? YOUR MICROPHONE IS LIT UP. >> LARRY, I'D LIKE TO ADD -- YOU TRIGGERED A LOT OF THOUGHTS AS YOU WERE TALKING, AND FIRST WITH REGARD TO THE PEER REVIEW ISSUE, PEER REVIEW HAS CERTAINLY SERVED US WELL FOR A VERY LONG PERIOD OF TIME. WE DON'T WANT TO THROW THAT BABY OUT WITH THE BATHWATER. HERE, BUT TIMES HAVE CHANGED, AND I AM VERY MUCH HOPEFUL WE COULD TAKE ON A TASK LIKE THIS FOR A LONG TIME. I'VE BEEN TRIGGERED BY SORT OF THE FLIP SIDE OF, ARTHUR, WHAT YOU RAISED, AND THAT IS AS I TALK TO REALLY OUTSTANDING RESEARCHERS IN THIS COUNTRY, NOT ONLY RELATED TO NIH BUT TO NSF AND OTHER INSTITUTIONS, WHEN YOU GET DOWN TO WHERE YOU HAVE A 15% PROBABILITY OF YOUR PROPOSAL BEING FUNDED, THE AMOUNT OF TIME THAT'S WASTED BY THESE REALLY CAPABLE PEOPLE WRITING DEAD-ENDED PROPOSALS IS A TRAGEDY. WE COULD PROBABLY INCREASE OUR RESEARCH OUTPUT BY SOME SUBSTANTIAL PERCENTAGE IF SOMEHOW WE COULD KNOW IN ADVANCE WHICH ONES WERE GOING TO SUCCEED AND WHICH WEREN'T. SOMEHOW WE'VE GOT TO ADDRESS THAT KIND OF QUESTION AND FIND A WAY TO MAKE THE SYSTEM MORE EFFICIENT. NOT ONLY DID I FIND AT TALKING TO PEOPLE -- BUT ALSO THIS RISK-AVERSE THAT GETS INTO THE SYSTEM THAT WE'VE ALL TALKED ABOUT A LOT THAT I THINK IS PART OF WHAT WE OUGHT TO TALK ABOUT HERE. THEN LASTLY, IT TAKES A LONG PERIOD OF TIME TO GO THROUGH PEER REVIEW BECAUSE APPARENTLY PEOPLE IN MOST GOVERNMENT INSTITUTIONS WHO DO PEER REVIEW, THEY DO IT FACE TO FACE. THE SCIENCE MOVES TOO FAST. I'VE GOT A FRIEND WHO RAN INTEL CORPORATION, HE TOLD ME THAT 90% OF THE REVENUES THEY RECEIVE ON THE LAST DAY OF ANY YEAR COME FROM PRODUCTS THAT DIDN'T EXIST ON THE FIRST DAY OF THAT YEAR, SO IF A PEER REVIEW PROCESS TAKES A YEAR, YOU'RE A WHOLE GENERATION BEHIND YOUR COMPETITORS. SO I THINK THERE ARE JUST A LOT OF REALLY COMPELLING REASONS TO TAKE THIS ON. I'M THRILLED YOU'VE PROPOSED IT. >> SO AGAIN, YOU TOUCH UPON THE MANY DIMENSIONS OF THIS ISSUE. IT'S DIFFICULT TO PARSE OUT THE BUDGETARY CONSTRAINTS AND HOW THAT'S TORQUING THE SYSTEM, BUT THAT SAID, YOU'RE QUITE RIGHT. IN AN ERA OF BIG DATA, ARE WE MAKING BEST USE OF ANALYTICS, ANALYTICAL TOOLS, OF ELECTRONIC MEANS? THIS REPRESENTS SORT OF THE WAY STUDY SECTIONS HAVE ALWAYS BEEN. ALL THAT'S MISSING ARE THE PINK-COLORED SHEETS THROWING THE THINGS INTO THE CENTER OF THE TABLE FOR THOSE OF WHO YOU SERVED ON STUDY SECTION BACK THEN. SO YES, WE ARE LOOKING FORWARD TO THIS GROUP'S DELIBERATIONS ON THIS. AND AGAIN, THIS WILL WORK TOGETHER WITH SOME ONGOING THINGS THAT ARE ALREADY GOING ON, AND ONCE THE GROUP LAUNCHES, WE'LL ENSURE THERE'S APPROPRIATE INTEGRATION AND CROSSTALK WITH THE OTHER VARIOUS THINGS THAT ARE ONGOING. >> SO I DO HAVE A DELICATE COMMENT. SO WE ARE ANSWERABLE TO CONGRESS AS WELL AS THE NIH, I MEAN, THERE'S A LOT OF CRITICISM OF NSF AT THE MOMENT AND I THINK BY IMPLICATION OR NIH REVIEW RELATED TO PEER REVIEW AMONGST THE VARIETY OF CONGRESS COMMITTEES AND SO ON, WHAT KIND OF INPUT DO WE THINK IS VALID IN TERMS OF OUR ROLE, IN TERMS OF GETTING THEIR OPINIONS AND SO ON, BECAUSE JUST TO IGNORE IT AND BE CRITICAL OF WHAT THEY'RE DOING ONLY GOES SO FAR, AND SOME OF IT WOULD BE BETTER TO INCORPORATE THEIR OPINIONS AND THINK ABOUT HOW THEY FEEL ABOUT WHAT WE'RE DOING, SO I'M TRYING TO DO A DELICATE BALANCE BETWEEN SAYING SOME OF THE QUESTIONS ARE VERY UNREASONABLE AND YET SAY THEY'RE THE PEOPLE WHO FUND US ANYWAY AND IT'S TAXPAYERS' MONEY, SO HOW DO WE KIND OF INCORPORATE THEIR OPINIONS AND CRITICISMS IN A LEGITIMATE WAY? >> SO NO, I THINK THE QUESTION IS VERY IMPORTANT. AND CERTAINLY PEOPLE HAVE NOT BEEN BASHFUL ABOUT ARTICULATING THEIR VIEWS, AND I THINK WHEN CHALLENGED, AS TO THE VALIDITY OF WHAT WE ARE DOING, WE HAVE TO BE ABLE TO DEFEND IT, AND IF WE CAN'T, THEN WE HAVE TO CHANGE IT, RIGHT? SO CERTAINLY YOUR ANALYSES WILL HELP US IN THAT REGARD. IT IS QUITE POSSIBLE THAT THE GROUP WILL FIND PORTIONS OF WHAT WE DO TO BE IN NEED OF ENHANCEMENT OR COMPLETE CHANGE. THAT'S OKAY. I THINK THE END GOAL HERE, AS I'M SURE YOU ALL APPRECIATE, IS TO MAKE OUR PEER REVIEW SYSTEM THE ABSOLUTE STRONGEST IT POSSIBLY CAN BE SO THAT IT CAN BE DEFENDED AGAINST ANY AND ALL-COMERS. SO I WOULDN'T SHIRK AWAY FROM THAT. I THINK WE NEED TO LOOK AT THAT IN A VERY OPEN AND TRANSPARENT WAY. >> I'VE HAD THE OCCASION TO MEET WITH MOST OF THE KEY PEOPLE ON THE HILL NOW ON THIS PARTICULAR DEBATE, AND AS YOU KNOW IT STARTED OUT WITH THE SOCIAL SCIENCES, MY GUESS IS IT'S GOING TO MOVE TO THE BIOSCIENCES NEXT AND THE PHYSICAL SCIENCES LATER. AND I THINK IT'S A REAL PROBLEM, BUT TO ME, IT SHOULD BE A PART OF WHAT WE'RE GOING TO TALK ABOUT. I THINK IT'S FAIR GAME, AND WE SHOULD EXPRESS OUR VIEWS, WHATEVER THEY MIGHT COLLECTIVELY TURN OUT TO BE. IT'S JUST TOO IMPORTANT AN ISSUE TO NOT TAKE THE OPPORTUNITY. >> AND CERTAINLY YOUR DELIBERATIONS TODAY ARE COMPLEMENTARY TO WHAT YOU WILL BE DISCUSSING. THE VALUE OF BIOMEDICAL RESEARCH IS CENTRAL TO THE DEFENSE OF WHAT WE DO AND HOW WE DO IT. AT THE END OF THE DAY, WE EITHER SUPPORT WHAT IS VALUABLE OR WHAT IS NOT. AND NONE OF US HAVE THE CRYSTAL BALL TO BE ABLE TO PREDICT NECESSARILY WHERE THE NEXT GREAT DISCOVERY WILL BE. BUT WE DO HAVE TO HAVE A TRANSPARENT SYSTEM THAT IS FAIR, AND THAT ENSURES THAT WE HAVE THE BEST CHANCE OF PROVIDING SUPPORT FOR THE THINGS THAT ARE MOST LIKELY TO MOVE THE AGENDA FORWARD. >> I THINK I DIDN'T SAY IT VERY WELL BEFORE. I MENTIONED THE QUALITY OF THE REVIEW AND THE QUALITY OF THE FEEDBACK. BUT TO ME, THIS IS REALLY AT THE VERY HEART OF THE PROBLEM, AND WHAT I SEE ARE SUMMARY STATEMENTS THAT CONTAIN INACCURACIES, SUMMARY STATEMENTS THAT CLEARLY FOCUS ON MINUTIA IN TERMS OF WHAT THE OVERALL POTENTIAL IMPACT MIGHT BE, AND I DON'T EVEN KNOW HOW YOU MIGHT EVEN GO ABOUT TRYING TO ASSESSOR IMPROVE THE CURRENT STATE OF AFFAIRS, BUT TO ME, THAT'S ONE AREA THAT IS ALMOST -- SHOULD BE TEASED OUT AND FOCUSED UPON BY IT SELF, AND THEN WHAT HAPPENS IS THAT YOU HAVE THE ACCOMPLISHED INVESTIGATORS TRYING TO RESPOND TO THE INACCURACIES, THE MINUTIA, AND WITH THE TRIAGE RULES THAT THERE ARE TODAY, IN FACT, THEY BURN OUT FAIRLY QUICKLY IN TERMS OF THEIR PATIENCE AND TOLERANCE AND OPPORTUNITIES TO RESUBMIT. SO I THINK WE'RE LOSING A LOT OF OUR INVESTIGATORS, AND IT'S NOT JUST THE QUALITY OF THE PEOPLE THAT YOU ULTIMATELY GET TO SERVE ON THE REVIEW GROUPS, BECAUSE MANY OF THEM ARE VERY FLIPPANT AND WON'T TAKE THE TIME TO EDUCATE IN TERMS OF FEEDBACK. SO SORRY TO GO ON, BUT HEARING AND THINKING A LOT ABOUT IT AND I DO THINK THE SUMMARY STATEMENT, LET'S JUST SAY, TO USE AN OLD TERM, MORE OR LESS IS REALLY -- HAS A MAJOR ISSUE. >> WELL, AND AGAIN, YOU ALL SHOULD APPRECIATE THAT WE HAVEN'T SAT BACK AND DONE NOTHING. KEITH, JEREMY AND I A FEW YEARS AGO WORKED THROUGH A PEER REVIEW ENHANCEMENT FOR THE ACD, AND WE ARE STILL EVALUATING THE IMPACT RKIMPACT,THE PERCEPTIONS ON STAKEHOLDERS THAT WE MADE. SOME GOOD AND WELL ACCEPTED, OTHERS NOT SO GOOD. TO BE BLUNT ABOUT IT. BUT AT THE END OF THE DAY, WE HAVE TO ENSURE THAT WE HAVE A PROCESS THAT MEETS ALL OF THE EXPECTATIONS SO THAT WHEN WE ARE ASKED TO DEFEND WHY HAVE YOU SUPPORTED THIS VERSUS THAT, WE CAN POINT TO A PROCESS THAT IS FAIR, THAT IS TRANSPARENT, AND THAT THE LOGIC MODEL GETS US TO A PLACE WHERE WE HAVE A VERY GOOD CHANCE OF SUPPORTING THE VERY BEST OPPORTUNITY. >> YOU HAD A COMMENT? >> YES, I'LL TALK MORE ABOUT IT LATER, BUT I WANT TO POINT OUT THAT THE TIME TO DO SOMETHING IS VERY CRUCIAL. THERE ARE MODELS IN THE COMMERCIAL WORLD TO DO A SCREENING, AND ONE THAT COMES TO MIND IS IN SAN DIEGO, THE ANGELS HOLD A BEAUTY CONTEST, AND THEY GIVE YOU, I THINK, FIVE MINUTES. YOU GET UP IN FRONT OF THE PEOPLE THAT HAVE THE MONEY, AND IT'S THE INITIAL SCREENING. AND AGAIN, YOU CAN DO A DEEP DIVE. I POINT OUT WHEN I WAS AT NASA AND I ASKED THE NATIONAL ACADEMIES TO DO CRUCIAL STUDIES FOR ME, THE ANSWER I GOT BACK, TWO TO THREE YEARS. I SAID, THE CONGRESS AND THE WHITE HOUSE DOES NOT OPERATE ON A TWO-TO THREE-YEAR SCHEDULE. IF THE NIH IN THE NEW WORLD OF BIG DATA IS GOING TO BE INSULAR TO PROTECT THE PERCEPTION OF INTEGRITY AND NOT TAKE A LOOK WHAT'S HAPPENING IN THE BROADER WORLD, IT'S GOING TO BECOME OBSOLETE, AND IT'S OF GREAT CONCERN TO ME, SO RELATIONSHIP TO TIMING IN A WORLD WHERE THAT'S CONTROLLED BY DIGITS HAS TO BE ADDRESSED WHETHER WE LIKE IT OR NOT. >> RIGHT. AND I THINK THAT YOU'RE SPOT-ON WITH THE CAVEAT THAT THE BEAUTY CONTEST, AS YOU PUT IT -- >> I USED THE WRONG WORD. >> NO, NO, NO, BUT IT'S ACTUALLY CRUCIAL, BECAUSE IF WE ADOPT NEW APPROACHES, WE HAVE TO MAKE SURE THAT THOSE APPROACHES ARE FAIR. SO THAT JUST HAS TO BE THE WAY IT IS. AND WE'VE GOT TO BE ABLE TO DEMONSTRATE THAT THEY ARE FAIR. BUT YOU'RE QUITE RIGHT, WE'VE GOT TO TAKE FULL ADVANTAGE OF THE MODERN APPROACH. >> SO ONE SHORT THING, IT SEEMS IN A LOT OF THIS KIND OF THING, PILOT RESEARCH OF SELECTED THINGS THAT COULD BE DONE RELATIVELY QUICKLY WOULD BE SOMETHING THE SMRB COULD CONSIDER. I KNOW SOME OF THAT HAS BEEN DONE INTERNALLY IN THE REVIEW THING, BUT IT ALWAYS SEEMS AT A VERY LOW LEVEL OR AT LEAST WHAT WE KNOW ABOUT IT SEEMS TO BE LESS THAN OPTIMAL, SO A NUMBER OF EXPERIMENTS LIKE NORM AND OTHERS AND DAN HAS SAID, DONE OVER A COUPLE OF MONTHS, YOU KNOW, WITH A LOT OF DATA THAT'S NOW AVAILABLE THAT COULD SOME BE RETROSPECTIVE COULD BE REALLY QUITE AN ECK SITTIN EXCITING THING TO DO S O WE COULD BASE SOME DECISIONS ON FACTS RATHER THAN JUST PEOPLE'S OPINIONS. I HOPE WE'LL TAKE THAT APPROACH. >> THERE IS A TREMENDOUS AMOUNT OF WORK THAT IS ACTUALLY ALREADY GOING ON IN THIS SPACE WHICH AT THE APPROPRIATE TIME, I AND OTHERS WILL HAVE TO BRIEF THE COMMITTEE OR SUBCOMMITTEE THAT THATACKLES THIS, BUT YOU'RE QUITE RIGHT, THE ITERATIVE PROCESS OF PILOT TO INFORM SUBSEQUENT DECISION-MAKING IS AN OUTSTANDING WAY TO GO. >> I GUESS I'D LIKE TO JUST MAKE ONE FINAL COMMENT, THEN WE NEED TO MOVE AHEAD. I DON'T WANT TO LET THE DISCUSSION OF THE INFORMATION ASPECTS THAT YOU TALKED ABOUT -- I THINK THEY'RE CLOSELY COUPLED. IT SEEMS TO BE WE'RE VERY CLOSE TO A WORLD WHERE THE TRANSMISSION AND PROCESSING, STORING OF INFORMATION IS FREE, AND THAT'S GOING TO JUST REVOLUTIONIZE EVERYTHING. THERE'S A RELATED THING GOING ON TO ME, AND I'VE MISSED THAT BEFORE, PEOPLE HAVE USED THE TERM "DISTANCE IS DEAD." DISTANCE IS LARGELY DEAD, AND IF THAT'S TRUE, IT COULD HAVE A HUGE IMPACT ON THE PEER REVIEW PROCESS AND SO ON THAT YOU DON'T HAVE TO BE IN THE SAME ROOM ANYMORE TO DO THINGS. I THINK WE'RE SEEING WHAT IS LIKELY TO BE A HUGE IMPACT ON OUR UNIVERSITIES FROM THIS REVOLUTION, AND THE UNIMPORTANCE OF DISTANCE, AND THAT'S LIKELY TO BE A REVOLUTION THAT'S GOING TO IMPACT NIH EVEN MORE THAN IT ALREADY HAS. IT'S ONE THAT MIGHT DESERVE SOME SPECIAL ATTENTION IN ITS OWN RIGHT. TO ME, SOME OF THE HUGE REVOLUTIONS OF THE ENGINEERING WORLD I LIVED IN AFTER THE COLD WAR WENT THROUGH A REVOLUTION. THEN THE PUBLISHERS TOWARD THE END OF THEIR REVOLUTION, I THINK EDUCATION -- HIGHER EDUCATION IS NEXT. AND IF THAT'S RIGHT, IT'S GOING TO HAVE A BIG IMPACT ON AN INSTITUTION LIKE THIS. PROBABLY SHOULD MOVE AHEAD. GIL, THAT MEANS WE TURN TO YOU FOR YOUR REMARKS AND INTRODUCE THE PANEL. THE FLOOR IS YOURS. >> I'LL TRY TO BE BRIEF SINCE WE'RE BEHIND A LITTLE BIT ALREADY, NORM. WHAT I WANT TO DO IS JUST FIRST OF ALL SAY THAT WHEN WE WERE GIVEN THIS CHARGE LAST JULY, MOST OF US SERVING ON THIS SUBCOMMITTEE JUMPED AT THE CHANCE BECAUSE WE THOUGHT WE KNEW EXACTLY WHAT WE WANTED TO DO AND MAYBE WHAT SHOULD BE DONE, BUT I WOULD JUST SAY THAT IT HAS BEEN ONE OF THE MORE CHALLENGING THINGS I'VE EVER DONE ON BEHALF OF NIH IN TERMS OF GETTING A GRASP ON THE CHARGE AND THEN KNOWING THAT WE'RE DOING THE RIGHT THING GOING FORWARD. SO WHAT I WANTED TO TALK TO YOU ABOUT THIS MORNING, REALLY, IS JUST TO TALK A LITTLE BIT ABOUT THE GROUP, BUT ALSO THE PROCESS THAT WE'RE TRYING TO USE TO TAKE ON OUR CHARGE. SO AS YOU SEE HERE, THIS IS OUR WORKING GROUP, YOU HAVE THIS, I WON'T READ OFF THE NAMES, AND YOU WILL NOTICE THAT MOST BUT NOT ALL OF THE INDIVIDUALS IN OUR WORKING GROUP ARE WITH US TODAY. HIGHLY SKILLED INDIVIDUALS COMING FROM VERY DIFFERENT ACTIVITIES. YOU CAN SEE HERE A LIST OF WHAT WE'VE BEEN PAST YEAR, AND MAINLY WE HAVE BEEN TRYING TO ASSESS THE LANDSCAPE, THAT IS, TO DETERMINE WHAT OTHER SIMILAR ACTIVITIES MAY BE BEING UNDERTAKEN BY OTHER GROUPS. EVERYBODY LIKES TO WEIGH IN ON THE VALUE, PARTICULARLY OF MAJOR INVESTMENTS, CONGRESSIONAL INVESTMENTS, IE, THE NIH BUDGET, SO WE'VE DONE, I THINK, A VERY THOROUGH ASSESSMENT WITH THE HELP OF THE STAFF ON THE LANDSCAPE AND WHAT IS GOING ON. THIS IS TO INCLUDE A THOROUGH LITERATURE REVIEW, ASSIGNMENT OF MEMBERS OF THE WORKING GROUP TO READ SOME OF THE BETTER PAPERS, AND THEN IN ADDITION TO SELECT FROM THAT GROUP, PEOPLE WHO CAME AND SPENT TIME AS A PANEL MEMBER, AS PANEL MEMBERS WILL DO TODAY, GIVING US THEIR THAT'S ABOUT THE VALUE OF BIOMEDICAL RESEARCH AND IN PARTICULAR THE ECONOMIC VALUE. IN ADDITION, THEN, WHAT WE HAVE DONE IS TO HAVE A LOT OF DELIBERATIONS BOTH IN PERSON, BUT ALSO IN -- ON THE PHONE BY HAVING A NUMBER OF DIFFERENT CONFERENCE CALLS BETWEEN OUR MEETINGS. IF YOU JUST LOOK AT THIS SLIDE HERE, IT ADDRESSES OUR DELIBERATIVE PROCESS. IN FACT, IN THE CHARGE, THE ESTABLISHMENT OF THE SMRB ITSELF, ONE OF THE THINGS THAT WAS DONE IN THE CONGRESSIONAL LANGUAGE WAS TO ENSURE IN FACT THAT ANY CHANGES THAT WERE TAKEN THROUGH NIH WOULD BE A THOROUGH AND DELIBERATIVE PROCESS, SO WE ARE TRYING TO DO THE SAME, AS WE DETERMINE HOW WE SHOULD APPROACH ASSESSING THE VALUE OF BIOMEDICAL RESEARCH, AND IF YOU LOOK AT THIS SLIDE, IN FACT, IF YOU LOOK AT THE GREEN ROW, WE'VE BEEN DELIBERATING THE FIRST TWO STEPS, AND ONE IS ASSESSING THE NEED FOR CHANGE, AND EVALUATING OPTIONS FOR CHANGE. IN OTHER WORDS, TO LOOK AT THE MORE TRADITIONAL WAYS OF ASSESSING VALUE AND WE'RE SAYING THAT MAYBE THAT'S NOT QUITE GOOD ENOUGH, USING SOME OF THE NEWER TECHNOLOGY, BUT ALSO INTERPRETATIONS OF VALUE AND HOW THEY CHANGE OVER TIE. AND THEN TO EVALUATE THE OPTIONS OF THE WAY YOU GO ABOUT MAKING CHANGE. IN OTHER WORDS, THE PROCESSES. WE HAVE ARGUED VEHEMENTLY THAT THE ATTRIBUTES OF THE PROCESS HAVE TO INCLUDE TRANSPARENCY, THAT THERE HAS TO BE AN OPTIMUM MEANS OF COMMUNICATION, AND ACCOUNTABILITY. SO HOW DO YOU DEFINE THE VALUE OF BIOMEDICAL RESEARCH? IT'S RATHER OBVIOUS TO ALL OF US THAT ONE OF THE BEST THINGS THAT COME FROM INVESTMENT IN BIOMEDICAL RESEARCH IS NEW KNOWLEDGE. AND THAT'S A RATHER IN MANY WAYS EASY SELL BECAUSE EVERYBODY UNDERSTANDS WHAT ACQUISITION OF NEW KNOWLEDGE CAN MEAN, AND ITS POTENTIAL. SO WE ALSO ARE BEGINNING TO APPRECIATE, AS I MENTIONED EARLIER, HOW YOUR CONCEPT OF VALUE CHANGES WITH YOUR AGE. ALSO I WOULD SAY THAT THE VALUE ALSO CHANGES DEPENDING ON THE PRICE YOU PAY FOR THAT. I HAPPEN TO BE MARRIED TO A BANKER, AND WHEN I STARTED MY CAREER, IT WAS RATHER EASY TO SELL HIM ON WHAT I WAS DOING AND WHY I WAS APPLYING FOR NIH GRANTS, BUT TODAY HE'S LOOKING AT THE NIH BUDGET AT OVER $30 BILLION AND LISTENING TO, YOU KNOW, A LOT OF THE REPORTS NOW BOTH IN THE PRINTED AND THE TV IN TERMS OF CONTROVERSIAL RESULTS AND WANTING TO KNOW WHAT THE HECK IS GOING ON, SO I THINK THAT WE DO NEED A DIFFERENT WAY OF EXPRESSING AND LOOKING AT VALUE FOR MANY DIFFERENT REASONS. SO WE HAVE CONCLUDED THAT THERE ARE THREE TYPES OF VALUE THAT WE WANT TO FOCUS ON. AND ONE IS, OF COURSE, THE ACCUMULATION OF SCIENTIFIC KNOWLEDGE, AND THEN TO LOOK AT THAT SCIENTIFIC KNOWLEDGE AND HOW IT TRANSLATES INTO IMPROVED PUBLIC HEALTH, AND THEN LASTLY, HOW IT TRANSITIONS INTO BROADER SOCIETAL EFFECT, PRIMARILY INCLUDING ECONOMIC BUT ALSO EDUCATIONAL IMPACT, DIPLOMATIC AND OTHER TYPES OF VALUE THAT DON'T NECESSARILY ALWAYS GET TAKEN INTO ACCOUNT. I THINK DURING THE LAST 10 TO 12 YEARS IN PARTICULAR, IT'S BECOME MORE AND MORE APPARENT HOW IMPORTANT OUR SCIENTIFIC DIPLOMACY AND OUR HEALTH DIPLOMACY PLAY A ROLE IN ESTABLISHING FOREIGN POLICY. IT'S WAY UNDERVALUED. AMY PATTERSON CAN TELL YOU FROM HER INTERNATIONAL EXPERIENCES HOW MUCH IT'S VALUED AND THE ROLE IT'S PLAYED IN MANY VERY PRECARIOUS SITUATIONS THAT I COULD DO OVER THE LAST 10 TO 15 YEARS WORKING WITH THE NATIONAL ACADEMIES IN THIS. IN FACT, IT'S REALLY HARD TO PLACE A VALUE ON THE IMPORTANCE OF SCIENTIFIC DIPLOMACY, BUT WITHOUT THE ACQUISITION OF NEW KNOWLEDGE, THE LEADERSHIP IN BIOMEDICAL RESEARCH, YOU LOSE A LOT OF MAJOR ADVANTAGES THAT WE HAVE BECAUSE OF THE INVESTMENT THAT WE'VE MADE IN BIOMEDICAL RESEARCH. SO WHAT ABOUT IF WE THEN MOVE ON AND TALK ABOUT THE BASIC FRAMEWORK FOR NIH VALUE. SO I DON'T WANT TO NECESSARILY WALK THROUGH THIS IN GREAT DETAIL IN THE INTEREST OF TIME, BUT JUST TO EMPHASIZE AGAIN THAT THERE IS A CONTINUUM, THE ACCUMULATION OF BASIC KNOWLEDGE THAT THEN LEADS TO IMPROVED INTERVENTIONS, IMPROVED HEALTH, AND THEN LASTLY, THEN, THE BROADER SOCIETAL IMPACTS AND WE'LL HAVE A LOT OF OPPORTUNITY, I THINK, TO TALK MORE ABOUT THESE AS THE DAY GOES ON. SO WHAT ABOUT THE MAJOR ELEMENTS OF CHANGE AND HOW WE ARE GOING TO BASICALLY EVALUATE THESE ELEMENTS OF CHANGE? SO THE FIRST MAJOR GOAL REALLY WITH REGARDS TO ELEMENTS OF CHANGE IS THE CHARGE OF IDENTIFYING TH THE GUIDING PRINCIPLES THAT SHOULD UNDERLIE ANY ASSESSMENT OF VALUE, AND WE'LL SPEND SOME TIME TODAY TALKING ABOUT THE PRINCIPLES AND OUR PANELISTS ARE GOING TO HELP US DO THAT. THE SECOND ELEMENT THAT WE DECIDED WOULD BE VERY IMPORTANT IS TO PROVIDE ADVICE TO NIH AS THEY MOVE FORWARD TO TRY TO BEGIN THEIR ASSESSMENT OF VALUE OF THE BIOMEDICAL RESEARCH IS THAT THEY ARE USING THE SOUNDEST METHODS AND STRATEGIES FOR ASSESSING VALUES, SO THE TOOLS, IN OTHER WORDS, THEN LASTLY ADVICE REGARDING STUDY QUESTIONS AND SELECTION OF STUDY TOPICS. WE'VE RECOMMENDED AND SUGGESTED THAT PERHAPS THIS COULD INVOLVE CASE STUDIES AND I'LL COME BACK AND SAY A FEW WORDS MORE ABOUT THE CASE STUDIES. SO LET'S TALK A LITTLE BIT ABOUT THE GUIDING PRINCIPLES, THE LIMITATIONS AND CAVEATS, AND THE FIRST ONE BEING THE PRINCIPLES THAT I MENTIONED, AND ONE MIGHT ASK WHY DOES NIH NEED TO BETTER ASSESS ITS VALUE, WHAT CAN WE ACCOMPLISH BY THIS EFFORT, AND SO THE GROUP HAS REALLY SPENT A LOT OF TIME BASICALLY TALKING ABOUT THIS. SO THE PRINCIPLES UNDERLYING VALUE ASSESSMENT THAT WE THINK ARE REALLY IMPORTANT ARE ATTRIBUTION, CAUSALITY, PRECISION, COMPREHENSIVENESS, DISCLOSURE OF ASSESSMENT LIMITATIONS, REFLECT VALUES OF SOCIETY, AND THEN WE ARE CONFIDENT THAT THERE ARE OTHERS. WHAT ARE THE MAJOR CHALLENGES WITH ASSESSING THE VALUE OF BIOMEDICAL RESEARCH? ESPECIALLY, WE THINK THAT ONE OF THE MOST MAJOR CHALLENGES WITH THE ASSIGNING OF ATTRIBUTION AND CAUSALITY, BECAUSE WE REALIZE THAT MULTIPLE FACTORS AND SECTORS INFLUENCE THE DOWNSTREAM EFFECTS OF NIH ACTIVITIES, JUST TO GIVE YOU ONE EXAMPLE, WE'VE TALKED A LOT ABOUT, ONCE YOU HAVE KNOWLEDGE IN HAND, HOW DO YOU TRANSLATE THAT INTO PRACTICE? THE INSTITUTE OF MEDICINE HAS JUST ISSUED A RECENT REPORT ON THE LEARNING HEALTHCARE SYSTEM, AND IF YOU GET THE DATA, WHAT YOU FIND, IT CAN BE DECADES FROM THE TIME THAT THE DEFINITIVE STUDIES ARE PUBLISHED BEFORE THEY ACTUALLY FIND THEIR WAY INTO PRACTICE GUIDELINES. SO IT IS DIFFICULT, NUMBER ONE, TO GET PEOPLE TO ACCEPT AND THEN TRANSLATE, BUT ALSO DIFFICULT WITH REGARDS TO THE FACT THAT MANY THINGS INFLUENCE THESE DOWNSTREAM EFFECTS. ONE IS THE ADAPTABILITY OF THE NEW KNOWLEDGE AND PUTTING IT INTO PRACTICE, BUT THERE ARE ALSO OTHER BEHAVIORAL FACTORS IN EVERYTHING ELSE THAT PRECLUDE THEIR ADOPTION. AT ANY RATE, IT'S ALSO DIFFICULT TO ESTIMATE IT AND ACCOUNT FOR THE LAG TIME BETWEEN RESEARCH AND IMPACT, THESE DECADE SOMETIMES THAT IT TAKES TO PUT INTO PRACTICE ONCE YOU KNOW THAT GROUP B CAUSES SEPSIS IN NEWBORN INFANTS. I'M ALWAYS AMAZED, ALLEN, THAT SOMEONE ACTUALLY DID THE STUDY TO SHOW THAT IT WAS ALMOST 15 YEARS BEFORE IT WAS KNOWN AND PROVEN, NO QUESTIONS ASKED, BUT BEFORE WE BEGAN ROUTINE SCREENING FOR GROUP B IN THE PERINATAL PERIOD. SO THE NEXT WAS WE ACKNOWLEDGE THAT THERE ARE MANY MYRIADS OF CHALLENGES IN COLLECTING AND ANALYZING THE DATA THAT CAN ACTUALLY CAPTURE THE OUTCOMES, AND THIS GOES BACK TO ALL THE BIG DATA ISSUES THAT WE'VE JUST TALKED ABOUT. AND IS THERE ANYTHING E UNIQUE FOR AGENCIES WITH ITS CHALLENGES, WE MAY HEAR A LITTLE OF THAT THIS AFTERNOON FROM ELLIOTT, OUR LAST PRESENTER TODAY. SO WHAT ARE THE COMMON ELEMENTS OF VALUE ASSESSMENTS? THERE ARE MANY MODELS FOR ASSESSING VALUE THAT HAVE DIFFERENT CONTEXTS, AND WE FEEL THAT THE IDENTIFICATION OF THE CRITICAL COMPONENTS OF OUR WORKING GROUPS MODEL ACTUALLY SHOULD BE A PART OF THE WORKING GROUP. I DIDN'T SAY THAT VERY WELL, BUT NEVERTHELESS, HOW SHOULD WE MEASURE VALUE IS WHAT I WAS TRYING TO SAY, SO THAT WE ACTUALLY HAVE COME UP WITH WHAT WE FEEL IS A VERY COMPLEX BUT NEVERTHELESS VALUABLE APPROACH TO ASSESSING THE VALUE, AND THIS SLIDE HERE WHICH IS RATHER COMPLEX AND YOU CAN'T SEE BUT SHOW, AGAIN, HOW ONE FEEDS INTO THE OTHER, THE SCIENCE, THE BASIC KNOWLEDGE, THE BASIC SCIENCE RESEARCH FEEDING INTO PUBLIC HEALTH AND THE BROADER SOCIETAL IMPACT AND THAT THE OUTPUTS ARE BOTH SHORT TERM, MEDIUM TERM AND LONG TERM, AND HOW DO YOU TRY TO MEASURE THAT. I'M NOT DOING THIS JUSTICE AT ALL, BUT YOU ALL HAVE COPIES OF THESE SLIDES, SO THIS JUST REALLY GETS AT -- OR LEADS US INTO THE NEXT QUESTION, WHICH IS WHAT TOPICS BEST COMMUNICATE AND REPRESENT NIH'S VALUE. ONE OF THE THINGS THAT OBVIOUSLY IS CHA CHALLENGING IS BECAUSE OF THE BREADTH OF THE RESEARCH FUNDED BY NIH. WITH RESPEN RESPECT TO THE CASE STUDIES THAT I MENTIONED IN TERMS OF BEING A WAY TO ILLUSTRATE VALUE, I JUST WANTED TO SAY A FEW WORDS TO MOVE US ALONG. THAT IS, ONE OF THE GOALS OF THE CASE STUDIES IS TO OBVIOUSLY TAKE INTO ACCOUNT THE APPLIED VERSUS BASIC RESEARCH AND THE DIFFERENCE IN THE PROPORTION WHICH IS NOW NOT AS GREAT AS IT ONCE WAS IN TERMS OF THE GAP. AND TO ALSO, THEN, SHOW THAT RATHER THAN CHERRY PICKING CERTAIN KINDS OF CASE STUDIES, WE WOULD HOPE THAT THE CASE STUDIES COULD ILLUSTRATE THE FULL SPECTRUM AS I SAID OF THE NIH RESEARCH, THIS BREADTH OF RESEARCH IN THE NIH, BOSS THE CLINICAL, THE SLOW AND THE QUICK, SUCCESSES AND FAILURES AND WHAT THOSE FAILURES MAY TELL US. ANOTHER VERY BROAD GOAL OF THE CASE STUDY SELECTION IS TO UNDERSCORE THE IMPORTANCE OF INVESTING IN CERTAIN TYPES OF RESEARCH BUT ALSO IN DIFFERENT DISEASE AREAS. AND HERE YOU JUST SEE THAT WE HAVE, IN FACT, SPENT A FAIR AMOUNT OF TIME EVEN SINGLING OUT WHAT WE THINK MIGHT BE SOME GOOD CASE STUDIES THAT WOULD ADDRESS EACH OF THE PHASES THAT I'VE MENTIONED. SO THE GOALS OF TODAY'S PRESENTATION ARE TO ACTUALLY HEAR FROM OUR PANELISTS ABOUT THE PRINCIPLES AND ATTRIBUTES OF SOUND ASSESSMENT EFFORTS, THE OPPORTUNITIES FOR IMPROVING ASSESSMENTS, A AND THEN TO HEAR A LITTLE BIT ABOUT PRIOR STUDIES AND CURRENT FUTURE EFFORTS TO ASSESS THE VALUE BOTH IN TERMS OF THE BASIC SCIENCE, BUT ALSO THE PUBLIC HEALTH AND THE BROADER SOCIETAL IMPACT. WHAT WE TRY TO DO IN SELECTING THE MEMBERS OF THE PANELS IS TO > GAIL, THAT WAS REALLY TERRIFIC. I'M ACTUALLY PINCH HITTING FOR STEVE KATZ AND I HAVEN'T BEEN PARTY TO THESE REALLY VERY THOUGHTFUL AND VERY USEFUL DELIBERATIONS TO DATE, BUT I AM REALLY IMPRESSED WITH THE RESEARCH THAT'S BEEN GIVEN TO US ALREADY, AND THE CONCEPTUAL FRAMEWORK FOR WHAT'S CLEARLY FOR US WHO HAVE THE TASK AND PRIVILEGE OF IMPLEMENTING THE DECISION-MAKING PROCESSES HERE BEEN A VERY THOUGHTFUL ANALYSIS. AND SO COMPLEMENTARY TO THE NEXT CHALLENGE, WHICH IS THINKING ABOUT THE WAYS IN WHICH THE TWO TIER PEER REVIEW PROCESS CAN BE STRENGTHENED TO HELP US ENSURE THE VALUE OF BIOMEDICAL RESEARCH AS STRONG AS WE CAN MAKE IT. SO AS I UNDERSTAND THE CHALLENGE TO OUR PANELISTS, IT HELPS US UNDERSTAND THE VARIOUS APPROACHES TO THE METRICS, TO THE LYMPH OU LIMB OF OUR PROCESSES AND WAYS THAT WE CAN STRENGTHEN THAT, AND I THINK IT SHOULD BE A THOUGHTFUL AND VERY VALUABLE DISCUSSION. >> THANK YOU, JOSIE. IT'S BEEN A PLEASURE WORKING ON THIS GROUP, AND WE'VE BEEN BUILDING TOWARD THIS SESSION TODAY FOR MONTHS ACTUALLY. THINKING ABOUT THE STRATEGIES OF THE NIH AND THINKING ABOUT THE NEEDS OF A COUNTRY, SO WE HAVE AN OPPORTUNITY HERE BOTH TO THINK ABOUT THE METH DO LOGICAL APPROACHES FROM SERIOUS SCIENTISTS, SOMETIMES AN AREA CALLED THE AREA OF SCIENCE POLICY, AND THE METRICS FOR EVALUATION. WE ALSO HAVE THIS VERY COMPLICATED MATTER OF DETERMINING WHAT IS AND HOW CAN IT BE DESCRIBED, WHAT IS VALUE. ONE WAY I THINK ABOUT THIS IS IN THE CONTEXT OF PORTFOLIO MANAGEMENT. AND IN FACT, IN YOUR TALK, LARRY, YOU MENTIONED THE SENSE OF INVESTMENT OF PUBLIC RESOURCES, AND THE RESEARCH ENTERPRISE. AS DOCUMENTED OVER MANY YEARS BY RESEARCH AMERICA AND OTHER GROUPS, THERE'S TREMENDOUS PUBLIC SUPPORT AND EXPECTATIONS IN THE PUBLIC INVESTMENT IN BIOMEDICAL RESEARCH, AND MAYBE WE SHOULD ALWAYS SAY BIOMEDICAL AND BEHAVIORAL RESEARCH. AS JOSIE JUST SAID, AND GAIL, IT'S BASIC CLINICAL, TRANSLATIONAL POPULATION RESEARCH, AND OVER THE DAY, WE'RE GOING TO HAVE A FOCUS FIRST ON THE KNOWLEDGE GENERATION, SECOND ON THE PUBLIC HEALTH BENEFITS TO OUR SOCIETY, AND THIRD, ON BROADER SOCIETAL OBJECTIVING. THE SCHEME OF QUESTIONS DIRECTED TO OUR PANELISTS ADDRESSES FIRST GUIDING PRINCIPLES, LIMITATIONS AND CAVEATS, AND THEN METHODS AND STRATEGIES, AND THEN STUDY TOPICS AND POTENTIAL CASE U.S. DIS. I WANT TO HIGHLIGHT JUST TWO THINGS. FIRST UNDER THIS CONCEPT OF PORTFOLIO MANAGEMENT, IT'S IMPORTANT FOR US TO ADDRESS WHAT IS THE STATED MISSION OF THE NIH. AND THAT MISSION IS TO ENHANCE FUNDAMENTAL KNOWLEDGE AND ITS APPLICATION TO ENHANCE HEALTH, LENGTHEN LIFE AND REDUCE THE BURDENS OF ILLNESS AND DISABILITY. SO IT'S IN THE DNA OF THE AGENCY THAT THE OBJECTIVES OF MAKING A DIFFERENCE IN THE HEALTH OF THE AMERICAN PEOPLE OR THE GLOBAL POPULATION IS CENTRAL TO THE MISSION JUST GENERATING MORE PUBLICATIONS OR MORE ACCUMULATED KNOWLEDGE THAT WOULD NOT SATISFY THAT MISSION STATEMENT. SECOND WITH REGARD TO THE BASIC KNOWLEDGE GENERATION ITSELF, I WOULD COMMEND EVERYONE A SPECIAL ISSUE OF SCIENCE MAGAZINE, JULY 2005. IT WAS A CELEBRATION OF THE 125TH ANNIVERSARY OF SCIENCE MAGAZINE BY THE AAA AMERICAN ASSOCIATION FOR THE ADVANCEMENT OF SCIENCE AND I HAPPEN TO BE THE AAAS PRESIDENT THAT YEAR. WHAT THEY DID WAS TO IDENTIFY ACROSS VAST RANGE OF SCIENTIFIC ACTIVITY AND RESEARCH INCLUDING ENGINEERING, 125 UNKNOWNS. TO TURN THE QUESTION AROUND INSTEAD OF BOASTING HOACH WE HOW MUCH WE LEARNED AND WHAT WE KNOW, WHAT DO WE NOT KNOW THAT WE THINK WOULD BE VERY IMPORTANT TO LEARN. THIS IS NOT ABOUT CURING DISEASE, THIS IS ABOUT BASIC SCIENTIFIC UNDERSTANDING. AND IT WAS A PRETTY INTERESTING WAY TO ADDRESS THE QUESTION. HOW MUCH PROGRESS ARE WE MAKING? ARE WE ADDRESSING THE MOST COMPELLING UNKNOWNS OR UNCERTAINTIES IN OUR KNOWLEDGE BASE FOR THE MISSION OF THE AGENCY AND THE COUNTRY. I COMMEND THOSE EXAMPLES TO YOU. I THINK IT MIGHT BE SOMETHING THAT WOULD BE A WAY TO THINK ABOUT HOW WE'RE DOING THE MOST IMPORTANT WORK HERE AT NIH. SO WITH THAT BIT OF INTRODUCTION, WE HAVE OUR THREE SPEAKERS TO ADDRESS US. DO YOU WANT TO START THE FIRST? >> RIGHT. >> DELLA IS FIRST. >> WE DECIDED THAT WE WOULD NOT REITERATE THE ILLUSTRIOUS CAREERS OF THE THREE SPEAKERS BECAUSE THE BIOGRAPHIES ARE IN FRONT OF YOU. DELLA HANN IS DEPUTY DIRECTOR OF THE OFFICE OF EXTRAMURAL RESEARCH AT THE NIH. DELLA. >> THANK YOU. GETTING SET UP HERE. GOOD MORNING. THANK YOU FOR THIS OPPORTUNITY TO PARTICIPATE IN THESE DISCUSSIONS. AS WE'VE HEARD ALREADY, THIS IS AN ENRICHING TOPIC. I THINK IT'S ONE THAT I'VE BEEN INVOLVED IN PERSONALLY FOR ALMOST AS LONG AS I'VE BEEN HERE AT THE NIH, WHICH NOW IS GOING ON A FEW YEARS, SHALL WE SAY. I'LL KEEP IT SIMPLE. WHAT I'D LIKE TO TALK WITH YOU THIS MORNING ABOUT IS POTENTIAL PICKING UP, I THINK IT DOVETAILS VERY NICELY WITH THE NUMBER OF THE TOPICS AND COMMENTS THAT HAVE BEEN RAISED ALREADY, PARTICULARLY BY GAIL AND THEN RECENTLY BY OTHERS ON THE PANEL THAT LOOKING AT THE VALUE AND TRYING TO UNDERSTAND HOW ONE CAN GO ABOUT ASSESSING THIS VALUE. WHAT I'M GOING TO BE TALKING ABOUT ULTIMATELY REALLY IS SORT OF AN INITIAL PROTOTYPE, IF UH-UH WILL, AN INITIAL SCAFFOLD, AN INITIAL CONCEPTUAL FRAMEWORK FRAMEWORK. BY NO MEANS DO I CONSIDER IT TO BE A FINAL PRODUCT, BY NO MEANS. IN FACT, YOU CAN TEAR IT APART, WHICH WOULD BE FINE TOO. BUT IT WAS AN ATTEMPT TO SORT OF, GIVEN WHERE I HAVE BEEN SITTING FOR THE NUMBER OF YEARS, THINKING THROUGH HOW WOULD WE GO ABOUT DOING THIS AND DEVELOPING WHAT I WOULD REFER TO AS CHANGE OF EVIDENCE IN TERMS OF SHOWING THE VALUE OF NIH RESEARCH. SO JUST TO SET THE STAGE, WHICH I PROBABLY DON'T NEED TO DO BUT I FELT COMPELLED IN A GOOD PRESENTATION TO DO SO, SORT OF WHAT WE'RE DEALING WITH, WHICH IS ESSENTIALLY BASED OFF OF 2009, WHEN MY OFFICE CONDUCTED AN ENUMERATION PROJECT TO THOROUGHLY LOOK AT THE NUMBERS AND TYPES OF INDIVIDUALS THAT WERE BEING SUPPORTED OFF OF RESEARCH PROJECTS. WE HAVE APPROXIMATELY 53,000 RESEARCH PROJECTS A YEAR THAT ARE FUNDED THROUGH THE EXTRAMURAL PROGRAM. WE ARE SUPPORTING OVER 300,000 RESEARCH POSITIONS OFF OF THOSE RESEARCH GRANTS. IN TERMS OF THE NRSA PROGRAM, WHICH IS OUR FORMALIZED TRAINING PROGRAMS, WE'RE TALKING ABOUT ROUGHLY 16,000 TO 17,000 INDIVIDUAL WHO ARE BEING SUPPORTED AND BEING TRAINED, AND THEN IF YOU LOOK AT THE POSITIONS ON THE RESEARCH GRANTS THAT ONE COULD REFER TO AS TRAINING FOR THE POST DOCS ON THE RESEARCH GRANTS, NOT THE NRSAs THAT, NUMBER JUST DOUBLES 28 TO 30,000 POSITIONS. ROUGH ESTIMATES. THEN OF COURSE WE HAVE OUR INTRAMURAL PROGRAM, WHICH ROUGHLY IS AROUND 6,000 SCIENTISTS WITHIN THE INTRAMURAL PROGRAM AND COUNTLESS OTHER, I SEE MICHAEL SITTING BACK THERE, IN TERMS OF TRAINEES THAT ARE ALSO PART OF THAT WORKFORCE. IN TERMS OF LOOKING AT NIH, WE HAVE ESSENTIALLY A SERIES OF PRODUCTS THAT WE CAN IMMEDIATELY GRAB ON TO, TO TALK ABOUT ISSUES OF KNOWLEDGE AND VALUE. THE COIN OF THE REALM ARE THE PUBLICATIONS, RIGHT? THAT'S HOW SCIENTISTS EXCHANGE KNOWLEDGE AND DEVELOP VALUE IN TERMS OF CITATIONS AND SO FORTH LIKE THAT. THERE'S NOTHING WRONG WITH THAT. WE ALSO HAVE CITATIONS IN CLINICAL GUIDELINES, AND JUST RECENTLY MY OFFICE HAS BEGUN LOOKING AT THIS IN TERMS OF THE PUBLISHED CLINICAL GUIDELINES AND LOOKING TO SEE THE CITATIONS IN THOSE GUIDELINES AND HOW THEY'VE CROSSWALKED BACK TO POTENTIAL NIH FUNDING. WE ALSO COLLECT INFORMATION HERE AT NIH FROM GRANTEES AND OTHERS ON INVENTIONS, PATENTS AND LICENSES. SO WE HAVE THAT SET OF DATA THAT'S AVAILABLE TO US. WE ALSO ARE IN THE PROCESS OF GATHERING MORE INFORMATION AND MORE ACCURATE INFORMATION, I SHOULD SAY, ON TECHNOLOGIES AND TECHNIQUES THAT OUR GRANTEES ARE PRODUCING, AND THEN OTHER PRODUCTS AS WELL. DATABASES, ANIMAL MODELS, INSTRUMENTS, ET CETERA. THE OFFICE OF EXTRAMURAL RESEARCH, WHICH IS THE LAND THAT I LIVE IN, IS RESPONSIBLE FOR DESIGNING THESE NEW AND MORE EFFICIENT METHODS OF DOING DAY A COLLECTION. AND IMPROVING THE BREADTH AND QUALITY OF THIS INFORMATION. SO OUR ELECTRONIC APPLICATION FORMS, TREMENDOUS AMOUNTS OF INFORMATION LIVE IN THOSE APPLICATIONS AND WE ARE TRYING TO REFINE THAT EVERY DAY IN TERMS OF OUR ABILITY TO CAPTURE THAT DATA AND USE IT MORE EFFECTIVELY. WE HAVE NOW THE RESEARCH PERFORMANCE PROGRESS REPORTS. THESE ARE LOVINGLY KNOWN AS THE RPTR. THIS IS THE ELECTRONIC PROGRESS REPORT. AND AGAIN, WE'RE MAKING EFFORTS TO TRY TO WORK WITH OUR FEDERAL PARTNERS BECAUSE THIS IS A FEDERAL-WIDE PROGRESS REPORT. SO WORKING WITH ALL THE PARTNERS IN TERMS OF DEVELOPING THIS PROGRESS REPORT AND BEING ABLE TO TUNE IT TO DIFFERENT KINDS OF ISSUES NOT ONLY IN TERMS OF THE MONITORING, WHICH IS CRUCIAL IN TERMS OF LOOKING FOR PROGRESS REPORTS, BUT ALSO WAYS TO BETTER CAPTURE THESE PRODUCTS THAT I JUST MENTIONED. AND INCLUDED WITHIN THE IS OUR REPORT ON THE PEOPLE. WHO IS IT THAT'S PARTICIPATING IN THE RESEARCH, WHAT ARE THE RESEARCH POSITIONS THEY HOLD, HOW MUCH TIME AND EFFORT ARE THEY DEVOTING TO THE PROJECT. AGAIN, WE'VE GONE THROUGH VERY RECENTLY ACTUALLY A SERIES OF EXERCISES TO REFINE THAT TOOL ESSENTIALLY TO CREATE STRUCTURED DATA ELEMENTS THAT WE CAN ACTUALLY THEN USE MORE EFFECTIVELY. THE OTHER THING THAT OER DOES IS TO HELP CREATE THE TOOLS NEEDED TO QUERY AND INVESTIGATE AND SYNTHESIZE THESE VARIOUS DATA SOURCES. AND THEN WE TRY TO FACILITATE TO THE BEST OF OUR ABILITIES IN TERMS OF MAKING SOME OF THAT INFORMATION ALSO TRANSPARENT. HERE OUR LEAD ESSENTIALLY ON ALL OF THIS IS CALLED THE REPORTER SYSTEM. HOPEFULLY YOU'VE SEEN THE NIH WEBSITE AND VISITED THIS. IT'S GREAT TOOL TO BE ABLE TO GET ACCESS TO UNDERSTAND WHAT'S IN THE NIH PORTFOLIO. WE ALSO CONDUCT THE RCDC SYSTEM FOR BETTER FOR WORSE, I REALIZE IT'S A CONTROVERSIAL TOPIC, BUT THERE IT IS. IN TERMS OF CATEGORIZING OUR RESEARCH PORTFOLIO ALONG DIFFERENT BUDGET ORIENTED CATEGORIES. SOMETHING NEW THAT'S COME DOWN THE ROAD WHICH I DO BELIEVE THAT THE SUBGROUP HAS HEARD ABOUT ALREADY IS SCIENCV. AN ATTEMPT BY NIH AS WELL AS WORKING WITH FEDERAL PARTNERS TO AN ELECTRONIC PLATFORM FOR RESEARCHERS WORLDWIDE TO USE FOR STORING THEIR BIBLIOGRAPHIC INFORMATION AND THEIR CV KINDS OF INFORMATION, SUCH THAT IT WILL BE POSITIONED WELL TO FEED INTO ELECTRONIC APPLICATION FORMS IN AN ATTEMPT TO REDUCE AND EASE BURDEN ON FOLKS AS WELL AS HOPEFULLY HAVE THE DATA STRUCTURED IN A WAY THAT MAKES IT MORE ACCESSIBLE. AND THEN, OF COURSE, THE COMMONS, WHICH THOSE OF YOU WHO COMPETE IN THE RESEARCH GRANT PROCESS ARE WELL AWARE OF, IT'S THE INTERFACE THAT NIH USES WITH THE GRANTEES TO SUBMIT THEIR APPLICATIONS AND TO GAIN INFORMATION ON THEIR APPLICATIONS. IT'S ALSO A SOURCE OF INFORMATION. WE HAVE EMBEDDED WITHIN THE COMMONS, WE ASK QUESTIONS ON WHO ARE YOU, WHAT DO YOU DO, AS WELL AS YOUR BASIC DEMOGRAPHIC INFORMATION THAT YOU CAN CHOOSE TO FILL OUT. SO THAT PROVIDES US ANOTHER WINDOW IN TERMS OF WHO ARE THESE FOLKS THAT WE'RE INTERACTING WITH. SO WE HAVE A LOT OF PRODUCTS, IMMEDIATE PRODUCTS THAT WE CAN GET AHOLD OF, AND ARGUABLY THESE ARE MEASURES OF KNOWLEDGE, AS WE'VE TALKED ABOUT. BUT THEN THE QUESTION THAT YOU ALL ARE STRUGGLING WITH IS HOW TO DO THE VALUE OF THIS KNOWLEDGE. AND IN MY MIND, THE VALUE OF KNOWLEDGE IS WHO'S USING IT. IS IT BEING USED AND WHO IS USING IT. AND IS IT BEING USED ESSENTIALLY BECAUSE OF THE MISSION OF OUR ORGANIZATION TO INFORM AND IMPROVE HEALTH. AS HAS ALREADY BEEN MENTIONED, THIS HAS BEEN INCREDIBLY CHALLENGING. THE NIH PORTFOLIO IN A WHOLE IS INCREDIBLY DIVERSE. IT'S DIVERSE IN TERMS OF TOPICS WITH ALL OF THE DIFFERENT INSTITUTES, EACH WITH DIFFERENT MISSIONS AND ORIENTATIONS. IT'S ALSO DIVERSE IN TERMS OF APPROACHES AND THAT WAS MENTIONED EARLIER IN TERMS OF CLINICAL, TRANSLATIONAL, BASIC EPIDEMIOLOGICAL, WE CAN GO ON AND ON FOR DAYS TALKING ABOUT ALL THE DIFFERENT KINDS OF APPROACHES THAT WE SUPPORT AS WELL. SO THIS ALL BECOMES RATHER COMPLEX THEN, TRY TO TAKE THIS FULL BASKET OF ISSUES, IDEAS, ET CETERA, AND FIGURE OUT HOW ONE GOES ABOUT ASSESSING THE VALUE. SO THIS IS A LITTLE SCHEMA THAT I WORKED ON TO TRY TO HELP, I THINK FIGURATIVELY, I THINK IF PICTURES SO BEAR WITH ME IN TERMS OF MY LITTLE PICTURES THAT I'VE OFFERED TO YOU. WE HAVE ESSENTIALLY NIH AND WE SUPPORT THE RESEARCH THAT LEADS TO THE PUBLICATION, THE PATENTS, THE TECHNOLOGIES, ET CETERA, ALL THE THINGS I JUST TALKED ABOUT. AND THEN WE ALSO HAVE THE RIGHT-HAND SIDE OF THE SCREEN WHAT ARE OFTEN MAJOR INDICATORS WITH REGARD TO PUBLIC HEALTH, IN TERMS OF MORTALITY, PREVALENCE, QUALITY OF LIFE, PRACTICE ISSUES, INSURANCE COVERAGE, YOU CAN GO ON AND AS YOU HAVE ALREADY IN YOUR WORK IN TERMS OF THE FULL DIAGRAM THAT GAIL SHOWED EARLIER IN TERMS OF WHAT SOME OF THESE LONGER RANGE OUTCOMES ARE. AND IN BETWEEN, WHICH UNFORTUNATELY YOU CAN'T SEE BECAUSE THEY'RE TOO LIGHT COLORED, THERE ARE LOTS OF LITTLE BUBBLES BECAUSE SOMEHOW, WE BELIEVE THAT THIS INFORMATION AFFECTS THAT AT THE END OF THE DAY. WITH A WHOLE LOT OF GAPS IN BETWEEN. AND IT'S GAPS NOT ONLY IN TERMS OF WHAT'S INSIDE OF THERE BUT IT'S ALSO GAPS AS WE'VE TALKED ABOUT BEFORE IN TERMS OF TIME. SO RIGHT NOW WE ARE REALLY BUSY HERE AT NIH. WE'RE SUPPORTING ALL THIS WONDERFUL SCIENCE, THESE PUBLICATIONS ARE COMING OUT, THESE TECHNOLOGIES, THESE PATENTS, THAT'S ALL RIGHT THERE, AND IT'S NOT GOING TO BE FOR A NUMBER OF YEARS AS WE'VE ALREADY TALKED ABOUT THAT WE'RE GOING TO BE ABLE TO SEE INFLUENCES ON THIS SIDE OF THE SLIDE. THAT'S VERY CHALLENGING WHEN PEOPLE WHO ARE PROVIDING YOU FUNDS SAYING WHAT ARE YOU GETTING ME NOW, WHAT ARE YOU BUYING ME NOW, HOW IS IT AFFECTING THE CONSTITUENTS IN MY DISTRICT. SO WHAT -- IN THINKING THIS THROUGH, SO THIS IS SORT OF LIKE OUR TRADITIONAL -- THIS IS NIH IN TERMS OF SUPPORTING RESEARCH. WHAT I DID WAS THINK OF A GIVEN KIND OF TECHNOLOGY ESSENTIALLY THAT CAN BE USED FOR DIAGNOSTIC PURPOSES. SO IN THIS CASE, IT'S 3D IMAGING THAT'S USED FOR UNDERSTANDING HEART DISEASE. IT'S A TECHNOLOGY THAT'S CURRENTLY AVAILABLE, SO THAT'S WHAT I'M -- MY LITTLE WORKING EXAMPLE IS GOING TO BE ORIENTED AROUND THAT PARTICULAR KIND OF TECHNOLOGY. SO WE HAVE AND WE KNOW, THERE WAS A GREAT DEAL OF RESEARCH THAT LED TO THE DEVELOPMENT OF THAT PARTICULAR TECHNOLOGY. WE CAN CLASSIFY THIS AS A SIMPLISTIC CLASSIFICATION OF THAT KIND OF RESEARCH. WE HAVE THE BASIC RESEARCH, THERE'S CLINICAL RESEARCH BOTH IN TERMS OF UNDERSTANDING CARDIAC DISEASE, BUT ALSO IN TERMS OF DEVELOPING THE TECHNOLOGY, THE 3D KIND OF TECHNOLOGY. THEN YOU HAVE THE DIAGNOSTIC KIND OF TESTING AND SO FORTH, IS THIS REALLY GOING TO BE AVAILABLE TO BE USING. SO IMMEDIATELY, WE GO TO OUR MEDIA PRODUCT. WE HAVE PUBLICATIONS, WE HAVE THE PUBLICATIONS, AND THEN THE BOTTOM SCHEMA WHICH IS A LITTLE HARD TO SEE AND I APOLOGIZE IS SORT OF A LITTLE SCHEMA FOR TECHNOLOGY. SO WE KNOW THOSE THINGS ARE PROBABLY PRODUCED AND UNDOUBTEDLY, THERE ARE PATENTS THAT ARE BEING RECORDED AND LICENSES THAT ARE BEING RECORDED RECORDED. AND THAT PRODUCED ESSENTIALLY THE FINAL -- THE 3D IMAGING TECHNOLOGY. WE CAN, WITH THE TOOLS THAT WE CURRENTLY HAVE AVAILABLE TO US, WE COULD START MAPPING THIS OUT. THIS WOULD NOT BE OVERLY DIFFICULT TO START MAPPING OUT IN TERMS OF WHAT WERE THE VARIOUS PIECES OF RESEARCH THAT ARE CITED WITHIN THE PATENT AND THE TECHNOLOGY TO PRODUCE THIS PARTICULAR DIAGNOSTIC TOOL. SO WE HAVE A TOOL. NOW IS THE CHALLENGING PART. SO THAT FIRST SLIDE IS SOMETHING THAT I DO THINK WE ARE CAPABLE OF DOING FOR -- IT COULD BE A DIAGNOSTIC, IT COULD BE A DEVELOPMENT OF A NEW THERAPEUTIC TOOL. I WAS LISTENING TO NPR LAST WEEK AND THEY WERE TALKING ABOUT THE NEW PROTON THERAPY, ET CETERA, AND SORT OF WHERE THAT IS IN THE PIPELINE, SO WE CAN DO THAT PART OF IT. THE NEXT PART GETS A BIT CHALLENGING. WHERE DO WE GO FROM THERE. HOW DO WE THEN FINALLY GET OUT TO AFFECTING PUBLIC HEALTH. AND I WOULD ARGUE THAT ONE WAY TO BEGIN TO THINK OF THAT IS TO START THINKING CONCEPTUALLY ABOUT WHERE WOULD I EXPECT TO SEE AN EFFECT? IF THIS TECHNOLOGY IS GOING TO TAKE OFF, WHERE WOULD I EXPECT TO SEE IT? WELL, ONE WAY I WOULD EXPECT TO SEE IT HAS TO DO WITH POTENTIALLY STARTUP COMPANIES. ARE PEOPLE PICKING UP THE TECHNOLOGY? ARE BUSINESSES SORT OF PICKING IT UP? ARE THERE -- IN THE CASE OF THE PROTON THING I WAS LISTENING TO, ARE THERE FACILITIES BEING BUILT, BECAUSE YOU HAVE TO HAVE VERY SPECIALIZED FACILITIES FOR THE PROTON IMAGING RADIATION TREATMENT, ARE THEY BEING BUILT? IS THE INFRASTRUCTURE HAPPENING? THOSE ARE CAPTURABLE PIECES OF INFORMATION. THE OTHER PART MIGHT BE ARE PEOPLE ACTUALLY BEING TRAINED TO USE IT. SO ARE THERE TRAINING COURSES BEING OFFERED TO USE THIS PARTICULAR TOOL. IS IT CREEPING -- IS IT BECOMING PART OF PRACTICE GUIDELINES, WHICH WILL BE ANOTHER. AND THIS ONE, THOUGH, IS THE ONE THAT I THINK IS PROBABLY THE BIGGEST TICKET WHICH HAS TO DO WITH INSURANCE COVERAGE. IS THIS GOING TO BE SOMETHING THAT THE MEDICATE/MEDICARE SYSTEM AND OTHERS ARE GOING TO BE PICKING UP AND COVERING AS PART OF SERVICE. THESE WOULD BE AREAS THAT COULD BE ASSESSED. THEN YOU HAVE OUT HERE THE MORTALITY INCIDENCE AND QUALITY OF LIFE, AND I WOULD ARGUE THAT GATHERING THAT DATA ON THAT INTERMEADE YEAR GULF BETWEEN -- AND I SUPPORTED AND DEVELOPED THE TECHNOLOGY TO X NUMBER OF YEARS LATER WHERE WE BEGIN TO SEE THE MORTALITY, INCIDENCE AND QUALITY OF LIFE, GATHERING INDICATORS ON THE INTERMEDIARY PLACE PUTS US IN A MUCH FIRMER PLACE. IT ALSO HELPS US IN TERMS OF OUR DISCUSSIONS WITH OTHERS WHO ARE SAYING, WHAT ARE YOU BUYING ME NOW? YOU'D SAY WE'RE ON THE ROAD. WE'RE ON THE ROAD TO THESE LONGER TERM PUBLIC OUTCOMES, BUT WE NEED TO GET INTO THE MIDDLE PART TOO, WE NEED TO START ASSESSING THOSE MIDDLE PIECES. SO THIS THEN THE FINAL CHAIN, IF YOU WILL, AGAIN, OVERLY SIMPLISTIC. I GRANT YOU THAT, IT IS, BUT IT'S JUST A TOOL ESSENTIALLY TO START US SORT OF THINKING THROUGH ON HOW TO MOVE BEYOND OUR COMFORT ZONE AND TRY TO FIGURE OUT HOW TO DO A BETTER JOB OF DESCRIBING THE ROAD ESSENTIALLY TO THESE LONGER TERM OUTCOMES. PEOPLE DON'T WANT TO WAIT, THEY WANT TO HEAR WHAT YOU'RE BUYING THEM, AND I THINK THESE COULD ALSO BE VERY, VERY INSTRUCTIVE, QUOTE, PURCHASING ELEMENTS IN THE CHAIN. WHAT DOES THAT MEAN IN TERMS OF THE TASK THAT YOU HAVE BEFORE YOU. I WOULD SUBMIT THAT THE ISSUE IS REALLY WHAT YOU'VE ALREADY STARTED, WHICH IS THE VISION AND THE BRAINSTORMING. WHAT'S REALLY IMPORTANT IS TO BE ABLE TO THINK ABOUT WHAT I SAID BEFORE, WHICH IS IF I HAVE A DEVELOPMENT, IN THIS CASE, IT WAS A TECHNOLOGY, AS I SAID, YOU COULD PUT IN A THERAPY, YOU COULD PUT IN OTHER KINDS OF PRODUCTS THERE, WHERE WOULD I EXPECT TO SEE A DIFFERENCE? WHAT DO I ANTICIPATE THAT LOOKING LIKE? ONCE YOU BUILT THE SCAFFOLD OF THE IDEA OF THE CONCEPTUAL FRAMEWORK, THEN YOU CAN TAKE ON THE TASK OF TRYING TO FIGURE OUT, ARE THERE EXISTING DAY A DATABASES ALREADY FORMED OR IS THIS DATA THAT I'M GOING TO HAVE TO FIGURE OUT HOW TO COLLECT. IN WHICH CASE, AGAIN, THAT MIGHT COME BACK POTENTIALLY TO THE NIH TO DO A BETTER JOB OF COLLECTING DIFFERENT KINDS OF PRODUCTS. BUT UNTIL WE HAVE THE CONCEPTUAL FRAMEWORK JUST AS YOU DO IN ANY SCIENCE PROJECT, UNTIL YOU HAVE THE CONCEPTUAL FRAMEWORK, YOU DON'T KNOW WHAT ALL THOSE DATA ELEMENTS ARE THAT YOU NEED TO HAVE. SO I THINK IN TERMS OF FROM WHERE I SIT WHAT CAN BE INCREDIBLY HELPFUL FOR SMRB IS TO BE ABLE TO THINK THROUGH AND HELP NIH TO THINK THROUGH WHAT THAT PROCESS WOULD LOOK LIKE. IT DOESN'T MEAN YOU HAVE TO DEVELOP EACH OF THE CHAINS. YOU CAN HAVE -- AND I APPLAUD THE IDEA OF DEVELOPING EXEMPLARY CHAINS OF EVIDENCE, BUT WHAT ARE THE THINGS THAT NIH SHOULD TAKE INTO CONSIDERATION IN DEVELOPING THESE CHAINS OF EVIDENCE GOING FORWARD SO THAT WE CAN CONTINUE TO DO IT OURSELVES. FOR EXAMPLE, ONE COULD DO THIS PROSPECTIVELY WITH BIG INITIATIVES. WE CURRENTLY HAVE A COUPLE OF BIG INITIATIVES GOING ON OR JUST GETTING LAUNCHED SUCH AS THE NEW BRAIN INITIATIVE. THAT'S A VERY EXCITING PROJECT THAT'S FILLED WITH PROMISE. WHAT WOULD WE EXPECT THAT TO LOOK LIKE? WHERE DO WE THINK THOSE PROMISES ARE GOING TO FULFILL IN TERMS OF HAVING PUBLIC VALUE. NOT ONLY IN TERMS OF THE SCIENTIFIC LITERATURE AND ALSO THE SCIENCE, IT'S VERY IMPORTANT, BUT WHERE DO WE THINK THAT'S GOING TO TAKE US? AND THEN ONCE WE HAVE THAT BUILT OUT, WHAT ARE THE DATABASES THAT WE COULD POTENTIALLY TAP TO DO THAT OR DATA PLATFORMS THAT WE COULD PUT INTO PLACE TO BE ABLE TO IS HAVE THAT. CLEARLY WITH THE EXAMPLE I SHOWED, IT WAS BUILT ON THE PRESUMPTION THAT YOU ARE COLLECTING THE EPIDEMIOLOGY TO KNOW WHAT THE MORTALITY INDICES ARE AND THE PREVALENCE INDICES ARE AND THE QUALITY OF LIFE. THAT WAS SORT OF A GIVEN THAT WE'RE ALREADY SUPPORTING THAT. BUT WE DON'T NECESSARILY HAVE SOME OF THOSE MIDDLE PIECES. THEN TURNING IT BACK, THIS IS WHERE MY OFFICE CAN BE HELPFUL ESSENTIALLY IN TERMS OF DESIGNING THE METHODS TO ACQUIRE THAT RELEVANT DATA, AS WELL AS THE TOOLS FOR INVESTIGATING IT AND DEVELOPING THE MODELS TO ANALYZE IT. SO THAT WAS IT. THAT WAS MY ATTEMPT ESSENTIALLY TO STIMULATE YOUR THOUGHT WITH REGARD TO AT LEAST FROM MY VANTAGE POINT WHAT COULD BE VERY HELPFUL. AND I JUST LEAVE YOU WITH THIS QUOTE BECAUSE I JUST THOUGHT IT WAS REALLY COOL. THE CRUCIAL VARIABLE IN THE PROCESS TURNING KNOWLEDGE INTO VALUE IS CREATIVITY. SO I ASK YOU ALL TO PLEASE BE VERY CREATIVE. THANK YOU. [APPLAUSE] >> DELLA, THAT WAS TERRIFIC. OUR INSTRUCTIONS WERE TO TRY TO WORK THROUGH ALL THREE PANELS BEFORE A REAL OPEN DISCUSSION, BUT WE DO HAVE ONE OF OUR SPEAKERS UNFORTUNATELY COULDN'T JOIN US, SO WE'RE NOT TOTALLY UNDER REALLY TIGHT TIME CONSTRAINTS, SO IF THERE ARE QUESTIONS FOR DELLA AT THIS POINT, WE COULD ENTERTAIN A FEW BEFORE WE KEEP MOVING. >> LET ME JUST ASK YOU TO CLARIFY ONE THING. I THINK THIS CHAIN OF EVIDENCE IS A GREAT IDEA AND YOUR EXAMPLE OF TECHNOLOGY IS JUST TO SHOW THAT TECHNOLOGY IS ONE OF THE OUTPUTS AND POSSIBLY CAN BE RELATED TO OUTCOMES FOR PATIENTS PATIENTS. BUT THERE IS A RISK HERE OF FEEDING THE IMPRESSION THAT THE MEASURE OF BIOMEDICAL RESEARCH IS HOW MUCH WE CAN RUN UP THE EXPENDITURES ON HEALTHCARE. AND ESPECIALLY WITH THE EXAMPLE OF PROTON BEAM THERAPY, WHICH IS EXTREMELY CONTROVERSIAL IN TERMS OF EFFICACY, AND MAYBE SOME OF THE OTHER TECHNOLOGIES, I'M SURE YOU INTEND IT SHOULD BE ACCOMPANIED BY MEASURES OF THE ACTUAL BENEFIT AND HEALTH STATUS QUALITY OF LIFE AND THE OTHER MEASURES THAT WERE IN YOUR SLIDE. THANKS. >> THAT'S QUITE TRUE. AND YOU'RE RIGHT ABOUT THE HEALTHCARE COSTS. AND I THINK THERE COULD BE EXAMPLES POTENTIALLY IF WE CAN GO BACK AND THINK OF OLD TIME PUBLIC HEALTH OF POTENTIAL INTERVENTIONS THAT MAY NOT NECESSARILY DRIVE UP COSTS, BUT THE THING IS TO THINK THAT TRUE AND TO THINK ABOUT WHAT YOU ANTICIPATE THAT TO LOOK LIKE WHEN DEVELOPING THE CHAIN. IN THIS EXAMPLE, WOULD WE EXPECT TO SEE THE COST OF HEALTHCARE TO INCREASE? PROBABLY SO. BUT THEN THAT'S OUTWEIGHED AS YOU SAID BEFORE IN TERMS OF POTENTIAL BENEFITS, IN TERMS OF DIAGNOSING DISEASE EARLIER, WHICH MAY OFFSET THE COURSE OF ILLNESS AND, THEREFORE, ACTUALLY HAVE A FINANCIAL BENEFIT IN THAT PEOPLE CAN WORK LONGER AND HEALTHIER. >> JUST SOMETHING I HOPE THE PANELISTS WILL ADDRESS AT THE END, I ENJOYED YOUR PRESENTATION VERY MUCH. TWO THINGS THAT I JUST WANT TO ADD. ONE IS HOW DO WE DEAL WITH THE LONG TIME FRAME BETWEEN CERTAIN KNOWLEDGE GENERATION AND APPLICATION? I MEAN, SOME OF THEM ARE 20 YEARS AND ENORMOUS VALUE, BUT IF WE ANALYZE THEM TOO SOON, YOU KNOW, WE MAY SAY THEY'RE NOT GOING TO BE WORKING BUT -- AND I CAN GIVE YOU MANY EXAMPLES AS YOU CAN SO I WON'T BOTHER YOU WITH THAT, BUT IT'S A BIG PROBLEM, I THINK, JUST TRYING TO ANALYZE BASIC SCIENCE OR RESEARCH FINDINGS AND ITS APPLICATION. THE SECOND IS JUST TO SAY WHAT GIL SAID, I THINK YOU TOUCHED A TENDER CORD WHEN YOU BROUGHT UP PROTON THERAPY. IT'S ALL OVER THE PLACE AND EVERYONE IS DOING IT WITH MARGINAL VALUE AT THE MOMENT THAT WE KNOW, SO THE SECOND THING IS HOW DO WE RELATE SCIENTIFIC DEVELOPMENT TO COST, WHICH IS KIND OF KEY FOR ALL OF US. SO I HOPE THE PANEL WILL ADDRESS THAT IN THE DISCUSSION. >> ART, YOU ALSO BRING UP AN ISSUE OF TRUST AS WE ASK THE PUBLIC TO TRUST US THAT THERE IS VALUE IN THIS, EVEN IF IT'S 20 YEARS FROM NOW. AND HOW DO WE CONVINCE THEM OF THAT. >> THERE ARE ADVOCATES BY DISEASE OR BY AGE WHO BEG US TO SHOW A GREATER SENSE OF URGENCY, AND THAT'S SOMETHING TO BE CONSIDERED WHEN YOU'RE TRYING TO BUILD THESE PARTNERSHIPS, THESE RELATIONSHIPS, THE TRUST THAT'S MENTIONED. >> OUR NEXT SPEAKER IS JIM ANDERSON, DEPUTY DIRECTOR FOR PROGRAM COORDINATION PLANNING AND STRATEGIC INITIATIVES. JIM. >> THANKS FOR THE OPPORTUNITY TO PARTICIPATE. THIS IS REALLY A VERY IMPORTANT TOPIC FOR US ALL TO THINK ABOUT. WE ARE CALLED TO TASK OFTEN ABOUT WHETHER WE ARE PRODUCING VALUE, SO I WANTED TO ADDRESS AN ISSUE OF EVALUATION OF VALUE. I REALIZE MY COMMENTS TODAY ARE VERY, VERY RELEVANT TO YOUR NEXT CHARGE. I'M GOING TO FOCUS ON RELE THE PROXIMAL OUTPUTS OF RESEARCH AND THAT'S MEASURING SCIENTIFIC KNOWLEDGE IN TERMS OF BIBLY OWE METRICS AND ALSO EXPERT OPINIONS OF OTHER WORKING SCIENTISTS. WHAT I'M GOING TO DO IS GIVE TWO EXAMPLES AND JUST WITH THESE EXAMPLES OFFER A REMINDER THAT THE ASSESSMENT OF VALUE IS ONLY AS GOOD AS THE DATA AND THE METHODS YOU USE TO INTERPRET THE DATA. SO I'LL ADDRESS VALUATION OF VALUE, REQUIRES RIGOR IN ORDER TO PRODUCE FINDINGS THAT ARE INTERPRETABLE AND USEFUL. I'M GOING TO USE THESE TWO EXAMPLES. THE FIRST ONE IS IN MY OPINION A VERY RIGOROUS AND RECENT EVALUATION OF OUR THE OUTCOMES OF OUR DIRECTOR'S PIONEER REWARD. HIGH RISK, HIGH REWARD PROGRAM FROM THE COMMON FUND, WHICH USED SOME NOVEL METHODS OF BILLY OWE METRIC AND EXPERT EVALUATION TO ASSESS WHETHER THIS AWARD TYPE WAS PRODUCING SOMETHING THAT RO1s WASN'T PRODUCING. AND THE SECOND ONE IS AN EXAMPLE, I'LL LEAVE YOU TO DETERMINE WHICH IS BETTER AND WHICH IS WORSE, THIS IS A DEBATE THAT WAS CONDUCTED IN THE PAGES OF NATURE ABOUT FIVE OR SIX MONTHS AGO. WITH A COMMENTARY AND THEN A RESPONSE FROM NIH FROM GEORGE IN OUR OFFICE OF PORTFOLIO ANALYSIS AND DAVID LITTMAN FROM NCBI. SO THE FIRST IS AN OUT COME EVALUATION OF THE NIH PIONEERS DIRECTORS AWARD AND I'LL SOMETIMES CALL IT THE DP1, THAT'S A CODE LIKE AN RO1. THIS WAS AN EVALUATION OF THE OUTCOMES OF THE FOLKS WHO WERE FUNDED FROM 2004 TO 2006. THOSE YEARS CHOSEN BECAUSE THEY HAD GONE THROUGH THEIR FIVE YEARS AND WE COULD COLLECT DATA ON WHAT HAD HAPPENED. THIS WAS DONE BY SOME ECONOMISTS AND STATISTICIANS FROM THE SCIENCE AND TECHNOLOGY POLICY INSTITUTE. THEY DEVELOPED SOME NEW METHODS FROM EVALUATING THE OUTCOMES FROM THIS STUDY. IT HAS NOT BEEN PUBLISHED BUT THE ENTIRE REPORT IS QUITE LENGTHY, QUITE MATHEMATICAL, LOTS OF DATA. IT'S ON THE COMMON FUND SITE, AND I PUT THE REFERENCE THERE, IT'S IN YOUR SLIDE SET. SO WHAT IS THE ORIGIN OF THIS AWARD? WHY DO IT? IT WAS STARTED AS ONE OF THE ORIGINAL ROAD MAP PROJECTS IN 2004, AND IT ADDRESSED A CONCERN IN THE COMMUNITY AND ALSO AT NIH THAT PERHAPS THE RO1 MECHANISM WAS NOT ALLOWING PEOPLE TO TAKE RISKS OR GO IN NEW DIRECTIONS. AND COULD YOU DESIGN AN AWARD THAT WOULD ALLOW PEOPLE, FREE THEM, TO PRODUCE SOMETHING DIFFERENT. AND IT WAS BASED ON THE PREMISE THAT A PERSON-BASED APPLICATION AND GRANT AS OPPOSED TO A PROJECT-BASED WOULD ALLOW YOU TO DO THAT. SO RECOGNIZE PEOPLE WHO HAD PROVEN PAST CREATIVITY AND ENCOURAGE THEM, ALLOW THEM TO GO IN A NEW DIRECTION. SO THIS RESEARCH HAD TO BE, REQUIREMENT WAS IT HAD TO BE A SUBSTANTIAL DEPARTURE FROM WHAT THIS PERSON OR ANYONE ELSE HAD BEEN DOING, SO THIS HAD TO BE A NEW PIONEERING DIRECTION, AND AGAIN, IT WAS REALLY AN EXPERIMENT IN SCIENCE MANAGEMENT. CAN YOU CREATE AN AWARD THAT WOULD JUST LET PEOPLE DO SOMETHING DIFFERENT. SO WHAT ARE THE ELEMENTS OF IT, THOSE OF YOU WHO HAVE WRITTEN RO1s WILL RECOGNIZE THIS AS VERY DIFFERENT. NO PRELIMINARY DATA REQUIRED. THE PROJECT DESCRIPTION IS A FIVE-PAGE ESSAY AS POSED TO THE STANDARD RO 1 APPLICATION. THERE ARE THREE LETTERS OF RECOMMENDATION REQUIRED. THIS IS EVEN FOR SENIOR INVESTIGATORS. SO IT'S ASKING WHAT -- TELL US ABOUT THIS PERSON, CAN THEY DO THIS PROJECT, ARE THEY VISIONARY, REQUIRES A LOT OF EFFORT, AT LEAST 51%. THE BUDGET IS LARGE, BACK THEN THIS WAS TWICE THE AVERAGE RO1, AND 5 YEARS -- SO IT'S ABOUT A YEARLONGER THAN THE AVERAGE RO1s. IT ALSO USED AN UNUSUAL AND NEW REVIEW PROCESS. THERE WAS A STANDARD REVIEW PANEL THAT LOOKED AT THE GRANTS AND THEN THERE WERE INTERVIEWS. WE DO THIS FOR ALL OF THE HIGH RISK, HIGH REWARD PROGRAMS. SO PEOPLE WERE BROUGHT IN, GIVEN A SHORT INTERVIEW ABOUT THEIR PROJECT TO ASK QUESTIONS ABOUT IT. SO VERY DIFFERENT MECHANISM. SO WHAT DID PEOPLE KNOW BACK IN 2004 ABOUT DESIGN OF MECHANISMS AND DIFFERENT OUTCOMES YOU WOULD GET? AND THERE REALLY WASN'T VERY MUCH. THE STUDIES THAT HAVE BEEN DONE ON THE OUTCOMES OF AWARDS AND MECHANISMS WERE ALL ANECDOTAL. THEY WERE NOT QUANTITATIVE, AND THEY DIDN'T HAVE CONTROL GROUPS. COMPARE THIS MECHANISM TO THAT MECHANISM AND SEE WHAT WE GET. SO THIS IS TOO SMALL TO READ BUT I OFFER IT IN CASE YOU'RE INTERESTED. IT HAS LINKS TO WEBSITES OF EVALUATIONS ABOUT THE LAST DECADE. THESE ARE EXCELLENT, VERY THOUGHTFUL, DETAILED, BUT THEY'RE FOR THE MOST PART ANECDOTAL. WE THINK THIS WAS THE OUTCOME OF OUR MARQUEE INVESTMENT OR DORIS DUKE OR THE WELCOME TRUST. THE ONLY REALLY IN DEPTH QUANTITATIVE ANALYSIS IS THE TOP ONE THAT COMES FROM THE SLOAN SCHOOL OF ECONOMICS AT MIT THAT LOOKS, AGAIN, RO1s, THE HOWARD HUGHES INVESTMENTS. SO I WOULD RECOMMEND THAT TOP ONE TO YOU ALSO AS THE SECOND SERIOUS ANALYSIS OF OUTCOMES. BUT IF YOU LOOKED AT ALL THESE, WHAT DID PEOPLE CONCLUDE? THEY CONCLUDED THAT YOU GET MORE INNOVATIVE RESEARCH IF THE APPLICATIONS WERE SHORTER, LESS PRELIMINARY DATA, BASED ON THE PERSON'S TRACK RECORD. THERE WAS FEEDBACK. HOWARD HUGHES GIVES EXTENSIVE FEEDBACK. THE GRANTS PROVIDE MORE SUPPORT FOR A LONGER PERIOD AND ALLOW GREAT FLEXIBILITY. SO THIS DIFFERS FROM AN RO1. DIFFERENT MECHANISM. BASED ON THIS COMMUNITY THINKING, THERE HAS BEEN A CHANGE IN THE WAY MANY AGENCIES OFFER THEIR GRANTS. AS AN EXAMPLE, WELCOME TRUST SINCE 2010 HAS BEEN USING ALL PERSON-BASED MECHANISMS AS OPPOSED TO STRICTLY PROJECT-BASED. AND I'D SAY IT'S ANECDOTAL THAT YOU GET SOMETHING DIFFERENT, IT'S JUST AN ASSUMPTION THAT THAT WOULD HAPPEN, SO THAT'S WHAT THIS ANALYSIS WAS ABOUT. SO WE COMPARED OR STIPI COMPARED THE OUTCOME OF THE PIONEER PROJECTS OVER THOSE THREE YEARS TO SEVERAL CONTROL GROUPS. I'LL SAY THERE WAS 39 PEOPLE FUNDED OVER THOSE THREE YEARS. THE INVESTMENT WAS ABOUT $85 MILLION. AND SO WE CHOSE SEVERAL CONTROL GROUPS TO LOOK AT THE OUTCOMES. FIRST WAS MATCHED RO1s. SO THEY WENT BACK TO THE GENERAL PORTFOLIO OF NIH AND ASKED FOR RO1s FROM SIMILAR-LOOKING P.I.s, PEOPLE FROM THE SAME INSTITUTION, CAREER STAGE, PEOPLE FROM -- WHO HAVE SIMILAR EARLY STAGE AWARDS, SO MATCHED FOR THE PHENOTYPE OF THE PERSON AND THEN ALSO MATCHED FOR THE TOPIC OF THE SCIENCE. SO THAT'S THE CLOSEST YOU COULD GET FOR MATCHING THE DP1 MECHANISM VERSUS AN RO1 MECHANISM AND THAT' ASK THE SAME KIND OF PERSON, WOULD THEY GET THE SAME OUTPUT WITH THE TWO DIFFERENT MECHANISMS. THE PROBLEM IS THE DP1s OFFERED TWICE AS MUCH MONEY, SO IN ORDER TO CONTROL FOR THE DOLLARS SPENT, THEY ALSO COMPARED THE OUTCOMES OF THE DP1s TO 30 RANDOM POOLS OF RO1s THAT ADDED UP TO THE SAME AMOUNT THAT WAS INVESTED IN THE PIONEERS. SO THIS HAS THE PROBLEM THAT THERE'S TWO OR THREE TIMES AS MANY GRANTS IN EACH OF THE POOLS, BUT IT'S A MATCH DOLLAR TO DOLLARS. NOT PERFECT, BUT IT'S TRYING TO ACCOMMODATE THE FACT THAT YOU DON'T HAVE AN IDEAL CONTROL GROUP. AND INEVITABLY, WE WERE ASKED TO LOOK AT THE HOWARD HUGHES OUTCOMES. TEASE ARE VERY DIFFERENT, THEY ALLOW A PERSON TO CONTINUE THEIR TRAJECTORY. I'M NOT MAKING EXCUSES AS YOU'LL SEE. IT ALLOWS A PERSON TO KEEP THEIR SAME TRAJECTORY IN RESEARCH RATHER THAN GO IN A DIFFERENT DIRECTION AND THAT MIGHT INFLUENCE THE OUTCOME OVER FIVE, SIX, SEVEN YEARS. IT'S ALSO MOR MORE MONEY. SO HERE ARE THE METRICS THAT WERE USED. FIRST OF ALL THERE WERE 20,000 PUBLICATIONS FROM THIS GROUP OVER THIS PERIOD OF TIME. AND THEY LOOKED -- LET'S LOOK AT THE BOTTOM LEFT. THEY LOOKED AT THINGS LIKE THE PUBLICATION, THE NUMBER OF PUBLICATIONS PER GRANT, THE NUMBER OF PUBLICATIONS PER DOLLAR, AND IN TERMS OF CITATIONS, THE CITATIONS PER GRANT, THE CITATIONS PER EACH OF THE PAPERS PRODUCED, THE CITATIONS PER DOLLAR, AND THE H INDEX OF THE COLLECTION OF PAPERS FROM THAT GRANT. SO AS YOU KNOW, H INDEX MEASURES BOTH THE NUMBER AND THE CITATIONS OF THE PAPERS AND AN H INDEX OF 20 MEANS THERE'S 20 PAPERS THAT ARE CITED AT LEAST 20 TIMES. THEY ALSO LOOKED AT THE JOURNAL IMPACT FACTOR, WHICH AS WE KNOW IS A TERRIBLY FLAWED MEASURE BECAUSE 80% OF THE CITATIONS COME FROM 20% OF THE PAPERS. THERE ARE PAPERS IN SCIENCE AND NATURE THAT ARE NEVER CITED. SO YOU CAN ONLY USE -- A MEAN IS ONLY USEFUL IF YOU'RE LOOKING AT A GAUSIAN DISTRIBUTION AND THE PROBLEM WITH IMPACT FACT CITATIONS IN THE JOURNALS IS THEY GROW SLEELY SLOWLY, AND MOST OF THE CITATIONS ARE DUE TO JUST A FEW PAPERS. NEVERTHELESS, THIS WAS AN INDICATION MORE OF THE ASPIRATION. WHERE DID THE PERSON THINK THEIR WORK SHOULD BE PUBLISHED. THEN THE GROUP OF THE TOP JOURNALS. WHAT THEY DID, VERY EFFORT-INTENSIVE WAS TO COMPARE THE BIB BIBLIOGRAPHICALLY OWE METRIC OUTPUT WITH ACTUAL EXPERT REVIEWS, SO THEY FOUND 94 EXPERTS IN THE TOPICS OF THE RESEARCH WHO SPENT A VERY LONG TIME LOOKING AT THE PAPERS, IT WASN'T QUITE BLINDED BUT THE ACKNOWLEDGMENT SECTION WAS REMOVED AND THEY ASKED WHAT WAS THE IMPACT AND INNOVATIVENESS OF THIS RESEARCH? BY IMPACT, THEY MEANT AS AN EXPERT IN THIS FIELD, CAN I ACKNOWLEDGE THAT IT CHANGED THE FIELD? I'LL GIVE YOU ONE EXAMPLE OF THE FIRST CLASS OF PIONEERS WAS CARL AT STANFORD IN OPT TOE GENETICS, WHICH HAS EXPLODED AS A NEW TECHNOLOGY THAT'S CHANGED FIELDS. FIELDS. SO THAT'S AN EXAMPLE OF IMPACT. IN TERMS OF INNOVATIVENESS, WAS IT NEW, DID IT BRING TOGETHER NEW DISCIPLINES THAT WEREN'T WORKING TOGETHER. DID IT ACCOMPLISH OR ACHIEVE A METHOD THAT PEOPLE THOUGHT WAS IMPOSSIBLE. SO THERE WERE LISTS OF CRITERIA FOR IMPACT AND INNOVATION THAT THEY WERE GIVEN AND THEY WERE ASKED TO ASSESS ALL THE INDIVIDUAL PAPERS PRODUCED AN THEN WHAT WERE DETERMINED TO BE THE TOP FIVE AS A PACKAGE. SO LOTS OF QUANTITATIVE APPROACHES TO THIS. AS I SAY, THE REPORT IS FULL OF A LOT OF DATA. I WILL GIVE YOU A FEW EXAMPLES REALLY SO THAT IT CAN BE COMPARED TO MY SECOND EXAMPLE IN TERMS OF RIGOR. SO FIRST OF ALL, ANOTHER CONTROL GROUP, I WILL NOT TALK ABOUT THE DOLLAR MATCHED EXCEPT TO MAKE THE STATEMENT AT THE END. THEY ALSO COMPARED WITH THE FINALISTS WHO DIDN'T RECEIVE FUNDING FOR THE PIONEERS. SO SEVERAL DIFFERENT CONTROL GROUPS. THE DP1 PUBLICATIONS APPEAR IN JOURNALS WITH A HIGHER IMPACT FACTOR AND AWARDS HAVE A HIGHER H INDEX RATING. SO HERE ARE THE PIONEERS, HERE ARE THE MATCHED RO1s AND THE MEANS AND 75% CAPTURED WITHIN THE BOX AND YOU CAN SEE THAT THERE IS SOMEWHAT HIGHER STATISTICALLY SIGNIFICANT HIGHER IMPACT FACTOR OF THE JOURNALS WERE THE PIONEERS PUBLISHED, AND THE H INDEX, MEANING THE CITATIONS ATTRIBUTED TO ALL OF THEIR PUBLICATIONS, WAS HIGHER COMPARED TO THE RO1s. THEN IF WE LOOKED AT THE EXPERT ASSESSMENTS OF THIS, THE DP1 RESEARCH WAS ASSESSED AS HAVING MORE IMPACT AND INNOVATION. I WON'T GO THROUGH THE NUMERIC SCALES THAT THEY HAVE FOR THAT BUT JUST TO ACKNOWLEDGE THAT THE PIONEERS WERE JUDGED TO BE MORE IMPACTFUL IN THEIR PUBLICATIONS AND INNOVATIVE IN THEIR PUBLICATIONS COMPARED TO THE RO1s. THEN AGAIN, I'M GOING TO SKIP TWO OF THE CONTROL GROUPS AND JUST GO TO THE HOWARD HUGHES GROUP. AGAIN WE'RE LOOKING AT THE PIONEERS, THE MATCHED AND FEEN TYPICALLY MATCHED RO1s AND THEN THE HOWARD HUGHES GROUP. YOU NOTICE THE HOWARD HUGHES FOLKS PUBLISHED MORE PAPERS, AND THEY ALSO HAVE MORE CITATIONS TO THEIR PAPERS THAN THESE TWO GROUPS. BUT THEN WE HAVE TO CONSIDER THE FUNDING. SO THAT COULD BE ONE CONFOUNDING FACTOR. SO IT TURNS OUT THAT THE PIONEERS, IF YOU LOOK AT THEIR PIONEER FUNDING PLUS ANY OTHER NIH FUNDING, THE AVERAGE WAS ABOUT 600,000 A YEAR IN DIRECT. IF YOU LOOK AT HOWARD HUGHES INVESTIGATOR COHORT, WHICH I'M SORE EE WASORRY, WAS THE CLASS OF 2005, WHICH TONS OUT TO BE 30 PEOPLE SO IT'S VERY CLOSER TO THE PIONEER GROUP, THEY HAD A TOTAL OF 960,000, AND IT'S ALSO NOT ATTRIBUTED TO A SINGLE PRAJ BECAUSE THEY CAN BE DOING SEVERAL DIFFERENT PROJECTS, BUT THEY HAVE MORE FUNDING FROM HUGHES PLUS NIH. IMPORTANT POINT, THE HUGHES FOLKS HAVE NIH FUNDING, WHICH BRINGS OUR TOTAL POOL UP. SO IF YOU NORMALIZE FOR THAT, IT'S REALLY THE SAME. THE NUMBER OF PUBLICATIONS AND THE CITATIONS PER DOLLAR IS REALLY THE SAME WHEN YOU NORMALIZE FOR DOLLARS. SO HOW DID THE EXPERTS ASSESS THE DIFFERENCE? AGAIN, BETWEEN THE PIONEERS, THE MATCHED RO1s AND THE HOWARD HUGHES FOLKS, AND EXPERTS ASSESSED THE IMPACT AND INNOVATIVENESS OF THE PIONEERS AND HUGHES FOLKS AS ABOUT THE SAME. SO AGAIN VERY LONG REPORT. I JUST GAVE YOU A FLAVOR FOR THE DATA PRODUCED HERE BIBLY OWE METRICALLY. SOME OF THE CONCLUSIONS WERE THAT THE PIONEER GRANT SCORED SIMILAR OR HIGHER ON THE MATCHED RO1s ON ALL THE METRICS USED. AND THE PIONEER PORTFOLIO PRODUCED MORE CITATIONS PER PUBLICATION AND THE PUBLICATIONS WERE IN HIGHER IMPACT JOURNALS THAN THE 30 RANDOMLY SELECTED PORTFOLIOS AT THE SAME DOLLAR AMOUNT, SO YOU'RE GETTING MORE FOR YOUR DOLLAR, MORE FOR THE MECHANISM. AND COMPARED TO THE HOWARD HUGHES GROUP, IT WAS FAIRLY SIMILAR OUTPUT WHEN YOU NORMALIZE FOR THE AMOUNT OF MONEY THAT WAS AVAILABLE. SO I WOULD JUST CONCLUDE THAT THIS IS AN EXAMPLE OF A RIGOROUS APPROACH TO ASSESSING THE NEAR TERM PROXIMAL SCIENTIFIC KNOWLEDGE OUTPUT USING BIBLIOMETRICS AND EXPERTS. IT'S VERY DETAILED, WE HOPE IT'S USEFUL AND WOULD CONVINCE PEOPLE THAT THIS IS A MECHANISM THAT GETS YOU SOMETHING THAT AN RO1 DOESN'T. IT'S NOT TO SAY THAT THERE'S NOT VALUE TO RO1-BASED RESEARCH, BUT IF YOU WANT PEOPLE TO GO IN A DIFFERENT DIRECTION AND TAKE A RISK, WE CAN QUANTITATIVELY, I THINK, DEMONSTRATE THAT THIS WILL DO IT. OKAY. I WANT TO CONTRAST THIS WITH AN EXAMPLE. OF A COMMENTARY THAT APPEARED IN NATURE AT THE END OF LAST YEAR TITLED RESEARCH GRANTS: CONFORM AND BE FUNDED. THIS STARTED WITH THE PREMISE THAT NIH PEER REVIEW ENCOURAGES CONFORMITY IF NOT MEADE Y MEADE YO KRI TEE AND MAY IGNORE TRULY INNOVATIVE THINKERS. THE GOAL WAS TO EXPLORE THE LINK BETWEEN HIGHLY CITED RESEARCH AND NIH FUNDING. THE CONCLUSIONS THAT THEY REACHED, THEY REACHED MANY, A MAJOR ONE WAS THAT THE AUTHORS OF THE MOST INFLUENTIAL AND INNOVATIVE PAPERS DO NOT RECEIVE NIH FUNDING, NOR DO THEY SERVE ON STUDY SECTIONS. SO I THINK THIS IS -- WHAT I'M PREND PRENTING TODAY I THINK IS ALSO RELEVANT TO YOUR NEXT CHARGE. AND BY INFOR INSTANCE INFERENCE, NIH PEER REVIEWERS ARE NOT INNOVATIVE AND FAIL TO SUPPORT INNOVATIVE SCIENCE. SO HOW DID THIS GROUP ARRIVE AT THIS CONCLUSION? FIRST OF ALL, THEY -- THROUGH THE SCOPUS DATABASE, THEY IDENTIFIED OVER 20 MILLION PAPERS THAT HAVE BEEN PUBLISHED BETWEEN 2001 AND 2012. THEY IDENTIFIED 700 OF WHICH WERE CITED AT LEAST A THOUSAND TIMES. SO THAT'S WHAT THEY CALLED HIGHLY CITED AND INFLUENTIAL. THE COMMUNITY RECOGNIZES THAT AS SOMETHING OF VALUE TO THEM BECAUSE THEY CITE IT IN THEIR WORK. THAT'S ONE INTERPRETATION. AS OF APRIL 2012. AND THESE WERE ALL IN THE LIFE SCIENCES OR HEALTH SCIENCES, AND THEY HAD AN AUTHOR AFFILIATION IN THE U.S. SO GOOD CHANCE THESE MIGHT BE NIH INVESTIGATORS OR NIH FUNDED. OF THESE 700 PAPERS, THEY HAD A TOTAL OF 1,172 PRIMARY AUTHORS. BY THAT THEY MEANT EITHER THE FIRST OR THE LAST AUTHOR. THEN THEY WENT TO, AS DELLA POINTED OUT,NIH REPORTER, ANYONE CAN GO THERE AND LOOK UP THESE AUTHORS AND ASK, DO THEY RECEIVE NIH FUNDING. SO THAT'S THE DATA APPROACH, ANALYSIS APPROACH THEY USED OH REACH THE CONCLUSIONS THAT I GAVE YOU. THEIR FINDINGS WERE THAT ONLY 40% OF THE PRIMARY AUTHORS HAVE NIH FUNDING -- 94% ARE NOT CURRENTLY SERVING ON STUDY SESSIONS. THAT STATIF YOU LOOK AT ALL THE MEMBERS SERVING ON STUDY SECTIONS IN THAT YEAR, ONLY .8% OF THEM WERE AMONG THESE HIGHLY CITED PRIMARY AUTHORS. THAT ALSO LOOKS LIKE WE'RE NOT ENCOURAGING THE MOTOR CITED AUTHORS, INFLUENTIAL AUTHORS. TO SERVE ON STUDY SECTIONS. APPARENTLY NIH IS NOT FUNDING THE MOST INFLUENTIAL BIOMEDICAL RESEARCH. END OF STORY. TURNS OUT THIS IS A DEEPLY FLAWED STUDY. THIS WAS REBUTTED BY GEORGE AND DAVID IN A FEW WEEKS IN "NATURE," AND SOME OF THE THINGS THAT THEY FOUND WHEN THEY ACTUALLY LOOKED AT THE DATA WERE THAT A MAJORITY OF THE PRIMARY AUTHORS NEVER RECEIVED NIH FUNDING BUT THEY ALSO NEVER APPLIED. SO IF YOU LOOK ON THE BOTTOM LINE, WE'LL PUT TOGETHER THE FIRST AND LAST -- LOOK AT ALL, NEVER APPLIED FOR NIH FUNDING, 60%, AND DID NOT APPLY DURING FISCAL YEAR 11 OR 12 IN ALMOST 90%. SO DON'T KNOW WHY, FIRST AUTHORS ARE TYPICALLY TRAINEES, MAYBE THEY HADN'T REACHED A POINT IN THEIR CAREER WHERE THEY WOULD BE A P.I. PEOPLE COULD HAVE RETIRED OR THESE COULD REPRESENT PUBLICATIONS AND AUTHORS FROM INDUSTRY. SO THEY WEREN'T APPLYING FOR NIH FUNDING. BUT IT'S SIMPLY WRONG THAT THESE WERE NIH FUNDED -- OR APPLIED FOR NIH FUNDING. YOU CAN'T GET THE MONEY IF YOU DON'T APPLY. THEN IF WE LOOK, WHAT THE GROUP DID WAS TO USE A SUBSET OF THEIR 700 PAPERS BECAUSE IT WAS QUITE LABORIOUS TO MAKE THE CONCLUSIONS THAT I SHOWED YOU. SO GEORGE AND DAVID LOOKED ACTUALLY AT THE PAPERS, AND UNFORTUNATELY FOUND THAT 60 -- MORE THAN 60% REPRESENT -- OF THIS SUBSET THAT THEY HAD PULLED OUT OF HIGHLY CITED ARTICLES DID NOT ACTUALLY INCLUDE ARTICLES, PUBLICATIONS, THAT ARE IN WHAT YOU WOULD CALL PRIMARY RESEARCH SPACE. SO FOR EXAMPLE, 11% WERE ACTUALLY NOT EVEN BIOMEDICAL SCIENCE. 22% WERE REVIEW ARTICLES. NOW THAT'S TRUE, PEOPLE CITE REVIEW ARTICLES. THEY CAN HAVE AN INFLUENCE ON THE FIELD BUT THEY'RE NOT PRIMARY HYPOTHESIS TESTING AND THEY DON'T DRIVE NEW RESEARCH IDEAS. 6.3% WERE JUST REPORTS. CANCER STATISTICS. YEAH, THAT'S VERY IMPORTANT. PEOPLE WANT TO CITE THAT, THEY NEED TO, BUT IT'S NOT INNOVATIVE RESEARCH. CLINICAL TRIALS, 18%, METHODS, EPIDEMIOLOGY, CLINICAL GUIDELINES. SO IT TURNS OUT OF THE PAPERS USED TO REACH THE CONCLUSIONS BY THESE AUTHORS, ONLY 31% WERE ACTUALLY BIOMEDICAL ARTICLES. AND IF YOU DRILL IN AND ASKED WHO WROTE THESE, 83% OF THESE WERE ACTUALLY FUNDED BY NIH, AND THE OTHER 17% WERE BY INDUSTRY. SO I WANT TO REST MY CASE WITH THAT. WHEN WE SET OUT TO MEASURE THE VALUE OF THE OUTPUT OF WHAT WE INVEST IN, REALLY HAVE TO USE GOOD DATA OR GOOD METHODS OR WE'RE NOT GOING TO COME UP WITH INTERPRETABLE AND USEFUL INTERPRETATIONS. SO I'LL STOP WITH THAT. >> THANK YOU, JIM. TWO VERY TELLING AND INFORM TIM INFORMATIV E EXAMPLES OF QUESTIONS FOR JIM? >> JUST A SHORT QUESTION WHICH AGAIN I HOPE YOU'LL DISCUSS IN THE DISCUSSION AT THE END. THERE'S BEEN AN EDITORIAL IN "SCIENCE" AND A REPORT RECENTLY CRITICIZING THE USE OF H FACTORS IN CITATIONS, AND A WHOLE LOT OF SOCIETIES, I THINK DR. ALBERT WROTE ABOUT IT, HAVE SAID WE SHOULDN'T USE THESE ANYMORE, WE NEED TO USE THEM MUCH LESS BEFORE. IT WAS REALLY VERY INNOVATIVE, BUT ALL THIS EFFORT, CITATION INDEXES, SO ON, ARE WE GOING TO DISARSTART THROWING THEM OUT A AS BEING NON-VALUABLE OR IS IT TIME TO FIND SOME MEDIAN THING WHERE THEY HAVE SOME VALUE AND WE NEED INTERPRETIVE ADVICE. I VALUE YOUR COMMENTS. >> I'LL COMMENT NOW, IS THAT ALL RIGHT? >> GO AHEAD. >> I THINK MOST PEOPLE WOULD AGREE THAT IMPACT FACTORS ARE WORTHLESS. BECAUSE THEY ARE BASED ON A MEAN FROM THE CITATIONS FROM ALL THE PAPERS IN THE JOURNAL AND ONLY A SMALL NUMBER OF PAPERS ACCUMULATE MOST IN CITATIONS, SO IT'S JUST NOT -- IT'S NOT RIGHT. HOWEVER, CITATIONS, I PROPOSE, ARE OF VALUE. BECAUSE IT'S A RECOGNITION OF WHO WANTED TO CITE YOUR WORK. THEY COULD CITE IT BECAUSE IT WAS INCORRECT. BUT SOMEHOW THEY CITED IT BECAUSE THEY WANTED OTHERS TO NOTE THAT AS A REFERENCE TO THEIR OWN WORK. THAT'S OF VALUE. A AND WE ARE ACTUALLY -- LARRY ALLUDED TO SOME OF OUR EVALUATIONS WE'RE DOING IN POUGHKEEPSIE AND WE'RE FINDING GREAT VALUE IN CITATIONS AND EVEN TRENDS OVER TIME IN CITATIONS. >> THAT'S A PRETTY HARD PROBLEM, THE NATIONAL ACADEMY, THE ONE REPORT THAT I CHAIRED WAS ASKED TO LOOK AT THE SEQUENCE OF PAPERS FROM DUKE UNIVERSITY ON CANCER CHEMOTHERAPY, RESPONSIVE OR NONRESPONSIVE MOLECULAR PATTERNS TO GUIDE THE DECISION WHICH DRUGS SHOULD BE GIVEN TO WHICH PATIENTS. THE FIRST FOUR ARTICLES IN THAT SERIES HAD OVER 1,000 CITATIONS. ALL HAVE BEEN RETRACTED. IN FACT, AT LEAST 27 ARTICLES OF THAT SERIES HAVE NOW BEEN RETRACTED. >> THAT'S THE CITATION ALSO. >> ANOTHER THING I'D LIKE TO SAY IS IT'S NOT SIMPLE. YOU CAN SET UP AN ALGORITHM THAT YOU THINK IS GOING TO GIVE YOU AN ANSWER AND YOU'LL SEE THAT ANSWER, BUT THEN YOU HAVE TO SAY IS THAT CORRECT. AS A HUMAN BEING, WHAT IS ANOTHER EXPLANATION? SO ALL OF THESE PROJECTS ARE ITERATIVE. YOU HAVE TO SAY THAT'S WHAT I SEE, NOW WHY, WHAT ARE THE EXPLANATIONS AND I CAN LOOK INTO THAT, AND MY CONCLUSION MAY BE CORRECT OR INCORRECT. SO THAT'S -- THE SECOND EXAMPLE IS THAT. IT'S SORT OF UNINFORMED USE OF BIBLIOMETRICS CAN LEAD TO VERY ERRONEOUS CONCLUSIONS. >> GRIFF, YOU HAD -- >> JIM, I APPRECIATE YOUR DISCUSSION ON THE DP1 AWARDS AND THE DIFFICULTY IN GETTING THE PERFECT MATCH IN TERMS OF CONTROL GROUP, BUT -- AND I KNOW YOU DIDN'T HAVE A CHANCE TO SPEAK ABOUT ALL THE CONTROLS. I'D BE INTERESTED TO KNOW WHETHER ANY SORT OF GENERALIZED IDEA THAT CAME OUT OF THAT GROUP OF INDIVIDUALS WHO AT LEAST MADE IT TO THE INTERVIEWS BUT WERE NOT SUCCESSFUL IN RECEIVING THE AWARD, DID THEY GO ON, FOR EXAMPLE, TO RECEIVE AWARDS, WHICH I SUSPECT THAT THEY WERE FROM THE INDIVIDUAL STANDPOINT AND HOW SUCCESSFUL WERE THEY. THAT WOULD SORT OF UNDERCUT THE SECOND -- OR YOUR DISCUSSION ABOUT THE OVERALL IMPACT AND INNOVATIVENESS THAT STUDY SECTIONS ALLOW PEOPLE IN THEIR CREATIVE SPACE, THAT GROUP, FOR EXAMPLE, ONCE YOU CONTROL FOR THE AMOUNT OF FUNDING THAT THEY GOT, HAD THE SAME TYPE OF IMPACT IMPACT. >> THEY WERE LESS SUCCESSFUL BY ALL THESE MEASURES. THEN AGAIN, THEY HAD LESS MONEY BECAUSE THEY ENDED UP WITH AN RO1. I THINK, THOUGH, IF YOU PUT ALL OF THESE TOGETHER, WHICH -- THERE'S NO SINGLE WAY TO GET AN ANSWER WITH THESE APPROACHES. YOU HAVE TO USE COMPLEMENTING APPROACHES, AND QUESTION THE RESULTS THAT YOU GET. AND WHAT ARE THE OTHER INTERPRETATIONS. BUT I THINK IF YOU PUT THIS ALL TOGETHER, IT'S PRETTY CONVINCING THAT YOU CAN DESIGN A VERY UNUSUAL APPLICATION PROCESS, A REVIEW PROCESS, A FUNDING PROCESS, A LENGTH AND AMOUNT OF MONEY THAT JUST LETS PEOPLE TAKE A RISK, AND YOU GET SOMETHING DIFFERENT WHEN YOU DO THAT. >> I JUST -- I SERVED ON THOSE INTERVIEW PANELS, TWO ROUNDS, AND THE APPLICANTS WERE REQUIRED TO MAKE THE CASE THAT THEY HAD BEEN UNSUCCESSFUL IN GETTING RO1 FUNDING FOR ANYTHING RESEMBLING THE WORK THAT WAS BEING PRESENTED, SO I DON'T KNOW, BUT I WOULD GUESS THAT PROBABLY THE ONES WHO WEREN'T FUNDED AS FINALISTS DIDN'T VERY OFTEN GET RO1 FUNDING IN THE SAME TIME PERIOD. >> GAIL? THEN WE SHOULD WRAP UP. >> JUST A FOLLOW-UP ON GRIFF'S QUESTION, I ACTUALLY HAD THE QUESTION DID YOU CAPTURE THE REASONS THAT THE INDIVIDUALS THAT WERE FINALISTS BUT DIDN'T GET FUNDED, WHAT WERE THEIR REASONS, MAJOR REASONS? WAS THERE ANY COMMON THEME? >> WHY THEY WEREN'T -- >> FOR WHY THEY WEREN'T FUNDED. >> LET ME ASK BETSY. IS THERE ANY CONCLUSION WE COULD DRAW? BETSY IS A DIRECTOR OF THE OFFICE OF STRATEGIC COORDINATION. [INAUDIBLE] >> THERE'S A REASON I THINK THIS IS IMPORTANT, AND MAYBE WE'LL DIVE INTO THIS ALI MORE DEEPLY BECAUSE IT RELATES DIRECTLY TO THE NEW CHARGE OF THIS GROUP. THESE ARE IDENTIFIED OUTSTANDING INVESTIGATORS, THOSE THAT WERE SUCCESSFUL AND THOSE WHO WERE NOT, MAYBE THAT TELLS US A LITTLE BIT ABOUT PROBING OUR WHOLE REVIEW AND EVALUATION AREA. SO MAYBE MORE LATER TO FOLLOW ON THIS. >> NORM? >> I WANTED TO -- JIM, THE MORE I LEARN ABOUT THIS, THE MORE STRUCK I AM BY HOW DIFFICULT IT IS TO -- IT'S REALLY A TRAN YENT PROBLEM, BUT ONE OF THE THINGS THAT WAS EMPHASIZED IN SELECTING INVESTIGATORS OR WHAT PROJECTS WERE TO BE FUNDED, WITH TRACK RECORD, AND IN MY OWN EXPERIENCE, THE BEST PREDICTOR OF FUTURE PERFORMANCE IS PAST PERFORMANCE. WHEN DEALING WITH INDIVIDUALS. BUT HOW DO YOU RECTIFY THAT WITH THE OTHER STUDIES THAT SHOW THAT THE MAJOR SCIENTIFIC BREAKTHROUGHS HAVE OCCURRED BY -- PRODUCED BY PEOPLE 35 YEARS AND YOUNG THEY'RE HAVE NO TRACK RECORDS. HOW DO YOU DEAL WITH THAT? >> I WILL LEAVE THAT TO YOU. [LAUGHTER] >> IT'S A CHALLENGE TO US. I'M TOLD WE ONLY HAVE 35 MORE MINUTES. >> I'LL TAKE TWO MINUTES. I WAS VERY STRUCK BY YOUR DP1 STUDY, AND IT WAS CLEARLY WELL THOUGHT OUT AND AN EXCELLENT EXAMPLE. COULD YOU SHARE WITH US FURTHER THOUGHTS OF WHERE YOU COULD GO WITH THIS KIND OF INNOVATIVE PROCESS? THAT WAS AN EXAMPLE OF ONE APPROACH TO HAVE A PARADOX IN LESS TIME, YOU GET BETTER RESULTS. HAVE YOU THOUGHT THROUGH WHERE YOU GO NEXT? >> WELL, ALL OF THE COMMON FUND SPACE IS AN EXPERIMENT. SO THIS WAS AN EXPERIMENT WITH A MECHANISM TO SEE WHAT WE COULD GET OUT OF DESIGNING IT A DIFFERENT WAY. I FIND THE RESULTS COMPELLING. THERE ARE VERY FEW -- I THINK THERE'S 12 TO 14 PIONEERS A YEAR, SO IT'S A VERY SMALL PROGRAM. ONE QUESTION IS, SHOULD THIS BE GENERALIZED, SHOULD MORE OF THE FUNDING BE PUT INTO PERSON-BASED GRANTS. >> OBVIOUSLY IMPORTANT QUESTIONS THAT WE'LL RETURN TO. OUR LAST PANEL MEMBER IS LAUREL HAAK, WHO IS THE EXECUTIVE DIRECTOR OF ORCID. >> I'M REALLY TORN BECAUSE I'M GIVING A GENERAL OVERVIEW ABOUT WHAT I THINK ABOUT WITH REGARD TO EVALUATION, HOW WE THINK ABOUT METRICS INDICATORS, REALLY WHAT I SHOULD HAVE DONE WAS GIVE SPECIFIC EXAMPLES OF ACTUAL EVALUATIONS THAT I'VE BEEN INVOLVED WITH THAT I THINK WOULD REALLY SHED LIGHT, SO WHAT I'D DO, I'D BE HAPPY TO SHARE WITH YOU SOME OF THE REPORTS THAT ARE OUT THERE. I THINK KIND OF THE LONG AND SHORT OF THIS, I REALLY THINK IT'S POSSIBLE TO DO A REALLY GOOD QUANTITATIVE EVALUATION BUT YOU NEED TO THINK ABOUT IT BEFORE YOU START YOUR PROGRAM. BECAUSE EVERYTHING WE'VE SEEN HERE ARE PEOPLE SAYING, O WE'VE LAUNCHED THIS NEW PROGRAM, AND FIVE YEARS LATER, WE DECIDE TO FIGURE OUT HOW EFFECTIVE IT WAS. I DON'T KNOW, I DON'T THINK THAT'S A REALLY EFFECTIVE WAY TO START IT AND ANY ENGINEER OUT THERE KNOWS IT'S BOUND FOR FAILURE. SO ASSESSING VALUE. YOU'LL NOTICE I STARTED OFF WITH ATTRIBUTING VALUE BECAUSE A LOT OF THE QUESTIONS THAT WE HAVE HERE ARE HOW CAN WE ACTUALLY SHOW THAT NIH IS IN SOME WAY RESPONSIBLE FOR WHAT'S HAPPENING DOWN THE ROAD. IT'S ALL FINE AND GOOD TO SHOW THERE'S CHANGES IN PUBLIC HEALTH BUT HOW MUCH CAN WE ATTRIBUTE BACK TO THE NIH? SO MUCH OF THAT IS DONE BY THE PUBLIC SECTOR. SO I THINK WE REALLY NEED TO THINK ABOUT WHAT CAN WE REALISTICALLY SAY THE NIH HAS DONE, OKAY? THINK ABOUT THAT AND DON'T TRY TO OVERSELL IT. WHAT NIH DOES IS VERY POWERFUL AND VERY IMPORTANT. BUT WE DON'T -- I DON'T THINK NIH CAN CLAIM THAT THEY'VE DONE THE CLINICAL TRIAL NECESSARILY. OR THEY'VE DONE THE PRODUCT DEVELOPMENT NECESSARILY. I DON'T THINK WE SHOULD OVERREACH. ANYWAY, SO ASSESSING VALUE. WE NEED TO FIGURE OUT WHAT SUCCESS IS. WE NEED TO THEN ESTABLISH MEASURES. THIS IS VERY STRAIGHTFORWARD. WE NEED TO CREATE SYSTEMS AND SUPPORTS TO MEASURE. NIH HASN'T REALLY DONE THIS, I WOULD ARGUE. WE NEED TO COLLECT THE DATA AS THE PROGRAM IS GOING ON. NIH DOESN'T DO THIS VERY MUCH. THERE ARE SOME GREAT EXAMPLES, HOWEVER. WE NEED TO ANALYZE PROGRESS TOWARDS SUCCESS AND WE NEED TO MAKE ADJUSTMENTS AND REPEAT IF ANY OF YOU IN HERE ARE SCIENTISTS WILL LOOK AT THIS AND SAY THIS IS WHAT I LEARNED IN SCIENCE 101, THIS IS THE SCIENTIFIC METHOD, THIS WHOLE ROOM IS FULL OF SCIENTISTS BY AND LARGE AND WE DON'T USE THE SCIENTIFIC METHOD TO ASSESS OUR FUNDING PROGRAMS. WHY IS THAT? I DON'T UNDERSTAND. WHY IS THAT? SO WHAT IS FAILURE? ALL THE ENGINEERS IN THE ROOM HAVE SEEN THESE PICTURES. I SHOULD PROBABLY GO TO MY NOTES. IN ALL THE YEARS WORKING IN SCIENCE POLICY AND RESEARCH EVALEVALUATION, WHAT IS SUCCESS IS REALLY DIFFICULT TO ANSWER. QUITE OFTEN, AND I'VE BEEN INVOLVED IN EVALUATION FOR MANY YEARS AT THE NIH AND PRIOR TO THE NIH -- WITH THE NIH WHEN I WAS WORKING AS A CONTRACTOR AND ALSO PRIOR TO THAT WHEN I WORKED FOR THE NATIONAL ACADEMIES. QUITE OFTEN PROGRAMS ARE DEVELOPED WITH NO MEASURABLE GOALS. NONE. AND THEY'RE PUT IN PLACE WITH NO EVALUATION PLAN. AND WE'RE CALLED IN AS CONTRACTORS TO EVALUATE THE PROGRAM, AND YOU KNOW WHAT WE FIRST DO? WE HAVE TO SIT DOWN WITH PROGRAM FOLKS AND SAY, WHY DID YOU DO THIS IN THE FIRST PLACE? AND WE SPEND ABOUT A MONTH COMING UP WITH THE GOALS OF THE PROGRAM, WHICH SHOULD HAVE BEEN THERE TO START OFF WITH. THIS IS THE SAME. I WOULD ARGUE IT'S PUTTING OUT A HYPOTHESIS IN AN EXPERIMENT AND CLAIMING IT AS A FACT WITH NO TESTING. IMAGINE WHAT A WORLD WE WOULD BE LIVING IN IF ENGINEERS DID NOT DO MODELING AND STRESS TESTING BEFORE BUILDING A BRIDGE. WE NEED TO DO THESE KINDS OF THINGS AS THE PROJECT IS GOING ON BEFORE THE PROJECT STARTS. SO WHAT IS SUCCESS? I THINK NIH HAS VERY CLEAR METRICS AND MEASURES FOR SUCCESS. THEY STATE IT ON THEIR HOME PAGE. ANYONE CAN GO THERE AND SAY, WHAT DOES NIH CONSIDER TO BE A SUCCESS? IT'S THERE. RIGHT? EVERYONE SAID IT ALREADY TODAY. FOSTER FUNDAMENTAL CREATIVE DISCOVERIES, INNOVATIVE RESEARCH STRATEGIES, AND THEIR APPLICATIONS. DEVELOP, MAINTAIN AND REVIEW SCIENTIFIC HUMAN AND PHYSICAL RESOURCES. NUMBER THREE, EXPAND THE KNOWLEDGE BASE. NUMBER FOUR, EXEMPLIFY AND PROMOTE THE HIGHEST LEVEL OF SCIENTIFIC INTEGRITY, PUBLIC ACCOUNTABILITY AND SOCIAL RESPONSIBILITY. THIS IS SUCCESS. THIS IS WHAT I WOULD ARGUE THAT NIH SHOULD BE MEASURING. HOW SUCCESSFUL IS NIH IN ACHIEVING ITS MISSION. SO WHAT DO WE MEASURE? WHAT WE JUST TALKED ABOUT, WE NEED TO MEASURE CREATIVE DISCOVERIES. WE NEED TO MEASURE INNOVATIVE RESEARCH STRATEGIES. WE NEED TO MEASURE APPLICATION OF THESE DISCOVERIES AND STRATEGIES. WE NEED TO MEASURE HUMAN AND PHYSICAL RESOURCES, NO ONE HAS TALKED ABOUT THIS VERY MUCH TODAY, KNOWLEDGE BASE IN MEDICAL SCIENCES, AND NO ONE HAS TALKED ABOUT INTEGRITY, ACCOUNTABILITY AND SOCIAL RESPONSIBILITY. WE NEED TO MEASURE ALL THESE THINGS IN NIH. SO WHAT I WANT TO SUGGEST TODAY AND REALLY GET ACROSS IS THE FACT THAT NIH ACTUALLY COLLECTS DATA ON ALL OF THESE AREAS, ALL OF THEM. THE DATA ARE THERE BUT THEY'RE LARGELY UNUSABLE, WHICH IS INCREDIBLY FRUSTRATING TO US THAT WANT TO USE THEM. SO THE DATA SOURCES ARE SILOED, COLLECTED IN NON-STANDARD WAYS, SOME ARE STORED IN DOCUMENT TEXT AND NOT IN SEARCHABLE FIELDS REQUIRING A HUGE AMOUNT OF MANUAL EFFORT. I SEE PEOPLE NODDING THEIR HEADS. FOR EXAMPLE, ONE OF THE ONES THAT I HAVE DESPERATELY WANTED TO USE IS THE H DAT DATABASE. IT'S A COLLECTION OF CLINICAL PRACTICE GUIDELINES. WOW. NOW WOULDN'T IT BE GREAT IF WE COULD UNDERSTAND WHICH NIH RESEARCH STUDIES, DATA SETS, FUNDED PROJECTS, GO INTO THE DEVELOPMENT OF CLINICAL PRACTICE GUIDELINES. WOULDN'T THAT BE FABULOUS? RIGHT? BUT THE PROBLEM IS, WE DON'T COLLECT PERSONS' NAMES, WE DON'T COLLECT WHO SITS ON THE COMMITTEE NECESSARILY, IT'S REALLY HARD TO PARSE THE PUBLICATION INFORMATION IF IT'S THERE. IT ALL HAS TO BE DONE MANUALLY. TRUST ME, I'VE TRIED. THIS WOULD BE HUGE. WHO RUNS THE H DAT DATABASE? HMMM. OKAY. SO NOT ONLY COULD THESE DATA SUPPORT AN ANALYSIS OF THE NUMBER OF SUCH GUIDELINES PRODUCED, STANDARD BEAN COUNTING, BUT ALSO WHAT PAPERS WERE USED TO CREATE THE GUIDELINES AND HOW MANY OF THEM WERE SUPPORTED BY NIH RESEARCH, WHO PARTICIPATED ON THE PANELS AND IF THEY WERE FUNDED BY NIH AT SOME POINT IN THEIR LIFE, BUT THIS EVIDENCE ISN'T AVAILABLE. SO BY MAKING THESE EXISTING DATA SOURCES MORE USABLE IN PARTICULAR FROM AN EVALUATION PERSPECTIVE, THE NIH COULD GO A LONG WAY TO ADDRESSING ITS EVALUATION NEEDS. I'VE GOT A FEW MORE SLIDES. AND HERE IS A LOGIC MODEL. THIS IS ESSENTIALLY THE SCIENTIFIC METHOD APPLIED TO PROGRAM EVALUATIONS. EVERY PROGRAM AT NIH SHOULD HAVE ONE OF THESE, BEFORE FUNDING IS SENT OUT THE DOOR. PERIOD. EVERY PROGRAM. THIS IS MY RECOMMENDATION, I THINK IT'S SOMETHING SMRB SHOULD STRONGLY CONSIDER. IF A PROGRAM DOESN'T HAVE A LOGIC MODEL THAT CLEARLY LAYS OUT WHAT ARE THE PROPOSED INPUTS, SO WHO ARE MY APPLICANTS, WHAT ARE THE PROGRAM CHARACTERISTICS, WHAT ARE THE SPECIFIC ACTIVITIES OF THE PROGRAM, IS IT TRAINING, IS IT TO PERFORM RESEARCH, IS IT WHATEVER, AND THE CONTEXT OF THIS, AND WHAT ARE OUR INTENDED OUTCOMES IN THIS PROGRAM? IF KE DON'T HAVIF WE DON'T HAVE THAT, THE PROGRAM WON'T BE FUNDED. WE DON'T KNOW WHAT SUCCESS IS. EVERYONE IS TALKING ABOUT LOOKING AT IMPACT FACTORS AND CITATIONS AND THOSE ARE GREAT, BUT IN A LOT OF CASES, THEY'RE COMPLETELY IRRELEVANT TO THE PROGRAM. SO HERE THIS IS A LOGIC MODEL FROM A PROJECT I WAS AN AD ADVISOR ON IN MY PREVIOUS LIFE, ONE OF THE FOLKS ON IT IS IN THE AUDIENCE, THIS IS A LOGIC MODEL WE DEVELOPED FOR THE NCI. YOU'RE ABLE TO USE THIS LOGIC MODEL TO THINK ABOUT WHAT SHOULD WE BE MEASURING, WHEN SHOULD WE BE MEASURING IT, HOW DO WE DEVELOP MEASURES THAT ACTUALLY TELL US SOMETHING, NOT JUST METRICS BUT ACTUAL INDICATORS. REALLY, REALLY IMPORTANT. WE'VE DONE SOME WORK, AGAIN IN MY PREVIOUS SLIDES, WORKING WITH THE OFFICE OF EXTRAMURAL RESEARCH ON HOW DO WE ANALYZE THE IMPACT OF DIVERSITY-TARGETED PROGRAMS. YOU KNOW YOU CAN ASK SOMEBODY HERE HOW MANY OF THOSE HAVE ACTUALLY BEEN EVALUATED. YOU KNOW, THEN YOU LOOK AT WHY ISN'T DIVERSITY INCREASING? WELL, I DON'T KNOW. WE'RE THROWING MONEY AT IT. BUT WAIT, HAVEN'T YOU EVALUATED THE PROGRAM? WHY NOT? BECAUSE THEY HAVEN'T THOUGHT ABOUT HOW TO EVALUATE THE PROGRAM IN THE LONG TERM. THERE'S TOUGH QUESTIONS THAT WE'VE GOT TO THINK ABOUT WHAT IS THE EXPERIMENT, HOW DO WE MEASURE IT, IS IT SUCCEEDING, AND IF IT'S NOT SUCCEEDING, TRY AGAIN WITH NEW PARAMETERS. SO WHAT DISCOVERIES HAS NIH FUNDED? SO NIH, AND I'VE GIVEN TALKS ALL OVER THE WORLD ON ALL KINDS OF FUNNY VALUATION TOPICS INCLUDING A NUMBER OF THE THINGS I DID WHILE I WAS AT NIH, AND ONE OF THE BIG THINGS IS THAT, MY GOD, NIH HAS SO MUCH DATA, YOU GUYS ARE SO LUCKY, OKAY? SO NIH IS WAY AHEAD OF THE GAME. PEOPLE ARE JEALOUS OF YOU GUYS. I'M NOT SAYING YOU'RE PERFECT, BUT THERE'S SO MUCH THAT CAN BE DONE HERE. THERE'S MISSING DATA, GAPING HOLES, BUT NIH BENEFITS FROM SOME POLICIES PUT IN PLACE THAT HELP US UNDERSTAND OUTCOMES. AND WHAT I'M TRYING TO GET ACROSS IN THIS TALK TODAY IS LET'S DO MORE OF THIS, LET'S REALLY LEVERAGE WHAT WE'VE GOT. SO IN 1985, THE NATIONAL LIBRARY OF MEDICINE STARTED CAPTURING NIH FUNDING INFORMATION IN PUBLICATIONS. IT'S A FAIRLY STRAIGHTFORWARD THING THAT THEY WERE DOING, TRANSFORMED OUR ABILITY TO UNDERSTAND THE PRODUCTS OF NIH FUNDED RESEARCH. NO OTHER FUNDING AGENCY HAS THIS INFORMATION. THIS IS REALLY, REALLY IMPORTANT IMPORTANT. I AM NOT SAYING THAT 100% OF INVESTIGATORS COMPLY WITH THIS, BUT THIS IS THE BEST PICTURE WE HAVE, OKAY, OF HOW FUNDING IS TIED TO PRODUCTIVITY AT LEAST AS IT RELATES TO PUBLICATIONS. THE OTHER GREAT THING THAT PUBMED DID IS IT MADE THIS DATA PUBLICLY AVAILABLE. SO ANYONE CAN GO IN AND ASSESS USING PUBMED DATA, NIH FUNDING AND PUBLICATIONS. IT'S GREAT. RIGHT? IN 2008, NIH REQUIRED, REQUIRED NIH INVESTIGATORS TO LIST THEIR NIH SUPPORT IN THEIR PUBLICATIONS. AND PUT STUFF IN PUBMED. OOO, REQUIRED. A VERY STRONG WORD AND I KNOW NIH HATES REQUIRING THINGS FOR A WHOLE NUMBER OF THINGS, BUT BECAUSE OF THESE TWO THINGS, A STANDARD DATA COLLECTION PROCESS AND REPORTING REQUIREMENTS, THE NIH HAS BY FAR THE BEST DATA OF ANY FUNDER ON OUTPUTS OF PUBLICATIONS -- TO GRANTS. I DON'T THINK IT WAS THAT HARD. NEXT SLIDE HERE. HOW MANY OF YOU KNOW ABOUT FUND REF? EITHER YOUR ARMS ARE TIED OR -- OKAY. FUND REF. THERE'S A RECENT OSTP MEMO ABOUT OPEN ACCESS. NIH IS FAR AHEAD OF THE GAME AGAIN WITH PUBMED. THIS IS THE U.S. GOVERNMENT SAYING WHEN FEDERAL FUNDS ARE USED FOR RESEARCH, THAT RESEARCH SHOULD BE PUBLICLY AVAILABLE AFTER A CERTAIN EMBARGO PERIOD. WHAT I WANT TO GET ACROSS IS THAT THE NIH SHOULD PARTICIPATE IN FUND REF. WHAT THIS DOES, THIS INITIATIVE WORKS WITH PUBLISHERS TO MAKE SURE THAT WHEN SOMEONE SUBMITS A PAPER, THAT THERE IS A STANDARD WAY OF ACKNOWLEDGING FUNDING, THE FUNDING AGENCY OR ORGANIZATION, AND THERE IS A WAY FOR AN INDIVIDUAL TO LIST THEIR GRANT NUMBER SO THAT THIS ISN'T BURIED IN THE ACKNOWLEDGMENT SECTION, BUT IT'S ACTUALLY A PIECE OF THE FIELDED DATA AND BECOMES SEARCHABLE. SO ANYONE CAN SEARCH FUND REF FREE, AND GET ACCESS TO THE PAPERS THAT WERE FUNDED BY ORGANIZATION X. THIS WAS NEW, IT WAS JUST LAUNCHED LAST WEEK. I KNOW THERE WAS A PILOT THAT DID NOT INVOLVE NIH BUT OTHER FEDERAL FUNDERS IN THE U.S. I STRONGLY ENCOURAGE THE NIH, ACTUALLY I RECOMMEND THAT NIH PARTICIPATE IN FUND REF AND ENCOURAGE PARTICIPATION BY OTHER FUNDERS. ANOTHER FUN ONE, BY BENEFIT OF THE -- DOLE ACT, WHICH MANY PEOPLE HATED, THE NIH AND OTHER U.S. FEDERAL AGENCIES CAN UNDERSTAND WHICH PROJECTS THEY FUNDED ARE RELATED TO PATENT APPLICATIONS AND GRANTED PATENTS. AGAIN, THIS IS NOT EASY BECAUSE THE DATA ARE BURIED INSIDE A GRANT AND YOU HAVE TO DO A WHO BUNCH OF TEXT LINING TO GET IT OUT BUT IT'S AT LEAST THERE. THERE'S NO OTHER FUNDER, ANY OTHER COUNTRY THAT DOES THIS. THIS IS HEUNL. HUGE. THE OTHER PIECE THAT'S FUNNY IN PATENTS IS PATENTS ALSO HAVE PRIOR ART, SO YOU CANNOT ONLY GET THE CITATION RELATIONSHIP TO A GRANT, YOU CAN ALSO GET WHAT OTHER PAPERS ARE CITED IN THIS GRANT AS PRIOR ART. IF THESE DATA WERE AVAILABLE IN A STANDARDIZED FORMAT, THAT DIDN'T REQUIRE ALL THIS TEXT MINING, WE COULD GO A LOT FARTHER IN UNDERSTANDING THE RELATIONSHIP BETWEEN NIH FUNDING, IN FACT, FOR THAT MATTER, FEDERAL FUNDING OR ANY FUNDING, AND PATENT. THE OTHER PIECE HERE IS THAT THE DATA ON INVENTORS FRANKLY IS A MESS. IT'S AN ABSOLUTE MESS. YOU CAN SAY THE SAME THING ABOUT AUTHOR DATA WHEN IT COMES TO PUBLICATIONS, AND THIS IS WHERE IT COMES INTO WHAT MY CURRENT JOB IS. THE USPTO NEEDS TO DO A BETTER JOB CAPTURING GRANT AND NAME INFORMATION IN A STANDARD MANNER. SO RECOMMENDATION HERE TO THE EXTENT POSSIBLE, THE NIH SHOULD BE WORKING WITH THE US PTO TO IMPLEMENT STANDARDS FOR NAME AND GRANT INFORMATION IN PATENTS DURING PATENT APPLICATION PROCESS. ALL RIGHT. CLINICAL TRIALS. THIS IS ANOTHER GREAT OPPORTUNITY, TRACKING NIH FUNDING AS IT RELATES TO CLINICAL TRIALS. CLINICALTRIALS.GOV WAS CREATED, IN 2008. THIS IS A GREAT OPPORTUNITY BUT -- AND I'VE TRIED OVER AND OVER AGAIN AND I KNOW THERE'S A FEW PEOPLE THAT HAVE TRIED WITH ME, IT IS ALMOST USELESS FOR APPLICATION PURPOSES. I'M BANGED MY HEAD AGAINST THE WALL SO MANY TIMES WITH THIS ONE, IT'S SO FRUSTRATING! SO THERE ARE FEW IF ANY PRINCIPAL INVESTIGATOR NAMES OR NIH GRANT ACKNOWLEDGMENTS IN THE CLINICAL TRIALS DATASET. I UNDERSTAND THAT THE DATASET WAS NOT CREATED FOR EVALUATION PURPOSES BUT FOR HEAVEN'SATION, LET'S FIGURE OUT A WAY TO MAKE IT USEFUL. THESE FEW ADDITIONS, PI NAMES AND GRANT ACKNOWLEDGMENTS, WOULD MAKE THE DATABASE A TREASURE TROVE FOR NIH IN DETERMINING THE LONGER RAGE RANGE VALU RANGE VALUE OF THEI R INVESTMENTS. LINKING THESE BACK TO THE REPORTER DATABASE WOULD BE FABULOUS. LET'S SEE IF WE CAN GET TO THAT POINT. THERE'S OTHER DATABASES OUT THERE. FDA HAS A MEDICAL DEVICES, BIOLOGICALS DATABASE, AND I'VE CALLED THEM A COUPLE TIMES, WE'RE NOT A LIBRARY AND YOU'RE LIKE, I KNOW, BUT CAN'T WE FIND A WAY TO MAKE THESE DATA USEFUL? SO FOR EVALUATION PURPOSES. WHO DID NIH TRAIN? THIS IS A BIG ONE. I'VE BEEN INVOLVED IN WORKFORCE STUDIES FOR QUITE A LONG TIME. AND SO I ACTUALLY HELP SUPPORT THE BIOMEDICAL WORKFORCE WORK GROUP AS A CONTRACTOR, AND THEY PUT FORTH A NUMBER OF RECOMMENDATIONS THAT DR. TABAK TALKED ABOUT EARLIER. THERE'S A LOT OF POOLS, A LOT, A LOT OF POLLS TRYING TO GET THE DATA. THERE'S A LOT OF STUFF OUT THERE, THOUGH. THE BIGGEST HOLE WE HAVE IN ANY KIND OF WORKFORCE STUDY, WELL, SHOOT, IS UNDERSTANDING WHO THE WORKFORCE IS. WE DON'T TRACK PEOPLE AT ALL. RIGHT? WE HAVE THE NIH GRANTS DATABASE AND PEOPLE HAVE -- IDs, BUT WE BARELY HAVE THEM LINKED TO EACH OTHER, WE DON'T HAVE THEM LINKED TO GRANTS. THERE'S SOME WORKFORCE PROGRAMS, SOME TRAINING PROGRAMS, COLLECT ACTUALLY NO PARTICIPANT NAMES. NONE. SO YOU FUNDED THIS PROGRAM TO TRAIN PEOPLE BUT YOU DON'T -- WHY IS THAT? WHY IS THAT? THERE'S -- SOMETIMES IT'S OWN THE P.I., SOME PROGRAMS COLLECT NAMES BUT THERE'S NO QUALITY ASSURANCE SO THERE'S A LOT OF MISSPELLINGS, NO LINKAGE TO THEIR MENTOR, YOU'RE TRYING TO FIND OUT IF THEY'RE INVOLVED IN A PAPER BUT YOU CAN'T FIGURE IT OUT. UNTIL RECENTLY, ONLY PIs WERE ASSIGNEASSIGNED A COMMONS I.D. AND THAT'S GREAT BUT THAT DOESN'T HELP TRACKING FOR THOSE INVESTIGATORS THAT MAY CONTINUE IN RESEARCH WITHOUT NIH FUNDING. WHICH IS ALSO IMPORTANT BECAUSE IT'S IMPORTANT TO NIH IF THEY FUNDED SOMEBODY AND THEY STAYED IN RESEARCH, EVEN IF THEY AREN'T FUNDED BY NIH, I WOULD ARGUE. SO IN THIS CONTEXT, IT IS REALLY GREAT TO SEE NIH ENGAGED WITH ORCID TO TEST OPTIONS FOR ADOPTION OF CERTAIN PERSISTENT RESEARCHER IDENTIFIERS THAT CAN BE USED ACROSS AN IDENTIFIER'S CAREER. FUNDING IN PUBLICATIONS, REPORTING BY INVESTIGATORS DID NOT REALLY TAKE OFF UNTIL IT WAS REQUIRED BUT THE REQUIREMENT HINGED UPON AN APPROPRIATE AND EFFECTIVE DATA COLLECTION STRATEGY. SO THIS IS BACK WITH PUBLICATION, RIGHT? YOU CAN'T REQUIRE SOMEBODY TO REPORT UNLESS YOU HAVE A WAY FOR THEM TO REPORT. IN THE CASE OF PERSISTENT RESEARCHER IDENTIFIERS, THE SAME IS TRUE. WE NEED TO MAKE SURE THAT THE NIH INFRASTRUCTURE IS IN PLACE TO COLLAPSE THESE AND STORE THEM AND LINK THEM. BUT THEN I STRONGLY RECOMMEND THE USE OF THESE IDENTIFIERS BY ALL INDIVIDUALS FUNDED BY NIH SHOULD BE MANDATORY. SO THESE ARE A FEW SLIDES FROM SCIENCE TV WHICH WAS DESCRIBED A LITTLE EARLIER, THE IDEA IS THAT SOMEBODY AS THEY'RE APPLYING FOR A GRANT WILL ALSO LINK THAT GRANT APPLICATION TO THEIR ORCID IDENTIFIER, A PERSISTENT UNIQUE IDENTIFIER FOR AN INDIVIDUAL THAT BELONGS TO AN INDIVIDUAL, NOT AN INSTITUTION, CAN BE USE BID THAT INDIVIDUAL THROUGHOUT THEIR CAREER. SO YOU CAN LINK THAT GRANTS APPLICATION TO THIS ORCID IDENTIFIER AND THAT IDENTIFIER BECOMES ACTUALLY A PIECE OF THAT GRANT APPLICATION, A SEARCHABLE FIELD THAT YOU CAN USE TO LINK AN INDIVIDUAL TO THAT GRANT, TO THEIR PUBLICATIONS, TO ANYTHING ELSE THEY DO. YOU CAN ALSO USE THIS TO REDUCE THE BURDEN ON INDIVIDUALS, YOU CAN IMPORT INFORMATION FROM YOUR ORCID RECORD, IT LINKS YOUR PERSISTENT IDENTIFIER TO YOUR RESEARCH WORKS, YOUR PUBLICATIONS, YOUR DATA SETS, REVIEW EXPERIENCE, ET CETERA. YOU CAN USE THESE TWO THINGS FOR INTEROPERABILITY BETWEEN DATA SETS AND SAVE PEOPLE A LOT OF TIME. WHY TYPE SOMETHING IN IF IT'S ALREADY IN A DATABASE OVER HERE? SO RECOMMEND -- THERE'S A LOT MORE GREEN ON THIS SLIDE BECAUSE THIS IS MY CURRENT BABY. I REMEMBER THAT THE NIH GO FARTHER TO IMPLEMENT ORCID IDENTIFIERS TO REQUIRE THE USE DURING THE APPLICATION PROCESS AND LINK THIS TO POST AWARD OUTCOMES REPORTING. SO IF AN INDIVIDUAL AT THE OUTSET INCLUDES AN ORCID IDENTIFIER, THE NIH WILL USE IT TO FIGURE OUT WHAT THEY HAVE DONE, WHAT PUBLICATIONS, WHAT DATA SETS ARE LINKED TO THIS GRANT. TO REQUIRE THE USE OF ORCID IDENTIFIERS FOR ALL PERSONS SUPPORTED ON THE GRANT FOR MORE THAN A MONTH. PERIOD. HAVE THEM GET ONE, MAKE SURE IT'S REPORTED. THE NIH WILL BE ABLE TO TRACK THESE INDIVIDUALS AFTER. IMPLEMENT TO WORKLOAD POST AWARDED GRANT INFORMATION TO A GRANTEE'S ORCID RECORD, SO THAT GETS LINKED TO THE ORCID RECORD AND THEN SOMEBODY CAN SEARCH ORCID OR FOR THAT MATTER REPORTER FOR THAT INFORMATION. IT'S EXPLICIT, YOU DON'T HAVE TO GUESS. TO IMPLEMENT A WORKLOAD TO ALLOW RESEARCHERS TO SEARCH AND LINK ORCID -- IMPACT TO PI PROFILE RECORDS. AND TO ENCOURAGE THE USE OF ORCID IDs BY THE USPTO. WOULDN'T IT BE GREAT IF WE KNEW WHO THE ASIG ASSIGNEES WERE? IT MAKES ALL THESE DATA SETS TALK TO EACH OTHER AT THE PERSON LEVEL AND TO UNDERSTAND WHEN WE FUND A PERSON, WHAT THAT PERSON IS DOING, AND HOW THIS PERSON IS COLLABORATING THIS PERSON WITH THIS FUNDING AGENCY, ET CETERA, IT MAKES ALL THESE THINGS COME TOGETHER. NIH NEEDS TO COMMIT TO EVALUATING ITS PROGRAMS. RIGHT NOW I THINK THERE'S COMMITMENT IN A FEW PLACE, NCI IS DOING A GOOD JOB OF TRYING TO GET THEIR VALUES EVALUATED. I SEE GOOD STUFF GOING ON AT NIGMS, LOOKING AT HOW DO YOU ESTABLISH COMPARISON GROUPS, UNDERSTAND THE DIFFERENCE BETWEEN PEOPLE WHO WERE FUNDED AND NOT FUNDED BY THESE PROGRAMS. IT IS POSSIBLE TO DO THIS BUT THE NIH NEEDS TO COMMIT, NEEDS TO KNOW WHO'S PARTICIPATING, WHAT THEIR RESEARCH ACTIVITIES ARE AND THOUSAND THEY'RE CONNECTED. THIS IS ABSOLUTELY FUNDAMENTAL. THIS WILL TAKE ACTION INTERNALLY TO ARTICULATE PROGRAM GOALS AT THE OUTSET AND EVALUATION MODELS AND EXTEND DATA SYSTEMS TO INCLUDE STANDARD DATA FIELDS AND COLLECT DATA, COORDINATE EXTERNALLY TO COORDINATE INTRAOPERABLE STANDARDS. YOU'RE GOING TO HEAR A TALK TOMORROW, DATA EXCHANGE STANDARDS, I STRONGLY ENCOURAGE NIH AND THE SMRB TO CONSIDER THESE DATA EXCHANGE STANDARDS: NIH DOES NOT HAVE TO CREATE A WHEEL HERE. THERE'S ALREADY A WHEEL OUT THERE TO ATTRIBUTE VALUE. FINALLY, LAST SLIDE, SO IN SHORT, NIH HAS A LOT OF THE PIECES. WE HAVE TO HAVE CONNECTORS SO THESE PIECES TALK TO EACH OTHER. SO WE NEED TO CONNECT THE DATA TO PEOPLE. EVERYTHING WE TALKED ABOUT THIS MORNING HAD TO DO WITH HOME PAPERS DO WE HAVE, WHICH IS GREAT AND IT'S REALLY IMPORTANT, BUT REALLY AT THE END OF THE DAY, IF WE LOOK AT THE NIH REVIEW PROCESS, WE'RE FUNDING PEOPLE HERE. WE'RE FUNDING PEOPLE. SO I THINK WE NEED TO EVALUATE PEOPLE, WHETHER IT'S IN THE CONTEXT OF A PROGRAM, WE NEED TO FIND THE PEOPLE AND EVALUATE THE PEOPLE AND THEIR RELATIONSHIPS. WE NEED ORCID TO DO THAT. ALSO THE LAST THING I WANT TO MENTION IS INITIATIVES SUCH AS STAR METRICS, I THINK MAKING THESE DATA AVAILABLE AS THE NLM HAS DONE PUBLICLY SO PEOPLE CAN USE IT FOR EVALUATION PURPOSES, WHETHER THEY'RE INSIDE NIH OR OUTSIDE SO THAT WE CAN TEST THE METRICS AND FIGURE OUT IF THEY'RE VALUABLE IS REALLY, REALLY IMPORTANT, SO I WOULD SAY ANYTHING THAT GETS RECOMMENDED TO NIH ALSO HAS TRANSPARENCY AND OPENNESS AND THE STANDARDS FOR MAKING THE DATA TALK TO EACH OTHER IS REALLY, REALLY IMPORTANT. SO I WILL END THERE AND GO SIT DOWN. >> SO WE HAD THREE VERY STIMULATING PRESENTATIONS. WE HAVE 10 MINUTES SCHEDULED TO BOTH THESE THREE INTERACTING PRESENTATIONS AND WITH A CHARGE TO GIL AND I TO SUMMARIZE IT, SO WE DO HAVE TO KEEP THIS MOVING QUICKLY. LET'S START WITH GAIL. >> I AGREE, THREE VERY EXCELLENT PRESENTATIONS, AND THINGS TO REALLY GRASP. LAUREL, I AM REALLY EXCITED ABOUT YOUR GUIDELINES DATA AND COMMENTS THERE. FOR ONE WHO WORKED WITH IOWA AND THE INTERNATIONAL JOURNAL EDITORS TO GET THE REGISTRATION OF CLINICAL TRIALS AND ALL THOUGH GOES WITH IT. I'M WONDERING, COULD YOU USE, IN LM, TO ALSO HAVE A REGISTRY FOR NEW PRACTICE GUIDELINES, BUT ALSO THEN BY HAVING THAT REGISTRY, WOULD IT GIVE YOU A MECHANISM OF UPDATING, IN OTHER WORDS ANY TIME NEW ARE RECOMMENDED OR ALTERATIONS, YOU CAN GET THAT DATA, AND HAVE YOU THOUGHT ABOUT THAT, IS THAT SOMETHING THAT COULD BE POSSIBLE >> THE DATABASE IS ACTUALLY RUN BY -- THEY ALREADY DO ALL OF THIS. IT'S THERE, NIH HAS THESE RESOURCES, THEY JUST ARE BASICALLY INACCESSIBLE. SO WHAT I WANT YOU GUYS TO THINK ABOUT IS HOW DO WE MAKE THESE THINGS ACCESSIBLE. >> AND ONE OTHER QUICK THOUGHT. EACH P.I. OF AN INSTITUTIONAL TRAINING GRANT IS REQUIRED TO TRACK EACH TRAINEE FOR 10 YEARS, AND THAT, I'VE ALWAYS THOUGHT WHAT AN UNBELIEVABLE RESOURCE, BUT YET NOT AWARE THAT THAT'S EVER BEEN USED IN TERMS OF THE DATA, BUT RELIGIOUSLY, EVERY YEAR FOR 15 YEARS, I TURNED IN THE DATA. THEN WHEN WE REVIEWED THE INTRAMURAL PROGRAM IN NIH AND THE SAME KIND OF DATA TRAINED IN THE INTRAMURALS PROGRAM, GUESS WHAT, OR FROM THE DOUBLE AMC AND THE SCHOOLS, GUESS WHAT, YOU'RE RIGHT, NO DATA. >> YEAH. >> LET ME PICK UP HERE. EVERYBODY HAS, I BELIEVE AT LEAST ALL THE PEOPLE AROUND THE TABLE, THE FIRST PAGE AT TAB 6, WHICH WAS A SET OF QUESTIONS THAT WERE PRESENTED TO THE SPEAKERS UNDER THE TITLES OF GUIDING PRINCIPLES, METHODS AND STRATEGIES AND STUDY QUESTIONS. YOU CAN JUDGE HOW WELL WE'VE COVERED THOSE IN THIS SESSION. I WOULD PARTICULARLY LIKE TO DIRECT YOUR ATTENTION TO THE ADDITIONAL QUESTIONS AT THE BOTTOM. A AND D. A, ARE THERE IMPORTANT CHARACTERISTICS OTHER THAN QUANTITY OF PUBLICATIONS AND CITATIONS THAT SHOULD BE CAPTURED, AND WHAT ARE THEY, AND THEY RANGE FROM INPUTS, OUTPUTS AND OUTCOMES. I THINK OUTCOMES IN YOUR LOGIC CHART MIGHT BE A LITTLE -- THERE'S NO OUTPUT, JUST OUTCOMES. CAN ADDRESS THAT LATER. D IS REALLY MY HIGHLIGHT FOR YOU. WHICH IS WHAT ARE THE MOST IMPORTANT LIMITATIONS OF CURRENT METRICS FOR THE GAINS IN SCIENTIFIC KNOWLEDGE, HOW CAN THESE LIMITATIONS BE OVERCOME. SPECIFICALLY, SHOULD WE BE ABLE TO IDENTIFY, QUOTES, BREAK THROUGHS. SHORT OF VERY EXPENSIVE 94 EXPERT PANEL THAT WE HEARD ABOUT. CAN WE MEASURE QUALITY AND NOT JUST QUANTITY OF PUBLICATIONS, AND DO WE REALLY THINK CITATIONS CAPTURE THAT ADEQUATELY. WHAT IS THE WAY WE MIGHT PICK UP THE POTENTIAL OF PUBLISHED WORK OR REPORTED WORK FOR TRANSLATION, AND HOW DO WE ADDRESS -- SEVERAL PEOPLE HIGHLIGHTED ALREADY TODAY THE TIME LAG BOTH FOR THE GENERATION OF SCIENTIFIC KNOWLEDGE AND THE UTILIZATION OF THAT KNOWLEDGE FOR THE GOALS OF THE NIH. >> SO I'LL TRY TO MAKE THIS SHORT. SO I THINK THERE HAS BEEN A FOCUS ON PUBLICATIONS BECAUSE WE CAN MEASURE IT TO SOME DEGREE. AND THERE'S SOME OTHER THINGS THAT RESEARCHERS DO. THERE'S COLLABORATIVE ACTIVITY, DATASET PRODUCTION WHICH WE TALKED ABOUT EARLIER, THERE'S TOOL DEVELOPMENT, THERE'S SOFTWARE, THAT WE DON'T TOUCH AT ALL AT NIH. AND I THINK WHAT NEEDS TO HAPPEN IS IT NEEDS TO BECOME MORE POSSIBLE -- THERE NEEDS TO BE AN ENCOURAGEMENT TO ACTUALLY MEASURE THIS TYPE OF ACTIVITY, TO PAY ATTENTION TO IT, TO CITE IT, SO THERE'S A HUGE CONVERSATION GOING ON NOW IN THE RESEARCH COMMUNITY ABOUT HOW TO DO THAT. HOW DO WE CITE SOFTWARE, HOW DO WE CITE DATA SETS, ET CETERA. AND THE NIH NEEDS TO ACTUALLY INCLUDE THESE KINDS OF ACTIVITIES IN ITS EVALUATION. THIS GOES BEYOND AND CAPTURES A LOT MORE OF THE RICHNESS OF WHAT HAPPENS. MENTORING, FOR EXAMPLE, DON'T CAPTURE MENTORING. TO SOME DEGREE -- HOW CAN YOU NOT CAPTURE MENTORING IF YOU'RE TRAINING AND IT'S SUPPOSED TO BE A KEY PIECE OF WHAT'S GOING ON. WE HAVE TO FIGURE OUT HOW TO CAPTURE THAT. I KNOW ONE OF THE QUESTIONS IS HOW DO WE EVALUATE WITHOUT INTERFERING, BUT RIGHT NOW WE'RE EVALUATING INT AND TO A LARGE EXTENT MEASURING PUBLICATIONS WHICH ARE IMPORTANT, BUT IT'S SKEWING RESEARCHER ACTIVITY TOWARDS WELL, I MUST PUBLISH, WHICH IS GREAT BUT THERE'S SOME OTHER THINGS YOU'RE SUPPOSED TO BE DOING. MENTORING, SHARING THEIR DATA, ALL THIS OTHER STUFF THAT THEY'RE NOT DOING BECAUSE THEY THINK, WELL, THE ONLY THING I'M GETTING MEASURED ON IS PUBLICATIONS. SO I THINK WE ALL NEED TO COME UP WITH WAYS TO INCENTIVIZE THIS RICHNESS OF BEHAVIOR AND WAYS FOR THE NIH TO CAPTURE THAT. THERE'S NOW PERSISTENT IDENTIFIERS NOT JUST ASSIGNED TO PUBLICATIONS BUT ALSO TO DATA SETS. WE'RE STARTING TO SEE THAT IN SOFTWARE. AND WITH ORCID, YOU CAN LINK THESE THINGS TO AN INDIVIDUAL SO IF YOU HAVE SOMEBODY'S I.D., YOU CAN GO. >> SO I WOULD LIKE TO TAKE A SLIGHTLY DIFFERENT TACT TO THAT. I AGREE WITH YOU AND I THINK IT IS IMPORTANT TO DO A BETTER JOB OF DESCRIBING OUTPUTS SUCH AS PUBLICATIONS. PUBLICATIONS IS OUR HAMMER AND EVERYTHING HAS BECOME A NAIL ESSENTIALLY TO BORROW A PHRASE, AND WE CAN DO A BETTER JOB, MANY OF THE THINGS SHE'S SUGGESTED, BUT THE FLIP OF THAT TOO IS, AND SOMETHING I TRY TO STRESS IS THAT IN THINKING THIS THROUGH, THERE'S A SENSE OF CREATIVITY IN TERMS OF WHAT DO WE EXPECT SOME OF THESE THINGS TO LOOK LIKE. AND ENVISION IT RATHER THAN FOCUSING ON DEET TAILS RIGHT NOW OF THE DATA AND THE QUALITY, WHICH IS IMPORTANT, BUT I THINK WE HAVE TO BE A LITTLE MORE VISIONARY AND THINK ABOUT THE BROADER KINDS OF THINGS SUCH AS, LIKE I SAID BEFORE, SOMETHING LIKE STARTUP COMPANIES, AND THE USES OF THESE TECHNOLOGIES AND THE TREATMENTS AND SO FORTH, AND WE TALKED EARLIER ABOUT INSURANCE COVERAGE AND HEALTHCARE COVERAGE AND ALL THESE PIECES. WHAT ARE SOME OF THOSE BIG MODULES, IF YOU WILL, OR NODULES THAT SIT OUT THERE THAT WE COULD LOOK AT, SOME OF WHICH ARE MORE PROXIMAL. THEY'RE NOT NECESSARILY 20 YEARS DOWN THE ROAD. THE ENTREPRENEURSHIP IS NOT NECESSARILY 20 YEARS DOWN THE ROAD FOR SOME OF THE THINGS WE'RE TALKING ABOUT. THEY'RE MUCH MORE PROXIMAL. AND WE TALKED EARLY ABOUT THE TIME LAG, IF WE CAN IDENTIFY SOME OF THESE MORE PROXIMAL TOPICS AND THEN IDENTIFY WAYS TO MEASURE THEM, I THINK WE WILL HAVE AN ARGUMENT BECAUSE WE WILL SAY THAT THAT'S OF VALUE. PARTICULARLY IF YOU'RE WORKING INTO THE BUSINESS COMMUNITY, THAT'S A VALUE. IN TERMS OF DOING THAT. BUT I ARGUE THAT WE HAVE TO HAVE THAT CONCEPTUAL FRAMEWORK. FOCUSING TOO MUCH ON THE QUALITY OF THE DATA RIGHT NOW, I THINK WE'RE LOSING SIGHT OF THE OPPORTUNITY TO DO THE VISIONARY PIECE. >> I THINK IT WAS YOU, DELLA, WHO MENTIONED THE -- OR MAYBE JIM -- THE POTENTIAL OF CASE STUDIES. IT SEEMS TO ME REALLY IMPORTANT TO COMPLIMENT WHAT YOU'RE DOING WHICH IS KIND OF THE TRADITIONAL EVALUATION OF DATA AND COLLECTION OF DATA WITH SOME EXAMPLES, SOME OF WHICH WILL PUT THAT APPROACH UNDER THE MICROSCOPE, SO LEAVING ASIDE A NUMBER OF EXAMPLES THAT I COULD GIVE, THAT EVERYONE COULD GIVE WHICH I WON'T BORE YOU WITH, IT ALSO DOES SEEM THAT THE HUMAN GENOME PROJECT IS A TREMENDOUS OPPORTUNITY BECAUSE IT'S RECENT, I GUESS A DECADE NOW, AND THE FUNDING WAS SO -- PRETTY CLEAR AND THE PUBLICATIONS ARE MORE RECOGNIZED BECAUSE OF THE NEW DATA, SO I WONDER IF A REALLY -- WHICH IS BEING DONE, I THINK, BUT A RIGOROUS EVALUATION GOING FORWARD AND BACKWARDS OF THAT PROJECT WOULD BE VERY, VERY INSTRUCTIVE, AS I THINK IT'S ALREADY TURNING OUT TO BE. THE OTHER QUESTION I WOULD JUST ASK, NOTHING COMES WITHOUT A COST AND WE'VE TALKED VERY, VERY LITTLE ABOUT COST TODAY. SO ONE IS THE COST OF DEVELOPMENT OF NEW TECHNOLOGIES WHICH I GUESS WITH OUTCOMES WOULD BE VALUED. ANOTHER IS WHAT IS THE THING YOU'RE RECOMMENDING AND WHERE IS THAT MONEY GOING TO COME FROM BECAUSE IT'S NOT NOTHING, RIGHT? WE'RE GOING TO HAVE TO FIND SOURCES TO PROVIDE US AND THEN WE'D ASK YOU HOW MUCH VALUE IT COULD PRODUCE, WHICH I GUESS YOU THINK WOULD PRODUCE A LOT OF VALUE. SO THESE ARE JUST SOME OF THE QUESTIONS THAT COME TO MIND. >> I DON'T DISAGREE WITH ANYTHING THAT YOU JUST SAID, AND I THINK IN TERMS OF THE COST OF THESE DATA, ESSENTIALLY, WE HAVE TO BE BROAD MINDED TOO IN TERMS OF BEING ABLE TO PARTNER, THAT MAY ACTUALLY BE MORE OF THE CAT BIRD SEED PRODUCING THIS DATA. IF YOU JUST HELPED YOUR DATA, IF YOU JUST INCLUDE THIS PARTICULAR VARIABLE, IT BECOMES A LITTLE MORE USEFUL FOR US, AND TO ENTER INTO THOSE KINDS OF DIALOGUES WITH PEOPLE. BECAUSE WE CAN'T -- WE CANNOT AFFORD IN A VERY PRACTICAL WAY TO BUILD ALL THESE DATA SETS. WE HAVE TO BE ABLE TO FIGURE OUT HOW TO PARTNER WITH EITHER FEDERAL OR PRIVATE TO ACQUIRE THEM, ONCE WE FIGURE OUT WHAT IT IS WE WANT TO ACQUIRE. >> AND I WOULD ARGUE THE DATA SETS, AT LEAST THE ONES I MENTIONED, I WANTED TO ALSO GET BACK TO ONE OF DELLA'S COMMENTS, THEY'RE ALL EXISTING. THE NIH DOESN'T HAVE TO CREATE ANY OF THESE DATA SETS. HOW DO WE ADD ANOTHER ONE OR TWO DATA FIELDS IN HERE, HOW DO WE PARTNER TO DO THAT, THAT IS NOT A HUGE EXPENSE, I WOULD ARGUE, COMPARED TO HOW MANY RO1 GRANTS ARE SUPPORTED BY THIS ORGANIZATION, AND I WOULD ALSO ARGUE THE OPPORTUNITY COSTS ARE NOT DOING ANYTHING, IT'S COSTING THE NIH ITS CREDIBILITY AND HAS -- SOME OF IT HAS TO BE DONE. TO YOUR POINT ABOUT DIVISION, I COMPLETELY AGREE, THAT JUST COLLECTING DATA FOR COLLECTING DATA'S SAKE IS NOT USEFUL. WHEN WHAT I WAS TRYING TO POINT OUT IS IF YOU GO IN WITH AN UNDERSTANDING OF VISION OF WHAT YOUR PROGRAM IS BE YOU WANT TO DO WHAT YOUR GOALS ARE AT THE BEGINNING, UH-UH CAN CONSTRUCTIVELY ENGAGE AND PRIORITIZE WHICH DATA SETS ARE GOING TO HELP ME UNDERSTAND AND THERE'S SOME FABULOUS DATA SETS LOOKING AT THE USE OF SPECIFIC TECHNOLOGIES, CURATIVE AGENTS, ET CETERA, THAT HAVE AN IMPACT ON HOW MANY DAYS SOMEBODY MISSES WORK, HOW MANY DAYS THEY'VE BEEN IN THE HOSPITAL, THAT'S ALREADY AVAILABLE, BUT RIGHT NOW WE CAN'T CONNECT THAT BACK BECAUSE WE DON'T HAVE THE CLINICAL TRIALS DATA. IT CAN BE FRUSTRATING. SO WE CAN CONNECT ALL THIS STUFF WITH, I WOULD SAY, A RELATIVELY MINIMAL AMOUNT OF WORK ON THE PART OF NIH AND SOME OTHER PARTNER ORGANIZATIONS TO GET THESE DATA SETS TO TALK TO EACH OTHER. >> I WANT TO PICK UP ON THIS CASE STUDY ISSUE. AND ALSO ACKNOWLEDGE THAT I THINK THE MOST FREQUENTLY USED WORD THIS MORNING IS DIFFICULT. [LAUGHTER] BECAUSE IT'S DIFFICULT. YOUR CHARGE IS VERY DIFFICULT. AND ONE OF THE REASONS I THINK IS THAT YOU'RE TRYING TO WORK IN AN AREA WHERE THERE'S NOT ANY STRICTLY LINEAR PURPOSEFUL CAUSALITY. INSTEAD IF ADVANCES ARE MADE BY ACCESSING IN A CHAOTIC WAY OPPORTUNITIES AND KNOWLEDGE THAT WE'VE CREATED. SO IT'S NOT LINEAR AND IT'S DIFFICULT TO EXPLAIN IN RETROSPECT. SO THE IDEA OF STORIES AT THE END CAN BE VERY COMPELLING FOR THE PUBLIC. JUST TO COMPLIMEN COMPLEMENT THE OTHER END, EVEN THE VERY PROXIMAL OUTPUTS LIKE CITATIONS, THEY'RE OFTEN MISUSED, THEY'RE MISINTERPRETED, THEY'RE DIFFICULT TO INTERPRET. WE'RE STILL ARGUING ABOUT THE VAL UNITED AND THAT'S YEARS AWAY FROM APPLICATION. SO I THINK THERE'S SOME VALUE IN JUST PROVIDING STORIES. >> SO WE ACTUALLY HAVE -- >> I'D LIKE TO GO BACK FOR A SECOND TO THESE QUESTIONS ON THE BOTTOM, A AND D, AND ONE OF THEM RELATES TO BREAKTHROUGHS, BECAUSE THIS PANEL WAS RELATED TO KNOWLEDGE IN TRYING TO MEASURE IT. ONE OF THE THINGS I'M ALWAYS IMPRESSED WITH, SO THIS IS NOT MOVING, I RO I LOVE THE IDEA OF PATENTS AND TECHNOLOGY AND MAKING SURE THAT WE MEASURE THAT, BUT THIS IS MEASURING BREAK THROUGHS, AND WHAT I SEE OFTEN IN SCIENCE IS IT'S LIKE THIS, CIRCULAR, AND SOMEBODY RECOGNIZES A FACTOR THAT'S -- THEY'VE ALWAYS THOUGHT WAS INVOLVED IN REGULATING MINERALIZED TISSUES AND ALL OF A SUDDEN IT'S NOW A HORMONE THAT'S REGULATING EVERYTHING ELSE. HOW DO YOU MEASURE THAT. SO A CITATION DOESN'T DO IT. >> SO THIS IS -- I'M NOT SAYING THIS IS THE WAY TO MEASURE IT, BUT WE -- IN MY PREVIOUS LIFE, WE DID ACTUALLY DEVELOP A PROJECT AND I KNOW WE DID SOME WORK WITH OTHER FOLKS AS WELL IN ACTUALLY TRYING TO IDENTIFY WHAT ARE SOME OF THE KEY HALLMARKS AFTER BREAKTHROUGH. ONE OF THE THINGS YOU CAN LOOK AT IS A VERY RAPID CITATION. IT'S NOT ALWAYS THE WAY YOU CAN IDENTIFY IT, AND IT WON'T IDENTIFY ALL BREAKTHROUGHS, BUT WHAT YOU CAN DO IS SAY, LOOK, I SHOULD BE PAYING ATTENTION TO THIS. I SEE SOMETHING HERE THAT SEEMS TO HAVE A HIGHER THAN EXPECTED RATE OF CITATIONS EARLY ON OR CERTAIN KINDS OF PATTERNS, AND THEN AS A PROGRAM OFFICER, IT PROVIDES ONE INDICATOR FOR YOU TO GO AND LOOK AT WHAT'S HAPPENING IN THIS PAPER OR WITH THIS GROUP OR WITH THIS FUNDING PROJECT AND INTERVENE IF UH-UH DECIDE YOU WANT TO, SO IT'S AN INDICATOR, IT'S USEFUL. AND THE FOLKS AT NIAID HAVE ACTUALLY USED THIS KIND OF THING. LOOK, I SHOULD BE PAYING ATTENTION RIGHT HERE. SHOULD I HOST A CONFERENCE OR SOME KIND OF GATHERING OF PEOPLE IN THIS AREA TO REALLY PUSH IT. AND MAYBE ONE OF THE REASONS WE DON'T SEE THE CITATIONS REALLY TAKE OFF IN SOME OF THESE IS BECAUSE THERE WASN'T AN EFFORT TO INTERVENE AND HELP PUSH IT. SO I WOULD ARGUE THAT LOOKING AT THESE KINDS OF AHEAD METRICS, EVEN THOUGH THEY'RE NOT INSOLUBLE, MAY BE ONE WAY OF PICKING UP ON A BREAKTHROUGH. >> I'D LIKE TO JUST AMPLIFY WHAT YOU SAID. I THINK -- WHAT I THINK OF THAT, I THINK IT'S A BUZZ FACTOR. SO THAT THERE IS AN ISSUE, SOMETHING HAS HAPPENED IN A PARTICULAR FIELD OF SCIENCE, AND WE'RE NOT QUITE SURE NECESSARILY BUT PEOPLE GET VERY EXCITED ABOUT IT. THERE'S A LOT OF INTEREST BEING ATTENDED TO. AN I THINK THINKING OF IT THAT WAY, THINKING ABOUT HOW YOU WOULD GO ABOUT ONE WAY IS CITATIONS, YOU COULD DO THE SAME THING IF YOU'RE LOOKING AT ABSTRACTS OF VARIOUS FIELDS IN TERMS OF CONFERENCES. THAT'S OFTEN WHERE I THINK IT'S PROBABLY FIRST POPS UP, BECAUSE YOU START TO SEE NODES OF EXPLOSIONS OF DISCUSSIONS GOING ON ON DIFFERENT TYPES OF TOPICS. THE OTHER THING WOULD BE TO -- TO CLEMENT WHAT YOU SAID WOULD BE TO START LOOKING AT APPLICATIONS IN TERMS OF THE -- TO SEE IF THAT ISSUE, WHATEVER IT IS, IS -- YOU'RE STARTING TO SEE AN INCREASE IN THE RATE OF THE APPLICATIONS INCLUDING THAT PIECE OF SCIENCE. DOESN'T MEAN THAT YOU'RE BEING FUNDED NECESSARILY BUT YOU'RE SEEING MORE APPLICATIONS COMING IN LOOKING AT THAT, IF IT'S SOMETHING THAT WE CAN GRAB OUT OF OUR TEXT MINING. SO LIKE EVERYTHING ELSE WE TALKED ABOUT THIS MORNING, THERE'S PROBABLY A MULTIPRONG STRATEGY SET YOU BY WHICH ONE WOULD TRY TO DO THAT. NO ONE IS NECESSARILY PERFECT. >> IT ALSO DEPENDS ON THE TIME RESOLUTION. IF YOU LOOK IN PUBMED AT THE CITATIONS TO ALL PAPERS, THEY PEAK AT 1 1/2 YEARS. SO IF YOU KNEW THAT, YOU COULD ACTUALLY SAY AT 1 1/2 YEARS, I COULD FIND THAT NEW IDEA IN RETROSPECT, SO MICRO RNAs, YOU COME UP WITH A NEW CONCEPT, IT'S PEAKING, THEN TAKING OFF. SO PEOPLE PLAYED WITH IDEAS OF LOOKING AT DOWNLOAD RATES, PDF DOWNLOADS. IS THAT GETTING ME A LITTLE BIT CLOSER TO THE DISCOVERY. BUT TO GET IT RIGHT AT THE DISCOVERY IS HUMANS. >> HOW MANY CITATIONS ARE BASED ON NOTHING MORE THAN MAYBE THE ABSTRACT, ESPECIALLY WITH PMC, FOR MANY JOURNALS, YOU HAVE TO PAY $30 TO FIND OUT IF THERE'S ANYTHING OF INTEREST IN THE ARTICLE SO YOU ONLY SEE THE ABSTRACT. THERE ARE A LOT OF CHALLENGES TO MAKING THIS SYSTEM WORK. I DIDN'T HEAR ANYBODY PICK UP ON THIS NOTION OF IDENTIFYING COMPELLING UNKNOWNS, THE BIG QUESTION -- IF WE RELY ALMOST COMPLETELY ON AN INDIVIDUAL INVESTIGATOR INITIATED PROPOSAL, IS THERE SOME ROLE FOR A COMPLEMENTARY DRIVER? I THINK THE INSTITUTES HAVE LONG HAD THESE RFAs, RFPs, TO TRY TO DO THAT. HOW DO WE LOOK AT THOSE IN ANY WAY SIMILARLY OR DIFFERENTLY. I DIDN'T HEAR ANYTHING MENTIONED ABOUT THAT. WE'VE HEARD A LOT OF INTERESTING COMMENTS TODAY AND WE'RE GRATEFUL TO YOU. I THINK AS WE THINK ABOUT THE TOPICS THE REST OF THE DAY, WE'LL HAVE AN OPPORTUNITY TO BUILD A LIST OF FURTHER QUESTION QUESTIONS, AND MAYBE SOME OF THESE, I'D LIKE TO SPEND BACK TO THE SPEAKERS WHO PUT A LOT OF EFFORT TO COMING OUR MEETING. I THINK WE'RE OUT OF TIME. >> NORM, I THINK WE'VE GONE OVER BY ABOUT 10 MINUTES. >> THIS HAS BEEN ABSOLUTELY TERRIFIC AND WELL WORTH THE EXTRA TIME. JOSIE AND GIL, THANK YOU VERY MUCH. LET'S TAKE A BREAK. WE'LL STICK TO THE ORIGINAL BREAK TIME. WE COULD GATHER AT 10 AFTER, THAT WOULD BE VERY GOOD. I'D LIKE TO WELCOME EVERYBODY BACK FROM THE BREAK. OUR LAST SESSION OF THE MORNING ON A VERY IMPORTANT TOPIC. WE'RE GOING TO BE LOOKING AT PUBLIC HEALTH OUTPUTS AND OUTCOMES OF BIOMEDICAL RESEARCH, AND I'D LIKE TO TURN THIS NEXT SESSION OVER TO GRIFF RODGERS AND ARTHUR RUBENSTEIN WILL BE THE COMODERATORS AND TO COME OUR GUEST SPEAKERS. >> THANK YOU, AMY, VERY MUCH. I WANT TO ALSO WELCOME AND THANK OUR VERY DISTINGUISHED PANEL OF SPEAKERS FOR THIS SECOND SESSION TODAY ON PUBLIC HEALTH OUTPUTS AND OUTCOMES OF BIOMEDICAL RESEARCH, AS I THINK OUR CHAIR ALLUDED, GAIL, IN HER INTRODUCTORY REMARKS, NIH HAS BEEN IN THE BUSINESS OF THE ACQUISITION OF KNOWLEDGE, AND I THINK ALONG THIS CONTINUUM THAT YOU SAW IN ONE OF HER SLIDE SETS, THIS IS REALLY THE ABILITY TO TRANSFER OR TRANSFORM THAT KNOWLEDGE NOW INTO ACTION. SO YOU'LL BE HEARING FROM THIS PANEL HOW WE MEASURE THE PUBLIC HEALTH BENEFITS OF BIOMEDICAL RESEARCH IN A PO POSITIVE WAY AND HOW THAT TRANSLATES TO THE VALUE OF THE AMERICAN PUBLIC. AS I SAID, WE HAVE THREE SPEAKERS IN THIS GROUP. WE'RE GOING TO TRY TO, AGAIN FOR THE BENEFIT OF TIME, REALLY ONLY ENTERTAIN SOME VERY SHORT QUESTIONS AT THE END BUT THEN HOLD SOMETIME AT THE END FOR A FULL DISCUSSION OF OUR WHOLE ENTIRE PANEL MEMBERS. AND WITH THAT, LET ME TURN IT OVER TO OUR CO-CHAIR OF THE SESSION. >> I TOO WANT TO WELCOME THE PANELISTS. WE'RE DELIGHTED YOU'RE HERE. WE BELIEVE THIS IS A REALLY IMPORTANT TOPIC. WE DID PROVIDE SOME QUESTIONS AHEAD OF TIME AND I KNOW SOME OF YOU WILL AT LEAST ATTEMPT TO ANSWER SOME OF THOSE. SO WE'LL BEGIN WITH THE FIRST PRESENTATION BY DR. JAMES CURRAN, DEAN THE PUBLIC HEALTH AT THE ROLLINS SCHOOL OF PUBLIC HEALTH, EMORY, SINCE I HAVE TO SAY 1995, WHICH MAKES IT 17 YEARS, MAYBE 18, JIM, WHICH TELLS YOU HIS CREDENTIALS ARE IMPECCABLE. BEFORE THAT, HE'S BEEN ALSO AT THE CDC AND GRADUATED FROM THE UNIVERSITY OF MICHIGAN AND HAS A LOT OF EXPERIENCE IN MANY OF THESE TOPICS. SO JIM? >> THANKS, ART AND GRIFF. IT'S A REAL HONOR TO BE HERE. I'D LIKE TO BEGIN BY THANKING THE SMRB FOR WHAT YOU ARE DOING. I THINK I KNOW THAT THIS IS PUBLIC SERVICE, SERVICE TO SCIENCE AND TO THE NIH, AND IT'S A VERY, VERY VALUABLE THING TO DO. I WANT TO JUST SAY A COUPLE THINGS IN ADDITION TO BEING THE DEAN OF THE ROLLINS SCHOOL OF PUBLIC HEALTH, I'M ALSO PRINCIPAL INVESTIGATOR ON CENTERS FOR AIDS RESEARCH GRANT SINCE 1997, SO THAT'S MY CONFLICT OF INTEREST, I GUESS, IN TERMS OF BEING AN INVESTIGATOR. I WAS A FORMER FEDERAL GOVERNMENT EMPLOYEE AT THE CENTERS FOR DISEASE CONTROL FOR MORE THAN 20 YEARS. SO I HAVE THE EXPERIENCE WITH THE PROGRAM AND NEED FOR BUDGET JUSTIFICATION AS WELL AS THE INDEPENDENT EY EVALUATION OF PROGRAM IMPACT AND ALSO THE TENDENCY IT MIX THOSE TWO UP. AND I THINK THAT THEY'RE BOTH VERY, VERY IMPORTANT, AND A LOT OF TIMES ONE GETS IN THE WAY OF THE OTHER AND IT'S A VERY DIFFICULT THING TO DO. I KNOW THE SMRB IS IN THE BUSINESS OF DOING THAT, SO THANK YOU FOR WHAT YOU'RE DOING. IF ONE OVERSHADOWS THE OTHER, THEN YOU END UP IN TROUBLE A LOT OF TIMES. I'M GOING TO BE JUST KIND OF INTRODUCING A TOPIC OF PUBLIC HEALTH, AND MAYBE IN TOO SIMPLISTIC A WAY BUT IT'S GOING FROM WHAT HAS BEEN FEASIBLE BEFORE TO WHAT IS EXTREMELY DIFFICULT NOW, AND LOOKING AT THE PUBLIC HEALTH IMPACT OF NIH EXPENDITURES. THE FIRST THING IN BROAD PERSPECTIVE IS TO SAY I THINK THROUGHOUT THE WORLD THE GOVERNMENT SUPPORTED BIOMEDICAL RESEARCH IN THE UNITED STATES IS OF COURSE THE MOST EXPENSIVE BUT ALSO THE MOST REVERED AS THE VERY BEST IN THE WORRELL. SIMILARLY, OUR HEALTHCARE ENTERPRISE, I DON'T WANT TO CALL IT A SYSTEM, BUT OUR HEALTHCARE ENTERPRISE IS BY FAR THE MOST EXPENSIVE IN THE WORLD, AND NOT AS REVERED THROUGHOUT THE WORLD. UNFORTUNATELY, OUR HEALTH OUTCOMES OR OUR PUBLIC HEALTH MORBIDITY AND MORTALITY MEASURES ARE WAY, WAY BELOW THE BEST IN THE WORLD. AS A MATTER OF FACT, THEY'RE AMONG THE WORST OF ALL THE DEVELOPED COUNTRIES. SO ONE OF THE QUESTIONS WE HAVE TO ASK IS, THE FIRST THING YOU HAVE TO SAY, OF COURSE, IS THAT NIH EXPENDITURES ARE ONLY ABOUT 1% OF TOTAL HEALTH EXPENDITURES. SO ONE WOULDN'T EXPECT RESEARCH TO BE EVERYTHING. AND THE QUESTION MIGHT BE, HOW MIGHT RESEARCH BEST BE USED TO HAVE A BETTER IMPACT ON OUR RATHER DISMAL HEALTH ME SOORS IN MEASURES IN THE UNITED STATES, AS WELL AS HEALTH MEASURES THROUGHOUT THE WORLD. IS IT RELEVANT OR IS IT TO BE EXPECTED THAT IT COULD DO BETTER FROM A RESEARCH POINT OF VIEW AND HOW MIGHT WE MEASURE IT. AND HOW IS IT DOING NOW AND HOW DO WE KNOW. THAT'S SORT OF THE BEGINNING. THE INSTITUTE OF MEDICINE HAD STUDIED PUBLIC HEALTH IN ADDITION TO SAYING THAT IT WAS IN DISARRAY IN THE UNITED STATES. IT DEFINED PUBLIC HEALTH AS WHAT WE AS A SOCIETY DO COLLECTIVELY TO ASSURE CONDITIONS IN WHICH PEOPLE CAN BE HEALTHY. MOST PEOPLE IN ATLANTA THINK IT'S THE SAME THING AS PUBLIC MEDICINE, THE SAME THING AS PUBLIC FINANCING, BUT, IN FACT, IT INCLUDES EVERYTHING. INCLUDING ALL OF HEALTHCARE AND ALL OF OTHER SOCIETAL ISSUES THAT IMPACT HEALTH. IT IS NOT JUST HEALTHCARE BUT IT'S EVERYTHING. SO IT INCLUDES LAWS AND REGULATIONS AND SMOKING REGULATIONS, MOTORCYCLE HELMET LAWS, ET CETERA. PUBLIC HEALTH DIFFERS FROM MEDICINE BY A FOCUS ON THE HEALTH OF POPULATIONS AND A FOCUS ON PREVENTION. NOW, THE ADVANTAGE OF THE HEALTH OF POPULATIONS ISSUE IS THAT THE ASSESSMENT HAS TO BE POPULATION-BASED. THAT IS THERE ARE DATA SYSTEMS THAT ARE NEEDED TO MEASURE THE HEALTH OF POPULATIONS RATHER THAN JUST THE HEALTH OF INDIVIDUALS. I'M ALSO A FACULTY MEMBER IN OUR MEDICAL SCHOOL WHEN I PREACH ABOUT POPULATIONS, I TELL SOME OF OUR INDIVIDUAL CLINICIANS THAT TREAT INDIVIDUALS THAT THEY DON'T SEE THE FOREST FOR THE TREES. THEY TELL ME I WOULDN'T KNOW A TREE IF I RAN INTO ONE. [LAUGHTER] >> SO I THINK OF COURSE THE HEALTH OF POPULATIONS DEPENDS UPON THE HEALTH OF INDIVIDUALS, BUT A LOT OF TIMES WE LEARN ABOUT HEALTH DISPARITIES, WE LEARN ABOUT DIFFERENCES AND WE LEARN ABOUT IMPORTANT REASONS FOR HEALTH DIFFERENCES BY STUDYING POPULATIONS. IN THE SAME INSTITUTE OF MEDICINE REPORT IN '87, THEY DEFINED THE MAJOR FUNCTIONINGS OF PUBLIC HEALTH AS ASSESSMENT, POLICY DEVELOPMENT AND ASSURANCE. SO PUBLIC HEALTH IS VERY MUCH INVOLVED IN DATA COLLECTION IN ORDER TO DO AN ASSESSMENT OF HEALTH. YOU'LL HEAR FROM DR. ARISPE A LOT OF SYSTEMS SET UP BY THE NATIONAL CENTER FOR HEALTH STATISTICS ONLY FOR US TO REMEMBER THAT POPULATION-BASED DAY IDATA SYSTEMS ARE VERY, VERY EXPENSIVE IF THEY'RE DONE WELL, SO THAT IF WE WANT THEM TO BE RESPONSIVE TO CHANGES IN RESEARCH FINDINGS, WE HAVE TO THINK ABOUT USING EXISTING DATA SYSTEMS AS WELL AS THINK ITING ABOUT MODIFYING THOSE IN AN ANTICIPATORY WAY. OF COURSE DATA HAS TO BE ACCURATE AND HAS TO BE TRACKED OVER TIME. POLICY DEVELOPMENT COMES AS A RESULT OF RESEARCH OFTEN. FOR EXAMPLE, THE LAWS TO REMOVE LEAD FROM GASOLINE CAME FROM NIH-SUPPORTED RESEARCH THAT RESULTED IN CONSIDERLY BLESS LESS LEAD POISONING IN THE UNITED STATES, SEAT BELT LAWS, BASED UPON SCIENCE BASED RECOMMENDATIONS FOR SECONDARY PREVENTION, SMOKING LAWS, CLEAN AIR LAWS, ALL OF THESE THINGS HAVE COME FROM CAREFULLY DOCUMENTED RESEARCH THAT HELPS, AND POLICY DEVELOPMENT IS A KEY PART OF PUBLIC HEALTH. ASSURANCE IS KIND OF THE REST OF IT, AND THAT MEANS THAT THERE'S AN ADEQUACY OF RESOURCES AND TARGETING OF RESOURCES TO GET THE JOB DONE. OF COURSE THAT'S ONE OF THE MAJOR PROBLEMS THAT WE HAVE IN THE UNITED STATES WITH 50 MILL MILLION UNINSURED PEOPLE AND A VERY, VERY DIVERSE POPULATION IN HEALTH LITERACY AND LITERACY ITSELF IN TERMS OF ACTUALLY IMPLEMENTING THE KIND OF SCIENTIFIC AND HEALTH PROCEDURES THAT ARE NEEDED. BUT UNLESS THERE'S ASSURANCE, OF COURSE, THEN A LOT OF THE FINDINGS MAY NOT MAKE AS MUCH DIFFERENCE. HOW DO YOU SET PRIORITIES IN PUBLIC HEALTH? THE MOST IMPORTANT PRIORITIES INVOLVE HOW MANY PEOPLE IN A POPULATION ARE AFFECTED. POPULATION CAN BE THE WORLD, THE CITY, THIS ROOM, COULD BE ALL CHILDREN, COULD BE ALL WOMEN, IT COULD BE ALL MINORITIES, OR IT COULD BE ALL AMERICANS. AND IT ISN'T JUST HOME PEOPLE ARE AFFECTED BUT HOW MANY ARE POTENTIALLY AFFECTED. SO FOR EXAMPLE, THE COMMON COLD IS VERY, VERY COMMON. AND INFLUENZA MAY OR MAY NOT BE COMMON BUT IT COULD BECOME EXTREMELY COMMON. SO PEOPLE POTENTIALLY AT RISK ALSO FIT INTO A LARGE NUMBER. OF COURSE THE SEVERITY OF CONDITIONS IS EXTREMELY IMPORTANT AS WELL SO THAT HIV INFECTION, CANCER, HEART DISEASE, ALZHEIMER'S DISEASE, ARE MORE IMPORTANT THAN THE COMMON COLD. RESPIRATORY INFECTIONS ARE MORE IMPORTANT IN AFRICA THAN THEY ARE IN THE UNITED STATES. BECAUSE IN AFRICA, THEY KILL CHILDREN. IN HERE WE HAVE A RESCUE MECHANISM TO PREVENT THEM FROM BEING AS IMPORTANT AS A PUBLIC HEALTH PRIORITY. BUT IMPORTANTLY, OF COURSE, IS OUR ABILITY TO IMPACT EITHER THE NUMBERS OR THE SEVERITY, AND THAT'S WHERE A LOT OF THE RUB COMES IN IN TERMS OF PUBLIC HEALTH PRIORITIES. SO THEREFORE, PREVENTING LUNG CANCER IS NOT LIKELY TO BE VERY GOOD, I MEAN, I KNOW THERE ARE SPIRAL CT STUDIES AND OTHER THINGS LIKE THAT. THE PRIORITY WOULD BE TO PREVENT SMOKING. IF YOU WANTED TO PREVENT LUNG CANCER PREVENTING ALZHEIMER'S DISEASE IS VERY DIFFICULT AND TREATING IT IS SUBOPTIMAL AT BEST, BUT DIAGNOSIS AND TREATMENT OF BREAST CANCER AND PREVENTION OF DIABETES, PREVENTION OF HEART DISEASE IS EXTREMELY IMPORTANT. SO THINKING ABOUT IT THAT WAY, HOW WOULD WE GO FROM THE BENCH TO THE POPULATION TO THINK ABOUT THE RESEARCH THAT'S NEEDED TO LOOK AT POPULATION IMPACT. AND ONE EXAMPLE I'LL USE, AND DR. ENTWISLE WILL MENTION THIS TOO, A STUDY THAT NIH SUPPORTED IN AFRICA, AN ANTIRETROVIRAL STUDY, I'LL BRING THIS UP. PART OF THE PROBLEM WITH POPULATION OUTCOME AND POPULATION-BASED RESEARCH IS THAT IT'S MUCH MORE CONTROVERSIAL. I'VE BEEN INVOLVED IN THIS IN A LOT OF MY CAREER AND I'VE BEEN CALLED TO -- BY SEVERAL MEMBERS OF CONGRESS WHO DIDN'T LIKE THE IDEA THAT WE WERE DOING STUDIES OF GAY SEXUAL BEHAVIOR. AND OF COURSE ONE OF THE QUESTIONS IS HOW DO YOU KNOW ABOUT IT UNLESS YOU CITY DID IT. BUT THEY'LL SAY WHY DON'T YOU GO BACK TO THE LAB AND DO GENETIC STUDIES. THERE'S A SAFETY IN THE LABORATORY. MOSTLY BECAUSE NOBODY UNDERSTANDS WHAT YOU'RE DOING, EVEN THE PUBLIC DOESN'T, AND THEY ALSO THINK IT LEADS TO PATENTS AND THINGS LIKE THIS. WHEN YOU START TO GET INTO THIS ISSUE OF COMMUNITY-BASED STUDIES, IT GETS TO BE VERY MESSY BUT ON THE OTHER HAND, IF YOU WANT TO MAKE POPULATION IMPACTS, THAT'S WHERE YOU ARE. SO YOU TAKE A STUDY, WHAT HAPPENS IS DR. MIKE COHEN AND HIS COLLEAGUES IN NORTH CAROLINA DID A STUDY OF SO CALLED DISCORDANT COUPLES, THAT IS COUPLES WHERE ONE MEMBER OF THE HETEROSEXUAL COUPLE HAD H.I.V. INFECTION AND THE OTHER MEMBER DIDN'T, AND THEN THEY PROVIDED VERY EARLY ON ANTI-RETROVIRAL THERAPY EARLIER THAN THE RECOMMENDATIONS FOR TREATMENT. AND HE SHOWED THAT EVEN THOUGH THEY PROVIDED CONDOMS TO BOTH PLACEBO GROUP AND THE CONTROL CONTROLS -- I MEAN AND THE PEOPLE WHO GOT THE DRUGS, THAT THERE WAS A BIG DROPOFF IN TRANSMISSION OF H.I.V. AN ELEGANT STUDY, I THINK IT WAS SCIENCE OR ONE OF THE JOURNALS, CALLED IT THE NUMBER ONE FINDING OF THE YEAR LAST YEAR, A BIG NIH-SUPPORTED STUDY. SO THE QUESTION IS, WHAT KIND OF IMPACT WILL THAT STUDY HAVE? IT'S GOING TO DEPEND UPON A NUMBER OF THINGS. FIRST OF ALL, HOW MUCH PREVENTION IS THERE TO BE DONE, AND HOW MANY PEOPLE WILL ACTUALLY RECEIVE THIS DRUG? IN AFRICA, THERE'S NOT ENOUGH MONEY AROUND AND ENOUGH RESOURCES FOR THE PEOPLE WHO NEED IT FOR MEDICAL REASONS. SO IT'S PRETTY UNLIKELY THAT THEY WILL BE ABLE TO UTILIZE THIS IN GREAT NUMBERS IN AFRICA. IT'S ALSO VERY HARD TO MEASURE PREVENTION AND EXTREMELY HARD TO MEASURE WHAT KIND OF IMPACT THIS PART OF PREVENTION WILL MAKE BECAUSE THE TESTING AND COUNSELING ITSELF REDUCES TRANSMISSION BY 25%, SO THERE'S ONLY 25% LESS. THE OTHER THING IS THAT PEOPLE IN RANDOMIZED CONTROL TRIALS IN WHICH EFFICACY IS DOCUMENTED ARE SELECTED PEOPLE WHO VOLUNTEER AND COMPLY WITH THE TRIAL. THEY'RE NOT LIKE THE REST OF US, COMPLETELY, WHO MAY NOT COMPLY AND MAY NOT BE LIKE THAT. SO MOVING TO EFFECTIVENESS, IN OTHER WORDS, WHAT EFFECT WOULD THIS HAVE ON ALL HETEROSEXUAL COUPLES IN AFRICA OR HETEROSEXUAL COUPLES IN THE UNITED STATES, AND THEN HOW MANY COUPLES WOULD EVEN TAKE THE DRUG OR EVEN BE TESTED TO BEGIN WITH. SO IF YOU WANTED TO LOOK AT THE IMPACT OF THAT STUDY, YOU CAN DO IT WITH MODELING AND OTHER TYPES OF THINGS BUT YOU HAVE TO RECOGNIZE THAT THAT PARTICULAR STUDY IT SELF, WHICH IS EXTREMELY VALUABLE IN REINFORCING THE LINK BETWEEN CARE AND PREVENTION IS NOT LIKE LIKELY TO HAVE A DEMONSTRABLE IMPACT BECAUSE IT CAN'T BE STUDIED IN LARGE DATA SETS, BUT IT'S REALLY IMPORTANT. ALTHOUGH I'M NOT MAKING IT SOUND IMPORTANT, IT'S NOT GOING TO BE DEMONSTRATED AS IMPORTANT BECAUSE IT SIMPLY CAN'T BE DEMONSTRATED AS IMPORTANT RELATIVELY. RESEARCH PRIORITIES ARE NOT THE SAME AS PUBLIC HEALTH PRIORITIES, BUT SHOULD THEY BE? I THINK THE QUESTION HAS TO BE WHY AREN'T THEY, NOT WHY ARE THEY DIFFERENT, BUT WHY AREN'T THEY. OF COURSE RESEARCH PROGRESS IS NOT ALWAYS LINEAR. WITH H.I.V., FOR EXAMPLE, THE FINDINGS OF THE INTRICACIES OF THE IMMUNE SYSTEM SUPPORTED BY NIH IN THE 70s MADE ENORMOUSLY HELPFUL THE D THE DISCOVERY OF AIDS, H.I.V. AND THE IMMUNE SYSTEM. THE DRUGS DEVELOPED FOR H.I.V., BACK WHEN I WAS IN MEDICAL SCHOOL, PEOPLE SAID THAT'S JUST A VIRUS, YOU CAN'T TREAT THAT. NOW HERE WE ARE TREATING A NUMBER OF CHRONIC VIRAL INFECTIONS, SOME OF WHICH WITH HEPATITIS AND OTHER DRUGS HAVE BENEFITED FROM DISCOVERIES WITH H.I.V. THE OPPORTUNITY FOR SUCCESS, OF COURSE, DIFFERS BETWEEN PUBLIC HEALTH PRIORITIES AND RESEARCH PRIORITIES, AND A LOT OF TIMES THE DISCOVERY OF A DRUG WHICH MAY LOOK LIKE IT HAS MINIMAL VALUE TURNS TO LARGER VALUE WHEN IT'S USED FOR OTHER AREAS. OF COURSE THERE IS THIS TIME LAG THAT WAS MENTIONED BEFORE BETWEEN RESEARCH FINDINGS, IMPLEMENTATION ON A WIDE SCALE, AND IN IN ADDITION, THERE'S A HUGE TIME LAG IN BEING ABLE TO IDENTIFY THOSE THINGS SO IT'S IMPORTANT WHEN YOU'RE THINKING ABOUT DATA SETS TO THENG I THINK FAR ENOUGH AHEAD OF TIME WHAT INFORMATION YOU WANT TO COLLECT, LOOK AT TRENDS IN FINDINGS, SO YOU'VE GOT A TIME LAG BETWEEN IMPLEMENTATION OF RESEARCH -- FROM RESEARCH TO IMPLEMENTATION, THEN YOU'VE GOT A TIME LAG IN YOUR ABILITY TO MEASURE IT AND ALSO MEASURE TRENDS IN IT. WE WERE FORTUNATE, FOR EXAMPLE, IN THE NIH-SUPPORTED STUDIES SHOWING THAT FOLIC ACID PREVENTED BIRTH DEFECTS, THAT THERE WERE SUFFICIENT NUTRITIONAL STUDIES AVAILABLE AND YOU COULD MEASURE FOLIC ACID FROM ED HANES AND OTHER THINGS TO GO BACKWARDS AND BE ABLE TO DETERMINE WHETHER FOLIC ACID WAS -- TO DO THIS, BUT THAT REALLY IS FORTUITOUS. OTHER TYPES OF THINGS ARE GOING TO REQUIRE A LOT MORE PLANNING AND A LOT OF STUDIES DON'T HAVE THOSE KINDS OF DATABASES. THE OTHER THING IS THAT RESEARCH FINDINGS ARE NOT, AS I MENTIONED WITH. 052 STUDY, ARE OFTEN NOT LIKELY TO BE IMPLEMENTED ON A WIDE SCALE, SO, THEREFORE, IF THEY'RE NOT IMPLEMENTED ON A WIDE SCALE, THEY'RE NOT LIKELY TO HAVE A POPULATION IMPACT. THEN THERE'S POLITICAL CONSIDERATIONS FOR RESEARCH PRIORITIES VERSUS PUBLIC HEALTH PRIORITIES. THE EASIEST TO UNDERSTAND POLITICAL CONSIDERATION IS THAT THE UNITED STATES IS -- I MEAN, THE NIH IS THE UNITED STATES AGENCY, AND IT HAS TO HAVE A GREAT DEAL OF PRIORITY TOWARD THE UNITED STATES HEALTH PROBLEMS. THAT ISN'T TO SAY THAT THE NIH HASN'T BEEN AN ENORMOUS CONTRIBUTOR TO GLOBAL HEALTH. BUT FOR EXAMPLE, IT HASN'T PRIORITIZED MALARIA TO THE EXTENT IT PRIORITIZES HEART DISEASE. THAT'S UNDERSTANDABLE BUT IT'S A POLITICAL CONSIDERATION. AND IT DOES PRIORITIZE MALARIA SOME. BUT IN TERMS OF COMPARISONS. BUT POLITICAL CONSIDERATIONS ALSO APPLY TO WE AT UNIVERSITIES AND OUR OWN LABORATORIES AND THE TYPES OF FUNDING THAT SHOULD BE DONE, AND I THINK THE SHYNESS OF GETTING INVOLVED IN IMPLEMENTATION RESEARCH AND THINGS BETWEEN THE INDIVIDUAL AND THE COMMUNITY, IN OTHER WORDS, REALLY STUDYING WHY THERE ISN'T MORE IMPACT OR WHY THE IMPACT ISN'T MORE DEMONSTRABLE RATHER THAN SIMPLY SAYING THERE'S A TIME LAG. I THINK A LOT OF TIMES THAT'S BECAUSE OF THE SOCIAL DETERMINANTS OF HEALTH AND OTHER FACTORS. NOW THE OTHER ISSUE IS ATTRIBUTION. THE EXISTING DATA SETS MAY BE ADEQUATE BUT THEY MAY BE INADEQUATE AND THEY'RE VERY EXPENSIVE ON A POPULATION BASIS. THE OTHER THING IS THAT MOST DATA SETS FOCUS ON DISEASE AND BIOMARKERS, NOT ON BEHAVIOR OR PREVENTION. SO FOR EXAMPLE, YOU CAN TRACK CARDIOVASCULAR MORTALITY, BUT IT'S VERY, VERY HARD TO TRACK CARDIOVASCULAR DISEASE IF PEOPLE HAVEN'T HAD AN M.I., AND IT'S VERY HARD TO ATTRACT WHO DOESN'T GET AN M.I. BASED ON NIH STUDIES. THE ABSENCE OF MORTALITY IS SOMETHING THAT IS EASILY TRACKABLE, BUT THE ABSENCE OF DISEASE IS VERY HARD TO TRACK. THEN OF COURSE THE TIME LAG I MENTIONED BEFORE, A LOT OF TIMES THE POPULATIONS THAT ARE AT HIGHEST RISK OR THE INTEREST ARE NOT ADEQUATELY SAMPLED IN THE LARGE SCALE STUDIES. AND THAT'S A BIG PROBLEM WHEN YOU'RE DEALING WITH A DISEASE OF RELATIVELY LOW INCIDENCE OR A DISEASE THAT IS FOLLOWED. THE NATIONAL CANCER INSTITUTE SEARS STUDY STRUGGLED WITH SOME OF THE ISSUES OF THE SAMPLE SIZE FOR CERTAIN AS YOU SUBDIVIDE BREAST CANCER, LOOK AT OUTCOMES, THINGS LIKE THIS. SO ITS IMPORTANT TO ANTICIPATE THE SPECIALIZED DATA NEEDS AND DO A SURVEY OF SURVEYS TO FIND OUT HOW YOU CAN SUPPLEMENT THOSE SURVEYS EARLY ON TO BEST ANTICIPATE IMPACT. THE OTHER THING IS THAT RESEARCH IS OFTEN COLLABORATIVE. THIS TAKES ME BACK TO THE BUDGET JUSTIFICATION VERSUS EVALUATION THING. I'M PART OF THE WOOD DRUF SCIENCES CENTER. WOODRUFF WAS HEAD OF COCA-COLA FOR ABOUT A HUNDRED YEARS, VERY LONG TIME. THERE ARE STATUES OF HIM EVERYWHERE, HE'S GIVEN LOTS AND LOTS OF MONEY TO EMORY AND EVERY PLACE ELSE IN THE ATLANTA METRO AREA. ONE OF THE SIGNS OF HIM SAYS IT'S AMAZING WHAT YOU CAN ACCOMPLISH IF YOU DON'T BOTHER TAKING THE CREDIT. THE IRONY, OF COURSE, IS THAT STATEMENT WAS MADE 100 YEARS EARLIER BY LOTS OF OTHER PEOPLE AND HE'S BEEN GIVEN CREDIT FOR THE STATEMENT. [LAUGHTER] BUT IT'S TRUE. THE PROBLEM, OF COURSE, IS WHEN YOU GET THE BUDGET JUSTIFICATION AND PROGRAM JUSTIFICATION, YOU HAVE TO GO AGAINST THAT. OF COURSE RESEARCH IS OFTEN COLLABORATIVE, AND SPONSORSHIP. AND WHEN YOU LOOK AT THINGS LIKE AIDS, IT'S REALLY OBVIOUS. TAKING DRUGS LIKE AZT, THE FIRST ANTI-RETROVIRAL INVENTED BY JEROME HORWITZ, THE MICHIGAN CANCER FOUNDATION IN 1970, PROVEN TO BE TOO TOXIC FOR CANCER, A STUDY BY BOB AND SAM BRODER AT THE NIH ALONG WITH A COMPANY THAT USED TO BE CALLED BURROWS WELCOME, AND THEY SHOWED THAT IT WORKED. AT LEAST A LITTLE BIT. THAT KIND OF WAS THE REVOLUTION THAT BEGAN IN THERAPY. A LOT OF WHICH WAS DONE BY PHARMACEUTICAL COMPANIES BUT CERTAINLY IN PARTNERSHIP WITH NIH AND NIH-SPONSORED WORK. SO WHO GETS CREDIT? MAYBE EVERYBODY GETS CREDIT OR MAYBE IT'S MORE DIFFICULT, BUT OF COURSE IT'S EXTREMELY IMPORTANT. THE OTHER THING IS SUCCESS IN IMPROVING HEALTH IS MULTIFACTORIAL. IT INVOLVES SOCIAL DETERMINANTS OF HEALTH, IT INVOLVES ASSURANCE AND ACCESS TO A LOT OF THE RESPONSIBILITIES. I REMEMBER WHEN -- THIS IS AN ASIDE, I GUESS, BUT WHEN H.I.V. -- WHEN AIDES WAS FIRST COMING ALONG, WHEN WE WERE INVESTIGATING PERSONS WITH HEMO FEEHEEHEMOPHILIA, ABOUT HALF OF WHOM IN THE UNITED STATES DIED DUE TO AIDS, AND WE FOUND THAT AFRICAN-AMERICAN HEMOPHILIACS HAD HALF THE RATE OF AIDS AS WHITE HEMOPHILIACS. THEY WERE LUCKY BECAUSE THEY HAD POOR ACCESS TO HEALTHCARE AND THEY DIDN'T GET THE CONCENTRATES WHICH ULTIMATELY INADVERTENTLY LED TO THE DEATH OF THE OTHERS. BUT OF COURSE THEY WEREN'T BENEFITING FROM THE ADVANCES OF THE FACTOR CONCENTRATES, AND I WOULD BET THEY STILL DON'T, AS A MATTER OF FACT, WITH THE NEW CONCENTRATES. SO EXAMPLES FROM AIDS, MULTIDISCIPLINARY -- THE RESEARCH DONE BY THE FOUNDATIONS AND INDUSTRY, THE BREAKTHROUGHS HAVE BEEN PROBLEM DEFINITION, ETIOLOGY, THERAPY AND IMPACT ON POPULATIONS. AND THE DATA AVAILABLE FOR PUBLIC HEALTH ARE REALLY BEST IN THE U.S. AND DEVELOPED COUNTRIES. NOT AS GOOD OUTSIDE THE U.S., WHERE MOST OF THE DISEASE BURDEN IS, AND IT'S BEST FOR TREATMENT OF MORTALITTREATMENT/MORTALITY AND REALLY LESS ADD QUAD FOR PREVENTION DUE TO H.I.V. PREVENTION EFFORTS. THE DATA FROM DEVELOPING COUNTRIES IS VERY LIMITED. SO ANYWAY, THAT'S JUST A FEW EXAMPLES TO GET PEOPLE TALK ABOUT REAL DATA. >> OUR NEXT SPEAKER IS DR. IRMA ARISPE, DIRECTOR OF THE OFFICE OF ANALYSIS AND EPIDEMIOLOGY AT THE NATIONAL CENTER FOR HEALTH STATISTICS, OFFICE OF SURVEILLANCE, EPIDEMIOLOGY AND LABORATORY SERVICES, ALL AT THE CENTERS FOR DISEASE CONTROL AND PREVENTION. ANOTHER ARM OF THE ATLANTA GROUP HERE. SO THANK YOU VERY MUCH FOR YOUR PRESENTATION, AND LOOK FORWARD TO HEARING IT. >> THANK YOU VERY MUCH. THANK YOU, DR. CURRAN, AND I APOLOGIZE FOR COUGHING SO MUCH DURING YOUR TALK. I WANT TO LET YOU KNOW THIS IS AN ONSET FROM YESTERDAY EVENING. I ALMOST PULLED OUT -- AND I REALLY DIDN'T WANT TO, SO -- BECAUSE I THINK THIS AREA IS SO IMPORTANT AND I'M FEELING SO POSITIVE BEING BACK, I REJOINED IN DECEMBER OF LAST YEAR, AND I'M JUST THRILLED AT THE NUMBER OF COLLABORATIONS THAT WE HAVE WITH NIH AND IN THE GROWTH AND PUBLIC HEALTH DATA CAPABILITY AND OUR FUTURE IN WORKING TOGETHER, SO I HOPE I CAN MAKE IT THROUGH WITH A MINIMUM OF COUGHING AND PLEASE EXCUSE ME. I'VE WORKED A LITTLE LESS THAN HALF MY CAREER IN THE LEGISLATIVE BRANCH AND A LITTLE MORE THAN HALF IN THE EXECUTIVE BRANCH. I'VE HAD A LIFELONG INTEREST IN LOOKING AT HOW DATA INFORM POLICY, AND HOW TO MAKE DATA USEFUL AND HOW TO USE IT TO IMPROVE CLINICAL PRACTICE. SO RATHER THAN DOING THE PRESENTATION OF USING PUBLIC HEALTH DATA TO EVALUATE BIOMEDICAL SCIENCE, WHICH TO ME IS KIND OF A LINEAR APPROACH, AS DR. CURRAN TOLD YOU, IT'S MUCH MORE OF AN INTERRELATED APPROACH, SO THIS IS KIND OF A MODEL I THOUGHT MIGHT BE WORTH TRYING OUT TO DEVELOP A LITTLE BIT. THIS IS A SILK ROAD, NIC ACTUALLY FEDERAL INVESTMENT AND RESEARCH HAS INTERDEPENDENT AND SYNERGISTIC. THIS IS THE SILK ROAD, IT'S A NETWORK OF COMPLEX INTERLINKING TRADE ROUTES CONNECTING THE EAST TO THE WEST, NORTH AFRICA TO CHINA, WENT BY LAND AND BY SEA, AND IT WAS ACTUALLY THE PASS THROUGH WHICH THE BUBONIC PLAGUE WAS TRANSMITTED, BUT ASIDE FROM THAT, THERE WERE MANY, MANY VERY POSITIVE ASPECTS OF THE SILK ROAD, IN ADDITION TO SELFING AS THE INTERSECTION OF CIVILIZATION, IT'S ACTUALLY THIS EXCHANGE WAS A SIGNIFICANT FACTOR IN THE EXPLORATION OF NEW TERRITORIES IN THE TRANSMISSION OF TECHNOLOGIES, RELIGIOUS THEORIES, THEORETICAL PRINCIPLES, CULTURAL EXCHANGE, AND I THINK THE MODEL THAT I WANT US TO THINK ABOUT IN TERMS OF BIOMEDICAL SCIENCE AND PUBLIC HEALTH IS THE INTERCONNECTEDNESS AND THE BACK AND FORTH THAT DR. CURRAN TALKED A LITTLE BIT ABOUT AND I'D LIKE TO TALK ABOUT TODAY. SO ON THE LEFT SIDE IS A MODEL THAT I KIND OF DRK THAT I LESHED WHEN I WAS MUCH YOUNGER, WHEN I WORKED AT THE AGENCY FOR HEALTHCARE POLICY AND RESEARCH WHEN IT WAS FIRST ESTABLISHED, CHARGED FROM TAKING EVIDENCE FROM RANDOMIZED CONTROL TRIALS AND DEVELOPING CLINICAL PRACTICE GUIDELINES, EVALUATION CRITERIA, PERFORMANCE MEASURES, CHANGES IN PRACTICE, IMPROVEMENTS AND OUTCOMES. SO A LINEAR MODEL WOULD TAKE US FROM RCTs TO A BODY OF EVIDENCE THAT INFORMS CLINICAL RECOMMENDATIONS TO THE DIFFUSION OF THESE RECOMMENDATIONS TO IMPROVED CLINICAL PRACTICE. AND THAT CERTAINLY HAPPENS. TAKING THE MODEL AND EXPANDING IT WESTWARD AS DR. CURRAN ALSO DID FROM HIS PRESENTATION, FROM THE BEDSIDE TO THE COMMUNITY. THE PUBLIC HEALTH MODEL CONCERNS ITSELF WITH A POPULATION HEALTH SIDE. THESE PATHWAYS, BY THE WAY, WOULD GO IN ALL DIRECTIONS. IN THE PUBLIC HEALTH MODEL, POPULATION HEALTH AT THE VERY TOP BULLET IS BOTH EYE POTASSIUM CYST GENERATING AND EVAL. WATIVE IN NATURE. COLLECTION OF DATA FROM AND ABOUT HEALTHCARE PROVIDERS, FROM AND ABOUT PATIENTS AND POPULATIONS, HELP US UNDERSTAND THE EPIDEMIOLOGY AND THE CONDITIONS THAT WE'RE OBSERVING IN SYSTEMS OF CARE, IN HUMAN SYSTEMS AND IN COMMUNITIES. AND THEN FINALLY THROUGH SURVEILLANCE IN AN ONGOING WAY, WE LOOK AT CHANGE OVER TIME AND WE USE POPULATION-BASED DATA TO PROVIDE FEEDBACK TO POPULATIONS ABOUT THEIR OWN PREVENTIVE BEHAVIOR TO PROVIDERS ABOUT THEIR PRACTICE AND TO THE RESEARCH ENTERPRISE TO FURTHER THAT. SO I AM FROM THE NATIONAL CENTER FOR HEALTH STATISTICS. WE MONITOR THE NATION'S HEALTH BY COLLECTING, ANALYZING AND DISSEMINATING HEALTH DATA, LOOKING ACROSS TIME, POPULATIONS, PROVIDERS AND GEOGRAPHIC AREA, WE IDENTIFY HEALTH PROBLEMS, RISK FACTORS AND DISEASE PATTERNS AND WE USE THIS DATA, WORKING WITH YOU AND THE OTHER AGENCIES OF THE PUBLIC HEALTPUBLICHEALTH SERVICE, AS WELL AS OTHERS, TO INFORM ACTIONS AND POLICIES TO IMPROVE THE HEALTH OF THE AMERICAN PEOPLE. MCHC HAS A VERY WIDE NUMBER OF DATA SYSTEMS, FAR TOO MANY TO REALLY DESCRIBE. I BROUGHT THIS SLIDE TO GIVE YOU A FLAVOR FOR THE ROBUST NATURE OF SOURCES OF DATA, AND WHAT WE USE THESE DIFFERENT SOURCES FOR IS TO TRIANGULATE FINDINGS, TO BUILD AN UNDERSTANDING OF THE ENVIRONMENT IN WHICH HEALTHCARE OCCURS AS WELL AS HET CARE ITSELF. WE WORK WITH THE STATES IN A A FEDERAL-STATE PARTNERSHIP TO COLLECT DATA ON BIRTH AND DEATH RECORD, WE COLLECT DATA VIA PERSONAL INTERVIEWS IN THE HOME AND BY TELEPHONE, WE CONDUCT PHYSICAL EXAMINATIONS AND LABORATORY TESTING IN MOBILE EXAMINATION CENTERS. WE EXAMINE PAPER AND ELECTRONIC MEDICAL RECORDS FROM A VARIETY OF HEALTHCARE ESTABLISHMENTS AND PROVIDERS, AND WE CONDUCT INTERVIEWS WITH HEALTHCARE PROVIDERS ABOUT THE CARE THEY DELIVER AND THE ENVIRONMENT IN WHICH THEY PRACTICE MEDICINE. SO THAT'S A LITTLE BIT ABOUT THE DATA SOURCES THAT YOU'RE GOING TO BE LOOKING AT. I'D LIKE TO QUICKLY GO THROUGH SOME FINDINGS AND ARE THESE FINDINGS CONCLUSIVE EVIDENCE THAT NIH RESEARCH RUTS IN OUTCOMES MAYBE NOT IN LABORATORY WAY BUT I THINK IT CAN CERTAINLY BE SAID THAT THESE FINDINGS WOULD NOT HAVE OCCURRED WITHOUT THE SCIENTIFIC DEVELOPMENT SPONSORED BY NIH, AND WE WOULD NOT HAVE STUDIED THEM WITHOUT THE ANTICIPATORY DATA COLLECTION COLLABORATIONS THAT WE HAVE WITHIN NIH. SO WHAT YOU SEE IS THAT PUBLIC HEALTH RESEARCH BOTH INFORMS BIOMEDICAL RESEARCH AND IS INFORMED BY IT THROUGH THE DEVELOPMENT OF DATA ITEMS. THESE ARE DATA FROM THE NATIONAL VITAL STATISTICS SYSTEM WHICH IS A FEDERAL-STATE PARLT NER SHIP. THESE ARE AGE-ADUSTED DEATH RATES FOR SELECTED CAUSES OF DEATH BY ALL AGES BY SEX. LOOKING AT BOTH MEN AND WOMEN COMBINED, BETWEEN 2000 AND 2010, RIGHT NOW THAT'S THE COMPLETE YEAR FOR WHICH ALL STATE DATA ARE AVAILABLE, AND I'M GOING TO TALK A LITTLE BIT MORE ABOUT A NUMBER OF INNOVATIONS WE HAVE IN VITAL STATISTICS. AT THE END OF THE PRESENTATION. BETWEEN 2000 AND 2010, AGE-ADJUSTED HEART DISEASE DEATH RATES DECLINED 30%, AND THE AGE-ADJUSTED CANCER DEATH RATE DECLINED BY 13%. WHAT YOU SEE IN THE NEXT SLIDE IS SOMETHING THAT WE REFER TO AS THE CANCER CROSSOVER. SO AS I SAID, 2010 IS THE MOST RECENT YEAR FOR WHICH WE HAVE COMPLETE DATA, BUT BASED ON OUR MODELING AND THE EARLY -- WE BELIEVE THE CROSSOVER HAS ALREADY OCCURRED. DEATHS DUE TO HEART DISEASE HAVE DECLINED SUCH THAT HEART DISEASE IS NO LONGER THE LEADING CAUSE OF DEATH IN THIS COUNTRY. IN 2010, 24% OF DEATHS WERE DUE TO HEART DISEASE AND 23% WERE DUE TO CANCER. THESE ARE DATA FROM THE NATIONAL HEALTH AND NUTRITION EXAMINATION SURVEY. THEY ARE DATA FROM PHYSICAL MEASUREMENTS, WE COLLECT BLOOD, URINE AND OTHER BIOMEDICAL DATA AS WELL AS INFORMATION FROM INTERVIEWS OF PATIENTS WHO ARE IN MOBILE EXAMINATION CENTERS. THE ENHANES IS A VERY INNOVATIVE SURVEY DESIGN, WE HAVE MOBILE SURVEY CENTERS ACROSS THE COUNTRY, BUT BECAUSE IT IS SO EXPENSIVE, WE HAVE TO COMBINE YEARS OF DATA IN ORDER TO PROJECT NATIONALLY. SO WHAT YOU SEE HERE IS FROM THE PERIOD OF 1988 THROUGH '94, TO THE PERIOD 2000 TO 2010, THE PERCENTAGE OF ADULTS 20 AND OVER WITH HIGH SEAR OWM CHOLESTEROL LEVELS DECLINED FROM 20 TO 14%. WHAT YOU SEE ON THE RIGHT IS THAT THE USE OF CHOLESTEROL LOWERING DRUGS IN THE PAST 30 DAYS HAS INCREASED. THUS WE SEE THAT THIS IS REFLECTIVE OF IMPROVEMENT IN OUR UNDERSTANDING OF DISEASE LEADING TO DEVELOPMENTS THAT IMPROVE HEALTH OF THE POPULATION. THE DATA THAT I SHOWED PREVIOUSLY WENT FROM 1950 TO 2010 OR FROM 1988 TO 2010, AND IF YOU'RE IN POLICY, PEOPLE LOOK AT THAT AND THINK THAT THAT'S VERY OLD DATA, THAT IT TAKES A VERY LONG TIME TO PRODUCE DATA AND, IN FACT, DR. CURRAN SPOKE ABOUT THAT GOOD DATA TAKE TIME AND THAT IS TRUE, BUT THERE ARE INSTANCES IN WHICH YOU CAN SEE VERY RAPID AND PRONOUNCED CHANGES IN CLINICAL PRACTICE, AND THIS IS WHAT YOU SEE HERE. THIS IS FROM THE NATIONAL AMBU LAMBULATORY -- THE DATA ARE REALLY OUT RELATIVELY QUICKLY AND YOU CAN SEE CHANGES IN PHYSICIAN BEHAVIOR. SO WHAT YOU SEE IS A PRECIPITOUS DECLINE FOLLOWING NHLBI, STOPPING THE TRIAL AND THE 2003 FINDINGS ON DEMENTIA AND COMBINATION THERAPY. SO I ACTUALLY PUT THE SLIDE UP, I REALIZE IT'S A VERY OLD SLIDE BUT I THINK IT IS AN IMPORTANT EXAMPLE OF HOW INFORMATION REACHING THE PUBLIC CAN HAVE AN IMMEDIATE EFFECT ON PRACTICE. THIS IS AN EXAMPLE OF DATA THAT DR. CURRAN DISCUSSED, AND IT'S A VERY GOOD CASE OF TRYING LAITION OF DATA FROM BROAD MULTIFACETED LONG TERM SURVEILLANCE AND THE COMBINATION OF THE EFFORTS OF PUBLIC HEALTH SERVICE AGENCIES. NIH RESEARCH EE LEWIS DATES THE ROLE OF FOLATES AND FOLIC ACID, THE MECHANISM BY WHICH IT WORKS, FOLLOWING NIH RESEARCH, THIS RESEARCH ESTIMATED THAT BETWEEN 20 TO 50% OF SPINA BIFIDA CASES COULD BE PREVENTED THROUGH TAKING FOLIC ACID, BOTH NIH AND CDC ISSUED CLINICAL RECOMMENDATIONS AND FDA ISSUED REGULATIONS. DATA FROM THE NATIONAL HEALTH AND NUTRITION EXAMINATION SURVEY MONITORED MEDIAN FOLATE LEVELS AND DATA FROM VITAL STATISTICS SHOW PRETTY DRAMATIC DROPS OF RIGHTS OF SPINA BIFIDA. THIS IS KIND OF ANOTHER EXAMPLE OF TRY ANGULATION OF DATA, TO TRACK HYPERTENSION. SO WE DEPEND ON FINDINGS FROM NIH RESEARCH AS WELL AS THE RECOMMENDATIONS, WE USE THESE DATA TO SET TARGETS FOR NATIONAL MONITORING NETWORKS SUCH AS HEALTHY PEOPLE AND WE TRACK OUR PROGRESS THROUGH SURVEYS SUCH AS THE NATIONAL HEALTH AND NUTRITION EXAMINATION SURVEY. I THINK THIS IS THE LAST EXAMPLE, THE ONE DR. CURRAN HAD REFERRED TO. EXPOSURE TO TOBACCO SMOKE, THE N IT IS HANENHANES 3 WAS THE FIRST NATIONAL EFFORT TO MEASURE SECONDHAND SMOKE AND FOUND 84% OF NON-SMOKER IT BEEN EXPOSED TO SECONDHAND SMOKE. THAT IS REALLY A VERY DRAMATIC DROP IN BIOMARKERS FOR EXPOSURE TO SECONDHAND SMOKE BASE OND THESE FINDINGS. THESE LAST TWO EXAMPLES ARE MODERN POLICY EXAMPLES. THIS IS -- I ADDED THEM FOR A COUPLE OF REASONS. ONE, AS A RESULT OF THE AMERICAN RECOVERY AND REINVESTMENT ACT, THE DEPARTMENT RECEIVED ABOUT $26 BILLION TO PROMOTE AND EXPAND THE USE OF HEALTH INFORMATION TECHNOLOGY AND WHAT YOU SEE IS DRAMATIC INCREASES IN THE USE OF ANY ELECTRONIC HEALTH RECORD, AND AS WELL AS BASIC SYSTEMS, WE CONTINUE TO MONITOR PROGRESS TOWARD MEANINGFUL USE USING OUR NATIONAL AMBULATORY MEDICAL CARE SURVEY. SO THIS IS BOTH AN EXAMPLE OF CHANGES IN PRACTICE AND CHANGES IN THE AVAILABILITY OF DATA BECAUSE AS WE PARTNER WITH THE PROVIDER COMMUNITY TO USE THESE DATA, WE HAVE A DATA SUPPLY WITH WHICH WE CAN WORK TO EVALUATE HEALTH PRACTICES AND POLICIES AND WORK WITH YOU TO DO THAT. THE LAST EXAMPLE IS AN IMMEDIATE CHANGE FOLLOWING THE PASSAGE OF THE AFFORDABLE CARE ACT. THE HEALTH INTERVIEW SURVEY IS NOW RELEASING QUARTERLY ESTIMATES OF HEALTH INSURANCE COVERAGE. THE ACA HAD A PROVISION ALLOWING YOUNG ADULTS AGE 19-25 TO BE COVERED UNDER THEIR PARENTS EMPLOYER SPONSORED OR INDIVIDUALLY PURCHASED HEALTH INSURANCE SO WE SEE AN IMMEDIATE DROP IN UNINSURED AND INCREASE IN PRIVATE COVERAGE. SO ACE MENTIONED, WE HAVE A NUMBER OF PARTNERSHIPS BETWEEN NIH AND NCHS. IN FACT, I COUNTED THAT WE HAVE COLLABORATIONS WITH 18 NIH CIOs. THE NATURE OF OUR COLLABORATION IS AT LEAST FOUR FOLD. YOU ARE COLLABORATORS ON SURVEYS WITH US SUCH AS THE NHAINSHANES AS WELL AS SUPPLEMENTS ON THE NATIONAL INTERVIEW HEALTH SURVEY. THE CANCER SUPPLEMENT, THERE HAVE ALSO BEEN SUPPLEMENTS ON ASTHMA, COMPLEMENTARY MEDICINE, CHILD MENTAL HEALTH, VISION, HEARING LOSS, PHYSICAL ACTIVITY AND SLEEP AND MANY OTHER AREAS. OUR STATISTICIANS AND SCIENTISTS WORK COLLABORATIVELY WITH YOURS ON METHODOLOGICAL PARTNERSHIPS, FOR EXAMPLE, THE DESIGN OF THE NATIONAL CHILDREN'S STUDY, AS WELL AS THE DESIGN OF ITEMS AND SURVEY MODULES FOR NCHS DATA SYSTEMS. A VERY IMPORTANT SOURCE OF COLLABORATION BETWEEN OUR TWO AGENCIES IS THE HEALTHY PEOPLE ENDEAVOR. HEALTHY PEOPLE -- NCHS IS THE STATISTICAL CONSULTANT TO THE HEALTHY PEOPLE EFFORT, SO WE COLLECT DATA, WE MONITOR PROGRESS TOWARD GOALS, BUT WE DO NOT DO GOAL SETTING. WE DEPEND OFTEN TH DEPEND ON THE SCIENTIST S OF NIH TO HELP US DO THAT. SOW SET THE TARGETS, EXAMPLES INCLUDE HEART DISEASE, STROKE, ASTHMA, MANY, MANY OTHERS. FINALLY, WE WORK WITH YOU ON NATIONAL REPORTING EFFORTS, TWO EXAMPLES OF WHICH ARE HEALTH HEALTH U.S. AIMED AT TREND ANALYSIS OF PUBLIC HEALTH DATA AS WELL AS THE AGING FORUM AND THE CHILDREN'S FORUM, WHICH ARE INTERDEPARTMENTAL EFFORTS. I WANT TO TURN BACK TO THE CONCEPT OF THE SILL ROAD BECAUSE I THINK WE HAVE -- THE SILK ROAD BECAUSE I THINK WE HAVE WONDERFUL OPPORTUNITIES, OPPORTUNITIES THAT CAME TO US THROUGH THE ERA AN OF THE AFFORDABLE CARE ACT. WE HAVE NEW INTERSECTIONS AND NEW PARTNERSHIP OPPORTUNITIES. IN THE AREA OF VITAL STATISTICS, WE ARE USING FUNDS FROM THE PUBLIC HEALTH AND PREVENTION FUND TO HELP THE STATES MOVE MORE TOWARD ELECTRONIC BIRTH AND DEATH RECORDS. THIS WILL ENABLE US TO HAVE MORE FASTER TRANSMISSION FROM THE STATES, BETTER STANDARDIZATION, AND IMPROVED QUALITY OF BIRTH AND DEATH DATA. WE ARE ALSO PARTNERING WITH DOD AND V.A., SO THAT FOR THE FIRST TIME, WE WILL HAVE A SYSTEM THAT COVERS THE BIRTHS AND DEATHS OF ALL PEOPLE WITHIN THE UNITED STATES AND BORN IN THE UNITED STATES. IN THE AREA'S HEALTHCARE, THESE ARE OUR ESTABLISHMENT AND PROVIDER SURVEYS, THE NAMSIS, HAMSIS WHICH COVERED OUTPATIENT SETTINGS AND OUR LONG-TERM CARE SURVEYS. FOR THE FIRST TIME IN MANY, MANY YEARS, WE HAVE BEEN ABLE TO EXPAND THE SAMPLE SIZE TO YIELD STATE ESTIMATES FOR THE LARGER STATES. THAT IS SOMETHING THAT WE DIDN'T EVEN REALLY DREAM OF WHEN I WAS AT NCHC BACK IN 1999 BUT IT'S SOMETHING WE HAVE BEEN ABLE TO DO NOW. WE HAVE ALSO BEEN ABLE TO ADD NEW ITEMS TO OUR SURVEY TO COLLECT THINGS, FOR EXAMPLE, SUF AS EVIDENCE-BASED PREVENTIVE SERVICES SO THAT WE CAN CONTINUE TO MONITOR AS WE HAVE CHANGES IN CLINICIAN PRACTICE. THE NHANES HAS UNDERTAKEN AT LEAST THREE NEW INNOVATIONS THAT I'D LIKE TO TOUCH ON, IF ONLY BRIEFLY. THE 24-HOUR URINE DATA COLLECTION PILOT, THE HEALTH MEASURES AT HOME PILOT, IN WHICH WE ARE VALIDATING THE METHODOLOGY USED BY THE NHANES FOR USE WITH THE NATIONAL HEALTH INTERVIEW SURVEY, AND THIS IS A TREMENDOUSLY POWERFUL INITIATIVE BECAUSE IT ENABLES US, AS I MENTIONED, THE -- WE TAKE A SAMPLE OF 5,000 PEOPLE A YEAR, AND WE HAVE TO COMBINE IN ORDER TO GET NATIONAL ESTIMATES BUT THE HIS IS A MUCH BIGGER SAMPLE AND SO WE HAVE BEEN ABLE TO COLLECT TESTING, PHLEBOTOMISTS AGAINST THE DATA COLLECTORS THAT CONDUCT THE HIS TO ENABLE US TO COLLECT SOME BIOLOGICAL MEASURES FOR NHANES WHICH WE ARE IN THE PROCESS OF TESTING NOW. WE ALSO HAVE -- THE NHANES COLLECTS SPECIMENS FOR FUTURE RESEARCH, WE HAVE A DNA BANK WHICH I CAN TALK ABOUT JUST BRIEFLY IF YOU'D LIKE TO. IT'S NOT AN AREA THAT I HAVE A GREAT DEAL OF EXPERTISE, BUT I DO THINK IT'S VERY IMPORTANT TO POINT OUT THAT THAT EXISTS, IT IS USED BY NIH, AND THAT WE HAVE TREMENDOUS POTENTIAL FOR FUTURE COLLABORATIONS. THE FINAL AREA IS THE HEALTH INTERVIEW SURVEY, AND WE HAVE NEW MEASURES FOR LBGT HEALTH AND MORE STATE LEVEL ESTIMATES AS WELL AS MANY NEW ITEMS RELATED TO THE AFFORDABLE CARE ACT. THE LAST SERIES OF INTERCONNECTIONS I WANT TO TALK ABOUT IS OUR DATA LINKAGE PROGRAM. SO WE HAVE THE NHANES, WHICH IS A VERY POWERFUL DATA SYSTEM BUT VERY SMALL, AND MANY OF OUR DATA SYSTEMS HAVE BEEN CROSS-SECTIONAL IN NATURE. THE HPPA BOTH PROTECTED HEALTH INFORMATION AND FOR US OPENED A NUMBER OF NEW DOORS THAT ENABLES US TO PARTNER WITH OTHER DATA SYSTEMS SO THAT WE COULD PROTECT HEALTH INFORMATION AND EXPAND THE ANALYTIC UTILITY OF OUR DATA SYSTEM. SO THROUGH OUR DATA LINKAGE PROGRAM, WE CONNECT DATA, FOR EXAMPLE, FROM THE NHANES TO THE NATIONAL DEATH INDEX TO CLAIMS DATA FROM CMS, AND WITH SUPPLEMENTAL SECURITY INCOME DATA FROM FSA. WE CREATE THESE THINGS CALLED LINK WAJ FILES. ONE THING THAT'S IMPORTANT TO TELL YOU, AGENCY OPERATE UNDER DIFFERENT LEGISLATIVE AUTHORITIES WITH RESPECT TO COLLECTING DATA SO THE WAY WE COLLECT DATA, THE WAY YOU COLLECT DATA DOESN'T AUTOMATICALLY CONNECT. WE HAVE TO OPERATE WITHIN THE BOUNDS OF THE LAW THAT GOVERN EACH AGENCY. SO WE HAVE DEVELOPED THE LINKAGE PROGRAM TO PERMIT US TO USE VARIABLES THAT WE HAVE TO LINK DATA SO THAT WE CAN CONNECT THESE THINGS AND MAKE THEM AVAILABLE TO THE RESEARCH COMMUNITY. NOW IN ORDER TO DO THAT, WE HAVE PUBLIC USE DATA SYSTEMS THAT ALLOW RESEARCHERS TO IDENTIFY WHETHER THEY HAVE ENOUGH CASES TO DO THEIR RESEARCH, BUT THAT ACTUAL CONNECTING RESEARCH MUST BE DONE THROUGH SOMETHING CALLED OUR RESEARCH DATA CENTERS. THE OFFICE OF ANALYSIS AND EPIDEMIOLOGY HAS CONDUCTED -- THE OFFICE THAT I HEAD CONDUCTS RESEARCH ON LINKAGE METHODOLOGIES, ANALYTIC METHODS AND WAYS THAT THESE DATA CAN BE USED. RESEARCHERS OUTSIDE OF NCHS CAN ACCESS DATA THROUGH OUR SERIES OF ON SITE RESEARCH DATA CENTERS IN HYATTSVILLE, ATLANTA, IN THE 12 SECRETARY O12 CENSUS OPERATED RDCs AND WE'RE IN THE PROCESS OF ESTABLISHING AN RDC IN THE HOME BUILDING BECAUSE OF THE EXTENSIVE -- TO ACCESS THESE DATA. SO I'M GOING TO CLOSE WITH A LINK TO OUR WEBSITE. I'M REALLY THRILLED TO SAY THE FIRST TIME I WAS AT NCHS, IT WAS IN THE LATE 1990S AND I'VE COME BACK NOW IN 2013 TO SEE REALLY TREMENDOUS GROWTH BOTH IN THE DATA SYSTEMS AND IN THE NATURE OF OUR PARTNERSHIPS. SO WE LOOK FORWARD TO WORKING WITH YOU. I'D LIKE TO ECHO SOMETHING DR. CURRAN SAID WHICH IS ABOUT THE VALUE OF ANTICIPATORY PARTNERSHIPS. THE EARLIER THAT WE CAN BEGIN WORKING WITH YOU, THE FASTER WE CAN BEGIN TO COLLECT DATA THAT WILL HELP EXAMINE THE EFFECTS OF YOUR RESEARCH. THANK YOU VERY MUCH. >> THANK YOU. ARE THERE ANY SHORT QUESTIONS? OR ELSE WE WILL KEEP THE QUESTIONS FOR OUR PANEL DISCUSSION AT THE END. GO AHEAD. >> THE INCREASE IN THE NUMBER OF -- OR THE DECREASE IN THE NUMBER OF UNINSURED, ARE THERE DATA THAT WOULD SUGGEST THERE'S ALSO BEEN A DECREASE IN NUMBER OF DEATHS OR OTHER INDICATORS OF IMPROVED HEALTH DUE TO THE IMPROVED ACCESS? >> I THINK THAT'S GOING TO DEFINITELY BE THE FUTURE OF EVALUATING THE ACA, SO THE PREMISE IS THAT INCREASING INSURANCE WILL INCREASE ACCESS, SO I THINK BY TRIANGULATING DATA, DOES GIVING PEOPLE HEALTH INSURANCE ENABLE THEM TO GET THE ACCESS TO HEALTHCARE TO IMPROVE THEIR OUTCOMES. I THINK THE ABILITY TO EXPAND SAMPLES TO LOOK AT VULNERABLE POPULATIONS, THAT IS WHERE THE FUTURE REALLY IS. IT'S SORT OF BACK TO THE FUTURE, THE ISSUE WE'RE STUDYING NOW, AND POLICY IS GOING TO BE ABOUT ACCESS. >> IT WAS A VERY GOOD STUDY IN OREGON RECENTLY THAT THAT LOOKED AT THE DIFFERENCE BETWEEN PEOPLE THAT PUT ON MEDICAID VERSUS OTHERS WHO WERE NOT AND THE OUTCOMES. IT WAS A SHORT STUDY BUT VERY INTERESTING. I PRESUME YOU'LL CAPITALIZE ON THAT KIND OF DATA. >> IT IT IS A TREMENDOUS MARKER. IT'S A VERY IMPORTANT VARIABLE TO COLLECT. WE HAVE BEEN WORKING WITH THE CENSUS ON EXAMING -- DEVELOPING ITEMS THAT WILL GO INTO THE FIELD IN DECEMBER ABOUT THE EXCHANGES, SO WE'LL BE ABLE TO FIND OUT FROM PEOPLE IF THEY GOT THEIR HEALTH INSURANCE THROUGH AN EXCHANGE. IT'S GOING TO BE VERY DIFFICULT BECAUSE PEOPLE'S UNDERSTANDING WHAT IF AN EXCHANGE IS, I THINK CMS CALL IT IS A MARKETPLACE AND PEOPLE'S KNOWLEDGE ABOUT WHAT KIND OF COVERAGE THEY HAVE, IT'S VERY DIFFICULT TO CAPTURE THAT FOR PEOPLE. BUT I THINK IT IS TREMENDOUSLY IMPORTANT TO DO SO. >> SHORT COMMENT, JIM? >> ALSO AN EXAMPLE OF SAMPLE SIZE ISSUE. MOST DEATHS OCCUR IN THE MEDICAID AGE POPULATION. THIS PARTICULAR STUDY WAS PEOPLE WHO REMAIN ON THEIR PARENTS' INSURANCE, THAT'S WHY THEY'RE MAINTAINED 19-25-YEAR-OLDS, VERY LOW MORTALITY RATES, SAMPLE SIZE WOULD HAVE TO BE HUGE IN ORDER TO ANTICIPATE SOMETHING TO -- >> SURE. >> OUR NEXT PRESENTER IS BARBARA ENTWISLE, VICE CHANCELLOR FOR RESEARCH AND KENAN DISTINGUISHED PROFESSOR UNIVERSITY OF NORTH CAROLINA CHAPEL HILL. SHE'S BEEN THERE SINCE I GUESS 1985, WITH HUGE IMPACT IN MANY, MANY FIELDS OF MEDICINE, SOCIOLOGY, STATISTICS, AND RESEARCH. SO BARBARA, WE'RE DELIGHTED TO ARE HAVE YOU HERE TODAY. >> THANK YOU. IT'S A PLEASURE. I'VE BEEN LOOKING FORWARD TO THIS REALLY HAVE ENJOYED THE DISCUSSION THIS MORNING. BEING INVITED TO SPEAK HERE TODAY IS A LITTLE BIT LIKE BEING GIVEN A PIECE OF A PUZZLE AND NOT KNOWING WHERE YOU SIT UNTIL YOU GET ON CAMPUS, SO I'VE BEEN FIGURING THAT OUT ALL MORNING. AS YOU HEARD, I'M VICE CHANCELLOR FOR RESEARCH. MY COLLEAGUE, DR. ROPER, WHEN I GOT THIS INVITATION, I ASKED HIM, YOU KNOW, WHAT WOULD HE RECOMMEND AND HIS ADVICE IS I SHOULD THINK OF THIS THE SAME WAY I THINK OF MY OWN JOB IN TERMS OF TRYING TO DEMONSTRATE VALUE. I'M VICE CHANCELLOR FOR RESEARCH AT A PUBLIC RESEARCH UNIVERSITY. HOW DO WE DEMONSTRATE VALUE TO THE STATE OF NORTH CAROLINA? I ALSO HAVE TO SAY I REPRESENT A CONSORTIUM OF PEOPLE LIKE MYSELF ENGAGED IN THIS PROCESS. IT'S AN OFFSHOOT OF SOME OF THE STAR METRICS WORK AND I'LL MAKE SOME NOTE OF THAT. I HAVE A PARTICULAR DISCIPLINARY PERSPECTIVE, AND SO YOU'LL HEAR A BIT ABOUT THAT TOO. THERE'S SOME THINGS I COULD HAVE TALKED ABOUT TODAY, IT'S REALLY KIND OF HARD TO DECIDE SO I HOPE I PICKED IT RIGHT AND THAT THIS WILL MAKE FOR AN INTERESTING CONCLUSION TO THE MORNING. THIS IS YOUR MISSION. YOU KNOW THIS ALREADY. BUT IT WAS A GOOD PLACE FOR ME TO START IN THINKING ABOUT WHAT IT IS THAT THE IMPACT WAS. AND THERE'S AN INTERESTING THING HERE. THE MISSION TALKS ABOUT ENHANCING HEALTH, LENGTHENING LIFE AND REDUCING ILLNESS AND DISABILITY AND YET ALL THE GOALS ARE AROUND THE CREATION OF KNOWLEDGE AND WORKFORCE, AND WE'VE ALL BEEN SORT OF NOTICING THAT AS WE'VE GONE LOU THIS MORNING AND I WILL ALSO POINT THAT OUT. HERE'S THE IMPACT STATEMENT. IF YOU'LL FORGIVE ME FOR MAKING A LITTLE QUIBBLE HERE. THE STATEMENT HERE IS THAT THANKS IN LARGE PART TO NIH RESEARCH, AMERICANS ARE LIVING NEARLY 30 YEARS LONGER THAN THEY DID IN 1900. MY QUIBBLE IS NOT WITH THE 30 YEARS LONGER. THAT'S TRUE. MY QUIBBLE IS, THANKS IN LARGE PART TO NIH RESEARCH. BECAUSE 20 OF THOSE 30 YEARS OCCURRED BEFORE THERE WERE MAJOR INVESTMENTS IN THE NIH. SO I THINK WHATEVER WE DO TO TALK ABOUT IMPACT, WE HAVE TO BE ABLE TO STAND BEHIND IT REALLY SOLIDLY AND ESPECIALLY SIMPLE STATEMENTS LIKE THIS SO APOLOGIES BUT AS A DEMOGRAPHER, I FELT THAT I NEEDED TO POINT THAT OUT. SO I WANT TO DRAW FROM A STUDY PUBLISHED IN THE PROCEEDINGS OF THE NATIONAL ACADEMY OF SIGH INS A COUPLE YEARS AGO BY KEN AND HIS COLLEAGUES, AGAIN COMING OUT OF THE DEMOGRAPHIC COMMUNITY, BUT IT ENABLES ME TO MAKE SOME GOOD POINTS. IT'S WHAT YOU MIGHT CALL A REDUCED FORM APPROACH TO THIS PROBLEM. IF YOU THINK OF FUNDS COMING IN AND LIFE EXPECTANCY COMING OUT, SO THE VERY ENDS OF THIS THAT WE'RE LOOKING AT. THIS IS, OF COURSE, AGE ADJUSTED DEATH RATES FROM MULTIPLE CAUSES AND IT'S FOR THE WHOLE POPULATION, SO THIS IS NOT JUST SPECIFIC GROUPS YET. AND THE SOLID LINE THERE IS -- LET'S SEE IF I CAN GET THIS WORKING. WELL, THE SOLID LINE THERE IS NIH FUNDS. AND THESE ARE BEING MEASURED IN TERMS OF 1,938-DOLLARS. SO THESE ARE CONSTANT DOLLARS, SO WE DON'T HAVE TO WORRY ABOUT INFLATION AS AN UNDERSTANDING OF WHAT'S GOING ON, AND YOU CAN SEE AND YOU ALREADY KNEW THIS THAT THERE HAVE BEEN SOME INTERESTING TRENDS IN THOSE FUNDS. LTS SHOWN ON THE GRAPH ARE SORT OF AGE-ADJUSTED DEATH RATES FOR PARTICULAR CAUSES. IN THIS CASE, CANCER, DIABETES, HEART, STROKE, ALL OF THESE ARE DEALT WITH IN THE ARTICLE. I'M JUST GOING TO PICK AND CHOOSE BECAUSE I WANT TO MAKE A FEW POINTS. SO THIS IS THE PICTURE, AND ACTUALLY DR. ARIS PESPE SHOWED A SIMILAR PICTURE IN WHAT SHE WAS DEMONSTRATING, BUT THE QUESTION IS, HOW CAN WE LINE THESE UP? AND THIS IS THE WORK AGAIN OF PROFESSOR MANTIN AND HIS COLLEAGUES, WHICH I THOUGHT WAS REALLY INTERESTING. THIS LOOKS LIKE A TIMELINE BUT IT'S NOT. SO AT THE BOTTOM ARE SIMPLY NIH FUNDS IN $1,938, LAGGED BY 10 YEARS. WHAT WE HAVE UP THE Y AXIS ARE THE AGE-ADJUSTED DEATH RATES. WHEN YOU MAKE THAT ADJUSTMENT, YOU CAN SEE THAT THEY TRACK REASONABLY WELL. I FELT THIS WAS A REALLY NICE STORY. IT DOESN'T GET AT WHAT'S HAPPENING, WHAT HAVE YOU DONE FOR ME LATELY, BUT IT'S A VERY NICE STORY AND YOU GET SOME INTERESTING SQUIGGLES THERE AS YOU HAVE FUNDING CUTS OR WHATEVER. SO THIS IS ONE PIECE, AN OVERALL PIECE I THOUGHT WAS KIND OF INTERESTING AND KIND OF NICE, AND I TRUST THE CITATION COUNT FOR THIS ARTICLE, IT WAS VERY LOW. SO I THOUGHT MAYBE YOU HADN'T SEEN IT. [LAUGHTER] JUST SAYING. SO THIS IS -- PART OF THE ARTICLE BREAKS IT OUT INTO PARTICULAR INSTITUTES AND CENTERS. SO THIS IS NHLBI AND WHAT WE HAVE HERE ON THE Y AXIS IS THE AGE-ADJUSTED DEATH RATE FOR HEART DISEASE. SO WE'RE TRYING TO SEE, WELL, WE HAD A NICE OVERALL PICTURE, WHAT DOES IT LOOK LIKE AT A MORE SPECIFIC LEVEL. SO HERE IT IS AGAIN, AND YOU HAVE THIS INTERESTING SORT OF QIG EL THERE. REMEMBER, THIS IS NOT A TIMELINE, THIS IS FUNDING LEVELS, AND IF THERE ARE UPS AND DOWNS IN FUNDING LEVELS, YOU'LL SEE A LITTLE BIT OF A SWITCHBACK. SO WHAT WE SEE HERE, AND WE DID SEE THIS A BIT IN THE FIRST ONE, FIRST OF ALL, AGAIN, CONFIRMATION OF THE IMPORTANCE OF LAGS. I'M REMINDING YOU THERE'S A 10-YEAR LAG HERE BUILT RIGHT IN. SECONDLY, A VERY INTERESTING THRESHOLD EFFECT THAT WE'RE SEEING, AND WE SAW THAT ALSO IN THIS OVERALL PICTURE, SO FUNDING GOES UP, THERE'S A LITTLE BIT OF A DECREASE, THEN THERE'S SOME THRESHOLD AMOUNT AND THEN WE SEE A BIGGER IMPACT. SO THE IMPACTS ARE NON-LINEAR ALSO IN THIS CASE. SO IN THIS ONE, WE HAVE THE THRESHOLD, WE HAVE ALSO THIS INTERESTING SWITCH BACK AND FUNDING GOES UP AND DOWN, AND THE -- IS OBSERVED AND THE OTHER IS A REGRE REGRESSION LINE. THERE'S NOTHING AUTOMATIC ABOUT THIS. SO PICKING AND CHOOSING AGAIN, HERE ARE THE NIDBK FUNDS AND WHAT'S GOING ON WITH DIABETES. SORRY ABOUT THAT. BUT YOU ALREADY KNEW THIS. IT RAISING SOME INTERESTING QUESTIONS. THERE'S NO APAIRNLT IMPACT YET. THERE'S BEEN SOME INVESTMENT. MAYBE THE INVESTMENT HASN'T BEEN ENOUGH YET. SO MAYBE THAT'S WHAT WE'RE SEEING, NO THRESHOLD YET, OR IS IT FAIR TO THINK ABOUT OTHER THINGS EQUAL BECAUSE IN OUR INTERPRETATION OF THE OTHER GRAPHS, WE WERE GIVING A LOT OF ATTRIBUTION TO NIH FUNDS AND WHAT WAS GOING ON, AND IN THIS CASE WE HAVE TO WONDER, WELL, MAYBE THERE ARE THINGS GOING ON THAT COUNTER THE TREND IN NIH INVESTMENTS, AND I WOULD POINT TO THE VERY OBVIOUS ONE HERE, IN TERMS OF THE PUBLIC HEALTH AND OBESITY AND THINGS OF THAT NATURE, WHICH MIGHT BE WORKING COUNTER TO THE IMPACT OF NIH FUNDING. AND RESEARCH AND ALL OF THOSE OUTCOMES. AND SO WHAT THAT RAISES, I THINK FOR US, IS THE VERY IMPORTANT QUESTION OF AS WE THINK ABOUT THESE IMPACTS, WHAT ARE WE ASSUMING ABOUT THE OTHER THINGS THAT ARE EQUAL. I WILL GET TO THAT POINT AGAIN IN THIS NEXT SLIDE. THIS IS THE FINAL SLIDE FROM THE PIECE. THIS WAS THEIR ATTEMPT TO MEASURE DEATHS AVOIDED. SO THEY DO THIS BY ASSUMING THAT STARTING IN 1950, THE AGE-ADJUSTED DEATH RATES WOULD HAVE STAYED THE SAME IN THE ABSENCE OF NIH INVESTMENTS. NOW, THAT'S A BIG ASSUMPTION. BUT IT'S A BIG ASSUMPTION EITHER WAY. MAYBE DEATH RATES MIGHT HAVE GOTTEN POSSIBLY WORSE GIVEN THE OTHER TRENDS THAT WERE GOING ON. MAYBE THEY WOULD HAVE GOTTEN BETTER ANYWAY, EVEN WITHOUT NIH INVESTMENT. WE DON'T KNOW. BUT THIS WAS THEIR ATTEMPT TO COME UP WITH SYMPTOMS AND WHAT THEY'RE SUGGESTING IS THAT OVERALL, SOMETHING LIKE 35 MILLION DEATHS WERE AVOIDED THROUGH NIH INVESTMENTS, WHICH IS A NICE NUMBER, SOMETHING TO TALK ABOUT. IT BASED ON VERY SIMPLE LOGIC. WE KNOW WE'RE MAKING ASSUMPTIONS ABOUT THINGS THAT WERE THE SAME THAT MAY NOT HAVE BEEN THE SAME, WE'RE ATTRIBUTING ALL TO NIH FUNDS. I'M A VERY SPHIS F SOPHISTICATED STATISTICIAN, MAYBE HAVING SOME SIMPLE RESULTS ARE A REAL BOONE IN THIS CASE, BECAUSE US A GET INTO THE MORE COMPLICATED PIECE, I'M NOT SURE THE PICTURE GETS A WHOLE LOT CLEARER. THAT WOULD BE ONE SUGGESTION FOR YOU. OKAY. I LOVE THIS. AND WHEN YOU THINK ABOUT THE IMPACT OF FUNDING ON HEALTH AND LONGEVITY, THERE'S THIS BIG MIDDLE IN THERE, AND THAT'S THE PART WE'VE BEEN TALKING ABOUT THIS MORNING. AND I ALSO WANT TO TALK ABOUT IT. BUT THERE IS A SENSE IN WHICH A MIRACLE OCCURS AND, TRUTHFULLY, ONE DOES AND HAS. BACK TO THE MISSION STATEMENT, THE POINT THAT'S BEEN MADE ALREADY THAT YOUR MISSIONARILY IS ABOUT THE CREATION OF KNOWLEDGE, HUMAN RESOURCES, PUBLIC ACCOUNTABILITY, THOSE THINGS ARE THE THINGS THAT YOU HAVE THE MOST CONTROL OVER. AND THE OTHER PIECE OF IT, YOU HAVE SOME CONTROL OVER, BUT THERE'S A LOT MORE TO IT THAN THAT. SO LET ME SAY A LITTLE BIT ABOUT KNOWLEDGE. I KNOW I WASN'T SUPPOSED TO BUT I DIDN'T FULLY UNDERSTAND THAT WHEN I MADE MY PRESENTATION AND I DO HAVE A FEW THINGS TO SAY. SO AS PART OF THE GROUP OF VICE CHANCE LORS FOR RESEARCH, WE'RE VERY INTERESTED IN TRYING TO UNDERSTAND IMPACT, AND OUR APPROACH TO THIS HAS BEEN REALLY TO LOOK AT PEOPLE AS INVESTMENTS INVESTMENTS. AND THINK ABOUT PEOPLE AND THINK ALSO ABOUT CONNECTIONS BETWEEN PEOPLE. SO PEOPLE ARE THE ONES DOING THE TRAINING, THEY CO-AUTHOR, THEY HAVE THE THOUGHTS, THEY IMPLEMENT THE SCIENCE, THAT'S WHERE THE INVESTMENTS ARE, AND WE OUGHT TO BE LOOKING AT OUTCOMES IN THOSE TERMS AS WELL AS PUBLICATIONS AND THE LIKE. I WANT TO PARTICULARLY DRAW ATTENTION TO THE IMPORTANCE OF SOCIAL NETWORKS AND THE CREATION DISSEMINATION APPLICATION OF NEW IDEAS. I WOULD SUBMIT TO YOU THERE'S BEEN AN AWFUL LOT OF ATTENTION ON INDIVIDUAL PROJECTS AND INDIVIDUAL INVESTIGATORS AND I THINK A LOT OF WHAT IS GOING ON SCIENTIFICALLY WHERE THE CREATIVITY COMES, WHERE THE TRANSFORMATIONAL VALUE COMES, IS IN NETWORKS OF PEOPLE BUILT OVER TIME, AND HOW ARE THESE DONE? SOME OF THESE ARE EXPLICITLY DUE TO PROJECTS. SOME OF THEM ARE DUE TO SHARED RESEARCH MATERIALS, SOME OF THEM ARE DUE TO STUDENTS AND POST POSTDOCS MOVING, SOME OF IT'S DUE TO SHARED DATA SETS. I'VE GOT AN EXAMPLE FOR YOU ON THAT. SOME OF IT'S JUST SPATIAL COLLISIONS. I MET SOMEBODY FROM MY OWN UNIVERSITY HERE THIS MORNING THAT'S SITTING OVER THERE, I DIDN'T KNOW, ALREADY WE'RE PLANNING TO DO SOMETHING THAT WILL BE VERY BENEFICIAL TO ME. OVERLOW PRESSURING PANEL MEMBERSHIP. THAT'S A VALUE, YOU KNOW, OF PEOPLE COMING IN PERSON TO STUDY SECTIONS THAT YOU OUGHT TO NOT LOSE SIGHT OF. AND OTHER KINDS OF PERSONAL CONTEXT. SO HERE'S A PICTURE OF DR. COHEN. DR. CURRAN REFERRED TO HIM. HE WAS THE WINNER OF THE 2011 BREAKTHROUGH OF THE YEAR FOR SCIENCE MAGAZINE FOR HIS WORK ON THE 052 STUDY. AND I COULDN'T BE MORE PROUD, I THINK IT'S REALLY TERRIFIC, BUT USING A DATABASE THAT WE HAVE AT UNC CALLED NC REACH, I WANT TO SHOW YOU DR. COHEN'S RESEARCH NETWORK. SO THAT'S HIM OVER THERE ON THE LEFT, HE'S THE RED DOT. AND THE GREEN LINES ARE TO ALL OF THE VARIOUS KINDS OF PEOPLE HE'S WORKING WITH THESE DAYS, THEN THE BLUE LINES ARE CONNECTIONS BETWEEN ALL OF THOSE PEOPLE. SO US A THINK ABOUT ALL OF THIS, I THINK THIS IS REALLY A MORE ACCURATE PICTURE FOR HOW SCIENCE GETS DONE, HOW BIOMEDICAL RESEARCH IS DONE. I THINK IT'S ALSO THE SOURCE OF CREATIVITY A LOT OF TIMES, NEW IDEAS, THESE INTERACTIONS BETWEEN PEOPLE, ULTIMATELY TO HELP TRANSFORM THE RESEARCH INTO APPLICATION. IF YOU HAVE A PICTURE LIKE THIS, IT'STIVE TO ATTRIBUTE A DISCOVERY IF YOU'RE GOING TO DO A SAMPLE, WHAT ARE YOU GOING TO SAMPLE, I THINK IS PART OF WHAT I WANT TO SAY TO YOU ABOUT THIS. IF YOU'RE GOING TO SAMPLE A SET OF PROJECTS, WHAT ARE YOU GETTING? JUST THE MOST RECENT ONES? AND WHO ARE YOU GETTING? IF YOU'RE INVESTING IN PEOPLE, WHO ARE YOU INVESTING IN? JUST THE INVESTIGATOR OR THE WHOLE NETWORK? THIS IS THE QUESTION I WOULD HAVE ASKED DR. ANDERSON, HAD I HAD THE OPPORTUNITY TO DO SO. WHERE ARE THOSE TWO STUDIES, THE PIONEER AWARDS AND THE NON-PIONEER AWAR DES MATCHED IN TERMS OF THEIR LOCATION IN THESE SOCIAL NETWORKS, I DON'T KNOW. JUST A QUESTION. I ALSO WANTED TO BRING UP THE ISSUE OF CREATED ANOTHER SORT OF OUT COME, DATABASE OUT COME, MENTION WAS MADE OF THE FACT THAT THERE WILL BE INCREASED FOCUS ON SHARING OF DATA. THIS IS THE NATIONAL LONGITUDINAL STUDY OF ADOLESCENT HEALTH THAT'S BEEN ONGOING FOR ALMOST TWO DECADES. IT'S BEEN EXTREMELY IMPORTANT IN THE KIND OF KNOWLEDGE THAT IT PROVIDED ABOUT THE HEALTH OF YOUNG PEOPLE AS THEY MOVE THROUGH THE TRANSITION TO ADULTHOOD, INCLUDING MOST RECENTLY DISCOVERIES ABOUT THE VERY YOUNG AGES AT WHICH NUTRITIONAL PATTERNS ARE SET IN OBESITY EMERGES, SORT HE PRECLINICAL SYMPTOMS OF VARIOUS KINDS ARE ALSO EMERGING. BUT I WANTED TO INDICATE TO YOU WHAT THE IMPACT HAS BEEN OF ADD HEALTH, I THINK IT'S LED THE FIELD IN MANY WAYS ON FIGURING OUT HOW TO BE USED BY EVERYBODY, NOT JUST THE INVESTIGATORS WHO DESIGNED IT. AND, IN FACT, THE DATA ARE AVAILABLE TO THE WHOLE FIELD ON THE SAME DAY THAT THEY'RE AVAILABLE TO THE INVESTIGATORS. THAT'S UNUSUAL. BUT YOU CAN SEE WHAT THIS HAS MEANT OVER TIME. IT'S A RESOURCE FOR OVER 10,000 RESEARCHERS WITH MORE THAN 600 FUNDED RESEARCH GRANTS, NOT ALL NIH, BUT SOME. YOU KNOW, OVER 2,000 REFEREE PUBLICATIONS, ET CETERA, ET CETERA, LOTS OF TRAINING, THESES DISSERTATIONS. I'M SORT OF GOING THROUGH THIS, FIRST OF ALL THIS CREATES A NETWORK, BUT AGAIN, IF YOU JUST PICKED ONE OF THESE FUNDED RESEARCH GRANTS, WOULD YOU UNDERSTAND ALL OF WHAT CAME BEFORE THAT AND WHAT CAME AFTER? AND IF UH-UH WERE ONLY TO LOOK AT WHAT THE PRODUCT WAS OF THE GRANT THAT CREATED ADD HEALTH, WOULD YOU KNOW ALL OF WHAT THIS IMPACT HAS BEEN? AND SO AGAIN, I'M ASKING YOU TO THINK HARD ABOUT HOW -- WHAT IT IS YOU SAMPLE AND HOW YOU DEFINE AND MEASURE IT. THE UNIT OF OBSERVATION MAY BE THE PERSON OR STUDY BUT THE UNIT OF ANALYSIS IS A DIFFERENT THING ALL TOGETHER. SO THERE'S WHAT'S CALLED A SCIENCE OF SCIENCE POLICY APPROACH, IT'S ALREADY BEEN MENTIONED HERE, BUT ONE OF THE THINGS THAT'S EXCITING ABOUT THAT IS THAT THE IDEA IS TO TRY TO USE SIGH PER TOOL CYBERTOOLS, USE DATA SETS AS THEY ARE CREATED. THE NETWORK I SHOWED YOU FOR DR. COHEN WAS DONE IN MUCH OF THAT SAME WAY, YOU COULD DO IT ON YOUR COMPUTER RIGHT NOW IN TWO SECONDS. THIS DIDN'T REQUIRE ANY SPECIAL SKILLS ON MY PART. SO TALKING NEXT ABOUT DISCOVERY TO IMPACT, THERE ARE STEPS HERE AND AS I INDICATED, THE NIH CAN CONTRIBUTE TO THESE THAT DOESN'T HAVE CONTROL OVER THESE IN THE SAME WAY THAT IT HAS CONTROL OVER THE OTHER PART. YOU NEED TO GET THE MESSAGE OUT TO THOSE WHO NEED TO KNOW. THERE'S A LOT OF KNOWLEDGE ABOUT PREVENTIVE PRACTICE THAT'S NOT USED. PEOPLE DON'T KNOW ABOUT IT. IN TERMS OF IMPLEMENTATION, SOMEHOW WE NEED TO TAKE A RESULT AND USE IT TO TRAFNS FORM POLICY, PROGRAMS, PRACTICE. ACTUALLY DR. CURRAN USED THE 052 STUDY AS AN EXAMPLE HERE. WE HAVE AN IMPORTANT RESULT, NOW WHAT? IS THERE GOING TO BE A RESULT OF THAT IN TERMS OF PUBLIC HEALTH, WHAT WOULD IT TAKE. FOR ME PERSONALLY, I'M INTERESTED IN THIS FROM THE STANDPOINT OF SCALING. HOW DO YOU SCALE UP A RESULT WHICH WAS DONE ON A SMALL SCALE, IN A PARTICULAR POPULATION, WITH VERY ENTHUSIASTIC PEOPLE USUALLY, THESE ARE SELECTED -- NOT ONLY SELECTED PARTICIPANTS BUT SELECTED PEOPLE INVOLVED IN ADMINISTERING WHATEVER THE TREATMENT IS. SO YOU TAKE THAT AND FIRST OF ALL YOU'VE GOT TO EXPAND IT TO A WHOLE GROUP OF PEOPLE WHO DON'T CARE IN THE SAME WAY, AREN'T PASSIONATE IN THE SAME WAY, AND YOU HAVE TO GET IT ACCEPTED IN CULTURES THAT MAY OR MAY NOT BE RECEPTIVE. IT'S NOT AN OBVIOUS STEP. IF YOU'LL FORGIVE ME ONE POLITICAL COMMENT, I'VE BEEN LISTENING WITH INTEREST TO THE DISCUSSION ABOUT WHAT SHOULD BE FUNDED AND WHAT SHOULDN'T BE BY SOME OF OUR CONGRESSMEN, AND THE STATEMENT IS MADE, WELL, WE WOULD BE MUCH BETTER TO TAKE ALL THAT RESEARCH IN SOCIAL MAY HAVE YOUR AND ECONOMIC RESEARCH AND PUT IT INTO CANCER RESEARCH SO WE CAN CURE CANCER. MY FEELING IS, WELL, THAT'S GREAT, ALTHOUGH YOU WOULDN'T NOTICE IT BECAUSE THERE'S SO LITTLE FUNDING RIGHT NOW FOR SOCIAL BEHAVIOR ON ECONOMIC RESEARCH, NOT SURE IT WOULD REGISTER, BUT SO YOU COME UP WITH THIS CURE FOR CANCER, THEN WHAT? IT'S GOT TO BE USED. FOR IT TO BE EFFECTIVE, IT HAS TO BE USED. SO THERE'S THIS VERY BIG STEP HERE THAT DR. CURRAN AND DR. ARISPE WERE BOTH TALKING ABOUT. IF I COULD SAY JUST A LITTLE BIT ABOUT COMMERCIALIZATION. I HAVE SOME OPPORTUNITY DO THAT IN TERMS OF MY ROLE AS VICE CHANCELLOR, AND HERE TOO, PATENTS ARE GREAT. BUT THEN WHAT? YOU CAN BUY PATENTS, IF YOU HAVE A BIG ENOUGH PATENT BUDGET, YOU CAN HAVE LOTS OF PATENTS. BUT THEN WHAT? ALL OF THIS TAKES TIME. IT'S VERY CHALLENGING, YOU CAN HAVE CHAINS OF EVIDENCE. IF YOU WANT TO KNOW WHAT THE NEXT STEP IS AFTER PATENTS, I THINK YOU OUGHT TO BE COLLECTING INFORMATION ABOUT LICENSES AND START-UPS. THERE ARE THINGS THAT CAN BE COLLECTED AND ACTUALLY AUTUMN, WHICH IS A PROFESSIONAL ORGANIZATION, IS COLLECTING IT, SO IT WOULDN'T BE HARD. BUT HERE'S A STANDARD GRAPHIC GROWN IN THE AREA OF -- THERE NEEDS TO BE A HANDOFF FROM THE DISCOVERY AND WHAT WE UNDERSTAND ABOUT HOW TO IMPLEMENT IT TO THE PEOPLE WHO ARE GOING TO USE IT. THIS IS A STANDARD WHAT'S CALLED VALLEY OF DEATH PRESENTATION ABOUT COMMERCIALIZATION OF TECHNOLOGY. DISCOVERY IN THE UNIVERSITY OR NIH, FUNDED BY NIH AND OTHERS. THAT'S WHERE NIH HAS ITS BIGGEST IMPACT IN THIS CHAIN. BUT THEN YOU HAVE YOUR SBIRs AND YOUR STTRs AND YOU'RE TRYING TO HELP SMALL BUSINESSES. GREAT. BUT THEN, THEN ONCE YOU'VE DONE A LITTLE BIT MORE OF THAT DEVELOPMENT, OTHER THINGS HAVE TO TAKE OVER. IT HAS TO BE AN ECONOMIC CLIMATE THAT IS GOING TO ENCOLONEL THAT DEVELOPMENT. YOU NEED FOUNDATIONS INVOLVED, INDUSTRY, THERE NEEDS TO BE A HANDOFF, AND THAT'S THE PART THAT NIH REALLY -- YOU CAN -- YOU CAN TRY AND DO BETTER BUT FUNDAMENTALLY AT THE END OF THE CHAIN, IT'S NOT UNDER YOUR CONTROL. AND SO I JUST WANT TO BE CLEAR ABOUT THAT, THAT THERE'S LOTS OF THINGS TO DO, WORTH DOING, BUT AT THE END THERE, YOU DON'T CONTROL IT. SO A FEW CONCLUDING REMARKS. YOU KNOW, YOU ALREADY KNEW THAT THIS WAS CHALLENGING, I THINK THE WORD WAS DIFFICULT BEFORE, BUT THAT YOU CAN HAVE CHAINS OF EVIDENCE THAT LINK THEM. LINKING FUNDING THE PROJECT IS THE EASY PART. REALLY STRAIGHTFORWARD. THE FORWARD LINKAGE OF PROJECTS TO DISCOVERY IS ALSO NOT SO HARD AS LONG AS WE RECOGNIZE THAT PROJECTS COMPLEMENT AND BUILD ON EACH OTHER. THESE ARE NOT ONE OFFS, AND THAT THESE ARE OCCUR OVER A PERIOD OF TIME. AND CONSTRUCTED AROUND NETWORKS AND IT'S REALLY HARD TO SHOW IMMEDIATE IMPACT IN TERMS OF THAT. I WANT TO JUST POINT OUT THE IMPORTANCE OF SHARED INFRASTRUCTURE. I'M FLYING THROUGH NOW BECAUSE I KNOW WE'RE RUNNING OUT OF TIME. THE BACKWARDS LINKAGE OF PROJECTS TO -- GOING THE OTHER WAY, I WANT TO MAKE THE POINT THAT DISCOVERY COMES FROM ALL KINDS OF PLACES. SO BACK TO DR. COHEN, HE DIDN'T INVENT ANTI-RETROVIRAL THERAPY, THAT WAS ALREADY THERE. SO IF YOU WANT TO THINK ABOUT ALL PARTS OF THAT, ONLY HAS A PIECE. SO IT DRAW FROM DISPARATE SOURCES AND IT CAN INVOLVE FUNDS OTHER THAN NIH FUNDS. I DON'T THINK THAT'S A NEGATIVE. SOMETIMES PEOPLE MIGHT SAY, WELL, YOU KNOW, WE DON'T NEED THE NIH, INDUSTRY WOULD STEP IN AND DO ALL THIS. WELL, I DON'T HAPPEN TO BELIEVE THAT, BUT I THINK IT'S MORE THE OTHER WAY. WHEN NIH INVESTS, THEN OTHERS ALSO INVEST. AND SO I THINK IT'S FINE. IN FACT, THAT'S A MEASURE OF SUCCESS. SO TO SAY A FEW OTHER THINGS, HERE AT THE END IN TERMS OF WANTING TO SAY THERE'S VALUE IN SIMPLICITY, THAT ALL OF THE STEPS IN THIS EVIDENCE CHANGE ARE CHALLENGING AND IN THE END, IF THE IDEA IS TO COMMUNICATE WITH THE PLIC, WE NEED SOME FAIRLY SIMPLE STALEMENTS TO CONCLUDE WITH. SO THANK YOU. >> THANK YOU. [APPLAUSE] >> SO ANYBODY WOULD LIKE TO ASK ANY QUESTIONS OR STIMULATE SOME DISCUSSION FROM THE PANEL? DID WANT TO START THAT? ON DR. COHEN'S SOCIAL NETWORK CHART, HOW FAR DID THAT GO? WAS IT JUST FOR THE CORESEARCHERS HE HAD OR DID IT GO DEEPER INTO THE LINKAGES THAT ARE NECESSARY TO MAKE THINGS HAPPEN? >> IT'S ONE TAKE ON THAT SOCIAL NETWORK. IT'S CURRENT AND IT WOULDN'T INCLUDE ALL OF -- IT INCLUDES STUDENTS AND POSTDOCS, IT DOESN'T INCLUDE ALL OF THE STAFF NECESSARILY. IT DOESN'T INCLUDE OTHER IMPORTANT ADVISORS AND COLLEAGUES THAT HE HAS THAT HE'S TALKING TO ALL THE TIME THAT WE WOULDN'T NECESSARILY KNOW ABOUT IN OUR DATABASE SYSTEM. SO IT'S JUST ONE SLICE, SO IT'S AN UNDERSTIM. >> A FOLLOW-UP QUESTION. >> GO AHEAD. >> THE SENSE I HAVE, HAVING LIVEED WITH A PHYSICAL SET OF RESEARCH, NOT BIOLOGICAL, I FIND AND STILL FIND TO THIS DAY A MUCH CLOSER COUPLING IN THE PHYSICAL RESEARCH DOMAIN, ESPECIALLY IN INFORMATION TECHNOLOGY, BETWEEN UNIVERSITY, SOURCES OF FUNDING, FILLING IN THE VALLEY OF DEATH, THAT'S A COMMON TERM ACROSS ALL TECHNOLOGIES. HOW DO YOU THINK THE BIOLOGICAL COMMUNITY COULD BUILD, ESPECIALLY WITH THE FOCUS OF THE NIH, BUILD THESE STRONGER LINKAGES AS PART OF THEIR RESEARCH PROJECT PROCESS SO THAT WE COULD GET THIS CLOSER COUPLING AND SPEED UP THE PROCESS? >> I THINK YOU MAY HAVE BEEN HINTING A THE@WHAT YO AT WHAT YOU THINK TH E ANSWER IS, WE NEED TO BUILD SOCIAL TIES BETWEEN THOSE FUNDERS AND THOSE INTERESTS AND THE SCIENTISTS THEMSELVES. SO MANY UNIVERSITIES NOW, FOR EXAMPLE, HAVE ENTREPRENEURS AND RESIDENTIN RESIDENCES WHO JOB IS TO BUILD THOSE SOCIAL NETWORKS. THERE ARE OTHER SORTS OF PROGRAMS FOR GRADUATES AND POSTDOCS PERSONALLY. THAT'S WHERE I THINK A LOT OF THIS COULD COME. WE TALKED ABOUT THE FUTURE OF THE BIOMEDICAL WORKFORCE, SOME OF THEM WILL BE ACADEMICS, SOME NOT. I WOULD LOVE TO POINT OUR PROGRAMS TO ENCOURAGING STUDENTS AND POSTDOCS TO HELP AND MOVE INTO THIS COMMERCIALIZATION SPACE, BUT THOSE ARE JUST A FEW OF MY IDEAS. >> TIM, DID YOU WANT TO COMMENT? I THINK, NORM, YOU WERE NEXT, AND THEN GAIL. >> I THINK I BROKE THE CODE HERE, BUT I'M INTRIGUED BY THE CORRELATION OF THE CHARTS YOU PROVIDED. THE CORRELATIONS ARE -- YOU WOULD FIND STRANGE CAUSE AND EVENTS WHEN YOU PLOT THINGS TOGETHER, BUT I WAS SO IMPRESSED WITH THE CORRELATIONS, THEN I SAW THE DIABETES ONE, AND WHAT HAS BEEN THE REACTION OF THE EXPERT COMMUNITY TO THIS BODY OF WORK IN TERMS OF ITS ACCEPTABILITY? >> I THINK THE CHALLENGE THERE IS HOW SIMPLE IT IS. SO I THINK YOU WOULD SEE EXPERTS COME IN AND SAY, OH, BUT YOU'RE ASSUMING THIS AND YOU'RE ASSUMING THAT, AND REALLY IT'S MUCH MORE COMPLICATED THAN THAT. I MEAN, HONESTLY, I THINK THAT'S PART OF WHAT YOU WOULD HEAR, BUT FOR ME, ONE OF THE THINGS I'VE LEARNED IN A LONG CAREER, REALLY SISIMPLICITY IS A LOT TOUGHER THAN COMPLEXITY. GETTING SOMETHING TO COMMUNICATE A MESS ABL MESSAGE REALLY CLEARLY IS NOT ALL THAT EASY TO DO. I WAS SORRY THAT THIS STUDY HADN'T BEEN PICKED UP MORE. I THOUGHT IT WAS INTERESTING. CAN I CRITIQUE IT? SURE, I CAN. AND HOW WOULD I CRITIQUE IT? IT'S ALL OF THE INTERVENING STEPS, IT'S ALL THE ASSUMPTIONS ABOUT WHAT CHANGED AND WHAT DIDN'T CHANGE. THIS IS OBSERVATIONAL RESEARCH AT ITS BEST. WE COULD EVEN SAY OH, IF YOU'RE GOOD AT FITTING TIME TREND, YOU CAN MAKE THIS WORK. ALL THAT'S TRUE. BUT IT'S A REALLY -- TO ME, A VERY COMPELLING PICTURE. >> THERE'S A QUOTE FROM EINSTEIN THAT SAID THINGS SHOULD BE AS SIMPLE AS POSSIBLE BUT NOT MORE SO, AND ONE OF THE THINGS YOU'D HAVE TO DO IS GET BACKUP DATA TO LOOK AT THE CHANGES, WHAT DO WE THINK ARE CAUSING THE CHANGES IN CARDIOVASCULAR MORTALITY AND DIKDIABETIC MORTALITY. FIRST THING I THOUGHT, A LOT OF DIABETICS ARE NOT DYING OF HEART DISEASE NOW, THERE'S OBESITY AND OTHER THINGS, SO YOU HAVE TO SAY, WELL, WHAT ARE THOSE BREAKTHROUGHS AND HOW DOES THAT RELATE TO TRENDS AND CABBAGE PROCEDURES AND ASPIRIN AND SMOKING ALL AFTE OF THESE THINGS, THE DOUBLING OF THE NIH BUDGET. SO YOU HAVE TO TRIANGULATE THIS WITH OTHER PIECES OF DATA THAT MAKE SENSE IN ORDER TO REALLY HAVE CREDIBILITY. BUT IT'S PRETTY COOL. I MEAN, IT LOOKS GOOD. >> OF COURSE ONE OF THE THINGS IS HEART DISEASE REALLY WENT DOWN, AND OTHER THINGS HAVE HAD MUCH LESS IMPACT SO YOU HAVE A REALLY STRONG CHANGE IN VALUE THAT UH-UH CAN GO BACK AND LOOK AT THE CAUSE. THERE IS A SIMILAR KIND OF THING WITH LUNG CANCER AND SMOKING, RIGHT? THOSE KIND OF CORRELATIONS ARE EXTREMELY TIGHT IN A PUBLIC HEALTH WAY. THIS IS NOT THE ONLY EXAMPLE THAT HAS BEEN THAT DRAMATIC. IT'S ONE OF SEVERAL REALLY BIG ONES THAT HAVE -- >> I ENJOYED YOUR PRESENTATION A LOT. YOU MENTIONED THE IMPORTANCE OF THE FACE TO FACE INTERACTIONS, ITS STUDY SECTIONS, AND I WORRY A LOT ABOUT THE LACK OF THAT IN TERMS OF DOING REVIEWS, YOU KNOW KNOW, BY PHONE, AND ALSO COUPLED WITH THAT THE TENDENCY IN TERMS OF ESPECIALLY THE YOUNG PEOPLE FOR A LOT OF REASONS NOT GOING TO AS MANY SCIENTIFIC MEETINGS AS THEY FORMERLY DID. SO IF YOU COUPLE THE LACK O OF INTERACTIONS, LACK OF ATTENDANCE AT MEETINGS, YOU'RE REALLY REDUCING A LOT OF THE FACE TO FACE INTERACTIONS, PARTICULARLY WITH THE INTERNATIONAL COMMUNITY, AND AS WE SEE FEWER AND FEWER INTERNATIONAL STUDENTS SAYING -- THE BEST ONES ARE BEING KEPT AT HOME -- TO YOUR POINT ABOUT THE SOCIAL NETWORKING AND HAVING ENTREPRENEUR PROGRAMS, I HAD THE OPPORTUNITY TO INTERACT A FAIR AMOUNT WITH THE SPARK PROGRAM AT STANFORD, AND BOY, I MEAN, IT IS REALLY AMAZING. ABSOLUTELY AMAZING WHAT THEY'VE BEEN ABLE TO DO, SOME REALLY BRIGHT, YOUNG STUDENTS THAT THEY'VE ATTRACTED TO PARTICIPATE IN THAT AND THEN HAVE GONE ON TO BE ABLE TO GET FUNDING FOR SOME OF THEIR WORK. SO I THINK THERE'S A LOT OF OPPORTUNITIES TO MAKE THINGS BETTER IN THAT REGARD. >> DID YOU HAVE A CHANCE TO LOOK AT THE QUESTIONS PAGE WE DISTRIBUTED? AT THE BOTTOM OF TH OF A PAGE -- B, GIVEN THE EXAMPLES THAT YOU'VE PRESENTED, THEY ALWAYS HAVE THIS STRUG TOLL TROO I TO OVERCOME THE SENSE THAT -- EXAMPLES MIGHT BE CHOSEN WITH A CERTAIN BIAS. IN ANY CASE, SOUND ANECDOTAL. DO YOU THINK THERE IS A CREDIBLE COMPREHENSIVE ASSESSMENT THAT COULD BE MADE BETWEEN PUBLIC HEALTH PARAMETERS AND NIH INVESTMENTS AND OUTPUTS? AND C, WE'RE AWARE THAT NIH DOES A FEW OTHER THINGS BESIDES GIVING OUT RESEARCH GRANTS. THERE'S A SIZABLE BUDGET FOR COMMUNICATIONS AND MAYBE INTERAGENCY ACTIVITIES COULD HAVE A LOT OF LEVERAGE AT VERY MODEST COST. IF THERE WAS A SENSE TO TRY TO SEE THROUGH TO THE PUBLIC HEALTH IMPACTS, PUBLIC HEALTH BENEFITS. FINALLY, WITHIN THE RESEARCH AGENDA ITSELF, D, THERE IS A HOST OF SCIENTIFIC METHODOLOGY AND SCIENTIFIC RESEARCH TOPICS THAT COULD PLAY ON THE BIGGEST PROBLEM THAT'S BEEN IDENTIFIED THIS MORNING AND FOR DECADES, WHICH IS THE IMPLEMENTATION OF WHAT WE KNOW. FOR PEOPLE TO ACTUALLY CHANGE THEIR CRITICAL HEALTH BEHAVIORS. WE HAD A DISCUSSION BEFORE ABOUT THE DIABETES SUBJECT, AS A MATTER OF FACT, WITH DIABETES AND OBESITY SUBJECT, AND OF COURSE THE IMMEDIATE RESPONSE IS TO THE INCREASE MORTALITY BESIDES AN INCREDIBLE PART ABOUT HEART DISEASE IS THAT OBESITY IS TOTALLY OUT OF CONTROL, WE'VE DONE A LOT OF STUDIES OF OBESITY, DOCUMENTED EXTENSIVELY, AND YET WE DON'T SEEM TO HAVE ANY HANDLE ON HOW TO ADEQUATELY INFLUENCE THE BEHAVIORS OF AMERICANS. IN MOST CASES, CRITICAL CASES, MENTAL HEALTH, DANGEROUS BEHAVIORS, RELATED TO OVERWEIGHT AND OBESITY, AND YOU COULD DO A LONGER LIST, THINGS ARE GOING VERY BADLY IN THE WRONG DIRECTION. THAT'S A TOPIC FOR RESEARCH. UNLESS PEOPLE HAVE JUST GIVEN UP AND SAID WE DON'T CARE FOR THAT KIND OF RESEARCH, EITHER IT'S NOT SUCCESSFUL OR NOT UP TO STANDARD OR IT'S NOT PORCH ENOUGH. IT IS A POINT IN THE PORTFOLIO MANAGEMENT THEME, WHAT KINDS OF RESEARCH AND WHAT KINDS OF EVALUATION METHODS SHOULD BE APPLIED. DOES THAT TRIGGER ANY FURTHER COMMENT? >> I COULDN'T AGREE WITH YOU MORE. THE QUESTION IS WHAT DRIVES THE PORTFOLIO, AND I THINK THAT'S A QUESTION FOR YOU YOUR BOARD TO DISCUSS. IF WE'RE TALKING ABOUT MY PERSONAL VIEWS ON THAT, YOU CAN SEE BY MY DISCIPLINARY AFFILIATIONS WHAT MY RESPONSE IS GOING TO BE THERE, I THINK THERE'S A TREMENDOUS POTENTIAL THERE. I DON'T THINK THERE'S ANY LACK OF INTEREST. I DON'T THINK THAT'S WHERE THE EXCITEMENT HAS BEEN. LET'S JUST PUT IT THAT WAY. IT'S NOT THE LARGEST BUDGET. THE NIH HAS DONE SOME WONDERFUL THINGS, AND SOME OTHER PIECES, BUT I DON'T THINK THAT'S WHERE ALL OF THE MAJOR PART OF THE MONEY HAS GONE. BUT AGAIN -- SO I'M AGREEING WITH YOU ON THAT. >> OF COURSE I AGREE WITH YOU TOO. THAT'S TWO BIASES AGAINST THIS. SO CALLED SOFT SCIENCE, IF YOU WILL, ONE IS THE HARD SCIENCE BIAS, WHICH IS THAT IF YOU'RE A REAL SCIENTIST, YOU'RE IN THE LABORATORY, AND I THINK THAT'S A VERY, VERY STRONG BIAS IN THE UNITED STATES. I DON'T AGREE WITH THAT BIAS BUT IT'S A VERY STRONG BIAS. MEDICINE AND FIE PHYSICIANS IN GENERAL LIKE PILLS, MA JIG MAGIC BULLETS. WE FINALLY GET THE PILL, WE FIND OUT THE PROBLEM IS PEOPLE DON'T TAKE IT, AND THEN WE SAY, OH, MY GOD, WHAT'S WRONG WITH THESE PEOPLE, HOW COME THEY DON'T GET THE FLEW VACCINE? NOT A VERY GOOD VACCINE, BUT HOW COME THEY DON'T? HOW COME THEY DON'T USE PREP, HOW COME THEY DON'T USE CONDOMS, THERE'S ALL THESE THINGS THAT WORK. AND THEN WE'RE BACK TO THIS ISSUE OF BEHAVIOR AGAIN AND IMPLEMENTATION, IMPLEMENTATION SCIENCE. PART OF IT IS -- THE CONGRESS DOESN'T LIKE IT VERY WELL, LOOK AT THE ARC BUDGET, ALWAYS SLAPPED AROUND AND BEAT UP. THEN THE HEALTH SYSTEM DOESN'T PAY PEOPLE TO REMAIN HEALTHY, IT PAYS FEE FOR SERVICE PROCEDURE-BASED STUFF AND IT DOESN'T PAY FOR -- USUALLY DOESN'T PAY FOR SMOKING PREVENS PREVENTION -- SO THE THING IS IT PAYS FOR BARIATRIC SURGERY, LOTS OF OTHER CRAZY STUFF, NOT THAT THERE ISN'T A ROLE FOR THAT, BUT YOU KNOW, WHEN WE HAD -- I MEAN, WHAT KIND OF COUNTRY DO WE LIVE IN WHEN THERE'S THIS EXPLOSION OF BARE YA TICK SURGERY IN BARRY ESSENCE, AND YET WE DON'T HAVE DOCTORS, NURSES, HEALTH SYSTEMS, DIET AND PHYSICAL ACTIVITY. IT'S BECAUSE OF THE INCENTIVES. YOU CAN DO RESEARCH ON THAT, ON THE INCENTIVES. >> I HATE TO BREAK HERE, THIS HAS BEEN FASCINATING. TERRIFIC MORNING, A GREAT PANEL. THANK ALL THREE OF YOU, THANK OUR MODERATORS, AND IT'S NOW TIME TO TAKE A BREAK FOR LUNCH. WE SHOULD RECONVENE AT -- I'LL SOFTEN THIS A LITTLE BIT, LET'S BE BACK AT 1:50. OUR THIRD PANEL IS A VERY INTERESTING ONE THAT IT BEGINS TO ADDRESS THE BROADER SOCIETAL IMPACTS OF BIOMEDICAL RESEARCH. DAN GOLDIN IS GOING TO LEAD THIS PANEL ALONG WITH ALAN GUTTMACHER, BOTH WHOM HAVE GIVEN THE TOPIC SERIOUS CONSIDERATION AS WELL AS THE IDENTIFICATION OF THE PANELISTS. SO I WOULD LIKE TO TURN IT OVER TO YOU, DAN. OR IS ALAN GOINGTIS? >> NO. >> FORTUNATELY WE THOUGHT AHEAD AND I'M GOING TO DO THE INVOCATION AND HE WILL DO THE BENEDICTION. >> OKAY. >> THIS IS A WONDERFUL TOPIC. AS I LOOK AT BROADER SOCIETAL IMPACTS OF BIOMEDICAL RESEARCH AND THE TERM VALUE, ONE NEEDS TO TAKE A LOOK AT THE VALUE SYSTEM IN THE MEASUREMENT -- AND THE MEASUREMENT SYSTEM THAT WAS, IS, AND WILL BE. THAT IS SHAPED BY MANY DIFFERENT FACTORS. I WAS TALKING TO THE GAIL THIS MORNING AND ONE OF THE ISSUES THAT WAS CONCERNING ME AS I REREAD THE MEETINGS TO THE PRIOR MEETINGS I CAN'T REMEMBER WHO TESTIFIED IN JANUARY BUT HE GAVE A TERM HE GAVE A TIME FRAME OF 17 TO 24 YEARS FROM THE TIME THE RESEARCH BEGAN UNTIL THE PRODUCT GOT TO THE FDA. THEN I THOUGHT ABOUT OUR MODERN WORLD. IN THE FINANCIAL WORLD IN TEN MILLISECOND YOU CAN USE BILLIONS OF DOLLARS, IF YOU LOOK AT WHAT HAPPENED TO TWITTER THERE ARE A HALF BILLION TWEETS A DAY OF 140 CHARACTERS. WHAT IS A WHOLE NEW MODE OF COMMUNICATION. AND THERE ARE 3 MILLION SERVERS AT GOOGLE, I DON'T REMEMBER ALL THESE NUMBERS. 35% OF THE WORLD'S PHOTOS ARE ON FACEBOOK. YOUTUBE HAS 4 BILLION VIEWS HER BAY. THERE ARE 6.1 TRILLION TEXT MESSAGES WORLD WIDE SO THAT'S ABOUT A THOUSAND PER PERSON PER YEAR. THERE SEEMS TO BE THIS MISMATCH. WHILE WE IN A SCHOLARLY MANNER TRY TO ADDRESS THE VALUE OF BIOLOGICAL RESEARCH, WE HAVE TO THE SAY WHO ARE WE ADDRESSING, IN WHAT TIME FRAME. AND ARE THE SCIENTIFIC ECONOMIC METRICS WE'RE LOOKING AT, ARE THEY RELEVANT IN THIS WHOLE PICTURE? Z I KNOW MY GRANDSON DOESN'T TALK TO ME OVER THE PHONE, HE ONLY TEXTS. THAT'S A REASON FOR THAT, BECAUSE HE THINKS IN SMALL BITS OF INFORMATION. SO WE NEED TO APPEAL TO THE AMERICAN PEOPLE, THIS IS AN ESSENTIAL FUNCTION OF AMERICA. IT IS ESSENTIAL THAT THIS ORGANIZATION BE HERE AND BE VIBRANT BUT ARE WE CONNECTING UP IN THE RIGHT WAYS. HOW DO WE MEASURE, I ALSO TOLD GAIL THIS MORNING I WHEN THROUGH ALL THE REFERENCE TECHS WE WERE ASKED TO READ AND A MAJORITY OF THEM ALL DEALT WITH ECONOMICS. BUT THERE ARE OTHER ISSUES. THAT HAS TO DO WITH HOPE AND FEAR. 9/11. ANTHONY FAUCI WHO HAS BEEN HERE FOR DECADES, COULD HAVE BEEN ELSEWHERE, CALMED THE AMERICAN PUBLIC ON THREATS THAT OCCURRED. HE CAME BACK WITH SARS. SO NIH PERFORMS MORE THAN AN ECONOMIC FUNCTION IS AT HEART AND SOUL OF THIS COUNTRY. AS WE GO THROUGH IT I CAN GIVE OTHER EXAMPLES WE NEED TO THINK BROADLY ABOUT THE SOCIETAL IMPACTS HOW WE CONNECT. WHO WE CONNECT TO AND THE MODE OF COMMUNICATION THAT WE HAVE. SO WITH THAT, ALAN, I INDICATED I'LL DO THE INTRODUCTION, YOU DO THE SUMMARY, AND IT WORKS OUT PERFECTLY. SO LET ME INTRODUCE OUR FIRST SPEAKER. DAVID M. CUTLER. HE'S THE OTTO EC KSTEIN PROFESSOR OF APPLIED ECONOMICS, I ALSO KNOW DR. CUTLER NEEDS TO LEAVE AT 3:30. >> THANK YOU FOR INVITING ME HERE. I APOLOGIZE, I DO HAVE TO LEAVE DIRECTLY AT 3:30 BECAUSE I HAVE TO DO SOMETHING BACK IN BOSTON IN THIS EVENING. I'M DELIGHTED TO BE HERE, THIS TOPIC IS SUFFICIENTLY EASY THAT IT SHOULDN'T OCCUPY US MORE THAN TEN OR 15 MINUTES OR SO. SO WE HAVE PLENTY OF TIME IN RESERVE. LET ME YOU HEAR DIFFERENT CLAIMS ABOUT BIOMEDICAL RESEARCH THE NIH VALUED BY, I LISTED A FEW HERE BUT A GROUP GONE ON FOR PAGES, IT HELPS AMERICANS WIN NOBEL P PRIZES, INCREWs EMPLOYMENT, -- INCREASES EMPLOYMENT, PEOPLE USED TO SAY COMMUTE COMPUTER CHIPS BUT I CHANGED IT TO PROTEIN CHIPS. THESE THINGS HAVE FEATURES SOMETIMES THEY OTHER TRUE BUT HARD TO VALUE, THAT'S A LITTLE BIT WHAT YOU'RE SAYING IN THE INTRODUCTION WHICH IS HOW YOU VALUE BEING CALM BECAUSE SOMEBODY YOU TRUST BEING IN CHARGE OF THINGS WHICH IS DIFFICULT TO VALUE. SOMETIMES THEY'RE IN THE TRUE, IT'S TRUE BIOMEDICAL RESEARCH SPENDING INCREASES EMPLOYMENT, ALSO TRUE THAT OVER TIME RESOURCES DRAWN INTO BIOMEDICAL RESEARCH, SO THEY'RE EMPLOYED SOMEWHERE SO I WOULDN'T WANT TO MAKE AN ARGUMENT THAT IF WE JUST SPENT MORE OVER TIME 150% OF THE U.S. POPULATION WOULD BE EMPLOYED IN BIOMEDICAL RESEARCH. SOMETIMES THEY'RE IRRELEVANT WE NEVER HAVE VERY STILL PEOPLE PRODUCING POTATO CHIPS AND WE PROBABLY NEVER WILL VERY UNSKILLED PEOPLE PRODUCING BIOMEDICAL RESEARCH SO THE FLOW ISN'T SO BIG. BUT I WANT TO GET TO SOMETHING THAT ARE TRUE, JUST TO START THIS WAY THE WAY ECONOMISTS WOULD START THIS OFF, I THINK BOB TOPPLE WILL PICK UP ON THIS. WILL PICK UP ON -- AT LEAST WHAT I GOT IN THIS COMMITTEE AND I THINK IS HELPFUL, FUNDAMENTAL KNOWLEDGE TO ENHANCE LIFE AND REDUCE BURDENS OF ILLNESS AND DISABILITY. WHAT I WANT TO EMPHASIZE IS THERE ARE DIFFERENCE LAYERS SO IT'S NOT JUST INVENTING CURES, IT'S THINKING THE CONSEQUENCES OF THE WORLD WE HAVE BEEN INVENTED CURES. ONE THEME THAT I WANT TO STRESS, THE NIH HAS BEEN MUCH BETTER AT INVENTING CURES THINKING CONSEQUENCES OF THE WORLD GOING TO GO INTO. THAT'S MEANT THAT SOME OF THE SOCIAL RETURNS TO THE NIH ARE PROBABLY SMALLER THAN THEY OTHERWISE WOULD HAVE BEEN. SO I WANT TO TALK ABOUT THAT AND DO IT IN TERMS OF LAYERS. SO I WANT TO THINK ABOUT THE NIH HAVING THREE LAYERS, THINK ABOUT UNPEELING AN ONION, BUILDING AN ONION. THE FIRST COROLLARY IS HEALTH IMPROVEMENTS FOR TARGETED POPULATIONS. WHEN -- WHAT IS THE NIH DO, WHAT THINGS THAT IT DOES IS UNDERSTAND MECHANISMS OF DISEASE AND TREATING DISEASE SO THAT HEALTH IS IMPROVED FOR THE POPULATION WE WOULD LIKE TO IMPROVE HEALTH FOR. THAT POPULATION, EITHER U.S. POPULATION OR MAYBE INCLUDE THE REST OF THE WORLD DEPENDING ON YOUR POINT OF VIEWFUL THE SECOND GETS A LITTLE AT THE SECOND LAYER OF THE ONION WHICH IS WHEN WE INVENT AND DEVELOP AND MOVE ALONG TOWARDS PRODUCTION, THOSE TREATMENTS WHAT IS THE ENVIRONMENT INTO, PARTICULARLY ENTERING INTO A METHOD HEALTHCARE SYSTEM. THE NIH NOT BY DESIGN BUT JUST INTENTIONALLY, HELPS TO CONTRIBUTE TO THE MESS IN THE ALMOST SYSTEM AND THAT CONTRIBUTION IS A VERY SIGNIFICANT IMPACT ON WHAT WE GET FOR THE TECHNOLOGICAL ADVANCES, I'M GOING TO COME BACK. THE THIRD ONE, HOW DO YOU DEAL WITH THE POPULATION THAT'S HEALTHIER, LIVING LONGER ALSO COSTING MORE MONEY. YOU HEARD IN THE PREVIOUS SESSION ABOUT THE VALUE OF NIH IMPROVEMENT, THIS IS WORK I HAVE DONE I WORK ON THIS OVER THE YEARS WITH MARK MCLAWN AND A NUMBER OF OTHER FOLKS. WHAT WE HAVE DONE HERE IS LOOK AT THE VALUE OF MEDICAL SPENDING BY DOLLARS PER YEAR OF LIFE. SO I HAVEN'T TAKEN QUALITY OF LIFE INTO JUST -- YOU WANT A LOWER NUMBER SO LOWER NUMBER MEANS YOU'RE SPENDING LESS BEAR ADDITIONAL YEAR LIFE EVENT SO THAT'S GOOD. GENERALLY THE NUMBERS ARE LOW BUT THEY'RE INCREASING SOMEWHAT OVER TIME IN PARTICULAR FOR THE ELDERLY SO THAT IS IF YOU SAY 1990 TO 2000 WHICH FOR WHICH WE HAVE DONE THIS THE COST OF THE YEAR OF LIFE IS ABOUT $150,000. QUALITY IMPROVEMENT SO THAT'S UNDERSTATING THE VALUE. BUT THAT'S A FAIR AMOUNT OF MONEY. PROBABLY SORT OF CONSENSUS ESTIMATE AMONG HEALTH ECONOMISTS YOU'RE SAFE ASSUMING ANYTHING LESS THAN $100,000 PER YEAR OF LIFE IS A GOOD INVESTMENT, ANYTHING OVER THAT YOU NEED TO GET INTO SOME DISCUSSION BOP T THEOPEL WILL TELL YOU IT'S $250,000, OTHERS WILL SAY $50,000, BUT SOMEWHERE AROUND THERE. YOU'RE IN A SITUATION YOU'RE INVENTING MORE WAYSES TO CURE PEOPLE MORE EXPENSIVE THAN WHAT IT IS THEY'RE BRINGING US IN RETURN. THAT'S AS MUCH A STATEMENT ABOUT PRICING WHEN THEY'RE OUT THERE THE UTILIZATION WHEN THEY'RE OUT THERE, AS IT IS ABOUT FUNDAMENTAL NATURE OF THE EVENTS STILL THE CASE THERE'S THINGS DEVELOPED WHICH ADVANCE IS SMALL AND PRICE IS HIGH. HOW DO YOU THINK ABOUT THE LAYER? THE FIRST LAYER IS THE HEALTH IMPROVEMENT. THE SECOND LAYER IS WHAT HAPPENS ONCE IT'S OUT THERE. WE SHOULD NOT BE BENIGN ABOUT THAT. IF YOU THINK MEDICAL CARE PROBLEM ASSOCIATED WITH OR INDUCED BY THE NIH, IT REALLY HAS A COUPLE OF DIFFERENT FEATURES TO IT. SO I WANT TO THINK ABOUT NOW ONCE WE HAVE DEVELOPED SOMETHING THE EXAMPLE I'LL USE IN A MOMENT THE SAY CORONARY ARTERY NARROWING. SOME CHASES IT'S CHEERILY BENEFICIAL IN THE AFTER MATH OF ACUTE MI, MORE TO DO AUSTIN THAN TO DO MEDICATION THERAPY OR INTERVENTION, SOME DOCTORS HAVE DONE THAT TO MAKE MONEY. THEN THERE'S A BROAD RANGE WHERE BENEFITS ARE UNCLEAR. HARD TO KNOW WHAT YOU SHOULD DO OR SHOULDN'T DO. WE GET EQUIVALENT OUTCOMES AND ONE THING IS THAT IN ALMOST ALL THE CIRCUMSTANCES WE WIND UP DOING MORE IN THAT AREA AND MORE IN INAPPROPRIATE AREA. FOCUSED TON MIDDLE PART YOU'LL DO GREAT, THAT'S BENEFICIAL THERAPY AND PEOPLE WHO NEED IT. THOSE OUTER LAYERS BUNCH OF ADDITIONAL COMPLICATIONS. OVER TIME THE COST OF REVASCULARIZATION IS $30,000 OR SO, PHENOMENAL ADVANCES THE FIRSTENING OWEPLASTY IS IN THE EARLY 1970s BY THE EARLY 1990s WE'RE DOGGING A MILLION WORLDWIDE BY CLOSE TO 2000 WE HAVE DONE ANOTHER MILLION SO THIS TWITTER ATIVAN IF YOU DREW THIS ON THE CHAR WOULD NOT LOOK UP WAR SLOPING AS NUMBER OF TWEETS SENT BUT A LITTLE BIT SIMILAR TO THAT. WELL ABOVE 2 MILLION, I COULDN'T GET MORE RECENT DATA. BUT KEY POINT IS NOT, WHAT IS THE NUMBER NOW, BUT HOW WELL ARE WE DOING? YOU SEE THIS TIME AN AGAIN, VERY VALUABLE USE, SO THERE'S NOTHING THE NIH HAS DEVELOPED FOR WHICH YOU SAY IT WAS NEVER WORTHWHILE DOING. THERE'S ALMOST NOTHING THE NIH HELPED DEVELOP WHEN YOU SAY YOU HAVE DONE IT IT'S ALWAYS WORTH DOING. YOU SEE EXAMPLES WHERE IT'S WORTHWHILE TO DO IT. YOU SEE EXAMPLES WHERE THE CLINICAL LITERATURE SUGGESTS YOU SHOULD NOT BE DOING IT. ABOUT A THIRD OF THE USE OF STINTS IS IN PATIENTS FOR WHICH THE CLINICAL LITERATURE SUGGESTS NO MORBIDITY OR MORTALITY BENEFITS. NOW, I'M NOT GOING THE TO SAY WE SHOULDN'T DO ALL OF THESE CASES BUT WHAT I'M TELLING YOU WHEN YOU LOOK AT THAT, THIS IS ABOUT $30,000 TO DO, WHEN YOU LOOK AT THAT, THAT'S A VERY BIG NUMBER. YOU CAN SEE HOW BIG THAT NUMBER IS HERE, IF YOU LOOK AT THE FEDERAL BUDGET GET, RELATIVELY CONSTANT, SPENDING IS PROJECTED TO GROW RAPIDLY. YOU CAN SEE SOCIAL SECURITY AT THE BOTTOM, IT'S A SMALL PART OF INCREASES IN PROJECTED SPENDING OVER TIME. OTHER IS ASSUMED CONSTANT AS SHARE OF GEP T WHALE THAT IN THIS PICTURE, IF YOU WANT TO INCLUDE IS MEDICAL SPENDING. MEDICARE, MEDICAID AND HEALTH INSURANCE EXCHANGES, CHILDREN'S HEALTH INSURANCE PROGRAM. ALL WHICH ARE GROWING REPLIED, ONE AND A HALF TO TWO PERCENT MORE RAPIDLY THAN THE ECONOMY. THIS IS WHY THE HIPPOBUDGETING IS BEING CUT, WHY FOOD STAMPS IS CUT AND EVERYTHING ELSE IS UNDER INTENSE PRESSURE. WHAT WE KNOW IS THAT A GOOD PART OF THAT IS BECAUSE OF BIOMEDICAL ADVANCE WHICH IS PROBABLY USED INCORRECTLY. LET ME ACTUALLY WHEN I SAID A LARGE PART OF THAT, LET ME BE MORE PRECISE LET ME BE EXTREMELY PRECISE. LESS FOR NIH, IT'S CLEAR THE LESS MONEY TO INVEST IN EVERYTHING ELSE INCLUDING THE NIH AND LET ME BE AS PRECISE AS ECONOMISTS WILL BE TO FOLLOW THE ADVICE OF FORMER MENTORS WHEN HE SAYS IF YOU GIVE A NUMBER DON'T GIVE A YEAR. IF YOU GIVE A YEAR DON'T GIVE A NUMBER. IF POSSIBLE DON'T GIVE EITHER. HOWEVER I WILL VIOLATE COROLLARY AND GIVE A NUMBER WITHOUT A YEAR AND OUR BEST GUESS IS ABOUT ONE-THIRD OF MEDICAL SPENDING SNOT CONTRIBUTING TO BETTER HEALTH. CLOSE TO A TRILLION DOLLARS A YEAR WITHOUT -- TRILLION DOLLARS A YEAR WITHOUT SUBSTANTIAL IMPROVEMENT IN POPULATION HEALTH. THE BANK BAIL OUT NOBODY LIKED THAT WAS SMALLER THAN THIS, IF STIMULUS BILL HALF THE PEOPLE DIDN'T LIKE IT WAS A LITTLE BIT SMALLER THAN THIS. SO DUMPING A BANK BAIL OUT OR STIMULUS BILL TO THE PA POTOMAC RIVER EVERY YEAR NEXT TO JIMMY HOFF. YOU GET UNNECESSARY SERVICES IS YOU FIRST HAVE SERVICES AND THEN DO THEM UNNECESSARILY. THE DIFFUSION OF MORE KINDS OF TECHNOLOGY CLEARLY CREATES ABILITY FOR MORE UNNECESSARY SERVICES. BARRIERS OF CARE DELIVERY, MISTAKES THAT OCCUR IN HOSPITALS, HOSPITAL ACQUIRED INFECTIONS, UNNECESSARY READMISSIONS TO HOPE, NOT DELIVERING THE CARE IT'S CAPABLE OF DOING. CHAIR COORDINATION, DIABETICS NOT APPROPRIATELY MANAGED, HYPERTENSIVE PEOPLE NOT MANAGED 50 YEARS AFTER KNOWING HOW TO DO SO. EXCESSIVE PRICES, IN MASSACHUSETTS F. YOU TRY TO PRICE OUT ABDOMINAL CT SCAN THE PRICES RANGE FROM $300 TO $1,300 OR ESSENTIALLY THE SAME QUALITY. MANY MORE PAY FOR THE $1,300 CT SCAN THAN OUGHT TO. ADMINISTRATIVE COSTS OF DEALING WITH THAT STUFF, IS INCREDIBLY COMPLEX JUST COLLECTING THAT DOLLAR, WHICH MAKES IT 100 TIMES MORE EFFICIENT THAN THE TYPICAL BUSINESS. THEN THERE'S FRAUD AND ABUSE, GOOD OLD FRAUD AND ABUSE WHICH IS ANOTHER HUNDRED OR 200 BILLION AS A GUESS. WORSE BY NOT TRYING TO DO IT BUT HAPPENSTANCE. THE COMPLEXITY THAT COMES WITH MORE TREATMENTS MAKE LACK OF BENEFITS MORE ACUTE. THERE'S MORE POSSIBILITY FOR WAYS AND CERTAINLY MORE UNCERTAIN TREATMENTS MEANS WHAT YOU GET IS FOCUSING ON WHERE PROFIT MARGINS ARE HIGH, BEST KNOWN THINGS IN HEALTH ECONOMICS IS THAT IF YOU PAY MORE FOR STUFF YOU GET MORE STUFF. THAT IS A DEEP ECONOMIC STATEMENT I WILL WIN THE NOBEL PRIZE ONE DAY. THE IMPLICATION IS YOU CANNOT THINK ABOUT THE MED -- ABOUT THE NIH BEING JUST BIOMEDICAL RESEARCH WITHOUT THINKING ABOUT THE ENVIRONMENT THAT IT GOES IN. AND THE SCIENCE OF DISEASE IS ONE PART OF IT WHICH WE DO VERY WELL. THE SCIENCE OF HEALTH SYSTEMS IS ANOTHER PART OF IT THAT WE DO VERY POORLY. AND IF YOU WANT TO THINK ABOUT THE RETURN FROM NIH, IT IS INCREDIBLY DEPENDENT UPON THE SCIENCE OF HEALTH SYSTEMS NOT JUST THE SCIENCE OF DISEASE. SO PUT IT ANOTHER WAY THE CHART THAT SHOWS DEFICIT EXPLODING AND THINK ABOUT HOW TO USE THINGS THAT WE KNOW WHEN WE SHOULD USE THEM AND HOW NOT TO DO THINGS WHEN WE SHOULDN'T DO THEM, YOU WOULD INCREASE RETURNS TO THE BASIC BIOMEDICAL RESEARCH IMMENSELY. LIKING GREAT TO HAVE CHEAP FOOD AS LONG AS YOU PREVENT FROM OVEREATING TOO MUCH AND KILLING YOURSELF BECAUSE YOU'RE TOO FAT. MAKE SURE THAT YOU'RE GETTING THE BENEFITS OF WHAT YOU'RE DOING BY MAKING SURE WHAT YOU DO IS WHEN YOU SHOULD BE DOING IT. SO THAT'S A BROAD STATEMENT, THERE IS WORK ON THIS, WE KNOW THERE ARE GOOD SYSTEMS AND BAD SYSTEMS. SYSTEMS WITH HIGH QUALITY LOW COST, SYSTEMS WITH HIGH CALL HIGH COST. WE KNOW FEATURES OF -- THERE IS SOME SCIENCE HERE, GOOD SYSTEMS USE IT. THAT'S VERY GOOD COMPENSATION SYSTEMS SO THEY'RE NOT PAYING MOTOR DOING THINGS, THEY'RE PAYING MORE FOR DOING LESS BUT FOR DOING THE RIGHT THING. THEY HAVE ENGAGED CONSUMERS SO THERE'S A VARIETY OF THINGS WE KNOW, WHAT THE SCIENCE IS WE HAVEN'T FIGURED OUT HOW TO TURN BAD ONES INTO GOOD ONES AND HOW TO MAKE ACTIONABLE THIS IDEA OF PROVIDING GOOD CARE. SO THAT'S THE FIRST BROADPOINT I WANT TO MAKE IS WE HAVE A MEDICAL SYSTEM THE RETURNS TO BIOMEDICAL RESEARCH ARE SIGNIFICANTLY REDUCED BY THE FACT THAT THE SYSTEM IS VERY POOR AND IT GETS WORSE AS WE INTRODUCE MORE STUFF, SO WE HAVE TO DEAL WITH THAT. I WANT TO GO BROADER TYPE SOCIAL IMPACT OF HEALTH AND I WANT YOU TO VOTE WHICH STATEMENTIOUS AGREE WITH. HERE IS WHERE AUDIENCE PARTICIPATION COMES. PEOPLE SHOULD BE ENCOURAGED TO WORK TO LATER AGES. HOW MANY SAY YES? HOW MANY SAY NO? ALL RIGHT. RAISE THE SOCIAL SECURITY MEDICARE ELIGIBILITY AGES. EQUIVALENT STATEMENTS. HOW MANY SAY YES? HOW MANY SAY NO. WE NEED TO HELP PEOPLE PREPARE FOR A LONGER RETIREMENT PERIOD. HOW MANY SAY YES? HOW MANY SAY NO. LET'S HAVE REQUIRED SAVINGS PROGRAM FOR OLE AGE. HOW MANY SAY YES? HOW MANY SAY NO? CLEARLY THERE ARE DIFFERENT WAYS OF PHRASING THE SAME THING OR CLOSE TO THE SAME THING THAT EVOKE DIFFERENCE PASSIONS IN PEOPLE. IN ADDITION TO SCIENCE OF HEALTH SYSTEMS WE NEED TO CONTINUE SCIENCE OF BEHAVIOR, TO GET PEOPLE TO ADAPT TO WHAT ARE THE CONSEQUENCES OF BIOMEDICAL RESEARCH. WE KNOW PEOPLE ARE LIVING LONGER AND HEALTHIER THAN THEY USED TO BE, ROUGHLY RETIREMENT AGE TODAY HAS THE SAME HEALTH PROFILE OF SOMEONE WHO IS TEN YEARS YOUNGER APPROXIMATELY TWO OR THROW DECADES AGO. AGING IS SLOWED DOWN BY PHYSIOLOGICAL TERMS BY ABOUT TEN YEARS. IT'S VERY NATURAL TO THINK THAT THAT PERSON SHOULD BE WORKING LONGER SAVING MORE AND SO ON BUT THE WAY TO GET PEOPLE TO DO THAT WE HAD A HARD TIME DOING THAT. IT'S NOT JUST THE KNOWLEDGE YOU CAN TELL PEOPLE, ADHERENCE TO MEDICATION RECOMMENDATIONS. WE KNOW HOW TO TREAT MEDICATIONS, WE CAN DO VERY WELL, GIVE THE NOBEL PRIZE TO THE PERSON WHO FIGURE OUT HOW TO GET PEOPLE TO TAKE MEDICINES THEY SHOULD BE TAKING THAN DEVELOPING A NEW MEDICATION. SO WE KNOW IT'S NOT JUST KNOWLEDGE, PEOPLE UNDERSTAND THESE THINGS. THEY UNDERSTAND THAT THEY SHOULD BE SAVING, THEY UNDERSTAND THEY SHOULDN'T OVEREAT, THEY COULD QUIT SMOKING, TAKE THEIR PILLS AND WORK LATER AND SO ON BUT THEY HAVE A DIFFICULT TIME DOING IT. PRICES MATTER ENORMOUSLY. IF THERE'S ONE THING WE HAVE LEARNED HOW TO REDUCE SMOKING IS RAISE TAXES ON CIGARETTES. IF THERE'S ONE THING ABOUT HOW TO REDUCE OBESITY IT'S HOW TO REDUCE PRICES OF UNHEALTHY FOOD AND SUGAR BEVERAGES. PRICES MATTER BUT ARE THERE WAYS TO USE THEM BETTER? MANY HAVE DIFFICULTY REMEMBERING THINGS. THE MEDICAL SYSTEM IS POOR AT HELPING REMEMBER THING. SOME HAVE FEAR OF THE UNKNOWN. WHEN YOU ASK WHY THEY DON'T TAKE THEIR MEDICATIONS WHEN THEY SHOULD, SOME NUMBER OF PEOPLE WILL TELL YOU THEY CAN'T AFFORD IT, SOME WILL TELL YOU THEY FORGOT, A FAIR NUMBER OF PEOPLE WILL TELL YOU THAT THEY THINK THE PHARMACEUTICAL COMPANIES BASICALLY RAPE AND PILLAGE PEOPLE, THAT THE DRUGS DON'T DO ANY GOOD AND JUST TRYING TO HOOK PEOPLE ON THEM. IMMENSE FEAR OF THE -- NOT TRUE IN THAT SENSE BUT WHETHER CHOOSING TO IGNORE THAT IS NOT RIGHT BECAUSE YOU'RE NOT HITTING WHERE PEOPLE ARE. THE SOCIAL IMPACT OF BIOMEDICAL RESEARCH ARE IMMENSE. BUT AS I WAS SAYING THEY HAVEN'T RECEIVED MUCH ATTENTION AND INDEED THERE ARE EFFORTS TO TRY AND CUT OUT THE NIH FROM ADDRESSING THESE. WHICH IS POOR BECAUSE IT WILL RESULT IN A SYSTEM WHERE WE'RE JUST DUMPLING FUELD ON FIRE ARETHER THAN THINKING HOW TO MAKE THE WHOLE THING WORK VERY WELL. BY ALL MEANS PLEASE DON'T FORGET ABOUT THESE KINDS OF IMPACTS. IT'S DIFFICULT TO SAY WHAT THE RETURNS BIOMEDICAL RESEARCH ARE WITHOUT THINKING ABOUT THE ENVIRONMENT IN WHICH IT'S GOING IN. I'M GOING TO STOP HERE. THANK YOU VERY MUCH. [APPLAUSE] >> ARE THERE A COUPLE OF QUICK QUESTIONS? >> FORGIVE ME I -- BUT WHY WOULD THIS OVERWHELMING AMOUNT OF DATA THAT PRETTY MUCH EVERYONE AGREES WITH ARE WE SO INCAPABLE OF DOING ANYTHING ABOUT IT? THE SCIENCE -- I WAS SAYING WHAT WE AGREE IS WE AGREE -- IT'S A LITTLE BIT LIKE WE UNDERSTAND HOW THE MOUSE IS DIFFERENCE -- NOT SURE I TRY THIS ANALOGY, I'M BAD AT IT. HOW DO YOU GO ABOUT TURNING SOMETHING THAT DOESN'T WORK INTO SOMETHING THAT WORK, A TECHNICAL ISSUE AKIN TO HOW DO YOU MAKE SURE A GENE DOES WHAT IT SHOULD AND NOT WHAT I SHOULDN'T. SO YOU NEED TO EXPERIMENT AND FIGURE HOW TO MAKE IT WORK. IT'S NOT AS SIMPLE AS SAYING OKAY, I NOW KNOW WHAT'S RIGHT. LET ME TELL PEOPLE AND WE'LL DO THAT. IT TEASE SYSTEM WHICH IT OPERATES, THE INCENTIVES IT OPERATES, THE WHOLE NATURE OF THE PRODUCTION. >> ARE WE MAKING PROGRESS AT ALL TO SOLVING THIS? SEEMS TO ME IT IS VERY, VERY SLOW. >> WE'RE MAKING SMALL AMOUNTS OF PROGRESS BUT NOWHERE NEAR AS RAPIDLY A WE SHOULD BE. >> YOU MADE A COMMENT THAT STRUCK ME, I WANT TO BE SURE I GOT IT RIGHT. A PERSON OF TO A YOUNGER PERSON THREE DECADES AGO? >> THAT'S CORRECT. MEASURE OF SELF-RATED HEALTH OR PHYSICAL IMPAIRMENT, THEY ACT LIKE PEOPLE THEY LOOK AND ACT AND FEEL LIKE PEOPLE YOUNGER.. >> LET ME INTRODUCE MARK MCCLELLAN, THE DIRECTOR OF THE INGLEBERG CENTER FOR HEALTHCARE REFORM AND THE LEE THAT WOULD SCHAFER CHAIR HEALTH POLICY STUDIES, BROOKINGS INSTITUTION. >> THANK YOU VERY MUCH FOR THE INTRODUCTION. VERY MUCH A PRIVILEGE TO BE HERE WITH YOU. I WILL PUT MY PRESENTATION WITH BOB AND DAVE ONE ADVANTAGE I HAVE BETTER ORGANIZE THAN I AM, BEFORE DOING MINE. SO ALL THE SLIDES IN MY PRESENTATION THEY COVER BETTER THAN I DID WHICH LEFT ME NOT MUCH SO I ADDED SOME NEW ONES ON EXTENSIONS YOU HEARD FROM DAVID WHAT YOU'RE GOING TO HEAR FROM BOB. I WANT TO TALK ABOUT THREE THINGS. FIRST IS CATEGORIES OF VALUE, LIKE THE BRING ATTENTION OF THIS GROUP, OBVIOUSLY FRONT AND CENTER IN THAT THE VALUE OF GREATER BIOMEDICAL KNOWLEDGE. DAVID DID A NICE LAY OUT AT THE BEGINNING OF HIS TALK, BOB WILL TUCK MORE ABOUT BENEFITS AND COSTS OF BIOMEDICAL KNOWLEDGE, BENEFITS INCLUDING THINGS LIKE CHANGES IN BEHAVIOR, WE KNOW BETTER THE CONSEQUENCES OF SMOKING, OTHER UNHEALTHY BEHAVIORS AND WE CHANGE ACCORDINGLY LEADING TO BETTER HEALTH. IMPORTANTLY, WE'RE ALSO MOST OF MY TIME AND MOST EFFORT IN THIS GROUP HAVE BEEN FOCUSED IS ON IMPACT OF BIOMEDICAL KNOWLEDGE ON INNOVATION IN HEALTHCARE, THE TECHNOLOGY AND TREATMENT IS AVAILABLE, AND CONTINUE TO OCCUR IN FUTURE YEARS IN BIOMEDICAL ENTERPRISE IMPACTING THE CALL AND LENGTH OF LIFE NOT ONLY AMERICANS BUT PEOPLE AROUND THE WORLD TODAY AND FOR THE FUTURE. AGAINST THIS ARE COSTS OF RESEARCH. I KEEP TELLING YOU IT'S NOT THE NUMBER OF JOBS CREATE BUD THE VALUE OF THINGS THAT THOSE JOBS ARE PRODUCING THAT WE CARE ABOUT. SO THERE ARE COSTS RELATED TO PRODUCING USING THE MEDICAL TECHNOLOGIES, DAVID WEPT THROUGH THESE EXTENSIVELY IN HIS TALK, BOB IS GOING TO DO SO AS WELL, IN MY VIEW THE DIRECT COSTS AND RESEARCH HERE THE AMOUNT SPENDING ON NIH RESEARCH AND AMOUNT IN PRIVATE SECTOR RESEARCH AND DEVELOPMENT ARE SMALL, PUNY COMPARED TO OVERALL BENEFIT AND COST IMPACTS FOR THE POPULATION, SO THAT'S WHERE IF YOU WAN THE ADDRESS THESE QUESTIONS OF VALUE YOU NEED TO FOCUS CAN YOUR EFFORTS. A DAVID ALSO MENTIONED, THERE IS A REAL VALUE IN UNDERSTANDING BEHAVIOR BETTER, THAT HELPS US THINK ABOUT POLICIES THAT COULD INFLUENCE WELL BEING, INFLUENCE EFFECTIVE USE OF TECHNOLOGIES AND SO FORTH. THAT'S ANOTHER IMPORTANT ASPECT THAT I'M NOT GOING TO SPEND TIME TALKING ABOUT NOW. I WANT TO TALK MEASURING VALUE AND PICK UP ON ISSUES THAT DAVID ALREADY DISCUSSED. YOU'LL HEAR ABOUT FROM BOB AS WELL. THERE ARE A NUMBER OF STUDIES DONE IN THIS AREA BY MY COLLEAGUES IN THIS PANEL DOING LEADING WORK, SOME OF THESE LOOK AT DAVID AND BOB TALK AD ABOUT AS WELL WHAT HAPPENED OVER TIME TO IMPORTANT HEALTH OUTCOMES LIKE SURVIVAL, CALL OF LIFE AS WELL, SINCE THAT'S SOMETHING PEOPLE VALUE A LOT TOO THE VERSUS THE COST OF HEALTHCARE OVERALL WHAT YOU MIGHT CALL REDUCE FLOOR MODELS BECAUSE THEY TAKE IN THE SUM TOTAL OF EVERYTHING THAT MIGHT INFLUENCE HEALTH AND MEDICAL COSTS BUT CERTAINLY INCLUDE MEDICAL INN INVESTIGATION BUT PROBABLY RELATED TO OTHER FACTORS AS WELL. AND LOOK AT COMPARISON. NOT ONLY DID THE EVIDENCE CLEARLY SHOW BENEFIT FAR OUTWEIGHED THE COST BUT THERE'S A COMMON SENSE VERSION OF THIS TOO BACK TO DAVID'S COMMENTS ABOUT LIKE TO GIVE A NUMBER, A YEAR, MAYBE NOT BOTH. IF YOU'RE DOING NUMBER AND YEAR AND ASK PEOPLE, THIS WILL BE GOOD AUDIENCE PARTICIPATION TOO, WOULD YOU RATHER HAVE MEDICAL CARE WITH TODAY'S COSTS OR 1990s MEDICAL CARE, 1990 ET CETERA COST IS NOT A CLOSE CALL. SO THAT'S THE INTUITIVE NOTION OF THE RESEARCH OVERALL IMPROVEMENTS IN BIOMEDICAL KNOWLEDGE AND BIOMEDICAL INNOVATION OVERALL HAVE BEEN WORTH SOMETHING. THAT SAID, THERE CLEARLY IS ROOM FOR IMPROVEMENT, I'LL COME BACK TO THAT AS WELL. THERE HAVE BEEN A NUMBER OF STUDIES THAT TRY TO BREAK OUT PIECES OF THE CONTRIBUTIONS DIFFERENT KINDS OF TECHNOLOGIES AN TRACING BACK TO DIFFERENT KINDS OF INVESTMENTS IN BIOMEDICAL SCIENCE UNDERSTANDING MODELS OF DISEASES LEADING TO THINGS LIKE BETA BLOCKER DRUGS OR OTHER MEDICATIONS TRYING TO TRACE THROUGH WHAT THE CONSEQUENCES OF THOSE ARE FOR HEALTH OUTCOMES. IN THE 1990s THEY TOOK A LONG TIME IN ONE STUDY WORKING WITH SOME LEADING CARDIOLOGISTS AROUND THE COUNTRY DO THIS BETTER THAN I DID TO STUDY THE IMPACT OF CONTRIBUTORS IMPROVEMENTS IN SURVIVAL AFTER ACUTE MYOCARDIAL INFARCTION FOR 30 DAYS AFTER SURVIVAL REQUIRE REVIEWING SOMETHING LIKE 400 STUDIES AND COMPILING A LOT OF EVIDENCE COMPLETELY CONSISTENTLY PUT TOGETHER, TO TRY TO GET ESTIMATES OF WHAT DIFFERENT FACTORS INCLUDING CHANGES IN THE MAKE UP OF THE POPULATION HAVING HEART ATTACKS, CHANGES MANY IN PRE-HOSPITAL TECHNOLOGIES, EMERGENCY RESPONSE, THINGS LIKE THAT, CHANGES THIS ACUTE MEDICATION, ICU, UCU USE ET CETERA TO BREAK OUT PIECES, IT TAKES WORK. IT CAN BE DONE BUT'S HARDER AS DISEASES BECOME MORE SOPHISTICATED AND INDIVIDUALIZED BUT THAT HEALTH CONDITION IMPACT MODEL I THINK HAS A LOT OF POTENTIAL FOR PROVIDING ILLUSTRATIVE INSIGHTS WHERE AND HOW IMPROVEMENTS IN BIOMEDICAL KNOWLEDGE AFFECT CARE AND IS AREA WHERE I THINK WOULD REALLY BENEFIT THIS GROUP AND NIH AND AMERICAN PUBLIC MORE GENERALLY IF WE HAVE A BETTER UNDERSTANDING HOW DIFFERENT KINDS OF BIOMEDICAL INSIGHTS TRANSLATE INTO IMPACT ON MEDICAL TREATMENT AND HEALTH OF THE POPULATION. AND CAPACITY TO MEASURE THIS IS IMPROVING. AND SINCE I HAVE SOME TIME THE ADD MORE SLIDES BECAUSE OF EVERYTHING ELSE THE REST OF THE GROUP IS COVERING I TALK A LITTLE BIT ABOUT ILLUSTRATIONS HOW CAPACITY TO MEASURE THE CHANGES IN MEDICAL TECHNOLOGY AND IMPACT ON HEALTH IS GETTING BETTER. FINALLY AS DAVE EMPHASIZED, IT'S A QUESTION ON AVERAGE FOR -- IN INVESTMENTS IN BIOMEDICAL RESEARCH BUT HOW WE INCREASE THEIR VALUE, HOW CAN WE GET TO RARE HEALTHCARE PRODUCTIVITY. WE TALKED ABOUT THAT AND COMPLETELY AGREE WITH DAVID THERE IS POTENTIAL FOR IMPROVEMENT AND I THINK WE'RE TAKING SOME STEPS IN THE RIGHT DIRECTION BUT A LONG WAY TO GO. WHAT I WOULD ALSO LIKE TO EMPHASIZE THOUGH IS OPPORTUNITIES FOR I WILL PROVING THE INNOVATION PROCESS ITSELF. SO IF ON NET BIOMEDICAL INNOVATIONS ARE BENEFICIAL TO THE PUBLIC, THERE CAN BE VALUE GETTING THOSE TO THE PATIENT WHOSE CAN BENEFIT, INDIVIDUALS BENEFIT SOONER RATHER THAN LATER. EVEN THOUGH WE MADE A TREMENDOUS AMOUNT OF PROGRESS UNDERSTANDING GENOMICS OR MOLECULAR BASIS OF DISEASE, SYSTEMS BIOLOGY, IT IS A LONG UNCERTAIN PROCESS TO GET THOSE TREATMENTS INTO PATIENTS INTO THE RIGHT PATIENTS. SO TAKE MY COMMENTS TO COMPLIMENT DAVIS IN HIS TALK ABOUT OPPORTUNITIES FOR IMPROVING PRODUCTIVITY NOT JUST HEALTHCARE DELIVERY BUT INNOVATION PROCESS ITSELF, THE BASIC BUILDING BLOCKS OF BIOMEDICAL PROGRESS OUT OF NIH RESEARCH AND THIS TOO IS MEASURABLE, PROGRESS AND HEALTHCARE PRODUCTIVITY, AND PROGRESS IN EFFICIENCY AND TIMELINESS BIOMEDICAL INNOVATION. IN TERMS OF IMPROVING CAPACITY FOR MEASURING THE VALUE OF BIOMEDICAL KNOWLEDGE, HARD FOR REASONS DISCUSSED BUT THERE'S SOME CHANGES THAT WILL MAKE IT EASIER, COIN SIDE WITH REFORMS IN HEALTHCARE THAT ARE HELPING TO ADDRESS THE INEFFICIENCY PROBLEMS, THE GAS AND HEALTHCARE PRODUCTIVITY TO CHASMS IN PRODUCTIVITY THAT DAVID DESCRIBED. MEDICAL TECHNOLOGIES MAKE IT THE MARKET AND USED IN WAYS LESS THAN IDEAL FOR NUMBER OF REASON,SOME POLICY RELATED, SOME EVIDENCE RELATED. PRE-MARKET CLINICAL TRIAL DON'T ANSWER ALL THE QUESTIONS PRACTICING CLINICS WANT TO KNOW ABOUT TREATMENT. IMPACT ON LONGER TERM OUTCOMES FOR PATIENTS, SAFETY EFFECTS, THE EFFECTS DIFFER ACROSS SUBGROUPS OF PATIENTS BASED ON CLINICAL CHARACTERISTICS, PREFERENCES AND SO FORTH. TO THERE ARE EVIDENCE GAPS. FILLING THOSE GAPS IS A GOAL, CERTAINLY A PREREQUISITE FOR FURTHER PROGRESS IMPROVING THATzV PRODUCTIVITY. REFORMS TO ADDRESS INEFFICIENCY PAID MORE FOR USING MORE STUFF. ARE MOVING FROM JUST MORE TREATMENTS TO SYSTEMS THAT PAY FOR EXAMPLE IN PART BASED ON SET OF IMPACT PROVIDER OUTCOME FOR PATIENTS SO IF YOU CAN GET YOUR BYPASS OPERATION FROM ONE FACILITY VERSUS ANOTHER, IT IS POSSIBLE, INCREASINGLY HAPPENING TO TRACK COMPLICATION RATES AND HEALTH OUTCOMES TO NOSE THOSE PATIENTS HOW RAPIDLY THEY WORK, LONG TERM SURVIVAL RATE, INCREASINGLY FUNCTIONAL STATUS MEASURES, BACK TO THIS TOO, THOSE ARE USED AS A BASIS FOR PAYMENT, SO THESE MEASUREMENT CAPABILITIES ARE IMPROVING. BENEFIT DESIGN AS WELL INCREASING NUMBER OF HEALTHCARE PLANS, SETTING UP NETWORKS OR DIFFERENCES IN CO-PAYMENTS BASED NOT JUST ON COSTS OF THE TREATMENT BUT HEALTH OUTCOMES, AND COST AND COMBINATION. AS A RESULT OF PROGRESS THAT'S HAPPENING THE DEVELOPMENT PROCESS, WE SEE VALIDATE DAYTIMED MARKERS OR PREDICTORS OF IMPACT OF TREATMENTS ON OUTCOMES. SO FOR HIV, 20 YEARS AGO, 15 YEARS, FOR HEPATITIS C TODAY WE DON'T HAVE TO WAIT FOR PEOPLE TO DEVELOP MAJOR COMPLICATIONS OF THE DISEASES LIKE OPPORTUNISTIC INFECTIONS OR TO DIE OR IN THE CASE OF HEPATITIS TO DEVELOP CIRRHOSIS OR HEPATIC CARCINOMA. WE LOOK AT VIRAL LOAD AND BLOODSTREAM AS A GOOD SHORT TERM PREDICTOR OF THAT LONG TERM OUTCOME, VALIDATED INDICATOR REAL IMPACT ON PATIENT HEALTH OBSERVED MORE QUICKLY AND IT WILL LOWER COST. SO EVIDENCE IS GETTING BETTER. THERE ARE GROWING OPPORTUNITIES TO GET MORE SYSTEMATIC EVIDENCE ON IMPACT OF BIOMEDICAL TECHNOLOGIES AND INDIRECTLY THE BIOMEDICAL RESEARCH UPON WHICH THEY ARE BASED THROUGH IMPROVING INFORMATION SYSTEMS, IN PRACTICE TODAY. THESE INCLUDE REGISTRIES, CLINICAL RESEARCH NETWORK, I'LL TALK MORE ABOUT THOSE RIGHT NOW. TO ILLUSTRATE ONE EXAMPLE OF BEING ABLE TO TRACK IN MORE DETAIL AT THE PATIENT LEVEL OVER TIME, SAFETY OUTCOMES INITIATIVE FDA IMPLEMENTED OVER THE LAST FEW YEARS WITH SUPPORT OF NIH AS WELL AS A NUMBER OF PRIVATE INSURERS, CMS AND STATES FROM MEDICAID PROGRAMS. THIS IS A COLLABORATION TO DEVELOP CONSISTENT DATA TO BE ABLE TO TRACK USE OF MEDICATION AND COMPLICATIONS IN MEDICATIONS IN NOW 150 MILLION AMERICANS, NOT COUNTING THE MEDICARE BENEFICIARIES INVOLVED SO THAT'S OVER 200 MILLION, SO MOST OF THE COUNTRY, THIS NOT DONE BY PUTTING DATA FROM DIFFERENT PARTICIPATING ORGANIZES TO ONE DATA WAREHOUSE SOMEWHERE. IT'S DONE BY THESE DIFFERENT ORGANIZATIONS WORKING COLLABORATIVELY TO SET UP A COMMON DATA MODEL, COMMON WAY OF IDENTIFYING PATIENTS USING ELECTRONIC DATA TO HAVE CERTAIN CONDITIONS TAKING CERTAIN MEDICATIONS AND SHARING SUMMARY TANGS FROM THOSE RESULTS. -- STATISTIC FROM THE RESULTS. THIS IS A SLIDE EFFORT TO SUMMARIZE THAT BRIEFLY BUT THERE IS AN OVERALL COORDINATING APPROACH FUNDD BY THE FEDERAL GOVERNMENT BUT STAFF BY ANALYST FROM ALL THESE PARTICIPATING ORGANIZATIONS COME UP WITH DEFINING POPULATIONS OF INTEREST FOR TRACKING TREATMENT USE AND TRACKING OUTCOMES. TO HIRE USING THE SAME MODEL, WALLS WITH FIRE ON THEM ARE REALLY MY EFFORT TO DESCRIBE A FIREWALL. AND THE POINT BEING THAT THE INDIVIDUAL PATIENT DATA STAYS WITH THE HEALTHCARE SYSTEM TRACKING -- CARING FOR PATIENTSP THESE ARE DATA USED TOKER CLINICAL CARE FOR ADMINISTRATION PURPOSES PAYMENT PURPOSES AN LIKE. BUT BECAUSE THEY'RE ALL FOLLOWING THE SAME RULE WHICH PATIENTS ARE TAKING A CERTAIN DRUG AND WHICH HAVE CERTAIN COMPLICATIONS, THEY CAN COMBINE SUMMARY STACKS. MR. JONES SAW DR. SMITH LAST TUESDAY FOR THE CONDITION AND TOOK THE DRUG HOW MANY PATIENTS LIKE MR. SMITH SAW DOCTORS WITH THIS CONDITION AND TOOK THIS MEDICATION. NUMERATOR AND DENOMINATOR THAT MATTERS, THE BASIC IDEA THIS IS MORE STACK AND MORE MULTI-VARYIAN. THAT'S THE BASIC IDEA. JUST TO GIVE AN EXAM) HOW THIS IS BEING USED NOW, FDA IS DOING NOW -- BUT AVERAGING OUTCOMES OF DAILY BASIS OF THESE RAPID QUERIES OF A POPULATION OF OVER 100 MILLION AMERICANS FOR ASSOCIATIONS BETWEEN OUTCOME SAFETY PROBLEMS OF INTEREST AND USE OF THE NEW USE OF MEDICATIONS OR LONG TERM USE OF MEDICATIONS OR LONG TERM USE OF MEDICATIONS IN PARTICULAR SUBGROUPS OF PATIENTS AND THE LIKE. SO TO GIVE YOU ONE EXAMPLE NOT TO PICK ON THIS PARTICULAR ARB, IT'S A DRUG THAT IS INTENDED TO HELP WITH HIGH BLOOD PRESSURE IN CERTAIN PATIENTS. THE FDA GOT SOME REPORTS ASSOCIATED WITH A VERY SERIOUS GASTROINTESTINAL DISEASE CALLED SEAL YAK DISEASE FROM RANDOM CASE REPORTS, DIDN'T HAVE A DENOMINATOR, COULDN'T REALLY TELL IF THIS WAS AN UNUSUAL PATTERN. HOW MANY TAKING THIS DRUG DEVELOP SILL YAK DISEASE COMPARE TO OTHER SIMILAR DRUGS FOR HIGH BLOOD PRESSURE. IN A DAY BECAUSE THIS POPULATION COVERED CLOSE TO 100 MILLION PATIENTS, A VERY SMALL FRACTION OF ADULT POPULATION ALL THE DRUGS ON THOUSANDS OF PATIENTS TAKING EACH OF THESE DIFFERENT KINDS OF DRUGS IN THIS CLASS AND 20, 30,000, 200,000 PATIENTS IN EACH CATEGORY, PRECISE ESTIMATES OF ASSOCIATION OF SERIOUS ADVERSE EVENTS HEREBY, SILL YAK DISEASE, HOSPITALIZATIONS AND COMPLICATIONS WITH THE DISEASE, FOR ALMOST ALL IN PARTICULAR, NOT OUT OF LINE WITH THE RACE SEEN FOR OTHER DRUGS, IT'S RIGHT IN LINE SO THAT SUGGESTED FOR FDA THERE WASN'T UNUSUAL SAFETY SIGNAL HERE DESPITE A FEW ADVERSE EVENT REPORTS THEY HAVE HAD. MY POINT FOR YOU IS THAT THIS IS ONGOING NOW FOR A DAILY BASIS AND FOR TRACKING A QUESTION LIKE THOSE OF INTEREST OF THIS GROUP, HOW THINK ASSOCIATE WITH WITH CHANGES, TRENDS AND OUTCOMES OVER TIME FOR PARTICULAR KINDS OF PATIENTS, THIS IS A MUCH BETTER CAPACITY TO DO THOSE STUDIES ROUTINELY THAN HAVING TO GO BACK AND REVIEW HUNDREDS OF THOUSANDS OF STUDIES FROM THE LITERATURE N. THESE DATABASES PUT TOGETHER IN THESE LARGE DISTRIBUTED DATA SYSTEMS, THERE'S INCREASINGLY SOPHISTICATED MEASURE OF OUTCOMES AND COSTS. SO A LOT OF THESE STUDIES INCLUDING THE ONES DESCRIBED USING FOR SAFETY STARTED OUT, CAPTURED ADMINISTRATIVE CLAIMS DATA BUT AS MORE SYSTEMS ADOPT ELECTRONIC RECORDS APPROACHES, WITH WELL DEFINED MEASURE OF CLINICAL OUTCOMES LIKE HEMOGLOBIN A 1C LEVELS, BLOOD SUGAR LEVELS, IN DIE PEE TEASE, FUNCTIONAL OUTCOMES, HOW WELL THEY'RE PERFORMING DAILY ACTIVITIES OR HOW MUCH ACTIVITY THEY CAN TOLERATE IF THEY GOT HEART DISEASE. IT'S BECOMING POSSIBLE TO DO MUCH MORE SOPHISTICATED TREND ANALYSES TO GET DATA SETS AND FUTURE, I THINK SUGGESTS MORE PROGRESS ALONG THESE LINES. MODE YUM AND LONG TERM RECOMMENDATIONS FOR HOW TO SYSTEMATICALLY GET A BETTER HANDLE ON THE IMPACT OF BIOMEDICAL RESEARCH, VERY IMPORTANT TO THINK ABOUT CAPACITIES THAT ARE EMERGING TO HELP UNDERSTAND BETTER HOW MEDICAL TREATMENTS ARE USED AND ASSOCIATION WITH IMPORTANT OUTCOMES OR COSTS AS WELL. SO WANT TO CONCLUDE BY SPENDING A FEW MINUTES ON OPPORTUNITIES TO IMPROVE PRODUCTIVITY, NOT JUST CHALLENGES OR OPPORTUNITIES FOR MEASURING IT, AND I'M GOING TO COMPLIMENT DAVID'S WORK OR DAVID EAT PRESENTATION BY FOCUSING ON THE INN INVESTIGATION PROCESS ITSELF F YOU TALK TO HUB IN THE FIELD, THEY WILL TELL YOU THEY THINK THERE'S ROOM FOR IMPROVEMENT. THERE ARE DIFFERENCE VIEWS BUT ONE WIDELY CITED STATISTIC SOMETHING LIKE THIS, THERE ARE MEASURES THAT CAN BE TRACKED THAT ARE BEING TRACKED, THIS IS A MEASURE OVER TIME AVERAGE APPROVED SPENDING BY PHARMACEUTICAL AND BIOTECH COMPANIES ON RESEARCH AND DEVELOPMENT AND YOU CAN SEE THERE'S A STEADY DOWNWARD LINE ON THIS EXPONENTIAL CHART. SOME HAVE CALLED THIS (INAUDIBLE) REVERSE OF MORE'S LAW, FOR THOSE FA MILL CONSIDER WITH THAT, WITH SEMICONDUCTORS, WE HAVE PRODUCTIVITY YEAR AFTER YEAR, AT LEAST BASED ON THIS KIND OF METRIC AND THE DEVELOPMENT PROCESS ITSELF, IF YOU LOOK BY DISEASE AREA, SHOWS VARYIABILITY. IT'S LOW PRODUCTIVITY SO OF THOSE TREATMENTS THESE ARE DRUGS. SO DRUGS THAT ENTER INTO CLINICAL TESTING IN HUMANS, AT LEAST SO CALLED PHASE 1 TRIALS HOW MANY GO ON TO BE APPROVED IS A PRETTY LOW RATE OVERALL BUT BURIED A LOT AT LEAST VARIED HISTORICALLY ACROSS DISEASE AREA S. AND IT TAKES TIME SO FAILURES OCCUR LATE IN THE PROCESS, NOT JUST PHASE 2 BUT DOWN THE LINE, SO THAT'S A LOT OF TIME AS WELL. YOU HAVE SEEN THE AVERAGE DEVELOPMENT TIME TRENDS OR SOME OF YOU HAVE SEEN AVERAGE DEVELOPMENT TRENDS FOR DRUGS NOT GOOD EITHER. 7 TO 12 YEARS DEPENDING ON THE CLINICAL AREA. A LITTLE GOOD NEWS IS MAYBE AFTER A DOWNTURN IN APPROVAL OF NEW DRUGS IN EARLY TO MID 2000S THE LAST COUPLE OF YEARS INCLUDING 2013 AS WELL, HAS SEEN A MARKED INCREASE IN THE NUMBER OF NEW DRUG APPROVALS AND WHAT FDA CALLS PRIORITY REVIEW, NOT A PERFECT MEASURE OF DRUGS THAT ARE BREAK THROUGHS FOR UNMET MEDICAL NEEDS BUT INDICATOR OF THAT, I WANT TO EMPHASIZE THERE'S MEASURES THAT COULD BE USED FOR HOW QUICKLY IDEAS THAT MAY GET INTO CLINICAL TESTING FROM BASIC BIOMEDICAL RESEARCH ARE REACHING AND IMPACTING PATIENTS OR METRICS THAT COULD BE USED TO DO EVALUATIONS OF THAT MORE SYSTEMATICALLY AND SEEMS TO BE ON UPTURN IN DISCUSSION TIME WE CAN TALK MORE ABOUT THAT. THERE ARE A NUMBER OF REPORTS RECENTLY. WE HAVE DONE SOME AT BREAKINGS, SCIENCE AND TECHNOLOGY REPORT OUT LAST FALL THAT IDENTIFIED A A NUMBER OF WAYS TO REDUCE TIME AND IMPROVE EFFICIENCY OF THIS DEVELOPMENT PROCESS. I'LL LEAVE THESE SLIDES FOR YOU BUT A LOT OF IT COMES DOWN TO MAKING EXPLICIT FOCUS MAKING THE DEVELOPMENT PROCESS MORE EFFICIENT. IMPROVING DEVELOPMENT SIGNS SO WE HAVE BIOMEDICAL SIGNS, DELIVERY SCIENCE DELIVERY SCIENCE, ALSO ADD IN DEVELOPMENT SCIENCE ANOTHER AREA WHERE THERE ARE SOME REAL EVIDENCE OF GAPS IN EFFICIENCY THAT COULD BE GETTING IN THE WAY OF MUCH MORE PRODUCTIVE IMPACT. MUCH MORE IMPACT OF BIOMEDICAL RESEARCH, FOR AMERICAN PUBLIC AND PUBLIC HEALTH AND AGAIN THESE ARE ALL -- THESE ARE ALL MEASURABLE STEPS. SO LET ME CONCLUDE THERE. THANKS AGAIN FOR THE OPPORTUNITY TO JOIN Y'ALL TODAY. [APPLAUSE] >> SHORT QUESTIONS? THANK YOU VERY MUCH. OUR NEXT SPEAKER HAS BEEN HERE ONCE BEFORE, ROBERT TOPEL, PROFESSOR OF URBAN AND LABOR ECONOMICS UNIVERSITY OF CHICAGO BOOTH SCHOOL OF BUSINESS. THANK YOU FOR HAVING ME. LET ME BEGIN BY PUTTING IN MY TWO CENTS WORTH ON THIS CONCH TROUVERES OVER THE SOCIAL AN ECONOMIC BUDGET HERE AT NIH. YEARS AGO DAVID COVERS TESTIFIED ON EXACTLY THIS POINT BEFORE CONGRESS WHEN THERE'S DISCUSSION OF GETTING RID OF THE NSF ECONOMICS BUDGET. YOU MIGHT NOT BE SURPRISED BUT THE ARGUMENT IS ECONOMIC. HE SAID IF YOU BELIEVE IT SHOULD BE -- WHAT YOU'RE SAYING IS THE WORST PHYSICS PROJECT IS BETTER THAN THE BEST ECONOMICS PROJECTS SO IN THE ABSENCE OF FIXED COST YOU HAVE TO BELIEVE IN ORDER FOR THAT TO BE TRUE. YEARS LATER I SPENT TWO TERMS ON THE NSF REVIEW PANEL THROWING AROUND NICKELS THAT WE HAD FOR ECONOMICS. ALLOCATED AND WHAT THE SIZE WAS, ON THE WAY HOME I WAS FLYING HOME, I WAS ON -- SITTING NEXT TO A PHYSICIST FROM RUTGERS AND WE STARTED TALKING ABOUT THE SIZE OF GRANTS, STUFF LIKE THAT. TURNS OUT AT THE TIME RUTGERS AS I UNDERSTAND FROM MY DISCUSSION WAS THE 20th RANKED PHYSICS DEPARTMENT THROUGHOUT THE COUNTRY. HIS INDIVIDUAL PHYSICS GRANT WAS BIGGER THAN THE ENTIRE NSF ECONOMICS BUDGET SO IF IT'S ANYTHING LIKE THAT WITHIN NIH I SUSPECT IT COULD BE, THEN YOU HAVE TO THINK MARGINAL PROJECT VERSUS BEST ECONOMICS PROJECT T. THE ORIGIN OF THIS WORK CAME WHEN THE PRESIDENT OF OUR UNIVERSITY WHO BECAUSE HE WAS THE PRESIDENT OF A MAJOR RESEARCH UNIVERSITY KIND OF LIKE NIH HAD AN INTEREST IN FUNDING FOR MEDICAL RESEARCH. WHAT'S IT WORTH? MY FRIEND KEVIN MURPHY AND I DID THIS WORK TOGETHER, WE WERE PLEASED, I WAS PLEASED TO ANSWER QUICKLY AND I DID. I SAID I DIDN'T KNOW BUT WE SAID WE'D GET BACK TO THEM. THAT WAS THE GENESIS OF THIS WORK. THE PROBLEM THAT WE'RE TALKING ABOUT IS REALLY A LOT TO DO WITH WHAT ECONOMISTS CALL PUBLIC GOODS BECAUSE THE THING PRODUCED HERE IS KNOWLEDGE, KNOWLEDGE IS PUBLIC GOOD. IT'S SOMETHING THAT ONCE PRODUCED MARGINAL COST OF EXTENDING USE TO GREATER POPULATION OR NUMBER OF USERS IS EFFECTIVELY ZERO. THAT'S A DIFFICULT THING TO VALUE. BACK IN THE 19th CENTURY THE GREAT ECONOMIST ALFRED MARSHALL JOINED THE DEBATE OVER THE VALUE OF EDUCATION, THE SOCIAL VALUE OF EDUCATION, NOT JUST THE PRIVATE VALUE, HOW MUCH IT INCREASES YOUR EARNINGS PRIVATELY BUT HOW MUCH DOES IT CONFER BENEFITS ON OTHERS. I GUESS AT THE I'M THERE WAS A DEBATE ABOUT ALL THE MONEY THAT WAS BEING SPENT EDUCATING THE GREAT UNWASHED IN 19th CENTURY ENGLAND. HE POINTED OUT FOR ALL MONEY WE'RE SPENDING, ALL PRODUCE BECAUSE WE'RE PRODUCING IDEAS IF WE PRODUCE A NEW TON IT'S WORTHWHILE. WHAT HE WAS TALKING ABOUT IS THIS RARE EVENT THAT COMES ALONG BUT PUBLIC GOOD, A NEW SET OF IDEAS THAT MAKE IT A BETTER PLACE. THOSE ARE NOT HARD TO VALUE. NOT LONG AFTER WE DID THIS PROJECT, NIH ISN'T THE ONLY SET OF INSTITUTIONS TO WANT THE SAY, LET ME -- FOR WHAT IT WAS WORTH. SO I WAS APPROACHED BY THE HEAD -- ONE OF THE NATIONAL LABS ARGON, THE NATIONAL LABS WANTED TO HAVE A PROJECT. WHAT IS THIS WORTH WORTH? WHAT IS THIS KNOWLEDGE WORTH? DO YOU HAVE METRICS LIKE WE USE AS YOU'LL SEE IN A MINUTE HOW MUCH LONGER PEOPLE ARE LIVING, THINGS LIKE THAT. METRIC LIKE THIS, WHAT DO YOU MAKE? I THINK WE HAVE SOMETHING TO DO WITH MAKING TANG. THAT WAS THE NATIONAL -- THAT WAS NASA. BUT THE PROBLEM WITH -- (OFF MIC) >> YEAH, BUT THE POINT BEING THAT IT'S VERY HARD TO NAIL DOWN THE VALUE OF THESE IDEAS THAT ARE THEMSELVES PUBLIC GOODS. SO LET ME GO INTO WHAT WE DO HERE IF I CAN MAKE THIS GO. HERE WE GO. SO WHAT ARE OUR GOAL? WE WANT TO MEASURE THE VALUE OF HEALTH IMPROVEMENTS AS THEY OCCURRED OVER TIME AND TO EXPAND IT SEEMS TO MEASURES TO INCLUDE MEDICAL ADVANCES SO SOMETHING TO SAY IN FEW MINUTES PRODUCTION OF HEALTH CAPITAL UNCOUNTERRED IN GROWTH ECONOMIC ACTIVITY. WE WANT TO MEASURE POTENTIAL GAPES FROM FUTURE ADVANCES THAT'S A BIG ROLE THINKING THE VALUE OF AN INSTITUTION LIKE NIH. WE WANT TO RELATE THOSE POTENTIAL GAINS TO THE COSTS OF ALLOCATED, METHODS ALLOCATING HEALTH RESOURCES PLAGUED OTHERS POINTED OUT BY THIRD PARTY PAYMENTS PUBLIC AND PRIVATE. THAT'S DISTORTION THAT LEADS TO DISTORTION OF INCENTIVES AND IN IT WE FIND RESEARCH INCENTIVES AS WELL. HERE IS A GRAPH OF LIFE EXPECTANCY IN THE UNITED STATES THAT CAME FROM OUR EARLIER RESEARCH. INADVERTENTLY I HAVE TWO GRAPH, THE THICK ONES ARE LIFE EXPECTANCY AT BIRTH AND THE THIN ONES ARE LIFE EXPECTANCY FOR FEMALES AND MALES AGE 50. INADVERSE TENLY IODATE OFF THE AXIS TO TELL YOU HOW MANY YEARS PEOPLE ARE GETTING BUT IT'S GOING UP. DESPITED THAT THESE CALCULATIONS ARE NOT AS IF PEOPLE LIVE THEIR ENTIRE LIVES WITH MORTAL LTY RATES OF THE YEAR ALONG THE HORIZONTAL AXIS. WE DIDN'T WANT TO THE LEAD YOUR ENTIRE LIFE WITH PANDEMIC OF 1919. THAT EAT THAT SPIKE DOWN THERE IS TELLING YOU. THE BIG DEAL IS THAT WE PICKED UP ADDITIONAL LIFE YEARS. IF YOU LOOK IN THE MIDDLE, CAN I MAKE THIS WORK? NO I CAN'T. IF YOU LOCK IN THE MIDDLE ESPECIALLY MALES, THINGS ARE FLAT FROM 1950 TO 1970. THERE WAS A SLOW DOWN IN GROWTH. IT'S NOT IMMEDIATELY OBVIOUS WHAT THAT IS BUT IT MAY HAVE TO DO WITH THE INCREASE IN SMOKING THAT OCCURRED IN THE WORLD WAR II GENERATION AND REDUCTION OF SMOKING AFTERWARDS LED TO ASSOCIATED WITH INCREASED LONGEVITY BUT MOST CAME FROM HEART DISEASE. HERE ARE A COUPLE OF KEY DISTINCTIONS IN VALUE BIOMEDICAL ADVANCES. THINKING RETROSPECTIVELY, BACK TO THINGS THAT HAPPEN IN THE PAST, WE CAN GIVE -- WE MEANING MY COLLEAGUE KEVIN AND ME, WE CAN PLACE A BALLPARK EVALUATION ON HEALTH GAINS WHAT THOSE PAST HEALTH GAMES HAVE BEEN WORTH. WE CAN'T CONFIDENTLY ATTRIBUTE WHAT GAINS WERE CAUSED BY. IT COULD BE CAUSED BY MYRIAD REASONS AND DAVID HAS ALREADY TALKED ABOUT SOME OF THOSE. OR SOME OF THAT PROBLEM. PROSPECT TESTIFILY WE CAN ESTIMATE THE GROSS VALUE THAT WE WOULD RECEIVE. I WANT TO PUT EMPHASIS ON GROSS VALUE. WHAT ADDITIONAL LIFE YEARS WOULD BE WORTH FROM REDUCING MORTALITY FROM HEART DISEASE OR CANCER OR DIABETES, BY A FEW PERCENTAGE POINTS. WE CAN'T AS ECONOMISTS AT LEAST THE TWO ECONOMISTS THAT AUTHORED THIS WORK ARE NOT IN A POSITION OF COST OF ACHIEVING THOSE GAINS SO WE CAN SAY WHAT LIFE YEARS WOULD BE WORTH, COST OF GET THEM IS ANOTHER MATTER. THAT'S AN IMPORTANT POINT COMING DOWN THE ROAD. SO BASIC CONCLUSIONS. HISTORICAL IMPROVEMENTS IN LIFE EXPECT TAPPICE IS VALUABLE. FROM 1900 TO 2000, A STATISTIC, BACK IN 1900, I THINK IT WAS 18% OF MALES DIED BEFORE THEIR FIRST BIRTHDAY. BY # THOUSAND IT WASN'T 62 OR 18% DIED THAT'S DRAMATIC REDUCTION IN MORTALITY. THESE GAPES, ABOUT 1.2 MILLION PER PERSON TO THE CURRENT POPULATION. THE UNCOUNTED PRODUCTION OF HEALTH CAPITAL THEN ACCOUNTS FOR 25% OF GDP. IT WAS A BIGGER NUMBER BACK IN THE EARLY PART OF THE CENTURY WITH SMALLER TOWARDS THE END, WE'LL TALK IF WE HAVE A CHANCE ABOUT REASONS FOR THAT TOO. THE GAINS FROM 19 -- OVER THREE DECADES FROM 1970 TO 2000 WERE WORTH $95 TRILLION OR ABOUT 3.2 TRILLION, A FLOW VALUE OF $3.2 TRILLION PER YEAR. GDP IS 14 TO $5 TRILLION, SO A LARGE PROPORTION OF OTHER MEASURES OF ECONOMIC ACTIVITY. SO THAT HAS IMPORTANT IMPLICATIONS FOR THE WAY WE THINK ABOUT ECONOMIC GROWTH AS WELL. HERE IS KEY GAMES PLAYED OUT. THIS IS GETTING TO THE 1.2 MILLION, YOU SEE RAPID GROWTH AT THE BEGINNING OF THE CENTURY WITH LOW HANGING FRUIT IF YOU WILL PICKED OFF BECAUSE WE REDUCED MORTALITY INFANT MORTALITY FROM CHILDHOOD DISEASES, THEN THERE WAS A SLOW DOWN AROUND 1950 TO 1970, BOTH MEALS AND FEMALES BUT ESPECIALLY MALES. THEN THERE WAS UPTICK AT THE EN. UNLIKE THE STUFF THE LOW HANGING FRUIT AT THE BEGINNING THAT WAS OCCURRED AT YOUNG AGE, THE STUFF AT THE END OCCURRED AT MORE ADVANCED AGES, PEOPLE AT 60 HAVE MORE LIFE YEARS TO GO THAN THEY WOULD HAVE HAD EXPECTED VALUE BACK IN 1950s OR 1960s. FUTURE GAINS ARE ALSO LARGE. IF YOU CAN CURE CANCER YOU MIGHT SAY THAT'S FARFETCHED BUT IF YOU COULD IT WOULD BE OPT ORDER OF 47 TRILLION TO THE U.S., VALUE TO THE CURRENT GENERATION AND FUTURE GENERATIONS AS YET UNBORN BECAUSE WE ARE MAKING A PUBLIC GOOD. IF WE HAVE THE KNOWLEDGE TO CURE CANCER IN TODAY'S POPULATION WE HAVE THE KNOWLEDGE TO CURE FUTURE GENERATIONS AS WELL THAT WILL BENEFIT BY LIVING LONGER AND NOT DYING FROM THE DISEASE. SAME IS TRUE FOR HEART DISEASE. REMARKABLY SIMILAR NUMBERS. PART OF THE REASON IS YOU HAVE SEEN, IN EARLIKE CHART MORTALITY FROM HEART DISEASE AND CANCER GET CLOSER TOGETHER AND SOMEBODY SAID THEY CROSS THIS YEAR OR SOMETHING. IT IMPLIES MODEST PROGRESS HAS GREAT VALUE. WHEN I TALK IN CONFERENCES LIKE THIS BEFORE OTHER MDs, ESPECIALLY ONCOLOGISTS I SAY IF WE GAVE YOU ANOTHER $10 BILLION A YEAR RESEARCH MONEY FOR ALL ONCOLOGISTS COULD YOU REDUCE MORTALITY FROM CANCER BY 10%? ANSWER IS ALWAYS YOU BET. GIVE US THE MONEY. SO LET'S ASSUME 10% IS ON HIS TABLE. 10% WOULD BE WORTH 4.7 TRILLION OR THIRD OF THE YEAR'S GDP, A BIG NUMBER. HISTORICAL REDUCTIONS IN HEART DISEASE FROM 1970 TO 2000 WERE WORTH ABOUT 35 TRILLION BECAUSE ALL THOSE ADDITIONAL LIFE YEARS PEOPLE WERE GETTING. THIS NUMBER IS FROM THE UNITED STATES ONLY. SO IT DOESN'T COUNT THE PUBLIC GOOD SPILL OVERS THAT OTHERS GAIN FROM KNOWLEDGE THAT WAS PRODUCED IN THE US. SO IF YOU TALK MORTALITY IN EUROPE AFRICA OR CHINA, REDUCED BECAUSE OF THE IDEA -- THIS IS ALL WITHIN THE US. IT VALUES LIFE YEARS ONLY, DOESN'T COULD UNDERSTAND HEALTH DRIVEN IMPROVEMENTS IN THE QUALITY OF LIFE. FIVE WEEKSING A I GOT MY KNEE REPLACED. IT WON'T MAKE ME LIVE LONGER. UNLESS I HAVE TO OUTRUN A BEAR OR SOMETHING LIKE THAT. BUT MY DOCTOR SAID YOU PUT IN TERMS ECONOMISTS CAN UNDERSTAND, IT WILL GIVE YOU A BUNCH OF CONSUMER SURPLUS, IT'S GOING TO HURT LESS BUT I CAN TRAN LATE INTO SOMETHING MORE MEANINGFUL TO ME. HERE IS MEDICAL RESEARCH COST OF CARE. SMALL BY COMPARISON OF GAIN. ABOUT 50 BILLION IN THE U.S., THERE'S BEEN DECLINE OF VALUE AT NIH, 3.5 OF DIRECT HEALTH EXPENDITURES IS ON BASIC RESEARCH. POTENTIAL GAINS FROM MEDICAL RESEARCH ARE LARGE BUT COULD BE OFFSET BY INCREASED COST OF CARE. WE'RE TALKING GROSS VALUE. WHAT WOULD THE LIFE YEARS BE WORTH? NOT HOW IT COSTS TO ACHIEVE THOSE. KEY ISSUE IS COST OF IMPLEMENTING THESE INNOVATIONS. IT'S MORE IMPORTANT DIRECT EXPENDITURES, I'LL GIVE AN EXAMPLE AT THE END IF WE HAVE THE TIME. FOCUS ON OUTPUTS OF RESEARCH RATHER THAN THE INPUTS. DISTORTIONS IN DISTRIBUTION AND USE FOR EXAMPLE CALL -- CAUSED BY THIRD PARTY PAYER SYSTEMS WHERE PEOPLE HAVE INCENTIVE TO OVERUSE HEALTHCARE, AFFECT THE VALUE OF INNOVATIONS CH HERE IS OUR BASIC METHOD. FOR PUBLIC POLICY PURPOSES IN THE UNITED STATES TODAY, THE ENVIRONMENTAL PROTECTION AGENCY USE A VALUE STATISTICAL LIFE OF ABOUT A LITTLE OVER $6 MILLION, THEY RAISED IT OF LATE. SO THAT IS OUR BASIC NUMBER VALUE OF LIFE BUT WE HAVE TO EXTRACT FROM THAT THE VALUE OF LIFE YEAR. BECAUSE WHEN WE TALK ABOUT REDUCE MORE IT WILLTY IN DIFFERENT AGES WE SAY BEING ALIVE AT RAREIOUS AGES OVER THE LIFE CYCLE SO $6.3 MILLION HAS TO BE DIVIDED UP, IF YOU WILL, ALLOCATEED TO LIFE YEARS OVER THE LIFE CYCLE. SO WILLINGNESS TO THE PAY FOR LIFE YEARS WHAT IS YEAR OF LIFE WORTH TO PERSON LIVING IT. THE WAY ECONOMISTS THINK IS THE FLOW OF CONSUMER SURPLUS ON CONSUMPTION AND INCOME IN CONSUMPTION WE HAVE IN A GIVEN YEAR. WHAT IT'S WORTH TO BE ALIVE A GIVEN YEAR? FLOW OF GOODS AND SERVICES, YOU UNTILTY AND EXPRESSED IN DOLLAR TERMS THAT'S CONSUMER SURPLUS. IT YIELDS A LIFE CYCLE VALUE FOR A REPRESENTATIVE PERSON SO I OFTEN ASK THE HYPOTHETICAL QUESTION, WE DON'T GO AROUND THE ROOM THE ANSWER BUT IF I CAN GIVE YOU ONE MORE YEAR AT ANY POINT IN YOUR LIFE, WOULD YOU RATHER HAVE IT AT 35 OR RATHER AT 85? SO YOU CAN LIVE ANY ONE OF THOSE TWO YEARS OVER AGAIN, MOST WOULD SAY GIVE ME 35, WE'RE HAVING MORE FUN THEN. THAT IS WILLINGNESS TO PAY IN THE EARLIER LIFE CYCLE OR PRIME YEARS IN THE LIFE CYCLE IS HIGHER THAN YEARS AT THE END OF LIFE CYCLE WHERE HEALTH IS DEPRECIATING. THAT IS MIND THE YELLOW LINE. DON'T WORRY PINK OR BLUE. IT EXPRESSES FOR THE 6.3 VALUE, 6.3 MILLION VALUE OF STATISTICAL LIFE FOR PRIME AGE PERSON IN THE UNITED STATES, WHAT WOULD THE VALUE OF LIFE YEARS LOOK LIKE OVER TIME TO THE ADD TO THAT, TAKING ACCOUNT OF THIS DEPRECIATION OF HEALTH, I WON'T GO INTO TECHNICAL DETAILS HOW WE BACK IT OUT OVER TIME SO VALUE GOING DOWN AS PEOPLE AGE, IT PEAKS AROUND AGE 50 SO THOSE ARE VALUABLE LIFE YEARS IN CALCULATIONS. SO HERE IS VALUE OF REMAINING LIFE, $6.3 MILLION VALUE, THAT'S SMALL BECAUSE DOWN TO PERIODS YOU HAVE LOW VALUED LIFE YEARS AND YOU DON'T HAVE MANY LEFT. THAT'S WHAT FIGURE IS SHOWING HERE. SO WHAT ARE THE IMPLICATIONS? ONE IMPLICATION IS WILLINGNESS TO PAY FOR HEALTH IS PROPORTIONAL TO INCOME. IF WE GET RICHER THE LIFE YEARS ARE GETTING MORE VALUABLE TO US. SO POLITICAL PRESSURE FOR INCREASED HEALTH RESEARCH AND STUFF HAVE GONE UP, WE HAVE IN OUR AGING POPULATION. BUT COUNTRIES ALSO SPEND A LARGER FRACTION OF INCOME BECAUSE THE INCOME WILL LAST DEMAND FOR HEALTH IS BIGGER THAN ONE. WE'LL SPEND LARGER PROPORTION OF INCOME ON PURCHASING HEALTH. ECONOMIC GROWTH THEREFORE RAISES THE VALUE OF HEALTH INNOVATIONS, RICH SOCIETIES WILLING TO PAY MORE. SHARE HEALTH SPENDING CONTINUES TO RISE, THAT'S A PREDICTION MOST ECONOMISTS GO ALONG WITH. INTERESTING ASPECT OF THIS STUFF IS THAT THE PROGRESS AGAINST THE DISEASE IS GREATER WHEN THE CURRENT AGE IS CLOSE TO BUT BEFORE AGE TYPICAL ONSET. SO PEOPLE WHO ARE REALLY YOUNG, THEY DON'T WORRY ABOUT HEART ATTACKINGS VERY MUCH. SO THE PRESENT VALUE OF THAT WHEN YOU'RE 50 OR 20 ISN'T BIG BUT WHEN YOU'RE 55, PROGRESS AGAINST HEART DISEASE IS A BIG DEAL. SPENT BE'S POINT WE CALL COMPLIMENTARITY. PROGRESS AGAINST HEART DISEASE, AS LONG AS THESE DISEASES ARE AGE RELATED, PROGRESS AGAINST HEART DISEASE MAKES IT MORE VALUABLE TO SOLVE ALZHEIMER'S. IN THE OLD DAYS WHEN WE ALL DIED OF HEART ATTACKS WHEN WE WERE 50, WE DIDN'T CARE ABOUT ALZHEIMERS. NOW IT IS. THE FACT THAT RAISES THE VALUE OF RESEARCH AGAINST THESE AGE RELATED DISEASES. SO HEALTH ADVANCES RAISE THE VALUE OF FURTHER HEALTH ADVANCES IS THE WAY TO THINK OF IT. SO WHAT WE DON'T DO, I'M GOING TO REPEAT SOME POINT THAT BOTH PREVIOUS LEADERS TALKED ABOUT. THE VALUE OF HEALTH IMPROVEMENTS IS NOT THE CONTRIBUTION OF HEALTHCARE EXPENDITURES TO MEASURE GDP. THOSE ARE COSTS NOT BENEFITS. NOT ADDITIONAL PRODUCTIVITY, NOT JOBS, WHEN I FIRST DID THIS, THAT'S ALL KNIFE, PROFESSOR BUT HOW MANY JOBS WILL THIS CREATE IN MY STATE? WHICH HAPPENED TO BE OREGON. AND THAT'S NOT THE POINT. WHAT THE VALUE OF THIS BEING CREATED IS THE VALUE PEOPLE GET FROM BEING ALIVE. ANOTHER WAY TO THINK ABOUT THE PRODUCTIVITY POINT IS IF EVERYBODY RETIRED AGE 65, AND THEN WE ADDED LIFE YEARS AFTER 65, THOSE YEARS ARE WORTH SOMETHING TO PEOPLE. WE'RE NOT MAKING THE ECONOMY MORE PRODUCTIVE, WE'RE MAKING THE WORLD A BETTER PLACE BECAUSE YOU GET TO ENJOY THAT TIME WITH GRANDKIDS DOING WHAT YOU DO OVER AGE 65. SO PEOPLE CARE A LOT MORE THAN PRODUCTIVITY. AND I HAVE ALREADY MADE THAT LAST POINT. WE DO MEASURE HEALTH AND LONGEVITY, THAT'S WHAT MATTERS. I'M GOING TO GO THROUGH THIS FAST. (OFF MIC) >> OKAY. SO OVER HERE I HAVE THE INCREASE LONGEVITY FOR MEN WE CALCULATED FROM 1970 TO 2000. I BROKE IT OUT BY DECADES. SO THE BOTTOM PART IS FROM 1970 TO 1980, THE RED PART IS 1980 TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST >> SINCE WE DO HAVE A MINUTE, YOU HAVE DONE A GOOD JOB, DAN, OF KEEPING US ON SCHEDULE HERE. I'LL RELATE -- I MAY HAVE RELATED THE FIRST PART OF THIS STORY ONCE BEFORE, IF P I HAVE, I APOLOGIZE. BUT I WAS TESTIFYING BEFORE THE CONGRESS A FEW MONTHS AGO AND ONE OF DAVID'S COLLEAGUES AT HARVARD WAS THERE, HE WAS A MEDICAL DOCTOR Ph.D. AND WAS TALKING ABOUT HIS INTEREST IN SEALS. WHEN HAIR -- AND HE BECAME VERY CURIOUS ABOUT HOW IT IS THEY STAY IN THE ICE SO LONG WITHOUT AIR. THEIR STUDY LED THEM TO AN IDEA THAT AN IDEA THEY WERE ABLE THE APPLY AND SURGERY FOR CERTAIN DISEASE, SOMETHING TO DO WITH THE LUNGS AND HE SAID DURING THE HEARING IT'S BEEN ABLE TO SAVE 15,000 LIVES A YEAR OF CHILDREN. THIS COUNTRY. WHICH STRUCK ME AS A VERY BIG DEAL. MADE THE IMPORTANCE OF BASIC RESEARCH. MORE RECENTLY JUST THE LAST COUPLE OF WEEKS I WAS ON THE HILL TALKING TO THE INDIVIDUAL WHO HAVE BEEN -- SOME INDIVIDUALS WHO WERE CRITICAL OF SOME OF THE RESEARCH UNDERWAY TODAY WHICH THEY CONSIDER FRIVOLOUS, PARTICULARLY THE SOCIAL SCIENCES BUT NOT ENTIRELY. I USE THAT EXAMPLE OF -- HOW DO YOU KNOW WHAT'S FRIVOLOUS? BASIC RESEARCH? I THOUGHT TESTIFY DANGEROUS FOR A POLITICIAN TO FIND WHAT THE WORD FRIVOLOUS MEANT. HIS COMMENT TO ME, I HOPE IT WAS IN JEST BUT I DON'T KNOW, WHAT -- YOU SAID WHAT YOU'RE SAYING MR. AUGUSTINE IS WE SHOULD SPEND MORE MONEY SPENDING -- STUDYING SEALS AN LESS ON NIH. WHICH I INDICATED WAS NOT MY POINT BUT THAT'S THE PROBLEM YOU RUN INTO. IT IS REALLY DIFFICULT TO CONVEY INPUT AND OUTPUT. YOU MADE THE POINT WE MEASURE NIH BEEN IN TERMS OF UNITED STATESERÖ BORDERS BUT IT GOES AROUND THE WORLD, NOT JUST CURRENT GENERATION, FUTURE GENERATIONS WILL BENEFIT FROM SOL'S WORK AND SO ON. SO I THINK THE DATA THAT KIM OUT IS THIS IS A COMPLICATED IMPORTANT TOPIC, BUT IF WE DOPE7Ť DO A HALFWAY DESEN JOB WE'LL MISS A LOT OF OPPORTUNITIES. WITH THAT IT'S TIME THE TO GIVE THE PUBLIC A CHANCE TO COMMENT. WE HAVE A SIGN UP LIST OUTSIDE. MARK YOU HAVE A COMMENT? (OFF MIC) >> PANEL IS THINNING. MARK, YOUR COMMENTS HAVE BEEN GRAY GREAT. THANK YOU SO MUCH FOR BEING HERE. [APPLAUSE] >> SO THIS IS THE TIME WE TURN TO THE PUBLIC. WE HAVE A SIGN UP LIST IN THE HALL, IS ANYBODY SIGNED UP TO COMMENT? THE ANSWER IS NO SO I DON'T NEED TO GO THROUGH THE QUALIFICATIONS. GIVEN THAT WE'RE TEN MINUTES AHEAD. AND DO YOU WANT TO SAY ANYTHING ABOUT THE PENDING COMPLICATIONS COMING UP OR IS THAT BEING SOLVED? >> BEING SOLVED. OKAY. WHY DON'T WE THEN PLAN TO MEET AT 10 AFTER. WILL THAT WORK FOR EVERYBODY? 15 AFTER. BOY. AMY IS GOING SOFT ON US HERE. >> IF EVERYBODY WILL PLEASE TAKE THEIR SEATS, WE'RE ONE MINUTE AHEAD OF SCHEDULE. AND WE CAN CONTINUE WITH OUR AGENDA. WE'LL INVITED SOMEONE WHO TRULY NEEDS NO INTRODUCTION AT THIS PLACE. AND I COUP ELIAS AS A DEAR FRIEND AND MUCH ADMIRED FRIEND. HE DID SO MUCH FOR THIS INSTITUTION, FOR A LOT OF OTHER INSTITUTIONS, GAVE ME GOOD ADVICE ALOK THE WAY. -- ALONG THE WAY. IT'S JUST A PRIVILEGE TO HAVE YOU BACK. AND HAVE YOU TALK TO US A LITTLE BIT. WE'RE FOCUSING ON HERE AS YOU KNOW, HOW DOES ONE VALUE INVESTMENTS IN BIOMEDICAL RESEARCH. THAT'S A TOPIC I KNOW YOU HAVE DEALT WITH BEFORE CONGRESS AND OTHER BODIES A LOT. WE HAVE BEEN HAVING PANEL DISCUSSIONS TODAY, GREAT SPEAKERS AND I COME AWAY CONCLUDING IT'S MORE DIFFICULT THAN WHEN I BEGAN THIS MORNING. BUT ANYTHING YOU WANT TO TALK ABOUT THAT YOU THINK MIGHT BE HELPFUL, WE'RE HONORED TO HAVE YOU HERE. SO THE FLOOR IS YOURS. FIRST I'M PLEASED TO BE IN THIS ROOM WHERE I SEE MY COLLEAGUES, WE HAVE MANY MEETINGS. I'M ALSO PLEASEDED TO SEE THE SMRB MEETING AND DISCUSSING IMPORTANT ISSUES. YOU RIB THIS IS FOR THE RE-AUTHORIZATION OF THE REFORM ACT IN 2006 AND QUITE A TOPIC, WHETHER OR NOT A FEDERAL AGENCY NEEDS A REAL (INAUDIBLE) TO TRULY STRATEGICALLY MANAGE THE AGENCY AND WHETHER OR NOT THAT COULD BE AN EFFECTIVE TOOL RATHER THAN THE TYPICAL ADVISORY COUNCILS WHO MAY NOT HAVE A VOICE. INFLUENCE WITH BOTH INSTITUTION AND CONGRESS. I'M GLAD TO SEE YOU'RE ADDRESSING THESE TOPICS WHICH IS A VERY DIFFICULT ONE. NOW I KNOW YOU HAVE HAD A PRETTY AUGUST GROUP OF PEOPLE COMING ABOUT HOW TO VALUE, HOW TO MEASURE, HOW TO -- AND I THOUGHT PERHAPS WHAT I WOULD DO, NORM, IF IT'S OKAY WITH WITH YOU, IS NOT REALLY GO INTO THE ACADEMIC DISCUSSION OF WHAT IT IS THAT VALUE OF RESEARCH HOW SHOULD IT BE MEASURED. RECALL WE HAD THIS QUESTION CONSTANTLY FROM CONGRESS. WHAT ARE WE GETTINGTOR THE $30 BILLION WE'RE INVEST SOMETHING I USED TO BE BOMBARDED BY BOTH FOLK WHOSE SAID WHERE ARE THE RESULTS, FOLK WHOSE SAID WHAT ARE YOU DOING X VERSUS Y, PRIORITY SETTINGS. I'M NOT SEEING THE RETURN ON INVESTMENT, AND HOW DO YOU PROVE THERE IS A RETURN INVESTMENT. SO WE DID SOME STUDIES AT THE TIME, THOSE OF YOU WHO REMEMBER AND CAME UP WITH SOME REAL ONE LINERS TO THE VALUE OF MEDICAL RESEARCH, TRAY DOLLARS A YEAR SPENT ON CARDIOVASCULAR RESEARCH THAT LED TO 2 1/2 TRILLION DOLLARS ECONOMIC VALUE EVERY YEAR. THE PROBLEM WITH THE STATEMENTS LIKE THIS IS THAT YOU CAN EASILY MAKE THEM, YOU CANNOT EASILY PROVE THEM. BECAUSE THE NOTION OF VALUE IS A NOTION THAT CHANGES WITH TIME. AND P NOTION OF WHAT IS MEDICAL VALUE FOR ONE PERSON IS NOT NECESSARILYw3 THE SAME FOR SOMEONE ELSE. WHAT IS THE TOTAL CAPTURE OF HOW YOU CAPTURE THAT VALUE. LET ME SHARE WITH YOU WHAT MY PERSPECTIVE I WAS ASKED TO GIVE YOU SENSE FROM INDUSTRY. HOW YOU CAN FRAME IN A WAY ESTABLISHED A VALUE THAT IS NOT THEORETICAL BUT VERY PRAGMATIC. I THINK VALUE DEPENDS ON THE MOMENT IN TIME SOCIETIES AREN'T IN. IF YOU'RE IN A PERIOD OF POOR LIFE EXPECTANCY, BAD HEALTH RESULTS VALUE IS GOING TO BE VERY DIFFERENT THAN IF YOU ARE IN THE PERIOD OF LONGER LIFE EXPECTANCY. AND BUDGET HIGHER THAN THE ECONOMIC GROWTH. THE PERCEPTION OF VALUE IS GOING TO BE DIFFERENT AT THOSE TWO TIME FRAMES. NO AMOUNT OF SUPPORT THAT CAN BE SPARED WHEN YOU HAVE EPIDEMICS AND REAL LOSS OF BOTH MEDICAL VALUE AND ECONOMIC VALUE AS IF YOU HAVE A PANDEMIC, SOMETHING LIKE THIS, OPPOSED TO STAGE OF SOCIETY WHERE CHRONIC DISEASE ARE DOMINATING THE CHALLENGES AND THE IMPACT ON THE BUDGETS AND ABILITY OF A SOCIETY TO KEEP UP WITH ALL OF ITS OBLIGATIONS IS CHALLENGED. SO WHEN YOU LOOK AT FEDERAL INVESTMENTS AND YOU SAY ALL RIGHT, WELL, THE BIG QUESTION THAT I THINK POLICY MAKERS ARE GOING TO ASK THEMSELVES IS THIS DOLLAR BETTER INVESTED HERE THAN HERE. SO THAT'S THE NOTION OF RELATIVE VALUE OF FUNDING RESEARCH. THE SECOND IS THE MATCH WITH THE SOCIETAL CAN HE BATIONS. I THINK IT'S VERY IMPORTANT NOT TO HAVE INTRINSICALLY DEFINED METRICS THAT ARE GREAT ACADEMICALLY, I'M NOT AGAINST THEM, I THINK IT IS IMPORTANT HOW MANY PATENTS, HOW MANY COMPANIES YOU CREATE, HOW MANY JOBS YOU CREATE. ALL THESE ARE OF COURSE THEY'RE SIN QUINONE HOW TO JUSTIFY INVESTING. A VERSUS B TODAY BUT WHAT IS THE DEFINITION OF THE BETTER INVESTMENT FOR A SOCIETY, OUR SOCIETY? IS IT BETTER TO SPEND MORE AT THE NIH OR LESS OR NSF MORE? I KNOW NORM, YOU WROTE THE GATHERING STORM REPORT AND YOU RIGHTLY POINT OUT THE FACT THAT YOU CAN'T REALLY ADVANCE KNOWLEDGE WITHOUT A BALANCE BETWEEN COMPLEX DISLYNNS THAT ARE REQUIRED TO DO THAT. HAVING SAID THAT LET ME IF I IF I MAY GO BACK TO WHAT I BELIEVE VALUE SHOULD -- FROM THE STANDPOINT OF HAVING BEEN IN ACADEMIA, HAVING BEEN IN GOVERNMENT NOW IN INDUSTRY, LOOKING A T IT FROM THE THREE DIFFERENT ANGLES. WHERE IS IT THAT THE DISCUSSION REALLY SHOULD GO. FROM MY POINT OF VIEW THE PRODUCT OF FEDERAL FUNDING IN MY ESTIMATION, PRODUCT OF NIH SHOULD BE KNOWLEDGE, NOT PRODUCTS. THAT'S THE BIG DIFFERENCE FRAMING CLEARLY IMMEDIATE MISSION OF THE NIH AND HOW IT DECLINES ITSELF IN A DIFFERENT AREAS IT HAS IMPACT ON WHAT SOCIETY IS EXPECTING. AND THAT IS REDUCTION OF THE BURDEN OF DISEASE AND BURDEN OF HEALTHCARE COSTS. SO YOU GO AROUND THE WORLD TODAY AND WHAT I'M SEEING FROM THE STANDPOINT OF INDUSTRY IS HOW DO YOU MAINTAIN INNOVATION WHICH COMES FROM THAT BASIC KNOWLEDGE THAT IS GENERATED BY THE FEDERAL INVESTMENT AND TRANSFORM IT INTO A SOCIETAL WHEN. I THINK IT'S VERY IMPORTANT TO DO IT THAT WAY. SO HOW CAN I SUGGEST WE DO THIS? I HAVE A SIMPLE MATRIX IN MY MIND. IF YOU TRULY LOOK AT THE -- WHAT PREVENT PES YOU FROM GENERATING VALUE, YOU HAVE SEVERAL STEPS WHICH ARE PRAGMATIC STEPS THAT YOU CAN MEASURE. SO THE FIRST ONE IS WHAT I CALL, AGAIN, REMEMBER, I ALWAYS LIKE TO TALK IN TERMS OF WHAT'S THE DIFFICULTY TRANSFORMING IDEAS TO ESTABLISHED KNOWLEDGE ESTABLISHED PATHWAYS, MAKING A GOOD PUBLIC, BUT THOSE STEPS ARE TRULY DIFFERENTIAL. FOR EXAMPLE, IF YOU LOOK AT TRANSLATION, SEVERAL STAGES TO IT. ONE IS T-1 TRANSLATION WHERE YOU TRULY ARE AT THE EDGE OF UNDERSTANDING A PROCESS OR DISEASE BIOLOGY OR SOMETHING THAT YOU CAN THEN USE FOR INDUSTRY TO USE AS TOOLS. I THINK THIS IS WHERE I THINK DISCUSSION CAN BE FOCUSED AND THAT DISCUSSION ON T-1 TRANSLATION IS GOING TO BE COMPLETELY DIFFERENT FROM T-4 TRANSLATION, WHICH IS ONCE YOU HAVE DEVELOPED KNOWLEDGE, ONCE YOU HAVE DEVELOPED A PRODUCT, ONCE YOU HAVE DEVELOPED A SYSTEM OF DELIVERY, WHAT IMPACT DOES IT HAVE ON PUBLIC HEALTH? TO ME THAT'S T-4 QUESTION NOT T-1 QUESTION. I SEE THAT THIS IS GETTING MIXED UP IN THE PUBLIC DEBATE. PEOPLE SAY WHY IS IT YOU SPEND MONEY ON HEALTH OUTCOMES OR ECONOMICS AS OPPOSED TO SPENDING IT ON UNDERSTANDING CANCER FOR THAT MATTER? I HAD THE SAME ISSUES ADS YOU KNOW I HAD TO WRITE A LETTER TO CONGRESS WHEN I WAS NIH DIRECTOR BECAUSE THEY WERE QUESTIONING THE ABILITY OF NIH TO STUDY BEHAVIORAL PROBLEMS. BUT BEHAVIORAL PROBLEMS ARE THE SOURCE OF 50% OF THE DISEASE BURDEN, SO I'M COURAGE THE COMMITTEE TO LOOK AT THE MISSIONS OF THE NIH ACCORDING TO WHAT IS THE CONVERSION OF VALUE BETWEEN KNOWLEDGE GENERATION AND TRUE SOCIETAL BENEFIT AS A FUNCTION OF THE TRUE CHALLENGE THAT WE HAVE AT THE T-1 TRANSLATION STAGE, THAT GENERATING OF VALIDATED VALUE DADEABLE IDEAS THAT CAN THEN BE USED TO TEST WHETHER OR NOT IT WILL INFLUENCE DISEASE. THAT'S THE VERY FIRST STEP PEOPLE CALL BASIC SCIENCE, EARLYD TRANSLATION, I DON'T KNOW BUT THAT'S A SET OF MEASURES THAT IS COMPLETELY DIFFERENT IN MY MIND THAN THE OTHERS. THEN YOU HAVE THE T-2 TRANSLATION. WHAT CAN NIH DO TO MAKE SURE AT THE END WHEN THAT TRANSLATION GOES FROM THE EARLY STEPS OF UNDERSTANDING IT IN MODELS GOES TO HUMANS. THAT STEP IS ANOTHER ACTIVATION ENERGY STEP IN MY VIEW IDENTIFIED SEPARATELY FROM OTHERS. IT RELATES TO CLINICAL TRIALS AND THE VALUE OF WHETHER NIH DOES MAKING SURE THAT WE TEST HYPOTHESES, MAKE SURE THAT WE TRAIN PEOPLE TO UNDERSTAND WHAT THE VALUE OF THAT T-2 TRANSLATION SHOULD BE. A GOD EXAMPLE OF THAT IS WHAT WE HAVE SEEN FOR EXAMPLE, CARDIOVASCULAR COMORBIDITY AND MORTALITY. IT WASN'T ENOUGH TO DIVULGE STATINS ON THE DISCOVERY OF MICHAEL BROWN ON CHOLESTEROL METABOLISM, THE FACT IS THE IMPACT WAS NOT REALLY GENERATED FROM THAT DISCOVERY, IT WAS GENERATED FROM THE FRAMINGHAM STUDY PICKED UP THE CHOLESTEROL SIGNAL. THE FRAMINGHAM STUDY IDENTIFIED THE FACT THAT THERE WAS A CORRELATION, RISK FACTOR CORRELATION SO IT'S A CYCLE BETWEEN THE TWO. IF YOU THINK ABOUT WHAT MADE THE REAL IMPACT IT WAS THAT SECOND TRANSLATION WHEN IN FACT DESIGN OF THE CLINICAL TRIALS WHICH WERE REALLY DRIVEN IN MANY WAYS BY NIH, CHANGE THE PRACTICE OF MEDICINE. SO HOW MANY TIMES HAS THE PRACTICE OF MEDICINE CHANGED BECAUSE OF FEDERALLY FUNDED RESEARCH TO ME IS ANOTHER SET OF QUESTIONS. YOU CAN SEE IN HIV AIDS, WE CHANGED THE PRACTICE IN MEDICINE , THERE ARE MANY OTHER EXAMPLES BY THE WAY, NOT JUST -- I MEAN STROKE, REDUCTION 70%. SO YOU CAN SEE HOW YOU CAN START TO BUILD A FRAMEWORK AROUND THE CONCEPT OF VALUE WHICH IS DIFFERENTIATED FROM THESE GENERAL METRICS WHICH REALLY TO ME DON'T REALLY CONNECT TO THE SOCIETAL BENEFIT AND IMFACT THAT YOU NEED TO JUSTIFY. THEN T-3 IS REDUCTION TO PRACTICE HOW IS -- INDEED HAS NIH CHANGED THAT FOR THE POPULATION WE KNOW 30% OF PATIENTS WITH DIABETES ARE COMPLIANT OR REACH GOALS. THIS YOUR THING, GIVE. 70% TO NOT. THIS IS APPLYING WHAT WE KNOW, GENERIC MEDICATIONS, CHEAP INSULINS BEEN AROUND 75 YEARS AND IT'S NOT COMPLICATED. RIGHT? WHAT HAS NIH DONE TO CHANGE THAT? WHAT IS THE NIH RESEARCH REALLY CHANGING IN THE T-3 SPHERE WHERE ONLY 50% OF PATIENTS HAVE -- ARE COMPLIANT WITH WITH -- SO COMPLIANT WITH HIGH BLOOD PRESSURE, MEDICATION. THAT TO ME IS CRITICAL. AT THE END OF THE DAY YOU NEED TO TIE IN THE RESEARCH YOU DO IN THAT SPHERE AND ITS IMPACT. SO I'M NOT SUGGESTING DECONSTRUCT THE CREATION -- VALUE CREATION PROCESS OF NIH. I'M SAYING SPECIFY IT BECAUSE DISCUSSION OF T-1 TRANSLATION AND T-1 TRANSFORMATION, IS COMPLETELY DIFFERENT THAN WHAT THE REGULATORS, NOT REGULATORS THE POLICY MAKERS CO-MINUTE IT WILL VALUES AN AT THE END THE IMPACT MAYBE MORE ECONOMIC VALUE MORE COMPANIES WORLDWIDE MORE INTELLECTUAL PROPERTIES. THEY'RE IN THE PRIMARY MEASURES VALUE EXPECTED BY SOCIETY. SO YOU TALK ABOUT T-4 TRANSLATION, IT'S VERY CLEAR, YOU LOOK AT THE MAP OF THE UNITED STATES, AND YOU LOOK AT THE FACT THAT WE HAVE ESSENTIALLY THE SAME KNOWLEDGE, RIGHT? YOU LOOK AT THE DIFFERENCE DISPARITIES. IT'S IN THE JUST SOCIAL ECONOMIC, THERE'S SYSTEMS ISSUES, IS WHAT I CALL IMPLEMENTATION RESEARCH WHEN YOU REALLY GET TO THE POINT OF CHANGING HOW THE HEALTHCARE SYSTEM IS DESIGNED. I DON'T SEE WHY A FEDERALLY FUNDED INSTITUTION CANNOT DO THAT. I GIVE YOU A GOOD EXAMPLE. NIH WAS CRITICIZED HIGHLY FOR DOING THE WOMAN'S INITIATIVE STUDY. AT THE TIME WHEN DOGMA WAS THAT HORMONAL REPLACEMENT WAS THE ABSOLUTE BEST THING YOU CAN DO. HOW COULD YOU DENY POPULATION FROM THAT? UNTIL THE HIPPOSTOOD UP AND SAIDUP WHAT, WE DON'T HAVE THE SCIENTIFIC EVIDENCE TO SUBJECT POPULATION TO THIS AND THEN YOU KNOW THE RESULTS. THE DOGMA WAS OVERTURNED, WE FOUND OUT IN TACT IT INCREASED CANCER RISK, INCREASED CARDIO VASCULAR RISK OPPOSED TO WHAT WE THOUGHT. IT DIDN'T HAVE BENEFITS EXCEPT OSTEOPOROSIS. THAT IS VALUE. I THINK IT'S IMPORTANT TO QUANTIFY THAT VALUE. SO THE QUALIFICATION IS YOU'RE SEEING RIGHT NOW IN CANCER STATISTICS YOU SEE REDUCTION IN THE CANCER RATES, IN BREAST CANCER, AND YOU LOOK AT THE HOT SPOTS OF BREAST CANCER IN THE COUNTRY, MAUREEN COUNTY AND LONG ISLAND, WHERE WE DID SEVERAL STUDIES BECAUSE SOCIETY, THE LOCAL BODY POLITICS WAS SUGGESTING THAT THERE WAS A ENVIRONMENTAL FACTOR IN THOSE COUNTIES THAT EXPLAINEDDED WHY CANCER WAS AT THE HIGH RATE. TRUTH WAS, BECAUSE THE WOMEN IN THOSE COUNTIES HAD THE MEANS, THESE ARE SOME OF THE WEALTHIEST COUNTIES IN THE US. THEY WERE A MUCH HIGHER PROPORTION OF WOMEN ON FULL HORMONAL REPLACEMENT. WE THEN LED TO HIGHER INCIDENCE OF CANCER. SO THIS RESEARCH, IMPACT IS EXTREMELY VALUABLE AND NEEDS TO BE QUANTIFIED. I WOULD SAY QUALIFY AND WE IMMEDIATE TO GET TO THE POINT WHERE WE QUALIFY THE VALUE OF WHAT NIH DOES IN LIVES QUALITY OF LIVES, LIFE YEARS, PRESERVE, NOT JUST ABSTRACT NUMBERS, NEEDS TO BE PERSONALIZED AND FRAMED IN A DISCUSSION WHERE YOU NEED TO JUSTIFY THE IMPACT OF FEDERALLY FUNDED RESEARCH, NOT INTERMEDIATE MEASURES BUT IN OUTCOMES THAT RELATE DIRECTLY TO SOCIETAL EXPECTATIONS. I DON'T KNOW IF THAT HELPS. TO ME IT'S THE ONE THING THAT FEDERAL AGENCIES NEED TO DO TODAY. AT THE END OF THE DAY IT WILL BE THE RELATIVE VALUE OF THE FEW DOLLARS I HAVE TO INVEST IN A, B, C. DON'T YOU ASSUME THAT THERE'S A GIVEN IT WILL BE HEALTHCARE RESEARCH. BUOY MEDICAL RESEARCH. ON THE INDUSTRY SIDE LET ME BE VERY CLEAR. ON THE CHALLENGES RIGHT NOW ARE THAT WE -- I DON'T SEE THE BODY OF KNOWLEDGE AND BECAUSE WHEN WE TALK NIH WE SHOULD REMEMBER NIH IS AN ECOSYSTEM, IT'S NOT AN INSTITUTION. WE SHOULD SEPARATE THE VALUE OF NIH FROM ITS INTRAMURAL PROGRAM, WHAT IT DOES UNDER ITS OWN CONTROL, FROM THE FACT THAT NIH IS THE TOTALITY OF THE 3,000 FUNDED INSTITUTIONS IN THE ACADEMIC SYSTEM THAT WE HAVE. WE NEED TO SEGREGATE THE TWO IN MY VIEW. IT'S A MINOR QUESTION BUT AN IMPORTANT ONE BECAUSE PEEP AT THE END JUSTIFY WHERE EVERY DOLLAR GOES BUT WHEN YOU LOOK AT -- WHEN WE SAY NIH, IT'S NOT NIH, NIH I'M ALWAYS SURPRISED WHEN PEOPLE SAY WHAT IS THE POSITION OF NIH, THE POSITION OF NIH REFLECTS A CONSENSUS OF INPUTS FROM DIFFERENT CONSTITUENCIES, DIFFERENT PROFESSIONS AND SO ON PRESENT IN ADVISORY COUNCILS PRESENT ON DIFFERENCE PEER REVIEW PANELS AND SO ON. AND THAT WORLD THE PRODUCTIVITY OF THAT WORLD, THE IMPACT IF YOU WILL, THE VALUE OF THAT WORLD IS TO BE CAP CHUED BECAUSE THAT'S WHAT NIH IS -- CAPTURED BECAUSE THAT'S WHAT NIH IS. WHEN YOU LOOK FROM THE STANDPOINT OF WHAT I HAVE LEARNED THEY'RE A HUGE DISCONNECT. GAIL, I'M SURE YOU KNOW. RIGHT NOW THERE IS A FUNDAMENTAL DISCONNECT BETWEEN6P THE NEED FOR KNOWLEDGE AND THE FACT WE HAVE BETTER AND BETTER SCIENCE, WE HAVE GREATER UNDERSTANDING OF BIOLOGICAL SYSTEMS, BETTER THAN EVER BEFORE. YOU KNOW HOW NOW WE CAN CONDUCT EXPERIMENTS IN THE LAB THAT NOW COULD DREAM OF 20 YEARS AGO. WE HAVE TRANSGENICS, WE HAVE MODELS, WE HAVE CELL LINES, NONE OF IT IS PREDICTIVE OF WHAT HAPPENS IN THE HUMAN POPULATION. YET THE SYSTEM CONTINUES VERY STRAIGHT, TO NOT INVEST IN WHAT IS THE KEY TO OUR FUTURE SUCCESS TO INVEST IN COHORTS OF PATIENTS SUFFERING FROM CHRONIC DISEASES THAT ARE LONG TERM, OTHERWISE WE'LL NEVER UNDERSTAND REALLY WHAT HAPPENS. WE'RE PRETTY GOOD UNDERSTANDING KNOWLEDGE TO BIOLOGY, BUT NOT THAT GOOD AS THE NIH ECOSYSTEM, NOT US NIH BUT FUNDING THE BEST RESEARCH THERE IS. ARE WE THAT GOOD IN TERMS OF NIH SYSTEM IN GENERATEING WHAT I CALL A DEEPER UNDERSTANDING OF DISEASE BIOLOGY IN HUMANS. MY SENSE IS THAT THAT IS VALUE THAT IS BEING LOST. I DON'T KNOW HOW TO SAY THIS LITTLE INDICTMENT AT ALL. IT'S BASICALLY AN OBSERVATION. ARE WE USING THE DOLLAR WE HAVE TO THE MAXIMUM EXTENT POSSIBLE? FIRST LEVEL QUESTION SHOULD WE USE THE DOLLAR AT NIH, SHOULD WE USE WITHIN NIH USE THE DOLLAR AS WE'RE USING IT. I THINK IT'S VERY IMPORTANT TO KEEP THAT IN MIND. TO CALCULATE THE VALUE THAT THE ECOSYSTEM IF YOU WILL IS INVESTING RELATIVE TO SOCIAL EXPECTATION OF IMPACT ON WHAT IS CONSUMING RIGHT NOW 80% OF HEALTHCARE COSTS AND THAT IS CHRONIC DISEASES. SO -- I THINK IT'S HAPPENING, THE IMPACT OF THE NIH FUNNED RESEARCH WITH ALZHEIMER'S DISEASE INITIATIVE IS REMARKABLE. FRANKLY, IT'S CHANGED THE THE WAY WE IDENTIFY THESE PATIENTS AND HOW WE'RE GOING TO DO THE RESEARCH. HOW DO YOU CAPTURE THAT IS NOT VERY CLEAR TO ME. SO THOSE ARE THE ROUGH POINTS, BE HAPPY TO ANSWER QUESTIONS. YOU NEED TO IN ADDITION TO WHAT YOU HEARD TODAY WHICH IS VERY IMPORTANT, THE ECONOMIC VALUE AND ALL THE FACTORS AND THE SURROGATE MEASURES I WOULD SAY THAT THE FEDERAL INVESTMENT NEEDS TO BE TIED TO SOCIAL SOCIETAL PRIORITIES OF TODAY BUT IT NEEDS TO BE FRAMED IN WAY THAT RELATES TO THE PRODUCT NUMBER ONE OF FED ALLEY FUNDED RESEARCH IS KNOWLEDGE. KNOWLEDGE USED BY THE ECOSYSTEM TO CHANGE PRACTICE OF MEDICINE OR CREATE INNOVATION THAT WILL CHANGE DISEASE BURDEN. IF YOU LOOSE THAT YOU START TO BECOME ACADEMICALLY ISOLATED AND THEORETICAL INTERACTING WITH THE PUBLIC FORCES OUT THERE. >> ELIAS, THANK YOU. THAT WAS TERRIFIC. I'M SURE WE HAVE QUESTIONS. I HAVE SOME BUT I'LL BEGIN WITH MY COLLEAGUES P GAIL WOULD YOU LIKE TO START OUT? >> THANK YOU ELIAS, SO MUCH FOR TAKING THE TIME WITH WITH US. WHEN I HEAR WHAT YOU JUST SAID TO US THE FIRST THING THAT I'M THINKING PARTICULARLY AS YOU RELATE TO TIE TO BROADER SOCIETAL IMPACTS, CURRENT NEEDS, PRIORITIES, I DO THINK IN A PUBLIC HEALTH I THINK OF THE INTERACTION OF NIH WITH OTHER AGENCIES LIKE THE CENTERS FOR DISEASE CONTROL AND ALSO FDA QUITE HONESTLY, WONDER COULD YOU JUST COMMENT ON WHETHER OR NOT YOU FEEL WE'RE DOING ENOUGH AT THE NIH WITH CDC, FDA IN ORDER TO MAXIMIZE WHAT WE'RE DOING, I STILL GET THIS FEELING THAT WE ARE NOT DOING AS MUCH AS WE COULD BE DOING OR SHOULD BE DOING. BASED ON EVERYTHING YOU HAVE SAID TO ME ANYWAY, THAT SEEMS LIKE MAYBE A GOOD PLACE TO START. TO LOOK AT WHAT I'M DOING. AND WE HAVE JIM CURRAN WHO CERTAINLY HAS CDC EXP PER TEASE AN PERSPECTIVE AS WELL AS OTHERS. >> I THINK IN MY MIND, AGAIN, I FRAME THE ISSUE OF HOW YOU REALLY MiK1JIZE THE LEVERAGING SYNERGY, IF YOU WILL, AGAIN ALONG THAT AXIS AT T-1. WHAT IS IT -- WHAT IS THE KNOWLEDGE EMERGING THAT'S ACTIONABLE, REALLY TO BE ENCOURAGED AND HAS POTENTIAL USING THE TOOLS THAT HAVE BEEN DEVELOPED OVER THE PAST 10, 15 YEARS NEED TO BE ACTED UPON TO MAKE THINGS HAPPEN. SO IN THE T-1 T-2 SPACE, FDA IS CRITICAL. I DON'T SEE CDC IN THAT SPACE, SOME IDENTIFYING ISSUES A LOT FOR EXAMPLE MANY THE INFECTIOUS WORLD, DISEASE THE WORLD. THAT TO ME IS THE WAY TO LOOK AT IT. WHEN I WAS DIRECTOR WE HAD A LEEANNSON -- LIAISON SYSTEM WITH THE AGENCIES, THERE IS A NEED TO DO MORE. IT'S BEING DONE, WE WORKED ACTUALLY AMY WAS PERSON WHO WAS DOING THIS ON ADVERSE EVENT REPORTING AND DOING -- HAVING A COMMON LIFE FORM TO DO THAT. FRANKLY WHERE NIH COULD PLAY A BIGGER ROLE IS IN WHAT I CALL THE LEADING EDGE OF REGULATORY INNOVATION, THE NEW SCIENCE, WHEN YOU LOOK AT HOW FDA DOES IT, FDA IS SORT OF HAM STRUNG BUDGET WISE SCIENCE WISE AND ABILITY TO MODIFY REGULATORY SYSTEM REQUIRES REAL SCIENCE. EVERYBODY WOULD LIKE MORE SCIENCE IN THE REGULATION. AND I THINK PEGGY IS -- IN MY VIEW THE FIRST COMMISSIONER WHO HAS FORMULATED THAT, THEN NIH NEEDS TO BE THERE. IN TERMS OF CDC, I SEE CDC MORE THE T-3, T-4 SPACE. WHERE WE NEED TO REALLY MAKE SURE THAT THE PUBLIC HEALTH IMPACT IS THERE. WHY SO MUCH DISPARITY. I ALWAYS USE THE -- I WAS ON THEIR DATABASE RECENTLY ABOUT DIABETES. AND SEEING THE DIFFERENCIAL RATES OF GROWTH IN THE COUNTRY. AND IT'S NOT JUST OBESITY, THERE ARE OTHER THINGS BUT IT'S VERY FASCINATING RESEARCH. HOW TO WE TAKE ADVANTAGE OF THAT IN THE COMPLIANCE TO THE POPULATION, BEST METHODS AND HOW IT IMPACTS PUBLIC HEALTH. THAT'S SOMETHING CDC ISSUE CDC HAS CONSTRAINTS, IT HAS STATES, A DIFFERENT MISSION, IT'S NOT GENERATING IT'S BLOCK GRANTS, SO ON. SO COMPLETELY. I SEE THE SAME THING WITH NSF. I THINK PHYSICAL SCIENCES ARE REALLY UNDERSERVED IN OUR COUNTRY. ESPECIALLY ENGINEERING SCIENCES. I THINK WE NEED TO BE REMEMBER THAT IT'S GREAT TO HAVE AN IDEA. IT'S GREAT TO HAVE AN INDUSTRY BUT AT THE END OF THE DAY YOU ALSO NEED YOUR OWN EXPERTISE AND KNOWLEDGE TRANSFORMING THAT KNOWLEDGE. THE SKILL SETS THAT YOU NEED IT'S NOT ALWAYS THERE. THERE ARE GAPS IN MY VIEW, A GOOD EXAMPLE IS THAT FRANCIS UNDERTOOK WHICH IS GREAT IS THIS COLLABORATION BETWEEN DARPA, NIH AND FDA ON THESE THREE DIMENSIONAL TISSUES FOR TOXICOLOGY. THAT'S A GREAT EXAMPLE OF SYNERGY. THAT'S HAPPENING. >> THANK YOU TOO FOR THAT EXCELLENT OVERVIEW I ENJOYED. I HAVE TWO QUESTIONS TO ASK I'M A LITTLE UNCERTAIN ABOUT EXACTLY WHERE YOU COME DOWN THERE IS A FAIR AMOUNT OF RELUCTANCE OR CRITICISM OF THE HIPPOTO BE FULLY INVOLVED IN BEHAVIORAL RESEARCH OR BORDERLINE BEHAVIORAL SOCIAL RESEARCH. MANY OF THESE APPLICATIONS THAT YOU TALK ABOUT ARE BASIC KNOWLEDGE TO APPLY THEN TO THE POPULATION REQUIRED THAT SERIOUS RESEARCH IN SO CALLED SOFTER SCIENCES THAN BIOMEDICAL. BASIC BIOMEDICAL RESEARCH. ARE YOU IN FAVOR OF NIH PUTTING MORE RESOURCES INTHE THAT KIND OF WORK? BECAUSE IT COULD HAVE A BIG IMPACT ON THE POPULATION IF THEY RESPONDED TO SOME OF THE KNOWLEDGE THAT'S ALREADY THERE. BUT IS NOT BROADLY APPLIED. >> SO AGAIN, I WOULD BE VERY FRAMING OF THAT CONVERSATION. I PERMLY BELIEVE YOU NEED TO BE CAREFUL TO WHAT IS IT YOU'RE APPLYING IT TO SO YOU CAN SHOW THE REALITY OF THE SOCIETAL NEEDS. I THINK IT'S TRUE IN THE T-3, T-4 SPACE. IT'S EVEN TRUE IN THE FIRST TRIALS BECAUSE BEHAVIOR DOESN'T -- DOES IMPACT YOUR UNDERSTANDING OF THE DISEASE PROCESS. SO IT'S CLEAR YOU HAVE TO ATTACH IT. THE PROBLEM THAT YOU HAVE IN BEHAVIORAL SCIENCE IS PEOPLE -- A LOT OF PEOPLE BY NATURE THE MEASUREMENTS IN BEHAVIORAL SCIENCE ARE FUZZIER BY NATURE BECAUSE OF THE VARIABILITY, HETEROGENEITY, SO ON, SO IT LENDS ITSELF TO BEING CRITICIZED. BUT I THINK WE SHOULD DEFEND IT AND TO ME I THINK IT'S ABSOLUTELY EXEMPLARY TO SEE SOME RESEARCH THAT HAS MADE A HUGE DIFFERENCE IN -- FROM THAT ANGLE. CURRENTLY THE THING THAT I FIND FASCINATING IS WHAT WE CALL GAME APPROACHES TO BEHAVIORAL ENGAGEMENT OF PATIENTS, SUSTAINING THEM. IN COMPLYING. IN COMPLIANCE. YOU HAVE RESEARCHERS WHO HAVE COME UP WITH THIS IDEA IF YOU ENGAGE A PATIENT BEYOND THE THREE -- VISIT EVERY THREE MONTHS, WHERE DOCTOR SAYS YOU SHALL DO -- IF YOU REALLY ENGAGE THEM WITH MODERN SOCIAL NETWORKING AND SO ON, THAT IN FACT YOU HAVE A MUCH HIGHER COMPLIANCE. THAT RESEARCH NEEDS TO BE VALIDATED. THAT RESEARCH IS BEHAVIORAL RESEARCH BY NATURE. SO THE REAL QUESTION TO ME IS WHAT KIND OF BEHAVIORAL RESEARCH ARE YOU TALKING ABOUT? IF IT'S REALLY FUZZY, NOT MEASURABLE, ENDLESSLY CIRCLING THE SAME CONCEPT SOME WHICH RETROSPECTIVELY YOU HAVE A HIGH PARTAGE OF ISSUES WITH PAPERS RETRACKED IN THAT FIELD, SO ON, I'M NOT THE NIH DIRECTOR, FREEDOM OF SPEECH. BUT IT'S TRUE. SO I THINK YOU NEED TO BE VERY IN MY VIEW STRATEGIC IN THE SENSE THAT BEHAVIORAL SCIENCES ARE NEEDED. THEY NEED A VERY CLEAR PATHWAY BUT ALSO THEY NEED INNOVATION. THERE'S NOT ENOUGH INNOVATION I THINK AND THEY REALLY NEED TO TIE TO THIS WHOLE CONCEPT OF SOCIAL NETWORKING, WHAT THEY CALL INFLUENCERS NETWORK. THERE'S GOOD RESEARCH IN THAT SPACE. I THINK THAT'S WHERE THE FIELD SHOULD BE. IT IS NOT NIH'S ONLY RESPONSE, ONLY ISSUE, AT THE END OF THE DAY HIPPOPRIORITIES ARE DETERMINED BY THE FEEL, NOT BY A FIAT FROM DIRECTOR OF WHATEVER INSTITUTE IS RESPONSIBLE. THAT'S THE TRUTH. THAT IS NOT UNDERSTOOD GENERALLY. WE'RE NOT THE SOVIET SYSTEM HERE. SO WHAT IS GOING TO HAPPEN IS DEFENSE OF THE DYNAMIC DEFENSE OF BEHAVIORAL SCIENCE IS DONE RELATIVE TO VALUE THEY PROVIDE REGARDLESS OF POLITICAL IMPLICATIONS AND SENSITIVITIES ABOUT THAT. >> FOLLOW-UP WITH ONE QUESTION, THIS RELATES TO INDUSTRY WHICH YOU'RE MUCH INVOLVED IN NOW. SEEMS TO ME THERE'S A LOT OF DISAPPOINT OR CONCERN TOO FEW NEW DRUGS IN THE LAST SEVERAL YEARS AND THE BASIC SCIENCE NOT TRANSLATED INTO POSITIVE THINGS FOR PATIENTS WHO OVER-- IN A VARIETY OF WAYS BUT ALMOST SEEMS TO ME THESE THINGS GO IN CYCLES AND I'M MUCH MORE OPTIMISTIC THAN THESE PESSIMISTIC PEOPLE WHO SAY WATCHING THE HUMAN GENOME TEN YEARS LATER AND JUST LOOKING AT THE UPTICK IN NEW DRUGS AND SOME OF THE EXCITEMENT P ABOUT NEW APPROACHES AND PERSONALIZED MEDICINE, SEEMS TO ME THE NEXT TEN YEARS ARE GOING TO BE REALLY EXCITING AND WONDERFUL AND WE TAKE TOO SHORT TERM VIEW HOW THESE INVESTMENTS PLAY OUT. SO ONE THING WE WERE TALKING ABOUT THIS MORNING WAS YOU CAN'T -- ONE SHOULD TAKE A LONGER TERM VIEW OF THESE INVESTMENTS BECAUSE THEY DO PAY OFF TEN YEARS, 15 YEARS, 20 YEARS. AND WHEN THEY DO, THEY MAY PAY OFF BIG TIME. SO LIKE MANY THINGS IN THIS COUNTRY WE FOCUS ON SHORT TERM ANALYSIS. I JUST LIKE YOUR OPINION FROM INDUSTRY WHETHER THERE'S THAT KIND OF FEELING THERE IN TERMS OF NEW DEVELOPMENTS. >> VERY GOOD POINT. WHEN YOU LOOK AT T-1, TIMES ZERO IS MINUS 15 YEARS. DISCOVER FUNDAMENTAL MECHANISM T-0. WITNESS YOU GET T-1, THE GOAL IS 13, 15 YEARS. T-2, GOAL IS 7, 8 YEARS, T-3 GOAL, YOU CAN DEFINITELY SEE AN IMPACT BETWEEN 3 AN 5 YEARS. T-4 YOU NEED TO TITRATE WHAT YOU DO DEPENDING ON HETEROGENEITY I WAS TALKING ABOUT. HOW DO YOU CHANGE THE BEHAVIOR OF SOMEONE LIKE ME, SOMEONE LIKE YOU, SOMEONE -- IS GOING TO BE OR HOW DO YOU MAKE SURE THERE IS THE RISK BENEFIT THAT MAKES SENSE FOR THESE PATIENTS, HOW DO YOU RETUESDAY THE COST, THOSE ARE SYSTEM CHANGES. IT'S ALMOST LIKE OPERATIONS RESEARCH. SO I THINK THAT FROM THAT POINT OF VIEW, I BELIEVE THERE'S A PROBLEM, THERE IS A RESILE PROBLEM, IT'S NOT JUST A CYCLE, I THINK IN MANY AREAS, COMPLETELY AGREE WITH YOU. IMMUNOMODULATION, ABILITY TO MODIFY THE BEHAVIOR OF THE IMMUNE SYSTEM, I THINK YOU'LL SEE A TREMENDOUS AMOUNT OF POSITIVE NEWS THERE IN CANCER. I THINK HETEROGENEITY PROBLEM IS STILL A HUGE ISSUE THAT DESPITE ALL THE ADVANCES THAT WE HAVE, WE'RE GOING TO HIT A LIMIT. THE LIMIT IS THE FOLLOWING. AS THE DISEASE PROCESS BECOMES FRAGMENTED AND COMPLEX WITH MULTIPLE PATHWAYS YOU REALIZE YOU HAVE TO STACK THINGS, YOU NEED TO ADDRESS THIS PATHWAY, GROWTH PATHWAY, TUMOR SUPPRESSOR PATHWAY, WHATEVER, OR IMMUNE PATHWAY WHICH IS THE BIG NEWS NOW, AND THAT MEANS THAT EVERY INNOVATION IS GOING TO BE STACKED ON EVERY OTHER INNOVATION WHICH WILL MAKE IT PROHIBITIVE TO THE PATIENTS. AND THE -- THE THIS, AGAIN T SOCIETAL EXPECTATION, YOU HAVE, YOU HAVE TO LOOK AT IT FROM A PATIENT POINT OF VIEW, YOU LOOSE THAT PROTECTION BETWEEN WHAT YOU DO AND HOW IT AFFECTS THE TAXPAYERS. WHAT THEY WANT -- YOU HAVE LOST -- YOU HAVE LOST THE DEBATE. I REALLY SEE THIS AS A ISSUE OF DEBATE AND ADVOCACY BUT REAL BASE FACT BASED DEBATE THAT HAS TO RELATE TO THE TO THE PATIENT TO RESPOND TO YOUR QUESTION I THINK IS ADVANCES IN CANCER FOR EXAMPLE ARE GOING TO BE AFFORDABLE. THE ANSWER IS DIFFERENT DEPENDING ON THE DIFFERENT AREAS. I AM -- I WOULD SAY IMMUNOLOGY, IMMUNOMODULATION YES, DIABETES, YES, IT'S HAPPENING. I THINK BRUCE CAN -- I WOULD SAY MENTAL HEALTH, I JUST DONE KNOW. I DON'T SEE IT, SO THE ANSWER IS YES FOR SOME, NO FOR OTHERS.X „ >> VERY COMPELLING FRAMEWORK, YOU LED WITH THAT T-1, T-2, T-3, T-4 FRAMEWORK AND VERY IMPORTANT. SO IT STARTS WITH FUNDAMENTAL KNOWLEDGE. AND IN THE SAME SENTENCE IT COMMITS TO APPLICATIONS TO ENHANCE HEALTH, INCREASE LONGEVITY, REDUCE BURDEN OF ILLNESS, BURDEN OF DISSENT. AND I THINK THE T-4 -- T-1 TO 4 COVERS ALL THAT. THE AGENDA AND ALLEGATION RESOURCE OF THE NIH AND THE NIH INFLUENCE COMMUNITY DOESN'T COVER THAT AGENDA SO WELL. IT'S WEAK IN LATER STAGES. AND IT IS NOT VERY POPULAR IN THE INSTITUTIONS AND THE PEER REVIEW PROCESS, AND IN THE CONGRESS FOR WHAT THAT MAY MATTER FOR THIS DISCUSSION. SO TRYING TO FIGURE OUT HOW WE CAN FRAME -- TAKE ADVANTAGE OF THE FRAMING YOU HAVE GIVEN US, WE PROBABLY NEED TO LOOK AT EACH ONE OF THOSE STAGES IN DEATH. THE T-0, T-1 FRAME HAS ITS OWN PROBLEMS. YOU TOOK VERY BOLDLY STATED FRUSTRATION OF THE DRUG DEVELOPERS AND MANY OTHERS OVER THE DIFFICULTY OF GOING FROM ANIMAL MODELS TO THE HUMAN BIOLOGY CH. IT'S BEEN MENTIONED SO MANY TIMES YOU THINK EVERYBODY UNDERSTAND BY NOW, IT'S STILL NOT CHANGED. OUR INVESTMENT STRATEGY IT SEEMS. INDUSTRY THERE HAVE BEEN SERIOUS WRITTEN COMPLAINTS IN THE JURY ROOMS ABOUT THE REPLICABILITY OF RESEARCH, THE MISSION STATEMENT OF THE NIH ALSO ADDRESSES IT PUBLIC ACCOUNTABILITY, INTEGRITY OF THE SCIENCE, RELATED CULTURAL ASPECTS, WE HAVE SOMEWHAT OF A WEAK SPOT THERE. I'M HAPPY, THAT'S A SERIOUS PROBLEM OR JUST AN IRRITANT. WE HAD A CHANCE IN THIS REPORT, I THINK TO HIGHLIGHT IN THE WAY YOU DESCRIBED REALLY, HOW THE COMMUNITY CAN COME TOGETHER AND LOOK FORWARD, NOT JUST TRY TO SUSTAIN, ALWAYS THERE'S A BUDGET STRESS, DOUBLING OF BUDGET YOU AND I RECALL THOSE DISCUSSIONS WAS ALMOST WORSE STRESS BECAUSE PEOPLE'S EXPECTATIONS WERE SO DISALIGNED. MISALIGNED. WITH WHAT WAS HAPPENING IN THE FLOOD OF APPLICATIONS AND INCREASED DEMAND FOR FUNDING. WE HAVE GONE FROM STEADY STATE 10,000 TRAINEES TO 17,000 TRAINEES, PLUS TWICE THAT NUMBER ON GRANTS. WE ARE FEEDING THE PIPELINE AT A REMARKABLE RATE. SO WE'RE NOT BEHAVING AS IF WE'RE DEALING WITH PRECIOUS RESOURCES AND A HIGH PRIORITIZATION OF ALLOCATION PROCESS. WE ARE TAKING FOR GRANTED CONTINUATION OF THE BASE AND ALL THE THINGS THAT ARE SO COMPELLING SEEM TO BE DONE ON TOP. NOT LIKE THEY WORK THAT WAY. FROM ALL THAT COMMENTARY I WOULD HIGHLIGHT FOR YOU THE ANIMAL MODELS, REPLICABILITY OF RESEARCH, AND WE WHETHER THE VERY POSITIVE STATEMENTS JUST MADE AROUND THE TABLE BY YOU ABOUT INVESTING IN HIGH CALL WELL FOCUSED RESEARCH ON BEHAVIORAL FACTORS THAT DRIVE ACHIEVING THE BENEFITS FROM BIOMEDAL RESEARCH -- BIOMEDICAL RESEARCH T-3, T-4, CAN BE HIGH PRIORITY AT NIH. >> RIGHT. IT IS A COST ISSUE, ISSUE YOU CAN GO BACK TO THE 1998 IOM REPORT AND PRIORITY SETTING. THERE WAS A HUGE NEED THERE TO GET PATIENT INPUT, PATIENTS ARE VERY IMPORTANT TO THIS PROCESS. THEY MEASURE BEAR THAN ANYBODY WHAT THE OUTPUT -- WHAT THE SYSTEM IS. I THINK IN TERMS OF IT'S A LITTLE BIT LIKE WHAT ARTHUR WAS SAYING WHERE ARE YOU IN THE CYCLE OF KNOWLEDGE? IF YOU START SAYING WELL, I'M GOING TO SLOW DOWN MY UNDERSTANDING, MY KNOWLEDGE BASE SO I CAN APPLY BETTER, YOU FAIL. I THINK BECAUSE YOU DONE KNOW ENOUGH TO TRANSLATE WHAT YOU DONE KNOW. SO I TEND TO BELIEVE THERE'S A BALANCE YOU NEED TO KEEP. I PERSONALLY BELIEVE 60% OF WHAT WE DO AT NIH REALLY BE DIRECTLY RELATED TO GENERATINGE-y UNKNOWN KNOWLEDGE, THE UNKNOWN AND KNOWNS. I THINK WE NEED TO DO THAT. AND KEEP IT GOING BECAUSE IF YOU DON'T, YOU LOSE A GENERATION OF SCIENTISTS. AND THISA‡ IS MY NUMBER ONE CONCERN IS THAT WHEN YOU LOOK AT THE IMPACT OF # 5% BUDGET CUT WHATEVER IT IS, IT'S FALLS ON THAT 20% DISTRIBUTABLE IN ONE YEAR BECAUSE YOU HAVE FIVE YEAR GRANTS. HALF THE GRANTS RENEW EVERY FIVE YEARS, PEOPLE THINK THAT NIH GRANTS ARE FOREVER AND THAT'S NOT TRUE. IT'S A HUGE TURN OVER, 50% EVERY FIVE YEARS, YOU DON'T GET THAT IN INDUSTRY. YOU DON'T GET THAT ANYWHERE. IT'S COMPETITIVE AND REALLY KEEPS UP. WHAT THAT MEANS IS YOU'RE RENEWING HALF SO IF YOU HAVE 20% LEFT, 10% GOES TO RENEWING, HALF OF IT WILL GO TO RENEWING AND THEN IF YOU HAVE A CUT OF 7% YOU'RE LEFT WITH 3%. GUESS WHAT, THE NEW IDEAS SUFFER AND THE NEW SCIENTISTS ARE GOING TO SUFFER. THE DEPRIMING OF THE INTELLIGENCE PUMP, YOUNG PEOPLE, THAT IS THE BIGGER THAN SAYING WE'RE NOT PUTTING ENOUGH MONEY MAKING SURE WE GET A RESULT IN HEALTHCARE. THAT IS THE REFINED BALANCE BETWEEN THE TWO. TO ME I THINK YOU NEED TO REALLY BALANCE ACCORDING TO THE OPPORTUNITY, NOT THE SCIENTIFIC OR OTHERWISE, NOT TO ANY TOP DOWN FILTER FREE OF FIAT FROM THE TOP. BECAUSE IT DOES REFLECT AT THE END OF THE DAY A LOT REFLECTS WHAT THE INTERESTS ARE. THE OTHER EXPERIENCE I WOULD RELATE TO YOU IS SOMETHING GAIL KNOWS, I WAS SURPRISED THAT WE DIDN'T INVEST ENOUGH IN TUBERCULOSIS MY FIRST YEAR SO I TALKED TO TONY, HOW COME? THE WORLD IS SUFFERING FROM TUBERCULOSIS AND WE'RE ONLY INVESTING 200 MILLION OR WHATEVER. WE DOUBLED THE BUDGET. THIS BUDGET HAS NOT DOUBLED. SO WE DECIDED I TOLD THEM WE NEED TO DO MORE. WHAT HAPPENS WHEN WE FORCED IT, WE REALIZED THAT IN FACT THE SCIENCE WAS NOT BETTER. AND THERE WERE NO NEW IDEAS. THE SAME INVESTIGATORS GOT TWICE AS MUCH. SO I ASKED LET'S UNDERSTAND WHY. YOU KNOW WHAT IT TURNED OUT TO BE? IT TURNED OUT THE WORLDWIDE, WORLDWIDE AT THE TIME, THERE WERE TOTAL OF 700 SCIENTISTS WHO WERE REALLY DOING REAL WORK ON TB. ALL WHOM CAME FROM 6 LABORATORIES. SO WHAT YOU HAD IS A FUNDAMENTAL LACK OF MATERIAL TO INVEST IN. THAT'S WHAT I'M ALL WORRIED ABOUT, LET'S MOVE MONEY HERE PUT IT HERE, UNLESS L THERE IS A POOL, IT'S DANGEROUS TO PUT -- HAVE A PUSH FROM THE TOP TO SAY WE KNEE MORE OF THIS AND MORE OF THAT. SO I DON'T HAVE A GOOD ANSWER FOR YOU. I BELIEVE I WOULD NOT REDUCE THE BASIC SCIENCE INVESTMENT AT THE NIH BELOW 60% OF BUDGET NO MATTER WHAT. AND YOUNG INVESTIGATORS -- >> MUST HAVE MISUNYOU THERE OR MISSTATED BECAUSE I DIDN'T MEAN TO IMPLY THAT IN THE SLIGHTEST. I THOUGHT YOU WERE MAKING A POWER CASE THAT THE BASIC SCIENCE SHOULD BE MORE HUMAN SPECIES. >> THAT'S DIFFERENT. >> THAT'S EXACTLY MY POINT. >> THAT'S A DIFFERENT -- >> I DON'T SEE THAT HAPPENING BY WAITING UNTIL THERE'S RUSH ALONG THE LINES FROM THE IMMUNITY WHICH IS RELYING ON CONTINUED FUNDING. ROUTINE ANIMAL STUDIES. >> I MISUNDERSTOOD YOU. I THOUGHT PUTTING MORE MONEY FROM T-1 TO 4. >> HIGH QUALITY WORK DONE THERE TOO. >> RIGHT. >> THIS IS A MORE SPECIFIC ITEM. >> ABSOLUTELY. I THINK IF YOU ASK ME WHAT IS THE THREE THINGS YOU HAVE LEARNED SINCE I LEFT THE NIH THAT YOU REALLY NEED TO FIX. ONE IS DON'T DAMAGE THE YOUNG INVESTIGATORS. ABSOLUTE HI NO MATTER WHAT HAPPENS, YOU HAVE TO GIVE A CHANCE TO THE YOUNG INVESTIGATOR S. GIVE IT EARLY AND DON'T KILL THEM WITH RIGIDITY FOUR YEARS OF THAT, FOUR YEARS OF THAT, SIX YEARS OF THAT, SIX YEARS OF THAT, BY THE END OF THE COMBAT THEY'RE EXHAUSTED. THAT'S WHAT I SEE. I WAS LUCKY, I WAS GIVEN A CHANCE, I WAS 28 YEARS OLD WHEN I WAS AT HOPKINS AND MY BOSS SAID YOU HAVE SOME GATED IDEAS, WHY NOT -- GREAT IDEAS. I GIVE YOU SOME TIME Q. IN THE LAB AND DO IT. HOW MANY PEOPLE DO YOU THINK THAT HAPPENS TO TODAY? WE HAVE TO FIGHT INTERNALLY AT THE NIH TO GIVE GRANTS TO POST-DOCS. YOU REMEMBER THE KANGAROO AWARD AND ALL THE FIGHTS WITHIN THE INSTITUTES ABOUT NOT DOING IT. AND SO ON. WE HAD THE FIGHT TO GET COKE TO PUT A FLOOR ON THE NUMBER OF NEW INVESTIGATORS FUNDED EACH YEAR. STORY LANDIS CAN SHOW YOU THE THING. SO THAT'S NUMBER ONE. ABSOLUTELY CRITICAL. THE SECOND IS WE HAVE MOVED AWAY AS A ECOSYSTEM OF RESEARCH FROM STUDYING HUMAN DISEASE IN HUMANS, AND WE BELIEVE FOR A LONG TIME THAT WE COULD DO IT. THAT YOU COULD DO IT SINGLY PATHOLOGY IN ANIMAL MODELS, YOU HAVE POWERFUL MANIPULATION TOOLS AND WE ALL DRANK THE COOL AID. ME INCLUDED, THAT IN FACT WE COULD DO THIS. SO WE FUNDED UM TEEN PROGRAMS SAYING A INSTEAD OF STUFF STUDYING DISEASE WITH PATIENTS AND LAWYERS WHO SUE US, SO ON, LET'S DO THIS IN MOUSE. AND IT MAKES SENSE BECAUSE WE KNOW THAT THE GENOMIC AND KNOCK OUT KNOCK IN ANY GENE ALAN, YOU -- WITH THE MOST KNOCK OUT PROJECT AND THESE THINGS THAT WE HAVEN'T FIGURED THAT OUT. YOU HAVE THESE FANTASTIC NEW TECHNOLOGIES COMBINATORIAL CHEMISTRY, SO ON. WE SAID NOT ONLY DO WE HAVE FIGURED OUT WE CAN SCALE UP, UM TEEN TIMES THE PROBLEM IS SO WE CAN ABSOLUTELY REFOCUS EFFORTS TO ADOPTING TECHNOLOGIES THAT EMERGED. THEY ARE SUPER POWERFUL, SINGLE CELL SEQUENCING, PROTEOMICS, THE ABILITY TO SORT OF THE 30, 40 CELL TYPES IN IMMUNE RESPONSE WITH GARY KNOW LAND TECHNOLOGY AND APPLY IT TO HUMAN BEINGS. IT'S NOT -- WE DO NOT HAVE ENOUGH CHARACTERIZED COHORTS OF PATIENTS FOLLOWED OVER LONG ENOUGH PERIOD OF TIME TO UNDERSTAND THE DISEASE PROCESS.L THAT IS THE SECOND THING THAT SCARES ME. >> YOU MAKE THAT MOVE. >> WHAT? >> THAT HAS TO BE DONE BY THE INSTITUTIONS ACADEMIA INDUSTRY, PUBLIC PRIVATE PARTNERSHIPS. YOU DO IT IN THE ALZHEIMER'S DISEASE NEUROINITIATIVE, IT WORKS. >> THEY ARE INCREASING DATA TOO, JUST THIS SPRING. MORE PAPERS SHOWING EVERY TIME HOW THE MICE RESPOND DIFFERLY FROM HUMAN. SO -- BUT PEOPLE -- THEY DON'T HAVE A LOUD ENOUGH VOICE. BARRY BLOOM HAS BEEN SAYING AND PUBLISHED ALMOST 20 YEARS AGO THAT THE MOUSE MODEL FOR TB SHOULDN'T BE USED. BECAUSE THE MOUSE MACROPHAGES DON'T PROCESS TB LIKE HUMAN MACROPHAGES BUT WHAT ARE WE USING FOR SCREENING COMPOUNDS? A GAMMA INTERFERON KNOCK-OUT MODEL. BEING IN MICE. >> I AGREE WITH YOU, I HADN'T THOUGHT ABOUT IT BUT TO YOUR POINT, I DON'T KNOW HOW TO CHANGE IT BUT WE -- IT'S NOT PART OF THIS BUT IT'S PART OF I THINK THE NEXT CHARGE IN TERMS OF GRANT REVIEW. I MEAN, IT'S CERTAINLY PLAYING A ROLE IN THAT. BUT IT'S TRUE IN DISEASE. >> THE THIRD FEAR I HAVE BUDGET PRESSURES ON HEALTHCARE WILL KILL ANY INKLING OF NEW INNOVATIVE -- INNOVATION, BEING USEFUL FIND A WAY TO REFLECT AND CARDIOVASCULAR DISEASE, ALZHEIMER'S DISEASE, YOU HAVE FIVE CONDITIONS, FIVE DISEASES THAT WILL BASICALLY BANKRUPT, BANKRUPT, NOT KIDDING, BANKRUPT SOCIETY, EVOLVED SOCIETIES MANY THE NEXT 40 YEARS IF WE DON'T FIND SOLUTIONS. THE GAMBLE, THE CHALLENGE IS HUGE. WE CANNOT GAMBLE I THINK WITH JUST SAYING JUST LET IT GO. SO I DO BELIEVE WE NEED TO HAVE SOME YARDSTICKS OF MEASURE HOW CAN WE REDUCE THAT. IT'S NOT GOING TO HAPPEN WITHOUT A CHANGE IN THE WAY WE DO"N RESEARCH. WE FUND RESEARCH. >> SPONTANEOUSLY. >> NO. >> I WOULD LIKE TO JUMP IN HERE AND SAY IT ISN'T ROCKET SCIENCE. >> HE IS A ROCKET SCIENTIST. >> ONE COULD FRET -- WHEN YOU HAVE A CLIA, I JUST LOVE THE CLARITY OF WHAT YOU SAID. WHEN YOU HAVE A CLEAR PATH TO FOLLOW AND SMT COMES TOGETHER AND AGREES TO A PATH PERHAPS A LITTLE MODIFIED EXECUTION IS EASIEST PART. HAVING CLARITY OF THOUGHT AND FOCUS YOU HAVE THE ABILITY IF THERE'S A PARTNERSHIP WITH THE LEADERSHIP AND COMMITTED TO DOING IT, THE FIRST STEP IS YOU WORK WITH THOSE WHO ARE YOUR KEY OVERSIGHT CONSTITUENTS IN THE WHITE HOUSE AN IN THE CONGRESS YOU HAVE TO BE CLEARLY OPEN ABOUT IT. YOU CAN'T BRING EVERYBODY ALONG. YOU'RE GOING TO LOSE PEOPLE AND THAT'S OKAY IF YOU HAVE THE RIGHT VISION. YOU ALSO CANNOT DO IT IN THE YEAR OR TWO, THIS IS SOMETHING THAT TAKES TIME AND YOU'RE TALKING ABOUT A 3 TO 5 YEAR PERIOD. BUT BECAUSE PEOPLE STAND UP AND SAY HELL, NO, IT CAN'T GO BECAUSE I'M COMFORTABLE BECAUSE I'M LOOKING OUT THE BACK OF THE BUS AND I SEE THE VISION CLEARLY, IT DOESN'T HAPPEN, IT TAKES PAIN, IT WOULD BE BREAKAGE BUT IF YOU SIGN UP KEY PEOPLE WHO ARE LEADERSHIP CONSTITUENTS IT CAN BE MADE TO HAPPEN AND I THINK THIS IS A GOOD APREACH BUT YOU CAN BACK OFF WITH THE RIGHT APPROACH BECAUSE OTHERWISE IT ISN'T GOING TO HAPPEN AND IT WILL ONLY GET WORSE. I LOOKED AT THE AGENERAL DA AND FRANKLY EVERY PANEL YOU CAN SAY PRINCIPLE METRICS STRATEGIES AND CALFIOTS THERE ARE DIFFERENT IN THE T-1 STAGE T-2, T-3, T-4, RELATIVE TO EXPECTATIONS OF SOCIETY. THEY CANNOT BE DIVORCED. YET WE'RE DOING WELL, AS I LOOK AT THE GLOBE U.S. IS STILL THE TOP PRODUCER OF KNOWLEDGE AND SMALL INNOVATIVE COMPANIES PARTNERSHIPS EASIER THAN EVER, TRY FRANCE, I LOVE PARIS BUT TRENCH SYSTEM NOT SURE. SO IF YOU LOOK AT THAT, PUBLIC HEALTH OUTCOMES AND OUTCOMES OF BIOMEDICAL RESEARCH THAT'S T-3, T-4, YOU CAN SEE HOW YOU FRAME IT, ALMOST LINE ALIGN IT TO WITH YOU NEED TO PROVE. AT THE END OF THE DAY FOR MEASURING VALUE. AND AT THE END OF THE DAY I AGREE WITH WHAT YOU SAID, IT'S SOMETHING THAT YOU HAVE TO HAVE A VISION AND FRAMEWORK TO TRANSFORM. IT'S VERY CLEAR IF YOU DON'T IDENTIFY WHAT ARE CREATORS OF VALUE, WHAT ARE THE LEVERS, IT'S YOUNG PEOPLE, SUPPORTING THEM, SUPPORTING NEW IDEAS AND DEFINITELY STUDYING HUMAN DISEASE IN HUMANS BETTER THAN WE DO TODAY. AND STOP DANCING AROUND THE POT. WE HAVE TO GO AWAY FROM IT, THERE'S NOT A PLACE TO GO TO THAT HAS CONSISTENT HUMAN COHORTS OF PATIENTS THAT YOU CAN STUDY THAT THE INDUSTRY CAN ACCESS. IT'S UNBELIEVABLY COMPLEX. NIH MADE IT MORE COMPLEX BY NOT BEING CLEAR. WHAT IS EXPECTED. I THINK ACADEMIA UNDERSTANDS THAT SLOWLY BUT IT'S HARD, IT'S HARD BECAUSE HEALTHCARE IS SO COSTLY TODAY. SO MUCH EASIER TO GET A PAPER AND PROMOTE IT ON A COUPLE OF MOUSE PAPERS. I DON'T MEAN TO DERIVE MOUSE PAPERS, THEY'RE VERY IMPORTANT, DON'T TAKE ME WRONG BUT THERE'S A MISSING PIECE CREATING THE VALUE THAT GETS YOU TO THAT BRIDGE, REAL VALUES PERCEIVED BY SOCIETY. >> ELIAS HAS ONE THAT CUTS ACROSS THE T-1, THE T-2 AREAS. GOING FROM FUNDAMENTAL KNOWLEDGE TO MODELS TO HUMANS. I WAS GRATIFIEDED TO HEAR YOU ENDORSE THE 3-D TISSUE MODELS PROJECT. AS YOU WILL RECALL WE INTRODUCED THAT AS A CONCEPT WHEN YOU WERE HERE. IT'S BECOMING A REALITY NOW. BUT I WOULD LIKE YOUR PERSPECTIVE WHAT YOU SEE AS POTENTIAL ROLE AND VALUE OF DEVELOPING MODELS COMPUTATIONAL MODELS AS WELL AS OTHER TYPES OF MODELS IN ORDER TO BE MORE EFFICIENT IN ADDRESSING PROBLEMS AND GETTING TO RATIONAL SOLUTIONS A BIT MORE EFFICIENT EFFICIENTLY. SOMETHING WE TALKED THEORETICALLY BUT WE DONE SEE IT HAPPENING AS MUCH AS MAYBE BEGINNING TO -- YOUR PERSPECTIVE ON THAT, IN PARTICULAR IN THE INDUSTRY SPACE. >> SO I THINK AGAIN, YOU USE THE FRAMEWORK, T-4 POPULATIONS, WE ABSOLUTELY NEED TO USE MODERN INFORMATION TECHNOLOGY TO CAPTURE MORE DATA MORE REAL TIME MORE PEOPLE ACROSS MULTIPLE PATTERNS OF TREATMENTS SO WE CAN MEASURE WHAT IS HAPPENING ON THE GROUND. WE DON'T KNOW HOW TO MANAGE BIG DATA, WE DON'T KNOW HOW TO QUERY BIG DATA, WE DOPE KNOW HOW TO UNDERSTAND WHAT HAPPENS WITH THE PATIENTS OUT THERE. WHO DOES? I DON'T THINK ANYBODY DOES, THAT'S AREA TECHNOLOGY IN YOUR MINE WHAT YOU'RE DESCRIBING IS GOING TO BE CRITICAL. THEN WHEN YOU TALK ABOUT T-1 THE 3-D DISCUSSION SEVERAL YEARS AGO, WE COULDN'T DO IT WITH MOUTH MOUSE, NOW WE' EEL DO IT WITH FUNNY MODELS TO REPRESENT HUMAN DISEASE. THE REASON I ENDORSE WHAT IS BEING DONE IS BECAUSE IT'S FOCUSED ON SAFETY AND TOXICOLOGY. HOW DO YOU CREATE CHIP MODELS IN THAT REPRESENT SPECIFIC BLOCK YOU HAVE IN T-1 AN T-2 TRANSLATION WHICH IS SAFETY. SO THAT'S A MUCH MORE FOCUSED GOAL IN MY VIEW, I CAN DEFINE IT, I CAN SEE IT. THE IDEA DEFINING DISEASE MODELS IN VITRO IS SOMETHING THAT COULD L WORK, I DON'T KNOW, IT COULD WORK FOR CERTAIN DISEASE BUT I DON'T BELIEVE IT'S A REPLACEMENT FOR UNDERSTANDING HUMAN DISEASE IN HUMANS. THAT WAS THE REASON. EVERY TIME IF YOU LOOK AT IT, YOU SEE THAT IN FACT THERE IS A PLACE FOR NEW TECHNOLOGY I COMPLETELY BELIEVE AND AGREE WITH THE FACT THAT FOR EXAMPLE, TODAY, THAT SPEAKS SPECIFICS, LET'S TALK ABOUT SPECIFICS, CANCER. WE HAVE SO MANY DIFFERENT UNDERSTANDINGS OF CANCER, NOT ONLY IN TERMS OF HETEROGENEITY, MOLECULAR HETEROGENEITY, MOLECULAR SIGNATURES, DIAGNOSTIC COMPANIES LOOKING AT EVERY ONE CANCER SEEING -- BUT IT CHANGES OVER TIME. SO HOW ARE WE GOING TO MEASURE THAT IN A HUMAN BEING? YOU NEED MICROSAMPLING, YOU NEED TO HAVE VERY HIGH RESOLUTION MASS SPEC TECHNOLOGY ALMOST CONNECTED TO THE BEDSIDE THIS IS A NEW WAY AND TECHNOLOGY IS -- THERE'S A COMPANY THAT SIMULATED A LYMPH NODE FOR IMMUNE RESPONSE, IT IS AMAZINGLY PREDICTIVE IN INFECTIOUS DISEASE AREA FOR VACCINES. FOR INFECTIOUS DISEASE. YOU TRY FOR SOMETHING ELSE, IS NOT AS PREDICTIVE. SO I THINK WE NEED TO HAVE MORE OF THAT JUNCTION, THAT'S WHY I THINK SUPPORTING NSF IS REALLY IMPORTANT THING TO DO BECAUSE IT COMES FROM THAT FUNDAMENTAL RESEARCH. >> NOT AS REPLACEMENT BUT ONE THING IN OUR TOOLBOX. >> I AGREE. >> ONE MORE. YEAH. SURE. >> CLIFF, GIVE YOU THE LAST WORD IN THIS SEGMENT. >> I JUST WANT TO AGREE WITH TRANSLATION HAL SPACE T-1 TO 4, BUT TO ADD ANOTHER DIMENSION TO THAT WHERE YOU PUT THE EFFORTS ALONG THAT TRANSLATION SPACE AND IT RELATES TO SOME OTHER LETTERS THAT YOU INVOKED A NUMBER OF YEARS AGO, THAT'S THE P, THE FOUR Ps. I THINK EVERYONE AGREES THE DISEASES THAT CAN BE PREVENTED WOULD BE MUCH MORE COST EFFECTIVE AND EFFICIENT THAN WAITING FOR SECONDARY AN TERTIARY PREVENTION. SO IF YOU HAVE A POPULATION YOU JUST MENTIONED WE HAVE BEEN THE CURVE TO PROBABLY FIVE DISEASES, CERTAINLY DIABETES IS ONE OF THOSE. BUT IF YOU HAVE EFFECTIVE WAYS TO PREVENT IT, PEOPLE AT HIGH RISK-BASED UPON PREDICTIVE MARKERS, AND OTHER PATTERNS HOW DO YOU SEE THAT IN THE TRANSLATION SPACE IF THERE'S NO ONE INTERESTED IN FUNDING PREVENTION, IT GETS TO QUESTION RAISED ABOUT BEHAVIORAL TYPES RESEARCH, NO ONE IS INTERESTED BECAUSE NET RESULTS MAYBE SO FAR IN THE FUTURE THAT WE'RE ON THIS REALLY SHORT TIME CYCLES. SO YOUR COMMENTS ON THAT YOU'RE POINTING OUT FAILURE OF THE SYSTEM, NOT FAILURE OF FED ALLEY FUNDED RESEARCH. IT'S THE WAY THE SYSTEM IS DESIGNED, BASICALLY CREATES THE WRONG INCENTIVES AND FRANKLY THAT HAS TO CHANGE, PERSONALLY I THINK -- I'M SAYING THAT NOT JUST FROM THE POINT OF VIEW INDUSTRY ACADEMIA BUT AS A SIMPLE WAY OF LOOKING AT THINGS WE CANNOT AFFORD THE STACKING OF MULTIPLE $50,000 CANCER DRUGS ON TOP OF ONE PATIENT SIX MONTHS OF LIFE EXPECTANCY, RIGHT? SO AT SOME POINT THE SYSTEM WILL SHIFT FROM PAYING FOR INPUTS, SYRINGE, PILL AND THINGS BEYOND THAT TO PAY FOR OUTCOMES. THE PAYING OF -- ON THE BASIS OF OUTCOMES HOPEFULLY WILL THEN CHANGE THE PERCEPTION THAT YOU HAVE ON THE P PREVENTION. THE SECOND POINT IS WE DO NOT HAVE AS Y'ALL KNOW, WE HAD THE BIOMARKERS INITIATIVE AND SO ON, I TALKED ABOUT PREDICTIVE PREEMPTIVE PERSONALIZED MEDICINE, AND -- BUT IT WILL REQUIRE PARTICIPATION, SOCIETY AND INDIVIDUALS TO DO THAT BUT WE HAVE TO PROVIDE THE SCIENTIFIC BASIS TO DO THAT. IT'S VERY CLEAR, IF YOU LOOK AT STATINS FOR EXAMPLE, WE GIVE STATINS TO 100 MILLION PEOPLE, STATISTICALLY WE KNOW THAT LESS THAN 10% OF THOSE ARE GOING TO BE PREVENTED FROM HAVING A EVENT. HOW DO WE KNOW THIS? BECAUSE WE KNOW BASIC EPIDEMIOLOGY OF THE INDIVIDUALS WITHOUT TREATMENT AND THEN WITH TREATMENT. SO 90% OF WHAT WE DO THERE IS NOT -- IS NOT HELPFUL. SO PAYERERS WILL BE VERY INTERESTED IN FINDING OUT WHICH 19% I SHOULD AVOID TREATING AND THAT WOULD BE THE SETTINGS. THE POINT YOU MAKE IS A VERY IMPORTANT POINT. THERE IS A SOCIETAL EXPECTATION OF CURE, NOT PREVENTION. AND SYSTEM IS DESIGNED THE PAY FOR THAT. THE SYSTEM IS DESIGNED AT TIME OF ACUTE DISEASE. THE SYSTEM WAS NEVER DESIGNED AT TIME OF CHRONIC DISEASES BEING TOP BURDEN. THE SOCIETAL NEEDS AND EXPECTATIONS HAVE CHANGED AND THE SYSTEM HASN'T. TODAY I TELL PEOPLE WE'RE TAKING CARE OF CHRONIC DISEASE IN THE HOSPITAL, THE MOST EXPENSIVE ENVIRONMENT POSSIBLE, AND IT'S ALMOST LIKE GOING TO GROCERY STORE WITH AN F-16 AIRPLANE AND LANDING NEXT TO IT WHEN YOU CAN GO THERE WITH A BIKE. IT'S NOT VERY SMART. >> GOOD IDEA FOR YOU. [LAUGHTER] >> SO THAT IS A QUESTION BEYOND THE VALUE. YOU LIKE THAT. I WISH I HAD THOUGHT ABOUT THAT A FEW YEARS AGO. >> TERRIFIC. THANK YOU SO MUCH. I HAD A QUESTION TO ASK YOU BUT I'M GOING TO SAVE IT FOR THE PANEL SINCE YOU HAVE KIND ENOUGH TO AGREE TO THE PANEL. AT THIS POINT OUR VARIOUS PAMMISTS DURING THE DAY WOULD BE KIND ENOUGH TO JOIN US IN THE FRONT. WE HAVE HAD A COUPLE THAT HAVE TO CATCH AIRPLANES AND WE'RE DOWN TO HARD CORE AT THIS POINT. SO I WILL -- THIS TIME GAIL, TO YOU, WE'LL BEGIN WITH A QUICK RECAP FROM PANELS ONE AND TWO. FROM THE WHAT WE CALL, MODERATORS. AND THEN WE'LL HAVE A CHANCE TO TALK TO THE PANELS AS A WHEEL. GAIL, YOURS AGAIN. >> GILL AND JOSEPHINE, COULD WE ASK YOU TO LEAD OFF? SUMMARIZING THE OUTCOMES PANEL 1 THAT WAS WHERE THE TOPIC WAS PRINCIPLES AN METRICS. >> I'LL START TO PICK UP WITH IMPORTANT ELEMENTS I LEAVE OUT. SO WE STARTED WITH A PRESENTATION FROM DELLA HAHN WHO TOLD US -- GAVE US NUMBERS ABOUT THE PROGRAMS AND PROCESSES THAT NIH HAS IN PLACE ALREADY TO ASSESS OUR PRODUCT SUCH AS PUBLICATIONS. AND A NUMBERS ABOUT THE LARGE NUMBER OF PEOPLE THAT THE NIH ENTERPRISE, THE ECOSYSTEM, IS SUPPORTING. SHE TALKED ABOUT THE DEVELOPMENT OF METRICS RELATED TO A SECOND SET OF OUTPUTS RELATED TO ACTUALLY IMPACTING ON THE NATION'S HEALTH. BUT WE DIDN'T DELVE INTO HOW WELL THOSE OUTPUTS WOULD WORK. THE NEXT PRESENTATION WAS JIM ANDERSON, WHO GAVE TWO TELLING EXAMPLES OF THE STRENGTHS AND WEAKNESSES OF BIBLIOMETRIC EVALUATION, INCLUDING THE CAREFUL PROCESS ESTABLISHED TO LOOK AT THE VALUES OF THE PIONEER AWARDS. AND THEN AN EXAMPLE FROM A RELATIVELY FLAWED BIBLIOMETRIC ANALYSIS THAT LOOKED AT NIH PRODUCTIVITY. THEN THE THIRD PRESENTATION WAS A LIVELY ENDORSEMENT BY LAUREL HACK OF THE IMPORTANCE OF STRENGTHENING OUR DATA RESOURCES FOR EVALUATION. AND STRENGTHENING EVALUATION PROCESSES INCLUDING IMPROVING THE DATA SYSTEMS THAT ARE AVAILABLE FOR EVALUATION PROCESS. I PERSONALLY TOOK OUT OF THIS CONVERSATION TWO THINGS, WE KID NEED TO EXERCISE CAUTION IN BIBLIOMETRIC TOOLS AND REMINDERS BY PUBLIC HEALTH PANEL LATER IN THE DAY AND THEN ELIAS'S CAREFUL REMINDER TO FRAME OUR EVALUATION AN VALUE ASSESSMENT IN THE TRANSLATIONAL STAGES THAT NONE OF THE METRICS THOUGH PART OF GOOD PORTFOLIO MANAGEMENT AND PART OF GOD ANALYSIS INTERNALLY, ARE GOING TO NECESSARILY SERVE IN BROADER CHALLENGE TALKING VALUES NEEDED FOR THE STAGES OF RESEARCH. I ALSO TOOK AWAY THAT WE DIDN'T HAVE CONCRETE SUGGESTIONS ABOUT METRICS RELATED TO PEOPLE. WE SAW A LITTLE BIT OF THE WAY PEOPLE ARE USING NETWORKING TOOLS AND SOCIAL MEDIA TOOLS BUT IT IS REALLY VERY CLEAR THAT MUCH OF THE VALUE OF THIS $30 BILLION DOES LIE IN THE SUPPORT OF ENTERPRISE OF CRITICALLY THINKING PEOPLE. AND THAT WE ANALYZE THEIR PRODUCT BUT WE DONE HAVE A GOOD WAYS TO CREATE METRICS AROUND WHETHER OUR INVESTMENTS IN PEOPLE ARE TARGETED TO THE RIGHT KIND OF RESEARCH. IN MANY WAYS WE INFLUENCE THE PEOPLE POWER THAT WE SUPPORT THROUGH THE PEER REVIEW PROCESS, THE PEOPLE WHO GET FUND REDIRECT EXAMINATION THE PEOPLE WHO SURVIVE IN THIS ECOSYSTEM AND WE DON'T STAND BACK OR HAVE GOOD WAYS TO STAND BACK AND SAY HOW WELL IS THAT WORKING. SO THAT WAS THE THOUGHTS I WOULD DERIVE FROM THIS. I WILLu#jy ADD THAT I AM SUPPOSED TO BE -- I WOULD ADD I AM POSED TO BE CHANNELING STEVE CATS, I HAVEN'T BEEN A MEMBER OF THE PANEL ALL ALONG SO I'M GIVING YOU THOUGHTS OF SOMEONE A LITTLE NEW TO THIS DIALOGUE. >> THAT WAS A SPLENDID -- YOU HAVE FILLED IN FOR STEVE ADMIRABLY TODAY. THIS WAS ENTIRELY NIH INSIDE PRESENTATION ABOUT THE KIND OF DATA AVAILABLE AND THE DATA ASSESSMENTS COULD BE USED ASK QUESTIONS PARTICULARLY TO TRACK PEOPLE THROUGH TEN OR NOR YEARS OF TRAINING EXPERIENCE AND EARLY CAREERS. IN A WAY THAT'S APPARENTLY NOT BEING DONE REGULARLY AT ALL. LAUREL'S POINT WAS A LOT OF DATA COLLECTED, IT'S IN DIFFERENT KIND OF DRAWERS AND SOME SERIOUS EFFORT RATHER MODEST COST MIGHT LINK THEM TOGETHER IN A WAY THAT WOULD BE QUITE USEFUL. OTHER INTERESTING COMMENTS, D DELLA HAHN REFERRED TO THE RCDC SYSTEM WHICH WE TALKED ABOUT A FEW MONTHS AGO. AND COVERED IT WITH A PHRASE FOR BETTER OR WORSE AND I DON'T KNOW IF ANYBODY WANTS TO ADDRESS THAT FURTHER BUT IT'S A HARD CHALLENGE HOW TO CATEGORIZE THE VAST ARRAY OF ACTIVITIES IN JUST THE RESEARCH ACTIVITIES OF THIS WONDERFUL ORGANIZATION. AND THE SPEAKER WHO WAS USING RCDs -- RCDC IN A MEMORIAL ARTICLE WHOSE PAPER IS IN OUR BOOK, WASN'T ABLE TO MAKE IT THE LAST MINUTE SO WE DON'T HAVE THAT DISCUSSION. THAT PAPER ITSELF WAS LIMITED BY THE RESOURCE THAT WAS -- HE JUST ACCEPTED EVERYTHING AT FACE VALUE. JIM ANDERSON GAVE QUITE AN ACCOUNT OF THIS THAT TRIED TO COMPARE MATCH SAMPLES OF PIONEER AWARDEES FOR THE FIRST THREE COHORTS. HOWARD HUGHES WERE THE SIMILAR NUMBERS SAMPLED FROM THE RO-1 POOL. THOUGH HE DIDN'T INCLUDE THE DATA, THE PIONEER AWARD FINALISTS WHO WERE NOT FUNDED SO THERE'S FOLLOW-UP ABOUT THOSE GROUPS, STATISTICAL ANALYSIS ON PUBLICATION RATES AND EVEN 94 PEER REVIEW EXPERTS TO LOOK AT THEM. QUALIFICATIONS TOP FIVE PUBLICATIONS AND THAT WAS NEVER SYSTEM. LITTLE HIGHER THERE, LOWER THERE. BUT IF IT WAS -- IF IT WERE REALLY TRUE, THAT THE PIONEER AWARDS WHICH WAS A WONDERFUL THAT WOULD NOT HAVE OTHERWISE BEEN FUNNED, AND WAS HIGH RISK THEY PERFORMED THE SAME LEVEL AS THESE HIGH PERFORMING COMPARISON GROUPS, THAT'S NICE REINFORCEMENT. SO THE EXPERIMENT TO TRY NEW THINGS IN NEW WAYS GOT AT LEAST THE SAME LEVEL OF PERFORMANCE BY THE TRADITIONAL MEASURES. THAT MAYBE STARTING NEW THINGS. >> GOOD. THERE WERE SOME TOPICS WE DIDN'T REALLY GET MUCH INPUT ON. MOSTLY HAD TO DO WITH QUALITATIVE ASSESSMENT OF T-1. THE FUNDAMENTAL KNOWLEDGE GENERATION PROCESS AND ITS POTENTIAL TRANSLATION. EVERYTHING WAS ABOUT COUNTING THINGS. IN WAYS THAT ARE BEGGING FOR BETTER INTERPRETATION LIKE QUALITY, BREAK THROUGHS, POTENTIAL FOR TRANSLATION AT SUBSEQUENT STAGES, RECOGNITION AND COMPENSATION OF TIME LIKES GENERATION KNOWLEDGE AND TOOLS AN UTILIZATION OF NEW KNOWLEDGE. ALL THOSE SEEM TO BE REALLY IMPORTANT DIMENSIONS FOR WHICH SOME METHODS SHOULD BE HELPFUL. AND THEN THERE WAS FUNDAMENTAL QUESTION OF THE CAUSEAL LINK. IT'S A LONG WAY. EVERYBODY HAS SAID HERE TODAY IT'S A LONG LINK FROM SERIES OF LINKS FROM EARLY IDEAS TO4h– DISCOVERY SCIENCE, ALL THE WAY TO MAKE IMPACT ON HEALTH LONGEVITY, DISABILITY AND SOCIETY. TO THE EXTENT PEOPLE ASKING US TO PERFORM THIS TASK, OUR TASK IS JUST TO SAY WHAT'S FEASIBLE AND HOW IT MIGHT BE DONE PRESUMABLY THE NIH HAS TO CONTRACT OR DO A SERIES OF CONTRACTS TO COVER THIS WATER FRONT. HOW TO COVER THAT STRETCH WITH ALL THE IMBEDDED ASSUMPTIONS AND VERY SUBSTANTIAL TIME DEMENTIONS DURING WHICH THINGS CHANGE IN SOCIETIES AND INDUSTRIES, THIS WE HAVE TO FOCUS ON MORE I THINK. WE DIDN'T GET TO THIS IN THIS TODAY.f‡ >> THANK YOU. ARTHUR AND GIVE, PANEL 2, THE PUBLIC HEALTH IMPACT. >> MAYBE ISLE START AND TURN OVER TO ARTHUR AND WE'RE FORTUNATE TO HAVE TWO OF THE DISCUSSANTS HERE SO MAYBE THIS IS A LITTLE BIT TRUNCATED. I'LL SPEND MOST OF THE TIME DISCUSSING THAT PRESENTER WHO HAD TO LEAVE. THIS DISCUSSION FOCUSED ON PUBLIC HEALTH ASPECTS OF THIS ON THE INDIVIDUAL METRICS OF WHETHER WE DO IN TERMS OF INPUT TO LOOKING AT TRANSLATING KNOWLEDGE ACQUISITION TO PUBLIC GOOD AND NEED WE HER FROM DR. CURRAN, IT WAS A VERSION OF YOUR T-1 THROUGH T-4 WAS MORE TRUNCATED T-1 NOT DIRECTLY T-4 NICE PERSPECTIVE ON THE COMMUNITY SOCIETAL BENEFIT OF BASIC DISCOVERY OF SCIENCE AND ILLUSTRATE AD COUPLE OF CAVEATS ABOUT ATTRIBUTION, HOW MUCH CAN BE CREDITED FROM THE BASIC DISCOVERY AND HOW THAT BENEFITS SCIENCE AND CAVEATING THAT WITH PRACTICAL EXAMPLES OF WORK YOU DID AND OTHERS IN HIV RESEARCH AND HOW THAT NOT ONLY BENEFITED THE U.S. BUT ALSO HAD MORE A GLOBAL IMPACT. DR. BATCHTHER HAD A RECIPE PROVIDED NATIONAL CENTERS FOR HEALTH STATISTICS AND PROVIDE AD LOT OF INFORMATION WHAT CDC COLLECTS TRENDS AT LEVEL OF STATE AND HOW EFFECT ON MORTALITY CHANGE OVER TIME AND AGAIN, ISSUE OF ATTRIBUTION, HOW ONE MIGHT CONSIDER DETERMINING WHAT NIH'S CONTRIBUTION RESEARCH HAD ON THOSE PARTICULAR IMPACT, ATTRIBUTION. BUT I THINK MORE IMPORTANTLY LOOKING AT THESE PUBLICLY AVAILABLE DATA SETS THAT THEY NOT ONLY HAVE BUT ALSO ABILITY NOW TO LINK TO ADDITIONAL DATA SETS THAT THAT EXIST TO BEGIN TO ASK VERY IMPORTANT QUESTIONS. AND THEN SAY DR. ENTWISTLES PRESENTATION, GAVE A VERY NICE TALK, I WAS TRUCK BY THIS PAPER FROM PNAS THAT YOU PRESENTED SHOWING THE INCREDIBLE IMPORTANCE OF TIME LAG BETWEEN THE DISCOVERY SCIENCE AND HOW YOU CAN ACTUALLY SUPER IMPOSE THAT UPON IMPROVEMENTS WE'RE SEEING AT LEAST IN MORTALITY IN CERTAIN DISEASES, AGAIN ATTRIBUTION IS AN IMPORTANT ELEMENT THAT ONE HAS TO CONSIDER. BUT I THINK THE BRIEF PRESENTATIONS WE HAD THREE PREGNANTS IN TOTAL GAVE A GLIMPSE HOW WE CAN BEGIN TO ADDRESS THE TOPIC OF WHAT IS A PUBLIC HEALTH SOCIETY TITLE VALUE AND PUTTING SOME ESTIMATES QUANTITATION OF THAT IN TERMS OF WHAT NIH HAS DONE IN TERMS OF VALUE BIOMEDICAL RESEARCH. >> WE HAVE TWO PANELISTS HERE, I WOULD LIKE THEM TO COMMENT IN RETROSPECT ABOUT WHAT -- THAT HERE TODAY. I BRING UP THE ONE ISSUE AGAIN WHEN YOU LOOK AT THE FUNDING OF THE NIH, I GUESS WHEN WE HAVE TO HAVE THE CDC AND SOME OTHER MODEST FEDERAL AGENCYINGS IN TERMS OF THE BUT LICK HEALTH IMPACT OF A LOT OF WHAT THE NIH GENERATED IN KNOWLEDGE, IT'S PRETTY MODEST AND ALSO UNDER A FAIR AMOUNT OF CRITICISM AT TIMES. ANOTHER QUESTION I ASK ELIAS, I'D LOVE THE TWO OF YOU TO THINK ABOUT OR ANSWER TO US WHEN WE GET TO THAT STAGE ABOUT THE OPPORTUNITIES YOU SEE FROM ALL THE WORK YOU HAVE DONE AND THE INSTITUTES YOU LEAD IN TERMS OF IF WE DID INVEST MORE IN THAT AREA, AND WE COULD BRING PEOPLE ALONG TO HAVE REALLY TOUGH SCIENCE THAT'S SUPPORTABLE, WOULD WE MAKE THE IMPACT FAIRLY SOON IN TERMS OF HEALTH OF PUBLIC LONGEVITY DECREASES SO ON. SO THAT QUESTION IS AN IMPORTANT ONE. IT COMES ALSO TO WHAT ELIAS SAID IN TB. DO WE ARE HAVE THE INFRASTRUCTURE READY TO APPLY THAT THING EVEN IF WE PUT THE MONEY THERE BECAUSE AT LEAST TO SOME OF MY THING, THIS BEHAVIORAL RESEARCH NUMBER OF PEOPLE DOING RESEARCH IN THAT AREA IS RELATIVELY MODEST. I WOULD BE INTERESTED IN YOUR OPINIONS FROM YOU STANDING FROM LOOKING AT PUBLIC HEALTH IMPACT AND SO ON. (OFF MIC) >> HELEN AND DAN. >> >> I WILL TRY TO BE CONSISTENT WITH WHAT I SAID BEFORE BUT MORE SUCCINCT. DAVID CUTLER OPENED WITH NOTING THAT THE NIH MISSION IS NOT JUST ABOUT INVENTING CURES. THEREFORE WE NEED TO STUDY THE SOCIAL CONSEQUENCES OF HEALTH IMPROVEMENT AS WELL AS OF CURES. BECAUSE OF THAT WE NEED TO ADVANCE THE SCIENCE OF BEHAVIOR. HE BROUGHT UP EACH OF THE THREE SPEAKERS RETURN TO THAT RETURN ON NIH SCIENCE IS DEPENDENT ON THE SCIENCE OF HEALTH SYSTEMS AND DELIVERY. MARK MCCLELLAN TALKED BEHAVE BEHAVIORAL KNOWLEDGE, BENEFITS AND COSTS AND AGAIN HIT THE FACT MOST COSTS ARE NOT DOING RESEARCH BUT IMPLEMENTING IT. THERE ARE VARIOUS WAYS TO MEASURE VALUE AND CAPACITY TO DO SO IS IMPROVING, EXAMPLES OF THIS, ALSO GAVE EXAMPLES OF VARIOUS WAYS TO INCREASE VALUE AND IP INVESTIGATE MORE EFFICIENTLY THAN WE CURRENTLY ARE. ROBERT TOPEL NOTED THAT KNOWLEDGE IS A PUBLIC GOOD BUT IT'S A DIFFICULT ONE TO MEASURE. BUT WE CAN PLACE BALLPARK EVALUATIONS ON PAST HEALTH GAINS BUT CAN'T CONTRIBUTE SPECIFIC CAUSES TO MOST OF THOSE GAINS. SIMILAR LAYER WHILE WE CAN ESTIMATE THE VALUE OF FUTURE GAINS WE HAVE A DIFFICULT TIME ESTIMATING WHAT THEIR COSTS WILL BE. HE MENTIONED WE SHOULD NOT FORGET IMPLICATIONS OF RESEARCH FOR THE REST OF THE WORLD, WHILE WE METRICS FOR THE U.S. IT GOES MORE BROADLY. THAT THE COST INNOVATIONS ARE MUCH GREATER THAN DEVELOPING THE INNOVATIONS. THAT THE VALUE OF HEALTH IMPROVEMENTS, CONTRIBUTIONS TO GDP OR JOBS CONTRIBUTIONS WELL BEING THE DISCUSSION AT THE END OF THE PAM THE POINT MADE UP MADE THAT THE GREATEST RETURN TO NIH RESEARCH IN THE FUTURE MAY NOT BE IN THOSE AREAS TRADITIONALLY FOCUSEDDED IN THE PAST BECAUSE HAVING FOCUSED THERE IN THE PAST WE HAVE HAD SOME REAL BENEFITS, WE HAVE GOTTEN MAYBE THE LOW HANGING FRUIT BUT ALSO I THINK WE'LL BE SORT OF TIE IN THEY DIP GIVE THIS EXAMPLE BUT WITH THE EXAMPLE ELIAS MADE ABOUT THE FRUSTRATIONS OF ANIMAL MODELS. >> I WOULD LIKE TO OPEN UP FOR BROADER DISCUSSION, JUST -- I FEEL LIKE THROUGHOUT THE WHOLE DAY, THERE'S BEEN A CLOUD OVER THE TABLE THAT HASN'T BEEN TOTALLY ADDRESSED WHICH IS HEALTHCARE COSTS. I WOULD BE INTERESTED IN THE OPINIONS OF ALL THREE OF YOU ABOUT THIS ISSUE PARTICULARLY NOW THAT THERE IS A NEW AGENCY, SPECIFICALLY GENERATED OR ESTABLISHED PARTLY TOH TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST TEST THE CLINICAL RESEARCH, BENCH SCIENTIST, I SEE GLIMMERS OF GOOD THINGS HAPPENING BETWEEN THE AGENCIES, TO SEE PEPFAR AIDS RESEARCH AND THEN INCREASINGLY THE NCHS NIH COLLABORATIONS SOME COLLABORATIONS OF DATA SYSTEMS THAT CAN TRACK BASIC SCIENCE FINDINGS WHICH THEN RESULT IN CHANGES IN NEURAL TUBE DEFECTS AND ALL THESE THINGS. THE THING I WORRY ABOUT THE MOST WITH THE SWALLOWING THAT ELIAS TALKED ABOUT WITH THE HEALTHCARE COSTS, NOW DOMINATING THE LANDSCAPE, IS EVERYONE WILL RETREAT TO THIS SILOS. WHEN THE CDC BUDGET IS CUT MORE THAN NIH'S, THEY HAVE ALWAYS BEEN GUN SHY ABOUT FUNDING RESEARCH. NOBODY FUNDS RESEARCH LIKE NIH DOES. MAYBE NEED TO BE CHANGEDDED BUT PEER REVIEWED SCIENCE AND ENGAGES THE ENTIRE RESEARCH INFRASTRUCTURE OF THE COUNTRY AND BEYOND WHAT USED TO BE CALLED AHRQ NOW UNPRONOUNCEABLE ACRONYM, HAS ALWAYS BEEN UNDER THREAT. I GET REAL NERVOUS ABOUT NEW AGENCY PCORI, THESE OTHER PEOPLE TRYING TO FIGURE HOW TO DO THIS, BEING ATTACKED BY CONGRESS, ALWAYS BEING THREATENED. TOO COST ORIENTED. THAT IF THE REAL THREAT IS HOW WE DECREASE DIABETES, THAT STARTS IN THE IMMUNITY, NOT THE HOSPITAL. DECREASE DEE BEE TEASE, SOMEBODY HAS TO DO THAT RESEARCH. SO I WOULD LIKE TO SEE NIH TAKE BIGGER ROLE IN LOOKING AT THE RESEARCH QUESTIONS THAT ARE IMPORTANT. AND I DON'T THINK CDC WILL DO IT AND I DON'T THINK ANYBODY ELSE WILL DO WELL. I DON'T KNOW THAT NIH WILL BE WILLING TO DO THAT BECAUSE THEY MIGHT BE CARVED FROM THE 60% WHICH SHOULDN'T HAPPEN. I WOULD LIKE TO SEE THE FEDERAL GOVERNMENT GET TOGETHER WITH THIS RESEARCHERS AND INSTEAD OF FIGHTING WITH EACH OTHER FOR THE CHROMOSOME, THE TAKE, COME UP WITH A RESEARCH AGENDA THAT COULD HELP CRAFT OUR SOLUTIONS TO THE DISEASES OF THE FUTURE. NOT JUST BEHAVIOR BUT IT'S NOT -- IT'S ALSO PHYSICIAN BEHAVIOR. IT'S HEALTHCARE PROVIDER BEHAVIOR. THERE'S A LOT OF FAT PEOPLE RUNNING AROUND GETTING DIABETES, IS IT JUST OKAY TO SAY PEOPLE ARE GETTING FAT OR LET PEOPLE BLAME IT ON THE SODA COMPANIES. OR SAY WE CAN'T DO ANYTHING ABOUT THIS. WE CAN'T DO ANYTHING ABOUT THIS AT ALL. AND WE'RE HAVE TO LET IT HAPPEN. THERE'S REALLY GOOD THAT IS THAT GOING -- WHERE IS THAT GOING TO GO? IS IT GOING TO SLIT AND LET PEOPLE SELL DRUGS INSTEAD EITHER OTHER QUESTIONS THAT SHOULD BE ANSWERED AND WHO -- THE ONLY PEOPLE -- THE ONLY AGENCY THAT HAS CREDIBILITY IS NIH. HIPPODOESN'T TAKE A BROADER ROLE IN TRYING TO SOLVE THE HEALTHCARE PROBLEMS OTHER THAN JUST REDUCING COSTS, WHICH IS A DIFFERENT ISSUE. I DON'T KNOW WHO WILL. >> I WOULD LIKE TO PICK UP WHAT HE JUST SAID, I DON'T DISAGREE WITH YOU BUT AT THE SAME TIME COMMUNITY RESEARCH DOING THE RESEARCH YOU'RE TALKING ABOUT IS VERY EXPENSIVE, SOME WAYS LIKE DOING PEOPLE ON DRUG DEVELOPMENT AND WE RECALL ALWAYS CAUGHTING NIH DONE GET TOO INVOLVED IN DEVELOPMENT BECAUSE NOBODY WILL FUND DISCOVERY PIECE LIKE HIPPOWILL FUND. SO COULD IT BE IF IN FACT WE EXPECTED NIH WILL BEGIN TO PLAY THIS ROLE AND AI AGREE WITH YOU IT'S LIKE THE QUESTION OF WHERE DOES CONGRESS PUT THAT $1.2 BILLION INCREASE IN BIODEFENSE RESEARCH, WE PUT IT IN THE DEPARTMENT OF DEFENSE AND PEGGY HAMBERG AND MYSELF TESTIFY ADAMANTLY IT HAS TO GO TO NIH, THEY'LL DO IT THE BEST, LEVERAGE IT SYNERGISTIC, YOU CAN MAKE THE SAME ARGUMENTS HERE. BUT SO COULD IT BE THAT MAYBE WE NEED THE LOOK AT OTHER AGENCIES THE NEW GROUP AND MAYBE THERE'S REDUCTIONS IN THOSE OTHER AGENCIES THAT SHOULD BE APPLIED TO DOING THIS TYPE OF RESEARCH. NIH TO DO RESEARCH OR COLLABORATION MUCH LIKE WHEN THE DEPARTMENT OF DEFENSE GOT THE INCREASE BREAST CANCER CLINICAL RESEARCH, THEY'RE WISE ENOUGH TO KNOW THEY COULDN'T DO IT AND DO IT WELL SO WHEY WHAT DO THEY DO? THEY CAME TO NIH AND ASKED HIM TO HELP THEM DO IT. SO MAYBE THAT'S WHAT WE NEED RIGHT NOW. BECAUSE I WORRY LIKE YOU DO, ABOUT NEW AGENCY. >> LET ME ONE THING, THAT IS TO SAY I THINK THE -- THE NIH CAN SCREW IT UP TOO. THEY CAN DO A COHORT STUDY IT WILL COST $700 BILLION OVER TEN YEARS WITH EVERY GENETIC TEST KNOWN TO MAN BUILT INTO IT AND SEQUENCING THE GENOME THE WHOLE c=55U9%Qe TRIAL. THAT IS WHAT I'M TALKING ABOUT. I'M TRYING TO SAY WHAT ARE THE -- I MEAN, THE COUNTRY LARGELY DRIVEN [LAUGHTER] >> I AGREE WITH YOU BUT IF WE'RE SITTING AN BEMOANING THE INCREASING HEALTHCARE COSTS, ARE THERE ANY RESEARCH QUESTIONS THAT SHOULD BE ASKED THAT CAN HELP REDUCE THOSE COSTS OTHER THAN HAVING THE ECONOMISTS DO IT? PANEL NUMBER THREE IS GOING TO DO THIS UNLESS NIH HAS SOME ROLE. PANEL NUMBER 3 IS GOING TO SAY DOCTORS MAKE TOO MUCH MONEY, FRAUD, ABUSE, BLAH, BLAH, BLAH. REFORM THE SYSTEM. THAT'S NOT REALLY RESEARCH BUT EITHER RESEARCH QUESTIONS SOMEBODY SHOULD BE ASKING THAT CAN HELP CHANG PROFILE OF OUR CHRONIC DISEASE EPIDEMIC, I DON'T THINK THEY'RE BASIC SCIENCE QUESTIONS. >> I AGREE. >> NECESSARILY BEHAVIORAL, I THINK WHERE ARE WE WITH THIS? >> LEARNING HEALTHCARE SYSTEM REPORT FROM IOM WOULD BE A GOOD PLACE TO START. I NEED TO LOOK AT MAYBE SOME SPECIFIC AREAS. >> THE DISCUSSION IS SO INTERESTING AND I'M A GREAT ADMIRER OF ELIAS AND SO ON BUT WHEN FRANCIS PUT FORWARD THE ROLE OF N CATS, THE NATIONAL CENTER FOR ADVANCING TRANSLATIONAL RESEARCH FOR $50 MILLION, THERE WAS AN OUTCRY THE WHOLE COUNTRY INCLUDING CONGRESS AND ONE THING OR ANOTHER. ARGUING THIS ISN'T THE MISSION OF THE NIH. AND I THINK WHAT'S JUST THROUGH FRANCIS' ABSOLUTE COMMITMENT TO IT AND SKILL AND SO FORTH, BUT THIS IS A TOPIC WE'RE TALKING ABOUT. IT WASN'T EMBRACED BY EVERYBODY, THIS IS A GREAT IDEA, IN FACT HE GAVE UP CREDIBILITY TO GET IT DONE. IT MAY PAY OFF LONG RUN BUT THE POINT I'M TRYING TO MAKE IS SOMETIMES WHAT SEEMS OBVIOUS TO A LOT OF PEOPLE IS NOT OBVIOUS TO OTHER PEOPLE. THIS DISCUSSION WE HAVE NOW WE ARE ALL ON BOARD, SEEMS TO BE THERE ARE A LOT OF PEOPLE WHO BELIEVE THIS IS AN NIH MISSION. AND THAT EVEN $50 MILLION WAS MORE MONEY IN GRANTS AND OTHER INSTITUTE AND SO ON IS NOT WHAT WE SHOULD BE ABOUT. SO THERE IS A SALIENT JOB WHICH IS ONE OF THE POINTS WE MADE EARLIER, THERE WAS A PUBLIC OR CONGRESS OR THE PEOPLE WHO SUPPORT THE NIH OR SO ON. ABOUT HOW NIH SHOULD POSITION ITSELF TO DO IT TO CARRY OUT ITS MISSION. I DON'T THINK WE'RE THERE YET THOUGH THIS MAY ENERGIZE THE DISCUSSION. >> ALAN, WOULD YOU LIKE TO RESPOND TO JOE'S COMMENT? >> GREAT RAIL TO CERTAIN LARGE COHORT OF STUDIES. >> THEN THERE WAS -- HE WAS A FRIEND FOR 30 YEARS. >> GETS TO SOME OF THE POINTS, INVERSE PARTS OF THE T -- CONTINUUM, THERE ARE CERTAIN KINDS OF STUDIES WHICH WILL BE RELATIVELY EXPENSIVE STILL WORTH DOING, AND I ARGUE NIH AND RESEARCH COMMUNITY, THAT'S IMPORTANT. ONE THIS CAN WE THINK ABOUT WHEN WE THINK ABOUT THE DIFFERENT ENTITIES WHETHER THEY BE FEDERAL OR OTHERS THAT SUPPORT RESEARCH, IT'S MORE THAN ENTITY, THE RESEARCH COMMUNITY BRING ALONG WITH THEM AND WE TEND TO HAVE DIFFERENT COMMUNITIES FOR INTEREST IN HISTORICAL, SOCIOLOGICAL OR OTHER REASONS THERE'S RESEARCH I'M OBVIOUSLY BIASED BUT I THINK NIH IS UNIQUELY ON POSITION TO DO BECAUSE OF THOSE COMMUNITIES OTHER KINDS OF THINGS, INCLUDING CERTAIN LARGE STUDIES. >> JUST WANTED TO MAKE A POINT. I THOUGHT THE WHOLE SESSION TODAY WAS DEVELOPING A WAY TO ASSESS VALUE. NOT WHAT THE VALUE SHOULD BE. OR WHAT VALUE WE HAVE TOO MUCH OF, NOT ENOUGH OF. THE MISSION VALUE OF FEDERALLY FUNDED BIOMEDICAL ENTER-- DEVELOPING MEDICAL INTERVENTIONS OF TREATMENTS CH T-1 TO 4 BECAUSE THAT'S RELEVANT PART. THERE'S ANOTHER T-0 HAS A DIFFERENCE CONNOTATION OF VALUE, I ONLYE '‡ONDERING WHAT EXTENT THE FIRST THING WE NEED TO DO, YOU CAN MANAGE SOMETHING YOU MEASURE, IT BECOMES OPINION DRIVEN YES, WE CAN CHANGE THE BEHAVIOR I REMEMBER SENATOR CLINTON ASKING THAT QUESTION AND IF YOU DO COST AND YOU COST OUT THE INTERVENTIONS, YOU TALK $110 BILLION. WE DID THE MODELING HOW YOU WOULD INTERVENE WITH THE HEALTHCARE SYSTEM, IT'S VALUABLE BUT NOT AFFORDABLE. THE QUESTION TO ME IS THE QUESTION IN THIS SMRB SAY LOOK, WE DONE KNOW THE REAL ANSWER BUT HERE IS THE FRAMEWORK OF ASSESSMENT. SHOW ME MY PROPOSAL I DIDN'T REALIZE YOU HAVE THE SAME, YOU REALLY CANNOT DO IT ONE SIZE FITS ALL. YOU HAVE CONCERN SET OF VALUE WHEN IT COMES TO KNOWLEDGE GENERATION AN DISCOUNTRY, WHEN IT COMES TO THAT FIRST BUT THE CONSENSUS WE SHOULD HAVE AS COMMUNITY IS NOT DO WE -- WE'RE DOING ENOUGH OR NOT BUT WHAT WOULD BE THE ASSESSMENT FRAMEWORK, MANAGEMENT REVIEW BOARD WOULD LIKE NIH TO EVENTUALLY ADOPT AND CONSIDER. AT LEAST THAT'S THE WAY I UNDERSTOOD THE REQUEST IN THE PANEL. >> ONE OTHER THING THAT STRUCK ME SO MUCH TODAY IS THE IMPORTANCE OF CHOOSING BENCHMARKS. I THINK EACH OF US UNDERSTOOD THE QUESTION MAYBE DIFFERENTLY. I UNDERSTAND IT WHAT IS THE VALUE. AND OVER THE DAY IT'S BECOME WHAT COULD BE THE VALUE. I THINK THAT'S A GREAT QUESTION, DEFINITELY ONE WORTH ASKING. IT'S A DIFFERENT QUESTION IT'S IMPORTANT TO BE CLEAR ABOUT THAT. I LOVE THE IDEA OF LOOKING AT WHAT COULD WE BE DOING, HOW DO WE TAKE THE CURRENT TRENDS AND DRIVE THE RESEARCH PROGRAM. THAT'S WONDERFUL BUT THAT'S WHAT -- PERHAPS BECAUSE IN MY OWN WORLD I'M CAUGHT UP IN THIS ARGUMENT WHAT IS THE VALUE OF THE PUBLIC RESEARCH UNIVERSE IN THE STATE OF NORTH CAROLINA. AND POLITICALLY IN WASHINGTON, WHAT IS THE VALUE OF THE RESEARCH PROGRAM TO THE HEALTH AND ECONOMY OF THE UNITED STATES. I WOULD POINT THAT OUT, THROUGH MY PRESENTATION TOO, THE ANSWER IS DEPENDENT ON HOW ONE SETS BENCHMARKS AN POINTS OF COMPARE SOON. -- COMPARISON. >> CAN I JUST SAY RIGHT QUICK, I'M SO GLAD TO HEAR YOU SAY THAT BARBARA, BECAUSE IT'S TAKEN A YEAR TO GET WHERE WE ARE BECAUSE WE KEEP GOING BACK AND P FORTH ABOUT THIS. >> SO I HAVE BEEN AGNOSTIC THROUGHOUT THE DAY AS SOMEBODY WHETHER PARACHUTED INTO THIS, FOR FRANCIS. THE LAST INTERCHANGE I THINK BEARS -- EVERYBODY WANTS TO DRIVE WHAT RESEARCH NIH FUNDS. IT IS A PAR ALREADY SPORT, THIS IS WHAT EVERYBODY WANTS TO DO. IT MAY IN FACT BE A APPROPRIATE QUESTION FOR THIS BOARD THE TAKE UP. THAT'S NOT WHAT YOU'RE TAKING UP I THINK THE EXERCISE HOW DOES ONE DETERMINE THE VALUE OF RESEARCH THAT WE ARE SUPPORTING. SO SUFFICIENTLY COMPLEX AND CHALLENGING THAT IF WE COULD GET THE BOARD'S BEST ADVICE ON THAT, THAT WOULD HELP US ENORMOUSLY AND IN DIFFERENT WAYS OFFER APPROACHES BUT YOU'RE RIGHT. WE HAVE BEEN HAVING PARALLEL CONVERSATIONS THROUGHOUT THE DAY, BECAUSE EVERYBODY'S INSTINCT DR. TONY FAUCI, EVERYBODY WANTS AN INSTITUTE, THIS PHRASE. SO EVERYBODY WANTS TO TELL NIH FUNDED. OKAY FINE. BUT IF WE COULD FOCUS BACK ON THE APPROACHES TO ASAYSING THE VALUE THAT WOULD BE SUCH AN ENORMOUS HELP BECAUSE DESPITE LOTS OF EFFORT, LOTS OF ENERGY, MANY, MANY PEOPLE THINKING ABOUT IT THE ONLY COMMONALTY IS IN ITS HEART. SO JUST THAT PLEA. IN FUTURE MEETINGS IF YOU WANT TO GO FROM THE OTHER THINGS, THAT'S BETWEEN YOU AND FRANCIS. >> RESPOND TO THAT. >> I'M NOT GOING TO RESPOND TO THAT REQUEST BUT A QUESTION I HAVE BEEN DYING TO ASK, THIS IS A PERFECT INTRODUCTION FOR IT. ALSO I WILL SAY LARRY, YOUR POINT ASSESSING THE VALUE OF PAST RESERVE IS A OBVIOUSLY VERY DIFFERENCE TASK FOR PROJECTING THE VALUE WHERE TO SPEND OUR MONEY. ELIAS, I WANT TO ASK YOU SOME YEARS AGO I REMEMBER YOU MAKING A PREGNANT TO THE BOARD OF TRUSTEES IN JOHNS HOPKINSND I WAS THINKING BACK ON THAT. TODAY AS YOU VISIT PEOPLE OF THE HILL AND RESEARCH IN THIS COUNTRY MANY ARE LOOKING FOR SIMPLE FORMULA WHERE IF YOU INVEST X IN RESEARCH YOU GET OUT Y, WHAT'S THE FORMULA? AND MY GUESS IS WE'RE NOT GOING TO FIND THAT IN MY LIFETIME. I PLAN TO LIVE TO BE 1 # 5 SO I PLAN TO BE AROUND A WHILE. BUT IF YOU -- IF THAT TALK YOU GAVE YOU USE CASE HISTORY FOR EXAMPLE, HEART RESEARCH WE INVEST IN LUNG RESEARCH, INVEST IN AND YOU SHOWED RESULTS, MORTALITY AND THE VARIOUS AREAS I THOUGHT IT WAS COMPELLING, MY QUESTION TO YOU IS ASSUME YOU GAVE THAT TALK IN RASHIUS PLACES, WHAT KIND OF REACTION DID YOU GET? DOES THAT PROVE USEFUL AND CONVINCING? >> I HAVE TO SAY FOR THOSE OF YOU, I KNOW I HAD SOME COLLEAGUES WHO WE USED TO GO DOWN TO HILL. FRANKLY I WAS BARRAGED BY THE CONGRESS STAFF AND CONGRESS AND SENATORS ABOUT WE DOUBLED YOUR BUDGET, WORE IN A+sx DEFICIT, WHY ARE WE DOING THIS, WHAT'S THE VALUE OF THIS. I BASICALLY TURNED AROUND AND NIH WAS -- I WOULD SAY THEY DIDN'T WANT THE ADDRESS THAT AT ALL BECAUSE THERE WAS THE SENSE OF DISCOVERY, PRODUCT IS KNOWLEDGE, KNOWLEDGE IS NOT MEASURABLE, BEING MEASURED IN CENTURIES AND SO ON. I HAVE TO SAY CHANGE THE CONVERSATION WITH CONGRESS. I HAVE -- I REMEMBER GOING THERE AND HAVING A NOT ONLY -- THE FIRST TIME I TESTIFIED TO THIS WAS AT A TIME THEY WANTED TO STOP DOUBLING OF THE NIH BUDGET. THERE WAS TIME THEY HAD FUNDED BY THE FEDS THROW THE DOUBLING. AND AND OMB WAS TO SLOW DOWN DOUBLING OF NIH BUDGET. FOR ME DIRECTOR, IT WAS LIKE NOT SURVIVABLE IF I WAS DIRECTOR THAT STOPD THE DOUBLING. IT WOULDN'T HAVE BEEN VERY GOOD. SO WE SCRAMBLED AND WE FOUND THE INFORMATION, HOW MUCH DID YOU INVEST? AND WE WANTED TO REALLY CREATE THE SENSE OF RELATIVE BENEFIT INVESTMENT BENEFIT. THAT YOU CAN'T DO UNLESS YOU CRYSTALLIZE IT TO THE DOLLARS PER YEAR PER AMERICAN. SO WE USE THAT. AND WHEN WE DID IT WE FOUND IT WAS $4 PER CAREER PER AMERICAN FOR HEART DISEASE RESEARCH, $9 FOR CANCER, $2 FOR ALZHEIMER'S DISEASE AND THEN I HAD A PATIENT WHO CAME. IF YOU ALL REMEMBER, WE ASKED THE PATIENT SAID I'M VOLUNTEERING, I WANT TO COME AND NEUROSCIENCE RESEARCH WAS $1.20 PER YEAR. IF REQUEST LOOK AT BUDGETS OVER 30 YEARS. SO WE HAD THIS PATIENT WHO WAS THERE AND WE SHOW THE VIDEO OF HIM BEFORE HAVING THE DEEP BRAIN STIMULATION, IT WAS A PARKINSON'S PATIENT AND WE SHOWED AFTER PLAYING WILL CROSS WITH CHILDREN. I HAVE TO TELL YOU, NORM, THE COMMITTEE STOOD UP, SOME CRYING AND APPLAUDED THE PATIENTS. AND FRANKLY THERE'S NOTHING THAT WILL TOUCH CONGRESS MORE THAN THE DIRECT CONNECTION BETWEEN WHAT IS IT WE'RE TALKING ABOUT? WE'RE TALKING ABOUT $100 TOTAL. THAT'S $90 TOTAL PER AMERICAN PER YEAR IS THE AMERICAN -- IS THE NIH INVESTMENT. YOU HAVE TO RELATE IT WITH THE INVESTMENT IS BECAUSE IF YOU JUST KEEP IT THEORETICAL ABOUT HAVE YOU REDUCED THE COST OF DIABETES OR HAVE YOU REDUCED -- YOU'RE NOT DOING YOUR JOB, THE REAL PROBLEM WE HAVE IS HEALTHCARE COSTS TOO MUCH. AND SOMEHOW PEOPLE CONFLATE THAT $90 PER YEAR PER AMERICAN INVESTING AS THE COMPLETE SOLUTION TO EVERY PROBLEM AN WHY IS IT WE'RE PAYING TWICE AS MUCH AS THE GERMANS. WHICH IS A ISSUE. IN HEALTHCARE. IT RELATES TO YOUR PANEL 3, COMPLETELY AGREE WITH YOU. SO I THINK WE NEED TO PUT THAT RELATIVE PERSPECTIVE AND IT'S EXTREMELY POWERFUL, I HAVE TO SAY FOR THOSE OF YOU WHO ARE THERE, THAT WAS A TURNING DAY, IT WAS A TURNING MOMENT. THE BUDGET COMMITTEE MET THAT AFTERNOON AND DECLINED TO RESPOND TO THE OMB REQUEST TO REMOVE -- IT WAS LIKE 50% INCREASE WE REQUESTED, THIS SHOULD BE 8%, JUST LEFT THAT PROVIDE OFFENSE. I COMPLETELY AGREE IT'S NOT ONLY TO COME UP WITH WITH ASSESSMENT, BUT IT'S YOU HAVE TO COME UP WITH A COMPELLING THAT RELATES, WHAT IS IT YOU EAR PUTTING OPT TABLE, GENTLEMEN? WHAT IS WHAT IT IS WHAT NIH IS, WHAT IS THAT INVESTMENT? THAT'S NOT TALKING THEORETICAL TERMS IN BILLIONS BECAUSE I CAN SHOW YOU'RE BILLIONS THAT DON'T HAVE THAT RETURN THAT WE'RE SPENDING. SO I THINK THE $4 PER AMERICAN PER YEAR FOR MORE IT WILLTIER, MORBIDITY, OF HEART DISEASE AND STROKE WHICH IS BASICALLY A MILLION LIVES A YEAR, ECONOMIC VALUE THAT IS 2 1/2 TRILLION DOLLARS, I COUNTER ARGUE THAT, IF YOU LOOK AT HIV AIDS FRANKLY TURNING IDEAS YOU LOOK AT 22 DRUGS IN A SHORT TIME. HOSPITAL BEFORE SO WE TESTIFIED TO THAT. I THINK THAT'S AGREE TO PROVIDE BUT IT HAS TO BE FOR DIFFERENT CATEGORIES OF FUNDING. -- >> EXCUSE ME. I NEED TO LEAVE SO I WANT TO THANK YOU ALL AGAIN FOR THIS OPPORTUNITY.2bo >> I WOULD LIKE TO ASK IF THERE ARE ANY OTHER QUESTIONS, COMMENTS FROM MEMBERS OF THE BOARD. THOSE THAT LISTENED SO PATIENTLY AROUND THE ROOM. AND NORM, COULD YOU TALK TO US ABOUT NEXT STEPS AND ADJOURN? >> I'LL DO THAT. THANK YOU TO OUR PANELISTS. I THINK ELIAS, YOU OR AWARE OF -- BEFORE YOU LEAVE, YOU CAN'T GET OUT THE WISCONSIN ENTRANCE AT THIS POINT O SO GO OUT THE OLD GEORGETOWN ROAD. THE PRESIDENT WEAPON THE BILLING ACROSS THE STREET. (OFF MIC) [LAUGHTER] Q. THANK YOU. ELIAS, THANK YOU. GREAT TO SEW YOU. GOOD LUCK. [APPLAUSE] >> IT'S BEEN A TERRIFIC DAY. WE%q– CERTAINLY YOUR GROUP GETS OUR THANKS. THE OBR GROUP WILL BE MEETING AGAIN TOMORROW, I LOOK FORWARD TO THAT, IT WILL BE HARD TO TALK TODAY BUT SURE IT WILL OR WE CAN POTENTIALLY -- WE'RE GOING TO BE HAVING SOME TELECONFERENCES OF THE COMMITTEE, I UNDERSTAND GILL AND THE STAFF HERE IS WORKING TO SET UP TIMES DURING THE SUMMER. WE CAN DO THAT. WE'LL HAVE MEETING IN THE FALL, THE ENTIRE COMMITTEE AND MAYBE I'LL ASK YOUEN YOUR FOLKS PUT OUT A COUPLE OF WARNINGS VERY EARLY SO PEOPLE KNOW ABOUT IT AND WE CAN GET IT ON EVERYBODY'S CALENDAR IF IT'S NOT THERE ALREADY. THE NEW CHARGE LARRY GAVE US THIS MORNING AMY AND I WILL BE WORKING TO COME WITH MEMBERS WHO HAVE SKILL SETS THAT MATCHES THAT TASK WHICH PROBABLY IS ALMOST EVERYBODY IN THIS GROUP AND ALSO BALANCE THE WORKLOAD OF OUR MEMBERSHIP BUT IF THERE ARE PEOPLE HERE WHO HAVE A PARTICULAR INTEREST, I WOULD ASK -- YOU PULL THOSE WHO ARE NOT HERE IF THEY WOULD LIKE TO VOLUNTEER TO SERVE ON THAT GROUP OR IF WE COULD KNOW THAT, IF YOU'LL TELL AMY THAT YOU'LL MAKE SURE YOU'RE ON THE THE GROUP. I THINK THAT'S THE EXTENT OF ADMINISTRATIVE ITEMS TO COVER. I ALWAYS LIKE AS PART OF OUR TRADITION TO GO AROUND THE TABLE AT THE END OF THE MEETING AND GIVER A CHANCE TO SAY ANYTHING ON YOUR MIND THAT WE MISSED, SOMETHING YOU WOULD LIKE TO THE SAY, SOMETHING YOU'RE NOT HAPPY ABOUT, WHATEVER. SO LET'S GO AROUND, WE'LL START OUT WITH YOU, DO YOU WANT MAKE COMMENTS AT ALL IN >> I CERTAINLY FOUND ELIAS' COMMENTS THIS EVENING VERY PROVOCATIVE IN THINKING ABOUT METRIC AN MEASURES THAT ARE SPECIFIC TO TRANSLATIONAL STAGES, I THINK WOULD BE HELPFUL. >> BILL GAIL. >> I WOULD LIKE TO THANK AMY, YOU AND YOUR ENTIRE STAFF, THIS HAS BEEN A CAN FEEL MEETING WITH THE PHONE CALLS AN MATERIALS RESPONDED SO EFFICIENTLY AN PROFESSIONALLY, I DO APPRECIATE THEM SO THANKS VERY MUCH. >> I THOUGHT IT WAS TERRIFIC. I CONGRATULATE AMY AN GAIL AND SO ON. I COME AWAY WITH THE CONCERN AND HE WILL WHEREAS'S EXCELLENT PRESENTATION HIGHLIGHTED IS WE DO FOCUS A LOT ON WHAT WE REALLY BELIEVE IS ACADEMIC UNDERPINNING OF WHATEVER QUESTION WE ASKED SO THE FIRST THROW PRESENTATIONS, TREMENDOUSLY GOOD IN TERMS OF STATISTICS AND ANALYSIS AND SO ON. AND POINT OF VIEW MANY TIMES BUT IT'S REALLY IMPORTANT, SO FORTH. BUT I JUST HOPE OUR REPORT CAN ALSO TAKE INTO ACCOUNT A LOT OF WHAT ELIAS SAID, THERE IS A WAY OF PRESENTING THINGS TO THE PUBLIC AND CONGRESS WHERE IF WE DO MUCH ACADEMIC HOW CONVINCING IT IS TOTALLY MAKES NO DIFFERENCE TO THEM AND THEY'LL LOOK AT THIS REPORT LIKE MANY OTHER REPORTS WITH IOM AND OTHERS AN WILL HAVE ABSOLUTELY NO IMPACT THOUGH THEY'RE GREAT BUT THEY SIT ON THE SHELF AND NO ONE PAYS ATTENTION. SO I THINK OUR CHALLENGE REALLY, I CERTAINLY DON'T KNOW THE ANSWER BUT I FEEL STRONGLY ABOUT IT, HOW WE COMBINE THOSE TWO AND PRESENT REAL GOOD DATA AND ANALYSIS BUT DONE MISTHE POINT THAT SOME OF THIS IS PUBLIC CONSUMPTION IN A WAY THAT MAYBE MORE ANECDOTAL OR MORE CASE BASED SIMPLIFIED IN TERMS OF A MESSAGE. IF WE CAN BALANCE THE REPORT THAT WAY WE MAKE PROGRESS. >> JUST A FEW COMMENTS ON THE DAY BEING ON THE SUBCOMITTEE, ONE OF HIS POEMS HE SAID I MET NEW -- I AM AT A NEW AWARENESS OF CONFUSION, THAT'S WHERE I AM RIGHT NOW BUT WHAT WAS INTERESTING TO ME DURING THE DAY IS THAT NO ONE ADDRESSED THE QUESTIONS, THEY JUST REALLY DECIDED TO3m DO WHAT THEY WANTED TO SAY. EXCEPT ELIAS, ONE OTHER POINT, I HAVE TO THINK ABOUT THIS, THIS EVENING, IS VALUE AND HOW DO YOU DEFINE VALUE IN ITSELF. THAT WAS ANOTHER POINT TO CONSIDER OVER TIME AS WELL. SO REALLY, THANK YOU VERY MUCH FOR ALL THE ORGANIZATION TODAY. IT WAS WONDERFUL. >> WE'LL GIVE YOU THE LAST WORD. >> THANK YOU, EVERYBODY, HAVE A GREAT EVENING. [LAUGHTER] PARTICULAR THANKS TO AMY & STAFF FOR ALL THE WORK. THIS WAS A PARTICULARLY CHALLENGING -- [APPLAUSE]