I'M BILL RILEY, ASSOCIATE DIRECTOR FOR BEHAVIORAL AND SOCIAL SCIENCES RESEARCH AT NIH, I WANT TO WELCOME YOU TO THIS WEBINAR ON NIH FUNDED BASIC EXPERIMENTAL STUDIES WITH HUMANS OR BESH, REGISTRATION AND RESULTS REPORTING. THANK YOU ALL FOR ATTENDING TODAY. JUST A REMINDER THAT THE WEBINAR WILL BE RECORDED AND ARCHIVED AND THAT THE SLIDES WILL BE MADE AVAILABLE ON THE NATIONAL LIBRARY OF MEDICINE WEBSITE, AND YOU CAN SUBMIT YOUR QUESTIONS AS YOU CAN SEE ON THE SCREEN BY E-MAILING SCIENCE POLICY@OD.NIH. GOV AND YOU CAN E-MAIL THOSE ALL ALONG OVER THE COURSE OF THE HOUR AND WE'LL TRY TO GET TO AS MANY OF THOSE AT THE END AS POSSIBLE. SO IN PREPARATION FOR OUR NATIONAL LIBRARY OF MEDICINE COLLEAGUES PRESENTING ON THEIR WORK, WE THOUGHT WE WOULD DO A BRIEF OVERVIEW OF BASIC EXPERIMENTAL STUDIES INVOLVING HUMANS AND HOW WE GOT TO WHERE WE ARE. SO MANY OF YOU KNOW THE RATIONALE FOR OUR CLINICAL TRIALS POLICIES. OUR COMMON GOAL ACROSS I THINK ALL THE SCIENCES IS ENHANCING RESEARCH TRANSPARENCY BOTH REDUCING PUBLICATION BIAS AND INCREASING THE REPORTING OF FINDINGS IN A TIMELY MANNER. MOST OF YOU HAVE SEEN DATA SUCH AS WHAT I HAVE ON THE RIGHT, THIS IS THE ROSS STUDY WHICH IS HIGHLY CITED IN CLINICAL TRIALS WITH THE PRIMARY OUTCOME FINDING THAT NEARLY HALF DO NOT PUBLISH THEIR PRIMARY OUTCOME FINDINGS IN A TILELY MANNER, AND THAT PROBLEM EX--- TIMELY MANNER, AND THAT PROBLEM EXTENDS BEYOND CLINICAL TRIALS. ANOTHER STUDY SHOWED THAT 39 PERCENT OF OBSERVATIONAL STUDIES FAIL TO PUBLISH. THEN A STUDY WE RECENTLY DID OF ALL RESEARCH, HUMAN, CELL BIOLOGY, GENETICS, ANIMAL RESEARCH, ALL RO1'S AND UO1'S FUNDED BY NIH FROM 2008 TO 2014 WITH ABOUT THE ENTIRE GRANT PROJECT PERIOD TO PUBLISH ANYTHING, NOT JUST THE PRIMARY RESULTS, 2.4 PERCENT OF THOSE RO 1'S AND UO1'S FAILED TO HAVE A SINGLE PUBLICATION ASSOCIATED WITH THE GRANT AFTER 60 MONTHS AND THAT WAS JUST RECENTLY PUBLISHED IN PLS1. SO IN ADDITION TO THOSE COMPLON GOALS ABOUT ENHANCING RESEARCH TRANSPARENCY THERE'S ALSO OBVIOUSLY THE COMMITMENT TO RESEARCH PARTICIPANTS AND TAXPAYERS AND I'LL POINT YOU TO A JOO REPORT FROM A FEW YEARS AGO -- GAO FROM 2006 SHOWING NIH NEEDS TO BE A BETTER JOB AND BE A BETTER STEWARD IN THESE CLINICAL TRIALS AND VARIOUS STUDIES WITH HUMANS THAT WE FUND MANY OF YOU KNOW THE VALUE OF SCIENTIFIC CENTRALIZED STRUCTURED REGISTRATION AND REPORTING, MINIMIZING PUBLICATION BIAS, IMPROVING ABILITY TO SYNTHESIZE STUDIES, A NUMBER OF META-ANALYSES HAVE COME FROM CLINICALTRIALS.GOV FOR INSTANCE, LIMITS ON P-HACKING AND HARKING, HYPOTHESES AFTER RESULTS ARE KNOWN, IDENTIFY RESEARCH GAPS AND REDUCING DUPLICATION, FACILITATING STUDY REPLICATION. AIDING STUDY RECRUITMENT AND ENCOURAGING COLLABORATION. WHAT WE'VE FOUND IS THAT AS YOU KNOW, BASIC STUDIES INVOLVING HUMANS ARE STOWJ THE NIH CLINICAL TRIALS POLICIES, THAT'S THE RESULT OF THE BREADTH BY WHICH INTERVENTIONS IS CONSIDERED AND THAT WOULD INCLUDE SORT OF EXPERIMENTAL MANIPULATIONS WITH HUMANS. BUT THERE ARE CLEAR DIFFERENCES BETWEEN BASIC AND APPLIED RESEARCH, AND THAT'S NECESSITATED ADDITIONAL CONSIDERATIONS FOR BASIC RESEARCH UNDER THESE POLICIES. ONE OF THE THINGS THAT WAS CHANGED IN THE LAST YEAR WAS THE INCLUSION OF OR ADDITION OF BESH PARENT GRANT APPLICATIONS, BASIC EXPERIMENTAL STUDIES INVOLVING HUMANS, THESE ARE HUMANS THAT MEET THE NIH DEFINITION OF A CLINICAL TRIAL BUT THEY ALSO MEET THE DEFINITION OF BASIC RESEARCH. SO ONE OF THE OTHER THINGS THAT HAS BEEN AFFECTED BY THIS IS THE DELAY IN ENFORCEMENT AND SHORT-TERM FLEXIBILITIES OF SOME OF THE REQUIREMENTS FOR THESE PROSPECTIVE BASIC SCIENCE STUDIES. ONE OF THE THINGS THAT WAS DONE BACK IN 2018 WAS TO LOOK BACK AND DECIDE THAT WE NEED TO GIVE A LITTLE BIT MORE FLECK FLEXIBILITY FOR REGISTRATION REPORTING OF THESE BASIC SCIENCE STUDIES OR BESH STUDIES TO ADD FOR ADDITIONAL FLEXIBILITY TO ALLOW REPORTING OF EXISTING BAISK SCIENCE RESEARCH -- BASIC SCIENCE RESEARCH. THAT STILL REQUIRES THAT BASIC SCIENCE OR BASIC BESH STUDIES STILL REGISTER AND STILL DO RESULTS REPORTING, BUT JUST FOR THIS FLEXIBILITY TIME, NOT NECESSARILY ON THE CLINICAL TRIALS.GOV WEBSITE. THE NIH PUT OUT AN RFI AND REPORTING STARNLDZ FOR PERSPECTIVE BASIC SCIENCE STUDIES INVOLVING PARTICIPANTS AND WE'VE ANALYZED THAT AND IDENTIFIED SPECIFIC CHALLENGES, AND THEN OUR COLLEAGES AT NLM WILL FLENT A MINUTE SOME OF THE WORK THEY'VE DONE SUBSEQUENT TO THAT TO BETTER UNDERSTAND THOSE CHALLENGES. AND THEN THAT EXTENSION OF FLEXIBILITIES HAS NOW BEEN EXTENDED INTO SEPTEMBER 24, 2021 , TO ALLOW US TO FURTHER UNDERSTAND AND FURTHER ADDRESS SOME OF THE BESH COMMUNITY ISSUES RELATED TO REGISTRATION AND RESULTS REPORTING. SO WITH THAT, I AM GOING TO CLOSE MY SHARING TO LET BECKY WILLIAMS AND ELLIS -- AND ELISA GOLFINOPOULOS AT THE NATIONAL LIBRARY OF MEDICINE AND BECK BECKY LEAD THE WORK THEY'VE DONE TO ANALYZE AND BETTER UNDERSTAND THE CHALLENGES OF BASIC STUDIES INVOLVE BEING HUMANS AND CLINICAL TRIALS REGISTRATION AND RESULTS REPORTING. BECKY? BECK BECK THANK YOU. >> THANK YOU SO MUCH, BILL. I'M REALLY HAPPY TO PRESENT THIS TOPIC WITH YOU ALL 20ED. THE ANALYSIS WE UNDERTOOK, AS BILL SAID, WAS REALLY IN THE CONTEXT OF TRYING TO BETTER UNDERSTAND MUCH OF THE INFORMATION THAT NIH HAD RECEIVED THROUGH THAT RFI EFFORT THAT HAD OCCURRED EARLIER. BUT BEFORE I DIVE INTO THE ANALYSIS, EFTS, I THINK IT'S -- ITSELF, I THINK IT'S IMPORTANT TO SET A LITTLE BIT OF CON TECH, JUST IN TERMS OF THE CONTEXT OF THE CLINICALTRIALS.GOV MODEL ITSELF, MAKE SURE WE ALL HAVE THE SAME SHARED UNDERSTANDING. AND IN GENERAL, THE MODEL FOR CLINICALTRIALS.GOV REALLY TRACES THE LIFE SPAN OF A CLINICAL TRIAL ITSELF. SO CLINICAL TRIAL INFORMATION IS REGISTERED AT CLINICALTRIALS.GOV WHEN THE STUDY IS INITIATED. THE INFORMATION IS KEPT UP TO DATE THROUGHOUT THE LIFE CYCLE OF THAT CLINICAL TRIAL. AND THEN FOLLOWING COMPLETION OF THE STUDY, RESULTS INFORMATION CAN BE SUBMITTED TO CLINICALTRIALS.GOV, AND YOU ALSO SEE CAPTURED HERE THAT THERE'S OTHER THINGS THAT OFTEN HAPPEN SORT OF EXTERNAL TO CLINICALTRIALS.GOV AS WELL. THIS MODEL REALLY SUPPORTS TRANSPARENCY BY DOCUMENTING THE PRESPECIFIED RESEARCH PLAN FOR THE CLINICAL TRIAL AND ALSO HELPS ENSURE ACCOUNTABILITY IN THE REPORTING OF RESEARCH RESULTS. BUT THERE'S A FEW KEY ASSUMPTION S THAT WENT INTO THE MODEL FROM SORT OF THE OUTSET OF CLINICALTRIALS.GOV AND THAT WAS BASICALLY THAT ONE PROTOCOL TYPICALLY RELATED TO ONE CLINICAL TRIAL, WHICH THEN RELATED TO ONE STUDY RECORD ON THE CLINICALTRIALS.GOV WEBSITE ITSELF, AND THAT THAT PROTOCOL DOCUMENT WAS REALLY THE DRIVER FOR THE REGISTRATION INFORMATION THAT WOULD BE SUBMITTED. AND THE PROTOCOL INCLUDES THE OVERALL RESEARCH PLAN WITH THE PRESPECIFIED APPROACH TO THAT TRIAL CONDUCT. AND THEN THE REGISTRATION INFORMATION ITSELF AGAIN IS BEING DRIVEN FROM THAT PROTOCOL DOCUMENT AS REALLY THE KEY ASPECTS THAT PEOPLE MAY NEED TO RND THE STUDY -- UNDERSTAND THE STUDY WHEN IT BEGINS. AND THEN OF COURSE FOLLOWING THAT STUDY RESULTS INFORMATION. THE STUDY RESULTS INFORMATION IN CLINICALTRIALS.GOV WAS REALLY DRIVEN BY SOME OF THE FOUNDATION AL REQUIREMENTS THAT WERE ESTABLISHED IN A LAW BACK IN 2007, THE AND RESULTS INFORMATION IS AGGREGATED SO IT'S SUMMARY LEVEL INFORMATION THAT APPEARS IN TABLES. WE DON'T HOST INDIVIDUAL PARTICIPANT LEVEL DATA, AND THE RESULTS TABLES DON'T SUPPORT FIGURES, IMAGES, OR CONCLUSIONS. SO FOR THOSE WHO AREN'T NECESSARILY FAMILIAR WITH WHAT THE RESULTS MODEL SPECIFICALLY LOOKS LIKE, WHAT YOU SEE HERE IS OVER IN A PUBLICATION, YOU WOULD OFTEN HAVE SOME COMBINATION OF NARRATIVE TEXT, MAYBE A FIGURE, MAYBE A TABLE REPRESENTING THE RESULTS, AND THEN WHAT YOU HAVE ON THE CLINICALTRIALS.GOV SIDE IS QUITE SIMILAR IN TERMS OF THE NARRATIVE DESCRIPTION OF THE OUT COME MEASURE ITSELF BUT THE RESULTS INFORMATION IS TRANSPOSED INTO JUST THIS BASIC TABULAR FORMAT THAT ALLOWS YOU TO SEE THE INFORMATION AS ORGANIZED BY EACH OF THE STUDY ARMS. AND THEN SECONDARILY, THE TABLES OF INFORMATION THAT WE COLLECT FOR RESULTS HAVE ALSO ALLOWED US TO REALLY ELUCIDATE SOME VERY SPECIFIC FRAMEWORK FOR UNDERSTANDING WHAT LEVEL OF DETAIL IS NECESSARY WHEN REPORTING RESULTS INFORMATION FOR AN OUTCOME MEASURE, AND WE'VE REPRESENTED THAT INFORMATION AS BASICALLY FOUR DIFFERENT LEVELS OF INFORMATION, YOU KNOW, STARTING BASICALLY WITH THE DOMAIN, YOU WANT TO KNOW WHAT YOU'RE STUDYING, FOLLOWED BY THEN THE SPECIFIC MEASUREMENT, WHAT TOOL ARE YOU GOING TO USE TO ASSESS THAT PARTICULAR DOMAIN, AND THEN ONCE YOU'RE USING THAT TOOL, THERE'S DIFFERENT WAYS IN WHICH YOU COULD AGGREGATE THE DATA TO SUPPORT THAT OUTCOME REPORTING. IT COULD BE SORT OF THE TIME TO EVENT, IT COULD BE A CHANGE FROM BASELINE OR JUST AN END VALUE, AND THEN OFTENTIMES YOU'RE FURTHER AGGRAVATING THAT INTO EITHER A CONTINUOUS OR CATEGORICAL TYPE MEASURE. AND THEN ALL ALONG, EACH PIECE OF AN OUTCOME MEASURE IS ALSO ASSOCIATED WITH A SPECIFIC TIME FRAME, AND SO THAT SORT OF IS GENERALLY REPRESENTED WITHIN THIS PARTICULAR FRAMEWORK. AND AS WE DISCUSS THESE ISSUES A LITTLE BIT FURTHER, IT HAS BECOME SORT OF CLEAR THAT THERE'S NOT NECESSARILY ALWAYS CLEAR BOUNDARIES BETWEEN THE DIFFERENT TYPES OF STUDIES THAT WE'RE TALKING ABOUT. YOU KNOW, YOU FIRST HAVE TO TAKE A LOOK AT THE DIFFERENCE BETWEEN AN INTERVENTIONAL STUDY AND AN OBSERVATIONAL STUDY, AND THEN WE'VE SEEN A LOT OF HETEROGENEITY BASICALLY LOOKING ACROSS DIFFERENT TYPES OF STUDIES IN TERMS OF THOSE THAT ALSO MEET SORT OF THE DEFINITION OF BESH AND THERE'S THIS RANGE OF SORT OF EXPLORATORY AND CONFIRMATORY TYPE STUDIES. AND WE'LL COME BACK TO THIS A LITTLE BIT LATER. BUT JUST IN TERMS OF CONTEXT FOR OUR ANALYSIS, AGAIN REFERRING BACK TO THAT RFI THAT WAS CONDUCTED EARLIER, THERE'S OVER ALL SORT OF GENERAL AGREEMENT ON THE GOALS AND THE IMPORTANCE THAT NIH WAS AIMING TO ACCOMPLISH, BUT IT BECAME VERY CLEAR THAT THERE WERE SPECIFIC TYPES OF STUDIES AND SPECIFIC RESEARCH DOMAINS THAT WERE PROVING TO BE PARTICULARLY CHALLENGING FOR RESEARCHERS. SO WHETHER WE FIRST LOOKED AT THAT ANL -- WHEN WE FIRST LOOKED AT THAT ANALYSIS WE KIND OF DOUG IN RIGHT AWAY AND LOOKED QUICKLY APT A SET OF PUBLICATIONS TO BETTER UNDERSTAND AND CHARACTERIZE THE IRB USE THAT RESEARCHERS WERE IDENTIFYING AND WE TALKED A LITTLE BUILT ABOUT THESE IN A BLOG POST THAT WAS ERB YOU HAD LAST YEAR. -- THAT WAS ISSUED LAST YEAR. BUT ONE LIMITATION OF THAT ANALYSIS IS THAT WE ARE RELYING ON THE PUBLISHED LITERATURE, AND AS YOU'VE SEEN WITH THE REPORTING MODEL FOR CLINICALTRIALS.GOV IT'S REALLY DEPENDENT ON THE INFORMATION IN THE PROTOCOL DOCUMENT TO DRIVE THAT REGISTRATION INFORMATION. SO WE DECIDED TO EXTEND OUR EARLIER ANALYSIS TO INCLUDE EVALUATION OF PROTOCOL DOCUMENTS AND TO ALSO TALK WITH RESEARCH ERS DIRECTLY. WE DIDN'T HAVE THAT OPPORTUNITY IN OUR PRELIMINARY ANALYSIS AS WELL. SO HERE IS A BRIEF DESCRIPTION OF OUR METHODS. WE LOOKED AT 12 CASE STUDIES AND 12 MAY SOUND BIG, MAY SOUND SMALL, DEPENDING ON YOUR POINT OF VIEW BURKT YOU'LL QUICKLY SEE -- BUT YO U'LL QUICKLY SEE HOW 12 IS ACTUALLY A MUCH LARGER NUMBER. WE TRIED TO INCORPORATE DIFFERENT DOMAINS OF RESEARCH. WE FERS PARTNERED WITH OUR -- FIRST PARTNERED WITH OUR INVESTIGATORS HERE AT THE NIH AND YOU CAN SEE THE RANGE OF INSTITUTES THAT WE WERE ABLE TO TALK WITH INVESTIGATORS FROM AND GET PROTOCOL DOCUMENTS. THEN ONCE WE HAD A PRETTY GOOD UNDERSTANDING OF THE INTRAMURAL SPACE, WE EXPANDED THAT LOOK OUT TO THE EXTRAMURAL SPACE, AGAIN MIRRORING THOSE DOMAINS YOU SAW BOTH IN THE RFI AND ALSO IN THE INTRAMURAL SAMPLE. ONCE WE STARTED LOOKING AT THE EXTRAMURAL STUDIES, WE FOUND MANY SIMILARITIES AND WERE NOT FINDING ANY NEW ISSUES. SO WE DIDN'T NECESSARILY THINK IT WAS NECESSARY TO GO MUCH FURTHER AND LOOK AT TOO MANY MORE EXAMPLES BECAUSE THEY WERE ALL SORT OF SELF-VALIDATING AND CONFIRMING. WE HAD INVESTIGATORS PROVIDE A LITTLE BIT OF INFORMATION IN A QUESTIONNAIRE FORMAT AND ASKED FOR MATERIALS THAT WOULD BE SUPPORTIVE OF BOTH REGISTRATION AND RESULTS REPORTING. STAFF HERE AT NLM SORT OF INDEPENDENTLY REVIEWED THOSE DOCUMENTS AND THEN TRIED TO THEY THEMSELVES LOOK AT HOW THEY MIGHT ACCOMMODATE THIS INFORMATION WITHIN CLINICALTRIALS.GOV, AND THEN WE'RE REALLY LUCKY, WE WOULD THE OPPORTUNITY TO DISCUSS WHAT WE FOUND DIRECTLY WITH THE PI'S OR THE INVESTIGATORS TO MAKE SURE THAT WE WERE FULLY UNDERSTANDING THEIR STUDIES CORRECTLY AND FOR THEM TO PROVIDE FURTHER FEEDBACK , AGAIN, ON OUR FINDINGS IN TERMS OF WHO WE INCLUDED, AGAIN IN THE SAMPLE, MOST OF THE INTRAMURAL INVESTIGATORS DID NOT HAVE ANY DIRECT EXPERIENCE WITH SUBMITTING INFORMATION TO CLINICALTRIALS.GOV. THE SUBMISSION SYSTEM IS CALLED THE PRS, WHICH YOU SEE ON THE SLIDE HERE, BECAUSE HERE WE HAVE A CENTRALIZED PROCESS FOR REGISTERING THOSE STUDIES AND NO ONE HAD EXPERIENCE WITH SUBMIT TING RESULTS INFORMATION. WHEN YOU LOOSK ON THE EXTRAMURAL SIDE, THERE'S ACTUALLY A LITTLE BIT MORE FAMILIARITY WITH THE REGISTRATION PROCESS AT CLINICALTRIALS.GOV AND NO ONE YET HAD SUBMITTED RESULTS VIA OUR SYSTEM EITHER. MANY OF THE INVESTIGATORS ON THE EXTRAMURAL SIDE USED OTHER PLATFORMS SUCH AS OSF AS WELL. HERE ON THE SLIDES I INCLUDE A SUMMARY OF THE 12 DIFFERENT CASE STUDIES THAT WERE INCLUDED IN OUR ANALYSIS. I'M NOT GOING TO READ THROUGH EACH AND EVERY ONE OF THESE, BUT I WILL JUST MENTION A LITTLE BIT LATER MY COLLEAGUE ELISA GOLFINO POULOS WILL GO THROUGH A COUPLE OF THESE TO BETTER ILLUSTRATE SOME OF THE CHALLENGES I WOULD LIKE TO TALK ABOUT NEXT. BUT AGAIN I HOPE YOU SEE THAT WE TRIED TO HIT A RANGE OF DOMAINS IN THIS SPACE. SO IN TERMS OF OUR OVERALL FINDINGS, WE ULTIMATELY VALIDATED THAT PRELIMINARY ANALYSIS, AND IT REALLY DID GEF US DEEPER INSIGHTS INTO THE -- DID GIVE US DEEPER INSIGHTS INTO THE REASONS FOR THE CHALLENGES, AND IT ALLOWED US TO BEGIN TO ARTICULATE SORT OF POINTS TO CONSIDER AS WE FURTHER EXPLORE REPORTING MODELS FOR BESH. IN TERMS OF THE SPECIFIC CHALLENGES, WE IDENTIFIED FIVE PRIMARY CHALLENGES THAT SEEM PARTICULARLY SPECIFIC TO BESH. FIRST AND FOREMOST IS THIS TOPIC OF MULTIPLE INTERRELATED SPERMINGTS, AND IF YOU RE-- EXPERIMENTS, AND IF YOU REFLECT BACK ON THE SLIDE I INCLUDED BEFORE THAT HAD TALKED ABOUT SOME OF THE UNDERLYING ASSUMPTIONS OF THE CLINICALTRIALS.GOV REPORTING MODEL, YOU CAN SEE HOW EACH OF THESE CHALLENGES ULTIMATELY ENDS UP MAPPING TO SOME OF THOSE UNDERLYING ASSUMPTIONS THAT DON'T NECESSARILY APPLY IN THE BESH SPEAS. THE FIRST THING I HAD MENTIONED WAS BASICALLY ONE PROTOCOL, ONE CLIRN -- ONE CLINICAL TRIAL, ONE STID DID I RECORD -- STUDY RECORD AND WHAT WE'RE SEEING IN THE CONTEXT OF BESH AND THERE ARE ACTUALLY MULTIPLE INTER RELATED EXPERIMENTS, AND SOMETIMES ONE EXPERIMENT ON ITS OWN MIGHT NOT BE MEANINGFUL, BUT IT'S REALLY ONLY IN THE CONTEXT OF OTHER EXPERIMENTS THAT THAT ONE EXPERIMENT BECOMES MEANINGFUL. WHEN LOOKING AT THE PROTOCOL DOCUMENTS, WE ALSO NOTICE THAT THE PROTOCOLS THEMSELVES TENDED NOT TO HAVE DETAILED PRE SPECIFIED PRIMARY OUTCOME MEASURES, SO AGAIN THINKING BACK TO THAT OUTCOME MEASURE REPORTING FRAMEWORK, NOT NECESSARILY THAT YOU WOULD NEED THAT LEVEL FOR REPORTING, BUT AT LEAST A LEVEL TWO OF REPORTING TO BE ABLE TO PRESPECIFY THAT OUTCOME MEASURE. AND WE FOUND THAT MANY PROTOCOL DOCUMENTS WERE ACTUALLY MORE INTENDED TO BE SUPPORTIVE OF OTHER ASPECTS OF REVIEW AS WELL, SUCH AS IRB REVIEW WHERE THEY MIGHT BE PROVIDING INFORMATION ON SORT OF THE HIGH LEVEL RESEARCH AIMS COMBINED WITH THE RANGE OF MEASURES AND EXPERIMENTS THAT MIGHT BE CONDUCTED BUT NOT NECESSARILY ORGANIZED SPECIFICALLY AROUND THE UNIT OF A SINGLE CLINICAL TRIAL. THEN WHEN WE STARTED LOOKING AT SOME OF THE RESULTS DATA, WE ALSO SAW THAT PUBLICATIONS AND OTHER REPORTS OFTEN HAD INDIVIDUAL PARTICIPANT LEVEL DATA, AS OPPOSED TO THE SUMMARY AGGREGATE LEVEL DATA THAT IS ACCOMMODATED BY CLINICAL TRIALS POILT GOV. SO THAT CLEARLY -- CLINICALTRIALS.GOV. SO THAT CLEARLY POSES ANOTHER CHALLENGE. A LITTLE BIT RELATED TO THE FERS ITEM I RAISED IS THAT MANY STUDIES ALSO SEEMED TO FALL MORE ON THAT SPECTRUM OF EXPLORATORY WHERE THEY MIGHT BE ITERATIVE AND THERE MIGHT BE MORE PROCESSES UNDERWAY TO REALLY SORT OF OPTIMIZE A RESEARCH DESIGN AS WELL, AND IT BECAME HARD TO KIND OF TELL WHERE THAT -- WHEN A STUDY KIND OF TRANSITIONED INTO SOMETHING THAT WAS MORE THAN JUST THIS PRELIMINARY EXPLANATION, EXPLORATION. AND THEN FINALLY, WE SAW A LOT OF PRESENT OF NONTABULAR FORMATS SO AGAIN, THE RESULTS INFORMATION THAT WE SEE IN PUBLICATIONS FOR APPLIED CLINICAL TRIALS CAN USUALLY EASILY BE TRANSFORMED INTO THAT TAB LORE FORMAT, BUT -- TABULAR FORMAT BUT WHAT WE SAW AT BESH IS THAT IT MORE OFTEN INCLUDED GRAPHICAL DATA, IMAGES, ESPECIALLY IN THE CONTEXT OF BRINGING RESEARCH, QUAL TIIVE INFORMATION -- QUALITATIVE INFORMATION AND AGAIN MANY STATISTICAL ANL SEEPS BUT NOT NECESSARILY SHOWING SORT OF THE UNDERLYING SUMMARY DATA THAT SUPPORTED SOME OF THOSE STATISTICS. SO WHAT ENDED UP SORT OF BEING A LOOSE CORRELATION HERE IS THAT WHEN YOU THINK ABOUT THAT SPECTRUM OF BESH ARE RESEARCH, THAT THE STUDIES THAT TENDED TO BE MORE ON THE EXPLORATORY SIDE OF BASIC RESEARCH WERE NOT AS WELL ACCOMMODATED BY CLINICALTRIALS.GOV, WHEREAS THE FURTHER THAT YOU WENT ALONG THE SPECTRUM AND GOT CLOSER TO REALLY BEING HYPOTHESIS TESTING AND CONFIRMATORY-TYPE STUDIES, THEN THOSE SEEMED TO MORE CLOSELY MIRROR AND FIT THE MODEL OF CLINICALTRIALS.GOV. AND SO I HAVE BEEN TALKING ABOUT THESE CHALLENGES THE IN ABSTRACT , BUT I THINK IT WILL BE MOST USEFUL IF I CAN TURN OVER TO MY COLLEAGUE, ELISA, WHO WILL BE ABLE TO SHOW SOME OF THE CASE STUDIES THAT WE INCORPORATED IN THE SAMPLE TO ILLUSTRATE THE THE CHALLENGES AND ISSUES I'VE JUST DESCRIBED. >> Elisa Golfinopoulos: THANK YOU, BECKY, AND GOOD AFTERNOON, EVERYONE. TODAY I WILL BE GOING OVER FOUR DIFFERENT CASE STUDY EXAMPLES. THESE CASE STUDIES HAVE CHARACTERISTICS COMMON OF BESH AND WOULD POSE CHALLENGES FOR REGISTRATION AND RESULTS REPORTING. ONE COMMON CHARACTERISTIC OF MANY BESH THAT WE SAW IN OUR ANALYSIS WAS THAT OFTEN MULTIPLE INTERRELATED STUDIES ARE PROPOSED WITHIN A SINGLE PROTOCOL. THESE TYPES OF PROTOCOLS POSE THE CHALLENGE OF HOW TO BEST REPRESENT ALL THE DIFFERENT TYPES OF EXPERIMENTS AND THEIR RESULTS IN A WAY THAT WOULD BE SCIENTIFICALLY MEANINGFUL AND USEFUL. THIS IS ONE OF THE PRIMARY CHALLENGES FOR THE PROTOCOL OF CASE STUDY NUMBER 3. AT ITS INCEPTION IN 1993, IT WAS ONE OF THE FIRST FMRI PROTOCOLS PROPOSING AROUND TEN DIFFERENT EXPERIMENTS. AS RESULTS WERE DISCOVERED, AMENDMENTS WERE MADE, AND THE PROTOCOL GREW TO ITS PRESENT VERSION INCLUDING OVER 50 DIFFERENT EXPERIMENTS ORGANIZED ACCORDING TO 7 DIFFERENT THEMES. THIS STUDY THAT WAS PUBLISHED IN ASSOCIATION WITH THE PROTOCOL DEMONSTRATES THE LEVEL OF COMPLEXITY THAT ANY ONE STUDY CAN HAVE. DEVELOPED BY COMBINING THE METHODS OF MULTIPLE STUDIES FROM THE PERCEPTION THEME WITHIN THE PROTOCOL, THIS NEW STUDY WAS CREATED. IT INVOLVED THREE DIFFERENT NEURAL IMAGING SESSIONS PER PARTICIPANT AND BEHAVIORAL PERFORMANCE AND NEURAL ACTIVITY WERE RECORDED, SO THAT THE INVESTIGATORS COULD UNDERSTAND THE FUNCTIONAL SPECIALIZATION OF VISUAL BRAIN AREAS. THIS STUDY REPRESENTS JUST ONE OF THE COMMON SET OF EXPERIMENTS THAT CAN STEM ACROSS THEMES AND FROM THE 66 DIFFERENT EXPERIMENTS THAT WERE PROPOSE IN THE PROTOCOL. THE PRACTICING TUD OF THIS PROTOCOL UNDERSCORES THE CHALLENGE. FOR PROTOCOLS WITH MULTIPLE INTERRELATED STUDIES, WE MIGHT ASK, WHAT SHUFD THE UNIT OF REPORTING -- SHUSMED THE UNIT OF REPORTING BE? SHOULD IT BE AT THE LEVEL OF THE PROGRAM OF RESEARCH DESCRIBING OVER 50 DIFFERENT EXPERIMENTS AND NEARLY 200 OUTCOME MEASURES? OR AT THE LEVEL. THEME, EVEN THOUGH EXPERIMENTS CAN BE COMBINED ACROSS THEMES? OR AT THE LEVEL OF THE EXPERIMENT RESULTING IN OVER 50 DIFFERENT PUBLIC RECORDS? CERTAIN BESH PROTOCOLS ONLY PROVIDE HIGH LEVEL OBJECTIVES THAT BROADLY CONVEY WHAT WILL BE STUDIED, THE INSTRUMENTS USED FOR MEASUREMENT, AND THE INTERVENTIONS ASSIGNED TO PARTICIPANTS. NOTABLY, THESE PROTOCOLS COULD NOT PRESPECIFY THE ENDPOINTS TO BE ANALYZED FROM THE OUTSET. THESE TYPES OF PROTOCOLS POSE THE CHALLENGE OF OFFERING SUFFICIENT LEVEL OF DETAIL FOR A REGISTRATION TO BE SCIENTIFICALLY MEANINGFUL AND USEFUL. THIS IS ONE OF THE PRIMARY CHALLENGES FOR THE PROTOCOL OF CASE STUDY NUMBER 6. THE OVERARCHING, EXPLORATORY OBJECTIVE OF THE PROTOCOL WAS TO UNDERSTAND HOW IGE-MEDIATED FOOD ALLERGY DEVELOPS AND A VARIETY OF ASSESSMENTS WERE DESCRIBED AT A HIGH LEVEL. AT THE POINT OF PUBLICATION, THE DESIGN AND ENDPOINTS OF THE STUDIES ASSOCIATED WITH THE PROTOCOL ARE FULLY FLESHED OUT. FOR EXAMPLE, THIS STUDY INVOLVED CHILDREN WITH AND WITHOUT PEANUT CELLS WERE QUANTIFIED AND PHENOTYPED. NOTE BLIRKS THESE ENDPOINTS HAD NOT BEEN DOCUMENT UNDERSTAND IN THE PROTOCOL. FOR PROTOCOLS WITH A HIGH DEGREE OF FLEXIBILITY, LIKE THIS ONE, WE MIGHT ASK, FROM WHERE SHOULD THE LEVEL OF DETAIL ABOUT THE STUDY DESIGN AND THE OUTCOME MEASURES BE DRAWN FOR A SCIENTIFICALLY MEANINGFUL AND USEFUL REGISTRATION? THE APPROACH OF THIS NEXT PROTOCOL IS SOMEWHAT OF A HYBRID OF THE APPROACHES WE JUST SAW. THE PRIMARY OBJECTIVE WAS TO PROVIDE STANDARD CARE THERAPY FOR DRUG-RESISTANT EPILEPSY. A SECONDARY GOAL OF THE PROTOCOL WAS TO INVESTIGATE NEUROPHYSIOLOGICAL CORRELATE OF COGNITIVE PROCESSES. TOWARDS THIS END, SEVERAL SMALL EXPERIMENTS ARE PROPOSED INVOLVING COGNITIVE AND MOTOR TASKS THAT ARE ONLY BROADLY OUT LINED IN THE PROTOCOL. IN ADDITION, THIS STUDY THAT WAS PERFORMED IN ASSOCIATION WITH THE PROTOCOL, IN ASSOCIATION WITH THE SECONDARY GOAL, INVOLVED INTRACRANIAL RECORDINGS WHILE PARTICIPANTS PERFORMED A VERBAL EPISODIC MEMORY TASK. THE ASSESSMENT ASSESSED THE ROLE OF RIPPLES IN MEMORY RETRIEVAL, WHICH HAD NOT ORIGINALLY BEEN DOCUMENTED IN THE PROTOCOL. IN ADDITION TO MULTIPLE INTER RELATED STUDIES AND IN ABSENCE OF PRESPECIFIED OUTCOME MEASURES, CASE STUDY NUMBER ONE DEMONSTRATES ANOTHER COMMON CHARACTERISTIC OF THE BESH THAT WE SAW, A NEED FOR NONTABULAR FORMATS AND INDIVIDUAL PARTICIPANT LEVEL DATA. THIS FIGURE ON THE RIGHT DEMONSTRATES THE DEGREE OF SIMILARITY IN THE PATTERN OF ACTIVITY DURING THE ENCODING AND RETRIEVAL PERIOD OF CORRECT AND INCORRECT TRIAL PERFORMANCE. THE TEMPORAL REGION OF INTEREST, OUTLINED IN BLACK ON THE RIGHT, IS UNIQUE TO THIS STUDY AND REPORTING RESULTS FROM IT WOULD NOT BE A SCIENTIFICALLY MEANINGFUL OR USEFUL WITHOUT THE FIGURE THAT DEMONSTRATES ITS TEMPORAL LOCATION. LIKEWISE, THE PUBLICATION INCLUDED THE RESULTS OF EACH INDIVIDUAL PARTICIPANT BECAUSE THE PLACEMENT OF THE ELECTRODES DIFFERED ACROSS PARTICIPANTS. THESE PARTICIPANT LEVEL DATA CAPTURE DISTINCT INFORMATION THAT CANNOT BE CONVEYED WITH THE GROUP LEVEL DATA. NONTABULAR FORMATS LIKE FIGURES AND INDIVIDUAL PARTICIPANT LEVEL DATA POSE A CHALLENGE FOR A SYSTEM THAT EXPECTS TABULATED DATA AND GROUP LEVEL INFORMATION THIS LAST, THE LAST CHARACTERISTIC OF BESH THAT I WILL COVER TODAY IS ITERATIVE PRELIMINARY STUDIES. CERTAIN BESH-INVOLVED PILOTING TO DEVELOP OR OPTIMIZE PARAMETERS OR PROCEDURES. THE PROTOCOLS ASSOCIATED WITH THESE BESH ARE MORE FLEXIBLE TO ACCOMMODATE ITERATIVE PRELIMINARY STUDIES. THESE TYPES OF PROTOCOLS POSE THE CHALLENGE OF HOW TO CAPTURE THE PROGRAM OF RESEARCH EVOLUTION AND POTENTIAL METAMORPHOSIS IN A WAY THAT WOULD BE SCIENTIFICALLY MEANINGFUL AND USEFUL. THIS IS ONE OF THE PRIMARY CHALLENGES FOR THE PROTOCOL OF CASE STUDY NUMBER 4. THE OVERARCHING GOAL WAS TO STUDY DIFFERENT STIMULATION PARAMETERS. INFORMATION EMERGING FROM THE RESULTS OF THESE EXPERIMENTS COULD LEAD TO OPTIMIZED PROCEDURES FOR HYPOTHESIS DRIVEN RESEARCH. PRELIMINARY PILOT STUDIES ARE RARELY CAPTURED IN PUBLICATION. HOWEVER, WE WERE GIVEN A COPY OF THE CONTINUING REVIEW DOCUMENT WITHIN WHICH ONE OF THE PILOT STUDIES WAS DESCRIBED. THE GOAL OF THIS PILOT WAS TO EXPLORE THE MODIFICATION OF EXISTING MOTOR MEMORIES. THE RESULTS INFORM THE DISIEFN A HYPOTHESIS DRIVEN EXPERIMENT RELATED TO MOTOR LEARNING AND REHABILITATION IN STROKE PATIENTS. THE FLEXIBLE NATURE OF THE PROTOCOL ALLOWED FOR THE DESIGN OF AN EXPERIMENT WHERE A HYPOTHESIS COULD BE TESTED. FOR PROTOCOLS LIKE THIS ONE, WITH ITERATIVE PRELIMINARY STUDIES, WE MIGHT ASK, AT WHAT TIME POINT SHOULD REGISTRATION AND RESULTS REPORTING TAKE PLACE SO THAT INFORMATION RELATED TO THE STUDY DESIGN AND OUTCOME MEASURES IS SCIENTIFICALLY MEANINGFUL AND USEFUL? AT THIS POINT, I WILL TURN THE PRESENTATION BACK OVER TO BECKY. >> Rebecca Williams: GREAT. THANK YOU SO MUCH, ELISA, AND I THINK SEEING SPECIFIC EXAMPLES REALLY DOES HELP ILLUSTRATE AND DRIVE HOME THE CHALLENGES THAT I DESCRIBED BEFORE. IN ADDITION IN THIS ANALYSIS, WE HAD THE BENEFIT OF BEING ABLE TO TALK MORE DEEPLY WITH SOME OF THE INVESTIGATORS AS WELL TO BETTER UNDERSTAND THEIR VIEWS. MANY OF THE VIEWS THAT ARE CAPTURED HERE ARE QUITE CONSISTENT WITH THE THINGS THAT WERE ALSO DOCUMENTED IN THE RESPONSE TO THE RFI. WHEN THINKING ABOUT BENEFITS RELATED TO A REGISTRATION AND RESULTS REPORTING MODEL, INVESTIGATORS SAW THAT THE PURPOSE OF HAVING A CENTRAL INDEX OR REPOSITORY FOR BESH COULD BE QUITE USEFUL IN THE CONTEXT OF PROMOTING RESEARCH COLLABORATION, HELPING TO MINIMIZE DUPLICATION OF EFFORT, AND TO PROVIDE A NICE FRAMEWORK FOR TRACKING AND MONITORING A PROGRAM OF RESEARCH, AND COULD ALSO POTENTIALLY FACILITATE MANAGEMENT OF THESE MULTIPLE INTERRELATED EXPERIMENTS, AND THEN MOST IMPORTANTLY, WHEN WE THINK ABOUT MANY OF THE GOALS OF THE NIH POLICY, IT'S AROUND PROMOTING SCIENTIFIC RIGOR AND TRANSPARENCY. WE DID SOLICIT VIEWS FROM INVESTIGATORS RELATED TO ANY SUGGESTIONS THEY MIGHT HAVE IN THIS CONTEXT. THERE'S A LOT OF ENCOURAGEMENT TO THINK ABOUT ALL THE EXISTING PROCESSES IN PLACE AND HOW THOSE COULD BE LEVERAGED. IT WAS EXPRESSED THE DESIRE TO HAVE NARRATIVE SUMMARIES WITH FIGURES AND IMAGES THAT WOULD BE ABLE TO PROVIDE SUFFICIENT CONTEXT FOR THE RESULTS REPORTING, AND REALLY THINKING ABOUT FLEXIBILITY AND REPORTING APPROACH, ESPECIALLY AS THE RESEARCH ADAPTS TO ITS OWN FINDINGS, BUT ALSO AS THE ENTIRE RESEARCH LANDSCAPE EVOLVES. AND THEN THINKING ABOUT BROADER, MORE INCLUSIVE PLATFORM AS WELL, PROPOSING A NEW PLATFORM, HUMAN RESEARCH.GOV WAS ONE PROPOSAL WE HEARD. SO NOTWITHSTANDING THE BENEFITS, OF COURSE, THE CHALLENGES WERE ALSO REINFORCED. AGAIN, JUST SUPPORTING TRANSPARENCY BUT GENERAL CONCERNS ABOUT HOW THE OVERALL APPROACH FOR THE POLICY CAN BE IMPLEMENTED. AND AGAIN, JUST REINFORCING THE CHALLENGES THAT WE IDENTIFIED AND THAT ELISA FURTHER ELUCIDAT ED IN TERMS OF DISSEMINATING EXPLORATORY BESH USING A MODEL THAT WAS PRIMARILY DESIGNED AROUND APPLIED CLINICAL TRIALS. AND THERE WAS CONCERN ABOUT REALLY BALANCING THE LEVEL OF BURDEN OF REPORTING WITH THE UTILITY OF THE INFORMATION EFTS ITSELF AND THAT WHEN THERE ARE LARGE NUMBER OF EXPERIMENTS, SOMETIMES HAPPENING RELATIVELY QUICKLY, WITH LARGE AMOUNTS OF DATA OR OVER LONG PERIODS OF TIME, YOU KNOW, HOW CAN YOU MANAGE THOSE PARTICULAR SPACES. AND THEN AGAIN JUST SORT OF ACKNOWLEDGING THE DISTINCTION AND WHEN SOME STUDIES MIGHT BE DIFFERENT FROM APPLIED CLINICAL TRIALS IN TERMS OF THE OVERALL FRAMEWORK, AND REALLY LOOKING FOR CLARIFICATION ON WHAT ASPECTS MUST BE PRESPECIFIED AND DOCUMENTED TRANCE TRANSPARENTLY AND REALLY THINKING ABOUT SOME OF THAT TIMING ASSOCIATED WITH THE RESEARCH ITSELF. SO CONSIDERING THE INFORMATION FROM THE RFI, CONSIDERING THE INFORMATION FROM OUR OWN ANALYSIS AND THE CASE STUDIES, WE WERE ABLE TO ELUCIDATE BASICALLY THE FACTORS THAT MUST REALLY BE TAKEN INTO CAREFUL CONSIDERATION WHEN WE THINK ABOUT BESH REGISTRATION AND RESULTS REPORTING. WHAT WE KEEP COMING BACK TO IS REALLY IDENTIFYING THIS UNIT OF REPORTING AND WHAT IS THE MOST APPROPRIATE UNIT TO BE REPORTED. I WILL FLASH AHEAD AND THEN COME BACK BUT BACK A LITTLE BIT TO WHAT ELISA WAS DESCRIBING WHERE THERE MIGHT BE ONE OVERALL RESEARCH PROGRAM, AND THEN WITHIN THAT RESEARCH PROGRAM THERE MIGHT BE A PROJECT OR THEME THAT'S COMPOSED OF A CLUSTER OF EXPERIMENTS, OR IT MEET JUST BE A STANDALONE EXPERIMENT ON ITS OWN. SO IN ORDER TO REALLY DEFINE AND DESCRIBE WHAT'S APPROPRIATE FOR BESH, IT'S REALLY IMPORTANT TO CONSIDER THIS ASPECT OF UNIT TO HELP DRIVE SOME OF THE OTHER INFORMATION THAT'S NEEDED. SO AGAIN, WHEN WE THINK ABOUT THE MODEL FOR APPLIED CLINICAL TRIALS, THE TIMING AND THE DUE DATES ARE REALLY DRIVEN BY THOSE ITEMS THAT WE SAW IN THE LIFE CYCLE OF THE CLINICAL TRIAL ITSELF. SOATS PRETTY CLEAR WHEN A CONSTITUTED -- SO IT'S PRETTY CLEAR WHEN A STUDY IS INITIATED AND PARTICIPANTS ARE BEING FIRST ENROLLED ANDETH ALSO PRETTY CLEAR WHEN THE FINAL COLLECTION OF DATA IS EITHER FOR THE PRIMARY OUTCOME OR THE STUDY OVERALL, AND THOSE ARE THE MILESTONES THAT THE CURRENT REGISTRATION AND CRS RESULTS REPORTING ARE FRAMED AROUND SO YOU HAVE TO REALLY THINK ABOUT WHAT MILESTONES MIGHT BE RELEVANT FOR THAT PARTICULAR UNIT. THEN OF COURSE WHAT'S THE MINIMUM REQUIRED CONTENT WITHIN CLINICALTRIALS.GOV THE CURRENT MODEL SPECIFIES A COMBINATION OF OPTIONAL AND REQUIRED DATA ELEMENTS THAT ARE EXPECTED TO BE COMPLETED AS WELL AS THEN THE FORMAT OF THE INFORMATION ITSELF WE HAVE A COMBINATION OF STANDARDIZED FIELDS THAT REALLY DO SUPPORT THAT INDEXING AND CENTRALIZED ABILITY TO FIND SIMILAR STUDIES, BUT THEN THERE'S OTHER DATA PO'S THAT ARE LESS STRUCTURED AND ALLOW FOR MORE FLEXIBILITY IN HOW THAT INFORMATION IS PROVIDED. SO AGAIN, WHEN THINKING ABOUT THIS UNIT, YOU HAVE TO CONSIDER WHAT FORMAT SHOULD BE IN PLACE. WE ALSO HAVE A QUALITY CONTROL REVIEW PROCESS IN PLACE AT CLINICALTRIALS.GOV THAT EVALUATES BASICALLY INTERNAL CONSISTENCY LOOKING FOR MEANINGFUL ENTRIES AND ENSURING THAT THE INFORMATION IS LOGICAL AND AGAIN SORT OF INTERNALLY CONSISTENT. SO THINKING ABOUT HOW MUCH THAT MATTERS. AND THEN WE ALSO I THINK CONTINUE TO RETURN TO DEFINITION S BECAUSE WE'VE SEEN SUCH A HETEROGENEOUS -- I SHOULD HAVE CHOSEN AN EASIER WORD TO SAY, BECAUSE THERE'S SO MUCH HETEROGENEITY IN THE STUDIES THAT WE'VE SEEN, THERE'S AIR LOT OF DESIRE FROM INVESTIGATORS, OF COURSE, TO HAVE FURTHER CLARIFICATION AROUND DEFINITIONS AND WHERE AND HOW THESE DEFINITIONS APPLY TO THEM. SO I WOULD JUST LIKE TO ACKNOWLEDGE THE TEAM THAT HAS BEEN SUPPORTING THIS WORK HERE AT THE NATIONAL LIBRARY OF MEDICINE. NONE OF THIS COULD HAPPEN WITHOUT A WHOLE CAST OF CHARACTERS TO SUPPORT IT, AND I ALSO MUST THANK OUR INTRAMURAL AND EXTRAMURAL INVESTIGATORS WHO WERE SO SUPPORTIVE AND WILLING TO SHARE INFORMATION TO BE ABLE TO SUPPORT THIS ANALYSIS AND TO HELP US LEARN AS MUCH AS WE COULD. I ALSO WANT TO HIGHLIGHT A FEW MATERIALS THAT WE HAVE AVAILABLE ON CLINICALTRIALS.GOV. WE HAVE A PAGE THAT'S DEDICATED TO TRAINING MATERIALS. YOU CAN FIND OLD WEBINARS AND INFORMATION THERE, BUT WE'VE RECENTLY LAUNCHED THIS NEW TOOL CALLED PRS GUIDED TUTORIALS, WHICH PROVIDES STEP BY STEP EN INSTRUCTIONS BOTH FOR REGISTRATION AS WELL AS RESULTS INFORMATION SUBMISSION. AND THEN WE'VE HAD THE OPPORTUNITY TO PARTNER WITH BILL 'S OFFICE TO DEVELOP STUDY DESIGN EXAMPLES THAT MAY BE MORE COMMON OR MORE CHALLENGING IN THE BEHAVIORAL AND SOCIAL SCIENCE RESEARCH COMMUNITY. THESE ARE THINGS AGAIN THAT STILL FOLLOW SORT OF THAT TRADITIONAL MODEL OF BEING WELL ACCOMMODATED IN CLINICALTRIALS.GOV BUT ADDRESS SOME REALLY INTERESTING CHALLENGES WITH REPORTING AND USING THE SYSTEM. SO THOSE INCLUDE A CLUSTER RANDOMIZED EXAMPLE AS WELL AS FRACTURIZED AND ARE COMING SOON AND THOSE ARE LISTED ON TRAINING MATERIALS PAGE AND ALSO EVENTUAL LY GET FOLDED INTO THE GUIDED TUTORIAL MATERIALS. I ALSO WANT TO EMPHASIZE THAT IF AT ANY POINT IN TIME AS AN INVESTIGATOR OR SOMEONE SUPPORTING AN INVESTIGATOR YOU HAVE QUESTIONS ABOUT HOW TO REGISTER A STUDY IN CLINICALTRIALS.GOV OR HOW TO REPORT YOUR RESULTS, WE OFFER ONE ON ONE ASSISTANCE. THIS IS USED MOST OFTEN ON THE RESULTS SUB METION SIDE -- SUBMISSION SIDE. SO IF YOU EVER HAVE QUESTIONS OR NEED THAT TYPE OF SUPPORT, PLEASE DON'T HESITATE TO REGISTER AT CLINICALTRIALS.GOV TO GET ASSISTANCE. THEN FINALLY, WE DO HAVE AN E- BULLETIN THAT WE CALL HOT OFF THE PRS THAT ALLOWS OUR USE TOWERS GET UPDATES RELATED TO THE SUBMISSION SYSTEM THAT IMPACT THEM, AND THAT HEATS WHERE WE WOULD -- AND THAT'S WHERE WE WOULD SEND OUT A NOTIFICATION TOO ABOUT THESE NEW STUDY DESIGN EXAMPLES BEING AVAILABLE. SO PLEASE DO TAKE ADVANTAGE OF SIGNING UP FOR THAT IF YOU'RE LOOKING TO STAY INFORMED ON WHAT IS HAPPENING OVER AT CLINICALTRIALS.GOV. AND WITH THAT, I WOULD LIKE TO TURN IT OVER TO OUR MODERATOR, ADAM BURGER, WHO IS WITH THE NIH OFFICE OF SCIENCE POLICY, WHO HAS ALSO BEEN VERY SUPPORTIVE OF THIS WORK, AND IF YOU WOULD LIKE TO, YOU CAN E-MAIL QUESTIONS TO THE E-MAIL BOX THAT YOU SEE HERE , OR THERE'S ALSO THE SEND LIVE FEEDBACK BUTTON AS YOU'RE WATCHING THE VIDEOCAST, AND WE DO HAVE SOME EXTRA TIME TO BE ABLE TO HOPEFULLY ADDRESS SOME OF THE QUESTIONS YOU MAY HAVE. >> ADAM BERGER: THANKS VERY MUCH, BECKY, I WANT TO THANK YOU AND BILL AND ELISA FOR GIVING SUCH GREAT TALKS. I'LL ASK AWELY YOU TO COME HERE AND JOIN ME SO WE CAN ANSWER QUESTIONS FROM THE COMMUNITY. I'M ADAM BERGER, DIRECTOR OF THE CLINICAL AND RESEARCH POLICY AND OFFICE OF SCIENCE POLICY PART OF THE OFFICE OF DIRECTOR ACCIDENT HAPPY TO JOAN YOU TODAY AND MODERATE THIS Q & A AND WE DO HAVE A FEW MINUTES HERE. AGAIN, JUST FEEL FREE TO SEND THOSE QUESTIONS INTO THE SCIENCE POLICY MAILBOX THAT YOU SEE UP ON THE SCREEN, OR CLICK THAT SEND LIVE FEEDBACK. I THINK MAYBE I'LL ASK AND SOMEWHAT ANSWER THE FIRST QUESTION THAT'S PROBABLY POPPING UP IN EVERYBODY'S MIND AS WE'RE WAITING FOR FOLKS TO SEND IN QUESTIONS, BUT, YOU KNOW, WHAT'S REALLY NEXT HERE? I JUST WANT TO REITERATE WHAT BECKY STATED ABOUT HOW MUCH WE APPRECIATE THE WILLINGNESS OF INVESTIGATORS AND THE COMMUNITY TO ENGAGE WITH US TO HAVE BEEN ABLE TO CONDUCT THIS ANALYSIS. OUR PLAN IS REALLY TO CONTINUE WORKING WITH THE BASIC REFN COMMUNITY TO ADDRESS BESH SPECIFIC CONCERNS WITH REGISTRATION AND REPORTING, AND WE ARE HOPING THAT WE'LL BE ABLE TO MAKE THIS ANALYSIS AVAILABLE TO THE COMMUNITY SO YOU CAN REVIEW IT AS WELL IN THE NEAR FUTURE. SO, YOU KNOW, I'LL SHIFT A LITTLE BIT HERE JUST BECAUSE I THINK THERE'S SOME REALLY INTERESTING POINTS THAT WERE BROUGHT UP OVER THE TALK SO FAR. AND BECKY, YOU HAD MADE A COMMENT ABOUT TRYING TO -- ONE OF THE SUGGESTIONS THAT WAS MADE FROM INVESTIGATORS WAS REALLY TO LEVERAGE EXISTING DOCUMENTS, AND I THINK THERE ARE SOME POSSIBILITIES HERE AND SO I'M WONDERING IF YOU MIGHT COMMENT ON WHERE THERE MIGHT BE SOME ABILITY TO KIND OF LEVERAGE SOME OF THE JUST IN TIME PROCESSES OR ANY OTHER PARTS OF THE GRANT APPLICATION TO BE ABLE TO FEED INTO THE REGISTRATION PROCESS FOR GRANT SUBMISSIONS. SO BECKY? >> Rebecca Williams: YEAH. SO IN GENERAL, WE'VE BEEN WORKING WITH THE OFFICE OF EXTRAMURAL RESEARCH PRETTY CLOSELY FOR A WHILE, RECOGNIZING THAT MUCH OF THE INFORMATION THAT GOES INTO A GRANT APPLICATION IN THE CLINICAL TRIALS SPACE IS REALLY SIMILAR TO INFORMATION THAT'S ALSO AVAILABLE IN CLINICALTRIALS.GOV AS REGISTRATION INFORMATION. SO WHENEVER THAT INFORMATION IS THE SAME, WE REALLY AIM TO ALIGN OUR DEFINITIONS, AS WELL AS THE NAMES OF THE DATA FIELDS THEMSELVES SO THAT SOME OF THAT INFORMATION COULD BE TRANSPORTED OVER TO CLINICALTRIALS.GOV TO INITIATE A STUDY REGISTRATION. AND I THINK THAT IS SORT OF JUST A SMALL PERCENTAGE, BUT IT'S A PERCENTAGE OF THE DATA ELEMENTS THAT SUPPORT A COMPLETE REGISTRATION, AND I THINK WHEN WE THINK ABOUT THE OTHER PROCESSES, YOU KNOW, IT WAS THE INVESTIGATORS THEMSELVES WHO SORT OF HAD THE BEST SUGGESTIONS ABOUT SORT OF THEY KNOW VERY WELL AND CLOSELY THEIR EXISTING REPORTING REQUIREMENTS AND WERE LOOK TO GO SEE BASICALLY WHERE THERE ARE OPPORTUNITIES FOR SYNERGIES IN THOSE SPACES, EVEN BEYOND WHAT WE CURRENTLY HAVE. >> ADAM: THANKS, BECKY. AGAIN, WHILE WE'RE WAITING FOR MORE QUESTIONS TO TRICKLE IN, I'LL KEEP ASKING A FEW MORE. ONE OF THE THINGS THAT YOU BROUGHT UP THAT YOU LISTED AS ONE OF THE FIVE PRIMARY CHALLENGES WAS THIS CONCEPT OF THE ITERATIVE DESIGN. AND I'M WONDERING IF YOU MIGHT COMMENT ON HOW CHANGES TO A TRIAL MIGHT BE HANDLED IN THE REGISTRATION PROCESS. SO FOR INSTANCE, AS YOU'RE GOING THROUGH THOSE ITERATIONS AND YOU'RE IDENTIFYING NEW MEASURES, FOR INSTANCE, THAT ARE GOING TO COME INTO THE NEXT ROUND, HOW MIGHT THAT BE HANDLED AS YOU GO FORWARD? >> Rebecca Williams: YEAH, SO I THINK THERE'S A LOT OF DIFFERENT THINGS SORT OF EMBEDDED IN THAT QUESTION, AND I THINK IN GENERAL , YOU KNOW, ONE OF THE ASPECTS OF THE CLINICALTRIALS.GOV REPORTING MODEL JUST IN GENERAL IS THE EXPECTATION THAT INFORMATION RELATED TO THAT CLINICAL TRIAL IS KEPT UP TO DATE AS THERE ARE PROTOCOL AMENDMENTS OR OTHER TYPES OF INFORMATION THAT ARE REFLECTED IN THE STUDY REGISTRATION INFORMATION ITSELF. YOU CAN SOMETIMES HAVE PROTOCOL AMENDMENTS THAT WOULDN'T GET DETECT UNDERSTAND IN THE REGISTRATION INFORMATION. SO THE SYSTEM, YOU KNOW, ITSELF IS REALLY WELL SET UP IN TERMS OF BEING ALIVE -- A LIVE RECORDING, IF YOU WILL, AS INVESTIGATORS NEED IT TO BE ABLE TO UPDATE THAT STAWD DID I REGISTRATION CONTENT -- STUDY REGISTRATION CONTENT, AND I THINK WHEN WE THINK ABOUT BESH AGAIN IT'S SORT OF THINKING ABOUT WHAT LEVEL OF DETAIL IS NEEDED WITHIN THOSE DIFFERENT UNITS, AND YOU SORT OF HAVE THE FIRST REALLY WELL DEFINED WHAT UNITS OF REPORTING ARE EXPECTED IN THIS SPACE. >> ADAM: THANKS, BECKY. JUST ONE MORE REMINDER, YOU CAN SEND IN ANY QUESTIONS YOU HAVE BY EITHER SENDING DIRECTLY TO THE IENS POLICY@OD.NIH.GOV OR FEEL FREE TO SEND THE FEEDBACK BUTTON JUST BE LOAF THE VIDEO ITSELF AND THOSE QUESTIONS WILL COME IN THAT WAY. SO, YOU KNOW, BECKY, SOME OF THE OTHER THINGS THAT MIGHT BE RELEVANT HERE ARE GOING TO BE DIFFERENT TYPES OF TRIAL DESIGNS , WHERE IT MAY BE DIFFICULT TO HAVE INFORMATION ABOUT THE PROTOCOL OR THE STUDY ITSELF BE MADE AVAILABLE IN ADVANCE, WHETHER THAT MIGHT TAINT RESULTS OR WHATNOT, FOR INSTANCE, IN A DECEPTION TRIAL. COULD YOU COMMENT A LITTLE BIT ABOUT HOW THAT MIGHT BE HANDLED IN A REGISTRATION PROCESS? >> Rebecca Williams: YEAH. SO WE'VE SEEN SOME EXAMPLES OF THAT TO DATE WHERE THERE MAY BE DECEPTION INVOLVED IN A STUDY AND MAKING THAT INFORMATION AVAILABLE ON A PUBLIC WEBSITE LIKE CLINICALTRIALS.GOV AT THE START OF THE STUDY, YOU KNOW, MIGHT POSE A CHALLENGE, AND TYPICALLY IT'S BEST SORT OF ADDRESSED ON A CASE-BY-CASE BASIS DEPENDING ON WHAT THE STUDY ITSELF IS, BUT GENERALLY WE ADVISE TO FOLLOW THE LEVEL OF DETAIL THAT MIGHT BE INCLUDED IN THE INFORMED CONSENT FORM BECAUSE PRESUMABLY THAT WOULD BE EQUIVALENT TO SORT OF THE LEVEL OF INFORMATION THAT HAD BEEN DEEMED APPROPRIATE AT THE OUTSET OF THE STUDY. AND THEN WHEN YOU'RE PAST THAT TIME PERIOD IN WHICH THE DECEPTION MATTERS, THEN YOU WOULD UPDATE THE STUDY RECORD WITH THE MORE DETAILED ASPECT AFTER IT WOULD NO LONGER INTERFERE WITH THE STUDY CONDUCT ITSELF. >> ADAM: THANKS, BECKY. I'M GOING ON SWITCH TO SOME OF THE QUESTIONS THAT HAVE COME IN. SO I WANT TO THANK THE COMMUNITY AGAIN FOR SUBMITTING. THE FIRST ONE I WANTED TO ADDRESS IS, YOU KNOW, IT'S A QUESTION COMING IN, IN SUMMARY, HOW DO THE DEADLINES FOR REGISTRATION AND RESULTS REPORTING FOR BESH CLINICAL TRIAL DIFFER FROM ACT'S? >> YOU'RE TURNING TO ME ON THIS ONE, IS THAT CORRECT? >> ADAM: WELL -- >> DO YOU HAVE -- >> OR BILL, MAYBE BILL WANTS TO COME IN HERE. >> Bill Riley: WHAT WAS THE COMPARISON? >> ADAM: APPLICABLE CLINICAL TRIALS, RELATED. >> Bill Riley: WE'RE TALKING ABOUT THE DELAY IN REGISTRATION REPORTING BECAUSE OF THE EXTENSIONS THAT WE PROVIDED TO THE COMMUNITY FOR REGISTRATION REPORTING OF BESH STUDIES? >> ADAM: I THINK THE QUESTION IS REALLY THE DIFFERENCE -- LET ME REPHRASE IT. DO BESH CLINICAL TRIALS TEND ON FALL UNDER THE ROUX BRISK AN APPLICABLE CLINICAL TRIAL AS IT WOULD BE DEFINED UNDER FIDAH? >> YEAH, SO I CAN ACTUALLY JUMP IN ON THE FIDAH PART BECAUSE I LIVE AND BREATHE THAT ONE EVERY DAY. [LAUGHTER] AND IN THAT FRAMEWORK, FOR APPLICABLE TRIN CAL TRIALS -- APPLICABLE CLINICAL TRIALS THE REQUIREMENTS ARE TO REGISTER WITHIN 21 DAYS OF THE PARTICIPANT FIRST BEING ROLLED AND EXPECTATION IS THAT THERE BE REGULAR UPDATES AT LEAST ONCE A YEAR, SOME INFORMATION NEEDS TO BE UPDATED MORE FREQUENTLY THAN THAT. AND THEN AGAIN THE RESULTS WOULD BE DEPOSITED GENERALLY WITHIN ONE YEAR OF THE COMPLETION DATE, WHICH IS BASED ON THE FINAL COLLECTION OF DATA FOR THE PRIMARY OUTCOME MEASURE. NIH'S POLICY BASICALLY TRACKS ALONG THOSE REGULATORY REERLTS SO THAT NIH FUND -- REQUIREMENTS SO THAT NIH FURNLDED CLINICAL TRIALS ARE FOLLOWING THOSE SAME TIME LINES FOR REPORTING, SO AGAIN, 21 DAYS FOR REGISTRATION, AND SAME ONE YEAR TIMELINE FOR RESULTS. AND THAT'S A HUGE ADVANTAGE BECAUSE IT MEANS THAT THERE'S ONLY ONE SET OF TIME LINES THAT YOU HAVE TO REMEMBER, ALTHOUGH I HAVE TO PUT ONE LITTLE ASTERISK ON THAT BECAUSE THE JOURNAL EDITORS GENERALLY REQUIRE IN ORDER TO BE ELIGIBLE FOR PUBLICATION THAT YOU REGISTER THE CLINICAL TRIAL BEFORE THE FIRST PARTICIPANT IS ENROLLED, RATHER THAN THAT 21 DAY TIMELINE THAT I MENTIONED. SO I KNOW A LOT OF ORGANIZATIONS SORT OF DEFAULT TO THE MORE CONSERVATIVE TIMELINE OF THE JOURNAL EDITOR SO AS NOT TO RISK NOT BEING ABLE TO PUBLISH. >> THANKS, BECKY. THAT'S ALSO TRUE FOR AWFUL THE BESH STUDIES, THERE'S NO DIFFERENCE BETWEEN A BESH STUDY AND ANY OF THE OTHERS IN TERMS OF THOSE REQUIREMENTS. SO BILL, I'M GOING ON THROW ONE OVER TO YOU BECAUSE I THINK THIS QUESTION IS VERY MUCH RELATED TO SOME OF THE DATA THAT YOU WERE MENTIONING AT THE FRONT END. AND I'M JUST GOING TO READ THE QUESTION AS ACCOUNTS. IT SAYS I FIND THAT A NUMBER OF CLINICAL TRIALS HAVE NOT SUBMITTED RESULTS. HAVE I BEEN UNLUCKY OR IS THIS MORE COMMON? SO BACK TO A COUPLE STUDIES YOU MENTIONED AT THE BEGINNING AS TO WHAT SOME OF THE RATIONALE FOR WHY WE EMBARKED ON A CLINICAL TRIAL STEWARDSHIP REFORM IN THE FRONT END. >> Bill Riley: SO THERE HAVE BEEN A NUMBER OF STUDIES THAT SHOW THAT SOMEWHERE BETWEEN A THIRD AND A HALF OF THE TRADITIONAL CLINICAL TRIALS WITH A PRIMARY OUTCOME ENDPOINT DON'T PUBLISH THEIR PRIMARY RESULTS IN A TIMELY MANNER, AND THAT VARIES FROM STUDY TO STUDY. TYPICALLY 30 MONTHS FROM THE TIME THE PROJECT ENDS, THERE STILL ISN'T ANYTHING IN THE LITERATURE ABOUT THAT PRIMARY OUTCOME, AND ANY PUBLICATION OF THAT PRIMARY OUTCOME. SO IT HAS BEEN A CONSISTENT PROBLEM, EVEN OUR OWN DIRECTOR OF THE OFFICE EXTRAMURAL RESEARCH COULDN'T BELIEVE SOME OF THE RESULTS AND REPEATED THEM AT NHLBI DURING HIS TIME THERE, AND SURE ENOUGH, IT WAS EXACTLY THE SAME. I THINK OUR FRIENDS AT NHLBI THOUGHT THAT THEY WERE BETTER BUT IN ACTUALITY ABOUT THE SAME. IT REALLY MUCH IS CONSISTENT ACROSS THE BOARD REGARDLESS OF WHAT TYPE OF RESEARCH THAT YOU DO, THAT IT'S BEEN A PROBLEM IN THE FIELD. AND AS I NOTED, THOUGH SLIGHTLY LESS SO IN OBSERVATIONAL STUDIES , STILL A PROBLEM THERE. AND THE RESULTS THAT WE RECENTLY DID WHERE WE LOOKED AT EVERY RO1 AND UO1, THE RATES ARE MUCH LOWER, BUT KEEP IN MIND THAT'S ALL THE STUDIES, NOT JUST HUMAN STUDIES, SO EVEN ACROSS ALL RESEARCH THAT THE NIH FUNDS, THERE'S STILL A SMALL PERCENTAGE THAT FAIL TO PUBLISH ANYTHING OF ANY TYPE RESULTING FROM THEIR GRANT. SO THAT'S PART OF OUR RATIONALE, ONE, FOREIGN COURAGING PUBLICATION, BUT IT'S ALSO CRITICALLY IMPORTANT FOR THE PURPOSES OF CLINICAL TRIALS.GOV AND THEIR RESULTS REPORTING, WHICH PROVIDES A WAY TO REPORT THOSE RESULTS EVEN IN THE SITUATION IN WHICH NULL RESULTS ARE NOT TYPICALLY PUBLISHED OR THERE'S BIAS AGAINST PUBLISHING THEM OR EVEN WHERE YOUR STUDY GROSSLY FAILS FOR SOME FLAW OR CRITICAL REASON, THAT THOSE RESULTS ARE ARE STILL SOMEWHERE IN THE PUBLIC FOR PEOPLE TO BE ABLE TO LEARN FROM THEM AND MOVE FORWARD. >> ADAM: THANKS, BILL. I'M I'M GOING ON COMBINE A COUPLE QUESTIONS THAT HAVE COME IN BECAUSE THEY'RE ALL ASKING ABOUT EITHER FUNCTIONALITY WITHIN CLINICALTRIALS.GOV OR THE REGISTRY THAT COULD BE CALLED HUMAN RESEARCH.GOV IS WHAT THE E-MAILS ARE SAYING. YOU KNOW, I GUESS THE WAY I'M GOING TO FRAME THIS IS REALLY, YOU KNOW, WHAT KIND OF CONSIDERATIONS WOULD NEED TO BE TAKEN INTO ACCOUNT TO BE REALLY THOUGHTFUL ABOUT WHAT MIGHT BE ABLE TO FIT WITHIN CLINICALTRIALS.GOV VERSUS WHAT MIGHT NOT BE ABLE TO BE WITHIN THE CURRENT FRAMEWORK, WHAT KIND OF CONSIDERATIONS MIGHT NEED TO BE TAKEN INTO ACCOUNT FOR EXPAND ING THE FUNCTIONALITY OF CLINICALTRIALS.GOV IFLTS ITSELF, YOU KNOW, VERSUS OTHER ALTERNATIVE KINDS OF STRUMPLETZ SO BECKY, MAYBE YOU CAN JUST KIND OF TALK TO THE THINGS THAT REALLY WOULD NEED TO BE THOUGHT THROUGH TO EVALUATE THAT. >> Rebecca Williams: YEAH, I WOULD BE HAPPY TO COVER THAT, AND I THINK ONE OF THE FUN CHALLENGES ABOUT RUNNING CLINICALTRIALS.GOV IS THAT WE'RE SORT OF REFLECTING THE RESEARCH ECOSYSTEM IN WHICH WE'RE OPERATING. AND THAT ECOSYSTEM IS ADAPTING EVERY DAY AND OBVIOUSLY WITHIN THE PAST YEAR WE'VE SEEN EXTRAORDINARY ADAPTATIONS IN THAT SPACE. SO, YOU KNOW, OUR GOAL IS TO REALLY BE COGNIZANT OF HOW THAT LANDSCAPE IS SHIFTING AROUND US TO MAKE SURE THAT AS WE GO INTO THE FUTURE, THAT WE CAN BEST ACCOMMODATE THAT. BUT I THINK THAT THERE'S SOME CONSIDERATIONS THERE TOO JUST IN TERMS OF OVERALL, YOU KNOW, WHAT SCOPE IS MOST APPROPRIATE AS THIS LANDSCAPE EVOLVES AROUND US TOO, YOU KNOW, THERE'S BEEN A LOT OF INTEREST ALSO IN I WOULD SAY REAL WORLD EVIDENCE, SORT OF RETROSPECTIVE-TYPE ANALYSES AND OUR SYSTEM AGAIN WHEN WE GO BACK TO SOME OF THOSE ASSUMPTIONS IS REALLY MODELED ON PROSPECTIVE DATA COLLECTIONS. SO THE IRB USE THAT WE'RE TALKING ABOUT TODAY, YOU KNOW, AREN'T NECESSARILY UNIQUE IN THE SENSE OF THIS LARGER RESEARCH ECOSYSTEM WE'RE OPERATING IN. BUT I THINK THAT LAST SLIDE THAT I SHOWED THAT SORT OF OUTLINED AND STARTED WITH THE UNIT OF REPORTING IS REALLY ESSENTIAL, AND I THINK WHAT NIH HAS BEEN REALLY GREAT ABOUT SAYING AND REINFORCING IS THAT, YOU KNOW, WE REALLY WANT TO ENSURE THAT THE INFORMATION THAT'S BEING PROVIDED IS SCIENTIFICALLY MEANINGFUL AND USEFUL, AND THERE DOES NEED TO BE I THINK SOME FURTHER ANALYSIS TO BETTER DEFINE SOME OF THAT SCOPING AROUND THAT UNIT THAT WOULD ACHIEVE THAT GOAL OF SCIENTIFICALLY MEANINGFUL AND USEFUL, AND I THINK THAT'S THE STARTING POINT AND THEN WE CAN GET INTO EACH OF THOSE SUB BULLETS AROUND THE UNIT. >> THANKS, BECKY. GOING TO SUMMARIZE A COUPLE OF THESE OTHER ONES BECAUSE THEY'RE ASKING VERY MUCH ABOUT HOW YOU WOULD ACTUALLY, WHAT IS THE BEST WAY TO RENG STER A -- TO REGISTER A STUDY FOR INSTANCE THAT HAS MULTIPLE EXPERIMENTS OR MULTIPLE OUTCOMES OR MULTIPLE TYPES OF REPORTING FACTORS. AND SO BECKY, MAYBE I'LL KIND OF FRAME THIS QUESTION AS TO WHAT KIND OF REGISTRATION AND REPORTING HELP OR RESOURCES IS N LM MAKING AVAILABLE TO REALLY HELP THE COMMUNITY BE ABLE TO REGISTER REPORTING CLINICALTRIALS.GOV? >> Rebecca Williams: YEAH. SO THESE CHALLENGES ARE REAL. IT'S THIS ISSUE OF EXACTLY HOW DO YOU COMBINE OR EXPAND, YOU KNOW, THOSE STUDIES TO ACCURATELY REPRESENT THEM. AGAIN, WE DO HAVE A LARGE NUMBER OF RESOURCES AVAILABLE ON THAT TRAINING MATERIALS PAGE THAT I MENTIONED IS A GREAT START, AND I'LL JUST REINFORCE AGAIN, IF YOU EVER RUN INTO TROUBLE, WE DON'T WANT YOU SITTING THERE STEWING TRYING TO FIGURE OUT WHAT TO DO. DON'T HESITATE TO REACH OUT TO REGISTER AT CLINICALTRIALS.GOV, AND THAT'S A MAILBOX THAT'S MONITORED BY MANY CAPABLE PEOPLE WHO WILL HELP GET YOU TO THE RIGHT PERSON AND THE RIGHT HELP FOR THAT KIND OF SUPPORT. AND WE DO FIND SOME THINGS REALLY DO NEED TO BE ADDRESSED ON A CASE-BY-CASE BASIS BECAUSE THE IRB USE ARE UNIQUE -- BECAUSE THE ISSUES ARE UNIQUE AND WE CAN WORK THROUGH THAT WITH YOU. >> ADAM: GREAT. SO I'M SEEING IT'S 3:58 SO WE HAVE TWO MINUTES LEFT, SO I THINK THIS MIGHT BE OUR LAST QUESTION BEFORE WE HAVE TO WRAP UP. IT'S A QUESTION THAT CAME IN ASKING, YOU KNOW, SO WHERE ARE BESH TRIALS REQUIRED TO REGISTER AND REPORT? I THINK I'LL TOSS THIS ONE OVER TO BILL SINCE HE DID MENTION THE FLEXIBILITIES THAT WE'VE BEEN ABLE TO PUT INTO PLACE FOR THESE SPECIFIC TYPES OF STUDIES. SO BILL, DO YOU MIND FIELDING THE QUESTION ABOUT WHERE WE ACTUALLY -- WHERE ARE THEY REQUIRED TO REGISTER REPORTING? >> Bill Riley: SURE. SO THERE HAS BEEN AT TIMES SOME MISCONCEPTION THAT MAYBE WE'RE SUPPOSED TO REGISTER STER AND REPORT, THAT'S NOT THE CASE, ALL BESH TRIALS DO NEED TO REGISTER AND DO RESULTS REPORTING ON SOME PLAT FOMPLET OUR PREFERENCE OF COURSE WOULD BE TO THE DEGREE THAT IT WORKS OUT FOR YOU TO REGISTER ON CLINICALTRIALS.GOV. WHEN YOU CANNOT, WE ASK THAT YOU WOULD CONSIDER PLATFORMS THAT ARE NOT HOME-GROWN BUT ARE WIDE AND HAVE A DIVERSE AND BROAD PERSPECTIVE IN TERMS. PEOPLE WHO CAN SEE IT AND OBSERVE IT, THAT HAVE SOME SUSTAINABILITY TO THAT PLATFORM, THAT YOU KNOW IT WILL BE AROUND A YEAR OR TWO OR FIVE OR TEN FROM NOW. ALL THE SORT OF THINGS THAT WE SEE AS IMPORTANT AND WHY WE HAVE A PREFERENCE FOR CLINICALTRIALS.GOV, PLATFORMS THAT COME AS CLOSE TO BEING ON A REPRESENT AS POSSIBLE BUT GIVE YOU MORE FLEXIBILITY IN TERMS OF BEING ABLE TO REGISTER AND REPORT I THINK ARE THE PLACES THAT YOU WANT TO TRY TO LOOK FOR >> ADAM: THANKS, BILL. SORRY WE CAN'T GET TO THE REST OF THE QUESTIONS. I THINK THERE'S SOME REALLY GREAT ONES COMING IN, INCLUDING ABOUT THE ONLINE PROTOCOL TOOL THAT IS AVAILABLE, INCLUDING IT DOES HAVE A PROTOCOL DEVELOPMENT SPACE FOR BEHAVIORAL STUDIES AS WELL. SO IT'S A GREAT QUESTION. UNFORTUNATELY, WE ARE COMPLETELY OUT OF TIME, SO WE CAN'T ANSWER IT. BUT WE CERTAINLY APPRECIATE EVERYONE COMING ON TO THE WEBINAR TODAY. AS MENTIONED EARLIER, THE WEBINAR ITSELF IS GOING TO BE ARCHIVED AND MADE AVAILABLE ON VIDEOCAST. A LINK TO IT WILL BE MADE AVAILABLE ALSO ON THE NLM WEBSITE AND WE'LL ALSO MAKE THOSE SLIDES THAT YOU SAW TODAY AVAILABLE THERE AS WELL. SO WE WILL MAKE SURE THAT THAT INFORMATION GETS OUT TO THE COMMUNITY AS TO WHERE THAT SPECIFICALLY IS LOCATED. BUT AGAIN, JUST WANT TO THANK ALL OF OUR SPEAKERS TODAY. BECKY, BILL AND ELISA, FOR DOING SUCH A GREAT JOB WALKING US THROUGH THIS ANALYSIS THAT WAS JUST COMPLETED, AND JUST WANT TO THANK YOU ALL FOR JOINING US. SO THANK YOU.