GOOD MORNING. THIS SEMINAR SERIES, EXPLORES RESEARCH DESIGN, MEASUREMENT, INTERVENTION, DATA ANALYSIS AND OTHER METHODS OF INTEREST TO PREVENTION SCIENCE. OUR GOAL IS TO ENGAGE THE PREVENTION RESEARCH COMMUNITY, IF TAUGHT-PROVOKING DISCUSSIONS, FOR THE BEST AVAILABLE RESEARCH SCIENCE AND SUPPORT THE DEVELOPMENT OF BETTER METHODS. HAVE I SOME HOUSEKEEPING ITEMS. TO PARTICIPATE BY TWITTER, FOLLOW US@NIHPREVEN. AND USE # NIHMTG FOR QUESTIONS. YOU MAY E-MAIL QUESTIONS TO PREVENTION@MAIL.NIH.GOV. THERE'S A LINK TO A FEEDBACK FORMAT THE BOTTOM OF THE VIDEO WHERE YOU CAN ADMIT QUESTIONS AT THE TALK. AT THE CONCLUSION, WE WILL OPEN THE FLOOR FOR QUESTIONS, THAT HAVE BEEN SUBMITTED VIA TWITTER AND E-MAIL. LASTLY, WE WOULD APPRECIATE YOUR FEEDBACK ABOUT TODAY'S SEMINAR. FOLLOWING TODAY'S TALK. VISIT THE ODP WEBSITE AT PREVENTION.NIH.GOV/MINDTHEGAP AND CLICK THE LINK FOR EVALUATION UNDER THE RESOURCE SEX. AT THIS TIME, I'D LIKE TO TURN THINGS OVER TO DR. DAVID N. MURRAY, ASSOCIATE DIRECTOR FOR PREVENTION AND DIRECTOR OF THE OFFICE OF DISEASE PREVENTION. >> THANK YOU, DR. K DICKERSON IS PROFESSIONAL OF A DEEM IOLOGY AT JOHN HOPKINS AND SERVES AS THE DIRECTOR FOR CENTER FOR CLINICAL TRIALS AND EVIDENCE SYNTHESIS, WHERE SHE IS INVOLVED IN METH LOGICAL RESEARCH, RELATED TO CLINICAL TRIALS. AND EVIDENCE SYNTHESIS. REPORTING BIASES, TRIALS, REGISTERS, PEER REVIEW, EVIDENCE-BASED HEALTHCARE AND PATIENT-SUITED OUTCOMES RESEARCH. SHE'S CONDUCTED STUDIES IN A NUMBER OF SUBJECT AREAS, INCLUDING WOMEN'S HEALTHY, EYES AND VISION AND SURGERY. SHE'S CURRENTLY ENGAGED IN EXAMINING PRACTICAL ISSUES FOR OPEN DEMANDS OF OPEN ACCESS FOR TRIAL DATA WHEN THE AVAILABLE DATA MAY BE DISCORDANT. LASTLY, DR. DICKERSON IS DIRECTOR OF THE U.S. COCHRAN CENTER, ONE OF 13 CENTERS WORLDWIDE, PARTICIPATING IN THE COCHRAN COLLABORATION. THE COLLABORATION AIMS TO HELP PEOPLE MAKE WELL INFORMED DECISIONS BY PREPARING, MAINTAINING AND PROMOTING THE ACCESSIBILITY OF SYSTEMATIC REVIEWS FOR AVAILABLE EVIDENCE ON THE BENEFITS AND RISKS OF HEALTHCARE. AT THIS TIME, I WOULD LIKE TO WELCOME DR. DICKERSON AND TURN THE SESSION OVER TO HER. >> GOOD MORNING, THANKS FOR ATTENDING SEMINAR. WHAT I PUT TOGETHER IS SOMETHING THAT I HOPE WILL HAVE A LITTLE SOMETHING FOR EVERYONE, AND NOT TOO MUCH OUTSIDE YOUR OWN AREA OF INTEREST. WHAT WHAT I'D LIKE TO DO, FIRST OF ALL, IS I WOULD LIKE TO TELL YOU A LITTLE BIT ABOUT MY DECLARATION OF INTEREST. I SHOULDN'T HAVE ANY CONFLICTS OF INTEREST, EXCEPT THAT I'M FUNDED BY NIH FROM THE NATIONAL EYE INSTITUTE, TO DO A SATELLITE FOR COCHRAN EYES AND VISION GROUP. I HAVE FUNDING FROM A NONPROFIT FROM HRQ PART OF THE PUBLIC HEALTH SERVICE, AND THE FDA. I'M NOT THE P.I. ON THE FDA GRANT THOUGH. SO WHERE I'D LIKE TO START IS TALKING ABOUT REVIEWS. THAT IS, REVIEWS OF THE EVIDENCE. REVIEWS OF WHAT'S AVAILABLE. AND YOU MAY FIND ME SLIPPING INTO CLINICAL TRIAL SPEAK AND THAT'S BECAUSE THAT'S MY ORIENTATION. BUT REVIEWS CAN APPLY TO ANY AREA OF HEALTH AND HEALTHCARE, SPECIFICALLY PREVENTION, ALL SORTS RELATING TO THE OFFICE OF DISEASE PREN. SO I HOPE I WON'T SLIP INTO IT, BUT IT'S DEFINITELY RELEVANT TO OUR TALKS TODAY. SO I'M MOST INTERESTED IN WHAT WE CALL SYSTEMATIC REVIEWS. THAT IS, WE. TO SUMMARIZE EXISTING RESEARCH AND FIND OUT WHAT WORKS, WHAT SORT OF INTERVENTIONS WORK, WHAT'S AT THIS TIME ED IOLOGY OF A DISEASE, WHAT IS THE PROGNOSIS FOR PATIENTS, WHICH DIAGNOSTIC TESTS WORK BEST, ET CETERA. AND ESSENTIALLY, ONE WANTS TO SUMMARIZE THE DATA THAT ARE OUT THERE FROM RESEARCH TO TRY TO MAKE A JUDGMENT, EITHER ABOUT WHAT RESEARCH TO DO NEXT, WHAT OUR PRIORITIES SHOULD BE, LOOK WHERE THERE ARE KNOWLEDGE GAPS, SETTING GUIDELINES FOR CARE. I THINK PEOPLE DO RECOGNIZE MORE AND MORE, THAT SYSTEMATIC REVIEWS ARE PERSON. WE WANT TO USE REVIEWS AND SYSTEMATIC REVIEWS ARE A LITTLE DIFFERENT. THAT IS, YOU WANT TO LOOK AT ALL THE RESEARCH THAT'S BEEN DONE, AND SEE WHERE DO WE STAND AT THIS POINT IN TIME. SO LET'S GUILTY TO SOME DEFINITIONS FIRST. WHAT IS A SYSTEMATIC REVIEW? SO AN EASY DEFINITION IS IT'S A REVIEW OF EXISTING KNOWLEDGE THAT USES EXPLICIT SCIENTIFIC METHODS AND SOMETIMES, BUT NOT ALWAYS, IT CONTAINS A META ANALYSIS, WHICH IS WHEN THE DATA ARE COMBINED QUANTITATIVELY, I'LL SHOW ON EXAMPLE OF THAT. I LIKE THIS. IT'S FROM MORE THAN 10 YEARS AGO, A PICTURE FROM MATER PIE'S PAPER IN THE MEDICAL JOURNAL OF ENDIA. IT GIVE US A NICE DEPICTION OF ALL REVIEWS, WHAT MANY OF US, INCLUDING ME, WERE TAUGHT, WERE REVIEWED AND NOW, WHERE WE ARE TODAY WITH SYSTEMATIC REVIEWS AND META ANALYSES. SO ALL REVIEWS INCLUDES A REVIEW OF EVIDENCE, LET'S SAY, IN THIS CASE, WHICH WOULD ENCOMPASS MORE THAN SYSTEMATIC REVIEWS. SYSTEMATIC REVIEWS, AND I'LL GO THROUGH THE STEPS IN ONE, ARE THIS SCIENTIFICALLY DONE REVIEW ARTICLE AND META ANALYSES ARE IN SOME SYSTEMATIC REVIEWS BUT NOT ALL. A SUBSET OF SYSTEMATIC REVIEWS WITH META ANALYSES ARE INDIVIDUAL PATIENT META ANALYSES, WHERE YOU ACTUALLY GET THE DATA FOR THE PATIENTS. YOU GET A DATA SET, AND THAT MAY BE MORE AND MORE POSSIBLE WITH DATA SHARING BUT RIGHT NOW, THERE ARE VERY FEW OF THOSE. LET'S GO BACK AND THINK ABOUT SYSTEMATIC REVIEWS. ONE OF THE THINGS THAT HELP ME TO UNDERSTAND SYSTEMATIC REVIEWS, WHEN I WAS JUST LEARNING ABOUT THEM. IS WHEN SOMEONE SAID TO HERE'S YOU KNOW, WE HAVE A DOUBLE STANDARD. WE INSIST ON CERTAIN CRITERIA FOR DOING THE PRIMARY STUDIES, THAT IS, WE KNOW GOOD STUDY DESIGNS FIT CERTAIN TYPES OF QUESTIONS YOU WOULD USE ARE FOR EXAMPLE, A RANDOMIZED CLINICAL TRIAL TO ASSESS A QUESTION ABOUT WHICH TREATMENT WORKS BEST AND WE KNOW HOW ONE IS SUPPOSED TO DO A RANDOMIZED TRIAL. BUT WE DON'T HAVE ANY STANDARDS LIKE THAT, FOR HOW WE DO A REVIEW. WE DON'T HAVE A METHODS SECOND. WE DON'T HAVE A RESULTS SECOND, WE DON'T HAVE A DISCUSS SECTION AND WE NEED THAT. WE CAN'T JUST DO REVIEWS WILLY-NILLY, HOWEVER WE FEEL IF WE HAVE STRICT CRITERIA FOR WHAT'S AN INDIVIDUAL STUDY FOR WHAT'S IN THAT REVIEW. SO A SYSTEMATIC REVIEW IS LIKE THE INDIVIDUAL STUDIES THAT ARE PART OF THE SYSTEMATIC REVIEW. IN THAT IT HAS A METHODS SECTION, A RESULTS SECTION AND A DISCUSSION SECTION AND INDEED AIRCRAFTS SYSTEMATIC REVIEW IS AN OBSERVATIONAL STUDY ITSELF. IT'S A CROSS SECTIONAL STUDY THAT MAY INCLUDE RANDOMIZED TRIALS, BUT IT'S STILL AN OBSERVATIONAL STUDY. ALL META ANALYSES FEED TO BE A PART OF A SYSTEMATIC REVIEW. BUT NOT ALL SYSTEMATIC REVIEWS NEED TO HAVE A META ANALYSES. YOU CAN'T JUST CHOOSE THE FIVE STUDIES OF COURSE INVOLVE WITH, AND DO A META ANALYSIS. BECAUSE THAT COULD BE BIAS. PARTICULAR BE JUST THE CHOICE OF INDIVIDUAL STUDIES THAT YOU'RE MADE AND IT MAY NOT REPRESENT ALL STUDIES ASKING A SIMILAR QUESTION. AND SO ALL META ANALYSES NEED TO BE PART OF A SYSTEMATIC REVIEW. BUT SYSTEMATIC REVIEWS MAY NOT INCLUDE A META ANALYSIS, IF, FOR EXAMPLE, THE SYSTEMATIC REVIEWERS HAVEN'T FOUND ENOUGH STUDIES THAT ARE SIMILAR ENOUGH TO MERIT A META ANALYSIS OR OUTCOMES THAT ARE SIMILAR ENOUGH. AND WE'LL TOUCH ON THAT A LITTLE BIT LATER. SO THERE ARE FORMAL STEPS IN A SYSTEMATIC REVIEW. THE FIRST S GNAW REALLY NEED TO WORK WITH A TEAM. SYSTEMATIC REVIEWS ARE STUDIES THAT INVOLVED PEOPLE WHO ARE SUBJECT MATTER EXPERTS, METH LOGIC EXPERTS SOME PATIENTS OR CONSUMERS SHOULD BE INVOLVED. YOU NEED A TEAM. YOU SHOULD IF HE EVER DO IT ALONE AND I TELL EVERYWHERE SYSTEMATIC REVIEWS ARE THE HARDEST RESEARCH STUDIES I'VE EVER DONE, BECAUSE THEY INVOLVE JUDGMENT EVERY STEP OF THE WAY, EVEN THOUGH THEY HAVE A FORMALIZED METHOD TO THEM. THEN UP WRITE A PROTOCOL. UNLIKE TRADITIONALLY REVIEW ARTICLE, THAT PROTOCOL CAN BE PUBLISHED SHOWING PEOPLE WHAT YOU INTENT TO DO, AND OPEN TOGETHER TO CRITICISM, IF THERE IS ANY. THIS IS WHERE YOU SAY WHAT YOUR QUESTION IS WHAT THE ELIGIBILITY CRITERIA ARE, HOW YOU WILL DO YOUR SEARCH, HOW YOU WILL EXTRACT DATA AND DO YOUR QUALITY OR RISK OF BIAS ASSESSMENT. THE SITH SIS THAT YOU'LL DO, ET CETERA. SO THE PROTOCOL IS VERY PERSON. IT'S A TON OF WORK BUT IT ACTUALLY MAKES THE NEXT PART, THAT IS, DOING YOUR REVIEW, LESS WORK, AND IT GIVES AN OPPORTUNITY FOR OTHERS TO COMMENT ON WHAT YOU SAY YOU'RE GOING TO DO THIRD STEP IS TO COLLECT THE DATA AND IN A SYSTEMATIC REVIEW, THIS IS USUALLY DOG A RESEARCH OF THE LITERATURE AND UNPUBLISH SOURCES IF POSSIBLE. THEN YOU A PHRASE WHAT YOU FOUND. YOU LOOK AND SEE WHAT MEETS YOUR ELIGIBILITY CRITERIA. YOU LOOK FOR RISK OF BIAS IN THE INDIVIDUAL STUDIES, ET CETERA. SO YOU'RE LOOKING AT THESE STUDIES TO SAY, SHOULD I REALLY BE INCLUDED THEM? ARE THEY GOOD ENOUGH. WHAT'S THE QUALITY OF THE DATA OR HOW LIKELY ARE THE DATA TO BE AFFECTED BY BIAS. WAS THE CONTROL GROUP THAT WAS CHOSEN, APPROPRIATE, ET CETERA. STEP 5. I SHOULD HAVE SAID 5 HERE. SYNTHESIZE SAY QUALITATIVE SYNTHESIS OF RESULTS. THIS IS WHERE, THIS IS, I WOULD SAY, THE HARDEST PIECE TO DEFINE. YOU TALK ABOUT THE RESULT YOU FOUND, AND SAY, WELL, ALL THE STUDIES WE FOUND WERE QUITE SMALL. AND THEY HAD QUITE DIFFERENT CONTROL GROUPS. THERE WAS A MIXTURE OF CASE CONTROL AND COHORT STUDIES AND WE WERE UNSURE WHETHERY WE WANTD TO MIX THOSE SO WE ANALYZED THOSE SEPARATELY. ET CETERA. YOU TALK ABOUT HOW THE DESIGNS OF THE STUDY, AND WHAT YOU FOUND COULD BE AFFECTING YOUR FINDINGS. THEN IN STEP 6. YOU ANALYZE THE DATA THIS IS A QUANTITATIVE SYNTHESIS, THAT MIGHT USE META ANALYSIS, ASSUMING THAT CAN BE DONE. STEP 7. AND I CAN SEE, I HAVE TYPOS WITH THESE STEPS ALL OVER THE PLACE. YOU INTERPRET YOUR DATA AND ASSESS THE LIMITATIONS OF YOUR ANALYSIS. THEN YOU WANT TO KEEP YOUR SYSTEMATIC REVIEW UP TO DATE. THIS IS VERY IMPORTANT. WHAT IF YOU JUST PUBLISH UNTILLED A JOURNAL, YOUR SYSTEMATIC REVIEW IF 2009. WOULD A PERSON LOOKING AT IT IN 2016, SAFELY ASSUME THAT NOTHING'S CHANGED SINCE 2009. FOR SOME SUBJECTS, THIS MIGHT BE TRUE AND OTHERS, IT'S UNLIKELY TO BE TRUE. TO BE REALLY UP TO DATE AND GIVE PATIENTS THE BEST POSSIBLE CARE, ONE NEEDS TO UPDATE A SYSTEMATIC REVIEW, AND THEORETICALLY, IT'S EASIER THAN THE INITIAL REVIEW BECAUSE YOU HAVE THE PROTOCOL, ET CETERA, AND YOU KNOW WHAT QUARE GOING TO DO OFTEN IN A HOT AREA, THERE COULD BE A LOT OF BRAND-NEW STUDIES AND IT CAN BE HARD. SO META ANALYSIS. THE QUANTITATIVE SYNTHESIS THAT I DESCRIBE IS VERY HELPFUL SOMETIMES. IT CAN HELP YOU ASSESS HOW STRONG THIS EVIDENCE IS, AND WHETHER THIS EFFECT APPEARS TO BE REAL OR NOT, AND WHAT DIRECTION IT'S GOING. IT ALLOWS YOU TO OBTAIN A SINGLE SUMMARY RESULT. SO SMOKING AFFECTS HEART ATTACK, EXACTLY, PUTS YOU AT THIS RISK. YOU COULD GET A SUMMARY RESULT AND ALLOWS YOU TO INVESTIGATE THE HETEROGENEITY, AMONG THE RESULTS OF THE STUDY. SO WHY ARE THEY SO DIFFERENT. YOUR HETEROGENEITY, THE QUANTITATIVE ASSESSMENT, CAN GIVE YOU, ALTHOUGH IT'S QUITE UNDER POWER, SOME SORT OF FEEL FOR WHAT THE HETEROGENEITY OF THE DIFFERENT RUTS IS SO I MENTIONED A META ANALYSIS, ANDS IN AN EXAMPLE OF A META ANALYSIS. MOST PEOPLE HAVE SEEN THESE. IT'S CALLED A FOREST PLOT AND ON THE LEFT SIDE, YOU WOULD SEE THE FIVE INDIVIDUAL STUDIES AND THE HORIZONTAL LINE ACROSS ACROSS, AND THE POINT ESTIMATE GOES ACROSS A RISK RATIO OF IDENTITY. ONE. SO IN THIS PARTICULAR EXAMPLE, THERE'S NO EVIDENCE OF EFFECT AND YOU CAN SEE IN THE RED DIAMOND AT THE BOTTOM, THE CONFIDENCE INTERVAL IS QUITE NARROW. IT'S FIVE STUDIES, PULLED TOGETHER IN A QUANTITATIVE SIN SITH IS. EACH SIDE INDIVIDUALLY HAS A RISK RATIO OF 1. BUTT CONFIDENCE OF INTERVALS CAN BE QUITE WIDE, IN AT LEAST FOUR OF THE STUDIES AND SO WE DON'T KNOW WHETHER THERE MIGHT BE A HARMFUL OR BENEFICIAL EFFECT. IT DOES TELL A LITTLE BIT OF THE QUANTITATIVE AS ASPECTS OF EACH OF THE FIVE INDIVIDUAL STUDIES. SO THERE ARE MANY REPORTS OUT THERE THAT SUMMARIZE THE KNOWLEDGE THAT WE HAVE FROM RESEARCH IN THEIR REVIEWS BUT THEY MAY NOT BE SYSTEMATIC REVIEWS. SO WHAT I'M GOING TO GO OVER HERE, WHAT WE'RE SEEING MORE AND MORE IN THE LITERATURE IS REVIEWS THAT EITHER CALL THEMSELVES SYSTEMATIC REVIEWS OR CALL THEMSELVES SOMETHING DIFFERENT AND IT'S NOT CLEAR DO MANY PEOPLE. WHETHER THERE ARE SYSTEMATIC REVIEWS OR NOT. WE CAME ACROSS, LOOKING FOR SYSTEMATIC REVIEWS WHERE WE WEREN'T SURE. IS THIS A SYSTEMATIC REVIEW OR NOT. IT LOOKS LIKE A GENOME WIDE ASSOCIATION STUDY, AND IF YOU LOOK AT THE ABSTRACT, YOU CAN SEE THAT IT JUST LOOKS LIKE A COHORT STUDY. WE'RE NOT SURE. BUT I BLOCKED OFF THIS THIRD SENTENCE THAT SAYS, WE PERFORMED A GENOME WIDE ASSOCIATION STUDY OF INTEROCULAR PRESSURE. BUT IF YOU GO DOWN, DOESN'T SEEM LIKE A SYSTEMATIC REVIEW YOU SEE THEY DID A META ANALYSIS. SO UH-OH. NOW, I'M WORRIED. IS IT A SYSTEMATIC REVIEW OR NOT? I DON'T HAVE AN ANSWER. YOU KNOW, THERE'S NOT ONE CORRECT ANSWER. NOOSE A CENSUS IN THE COMMUNITY ABOUT WHAT CONSTITUTE ACE SYSTEMATIC REVIEWS AND ARE THESE OTHER TYPES OF REVIEWS WHERE PEOPLE ARE IN MORE OF A HURRY THAN YOU ARE OF A SYSTEMATIC REVIEW AND HOW GOOD ARE THEY. ARE THEY LIKE A TRADITIONAL REVIEW OR ARE THEY CLOSER TO A SYSTEMATIC REVIEW. I DON'T HAVE AN ANSWER FOR YOU, BUT I WANT TO CALL IT TO YOUR ATTENTION, THIS IS AN OUTSTANDING QUESTION. THAT NEEDS TO BE RESOLVED SO WHAT I SAY HERE, THIS ARTICLE REPORTS A META ANALYSIS, BUT IS IT A SYSTEMATIC REVIEW. AND REMEMBER, ALL META ANALYSIS NEED TO BE PART OF A SYSTEMATIC REVIEW. SO ONE MIGHT SAY WHY BOTHER WITH A SYSTEMATIC REVIEW. WHAT I HAVEN'T TOLD SUTHAT SYSTEMATIC REVIEWS TAKE A TON OF TIME. WE ESTIMATE THEY TAKE ABOUT TWO YEARS, IF YOU'RE LUCKY, GIVEN EVERYTHING ELSE WE'RE DOING. EVEN IF THEY ARE FUNDED, THEY TAKE A LONG TIME. EACH OF THOSE STEPS TAKES A LONG TIME. IT'S A BIG JOB. IT'S AN OBSERVATIONAL STUDY AND SO A LOT OF GROUPS OUT THERE SAY, WELL, WE DON'T HAVE TIME OR THE RESOURCES TO DO A SYSTEMATIC REVIEW SO WE'RE JUST GOING TO DO A NARRATIVE REVIEW OR A SCOPING OR RAPID REVIEW. ALL OF THESE DIFFERENT TYPES OF REVIEWS. SO THEY USE FEWER RESOURCES. THAT'S TRUE. BUT THE QUESTION IS DO THEY GIVE YOU THE QUOTE RIGHT UNQUOTE, ANSWER. MANY EXERTS IN A FIELD BELIEVE THEY KNOW THE LITERATURE SO THEY CAN AVOID DOING A FULL SYSTEMATIC REVIEW BUT THAT ISN'T TRUE, ESPECIALLY WHEN OTHER LANGUAGES ARE IMPORTANT IN IDENTIFYING ALL THAT IS KNOWN. SO THERE HAVE BEEN SOME STUDY THAT IS LOOK AT DIFFERENT WAYS OF SITH SIDING AND SUMMARIZE WHAT IS KNOWN IN IT ARTICLE BY ANDREA TRICO THIS YEAR, SHE DID A WORD CLOUD OF THE MOST FREQUENT METHODS THAT ARE USED OUT THERE FOR EXAMPLE, A META ETHNOLOGY AGGRESSIVEY AND META SYNTHESIS. MANY OF THESE THAT ARE USED LATERAL VIOLENCE, YOU'LL SEE IN THE NURSING LITERATURE THERE ARE STANDARDS ON HOW TO CONDUCT, AND REPORT A SYSTEMATIC REVIEW. SO FAR, WE DON'T HAVE THE SAME KINDS OF STANDARDS FOR THESE OTHER TYPES OF REVIEWS. SO THOSE ARE THE COCHRAN STANDARDS. MESSYER IS WHAT IT'S CALLED. BECAUSE IT IS VERY MESSI, BUT IT'S HARD TO DO A GOOD SYSTEMATIC REVIEW. AND YOU CAN LOOK IN THE COCHRAN LIBRARY, AND SEE COCHRAN REVIEWS THAT HAVE ADHERED TO THE STANDARDS. THERE IS AN OVER ARCHING EDITORIAL GROUP AND THEY WILL MAKE SURE EACH OF THE GROUPS ADHERES TO THESE STANDARDS. THE INSTITUTE OF MEDICINE, THE NATIONAL ACADEMY, ALSO PROVIDES STANDARDS FOR DOING SYSTEMATIC REVIEWS IN A BOOK YOU CAN GET FOR FREE. IT'S ONE ON THE LEFT. BY GOING TO THE WEB OR YOU CAN PURCHASE IT. EITHER WAY. BUT THIS ALSO GIVES STANDARDS FOR DOING SYSTEMATIC REVIEWS AND GUIDELINES IF YOU WANT TO DO A SYSTEMATIC REVIEW OR A SHORTER VERSION. WHICH STEP WILL YOU OMIT? THAT'S THE QUESTION. SO REMEMBER, WE HAVE THESE STEPS, WHERE I GOT THE NUMBERING ALL WRONG. THEY STILL ARE. I'VE SHORTENED THEM FOR HOW LONG THEY REALLY TAKE. BUT WHICH STEP WOULD YOU OMIT? WHAT I'M SEEING IN THE LITERATURE I READ, WHICH HAS A LOT TO DO WITH THE SCOPING ROW VIEWS AND NARRATIVE REVIEWS, PEOPLE ARE DOING PRETTY GOOD SEARCHES THAT COLLECT DATA STEPS, BUT THEY AREN'T APPRAISING THE STUDIES THAT THEY FIND FOR RISK OF BIAS. SO SO THIS TENDS TO BE THE STEP THAT THEY ARE TARTOWSKY LEAVING OUT, AND ALSO, UPDATING REVIEWS. SO THERE IS A PROBLEM. WE USE COCHRAN METHODS AND THEY AREN'T COCHRAN REVIEWS. BEWARE OF THOSE BECAUSE THEY DIDN'T NECESSARILY USE COCHRAN METHODS. SO WHAT TYPE OF BIAS AM I TALKING ABOUT? THERE ARE TWO. THE FIRST ARE THE STUDY THAT IS YOU IDENTIFY THAT FIT YOUR CRITERIA, COULD HAVE A BIAS IN THE METHODS THEY USE. SO THIS IS ABEXAMPLE FROM A ABDOMEN ONLYIZED CLINICAL TRIAL. YOU PROBABLY LEARNED IN SCHOOL, IF YOU DON'T PRACTICE CLINICAL TRIALS, THERE CAN BE BIAS IN SELECTION BIAS WHICH MEANS WHETHER A PERSON IS ASSIGNED TO A CONTROL GROUP OR INTERVENTION GROUP, THE WAY A RANDOMIZED TRIAL MITIGATES THE SELECT BIAS ISSUE IS WITH RANDOM ALLOCATION. THIS IS, OF COURSE, SOMETHING THAT IS A CONTINUING PROBLEM FOR OBSERVATIONAL STUDIES INFORMATION BIAS, THAT IS, MISCLASSIFICATION OF THE DATA RECEIVE FROM PEOPLE, FOR EXAMPLE, THAT CAN HELP IN USING A MEDICAL RECORD OR JUST HOW WE ALL INTERPRET THINGS AS METHODOLOGIES OR PATIENTS OR CARRIERS. THE WAY A RANDOMIZED TRIAL WOULD TRY TO MITIGATE THE RISK OF BIAS FROM INFORMATION BIAS IS THROUGH MASKING OR BLINDING. SO THAT YOU DON'T KNOW WHICH INTERVENTION YOU'RE GETTING. FINALLY, THERE CAN BE BIAS IN THE ANALYSIS. WE SAY WE'LL DO INTENTION TO TREAT ANALYSIS, WITH PREDEFINED OUTCOMES. DON'T ANALYZE PEOPLE IN THE FIRST ANALYSIS. BY THE TREATMENT THEY ACTUALLY RECEIVE, BUT ANALYZE THEM AS THE TREATMENT THAT THEY WERE ASSIGNED TO. DOING A SYSTEMATIC REVIEW OF OBSERVATION STUDIES AND TRYING TO ADDRESS THE BIAS ISSUE IS MUCH MORE DIFFICULT THAN A RANDOMIZED CONTROL TRIAL AND THERE'S A WAY TO ASSESS BIAS. THIS IS CALLED ACROBAT. THERE'S ALSO THE NEW CASTLE IT'S SCALE FOR ASSESSING OBSERVATION STUDIES BUT YOU HAVE GOT THE SAME BASIS PROBLEMS I WON'T GO INTO ANY DETAIL OF HOW TO ASSESS THIS, BUT THERE IS A LITERATURE OUT THERE. IT'S VERY DIFFICULT IT HAS TO BE DONE IN A PROPER, SYSTEMATIC REVIEW. THE SECOND WAY BIAS CAN BE INTRODUCE FEDERAL A SYSTEMATIC REVIEW IS SOMETHING WITH A NEW WORD. META BIAS. I DON'T USUALLY LIKE NEW WORDS BUT IN THIS CASE, OUR STUDENTS AND CLASSES ARE GETTING CONFUSED BETWEEN THE BIAS IN THE STUDIES THAT ARE INCLUDED IN THE SYSTEMATIC REVIEW AND THE BIAS THAT OCCURS BY THE WAY THE SYSTEMATIC OCCURS. HERE, WE HAVE SELECTION BIAS, INFORMATION BIAS AND BIAS IN THE ANALYSIS. I'M ONLY GOING TO COVER ONE OF THE META BIASES, AND THAT IS REPORTING BIAS. AT LEAST FOR BETWEEN YEARS, EVEN TRYING TO ADDRESS WITH RANDOMIZED TRIALS, AND I WOULD SAY, WE'RE GETTING CLOSE NOW TO MAYBE SOLVING THE PROBLEM BUT IT WILL PROBABLY NEVER BE COMPLETELY SOLVED. MAYBE I'M SOUNDING TOO NEGATIVE. BUT REPORTING BIAS IS A HUGE PROBLEM IN KNOWING WHAT THE TRUTH IS IN TERMS OF THE RESULTS, THE KNOWLEDGE THAT'S OUT THERE. WHAT IS REPORTING BIAS? THERE ARE LOTS TYPES. THE MAIN ONCE WE'RE TALKING ABOUT IS PUBLICATION BIAS. THAT IS, THAT WHAT REMAINS THE WHOLE STUDY THAT REMAINS UNPUBLISHED, PROBABLY BECAUSE OF THE RESULTS SO A PUBLICATION PUBLICATION BIAS IS UNPUBLISHED RESULTS BECAUSE OF PUBLICATION STUDIES. YOU'RE YOU'RE DOING A SYSTEMATIC REVIEW IN WHETHER ASPIRIN IS EFFECTIVE IN PREVENTING HEART ATTACK. IF OUR PUBLISHED STUDIES ARE ALL POSITIVE RESULTS, AND OUR UNPUBLISHED STUDIES ARE NOT POSITIVE RESULTS. YOU CAN SEE THE SYSTEMATIC REVIEW. DEPOSTI ON JUST UNPUBLISHED STUDIES, WOULD INCORRECTLY, OVER STATE THE USEFULNESS OF ASPIRIN IN PREVENTING HEART DISEASE. SO THE OTHER TYPE OF MAIN REPORTING BIAS, WHAT'S CALLED SELECTIVE OUTCOME REPORTING. I'LL TELL YOU, THAT WAS DESCRIBED IN FULL, AND REALLY PAID ATTENTION TO IN THE LAST 10 OR 15 YEARS. WHAT THIS IS, IS, THE STUDY ITSELF MAY BE PUBLISHED BUT MAYBE SOME OF THE OUTCOMES AREN'T REPORTED. IT MAY BE THAT OUTCOMES WERE REPORT THE FOR 6 MONTHS BUT NOT 12 MONTHS OR WORKS YEARS BECAUSE MAYBE THOSE RESULTS WEREN'T WEREN'T STATISTICALLY SIGNIFICANT. OR MAYBE THEY REPORTED SIGNATURE STATISTICAL P-VALUES AM SO THESE ARE REALLY BIG PROBLEMS OR CHALLENGES, READING TO DOING A SYSTEMATIC REVIEW. AND SOMETHING, PEOPLE ARE SPENDING A LOT OF TIME ON. ME, INCLUDED. SO RELATED TO THE SLEEKED, THE SELECTIVE OUTCOME REPORTING AND OTHER TYPES OF REPORTING BIASES IS WHERE DO WE GUILTY INFORMATION FOR TRIALS. OBSERVATIONAL STUDIES AREALLY DIFFERENT. BUT NOT SO DIFFERENT. I JUST THINK THEY AREN'T AS FAR LONG IN THE ANALYSIS, THE METHOD LOGIC ANALYSIS. SO THERE'S PUBLIC REPORTING, WHICH COULD BE A CONFERENCE ABSTRACT. JOURNAL ARTICLES, WE HAVE TRIAL REGISTRY, AND CLINICAL TRIALS.GOV AND W.H.O. HAS A REGISTER OF REGISTERS. SOME OBSERVATION STUDIES ARE THERE AS WELL. BUT THERE ISN'T THERE ISN'T A LOT LOOT TO HELP YOU ALONG. WE HAVE FDA REVIEWS. CAN YOU GO TO THE FDA WEBSITE AND LOOK AT DRUGS AT FDA. SORRY, I HAVE TRIAL REGISTRATION TWICE. I MUST HAVE BEEN THINKING OF W.H.O., THE REGISTER OF REGISTERS, AND CAN YOU ACTUALLY LOOK AT STUDY PROTOCOLS AND AT THE ANALYSIS PLAN, AND SEE WHAT WAS PLANNED. WE KNOW, PERHAPS, THEY'VE BEEN RELEASED DURING LITIGATION. WHAT'S IN THE NONPUBLIC SOURCE. OFTEN DOESN'T AGREE WITH WHAT'S IN THE PUBLIC SOURCE. THAT'S NOT TO SAY, ALL PUBLIC SOURCES AGREE WITH ONE ABOUT THE SAME REAL. BUT NONE PUBLIC CAN GIVE YOU A LOT MORE INFORMATION AND DISAGREE WITH WITH WHAT'S IN THE PUBLIC DOMAIN. SO THERE'S INCREASINGLY, A DEMAND FROM MAKING DATA FROM A CLINICAL TRIAL OPEN ACCESS, AND FROM ALL STUDIES, OBSERVATION OR EXPERIMENTAL OPEN ACCESS. SO THAT EVERYBODY CAN CHECK WHAT YOU GET AND DO ADDITIONAL ANALYSIS. SO IN THE INTEREST OF TIME, I'M GOING TO SKIP THE FIXTURE COUPLE OF SLIDES. YOU CAN READ THE RESULTS IN THIS ARTICLE. BUT IT SHOWS HOW AN EXAMPLE, AN EXTREME EXAMPLE OF SELECTIVE OUTCOME REPORTING, WHERE THE DATA, NOT ONLY WHERE THE OUTCOMES CHANGE, BETWEEN THE PROTOCOL AND WHAT WAS REPORTED ABOUT HALF THE TIME, BUT ALSO, WHAT WAS REPORTED, WERE THE STATISTICALLY SIGNIFICANT P-VALUES IF WHAT WAS NOT REPORT THE, WERE P-VALUES THAT TENDED TO BE NOT STATISTICALLY SIGNIFICANT. THERE WAS A CASE OF SELECTIVE OUTCOME REPORTING THEN AND ALSO, SOME CASES OF PUBLICATION BIAS. SO THIS PAPER HAS A LOT OF MEET TO IT, AND IT MIGHT BE WORTH READING IF THIS IS AN INTEREST OF YOURS. I'M GOING TO SKIP THROUGH THESE SLIDES BUT THE DATA ARE OUT THERE. SO BECAUSE OF SELECTIVE OUTCOME REPORTING AND ESPECIALLY FAILING TO REPORT SOME OUTCOMES WHERE YOU REPORT OTHERS, PERHAPS RELATED TO THE RESULTS OR, AND I'M NOT GOING TO TALK ABOUT THIS MUCH, BUT THE PROBLEM THAT TRIALS REPORT SO MANY DIFFERENT OUTCOMES AND SYSTEMATIC REVIEWS, TOO. SO THAT YOU CAN HAVE 400 TRIALS, AND HALF OF THE OUTCOMES MEASURED, WERE INN THE REPORTED ANYWHERE, EXCEPT ONE TRIAL. NOW, THAT MAKES IT REALLY HARD TO DO A SYSTEMATIC REVIEW AM THAT'S A SELFISH VIEW OF MINE, BUT IT ALSO ISN'T VERY HELPFUL THAT THE PATIENT OR DOCTOR, IF ALL THESE OUTCOMES ARE DIFFERENT. SO WHAT'S BEING ADVOCATE SIDE A CORE OUTCOME SET. IT'S NOT TO SAY, YOU COULDN'T MEASURE OUTCOMES FOR YOU OR THE PATIENT OR ANYBODY PARTICIPATING IN THE TRIAL. BUT AT LEAST EVERYONE AGGRESS, THERE'S A COURSE OF OUTCOMES THEN THEY CAN BE COMBINED IN A SYSTEM META ANAL SAY. COMMENTS, GROUPS, IT DOESN'T DO THEM ITSELF. IT'S A VEX OF WHO'S DOING IT, AND ONE OF THE GROUPS, I HAVEN'T TALKED ABOUT THEM HERE. LET'S SEE. MY ANIMATION MUST NOT BE WORKING BUT YOU CAN SEE PERMANENT, WHICH IS THE PURPLE AND GROOVE HEALTH MEASURES. AND THIS IS THE NIH INITIATIVE CALLED PROMISE WHERE THEY'RE TRYING TO STANDARDIZE MEASURES ACROSS DIFFERENT TOPICS SO THAT IS ONE OF THE CORE OUTCOME SETS THAT'S BEEN DEVELOPED. SO WHO'S DOING SYSTEMATIC REVIEWS? YOU'LL SEE THEIR SYSTEM AT SYSTEMATIC REVIEWS OF ALMOST ANYTHING YOU CAN IMAGINE REMEMBER YOU MAY OR MAY NOT KNOW. REVIEWS WILL INCREASE THE IMPACT FACTOR FOR A JOURNAL. SO JOURNALS LOVE REVIEWS OF ANY TYPE. AT A DON'T HAVE TO BE SYSTEMATIC BUT SYSTEMATIC REVIEWS WOULD INCREASE THEIR IMPACT FACTOR BECAUSE THEY GET CITED A LOT. SO INDEPENDENT AUTHORS ARE ALSO INTERESTED IN DOING SYSTEMATIC REVIEWS. SOME, BECAUSE THEY REALLY ARE GENUINELY INTERESTED AND SOME BECAUSE THEY THINK IT'S A FAST WAY OF GETTING A PUBLICATION. IT'S NOT AT ALL FAST WHEN YOU DO IT WELL. BUT THAT'S NOT WHAT PEOPLE THINK. THE CONFERENCE COLLABORATION, I'LL TALK ABOUT BRIEFLY IN A FEW SLIDES IT'S AN INTERNATIONAL EFFORT OF ABOUT 30,000 PEOPLE 84 TRYING TO DO SYSTEMATIC REVIEWS ACROSS ALL OF HEALTH AND HEALTH CARE. POLICY MAKERS, IN MANY COUNTRIES ARE DOING SYSTEMATIC REVIEWS THEY WANT TO SEE WHAT WORKS BECAUSE THEY WANT TO PRODUCE GUIDELINES, AND RECOMMEND WHAT SHOULD BE PAID. AND BUSINESSES ARE DOING A LOT SYSTEMATIC REVIEWS. A COMPANY MAY WANT TO KNOW, SHOULD WE PURSUE THIS DRUG. WHAT'S THE EVIDENCE OUT THERE ALREADY, AND IS THIS WORTH PURSUING. IF YOU'RE TALKING ABOUT KNOWLEDGE, THE SYSTEMATIC REVIEW IS BETWEEN THE CENTER, EVIDENCE GENERATION, THAT IS, DOING THE ORIGINAL PRIMARY STUDIES, AND POLICY WHERE YOU WOULD HAVE GUIDELINES OR APPLICATION TO MAKE HEADACHE. SO THAT'S WHERE WE ARE. NOW, ONE THING THAT HAS BEEN SUGGESTED ACCIDENT AT LEAST IN THE ... SHOULD FUNDERS REQUIRE APPLICANTS TO REFER TO A SYSTEMATIC REVIEW EITHER THEIR OWN OR SOMEBODY ELSE'S. TO SHOW, THAT WHAT THEY'RE PROPOSING NEEDS TO BE DONE. THERE REALLY SAN EVIDENCE GAP. AND YOU CAN SEE THE NIHR. THE MEDICALS RESEARCH, YES, THEY DO REQUIRE A SYSTEMATIC REVIEW OF EVIDENCE. BUOTHERS REQUIRE A PARTIAL SYSTEMATIC REVIEW. SO JUST TO SHOW YOU SOME SYSTEMATIC REVIEW, HERE'S ONE ABOUT PREVALENCE, AND ELEVATED RISK. HERE'S LOOKING AT THE ASSOCIATION BETWEEN MORTALITY AND OVER WEIGHT. THE COCHRAN LIBRARY HAS A PUBLIC HEALTH GROUP THAT LOOKS AT TABLE WEAR IN PORTION SIZE FOR CHANGING SLEEKED AND CONSUMPTION OF FOOD. HERE, THEY'RE LOOK AT HEALTH SYSTEMS, TELEMEDICINE AND IMPROVING METHODS -- HERE'S THE COCHRAN REVIEW ON WORKPLACE INTERVENTIONS FOR REDUCING SITTING AT WORK. I NEED TO REALLY TAKE SOME OF THIS ADVICE. I AT MY COMPUTER A LOT. AND HERE IS JUST IN THE PROTOCOL STAGE. THIS REVIEW ISN'T FINISHED YET. PROTOCOLS ARE PUBLISHED SO THEY CAN GET COMMENTS. IMPROVE INHALER TECHNIQUES FOR PEOPLE WITH ASTHMA. EDUCATION, JUDICIAL SYSTEM AND SOCIAL SCIENTISTS IFS IT REVIEWS REVIEWS. MATH BOOKS ARE NEVER UP TO DATE. YOU MY WANT TO LOOK TUP. IN THE CLEARING HOUSE TO SEE WHAT THE EVIDENCE ACTUALLY S. THE E.P.A., ENVIRONMENTAL PREVENTION AGENCY IS DOING SYSTEMIC REVIEWS TO TRY TO DETECT HOW MUCH HOW MUCH OF VARIOUS TOXINS ARE OKAY TO RELEASE INTO THE IT'S IT'S FEAR OR WATERS, AND WHETHER THE LEPTS SHOULD BE ALLOWED AND ALL KINDS OF PEOPLE ARE USING SYSTEMATIC REVIEWS AS WELL. I ALWAYS LOOK IF MY MORNING NEWSPAPER TO, SEE IF THE JOURNALISTS HAS CITED A SYSTEMATIC REVIEW OR A SINGLE PAPER. OTHER NEWSPAPERS MAY BE LESS GOOD, BUT THEY'RE ALL COMING AROUND. IT'S PRETTY EXCITING. SO THE NATIONAL LIBRARY OF MEDICINE HAS SOMETHING CALLED PUB MED HEALTH. AND YOU CAN GO TO PUB MED HEALTH AND IT WILL SHOW YOU THE REVIEWS OF EVIDENCE. EVERYBODY'S REVIEWS. INDEPENDENT UPON REVIEWS. HRQ'S. COCHRAN'S, EVERYBODY'S, IF THEY'RE HIGH QUALITY. SO THAT'S A REALLY GOOD SOURCE TO FIND OUT REVIEWS RELATED TO HEALTH. JUST A FEW LAST MINUTE THINGS, BECAUSE MY TIME IS UP. BUT THERE ARE THINGS CAN YOU DO WITHIN A SPECIFIC FIELD TO ENSURE THE QUALITY OF PUBLISHED SYSTEMATIC REVIEW. IN GENERAL, I WOULD SAY, AND THERE HAVE BEEN A NUMBER OF PAPERS ON THIS, THE QUALITY IS STILL NOT GREAT: IN THE OP THAT MOLOGY, AND OPTOMETRY FIELD WHERE WE WORK. WE HAVE RELAYED EDITORS AT EYES AND VISION JOURNALS WHERE THE EDITORS ARE WILLING AND THEY LOOK AT ALL SYSTEMATIC REVIEW THAT IS COME IN. THE EDITORS WANT TO BE ABLE TO PUBLISH THEM, BUT THEY ONLY WANT TO PUBLISH GOOD STUFF. ARCH THE FIRST YEAR, I THINK THE AUTHORS GET USED TO WHAT IS REQUIRED. FOR EXAMPLE, HERE'S THE INSTRUCTIONS FOR AUTHORS IN OPHTHALMOLOGY THAT LEE OVER ON THE RIGHT, HAS BEEN EDITING THE SYSTEMATIC REVIEW SECOND FOR FOUR, FIVE YEARS AND THEY PUT TOGETHER GOOD STRUCKS FOR AUTHORS. SO CAN YOU WORK WITH A FIELD AND GET THE FIELDS PUBLISH BETTER SYSTEMATIC REVIEWS THE GROUP SETTING GUIDELINES, ALSO CAN HELP WITH USING SYSTEMATIC REVIEWS AS A BASIS FOR THEIR GUIDELINES, AND CAN YOU USE IT AS SYSTEMATIC REVIEWERS. WE DO IT AS THE COCHRAN EYES AND VISION GROUP, SETTING THE PRIORITIES FOR DOING SYSTEMATIC REVIEWS. I WON'T GO THROUGHOUT PROCESS BECAUSE IT TAKES TOO MUCH TIME. THIS IS TO DESCRIBE IN AN INTERNAL MEDICINE ARTICLE. HIGHWAY YOU WOULD GO ABOUT AND DO THIS. STARTING WITH THE THE GUIDELINE, BECAUSE THESE ARE OBVIOUSLY, RECOMMENDATIONS THAT MATTER TO CLINICIANS, AND WORK BACKWARDS FOR AN AGENDA FOR DOING SYSTEMIC REVIEWS I HOPE THEY COULD USE A LITTLE EXTRA TRAINING AND IT DOESN'T HAVE TO COST MONEY THERE ARE THESE MASSIVE ONLINE COURSES. YOU CAN ALWAYS PAY AND TAKE COURSES ON HOW TO DO A GOOD SYSTEMATIC REVIEW. THERE'S A MASSIVE, OPEN ONLINE COURSE THROUGH COURSE SERA THAT DR. LEE ORGANIZED AND IT'S FREE. IT'S ENTIRELY FREE OF CHARGE. IT TAKES SIX WEEKS, IT'S ONLINE, NOBODY CEASE HOW YOU DO. THERE ARE QUIZES AND YOU LEARN HOW TO DO A SYSTEMATIC REVIEW. IT'S NOT HANDS ON. MOST PEOPLE DOING SYSTEMATIC REVIEWS, BELIEVE THAT HANDS ON IS HOW YOU'RE GOING TO GET THE BEST PRACTICE FOR DOING A SYSTEMATIC REVIEW, BUT IT'S A STARTING POINT AND IT'S AN EDUCATION. BETTER TO TAKE IT THAN NOT. SO IN SUMMARY, EVERYBODY NEEDS FORMAL TRAINING AND MENTORING. AND FINALLY, THERE ARE SYSTEMATIC REVIEWS THAT ARE STRANDS PARENT AND GOOD ONCE ADHERE TO THE STANDARDS I MENTION. A LOT OF GROUPS ARE DOING SYSTEMATIC REVIEWS, TAKING ON THESE SHORTER ONCE, LIKE SCOPING REVIEWS AND LANDSCAPER VIEWS BUT THEY HAVEN'T UNDERGONE SCRUTINY. THERE AREN'T STANDARDS FOR THEM. THEY OFTEN DON'T INCLUDE A RISK OF BIAS ASSESSMENT, AND SO THEY ARE WORRISOME AND STAY TUNED, AND ESPECIALLY PARTICIPATE IN DOING RESEARCH ON THEM. IT WOULD BE GREAT IF WE CAN SHORT INTHIS PROCESS. LOTS OF PEOPLE ARE USING AND PRODUCING SYSTEMATIC REVIEWS FOR DIFFERENT THINGS, INCLUDING POLICY MAKING, FUNDING, DIFFERENT HEALTH GUIDELINES, AND I DID MENTION THAT COCHRAN IS AN INTERNATIONAL COLLABORATION. ANYONE CAN PARTICIPATE, I'M AT THE U.S. COCHRAN CENTER. IT WAS SAID BY DAVID AT THE BEGINNING, AND WE'RE HAPPY TO HELP YOU GET INVOLVED IF UP THE TO. YOU AND I'D LIKE TO ACKNOWLEDGE NIH, THE NATIONAL EYE INSTITUTE FUNDS THE EYE AND VISION GROUP. THE NICHD FUNDS THE NEONATAL GROUP AND COMPLEMENTARY MEDICINE FUNDS THAT FIELD WITHIN COCHRAN AND LOOKS AT COMPLEMENTARY MEDICINE AND HEALTH INTERVENTION. SO NIH HAS BEEN TREMENDOUSLY SUPPORTIVE OVER THE YEARS. SO THAT'S ALL I HAVE, AS FAR AS SLIDES GO, AND I'M HAPPY TO ANSWER YOUR QUESTIONS. I'M SORRY I RAN A LITTLE LATE. >> ALL RIGHT. THANK YOU, DR. DICKERSON, VERY MUCH, FOR THAT PRESENTATION. ONE OF THE QUESTIONS THAT MANY PEOPLE ARE GOING TO BE WONDERING ABOUT IS THEY'RE THINKING ABOUT DOING SYSTEMIC REVIEW SYSTEM HOW THEY GET SUPPORT FOR IT. MOST FOLKS 84 DOING BIOMEDICAL RESEARCH NEED FOOTAGE SUPPORT TO DO THAT RESEARCH AND DOINGIC REVIEW SYSTEM PROBABLY NO DIFFERENT. THEY TAKE TIME AND ENERGY AND EFFORT. WHAT ARE THE POSSIBLE SOURCES OF SUPPORT FOR INVESTIGATORS WHO WANT TO DO SYSTEMATIC REVIEWS AND HOW DO THEY LEARN MORE ABOUT THAT? >> WELL, THAT'S AN ONGOING CHALLENGE HERE IN THE UNITED STATES. IN THE U.K. THEY DO GIVE SUPPORT TO THE GROUPS THAT ORGANIZE THE REVIEWS FOR CONFERENCE. AND IT'S MUCH MORE AN INTERGAL PART. HERE IN THE U.S., THERE ARE VARIOUS MODELS. GETTING FUNDING FOR A SINGLE, SYSTEMATIC REVIEW IS FUN. YOU CAN LOOK ON REPORTER, FOR EXAMPLE, AND SEE THE INDIVIDUAL SYSTEMIC REVIEWS ARE FUNDED. IF YOU HAVE AN IDEA FOR ONE, BUT MAY NOT GET FUNDED, IT IS ANNEX PENSIVE WAY TO GO. YOU COULD WORK WITH AN EXISTING TEAM, AND GET YOUR REVIEW DONE THIS WAY. WE HAVE METHODOLOGIES, AND THEY WILL HELP CLINICIANS, OFTEN, CLINICIANS, ABOUT ALSO CONSUMERS, WHO MAY BE WORKING WITHOUT COMPENSATION, TOO DO THE REVIEW AND THE CLINICIANS ARE HAPPY, THEY'RE LEARNING HOW TO DO A SYSTEMATIC REVIEW. WE ENCOURAGE COPUBLICATION AND THEY'RE WORKING WITH PEOPLE WHO WHO DO A LOT OF THE WORK, AND SEAR THEM ON THEIR WAY. OFTEN, THERE ISN'T FUND FOR EXAMPLE A SYSTEMIC REVIEW BUT FUNDING FOR AN INFRASTRUCTURE, SUCH AS THE ONE I JUST DESCRIBED. NOT ALL THE RESULTS ARE REPORTED FROM A GIVEN TRIAL OR MAYBE OUTCOME, NOT ALL THE OUTCOMES ARE REPORTED. IT'S ALWAYS CURIOUS TO ME, TOO SEE RESULTS COMING OUT OF A TRIAL THAT MAY BE DIFFERENT FROM WHAT WAS PROMISED IN A PROTOCOL, THAT WAS PUBLISHED EARLIER, LIKE THE PRIMARY OUTCOME CHANGE, SOMEHOW. THINGS LIKE THAT HAPPEN. HOW CAN WE DO A BETTER JOB OF GETTING PEOPLE TO REPORT WHAT THEY OUT TO REPORT AND REPORT MORE FULLY, SO IT'S EASIER TO DO SYSTEMATIC REVIEWS, AND THEY CAN BE MORE ACCURATE? >> SO PEOPLE ARE WORKING ON A NUMBER OF FRONTS ON THIS. IT'S TRICKY, IT'S NOT GOING TO HAPPEN OVERNIGHT. THERE ARE REPORTING STANDARDS NOW THAT THE EQUATOR GROUP, AND YOU CAN REPORT ON EQUATOR, AND REPORTING STANDARDS, AND YOU CAN FIND WHAT YOU'RE LOOKING FOR. REPORTING STANDARDS HELP YOU TO MAKE SURE WHAT YOUR REPORT SUGGEST COMPREHENSIVE. AND ALSO, EDITORS CAN ENDORSOR USE REPORTING STANDARDS TO INSIST THAT THE REPORTS THEY GET, ADHERE TO THOSE STANDARDS. MOST PEOPLE ARE FAMILIAR WITH CONSORT FOR REPORTING CLINICAL TRIALS BUT THEY'RE ALSO, PLENTY OF REPORTING STANDARDS FOR OBSERVATIONAL STUDIES, AND OTHER TYPES EVER STUDY DESIGNS. SO REPORTING STANDARDS ARE IMPORTANT. DOESN'T MEAN THAT YOU OVERCOME REPORTING BIASES, BUT IT'S A BEGINNING. CLINICAL TRIALS.GOV IS REALLY SUPPORTED BY THE LAW WHICH MANDATE THAT IS ANYTHING COMING THROUGH THE FDA FOR APPROVAL HAS TO BE REGISTERED. MOW OF THOSE ARE RANDOMIZED TRIALS. BUT NIH REQUIRES ANY STUDIESES IT FUNDS ARE REGISTERED, REGARDLESS OF DESIGN. SO CAN YOU REGISTER YOUR STUDY, REGARDLESS OF DESIGN AT CLINICAL TRIAL.GOV. A LOT OF PEOPLE SEE THIS AS A HASSLE. BUT REGISTER FIRST, IT WAS YOUR IDEA. YOU CAN REGISTER YOUR PROTOCOL. YOU CAN ACTUALLY ASSOCIATE THAT EITHER WITH CLINICAL TRIALS.GOV NOW OR THROUGH ANOTHER SOURCE ALL THE DATA HAVE TO BE MADE PUBLICLY AVAILABLE. OR YOU COULD GO TO A LARGER REPOSITORY, DEPENDING ON THE FUNDER, SO THAT OTHER PEOPLE CAN FIND YOUR DATA BUT I THINK THE BIAS IS MULTIPLE. SO YOU KNOW, JUST CHOOSING OUTCOMES IS A RELATIVELY SMALL NUMBER AND ADHERES TO A CORE OUTCOME SET THE WE'LL PEAL MAKE SURE WE HAVE DATA ON THOSE OUTCOMES. SO IT'S A HUGE PROBLEM. AND I THINK THERE ARE MANY, MANY DIFFERENT APPROACHES AND WE CAN ALL CONTRIBUTE. THERE'S A CONFERENCE ON PEER REVIEW. I THINK IT'S SEPTEMBER 2017 IN CHICAGO AND I ENCOURAGE EVERYBODY TO ATTEND. IT'S A WONDERFUL CONFERENCE. ONLY EVERY 4 YEARS, AND IT REALLY ADDRESSES PROBLEMS LIKE REPORTING BIAS. MANY STUDIES THAT ARE COMPLETED ARE NEVER PUBLISHED. THERE ARE A NUMBER OF PAPERS SUGGESTING THAT MAYBE HALF OF THE CLINICAL TRIALS ACTUALLY GET PUBLISHED, AND THE OTHER HALF NEVER SEE THE LIGHT OF DAY, NOT FOR ANY OUTCOMES. WHAT DO YOU THINK ABOUT THE IDEA OF HAVING CLINICAL TRIAL.GOV OR HAVING INVESTIGATORS BE REQUIRED TO REPORT RESULTS FROM ALL STUDIES, NOT JUST REGISTER THEM, BUT REPORT ALL STUDIES FUNDED BY NIH AND CLINICAL TRIALS.GOV. WOULD THAT HELP TO ADDRESS THIS PROBLEM? >> I THINK IT WOULD. I THINK GET GETTING OUR RESULTS IN MULTIPLE PLACES. IT MAY BE IN FACT, THAT JOURNALS DO SOMETHING ENTIRELY DIFFERENT 20 YEARS FROM NOW. THAT WE PUBLISH OUR RESULTS IN A STARNIZED WAY ON SOMETHING LIKE CLINICAL TRIAL.GOV. SORRY, I'M CALLING IT PUBLISHED, BUT I MEAN REPORT. AND THEN WE USE PUBLICATIONS, MAYBE IN A DIFFERENT WAY THAN WE'RE CURRENTLY USING. THERE IS A STUDY FROM OH. ABOUT TWO, THREE YEARS AGO, ROB, ROS, I THINK IS THE FIRST AUTHOR, THAT SHOWS THAT ADVERSE EVENTS, ESPECIALLY ARE WELL REPORTED ON CLINICAL TRIAL.GOV. BETTER REPORTED THAN WHAT YOU WOULD FIND IN A JOURNAL ARTICLE, AND THEY HYPOTHESIS, THE AUTHORS HYPOTHESIS THAT THIS IS BECAUSE CLINICAL TRIAL.GOV HAS A STANDARDIZED FORMAT. SO PEOPLE KNOW AHEAD, WHAT'S WANTED, AND THAT JUST REALLY RANG A BELL WITH ME. IF YOU TELL PEOPLE WHAT'S WANTED, THEY'RE GOING TO LIKE NOT DO IT. IF THEY HAVE TO COME UP WITH IT ON THEIR OWN. THEY MAY NOT REPORT DATA THAT'S USEFUL. FOR EXAMPLE, WE'RE DOING A STUDY THAT'S SHOWING THAT SOMETIMES VERY OFTEN, DATA CAN'T BE INCLUDED IN A META ANALYSIS, BECAUSE PRECISION, WAYS OF MEASURING PRECISION, LIKE A STANDARD DEVIATION, AREN'T PROVIDED. AND SO YOU CAN'T THEN INCORPORATE THE DATA IN A META ANALYSIS AND TRIAL RESULTS STAND ALONE. IN CLINICAL TRIAL.GOV. IF IT ASKS FOR A STANDARD DEVIATION, YOU'RE GOING TO TRY TO FIND T. BUT ONE OF THE DOWNSIDES OF THE REPORTING IN CLINICAL TRIALS.GOV, AND YOU MENTIONED, IT SOMETIMES DIDN'T AGREE WITH WHAT'S IN A PUBLICATION. PEOPLE DON'T SEEM TO BE PUTTING THEIR OWN RESULTS OR OWN INFORMATION IN THEMSELVES. THAT IS THE P.I. OR CALL INVESTIGATORS ARE NOT DOING T. THEY'RE LEAVING IT UP TO SOMEONE WHO MAY OR MAY NOT KNOW THE DETAILS OF THE INDIVIDUAL. THIS IS AN INCREDIBLY IMPORTANT PART OF THE STUDY. WHAT YOU PUT IN CLINICAL TRIALS.GOV. THIS IS. >> SO OUTSIDE OF META ANALYSIS, WHAT OTHER PROBLEMS ARE PRODUCED BY SYSTEMATIC REVIEWS? SO THE MAJOR RESPONSE, SYSTEMATIC REVIEW IS AN OBSERVATIONAL STUDY AND SO WHAT YOU GET THEN, IS THE SYNTHESIS OF ALL AVAILABLE DATA AND THE META ANALYSIS, MAY OR MAY NOT BE A COMPONENT OF A SYSTEMATIC REVIEW. SO THE PRODUCT IS JUST THE STUDY ITSELF IT CAN BE USED BY OTHERS, FOR EXAMPLE, THOSE PRODUCING GUIDELINES, THE GUIDELINES PRODUCERS MAY BE CLINICIANS OR PUBLIC HEALTH OFFICIALS WHO WANT TO IMPLEMENT POLICY OR SEAT RESULTS OF RESEARCH HAVE AN IMPACT, THAT THEY'RE IMPLEMENTED THEY ARE TRAINED AND EXPERIENCED IN IMPLEMENT DATA ALTHOUGH IT'S NOT A DIRECT PRODUCT OF SYSTEMATIC REVIEW. I WOULD SAY THAT GUIDELINES PRODUCERS USE THE FINDINGS FROM SYSTEMATIC REVIEWS. >> DO YOU COLLABORATE OR USE HEALTH SCIENCE LIBRARIANS AT YOUR INSTITUTION? >> THE HEALTH SCIENCE LIBRARIANS ARE A REALLY IMPORTANT PART OF ANY SYSTEMIC REVIEW, AND WE USE THEM AND WE HAVE LEARNED A LOT I SHOULDN'T SAY WE USE THEM. WE COLLABORATE WITH THEM. WE HAVE LEARNED A LOT DURING THAT PROCESS. FIRST OF ALL, WE TEACH WITH THEM. SO OUR COURSES HAVE ALWAYS, FOR MORE THAN 20 YEARS, OUR COURSES ON HOW TO DO A SYSTEMATIC REVIEW HAVE ALSO INCLUDED THE LIBRARIANS OR INFORMATION AS SUCH AN INTERGAL PART. THE INFORMATION HELPS DO THE RESEARCH OF ALL AVAILABLE SOURCES. THEY KNOW WHAT SOURCE ARE OUT THERE. IT'S NOT JUST PUB MED AND M BASE. BUT THERE ARE MANY MORE SOURCES WE CAN USE, DEPENDING ON WHAT WE'RE LOOKING FOR. AND THEY ALSO, IF THEY'RE GOOD, THEY FOLLOW BLOGS, FOR EXAMPLE, LIST SERVES, ABOUT HOW TO DO RESEARCHES FOR SYSTEMIC REVIEWS. I WAS INTERESTED, SOMEONE JUST SENT ME LAST WEEK, A NOTICE THAT THE TOP 10 PAPERS THE MEDICAL LIBRARY ASSOCIATION MEETING, ALMOST ALL OF THEM WERE ABOUT SYSTEMATIC REVIEWS. SO IT'S REALLY, INFILTRATED THE MEDICAL LIBRARY ASSOCIATION AS WELL. THIS NEED TO BE AN ACTIVE PARTICIPANT TO MAKE A SYSTEMATIC REVIEW GOOD. >> SOMEONE WANTED TO SAY THANKS FOR TEACHING THEM A NEW WORD, THE META BIAS WORD. >> OH. OKAY. [LAUGHING] >> WHAT WAS THE NATIONAL INDICATE ME PUBLICATION DESCRIBED DOING A SYSTEMATIC REVIEW. >> IT WAS THE ONE I HAD ON MY SCREEN. CAN I JUST GO UP ON MY SHELF AND GET IT. JUST A S.E.C. >> ALL OF THE SLIDES WILL BE AVAILABLE ON THE ODP WEBSITE AFTER THE TALK AND PEOPLE CAN REFER BACK TO THAT SLIDE. FIND WHAT WORKS IN HEALTHCARE. >> CAN YOU STILL E-MAIL QUESTIONS TO PREVENTION@ PREVENTION@MAILNIH.GOV. ONE MORE QUESTION, DR. DICKSON. WHAT CAN INVESTIGATORS DO IN REPORT BEING THE RESULTS OF THEIR PAPERS? THAT WOULD MAKE IT EASIER. I GET INQUIRIES FROM PEOPLE WHO ARE DOING A REVIEW AND THEY'RE ASKING ME ABOUT A PAPER THEY WROTE 20 YEARS AGO, AND ASKING ME FOR INFORMATION I DIDN'T INCLUDE IN IT. AND SOMETIMES, THAT'S VERY DIFFICULT TO ACCESS. 20 YEARS LATER. WHAT ARE THE CRITICAL PIECES THAT INVESTIGATORS NEED TO INCLUDE IN THEIR PAPERS, IF THEY FOLLOW THE CONCERT GUIDELINES, IS THAT GOING TO TAKE CARE OF IT? >> YEAH, THAT WOULD BE GREAT FOR RANDOMIZED TRIALS. SO THEY SHOULD FOLLOW THE REPORTING GUIDELINES, APPROPRIATE FOR THE TYPE STUDY DESIGN THEY USE. THE WHOLE CORE OUTCOME, I DON'T THINK PEOPLE ARE TALKING ABOUT IT. IT'S REALLY CRITICAL. FOR EXAMPLE, YOU MIGHT HAVE SEEN A SYSTEMATIC REVIEW THAT HAD 25 STUDIES MEETING THE INCLUSION CRITERIA, BUT ONLY FOUR CONTRIBUTED TO THIS OUTCOME AND FIVE CONTRIBUTED TO THAT OUTCOME, ET CETERA. WELL, THAT'S RESEARCH WASTE. AND I THINK THERE'S A LOT MORE TENACIN BEING PAID NOW TO WASTE AND IF WE DON'T REPORT OUTCOMES, THAT EVERYBODY IS INTERESTED IN, THEN WHAT ARE WE DOING? SO FOR EXAMPLE, ON A RECENT SYSTEMATIC REVIEW WE DID, WE SAID IT WAS A DRUG TREATMENT FOR PAIN. WE SAID THE EARLIEST YOU WOULD WANT TO LOOKED BE EIGHT WEEKS, BUT PROBABLY, YOU WANT DATA FROM TWO YEARS, HOW WELL IS THIS DRUG WORK. AND WHEN WE WENT TO THE LITERATURE AND FOUND 50-SOMETHING STUDIES, ALL OF THEM STOPPED AT 8 WEEKS. THAT WAS OUR EARLIEST POSSIBLE TIME POINT. EXCEPT FOR ONE, WHICH WENT TO 14 WEEKS. SO THE SYSTEMATIC REVIEWER, THAT WAS US, IN THIS CASE, THE CLINICAL TRIALIST OR THE PERSON DOING OBSERVATION STUDY SETS OUTCOMES AND I DIDN'T SEE NCAA IT IN THE TALK. BUT I THINK WHEN PEOPLE THINK OUTCOMES, THEY THINK JUST ONE PIECE OF THE OUTCOME. SO FOR EXAMPLE, PAIN IS NOT THE ONLY THING THAT NEEDS TO BE REPORTED. IT'S PAYING AT ONE TIME POINT. WHAT DID YOU USE TO MEASURE PAIN. WAS IT A VISUAL AN LOG SCALE OR THE McGILL PAIN QUESTIONNAIRE. WHAT DID YOU USE? AND HOW DID YOU REPORT IT? IT WAS A CONTINUOUS OR DICHOTOMIST OUTCOME MEASURE. ALL OF THESE ELEMENTS, WHAT ARE CALLED ELEMENTS ARE COM.S OF A DEFINITION OF AN OUTCOME. CAN YOU SEE THIS IN CLINICAL TRIAL.GOV AND IT HAS TO BE INCLUDED WHEN PEOPLE WRITE PAPERS. WE DIDN'T EVEN THINK OF IT, I DIDN'T, ANY WAY, 20 YEARS AGO. BUT PEOPLE ARE THINKING ABOUT IT NOW. SO NOT ONLY DO YOU NEED TO CONTRIBUTE TO CORE OUTCOMES AND DEMAND THAT THEY EXIST FOR THEIR FIELD, BUT YOU NEED TO MAKE SURE YOU REPORT ALL THE ELEMENTS OF AN OUTCOME AND NOT JUST A DEMAIN, WHICH IN THE EXAMPLE I GAVE YOU IS PAIN. >> THANK YOU, DR. DICKERSON AND THANK YOU TO EVERYONE WHO PARTICIPATED IN TODAY'S WEBINAR. >> THANK YOU VERY MUCH. >> THE MEDICINE MIND THE GAP WEBSITE, WHICH IS AGAIN, PREN PREN.NIH.GOV/MINDTHEGAP, YOU WILL FIND SEVERAL RESOURCES FOR THIS TALK. INCLUDING SLIDES, AND REFERENCES AND YOUR FEEDBACK IS IMPORTANT FOR US AS WE PLAN FUTURE SESSIONS. THANK YOU AGAIN FOR YOUR TIME. >> THANK YOU.