>> GOOD AFTERNOON, EVERYONE.
I'M STATISTICIAN IN CLINICAL
TRIALS UNIT.
I WANT TO FIRST GIVE A BIG THANK
YOU FOR JOINING THE FIFTH PART
OF THE CLINICAL TRIALS
STATISTICAL LECTURE SERIES,
TODAY IS A META-ANALYSIS, SO I
WILL BEGIN WITH AN OVERVIEW OF
THAT METHODOLOGY.
IN SHORT A META-ANALYSIS IS A
STATISTICAL METHOD USED TO
SYNTHESIZE RESULTS FROM A SERIES
OF SCIENTIFIC STUDIESF I PUT A
NUMBER ON IT, IT ENTAILS
CONDUCTING SYSTEMATIC LITERATURE
REVIEW.
THIS ENTAILS CONDUCTING ON A
SYSTEMATIC LITERATURE REVIEW.
WE WILL BE ABLE TO COVER
SECTIONS 2 THROUGH 5 OF THE
PRESENTATION, ALL RELATED TO THE
SYSTEMATIC REVIEW PROCESS AND
SECTION 6 THROUGH 9 FOR THE MOST
PART RELATED TO THE STATISTICAL
SYNTHESIS.
WE WILL DISCUSS THE COMPONENTS.
SYSTEMATIC REVIEWS AND
META-ANALYSIS FORM TOP HIERARCHY
OF EVIDENCE DUE TO OBJECTIVE AND
COMPREHENSIVE NATURE.
SYSTEMATIC REVIEWS ARE
COMPREHENSIVE AND UNBIASED THESE
OF MANY RELEVANT STUDIES USING
RIGOROUS TRANSPARENT METHODS.
THIS PROCESS HELPS RESEARCHERS
MAKE INFORMED DECISIONS AND
IDENTIFY GAPS IN THE LITERATURE.
AS STATED, META-ANALYSIS IS A
STATISTICAL METHOD THAT COMBINES
RESULTS FROM MULTIPLE STUDIES TO
ADDRESS A SIMILAR, THAT IS THE
KEY, WELL-DEFINED RESEARCH
QUESTION.
IT INCREASES STATISTICAL POWER,
PROVIDING MORE ROBUST
CONCLUSIONS, META-ANALYSIS IS
BEING USED INCREASINGLY AND
PLAYS AN IMPORTANT ROLE IN
MEDICAL RESEARCH.
AS SHOWN HERE, THE HIERARCHY OF
SCIENTIFIC STUDIES, SYSTEMATIC
REVIEWS AND META-ANALYSES SIT AT
THE TOP OF THE PYRAMID OF
EVIDENCE-BASED CLINICAL STUDIES.
JUST BELOW IT YOU HAVE
RANDOMIZED CONTROLLED TRIALS,
COHORT STUDIES, CASE-CONTROLLED
STUDIES, CASE AND REPORTS, AND
BASE OF THE PYRAMID IS FORMED BY
BACKGROUND INFORMATION AND
EXPERT OPINION.
THE FLOW DIAGRAM OF
META-ANALYSIS DE DEPICTS FLOW OF
INFORMATION THROUGH PHASES OF 14
STEPS.
SO, THIS DIAGRAM HAS 14 STEPS
BEGINNING WITH DEFINING THE
WELL-DEFINED RESEARCH QUESTIONS.
STEPS 2 THROUGH 9 INVOLVE, MAYBE
HALF OF THE META-ANALYSIS, AN
EXHAUSTIVE AND THOROUGH RESEARCH
OF THE LITERATURE.
STEP 10 IN THIS 14-STEP PROCESS
INVOLVES DATA EXTRACTION ANT
QUALITY ASSESSMENT, AND QUALITY
ASSESSMENT IS ALSO REFERRED TO
AS THE RISK OF BIAS ASSESSMENT.
SO, ALTHOUGH THIS IS THE ENTIRE
FIELD, META-ANALYSIS STILL
FAIRLY NEW, THIS QUALITY
ASSESSMENT I WOULD SAY HAS
BECOME AN ESSENTIAL COMPONENT
WITHIN THE LAST A 5-10 YEARS.
SO WE WILL SEE AN EXAMPLE OF HOW
ONE WAS PERFORMED JUST LOOSELY,
AS AN OVERVIEW.
STEP 11 IS DATA CHECKING TASK.
STEP 12 IS WHEN WE CONDUCT
STATISTICAL ANALYSIS.
SO, SOME OF OUR COLLEAGUES THINK
THE META-ANALYSIS IS THE
STATISTICAL ANALYSIS BUT REALLY
THE META-ANALYSIS IS THIS
COMPLETE 14-STEP PROCESS.
STEP 13, DOUBLE CHECKING AGAIN
DATA.
AND 14 IS DISSEMINATION.
SHARING OF RESULTS, WRITING THE
MANUSCRIPT, REVISIONS, ALSO
SUBMISSION.
HOW DOES IT WORK?
THE PROCESS INVOLVES SEARCHING
VARIOUS DATABASES OF STUDIES,
RELEVANT TO A SPECIFIC QUESTION.
I'M HIGHLIGHTING THAT AGAIN,
VERY IMPORTANT.
CRITERIA APPLY 20 CHOOSE WHICH
STUDIES ARE INCLUDED IN REVIEW,
DATA ARE SUMMARIZED, SYSTEMATIC
REVIEW.
THE VERY FIRST STEP IN THIS
PROCESS IS DEVELOPING A FOCUSED
RESEARCH QUESTION, USING WHAT'S
CALLED THE PICOT FRAMEWORK.
THAT IS AN ACRONYM.
P FOR POPULATION, POPULATION
REFERS TO SAMPLE SUBJECTS THAT
WERE RECRUITED IN A STUDY.
THAT'S AN ESSENTIAL COMPONENT I
FOR INTERVENTION, TREATMENT
PROVIDED TO SUBJECTS ENROLLED IN
YOUR STUDY.
C REFERS TO COMPARISON GROUPS.
O REFERS TO THE OUTCOME
MEASUREMENT.
AND T DESCRIBES THE DURATION FOR
YOUR DATA COLLECTION.
WHEN DEVELOPING THIS VERY
IMPORTANT RESEARCH QUESTION, IT
IS ESSENTIAL TO TAKE INTO
ACCOUNT EACH OF THOSE COMPONENTS
OF THE PICOT FRAMEWORK.
A COMPREHENSIVE LITERATURE
INVOLVES DATABASES LIKE MEDLINE,
EMBAS EE COCHRAN, GRAY
LITERATURE, CLEAR CRITERIA FOR
INCLUSION AND EXCLUSION HELP
MAINTAIN CONSISTENCY AND
RELIABILITY IN SELECTING
RELEVANT STUDIES.
MULTIPLE REVIEWERS MAY BE NEEDED
TO ENSURE OBJECTIVE AND UNBIASED
INCLUSION OF STUDIES IN THE
REVIEW PROCESS.
A COMMON TECHNIQUE FOR THE
RESEARCH TEAMS IS TO USE TWO TO
THREE REVIEWERS TO CONDUCT
INDEPENDENT SYSTEMATIC REVIEWS,
AND THEN RECONCILE THOSE
DIFFERENCES TO MAKE SURE REVIEWS
OF AMOUNT OF LITERATURE
CONSIDERED IS AN EXHAUSTIVE AND
RELEVANT REPRESENTATION OF THE
LITERATURE RELATIVE TO THAT
PARTICULAR RESEARCH QUESTION.
THERE'S SEVERAL GUIDELINES
COMMONLY USED TO PROVIDE WHAT I
WOULD CONSIDER TO BE A BLUEPRINT
FOR HOW TO CONDUCT A
META-ANALYSIS.
MOST RELEVANT FOR RANDOM
CONTROLLED TRIALS IS PRISMA
STATEMENT.
IN 2009, THE PREFERRED REPORTING
ITEMS FOR SYSTEMATIC REVIEWS AND
META-ANALYSES ALSO KNOWN AS
PRISMA STATEMENT WAS LAUNCHED.
PRISMA PROVIDES CLEAR
DOCUMENTATION OF SEARCH
CRITERIA, RECORDS IDENTIFIED,
EXCLUSIONS, AND THEN YOU ALSO
HAVE TO GIVE REASONS FOR THOSE
EXCLUSIONS.
THE PRISMA STATEMENT ITSELF IS A
DOCUMENT, A DOCUMENT CONSISTING
OF A 27-ITEM CHECK LIST AND A
FLOW DIAGRAM.
THIS IS THE ACTUAL COPY OF THE
PRISMA 2020 CHECKLIST, ITEMS
1-15.
AS YOU WORK YOUR WAY THROUGH THE
META-ANALYSIS EACH ITEM HAS TO
BE ADHERED TO.
SECTIONS INCLUDE TITLE,
ABSTRACT, INTRODUCTION, METHODS.
OBVIOUSLY WITHIN THE METHODS YOU
WANT TO PROVIDE ELIGIBILITY
CRITERIA, INFORMATION SOURCES,
SEARCH STRATEGIES, DATA
COLLECTION PROCESSES.
RANDOM ESTIMATION APPROACH.
ITEMS 16-27 CONTAINS THE TOPICS
FOR RESULTS, DISCUSSION, AND
OTHER INFORMATION.
AGAIN, THIS IS THE BLUEPRINT,
THIS WOULD BE THE EXPECTED, LIKE
A TEMPLATE.
WHEN YOU SUBMIT THE ARTICLE TO A
JOURNAL, THIS IS THE EXPECTED
LAYOUT.
IN THE RESULTS SECTION YOU WOULD
DESCRIBE THE STUDY SELECTION,
STUDY CHARACTERISTICS, RISK OF
BIAS IN STUDY, STILL FAIRLY NO
SO IF -- NEW SO IF POSSIBLE TO
DO SO.
RESULTS OF INDIVIDUAL STUDIES.
AND THEN RESULTS OF THE
SYNTHESIS, WHICH IS GOING TO BE
IN MOST CASES THIS SUMMARIZED
EFFECT BASED ON RANDOM EFFECTS
ESTIMATION PROCESS.
THE PRISMA FLOW DIAGRAM DEPICTS
THE FLOW OF INFORMATION THROUGH
THE DIFFERENT PHASES OF
SYSTEMATIC REVIEW, IT MAPS OUT
THE NUMBER OF RECORDS
IDENTIFIES, INCLUDED AND
EXCLUDED, AND THE REASONS FOR
THOSE EXCLUSIONS.
THIS IS OFTEN THE VERY FIRST
FIGURE PROVIDED OR THAT YOU
WOULD ENCOUNTER IN THE
META-ANALYSIS.
I DO WANT TO SAY A FEW WORDS
ABOUT RISK OF BIAS.
BIAS ITSELF IS THE THEORETICAL
MEASUREMENT.
WE KNOW IT EXISTS.
IT'S VERY DIFFICULT TO QUANTIFY
BIAS.
BIAS IS THIS DIFFERENCE BETWEEN
THIS OBSERVED MEASUREMENT OF
ESTIMATE AND TRUE VALUE.
SO IN THE CONTEXT OF A
META-ANALYSIS, IT'S GOING TO BE
THE DIFFERENCE BETWEEN THE
OBSERVED EFFECT VERSUS THE TRUE
EFFECT.
THE RISK OF BIAS ASSESSMENT ON A
QUALITY ASSESSMENT TRIES TO BE
TRANSPARENT.
IT IS A TOOL THAT ALLOWS THE
TEAM TO BE TRANSPARENT ON THE
POSSIBLE SOURCES OF BIAS.
SO, FOR EXAMPLE, BIAS CAN ARISE
AT MULTIPLE STAGES.
SYSTEMATIC REVIEWS AFFECTING
VALIDITY OF THE CONCLUSIONS
DRAWN, TIMES OF BIAS INCLUDE
SELECTION BIAS, PERFORMANCE
BIAS, DETECTION BIAS,
PUBLICATION BIAS.
EACH STUDY SHOULD BE ASSESSED
RIGOROUSLY FOR POTENTIAL BIAS
USING STANDARDIZED TOOLS AND
REPORTING GUIDELINES.
WE'LL LOOK AT AN EXAMPLE WHAT
WAS DONE FOR A META-ANALYSIS
PUBLISHED IN THE COCHRANE
COLLABORATION LIBRARY.
COMMONLY USED TOOLS TO DETECT
BIAS ARE TO CLASSIFY BIAS ON THE
COCHRAN RISK OF BIAS TOOL, THE
NEWCASTLE OTTAWA SCALE,
QUADAS-2.
THESE ARE ALL NEW.
THE GENERAL THOUGHT IS THIS.
THE COCHRAN TOOL HAS ABOUT SIX
TO NINE SECTIONS OR ELEMENTS
THAT HAS TO BE SCORED.
BASED ON THE OVERALL RISK OF
BIAS OR QUALITY ASSESSMENT
SCORE, THE RESEARCH TEAM WOULD
THEN CONSIDER CLASSIFYING A
STUDY AS HAVING LOW RISK OF
BIAS, MODERATE RISK OF BIAS, FOR
EACH INDIVIDUAL STUDY.
THERE WOULD BE THIS OVERALL
ASSESSMENT DONE FOR THE
META-ANALYSIS THIS PROCESS IS
USED WHEN PROBLEM TO HELP
DETERMINE WHICH OF THE FOUR
ARTICLES WILL BE INCLUDED IN THE
META-ANALYSIS.
SO SHOWN HERE IS THE COCHRANE
COLLABORATION TOOL FOR ASSESSING
BIAS.
YOU HAVE DOMAINS IN THE FIRST
COLUMN.
HERE WE HAVE SIX DOMAINS.
SEQUENCE GENERATION, ALLOCATION
CONCEALMENT, POSSIBLE BIAS FROM
BLINDING, POSSIBLE BIAS DUE TO
INCOMPLETE OUTCOME DATA,
POSSIBLE BIAS DUE TO COLLECTIVE
REPORTING.
THE COCHRANE COLLABORATION
TOOLS, SECOND COLUMN SHOWS
SUPPORT OF JUDGMENT, GUIDE FOR
RESEARCH TEAM.
THIRD COLUMN SHOWS REVIEW OF
AUTHOR'S JUDGMENT.
THE AUTHOR HAS TO IN THE
ARTICLE, IN THE META-ANALYSIS,
PROVIDE JUSTIFICATION TO SHOW OR
EXPLAIN, IT'S NOT AN EXACT
SCIENCE, HOW THEY CLASSIFIED AN
ARTICLE AS HAVING LOW RISK,
MODERATE RISK, OR HIGH RISK OF
BIAS.
DATA EXTRACTION.
AGAIN, ALL THIS, AS YOU CAN SEE,
THE META-ANALYSIS FOR THE MOST
PART, I MAY SAY THIS TWO OR
THREE ADDITIONAL TIMES, IS NOT A
COMPLICATED APPROACH BUT DOES
REQUIRE ATTENTION TO DETAIL.
DATA EXTRACTION, I THINK THIS
WAS STEP 10 IN THIS 14-STEP
PROCESS, DATA EXTRACTION
INVOLVES COLLECTING ONLY THE
NECESSARY INFORMATION TO ANSWER
THE RESEARCH QUESTION
EFFECTIVELY AND ACCURATELY.
SO ONE GOAL OF THE EXTRACTION IS
TO COMPUTE A SUMMARY EFFECT, ITS
MEASURE OF PRECISION AND A
P-VALUE.
I'LL SAY IN THEORY BECAUSE IT
SOUNDS SIMPLE.
THAT PART CAN BE VERY
FRUSTRATING BECAUSE WE'RE
WORKING BACKWARDS.
WE HAVE A GOAL IN THIS REVIEW
PROCESS OF BEING ABLE TO EXTRACT
THESE EFFECTS.
AND VERY SPECIFIC MEASURES OF
PRECISION ON A VERY SPECIFIC
MEASURE OF UNCERTAINTY.
HOWEVER, FOR THE INDIVIDUAL
STUDY IT'S VERY POSSIBLE THOSE
MEASURES OF PRECISION COULD BE
DIFFERENT.
THEY COULD BE MISSING.
YOU MAY OR MAY NOT HAVE A
P-VALUE.
SO, THAT'S FRUSTRATING.
IT COULD IMPACT THE ABILITY TO
INCLUDE THOSE STUDIES, AND THE
META-ANALYSIS, STATISTICAL
CALCULATIONS PART OF THIS
PROCESS.
ALL RIGHT.
ARE WE READY FOR THE FIRST
EXAMPLE?
THIS IS EXAMPLE 1.
THIS IS EXAMPLE OF A STUDY FLOW
DIAGRAM.
FROM AN ARTICLE PUBLISHED IN THE
COCHRANE LIBRARY.
IT'S BOTULINUM TOXIN TYPE A
THERAPY, DYSTONIA.
WE'LL CAULK THROUGH DIFFERENT
PARTS OF THE ABSTRACT.
IT IS A SYSTEMATIC APPROACH, SO
YOU CAN SEE REMNANTS OF THOSE 14
STEPS, YOU'LL SEE IT HERE EVEN
IN THE ABSTRACT.
SO BACKGROUND INFORMATION FOR
THIS PARTICULAR STUDY, UPDATE OF
COCHRANE REVIEW PUBLISHED IN
TWIEF, THE MOST COMMON FOLK OF
FOCAL DYSTONIA CHARACTERIZED BY
INVOLUNTARY, USUALLY PAINFUL,
HEAD POSTURING.
BTA IS CONSIDERED FIRST LINE OF
THERAPY FOR THIS CONDITION.
COME DOWN TO THE FIRST METHOD.
THIS IS THE ABSTRACT.
IN THE ABSTRACT YOU HAVE TO
STATE TO IDENTIFY STUDIES FOR
THIS REVIEW WE SEARCH THE
COCHRANE MOVEMENT DISORDERS'
TRIALS REGISTER, CENTRAL
DATABASE, MEDLINE DATABASE,
EMBASE DATABASE, REFERENCE LIST
OF ARTICLES, AND COCHRANE
PROCEEDINGS, THAT'S VERY
IMPORTANT, THAT IS A SOURCE OF
PUBLICATION BIAS THAT OCCURS
WHEN THE RESEARCH TEAM ONLY
SELECTS THOSE STUDIES FROM THE
PEER-REVIEWED JOURNALS, ONLY
SELECT STUDIES WITH SIGNIFICANT
DIFFERENCES, IN THE EFFECT,
COMPARISON BETWEEN TREATMENT
GROUP AND PLACEBO GROUP.
WITHOUT INCLUDING STUDIES,
WHETHER IT DID NOT SHOW A
STATISTICAL SIGNIFICANT
DIFFERENCE.
THAT IS A SOURCE OF PUBLICATION
BIAS.
SO WE CONTINUE HERE.
I HAVE A FEW PARTS OF THIS
ABSTRACT HIGHLIGHTED TO AGAIN
SHOW THAT IN THIS 14-STEP
PROCESS, AT DIFFERENT PHASES OF
THE MANUSCRIPT, YOU HAVE TO
CLEARLY STATE AND JUSTIFY THE
PROCESS THAT WAS USED IN THIS
META-ANALYSIS.
SO HERE, FOR EXAMPLE, DATA
COLLECTION AND ANALYSIS,
HIGHLIGHTS SHOW TWO REVIEW
AUTHORS INDEPENDENTLY ASSESSED
THE RECORDS, SELECTED THE
STUDIES, EXTRACTED DATA USING A
PAPER PRO FORMA, EVALUATED RISK
OF BIAS.
RESULTS ARE HIGHLIGHTED.
WE INCLUDED EIGHT RANDOMIZED
CONTROLS TRIALS OF MODERATE
OVERALL RISK OF BIAS.
ALL OF THIS IS FAIRLY NEW, AND
THIS IS A FAIRLY REPUTABLE
JOURNAL, COCHRANE COLLABORATION
LIBRARY.
IT IS EXPECTED YOU WOULD USE
THIS QUALITY ASSESSMENT TOOL OR
RISK OF BIAS TOOL TO HELP
DETERMINE THE PAPERS THAT WOULD
BE INCLUDED IN THE META-ANALYSIS
OR SYSTEMATIC REVIEW.
HOPEFULLY YOU CAN SEE THIS.
THIS IS THE FIRST FIGURE FROM
THAT META-ANALYSIS.
SO THIS IS THE FLOW DIAGRAM,
HOPEFULLY YOU CAN SEE IT.
IT SHOWS IN TOTAL 208 RECORDS
WERE IDENTIFIED THROUGH THE
CENTRAL DATABASE SEARCH.
182 RECORDS WERE IDENTIFIED
THROUGH MEDLINE.
1256 RECORDS WERE IDENTIFIED
THROUGH EMBASE DATABASE SEARCH.
IN TOTAL 1,624 RECORDS
IDENTIFIED, 1599 RECORDS
INCLUDED AFTER DUPLICATES.
CONTINUE TO DRILL DOWN.
1569 RECORDS WERE IN TOTAL
EXCLUDED.
23 FULL-TEXT ARTICLES WERE
ASSESSED FOR ELIGIBILITY.
THERE'S 23 FROM THE INITIAL 1646
RECORDS.
OF THOSE 23 FULL TEXT ARTICLES,
15 WERE EXCLUDED.
SO THE RESULTING AT THE BOTTOM
RESULTING NUMBER OF STUDIES
INCLUDED FOR THIS META-ANALYSIS
WERE 8.
IT'S VERY IMPORTANT TO SHOW YOU
CAST A WIDE NET TO EVALUATE THE
LITERATURE ALL RELATED TO THAT
PARTICULAR RESEARCH QUESTION.
THIS IS FIGURE 2 IN
META-ANALYSIS.
IN THIS META-ANALYSIS EXAMPLE
RISK OF BIAS SUMMARY OF INCLUDED
STUDIES IT PRESENTS THE AUTHOR'S
JUDGMENT OVERALL ABOUT THE RISK
OF BIAS FOR EACH OF THE DOMAINS.
AT THE BOTTOM YOU SEE THE
LEGEND, COLOR-CODED LEGEND.
THE COLOR CODE OF GREEN
INDICATES LOW RISK OF BIAS.
COLOR CODE OF YELLOW INDICATES
UNCLEAR RISK OF BIAS, WHICH
WOULD HAPPEN QUITE OFTEN, FOR
THE MOST PART.
AND RED, SO THE COLOR CODE OF
RED DENOTES A HIGH RISK OF BIAS.
THIS RISK IS PRESENTED AS A
PERCENTAGE, IN THIS EXAMPLE.
SO 80% OF THE STUDIES WERE
CLASSIFIED AS HAVING A HIGH RISK
OF BIAS, FOR THE FOR HAD BEEN
PROFIT DOMAIN.
OFTEN WITH CLINICAL TRIALS
INVOLVING TREATMENTS, THOSE
TRIALS ARE FUNDED BY THE
INDUSTRY, PHARMA INDUSTRY OR
FOR-PROFIT INDUSTRY, WHICH IS
GREAT BUT IT CAN ALSO BE A
SOURCE OF WHAT'S CALLED
FOR-PROFIT BIAS.
SHOWN ON THIS SLIDE IS FIGURE
NUMBER 3, IN OUR EXAMPLE
META-ANALYSIS.
A RISK OF BIAS SUMMARY IS
PROVIDED FOR EACH INDIVIDUAL
STUDY.
AGAIN USING THE SAME COLOR CODE
HOPEFULLY YOU CAN SEE THIS.
THIS IS ACTUALLY A SUMMARY OF
THE EFFECT SIZE.
USING RANDOM EFFECTS
DESTINATION, THIS RESEARCH TEAM
HERE SHOWS OR ILLUSTRATES FOR
EACH STUDY, AND HERE WE'RE
LOOKING AT JUST SEVEN STUDIES,
THE RESULTS FOR THE COMPARISON
OF BOTULINUM TOXIN TYPE A VERSUS
PLACEBO, OUTCOME MEASURE IS
IMPROVEMENT OF CERVICAL
DYSTONIA.
FOR THESE ANALYSES THE RESEARCH
TEAM USES AN APPROACH CALLED
TWISTERS, AN INDEX THAT CAPTURES
SOME TYPE OF IMPROVEMENT.
ALL RIGHT.
SECOND AND THIRD COLUMNS YOU
HAVE THE ACTUAL NUMBER OF
PARTICIPANTS IN THE TREATMENT
GROUP FOR EACH STUDY, NUMBER OF
PARTICIPANTS THAT WERE IN THE
PLACEBO GROUP, THE FOURTH COLUMN
ILLUSTRATES THE STANDARDIZED
MEAN DIFFERENCE.
THAT'S THE EFFECT SHOWN FOR EACH
INDIVIDUAL STUDY.
THE GOAL HERE WOULD BE TO
SOMEHOW SYNTHESIZE THESE RESULTS
TO CALCULATE AN OVERALL EFFECT.
ALL RIGHT?
FIFTH COLUMN SHOWS THE FOREST
PLOT, EFFECT SIZE, CORRESPONDING
CONFIDENCE INTERVAL.
FOREST PLOTS ARE USED TO SHOW
THE EXTENT OR GAUGE
HETEROGENEITY THAT EXISTS
BETWEEN THESE STUDIES.
IN THEORY, IN THEORY, IF YOU DO
NOT SEE MOVEMENT, IF THE EFFECTS
BETWEEN THE STUDIES, LOOKING AT
THE PLOT, ALL LINES UP,
UNDERNEATH EACH OTHER, AND YOU
DON'T HAVE OVERLAP, THIS IS A
CLEAR SIGN THAT VARIATION
BETWEEN STUDIES IS ALMOST ZERO,
CLOSE TO ZERO.
YOU COULD THEN USE AN APPROACH
CALLED FIXED EFFECTS ANALYSIS,
FIXED EFFECTS MODELS IN MOST
CASES THAT WOULD NOT BE GOOD.
SO EVEN IN THIS CASE YOU DO SEE
THERE'S SOME MOVEMENT, THE
EFFECTS ARE MOVING SLIGHTLY BUT
ENOUGH TO JUSTIFY USING WHAT WE
CALL RANDOM EFFECTS MODELS AND
WE'LL TAKE A CLOSE LOOK AT THAT
IN THE NEXT TWO SLIDES.
ALL RIGHT.
SO, THE CALCULATION OF THIS
OVERALL EFFECT, SUMMARY EFFECT,
IS DONE BY ASSIGNING A WEIGHT, A
WEIGHT TO EACH STUDY, SO THAT IS
SHOWN IN THE FIFTH COLUMN.
WE WILL LOOK AT BACK OF ENVELOPE
CALCULATIONS TO SEE HOW THAT IS
DONE, THIS BUT THIS WEIGHT IS
INVERSELY RELATED TO PRECISION
OF THE EFFECT ON THE ESTIMATE.
THE PRECISION IS LIKE THE
STANDARD ERROR.
SO, STUDIES HAVING SMALL
STANDARD ERRORS CORRESPOND TO
THOSE STUDIES HAVING A LARGE
SAMPLE SIZE, SO THOSE STUDIES
WILL BE GIVEN MORE WEIGHT.
SO THEN WE WOULD THEN USE THOSE
WEIGHTS TO APPLY TO EACH OF THE
EFFECTS, AND THEN TAKE THE
WEIGHTED AVERAGE.
AND THEN THAT'S WHAT YOU HAVE AT
THE BOTTOM.
SHOWN HERE IS THE SUMMARIZED
EFFECT AND CORRESPONDING
CONFIDENCE INTERVAL.
SO, AGAIN, WE'RE GOING TO
ACTUALLY WALK THROUGH AND
REPLICATE THESE ANALYSES, THEY
ARE NOT VERY DIFFICULT.
JUST TO MAKE SURE THAT YOU HAVE
AN UNDERSTANDING OF WHAT IS DONE
IN THE BACKGROUND FOR MOST
SOFTWARE PACKAGES.
SHOWN IN THE BOTTOM LEFT-HAND
CORNER ARE YOUR MEASUREMENTS OF
HETEROGENEITY.
SO HETEROGENEITY OCCURS WHEN
YOU -- THE EFFECTS ARE MOVING.
SO THERE'S SOME RANDOM
DIFFERENCES IN THE POPULATIONS
THAT CAN BE DIFFERENCES IN HOW
THOSE STUDIES WERE CONDUCTED,
ALL OF THOSE THINGS COULD RESULT
IN THE EFFECTS BEING SLIGHTLY
DIFFERENT.
SO WE ARE POOLING, TAKING A
WEIGHTED AVERAGE, OF THESE
EFFECTS OF INDIVIDUAL STUDIES.
WE CAN JUSTIFY DOING THIS, IF
THE HETEROGENEITY IS REASONABLE.
REASONABLE.
SO YOU WANT TO MAKE SURE THAT
YOU ARE POOLING OR TAKING THE
WEIGHTED AVERAGE OF EFFECTS THAT
ARE SOMEWHAT SIMILAR.
AND SO WE'LL LOOK AT SOME OF
THESE MEASUREMENTS TAU SQUARED,
CHI SQUARED, ITEMS SHOWN HERE
ARE EXPECTED IN THE
META-ANALYSIS, THIS IS EXHIBIT
8.
WHAT WOULD BE EXHIBITED WHEN
SUBMITTING YOUR ANALYSES TO
ADJOURN.
BEFORE WE LOOK AT BACK OF THE
ENVELOPE CALCULATIONS I WANT TO
JUST STAY A FEW WORDS ABOUT
HETEROGENEITY.
HETEROGENEITY REFERS TO
VARIABILITY IN STUDY
POPULATIONS, INTERVENTIONS, OR
OUTCOMES IN SYSTEMATIC REVIEWS.
COCHRAN'S Q TEST, TAU SQUARED
USED TO ASSESS HETEROGENEITY.
I SQUARED AND TAU SQUARED ARE
FUNCTIONS OF THE COCHRANE Q
TEST, THE ONLY TRADITIONAL TERT
OF HETEROGENEITY IN THAT
ANALYSIS.
BASED ON TIME SQUARED
DISTRIBUTION GENERATES THE
PROBABILITY WHEN THAT STATISTIC
IS LARGE IT INDICATES LARGE
VARIATION ACROSS STUDIES, RATHER
THAN WITHIN SUBJECTS, OF THE
INDIVIDUAL STUDIES.
UNDERLYING HYPOTHESIS ASSUMES
THE TRUE EFFECT IS THE SAME
ACROSS STUDIES.
BORENSTEIN HAS A VERY GOOD BOOK,
"COMMON MISTAKES WHEN MAKING A
DECISION."
AT SOME POINT THE RESEARCH TEAM
HAS TO MAKE A DECISION TO USE
RANDOM EFFECTS ESTIMATION,
RANDOM EFFECTS MODELS, OR FIXED
EFFECTS MODELS.
IT'S NOT A SETTLED AREA.
THERE'S DIFFERENT THOUGHTS ABOUT
THIS.
SO WE WILL SEE, LOOKING AT THE
ACTUAL ANALYTIC FORMULAS.
IT DOESN'T HURT TO ASSUME A
RANDOM EFFECTS MODEL BECAUSE IN
THE ABSENCE OF HETEROGENEITY,
THOSE ESTIMATES CONVERGE OR MOVE
TOWARDS THE ESTIMATE THAT YOU
WOULD GET WHEN USING THE FIXED
EFFECTS MODEL.
SO, IT'S AN AREA THAT'S NOT
SETTLED BUT I THINK IT'S GOOD TO
NOT OVERTHINK THIS PROCESS.
ALL RIGHT.
ARE WE READY FOR CALCULATIONS?
THESE CALCULATIONS, THIS IS THE
EXAMPLE 3.
THESE CALCULATIONS ARE FAIRLY
SIMPLE.
I REPLICATED THE ANALYSIS, JUST
SHOWN TO YOU IN THE ARTICLE,
PUBLISHED BY THE COCHRANE
COLLABORATION LIBRARY.
BASED ON THE INFORMATION THAT
WAS PROVIDED.
SO THAT'S THE GOAL THAT YOU'RE
PROVIDING ENOUGH INFORMATION IN
THE META-ANALYSIS THAT ANYONE
COULD QUICKLY REPLICATE YOUR
ANALYSES.
SO, IN THIS PARTICULAR SLIDE YOU
SEE THE INDIVIDUAL STUDY, YOU
SEE THE NUMBER OF RESPONDENTS
TREATED WITH BTA.
BOTULINUM TOXIN TYPE A, NUMBER
TREATED WITH PLACEBO, FOURTH
COLUMN SHOWS STANDARDIZED
DIFFERENCE, TWISTER SCORES,
CORRESPONDING ERROR,
STANDARDIZED MEAN DIFFERENCE.
THE SIXTH COLUMN IS THE SQUARE
OF THE STANDARD ERROR.
AND THEN THE BETWEEN-STUDY
VARIANTS, THIS IS TAU SQUARED.
SO WE'RE LOOKING AT CALCULATION
OF TAU SQUARED.
IT IS A FUNCTION OF THE COCHRANE
Q STATISTIC.
ALL OF THIS INFORMATION IS USED
SO WE CAN CALCULATE A WEIGHT, SO
THE WEIGHT HAS TO BE BASED ON A
FUNCTION OF THE VARIANCE, IN
THIS CASE IT IS THE TOTAL
VARIANCE.
TOTAL VARIANCE MEANS THE
VARIANCE WITHIN THE STUDIES AND
VARIANCE BETWEEN THE STUDIES.
TOTAL VARIANCE.
THE NEXT COLUMN YOU HAVE THE
ACTUAL WEIGHTS.
AND THEN THE SECOND TO THE LAST
COLUMN YOU HAVE THE WEIGHTS AS
PERTAINS TO WHAT PERCENTAGE OF
OF THE OVERALL WEIGHT IS
ATTRIBUTED TO EACH STUDY.
THEN THE LAST COLUMN SHOWS THE
PRODUCT OF THE WEIGHT TIMES THE
EFFECT IN EACH STUDY.
YOU SUM THOSE UP.
AND THEN DIVIDE BY THE ACTUAL
POPULATION SIZE TO GET THE
RANDOM EFFECTS ESTIMATION --
ESTIMATE.
IN THIS CASE, I CAME PRETTY
CLOSE.
SO OUR EFFECT SHOWN HERE IS
ABOUT 6.9, IN -- I THINK IN THE
ARTICLE IT'S 6.7.
STANDARD ERROR .08, AND THEN THE
CONFIDENCE LIMIT, LOW CONFIDENCE
BALANCE OR UPPER CONFIDENCE
BALANCE ARE IDENTICAL TO WHAT
WAS PUBLISHED IN THE
META-ANALYSIS.
THIS SLIDE SHOWS THE ACTUAL
FORMULAS.
SO THE WEIGHTS, INVERSELY
RELATED TO VARIANCE, IN THIS
CASE FIXED EFFECTS.
THOSE WHO ARE NOT FAMILIAR, SOME
OF THIS, SYMBOLS AND NOTATIONS,
SIGMA SUMMATION REFERS TO WE'RE
ADDING.
MATHEMATICAL JARGON FOR ADDING.
I IS STUDY.
SO W IS RATIO OF 1 OVER THE
VARIANTS, WEIGHTED MEAN WILL BE
THE AVERAGE.
THE WEIGHT APPLIES TO THE EFFECT
OF EACH STUDY, DIVIDED BY TOTAL
WEIGHT.
SUM FROM ONE TO K, IN OUR CASE
EXAMPLE THERE WERE ONLY SEVEN
STUDIES, SUM FROM THE FIRST
STUDY THROUGH THE SEVENTH STUDY.
THE THIRD IS VARIANCE OF SUMMARY
EFFECT.
ONE OVER SUMMATION.
FOURTH STANDARD ERROR.
SUMMARY EFFECT.
AND THEN THE CONFIDENCE
INTERVAL.
IT CAN BE SHOWN THAT THE RANDOM
EFFECTS ESTIMATE, THIS OVERALL
EFFECT IS NORMAL DISTRIBUTED,
SHOWN THE CONFIDENCE INTERVAL IS
PROVIDED BY THE POINT ESTIMATE
PLUS OR MINUS MARGIN OF ERROR,
AND 1.96 TIMES STANDARD ERROR OF
THE EFFECT.
IN THE PRESENCE OF
HETEROGENEITY, THE CALCULATIONS
ARE A LITTLE DIFFERENT.
THE MEASUREMENT TAU SQUARED
WHICH MEASURES, THIS IS THE
SCRIPTIVE STATISTICS THAT
MEASURED PERCENTAGE OF VARIATION
DUE TO DIFFERENCES BETWEEN THE
STUDIES, IS THE RATIO OF Q TO
COCHRANE Q MINUS DEGREES OF
FREEDOM DIVIDED BY C.
WE HAVE THE ANALYTIC FORMULA FOR
Q.
IT'S GIVEN.
THE DEGREES OF FREEDOM, NUMBER
OF FEE VAIRALS, NUMBER OF
STUDIES K MINUS 1.
C IS ALSO GIVEN.
SO THE RANDOM EFFECTS
STIMULATION IS GOING TO BE 1
OVER THE WEIGHT, 1 OVER THE
VARIANCE WITHIN STUDIES PLUS TAU
SQUARED, TAU SQUARED IS THE
COMPONENT THAT DESCRIBES THE
VARIANCE DUE TO HOMOGENEITY OR
DIFFERENCES BETWEEN THE STUDIES.
THE LAST ONE SHOWS THE WEIGHTED
MEAN.
AGAIN, COMPLICATIONS ARE NOT
THAT OVERLY COMPLICATED.
YOU CAN DO IT IN EXCEL.
COMPREHENSIVE META-ANALYSIS, IT
IS VERY IMPORTANT FOR THE MOST
PART, I WOULD SAY USE A PACKAGE,
SO CMA WAS DEVELOPED IN PART BY
FUNDING THROUGH CDC, AND NIH.
IT IS COMMONLY USED AS A
PACKAGE, AND SO IF YOU INFORM
SOMEONE YOU USE A PARTICULAR
PACKAGE, IT ISN'T JUST SO EASY
FOR THEM TO REPLICATE YOUR
RESULTS.
ALL RIGHT.
I SQUARED IS SIMILAR TO TAU
SQUARED, DESCRIBES PERCENTAGE OF
VARIATION ACROSS STUDIES, THAT
IS DUE TO HETEROGENEITY, RATHER
THAN CHANCE.
I'M SAYING IT'S NEW, IT'S NEW TO
ME.
2002, 2003, HIGGINS ET AL., IT'S
EXPECTED IN THE META-ANALYSIS
YOU NOT ONLY PROVIDE SUMMARIZED
EFFECT, SUM MEASUREMENT OF
UNCERTAINTY, KV DENSER INTERVAL,
P-VALUE, INCLUDE MEASUREMENTS OF
HETEROGENEITY.
COCHRAN'S Q TEST, TAU SQUARED,
AND ALSO I SQUARED.
I'M PROVIDED REFERENCES I THINK
THAT WOULD BE VERY HELPFUL.
PROFESSOR BORENSTEIN HAS A
SERIES OF VIDEOS ALSO AVAILABLE
ON YouTube.
HE'S WELL PUBLISHED.
ARE THERE ANY QUESTIONS?
SEEING ANY QUESTIONS IN THE
CHAT?
>> NO, I DON'T SEE ANY AT THE
MOMENT.
>> SO THIS IS JUST A QUICK
OVERVIEW OF META-ANALYSIS.
I'M CURRENTLY WORKING THROUGH
TWO META-ANALYSES PROJECTS FOR
OUR COLLEAGUES HERE WITHIN
NINDS.
THEY ARE NOT COMPLICATED BUT IT
INVOLVES A LOT OF DETAILS AND
HAS A LOT OF MOVING PARTS.
IF YOU CAN THINK OF ANY
QUESTIONS, PLEASE DO NOT
HESITATE TO CONTACT ME.
I'LL BE MORE THAN HAPPY TO SHARE
WHATEVER RESOURCES I HAVE WITH
YOU.
>> I SEE ONE QUESTION.
IT SAYS IN EXCLUSION OF RECORDS
IN EXAMPLE HOW IS THAT DONE?
>> EXCLUSION OF RECORDS?
YES, SO THE RECORDS CAN BE
EXCLUDED BASED ON IF THE DESIGN
OF THE STUDY IS NOT IDEAL, SO
PERHAPS RANDOMIZATION IS NOT
IDEAL, THERE'S SOMETHING ABOUT
HOW THAT STUDY WAS CONDUCTED
THAT DOES NOT FIT THE EXCLUSION
CRITERIA.
SO THE RESEARCH TEAM HAS TO
DEVELOP THEIR OWN INCLUSION AND
EXCLUSION CRITERIA.
AND THEN YOU PROVIDE
JUSTIFICATION.
SO FOR THOSE RECORDS THAT
EXCLUDED IN THE META-ANALYSIS,
THAT JUSTIFICATION HAS TO BE
THERE.
WHY WERE THOSE ARTICLES NOT
INCLUDED?
BUT FOR THE MOST PART IT JUST
MEANS THE CONDUCT OF THOSE
STUDIES WAS NOT WHAT WAS
EXPECTED FOR YOUR META-ANALYSIS.
THE RESEARCH QUESTION YOU HAD IN
MIND, IT WAS VERY IMPORTANT THE
TRIAL BE CONDUCTED IN A CERTAIN
WAY.
>> A FOLLOW-UP QUESTION, 1500
EXCLUDED IS THAT TYPICAL?
>> I WOULD SAY IT IS TYPICAL.
I WOULD SAY FOR THE MOST PART IT
IS TYPICAL.
IT'S VERY DIFFICULT.
THERE'S A TON OF LITERATURE IN
THE UNIVERSE, YOU'RE LOOKING FOR
ANSWERS FOR A VERY NARROW, VERY
SPECIFIC RESEARCH QUESTION.
AND SO AMONG ALL THE ARTICLES
THAT ARE AVAILABLE, IT HAS TO BE
RELEVANT, AND CONSISTENT, WITH
THAT PARTICULAR QUESTION.
OTHERWISE WHEN YOU SYNTHESIZE
THE RESULTS, IT COULD BE
MEANINGLESS.
>> ANY FURTHER QUESTIONS?
>> I'M HOPING YOU PROVIDE
FEEDBACK.
THIS IS MY FIRST TIME GIVING
THIS TALK.
SO WE'RE DEVELOPING THESE TALKS
INVOLVING META-ANALYSIS, SO
BASED ON YOUR FEEDBACK, THAT
WILL GIVE ME GUIDANCE ON PERHAPS
TO INCORPORATE ADDITIONAL TOPICS
OR TO REMOVE CERTAIN TOPICS FROM
THE PRESENTATION.
AGAIN, I THANK YOU ALL FOR
JOINING US.
I LOOK FORWARD TO HEARING FROM
YOU OR MEETING YOU IN PERSON.
THANK YOU.
[END OF PROGRAM]
>> THANKS, HENRY.
I THINK THAT'S ALL.
I DON'T SEE ANYTHING ELSE COMING
IN ADDITIONAL.
I'LL GO AHEAD AND END THE
MEETING NOW FOR EVERYONE.
THANKS FOR COMING.
THANKS TO THE BROADCAST TEAM.
[END OF PROGRAM]