PRESENT NOT ONLY OF THE OBSERVATIONAL COMPARATIVE
EFFECTIVENESS RESEARCH WORKING GROUP, BUT ALSO I'VE REALIZED
THAT THERE'S HEAPS OF CROSSOVER BETWEEN DIFFERENT WORKING
GROUPS, SO THE DATA SCIENCE AND HARMONIZATION AND OTHERS AS
WELL.
SO THE CLINICAL TRIALS GROUP, THERE'S A LOT OF CROSSOVER, THE
IMAGING GROUP AND SO FORTH.
SO THE CONVERSATION THAT WE HAVE TODAY I THINK WILL BE RELEVANT
FOR LOTS OF THOSE.
SO FIRSTLY, I THINK THAT IT'S JUST MOVING TO THE NEXT SLIDE.
THE, THE PURPOSE OF WHAT WE'RE TRYING TO DO TODAY IN THIS ONE
HOUR SESSION IS WE'VE BEEN CHARGED BY DAVID MENON TO COME
UP WITH A A SINGLE SLIDE AT THE END OF THE CONVERSATION WHICH
WOULD BE WHAT WOULD IMPROVE TBI OUTCOMES, WHAT WOULD BE A
FUNDABLE PROPOSITION IN THIS FIELD OF CLINICAL DATA,
OBSERVATIONAL, COMPARATIVE AND EFFECTIVENESS RESEARCH THAT
WOULD IDEA THAT WE COULD AND AND NOT JUST THUNDERBALL, SOMETHING
THAT WOULD CHANGE THE GAME, WHAT WOULD BE IMPORTANT FOR PEOPLE,
WHAT WOULD IMPROVE OUTCOMES.
SO IN ORDER TO DO THAT IN THE FIELD WE NEED TO GET TO THE
POINT WHERE CLINICIANS KNOW WHEN TO TRUST THIS KIND OF RESEARCH
IN TRAUMATIC BRAIN INJURY AND HELP CLINICIANS INTERPRET THESE
KIND OF STUDIES.
AND SO YOU HEARD THIS MORNING ABOUT THE WHITE PAPER THAT'S
BEEN DISCUSSED AND WE'LL TALK ABOUT THAT A LITTLE BIT MORE IN
A MOMENT.
WE ALSO WANT TO BE ABLE TO INCREASE THE NUMBER OF OUTPUTS
THAT CHANGE CLINICAL PRACTICE BECAUSE THERE'S NO POINT REALLY
DOING THESE STUDIES UNLESS THEY ARE GOING TO CHANGE CLINICAL
PRACTICE IF WE FIND SOMETHING USEFUL.
SO PERHAPS REPLICATING THAT FLUID BALANCE PAPER THAT THAT
MATTHEW TOLD US ABOUT IN, IN OTHER AREAS AS WELL AND SO THAT
IT FLOWS THROUGH INTO THE GUIDELINES.
SO THIS DIAGRAM, OH, I SEEM TO HAVE LOST MY DIAGRAM.
I HAVE LOST MY DIAGRAM.
THERE WAS A DIAGRAM WHERE WE SORT OF TALKED A LITTLE BIT
WHICH SHOWED THAT IN A BIT OF A SCHEMATIC FORM AND BUT SORT OF
TO TO SHOW HOW.
BUT THE REASON FOR HAVING OBSERVATIONAL COMPARATIVE
EFFECTIVENESS REALLY THIS KIND OF DATA COLLECTION IN TRAUMATIC
BRAIN INJURIES IS PARTICULARLY IMPORTANT BECAUSE IT'S SUCH A
VARIABLE DISEASE STATE.
IT'S VERY DIFFICULT TO ENROLL PEOPLE IN RANDOMIZED CONTROL
TRIALS WHEN EVERYBODY IS DIFFERENT.
YOU KNOW, THIS IDEA OF AN END OF ONE FOR EACH INDIVIDUAL PERSON
WITH AN INJURY.
SO WE WE, YOU KNOW, EVERYONE IN THIS ROOM, WE'RE PREACHING TO
THE CHOIR THAT WE BELIEVE THIS KIND OF DATA CAN BE IMPORTANT,
BUT WE NEED TO TAKE THE NEXT STEPS TO SEE HOW IT CAN BE USED
MORE BROADLY.
SO TODAY I JUST WANT TO HAVE A BIT OF AN OPEN CONVERSATION.
WE'VE GOT A RANGE OF PEOPLE WHO ARE GOING TO HELP TO LEAD THE
DISCUSSION AND I'M GOING TO THROW TO TO HESTER IN A LITTLE
WHILE TO TALK A LITTLE BIT MORE ABOUT THE WHITE PAPER AND WHAT
MIGHT BE IN IT, WHAT WE SHOULD COVER.
BUT BEFORE WE DO THAT WE MIGHT JUST HAVE A A BIT OF A
CONVERSATION ABOUT SOME OF THE BASIC PRINCIPLES OF OF WHAT
WE'RE TRYING TO DO.
FIRSTLY I'D LIKE I HAD A CONVERSATION YESTERDAY WITH
JENNIFER ALBRECHT ABOUT THE THE WORDING OF COMPARATIVE
EFFECTIVENESS RESEARCH.
THE TITLE IS IT A GOOD ONE.
THE PEOPLE KNOW WHAT WE MEAN WHEN WE TALK ABOUT THAT.
I CERTAINLY DIDN'T KNOW WHAT IT MEANT UNTIL I CAME TO KNOW ABOUT
CENTRE.
SO JENNIFER, DID YOU WANT TO SORT OF GET THE BALL ROLLING ON
THAT ONE AND THEN WE CAN JUST OPEN UP THE CONVERSATION ON THAT
FOR A FEW MINUTES.
SO ARE WE TALKING ABOUT THE RIGHT THING?
YEAH, SURE.
GOOD NUMBER HERE.
OK, GREAT.
SO, SO MAYBE TAKE THE, YEAH, TAKE THE MIC OUT AND WHAT
HAPPENED AS A ROVING TURN THE OTHER WAY.
YEAH.
SO THAT YOU'RE NOT LOOKING AT MY BACK.
OK.
SO FIRST OFF, MY BACKGROUND IS AN EPIDEMIOLOGIST, SO.
SO I, I AM A METHODOLOGIST.
TO ME COMPARATIVE EFFECTIVENESS RESEARCH IMPLIES THE DIRECT
COMPARISON OF TWO TREATMENTS.
I DON'T TEND TO THINK OF IT SO MUCH AS A A TREATMENT VERSUS NO
TREATMENT.
I I BELIEVE THAT WHAT THIS GROUP WOULD LIKE TO DO IS IS ACTUALLY
ESTIMATE CAUSAL EFFECTS USING OBSERVATIONAL DATA.
AND I'LL JUST PUT IT OUT THERE FOR FOR CONSIDERATION AND
DISCUSSION THAT THERE ALREADY IS A VERY WELL ESTABLISHED
FRAMEWORK FOR ESTIMATING CAUSAL EFFECTS FROM OBSERVATIONAL DATA.
DOES ANYONE WANT TO COMMENT ON THAT?
DO WE LIKE THE TERM COMPARATIVE EFFECTIVENESS RESEARCH?
WHO WANTS TO SPEAK AS A CHAMPION OF IT?
MAYBE I'LL WALK AROUND WITH THE WITH THE MIC AND THAT MIGHT
FACIL THE DISCUSSION.
MAYBE TO END ON THAT, I THINK ANOTHER POINT FOR DISCUSSION IS
WHETHER WE WANT THE WHITE PAPER, DO WE?
DO WE NEED IT AND DO WE THINK IT ADDS TO THE FIELD?
SO YES, I COMPLETELY AGREE WITH THAT.
I THINK WE HAVE IN MEDICINE BEEN VERY ADHERENT TO A GOLD
STANDARD, AND I'M NOT SUGGESTING THE GOLD STANDARD ISN'T A GOLD
STANDARD, WHICH IS A RANDOMIZED CONTROLLED TRIAL, BUT IT SIMPLY
DOES NOT WORK FOR A WHOLE LOT OF REASONS THAT I'M SURE EVERYONE
IS COMPLETELY FAMILIAR WITH.
SO WE HAVE TWO OPTIONS, EITHER WE CARRY ON DOING INEFFECTIVE
RANDOMIZED CONTROLLED TRIALS OR WE ACTUALLY TRY AND FIND A
DIFFERENT GOLD STANDARD.
AND I WOULD ALSO SUBMIT THAT THE GOLD STANDARD ISN'T ACTUALLY
COMPLETELY GOLDEN.
THERE ARE PROBLEMS WITH RANDOMIZED CONTROLLED TRIALS
TOO.
AND AND OF THAT CAUSAL EFFECT MODELING, YOU KNOW, THERE'S A
WHOLE SPECTRUM FROM PROPENSITY SCORE MATCHING, FAIRLY SIMPLE
STUFF AT ONE END ALL THE WAY THROUGH TO THE ULTIMATE GOAL,
WHICH I THINK WOULD BE SORT OF INDEPENDENT TREATMENT EFFECTS,
INDIVIDUALIZED TREATMENT EFFECTS TYPE MODELS, WHICH WOULD ENABLE
US TO TAKE THINGS FROM THE TRIAL ALL THE WAY DOWN TO THE
INDIVIDUAL PATIENT AND MAKE PREDICTIONS AT PATIENT LEVEL AS
WELL IF WE HAVE ENOUGH DATA.
SO I DO COMPLETELY SUPPORT THAT.
I THINK IT'S A REALLY GOOD IDEA MAYBE TO KIND OF EXPLAIN AND
BUILD ON THAT.
WE HAD A FIGURE AND IT IT FELT OUT.
BUT TO TO TEASE OUT A LITTLE BIT THE DISCUSSION IS WE HAVE THE
BTF GUIDELINES WHICH PERSONALLY I THINK THEY'RE NON INSTRUCTIVE
FOR CLINICAL PRACTICE BECAUSE IT'S THE HIGHEST LEVEL OF
EVIDENCE.
WE DON'T HAVE ANY EVIDENCE.
SO THERE'S NO NO ADVICE FOR CLINICAL PRACTICE.
THEN AT THE BOTTOM OF AUTUMN WE HAVE THE CONSENSUS GUIDELINES
WHICH MAY HELP HARMONISE TREATMENTS AMONG DIFFERENT
SETTINGS, BUT WE DON'T KNOW IF IT IMPROVES OUTCOME AND THEN IN
BETWEEN MAYBE THAT'S FOR FOR DISCUSSION.
NOW WE HAVE OBSERVATIONAL STUDIES WITH DIFFERENT LEVEL OF
POSSIBILITY TO TO TO DO CAUSAL INFERENCE AND THAT DEPENDS ON
WHICH STATISTICAL METHODS YOU YOU USE.
SO ONE OF THE DISCUSSION MIGHT BE IS IN ORDER TO INFORM
CLINICAL PRACTICE GUIDELINES, SHOULD WE DRAFT THIS WHITE PAPER
TO TRY AND BE BRIDGE THE GAP BETWEEN CONSENSUS PAPERS,
EVIDENCE BASED GUIDELINES WHICH ARE ESSENTIALLY EMPTY AND
ANOTHER LAYER OF EVIDENCE WITH A CERTAIN LEVEL OF CAUSAL
INFERENCE POSSIBILITY TO REALLY INFORM GUIDELINES.
AND I THINK AS LONG AS WE HAVE THIS DICHOTOMY OF CONSENSUS AND
BTF GUIDELINES AND NOTHING IN BETWEEN THAT'S THAT FEELS LIKE
WE'RE MISSING SOMETHING REGARDING CLINICAL PRACTICE
ADVICE.
SO I I THINK THAT'S THAT MIGHT BE THE DISCUSSION, ONE OF THE
POINTS FOR DISCUSSION.
YEAH.
THEN I, YEAH, IT'S A LITTLE BIT STRANGE HOW TO TALK.
I TALK LIKE THIS TO EVERYBODY BECAUSE AND I ALSO LOOK AT YOU
BECAUSE TREATMENT.
WHAT IS THEN FOR THE OBSERVATION OF STUDIES TO EVALUATE
TREATMENT, BECAUSE GUIDELINES IS ALSO ABOUT POLICY ABOUT
ADMISSION OBSERVATION, SO IS FOR FOR THE SER IS ALSO TREATMENT
ADMISSION POLICY FOR EXAMPLE, BECAUSE IF YOU ADMIT PATIENTS
FOR A SHORT TIME, YOU GIVE THEM INFORMATION, YOU GIVE THEM
PSYCHOEDUCATION.
SO I JUST WANTED TO DISCUSS HOW BROAD IS THE CONCEPT OF
TREATMENT WITHIN THE SER APPROACH BECAUSE OTHERWISE I SAW
SOME EXAMPLES WHICH ARE ONLY GIVEN AT THE NEUROCARE UNIT
ALREADY ICU AND I THINK IT'S BROADER THAN THAT RUNS MY
HEALTH.
I THINK IT'S ANY INTERVENTION THAT YOU WOULD APPLY TO IMPROVE
OUTCOME.
AND SO I LIKE THE DISCUSSION AND ONE THING, AND I THINK LENNY
BROUGHT IT UP AS WELL AS A QUESTION.
SO I THINK THERE'S SOME AGREEMENT THAT WE WANT TO FILL
THIS GAP BETWEEN THE GOLD STANDARD AND THEN THE LOWER ENDS
OF EVIDENCE, WHICH IS EXPERT OPINION.
BUT YOU START BY ASKING, IS COMPARATIVE EFFECTIVENESS
RESEARCH THEN SORT OF THE RIGHT TERM STILL TO USE?
SUZANNE MISHIGL, I'M A NEURO INTENSIVIST WHO'S VERY KEEN ON
FINALLY SEEING DATA FROM COMPARATIVE EFFECTIVENESS.
I THINK TRIALS ARE TOO EXPENSIVE, TAKE TOO LONG,
REQUIRE FUNDING, AND I MEAN IT'S TOO LONG UNTIL WE HAVE AN
ANSWER.
I THINK WE'RE DOING A LOT IN ICU AND IT MAY NOT BE ICU AND MAYBE
YOU KNOW WHATEVER INTERVENTION THAT WE THINK WORKS
PHYSIOLOGICALLY MAKES SENSE, BUT WE HAVE NO DATA ON THAT.
I THINK WITH CENTER AND TRACK TBI COMBINED LOOKING AT YOU GUYS
RIGHT, THE RIGHT DATA, DOING VERY RIGOROUS ANALYSIS, I THINK
THAT ACTUALLY WILL DRIVE THE FIELD FORWARD.
I WILL TELL YOU, MY COLLEAGUES AND I, WE'RE VERY KEEN TO SEE
COMPARATIVE EFFECTIVENESS RESEARCH AND I MEAN NIH IS ALSO
RECOGNIZING THAT AND HAS OUR EYES OUT FOR THAT.
I THINK IT'S KNOWN UNDER THAT I THINK IT HAS AI MEAN.
WHAT ELSE WOULD YOU CALL IT?
I MEAN, I DON'T CARE WHAT YOU CALL IT AS LONG AS IT'S
RIGOROUS.
I THINK A WHITE PAPER HAS A ROLE TO EXPLAIN IN SIMPLE TERMS TO
CLINICIANS THE ADVANTAGES AND DISADVANTAGES.
EARLIER I SAW A SLIDE WITH GREEN PROS AND THE RED CONS OF WHAT
COMPARATIVE FACTORS RESEARCH CAN GIVE US.
IF WE CAN OUTLINE THAT TO MAKE CLINICIANS WHO DON'T KNOW
STATISTICS UNDERSTAND WHAT THE ADVANTAGES OF SUCH A STUDY ARE
OVER A CLINICAL TRIAL, I MEAN, IT'S REAL WORLD DATA.
I THINK THAT'S THE BIGGEST ADVANTAGE.
YEAH.
SO SO TO SUMMARIZE YOUR OPINION AND I THINK MANY OF US WOULD
AGREE THAT THERE'S A ROOM FOR A WHITE PAPER TO TO KIND OF
POSITION WHAT COMPARATIVE EFFECTIVENESS CAN BRING TO THE
TABLE AS DOES ANYONE HAVE AN OPPOSING OPINION ON THAT AND WHY
OR SUPPORTS IT AND WHY.
SO I I THINK IT'S A GOOD IDEA BUT MY MY THOUGHT IS I JUST WANT
TO MAKE SURE THAT PEOPLE ARE AWARE THAT IT IT DOESN'T
NECESSARILY HAVE TO ONLY BE A A WHITE PAPER IT COULD BE A
METHODS PAPER.
SO I'M ONE OF THE I'M THE ASSOCIATE EDITOR FOR METHODS AND
STATISTICS AT NEUROLOGY AND WE HAVE METHODS AND IN NEUROLOGY
PAPERS AND IT'S A SUBSECTION SO IT DOESN'T HAVE TO BE A WHITE
PAPER.
WE AGREE AND WE WE WE HAD A VERY ROUGH DRAFT THAT HAD BOTH METHOD
SECTIONS AND.
BUT IN THE END THE QUESTION IS THIS PRIMARILY FOR CLINICIANS OR
FOR METHODOLOGIST OR BOTH OR SHOULD YOU HAVE A CLINICIANS
PAPER AND A METHODS PAPER WHICH CAN REFERENCE TO EACH OTHER
WHICH CAN ALSO BE DO.
YEAH, YOU MEAN THAT'S THAT'S I MEAN I'M ALWAYS CONFUSED ABOUT
THIS TERM WHITE PAPER, BUT YOU MEAN THAT YOU CAN ALSO RECORD
THE METHODS PAPER AND MORE.
YEAH, EXPLAIN THE METHODS BUT ALSO HOW THEY SHOULD BE
INTERRUPTED.
AND YEAH AND TO COME BACK ON THE COMPARATIVE FACTORS.
SO I THINK AND JENNIFER WE HAVE BEEN MAILING AND TALKING ABOUT
THAT.
IF YOU LOOK AT THE ORIGINAL DEFINITIONS OF COMPARATIVE
EFFECTIVENESS RESEARCH, IT'S IT SAYS IT HAS TO USE THE
APPROPRIATE METHODS FOR THE QUESTION AT HAND OR SOMETHING.
SO A PRAGMATIC RANDOMIZED CONTROLLED TRIAL WITH BROAD
INCLUSION CRITERIA AND PATIENT OUTCOMES WOULD ALSO BE A
COMPARATIVE EFFECTIVENESS RESEARCH STUDY.
SO THAT IS THE ONLY THING.
BUT I I NOW I THINK WE JUST HAVE TO MAKE CLEAR THAT IN THIS WHITE
PAPER WE TALK ABOUT OBSERVATIONAL STUDIES OR
ESTIMATING TREATMENT EFFECTS.
SO THAT'S THE ONLY I WONDER IF WE SHOULD JUST CATCH IN TERMS OF
REAL WORLD EVIDENCE RATHER THAN TRIALS EVIDENCE AND ACTUALLY
ALLOW IT TO BE QUITE HETEROGENEOUS.
BECAUSE I'D LIKE TO COIN A NEW PHRASE WHICH IS COMPARATIVE
INEFFECTIVENESS STUDIES.
BECAUSE I THINK YOU KNOW IN REALITY THERE'S THERE'S WHAT WE
SHOULD DO WHICH WE OFTEN DON'T KNOW, WHAT WE THINK WE SHOULD
DO, WHAT WE THINK WE DO, WHAT WE THINK WE DID AND WHAT ACTUALLY
HAPPENED.
AND THEY'RE OFTEN THE COMPLETELY DIFFERENT THINGS.
SO WE THINK WE DO CERTAIN THINGS ACTUALLY.
YOU KNOW, WE, I THINK ONE OF THE PROBLEMS ABOUT INTERVENTIONAL
STUDIES IS THAT WE'RE LOOKING FOR INCREMENTALLY TINY LITTLE
BENEFITS WHICH ARE PROBABLY THERE SOMEWHERE, RIGHT, IF YOU
DID ENOUGH STUDY OR WHATEVER, BUT THEY'RE WASHED OUT BY THE
FACT WE SCREW STUFF UP ALL THE TIME WE DO, WE DO, THERE'S
HEALTHCARE, HEALTHCARE HAPPENS ALL THE TIME.
SO ACTUALLY THE REAL WORLD EVIDENCE IS ABSOLUTELY CRUCIAL,
IS ABSOLUTELY CRITICAL AND THAT'S THE KEY BIT I THINK FOR
ME THAT'S A NICE DIMENSION.
SO FOR INSTANCE FLUID PAPER, IT WAS NOT ONLY OR MAYBE NOT
SPECIFICALLY ABOUT THE FLUIDS, BUT IT WAS ALSO ABOUT DO WE DO
WHAT WE THINK WE DO BECAUSE WE ALL WANT EUVOLEMIA, BUT IN THE
END SOME GIVE A LOT OF FLUIDS OTHER VERY MUCH LESS AND WE WE
PURSUE THE SAME.
SO AND STRANGE THINGS HAPPEN IN PRACTICE.
SORRY, THERE WAS A QUESTION I COME BACK TO YOU, APOLOGIES THAT
MAY BE COMING TO THIS OR NOT IN HERE QUITE RIGHT.
BUT YOU KNOW THE COMPARATIVE EFFECTIVENESS RESEARCH, THE
PATIENT CENTERED OUTCOMES RESEARCH INSTITUTE HERE IN THE
US HAS DONE A VERY SPECIFIC DEFINITION OF THAT.
YOU HAVE TO HAVE TWO EVIDENCE BASED APPROACHES, 2 OR MORE THAT
THEN YOU'RE COMPARING AGAINST EACH OTHER.
YOU COULD DO THAT IN A RANDOMIZED CLINICAL TRIAL OR YOU
COULD DO IT WITH ONE OF THE OBSERVATIONAL METHODS THAT THAT
GET AT CAUSALITY.
YOU HAVE A CHOICE.
WELL, I THINK WHAT WE'RE TALKING ABOUT HERE IS THE LARGER
OBSERVATIONAL RESEARCH AND USE YOU KNOW LIKE A TECHNIQUE LIKE
PROPENSITY SCORE ANALYSIS IS NOT COMPARATIVE EFFECTIVENESS.
IT'S A YOU KNOW IT'S A TEMPTING TO GET AT CAUSALITY BUT WITHOUT
A COMPARATIVE EFFECTNESS.
SO I THINK OBSERVATIONAL RESEARCH IS OUR LARGER
CONSTRUCT.
OK, THANK YOU.
AND I THINK THIS ALSO ILLUSTRATES THAT WE, WE MAY HAVE
THIS WHITE PAPER TO CLARIFY WHAT WE'RE TALKING ABOUT BECAUSE
EVERYONE, DEPENDING ON YOUR BACKGROUND, THINKS SOMETHING
ELSE ABOUT IT OR HAS HIS OWN DEFINITIONS.
YEAH.
THANKS SO MUCH.
AND THANK YOU, JOHN, FOR SAYING EXACTLY WHAT I WANTED TO SAY,
BUT DIDN'T SAY CLEARLY ENOUGH.
AND I THINK IT'S POSSIBLE THAT THIS IS JUST AUS EUROPEAN SORT
OF DEFINITION ISSUE WHERE FOR ME COMPARATIVE EFFECTIVENESS
RESEARCH DOES INVOLVE THE COMPARISON OF TWO EVIDENCE BASED
TREATMENTS.
SO, SO WHAT I'VE BEEN SEEING THAT'S BEING CALLED COMPARATIVE
EFFECTIVENESS RESEARCH IN SOME OF THE STUDIES TO ME DOESN'T
QUITE MEET THAT DEFINITION.
HOWEVER, I LOVE REAL WORLD EVIDENCE FOR TBII THINK THAT
THAT ENCOMPASSES ALL OF IT AND IS REALLY AN EXCELLENT TERM.
SO I'M TRYING.
I'M TRYING TO MAKE SURE THAT WE UNDERSTAND THIS, BUT THIS IS THE
GENESIS OF HESTER'S QUESTION.
AND THEN THIS IS THINGS THAT HAVE COME UP REPEATEDLY ABOUT
WHAT IS THE CORRECT TERM.
AND SOMETIMES THESE ARE OPERATIONAL TERMS, OPERATIONALLY
DEFINED TERMS FOR PEOPLE TO STAY IN SCOPE, OK.
SO FOR THE FOR PAKORI, THE PATIENT CENTERED OUTCOMES
RESEARCH INSTITUTE WHICH IS A FEDERAL FUNDING AGENCY IN THE
UNITED STATES, THEY HAVE TO BE VERY, VERY SPECIFIC ABOUT WHAT
FALLS UNDER THEIR SCOPE SO THAT PEOPLE CAN ALSO UNDERSTAND,
PLEASE DON'T SEND YOUR APPLICATION TO US BECAUSE IT'S
NOT IN OUR SCOPE.
AND SO LIKE IS THIS OR EVAL DOES THIS WHEN YOU HEAR THIS, THIS US
BASED DEFINITION OF COMPARATIVE EFFECTIVENESS RESEARCH, DOES IT
CONFLICT WITH YOUR UNDERSTANDING OF WHAT COMPARATIVE
EFFECTIVENESS RESEARCH IS?
AND THEN ARE WE BETTER OFF SAYING ARE WE BETTER OFF JUST
USING THE WORD REAL WORLD EFFECTIVENESS OR SOMETHING ELSE
SO THAT WE'RE JUST SO THAT WE'RE NOT CREATING THE ISSUE OF
CONFUSION.
I I THINK MY, AS AN AUSTRALIAN, SOMEONE WHO'S COMPLETELY OUT OF
BOTH OF THOSE SYSTEMS, MY DEFINITION OR MY UNDERSTANDING
OF THE TERM COMPARATIVE EFFECTIVENESS RESEARCH WAS
ACTUALLY JUST COMPARING WHAT HAPPENS IN HOSPITAL A IN THIS
COUNTRY TO WHAT HAPPENS IN HOSPITAL B IN THAT COUNTRY.
THAT'S WHAT I THOUGHT IT WAS.
SO INDEED, AND I THINK, AND THAT IS PROBABLY THE FAULT OF CENTRE,
TBI AND MAYBE THEN MYSELF BECAUSE THAT'S HOW WE HAVE
INDEED PRESENTED.
BUT IT'S ACTUALLY A SPECIFIC METHOD, INSTRUMENTAL VARIABLE
ANALYSIS RATHER THAN PROPENSITY SCORE MATCHING OR ANOTHER METHOD
TO TRY TO ADJUST FOR CO FOUNDING BASICALLY.
SO IT'S ONE METHOD IN THE COMPARATIVE EFFECTIVENESS REACH
THAT YOU CAN USE FOR COMPARATIVE EFFECTIVENESS RESEARCH.
AND ONE MORE THING TO JENNIFER, I THINK WE KNOW IN EUROPE AS
WELL THESE THE, THE, YEAH, THE TERM AND AND THE FORMAL
DEFINITION AS THE OR AS THE FORMAL, I DON'T KNOW HOW IT WAS
ONCE DEFINED BY THE INSTITUTES OF HEALTH HAS THESE ELEMENTS OF
PATIENTS RELEVANT OUTCOMES COMPARING TO THINGS THAT ARE
ALREADY COMMON PRACTICE ETCETERA.
I THINK THE MAIN PROBLEM IS THE CONFUSION WITH RANDOMIZED OR NOT
RANDOMIZED IN MY MIND AND THEN REAL WORLD EVIDENCE.
YEAH, I DON'T KNOW.
I MEAN, I THINK THERE'S A THERE'S A SORT OF STATISTICAL
COMMON GOLDEN THREAD THAT RUNS THROUGH A LOT OF THESE THINGS,
WHICH IS ESSENTIALLY WHAT A LOT OF THESE THINGS WE'RE TRYING TO
DO IS EXPLOIT HOPEFULLY RANDOM OR AT LEAST PREDICTABLE
VARIATIONS IN CARE.
AND YOU MIGHT EXPLOIT THAT FROM CENTRE TO CENTRE OR YOU MIGHT
EXPLOIT IT JUST FROM BETWEEN PATIENTS OR YOU MIGHT EXPLOIT IT
BETWEEN PRACTITIONERS OR YOU MIGHT EXPLOIT IT BETWEEN
PRACTITIONERS BEFORE AND AFTER LUNCH.
IT DOESN'T MATTER.
AS LONG AS YOU'VE GOT A BIG ENOUGH COHORT THEN YOU CAN KIND
OF FIND A MATCHING TWIN, IF YOU LIKE IN SOME STATISTICAL WAY
THAT THAT LOOKS SIMILAR BUT GOT A DIFFERENT TREATMENT.
THAT'S THE KIND OF THREAD.
I MEAN, YOU KNOW, I'M RATHER SIMPLIFYING ALL OF THESE MODELS
METHODS.
BUT BASICALLY I MEAN I I THINK I KNOW I CAME UP WITH IT.
I THINK REAL WORLD EVIDENCE IS, AND I'M HEARING IT A LOT PEOPLE
ARE USING THIS IN OTHER DOMAINS.
AND IT IS ABOUT, IT'S ALL ABOUT USING THE DATA WE'VE GOT IN THE
REAL WORLD TO FIND OUT WHAT WHAT WORKS AND WHAT DOESN'T.
AND I THINK IT DO YOU, WOULD YOU AGREE THAT IT EXCLUDES
RANDOMIZATION OR INTERVENTIONAL?
YEAH.
CAN I JUST ASK THE ROOM, DO YOU THINK THERE ARE ANY UNINTENDED
CONSEQUENCES FROM CHANGING AND USING THIS KIND OF A TERM?
WE MENTIONED FUNDING BODIES AND BEING ABLE TO USE, YOU KNOW,
LIKE IS THIS GOING TO UPSET FUNDERS OR UPSET INSTITUTES THAT
ARE SUPPOSED TO USE A PARTICULAR TYPE OF RESEARCH AS LONG AS
WE'RE CREATING A TERM, A PHRASE I'VE ALWAYS LIKED A LOT IS
PRACTICE BASED EVIDENCE.
BECAUSE IT, YOU KNOW, IT PLAYS OFF OF EVIDENCE BASED PRACTICE
OBVIOUSLY.
BUT WHEN YOU'RE USING ACTUAL REAL WORLD TREAT, YOU KNOW,
PRACTICE DATA IN SOME WAY, THEN YOU CAN USE IT ANY WAY YOU WANT,
FRANKLY.
YOU CAN DO COMPARATIVE EFFECTIVENESS.
YOU CAN DO PROPENSITY SCORE ANALYSIS OR OTHER OTHER METHODS.
NOT AT THAT.
I'M COMPLETELY AGAINST RETERMING IT, BUT I I DO WANT TO PUT OUT A
WORD OF CAUTION FOR NON STATISTICALLY EDUCATED
CLINICIANS, WHICH IS PROBABLY 90% OF THEM OUT THERE.
THAT CER IS A RESPECTED TERM FOR A SOLID METHODOLOGY EVEN IF THEY
DON'T UNDERSTAND IT.
SO IF WE NOW COME UP WITH A NEW TERM THAT NO ONE'S HEARD BEFORE
AND WHILE WE THINK REFLECTS MORE OF WHAT WE'RE ACTUALLY DOING, I
WORRY THAT IN THEIR HEAD, AS THEY'RE READING THE ABSTRACT
THAT COMES OUT IN THE IN THE JOURNAL, THEY MAY NOT TAKE IT AS
SERIOUSLY OR MAY NOT THINK OF IT AS THIS IS, YOU KNOW, INCREDIBLY
IMPORTANT DATA OR MAY NOT EVEN THINK, OH, MAYBE WE SHOULD PUT
THIS IN GUIDELINES.
AGAIN, IT'S MORE OF THE PERCEPTION OF THE UNEDUCATED
CLINICIAN AND THAT'S WHAT WE'RE TRYING TO CHANGE, RIGHT.
I MEAN LOOKING BACK AT THE FLUID THING, WE'RE TRYING TO GIVE
PATIENTS LESS FLUID.
SO WE'RE TRYING TO CHANGE PRACTICE.
SO IT NEEDS TO BE PRACTICAL.
SO THAT'S JUST A WORD OF CAUTION.
SO LET'S IN THE INTERESTING, I DO AGREE THAT WE, I THINK IT'S A
GOOD DISCUSSION, BUT WE SHOULDN'T TRY TO KIND OF CHANGE
ESTABLISHED TERMINOLOGY.
WE SHOULD CLARIFY WHAT'S THERE AND THEN LET'S SEE HOW TO MAKE
BETTER USE OF IT.
YEAH.
AND I APPRECIATE.
I APPRECIATE THE THE THE CONCERNS ABOUT TIME BUT THE
IRONY HERE IS THAT WE'RE THIS IS LIKE A MAJOR OUTPUT OF THIS WORK
GROUP AND WE DON'T HAVE WE DON'T HAVE CLARITY OR AGREEMENT ON
WHAT TO WHAT TO CALL THIS.
AND THEN WHEN IT COMES TO FUNDING AGENCIES, THE ORIGINAL
TRACK GRANT THAT WE GOT IN 2009 WAS 1 MECHANISM BUT THEN IN 2000
WE NOW THAT CALL THAT THAT TRACK PILOT.
IN 2013 WHEN WE GOT THE MAJOR GRANT THAT WE NOW START TO REFER
TO AS TRACK, IT WAS BY DEFINITION MANDATORY FOR US TO
DO COMPARATIVE EFFECTIVENESS RESEARCH.
AND BY THE DEFINITION THAT THE TWO OF YOU JUST USED NOT ONE
SINGLE THING IN THAT GRANT WAS COMPARATIVE EFFECTIVENESS.
SO THAT'S TRICKY AND AND ARE YOU KNOW SO ARE WE BETTER SERVED
LIKE REACHING AN AGREEMENT ON WHAT IS BEST AND SO WHAT I'VE
HEARD SO FAR IS PRACTICE BASED EVIDENCE REAL WORLD DATA OR TO
STICK WITH COMPARATIVE EFFECTIVENESS RESEARCH OR ARE WE
MISSING SOMETHING?
I THINK I THINK IT WOULD BE BENEFICIAL TO ACTUALLY HAVE AN
AN AGREEMENT ON WHAT TO CALL THIS.
YEAH BUT THAT BUT AGAIN WILL AGAIN A LITTLE BIT OPPOSING LIKE
WITH THE TROPONIN I I THINK IT'S ABOUT THE THE LEVEL OF CAUSALITY
BASED ON OBSERVATIONAL DATA AND THE LEVEL HOW THESE
OBSERVATIONAL ANALYSIS MIGHT INFORM PRACTICE.
I I DON'T ACTUALLY CARE MUCH ON HOW YOU MENTION IT BUT IT'S IT'S
ABOUT WHAT GOAL ARE WE TRYING TO ACCOMPLISH WITH CLARIFYING
METHODOLOGY AND WHAT WHAT CAN WE EXTRACT THE MAXIMUM OUT OF
OBSERVATIONAL DATA OR AND CAN IT BE INSTRUCTED FOR GUIDELINES AND
AND INFORMAL PRACTICE.
SO YEAH HOWEVER YOU YOU MENTION IT THAT'S SOMETHING WE CAN
DISCUSS IN THE IN THE WHITE PAPER.
I WANTED TO MENTION THE THE AVIATION INDUSTRY BECAUSE THEY
HAVE BEEN SUCCESSFUL IN FINDING CORE ROUTES TO PROBLEMS AND THEY
USE A METHOD KNOWN AS CASE BASED REASONING WHICH KIND OF FITS
INTO WHAT YOU'RE TALKING ABOUT ABOUT DIGITAL DEPLOYING.
I DON'T KNOW HOW WELL IT OR HOW CLOSE IT IS TO COMPARATIVE
EFFECTIVENESS RESEARCH.
I THINK WITH CR IT ASSUMES YOU HAVE ENOUGH PATIENTS THAT ARE
SIGNIFICANTLY SIMILAR TO ONE ANOTHER IN ORDER TO MAKE AN
INFORMED SPECIFICALLY STATISTICALLY SIGNIFICANT
DECISION.
SO THAT THAT'S JUST MY THOUGHTS.
CAN I JUST SUGGEST THAT WE MOVE ON BUT PARK THIS AND NOT LEAVE
THE ROOM AT THE END UNTIL, BUT I THINK THE DECISION, THE
DISCUSSION THAT WE CONTINUE TO HAVE WILL HELP TO INFORM THIS.
SO LET'S, I THINK WE'VE ALREADY COVERED IS IT WORTH DOING.
WE'VE BEEN TALKING ABOUT A DEFINITION, SO A LOT OF WHAT
WE'VE TALKED ABOUT WILL WILL HELP.
SO I'D LIKE TO HAND OVER TO HESTER TO TAKE US THROUGH SOME
OF THE THE OUTLINE FOR THE WHITE PAPER AND WHAT SHOULD BE COVERED
AND HOW WE CAN PITCH IT SO THAT IT IS PERCEIVED AS SOMETHING
THAT CAN HELP CHANGE PRACTICE FOR THE BETTER.
SO DO YOU WANNA GIVE THE OVERVIEW FIRST OR PER SLIDE OPEN
THE DISCUSSION WHILE YOU'RE GOING THROUGH THE THE THE
HEADINGS, LET'S SEE WHAT EVERYBODY WANTS.
IT DOESN'T MATTER ACTUALLY.
I THINK WE SHOULD COME FIRST.
NOW LET'S GO TO THE OUTLINE AND THINGS WILL COME UP.
SO PLEASE DON'T RESTRICT YOURSELF TO OPINIONS THAT ARE OR
TO THINGS THAT ARE ONLY THINGS THAT ARE ON THE SLIDES.
SO IT IS NICE THAT WE HAVE THESE DISCUSSIONS BECAUSE IT'S I THINK
IT HIGHLIGHTS THE NEEDS TO WRITE DOWN FOR OURSELVES ALSO WHAT WE
MEAN EXACTLY AND WHAT WE WANT TO DO.
SO WE HAVE MADE SORT OF AN THERE MANY PEOPLE HAVE ALREADY
PROVIDED INPUT ON THE ON THE DRAFT TEXT OF THE WHITE PAPER.
WE TOOK ONE STEP BACK AND MADE MORE OF AN OUTLINE TO ALL AGREE
ON WHAT SHOULD BE COVERED AND THE OUTLINE IS ON THE SLIDE SO
WE CAN GO THROUGH IT.
BUT AS I SAID THIS MORNING IN MY OVER, IN OUR VIEW THE AIM WOULD
BE TO CLARIFY CONCEPTS.
NOW I THINK THIS SHOWS THAT THAT IS USEFUL AND THEN TO PROVIDE
RECOMMENDATIONS ON HOW TO DESIGN COMPARATIVE EFFECTIVENESS
OBSERVATIONAL SPECIFICALLY.
SO I GUESS WE SHOULD WHATEVER WE CALL IT RIGHT FROM THE START, WE
WILL SAY THAT WE WILL NOT TALK ABOUT RANDOMIZED TRIALS BUT ON
OBSERVATIONAL REAL WORLD DATA.
SO HOW DO YOU DESIGN SUCH A STUDY AND THERE HOW YOU ANALYZE
IT AND HOW YOU INTERPRET IT AND WHAT COULD BE THE IMPLICATIONS
FOR CLINICAL PRACTICE.
AND OF COURSE THE LAST TWO THINGS ARE CLOSELY RELATED AND
IN THE METHODOLOGIST VIEW THE THE INTERPRETATION OF COURSE
CLOSELY RELATED TO THE DESIGN AND THE METHODS REMARKS TO THAT.
SO THAT OVERALL CONTENT OR OUTLINE THIS SPEAK UP OF COURSE
IF YOU INTERRUPT, YEAH.
SO, SO THE START WOULD BE THE RATIONALE FOR COMPARATIVE
EFFECTIVENESS RESEARCH AS IT WAS INITIALLY OUTLINED SPECIFICALLY
IN THE US THE FACT THAT MANY THE PRACTICE IS ACTUALLY NOT BASED
ON EVIDENCE BECAUSE THE RANDOMISED TRIALS USUALLY NOT
MEET OR NOT INFORM THE REAL CLINICAL QUESTIONS THAT ARE AT
HAND DUE TO RESTRICTED POPULATIONS, NOT PATIENT
RELEVANT OUTCOMES ETCETERA.
BUT THEN AND THEN MOVE TO TBI THAT IN TBI SPECIFICALLY RCTS
HAS NOT HAVE NOT BROUGHT US WHAT WE WOULD HAVE HOPED FOR.
AND THEN TO MAKE THE POINT THAT WE THEREFORE NOW HAVE THESE
LARGE OBSERVATIONAL DATA SETS THAT WE NOW CAN ANALYZE,
PREFERABLY JOINTLY BUT ALSO SEPARATE.
BUT IT'S IMPORTANT TO AVOID BOTH OVER AND UNDER INTERPRETATION.
SO NOT TO USE TOO SIMPLE METHODS TO DEAL WITH CONFOUNDING AND
THEN DRAW CONCLUSION THAT IS BIAS OR THAT IS NOT TRUE.
SO THEN THE AIM IS NOW FORMULATED LIKE THIS TO MAKE
MAXIMAL USE OF SO TO LINK IT TO THE FACT THAT WE HAVE ALL THESE
EXISTING DATA.
WHEREAS WE ALSO WANT TO SAY SOMETHING I GUESS WHEN YOU NOW
SET UP A PROSPECTIVE STUDY.
BUT THAT THE THE RATIONALE IS ALSO ABOUT USING THE DATA THAT
WE HAVE ALREADY COLLECTED AND ALL THE REAL WORLD DATA THAT ARE
EVERYDAY COLLECTED IN EHR FOR EXAMPLE.
SO WE AIM TO GUIDE CLINICIANS RESEARCH AND FUNDERS ARE QUITE
BROAD WITH DESIGN EXECUTION AND INTERPRETATIONS OF PROFESSIONAL
SEARCH STUDIES.
COMMENTS.
THOUGHTS.
ARIE?
NO, I AGREE.
I'LL DEFINITELY MAKE THE THING ABOUT ELECTRONIC HEALTH RECORDS
ROUTINE EXCESS DATA.
NOW THIS IS FOR THE HIGH INCOME COUNTRY SPECIFIC RATHER THAN
LONDON, SO I APPRECIATE IT NOT, YOU KNOW, GLOBAL, BUT THE SPACE
OF HOSPITALS THAT DON'T HAVE ELECTRONIC HEALTH RECORDS IS
GOING TO BECOME SMALLER AND SMALLER.
CENTRE AND TRACK WERE PHENOMENALLY EXPENSIVE IN TERMS
OF TIME AND ACTUAL MONEY.
ACTUALLY WITH A BIT OF AN EFFORT WE CAN BE COLLECTING THE RIGHT
DATA AS PART OF CLINICAL PRACTICE AND WE SHOULD BE
BECAUSE THE LAST TWO DAYS HAVE SHOWN US THAT THESE ARE DATA
THAT WE SHOULD BE COLLECTING BECAUSE WE THINK IT'S IMPORTANT
TO DEFINE THE TBI PHENOTYPE.
SO I AND SO THAT WILL JUST THINK GROW AND GROW AND GROW AND IT'S
A BIG OPPORTUNITY I THINK.
SO WE SHOULD EXPLICITLY NO I AGREE.
AND THAT WOULD THEN IN THE RECOMMENDATIONS REQUIRE MORE
ATTENTION TO ALSO DEFINITIONS, HOW TO IDENTIFY POPULATIONS
ETCETERA.
SO THAT WOULD BE THAT SORT OF AN EXTRA BUT I I AGREE OTHER
PEOPLE, SPONSORS MAYBE A COUNTER QUESTION.
ARE THERE CLINICIANS HERE OR METHODOLOGIES THAT THAT THINK WE
SHOULD CLARIFY THIS TO TO HELP CLINICAL PRACTICE.
IS THERE A HOW DO YOU SAY BOTH THERE NEED, IS THERE A NEED TO
TO DO THIS EXERCISE TO BE ABLE TO BETTER INTERPRET DIFFERENT
LEVEL OF LEVELS OF OBSERVATIONAL STUDY WITH A COMPARATOR
TREATMENT ONE TREATMENT VERSUS THE OTHER.
I THINK SUSANNA AND ELOQUENTLY SAID THAT THE NEED IS THERE AND
I THINK MANY OF US AGREE, BUT I I MAYBE IT'S GOOD THAT WE DON'T
HEAR MANY COUNTER ARGUMENTS SO MAYBE WE SHOULD LEAVE THAT THAT
THEY'LL GO.
THANK YOU.
WE ARE HERE WITH ALL PEOPLE DOING CLINICAL SCIENCE AND AND
WE ARE NOT THE GENERAL PUBLIC.
SO IF I AGREE WITH THE WHITE PAPER, BUT IT SHOULD ALSO TAKE
INTO ACCOUNT THAT THE ANALYZES YOU USE IN CER COMPARED TO RCT
IS FOR NORMAL CLINICIANS VERY DIFFICULT TO UNDERSTAND.
SO IT TOOK WITH ME ONE YEAR TO UNDERSTAND WHAT YOU AND THOMAS
DID AND I WAS NOT VERY HAPPY IN THE BEGINNING, YOU REMEMBER.
SO I THINK THAT IF YOU WRITE A WHITE PAPER, THE AUDIENCE SHOULD
BE BROADER THAN PEOPLE SITTING HERE.
IT SHOULD BE PROBABLY FOR A VERY BIG AUDIENCE OF PEOPLE WHO ARE
NOT STATISTICAL TRAITS OR A LITTLE BIT.
I THINK THAT'S A VERY IMPORTANT COMMENT BECAUSE WHEN WHEN WE
UNDERSTAND IT AND ADOPT IT, THAT MIGHT NOT HOLD FOR ANYONE WHO'S
NOT INVOLVED IN ANY CLINICAL RESEARCH.
YEAH, SO, SO DEFINITELY THAT THERE MAY BE WIDER SCOPE FOR
COMPARATIVE EFFECTIVENESS AND EXPLAINING IT TO MORE NORMAL
PEOPLE IN A SENSE.
AND ALSO WHAT I WONDERED ABOUT IN THE AIM AS YOU PUT IT, YOU
SAID MAXIMIZE USE CAN CAN BE REPLACED THAT BY TO MAKE DEFINE
SAY MINIMAL REQUIREMENTS.
SO NOT MAXIMIZE BUT MINIMIZE REQUIREMENTS IN LINE WITH WHAT
WE KNOW FROM CAUSAL INFERENCE LIKE POSITIVITY DEFINITION.
YOU NEED TO KNOW WHAT YOU'RE TALKING ABOUT THAT IF YOU SAY
THIS IS A TREATMENT, IT'S NOT VERY LOCALLY DEFINED BY PRACTICE
IN ONE CENTER, BUT IT'S REALLY IT'S A POSITIVE THE ASSUMPTION
IN THE CAUSE OF INFLUENCE RESEARCH, RIGHT THAT YOU CAN
INTERPRET THAT ACROSS SETTINGS AND THE COMPARABILITY ISSUE, THE
CONFOUNDING ACCELERATION.
YOU CAN GO THROUGH ALL THESE REQUIREMENTS THAT WE HAVE IN THE
CAUSE INFLUENCE LITERATURE AND SEE TO WHAT EXTENT IT'S FEASIBLE
TO DO THAT.
THEN FOR TBI.
SO THE WAY TO THEN ILLUSTRATE THE PAPER I GUESS YOU WILL COME
TO THAT IS EITHER GIVE EXAMPLES OR WHERE IT'S FAILED, MOST
PEOPLE CRITICIZE THE FINDINGS SAY YEAH, YOU YOU, YOU CLAIM
THIS IS COMPARATIVE BUT IT'S NOT COMPARABLE AND IT'S JUST BIASED
IN SOME SENSE OR IT IS TO SOME EXTENT SUCCESSFUL AND WELL
EXAMPLES AND ACROSS THIS RANGE MIGHT BE VERY USEFUL.
I WOULD LEARN FROM THAT AT LEAST TO UNDERSTAND WHERE WE ARE IN
TBI.
YEAH, YEAH, THANKS NOW.
SO WE DEFINITELY INDEED HAVE THE EXAMPLES NOW.
SO THAT MAXIMAL USE IS MAYBE IT IS MEANT WHAT I SAID TO TO TO
GET A BIT ANGLE OF OK WE HAVE DATA AND HOW CAN WE ANALYZE THAT
TO GET INFORMATION THAT IS USEFUL FOR CLINICAL PRACTICE
RATHER THAN BECAUSE I THINK IF YOU MANY PEOPLE WOULD SAY
OBSERVATIONAL STUDY SHOULD BE DESIGNED AS A CLINICAL TRIAL
EXCEPT THAT YOU DON'T RANDOMIZE SO SAME WAY OF DEFINING
POPULATIONS ETCETERA.
AND HERE WE BUT YOU'RE RIGHT THAT WE HAVE TO DO IT AND IT
NEEDS TO MEET CERTAIN CRITERIA BUT WE DO A CERTAIN.
SO IT WOULD BE DIFFERENT CONSIDERATION SLIGHTLY I WOULD
SAY TO SETTING ADVISING PEOPLE ON HOW TO SET UP A NEW
PROSPECTIVE STUDY VERSUS HOW TO ANALYZE DATA THAT ARE ALREADY
THERE.
SO THAT IS A BIT THE BACKGROUND, BUT THE TERM MAXIMAL I DON'T
KNOW.
BUT ANYWAY, THE AND THEN IT WOULD BE THE KIND OF WHATSOEVER.
AND THEN TRY TO.
YEAH, FOR ALSO AND WE'LL GO.
THAT'S WHY I ALSO THINK WE HAVE TO DO THIS ALTOGETHER, TO AVOID
US FROM WRITING THINGS THAT NO ONE UNDERSTANDS.
I'M WONDERING IF THERE'S NOT MAYBE TWO PAPERS, ONE HERE THAT
HELPING PERMISSIONS INTERPRET WHAT THEY SEEM TO ALL RECOGNIZE
AS COMPARATIVE EFFECTIVENESS RESEARCH BY PROVIDING THE THE
BASIS FOR HIGH QUALITY PRIOR TO THE BACK OF THIS RESEARCH IN
LIKE DIFFERENT TYPES OF DESIGNS.
FOR EXAMPLE, IF THERE'S A STUDY THAT USES INSTRUMENTAL VARIABLE
ANALYSIS, THEN AS A CLINICIAN YOU SHOULD BE LOOKING TO MAKE
SURE THAT THE ASSUMPTIONS OF THAT ANALYSIS WILL MET AND MAY
BE DISCUSSED IN THE PAPER AND SO INFLUENCING RESEARCH THIS WAY.
WHEREAS PERHAPS ANOTHER PAPER, MAYBE IT'S A SERIES, MAYBE
THERE'S ONE PAPER CALLED CLINICIANS INTERPRET AND AND TO
APPLY AND THEN ANOTHER TO HELP PEOPLE ACTUALLY DESIGN AND
EXECUTE.
BECAUSE GETTING TO THE POINT THAT IT WAS MADE EARLIER, THE
STATISTICAL METHODS INVOLVED ARE ACTUALLY VERY COMPLEX.
SO IT PERHAPS IT'S IT'S IT'S NOT THE CLINICIAN WHO'S GOING TO BE
DOING THESE ANALYSIS AND THEN THAT MAY BE TIERED TOWARD
ANOTHER BOTTOM.
I PREVIOUSLY MENTIONED THIS MORNING THAT I I THINK REALLY
ABOUT THE DISCUSSION BETWEEN THE NATION AND THE AND THE STAFF
STATION WHATEVER.
SO AND THAT BUILDS TRUST.
SO WHEN I WORK WITH HEFTER AND I DON'T UNDERSTAND WHAT'S WHAT'S
GOING ON WITH DATA.
I DISCUSSED LONG ENOUGH TO THAT I'M CONVINCED THAT WHAT SHE SAID
AND ACTUALLY EXPLAINED IT TO ME THAT IT CAN BE THAT I CAN TRUST
IT.
SO I THINK THAT THAT DIALOGUE, IT WOULD BE GOOD TO KIND OF HAVE
IT ON PAPER, THE DIALOGUE AND THE ELEMENT OF THE DIALOGUE TO
HELP CLINICIANS TRUST OR DISTRUST.
CAN I, CAN I JUST ASK THE QUESTION.
SO ONE THING THAT IS DEFINITELY REAL WORLD EVIDENCE AND WE'VE
JUST DONE A SMALL EXAMPLE OF THIS IN TBR IS REALLY, REALLY
TINY IS THE IDEA OF HISTORICAL CASE, HISTORICAL CONTROLS.
SO TO USE OBSERVATIONAL DATA AND THEN YOU SCORE WHATEVER MATCH
THE OBSERVATION RELATED TO SOME INTERVENTION THAT'S SOMEWHERE
BETWEEN THE TWO.
BUT IT'S DEFINITELY REAL WORLD DATA.
EVERY REAL WORLD EVIDENCE FALLS INTO THAT COUNT.
WOULD THAT BE PARTS OF WHAT WE GET?
I DON'T I DON'T HAVE SHARES IN IN THIS METHODOLOGY BUT I JUST
YEAH I THINK SO A LOT ALSO MADE ME THINK SO I THINK THERE'S SORT
OF RISK OF LISTING ALL POTENTIAL NAMES OF DESIGNS LIKE
INTERRUPTED TIME SERIES IVY HISTORICAL CASE IS IS A BIT THE
RISK OF I DON'T KNOW CONFUSING PEOPLE AND SO TO ME BUT I VERY
MUCH LIKE WHAT YOU SAID BEFORE YOU ARE TRYING TO FIND SOME
IDEALLY RANDOM AND IF NOT CONDITIONED ON THE COVARIUS ALL
THAT WE HOPE TO KNOW.
SO I WOULD RATHER NOW I'M THAT MADE ME THINK IN DIRECTION AND
IT ALSO RELATES TO WHAT I WOULD SAID THAT YOU SORT OF SAY OK,
WHAT ARE WE COMPARING AND HOW VALID OR WHAT ASSUMPTIONS DO YOU
THEN NEED TO MAKE TO BELIEVE THAT THAT IS A UNBIASED
COMPARISON.
THANK YOU.
THE REASON I BRING THAT ONE UP SPECIFICALLY IN TBI IS THAT IT
IT DOES POTENTIALLY ADDRESS A NUMBER OF ISSUES WITH TRAUMATIC
BRAIN INJURY TRIALS WHICH IS THEY'RE EXPENSIVE HARD TO GET
THEM AND YOU COULD HAVE ISH YOU KNOW THE NUMBER OF PEOPLE YOU
NEED TO YOU NEED TO RECRUIT.
SO AGAIN I'M NOT VESTED IN IT, BUT I JUST WONDER WHETHER IT'S
IT'S AN AREA THAT WE WANT TO EXPLORE OR NOT.
YEAH.
WE HAVE 10 MINUTES LEFT.
OK WELL, I DON'T THINK.
LET'S SEE.
OH YEAH.
MAYBE WE CAN DISCUSS ABOUT ANOTHER QUESTION.
I ONE THOUGHT I HAD AS I'VE JUST BEEN LISTENING IS PERHAPS SOME
SORT OF FRAMEWORK WOULD BE MOST USEFUL HERE.
AND I, YOU KNOW, I COME AT IT FROM SORT OF AN EPIDEMIOLOGY
PERSPECTIVE.
A LOT OF TIMES PEOPLE THINK ABOUT THE QUALITY OF EVIDENCE,
RIGHT?
AND HERE YOU'RE TRYING TO HELP CLINICIANS UNDERSTAND THE
QUALITY OF COMPARATIVE EFFECTIVENESS RESEARCH STUDY.
SO PERHAPS MAYBE PART OF THE WHITE PAPER, IT COULD BE VERY
SOMETHING VERY SIMPLE BUT LIKE A VISUAL SORT OF IF THIS, THEN
THAT OR SOMETHING LIKE THAT FRAMEWORK.
AND YOU KNOW, PEOPLE LOVE ACRONYMS AND THINGS LIKE THAT,
BUT REALLY USING THAT.
SO SOMETHING THAT'S EASY TO THINK THROUGH, EASY TO SAY THIS
MEETS THIS CRITERIA AND THIS DOES NOT.
I FOUND THAT VERY HELPFUL IN MY WORK.
SO YEAH, VERY NICE SUGGESTION.
THANKS.
I LIKE IT.
ANOTHER QUESTION I I, I DON'T KNOW ABOUT ANYBODY ELSE, BUT I'M
GETTING REALLY STUCK ON THE TERM NOW.
WHAT ARE WE COMPARING THE EFFECT OF?
GOOD QUESTION.
YEAH.
I MEAN TO ME ARI'S LAST SUGGESTION WAS THE FIRST TIME
WE'VE ACTUALLY TALKED ABOUT COMPARING EFFECTIVENESS RESEARCH
AT LEAST BY THE YOU KNOW, TYPICAL AMERICAN DEFINITION
COMPARING AN INTERVENTION TO PEOPLE WHO WEREN'T TOLD TO DO
ANYTHING IN PARTICULAR DOES CENTRE TBI HAVE DATA ON FOLLOW
UP INTERVENTIONS THAT THAT PEOPLE VOLUNTARILY SUBMITTED TO
NOW SO CENTRE TBI WAS PURELY OBSERVATIONAL, SO NO ONE WAS
TOLD TO DO ANYTHING BASICALLY.
SO THAT'S THAT'S LEADS TO VARIABILITY BOTH BETWEEN PAGE IN
TREATMENT I MEAN AND TREATMENTS DEFINES I THINK IN THE.
BROAD SENSE AND JENNIFER CORRECT ME IF I'M WRONG, BUT IN MY VIEW
IT CAN BE LIKE YOU SAID ADMISSION OR NOT ADMISSION, BUT
WE WILL NOT THE IN CONTEXT WE ARE TALKING ABOUT WE WILL NOT
TELL PEOPLE TO ADMIT A SPECIFIC GROUP OF PATIENTS OR AND ALSO
NOT RANDOMIZED PATIENTS BETWEEN.
BUT IN REALITY SOME PATIENTS WILL BE ADMITTED AND SOME WILL
BE NOT.
SOME PATIENTS WILL BE OPERATED ON SOME WILL NOT.
SO THE THE THE AND THAT IS THAT IS WHAT WE CONSIDER THE
TREATMENT GROUPS LIKE YOU KNOW AND AND BUT YEAH THAT THEY COULD
BE DISTINGUISHED ON EVERYTHING THAT WE WANT TO ESTIMATE THE
EFFECT OF THE THE PREMISE OF SANTA TBI IF I SAY CORRECTLY WAS
THE LEVERAGING NICE WORD LEVERAGING THE THE VARIABILITY
IN TREATMENTS TO IDENTIFY BEST PRACTICES AND THE VARIABILITY IS
IN PART CAUSED BY RANDOM VARIABILITY BECAUSE WHEN YOU ARE
TREATED IN HOSPITAL A, THERE WILL BE ANOTHER HISTORY ANOTHER
CUSTOM AMONG CLINICIANS THEN IN ANOTHER CENTRE AND THAT'S RANDOM
VARIATION AND THAT IS ON TOP OF VARIATION BASED ON CONFINING BY
INDICATION SO BASED ON PATIENT CHARACTERISTICS.
AND THAT'S I THINK WHAT IT'S ABOUT.
IF I IT'S NOT RANDOM VARIATION YOU HAVE TO DO ALL KIND OF
TRICKS TO I BELIEVE THAT IT'S RANDOM VARIATION.
BUT ANYWAY IT DOESN'T MATTER.
I'M NOW REALLY HESITANT TO SAY WHAT I WAS GOING TO SAY AS A AS
A VERY NAIVE POINT.
I'M SORT OF THINKING IS WHAT WE'RE TRYING TO FIND A CAUSAL
EFFECT IS SOMETHING CAUSAL.
YEAH.
SO HOW DO WE PUT THAT INTO A TERMINOLOGY THAT MAYBE DOESN'T
GET US CAUGHT UP IN EVERYBODY'S PRECONCEPTION OF COMPARATIVE
EFFECTIVENESS RESEARCH.
AND AND I I TAKE THE POINT WELL MADE ABOUT IT, THAT IT'S THAT
THE TERM IS WELL KNOWN.
I'M WONDERING HOW WELL THE TERM IS WELL KNOWN BEYOND THE PEOPLE
IN THIS ROOM AND THE PEOPLE WHO LIKE IT DOES A SHOW OF HANDS.
DOES ANYONE ELSE THINK IT'S WELL KNOWN WHEN I TALK ABOUT IT, NO
ONE KNOWS ABOUT IT IN BUT I DON'T KNOW.
I THINK IN THE US IT'S EXTREMELY WELL KNOWN, BUT PLEASE CORRECT
ME IF I'M WRONG, DAVID AND OTHERS.
I GET THE SENSE THAT THE WHITE PAPER WOULD HAVE BEEN WELL TIMED
BEFORE SENT TO TBI BECAUSE OF THIS DISCUSSION.
JUST A COMMENT, I THINK I REALLY LIKE THE IDEA OF A FRAMEWORK AND
I THINK THERE'S A CONCEPTUAL FRAMEWORK WHICH COULD BE QUITE
PEDAGOGICAL AS WELL AND ACTUALLY EXPLAINING THESE THINGS BECAUSE
IN FACT, RIGHT.
SO, YEAH, SO IN A A INDIVIDUALIZED TREATMENT EFFECTS
MODEL IS A VERY, VERY COMPLICATED MACHINE LEARNING
MODEL AND THAT'S REALLY HARD THING TO GET ACROSS TO PEOPLE.
OK.
BUT AT THE VERY OTHER END OF THE SPECTRUM IS BRADFORD HILL CAN
WRITE CRITERIA SAYING ACTUALLY IF THE STRENGTH OF THE
ASSOCIATION WITH THIS RESPONSE AND THE TIMING IS RIGHT, THEN IT
MAY WELL, IT'S MORE LIKELY TO BE CAUSAL THAN ASSOCIATIVE, RIGHT.
THAT'S MOST PEOPLE GET THAT THE NEXT THING WHICH IS PROPENSITY
SCORE IS A LITTLE BIT MORE COMPLICATED.
SO ACTUALLY JUST FIND, LET'S FIND TWO GROUPS OF PATIENTS THAT
ROUGHLY SPEAKING LOOK THE SAME BUT GOT DIFFERENT TREATMENTS.
THAT'S QUITE EASY TO EXPLAIN.
INVERSE PROBABILITY WEIGHTING IS A LITTLE BIT MORE DIFFICULT.
BUT AGAIN, YOU'RE SORT OF TRYING TO GET ALONG TO THAT KIND OF
LINE, YOU KNOW, STRUCTURAL EQUATION MODELING.
OK, BUT WE CAN DRAW SOME DIAGRAMS OF THAT AND SAY, WELL
ACTUALLY WHAT ABOUT IF IF THIS EFFECT IS DIFFERENT THAN THIS,
THEN MAYBE THERE'S A DIFFERENT AND THEN EVENTUALLY GET
SOMETHING REALLY COMMON.
SO THERE IS A GRADUATION OF OF INTELLECTUAL COMPLEXITY AS
OPPOSED TO THESE MODELS AND A FRAMEWORK MIGHT BE A REALLY GOOD
THING.
I DON'T THINK WE HAVE TO WRITE A PAPER.
WE HIRE YOU TO MAKE A 15 MINUTE VIDEO AND EVERYBODY WILL.
I SEE 22 MODELS.
THE ID MODEL WHEN YOU'RE A PHYSICIST AND AN INTENSIVIST AND
VERY SMART ON IT.
AND MY MODEL LIKE TALKING A LOT TO KNOWLEDGE STATISTICIANS.
SO SOMEWHERE IN BETWEEN, THERE MUST BE A GREY AREA.
YEAH, GO AHEAD.
YEAH, SO SURE.
SO I THINK WE HAVE A LOT, WE HAVE HEARD A LOT OF USEFUL
THINGS.
SO I SUGGEST WE JUST BECAUSE WE, SINCE WE DON'T HAVE SO MUCH
TIME, WE JUST LEAVE THE SLIDES AND PLEASE BRING UP EVERYONE
THING THAT YOU STILL WANTED TO SHARE.
YEAH.
SO THE ONLY THING THAT'S WORRYING ME, AND I'M, I'M A
REHABILITATION PERSON, SO I'M SORT OF RIGHT AT THE OTHER END
OF THE SPECTRUM IS JUST ALL OF THE FACTORS WE WOULD HAVE TO BE
TAKING ACCOUNT AND THINKING ALL THE TIME OF THE FACTORS THAT
INFLUENCE DECISION MAKING.
SO FOR EXAMPLE, PEOPLE WHO GET MORE REHABILITATION SEEM TO DO
WORSE AND THAT'S BECAUSE THEY NEED MORE REHABILITATION BECAUSE
THEY'VE GOT BIGGER DEFICITS AND I'M SURE ALL THOSE SORTS OF
DECISIONS ARE MADE IN THE INTENSIVE CARE SETTING AS WELL.
SO IT ALL, IT KIND OF SCARES ME BECAUSE ALL THE TIME THIS TYPE
OF RESEARCH ENDS UP FINDING OUT THAT THE TREATMENT DIDN'T WORK
FOR FOR EXACTLY THOSE REASONS BECAUSE THE PEOPLE WHO NEEDED
THE TREATMENT WERE WORSE TO START WITH.
SO THAT'S JUST SOMETHING YOU HAVE TO REALLY THINK THROUGH AND
TAKE INTO ACCOUNT.
YEAH, IN DOING THIS, NO, I AGREE.
SO WE HAVE ALL THESE METHODS TO TRY TO AVOID THIS, BUT THEY ALL
HAVE THEIR LIMITATIONS AS WELL.
SO I, I, I CALL THIS OVER TO AVOID OVER AND UNDER
INTERPRETATION.
I THINK ON BOTH ENDS OF THE SPECTRUM THERE IS ROOM FOR
IMPROVEMENT STUDIES THAT WE THINK ARE ALL WELL DONE AND
DESERVE FOR CAUSAL INTERPRETATION THAT THEY
ACTUALLY GET THAT.
BUT THEN ON THE OTHER END OF THE SPECTRUM, THERE ARE MANY STUDIES
WHERE WE WILL PROBABLY ALL THINK THAT THEY DO NOT DESERVE A
CAUSAL INTERPRETATION.
BUT THEN IN THE CONCLUSION IT SAID PATIENTS DO BETTER WHEN
THEY RECEIVE LESS REHABILITATION AND THAT'S WHAT WE ALSO WANT TO
AVOID.
SO YEAH, MAKING PEOPLE CLEAR ON BOTH THE RISKS AND MAYBE THE THE
FACT THAT THEY ONLY CONSIDER AN RCT A GOLD STANDARD, I THINK
THERE'S BOTH SHOULD BE ADDRESSED AT THE START OF THE CENTER TV.
I HAVE MADE NOTHING WORRIED FOR WHAT WHICH IS DISABLED TO SHOW.
SO TO MY UNDERSTANDING YOU ARE MR.
HAPPY MOST OF THE ANALYZED DPI.
SO MY QUESTION IS, HAVE YOU FOUND ANY EVIDENCE THAT SOME
PRACTICES ARE BETTER THAN OTHERS?
YEAH.
WERE YOU IN ANTWERP LAST YEAR?
THERE I HAD TO GIVE A TALK ON THE HIGHLIGHTS OF COMPARATIVE
FACTORY SHARES FROM CENTER TBI.
SO I THINK THERE ARE ACTUALLY 2 EXAMPLES SO FAR FROM CENTER TBI.
IT'S THE PAPER THAT WE WROTE WITH MACHO AND WITH THOMAS.
AND I KNOW SOMEWHAT MORE EXAMPLES REALLY MADE IT TO HIGH
IMPACT JOURNALS.
AND WHERE I DARE TO CLAIM THAT'S WHAT WE FIND IS THE YEAH OR TRUE
OR CAUSAL EFFECT.
BUT THERE ARE ALSO MANY OTHERS WHERE WE TRIED AND WE MISSED
EITHER AN INTERMEDIATE ENDPOINT OR WE WERE NOT CONFIDENT THAT
THE GROUP GROUPS WERE COMPARABLE IN THE END.
SO YEAH, IT'S TRICKY.
BUT YEAH, LIKE I SAID, WE HAVE THESE RULE DATA.
THERE'S YEAH.
WHAT'S THE OTHER WAY TO MOVE FORWARD?
I DON'T KNOW WHAT THAT SHOULD BE.
IT'S IT'S WORTH REFLECTING THAT RANDOMIZED CONTROL TRIALS DON'T
PROVE CAUSALITY.
OK, THEY'RE QUITE GOOD EVIDENCE FOR CAUSALITY, BUT THE REASON WE
DO A RANDOMIZATION IS TO TRY AND MAKE SURE THAT ALL THE FACTORS
THAT WE THINK MIGHT BE CONFOUNDERS OR INFLUENCE STUFF
ARE EQUALLY DISTRIBUTED IN THE TWO DIFFERENT GROUPS.
AND YOU CAN THEN POST TOP CHECK THAT AND SAY, WELL LOOK, THE
AGES WERE THE SAME IN THE TWO DIFFERENT GROUPS, AND THE
ADMISSION WHATEVER WAS THE SAME AND WHATEVER.
BUT YOU DON'T CHECK FOR THE THINGS YOU HAVEN'T THOUGHT OF,
SO YOU CANNOT BE 100% SURE THAT THERE WASN'T A MASSIVE IMBALANCE
OR SOMETHING YOU HAVEN'T THOUGHT OF LOOKING FOR BETWEEN THE TWO
GROUPS, AND THEREFORE YOU HAVEN'T PROVEN ANYTHING.
NOW IT'S PROBABLY VERY UNLIKELY IF YOU'VE DONE THINGS WELL, BUT
IT'S NOTHING IS ACTUALLY A PROOF OF ANYTHING.
THERE'S ALWAYS VARIABILITIES.
IT'S IT'S STATS, LINDA.
YEAH, YOU CAN WRAP UP.
SO ACTUALLY WE DO HAVE ABOUT 10 MINUTES LEFT TO TO THINK ABOUT
WHAT WE'VE ALL TALKED ABOUT.
I'M GOING TO FLICK THROUGH ALL OF THESE, BUT TO TAKE US BACK TO
THAT ORIGINAL QUESTION THAT WE STARTED WITH, WHICH WAS WHAT
WOULD BE WHAT WOULD WE PUT FORWARD WITH THE WEALTH OF DATA
THAT WE HAVE AND THE WEALTH OF DATA THAT WE COULD COLLECT AND
WHAT WOULD THAT LOOK LIKE AND WHEN WE ALSO NEED TO REVISIT THE
TERM NOW THAT WE'VE HAD THAT CONVERSATION.
SO BECAUSE WE HADN'T, WE HADN'T RESOLVED THAT ONE.
SO DOES ANYONE HAVE ANY COMMENTS THEY'D LIKE TO MAKE?
WE'VE, YOU KNOW, PULLING THIS TOGETHER WITH, YOU KNOW, WE'VE
LANDED ON THE FACT THAT A WHITE PAPER WOULD BE GOOD, THAT A
FRAMEWORK AND THAT THERE'S A LOT OF EVIDENCE OR THERE'S A LOT OF
DATA OUT THERE AND IT COULD BE PASSED INTO THE DIFFERENT KINDS
OF LAYERS OF EVIDENCE AND WE COULD GET OUTCOMES.
SO IF WE'RE GOING TO ASK FOR POTS OF MONEY IN THIS PLAY IN
THIS FIELD, WHAT WOULD WE ASK FOR OR WHAT WOULD WE DO WITH
THAT MONEY OR WHAT WOULD WE DO WITH THAT MONEY?
SO IF WE WERE, IF WE WERE GOING TO DO SOMETHING IN THIS FIELD
THAT COST MONEY, WHAT WOULD BET THAT DO AND HOW WOULD IT HELP
PATIENTS?
YOU KNOW WHAT?
WHAT WHAT COULD WE DO THAT WOULD HAVE THE MOST BANG FOR THAT
BUCK.
SO MY ALL OF THESE TECHNIQUES REQUIRED TO HAVE REALLY LARGE
THAT'S THE BEST TRADE OFF.
SO YOU CAN FIND AN EQUIPPED TWIN WITH ALL THE OTHER THINGS THAT
EXACTLY AS YOU SAY THAT THAT THAT HAD THE SAME ARE IDENTICAL
IN EVERY WAY EXCEPT FOR THE THING THAT RANDOMLY THEY DID OR
DIDN'T GET.
SO FOR ME, IT'S ABOUT GETTING REALLY BIG SETS OF DATA.
AND I AND I, AGAIN, THE APOLOGIES HAVE BEEN SORT OF
BORING ABOUT IT.
I THINK ONE OF THE REALLY IMPORTANT THINGS IS THAT WE
STANDARDIZE WHAT WE'RE TRYING TO COLLECT SO THAT WE CAN EXTRACT
THINGS OUT OF BIG ELECTRONIC HEALTH RECORDS.
AND IN ORDER TO DO THAT, SOMEBODY NEEDS TO DO THE REALLY,
REALLY BORING, REALLY TEDIOUS BUT UNBELIEVABLY INCREDIBLY
IMPORTANT TASK OF MAPPING ALL THE THINGS WE THINK OF IN
TRAUMATIC BRAIN INJURY TO SOME KIND OF COMMON DATA MODEL LIKE
LIKE O MOP OR OR WHATEVER IT IS.
SO I WAS JUST LOOKING HERE.
SO YOU KNOW, TRAUMATIC BRAIN INJURY.
IT CAN BE 127295002 IN SNOW MED OR 124209 IN CIL OR 127-295-0202
IN NEBRASKA LEXICON.
AND AS QUITE A LOT OF THESE OR ICD 10 S 06, I KNOW THAT ONE OFF
MY HEART.
YOU NEED TO DECIDE WHICH ONE I'M GOING TO DO AND THAT'S BEEN DONE
BY PEOPLE ALREADY AND AND REALISTIC THAT WITH MY NON TBI,
NON HOSPITALS CCIO HAT ON.
I AM NEVER GOING TO MAP MY ELECTRONIC HEALTHCARE RECORD TO
NINDS COMMON DATA ELEMENT.
SORRY IT'S NOT GOING TO HAPPEN.
IT'S TOO NICHE.
I'M NEVER GOING TO DO THAT.
BUT WHAT I AM GOING TO DO IS MAP IT TO O MOB.
BECAUSE O MOB HAS BEEN USED BEFORE FOR DOING COMPARATIVE
DRUG TRIALS AND THAT ALLOWS MY ELECTRONIC HEALTH RECORDS TO BE
COMPARED WITH YOUR ELECTRONIC HEALTH RECORD COMPARED TO BE
SENT TO TBI EVERYTHING SUDDENLY FOR FREE I CAN I CAN REPEAT SENT
TO TBI FOR YOU.
SO I WOULD I WOULD DO THAT.
SO THERE IS TO SAY WHAT ALL THAT IS THERE ALREADY AN OMOP MODEL
FOR TBI.
WELL OMOP AND THERE ARE OTHER OTHER COMMON DATA MODEL BUT I
THINK OMOP'S A VERY GOOD ONE.
SO WHAT OMOP WILL DO IS SAY WHAT THOSE ONES I'VE JUST TOLD YOU
THE CONCEPTS ARE NON STANDARD, NON STANDARD.
THE ONE THAT IT SAYS IS STANDARD IS 127295 THE ONE I SET UP FIRST
WHICH IS THE SNOW MADE ONE.
SO THAT'S THE ONE YOU CHOOSE.
NOW IF ANYBODY ELSE IN THEIR ELECTRONIC HEALTH RECORD HAS
USED CIL, NEBRASKA, PPI DOESN'T MATTER BECAUSE IT ALSO SAYS THAT
IT LINKS TO THE SNOW MADE ONE.
SUDDENLY YOU GET COMPLETE INTEROPERABILITY OF ALL DATA
SETS.
BUT SOMEONE HAS TO GO THROUGH THE REALLY TEDIOUS TASKS SAYING
THESE ARE THE THINGS THAT WE THINK ARE IMPORTANT ON THE BASIS
OF THE LAST TWO DAYS FOR DEFINING THE SENSITIVITY THE THE
TBI PHENOTYPE AND THERE MAY BE THINGS THAT AREN'T IN SNOW MED,
RIGHT, LIKE FOR EXAMPLE POST TRAUMATIC AMNESIA ASSESSMENT.
IF THAT'S NOT WE NEED TO GET IT ADDED TO SNOW MADE AND ACTUALLY
THAT'S QUITE EASY TO DO.
SO THOSE ARE VERY IMPORTANT AND DEEP IN THE WEEDS POINTS ONE OF
THE OUTPUTS OF TODAY AND TOMORROW IS INTENDED FOR US TO
UNDERSTAND PRIORITIES FOR COMMUNICATION WITH FUNDING
AGENCIES TO PROPEL THIS EFFORT INTO ITS NEXT PHASE.
AND AS WE'VE DISCOVERED AND YOU KNOW HESTER ALMOST MORE THAN
ANYONE IN THIS ROOM EXCEPT MAYBE ABOUT HAVE DISCOVERED THAT WE'VE
BEEN CHALLENGED OVER THE LAST FIVE YEARS TO CONVINCE FUNDING
AGENCIES TO SUPPORT THIS VERY WORK.
AND THE TRICK IS WHAT LINDY'S GETTING AT, WHICH IS WHAT'S THE
DELIVERABLE AT THE END?
BECAUSE IF THE DELIVERABLE AT THE END IS GOING TO CHANGE
CLINICAL PRACTICE, THEN WE CAN JUSTIFY THE RESOURCES NECESSARY
TO GO AS DEEP INTO THE WEEDS AS AS NECESSARY.
SO I THINK AN IMPORTANT THING AND THE THE WISDOM OF THE CROWD
IS HOW DO WE IDENTIFY THE PRIORITIES INSIDE OF
OBSERVATIONAL DATA THAT WOULD MOST IMPACT CLINICAL PRACTICE SO
THAT WE COMPEL THE NECESSARY RESOURCES TO DRIVE THE EFFORT.
SO AGAIN, I MEAN, I DO HAVE A BIAS HERE, OF COURSE, BUT SO I
THINK WHAT, WHAT A MAPPING ALLOWS YOU TO DO IS 2 THINGS.
THE FIRST THING IS THAT THAT GROUP'S OUTPUT ALLOWS US NOT TO
HAVE TO REINVENT THE WHEEL EVERY SINGLE TIME.
SO THE MOMENT IF I WANT TO HARMONIZE CENTRE AND TRACK, I'LL
HAVE TO HARMONIZE CENTRE AND TRACK.
AND THEN WHEN YOU COME ALONG WITH TRACK X OR SOME NEW STUDY,
I'LL HAVE TO DO IT ALL OVER AGAIN AND HARMONIZE IT AGAIN.
WHEREAS ACTUALLY, IF EVERYBODY AGREES ON THE SAME SET OF DATA
HARMONISED TO AN AGREED INTERNATIONAL STANDARD, THEN YOU
DON'T HAVE TO DO THAT.
YOU JUST NEED TO COLLECT IT.
AND IF YOU COLLECTED IT IN A DIFFERENT STANDARD, THERE'S A
MAPPING BACK TO THE INTERNATIONAL ONE.
SO THAT'S THE FIRST THING THAT ALLOWS YOU.
IT SAVES A HUGE AMOUNT OF TIME AND OPENS UP ALL OF THE
ROUTINELY COLLECTED DATA, AT LEAST AS FAR AS ELECTRONIC
HEALTH RECORDS ARE CONCERNED.
THE CAVEAT OF HIGH INCOME COUNTRY NOTWITHSTANDING.
THE OTHER THING IT DOES IS IT ACTUALLY ALLOWS US TO DEFINE AS
A GROUP WHAT IS IT THAT THE DATA ELEMENTS THAT WE THINK WE ARE
IMPORTANT.
SO FOR EXAMPLE, I MISSED A TRICK IN 2014 WHEN I BUILT THE EHR FOR
MY ICU.
SO WE DIDN'T, I DIDN'T CODIFY GLASGOW OUTCOME SCORES IN AN IN
A CODED WAY, I DIDN'T CODIFY THE SOME OF THE IMPACT VARIABLES.
I WILL CHANGE THAT NOW AND I WILL CODE THEM IN ACCORDING TO
OMAR SO THAT I'VE GOT THEM IN THE FUTURE AND OTHERS MAY WANT
TO GO AWAY.
SO DO YOU KNOW WHAT THESE ARE THINGS THAT WE'RE NOT ACTUALLY
CAPTURING IN A IN A STRUCTURED WAY.
WE SHOULD BE DOING IT AS PART OF CLINICAL CARE BECAUSE IT'S PART
OF THE ATVI PHENOTYPE AND THEREFORE RELEVANT TO US.
BUT THAT FINAL LIST OF THINGS ALLOWS PEOPLE TO DO IT IN THE
APPROPRIATE WAY AND NOT COLLECT THE WRONG THING.
SO CAN I JUST CARRY ON FROM DAVID'S POINT, I I THINK THAT'S
REALLY, REALLY VALID AND WOULD ENABLE THE WORK.
BUT WHAT IF THERE WAS ONE THING THAT WE WANTED TO BE ABLE TO
TEST WITH THE OBSERVATIONAL DATA THAT WE HAVE THAT YOU THINK WE
WOULD BE ABLE TO THEN MODIFY TO IMPROVE OUTCOMES IF WE KNEW THAT
SUCH COULD IMPROVE OUTCOMES?
IS THERE SOMETHING THAT, YOU KNOW WHAT?
WHAT WOULD WE TEST?
SO YOU'RE ASKING FOR A SPECIFIC INTERVENTION?
YEAH.
BECAUSE IF I'M GOING TO PUT A SLIDE TOGETHER, I'VE GOT A LIST
OF ALL THESE GREAT POINTS EVERYONE'S MADE OF HOW WE'D
HARMONIZE THE DATA AND ALL THAT.
BUT THE LAST BULLET WHO WANT TO SAY, AND WE'D USE THIS TO PROVE,
TO ADDRESS THIS SPECIFIC RESEARCH QUESTION, CLINICALLY
RELEVANT.
YEAH.
BUT DO WE NOW HAVE TO PICK, I TOTALLY AGREE WITH WITH DAVID
THAT THERE SHOULD BE KIND OF A DELIVERABLE TO KEEP THAT WE MUST
HAVE A REASON TO TO PURSUE THIS.
BUT JUST TO TRY TO END ALSO THE DISCUSSION, SHOULDN'T WE TRY TO
THINK ABOUT HAVING IN THE WHITE PAPER THE INFORMATION THAT
DRIVES?
I I THINK WE DEFINITELY DO THAT.
BUT THE FUNDERS ARE NOT GOING TO SAY WE'RE GOING TO GIVE YOU POTS
OF MONEY TO WRITE A WHITE PAPER, SORRY.
SO YEAH, I THINK SO WE DEFINITELY DO THAT BUT AS WELL
AS WHAT WHAT CLINICAL THING WOULD WE TEST, I THINK.
BUT THEN I THINK THAT'S NOT THE DISCUSSION.
I'M NOT SURE TIME ALLOWS.
IF I COULD GIVE YOU A SENSE OF TBI DATA SET OR TRACK TBI DATA
SET, THAT'S 20,000 PATIENTS.
WHAT FEDERATION ANALYSIS WOULD YOU DO?
YEAH, WHAT WOULD BE INTERESTED IN FOR EXAMPLE, GUIDELINES IS
ABOUT, I THINK ALSO IT'S NOT ONLY ABOUT ICU ADMISSION OR
ABOUT NEUROSURGICAL INTERVENTIONS, BUT IT'S ALSO
ABOUT DISCHARGED FROM THE EMERGENCY DEPARTMENT OR NOT.
SO AND WE DON'T KNOW A LOT OF THOSE PEOPLE WHO ARE DISCHARGED
FROM THE EMERGENCY DEPARTMENT.
THERE'S A LOT OF DISCUSSION WITHIN GUIDELINES HOW TO LOOK AT
THESE PEOPLE'S GET ACT SCAN THEN DISCHARGE THEM OR EVEN IF THEY
HAVE ACT CAN JUST DISCHARGE THEM.
WHATEVER THE CT CAN SCAN DOES IS WHAT YOU SEE IN THE CT SCAN.
SO I WOULD LIKE TO KNOW WHAT'S THE POLICY OVERALL, FOR EXAMPLE
IN THOSE WITH MILD TBI ADMITTED TO THE EMERGENCY DEPARTMENT WHO
IS DISCHARGED OR NOT, WHAT'S THE POLICY IN CT, YES OR NO,
ABNORMALITY YES OR NO?
AND WHAT'S THE OUTCOME?
I THINK THAT WOULD BE COULD BE A QUESTION TO BE ANSWERED FROM THE
DATA.
THANKS VERY MUCH.
ANYONE ELSE, LET'S GET A FEW IDEAS.
SO I'M NOT A CLINICIAN.
HOWEVER, I'VE WORKED WITH MANY AND THEY ALWAYS SEEM TO HAVE
THIS SAME QUESTION REGARDING OLDER ADULTS AND WHETHER
NEUROSURGICAL INTERVENTION AMONG OLDER ADULTS.
I MEAN, THIS IS JUST PUTTING THIS OUT THERE.
I'M SURE THERE'S TONS OF CLINICIANS OUT HERE, BUT WHETHER
NEUROSURGICAL INTERVENTION IN OLDER ADULTS WAS BENEFICIAL, BUT
UNRELATEDLY SOMETHING THAT WAS FUNDABLE.
I WAS GOING TO SAY YOU KNOW METHODS SOMETIMES SOMETIMES CAN
BE FUNDABLE AND I KNOW THAT RIGHT NOW FOR EXAMPLE, THERE ARE
CALLS AND AGING RESEARCH TO FUND METHODS THAT ARE SPECIFIC TO
THAT KIND OF RESEARCH, SO SPECIFIC TO THAT POPULATION.
SO IS THERE ANY WAY TO FUND METHODS SPECIFIC TO THESE TBI
DATA SETS THAT WOULD THAT WOULD INFORM THE RESEARCH?
SHANA, FIRST WEEK, SHANA, YOU WE'RE LOOKING FOR TO HAVE THE
RESULTS FROM GENDER AND YOU HAVE SPECIFIC EXAMPLES FROM YOUR THAT
YOU DON'T KNOW YET.
WELL, I PERSONALLY HOPE YOU KNOW I'M PARTICULARLY INTERESTED IN
THE BIAS BY CLINICIANS THAT GOES INTO THE WITHDRAWAL BY STATE
THERAPIES ESPECIALLY FOR THREE DAYS.
SO AS WE ALL KNOW, IN SOME GUYS.
FOR HOW PEOPLE LIVE FOR, LET'S SAY, 12 MONTHS, SOME OF THEM
MAJOR SURPRISINGLY WELL.
AND SO, YOU KNOW, YOU MIGHT WANT TO CALL THIS POSITIVE DEVIANCE
RESEARCH.
SO WHY DO THOSE PEOPLE DO WELL AND IS IT REALLY THE TREATMENTS
THEY'RE RECEIVING AND THE AGGRESSIVENESS OF THE CENTER OR
THE NEUROSURGEON?
I'M VERY, YOU KNOW, BASED ON WHAT I SEE IN EVERYDAY PRACTICE,
THERE'S A HUGE AMOUNT OF HEURISTICS THAT GOES INTO
DECISION MAKING AND JUST DECIDING.
I'M NOT GOING TO OFFER SURGERY TO AN OLD PATIENT, 'CAUSE I KNOW
THEY'RE GOING TO DO WELL, POORLY.
WELL, HOW DO WE KNOW THAT, RIGHT?
I MEAN, WE'VE HAD THAT CONVERSATION MANY TIMES.
WHAT I DON'T KNOW IS WHETHER OBSERVATIONAL DATA IS SUFFICIENT
TO STUDY THAT.
THAT I I DOUBT THAT TO BE HONEST, BUT WOULD LOVE YOUR
INPUT.
I THINK THAT'S THE BIGGEST PROBLEM IN OUR WORLD IS I'M
OBVIOUSLY BIASED, BUT I WOULD ALSO LOOK AT VARIABLES THAT MANY
PATIENTS ARE EXPOSED TO INCENTIVE TBI, LET'S SAY GLUCOSE
AND IT'S MANAGEMENT TEMPERATURE AND IT'S MANAGEMENT.
SO AS TO KIND OF COPY PASTE WHAT WE DID WITH THE FLUIDS TO SEE IF
CLINICS WHERE THEY HAVE OTHER THRESHOLDS MIGHT FARE WORSE OR
THE PATIENTS VERSUS BETTER.
SO THAT WOULD BE MY FIRST.
YEAH.
YEAH, I I'M, I THINK THEY BOTH ARE FASCINATING AND I THINK THE
THE HEALTHCARE UTILIZATION OR OUTCOMES OF YOU KNOW OR
EFFECTIVE NIHILISMS REALLY.
AND I THINK IT IS ACTUALLY SOMETHING THAT YOU CAN GET A
HANDLE ON USING OBSERVATIONAL DATA BECAUSE YOU WILL FIND 2
PATIENTS WHO ARE, IF THE DATA IS SUFFICIENTLY GOOD DESCRIPTION OF
THE PHENOTYPE OTHERWISE IDENTICAL, RIGHT IN TERMS OF NOT
ONLY THEIR PHENOTYPE BUT EVERYTHING THAT HAPPENED TO
THEM.
YOU'D BE A LOT OF DATA FOR THAT.
BUT YET ONE DOES IS WITHDRAWAL, THE OTHER ONE ISN'T.
AND YOU CAN START TO GET A HANDLE ON THAT WITH ON SHIBAYA'S
ORIGINAL PAPER THAT I DID, THE METHODOLOGY THAT SHIBAYA THEN
EXTENDED, HE'S LOOKING AT ALL OF DATA WITHIN MIMIC, WHICH IS A
BIG ICU, AMERICAN ICU DATA SET.
WE JUST DID SOME MORTALITY PREDICTION AND ACTUALLY ONE OF
THE THINGS THAT JUST DROPPED OUT OF THE OF THE MACHINE LEARNING
MODEL, PERHAPS NOT SURPRISINGLY WITHOUT IT WAS THE REFERRAL TO
THE PRIEST ACTUALLY BECAUSE IT'S CODED RIGHT IS A STRONG NEGATIVE
PROGNOSTIC SIGNIFICANCE.
THOSE THAT THAT THAT DATA ABOUT WHAT PEOPLE ARE DOING AND WHAT
IS ACTUALLY IN THERE.
IN SOME WAYS SOMETHING A BIT SNEAKY ABOUT HOW YOU HOW YOU GET
AT IT, BUT I THINK YOU PROBABLY CAN GET A HANDLE ON THAT.
THOSE SORTS OF QUESTIONS YOU MENTIONED THE WORD IDENTICAL,
ARE THEY THEY HAVE TO BE IDENTICAL ACROSS ALL DIMENSIONS
OR DO MORE?
DO SOME DIMENSIONS MATTER MORE?
AND I THINK THAT NEEDS TO BE ADDRESSED BEFORE YOU CAN GET
THERE.
IT WILL NEVER BE ENTIRELY UNETHICAL, I THINK WE AGREE ON
THAT.
SO JUST BEING A MEMBER OF THE PSYCHOSOCIAL ENVIRONMENTAL
MODIFIERS GROUP, WE HAD A REAL LACK OF DATA TO SUPPORT OUR
MODIFIERS, AND I DON'T KNOW HOW MUCH, HOW COMPREHENSIVELY.
SO FOR EXAMPLE, ABOUT CULTURE OR LANGUAGE OR SO IF, AND IT SEEMS
LIKE THIS IS SOMETHING WE COULD GATHER, YOU CAN LOOK AT
COMPARATIVE INFLUENCE OF FACTORS ON OUTCOME WITHOUT IT BEING TOO
CONFOUNDED.
SO YEAH, THAT WOULD BE SOMETHING TO PICK SOME KEY ENVIRONMENTAL
SOCIAL FACTORS.
IF YOU HAVE THE DATA, I DON'T KNOW IF YOU COLLECT THE DATA.
SO THAT'S WHAT I'D LIKE.
YES, I DON'T WANT TO REPEAT MY FLASH PRESENTATION, BUT I OF
COURSE HEAVILY INVESTED IN THE SURGICAL QUESTIONS BUT FOR THE
FUTURE MATH ANALYSIS BETWEEN THE THREE LARGE STUDIES NOW.
BUT THE OTHER THING IT OFFERS IS THAT IT IS ONE OF THE RARE
OCCASIONS AND WE CAN COMPARE A SURGICAL INTERVENTION WITHIN
CENTER TBI.
IT WAS ACTUALLY ALSO STUDIED IN A RANDOMIZED SESSION WHICH
RESCUE ACH OF COURSE AND WE DID THAT CRANIOTOMY FIRST CRANECTOMY
PARALLEL TO RESCUE HDH AND IT SHOWED THE EXACT SAME EFFECT
ESTIMATES EVEN THE CONFIDENCE IS OVER VERY COMPARABLE.
OF COURSE RESCUE HH IS WAY MORE VALID, BUT IT SHOWS THE STRONG
MARKETING POTENTIAL OF OBSERVATIONAL EVIDENCE BECAUSE
YOU CAN SEE WELL IF WE JUST ADHERE TO ALL THE ASSUMPTIONS
FOR THE FOR THE OBSERVATIONAL EVIDENCE OF THE IVNS, YOU CAN
ALMOST ACQUIRE THE SAME LEVEL OF VALIDITY.
AND OF COURSE BOTTOM LINES, YOU CAN NEVER PROVE CAUSAL INFERENCE
OF COURSE, BUT THAT'S NOT RELEVANT.
YOU HAVE TO SHOW WHICH ASSUMPTIONS ARE ACTUALLY MET OR
NOT, AND THAT'S WHAT THE DISCUSSION SHOULD BE I THINK.
AND A SURGERY MAKES A STRONG CASE, BUT I'M VERY HEAVILY
CONFOUNDED BEING A NURSE SURGEON, BUT I REALLY LIKE THAT
EXAMPLE WITH ALSO IN THAT WHITE PAPER.
AND THAT'S THE OTHER THING IS BECAUSE YOU'VE BEEN MAKING
GUIDELINES TO ALSO DEFINE KNOWLEDGE GAPS.
AND THOSE KNOWLEDGE GAPS IS ALWAYS RELATED TO WHAT DO WE
KNOW ABOUT WHAT WE DO AND IT GIVES THAT ENOUGH INFORMATION TO
DISCHARGE PATIENTS.
OR IS IT JUST AN IDEA THAT YOU REALLY WANT TO KNOW WHAT ACT
SCAN LET'S SHOWS TO US MAYBE WE CAN DISCHARGE EVERYBODY WITH A
MILD, I'M NOT SUPPOSED TO SAY THAT, BUT MAYBE WE CAN DISCHARGE
EVERYBODY WITH A MILD TBI.
IT DOESN'T MATTER.
THEY COME BACK WHEN THEY HAVE COMPLAINTS.
SO THE KNOWLEDGE GAPS GIVE YOU SOME IDEAS OF HOW YOU COULD
CHANGE THE GUIDELINES AND COMPETITIVE EFFECTIVENESS.
RESEARCH IS ABOUT GUIDELINES, SO YOU THAT'S HOW YOU CAN LOOK AT
IT.
IN THE END, I KNOW NOTHING ABOUT BUSH.
I'LL BE REALLY EDUCATED.
I WORK ON SEARING, BUT NOW I KNOW THAT ACTUALLY YOU'RE
INTERESTED IN THAT AND IT'S PART OF, IT'S A REALLY IMPORTANT PART
OF THE MILD MILDER END, TO USE THE OLD TERMINOLOGY SCORING,
THAT WE SHOULD BE COLLECTING THAT.
OK, SO I NEED TO GO BACK AND CLEARLY THERE'S TWO THINGS.
ONE, I NEED TO CONVINCE ALL THE OTHER PEOPLE AROUND ME TO
ACTUALLY COLLECT THAT DATA.
WELL, THAT'S THE JOB OF THE NOMENCLATURE THE LAST TWO DAYS.
BUT WHAT WE NEED TO KNOW IS I NEED TO TELL YOU THAT IN OMAR
THAT'S CONCEPT 717323005 WESTMEAD POST TRAUMATIC AMNESIA
SCALE SCORE RESERVABLE ENTRY VALID SNOW MADE, RIGHT.
SO THAT YOU CAN GO AND START COLLECTING IT ACCORDING TO THAT
SO THAT I CAN JUST ASK FOR THAT WHEN IN YOUR ELECTRONIC HEALTH
RECORD MUCH.
SO THIS IS FASCINATING.
WE NEED.
I THINK WE'RE REALLY WE MUST WRAP UP.
YEAH.
SO THANK YOU ALL INCREDIBLY MUCH FOR YOUR INPUT.
WE WILL BE HAVING A WORKING GROUP MEETING AT 2:00 TOMORROW
FOR THE PEOPLE WHO WANT TO BE INVOLVED IN WRITING THE PAPER
AND WANT TO BE A NAMED AUTHOR.
SO IF YOU WANT TO BE CONTRIBUTING TO THE MORE FINE
DETAIL ABOUT THE STATISTICAL METHODS AND ABOUT THE
HARMONISATION OF DATA AND ALL OF THE THE SORT OF THINGS THAT WE
UNFORTUNATELY HAD TO GLOSS OVER BECAUSE THERE WASN'T TIME TODAY,
PLEASE BE HERE AT 2:00 TOMORROW.
AND BEFORE WE CLOSE, CAN I JUST HAVE A SHOW OF HANDS ON THE
TITLE ON THE COMPARATIVE EFFECTIVENESS RESEARCH?
WHO WANTS TO KEEP COMPARATIVE EFFECTIVENESS RESEARCH HAS A
NAME.
I I'LL TALK TELL WE'LL TALK ABOUT ALTERNATIVES.
BUT WHO DESPERATELY WANTS TO KEEP COMPARATIVE EFFECTIVENESS
RESEARCH?
WELL, ALL RIGHT.
WHO WANTS TO CALL IT REAL WORLD EVIDENCE FOR TBI GOT ABOUT FOUR
OF THEM I DON'T THINK.
WELL, I I HAVEN'T SAID MUCH I NORMALLY DO BUT YOU YOU HAVE TO
GO IN TERMS OF OTHER PEOPLE UNDERSTAND AND WE GET IT IN THIS
ROOM.
BUT WHEN YOU GO TO YOUR GRAPH, I MEAN ONCE IT WOULD BE LIKE
CHANGING THE WORD RCT.
IT'S SO EMBEDDED IN CULTURE.
I WOULD BE VERY CAUTIOUS ABOUT LIVING AWAY.
I THINK IT'S A WELL KNOWN ONE AND AND I REALLY HAVE TO DO WHAT
MY PERSPECTIVE TO DO.
BUT IT IS THE REAL TERM.
IT'S NOT REAL THING YOU LIKE BUT IT IS ANY OTHER WHAT IS
COMPARATIVE EFFECT REAL WORLD EVIDENCE.
OK AND WHAT ABOUT PRACTICE BASED IT IT THE PEOPLE OUTSIDE THIS
ROOM WILL, WILL UNDERSTAND WHAT IT IS STILL.
I MEAN IT WILL UNDERSTAND WHAT COMPARED TO EFFECTIVENESS AND
YOU KNOW IT'S THERE AND IT'S IT'S A RECOGNIZED METHODOLOGY
WE'VE ALREADY PROVED GREAT SUCCESS CENTRE TRACK AND YOU
KNOW I'D BE VERY KEEN TO KEEP IT ALL RIGHT JUST BEARING THAT IN
MIND TOTALLY TOTALLY HEAR THAT ANY HANDS FOR PRACTICE BASED
EVIDENCE.
SORRY THAT ONE GOT VERY LITTLE AND SOME WORDING THAT HAS CAUSAL
EFFECT IN IT.
NO, SURE.
NOBODY IS FOR CARE BECAUSE IT'S ABOUT CARE.
IT'S NOT ONLY ABOUT.
SO I THINK THE MOST HANDS WERE FOR COMPARATIVE EFFECTIVENESS
RESEARCH.
SO ARE WE COMFORTABLE LEAVING THE ROOM WITH TAKING IT LIKE
THAT?
YEAH.
COOL.
THANK YOU VERY MUCH AND THANK YOU VERY MUCH FOR PEOPLE ONLINE. 1260 01:09:54,957 MUCH APPRECIATED.