THANK YOU FOR JOINING US THIS
MORNING AGAIN.
WE HAD AN AMAZING DAY YESTERDAY
WITH OUR PRAGMATIC TRIALS
COLLABORATORY ANNUAL WORKSHOP.
OUR TOPIC YESTERDAY AND TODAY,
GETTING THE RIGHT EVIDENCE TO
DECISION MAKERS FASTER, INSIGHTS
FROM THE PRAGMATIC TRIALS
COLLABORATORY.
IT'S MY PLEASURE TO WELCOME YOU
AGAIN TODAY THIS MORNING.
MY NAME IS WENDY WEBER, AT THE
NATIONAL CENTER FOR
COMPLEMENTARY AND INTEGRATIVE
HEALTH HERE AT THE NATIONAL
INSTITUTES OF HEALTH.
I'VE BEEN WORKING WITH THE
PRAGMATIC TRIALS COLLABORATORY
SINCE WE LAUNCHED IN 2011-2012,
AND WE LEARNED A LOT OVER THE
LAST 11+ YEARS IN THE PROGRAM.
SO, WE'RE LOOKING FORWARD TO
ANOTHER REALLY EXCITING DAY
TODAY.
YESTERDAY WE HEARD FROM SEVERAL
TRIALS AS EXAMPLES WHAT THEY
HAVE LEARNED AND FROM THEIR
HEALTH CARE SYSTEM PARTNERS
REALLY LOOKING AT WHAT HAPPENS
WITH THE RESULTS OF THEIR
STUDIES AND WHAT WAS
IMPLEMENTED, SUSTAINED, CHANGED,
MODIFIED.
TODAY WE'LL TALK MORE ABOUT IS
THERE ANY WAY WE CAN GET THIS
MORNING AND LEARN RESULTS OF
TRIALS MORE QUICKLY, DO WE GET
SIGNALS EARLIER, WE'LL HAVE AN
INTERESTING CONVERSATION ABOUT
HOW TO TEASE OUT WHAT'S A SIGNAL
AND HOW MUCH NOISE IS THERE
AROUND THAT SIGNAL AND DO WE
REALLY KNOW WHAT'S HAPPENING OR
DO WE NEED TO KEEP GOING.
SO I THINK IT'S GOING TO BE A
REALLY EXCITING DISCUSSION
TODAY.
I WANT TO THANK OUR PLANNING
COMMITTEE MEMBERS.
TODAY YOU GOAT HEAR FROM SEVERAL
OF OUR OTHER PLANNING COMMITTEE
MEMBERS, FIRST SESSION MODERATED
BY DR. KEVIN WEINFURT, SECOND BY
DR. RICHARD PLATTE.
WE HAVE JUST A REALLY EXCITING
GROUP OF SPEAKERS, SOME ARE
PRINCIPAL INVESTIGATORS FROM
SOME TRIALS, AS WELL AS SOME OF
OUR STAFF, NATIONAL INSTITUTES
OF HEALTH, AS WELL AS OTHER
PARTS OF THE FEDERAL GOVERNMENT,
YOU'LL HEAR FROM PEOPLE IN
SEVERAL PERSPECTIVES AS WE THINK
ABOUT HOW DO WE LEARN FASTER AND
THEN OUR LAST SESSION TODAY IS
GOING TO BE WHAT COULD WE AS
POTENTIALLY FUNDERS AS THOSE
PROPOSING APPLICATIONS FOR
PRAGMATIC TRIALS HOW MIGHT WE
STRUCTURE OR PUT INCENTIVES TO
PLACE TO LEARN MORE QUICKLY AS
WE'RE CONDUCTING THESE TRIALS.
I KNOW IT'S EARLY ON THE WEST
COAST.
BUT THANK YOU FOR JOINING US.
FOR EVERYONE THAT IS THERE ON
THE WEST COAST JOINING US THIS
MORNING, AGAIN, THANK YOU TO OUR
KEYNOTE SPEAKER.
YESTERDAY A GREAT PERSPECTIVE
THINKING ABOUT HOW TO THINK
ABOUT LEARNING HEALTH CARE
SYSTEMS AND WHAT WE NEED TO DO
IN ORDER TO MAKE THAT HAPPEN.
SO, I THINK I'M GOING TO GO
AHEAD AND TURN THINGS OVER TO
OUR FIRST PANELIST.
OH, THANK YOU FOR THE REMINDER.
ONE HOUSEKEEPING TIP, WE WILL
PUT THIS E-MAIL UP AGAIN AS WE
GET TO THE Q&A SESSIONS BRIEFLY.
IF YOU HAVE QUESTIONS, PLEASE DO
SEND THEM TO THIS E-MAIL FOR THE
WORKSHOP.
I'M ACTUALLY MONITORING IT AND
SO I'LL BE SURE THE QUESTIONS
GET TO OUR PANELISTS AND TO OUR
SPEAKERS.
YOU CAN ALSO USE THE "LIVE
FEEDBACK" FEATURE IN THE
VIDEOCAST.
IT WILL END THE E-MAIL TO THE
SAME E-MAIL BOX TO SHARE
QUESTIONS WITH PANELISTS AND
SPEAKERS.
WE DO REALLY WANT TO MAKE THIS
ENGAGING AND SO IF YOU HAVE
QUESTIONS PLEASE DO SEND THEM IN
TO US.
SO, KEVIN, I'M GOING TO TURN
THINGS OVER TO YOU TO GET US
STARTED IN OUR FIRST PANEL
TODAY, LEARNING FASTER.
>> THANK YOU SO MUCH.
GOOD MORNING, EVERYONE.
I'VE KEVIN WEINFURT FROM DUKE
UNIVERSITY.
I'M ONE OF THE CO-P.I.s OF THE
COORDINATING CENTER OF THE
COLLABORATORY.
GOOD TO HAVE EVERYONE BACK WITH
US THIS MORNING.
AS WENDY POINTED OUT, YESTERDAY
A FEW DIFFERENT TIMES, THIS
THEME OF TRYING TO LEARN FASTER
CAME UP.
AND WE'RE GOING TO REALLY FOCUS
ON THAT IN THIS THIRD PANEL.
AND WE'LL HEAR SOME
REVERBERATIONS OF SOME PRIOR
TOPICS THAT WERE DISCUSSED, BUT
WE'LL BE LOOKING AT THEM IN THE
CONTEXT OF HOW WE MIGHT LEARN
FASTER.
AND, YOU KNOW, WE'VE OBSERVED AT
LEAST THREE DIFFERENT SITUATIONS
WITH PRIOR TRIALS, WHERE IT
MIGHT BE APPROPRIATE TO MODIFY
OR END A STUDY.
ONE SITUATION IS DETERMINATION
IN THE PLANNING FACE -- PHASE
THE INTERVENTION IS NOT FEASIBLE
OR ETHICAL.
WE'VE SEEN SITUATIONS WITH
INABILITY TO EFFECTIVELY
IMPLEMENT THE INTERVENTION.
AND THERE'S ALSO THE POSSIBILITY
THAT WE COULD DETERMINE THE
INTERVENTION HAS ALREADY MET ITS
MAJOR AIM BEFORE THE END OF THE
STUDY, OR IT'S UNLIKELY TO DO SO
WHICH YOU MIGHT REGARD AS
FUTILITY.
SOME OF THESE ISSUES ARE GOING
TO SOUND FAMILIAR FOR
TRADITIONAL TRIALS, WHERE WE
MIGHT HAVE A DSMB ADVISING A
TRIAL BE STOPPED BECAUSE OF
OVERWHELMING EVIDENCE OF BENEFIT
OR FUTILITY.
BUT WE'LL BE TALKING ABOUT SOME
OF THE ISSUES THAT COME TO PLAY
IN PRAGMATIC TRIALS.
THERE'S SOME INTERESTING WRING
WRINKLES, ONE INGREDIENT OF THE
INTERVENTION BEAK TESTED MIGHT
BE EXPLICIT FOR MONITORING
INTERVENTIONS, WE HAVE TWO MAJOR
QUESTIONS THAT WE'RE LOOKING AT
IN THIS SECTION.
THE FIRST IS HOW SOON
INVESTIGATORS MIGHT KNOW THAT A
TRIAL IS UNLIKELY TO SUCCEED IF
IT STAYS ON ITS PRESENT COURSE,
WHAT ARE OPPORTUNITIES AND
CHALLENGES FOR MONITORING
SIGNALS IN REAL TIME ABOUT THE
INTERVENTION AND THE
IMPLEMENTATION OF THE
INTERVENTION.
SO, WE'LL HAVE ONE GROUP OF
DISCUSSANTS ADDRESSING THIS
QUESTION OF WHAT ARE THE
OPPORTUNITIES AND CHALLENGES FOR
GETTING EARLIER SIGNALS.
THEN THE SECOND POINT OF
DISCUSSION IS WHAT ARE THE
CONSIDERATIONS FOR DECIDING WHEN
AND HOW TO ACT ON THESE SIGNALS
THAT WE MIGHT BE GETTING, AND
WE'RE GOING HEAR FROM DIFFERENT
PERSPECTIVES, THOSE INVOLVED
WITH DSMBs, SPONSOR
PERSPECTIVE, BOTH ETHICS,
BIOSTATISTICS.
WE SHOULD HAVE A GREAT
DISCUSSION TO SURFACE SOME OF
THESE CONSIDERATIONS WHEN TRYING
TO ACT ON EARLY SIGNALS THAT
MIGHT BE COMING IN.
SO, LET'S START IN WITH OUR
FIRST SECTION HERE.
AND WE HAVE THREE OF OUR
COLLABORATORY PRINCIPAL
INVESTIGATORS, WHO ARE GOING TO
BE SPEAKING AND SHARING A LITTLE
BIT OF THEIR EXPERIENCE.
AND SO I'D LIKE TO START OUT
WITH DR. GLORIA CORONADO,
TALKING ABOUT THE STOP CRC AND
RELATED EXPERIENCES.
GOOD MORNING, GLORIA.
>> GOOD MORNING.
GOOD MORNING.
>> THANK YOU FOR GETTING UP
EARLY FOR US.
>> YOU'RE VERY WELCOME.
YEAH, THANK YOU FOR KICKING OFF
THIS VERY INTERESTING PANEL.
SO I'M GOING TO START JUST BY
GIVING YOU A TWO SENTENCE
BACKGROUND ABOUT STOP CRC AND
THEN SOME OF THE WAYS I FEEL
LIKE WE GOT SOME EARLY SIGNALS
OR INFORMATION ABOUT THE STUDY
THAT I THINK WERE REALLY
CRITICAL AND HELPED US MAKE
DECISIONS ABOUT GETTING
INFORMATION TO DECISION MAKERS
IN ORDER TO REDUCE SOME BARRIERS
THAT WE WERE ENCOUNTERING.
STOP CRC WAS A PRAGMATIC STUDY,
ONE OF THE FIRST -- FUNDED IN
THE FIRST ROUND OF THE NIH
COLLABORATORY AND FOCUSED ON
TESTING INTERVENTION TO REDUCE
RATES OF COLORECTAL SCREENING IN
FEDERALLY QUALIFIED HEALTH
CENTERS, RECRUITED 26 QUALIFIED
CLINICS IN OREGON OR NORTHERN
CALIFORNIA, AND WE DID A CLUSTER
RANDOMIZED DESIGN TO TEST MALE
FIT INTERVENTION DELIVERED
PRAGMATICALLY BY CLINIC STAFF.
WE DEVELOPED ELECTRONIC HEALTH
RECORD TOOLS TO IDENTIFY
PATIENTS DUE FOR SCREENING AND
DELIVERING EACH STEP OF THIS
MULTI-STEP INTERVENTION TO MAIL
FIT KITS TO PATIENTS DUE FOR
CAREENING.
SOUNDS LIKE A STRAIGHT FORWARD
INTERVENTION.
WE ASSEMBLED -- ONE OF THE
CRITICAL THINGS AT THE BEGINNING
WAS ASSEMBLE AN ADVISORY BOARD
THAT CONSISTED OF MOST OF THE
MEDICAL DIRECTORS FROM OUR
FEDERALLY QUALIFIED HEALTH
CENTERS, AS WELL AS OTHER
STAKEHOLDERS AND REPRESENTATIVES
FROM COMMUNITY ORGANIZATIONS.
I REMEMBER KIND OF OUR FIRST
IN-PERSON MEETING.
I THINK WE'VE SCHEDULED LIKE
FIVE HOURS DURING A DAY.
WE BROUGHT EVERYBODY IN TO OUR
HEADQUARTERED LOCATION IN
PORTLAND, OREGON.
WE SHARED ABOUT THE STUDY AND WE
KIND OF FACILITATED A VARIETY OF
CONVERSATIONS, AT THAT TIME THIS
WAS LIKE 2012 OR 2013.
AND AT THAT TIME, THE AFFORDABLE
CARE ACT HAD JUST RECENTLY
PASSED.
AS PART OF THE AFFORDABLE CARE
ACT THERE WAS A PREVENTIVE
HEALTH MANDATE THAT NOW PAID FOR
SCREENINGS, ALL KINDS OF CANCER
SCREENINGS THAT WERE RECOMMENDED
BY U.S. PREVENTIVE SERVICES TASK
FORCE WITH NO COST, NO CO-PAYS
TO PATIENTS.
SO ONE OF THE THINGS THIS BOARD,
THE MEDICAL DIRECTORS OR
CONCERNS THE MEDICAL DIRECTORS
RAISED IS THAT BY REFERRING
PATIENTS TO FIT TESTING OR
OFFERING PATIENTS FIT TESTING IF
THAT PATIENT HAD ABNORMAL FIT
TEST RESULT THEY WOULD FACE
CO-PAYS FOR THE FOLLOW-UP COLON
ONOSCOPY.
IF THE PATIENT WERE REFERRED
DIRECTLY FOR COLONOSCOPY THERE
WOULD BE NO CO-PAYS ACCORDING TO
PREVENTIVE HEALTH MANDATE.
ONE OF THE THINGS THIS BOARD DID
WAS, YOU KNOW, WORK WITH OUR
LEGISLATIVE LEAD THAT WAS ON OUR
BOARD, SO MITCH GREENLAKE WAS
FOUND EVERY MY RESEARCH CENTER.
I HAVE A POSITION THAT'S NAMED
AFTER HIM.
HE WAS ON OUR BOARD.
AND SO IT WAS PRETTY EASY TO
WORK WITH HIM TO SAY, GOSH, THIS
IS AN IMPORTANT CHALLENGE THAT
ALL OF OUR CLINICS ARE FACING IN
TERMS OF IT'S REALLY IMPACTING
HOW THIS PROGRAM WILL BE
IMPLEMENTED AND HOW THE
PATIENT -- WHETHER OR NOT THE
PATIENTS WILL EXPERIENCE OR
PHASE, YOU KNOW, SUBSTANTIAL
CO-PAYS TO GETTING THE
RECOMMENDED SERVICE.
AND SO WE WERE ABLE TO WORK WITH
MITCH TO EDUCATE LEGISLATORS WHO
EVENTUALLY CHANGED THE POLICY IN
OUR STATE, AND FOR THOSE THAT
TRACK THIS POLICY YOU'LL KNOW
THAT IT'S NOW A LOT OF THE LAND.
SO NATIONAL POLICY HAS BEEN
CHANGED IN ACCORDANCE WITH THE
POLICY THAT WAS CHANGED HERE IN
OREGON YEARS AND YEARS AGO.
SO, I THINK THAT'S AN EXAMPLE OF
SOMETHING THAT WE LEARNED THAT
WAS ESSENTIALLY A POLICY LEVEL
BARRIER THAT WE WERE ABLE TO
CHANGE BECAUSE WE HAD CONVENED
THIS GROUP OF STAKEHOLDERS WHO
WERE FACING THIS ISSUE IN THEIR
CLINIC DAY IN AND DAY OUT.
AND SO THAT HAPPENED TO BE AN
EARLY LEARNING WE WERE ABLE TO
TAKE ACTION ON TO IMPROVE
REDUCED BARRIERS TO IMPLEMENTING
OUR PROGRAM OVER THE COURSE OF
THE STUDY.
THE OTHER EXAMPLE THAT I WANT TO
GIVE, AND I THINK I CREDIT RUSS
GOSGO, A PROGRAM CENTER, AN
APPROACH.
THIS IS A VERY COMMON AND WELL
KNOWN STRATEGY TO -- OF
ITERATIVE SMALL TEST OF CHANGE,
CLINICS ARE TYPICALLY TRAINED
AND FAMILIAR WITH, THAT ALLOWS
THEM TO REALLY IMPROVE A
PROCESS.
AND SO WE DECIDED TO TAKE RUSS'
ADVICE, AND USE PLANNED USE
STUDIES AS OUR STRATEGY FOR
SUPPORTING AND FACILITATING THE
IMPLEMENTATION OF THE PROGRAM
WITHIN OUR CLINICS.
AND SO WE WORKED WITH OUR
PRIMARY CARE ASSOCIATION TO KIND
OF DO A BOOSTER TRAINING AT EACH
ONE OF OUR KIND OF LEADERSHIP
TEAMS OF EACH HEALTH CENTER AND
CREATED A FORUM THAT WOULD HAVE
THEM GIVE US THEIR IDEA FOR A
SINGLE CYCLE AND WE WERE GOING
TO ASK THEM TO REPORT BACK ON
THE FINDINGS FROM THAT CYCLE.
AND SO GOING INTO THIS PROCESS,
WE ANTICIPATED THAT OUR CLINICS
WOULD TEST OUT DIFFERENT
REMINDER APPROACHES, LIKE MAKE
ADDITIONAL PHONE CALL TO
PATIENTS OR REALLY KIND OF FOCUS
IN HOW TO INCREASE RETURN RATES
ON THE FIT KITS BEING MAILED OUT
TO PATIENTS FOR COLORECTAL
CANCER SCREENING.
TO OUR SURPRISE, WE FOUND ALL
KINDS OF ADDITIONAL
IMPLEMENTATION BARRIERS THAT WE
PROBABLY WOULDN'T HAVE FOUND OUT
ABOUT OTHERWISE.
AND SO ONE OF OUR CLINICS SAID,
WELL, WE WANT TO FOCUS OUR PDSA
ON THE ISSUES WE'RE HAVING WITH
OUR LAB BECAUSE WE'RE GETTING
COMPLETED FIT TESTS RETURNED BUT
THROWING THEM OUT BECAUSE THEY
DON'T HAVE A COLLECTION DATE.
OUR REFERENCE LAB WILL NOT
PROCESS WITHOUT A COLLECTION
DATE.
THIS LED TO THINKING THROUGH THE
PROCESS, AND CHANGING THE
MATERIALS THEY WERE SENDING TO
PATIENTS TO EMPHASIZE THE
IMPORTANCE OF ADDING THAT
COLLECTION DATE.
SO THIS IS A VERY PRACTICAL
EXAMPLE OF SOMETHING THAT MIGHT
BE INTERFERING WITH THE RESULTS
OF YOUR STUDY.
SO IN OUR CASE, THOSE KITS THAT
WERE BEING THROWN OUT WERE NOT
BEING CREDITED AS A SUCCESS OF
THE STUDY AND SO IT WAS REALLY
CLEAR PROBLEM THAT ALL OF US
WANTED TO FIX.
AND, AGAIN, IT'S SOMETHING THAT
WE PROBABLY WOULD NEVER HAD
LEARNED ABOUT HAD WE NOT USED
THE PDSA CYCLES.
SO MY EXAMPLES ARE TANGIBLE TO
HOW YOU MIGHT LEARN FASTER,
ABOUT CON CAREENING MEDICAL
DIRECTORS AND PEOPLE --
CONVENING MEDICAL VECTORS AND
POLICY LEVERS THAT MIGHT NEED TO
BE CHANGED, ALSO QUALIFY
IMPROVEMENT PROCESS PDSAs THAT
WAS HELPFUL.
SO I'LL STOP THERE.
>> THANK YOU SO MUCH, GLORIA.
THAT WAS TERRIFIC.
OUR NEXT COLLABORATORY P.I., DR.
NATALIA MORONE, EXPERIENCES IN
THE OPTIMUM TRIAL AND OTHER
RELATED EXPERIENCES.
GOOD MORNING.
>> GOOD MORNING, KEVIN AND
EVERYONE.
I'M GLAD -- THANK YOU FOR
INVITING ME TO SPEAK TODAY.
A LITTLE BIT OF QUICK
BACKGROUND, THE OPTIMUM TRIAL IS
A TELEHEALTH-DELIVERED GROUP
MINDFULNESS PROGRAM FOR PERSONS
WHO HAVE CHRONIC LOW BACK PAIN.
AND IT'S BEING OFFERED CURRENTLY
SO WE'RE STILL DOING THE
CLINICAL TRIAL.
WE'RE TOWARDS OF END OF
RECRUITMENT.
WE'RE IN OUR YEAR FOUR OF OUR
UH3 PRAGMATIC -- AWARD, A
PRAGMATIC TRIAL.
I REALLY ENJOYED LISTENING TO
WHAT GLORIA HAD TO SAY, AND I
THINK WE MAY BE STARTING A
SCENE.
WHEN WE STARTED THE TRIAL, WE
HAD ALREADY AHEAD OF TIME
PLANNED TO DO -- HAVE A
COMMUNITY ADVISORY BOARD AND
PLANNED TO INTERVIEW DIFFERENT
STAKEHOLDERS AND PARTICULARLY
WHAT ENDED UP BEING PRESCIENT IN
RETROSPECT IS WE DID A LITTLE
PILOT STUDY OF THE INTERVENTION.
THIS IS ALL PLANNED AHEAD OF
TIME, BEFORE WE ACTUALLY
DELIVERED THE LARGER
INTERVENTION.
AND THAT'S HOW THE UG3/UH3s
ARE SET UP.
IT TURNS OUT WE HAD TO QUICKLY
PIVOT IN 2020 TO A
TELEHEALTH-DELIVERED
INTERVENTION INSTEAD OF IN
PERSON BECAUSE OF THE PANDEMIC.
AND SO AT THE TIME, WE WERE NOT
PLANNING THAT, WHICH MEANT WE
HAD TO PIVOT IN THE WAY THAT THE
INTERVENTION WOULD BE DELIVERED
BECAUSE IT WAS NOT GOING TO BE
DELIVERED BY A GROUP AND ONLINE.
AND ALSO MAKING SURE THAT OUR
PARTICIPANTS WHICH WE HAVE
DIVERSE PARTICIPANTS, WE'RE AT A
SAFETY NET HOSPITAL HERE IN
BOSTON, WE HAD TO MAKE SURE OUR
PARTICIPANTS COULD ACCESS THE
PROGRAM THROUGH THEIR
TECHNOLOGY.
AND SO WHEN WE RAN THE PILOT, WE
DID IT ONLINE.
AND WE WERE ABLE TO ASK THE
PARTICIPANTS BECAUSE IT WAS
OFFLINE AHEAD OF TIME, WE WERE
PLANNING TO DO EXIT INTERVIEWS
BUT WE WERE ABLE TO PIVOT VERY
QUICKLY TO GET FEEDBACK ON THOSE
EXIT INTERVIEWS ABOUT THE
DELIVERY OF THE INTERVENTION,
AND HOW AS A PARTICIPANTS AND AS
A PATIENT HOW IT WAS FOR THEM.
SO WE WERE -- I WAS VERY FOCUSED
INITIALLY AS P.I. OF OPTIMUM
MAKING SURE WE WERE FAITHFUL TO
THE INTERVENTION WHICH NOW HAD
TO BE DELIVERED ONLINE.
THAT WAS VERY, VERY HELPFUL.
AND WHAT WAS SO HELPFUL ABOUT
DOING THAT VERY EARLY PILOT WAS
THE TYPE OF COMMENTS THAT PEOPLE
WERE SAYING ABOUT THE
INTERVENTION AND WAS IN LINE
WITH WHAT QUALITATIVE DATA WITH
MINDFULNESS INTERVENTION.
SO THEY WERE APPLYING THE
METHODS NOT ONLY TO THEIR PAIN
BUT ALSO TO THEIR DAILY LIVES.
SO THERE WAS REASSURANCE EARLY
THAT DELIVERING IN THIS MODE WAS
GOING TO BE OKAY.
AND SO THAT WAS VERY HELPFUL.
THE OTHER, AGAIN, PLAN AHEAD OF
TIME WHICH ENDED UP BEING
HELPFUL ANTICIPATING IT WAS VERY
EARLY COLLABORATION AMONG THE
MINDFULNESS INSTRUCTORS AS WELL
AS PRIMARY CARE PROVIDERS.
THIS IS AN INTERVENTION
DELIVERED IN PRIMARY CARE.
AND I HAVE ALWAYS BEEN
INTERESTED IN HAVING THE PRIMARY
CARE PROVIDER INVOLVED.
AND SO FROM THE BEGINNING THERE
WAS VERY CLOSE COMMUNICATION
BETWEEN THE TWO.
AND WE HAD TO PIVOT IN PERSON TO
ONLINE, THIS BECAME EVEN MORE
IMPORTANT.
THOSE ALREADY PLANNED
COMMUNICATIONS AMONG THE TEAM
ITSELF, TO BE AS, AGAIN, FOR
FIDELITY OF DELIVERY OF THE
INTERVENTION.
THAT ENDED UP BEING VERY, VERY
HELPFUL.
THIS IS SIMILAR TO WHAT GLORIA
WAS SAYING, PLANNED AHEAD TO
HAVE STAKEHOLDER INVOLVEMENT.
IN OUR CASE WE HAD IMMEDIATELY
INVOLVEMENT FROM OUR
PARTICIPANTS EVEN IN THE PILOT
THAT REALLY GAVE ME A LOT OF
REASSURANCE THAT THE PIVOT THAT
WE WERE MAKING WAS STILL GOING
TO DELIVER THE INTERVENTION THE
WAY I HAD INTENDED.
THE OTHER -- LOOKING BACK TO
WHAT KEVIN HAD MENTIONED, YES,
SO ADDRESSING THE ISSUES AS THEY
AROSE WAS, AGAIN, VERY HELPFUL.
THESE, AS WE STARTED THE MAIN
TRIAL, HAVING THE FEEDBACK FROM
PARTICIPANTS AND FROM THE VERY
BEGINNING ENDED UP BEING VERRY
HELPFUL PUTTING SIGNIFICANT
AMOUNT OF TIME TEACHING THEM HOW
TO USE TECHNOLOGY.
SIMPLE THINGS LOCALLY THAT REAL
MADE A DIFFERENCE IN BOSTON, A
LOT OF OUR PARTICIPANTS USE
PHONES AND THEY WERE VERY HAPPY
USING THE INTERVENTION ON THEIR
PHONE.
WHEREAS IN THE PITTSBURGH THEY
WERE MORE LIKELY TO USE LIKE A
LAPTOP, FOR EXAMPLE.
THESE SIMPLE THINGS GAVE ME A
LOT OF REASSURANCE THAT PIVOTING
AND HOW THE INTERVENTION WAS
BEING DELIVERED, WHICH WAS A
MAJOR CHANGE GOING FROM
IN-PERSON TO BASICALLY PEOPLE
BEING IN THEIR HOMES, WATCHING A
SCREEN, WAS STILL GIVING WHAT WE
WERE INTENDING.
THESE WERE WAYS WE WERE ABLE TO
HARNESS WHAT WE ALREADY BUILT
INTO THE TRIAL TO ACTUALLY DO A
VERY QUICK PIVOT THAT WAS AT THE
TIME UNEXPECTED BUT I THINK THIS
IS WHERE HAVING THINKING-AHEAD
TIME AS COMMUNITY ADVISORY
BOARDS AND STAKEHOLDERS AS WELL
AS PARTICIPANTS.
THE OTHER, AGAIN, THAT WE BUILT
IN AHEAD OF TIME WAS THE TEAM.
SO TEAM MEETINGS ACROSS ALL
THREE SITES HAVE ALWAYS MET
FREQUENTLY THROUGHOUT THE TRIAL,
AND I MAKE SURE TO INVITE PEOPLE
ALL THE DIFFERENT PARTS OF THE
TEAM TO COMMUNICATE.
AND SO, FOR EXAMPLE, THE TEAM
THAT WAS DOING THE TECH
ORIENTATIONS FOR THE CLINICAL
TRIAL, THEY WOULD BE GIVING US
VERY USEFUL FEEDBACK, REALLY
REALTIME FEEDBACK, WHAT
PARTICIPANTS NEEDED TO REALLY
LEARN HOW TO USE TECHNOLOGY.
THIS REALTIME FEEDBACK WAS VERY,
VERY IMPORTANT.
SO, MORE OF THE POLICY LEVEL
LIKE GLORIA WAS TALKING ABOUT
BUT REALLY BEING ABLE TO
ACTUALLY MAKE SURE YEAR
DELIVERING INTERVENTIONS YOU
INTENDED AND THAT IS SIGNALS OF
EARLY RESPONSE THAT IT ACTUALLY
IS WORKING.
SO THOSE ARE SOME BRIEF COMMENTS
HOW WE WERE ABLE TO TAKE
ADVANTAGE OF THE -- OF BOTH
PARTICIPANTS, STAKEHOLDERS, TEAM
ENGAGEMENT DURING A PERIOD OF
TIME WHEN WE HAD TO REALLY DO A
180 HOW OUR INTERVENTION WAS
GOING TO BE DELIVERED.
>> THANK YOU SO MUCH, NATALIA.
LET'S GO TO OUR THIRD SPEAKER,
DR. CORITA GRUDZEN TO HEAR ABOUT
THE STUDY AND EXPERIENCES.
>> THUMBS UP IF YOU CAN HEAR
ME.
OKAY, THANK YOU.
SO I'M GOING TO PIVOT A LITTLE
BIT, IT WAS REALLY INTERESTING
TO HEAR FROM BOTH OF YOU.
I GUESS PRIMARY PALLIATIVE CARE
FOR EMERGENCY MEDICINE WAS QUITE
A COMPLEX INTERVENTION, AND SO I
THINK LOOKING FOR A SIGNAL EARLY
IS A VERY COMPLICATED QUESTION
AND I WOULD ARGUE ALMOST
DANGEROUS.
I'LL EXPLAIN WHY.
YOU KNOW, OUR INTERVENTION
INVOLVED MULTIPLE COMPONENTS,
INVOLVED MULTIPLE TARGETS, IT
WAS EMERGENCY PHYSICIANS,
TEACHING EMERGENCY PHYSICIANS,
PRIMARY PALLIATIVE CARE SKILLS,
THERE WERE TECHNOLOGICAL
ENHANCEMENTS TO REMIND THEM.
SO WE TARGETED PHYSICIANS,
APPs, NURSES, SOCIAL WORK.
AND AT MULTIPLE LEVELS SO THE
LEVEL OF THE E.D., LEVEL OF THE
HEALTH SYSTEM.
AND SO BECAUSE OF THIS, I WOULD
SAY LOOKING FOR AN EARLY SIGNAL
IS A LITTLE MORE COMPLEX.
SO I WOULD ARGUE THAT IN OUR
CASE OF COURSE WE -- IT WAS A
UH3 WE PILOTED AT ONE INTERNAL
SITE, ONE EXTERNAL SITE, AND
FOCUSED MAINLY ON IMPLEMENTATION
OUTCOMES, NOT EFFECTIVENESS,
ALTHOUGH THAT COULD BE AN
IMPLEMENTATION OUTCOME BUT WE
FOCUSED ON REALLY MAKING SURE
THAT THE INTERVENTION WAS
FEASIBLE, THAT THERE WAS REACH,
ADOPTION, YOU KNOW, THAT THE
CLINICAL DECISION SUPPORTS TO
IDENTIFY PATIENTS IN NEED OF
PALLIATIVE CARE AND MAYBE PRIOR
ADVANCE DIRECTIVES WERE UP AND
RUNNING BUT WE DIDN'T LOOK AT
SIGNALS FOR PRIMARY OUTCOME
MEASURES ON UTILIZATION AND
SECONDARY MEASURES.
THE REASON IS BECAUSE COMPLEX
INTERVENTIONS REQUIRE SO MUCH
ADAPTATION TO LOCAL CONTEXT THAT
I THINK WHILE OUR IMPLEMENTATION
OUTCOMES WERE GREAT, IF WE HAD
SIMPLY SAID, OKAY, THIS IS --
UPTAKE IS GREAT, WE HAVE REACH,
ADOPTION, I'M NOT SURE THAT
WOULD HAVE TRANSLATED.
NOT ONLY THAT BUT I THINK, YOU
KNOW, IF YOU LOOK AT THE WORK OF
BRIAN MITTMAN, HE WAS PART OF
THE PCORI STANDARDS FOR COMPLEX
INTERVENTIONS, HE ARGUES I THINK
VERY -- IN MY CASE I THINK VERY
WELL THAT THE MORE COMPLEX AN
INTERVENTION, THE LESS LIKELY IT
IS TO WORK ACROSS THE BOARD.
AND THE QUESTION IS NOT WILL IT
WORK ON AVERAGE ACROSS ALL OF
THE HEALTH SYSTEMS BUT IN WHAT
LOCAL CONTEXT DOES IT WORK AND
WHY.
AND I THINK WE MAKE A MISTAKE TO
THINK AND TO SORT OF HAVE AN
APPROACH THAT ASSUMES IT'S A YES
OR NO ANSWER AND REALLY IT'S ALL
ABOUT CONTEXT.
AND SO WE WERE AT THE STAGE THAT
WE HAVE ALL OF OUR FINAL DATA,
TO LOOK AT OUR OUTCOMES.
WE'VE, YOU KNOW, AD NAUSEAM
PUBLISHED OUTCOME OUTCOMES,
QUANTITATIVE AND QUALITATIVE
WAYS BUT EFFECT WHICH EFFECT
HEALTH CARE UTILIZATION WILL BE
MUCH MORE CHALLENGING TO
UNDERSTAND AND -- NOT TO
UNDERSTAND BUT I THINK WILL
BE -- THERE WILL BE A LOT MORE
HETEROGENEITY IN TERMS OF HOW
THEY IMPACT PATIENT OUTCOMES.
AND PROBABLY A LOT OF THAT WILL
RELATE TO LOCAL LEADERSHIP, YOU
KNOW, A HOST OF FACTORS.
PROBABLY MOST IMPORTANTLY THE
PANDEMIC, BECAUSE WE HAD TO STOP
RECRUITMENT OR STOP OUR -- IT'S
A STEPPED WEDGE, HAD TO PAUSE
FOR SIX MONTHS.
SO, I I DON'T HAVE MUCH ELSE TO
SAY SO I'LL LEAVE MORE TIME FOR
DISCUSSION.
THAT'S MY MAIN POINT.
>> THANK YOU SO MUCH.
I WANT TO FIRST BEGIN BY ASKING
THE THREE OF YOU, ASK THERE
ANYTHING THE OTHERS SAID THAT
MADE YOU THINK OF SOMETHING ELSE
YOU WANTED TO ADD OR COMMENT ON
ABOUT THE OTHER TRIALS?
WE HAD SOME COMMON THEMES, BUT
ANYTHING -- ANYONE WANT TO ADD
BEFORE WE GO FOR DISCUSSION?
>> I'LL JUST SAY I RELATE TO
THE CYCLE.
IT'S A DIFFERENT ISSUE.
I LOVED HEARING ABOUT THAT
BECAUSE I THINK THAT'S KEY TO
FIGURING OUT HOW TO EVEN --
BEFORE YOU EVALUATE THE
INTERVENTION YOU HAVE TO FIGURE
OUT WHAT ALL OF THE PAINFUL
POINTS ARE THAT NEED TO BE
FIXED.
SO I RELATE.
>> THAT'S A GOOD PLACE TO START
A DISCUSSION.
YESTERDAY HAYDEN BOSWORTH IN HIS
SUMMARY HAD A REALLY INTERESTING
QUESTION WHICH WAS WHEN IS IT
TIME TO STOP ADAPTING AND JUST
STOP?
AND SO IN THESE TRIALS WHERE
THERE'S THE INTENTION TO
EXPLICITLY MONITOR IN REAL TIME
AND MAKE LOCAL ADJUSTMENTS, HOW
DO YOU THINK ABOUT WHEN FURTHER
ADAPTATIONS ARE NOT GOING TO BE
USEFUL OR DO YOU?
HOW DO WE FIGURE THAT OUT?
>> THIS IS GLORIA.
SUCH A GOOD QUESTION.
I ACTUALLY DON'T KNOW IF IT'S
GOING TO BE THE SAME ANSWER FOR
EVERY PROJECT OR EVERY SCENARIO
BECAUSE I THINK IT'S GOING TO
BE -- IT'S GOING TO REALLY
DEPEND ON CONTEXT BUT IT'S
SOMETHING WE'VE THOUGHT A LOT
ABOUT, BECAUSE THERE'S SOME
CLINICS THAT WILL TAKE ON AN
INTERVENTION WITH GUSTO, AND
THEN THEY WILL LOOK AT THE
FINDINGS AND SAY, WELL, THIS
ISN'T WHAT I EXPECTED.
I EXPECTED TO SEE A, YOU KNOW, A
60% INCREASE IN THIS OUTCOME AND
I GOT A 10 OR 12%.
AND THEN THEY WILL ABANDON THE
EFFORT.
THERE ARE OTHER CLINICS THAT
MIGHT TAKE A MORE KIND OF
MEASURED APPROACH AND REALLY
THINK ABOUT HOW THEY CAN DESIGN
THE STUDY SO THEY CAN CLAIM SOME
EARLY SUCCESS, AND BUILD ON THAT
MOMENTUM AND EXPAND OUT FURTHER.
AND SO I DO SEE, YOU KNOW,
CLINICS TAKING VERY DIFFERENT
APPROACHES AND I CERTAINLY THINK
THAT THE MORE MEASURED APPROACH
TENDS TO BE ONE THAT PERHAPS IS
PLANNING FOR LONGER TERM
IMPLEMENTATION.
I THINK I MEAN WE HEAR A LOT
ABOUT PLANNING FOR
SUSTAINABILITY AT THE VERY
BEGINNING, WHICH I THINK DOES
MAKE ME THINK HOW DO DO YOU
SOMETHING SMALL AND THEN EXPAND
IT OUT VERSUS GOING ALL IN AND
POTENTIALLY ABANDONING THE
EFFORT LATER.
>> THANKS.
>> THIS IS AN INTERESTING ONE.
PRAGMATIC TRIALS WE'RE WORKING
WITH EVIDENCE-BASED MEDICINE SO
TRYING TO PUT FORWARD EVIDENCE
BASE.
SO, THAT'S WHY IT'S A TRICKY
QUESTION BECAUSE IT'S LIKE
SAYING, YOU KNOW, WE WANT TO PUT
FORWARD SOME TYPE OF WALL TO
MAKING THAT EVIDENCE GO FORWARD
AND REALLY GETTING IT
IMPLEMENTED.
SO I THINK THAT'S WHY -- WHAT
I'VE HEARD IS HOW ALL OF US
WORKED HARD AT SHIFTING SO THAT
WE DON'T HAVE TO GIVE UP THE
GHOST, SO TO SPEAK.
SO I THINK THAT OUR SIGNALS
THOUGH THAT POTENTIALLY IT NEEDS
TO CHANGE, AND THIS IS WHY I
LOVE GLORIA SAYING THE PDSA
CYCLE, REMINDS ME OF RESIDENCY
WHERE THAT'S A REQUIREMENT, I'M
AN INTERNIST, THAT WE DO THAT.
AND IT'S VERY USEFUL IN THE
CLINIC AND IT'S ACTUALLY A
MINDSET THAT PRIMARY CARE
PROVIDERS WILL UNDERSTAND, IF
YOU APPROACH THE CLINIC WITH
THAT.
SO, I THINK PART OF IT IS SOME
OF THE ADAPTATIONS MAY BE MORE
THAN WE HAVE EXPECTED.
EVIDENCE BASE MIGHT HAVE THE
IMPLEMENTATION ONE WAY AND WE'RE
GOING TO POTENTIALLY BE CHANGING
IT A LOT MORE THAN EXPECTED.
I THINK THAT'S ONLY FAIR GIVEN
THE HETEROGENEITY OF EACH OF THE
CLINICS THAT WE'VE TALKED ABOUT.
SO, SOMETIMES I WONDER HOW MUCH
IS ACTUALLY STOPPING, THIS IS
REALLY MAKING A CHANGE, THAT
WASN'T THE ORIGINAL RANDOMIZED
CONTROLLED TRIAL.
>> YEAH, THAT TERM
HETEROGENEITY WAS ONE CORITA
USED A FEW MINUTES AGO.
FOR FOLKS NOT USED TO THINKING
ABOUT PRAGMATIC TRIALS
ESPECIALLY INVOLVING COMPLEX
INTERVENTIONS IT MAY STRIKE
PEOPLE AS WEIRD THE ABILITY TO
ADAPT LOCALLY IN WAYS NOW THAT
IT'S DIFFICULT TO SAY WHAT THE
INTERVENTION IS, APART FROM AN
EXPLICIT PROCESS PEOPLE SHOULD
GO THROUGH TO FIGURE OUT WHAT'S
GOING TO HAPPEN LOCALLY.
I THOUGHT IT WAS INTERESTING
CORITA, YOU WERE SAYING THAT
WHEN IT COMES TO LOOKING AT THE
BIG PRIMARY AND SECONDARY
EFFICACY ENDPOINTS, IT MIGHT BE
CHALLENGING NOW BECAUSE YOU'RE
ASSUMING THERE'S GOING TO BE
HETEROGENEITY THERE.
AND I GUESS A PROVOCATIVE
QUESTION MIGHT BE IS THIS
HETEROGENEITY OF TREATMENT
EFFECTIVE A WELL-DEFINED
TREATMENT OR TO WHAT EXTENT ARE
WE ACTUALLY DOCUMENTING EFFECTS
OF DIFFERENT TREATMENTS.
>> I THINK IT'S THE LATTER,
UNFORTUNATELY.
I WISH IT WAS SIMPLER AND WE
WERE JUST -- THERE WAS JUST
HETEROGENEITY OF TREATMENT
EFFECT, YOU KNOW, BUT I THINK
IT'S UNFORTUNATELY MORE
COMPLICATED THAN THAT.
THE BEER VENGES IS BY NATURE
GOING TO BE A LITTLE BIT
DIFFERENT, OR IT'S NOT -- THE
INTERVENTION IS GOING TO BE
DIFFERENT OR IF YOU HAVE COMPLEX
INTERVENTION, YOU DON'T ADAPT
IT, YOU'RE TOAST.
>> YEAH, I THINK THAT'S -- IT
REALLY MAKES IT FUZZY TALKING
ABOUT EARLY SIGNALS THE
INTERVENTION MIGHT NOT WORK,
INTERPRETED DIFFERENTLY, IN A
SITUATION LIKE THIS WHERE IT'S
PART OF THE INTERVENTION TO DO
THESE ADAPTATIONS AND SO THAT
RAISES THAT QUESTION ABOUT AT
WHAT POINT DO YOU SAY WE CAN'T
ITERATE FURTHER, ITERATING
FURTHER SEEMS TO BE FRUITLESS,
SEEMS LIKE THAT'S AN INTERESTING
BUT CHALLENGING DECISION POINT
TO CONSIDER.
>> YEAH, I MEAN, I WAS
STRUGGLING, I STRUGGLE WITH
THAT, YOU KNOW, DO YOU KEEP
ADAPTING?
BUT I THINK IN MY MIND THE MOST
IMPORTANT THING IS JUST FOCUSING
ON IMPLEMENTATION AND JUST
KEEPING THOSE IMPLEMENTATION
OUTCOMES IN MIND BECAUSE IF YOU
PASS OVER THAT, THEN WHAT IN THE
WORLD ARE YOU DOING, WHAT ARE
YOU TESTING, BECAUSE YOU'RE NOT,
YOU KNOW, AND AS LONG AS YOU ARE
RELIGIOUS ABOUT THE FUNCTIONS OF
THE INTERVENTION, THAT'S WHERE
THE FORM FUNCTION COMES IN, I
THINK PRE-SPECIFYING WHAT THE
FUNCTIONS ARE, OURS WERE
IDENTIFYING PATIENTS WITH
SERIOUS ILLNESS WITH PALLIATIVE
CARE NEEDS, ADVANCE LIFE
DIRECTORS, ACTING ON THOSE PER
THE PATIENT'S WISHES, THOSE WERE
THE FUNCTIONS.
THE FORMS THEY MIGHT TAKE,
WHETHER IT'S A BANNER OR
INTERRUPTIVE ALERT, LIKE SOME
HEALTH SYSTEMS DID NOT USE
INTERRUPTIVE ALERTS.
THERE WAS A NON-STARTER.
THEY DIDN'T DO IT.
OTHER ONES, THEY LOVED IT.
WE CAN'T IMPOSE OUR WILL AND SAY
YOU HAVE TO HAVE AN INTERRUPTIVE
ALERT.
I JUST PUT IT IN THE BANNER,
WHATEVER THE LOCAL CONTEXT IS.
AS LONG AS YOU'RE FOCUSED ON THE
FORMS IS, YEAH, THE WAY TO MAKE
SURE YOU'RE I GUESS KEEPING WITH
THE PURPOSE OF THE WHOLE
INTERVENTION.
>> THIS IS GLORIA, TO ADD TO
THE CONVERSATION, IN OUR CASE
THERE WAS QUITE A BIT OF
EVIDENCE MAIL KIT OUTREACH WAS A
SUCCESSFUL INTERVENTION.
I FEEL AS IF PART OF WHAT I
NOTICED IS THAT AMONG THE
COLLABORATORY-FUNDED PROJECTS
THERE WAS A CONTINUUM, A RANGE
AT WHICH THEY WERE EMPHASIZING
EFFICACY VERSUS IMPLEMENTATION,
OUR PROJECT WAS PERHAPS MORE ON
THE IMPLEMENTATION END BECAUSE
WE DIDN'T NECESSARILY NEED TO
PROVE IT WOULD WORK AS A WHOLE.
WHAT WE NEEDED TO REALLY
DEMONSTRATE OR WE WANTED TO
DEMONSTRATE IS THAT THIS COULD
WORK IN MULTIPLE SETTINGS, IT
MIGHT NEED TO BE ADAPTED FOR
THOSE SETTINGS BUT IT COULD WORK
IN THESE SAFETY NET CLINICS,
RIGHT?
AND SO WHAT WE LEARNED THROUGH
THE PROCESS, AGAIN, IS IT
WASN'T -- THERE WERE A LOT MORE
QUESTIONS THAT WERE PUT ON THE
TABLE, LIKE DURING THE PDSA
APPROACH THERE WAS ONE CLINIC
THAT SAID, WELL, WE DON'T WANT
TO LOOK BACK TO IDENTIFY
PATIENTS WITH A CLINIC VISIT IN
THE LAST YEAR, WE WANT TO JUST
LOOK BACK AT THE LAST TWO MONTHS
AND MAIL OUT TO THOSE PATIENTS
WHO WEREN'T GIVEN ONE DURING THE
CLINIC VISIT.
IT'S LIKE A SWEEP OF THOSE
PATIENTS WHO WERE ESSENTIALLY
MISSED.
AND SO IT'S REALLY -- IT WAS AN
INTERESTING IDEA ABOUT HOW YOU
CAN ADAPT THE PROGRAM TO PERHAPS
GET EVEN HIGHER RETURN RATE
BECAUSE THESE PATIENTS HAVE HAD
A RECENT CONTACT WITH THE
CLINIC, TYPICALLY HAVE GOOD
CONTACT INFORMATION IN THE
ELECTRONIC HEALTH RECORD, JUST
HAD A VISIT, SO TO SEND A LETTER
THAT SAYS, OOPS, WE FORGOT TO
GIVE YOU A TEST, MAIL IT BACK,
WAS FEDERALLY QUALIFIED CLINICS
MAY CLAIM EARLY SUCCESS IF
BUILDING OVER TIME OR KEEPING IT
A SMALL SCALE IF THAT'S WHAT BEE
NEED TO DO OR ANY NEED TO BE
FILLED THERE'S TYPICALLY A WAY
TO ADAPT THE PROGRAM TO MAKE IT
WORK.
I THINK THAT'S SOME OF THE MOST
VALUABLE FINDINGS FROM OUR WORK
IS REALLY WHAT ARE THOSE PLACES
WHERE THINGS CAN GO OFF THE
RAILS OR THINGS THAT YOU NEED TO
PAY ATTENTION TO IN ORDER TO
SUCCESSFULLY IMPLEMENT THIS
PROGRAM.
>> I REALLY APPRECIATE BOTH
YOUR COMMENTS, AND I WANT TO
UNDERCORE ALSO WHAT CORITA SAID,
THERE'S A DIFFERENCE BETWEEN
WHAT'S AT THE CORE AT WHAT YOU
WANT TO DELIVER AND WHAT'S THE
ADAPTATION.
SO, YOU KNOW, THE ANALOGY, YOU
KNOW, IS IF YOU'RE TRYING TO GET
GLORIA, PEOPLE, TO GET THEIR
SCREENING DONE, HOW YOU REACH
THEM, WHETHER YOU DO IT BY MAIL,
BY PHONE, WHETHER YOU SEND A
TEXT, THAT'S THE ADAPTATION.
WHETHER YOU DO IT OVER THE PAST
TWO, THAT IS NOT GOING TO
COMPROMISE THE CORE OF WHAT
YOU'RE TRYING TO DO.
I JUST WANTED TO UNDERSCORE
BECAUSE THERE'S A DIFFERENCE
BETWEEN THE TWO.
AS INVESTIGATORS WE HAVE TO KEEP
THAT IN MIND AND NOT FORGET
THAT, WHAT IS IT THAT WE'RE
TRYING TO ACTUALLY DO VERSUS
WHAT ARE THE ADAPTATIONS, THAT'S
A REALLY NICE POINT.
>> I SEE A COMMENT FROM KEVIN
CHAN AT NIDDK WHO AGREES WITH
THE ISSUE OF FORM AND FUNCTION
AND KEVIN SAYS IT'S IMPORTANT
THAT YOU CAN DESCRIBE YOUR
INTERVENTION AS PRECISELY AS
POSSIBLE SO IT COULD BE
REPLICATED IN OTHER CENTERS
ACROSS THE U.S.
GREG SIMON HAD A QUESTION ABOUT
SIGNAL TO NOISE, SIGNAL VERSUS
NOISE.
GREG, DO YOU WANT TO POSE YOUR
QUESTION?
>> SURE.
IT'S SOMETHING I'VE BEEN
THINKING ABOUT BUT GLORIA'S
POINT ABOUT CO-PAYMENTS FOR
COLONOSCOPY IS A NICE EXAMPLE OF
WHEN YOU RUN INTO A BARRIER HOW
DO YOU DECIDE WHETHER THIS
BARRIER IS AN ARTIFICIAL THING,
WHICH SHOULD BE AT LEAST IN THE
CONTEXT OF THE TRIAL REMOVED
BECAUSE IT'S NOISE, VERSUS THIS
BARRIER IS SIGNAL, THIS IS A
TRUE REALITY OF THE PLACE TO
WHERE WE WOULD HOPE TO
GENERALIZE TRIAL RESULTS,
THEREFORE YOU MUST ADAPT OR
ACCOMMODATE TO IT.
TO ME IT'S A REALLY IMPORTANT
QUESTION WHEN YOU THINK IF
THERE'S A BARRIER DO YOU NEED TO
CHANGE THE ENVIRONMENT TO TAKE
AWAY THE BARRIER OR DO YOU NEED
TO CHANGE THE INTERVENTION TO
ACCOMMODATE THE BARRIER?
>> YEAH, SUCH A GOOD QUESTION,
GREG.
I HAVE TO SAY THE EXAMPLE I GAVE
AROUND CO-PAYMENTS FOR
COLORECTAL CANCER SCREENING AND
A DECISION THE PROVIDER MIGHT
MAKE GIVEN BOTH WOULD HAVE NO
CO-PAYMENTS BUT THE FOLLOW-UP
COLONOSCOPY WOULD IF THE TEST
WAS ABNORMAL, WE'RE TRAINED TO
ADAPT, RIGHT?
OH, WE NEED TO CHANGE THE
INTERVENTION, NEED TO CHANGE HOW
IT'S BEING DELIVERED, AND THIS
WAS ACTUALLY AN ISSUE THAT WE'RE
LIKE WHAT CAN WE DO?
LIKE WE COULDN'T THINK OF HOW WE
WOULD ADAPT THE PROGRAM TO DEAL
WITH THIS BARRIER.
SO IT REALLY FELT, TO US, THAT
THERE WAS ONLY A POLICY
SOLUTION.
AND I THINK THAT'S WHY WE
PURSUED THAT APPROACH BECAUSE WE
COULDN'T IMAGINE, COULDN'T DREAM
UP A WAY WE COULD STILL
ENCOURAGE THESE PROVIDERS WHO
FELT LIKE THEY WERE FACING AN
ETHICAL DILEMMA, RIGHT?
LIKE IF I GIVE THIS TO MY
PATIENT THEY MAY FACE CO-PAYS
AND THEY ARE NOT GOING TO GET
THE COLONOSCOPY AND WHAT DO I
DO?
IT WAS BOTH ETHICAL AND
PRACTICAL CHALLENGE THEY WERE
ENCOUNTERING AND WE COULDN'T
DREAM OF A WAY TO FIX IT OTHER
THAN POLICY.
>> YEAH, IT'S AN INTERESTING
QUESTION BECAUSE THE TRIAL
RESULTS ARE NOT FOR NOW, THEY
ARE FOR THE FUTURE, TO INFORM
WHAT PEOPLE CAN DO IN THE FUTURE
AND WE CAN'T KNOW WHAT THE
FUTURE WILL BE LIKE.
BUT OUR LAWS ARE EASIER TO
CHANGE THAN PEOPLE, YOU KNOW.
ONE OF MY RULES WOULD BE TO THE
EXTENT THE BARRIER INVOLVES
POLICIES AND LAWS THEY CAN
CHANGE TO THE EXTENT THE BARRIER
INVOLVES FUNDAMENTALS OF HUMAN
BEHAVIOR NOT GOING TO CHANGE.
>> NICE PERSPECTIVE.
>> I WANT TO SHIFT GEARS A
LITTLE BIT, NATALIA WAS
MENTIONING THE UG3/UH3 PHASES
THAT THE COLLABORATORY USES, AND
WE KNOW THAT OTHER MECHANISMS
AND FUNDERS ARE USING SOME
SIMILAR APPROACHES OF HAVING,
YOU KNOW, A PLANNING PHASE AND
THEN THE ACTUAL IMPLEMENTATION
OF THE TRIAL PHASE, AND WHEN WE
THINK ABOUT EARLY LEARNING AND
OPPORTUNITIES TO KIND OF SQUEEZE
MORE INFORMATION OUT OF OUR
EXPERIENCES SO THAT WE CAN
ANTICIPATE WHAT MIGHT BE COMING,
DO YOU THREE THINK THERE WERE
ANY OPPORTUNITIES TO DO
ADDITIONAL WORK IN THE PLANNING
PHASE, THAT FIRST YEAR, AND WHAT
IF IT HAD BEEN TWO YEARS, WHAT'S
THE OPPORTUNITY FOR GETTING EVEN
MORE LEARNING IN THAT PHASE
BEFORE THE TRIAL IS IMPLEMENTED?
SO WHAT ARE YOUR THOUGHTS ABOUT
THAT?
>> THIS IS GLORIA.
I'LL START.
ONE OF THE APPROACHES WE TOOK
FOR OUR PILOT, AND WE WERE BUSY
DURING THE PILOT YEAR, BUT TO
REALLY PILOT TEST OUR
INTERVENTION, IN OUR MOST
DIVERSE CLINIC.
THIS REALLY HELPED US DEVELOP
MATERIALS IN ENGLISH AND SPANISH
AND DO SOME FORMATIVE WORK SO WE
WOULD HAVE A SET OF MATERIALS
THAT WERE -- AND APPROACHES
GENERALIZABLE TO THE OTHER
SITES, EFFECTIVE APPROACH.
WE DEVELOPED TOOLS, GOT
FEEDBACK, WORKED WITH OUR I.T.
SPECIALISTS FROM EACH OF OUR
HEALTH CENTERS TO DEVELOP THE
TOOLS IN A WAY THAT MADE SENSE
TO THEM.
WE VALIDATED WHETHER WE WERE
ACTUALLY CAPTURING PATIENTS WHO
WERE DUE BECAUSE WE KIND OF WENT
INTO THIS THINKING, OH, THE
CLINICS, YOU KNOW, ARE DOING A
LOT OF PROMOTION, WE'RE GOING TO
KIND OF BOOST THEIR EFFORTS.
WHAT WE REALIZE IS THEY WEREN'T
DOING A LOT OF FIT PROMOTION, IN
SOME CASES CLINICS WERE STILL
USING A THREE SAMPLE FOBT AND
WANTED TO SWITCH BEFORE THE
TRIAL STARTED, A LOT OF STUFF WE
DID IN THE SINGLE YEAR.
I WOULD SAY IF ANYTHING ANOTHER
SIX MONTHS PROBABLY WOULD HAVE
BEEN VALUABLE TO TEST OUT A FEW
THINGS, BUT WE DID A HECK OF A
LOT IN THAT SHORT AMOUNT OF
TIME.
>> THANKS, GLORIA.
CORITA AND NATALIA WHAT ARE YOUR
THOUGHTS FOR OPPORTUNITIES FOR
EARLY LEARNING.
>> I AGREE WITH GLORIA, ANOTHER
SIX MONTHS WOULD HAVE BEEN NICE.
TWO YEARS FEELS A LITTLE LONG,
TO BE HONEST, BECAUSE OPTIMUM IS
THREE SITES IT WAS REALLY
IMPORTANT FOR ME TO PILOT AT ALL
THREE SITES.
THE MEDICAL CENTERS WERE VERY,
AGAIN, FOR HETEROGENEITY,
MEDICAL CENTERS WERE VERY
DIFFERENT.
WHAT WAS UNEXPECTED, WHAT WE
LEARNED, IS PEOPLE -- BECAUSE IT
WAS PILOTED AT ALL THE SITES,
SOME OF THE SITES EXPERIENCED
CERTAIN BEHAVIORS IN
PARTICIPANTS THAT THE OTHER
SITES DIDN'T.
FOR EXAMPLE, NORMALLY IN AN
IN-PERSON GROUP, MEDICAL GROUP,
PEOPLE FELT COMFORTABLE SMOKING
AND SO WE DECIDED WE WERE GOING
TO DO SOME RULES FOR
PARTICIPANTS THAT WOULD NOT HAVE
HAPPENED IF WE HADN'T DONE THE
PILOT AHEAD OF TIME.
THERE ARE OTHER THINGS WE
DISCUSSED THAT WE DECIDED IT WAS
ACCESSIBLE, WORKING WITH CHRONIC
PAIN POPULATION, IT WAS TOO HARD
FOR THEM TO STAY FOR THE
TWO-HOUR SESSIONS.
SO IT WAS INTERESTING.
I THINK AHEAD OF TIME I HAD
REALLY WANTED TO PILOT IT AT ALL
THREE SITES BUT I GOT
INFORMATION, THE WHOLE TEAM GOT
INFORMATION, ONE SITE WAS ABLE
TO SHARE WITH OTHER SITES THAT
ACTUALLY IMPACTED WHAT WE DID.
THERE'S SO MUCH TO DO IN THAT
FIRST YEAR, IT GETS COMPACTED
BECAUSE OF THE REQUIREMENTS,
WHEN THINGS ARE DUE, FOR
EXAMPLE.
BUT THIS IS WHERE MORE TIME
WOULD HAVE BEEN NICE.
BUT ON THE OTHER HAND IT WAS
ALSO NICE TO GET THAT YEAR OVER
WITH.
SO, YOU KNOW, AND MOVE FORWARD.
I DON'T MEAN IT THAT WAY BUT
JUST GET THAT PILOTING PART DONE
AND OVER WITH.
I WOULD SAY THE DECISION TO
PILOT IN ONE, ONE SITE, WAS
REALLY CRITICAL FOR US.
>> GREAT, THANKS.
>> YEAH, I WOULD SAY TWO SITES
ABSOLUTELY.
I THINK ONE YEAR IS REALLY
CHALLENGING WHERE YOU'RE WORKING
IN THE SAFETY YET HOSPITAL
BECAUSE THEY CAN'T EVEN GET A
CONTRACT THROUGH, SUBCONTRACT
THROUGH.
WE PILOTED A NYU, WHICH WAS
EASY, YOU KNOW, BECAUSE WE
DIDN'T HAVE THE MONEY TO PAY
INCENTIVES OR WHATEVER, WE
COULD, YOU KNOW, BORROW AND PAY
BACK LATER.
BUT OUR SECOND SITE WAS
UNIVERSITY HOSPITAL IN NEWARK,
WHERE THERE IS JUST -- I DON'T
THINK WE PAID THEM A DOLLAR TO
THIS DAY, IT WAS SO CHALLENGING
TO GET THINGS THROUGH THEIR
CONTRACTS OFFICE, THEY HAVE NO
SUPPORT FOR RESEARCH.
SO I THINK, YEAH, OUTSIDE OF TWO
YEARS WOULD HAVE BEEN EASIER.
MY VOTE IS FOR TWO PLUS FOUR BUT
FOR THE UG3/UH3, BUT ESPECIALLY
WHEN WORKING IN SAFETY NET
ORGANIZATIONS AND HEALTH SYSTEMS
IT'S VERY CHALLENGING TO GET
THINGS DONE IN REAL TIME.
>> SUCH A GREAT POINT, CORITA.
AND KNOWING I THINK THERE'S A
GENERAL PUSH TO DO RESEARCH IN A
BROADER RANGE OF SETTINGS,
INCLUDING UNDERRESOURCED
SETTINGS, SOUND LIKE IT WOULD BE
USEFUL TO CONSIDER THE LONGER
TIME IT WOULD TAKE TO EVEN DO
THAT PILOT LEVEL LEARNING TO GET
THOSE EARLY SIGNALS OF SUCCESS,
YEAH.
YOU KNOW, THERE WAS ANOTHER
THEME THAT I HEARD, THE
IMPORTANCE OF A
MULTI-STAKEHOLDER PANEL IN
HELPING TO FIGURE OUT WHAT
THINGS NEEDED TO BE LEARNED AND
REACT TO LEARNINGS THERE.
I WONDER IF YOU ALL HAVE ANY
TIPS FOR HOW BEST TO USE SUCH A
PANEL TO IDENTIFY THOSE EARLY
SIGNALS AND DIGEST THEM
INTERNALLY.
>> I GUESS I CAN START.
SO, OUR PANEL MEETS MONTHLY.
AND IT TAKES TIME, COMMUNITY
ADVISORY BOARD MEETS MONTHLY.
AND SO ONE IS I KNEW TO GET A
REALTIME FEEDBACK THAT WOULD BE
USEFUL FOR THE BOARD, WE WOULD
NEED TO MEET MORE FREQUENTLY.
NOT ALL COMMUNITY ADVISORY
BOARDS MIGHT MEET THAT
FREQUENTLY.
FOR US IT'S BEEN VERY, VERY
LEEPFUL.
THIS -- HELPFUL.
IT DOESN'T JUST HAPPEN AT
SPECIFIC TIMES.
AND SO WE COULD REAL TIME BRING
UP ISSUES TO OUR BOARD AND THEN
GET FEEDBACK.
BECAUSE WE ALREADY HAD A SYSTEM
IN PLACE WITH COMMUNICATION, I
MENTIONED BEFORE WITH THE TEAM,
WE COULD RAPIDLY IMPLEMENT IT.
SOME HAS TO DEAL WITH ADDING
CONFIDENTIALITY LANGUAGE, FOR
EXAMPLE, TO THE TELEPHONE
SCRIPTS THAT WE USE.
REALLY REAL TIME, THAT WAS ONE
OF THE WAYS WE HAVE DONE IT, BY,
ONE, FREQUENCY OF MEETINGS, TWO
HAVING AGENDAS WITH FLEXIBILITY
TO ADD AGENDAS, AND ABILITY TO
IMPLEMENT EFFECTIVELY WHAT'S
RECOMMENDED.
I'D BE INTERESTED IN THE PDSA
CYCLE YOU MENTIONED, GLORIA, HOW
LONG IT TOOK TO DO THOSE CYCLES
RELATED TO HOW TO DECIDE WHAT TO
DO AND THE IN THE CYCLES.
>> YEAH, SO WE, YOU KNOW, WENT
TO EACH OF OUR HEALTH CENTERS,
SO 26 CLINICS, I THINK ABOUT
EIGHT FEDERALLY QUALIFIED HEALTH
CENTERS, WE MET WITH LEADERSHIP
TEAMS, CARVED OUT A TWO-HOUR
MEETING TO REFRESH KNOWLEDGE OF
PDSAs, INTRODUCE FORMS AND
REQUIREMENTS.
THEN WE ARRIVE AT THAT MEETING,
ALLOWED TIME FOR BRAINSTORMING
BECAUSE THEY WERE MEETING AND
MIGHT THINK OF IDEAS RIGHT AWAY.
THEN I THINK IT WAS ABOUT TWO
MONTHS LATER THAT WE ASKED THEM
TO PRESENT THEIR FINDINGS AT A
BOARD MEETING, TWO OR THREE
MONTHS LATER THAT WE HEARD ABOUT
THEIR FINDINGS.
IT WAS SUCH A GREAT PROCESS FOR
OUR PROJECT FOR A COUPLE
REASONS.
ONE IS THAT WE, AGAIN, FOUND ALL
THESE REALLY INTERESTING
CHALLENGES THAT WE WOULDN'T HAVE
OTHERWISE THOUGHT ABOUT OR
LEARNED ABOUT WAY TOO LATE,
BECAUSE WE HAD ACTIVITIES
PLANNED AT THE END BUT THEY
WOULDN'T HAVE AN ACTIONABLE.
SO THAT WAS REALLY HELPFUL.
BUT THE OTHER THING THAT WE
ENCOUNTERED IS THAT I THINK
PERSPECTIVE OF OUR CLINICS WAS
LIKE, OKAY, YOU'RE WORKING WITH
THAT CLINIC, SO THAT ONE IS
YOURS TO WORK WITH, AND SO WE'LL
DO STUFF IN OUR USUAL CARE
CLINICS, SO REALLY INITIALLY
HANDS OFF WITH THE CLINICS THAT
WERE INTERVENTION CLINICS, WHICH
IS NOT HOW WE WANTED IT TO BE.
LIKE WE NEEDED TO ENGAGE THEM,
NEEDED TO WORK WITH THEIR TEAMS
AND HAVE THEM, YOU KNOW,
CHAMPIONING THE INTERVENTION.
SO THE OTHER THING THAT REALLY
CHANGED WAS THAT WILLINGNESS TO
DIG IN AND GET YOUR HANDS DIRTY
AND THINK ABOUT THE CHALLENGES
AND THINK ABOUT HOW YOU MIGHT
SOLVE THOSE CHALLENGES.
AND SO IT WAS REALLY HELPFUL
FROM THAT PERSPECTIVE AS WELL,
JUST REALLY HELPING THEM
UNDERSTAND THAT THAT'S AN OKAY
THING TO DO BECAUSE THIS IS A
PRAGMATIC STUDY AND WE'RE ASKING
THEM TO ACTUALLY DO IT, NOT US
TO COME IN AND DO THE STUDY WITH
THEIR PATIENTS.
>> THANKS, GLORIA.
OKAY.
WELL, THIS HAS BEEN REALLY
INTERESTING.
AND I WANT THE THREE OF YOU TO
HANG IN THROUGH THE NEXT FOUR
WHEN WE HAVE BROADER DISCUSSION,
PLEASE CAN'T TO SHARE YOUR
EXPERIENCES.
WE'LL MOVE NOW TO CONSIDER THE
DIFFERENT THINGS THAT DECISION
MAKERS MIGHT LOOK AT AS THEY ARE
MONITORING AND MAKING DECISIONS
ABOUT WHAT TO DO WITH THE STUDY
BASED ON EARLY SIGNALS THEY ARE
GETTING AND WE'LL HAVE FOUR
PERSPECTIVES AND WE'LL START
WITH DR. KEVIN CHAN, WHO IS AT
NIDDK, TO KICK US OFF.
KEVIN, TAKE IT AWAY.
>> THANK YOU, KEVIN, I'M FROM
NIDDKA SPONSOR OF TRIALS, WORK
WITH P.I.s AND DSMBs, AND
HAVE SLIDES TO TALK HOW WE
PROCESS INFORMATION ABOUT
TRIALS.
NEXT SLIDE.
EMBEDDED CLINICAL TRIAL,
EFFECTIVENESS, TESTED EMBEDDED
IN USUAL CLINICAL CARE.
I WANT TO START, I THINK OF
THESE OF QUALITY IMPROVEMENT
PROJECTS, INHERENTLY CHALLENGING
WHEN YOU START CHANGING HOW A
HEALTH ECOSYSTEM OPERATES, OR
YOU START CHANGING PEOPLE'S JOB
DESCRIPTIONS, IT WAS MENTIONED
IT'S EASIER TO CHANGE LAWS THAT
BE PEOPLE, IT'S A COMPLEX
PROCESS TO GET THIS DONE IN
HEALTH SYSTEMS.
TO DO THIS WE DO THESE LARGE
QUALITY IMPROVEMENT PROJECTS,
THE WHOLE RESOURCES THAT HELP
YOU BE SUCCESSFUL, SUCH AS
TRAINING, OPERATIONS, REALTIME
DATA REPORTING, MONITORING, ROOT
CAUSE ANALYSIS.
NOT ALL TRIALS MAY HAVE ACCESS
TO ALL THESE RESOURCES TO
IMPLEMENT COMPLEX AND
HETEROGENEOUS INTERVENTIONS WE
TALKED ABOUT.
AND IT GETS MORE CHALLENGING
WHEN THE GOALS OF THE RESEARCH
DON'T ALIGN WITH THE HEALTH CARE
SYSTEM AND YOU'RE WORKING
AGAINST WHERE THE HEALTH SYSTEM
IS GOING.
NEXT SLIDE.
SO, I WAS ASKED TO TALK ABOUT
HOW TO MAKE GO AND NO GO
DECISIONS ON TRIALS.
I BELIEVE IN SECOND CHANCES, I
WANT YOU TO KEEP THAT IN MIND.
BUT IN A TRIAL INDICATIONS FOR
STOPPING THE TRIAL EARLY,
BUCKETED THEM INTO THREE
CATEGORIES.
ONE IS FUTILITY, A COUPLE WAYS
TO LOOK AT FUTILITY.
ANOTHER CATEGORY IS
NON-RANDOMIZED BIAS, MAYBE ITS
OWN CATEGORY.
SAFETY OR HARM, OR OVERWHELMING
EFFECTIVENESS OR NEW STUDIES
HAVE COME OUT THAT MAY CHANGE
CARE.
BUT GO AND NO GO, IT'S NOT ONE
SIGNAL.
IT'S A TOTALITY OF DATA.
I'LL TALK MORE AS WE GO ALONG.
NEXT SLIDE.
SO, IF YOU WANT TO THINK ABOUT
CHANGING OR STOPPING A TRIAL
EARLY, HAVING DATA AND GOOD DATA
EARLY IS I THINK ONE OF THE
FIRST FOUNDATIONS.
THIS IS FROM A PAPER BY
ELLENBERG, I'M NOT GOING TO GO
OVER IT IN DETAIL.
YOU CAN LOOK AT THE ARTICLE.
IT OUTLINES HOW TO MONITOR
PRAGMATIC CLINICAL TRIAL.
IN A PRAGMATIC TRIAL, A COUPLE
THINGS, DATA QUALITY, USING REAL
WORLD DATA FROM EHRs, NOT
PERFECTLY ASKER -- ASCERTAINED.
WE CAN BUCKET INTO THREE
CATEGORIES OF HOW WOULD WE LOOK
AT THESE AND FOCUS ON LOOK AT
POWER, SEPARATION OR FIDELITY OF
THE INTERVENTION, NONRANDOM BIAS
IN AN UNBLINDED STUDY, I'LL TALK
MORE ABOUT THAT LATER.
THE LAST BUCKET IS SAFETY, IS
THIS CAUSING HARM TO PATIENTS.
NEXT SLIDES.
I WANT TO TALK ABOUT TWO THEMES
THAT SEEM TO SHOW UP MORE IN
PRAGMATIC TRIALS THAN TYPICAL
WELL-CONTROLLED TRIALS, FIDELITY
AND ADHERENCE.
WE'VE HAD LESS TIGHTLY
CONTROLLED BY RESEARCH TEAM THAT
PRAGMATIC TRIAL AND
INTERVENTION, AND WILL THE STUDY
ACHIEVE ADEQUATE SEPARATION,
SOMETHING THAT IS A BIGGER
CONCERN BECAUSE WE HAVE LESS
CONTROL OF THE INTERVENTION WHEN
THE INTERVENTION IS EMBEDDED ON
TOP OF TYPICAL CLINICAL CARE.
SO, ONE THING IS THE
REENGINEERING, WE'VE HEARD ABOUT
THE INTERVENTION AND MAKING
CHANGES AS WE GO ALONG.
THE OTHER DESIGN CONSIDERATION,
NOW I'VE BEEN HEARING FROM
LECTURES YOU SHOULD TRY TO
OVERPOWER YOUR TRIAL.
SO WE OVERPOWER TO ACCOUNT FOR
DROPOUT, WE SHOULD ALSO CONSIDER
A FACTOR TO OVERPOWER FOR LOWER
ADHERENCE OF UPTAKE OF THE
INTERVENTION, ANOTHER WAY TO
PROTECT AGAINST THIS.
NEXT SLIDE.
SO, THE OTHER ISSUE IN PRAGMATIC
CLINICAL TRIALS IS TRIALS ARE
NOT BLINDED.
SO, THEY ARE RANDOMIZED, WE DID
A LOT OF OBSERVATIONAL STUDIES,
SENT MANUSCRIPTS, THEY COME BACK
AND SAY IT'S NOT RANDOMIZED SO
WE DON'T KNOW THE UNMEASURED
FACTORS OF DIFFERENCES.
ONE THING OVERLOOKED IS POWER OF
BLINDED STUDY, TO PROTECT THE
TRIAL TO PROTECT IT FROM
NONRANDOM BIAS, ONCE THE TRIAL
STARTS.
PRESERVING INTERNAL VALIDITY OF
THE STUDY IN TRADITIONAL TRIALS.
AND THE CHALLENGE OF PRAGMATIC
TRIALS, IT'S OFTEN DIFFICULT OR
NOT FEASIBLE TO BLIND THE
INTERVENTION TO THE HEALTH CARE
PROVIDERS, OR WHO HAS DELIVERED
THE INTERVENTION, AND TO BLIND
THIS TO THE PARTICIPANTS.
AND THE CRUX OF THE TRIAL
ASSUMES THERE'S EQUI POISE
BETWEEN THE TWO TREATMENTS.
IF THERE'S NO EQUIPOISE THEN IT
MAKES IT MORE DIFFICULT TO
JUSTIFY A TRIAL.
NEXT SLIDE.
SO THIS IS A DIAGRAM.
AND WHEN I THOUGHT ABOUT THINGS
AND LOOKED AT THE LITERATURE,
ALL THE THINGS THAT COULD GO
WRONG AND CAUSE NONRANDOM BIAS
WHEN YOU'RE NOT BLINDED, COULD
START AT THE SCREENING PHASE
WHERE THERE'S DIFFERENT
APPLICATIONS, WHO YOU INCLUDE
AND EXCLUDE IF YOU KNOW WHAT THE
TREATMENT IS, THIS WOULD BE A
DIFFERENCE IN HOW PEOPLE ARE
CONSENTED.
AND ALSO PATIENTS HAVE
PREFERENCES, MAY CLUES TO HE
BE - MAY CHOOSE TO ENROLL IN ONE
GROUP AND NOT THE OTHER IF THEY
KNOW WHAT THEY ARE BEING
RANDOMIZED TO, RANDOMIZATION
OCCURS BEFORE ENTHUSIASM.
ENTHUSIASM, CROSSOVER FOR
PATIENTS WHO DON'T GET PREFERRED
CHOICES, THEY KNOW IT'S NOT
BLINDED, AND DIFFERENTIAL
REPORTING ESPECIALLY IF THERE'S
A HEALTH CARE PROVIDER
ASCERTAINING OUTCOMES.
THERE'S WAYS TO DEAL WITH THESE
ISSUES THAT IS NOT -- DOES NOT
USE BLINDING BUT THAT'S I THINK
SOMETHING FOR A DIFFERENT
LECTURE.
BUT I WANT YOU TO THINK ABOUT
THIS WHEN YOU'RE LOOKING AT A
TRIAL.
NEXT SLIDE PLEASE.
SO FOR ME AS A SPONSOR I THINK
ABOUT DE-RISKING TRIALS, NOT
TRYING TO HIT THE HOME RUN.
I WANT TO DE-RISK THE TRIAL AND
TRY TO MAKE SURE WE CAN FINISH
THE TRIAL AND GET SOMETHING
SENSIBLE OUT OF IT.
AND THE COMPLEXITY OF THESE
TRIALS MAKE THESE TRIALS I
BELIEVE RISKIER.
AND INCREASED CHANCES ONCE WE
START THESE TRIALS WE CAN
UNCOVER EARLY UNANTICIPATED
SIGNALS THAT SUGGEST THERE'S A
PROBLEM IN HOW THE TRIAL IS
GOING.
AND ONCE AGAIN, I THINK OF THESE
AS QUALITY IMPROVEMENT TRIALS,
LOOKING AT AHRQ THERE'S A
PROCESS OF ENGAGING
STAKEHOLDERS, DEMONSTRATION
PROJECTS, IT'S ITERATIVE.
WE GO BACK AND CHANGE OR
REENGINEER BECAUSE OF
UNANTICIPATED OUTCOMES,
ESPECIALLY COMPLEXITY AND
HETEROGENEITY IN THE POPULATION.
IS IT OKAY TO FAIL?
I THINK IT'S OKAY TO FAIL.
IN MANY QUALITY IMPROVEMENT
PROJECTS WE DON'T GET IT RIGHT
THE FIRST TIME.
IF WE SEE SOMETHING NOT GOING
RIGHT IN THE FIRST 10 TO 20% OF
PATIENTS ENROLLED, IS THIS AN
OPPORTUNITY TO STOP AND
REENGINEER AND CHANGE THE COURSE
OF THE TRIAL?
IT GETS MUCH MORE DIFFICULT WHEN
50% OF THE POPULATION IS
ENROLLED BUT WHEN IT'S AT 10%
WE'RE JUST GETTING STARTED AND
THERE'S STILL A LOT OF -- IT'S
STILL EARLY IN THE TRIAL AND I
BELIEVE WE SHOULD FIND OUT
THINGS EARLIER AND MAKE CHANGES
EARLIER.
I THINK THAT'S THE BEST THING TO
HAVE A BEST TO CHANGE AND
REENGINEER.
NEXT SLIDE.
SO PAUSING SOME TRIALS, I JUST
WANT TO CLOSE OUT WITH THIS,
THAT TO STOP OR PAUSE OR CHANGE
A TRIAL, IT PLACES -- DSMB HAS
TO MAKE A DIFFICULT DECISION.
HAVING ROBUST DATA IS IMPORTANT.
IT HAS TO BE OBJECTIVE.
SO IT'S NOT ONE SIGNAL.
IT'S ABOUT THE TOTALITY OF ALL
THE DATA.
YOU HAVE TO LOOK AT TOTALITY AND
HAVE AN EXPERT GROUP OPINION AND
DISCUSSION ABOUT WHAT DO YOU SEE
HERE.
IT'S NOT JUST ONE ISOLATE THAT
TAKES OUT.
YOU HAVE TO THINK ABOUT THIS AS
A BURDEN OF PROOF.
IT'S NOT PREPONDERANCE OF
EVIDENCE, HOW IT STAYS CLEAR AND
CONVINCING, HIGHLY LIKELY TRUE
THAN UNTRUE IS I WOULD SAY THE
BURDEN THAT WE ARE TRYING TO
THINK ABOUT, WHAT DOES THE
TOTALITY OF DATA TELL US?
IT SHOULD NOT COME AS A SURPRISE
DSMB INVESTIGATORS SHOULD SEE
THIS COMING.
WE SHOULD DISCUSS THIS.
IDEALLY BEFORE, DISCUSS WHAT ARE
THE GOALS, ARE THERE SECONDARY
RESEARCH OBJECTIVES GETTING OUT
OF THIS TRIAL AND DATA?
INVESTIGATORS ARE NOT BLINDED,
IT'S DIFFICULT FOR A COLLEAGUE
SITTING ON THE DSMB TO STOP
SOMEBODY ELSE'S TRIAL.
SO, THERE'S SOME DSMB HESITANCY
TO MAKE A DECISION.
ARE WE NOT GOING TO ANSWER THE
PRIMARY QUESTION, SHOULD WE
REBOOT OR REENGINEER OR ARE
THERE SECONDARY HIGH --
HYPOTHESES.
WE USE BIOETHICIST, HELP FOLLOW
ASK IF WE ASKED PATIENTS TO GIVE
THEIR TIME, ASKED HEALTH CARE
PROVIDERS TO GIVE TIME AND
RESOURCES, AND WE'VE ASKED THEM
TO BE PART OF THIS BIGGER STUDY
TO ANSWER A CERTAIN QUESTION AND
WE WANT TO BE ABLE TO SAY THAT
WE CAN ETHICALLY AND
REALISTICALLY ANSWER THIS
QUESTION GIVEN WHERE WE ARE
TODAY.
THE LAST THING, IT'S NOT
PERSONAL, BUT INVESTIGATORS WILL
SURE FEEL THAT WAY IF THERE'S A
MAJOR CHANGE IN THE COURSE OF
THE DSMB RECOMMENDATIONS.
NEXT SLIDE.
WITH THAT, I'LL END MY TALK AND
THANK YOU FOR HAVING ME.
>> THANK YOU SO MUCH, KEVIN.
NEXT UP DR. SHERYL BOYCE, WHO
WILL GIVE US THE PERSPECTIVE
FROM OFFICE OF STRATEGIC
COORDINATION AT THE COMMON FUND.
>> IN THINKING ABOUT WHAT WE'RE
TALKING ABOUT TODAY I WANTED TO
ALSO ADD IN TERMS OF SOME OF THE
CHALLENGES NOT ONLY THE DATA
THAT WE MIGHT TRADITIONALLY
THINK ABOUT BUT ALSO VALUE OF
QUALITATIVE DATA, CONTEXT, IN
TERMS OF DIFFERENT POPULATIONS
AND ROLES OF PATIENTS AND
COMMUNITIES AND HOW THAT CAN BE
IMPORTANT IN TERMS OF WHAT WE DO
IN TERMS OF EARLY SIGNALS AND
HOW WE MAKE CONSIDERATIONS FOR
HOW WE ACT.
IN TERMS OF THE DIFFERENT
LEVELS, WE HAVE PATIENTS,
PRACTITIONERS, TEAMS, HEALTH
SYSTEMS, AND COMMUNITIES WHO ARE
ALL A PART OF PRAGMATIC TRIALS
AND THE DECISIONS THAT WE MAKE.
AND ALSO THINKING ABOUT
INVOLVING SOMEWHAT OF OUR
COMMUNITY BASE REVIEW PROCESS
AND HOW WE MAKE SOME OF THESE
DECISIONS.
WE DO THINK ABOUT COMMUNITY
BASED RESEARCH AND
PATIENT-INVOLVED RESEARCH IN NEW
WAYS, ESPECIALLY AS WE'RE
REENGINEERING RESEARCH
ENTERPRISE WITHIN THE OFFICE OF
STRATEGIC COORDINATION OR COMMON
FUND WE ARE MOST RECENTLY HAVE A
COMMUNITY PARTNERSHIP TO ADVANCE
SCIENCE FOR SOCIETY WHERE WE ARE
FUNDING COMMUNITIES DIRECTLY TO
GIVE THEM MORE POWER IN THE
ACTUAL RESEARCH PROCESS.
BUT IN TERMS OF THINKING ABOUT
SOME OF THE INDIVIDUAL AND
COMMUNITY LEVEL PROTECTIONS AND
ETHICS, AND ALSO TO REALLY
ENHANCE WHAT WE DO, AND ENHANCE
AND BUILD OUR CAPACITY AND HOW
WE WORK WITHIN DIFFERENT
COMMUNITIES AND PATIENT
COMMUNITIES TO SHAPE OUR
RESEARCH, AND REALLY THINK ABOUT
SUSTAINABILITY THAT EFFECTIVE
COMMUNITY REVIEW MAY BE
IMPORTANT.
AND THAT COMMUNITY CAN BE
DEFINED IN DIFFERENT WAYS, BUT
THINKING ABOUT THAT IN TERMS OF
WHAT ARE THE OPPORTUNITIES IN
TERMS OF THE CHALLENGES THAT WE
FACE WITH DETECTING EARLY
SIGNALS.
AND AS WE THINK ABOUT HOW WE MAY
PROCEED, UNDERSTANDING AS WE
MOVE FORWARD INTO IMPLEMENTATION
AND SUSTAINABILITY THAT THERE'S
THE ROLE OF DEIMPLEMENTATION,
AND THAT NO MATTER WHAT THERE
WAS SOMETHING ELSE THAT PEOPLE
WERE DOING, WHETHER IT MAY BE
DIFFERENT TREATMENT OR DIFFERENT
BEHAVIORS, AND THOSE ALL PLAY
VERY WELL INTO HOW WE MAY BE
ABLE TO SUSTAIN OR ACT IN TERMS
OF THE FINDINGS THAT WE HAVE.
AND MAKING SURE THOSE ARE
CONSIDERED AND THAT THERE'S
INTROSPECTION AS WE MOVE
FORWARD.
IMPACT ON SOMETHING STARTING
EARLY OR STOPPING REALLY CAN BE
QUITE PROFOUND, AND WITHOUT
INVOLVING PATIENTS AND
COMMUNITIES THROUGHOUT THE
PROCESS WE REALLY ARE LACKING AN
IMPORTANT PERSPECTIVE.
IMPORTANT PERSPECTIVE THAT COULD
BE AN OPPORTUNITY TO STRENGTHEN.
AND HOW WE DESIGN OR ADAPT OR
THINK ABOUT HOW WE MIGHT CHANGE
IN TERMS OF WHEN WE PLAN AND
WHEN WE DO.
AND WE ALSO HAVE A GREAT
OPPORTUNITY IN TERMS OF THE
CONSIDERATIONS AND WHEN WE DO
HAVE TO ACTUALLY PROCEED WITH
MAKING A CHANGE IN TERMS OF
ADAPTATION OR STOPPING A TRIAL.
SO WITH THAT I REALLY WANTED TO
ENCOURAGE SOME ADDITIONAL
PERSPECTIVES ACROSS THE LEVELS
THAT INFORM WITH COMMUNITY
REVIEWS AND THOSE WITH LIVED
EXPERIENCE THAT MAY INFORM HOW
WE PROCEED AND SOME OF THE
RECOMMENDATIONS THAT WE MOVE
FORTH WITH, WITH THE PANEL
TODAY.
THANK YOU.
>> THANK YOU, CHERYL.
LET'S GO FROM BIOETHICS
PERSPECTIVE FROM COLLABORATORY
COLLEAGUE, PEARL O'ROURKE.
>> I HAVE SLIDES.
THANK YOU.
I WANT TO SPEND A FEW MINUTES
TALKING ABOUT THE ETHICAL AND
REGULATORY CONSIDERATIONS FOR
PIVOTING ON A STUDY.
BOTTOM LINE, THERE'S ETHICAL AND
REGULATORY REASONS FOR PIVOTING.
EVERYTHING IS IN FAVOR OF
PIVOTING.
NEXT SLIDE PLEASE.
>>PEARL, IF YOU PAUSE FOR A
SECOND, WE HAVE TO GET IT OUT OF
PRESENTER MODE THERE.
>> THAT'S SOMETHING YOU HAVE TO
DO OR I HAVE TO DO?
>> I THINK WE DO IT ON OUR END.
>> I'LL JUST KEEP TALKING, IF
THAT'S OKAY.
>> SURE.
>> ALL RIGHT.
JUST TO REMIND EVERYBODY, THE
REAL BASIC QUESTIONS THAT YOU
HAVE TO LOOK AT IF YOU'RE DOING
A STUDY, FROM ETHICS AND
REGULATION, IS -- NOW WE'VE
TOTALLY DISAPPEARED -- IS
LOOKING AT THE RISK/BENEFIT
ANALYSIS.
AND BOTH FOR ETHICAL REASONS AS
WELL AS REGULATORY, YOU WANT TO
MAXIMIZE BENEFIT, THE QUESTION
THERE IS THE STUDY WORTHWHILE
DOING, DOES IT MATTER, WILL IT
BE IMPLEMENTED?
WE TALKED YESTERDAY ABOUT
IMPLEMENTATION.
I DON'T KNOW IF WE LOOKED AT
THAT IN ENOUGH DETAIL.
YOU WANT TO DECREASE RISK.
I THINK WHAT ONE OF THE POINTS
KEVIN MADE JUST NOW IS RELEVANT
IN TERMS OF, YOU KNOW, IF IT'S
GOING BAD, EVEN IF MINOR RISK,
SHOULDN'T BE DONE.
ANY RISK IS NOT APPROPRIATE IF
THERE'S GOING TO BE NO BENEFIT.
DOWN TO THE NEXT SLIDE PLEASE.
OKAY.
ONE BEFORE THAT.
FABULOUS, OKAY.
SO, BASICALLY, I THINK ALSO AS
KEVIN NOTED, IF YOU KNOW THE
ANSWER, YOU SHOULDN'T BE DOING
THE RESEARCH.
IT'S NOT ETHICALLY APPROPRIATE
TO HAVE PEOPLE GO THROUGH ANY
LEVEL OF RISK.
THE OTHER IS TRANSPARENCY AND
HONESTY.
IT TALKED ABOUT COMMUNITY
GROUPS.
I THINK THAT'S PART OF THE
ANSWER.
WE TALK ABOUT PRAGMATIC TRIALS,
MUCH GETS A WAIVER OF INFORMED
CONSENT SO IT'S BECOME THE
STICKIER WICKET IN TERMS OF DO
YOU NOTIFY OR NOT.
IT'S ALWAYS REASONABLE TO TELL
PEOPLE, POSSIBILITY OF CHANGE,
POSSIBILITY OF EARLY
TERMINATION, WHAT HAPPENS TO
THAT PERSON?
NEXT SLIDE PLEASE.
THANK YOU.
ONE OF THE ISSUES ABOUT IRB
REVIEW YOU ALL KNOW IT'S
FRONT-ENDED.
YOU'RE SUBMITTING YOUR BEST
GUESS OF THE WAY THINGS WILL GO,
BUT NOT UNTIL YOU HAVE BOOTS ON
THE GROUND DO YOU REALLY HAVE AN
IDEA, BETTER GRASP OF WHAT IS
HAPPENING.
THE IRB IS ONLY LOOKING AT FRONT
END BEFORE IT GIVES YOU
APPROVAL.
REVIEW DOES INCLUDE STUDY
DESIGN.
WE TALKED ABOUT SECONDARY AIMS,
IF YOU HAVE OTHER AIMS OF THE
STUDY YOU SHOULD INCLUDE THAT,
JUST SAY SO.
AND THE IRB LOOKS AT CONTEXT.
WHAT YOU'RE PROPOSING TO DO IS
ACTUALLY RESEARCH VERSUS
STANDARD OF CARE THAT WOULD
HAPPEN ANYWAY.
RELEVANT POLICIES, WE GOT A GOOD
TALK IN TERMS OF STOP CRC,
POLICIES THAT CAME INTO PLAY.
RELATED RESEARCH, IF THERE'S
ALREADY BEEN EXTRAORDINARILY
GOOD RESEARCH THAT GIVES YOU THE
ANSWER, PERHAPS RESEARCH SHOULD
NOT BE DONE.
AGAIN, IMPLEMENTATION ISSUES.
NEXT SLIDE PLEASE.
NOW, ETHICAL REVIEW, ETHICS AND
REGULATORY ISSUES CONTINUE
THROUGH THE ENTIRE LIFE OF THE
PROTOCOL AND NOT JUST INITIAL
IRB APPROVAL.
AND I THINK YOU HAD GREAT
EXAMPLES HOW CIRCUMSTANCES CAN
CHANGE.
OBVIOUSLY COVID THROUGH A MAJOR
SAND INTO THE GEARS OF RESEARCH,
NEW RELEVANT POLICIES AGAIN I
WOULD GO TO THE STOP CRC HAS AN
EXAMPLE.
INFORMATIVE NEW RESEARCH, IRBs
RELY ON INVESTIGATORS TO HELP.
WHAT HAS HAPPENED IN THE
INTERIM?
HAVE NEW STUDIES COME OUT?
HAVE YOU FOUND OUT THE STUDY IS
NO LONGER FEASIBLE OR YOU CAN'T
DO IT FOR, YOU KNOW, JUST WHEN
YOU MADE THE DESIGN YOU DID YOUR
BEST EFFORT, UNFORTUNATELY IT
DOESN'T MEET WELL ARE REAL LIFE.
ALSO ARE THERE INTERIM RESULTS
THAT SUGGEST THAT, WELL, MAYBE
YOU ALREADY HAVE YOUR ANSWER AND
YOU CAN STOP THE STUDY EARLIER,
OR ELSE IT'S JUST NOT FEASIBLE
TO CONTINUE.
NEXT SLIDE PLEASE.
SO WHAT TO DO.
THERE'S DEFINITELYETHICAL AND
REGULATORY SUPPORT TO PIVOT.
THE THING ABOUT PIVOTING, YOU DO
MAXIMIZE BENEFIT, MAKING THE
INTERVENTION EARLIER TO OTHERS,
REMEMBERING RESEARCH IS GOING
FOR GENERALIZABLE KNOWLEDGE, NOT
JUST AT YOUR INSTITUTION,
HOPEFULLY OTHER INSTITUTIONS,
AND MAKING IT A STANDARD OF CARE
ELSEWHERE, AND IT WOULD BE A
GREAT BENEFIT.
YOU ALSO MINIMIZE RISK IN THAT
BOTH TO ENROLLED PARTICIPANTS
YOU CAN SAY, OKAY, WE'RE NO
LONGER IN A STUDY, YOU NO LONGER
HAVE TO MAKE OUT SURVEYS, NO
LONGER HAVE TO LOOK AT MEDICAL
RECORDS AND ALSO ENROLL FEWER
PARTICIPANTS.
BUT I REALLY WANT PEOPLE TO BE
CAREFUL BECAUSE PIVOTING, YOU
REALLY NEED TO HAVE A GOOD
JUSTIFICATION.
AND I THINK CORITA MADE THE
COMMENT THERE'S A DANGER TO
PIVOTING TOO SOON, ARE YOU
GETTING A SIGNAL, HARD TO TELL
NOISE FROM SICK -- SIGNAL AT
SOME POINT.
JUSTIFICATION IN MY MIND SHOULD
INCLUDE THOSE SAVVY WITH STUDY
DESIGN, YOUR STATISTICIAN, IS
THIS A GOOD MOVE TO MAKE.
TONGUE IN CHEEK, TRY TO AVOID
ERRORS OF ENTHUSIASM.
I THINK AS PEOPLE START FIRST
FOUR OR FORTY PARTICIPANTS,
GOING TERRIFICALLY, CAN BE
TEMPTING TO SAY, WOW, WE ALREADY
HAVE THE ANSWER AND YOU NEED TO
GO BACK TO THE STUDY TEAM AND
EMPHASIS WITH STATISTICIANS.
ANY CHANGE TO THE IRB, RESEARCH
PROTOCOL, NEEDS IRB APPROVAL.
THE IRB MAY WELL - AMENDMENT TO
INITIAL PROTOCOL MAY BE ALL
THAT'S NEEDED.
SOME BECOME LIKE CORAL REEFS,
HAVE YOU INITIAL PROTOCOL, ADD
ON, ADD ON, AT SOME POINT THIS
MAY BECOME THE NEW PROTOCOL SO
YOU MAY BE ASKED TO WRITE A NEW
PROTOCOL.
REMEMBER WITH CHANGES, IT MAY
CHANGE THE RISK/BENEFIT
ANALYSIS.
AND SUDDENLY YOU MAY NOW BE MOE
THAN MINIMAL RISK, WHAT WAS
APPROVED AS WAIVER OF INFORMED
CONSENT, NOW YOU WOULD HAVE TO
GET INFORMED CONSENT.
AS YOU ALTER PROTOCOL DO
REMEMBER THE ANALYSIS MAY ALSO
ALTER WITH THAT.
NEXT SLIDE PLEASE.
THE CHALLENGES I SEE WITH
PRAGMATIC TRIALS AND I THINK
KEVIN WENT THROUGH A LONG LIST
OF MANY OF THE CHALLENGES, BUT
FOR THE IRB AND I THINK FOR
ANYONE LOOKING AT ETHICAL
ISSUES, MOST OF THE TRIALS ARE
MULTI-SITE.
SOME ARE MULTI-SITE WITHIN A
SYSTEM SO YOU MAY HAVE MORE
STANDARDIZATION.
BUT MANY ARE MULTI-SITE BETWEEN
DIFFERENT STATES, DIFFERENT
SYSTEMS, DIFFERENT TYPES OF
ORGANIZATIONS.
AND YOU HAVE INCREDIBLE
VARIATION OF THINGS, I LISTED
SOME HERE.
BUT THE STANDARD OF PRACTICE IN
SOME INTUITIONS YOUR RESEARCH
PROTOCOL MAY ACTUALLY BE THE
STANDARD.
IN OTHERS, SOMETHING THEY NEVER
ACTUALLY PROVIDED.
SORRY, MY DOG IS BARKING.
SORRY FOR THAT.
THERE MAY BE VARIATIONS IN LOCAL
POLICIES.
ALSO RESOURCES FOR
IMPLEMENTATION.
AND THE CHALLENGE, OTHER THAN MY
DOG, IS SINGLE IRB REVIEW.
I THINK IT'S VERY DIFFICULT FOR
ONE SINGLE IRB TO BE ASSESSING
ALL OF THESE CHALLENGES,
CONTEXTUAL CHALLENGES, FOR
INSTITUTIONS.
MY FINAL SLIDE, PLEASE.
SO, MY SUMMARY IS DEFINITELY
PIVOT.
IF IT'S NOT WORKING, CHANGE IT.
IF YOU'RE GETTING THE ANSWER,
STOP IT.
AND DEFINITELY INVOLVE YOUR
TEAM, YOUR STATISTICIANS, AND
REALIZE THE IRB DEFINITELY NEEDS
TO BE INVOLVED IN ANY DECISION
MAKING AROUND THIS.
THANK YOU.
AND I APOLOGIZE FOR MY CANINE
INTERFERENCE.
>> THANKS, PEARL.
LET YOUR DOG KNOW THEY NEED TO
WAIT UNTIL THE DISCUSSION
PERIOD.
WE STILL HAVE ONE MORE
PRESENTATION.
>> AND I JUST THREW HER
OUTSIDE.
HOPEFULLY THAT WILL STOP IT.
>> OKAY.
GREAT, THANK YOU.
OUR LAST PERSPECTIVE HERE WILL
BE, NOT THE LEAST, BIOSTATISTICS
PERSPECTIVE.
COLLABORATOR, ANDREA COOK WILL
DISCUSS THESE THINGS.
HI, ANDREA.
>> GOOD MORNING.
IT'S EARLY ON THE WEST COAST.
I'M COLLABORATORY BIOSTATISTICS,
ON A DSMB AS WELL, TO GIVE THAT
PERSPECTIVE OF WHAT WE'RE
LOOKING AT IN THE COURSE OF A
PRAGMATIC TRIAL, DESIGN CHOICES,
STOPPING AND HOW WE TRY TO HELP
WITH STUDIES PIVOT IF NECESSARY
GIVEN DATA WE'RE REVIEWING.
MY EXPERIENCE, PCT, FORMAL
STOPPING FOR EFFECTIVENESS, I
HAVE IT MORE IN PHASE 4 TRIALS
WHERE IT'S MORE CLEAR WHEN YOU
WOULD STOP.
PRAGMATIC TRIAL BY DESIGN, USING
NOT JUST ONE AIM, EFFECTIVE WITH
TRIALS, BUT REALLY LOOKING AT --
WE KNOW THE ACTION IS HAPPENING,
DESIGN IS COMPLICATED, CLUSTER
CENTER WEDGE, STOPPING EARLY
MIGHT NOT BE THE RIGHT AIM OF
THE TRIAL, ALMOST NOT USEFUL FOR
RESOURCES GIVEN BECAUSE YOU WANT
TO SEE SUBGROUPS AND SORRY
THINGS, GENERALIZABILITY OF THE
INTERVENTION.
SOMETIMES INTERESTED IN LONGER
TERM OUTCOMES, MAYBE EFFICACY,
IT'S HARDER TO STOP FOR
EFFECTIVENESS.
HOWEVER, WHAT WE TEND TO LOOK AT
A LOT IS RECRUITMENT GOALS, SOME
THINGS THAT OFTEN HOW WELL THEY
ARE GENERALIZABLE, ANALYZE HOW
WELL THEY ARE RECRUITING AND
RETAINING CLUSTERS, HOW WELL
PARTICIPANTS ARE PARTICIPATING
IN THOSE GOALS.
IF THOSE AREN'T TO BE MET A LOT
OF TIMES WE'LL START WORKING
WITH YOUR TRIALS AND THIS IS
STUFF TO THINK ABOUT, REALLY
ABOUT THEIR ABILITY OF OUTCOME
AND CONTROL GROUP, SO WHAT'S
NICE PCTs A LOT OF OUTCOMES
FROM EHR, YOU CAN PULL OUT AND
CHECK SAMPLE SIZE ASSUMPTION.
IN COLLABORATORY, PHASE 1
THROUGH UG3 PHASES, A TON OF
WORK USING REAL LIFE DATA,
SHOULD ALSO DO IT OFTEN IN
MONITORING OF THE TRIAL.
IPC IS REALLY IMPORTANT.
I THAT COULD SHOW EFFECTIVENESS
IF YOU UNBLIND YOUR RESULTS.
WE DO MONITOR FOR FUTILITY AND
WAS THINKING WHEN LISTENING, WE
DON'T NORMALLY DO IT FORMALLY.
WE LOOK AT IMPLEMENTATION
OUTCOMES, ADHERENCE, UPTAKE OF
INTERVENTION.
AT ONE TRIAL I'VE BEEN ON WHERE
REDUCING AND CHANGING BLOOD
PRESSURE, OUTCOME IS
CARDIOVASCULAR OUTCOME, WITH
FUTILITY, BLOOD PRESSURE
SEPARATIONS, A PROCESS TO GET TO
THE HARDER OUTCOME, WONDERING IF
PCTS, MONITOR FOR FUTILITY MORE
ON THE IMPLEMENTATION SIDE TO
MAKE SURE UPTAKE IS HAPPENING
AND MAYBE MORE FORMAL PROCESS OF
WHAT YOU DO IF YOU'RE NOT SEEING
UPTAKE, HELP FROM MEMBERS TO
HAVE MORE FORMAL PROCESSES.
SOMETIMES IT IS HARD TO MAKE A
DECISION GIVEN WHAT'S GOING ON,
A LOT OF INVESTMENT, HOW TO MAKE
DECISIONS AHEAD OF TIME OR WHEN
PEOPLE SHOULD PIVOT AND WHAT
SHOULD HAPPEN.
I WAS ALSO INVOLVED IN A LOT OF
TRIALS DURING COVID.
THAT'S WHERE A LOT OF CHANGE IS
HAPPENING, UNFORTUNATELY,
THROUGH THE PANDEMIC AND WE HAD
TO MAKE DIFFICULT DECISIONS NOT
NECESSARILY STOPPING BUT SOME
TRIALS WERE ACTUALLY STOPPED,
NEW TRIAL WAS RESTARTED BECAUSE
OF THE INTERVENTION WAS SO
DIFFERENT WE COULDN'T CALL IT
THE SAME INTERVENTION.
THAT I THINK IS THE RIGHT THING
TO DO THAT AND PIVOTING, IRB
REVIEW IS REALLY NEW STUDY.
HOPEFULLY WE DON'T HAVE ANOTHER
PANDEMIC BUT IF AN INTERVENTION
DOES MAKE SUCH MAJOR CHANGES
THAT YOU WOULDN'T REALLY WANT TO
PULL THAT DATA TOGETHER I DO
THINK ALMOST TREATING IT AS
MAYBE A FULL PIVOT OF A NEW
TRIAL IS SOMETHING TO REFLECT
ON.
STATISTICALLY WE OFTEN -- SINCE
COVID STARTED INTRODUCING
ESTIMATES OVER TIME TO TRY TO
CAPTURE THIS IN THE PANDEMIC,
WITH IMPLICATIONS OF POWER, IF
YOU CAN'T COMBINE EFFECTS DUE TO
COVID, THE STUDY IS UNDERPOWERED
THERE'S IMPLICATIONS FOR BOTH
AND WE HAD TO THINK ABOUT WHAT
TO DO WITH THAT INFORMATION,
POSSIBLY REDESIGN A STUDY AND
MAKE IT GO LONGER IF THERE'S
FUNDS OF COURSE AVAILABLE TO
ANSWER QUESTIONS OF INTEREST.
IN THE CONTEXT, WE'VE LEARNED A
LOT IN COVID, WORST CASE
SCENARIO HOPEFULLY WHAT CAN
HAPPEN IN TRIALS AND PIVOTING.
PEOPLE HAVE BEEN GOOD ABOUT
DOCUMENTING CHANGES, TRYING TO
WORK WITH THE FUNDER AND DSMB OF
WHEN THINGS NEED TO FULLY PIVOT
TO A NEW TRIAL, I THINK THE
DESIGN CHOICES, I THINK IT'S
CHALLENGING FOR SEPARATE PIVOTS.
SO I THINK IF YOU THINK THERE'S
GOING TO BE A LOT OF ADAPTATION
AND CHANGES, OTHER MAYBE
PARALLEL GROUP OF CLUSTER
RANDOMIZED DESIGN MIGHT BE
BETTER OUTLET, OTHER DESIGN
CHOICES, IF THIS THING IS OF
WORRY, BECAUSE YOU HAVE TIME
CONFOUNDING.
STEPPING BACK A LITTLE BIT OF
DESIGNING YOUR STUDY BEFOREHAND
TO HANDLE THESE COMPLEXITIES AS
MUCH AS YOU CAN AND THEN YOU DO
NEED TO PIVOT OVER TIME BUT
MAKING SURE YOU'RE ESTIMATING
SOMETHING THAT IS GOING TO BE
IMPORTANT TO DESCRIBE THE
INTERVENTION, AND IF THAT
ESTIMATE IS CHANGING SO MUCH
DURING THE COURSE OF THE TIME
REALLY MULTIPLE INTERVENTIONS
THAT YOU NEED TO BE TESTING, AND
REALLY WORKING TO TRY TO GET THE
RIGHT ESTIMATE GIVEN YOUR
COMPLEXITY OF INTERVENTION AND
UPDATE YOUR POWER AND SAMPLE
SIZE CALCULATION AS WELL.
THAT HAPPENS A LOT ACTUALLY IN
THE TRIALS.
LUCKILY MOST PEOPLE ARE
CONSERVATIVE WITH POWER.
I THINK WE OFTEN INFLATE SAMPLE
SIZE, CONFOUNDING AS WELL,
SELECTION BIAS IN CLUSTER
RANDOMIZED TRIALS, NOT AS CLEAR
HOW TO HANDLE ALL THESE ISSUES
OF UPDATING SAMPLE SIZE EVERY
TIME IS REALLY IMPORTANT WHEN
YOU'RE DOING THIS, NOT JUST
BASED ON WHAT YOU PLANNED BUT
BASED ON WHAT YOU'RE DOING AND
SEEING.
THAT'S MOSTLY WHAT I WANTED TO
DESCRIBE, AND HAVE DISCUSSION AS
A WHOLE GROUP TOGETHER.
>> THANK YOU SO MUCH, ANDREA.
LET'S OPEN UP FOR DISCUSSION.
I WANT TO BEGIN BY ASKING PRIOR
PANELISTS HERE WHETHER ANY OF
THE DISCUSSION CUED ADDITIONAL
THOUGHTS, THINGS YOU WANT TO
COMMENT ON, QUESTIONS YOU HAD
FOR EACH OTHER.
A LOT OF STUFF WAS PUT ON THE
TABLE.
>> THIS IS PEARL.
I LOVED WHAT ANDREA SAID LOOKING
AT STEP WEDGE, IF YOU THINK A
STUDY MAY NEED NIP AND TUCK AND
ALTERATIONS GOES BACK TO STUDY
INFORMED DESIGN.
THANKS FOR BRINGING THAT UP,
ANDREA.
>> YEAH, I SECOND THE AVOID THE
STEP WEDGE AT ALL COSTS.
>> I DIDN'T WANT TO SAY EXACTLY
THAT.
>> I DID -- YEAH, PANDEMIC IN
THE MIDDLE OF IT.
IT'S REALLY HARD WHEN YOU HAVE
COMPLEX INTERVENTIONS THAT
REQUIRE A LOT OF HUMAN RESOURCES
TO ROLL OUT.
WE COULDN'T DO, AND IT HAD TO BE
A CLUSTER DESIGN, AND TOOK A TON
OF RESOURCES.
THERE'S NO WAY TO DO IT AT TOO
MANY SITES IN REAL TIME N FACT
REALLY MORE THAN ONE OR TWO.
SO THAT WAS OUR DIFFICULTY.
>>ONE THING I'D LIKE TO ASK KEN
ABOUT, YOU COMMENT A LOT OF
THESE TRIALS ARE QUALITY
IMPROVEMENT.
I WANTED TO MAKE PEOPLE BE AWARE
FROM A REGULATORY STANDPOINT,
QUALITY IMPROVEMENT AND RESEARCH
ARE DIFFERENT.
>> YES.
>> QUALITY IMPROVEMENT DOES NOT
FALL UNDER REGULATIONS, WHEREAS
RESEARCH DOES.
NIH FUNDS RESEARCH, IT DOESN'T
FUND QUALITY IMPROVEMENT.
AT LEAST THAT I'M AWARE OF.
SO I THINK WHILE A LOT OF THIS
RESEARCH INVOLVES PROBABLY
IMPROVING ISSUES, JUST BE
CAREFUL IF YOU CALL YOUR STUDYA
QUALITY IMPROVEMENT STUDY, IF
YOUR IRB AGREES, THAT GOES DOWN
A DIFFERENT ROUTE.
>> QUALITY IMPROVEMENT LIKE,
ISN'T THERE A SEPARATE TRIAL FOR
QUALITY IMPROVEMENT RESEARCH?
>> AS SOON AS YOU PUT THE WORD
"RESEARCH" IN THERE, BUT MOST
PLACES WILL SAY RESEARCHERS
CREATE GENERALIZABLE KNOWLEDGE.
QUALITY IMPROVEMENT, MOST PEOPLE
SAY IT'S MORE OF A LOCAL THING,
I'LL IMPROVE WHAT I'M DOING
HERE.
WE GO ON TO SAY RESEARCH HAS
RIGID RULES, HERE IS WHERE YOU
BRING PRAGMATIC TRIALS.
IT STARTS TO BLUR LINE AND
ASSESSMENT BETWEEN QUALITY
IMPROVEMENT AND RESEARCH.
IRBs INTERPRET, I DON'T THINK
THERE WOULD BE 100% AGREEMENT QI
AND RESEARCH.
>> A LOT IS GENERALIZABLE
RESEARCH.
IF IT'S NOT, IT'S A LOCAL THING
TO IMPROVE PATIENT CARE,
SOMETIMES RETROSPECTIVE, YOU
HAVE TO GO TO THE IRB AND SAY
RETROSPECTIVELY.
>> YOU DON'T HAVE TO DO THAT
ANYMORE.
>>GETTING PUBLISHED IS NOT A
TRIGGER FOR IRB REVIEW.
UNFORTUNATELY, A LOT OF JOURNALS
WILL SAY WE NEED AN IRB --
>>OKAY.
>> OFTENTIMES WE SAY THIS IS
SEEN BY THE IRB, DETERMINED TO
BE QUALITY IMPROVEMENT, NOT
UNDER THE REGULATIONS.
PEOPLE TO BE AWARE, IT'S A GRAY
AREA, PRAGMATIC TRIALS MAKE IT
EVEN MORE GRAY.
>> GREAT.
THANKS.
CORITA?
>> I WAS GOING SAY, WELL, I WAS
GOING TO ARGUE I MEAN MY TRIAL
WAS QUALITY IMPROVEMENT AND THE
NIH FUNDED IT SO MAYBE I'M JUST
AN EXCEPTION, BUT THERE IS AN
IMPLEMENTATION STUDY SECTION,
AND PEOPLE WERE VERY CONFUSED SO
OUR EVALUATION WAS NOT QUALITY
IMPROVEMENT.
SO OUR EVALUATION WAS RESEARCH
BUT THE INTERVENTION ITSELF WAS
NOT -- IT WAS QUALITY
IMPROVEMENT, IT WAS IMPROVING
THINGS AND DOING THINGS THAT ARE
ALREADY IN EXISTENCE BUT
BUNDLING THEM TOGETHER AND SO
OUR OUTCOMES WERE BY CMS DATA
ONLY AT NYU WENT THROUGH THE IRB
THERE BUT DID NOT GO THROUGH THE
IRB AT ANY OF OUR 33 OTHER
A.D.s, SO WE HAD A SUBCONTRACT
DO PAY THEM TO DO QUALITY
IMPROVEMENT BUT THERE WAS NO
DATA EXCHANGED.
PEOPLE WERE COMPLETELY CONFUSED,
AGREE.
BUT THERE WAS NO RESEARCH
HAPPENING AT THE SITES.
IT WAS QUALITY IMPROVEMENT.
SO, IT WAS CHALLENGING BUT WE
DID EVENTUALLY EXPLAIN IT TO
EVERYBODY.
AND GOT AROUND, YOU KNOW, A LOT
OF THAT.
BUT --
>> I THINK IT'S A GREAT TOPIC
FOR MORE GUIDANCE AND
CLARIFICATION.
>> ABSOLUTELY.
>> GIVEN THE TYPE OF RESEARCH.
IT IS DIFFICULT.
DOING THESE QUALITY IMPROVEMENT
OR STUDIES, IT'S SO IMPORTANT.
AND WHEN YOU CAN MAKE
GENERALIZABLE KNOWLEDGE, WE
SHOULD BE.
ANYWAY, ENOUGH ON THAT.
>> YEAH, THERE'S NO GRAY SPACES
IN THE PCT WORLD.
OTHER COMMENTS OR QUESTIONS FOR
ONE ANOTHER?
>> I WAS -- THIS IS NATALIA.
AS I WAS LISTENING TO ANDREA, A
COUPLE THINGS CAME TO MIND.
ONE IS AS P.I. RUNNING A TRIAL
THERE NEEDS TO BE CONSTANT
MONITORING OF ALL ASPECTS, THAT
REMINDED ME LISTENING TO ANDREA,
ALSO REMINDED ME OF HOW HELPFUL
IS IT TO HAVE THE COLLABORATORY,
SO THE COLLABORATORY CORE AND,
YOU KNOW, THE BIOSTATISTICS CORE
THAT THEY HAVE IN PLACE ALLOWS
YOU TO THINK BROADLY ABOUT THE
TOPIC THAT WE'RE JUST TALKING
ABOUT AND HOW MIGHT WE MONITOR
THROUGHOUT THE TRIAL AND HOW
MIGHT WE ACTUALLY DO THAT
BECAUSE EVERY TRIAL WILL BE
SLIGHTLY DIFFERENT, JUST WANTED
TO SAY I APPRECIATE THAT.
ONE, THE MONITORING HAPPENS,
THAT ANDREA WAS TALKING ABOUT I
MIGHT NOT HAVE THOUGHT ABOUT,
BUT HAVING A GROUP OF PEOPLE TO
COME TOGETHER TO THINK IS VERY
HELPFUL.
>> OTHER COMMENTS?
QUESTIONS?
BEFORE WE OPEN IT UP, I WAS
STRUCK AS WELL, YOU KNOW, THE
DIFFERENT THINGS THAT PEOPLE
HIGHLIGHTED THAT WILL MAKE IT
MORE LIKELY THAT YOU'LL BE ABLE
TO RECEIVE SIGNALS THAT
SOMETHING MIGHT BE HAPPENING AND
RESPOND TO THEM IN A WAY THAT
ALLOWS US A STUDY OR SOME KIND
OF STUDY TO KEEP GOING FORWARD,
ANDREA'S COMMENT ABOUT SELECTING
A DESIGN THAT PERMITS YOU TO
MONITOR AND RESPOND TO SIGNALS
WITHOUT REALLY COMPROMISING THE
ESTIMATES.
HAVING A MULTI-STAKEHOLDERS
PANEL IN PLACE TO BE IDENTIFYING
PLACES TO LOOK AND HELPING
REACT, THERE ARE OTHER THINGS AS
WELL.
WE'VE GOT A COMMENT FROM -- THAT
CAME IN.
WENDY, DO YOU WANT TO RELAY THAT
QUESTION?
>> SURE.
THIS MIGHT BE ADDRESSED TO
ANDREA, SEEMS TO BE A DESIGN
QUESTION.
IT MAY HAVE COME UP FOR SOME
OTHER INVESTIGATORS AS WELL.
AND THE QUESTION IS REALLY
CLUSTER RANDOMIZED TRIALS.
THE QUESTION IS AS MANY CLINICAL
PRACTICES ARE NOW HOUSED UNDER
LARGE GROUPS OR ORGANIZATIONS,
HAVE YOU ADVISED DEALING WITH
SEPARATE CLINICAL PRACTICES
WITHIN ONE ORGANIZATION, I THINK
TALKING MAYBE ABOUT
CONTAMINATION ISSUES POTENTIALLY
HERE, WHERE THE CONTROL
PRACTICES MAY LEARN ABOUT THE
INTERVENTION BEING OFFERED AT
SOME OTHER CLINICAL SITES.
I THINK WE'VE ALSO BEEN
CONCERNED WITH PROVIDERS THAT
MIGHT BE SEEING PATIENTS IN MORE
THAN ONE CLINIC IS SOMETHING
THAT HAS COME UP AS WELL, AND SO
ANY THOUGHTS OR SUGGESTIONS FOR
HOW TO HANDLE THAT?
>> WELL, HARD TO HANDLE,
EXACTLY.
FOR A LOT -- WHAT I'VE LEARNED
OVER THE YEARS, WE TYPICALLY
WORRY ABOUT THIS A LOT
BEFOREHAND AND THEN WE REALIZE
IT'S HARD, PROVIDERS ARE REALLY
BUSY, WHEN WE DOCUMENT POTENTIAL
FOR CONTAMINATION WE CUE THE
DESIGN AS A LEVEL YOU CAN,
THINKING OF MINIMAL
CONTAMINATION, WE WANT BALANCE,
YOU WANT STATISTICAL POWER.
SOMETIMES YOU PROMISE A STEP
WEDGE, WEIGHTLESS CONTROL, WE'LL
IMPLEMENT THIS A YEAR OUT OR
DEPENDING ON THE DESIGN, PROVIDE
MATERIAL FOR EVERYONE A YEAR
OUT.
WE'VE HAD TROUBLE WITH -- HERE
AT KAISER, WORKING ACROSS
CLINICS, POTENTIALLY A FORM OF
CONTAMINATION THERE.
HASN'T BEEN WHEN WE DOCUMENT,
HAS NOT BEEN AS BIG OF A DEAL
BECAUSE OF HOW BUSY PEOPLE ARE
TO IMPLEMENT THESE INTERVENTIONS
SO UNLESS THERE'S A LOT OF
TRAINING, IF YOU THINK IT'S
REALLY TO IMPLEMENT, WE HAVE TO
STEP BACK AND THINK HOW MUCH --
HOW BIG COULD CONTAMINATION BE.
I THINK NIH COLLABORATORY FROM
STATISTICIANS TALKING ABOUT
EFFECT BETWEEN GROUPS, AS LONG
AS THERE'S SOME SEPARATION OF
THE EFFECT YOU CAN STILL
ESTIMATE SOMETHING, DIFFERENCE
BETWEEN GROUPS, RIGHT?
AS LONG AS MOST OF IT IS NOT
CONTAMINATED, APPROACH
ESTIMATION YOU STILL WOULD HAVE
POWER FOR.
IT'S NOT A CLEAR ANSWER.
IT'S INTERVENTION DEPENDENT.
YOU HAVE TO DECIDE IF
CONTAMINATION IS A PROBLEM, A
FEW PEOPLE OVERALL CONTAMINATION
WILL NOT BE ADDRESS, HOW HARD TO
IMPLEMENT LIKE IF THEY NEED MORE
TRAINING, BUT IF YOU THINK IT'S
PRETTY EASY TO HAVE THE
CONTAMINATION I THINK STEP BACK
AND RCT OR EVEN (INDISCERNIBLE)
IF YOU HAD TO STEP BACK, IT'S
MORE OBSERVATIONAL, THE
INTERVENTION, AT DIFFERENT TIME
POINTS.
IT'S NOT A CLEAR ANSWER, SORRY.
I DON'T KNOW IF OTHERS WANT TO
SPEAK.
>> YEAH, SO IT'S NOT A BLINDED
STUDY.
HERE IS ONE AREA, IT'S NOT
BLINDED, CROSSOVER TO PREFERENCE
OR FOR WHATEVER REASON.
YOU CAN TRY TO FIX THIS
STATISTICALLY BY INCREASING
SAMPLE SIZE, CHOOSING THE
DESIGN, STEP WEDGE MAY NOT BE
THE BEST ONE.
THE OTHER WAYS, CHARGING PEOPLE
IN THEIR LANES, RIGHT?
IF YOU HAVE, SAY, NAVIGATOR WHO
IS EMBEDDING IN PRACTICE, YOU
TRY TO KICK THAT NAVIGATOR AWAY
FROM I GUESS THE PRACTITIONER SO
THAT DOESN'T CREEP OVER INTO
THEIR STANDARD OF CARE, AS BEST
YOU CAN.
YOU HAVE TO BE CREATIVE TO
COMPARTMENTALIZE AND KEEP PEOPLE
IN THEIR LANE AND NOT CROSSING
OVER.
>> GREAT.
THANKS, KEVIN AND ANDREA.
OUR COLLABORATORY COLLEAGUE
STEPHANIE MORONE HAD A QUESTION.
>> YEAH, THANKS TO THOSE.
ON THE WEST COAST, YOU REQUIRE
MORE COFFEE TO GET GOING.
REALLY INTERESTED, KEVIN, WOULD
LOVE YOUR THOUGHTS, MAYBE WENDY,
IF YOU HAVE REFLECTIONS AS WELL,
THE QUESTION ABOUT WHEN AND WHAT
CIRCUMSTANCES IT MIGHT BE
APPROPRIATE TO STOP A TRIAL
EARLY.
FUTILITY OR EFFECTIVENESS,
THINKING ABOUT HOW THE DECISION
SHOULD BE APPROPRIATELY MADE,
CASHING OUT RESPONSIBILITY TO
SPONSOR VERSUS DSMB.
IN PRIOR DSMB EXPERIENCE SEEING
SOME INTEREST, PERHAPS PASS THE
BUCK, HAVING SOMEONE ELSE MAKE
THE HARD CALL TO STOP EARLY.
AND IT STRIKES ME COULD BE EVEN
MORE CHALLENGING FOR DSMB
MEMBERS IN THE CONTEXT OF PCT,
PART OF THE MOTIVATION IS TO
CHANGE PRACTICE, THE EXTENT TO
WHICH THE WAY WE COMPRISE
DSMBs AND EXERCISE, WHETHER
INDIVIDUALS HAVE INSIGHT WHETHER
OR NOT THE LEVEL OF EVIDENCE
WILL BE HIGH ENOUGH TO MOTIVATE
A HEALTH SIS TOMORROW TO CHANGE
PRACTICE.
THERE'S A LOT IN THERE BUT ANY
REFLECTIONS?
>> IT'S VERY DIFFICULT.
IT'S A COMPLEX QUESTION.
IT'S WHY YOU HAVE A TEAM AND
IT'S A DSMB.
IT'S A GROUP DECISION TO MAKE
AND GO TO GO/NO-GO, STOP TRIAL,
MAKES IT EVEN MORE DIFFICULT.
CHANGING THE CULTURE, SAYING
IT'S OKAY TO FAIL, TO MESS UP,
ESPECIALLY WHEN YOU HAVE A
COMPLEX TRIAL, IT'S OKAY TO
REBOOT AND TO REENGINEER AND
RESTART.
AND I THINK THAT DOES TAKE SOME
OF THE HEAT OFF THE DSMB AND
INVESTIGATORS THAT, YOU KNOW,
IT'S NOT THE END OF YOUR TRIAL;
IT'S PART OF -- IT'S A COMMON
THEME THAT REENGINEERING IS
GOING TO HAPPEN.
THE OTHER THING IS IT'S NOT ONE
SIGNAL, ONE SAFETY SIGNAL.
WE LOOK FOR OTHER POINTS, TO
TRIANGULATE, MAYBE A CAUSAL
PATHWAY OR ROOT CAUSE THAT LINES
UP.
AND I USE THE WORD, THE BURDEN
OF PROOF IS CLEAR, CONVINCING,
IS WHAT THEY USE, THE LEGAL
TERM, YES OR NO, RIGHT?
IT'S NOT BEYOND REASONABLE
DOUBT.
IT'S NOT PREPONDERANCE OF
EVIDENCE, BUT I THINK CLEAR AND
CONVINCING IS PROBABLY THE WORD
I WOULD CHOOSE TO THE LEVEL,
IT'S A GROUP JUDGMENT CALL.
>> YEAH, THIS IS WENDY.
YOU KNOW, I THINK THESE ARE
REALLY CHALLENGING SITUATIONS.
AND AT OUR INSTITUTE OR CENTER
AT NCCIH WE SEE THE DSMB MAKING
A RECOMMENDATION TO THE DIRECTOR
OF INSTITUTE OR CENTER, BUT THEY
ARE SEEING THE DATA, THE MOST
CLOSELY, RIGHT?
THEY NEED TO ASK LOTS OF
QUESTIONS AND, YOU KNOW, I THINK
THAT IT IS MORE THAN JUST A
ONE-TIME SIGNAL, THEY ARE OFTEN
LOOKING FOR A PATTERN IN WHAT
THEY ARE LOOKING AT.
MANY OF YOU HAVE SERVED ON
DSMBs AND HAVE BEEN IN THE
SITUATION SO IF YOU WANT TO
SPEAK TO GENERALITYS OF HOW YOU
SORT OF CONSIDER THAT, THAT'S
HELPFUL.
WHAT WE DO AT NCCIH ANYWAY TRY
TO TAKE THE BURDEN OFF THE DSMB
AND THEY ARE MAKING A
RECOMMENDATION THE FINAL
DECISION DOES COME DOWN TO THE
INSTITUTE OR CENTER DIRECTOR
MAKING A DECISION ABOUT IT
BECAUSE SOMETIMES THERE ARE
OTHER OUTCOMES THAT WE'RE
INTERESTED IN, AND WE, YOU KNOW,
MAY ASK FURTHER FOLLOW-UP
QUESTIONS OF THE DSMB MEMBERS
AND WANT TO GET SOME MORE
INFORMATION.
YOU KNOW, I THINK ONE OF THE
THINGS THAT I'VE NOTICED THAT'S
REALLY DIFFERENT ABOUT PRAGMATIC
TRIALS IS WE -- IT'S MUCH LESS
COMMON FOR US TO SEE PLANNED
ANALYSES OR ASSESSMENT AND
THRESHOLDS AROUND UPTAKE OR
INTERVENTION FIDELITY OR THINGS
LIKE THAT.
SO IT'S A QUESTION WE'RE
STARTING TO ASK MORE ABOUT
SHOULD THE DSMB BE LOOKING AT
THIS BECAUSE WE'VE SEEN SEVERAL
TRIALS GET TO THE VERY END AND
REALIZED THERE PROBABLY WAS A
SIGNAL EARLIER ON IN THE PROCESS
THAT WE COULD HAVE CAUGHT.
AND SHOULD THAT HAVE BEEN
SOMETHING THAT WE CAUGHT AND
MAYBE HAD A DISCUSSION AT THAT
TIME POINT?
I DON'T KNOW IF OTHERS HAVE
THOUGHTS OR IDEAS, I'M
DEFINITELY OPEN TO OTHERS'
IDEAS.
>> TISHA WILEY, DO YOU WANT TO
MAKE A COMMENT?
>> I WAS GOING TO ADD OUR
APPROACH AT NIDA, THE COMPLEXITY
DEPENDING ON THE INITIATIVE
WHERE THE INSTITUTE IS MANAGING
IN HOUSE OR DEFERRING TO DSMB
THE INVESTIGATOR IS MANAGING,
AND ACTUALLY THERE'S A LOT OF
ISSUES ABOUT HOW THE INFORMATION
GETS REALLY BACK TO PROGRAM,
WHEN IT'S A DSMB YOU'RE MANAGING
AS INVESTIGATORS BECAUSE OUR
SYSTEMS AREN'T SET UP TO REMIND
TO YOU REGULARLY GIVE US THAT
INFORMATION.
IN TERMS OF DSMB WE MANAGE IN
HOUSE BECAUSE WE'VE GOT REASON
WHY WE REVIEW A PARTICULAR STUDY
OR GROUP OF STUDIES AS BEING
HIGHER IN RISK OR CLOSENESS OF
US MANAGING, EVEN THEN WE STILL
VIEW THE DSMB RECOMMENDATION AS
ONE THAT IS BEING MADE TO THE
INSTITUTE DIRECTOR AND THAT
EVERY TIME THE DSMB MEETS
REREALLY UP THE DSMB'S
RECOMMENDATIONS TO OUR INSTITUTE
DIRECTOR TO MAKE SURE THEY ARE
AWARE.
THE TENSIONS, WHETHER THE DSMB
VIEWS THEMSELVES MAKING A
DECISION, THE WAY WE OPERATE, WE
VIEW THEM AS MAKING A
RECOMMENDATION TO THE INSTITUTE
DIRECTOR WHO THEN ULTIMATELY
MAKES THE FINAL DECISION.
>> THAT'S HOW WE OPERATE HERE
AT NIDDK AS WELL.
>> STEPHANIE, ANY FOLLOW-UP
QUESTIONS FROM THAT?
>>REFLECTIONS ARE HELPFUL.
IT DOES MAKE ME WONDER FURTHER
ABOUT WHO IS SITTING AROUND THAT
TABLE FOR DSMB, WE MADE
RECOMMENDATIONS FOR PRAGMATIC
CLINICAL TRIALS, WANT TO MAKE
SURE WE HAVE INDIVIDUALS
FAMILIAR WITH THE
CONSIDERATIONS.
WE TALK ABOUT POTENTIALLY
PATIENTS HAVING MORE IMPORTANT
VOICE, I DON'T KNOW THE EXTENT.
I WONDER IF HEALTH SYSTEM
LEADERS OR OTHERS MIGHT BE
ANOTHER PERSPECTIVE, IF THAT'S
THE GROUP.
ARGUABLY ONE OF THE KEY GROUPS
WE'RE TRYING TO INFLUENCE WITH
THESE TRIALS BUT I HAVE
DISCOMFORT, MY OWN EXPERIENCE
WITH DSMB, INCLUDE TO
OVEREMPHASIZE WITH THE
INVESTIGATORS, IF THE TRIAL GETS
STOPPED IT MAY HAVE
IMPLICATIONS, RATHER THAN THE
PATIENTS FOR WHOM WE MIGHT BE
CONTINUING TO BURDEN EVEN IF THE
RISK IS LOW AND CERTAINLY
CHALLENGES WITH THAT STRUCTURE,
SOMEONE WHO SIMILARLY IS
DEPENDENT ON FUNDING STRUCTURE,
BUT MAYBE ONE OF THE
REFLECTIONS, KEVIN CHAN,
APPRECIATING YOUR REFLECTIONS,
THAT IT SHOULD BE AN EXPECTATION
THAT SOME WILL REQUIRE TO FAIL
AND PIVOT AND HAVE A DIFFERENT
OFF-RAMP, MORE COMFORT MAKING
THE CALL.
>> PEARL CAN COMMENT MORE.
LOOKING AT FROM ETHICAL LENS,
STATISTICAL LENS, OPERATIONAL
LENS, RIGHT?
YOU CAN LOOK AT PATIENTS,
MANAGING POPULATIONS, NOT
INDIVIDUALS, AFFECTS
INDIVIDUALS, THE AIM.
BUT ETHICISTS HELP US FRAME
THINGS, ETHICALLY IS TO PUT
PATIENTS AND PROVIDERS THROUGH
THIS ACTIVITY, GIVEN THE CHANCES
OF SUCCESS OR FAILURE OF THIS
TRIAL.
>> I THINK THAT'S A GOOD POINT.
THE DSMB JOB IS A LOT EASIER
WHEN IT'S A SAFETY ISSUE.
I MEAN, THAT'S EASIER TO GET
YOUR ARMS AROUND.
WHEN IT'S MORE FUTILITY ISSUE,
YOU KNOW, WHAT'S THE RISK OF THE
RESEARCH, WHAT ARE YOU EXPECTING
PARTICIPANTS TO DO?
AND THEN YOU GET IN THERE MIGHT
BE SECONDARY AIMS, THAT'S A MUCH
MUDDIER WATER.
BUT I THINK PURELY -- I HANG OUT
WITH BIOETHICISTS, I'M NOT A
BIOETHICISTS.
IF THE RESEARCH IS NOT GOING
FORWARD IN A MEANINGFUL WAY THEN
ANY RISK IS NOT WORTH PUTTING
SOMEONE THROUGH BUT IT'S MAKING
THAT DECISION OF IT'S NOT GOING
TO GO FORWARD IN ANY WAY IS THE
DIFFICULT PART.
>> PEARL, YOU'RE GIVING US
PERMISSION TO SAY THAT.
SOMETIMES WE NEED SOMEONE TO BE
ABLE TO DO THAT.
YOU'RE IN THE ROOM.
>> YEAH.
>> BECAUSE IT'S NOT EASY TO BE
THE FIRST ONE UP AND SAY NO.
>> WHY ARE WE DOING THIS,
EXACTLY.
>> THIS IS REMINDING ME OF
SOMETHING THAT CHERYL SAID IN
HER PRESENTATION.
CHERYL, YOU MADE A NOTE THE
DECISION TO STOP OR MASSIVELY
CHANGE A STUDY IS A PROFOUND ONE
THAT SHOULD INCLUDE PATIENTS IN
THE COMMUNITY.
DO YOU WANT TO SAY A LITTLE BIT
MORE ABOUT KIND OF WHAT YOUR
THINKING IS AS REGARDS THIS
CONVERSATION, WHAT TYPE OF INPUT
MIGHT BE NEEDED AND WHY IT'S
IMPORTANT?
I THINK YOU'RE MUTED, CHERYL.
OR YOU'VE GOT SOME AUDIO ISSUE
THERE.
>> THERE YOU GO.
CAN YOU HEAR ME?
>>YEP, YEP.
>> SORRY ABOUT THAT.
WELL, I THINK NUMBER ONE, WE DO
HAVE SOME THAT ARE INVOLVED IN
TERMS OF IRBs, BUT IT'S
THINKING ABOUT IS IT REALLY
NECESSARY TO HAVE AN ACTUAL
DELIBERATE COMMUNITY REVIEW OR
PATIENT REVIEW WHEN WE'RE AT
THAT POINT BECAUSE OF THE
IMPLICATIONS BECAUSE YOU ARE
REALLY MAYBE DEIMPLEMENTING
SOMETHING THAT HAS AN IMPACT OR
BE LEAVING A VOID IN TERMS OF
SOME PART OF PRACTICE OR WHAT
THAT MIGHT MEAN IN TERMS OF THE
COMMUNITY.
AND I THINK THAT WE CAN
STRENGTHEN THAT, BESIDES OUR
REGULAR REPRESENTATIVES FROM IRB
AND THINK ABOUT WHETHER OR NOT
SOMETHING AS IMPACTFUL AS THAT
REQUIRES ADDITIONAL BODY IN
TERMS OF COMMUNITY REVIEW BOARD
OR PATIENT REVIEW BOARD WHEN
WE'RE GOING TO DO SUCH AN
ACTION, BE AN ADDITION TO WHAT
WE TYPICALLY DO.
AND WE'RE REALLY DOING, WHEN
WE'RE STOPPING A STUDY WE'RE
DEIMPLEMENTING.
DEIMPLEMENTATION IS SOMETHING
THAT WE REALLY STILL DON'T TALK
A LOT ABOUT AND KNOW A LOT
ABOUT, AND IT IS A DIFFERENT
ACTION THAT HAS -- CAN HAVE
CONSEQUENCES IN TERMS OF ETHICS,
AND AS WELL AS OUR RELATIONSHIP
WITH PATIENTS AND COMMUNITY, IF
WE'RE GOING TO CONTINUE TO DO
PRAGMATIC TRIALS.
>> THANK YOU, CHERYL.
IS THERE ANYTHING THAT WE
HAVEN'T ADDRESSED YET ABOUT WAYS
TO INCREASE OPPORTUNITIES FOR
EARLY LEARNING, FOR GETTING
EARLY SIGNALS, AND HOW BEST TO
PROCESS THOSE EARLY SIGNALS TO
INFORM DECISIONS ABOUT WHAT
HAPPENS NEXT?
ARE THERE ANY OTHER
CONSIDERATIONS THAT WE HAVEN'T
RAISED YET?
>> I DID MENTION BEFORE THE
ROLE OF QUALITATIVE RESEARCH.
AND THAT'S IMPORTANT IN TERMS OF
CONTEXT AND IN UNDERSTANDING
SOME OF THE DATA BECAUSE
ADDITIONAL MULTI-MODES OF DATA
INCLUDING QUALITATIVE RESERVE
ALONG THE PROCESS AND ANALYZING
THAT AS WELL AS PART OF AN EARLY
DECISION REALLY INFORMS CHOICES.
AND REALLY MAY GIVE SOME BETTER
UNDERSTANDING, AND THAT'S NOT
ALWAYS A COMPONENT OF HOW WE
PROCEED.
IT MAY BE THAT WE HAVE IT AT THE
BEGINNING WHEN WE'RE DESIGNING,
BUT TO CONTINUE TO INVOLVE
QUALITATIVE METHODS AND HOW WE
LOOK AT DATA I THINK IS
IMPORTANT.
>> THAT'S A GREAT POINT.
I THINK NATALIA MADE A SIMILAR
POINT WHEN THEY WERE CONSIDERING
HOW TO ADAPT THE INTERVENTION
AFTER COVID HIT, A GOOD
REMINDER, IT'S A TOOL IT COULD
BE USED THROUGHOUT TO PROVIDE
VALUABLE INFORMATION.
>> CAN THE BIOSTATISTICIANS
PROVIDE SOME DEGREE OF
CONFIDENCE BEHIND?
WHEN DECISIONS ARE MADE, THERE'S
USUALLY NOT -- P-VALUE IS NOT
THERE, RIGHT?
IT'S A LOT OF THESE NUMBERS ARE
MOVING AND LINING UP, RIGHT?
IS THERE A WAY YOU CAN SAY
THERE'S A 95% OR 80% CHANCE IF
WE KEEP DOING THIS, THIS IS
WHAT'S GOING TO HAPPEN?
>> CONDITIONAL POWER, FUTILITY
BOUNDARY FORMAL, DEPENDS ON WHAT
YOU WANT TO ESTIMATE.
IF YOU WANT TO ESTIMATE BASED ON
PRIMARY OUTCOME, IT'S BEEN
AROUND FOR A LONG TIME FOR
REGULAR RCTs.
I THINK IF YOU'RE MORE CONCERNED
ABOUT HETEROGENEITY AND
VARIABILITY, THAT'S WHERE YOU
HAVE TROUBLE.
ESTIMATING THE THE SAME EFFECTE
NEXT YEAR WITH INTERVENTION
ADAPTS, IT'S EASIER WORKING WITH
A FEW DRUGS, IT'S NOT GOING TO
CHANGE, MAYBE THE POPULATION
YOU'RE RANDOMIZING TO CHANGES
BUT NOW YOU HAVE INTERVENTION
THAT'S DYNAMIC THAT MAY BE
IMPROVING OVER TIME, EARLY
EFFECT AT INTERVENTION MAY NOT
BE THE EFFECT AFTER YOU ADAPT IT
FOUR MONTHS FROM NOW.
YOUR ESTIMATE OF INTEREST
CHANGES, BUT YOU DO HAVE
CONDITIONAL POWER, FUTILITY
BOUNDARIES, SAFETY BOUNDARIES
FOR CLINICAL TRIALS, WORRIED
ABOUT SAFETY RISK.
THERE IS STATISTICS, IT'S THAT
ESTIMATE, WHAT YOU THINK WILL
CHANGE OVER TIME SO TO GET US TO
BE ABLE TO GET TO THAT ESTIMATE
THAT YOU NEED, TO SAY
EFFECTIVENESS.
FUTILITY YOU DO ON
IMPLEMENTATION OUTCOMES,
DEFINITELY.
IF YOU THINK THERE'S A PATHWAY
FOR YOUR STUDY FROM
IMPLEMENTATION TO GET THAT
EFFECTIVENESS OUTCOME AND YOU'RE
NOT SEEING UPTAKE OR SEEING
ASPECTS THOSE ARE WHERE WE CAN
PUT BOUND REIS -- BOUNDARIES.
WHICH ONE YOU WANT TO ESTIMATE,
IT'S HARD.
>> SO I WOULD SAY WITH
DIFFERENTIAL WITHDRAWAL IS ONE
GROUP, OR TABLE 1, DIFFERENCE OF
AGE OF, SAY, FIVE YEARS BETWEEN
THE TWO GROUPS, SO THAT'S A
LARGE DIFFERENCE, RIGHT?
SUGGESTING NONRANDOM BIAS.
WHAT'S IT GOING TO TAKE TO
BALANCE THINGS OUT, IS THERE A
WAY TO SAY THERE'S A 95% CHANCE
THE TRIAL WILL END UP IN BALANCE
AND BIAS?
>> WITHDRAWAL, NOT JUST
INTERVENTION WITHDRAWAL.
>> YES, INTERVENTION IS
BURDENSOME SO PEOPLE WITHDRAW AT
INTERVENTION SITE, OR TABLE 1,
THERE'S PREFERENCE IN ENROLLING
PATIENTS, DIFFERENCE IN TABLE 1,
THAT RANDOMIZATION OF CLUSTERS.
>> I THINK PULLING WORK FROM
ALTERATION TRIALS, SELECTION
BIAS, MISSING DATA OUTCOMES,
QUANTIFY FROM OBSERVATIONAL
TRIALS.
THAT'S WHAT I THINK OF THE PCT,
IN BETWEEN A CONTROLLED
RANDOMIZED TRIAL, IMPROVING WHAT
YOU CAN FROM OBSERVATIONAL
TRIAL, SOME ESTIMATES OF THE
BIAS, WITH PEOPLE BEING
COLLECTED IN POPULATION, TO LIKE
MISSING DATA BIAS, WORKING WITH
OUTCOME DATA.
ADHERENCE ALSO, IF YOU ESTIMATE
BASED ON ADHERENCE PATTERNS,
THERE'S METHODS TO LOOK AT -- I
CAN'T REMEMBER THE STATISTIC TO
ESTIMATE CONFOUNDING POTENTIAL.
IT DOES HELP WITH DATA, WHAT
PEOPLE HAVE, IF YOU THINK IT'S
GOING TO CONTINUE THROUGHOUT THE
STUDY.
YOU COULD PUT A BAN ON WHAT YOU
CAN ESTIMATE, THAT'S WHERE IT
GETS TRICKY TO MAKE THE
DECISIONS.
>> THANKS, ANDREA.
OTHER COMMENTS OR OBSERVATIONS?
>> THIS IS PEARL.
KEVIN, I HOPE I'M ARTICULATE
ENOUGH TO GET THIS ACROSS.
ALL TRY.
I'M TRYING TO PUT TOGETHER,
YESTERDAY AND TODAY, THE MEETING
YET THERE WAS A LOT OF
DISCUSSION ABOUT IMPLEMENTATION
YESTERDAY, YOU SHOULD YOU ONLY
DO RESEARCH CAN YOU SHOW IT WILL
BE IMPLEMENTED, THIS IS
IMPLEMENTABLE.
SOMEONE SAID ACTUALLY THEY ARE
REQUIRING THERE BE A COMMITMENT
TO IMPLEMENTATION AND HENCE
LEADERSHIP WEIGHING IN EARLY.
I'M WONDERING HOW THAT AFFECTS
GO/NO-GO DECISIONS OR CHANGES?
IF YOU HAVE TO -- THINKING OF
LET'S SAY WITH COVID, WHERE I
THINK IT WAS NATALIA'S STUDY,
THERE WAS GOING TO BE IN-PERSON
AND HAD TO GO TO REMOTE BECAUSE
OF COVID.
WHAT IS THE INSTITUTION SAYS WE
CAN'T SUPPORT REMOTE?
HOW WOULD THAT KIND OF NO-GO
DECISION HAPPEN?
WE WERE TALKING REALLY MORE
ABOUT SAFETY AND STATISTICAL
FINDINGS, BUT I THINK WE NEED TO
THINK ABOUT IS THIS RESEARCH
GOING TO REALLY ADD TO HELPING
PEOPLE, HENCE BE IMPLEMENTABLE,
HOW DO WE HANDLE NEW POLICIES,
NEW ISSUES THAT CAME IN?
I HOPE THAT MADE SENSE?
>> YEP, I THINK YOU'RE LOOKING
FORWARD, ANTICIPATING.
I THINK DECISIONS NEED TO BE
MADE BACKWARDS IN TIME DATA, ROO
IT?
IF IT'S A COVID ISSUE, TEAMS TRY
TO PIVOT AND CHANGE THE
INTERVENTION, AS THAT HAPPENS
DATA WILL LEAVE A TRACK TO SAY
WHETHER THIS IS WORKING OR NOT
AND HAVE CONTEXT, IT'S NOT JUST
ONE NUMBER, IT'S A NUMBER IN THE
CONTEXT OF THE SITUATION OF MANY
NUMBERS, RIGHT?
DOES THAT MAKE THE CASE THAT THE
LIKELIHOOD THIS IS NOT GOING TO
WORK OUT?
>> RIGHT.
>> UNLESS IT'S AN ISSUE OF
ETHICS, SAFETY, IN THE PIVOT OR
WHATEVER CHANGE THEY MAKE BUT
THAT'S A DIFFERENT ISSUE, THAT
WILL GO THROUGH THE IRB.
>> YEAH, NO, I PROMISE I WILL
NOT BEAT THIS DEAD HORSE, BUT I
WAS MORE THINKING IF YOUR PIVOT
ACTUALLY REQUIRED DIFFERENT
RESOURCES, ET CETERA, THAT THE
INSTITUTION HAD NOT AGREED TO AT
THE BEGINNING, THAT THEY WOULD
BE ABLE TO IMPLEMENT.
IT SEEMS LIKE SOMETHING WE DON'T
CONSIDER.
I JUST -- TODAY AND YESTERDAY,
I'M JUST THINKING WE NEED TO PUT
A LITTLE MORE ENERGY INTO
LOOKING AT THE SO WHAT QUESTION
ABOUT THE RESEARCH, IS THIS
GOING TO SOMETHING THAT CAN BE
IMPLEMENTED?
I DON'T THINK WE SPENT ENOUGH
TIME THERE, BOTH INITIALLY AND
ONGOING IN THE STUDY.
I'LL STOP THERE.
>> I AGREE.
>> THANK YOU, SHERYL.
>> I THINK IT'S IMPORTANT WE
HAVE MULTIPLE STAKEHOLDERS IN
TERMS OF INPUT BECAUSE AT THAT
TIME WE MAY BE ABLE TO HAVE A
BETTER DESIGN WHICH WOULD LEAD
TO A BETTER DESIGN THAT IS GOING
TO BE SUSTAINABLE.
>> CORITA, WHAT DO YOU WANT TO
ADD?
>> COULD NOT AGREE MORE WITH
PEARL, YOUR STATEMENT, AND I'LL
JUST SAY ANECDOTALLY PCORI DOES
THIS REALLY WELL.
THEY HAVE GO NO GO CRITERIA, IF
YOU CANNOT IMPLEMENT YOUR TRIAL,
COMPARATIVE EFFECT, WHATEVER
YOU'RE DOING, YOU HAVE TO BE
DESCOPE THE STUDY, THAT HAPPENED
TWO OUT OF THE TEN PALLIATIVE
CARE STUDIES, PHASE AWARDS ARE A
GREAT WAY TO DO THAT.
>> CAN YOU EXPLAIN DESCOPING
FOR FOX NOT FAMILY?
>> YOU TO HOWE CAN YOU REVISE
AMIES -- HOW CAN YOU REVISE THE
AIMS.
THERE'S AN EXAMPLE FROM KAISER,
IN SOUTHERN CALIFORNIA A THEY
WERE GOING TO DO A STUDY OF
TELEHEALTH PALLIATIVE CARE IN
THE HOME, I CAN'T REMEMBER THE
CONTROL OR INTERVENTION BUT
KAISER DECIDED THEY WERE JUST
GOING TO DO IT IN EVERYONE
WITHOUT EVIDENCE BUT DECIDED TO
IMPLEMENT FOR EVERYONE SO THERE
WAS NO LONGER ANY QUESTION THEY
COULD ASK.
THAT WAS -- SOUTHERN CALIFORNIA
SEEMS PARTICULARLY PROBLEMATIC.
I DON'T KNOW WHY THE OTHER STUDY
WAS ALSO SOUTHERN CALIFORNIA.
AND SO DESCOPING IS LIKE
REVISING THE AIMS, REVISING THE
BUDGET, AND SAYING HOW CAN I GET
THE MOST OUT OF WHAT HAPPENED TO
CONVEY LEARNINGS TO THE
SCIENTIFIC AND BROADER
COMMUNITY.
>> OKAY.
>> IT'S LIKE REWRITING YOUR
GRANT.
SOUNDS REALLY PAINFUL.
I'VE NOT HAD TO DO IT.
>> SOUNDS MORE LIKE RESCOPING.
DESCOPEING SOUNDS LIKE YOU'RE
REMOVING ANY POINT TO THE WHOLE
THING.
>> WELL, THE SCOPE IS MUCH
SMALLER BECAUSE THESE WERE
MULTIPLE MILLION DOLLAR AWARDS,
LIKE $10 MILLION AWARD.
DESCOPE MEANING THE SCOPE IS NOW
A MILLION DOLLARS PROBABLY
BECAUSE YOU'RE JUST GOING
QUALITATIVE INTERVIEWS AND
UNDERSTANDING WHY IT DIDN'T WORK
AND, YEAH, THAT'S WHAT I MEANT
BY DESCOPE.
>> GREAT, THANK YOU.
WE'RE APPROACHING END OF OUR
SESSION HERE.
WE HEARD SO MANY GOOD POINTS.
WE STARTED OUT WITH OUR THREE
COLLABORATORY P.I.s,
DISCUSSING THIS QUESTION OF WHAT
OPPORTUNITIES AND CHALLENGES ARE
THEREFORE LEARNING SOONER, AND
WE'RE EMPHASIZING LEARNING
SOONER, NOT FAILING FASTER,
WHICH WAS AN INITIAL PHRASE WE
USED WHEN PLANNING THIS
WORKSHOP, BUT LEARNING SOONER.
AND HEARD SOME EXAMPLES OF
EXPLICITLY BUILDING IN THAT
LEARNING AS PART OF THE FORM OF
THE INTERVENTION.
AND HOW TO REACT TO THAT AND THE
ROLE OF MULTI-STAKEHOLDER
PANELS, AND THAT WE ALSO HEARD
ABOUT CHOOSING DESIGNS THAT WILL
ALLOW US MAXIMUM FLEXIBILITY IN
RESPONDING TO INCOMING SIGNALS.
WE HEARD ABOUT THE POTENTIAL FOR
EARLIER AND MORE LEARNING, WITH
A BIT MORE TIME AVAILABLE IN THE
PLANNING PHASE, ESPECIALLY IN
CASES WHERE WE'RE GOING TO BE
DOING THE RESEARCH IN SETTINGS
THAT ARE LESS EXPERIENCED WITH
RESEARCH OR UNDERRESOURCED FOR
WHICH MORE TIME THAN IS
TYPICALLY THE CASE WILL BE
REQUIRED TO FIND OUT WHAT THINGS
WE NEED TO LEARN IN THOSE
SETTINGS.
SO I THOUGHT THAT WAS A REALLY
INTERESTING POINT.
AND THEN WE HEARD A WHOLE HOST
OF CONSIDERATIONS, LOGISTIC,
ADMINISTRATIVE, BIOETHICAL,
STATISTICAL, IN EXAMINING THE
RANGE OF SIGNALS THAT MIGHT BE
COMING IN AND MAKING DECISIONS
BASED ON WHAT HAPPENS NEXT AND
KEVIN TALKED ABOUT CHANGING THIS
CULTURE TO MAKE IT MORE
ACCEPTABLE TO SAY, HEY,
SOMETHING'S HAPPENED, WE NEED TO
PIVOT IN SOME WAY.
I THINK ONE OF THE THINGS THAT
OUR WORKSHOP PLANNING GROUP
RECOGNIZED IS THAT CHANGING
CULTURE MIGHT ALSO NECESSITATE A
CHANGE IN THE STRUCTURES AND
PROCESSES AND INCENTIVES FOR
RESEARCH IN WHICH ALL OF OUR
RESEARCHERS ARE PRACTICING.
AND SO THAT'S THE ISSUE THAT
WE'LL ADDRESS IN OUR FOURTH
PANEL.
MY COLLEAGUE RICH PLATT WILL
LEAD THAT AFTER THE BREAK.
WE'RE GOING TO TAKE A BREAK
RIGHT NOW AND COME BACK AT
12:25.
I WANT TO THANK ALL THE
PANELISTS FOR A REALLY, REALLY
INTERESTING DISCUSSION AND FOR
SHARING ALL OF YOUR EXPERIENCES
AND PERSPECTIVES.
AND EVERYONE HAVE A GOOD BREAK,
WE'LL SEE YOU BACK AT 12:25.
THANKS SO MUCH.
I'M RICHARD PLATT, I'M GOING TO
MODERATE THIS LAST SESSION.
THIS IS UP TO A ONE-HOUR SESSION
TO BRING TOGETHER A NUMBER OF
THREADS THAT WE'VE BEEN
DISCUSSING, PARTICULAR TO
CONTINUE THE DISCUSSION WE HAD
IN THE FIRST PART OF TODAY'S
MEETING.
IN PARTICULAR, SO FIRST, BEFORE
I INTRODUCE THE PANELISTS, WE
MADE MUCH IN THE LAST SESSION
ONE NEEDS TO ADAPT THE
INTERVENTION BEING EVALUATED,
BUT THE DECISION BETWEEN
ADAPTING AND JUST CHANGING THE
INTERVENTION FALLS ON A
SPECTRUM.
AT LEAST IN THE CONTEXT OF THE
COLLABORATORY INVESTIGATIONS,
THOSE ARE IN THE CONTEXT OF
PROPOSALS REVIEWED AND APPROVED
BY A -- RECOMMENDED BY A STUDY
SECTION, AND THERE ARE REAL
QUESTIONS ABOUT HOW MUCH CHANGE
IS ACCEPTABLE IN THE CONTEXT OF
THE APPROVED PROTOCOL.
SO, TO SORT OF THINK ABOUT HOW
TO ACCOMMODATE THE LEGITIMATE
NEED FOR CHANGE WITH APPROVED
PROTOCOLS WE'RE CONSIDERING, WE
HAVE THREE NIH PROGRAM STAFF TO
LEAD OFF DISCUSSIONS, AND WE
HAVE A NUMBER OF INVESTIGATORS
WHO ARE GOING TO PARTICIPATE IN
THE CONVERSATION, FROM THE NIH
DAVID CHAMBERS WHO IS DEPUTY
DIRECTOR FOR IMPLEMENTATION
SCIENCE AND OFFICE OF DIRECTOR
OF CANCER CONTROL POPULATION
SCIENCES, DIVISION OF THE NCI IS
HERE.
WYNNE NORTON FROM THE SAME
DIVISION, AND TISHA WILEY, CHIEF
OF SERVICES RESEARCH BRANCH OF
NIGHT.
GEORGE SOPKO HAD AN UNAVOIDABLE
CONFLICT AND WON'T BE WITH US.
AND GLORIA, NATALIA, AND CORITA
WHO HAVE BEEN WITH US ALL
MORNING AGREED TO STAY ON AND
KEN SANDS AND SEPTIMUS, PART OF
THE ABATE DISCUSSION YESTERDAY.
WE'VE DAVID AND WYNNE TO SPEAK
AND INVITED REST TO COMMENT AND
WE'LL HAVE A FINAL MINUTES IF WE
WANT TO USE IT FOR THE WHOLE
GROUP AND EVERYONE IN THE
AUDIENCE TO PARTICIPATE.
WITH THAT AS A LEAD-OFF, DAVID,
THE FLOOR IS YOURS.
>> THANKS, RICH.
THANKS TO EVERYONE FOR ENGAGING
IN SUCH GREAT DISCUSSIONS OVER
THE LAST DAY-PLUS.
THERE'S BEEN A LOT OF
CONVERSATION ABOUT
IMPLEMENTATION SCIENCE,
IMPLEMENTATION QUESTIONS.
SO ONE OF THE THINGS WE THINK
ABOUT IN IMPLEMENTATION SCIENCE
IS HOW IMPORTANT CONTEXT IS AND
NO QUESTION, THERE'S BEEN A LOT
OF COMMENTS TO THIS END.
I LIKE TO START WITH JUST BRIEF
CONTEXT FOR THE COMMENTS THAT
I'M ABOUT TO MAKE.
IT STARTS WITH GRATITUDE FOR MY
FELLOW NIH FOLKS AS PART OF THIS
GROUP AND BEYOND.
CERTAINLY THE WONDERFUL WORK IN
THE COLLABORATORY, PARTICULARLY
WANT TO SHOUT OUT WYNNE LEADING
EFFORTS IN OUR DIVISION TO
ADVANCE OUR COLLECTIVE THINKING
ABOUT PRAGMATIC TRIALS ACROSS
CANCER, LOOKING FORWARD TO HER
COMMENTS AS WELL.
I HAVE BEEN ENGAGED IN THE
COLLABORATORY SINCE THE
BEGINNING AND REALLY RESONATE
WITH PEARL'S NOTE OF THE "SO
WHAT" QUESTION, IT'S SOMETHING
WITHIN THE PRAGMATIC TRIALS
ENTERPRISE COMES UP AGAIN AND
AGAIN, INCREDIBLY IMPORTANT.
I WANT TO QUICKLY NOTE SOME
COMMENTS GENERATED BY THE
WONDERFUL WORK OF FOLKS WITHIN
THE MENTAL HEALTH RESEARCH
NETWORK, WHICH HAS BEEN FUNDED
BY NIMH FOR A WHILE THAT I HAD
THE PRIVILEGE OF GETTING TO WORK
WITH IN EARLY DAYS.
OTHER COMMENTS RELEVANT TO OUR
ONGOING IMPLEMENTATION SCIENCE
CENTERS AND CANCER CONTROL, A
SET OF CANCER MOONSHOT-FUNDED
INVESTIGATORS THINKING ABOUT THE
NEXT STEPS IN THE SPACE OF
IMPLEMENTATION SCIENCE AND
CERTAINLY WHERE EFFECTIVENESS
ALSO TAKES ROOT.
AND THEN THE OTHER BIT OF
CONTEXT IS JUST HAVING THE
ABILITY TO LEARN FROM SO MANY
EFFECTIVENESS AN IMPLEMENTATION
TRIALISTS OVER THE YEARS IN
MENTAL HEALTH AND CANCER AND
BEYOND.
CORE TO ALL OF THESE PARTICULAR
ACTIVITIES IS SOMETHING THAT I
THINK IS CENTRAL TO A LOT OF
WORK OF COLLABORATORY, DIRECT
AND ONGOING DIRECTIONS BETWEEN
COMMUNITY LEADERS AND PRACTICE
PARTNERS.
I THINK IT'S BOTH IN TERMS OF
WHAT CAN WE ANTICIPATE AND THEN
WHAT -- AS A RESULT OF WHAT WE
CAN ANTICIPATE, HOW DO WE
DEVELOP STRATEGIES TO ADDRESS
CHALLENGES.
AS RICH WAS SAYING, WHEN WE --
WHEN THINGS COME UP WE'RE NOT
EXPECTING HOW CAN WE TAKE
ADVANTAGE OF SOME PARTNERSHIPS
TO TRY AND MANAGE, TO STUDY, TO
LEARN FROM THESE UNEXPECTED
EVENTS.
THERE'S SORT OF TWO BROAD
BUCKETS THAT I THINK THE MAIN
COMMENTS THAT I'D LIKE TO MAKE
FALL WITHIN.
THE FIRST ONE IS REALLY RELATED
TO THIS NOTION OF WHAT'S BASED
WITHIN OUR CLINICAL AND
COMMUNITY SYSTEMS AND THEN WHAT
ARE WAYS THAT WE CAN LEVERAGE
PARTNERSHIPS TOWARD ADDRESSING
THEM AND THE OTHER ONE IS
RELATED TO UNEXPECTED EVENTS
THAT CROPPED UP CERTAINLY OVER
THE LAST FEW YEARS BUT HAVE
ALWAYS BEEN WITH US, THE IDEA
THAT AS MUCH AS WE PLAN WE KNOW
THAT THE UNKNOWN IS LURKING AND
SOMETIMES THAT CAN BE A POSITIVE
THING, CAN REALLY HELP US LEARN,
OTHER TIMES IT BECOMES A HASSLE.
ON THE CLINICAL OR COMMUNITY
SYSTEM-BASED SIDE CERTAINLY SOME
STRATEGIES THAT WE THOUGHT ABOUT
AND CERTAINLY THAT OUR
INVESTIGATORS HAVE TAKEN
ADVANTAGE OF IS THINKING ABOUT
THE DIRECT INVOLVEMENT OF OUR
CLINICAL SYSTEM LEADS ON THE
INVESTIGATIVE TEAM NAMED AS
INVESTIGATORS, OFT-DIFFERENT
APPROACHES WHERE THERE'S BEEN A
REQUIREMENT THAT PRINCIPAL
INVESTIGATORS MAY BE PAIRED,
MULTIPLE P.I.s, WHERE P.I.s
REPRESENT LEADERSHIP OF THE
CLINICAL SITES.
THIS IS SOMETHING SAID
FREQUENTLY IN PREVIOUS PANEL,
ENSURING RESEARCH QUESTIONS,
MILESTONES WE'RE DEVELOPING
AROUND TRIALS ARE MEANINGFUL TO
PARTNERS AT MULTIPLE LEVELS OF
THE ORGANIZATION OR COMMUNITY
SETTINGS, AND THEN POTENTIALLY
RESILIENT TO SOME OF THESE
CHANGES, THE PRIOR PANEL TALKED
ABOUT CONCENTRATING ON
IMPLEMENTATION OUTCOMES IN
ADDITION OR PRIMARY AROUND
CLINICAL OUTCOMES AS WELL, THE
TYPICAL EFFECTIVENESS TRIAL
OUTCOMES WE CONCENTRATE ON.
IMPORTANT WHERE WE'VE SEEN THE
ABILITY, THIS CAME UP AS WELL,
CAPTURING CONTEXTUAL DATA AS
PART OF THE TRIAL.
SO WE'RE ABLE TO TEASE APART THE
IMPACT OF THE CONTEXT ON TRIAL
IMPLEMENTATION AS WELL AS ON THE
EFFECTIVENESS OF OUR
INTERVENTIONS.
I THINK THE CLINICAL AND
INVESTIGATOR PARTNERSHIPS ENABLE
US TO JOINTLY PLAN FOR
MITIGATION STRATEGIES.
OUR INVESTIGATORS HAVE THOUGHT
ABOUT OVERSAMPLING OF SITES,
ANTICIPATING TRANSITIONS IN
LEADERSHIP, HOW DO WE MAKE SURE
THE SUCCESS OF STUDY ISN'T BASED
ON RELATIONSHIP BETWEEN JUST A
COUPLE FOLKS WHERE ONE MAY BE
HEADED FOR DIFFERENT PASTURES,
OTHER THINGS IN THE
IMPLEMENTATION SPACE AND TRIALS,
THE IDEA OF A PRE-MORTEM, A
TECHNIQUE USED IN INDUSTRY AND
ELSEWHERE PLANNING FOR WHAT'S
GOING TO GO WRONG.
A GROUP OF FOLKS SIT AROUND THE
TABLE, IDEALLY PARTNERS, BETWEEN
INVESTIGATORS AND CLINICAL OR
COMMUNITY MEMBERS, TO TRY AND
TALK THROUGH ALL OF THE WAYS
THAT THIS PARTICULAR TRIAL IS
NOT GOING TO WORK OUT HOW WE
PLANNED.
AND HOW DO WE TRY TO ANTICIPATE
THOSE IN FLEXIBLE DESIGN OR WAYS
IN WHICH WE SEE ALTERNATE
STRATEGIES FOR MOVING FORWARD.
THOSE ARE ALL CLINICAL OR
COMMUNITY SYSTEM-BASED THINGS.
OF COURSE, IN THIS SORT OF
DOMAIN OF UNEXPECTED EVENTS,
FIRST AND FOREMOST, COULD GET
MENTIONED IN RESPONSE TO EVERY
QUESTION, IMPORTANCE OF ENGAGING
WITH OUR NIH PROGRAM STAFF OR
GRANTS MANAGEMENT FOLKS
THROUGHOUT THE LIFE OF THE
STUDY.
I THINK THEY ARE THERE, WE ARE
THERE TO HELP INVESTIGATORS AS
MUCH AS WE CAN.
OFTEN ANSWERS TO QUESTION CAN WE
DO THIS OR NOT, IT DEPENDS.
WHAT IT DEPENDS ON OF COURSE ARE
OPPORTUNITY FOR EXPERTS WITHIN
THE NIH, EXPERTS WITHIN THE
INVESTIGATIVE FIELD TO TALK
THROUGH POTENTIAL CHALLENGES AND
HOW CAN WE OVERCOME THEM.
THERE'S BEEN AT TIMES HESITATION
TO ENGAGE WITH NIH STAFF FOR
FEAR OF SOME THINGS UNTOWARD BUT
AT LEAST IN MY EXPERIENCE IT'S
BEEN CERTAINLY THE CASE THAT THE
MORE ENGAGEMENT THERE IS, THE
MORT AWARENESS THERE ARE OF
PROBLEMS AS THEY OCCUR, THE
EASIER IT IS TO MITIGATE SOME
UNEXPECTED CHANGES FROM OUR NIH
COLLEAGUES IN THE PRIOR SECTION.
WE'VE SEEN IN TERMS OF
UNEXPECTED EVENTS JUST THE IDEA
HOW FLEXIBLY WE CONSIDER DESIGN
AND INTERVENTIONS.
WE SAW TRIALS SHIFT TO
TELEMEDICINE, TECHNOLOGY-BASED
RECRUITING, WE DON'T KNOW WHAT
EVENTS MAY GET IN THE WAY OF OUR
TRIAL EFFORTS.
CERTAINLY MODIFICATIONS TO
TIMELINES, WE'VE SEEN IN THE
PAST WHERE THERE'S BEEN
REALLOCATION DEPENDING ON THE
KIND OF DESIGN, IF IT IS A
DYNAMIC WEIGHTLESS CONTROL
DESIGN USED, SOMETIMES THERE'S
BEEN REALLOCATION OF SITES TO
LATER COHORTS IF THEY ARE NOT
READY TO PARTICIPATE AT THE
PARTICULAR TIME THAT THEY WERE
FORECAST TO BE READY.
AND THEN I THINK ANOTHER THING
THAT FITS BOTH CATEGORIES THE
IDEA HOW WELL WE'RE IN
ANTICIPATE OF WHAT MIGHT HAPPEN
IN THE FUTURE CAPTURING
CONTEXTUAL DATA TO CAPTURE
IMPACT.
TO LAY THESE OUT AT DIFFERENT
STRATEGIES, I'VE SEEN COLLEAGUES
AT THE NIH USE TO BOTH
ANTICIPATE WHAT MIGHT HAPPEN AND
THEN COPE WITH IT AS IT IS
HAPPENING.
WITH THAT I'LL PAUSE AND LOOK
FORWARD TO THE CONVERSATION.
QUICK COMMENTS FROM ANY OF THE
OTHER PANELISTS?
OKAY.
THEN WYNNE, TAKE THE BATON.
>> I THINK THERE WAS A COMMENT.
>> I CAN JUST COMMENT QUICKLY,
RICH, I APPRECIATED COMMENTS
ABOUT PARTNERSHIP AND WE'VE HAD
A TEN-YEAR PARTNERSHIP WITH RICH
PLATT'S GROUP, THAT'S REALLY
HELPED UNDERSTAND PER PERSPECTS
THAT CAN BE DIFFERENT BUT WE
UNDERSTAND THEM, THAT ALLOWS FOR
SUCCESS.
I THINK THE OTHER ISSUE THAT
RESONATED IS THE ANTICIPATING OF
CHANGES AND JUST ONE THING THAT
COMES TO MIND, FOR EXAMPLE, IN
OUR STUDIES WHICH INVOLVES
MULTIPLE INSTITUTIONS IS YOU CAN
HAVE AN ENTIRE INSTITUTION DROP
OUT FOR UNFORESEEN REASONS LIKE,
YOU KNOW, THEY CHANGED THEIR
ELECTRONIC EMR OR THEY ARE SOLD
TO ANOTHER ORGANIZATION, AND
THOSE ARE INEVITABLE AND THE
KINDS OF THINGS WE'VE LEARNED TO
ANTICIPATE AS PART OF THE
INITIAL DESIGNS.
>> OKAY.
ED, DO YOU WANT TO COMMENT?
>> YEAH, TO PIGGYBACK, ANOTHER
THING WE'VE DONE IN OUR DESIGNS
IS MADE SURE WE HAD THE RIGHT
STAKEHOLDERS AROUND THE TABLE.
SO IT'S IMPORTANT BECAUSE THEY
ARE THE ONES THAT WILL BE
EXECUTING THE INTERVENTION.
SO FOR INSTANCE FOR A BABY WE
WANT TO MAKE SURE WE HAD NURSING
AROUND THE TABLE, THE ONES DOING
THE BATHING, WE WANT PHARMACY
AROUND THE TABLE TO TRACK, WE
WANT OUR BETA SCIENTISTS AROUND
THE TABLE BECAUSE THEY DO
REPORTS FOR US.
WE TRY TO TRACK ADHERENCE,
THAT'S OUR GUIDE, HOW WELL ARE
WE DOING CHT ABATING FOR ABATE
TRIAL, TRACK THAT, IF WE SEE
SYSTEMS OR ORGANIZATIONS OR
HOSPITALS ARE LAGGING, WE TRY TO
ADDRESS THAT ON THE FLY.
>> NATALIA?
>> HI.
YEAH, I JUST WANTED TO MAKE ONE
COMMENT ON THE OUTCOMES
MEANINGFUL TO MULTIPLE PARTNERS.
I THOUGHT THAT WAS INTERESTING
THAT SHOWED UP FOR THE OPTIMUM
STUDY, WE ADOPTED THE PAY AS A
RECOMMENDATION BY THE STUDY
SECTION FOR MAIN OUTCOME
MEASURE, PAY IS
MULTI-DIMENSIONAL ASSESSMENT OF
PAIN, WHAT I LIKE, IT DOES HAVE
WIDESPREAD ADOPTION ACROSS
MULTIPLE PARTNERS, CDC
RECOMMENDS IT, IT'S RECOMMENDED
IN PRIMARY CARE, IT'S IN THE
ELECTRONIC HEALTH RECORDS, IT'S
EASY FOR PARTICIPANTS IN A STUDY
TO ANSWER IT, SO I THINK -- I
WANTED TO UNDERSTORE THAT WHEN
THE OUTCOME IS MEANINGFUL TO
MULTIPLE PARTNERS, IT'S GOING TO
BE EASIER TO MOVE FORWARD TO
THAT NEXT STEP OF
IMPLEMENTATION.
>> OKAY.
WYNNE, NICE TO SEE YOU ON
CAMERA.
>> NICE TO SEE YOU AS WELL.
GOOD AFTERNOON, GOOD MORNING, OR
EVENING, WHEREVER FORCE -- FOLE
JOINING TEAM.
I WANT TO NOTE BEFORE I START
THANK YOU VERY MUCH FOR THE
OPPORTUNITY TO BE PART OF THIS
PANEL, A THOUGHTFUL KIND OF YOU
TO ASK ME.
APPRECIATE YOUR COMMENTS ABOUT
LEADING THE EFFORTS.
HE SETS THE BAR HIGH FOR THESE
PANELS.
NEXT TIME I'LL TRY TO GO BEFORE
VERSUS AFTER, I APPRECIATE BEING
INVITED TO CHAT.
I WANTED TO SHARE THOUGHTS TODAY
AROUND REALLY FOCUSING IN MORE
ON THE FIRST PHASE, UG3 PHASE
VERSUS UH3 PHASE, ALTHOUGH
RECOGNIZING THERE ARE SOME
DIFFERENT TERMINOLOGY HERE SO
I'LL REFER TO IT AS THE FIRST
PHASE.
WHAT COULD BE DONE DURING THE
FIRST PHASE, WHICH ALSO IS A
PRODUCT OF WHAT IS PROPOSED IN
THE GRANT, THAT COULD TRY TO
MITIGATE SOME OF THE CHALLENGES
THAT HAVE BEEN DISCUSSED
YESTERDAY AND TODAY AROUND
RECRUITMENT IN PARTICULAR.
AND SO REALLY TRYING TO THINK
ABOUT AND IT'S A PATE MOST
TRIALS I WOULD SAY HAVE
CHALLENGES WITH RECRUITMENT AND
ACTUALLY THAT'S ONE OF THE
REASONS MOST TRIALS ARE SHUT
DOWN EARLY.
ALSO KIND OF DOVETAIL INTO,
RICH, WHAT YOU WERE TALKING
ABOUT IN TERMS OF PILOT TESTING
THE INTERVENTION AND HOW MUCH
CHANGE IS APPROPRIATE AND
NECESSARY, WHAT THAT MEANS FOR
THE PROTOCOL.
SO, WANTED TO OF COURSE
RECOGNIZE THAT IN YOUR
APPLICATION OR IN APPLICATIONS
FOLKS OBVIOUSLY ENGAGE THEIR
STATISTICIAN, ACTUALLY AS IT'S
BEEN MENTIONED ENGAGE THEIR
PARTNERS, WHETHER CRITICAL
PARTNERS, PATIENT
REPRESENTATIVES, DIFFERENT
LEADERS IN THE C-SUITE FOR
DIFFERENT CLINICS AND HEALTH
SYSTEMS, TO THINK ABOUT ISSUES
THAT MIGHT COME UP AND THE WAYS
IN WHICH THINGS MIGHT GO WRONG
AND HOW MUCH AND TO WHAT EXTENT
ONE MIGHT NEED TO OVERSAMPLE AND
PLANNED FOR THOSE.
WE'RE IN PRAGMATIC TRIALS
FOCUSED ON QUESTIONS IMPORTANT
TO DECISION MAKERS AND RESEARCH
PARTNERS, QUITE A CHALLENGE
REGARDLESS OF THE TYPE OF TRIAL
AND YOU HAVE THOSE FOLKS ENGAGED
IDEALLY BUT THINGS CAN GO
SIDEWAYS HERE.
I WANTED TO SPLIT IN THINGS THAT
COULD BE DONE BEFORE THE FULL
TRIAL IS LAUNCHED IN THE SECOND
PHASE AND THINGS THAT COULD BE
TRIED WITHIN THE FIRST PHASE TO
TRY AND PREVENT THAT FROM
HAPPENING OR LIMIT THE EFFECT IT
HAS ON MOVING THE TRIAL FORWARD
IN THAT SECOND PHASE.
OF COURSE, NUMBERS,
REPRESENTATIVENESS, REESSENTIAL
IN THE STUDY.
WE TALKED ABOUT PATIENT CENTERED
RESEARCH IN PCORI, BROUGHT UP
INCLUDING TWO REPRESENTATIVES ON
REVIEW, WE ALSO TALK ABOUT
PRACTITIONER-CENTERED RESEARCH.
TO WHAT EXTENT IS THIS TRIAL
ANSWERING A QUESTION OR
QUESTIONS THAT ARE IMPORTANT TO
PRACTITIONERS, WHETHER IT BE
SOMETHING THAT HELPS THEM DO
THEIR WORK BETTER, WHETHER IT BE
SOMETHING MORE EFFICIENT,
RESPONSIVE TO PATIENT NEEDS,
DOES IT REFLECT PERHAPS CURRENT
OR FORTHCOMING POLICY CHANGES,
TRYING TO BLEND THAT WITH
ADVANCING THE SCIENCE, ADVANCING
RESEARCH IN THIS PARTICULAR
AREA.
ENGAGE WITH STAKEHOLDERS EARLY
ON, WHAT ARE THE REASONABLE
NUMBERS, WORKING WITH YOUR
STATISTICIAN AND METHODOLOGISTS,
TRYING TO CONSIDER WHAT MIGHT BE
SOME SOLVENT STRATEGIES, SO TO
SPEAK, MIGHT YOU NEED TO EXPAND
ELIGIBILITY, TO ANALYZE PROVIDER
VERSUS CLINIC LEVEL WHICH HAS
BEEN DONE IN SOME TRIALS,
BECAUSE YOU DON'T HAVE ENOUGH
CLINICS.
ANOTHER ISSUE I THINK THAT ONE
COULD CONSIDER IS HOW MUCH
EFFORT DO YOU WANT TO PUT INTO
RECRUITMENT.
CERTAINLY YOU WANT TO MAXIMIZE
YOUR ABILITY TO RECRUIT PATIENTS
INTO THE TRIAL, OTHERWISE YOU
WON'T BE ABLE TO CONDUCT THE
TRIAL, BUT BALANCING THIS WITH
IF WE LOOK BACK AT THE
RECRUITMENT DOMAIN AND
ELIGIBILITY DOMAIN.
HOW MUCH EFFORT YOU PUT INTO
RECRUITMENT WOULD BE A FUNCTION
TO WHETHER THE TRIAL MORE OR
LESS PRAGMATIC WITHIN THAT
PARTICULAR DOMAIN.
THERE ARE PROS OF INCREASING THE
AMOUNT OF EFFORT ONE USES FOR
ENHANCING RECRUITMENT BUT
CERTAINLY CONS AS WELL TO THE
EXTENT THAT MAY BIAS THE TRIAL
MORE TOWARD THE EXPLANATORY END
FOR THAT DOMAIN RATHER THAN THE
PRAGMATIC END.
AND ALSO CERTAINLY BRINGS UP
QUESTIONS AROUND IF YOU CAN'T
RECRUIT, WHAT DOES THAT SAY
ABOUT THE QUESTION, WHAT DOES
THAT SAY ABOUT THE STRUCTURE,
AND REALLY LEANING MORE TOWARD
THE EXPLANATORY END, TO THAT
DOMAIN, IS THIS SOMETHING THAT
SHOULD BE DONE IN THE FIRST
PLACE, IMPLICATIONS FOR THE
TRIAL OVERALL.
SOME POTENTIAL SOLUTIONS TRYING
TO MITIGATE CHALLENGES IN THE
SECOND PHASE AROUND RECRUITMENT
AND PARTICIPATION AND ENGAGEMENT
IN TRIALS I THINK ONE OF THE
FIRST CONSIDERATIONS COULD BE
LOOKING TO THE LITERATURE.
SO THERE'S A BURGEONING
LITERATURE ON TRIAL ENGAGEMENT
AND PARTICIPATION, TRYING TO
BETTER UNDERSTAND WHO AND HOW
AND WHY FOLKS DO AND DO NOT
PARTICIPATE IN TRIALS.
SOME OF THE RECENT WORK WE'VE
DONE AT NCI IS USING THE HEALTH
INFORMATION, NATIONAL TREND
SURVEY, A REGULARLY -- SURVEY
COLLECT, NATIONAL REPRESENTED
DATA IN THE U.S.
WE FOUND THERE'S SOME RESEARCH
INDICATING THAT BLACKS ARE
ACTUALLY MORE LIKELY TO BE
OFFERED TO PARTICIPATE IN
CLINICAL TRIALS THAN WHITES BUT
ARE LESS LIKELY TO ACTUALLY
PARTICIPATE WHICH IS PROBABLY NO
SURPRISE TO FOLKS.
SO WHERE IS THE BREAKDOWN?
WHAT ARE SOME OF THE BARRIERS
TOWARD PARTICIPATING IN CLINICAL
TRIALS?
WHAT ARE SOME FACILITATORS, IS
IT FINANCIAL, IS IT OTHER
INCENTIVES SUCH AS CHILD CARE,
IS IT MORE EFFICIENCY IN TERMS
OF RECRUITMENT AND CONSENT WHICH
HAS BEEN DONE DURING COVID THAT
ARE VIRTUAL AND TELEHEALTH BASED
RATHER THAN IN PERSON, ARE THERE
STRATEGIES THAT COULD ENHANCE
UNDERSTANDING OF TRIALS AS A WAY
TO TRY TO EMPHASIZE AND INCREASE
PARTICIPATION IN TRIALS, SO
THAT'S ONE ASPECT LOOKING TO THE
LITERATURE IN TERMS OF BETTER
UNDERSTANDING OF COURSE,
CONTRIBUTING TO THE LITERATURE
AS THESE TRIALS UNFOLD.
TO THIS END, AS HAS ALSO BEEN
BROUGHT UP, WE'VE CONSIDERED AT
THE NCI AND HAVE EXPERIENCE WITH
UG3/UH3 PHASE IN INTERNAL
DISCUSSIONS AS WELL AS
DISCUSSIONS EXTERNALLY WITH
OTHER NIH COLLEAGUES, AND AS
WE'VE SEEN SOME OF THESE
DIFFERENT TRIALS UNFOLD USING
THIS PHASED APPROACH WE RECENTLY
ISSUED A PAR ON PRAGMATIC TRIALS
FOR OUR DIVISION, SOME MORE
CANCER, BEHAVIOR, EPIDEMIOLOGY,
SURVIVE RESEARCH TYPE OF TRIALS
AND DECIDED TO INCREASE OR
EXPAND FROM ONE YEAR FOR THE
FIRST PHASE TO A MAX OF TWO
YEARS.
REALLY SAW THIS AS AN
OPPORTUNITY AGAIN TO REFINE AND
HONE STUDY DESIGN AND
IMPLEMENTATION OF THE TRIAL,
WHICH WOULD ALSO THEN LEND
ITSELF TO IDEALLY AS WE'VE
SPECIFIED NOT ONLY ADAPTING AND
REFINING THE INTERVENTION AS
RICH WAS MENTIONING, AS FOLKS
WERE TALKING ABOUT HOW MUCH
ADAPTATION AND HOW MUCH IS
NECESSARY OR WOULD BE NEEDED FOR
THE TRIAL AND CAN YOU
ANTICIPATE, AND PILOT THAT
DURING THOSE TWO YEARS, IF YOU
OPT FOR UP TO THE TWO YEARS, BUT
ALSO KIND OF THE OPPORTUNITY AS
WAS ALSO WAS DISCUSSED AROUND
CYCLES, THE OPPORTUNITY TO
CONDUCT SMALL RAPID CYCLE
STUDIES WITHIN THE UG3 PHASE.
AND NCI HOSTED A WORKSHOP LAST
YEAR ON RAPID CYCLE RESEARCH
WHICH INCLUDED AMY KILBORN,
TALKED ABOUT HER WORK YESTERDAY,
APPLIED TO RECRUITMENT PHASE AND
OPTIMIZATION.
THERE'S ONE PARTICULAR STUDY
DESIGN THAT COMES FROM OUR
COLLEAGUES AT TRIAL FORGE IN THE
U.K. WHICH ARE CALLED SWAPS.
STUDIES WITHIN A TRIAL.
AND THESE ARE SELF-CONTAINED
RESEARCH STUDY EMBEDDED IN A
HOST TRIAL TO EVALUATE OR
EXPLORE ALTERNATIVE WAYS OF
DELIVERING OR ORGANIZING A
PARTICULAR TRIAL PROCESS.
IT'S A FORMAL DEFINITION.
IN PRACTICE ARE THERE WAYS TO
IMPROVE EFFICIENCY OF THE
CONDUCT OF A TRIAL, INCLUDING
BUT NOT LIMITED TO TESTING
DIFFERENT RECRUITMENT STRATEGIES
IN THAT FIRST PHASE, IN-PERSON
PLUS E-MAIL RECRUITMENT MORE
EFFECTIVE VERSE U.S. -- VERSUS
IN-PERSON ALONE?
PHONE CALL PLUS E-MAIL PLUS IN
PERSON?
IT'S AN OPPORTUNITY TO LOOK AT
RECRUITMENT IN TERMS OF NOT AT
THE PATIENT LEVEL BUT AT THE
PARTICIPANT LEVEL, PRACTITIONERS
OR CLINICS SO WE'VE DONE SOME
WORK TRYING TO EXPAND THIS TO
TRIALS THAT ARE TESTING
IMPLEMENTATION STRATEGIES, AND
WAYS IN WHICH YOU MIGHT BE ABLE
TO EMBED A SWAP, NOT JUST
PATIENTS AS PARTICIPANTS BUT
ALSO OR INSTEAD OF JUST LOOKING
AT PRACTITIONERS OR CLINICS,
ORGANIZATIONS, WHO ARE THE
PARTICIPANTS IN THAT TRIAL.
LAST, REALLY TRYING TO CONTINUE
TO BUILD CAPACITY INTERNALLY AND
EXTERNALLY IN CONDUCTING
PRAGMATIC TRIALS AND ALL THE
WONDERFUL WORK THAT YOU HAVE
DONE AND LIVERING TEXT BOOK AND
SO MUCH RICH INFORMATION
COLLECTED AND SHARED.
TRYING TO FOCUS ON THAT, WE'VE
TRIED TO BUILD INTERNAL CAPACITY
THROUGH TRAINING FOR PROGRAM
DIRECTORS AS WELL AS AGAIN
EXPAND TRAINING MODULES FOR
EXTRAMURAL RESEARCH COMMUNITY
AGAIN CONTRIBUTING TO AND
BUILDING ON EVIDENCE AROUND
RECRUITMENT.
I WILL STOP THERE.
THANK YOU.
>> TERRIFIC.
THANK YOU.
COMMENTS BEFORE WE MOVE ON TO
TISHA, ANYONE WANT TO WEIGH IN?
TISHA, THANK YOU.
>> GREAT.
THANK YOU ALL FOR THE
OPPORTUNITY TO BE HERE.
I'VE NOT BEEN INVOLVED IN THE
COLLABORATORY BUT I'VE BEEN A
LONG ADMIRER, GRATEFUL TO TO THE
VIEWPOINT -- TO LISTEN TO
REFLECTIONS.
MY PERSPECTIVE, I WORK AT THE
NATIONAL INSTITUTE
NATIONAL INSTITUTE ON DRUG
ABUSE, BRANCH CHIEF FOR FIVE
YEARS, I HAD THE CHANCE TO LEARN
FROM A LOT OF INVESTIGATORS, AND
OUR BRANCH, THE WORK WE DO, IS
AS BEING HEALTH SERVICES
RESEARCH WE HAVE A LOT OF
EFFECTIVENESS TRIALS, A LOT OF
IMPLEMENTATION SCIENCE TRIALS, A
LOT OF PRAGMATIC TRIALS, AND THE
WORK OF OUR BRANCH IS PUSHING ON
THAT "SO WHAT."
WE'RE AT THE END OF THE
TRANSLATIONAL CONTINUUM, IF ANY
RESEARCH THAT SHOULD MAKE IMPACT
AND MOVING THE NEEDLE ON
ADDICTION SERVICES DELIVERY IN
THE REAL WORLD SHOULD BE THE
WORK OF MY BRANCH SO I WANT TO
SHIFT GEARS AND TALK ABOUT HOW
WHEN THINGS ARE GOING WRONG WHAT
INTERACTION WITH PROGRAM STAFF
LOOKS LIKE AND HOW REVIEWING OUR
JOBS OR AT LEAST I VIEW MY JOB
AND TEAM VIEWS MY JOB, I'M GOING
TO BROADEN THE DISCUSSION.
YOU ALL HAD A COUPLE UNIQUE
FEATURES, ONE, A LARGE
INITIATIVE WITH A LOT OF
INFRASTRUCTURE SUPPORTS.
YOU HAD A U MECHANISM WHICH
ALWAYS GIVES A LOT MORE
FLEXIBILITY THAN A TYPICAL R
MECHANISM AWARD.
YOU HAVE A PHASED STRUCTURE
WHICH CAN OFFER A NICE
OPPORTUNITY TO DO SOME OF THE
PILOT WORK AND THEN TRANSITION
IMMEDIATELY INTO THE LARGER
SCALE TRIALS.
THINGS I'M GOING TO SHARE ARE
APPLICABLE, WITH THOSE UNIQUE
CHARACTERISTICS, YOU CAN OFTEN
WRESTLE MORE DIRECTLY AND
ACTIVITY WITH R MECHANISM, NOT
ALWAYS HAVING DIALOGUE WHERE YOU
DON'T HAVE STRUCTURES BUILT IN
SO I'LL TALK ABOUT HOW THAT
CONVERSATION WITH THE P.O. MIGHT
GO AND HOW THE DISCUSSIONS ARE
SHAPING THE WAY THAT WE INFORM
WHAT WE'RE DOING AS WE'RE
THINKING ABOUT DESIGNING
PROGRAMS OR PUTTING OUT RFAs,
OFFER YOUR FEEDBACK ABOUT HOW WE
CAN AGAIN HELP MOVE THE NEEDLE
AND ADVANCE THAT "SO WHAT"
QUESTION SO WE'RE ADVANCING THE
IMPACT OF OUR WORK.
SO, HOW I VIEW MY JOB, IN MY
ROLE TO HELP BE A GOOD STEWARD
OF TAXPAYER DOLLARS, TO MAKE
SURE WE'RE INVESTING IN RESEARCH
IN A WAY THAT ADVANCES PUBLIC
HEALTH IMPACT.
MY BRANCH IN PARTICULAR, I VIEW
US AS BEING AT AT THE END OF TE
CONTINUUM.
THERE'S A LOT OF RULES AT NIH,
LOTS OF WAYS TO TERM THAT.
WE NEED TO MAKE SURE WE'RE
COGNIZANT OF POLICY BUT ALSO
DOING THAT IN THE CONTEXT OF
UNDERSTANDING HOW WE MAXIMIZE
IMPACT OF TAXPAYER DOLLARS.
I'VE ENCOUNTERED A LOT OF
SITUATIONS WHERE STUDIES DIDN'T
GO AS PLANNED.
OFTEN THAT IS DESPITE THE BEST
EFFORTS AND HEROIC EFFORTS OF
THE INVESTIGATORS TO NAVIGATE
TOUGH SITUATIONS.
WHEN I ENCOUNTER THESE
SITUATIONS I'M ASKING FOUR KEY
QUESTIONS.
THE FIRST QUESTION IS WHAT IS
GOOD SCIENCE HERE?
WHAT LOOKS LIKE GOOD SCIENCE?
I COME BACK TO THE POINT IN A
MINUTE.
THE SECOND QUESTION I ASK MYSELF
IS HOW DO WE MAXIMIZE THE VALUE
OF THE INVESTMENT WE'VE MADE?
WHAT WERE THE ORIGINAL GOALS AND
AIMS OF THE STUDY AND WHAT THE
INVESTIGATOR PROPOSING, HOW DOES
THE PROPOSAL FIT WITH THAT AND
WE TALK AT NIH ABOUT IS THAT IN
SCOPE.
AND DEPENDING ON DIFFERENT
INSTITUTES AND P.O.s MAY HAVE
DIFFERENT VIEWS OF WHAT IS IN
SCOPE.
BUT MY THINKING ABOUT THAT TENDS
TO BE INFORMED AGAIN BY WHAT WAS
THE BIG PICTURE, WHAT WERE WE
TRYING TO DO WITH THIS STUDY,
WHAT DID WE HOPE TO LEARN?
AND THEN THE MOST -- THE SORT
OF -- THAT TAKES YOU TO THE
QUESTION DOES CONTINUED
INVESTMENT IN THE INFRASTRUCTURE
THAT'S BEEN DEVELOPED BY THIS
STUDY REPRESENT A GOOD
STEWARDSHIP OF TAXPAYER DOLLARS?
IS THERE A GOOD -- IS THERE A
SCIENTIFIC OPPORTUNITY WITHIN
THAT STUDY THAT WE CAN LEARN
SOMETHING AND MOVE THE NEEDLE OR
IS IT TIME TO THINK ABOUT AN
ORDERLY CLOSEOUT?
I KNOW THAT'S ALWAYS A REALLY
TOUGH CONVERSATION, SOMETHING
THAT BRINGS INVESTIGATORS
ANXIETY.
FORTUNATELY IT'S A RARE
SITUATION WHERE A NUMBER OF
THINGS HAPPEN THAT THE STUDY
BECOMES INFEASIBLE.
MY EXPERIENCE, THERE'S ALMOST
ALWAYS SOMETHING THAT CAN BE
SALVAGED OR THOUGHT ABOUT IN A
DIFFERENT WAY AND STILL BRING IN
SCIENTIFIC VALUE FROM THE STUDY.
I THINK I CAN SHARE A COUPLE
EXAMPLES SO YOU CAN SEE WHAT
THIS MIGHT LOOK LIKE.
ONE EARLY IN MY CAREER AT NIH I
ENCOUNTERED A A COMPLEX STUDY D
OF FOUR YEARS RECRUITED 5% OF
SAMPLE, IT WAS A VERY EXTENSIVE
STUDY, AND THAT'S ONE OF THE FEW
CASES WHERE A WHOLE LOT OF
THINGS HAVE GONE WRONG AND THERE
WASN'T MUCH THAT COULD GET THEM
BACK TO THE ORIGINAL GOAL BUT WE
FOUND A SET OF THINGS THAT STILL
MADE SENSE WITHIN THE OVERALL
GOALS OF THE ORIGINAL STUDY AND
CAME UP WITH A PLAN WHERE THE
INVESTIGATORS THEN DID A NUMBER
OF OTHER ACTIVITIES STILL IN
SCOPE OF THE ORIGINAL STUDY BUT
SORT OF CALLED IT QUITS WHEN
THEY HIT A DEAD END ON A
RECRUITMENT CHALLENGE.
I HAD ANOTHER STUDY THAT
ENCOUNTERED -- THIS IS ONE THING
I WANT TO OFFER BECAUSE
SOMETIMES BEING FLEXIBLE IS
IMPORTANT.
THERE'S ONE STUDY I ENCOUNTERED
EARLY IN MY CAREER,
UNDERPOWERED.
WE HAD CONVERSATIONS WHETHER
THERE WAS GOING TO BE VALUE IN
CONTINUING THE STUDY GIVEN THE
CHALLENGES.
THAT UNDERPOWERED STUDY ENDED UP
GETTING PUBLISHED IN JAMA, AND
ENDED UP BEING QUITE
INFLUENTIAL.
IT PROVIDED SORT OF EVEN THOUGH
IT WAS NOT FULLY POWERED STILL
PROVIDED SOME REALLY PROVOCATIVE
DATA POINTS THAT SPURRED A LOT
OF ADDITIONAL RESEARCH AND
REALLY MOVED THE VIEWS OF THE
LEADERSHIP AT NIDA AS WELL AS
LEADERS IN THE FIELD AND ENDED
UP BEING EXTREMELY INFLUENTIAL
STUDY, ONE STUDY I KEEP IN MIND
AS I THINK ABOUT THESE DIFFICULT
CONVERSATIONS BECAUSE THERE IS
THE BROADER PICTURE, NOT JUST
ARE WE HITTING RECRUITMENT GOALS
BUT WHERE IS THE BIGGER
SCIENTIFIC QUESTION GOING.
I'LL SHARE ANOTHER SORT OF
OBSERVATION THERE'S A NEED, THIS
PIVOT, WHERE A PIVOT HAS BEEN
IMPORTANT.
A LOT OF WORK I DO AND A LOT OF
WORK MY BRANCH FUNDS IS IN THE
JUSTICE SECTOR AND WE DO A LOT
OF WORK HOW PEOPLE ARE
TRANSITIONING BETWEEN HEALTH
CARE SYSTEMS AND JUSTICE
SYSTEMS.
A LOT OF TRIALS WE FUND RECRUIT
OUT OF JAILS AS PEOPLE ARE
RETURNING TO THE COMMUNITY, IF
YOU FOLLOW THE NEWS THERE HAVE
BEEN A LOT OF MAJOR UPHEAVALS IN
THE JUSTICE SYSTEM OVER THE LAST
FEW YEARS, CASH BAIL REFORM,
MASS RELEASES DURING COVID, TONS
OF POLICY ISSUES.
AND WE'VE HAD A NUMBER OF
STUDIES THAT JUST AS THESE
HAPPENED IN REAL TIME WHILE
COLLECTING SUDDENLY RECRUITMENT
STRATEGY OF RECRUITING FROM
JAILS BECOMES IMPOSSIBLE BECAUSE
THE FOLKS AREN'T WHERE THEY
EXPECT.
THAT ACTUALLY REPRESENTS A GOOD
MOVEMENT IN TERMS OF WHERE WE'D
LIKE THE FIELD TO GO BUT A
CHALLENGE FOR THE STUDY.
WE'VE BEEN ABLE TO FIND WAYS TO
PIVOT RECRUITMENT STRATEGY TO
CAPTURE THE GROUP OF FOLKS WHERE
WE CAN FIND THEM AND ADDRESS THE
BROADER QUESTIONS OF THE STUDY
DESIGN.
I WANT TO TOUCH ON WHAT IS GOOD
SCIENCE.
ONE THING IN SOME OF MY
EXPERIENCES WORKING ON
IMPLEMENTATION TRIALS I WOULD
ENCOURAGE THINKING FROM A DESIGN
PERSPECTIVE IS HOW DO WE THINK
ABOUT -- IT ALIGNS WITH DAVID'S
COMMENTS ABOUT THE PRE-MORTEM
APPROACH, STRUCTURING STUDIES SO
EVEN IF THINGS DON'T GO AS
PLANNED THAT'S INFORMATIVE,
THAT'S AN OUTCOME.
IF LEADERSHIP WON'T ADOPT AND
IMPLEMENTATION PLAN DEVELOPED BY
A LOCAL TEAM, THAT COULD BE AN
OUTCOME THAT IS OF INTEREST IN
OUR STUDY.
ALSO, YOU KNOW, WHEN THINGS SEEM
LIKE A GOOD IDEA AT THE TIME BUT
THEN IN THE END DON'T QUITE WORK
AS PLANNED, THERE'S OFTEN
RESISTANT -- THERE'S
HISTORICALLY BEEN DIFFICULTY
PUBLISHING NULL FINDINGS IN
LITERATURE BUT SOMETIMES WHEN WE
FIND THESE THINGS DON'T GO AS
PLANNED WE'VE GOT TO PUT THAT
OUT IN THE LITERATURE.
SO I KNOW THE STUDY RECENTLY I
SAW WHERE THEIR GOAL WAS TO GET
PEOPLE REFERRED INTO TREATMENT.
THINKING WITH THAT REFERRAL TO
TREATMENT IN MY WORLD WE'RE
OFTEN FOCUSED ON TRYING TO GET
PEOPLE TO USE MEDICATIONS FOR
OPIOID USE DIS DISORDER, GETTING
REFERRALS TO TREATMENT WOULD
LEAD TO THE UPTAKE OF
MEDICATIONS FOR OPIOID USE
DISORDER BUT IT DIDN'T QUITE
WORK.
THEY GOT REFERRALS BUT NOT
UPTAKE OF THE MEDICATIONS.
SO AGAIN FOR THAT STUDY WE'RE
LOOKING AT FUNDING ADDITIONAL
WORK TO SEE WHAT BROKE DOWN IN
THE ASSUMPTION OF HOF THIS IS --
HOW THIS IS GOING TO WORK.
STAKEHOLDER ENGAGEMENT IS
CRITICAL TOO, YOU'RE BETTER
POSITIONED TO TAKE THAT APART
AND THINK ABOUT WHY DID THINGS
NOT GO AS PLANNED.
INCLUDING THINKING ABOUT HOW
THIS IS ALIGNING WITH IN INCENTS
FROM STAKEHOLDERS.
HOW THESE HAVE INFORMED HOW WE
STRUCTURE FUNDING ANNOUNCEMENTS
AND INITIATIVES, AND IF YOU LOOK
AT THE RFAs MY TEAM HAS BEEN
PUTTING OUT OVER THE LAST COUPLE
YEARS, WE TALKED ABOUT A LOT OF
THESE ISSUES AND OBSERVATIONS IN
A RETREAT.
ONE OF THE PANELISTS YESTERDAY
TALKED ABOUT THE FOUR Ss.
FOR US WE CALL THEM SPECS,
SCALABILITY, SUSTAINABILITY,
STAKEHOLDER ENGAGEMENT.
FOR US IN SERVICES RESEARCH AT
NIDA IF WE'RE GOING TO MOVE THE
NEEDLE WE NEED TO BE THINKING
ABOUT THOSE ISSUES IN ALL OF OUR
RESEARCH.
WE'RE HOPING WE'RE GOING TO SEE
OVER THE NEXT FEW YEARS WE'RE
MAKING PROGRESS, THAT WE AREN'T
WHERE WE ARE TODAY, THAT WE'RE
IN FIVE YEARS FROM NOW ABLE TO
MOVE THE FIELD EVEN FURTHER.
THE LAST THING I WANT TO SHARE
TOO IS AGAIN AS WE THINK ABOUT
THIS WORK, I'VE HEARD THESE
TENSIONS BETWEEN WHAT IS
GENERALIZABLE VERSUS WHAT'S
SPECIFIC TO A CONTEXT, WHERE IS
SCIENTIFIC KNOWLEDGE, WE DON'T
HAVE RESOURCES TO TEST EVERY
INTERVENTION AN SETTING, WE'RE
STRUGGLING ARE WE MOVING THE
SCIENTIFIC NEEDLE, MOVING
PRAGMATIC NEEDLE OF HOW WE'RE
ACHIEVING OUR DESIRED OUTCOME AS
AN INSTITUTE, WHICH IS IMPROVING
DELIVERY OF SERVICES TO WORK TO
POPULATIONS THAT NEED THEM, AND
HOW DO WE BALANCE MAKING SURE
WE'RE ADDRESSING EQUITY, BEING
MINDFUL OF CONTEXT, ALSO DON'T
FALL INTO THIS TRAP OF TESTING
EVERY INTERVENTION AND POSSIBLE
ITERATION, AND NOT MOVING THE
BALL DOWN THE FIELD.
THANK YOU FOR THE OPPORTUNITY TO
SHARE SOME THOUGHTS.
>> THAT WAS TERRIFIC.
THANKS, TISHA.
WE'RE IN THE OPENING COMMENT
SECTION OF THIS PANEL, HANDS UP
IF YOU'D LIKE TO COMMENT.
WHILE WE'RE WAITING FOR THE
HANDS TO GO UP, COULD I ASK THE
THREE OF YOU NIH FOLKS TO TALK A
LITTLE MORE ABOUT WHAT
CONSTITUTES IN SCOPE, AND I'M
SURE THERE'S NO CARVED-IN-STONE
GUIDANCE ON THAT.
BUT TAKE THE HYPOTHETICAL
EXAMPLE OF A STUDY THAT REACHES
ITS GOAL A COUPLE YEARS EARLY,
FIVE YEAR STUDY, THREE YEARS,
TWO MORE YEARS OF FUNDING, IS IT
IN SCOPE TO PROPOSE HIGH-QUALITY
EXTENSION OF THE WORK THAT THEY
HAVE DONE?
OR NO?
>> I LOVE THE HYPOTHETICAL.
IT'S ONE WE LOVE TO SEE AS MANY
TIMES AS POSSIBLE.
I MEAN, I DO THINK, CERTAINLY
CURIOUS WHAT TISHA AND WYNNE
THINK AS WELL, WITHIN SCOPE HAS
GENERALLY BEEN A REASONABLY
BROAD CATEGORY, JUST WITH THE
IDEA THAT, YOU KNOW, THERE'S A
PREMISE AROUND A DIFFERENT
STUDY, STUDIES MAY HAVE, YOU
KNOW, A PARTICULAR SET OF
SPECIFIC AIMS BUT THERE MAY BE
OTHER RELEVANT QUESTIONS THAT
CAN BE ASKED AND ANSWERED WITHIN
THE CONFINES OF THE
INVESTIGATION.
SO, YEAH, I WOULD ASSUME THAT IN
CONVERSATION WITH PROGRAM STAFF,
WITH GRANTS MANAGEMENT, AND THE
INVESTIGATIVE TEAM IDENTIFYING
WHAT ARE SORT OF ADDITIONAL
OPPORTUNITIES WITHIN THE SAME
GENERAL RESEARCH AGENDA THAT
COULD BE TAKEN ON WITH APPROVAL,
WOULD MEET THAT VERY HAPPY
HYPOTHETICAL THAT YOU -- I DON'T
KNOW, TISHA AND WYNNE IF YOU
WANT TO ADD.
>> I WOULD BE OVER THE MOON TO
SEE -- SO IN THOSE CASES I'D BE
OVER THE MOON TO SEE A STUDY
THAT COMPLETED IN THREE YEARS
WHEN THEY PROPOSED FIVE
ORIGINALLY, BUT IT MAY MAKE YOU
QUESTION THERE WAS WORK THAT
HAPPENED BEFORE THE STUDY
PROPOSAL BECAUSE IN GENERAL WHAT
WE SEE IS IT TAKES PEOPLE JUST A
YEAR TO GET STARTED UP.
BUT LET ME NOT MAKE ANY NEGATIVE
ATTRIBUTIONS AND ASSUME IT JUST
REFLECTS AMAZING SUCCESS AND A
PRODUCTIVE TEAM.
WE WANT TO CONTINUE TO SUPPORT
SUCCESS UPON SUCCESS, SO IN
TERMS OF IN SCOPE I THINK I
ALWAYS TRY TO THINK OF IT AS
LIKE COULD I HAVE SEEN IT AS A
FOURTH AIM, AN UNSPOKEN FOURTH
AIM, YOU HAVE TO GIVE THREE
AIMS, COULD WHAT'S BEING
PROPOSED -- DOES IT LOOK LIKE IT
WAS SORT OF AN AIM THAT WASN'T
QUITE ARTICULATED OR COULD HAVE
BEEN ARTICULATED IN THE CONTEXT
OF THE WHOLE GRANT?
THE MORE THAT YOU -- IN THOSE
LOTS OF WAYS TO VIEW IT, IT'S
OFTEN IN THE EYE OF THE
BEHOLDER, SOME INSTITUTES AND
P.O.s ARE MORE CONSERVATIVE
HOW THAT VIEW THAT QUESTION IN
SCOPE, MIGHT BE CONTEXTUAL OTHER
THINGS BEYOND THE GRANT WITHIN
THE INSTITUTE BUT IF I COULD
ENVISION IT HAVING BEEN PART OF
THE ORIGINAL PROPOSAL IN SOME
WAY, SHAPE OR FORM, I'M OFTEN
MORE COMFORTABLE THINKING ABOUT
IT AS IN SCOPE.
>> I'LL JUST ADD ON
HYPOTHETICAL LINES ECHOING WHAT
TISHA AND DAVID SAID AROUND
WORKING WITH PEER GRANTS
MANAGEMENT AND YOUR P.O. AROUND
POTENTIAL EXTENSION OF THAT, I
THINK AN ADDITIONAL YEAR OR TWO
ONCE YOU COMPLETED YOUR TRIAL,
THIS MIGHT BE SEEN AS WITHIN
SCOPE, TO STUDY SUSTAINABILITY,
AND THAT IS OFTEN QUITE
DIFFICULT BECAUSE, YOU KNOW,
ONCE YOU GOT A TRIAL GOING
THERE'S USUALLY A LAG TIME
BETWEEN WHEN THE TRIAL IS DONE
AND YOU MIGHT HAVE RO3 OR R21 TO
STUDY SUSTAINABILITY.
IF IT OCCURS WITHIN THE SAME
GRANT PERIOD WOULD BE A NICE
OPPORTUNITY TO LOOK AT WHAT
EXTENT THE INTERVENTION IS
SUSTAINED, TO WHAT EXTENT YOU
SEE THE SAME EFFECTS OVER TIME,
AS WELL A LITTLE WHAT WE'VE DONE
A WHILE AGO AT NCI IS
SUPPLEMENTS TO REALLY TRY AND
PACKAGE THE INTERVENTION, AND
MAYBE PILOT TEST IT OR GET SOME
QUALITATIVE AND QUANTITATIVE
DATA WITH OTHER SITES WHO MIGHT
BE INTERESTED IN USING THIS
INTERVENTION, AGAIN KIND OF AT
THE SCALE OF CONSIDERATION IN
ADDITION TO SUSTAINABILITY
THINKING ABOUT THAT.
AGAIN, IT'S TRIAL BY TRIAL BASIS
BUT I COULD SEE THOSE AS FALLING
WITHIN BROADLY SPEAKING WITHIN
SCOPE AND REALLY IMPORTANT
RESEARCH QUESTIONS AS WELL.
>> TERRIFIC.
DAVID, I SEE YOU'VE POSTED LINKS
IN THE CHAT.
DO YOU WANT TO MENTION WHAT THEY
ARE?
>>YEAH, SURE.
I DON'T KNOW, DOES EVERYONE HAVE
ACCESS TO THE CHAT?
THIS IS FROM A COMMENT THAT
ERRED HAD MADE WANTING TO HEAR
MORE ABOUT PRE-MORTEM APPROACHES
AND EXAMPLES.
YEAH, SO THE GENERAL IDEA IS
THAT YOU'RE LOOKING AFTER THE
FACT PRE-MORTEM TO SEE WHAT WENT
WRONG.
PRE-MORTEM IS GATHERING RELEVANT
FOLKS TO ANTICIPATE WHAT ARE ALL
THE THINGS THAT MIGHT GO WRONG.
THERE WAS A RECENT EXAMPLE OF A
GROUP OF IMPLEMENTATION
SCIENTISTS WHO GOT TOGETHER AND
BASICALLY THROUGH GATHERING A
FAIR AMOUNT OF INFORMATION WERE
PROJECTING WHAT ARE DIFFERENT
WAYS IN WHICH SOME OF OUR
EFFORTS DON'T HAVE THAT IMPACT
THAT WE'D LIKE THEM TO HAVE SO
THAT WAS A PRE-MORTEM APPROACH
AND THERE'S BEEN INDUSTRY,
THERE'S BUSINESS, INDUSTRIES
HAVE USED PRE-MORTEM ANALYSIS TO
TRY AND PRIOR TO LAUNCHING A
PRODUCT OR PRIOR TO INITIATING
SOME LARGE SCALE CHANGE OF
TRYING TO BRAINSTORM DIFFERENT
THINGS THAT MIGHT HAPPEN AND HOW
DO WE TRY, BY RECOGNIZING WHAT'S
GOING TO GO WRONG, THE
HYPOTHETICAL FOR THIS
PRE-MORTEM, WHAT DO WE DO TO
INSULATE OUR PROCESS OR EFFORTS
FROM THAT.
IF PEOPLE DON'T HAVE ACCESS TO
THE CHAT WE CAN MAKE SURE TO
PROVIDE THE LINKS.
>> I'M TOLD THAT NOT EVERYONE
DOES HAVE IT SO WE'LL PUT THEM
IN THE SHOW NOTES.
>> PERFECT.
>> SO OTHERS CAN SEE THEM.
ONE OF THE LINES OF OUR
CONVERSATION IS THERE'S NO
SUBSTITUTE FOR PLANNING FOR
CONTINGENCIES.
EVEN UNKNOWNS ARE WORTH TRYING
TO ARTICULATE.
THE UNKNOWN UNKNOWNS SUFFER.
WENDY, YOUR HAND IS UP.
>> YEAH, WITH THIS PANEL OF
IMPLEMENTATION SCIENTISTS I
WANTED TO ASK A QUESTION THAT
COMES UP FREQUENTLY IN PEER
REVIEW OF THE TYPES OF
APPLICATIONS AS WELL AS
SOMETHING WE STRUGGLE WITH WHEN
WE LOOK AT THESE APPLICATIONS
PROGRAMATICALLY WHICH IS, YOU
KNOW, OFTEN WHEN I TALK TO
INVESTIGATORS ABOUT SUBMITTING
APPLICATIONS WE WANT THERE TO BE
AN EVIDENCE BASE BEFORE WE'RE
IMPLEMENTING INTERVENTIONS IN
HEALTH CARE DELIVERY SAYS --
SYSTEMS.
WE TALK ABOUT ADAPTATIONS.
THE QUESTION THAT COMES UP HOW
MUCH ADAPTATION BEFORE YOU GOT A
NEW INTERVENTION, BEFORE WE
SHOULD REALLY BE PUTTING IT INTO
HEALTH CARE DELIVERY SYSTEMS.
DO YOU HAVE GUIDANCE FOR
INVESTIGATORS AS THEY ARE
THINKING ABOUT THIS?
OR EVEN GUIDANCE FOR US ACROSS
NIH AS WORK THINKING ABOUT HOW
TO WORK WITH THESE APPLICATIONS
AND THE EVIDENCE THAT DOESN'T
PERTAIN TO WHAT THEY ARE
PROPOSING IN THE NEXT STEP?
>> SO LOOKING AT MY COLLEAGUES,
IF ANYONE WANTED TO GO OFF
BEFORE I DID, SO GREAT QUESTION,
WENDY.
I'LL GIVE A COUPLE THOUGHTS AND
THEN TISHA AND WYNNE MAY WANT TO
CORRECT MINE.
IT DOES DEPEND OBVIOUSLY.
PART OF IT IS AT TIMES WE MAY DO
OURSELVES A DISSERVICE BY NOT
SPECIFYING WHAT WE SEE AS THE
MECHANISMS OF ACTION WITHIN OUR
INTERVENTIONS, WHAT WE SEE AS
CORE COMPONENTS, PULLING
TOGETHER A PACKAGE WITHOUT A
GOOD SENSE OF MINIMAL
INTERVENTION NEEDED TO CHANGE OR
IS THIS SCOPED AS FRAMEWORK FOR
IMPLEMENTATION RESEARCH MODEL
SAYS AROUND CORE COMPONENTS AND
ADAPTABLE PERIPHERY.
I WOULD SAY GOING IN WHEN WE'RE
DEVELOPING AND WE'RE TESTING OUR
INTERVENTIONS, WE'RE DOING IT
WITH AN EYE TOWARD WHAT IS
ESSENTIAL AND WHAT ARE THINGS
THAT MAY BE FLEXIBLY ADAPTED.
WITHIN REVIEW COMMITTEES OR
APPLICATIONS, HOW WE DEAL WITH
THIS SORT OF CONSISTENCY VERSUS
ADAPTATION TO SOME DEGREE GIVES
US A CHANCE TO THINK THROUGH IS
AN AN OPPORTUNITY FOR HYBRID
DESIGNS WHERE WE'RE ABLE TO ASK
QUESTIONS ABOUT EFFECTIVENESS OF
INTERVENTION, WHETHER THERE'S A
LOT OF ADAPTATION, A LOT OF
FLEXIBILITY, AND AT THE SAME
TIME TO FOCUS ON IF IT'S BEING
IMPLEMENTED HOW DO WE UNDERSTAND
THE EFFECTIVENESS OF DIFFERENT
STRATEGIES TO GET IT INTO PLAY.
I WOULD SAY THE LAST THING I'D
SAY IS PART OF IT IS A
PERSPECTIVE OF WHO IS MAKING THE
DECISION, IF I'M MAKING THE
DECISION THAT THIS IS SORT OF
READY TO BE IMPLEMENTED THEN I
NEED TO JUSTIFY THE EVIDENCE IS
THERE, THAT THIS IS INTERVENTION
THE WAY IT'S BEING DEFINED IS
EFFECTIVE.
IF I'M ABLE TO EVALUATE WHAT IS
HAPPENING ANYWAY, THAT MAY GIVE
ME A LITTLE BIT MORE FLEXIBILITY
NOT TO HAVE TO DICTATE THIS
LOOKS THE SAME BUT THE
SCIENTIFIC ENDEAVOR IS TO
UNDERSTAND HOW DIFFERENT
PERMUTATIONS WHEN IMPLEMENTED BY
OTHERS MAY HAVE POSITIVE OR
NEGATIVE EFFECT.
JUST A FEW THINGS ON THE TABLE.
>> I'LL JUST ADD A COUPLE
THINGS HERE.
IT'S A GREAT QUESTION.
I THINK REFERRING BACK TO SOME
DISCUSSIONS EARLIER TODAY AROUND
FORM AND FUNCTION, SOME WORK
BRIAN HAS BEEN LEADING TRYING TO
SPECIFY PARTICULARLY FOR
INTERVENTIONS WITH MULTIPLE
COMPONENTS, SO WHAT MIGHT YOU
IMAGINE WOULD CHANGE IN TERMS OF
THE FORM IT MIGHT TAKE VERSUS
KEEPING TRUE TO THE FUNCTION AND
HOW AND WHY THAT IS PROPOSED TO
ACTUALLY HAVE AN EFFECT.
I THINK ALSO TRACKING
ADAPTATIONS IS REALLY IMPORTANT,
SO ANTICIPATING WHAT ADAPTATIONS
MIGHT BE, HAVING A STRUCTURE IN
PLACE IN THE APPLICATION AS WELL
AS PRACTICALLY WITHIN YOUR TRIAL
WHAT ARE SOME ADAPTATIONS THAT
MIGHT OCCUR, HOW YOU TRACK THAT
OVER TIME AND WE ALSO HAVE SOME
WORK AROUND EMERGING WORK AROUND
TRACKING ADAPTATIONS TO
IMPLEMENTATION STRATEGIES.
TO THE EXTENT APPLICANTS CAN
REFER TO EXISTING FRAMEWORK,
SPECIFYING WHAT INTERVENTION
ADAPTATIONS ARE FRAMED BY
SHANNON STERNMAN, HAPPY TO SHARE
IN THE CHAT OR AFTERWARD,
REFERRING TO THAT AND
IDENTIFYING TYPES OF ADAPTATION,
WHAT'S IN SCOPE, WHAT'S
CONSIDERED DIFFERENT, I DON'T
KNOW IF THIS IS DIFFERENT AND
NEEDS MORE EFFICACY VERSUS THIS
IS STILL WITHIN SCOPE BUT AGAIN
I THINK IN GRANTSMANSHIP TO
ANTICIPATE AND ACKNOWLEDGE WHO
IS GUIDING THE DECISION AROUND
ADAPTATION IS REALLY IMPORTANT.
>> I'LL JUST ECHO SOME
COMMENTS, IT'S A GREAT QUESTION
AND A TOUGH ONE.
I THINK IT IS A PLACE FOR MORE
OF HYBRID DESIGNS AND WE WOULD
DO WELL TO USE HYBRID DESIGNS
EARLY ON EVEN DOING EFFICACY
WORK, MAYBE NOT VERY EARLY
STAGES BUT WE SEE A LOT, AT NIDA
WE HAVE THE QUIRK WHERE WE
DIVIDE OUR TREATMENT DEVELOPMENT
AND THEN OUR SERVICES RESEARCH
SO WE HAVE THIS ACROSS DIFFERENT
DIVISIONS, WE HAVE THIS WEIRD
DIVIDE.
I DON'T THINK IT SERVES THE
SCIENCE WELL BECAUSE IT IS THESE
QUESTIONS, IS THIS A CHANGE THAT
IS FUNDAMENTAL, IS TAKING US
BACK TO THE EARLIER STAGE?
OR IS THIS A CHANGE THAT'S
REALLY ABOUT PUTTING TOGETHER
EVIDENCE BASED COMPONENTS, AND
WE'RE MAKING SURE IT WORKS AND
WE'RE DOING SO IN A WAY THAT IS
SMART, WHAT ARE THE CORE
COMPONENTS, WHAT IS MINIMUM
VIABLE INTERVENTION THAT WILL
PRODUCE OUR OUTCOME?
WE SEE A LOT OF IT.
I THINK THAT THE HYBRID DESIGNS
CAN BE MORE EFFICIENT WITH THE
WAY WE MAKE THESE INVESTMENT IN
THE CORE COMPONENT THINKING
ABOUT WHY WOULD THIS PIECE NOT
WORK AND WHAT ARE WE ADAPTING
AND FOR WHAT PURPOSES, SO, YEAH,
IT'S A TOUGH QUESTION.
THERE'S MANY SHADES OF GRAY IN
THIS SPACE.
>> OKAY.
GREAT.
AN INTERESTING PLACE TO BE.
WENDY WAS SURMISING PRIVATELY ON
THE CHAT -- I'M GLAD YOU CAME
BACK, WENDY.
WENDY WAS FEEDING ME QUESTIONS
BUT YOU SHOULD ASK THEM ON YOUR
OWN.
>> SURE.
SO WE HAVE A FEW OF OUR
INVESTIGATORS AND A COUPLE
HEALTH CARE SYSTEM PARTNERS ON
THE PANEL AS WELL.
AND, YOU KNOW, ONE QUESTION THAT
I WOULD ASK TO YOU AS
INVESTIGATORS IS, YOU KNOW, WHAT
DO YOU WISH YOU COULD HAVE ASKED
NIH?
WHEN THINGS WEREN'T GOING AS
PLANNED WHAT DO YOU -- WHAT DID
YOU WANT TO ASK, MAYBE YOU
DIDN'T, WHAT DO YOU WISH COULD
YOU HAVE DONE WHEN THINGS
WEREN'T EXACTLY GOING AS
PLANNED?
WE HAVE A COUPLE OF, YOU KNOW,
INVESTIGATORS STILL IN THE MIDST
OF THEIR TRIALS, MAY NOT WANT TO
BE FORTHCOMING BUT MAYBE YOUR
PREVIOUS TRIALS, MAYBE WHEN YOU
HAD OTHER SENDERS, OTHER THINGS
YOU WISH YOU COULD ENGAGE IN
MORE CONVERSATION OR DO YOU
THINK THE STRUCTURE OF THE
COLLABORATORY WHERE WE HAD TO
MAKE CHANGES DURING COVID SO I
DON'T KNOW IF ANY OF OUR
INVESTIGATORS WANT TO MAYBE TALK
ABOUT, YOU KNOW, WHEN DID YOU
DECIDE TO MAYBE EVEN HAVE THAT
CONVERSATION WITH NIH, AND HAVE
THOSE DISCUSSIONS TO THINK ABOUT
MAKING SOME ADAPTATIONS AND
CHANGES?
I KNOW NATALIA AND I HAD MANY
CONVERSATIONS OF THINGS WERE NOT
GOING SO WELL, THINGS WERE
CHANGING ALL AROUND YOU MAYBE IS
A BETTER WAY TO NAUGHT.
>> THIS IS RELEVANT TO WHAT YOU
JUST ASKED, WENDY.
DAVID HAD MENTIONED THERE MIGHT
BE FEAR TO ENGAGE BY P.I.s
WITH THEIR PROGRAM OFFICERS.
IN SOME WAYS A WAY TO
OPERATIONALIZE IT, TO TAKE AWAY
THAT EMOTIONAL COMPONENT WOULD
BE HELPFUL.
JUST KIND OF A POLICY.
I'M ALSO A PERSON TO JUST JUMPS
OFF A CLIFF, I KNOW I HAD TO
TALK TO YOU SOONER RATHER THAN
LATER BUT THE FEAR WAS REAL,
YES, WE WERE UNDER EMERGENCY
CIRCUMSTANCES WITH THE PANDEMIC
BUT THE FEAR WAS REAL BECAUSE
THIS WAS A REALLY DRAMATIC
CHANGE TO OUR INTERVENTION.
THE WORK THE WAY DAVID SAID.
I DIDN'T HEAR THAT AT THE TIME
BUT YOU WERE THE WAY HE SAID,
THAT YOU WOULD BEHAVE.
YOU DID LISTEN AND REALLY HELPED
US WORK THIS THROUGH, CONNECTED
US WITH THE COLLABORATORY, SO
THERE WERE A LOT OF -- THERE WAS
A LOT OF DISCUSSION.
IT WASN'T A UNILATERAL DECISION,
HAD TO THROUGH DSMB, NICCH.
I TOOK AN AMOUNT OF COURAGE, I
WISH IT WAS MADE THAT IT WAS
OKAY TO APPROACH EARLY SO THAT
IT COULD BE ADDRESSED SOONER.
>> GREAT.
KEN, YOU WERE NEXT TO LIGHT UP.
>> I SEE ED HAS A HAND UP, I'LL
LET HIM GO FIRST.
ED?
>> OH, OUR TRIAL WAS PRE-COVID.
AND WE MET I THINK IT WAS RICH
AND ROD CALIFF WERE THE
CO-CHAIRS OF THAT, RIGHT, RICH?
>> YEP, THERE WAS A TIME.
>> THERE WAS A TIME, YEAH.
ROD HAS GONE ON TO BE FDA
DIRECTOR.
I WANT TO SAY SOMETHING
POSITIVE, SUES AND I WENT TO ALL
THE MEETINGS, FOR THE FIVE OR
SIX YEARS WE DID THIS I CAN ONLY
SPEAK FOR MYSELF, I THINK SUSAN
WOULD AGREE, I LEARNED SO MUCH
FROM ALL OF YOU.
IT WAS AN INCREDIBLE EXPERIENCE.
I WISH WE COULD REPLICATE IT NOW
POST-COVID.
THAT'S JUST MY ANSWER.
>> OKAY, KEN, , YOU CAN HAVE TE
FINAL COMMENT.
>> I WASN'T THERE AT THE
BEGINNING OF ABATE BUT I WOULD
SAY, AGAIN, IT COMES DOWN TO THE
PARTNERSHIP AND THE FACT WE HAVE
A GOOD RELATIONSHIP WITH THE
RESEARCH TEAM AT HARVARD, AND
THAT THAT GROUP AS A GOOD
RELATIONSHIP WITH NIH THAT IS
I -- IT ALLOWED THE APPROPRIATE
BACK AND FORTH IN TERMS OF
QUESTIONS.
AND THE ABILITY TO APPROACH THE
FUNDERS IN TERMS OF RESEARCH
STUDIES, SO I'M GLAD ED HAD
COMMENTS BECAUSE HE WAS CLOSER
TO THE FRONT END OF THIS STUDY.
BUT I THINK IT DOES GET BACK TO
PARTNERSHIP AS BEING AN
IMPORTANT ELEMENT OF BEING ABLE
TO PROBLEM SOLVE ALONG THE WAY.
>> YEAH, IT IS -- I'LL JUST SAY
SINCE I'M MOST OFTEN INTERACTING
WITH NIH AS RECIPIENT OF
FUNDING, THAT RELATIONSHIP MAKES
IT TOUGH TO BE FULLY CANDID.
YOU KNOW, IT TAKES YEARS AND
YEARS OF TRUST, AND EVEN SO IT'S
A BIG ASK.
GLORIA, DO YOU WANT TO WEIGH IN?
>> YEAH, I THINK YOU'RE RIGHT
ON, RICH.
I WOULD SAY JUST BUILDING ON
WHAT NATALIA SAID, I WOULD SAY
ONE OF THE THINGS WE APPRECIATED
WAS THE INFRASTRUCTURE AND
SUPPORT OF COORDINATING CENTER.
AND SO, YOU KNOW, OUR
STATISTICIAN MET WITH THE OTHER
STATISTICIANS FROM THE OTHER
PROJECTS, THERE WAS REVIEW OF
ALL OF THE STATISTICAL ANALYSIS
PLANS OF ALL THE FUNDED
PROJECTS, AND I APPRECIATED KIND
OF THAT TYPE OF INFRASTRUCTURE
THAT REALLY KIND OF, YOU KNOW,
REVIEWED PROJECTS IN ADVANCE AND
DID ADDITIONAL REVIEWS AND KIND
OF THOUGHT THROUGH YOUR METHODS.
WE WERE IN A SITUATION WHERE WE
WERE ABOUT TO PUBLISH, AND THERE
WAS AN ERROR IN OUR PROGRAM THAT
WAS FOUND BY THAT STATISTICAL
GROUP, AND SO WE REALLY
BENEFITED THROUGHOUT BY HAVING
THAT TYPE OF INFRASTRUCTURE THAT
MADE IT SO IT WASN'T AS
CHALLENGING TO TALK ABOUT THE
ISSUES THAT YOU ARE ENCOUNTERING
BECAUSE YOU WERE IN A GROUP OF
PEERS AND SO I JUST WANTED TO
SHOUT OUT TO THE COORDINATING
CENTER FOR THAT TYPE OF
INFRASTRUCTURE.
I WILL SAY THAT I'M BEEN FUNDED
ON OTHER U PROJECTS, PART OF
CONSORTIUMS, PEOPLE SAID WE GET
A CHANCE TO WORK WITH THE
STATISTICIANS AND THEY SAID
ABSOLUTELY NOT, LIKE WE DON'T DO
THAT, WE WOULD NEVER ADVISE ON A
PROJECT BECAUSE THOSE ARE
DECISIONS THAT SHOULD BE MADE AT
THE PROJECT LEVEL SO I'VE ALSO
HAD TO NAVIGATE AND CHANGE
EXPECTATIONS BECAUSE NOT ALL
PROJECTS ARE RUN WITH THIS TYPE
OF FULL SERVICE COORDINATING
CENTER.
SO ANOTHER COMPLIMENT THAT I
THINK THOSE TYPES OF -- THAT
TYPE OF INFRASTRUCTURE CAN
REALLY HELP WITH THE
RELATIONSHIPS, BECAUSE YOU HAVE
YOUR PEERS THEY ARE IN THE SAME
SITUATION, REFLECTING ON THE
CHALLENGES AND I DO FEEL LIKE
OUR GROUP WAS COMING ABOUT
CHALLENGES AS THEY WERE
OCCURRING ON THE STUDIES, REALLY
HELPFUL.
>> GREAT.
AND CORITA, DO YOU WANT TO
WEIGH?
>> JUST GRATEFUL FOR THE HELP,
TO ECHO WHAT GLORIA JUST SAID,
IT'S KEY TO BE ABLE TO SHARE IN
AN HONEST WAY AND IT'S NICE TO
BE BACK, MY FIRST FACE-TO-FACE
MEETING OF THE COLLABORATORY,
LAST MONTH, TWO MONTHS AGO.
YEAH, NOTHING LIKE HAVING A
COMMUNITY YOU PEOPLE YOU CAN
TALK TO WHO ARE HAVING THE SAME
PROBLEMS YOU'RE HAVING.
>> TERRIFIC.
THIS IS THE PLACE TO DECLARE
VICTORY FOR THIS PANEL.
AND SO PANELISTS, THANK YOU,
REALLY, ON BEHALF OF EVERYONE,
REALLY APPRECIATE YOUR
THOUGHTFUL COMMENTS AND YOUR
PARTICIPATION.
WE HAVE A FEW MINUTES FOR
WRAP-UP.
THAT'S ADVERTISED ON THE PROGRAM
AS WENDY, KEVIN, AND RICH.
AND SO WE'RE NOT REQUIRED TO US
A ALL THE TIME BETWEEN NOW AND
2:00.
LET'S PRAY WE DON'T BECAUSE
OTHERWISE IT WILL BE JUST THE
THREE OF US.
IT'S TIME FOR EVERYONE TO WEIGH
IN.
SO, BY WAY OF SUMMARY, ONE OF
THE THINGS BY WAY OF SUMMARY, I
THINK ONE OF THE THINGS THAT WE
HAD BEEN -- HAD AS FRONT OF MIND
WHEN WE PUT THIS PROGRAM
TOGETHER WAS TO ASK HOW CAN
COLLABORATORY PROJECTS LEARN
FASTER AND I THINK WE
ACCOMPLISHED SOME OF THAT PART
OF WHAT, FOR ME, COME THROUGH
HOW MUCH THE COLLABORATORY HAS
FACILITATED BETTER, FASTER
LEARNING.
MAYBE THAT'S NOT A SURPRISE BUT
IT SEEMS TO BE ONE OF THE
THROUGH LINES OF EIGHT HOURS OF
DISCUSSION IS REALIZING HOW MUCH
THE VERY SUBSTANTIAL INVESTMENT
THAT NIH HAS MADE IN THE
COLLABORATORY HAS CONTRIBUTED TO
THE COLLABORATORY PROJECT BEING
MORE SUCCESSFUL THAN THEY WOULD
HAVE BEEN.
WENDY, YOUR BEST POSITION TO SAY
HOW MUCH WHAT THE COLLABORATORY
IS DOING IS DIFFUSING MORE
BROADLY ACROSS THE AGENCY.
GO AHEAD.
>> YEAH, I THINK THE DAY AND
I'M CURIOUS, LIKE YOU SAID,
RICH, THIS IS OPEN TO ANYBODY
WHO WANTS TO PROVIDE, SO MY
COMMENTS, WE HAVE LEARNED A LOT.
YOU KNOW, TRYING TO USE THESE
WORKSHOPS AS OPPORTUNITIES TO
SHARE THAT INFORMATION, I THINK
YESTERDAY, YOU KNOW, HEARING
FROM OUR HEALTH CARE SYSTEM
PARTNERS AND, YOU KNOW, THAT
OFTENTIMES WHAT THEY FIND MOST
INFORMATIVE ARE SECONDARY OR
SUBGROUP ANALYSES THAT REALLY
HAVE BIG INFLUENCE AND ARE NOT
JUST THE PRIMARY OUTCOMES.
AND SO I THINK THAT THAT'S
SOMETHING THAT'S REALLY
IMPORTANT FOR US TO THINK ABOUT
AS FUNDERS OF, YOU KNOW, ARE
THOSE ADDITIONAL QUESTIONS
INCLUDED IN THE AIMS AND, YOU
KNOW, IS THERE A REALLY GOOD
PARTNERSHIP BETWEEN THOSE HEALTH
CARE SYSTEMS AND INVESTIGATORS
SUCH THAT THEY HAVE REALLY
WEIGHED IN AND GIVEN INPUT ON
WHAT THOSE ADDITIONAL AIMS MAY
NEED TO BE AND WHAT ELSE SHOULD
BE COLLECTED BESIDES PRIMARY
OUTCOMES.
YOU KNOW, AS WE WERE TALKING,
HEARING FROM INVESTIGATORS HOW
HOPEFUL WORK GROUPS HAVE BEEN AS
PART OF THE STRUCTURE OF THIS
PROGRAM, IT ALWAYS MAKES ME
THINK ABOUT WHAT ABOUT RESEARCH
THAT CAN'T BE PART OF THE
PROGRAM?
YOU KNOW, WE CAN'T SUPPORT ALL
OF THESE TRIALS THAT ARE BEING
DONE, AND WE'RE SEEING MORE AND
MORE PRAGMATIC RESEARCH AND
CERTAINLY LOSS OF IMPLEMENTATION
SCIENCE HAPPENING OVER THE LAST
DECADE.
AND, YOU KNOW, I THINK WHAT
WE'VE TRIED TO DO WITH THIS
PROGRAM IS TO SHARE WHAT WE'VE
LEARNED BOTH THROUGH WORKSHOPS,
ALSO ON THE WEBSITE AND ON THE
TEXT BOOK ON THE BOOK,
clinicaltrials.gov, EVERYONE
HAS ACCESS TO THE INFORMATION,
WHERE WE PROVIDE TRAINING AND
OTHER THINGS TO REACH THE
BROADER COMMUNITY SO THAT YOU
DON'T HAVE TO FIGURE THIS OUT ON
YOUR OWN BUT YOU CAN ACTUALLY
FIND SOME OF WHAT'S LEARNED AND
HELP AROUND THINGS WE'RE
LEARNING LIKE BIOSTATISTICS AND
CLUSTER DESIGNS, A LOT OF
INTERESTING CONVERSATIONS WE'VE
HAD TODAY.
ONE THING THAT COMES TO MIND FOR
ME, WE DIDN'T SHARE A LOT OF
THOSE RESOURCES DIRECTLY AS PART
OF THIS WORKSHOP BUT I THINK
THERE'S A LOT OF INFORMATION
THAT PEOPLE WOULD FIND REALLY
RELEVANT IN HELPING DESIGN AND
CONDUCT THESE TYPES OF STUDIES,
ON THE WEBSITE.
KEVIN OR RICH, ADDITIONAL
THOUGHTS THAT YOU HAVE?
>>YEAH, I THINK IT WAS A GREAT
WORKSHOP.
I KNOW I LEARNED A LOT.
IT DID SHOWCASE THE WAYS IN
WHICH, YOU KNOW, BRINGING
TOGETHER A COMMUNITY WITH SOME
SHARED COMMITMENT TO
TRANSPARENCY AND MUTUAL LEARNING
REALLY HAS BEEN BENEFICIAL AND
SO WHEN WE THINK ABOUT TRYING TO
LEARN FASTER, IT SEEMS LIKE
THERE ARE SOME LESSONS TO BE
LEARNED FROM LOOKING AT THAT,
THE WAY THAT CULTURE IS CREATED
AND MAINTAINED, IN A NETWORK
THAT CONTINUES TO GET NEW
ENTRANCE.
SO, THAT SEEMS TO BE AN
IMPORTANT MESSAGE AS YOU SAID.
I THINK ANOTHER THING THAT STOOD
OUT TO ME TOO WAS THE WAYS IN
WHICH PRAGMATIC TRIALS PRESENT
YOUR NICE CHALLENGES AND
OPPORTUNITIES IN TRYING TO --
UNIQUE CHALLENGES AND
OPPORTUNITIES LOOKING FOR EARLY
SIGNS SOMETHING MIGHT BE AMISS.
AND RESPOND TO THOSE.
AND THERE ARE BOTH CHALLENGES
AND OPPORTUNITIES DISTINCT FROM
TRADITIONAL TRIALS.
SO THAT SORT OF IS A THEME FOR
JUST ABOUT ANY TOPIC WE TALK
ABOUT IN THE COLLABORATORY, BUT
IT WAS INTERESTING TO SEE IT
MANIFESTED HERE AS WELL.
AND HOW THE DECISION MAKING
AROUND THAT BORDER, THAT
CONTINUUM, I FORGET WHO SAID IT
WAS A CONTINUUM, WE HAVE A NEW
INTERVENTION WE'RE TESTING CAN'T
BE RESOLVED BUT INTERESTING TO
TRACK THAT IN THE FUTURE TOO.
A LOT OF OTHER STUFF BUT I'LL
STOP THERE.
>> FOR ME THE MOST IMPORTANT
TAKEAWAY OF THE MEETING WE'VE
HAD AND THE SEVERAL PLANNING
MEETINGS THAT WENT INTO THIS IS
THE IMPORTANCE OF CONTINGENCY
PLANNING, BEING EXPLICIT ABOUT
ARTICULATING, AS MANY OF THE
SORT OF POSSIBLE SITUATIONS THAT
MIGHT ARISE, AND BEING CLEAR, AS
CLEAR AS POSSIBLE, ABOUT HOW ONE
WOULD DEAL WITH THAT.
THE MORE YOU CAN AVOID OPERATING
IN THE COMPLETELY GRAY SPACE,
WHERE WE DEPEND ON PEOPLE WHO
ARE AS THOUGHTFUL AS TISHA TO
DECIDE WHETHER SOMETHING IS IN
SCOPE OR NOT WILL SERVE US WELL.
NOT THAT INVESTIGATORS HAVEN'T
BEEN ATTENTIVE TO THAT BUT
THERE'S ROOM TO DO MORE, TO PUT
MORE TIME AND ENERGY INTO THAT
KIND OF PREPARATION I THINK
WOULD SERVE US WELL.
>> GREG HAS A HAND UP.
>> SURE DOES.
GREG?
>> YEAH, THANKS.
I WANT TO SAY, RICH, YOU HAD
BROUGHT THIS UP, THE LANGUAGE IN
PLANNING MEETINGS, THE IDEA OF
FAILING FASTER, SAID NO, IT
SHOULD BE LEARNING FASTER.
BUT ONE OF THE THINGS WE TOOK AT
THE PLANNING MEETINGS, IT'S
IMPORTANT TO EMPHASIZE, IS THAT,
YOU KNOW, IF THE GOAL IS TO
BUILD A PLATFORM TO ANSWER
QUESTIONS, YOU KNOW, IT'S NOT
THE IDEA OF THIS THING WE'RE
TRYING DOES NOT WORK.
WE FIGURED THAT OUT QUICKLY,
WHAT'S NEXT, IT'S THE WHAT'S
NEXT THAT'S OFTEN MISSING.
AND, YOU KNOW, ESPECIALLY IF
WE'RE TALKING ABOUT THIS PHASED
AWARD PROCESS, TWO-STAGE THING,
THEY ARE DIFFERENT LETTERS AND
NUMBERS USED, BUT SAY UH2/UH3
MODEL, WE NEED TO HAVE THE
ABILITY TO HAVE A UT 2A, UH2C,
UH3C MODEL, WHEN APPROPRIATE,
BECAUSE IF THE ONLY OPTION IS,
YOU KNOW, AFTER SAYING THE FIRST
THING WE TRIED DIDN'T WORK,
LET'S SHUT DOWN, IT CREATES BAD
INCENTIVES TO TRY TO MAKE
SOMETHING LOOK LIKE IT'S WORKING
WHEN IT DOESN'T BUT ALSO IS
WASTEFUL.
THIS IS COMPLICATED THOUGH, IN
TERMS OF ASKING THE QUESTION
ABOUT WHEN IN TERMS OF SORT OF
STRATEGICALLY WHEN FUNDERS, NIH
OR PCORI, THINKING ABOUT THE BIG
PICTURE SEPARATELY CONSIDERING
THE ISSUES IS THIS THE RIGHT
QUESTION, FRAMED THE RIGHT WAY?
IS THIS THE RIGHT SETTING OR ARE
THESE THE RIGHT SETTINGS TO
ANSWER THE QUESTION?
YOU KNOW, ARE THESE THE RIGHT
PEOPLE, ARE THESE PEOPLE
QUALIFIED?
AND IS THIS THE BEST SOLUTION OR
BEST THING TO DO BASED ON WHAT
WE KNOW RIGHT NOW, AND THEN THE
THING WE DON'T CONSIDER IS WHAT
WOULD BE THE BACKUP PLAN?
YOU KNOW, IS THERE A BACKUP PLAN
IF THIS DOESN'T WORKS THERE
SOMETHING ELSE WE MIGHT TEST?
IT'S COMPLICATED.
CERTAINLY THIS COULD NEVER
HAPPEN WITH R MECHANISMS GRANTS.
THE U MECHANISM THAT'S BEEN USED
FOR THESE COLLABORATIVE TRIALS
IS A STEP IN THE RIGHT DIRECTION
BUT MAYBE WE NEED TO MOVE
FURTHER IN TERMS OF
COLLABORATIVE PLANNING FOR THE
"WHAT IF."
>> GREAT.
WENDY, YOU SPEAK FOR THE ENTIRE
NIH SO CAN WE DO THAT?
>> I CERTAINLY DON'T SPEAK FOR
THE WHOLE NIH, BUT I THINK, YOU
KNOW, COOPERATIVE AGREEMENTS, U
MECHANISMS, DEFINITELY GIVE US
MORE FLEXIBILITY, FOR SURE, THAN
R PROJECT GRANTS DO.
SO I THINK WE'RE ALSO STARTING
TO SEE AN APPLICATION MORE
COMMONLY CONTINGENCY PLANS AND
SORT OF WHAT IF SORT OF
SCENARIOS IF THINGS DON'T GO AS
PLANNED.
AND SOMETHING THAT WE KIND OF
TALKED ABOUT WAS MONITORING FOR
EITHER FIDELITY OR UPTAKE OF
INTERVENTION, MONITORING
ELEMENTS OF STUDIES ONGOING AND
THEN, YOU KNOW, WHAT I OFTEN ASK
WHEN WE SEE THESE APPLICATIONS
COME IN TO US HERE AT NCCIH, I'M
GOING TO MONITOR BUT WHAT
HAPPENS IF YOU SEE THIS OR IF
YOU SEE THAT, ARE YOU GOING TO
DO ANYTHING ABOUT IT?
AND CAN YOU DO ANYTHING ABOUT
IT?
WE'VE SEEN EXAMPLES ACROSS THE
COLLABORATORY WHERE THAT
TRIGGERS AN ACTION WHICH IS
REALLY CLEAR AND PRE-PLANNED AND
EASILY UNDERTAKEN AND IN OTHER
CASES WHERE IT'S NOT GOING AS
PLANNED, AND YES, SOMETHING
COULD BE DONE BUT THE COST TO DO
SOMETHING IS JUST BEYOND THE
SCOPE OF THE AWARD, AND SO IT
ACTUALLY CAN'T ACT TO MAKE
CHANGE IN TERMS OF UPTAKE OR
ADHERENCE OR THESE TYPES OF
THINGS.
BEING REALISTIC IN WALKING
THROUGH SOME OF THE THOSE
DISCUSSIONS UP FRONT I THINK
WITH YOUR PROGRAM OFFICERS AND
IN THE PLANNING PHASES AND THEN
IN WHAT YOUR PROTOCOL LOOKS
LIKE, AND WHAT YOUR CONTINGENCY
PLANS WILL BE, PLANNING FOR
THINGS YOU CAN ANTICIPATE WOULD
BE REALLY HELPFUL.
>> I WAS GOING TO ADD IF WE
LOOK AT THE HISTORY, NOW THERE'S
A HISTORY ABOUT THESE
DEMONSTRATION PROJECTS AND
COLLABORATORY, I THINK WE CAN
POINT TO A COUPLE OF EXAMPLES
THAT DIDN'T DO EXACTLY THAT
THING I WAS TALKING ABOUT BUT
WE'RE KIND OF GETTING THERE.
FOR INSTANCE, DIALYSIS.
THERE'S A LOT OF QUESTIONS ABOUT
WHAT TO DO IN DIALYSIS.
THE ORIGINAL TIME TRIAL, WHICH
WAS ABOUT DURATION OF DIALYSIS,
IT WAS ONE THAT DID LEARN FAST,
THIS ISN'T GOING TO WORK.
THEN WENT TO THE HIGH/LOW STUDY,
WHICH WAS A DIFFERENT BUT VERY
IMPORTANT QUESTION ABOUT DEFAULT
SETTINGS OF DIALYSIS, WHAT
SHOULD BE OUR DEFAULT WAY OF
APPROACHING HEMODIALYSIS?
IT EVEN HAD TO DO A PIVOT.
NOW, THERE WAS A LONG GAP
BETWEEN TIME AND HIGH/LOW, YOU
KNOW, IN THIS IMAGINARY WORLD, I
THINK WE WOULD SAY WHAT ARE THE
QUESTIONS ABOUT DIALYSIS, LET'S
SET UP THE PLATFORM TO USE THE
TERM THAT SOME PEOPLE USE, TO
STUDY QUESTIONS ABOUT DIALYSIS
AND LET'S SAY WHAT'S THE HIGHEST
PRIORITY QUESTION FOR RIGHT NOW,
WHAT'S THE THING WE WANT TO DO,
BUT BE READY TO PIVOT QUICKLY.
THAT DID HAPPEN.
BUT IT TOOK A WHILE.
YOU KNOW, IN TERMS OF ISSUES
ABOUT HOSPITAL-ACQUIRED
INFECTION, INFECTION CONTROL IN
HOSPITALS, I THINK IT'S FAIR TO
SAY MAYBE THAT HCA IS THE TRIAL
PLATFORM.
HERE WE ARE AT ROUND 2.
THERE WAS A DAY IN SAYING WHAT'S
THE NEXT QUESTION.
I THINK IN A PERFECT WORLD WE
MIGHT BE ALREADY PLANNING FOR,
OKAY, IF THIS IS GOING TO BE OUR
PLATFORM, NOW, YOU MIGHT NEED TO
MAKE ALLOWANCES FOR THE PLATFORM
EXPANDING OR SHIFTING OVER TIME
AS THE WORLD CHANGES, BUT THE
POTENTIAL SPEED, IF WE WANT TO
GO FASTER IT'S THE REUSABILITY
OF THE INFRASTRUCTURE I THINK
THAT'S A REALLY KEY THING.
>> YEAH, YEAH.
CERTAINLY THERE ARE EXAMPLES AND
YOU'RE WELL AWARE, GREG,
EXAMPLES WHERE INSTITUTES HAVE
SET UP EITHER A TRIAL NETWORK OR
A PLATFORM TYPE OF STRUCK TUSH
STRUCTURE TO
DO A SERIES OF TRIALS AND
UTILIZE THE SAME INFRASTRUCTURE.
WHAT WE STRUGGLED WITH IN
PRAGMATIC TRIAL SPACE IS HOW WE
SET THESE PROJECTS UP, EVERYBODY
COMES IN WITH THEIR OWN PARTNERS
FOR EACH INDIVIDUAL APPLICATION,
AS OPPOSED TO HAVING SORT OF A
NETWORK THAT HAS BEEN SET UP, I
THINK OTHER FUNDING GROUPS,
PCORnet, PCORI IS A GREAT
EXAMPLE TO TRIED TO SET UP A
NETWORK EXAMPLE THAT CAN DO THAT
IN HEALTH CARE SYSTEMS THAT WAY.
I DON'T KNOW, RICH, IF YOU WANT
TO COMMENT ON THAT AS A
DIFFERENT EXAMPLE, OF OF ANOTHER
WAVE TRYING TO DO MORE TRIALS
QUICKLY IT'S NOT ALWAYS EASY TO
DO IT THAT WAY, DEPENDING HOW
BROAD THE SCOPE OF THE TRIALS
COULD BE.
I THINK THE ONES MOST SUCCESSFUL
ARE COLLEAGUES AT NIDA AND THEIR
CLINICAL TRIAL NETWORK AROUND
OVERDOSE AND SUBSTANCE USE, THEY
ARE ABLE TO DO SORT OF NEXT
TRIAL AND THE NEXT TRIAL WITHIN
THE INFRASTRUCTURE THEY HAVE,
IT'S A LITTLE BIT EASIER,
PEDIATRIC ONCOLOGY, CHILDREN'S
ONCOLOGY GROUP, THEY PRIORITIZE
AND MOVE INTO THE NEXT STUDY.
AS YOU GET LESS SPECIFIC AND WHO
THE FUNDER IS THE GETS MORE
CHALLENGING.
>> AGREED.
>> I HAVE A COUPLE LAST-MINUTE
CLOSING COMMENTS TO THANK
EVERYBODY.
RICH OR KEVIN, ANYTHING YOU WANT
TO ADD BEFORE WE WRAP UP FOR THE
AFTERNOON?
>> NO, SEND US HOME.
>> JUST OUR GRATITUDE TO ALL
THE PEOPLE THAT PARTICIPATED AND
EVERYBODY WHO HUNG IN THERE AND
ATTENDED.
>> YES, YES.
YEAH, THANK YOU TO THE
ATTENDEES.
WE'LL FOLLOW UP WITH ANSWERS TO
THE SPECIFIC QUESTIONS BUT WE
GOT TO MANY.
I WANT TO THANK AGAIN ALL OF OUR
SPEAKERS, ALL OF OUR HEALTH CARE
SYSTEM PARTNERS, OUR
INVESTIGATORS, MY COLLEAGUES
ACROSS NIH WHO BRAVELY CAME ON
AND TALKED ABOUT CHALLENGING
SCENARIOS AND HOW THEY HANDLE
THEM.
THE PLANNING COMMITTEE, RICH
AND KEVIN, THE KEYNOTE SPEAKER
YESTERDAY, BUT THOSE PEOPLE
BEHIND THE SCENES THAT WE DON'T
ALWAYS SEE ON SCREEN, OUR
COLLEAGUES AT NIH WHO HELPED US
WITH VIDEOCAST, THANK YOU
TREMENDOUSLY FOR MAKING THIS
AVAILABLE, AND ARCHIVED, SO THAT
ANY OF YOU WHO MISSED THIS LIVE
CAN TUNE IN, ALSO THE
COLLABORATORY COORDINATING
CENTER TEAM, AND ALL THE OTHERS,
I KNOW HELPING WITH THIS AS
WELL.
THANK YOU SO MUCH FOR ALL OF
YOUR HELP AND MAKING THIS
HAPPEN.
I THINK THAT'S IT FOR US TODAY.
AND SO I THINK WE'LL GO AHEAD
AND CLOSE AND JUST THANKS FOR
JOINING US.
FEEL FREE TO FIND THIS ON THE
RETHINKINGCLINICAL TRIALS.ORG.
TUNE IN NEXT YEAR.
BYE-BYE.