Welcome to the Clinical Center Grand Rounds, a weekly series of educational lectures for physicians and health care professionals broadcast from the Clinical Center at the National Institutes of Health in Bethesda, MD. The NIH Clinical Center is the world's largest hospital totally dedicated to investigational research and leads the global effort in training today's investigators and discovering tomorrow's cures. Learn more by visiting us online at http://clinicalcenter.nih.gov JUST A COUPLE OF REMINDERS FOR ANYBODY WHO MIGHT BE NEW TO THESE SESSIONS. SO WE DO THIS FOUR OR FIVE TIMES A YEAR. IT'S SPONSORED BY THE DEPARTMENT OF BIOETHICS AT THE CLINICAL CENTER. THE STANDARD FORMAT WHICH WE'LL FOLLOW IS WE PRESENT WHAT'S BROUGHT TO OUR COUNCIL SERVICE. THERE IS A 24 HOURS A DAY BIOETHICS CONSULTATION SERVICE AT THE CLINICAL CENTER WHICH IS AVAILABLE TO ANYBODY AT THE CLINICAL CENTER, ANYBODY AT THE NIH WHO HAS AN ISSUE WHERE THEY'D LIKE TO GET SOME DISCUSSION ON THE ETHICAL ISSUES THAT MIGHT BE RAISED BOO IT. WE PRESENT ONE OF THOSE CASES FOUR TIMES A YEAR, THEN WE BRING IN AN OUTSIDE EXPERT TO LEAD THE DISCUSSION ABOUT WHAT THEY SEE AS THE MOST IMPORTANT OR PROMINENT ISSUES RAISED AND HOPEFULLY THOUGHTS ON THE BEST WAY TO TRY TO ADDRESS THEM, AND AS A COUPLE OF REMINDERS, ONE, EVERYTHING THAT I SAY IS MY OWN VIEWS. EVERYBODY ELSE SAYS IS THEIR OWN VIEWS, DOESN'T REPRESENT THE VIEWS OF THE CLINICAL CENTER, THE NIH OR INSTITUTIONS WHERE THEY WORK. THE SECOND THING, THIS GETS STREAMED ONLINE. WE TYPICALLY GET ABOUT 200 PEOPLE THAT WATCH. THOSE GO VERY WELL T ONLY COMPLAINTS I EVER GET IS WHEN PEOPLE CAN'T HEAR THE QUESTIONS THERE THE AUDIENCE. SO WHEN WE GET TO THE QUESTIONS, IF YOU CAN, THE IDEAL THING, IF YOU GET TO A MIC AND IT'S PICKED UP. IF YOU'RE STUCK IN THE MIDDLE AND CAN'T GET TO A MIC, SAY IT LOUD ENOUGH SO I CAN HEAR IT AND I'LL REPEAT IT BEFORE WE GET THE ANSWER. SO TODAY WE'RE GOING TO TALK ABOUT A VERY GENERAL ISSUE THAT HAS GOTTEN AS FAR AS I CAN TELL LOOKING THROUGH THE LITERATURE, AND KNOWING THE DISCUSSIONS WITH PEOPLE WHO RUN CLINICAL TRIALS. VERY LITTLE DISCUSSION AT LEAST IN THE ETHNICS LITERATURE. SO THERE HAS BEEN A LOT OF DISCUSSION PARTICULARLY OVER THE LAST FIVE YEARS ON INFORMED CONCEPT FOR CLINICAL RESEARCH TRIALS. IN PARTICULAR, THERE HAS BEEN A LOT OF DEBATE ABOUT WHAT YOU NEED TO TELL PEOPLE BEFORE YOU ENROLL THEM AND SOLICIT THEIR INFORMED CONSENT FOR A CLINICAL TRIAL. AS EVERYBODY KNOWS, CONSENT FORMS ARE LONGER AND LONGER, 20, SOMETIMES 340 PAGES THESE DAYS -- 30 PAGES. OBVIOUSLY THERE IS TOO MUCH INFORMATION. SO THERE HAS BEEN A PUSH OR AT LEAST MAY BE A HOPE TO TRY TO REDUCE THAT INFORMATION TO A MANAGE THEN SIZE. THAT RAISES THE QUESTION, WHAT SHOULD BE IN THAT -- IF WE HAVE A TRUNK THED SET OF INFORMATION THAT WE GIVE TO POTENTIAL RESEARCH SUBJECTS WHAT, IS THE MOST IMPORTANT THINGS TO BE IN THAT SET. >> AND SO THAT'S ONE OF THE QUESTIONS WE'RE GOING TO ASK, WHICH IS THERE IS GENERALLY WIDESPREAD AGREEMENT THAT IF YOU'RE GOING TO ENROLL SOMEBODY WITH A CONDITION IN THE CLINICAL TRIAL, AND THERE IS SOME ALTERNATIVE TREATMENT THEY COULD GET INSTEAD OF BEING IN THE TRIAL YOU NEED TO TELL THEM ABOUT THE POSSIBILITY OF GETTING THAT. THAT'S PRETTY WIDELY ACCEPTED. A RELATED QUESTION WHICH GOT ALMOST NO ATTENTION, WHAT IF YOU KNOW -- THERE IS THE POSSIBILITY OF OTHER CLINICAL TRIALS THAT THE PERSON COULD ENROLL IN, INSTEAD OF ENROLLING IN YOUR CLINICAL TRIAL? THAT SEEMS LIKE A REASONABLE ALTERNATIVE. YOU SHOULD TELL PEOPLE ABOUT IT. ON THE OTHER HAND, WE WANT TO GET PEOPLE INTO OUR CLINICAL TRIALS, NOT INTO OTHER PEOPLE'S, SO HOW DO WE NEGOTIATE THE TENSION BETWEEN THOSE 2 THINGS? SO TO START US OFF IS JAMIE ROBERTS WHO AT THE TIME OF THIS WAS A CLINICAL TRIAL SPECIALIST, HAS MOVED TO DUKE UNIVERSITY. COME BACK TO PRESENT THE CASE TO US. >> THANK YOU VERY MUCH. I'M EXCITED TO BE BACK. IN TERMS OF THE DISCUSSION ON INFORMED CONSENT I NOW WORK FOR THE CLINICAL TRIALS TRANSFORMATION INITIATIVE. BE ON THE LOOK OUT FOR THE FINAL RECOMMENDATIONS AND SOME NEW IDEAS ON CONSENT TEMPLATES AND APPENDIXES, SO YOU GIVE ONLY THE MOST SALIENT INFORMATION TO HELP PATIENTS. WE'RE VERY EXCITED ABOUT THOSE RECOMMENDATIONS. WHEN THIS CAME UP -- I HAVE NO FINANCIAL DISCLOSURES. THESE ARE OUR OBJECTIVES. AND ESSENTIALLY WHAT HAPPENED, I RECEIVED AS A CLINICAL TRIAL RECRUITMENT SPECIALIST, RECEIVED AND E-MAIL FROM AN INVESTIGATOR FOR NEWLY DEVELOPED STROKE NET AT NINDS, ASKING WHAT OUR POLICY IS, CONCERNS AND CONSIDERATIONS MIGHT BE IN TERMS OF RAMMEDLY ALLOCATING PATIENTS TO COMPETING STUDIES IN THE STROKE ENVIRONMENT. WE SPOKE A LOT ABOUT THIS INTERNALLY. AT NINDS. I FELT IT WAS BE HELPFUL TO TALK TO THE ETHICISTS AND GET A BROADER OPINION, OTHER THAN THE DISCUSSION WE WERE HAVING INTERNALLY, WHICH WAS, THEY'LL PUT PATIENTS IN THE STUDIES FOR WHOM THERE IS THE GREATEST EQUIPOISE. SO HERE IS THE SCENARIO, TREATMENT -- IMAGINE, THESE ARE ACUTE STROKE. ELIGIBLE SUBJECTS CAN BE ENROLLED WITHIN 6 HOURS OF SYMPTOM ONSET, 12 OR 24 HOURS. WITHIN 3 HOUR WINDOW WOULD BE ELIGIBLE FOR ALL 3 STUDIES. HOW DO YOU DECIDE WHICH OF THOSE TO ENROLL THAT PATIENT IN? THEY CAN ONLY PARTICIPATE IN ONE. SO THE U.S. FEDERAL REGULATIONS SAYS THAT WE MUST -- PATIENTS MUST BE INFORMED OF MOST APPROPRIATE ALTERNATIVE PROCEDURES OR COURSES OF TREATMENT, IF ANY, THAT MIGHT BE ADVANTAGEOUS. DOES THAT INCLUDE IF THERE ARE CLINICAL -- OTHER CLINICAL TRIALS FOR WHOM THAT MIGHT POSE AN ADVANTAGE FOR THEM? BRINGS US TO THE RESEARCH VERSES CARE. THIS REQUIREMENT IS WIDELY, NOT ALWAYS, NOT NECESSARILY OTHER CLINICAL TRIALS OR RESEARCH OPPORTUNITIES. SO THEY'RE NOT. FINALLY, WHEN AN INDIVIDUAL SUBJECT MAY BE ELIGIBLE FOR MORE THAN ONE INVESTIGATION, THE DECISION AS TO WHICH TRIAL IS MOST APPROPRIATE NEEDS TO BE MADE CASE BY CASE? MAY NOT BE FEASIBLE WITH A STROKE PATIENT, 6 HOURS WITHIN SYMPTOM ON SET. MAY NOT BE COHERENT ENOUGH TO TELL YOU WHO THEIR PRIMARY PROVIDEer IS, OR THEY MAY NOT HAVE ONE. THIS IS A CONSIDERATION THAT MAY BE PRECLUDED IN THIS STROKE ENVIRONMENT. WE NEED TO TALK ABOUT THE BENEFITS AND BURDENS. IF WE INFORM SUBJECTS OF ALL OPTIONS AVAILABLE TO THEM AND ALLOW THEM TO CHOOSE WHICH STUDY THEY MIGHT PREFERTIOUS HOW DO THEY MAKE THOSE DECISIONS? HOW MUCH BURDEN ARE YOU PUTTING ON A PATIENT WHO HAD AN ACUTE, OR LIFE THREATENING EVENT BY PRESENTING THEM WITH A CALF TIER IA OF POTENTIAL STUDIES? ESPECIALLY WHEN WE KNOW THAT SUBJECTS DON'T EVEN UNDERSTAND OFTEN THE CONSENT FORM FOR ONE STUDY, LET AIR LOAN 3 OR 4. AND HOW MANY MORE DO WE IMPACT THAT WHEN THEY'RE ACUTELY ILL AND PERHAPS COGNITIVELY EFFECTED, AND FACING A LIFE THREATENING SITUATION. IF SUBJECTS ARE ALLOWED TO CHOOSE, THE RESULT MIGHT BE BIASED DISTRIBUTION. PATIENTS MAY CHOOSE THE STUDIES THAT ARE THE LEAST BURDENSOME. RIGHT? AND IF NOT CLEAR, WHEN WE CAN JUSTIFY NOT DISCLOSING ALL THE AVAILABLE ALTERNATIVE STUDIES, WHETHER THEY HAVE TO BE ALTERNATIVE ONLY WITHIN OUR OWN INSTITUTION, WHETHER AN INVESTIGATOR NEEDS TO BE AWARE OF STUDIES ONGOING ACROSS THE STREET OR DOWN THE ROAD. SO REALLY, WE DON'T HAVE A LOT OF GUIDANCE FROM FDA OR OHRP ON THESE THINGS. IF WE DISCLOSE ALL AVAILABLE STUDIES AND ALLOW THE POTENTIAL SUBJECT TO CHOOSE WE'RE IN THAT SITUATION OF BIASING BECAUSE PATIENTS MAY CHOOSE THOSE STUDIES WHICH APPEAR TO BE THE LEAST BURDENSOME TO THEM. OR, MAYBE -- APPEARS TO BE THE MOST SECTIONY TO THEM. I'M GOING TO GET THIS DEVISE VERSES THAT DRUG. WE COULD GO WITH INVESTIGATOR CHOICE. I THINK THIS IS OFTEN WHAT HAPPENS. INVESTIGATORS MIGHT CHOOSE WHICH STUDY TO DISCLOSE. IF ONE STUDY CLEARLY BEST FOR THE PATIENT, THEY'LL DISCLOSE THAT ONE, HOWEVER, WHO DECIDES WHICH ONE IS BEST FOR THE PATIENT? OTHERWISE, THEY'LL DISCLOSE WHICHEVER ONE THEY PREFER. AND THIS -- PHYSICIANS COME TO THE TABLE WITH A BACK LOAD OF EXPERIENCE, AND KNOWLEDGE AND UNDERSTANDING WITH THEIR PATIENTS. MAY ACTUALLY HAVE PREFERENCES AMONG SMORING GUESS BOARD OF STUDIES OPEN AT THEIR SITE. WE COULD PRIORITIZE THE STUDIES. THIS WAS THE TABLE -- THIS WASSON ABRIDGED VERSION OF THE TABLE SENT TO ME, AND WE COULD GO WITH FELLING THE MOST IMPORTANT STUDY FIRST AND THEN THE SECOND. THE QUESTION IS WHO DECIDES WHICH IS THE MOST IMPORTANT STUDY? AND HOW MUCH DO WE LIMIT THE PATIENTS AUTONOMY BY NOT TELLING THEM ABOUT OTHER AVAILABLE STUDIES. BUT MOST IMPORTANTLY, HOW TO DECIDE WHICH IS MOST IMPORTANT. ARE THEY TIME SENSITIVE OR MORE DIFFICULT TO ENROLL? BECAUSE THEY HAVE A MORE CRITICALLY IMPORTANT SCIENTIFIC QUESTION? WHAT IF YOU HAVE INVESTIGATORS THAT THINK THEIR STUDY IS THE MOST IMPORTANT? WE CAN TRIKE RANDOM ASSIGNMENT, TO RANDOMIZE ON A DAILY BECAUSIS, THE STUDIES OFFERED TO EACH SUBJECT. I THINK THAT ADDS MORE BURDEN. DO YOU HAVE TO DISCLOSE TO PATIENTS THAT YOU'RE RUNNING A RANDOM PROCESS TO DECIDE WHICH STUDY TO OFFER THEM? OR KEEP STUDIES OPEN ON DIFFERENT DAYS. OVER STUDY A ON SUNDAY, WEDNESDAY, FRIDAY AND IF THAT ISN'T CHOSEN, MAYBE MOVE DOWN AKIN TO THIS TABLE, TO STUDY B. HOWEVER, THE IDEA IS THAT FOR THE MOST PART PATIENTS WILL ALWAYS BE OFFERED STUDY A FIRST ON ONE OF THESE DAYS, THEN B ON OTHER DAYS. SO LIKE I SAID EARLIER, WE DISCUSSED THIS INTERNALLY AT NINDS, TRIED TO COME TO AN AGREEMENT ON WHAT OUR POSITION WOULD BE. WHETHER OR NOT WE FELT WE NEEDED TO SAY ANYTHING ON IT AT ALL OR WHETHER IT WAS AN IRB DECISION ULTIMATELY. AND I ULTIMATELY MADE THE DECISION TO COME TO THE BUY ETHICS TEAM AND ASK FOR ADVICE BECAUSE I COULDN'T FIND ANYTHING IN THE LITERATURE. TURNS OUT THERE IS A PAPER WAY BACK IN 2002 IN THE JOURNAL IRB, THAT DISCUSSES THIS ISSUE VERY SPECIFICALLY. HOWEVER, THERE IS NOT MUCH, JUST A POSTIE OF LITERATURE ON THIS PARTICULAR ISSUE. SO THE QUESTIONS THAT AROSE THERE THAT CONSULTATION WERE THESE. HOW DO WE DETERMINE WHICH STUDIES ARE DISCLOSED TO POTENTIAL SUBJECTS FREEWAY ANYBODY? SHOULD RESEARCH BE HANDLED THE SAME IN THIS REGARD? DOES THE NEED TO DISCLOSE AVAILABLE ALTERNATIVES SUGGEST THAT INVESTIGATORS SHOULD DISCLOSE STUDIES CONDUCTED AT OTHER SITES, MEANING THEY WOULD NEED TO BE AWARE OF WHAT'S GOING ON GEOGRAPHICALLY AROUND THEM. SHOULD A GIVEN SITE OPEN MORE THAN ONE STUDY FOR THE SAME POPULATION? AND IF SO, HOW SHOULD THEY IN AN UNBIASED WAY ALLOCATE PATIENTS TO THOSE COMPETING STUDIES? SO THOSE ARE THE QUESTIONS THAT CAME UP FROM OUR BIOETHICS CONSULTATION. I WONDER, BASED ON THE EXAMPLE OF THE THREE STUDIES WITH THE WINDOW OF ON SET OF SYMPTOMS, HAS SCIENTIFIC IMPLICATION AND THE RESERVE THAT YOU MIGHT -- I MEAN NOT HAVE A [INDISCERNIBLE]. >> IF IT'S DIFFICULT TO RECRUIT TO THE THREE HOUR WINDOW, THEM CHANCES ARE WHENEVER YOU GET A PATIENT IN WHO IS GOING -- WHO IS ELIGIBLE FOR THE 3 HOUR WINDOW YOU'LL FUNNEL THEM INTO THAT STUDY. THEN WHEN IT COMES TO THE 24 HOUR WINDOW STUDY YOU'RE NOT GOING TO HAVE ENOUGH PATIENTS THAT REPRESENT HAVING COME IN WITHIN THAT THREE HOUR WINDOW. SO THERE ARE SCIENTIFIC IMPLICATIONS. THAT WAS ONE OF THE QUESTIONS WE HAD. HOW WOULD YOU, IN AN UNBIASED WAY, TRY TO MAKE SURE THAT THAT 24 HOUR STUDY WINDOW DOESN'T END UP BEING THE VICTIM OF EVERYBODY ENROLLING -- BEING ENROLLED IN THE 3 OR 6 HOUR STUDY WINDOW. >> YOU CAN IMAGINE, IT'S GOING TO BE HARDER TO FIND PEOPLE FOR THE 6 HOUR WINDOW. SO YOU GET SOMEBODY AT 3 HOURS POST STROKE. WHO WILL BE THE STRONG INCLINATION TO FUNNEL THEM INTO THE FIRST TRIAL. AND ONE WAY TO ENSURE THAT IS TO DOWN PLAY THE OTHER ONES OR NOT MENTION THE OTHER ONES. THAT'S A QUESTION OF WHETHER OR NOT THAT'S APPROPRIATE. THERE IS ONE OTHER HAND. YES? >> [INAUDIBLE QUESTION] LET ME JUST ANSWER A LITTLE BIT OF THAT. WHICH IS TO SAY I THINK THIS IS A GOOD EXAMPLE FOR US TO THINK ABOUT. I DON'T WANT TO THINK ABOUT JUST THIS EXAMPLE AND JUST THIS TYPE OF STUDY DESIGN. I WANT TO THINK ABOUT THIS ISSUE FAIRLY PROCEEDLY SO THE WAY I WANT TO THINK ABOUT YOUR QUESTION, THEN, IS IN GENERAL, AND WE'RE GOING TO HOPEFULLY GET SOMEBODY TO GIVE US A PROPOSAL ON THIS QUESTION, IN A MINUTE, BUT IN GENERAL TO WHAT EXTENT ARE THE INTERESTS OF THE PATIENT RELEVANT? ONLY, THEY ARE. RIGHT? SO DO YOU WANT TO SAY LOOK, ANY OTHER CHOICE OF LIKE INVESTIGATOR CHOICE THAT JAMIE MENTIONED OR RANDOMIZATION, THAT'S ONLY ALLOWED WHEN, WHAT? WHEN THERE IS NO REASON AT ALL TO THINK ONE IS BETTER THAN THE OTHER FOR THE PATIENT OR NOT MUCH OF A DIFFERENCE, OR NOT A BILLING DIFFERENCE. THAT'S EXACTLY THE KIND OF QUESTIONS THAT WE WANT TO TALK ABOUT. HOW DO WE BALANCE OFF FOR THE SCIENTIFIC DESIGN REASONS FOR MAYBE, FOR INSTANCE, A RANDOM ALLOCATION VERSES PATIENTS GETTING A CHOICE FOR EITHER WHICH THEY JUST MIGHT PREFER PERMANENTLY OR MIGHT BE BETTER FOR THEM MEDICALLY. SO THANKS FOR THAT. JAMIE -- ONE MORE THING? >> AND I THINK THIS IS AN IMPORTANT ISSUE. NOT JUST IN THE ACUTE STROKE SETTING BUT ACROSS A MULTIPLE OF DISEASE SETTING. TO ADDRESS THAT QUESTION, IF TPA WERE GIVEN TO EVERY ONE IN THAT STUDY, BUT YOU WERE LOOKING AT ADDITIONAL FINANCE MINISTER AT 3 HOURS, DIFFERENT -- TREATMENT AT 3 HOURS, DIFFERENT AT 6 AND DIFFERENT AT 234 HOURS, EVERYBODY IS GETTING THE STANDARD OF CARE BUT MAYBE -- ROB, YOU CAN CORRECT ME IF I'M WRONG. ROB IS AN ER PHYSICIAN, PROBABLY WITH MORE EXPERIENCE. BUT THERE IS THE POSSIBILITY THAT YOU WOULDN'T BE BEHAVING UNETHICALLY DEDRIVING PEOPLE OF EARLY TREATMENT. YOU WOULD SAY EVERYBODY IS GETTING TPA. WE'RE GOING TO ADD THESE THERAPIES AT DIFFERENT TIME POINTS. >> THANKS. THANKS, JAMIE. SO TO HELP US THINK ABOUT THESE ISSUES IS ROBERT SILBERGLEIT, WHO IS AN ER DOCTOR AT THE UNIVERSITY OF MICHIGAN. AND THE REASON WHY I WAS VERY EXCITED TO HAVE ROB COME AND TALK AND LEAD US IN THIS DISCUSSION, WE MENTIONED NOW TWICE, THERE IS NOT MUCH ON THIS. I WANTED TO GET SOMEBODY WHO COULD REALLY GIVE US AN INFORMED OPINION. AND THE UNIVERSE OF PEOPLE WHO DO SERIOUS CLINICAL TRIALS, WHO ARE VERY MUCH INVOLVED IN DOING SERIOUS CLINICAL TRIALS. AND ALSO, INSIGHTFUL PEOPLE THAT CAN THINK ABOUT AND WORK THROUGH THE ETHICAL ISSUES THAT THOSE TRIALS RAISE IS NOT THE LARGEST UNIVERSE IN THE WORLD. AND ROB HAPPENS TO BE ONE OF THE GREAT EXAMPLES OF THAT SMALL GROUP OF PEOPLE. AND GREAT TO HAVE YOU HERE, ROB. >> SO, YOU KNOW, THEY CHOSE ME BECAUSE THERE WEREN'T A LOT OF PEOPLE TO CHOOSE FROM [LAUGHTER] SO -- I DON'T HAVE A LOT OF ANSWERS HERE. I WILL DO MY BEST TO LEAD THE DISCUSSION AND TELL YOU WHERE WE COME FROM, FROM MY PERSPECTIVE. MY PERSPECTIVE IS THAT WE RUN A THIRDS FUNDED NEUROLOGICALLY FUNDED NETWORK, WHICH ENROLLS AT LEAST -- STROKE NET DOES ACUTE STROKE AS WELL AS OTHER FORMS OF STROKE. BUT WE DO JUST ACUTE MANAGEMENT. AND SO WE'RE ENROLLING IN THAT VERY EARLY WINDOW. AND THIS IS A PROBLEM THAT COMES UP FOR US A LOT. SO I'LL TELL YOU HOW WE AIR PROBLEMED IT AND -- AIR PROBLEMED IT AND WHY. THIS IS -- AIR PROBLEMED IT. [TECHNICAL DIFFICULTIES] ESTHE THE BRITISH NAVY. TACKLING THE PROBLEM OF SEARCHY AND DECIDED HE WOULD TRY TO SOLVE THIS PROBLEM BY TAKING A SINGLE CREW AND TWO MONTHS AFTER THEY WENT TO SEA, TOOK 12 SERVEY SYMPTOMATIC SAILORS, AND WANTED TO STUDY ON THEM. HE HAD ABOUT 6 DIFTHINGS HE THOUGHT WOULD WORK. HE DIDN'T HAVE TO DEAL WITH ETHICAL ISSUES. HE GAVE TWO PATIENTS ONE TREATMENT, TWO ANOTHER, AND TWO ANOTHER. 2016 -- 6 GROUPS. ONE WAS THE GROUP THAT DID -- WAS RANDOMIZED TO CITRUS FRUITS. THE BRITISH SOLDIERS BECAME LIMESE BECAUSE THE GROUP DID VIL. THE CIDER GROUP DID FAIRLY WELL. THE OTHER GROUPS ALL HAD NO BENEFIT AT ALL. MY DISCLOSURES, NONE RELEVANT TO THIS PRESENTATION. I DO HAVE FUNDING TO MY INSTITUTION TO SUPPORT RESEARCH T OBJECTIVES ARE TO CONTEXTUALIZE THIS PROBLEM, MORE THAN GIVE YOU SPECIFIC ANSWERS, TALKING ABOUT WHAT TO DO WITH SUBJECTS AND HOW WE ALLOCATE AMONG MULTIPLE COMPETING TRIAL. I'LL DESCRIBE OUR APPROACH AND CONSIDER THE RATIONALE. THE ULTIMATE PROBLEM IN ALL CLINICAL TRIALS, YOU HAVE A POPULATION OF INTEREST OF SUBJECTS. AND WHAT YOU'D LIKE TO DO IS DIVIDE THEM INTO 2 GROUPS THAT ARE THE SAME IN EVERY WAY EXCEPT FOR THE THERAPY YOU'RE GOING TO GIVE THEM. SO YOU WANT THIS GROUP TO LOOK LIKE THIS GROUP, IN EVERY WAY, EXCEPT THE THERAPY T PROBLEM WE'RE SUFFERING, WE ALSO NEED TO MAKE SURE WE'RE CLEAR THAT BOTH OF THESE GROUPS THAT WE STUDY LOOK LIKE THE POPULATION OF INTEREST. AND RANDOMIZATION, WE SPEND A LOT OF TIME ON, TALKED ABOUT A LOT. MAKE SURE THESE TWO ARE THE SAME. WE HAVE A BUNCH OF THINGS THAT INTERVENE HERE THAT MIGHT KEEP THIS GROUP BEING DECIDED FROM BEING DIFFERENT THAN THIS GROUP. THAT'S WHAT WE NEED TO WORRY ABOUT. SO THE THINGS THAT COME UP THAT WE TALK ABOUT A LOT ARE THAT THEY NEED TO BE ELIGIBLE FOR THE STUDY. AND SO ELIGIBILITY QUESTIONS COME IN. THEY MIGHT HAVE REGULATORY RAMIFICATIONS OR SAFETY PERMUTATIONS SPECIFIC TO THE INTERVENTION. THERE MIGHT BE SOME REASONS TO NARROW DOWN THIS POPULATION. HOPEFULLY, NONE OF THAT KEEPS THESE PEOPLE FROM BEING VERY DIFFERENT. IF THEY -- WE END ONE A BUNCH OF PEOPLE THAT ARE SHUFFLED OUT. IF THESE PEOPLE LOOK VERY DIFFERENT THEN WE HAVE A PROBLEM. CONSENT DOES THE SAME THING. WE HOPE FOR, YEAH, WE HAVE ELIGIBILITY AND CONCEPT ISSUES THAT ARE IMPORTANT. HOPEFULLY IT DOESN'T DIVERT TOO MANY PEOPLE OUT. IT GETS A LOT MORE COMPLICATED ONCE YOU START TO PUT IN OTHER TRIALS. SO WHAT HAPPENS WITH OTHER TRIALS? NOW YOU'VE GOT TO DIVIDE THIS POPULATION OF INTEREST AND HOPE THIS STILL LOOKS LIKE THIS, THIS, THIS, AND THIS. AND THEY'RE ALL A LITTLE DIFFERENT. ALL A LITTLE DIFFERENT WAYED HOW THEY GET -- BASED HOW THEY GET SELECTED ALONG 2 BAY HERE. SO WHAT WE NEED IS AN ALLOCATION PLAN. AND WHAT WE NEED TO DO IS TALK ABOUT THE PERMUTATIONS OF THAT PLAN. IF YOU DON'T HAVE A PLAN, ANYTHING CAN HAPPEN. YOU HAVE NO WAY OF PREDICTING IT AND CAN END UP WITH PROBLEMS. REALIZE, THE PROBLEMS YOU NEED TO DEAL WITH ARE MULTIFOLD. SO YOU WANT TO MAKE SURE TOO MANY PATIENTS ARE NOT SHUFFLED OFF HERE ZILL, OR HERE. THAT WOULD MAKE THESE GROUPS DIFFERENT. BUT YOU ALSO HAVE TO DEAL WITH IF YOU ALLOCATE PEOPLE TO THIS TRIALUISH AND THEY'RE NOT ELIGIBLE, DO I MAKE THEM ELIGIBLE HERE? IF THEY DON'T CONSENT, DO THEY -- ARE THEY OFFERED DEOPPORTUNITY TO CONSENT TO THE OTHER TRIAL? THERE IS LOTS OF WAYS TO CROSS OVER. SO THE WAY THAT -- AND NORMALLY, I WOULD SORT OF GO THROUGH A LECTURE AND VIEW THE WHOLE RATIONALE HOW WE GOT TO WHERE WE ENDED AND GIVE THE CONCLUSION. IN THIS CASE IT'S BETTER TO SAY WHAT DID WE DO, AND TRY TO JUSTIFY IT AFTERWARDS. THE IMPORTANT THING IS TO HAVE A PLAN. AND THE PLAN IS GOING TO BE TO MANAGE ELIGIBILITY. WE'RE GOING TO -- THE MOST IMPORTANT THING ABOUT MANAGING THE ELIGIBILITY IS TO MANAGE THE BIAS. WE'RE NOT GOING TO DO THIS IN AN UNBIASED WAY. IF WE SHUFFLE PEOPLE OFF FROM ONE TRIAL TO ANOTHER, THERE IS SOME FORM OF BIAS. SOME PEOPLE END UP IN ONE STUDY, THE OTHER STUDY HAS LESS OF THOSE PATIENTS. SO WHAT WE'RE GOING TO TRY TO DO IS ALLOCATE OR MANAGE OUR BIAS. WHAT WE'VE DONE IN THE NET IS THAT WHENEVER WE APPROACH -- WHENEVER WE APPROACH A NEW TRIAL, WE HAVE A FIXED SET OF SITES. WE BRING THAT TRIAL TO THAT SITE AND SAY AS PART OF THE START UP OF THIS TRIAL, YOU HAVE TO TELL US WHAT OTHER TRIALS, COMPETING TRIALS YOU HAVE TORE THIS STUDY POPULATION. YOU HAVE TO TELL US A PLAN. WE'LL LET EACH SITE INDIVIDUAL ON CASE BY CASE. I DON'T THINK THEY MEANT PATIENT BY PATIENT, I THINK THEY MEANT TRIAL TO TRIAL. SO THE QUESTION IS WHAT IS A CASE BY CASE BASIS? PATIENT BY PATIENT? I DON'T THINK THIS IS A WAY TO MANAGE ON PATIENT BY PATIENT. NEEDS TO BE MANAGED TRIAL BY TRIAL. WE HAVE THEM PROPOSE A SOLUTION AND ALLOCATION SCHEME THAT WORKS FOR THEIR SITE. AND THE REASON HAS TO BE DONE ON A SITE LEVEL IS BECAUSE EVERY SITE HAS A DIFFERENT COMBINATION OF TRIALS THAT THEY'RE RUNNING. AND THEN WHEN THEY COME UP WITH THAT PLAN WE'RE GOING TO TAKE THAT PLAN BACK TO OUR TRIAL LEADERSHIP AND MAKE SURE THAT IT'S ACCEPTABLE TO THEM, AND ALSO GOING TO MAKE SURE THAT THEY TAKE IT TO EVERY OTHER TRIAL LEADERSHIP THAT'S GOING TO BE IMPACTED BY OUR NEW TRIAL. AND MAKE SURE IT'S OKAY WITH THOSE. THEN A NEGOTIATION AROUND TO MAKE SURE THAT THIS IS ACCEPTABLE TO ALL THE PARTICIPANTS. HAS TO BE ACCEPTABLE TO THE SITE AND TO EACH OF THE TRIALS. IF IT'S NOT, THEN ONE CAN ALWAYS DROP OUT OF THE TRIAL. AND THEN WE REVIEW IT ANNUALLY, BECAUSE THINGS KEEP COMING UP AND OF THE ONLY TIME WE KNOW THAT A NEW TRIAL WITH COMPETING POPULATIONS HAS COME UP, ALL OF A SUDDEN ONE SITE PRODUCING AT THIS RATE DROPS OFF AND WE SEE A DROP IN ENROLLMENT. SO WE HAVE A PLAN. WE CAN TALK ABOUT DIFFERENT PERMUTATIONS OF THE PLAN. IT'S OWN A SITE BY SITE BASIS. IT'S VERY IMPORTANT TO CONTINUALLY REAPPROACH IT BECAUSE EVERYBODY HAS A PLAN. AND EVERYBODY HAS A PLAN UNTIL YOU'RE PUNCHED IN THE FACE. WHEN YOU OPERATIONALIZE THINGS THE PLAN OF GOES OUT THE WINDOW. YOU HAVE TO HAVE A PLAN AND BE ABLE TO THINK ABOUT THESE ISSUES AND RATIONALE BECAUSE YOU HAVE TO ADJUST IT ON THE FLY. SO I'M ACTUALLY GOING TO SAVE THE ETHICS QUESTIONS FOR LAST. I THINK THAT THIS PROBLEM IS REALLY MULTIFOLD. I WOULD ARGUE THAT IT'S A SCIENTIFIC PROBLEM PRIMARILY, AND WE NEED TO SOLVE IT ON A SCIENTIFIC BASIS BUT THEN AN OPERATIONAL POLITICAL ADMINISTRATIVE PROBLEM SECONDLY, AND THEN THE ETHICS OF IT MIGHT NOT BE THE HARDEST PART OF THAT. AND SAVE THOSE FOR LAST PAUSE THERE IS A LOT MORE PEOPLE HERE THAT CAN DISCUSS THE ETHICS THAN I. SO FROM SCIENTIFIC PERSPECTIVE, THE MOST IMPORTANT GOAL THAT WE HAVE IS TO HAVE A POPULATION OF CONSECUTIVE PATIENTS, AS I SAID, WITH SIMILAR CHARACTERISTICS. IF WE DON'T MEET OUR SCIENTIFIC GOALS EACH ELSE FALLS BENEATH THAT. IF WE DON'T MEET OUR GOALS THERE IS NO POINT IN ADMINISTERING THE TRIAL REALLY WELL. THE ROMNEY WE'RE -- REASON WE'RE EXPERIMENTING ON PEOPLE IN THE FIRST PLACE IS TO MEET A SCIENTIFIC AIM. IF WE FAIL TO MEET THE AIM, ARGUABLY, RESEARCH IS NOT ETHICAL. WE WANT TO AVOID THE LOSS OF ELIGIBLE PATIENTS WHERE THE LITERATURE IS, TO THINK ABOUT WHAT THE QUESTIONS ARE AKIN TO. THE LOSS OF ELIGIBLE PATIENTS TO ANOTHER TRIAL IS THE SAME AS THE LOSS TO CLINICAL PRACTICE OR TO A FAILURE TO ENROLL AT ALL. FROM THAT TRIAL'S PERSPECTIVE, AND THERE IS A LOT WRITTEN ABOUT THOSE PROBLEMS. AND THEN WE WANT TO AVOID HIGH RATES OF NON RANDOM OPT OUTS THAT CAN BE SCIENTIFICALLY DIFFICULT TO INTERPRET. I'LL TALK ABOUT ONE TRIAL THAT HAS A GREAT EXAMPLE, IF YOU LOSE A PATIENTS TO NON RANDOM OPT OUT, YOU LEAD TO PROBLEMS THAT ARE HARD TO INTERPRET. THIS TRIAL WAS VERY IMPORTANT, EXPENSIVE TRIAL COMPETED A COUPLE YEARS AGO. AFTER SORT OF LABORIOUS COURSE. THE TRIAL WAS A RANDOMIZED TRIAL OF UNRUPTERED BRAIN AVMs, A LOT OF PEOPLE WITH THESE USUALLY CONGENITAL VENOUS MALFORMATIONS, PROBLEMS WITH THE VASCULATURE OF THE BRAIN, THAT RESIDE AND DON'T CAUSE ANY PROBLEMS UP ITEM THE POINT WHERE A SMALL NUMBER RUPTURE AND BLEED AND HAVE TERRIBLE OUTCOMES. AND IT USED TO BE WE NEVER FOUND OUT ABOUT THESE UNTIL PEOPLE HAD RAPTURE. AS WE'VE GOTTEN MORE ADVANCED IMAGES, WE DISCOVER ALL THE PATIENTS IN THE WORLD WHO ARE LIVING ASYMPTOMATICALLY WITH THESE, AND WE SAY WE HAVE TO DO SOMETHING ABOUT THIS. ONLY A SMALL NUMBER OF THEM RUPTURE, AND THEY RUPTURE NOW OR TEN YEARS OR 20 YEARS FROM NOW. BUT IT'S A HUGE QUESTION. THERE ARE SURGICAL THINGS YOU CAN DO TO MODIFY THEM BUT IT'S A BIG SURGERY. AND A LOT OF MORBIDITY. SO IT'S A HUGE QUESTION. AND MORE AND MORE PEOPLE GET UNNECESSARY IMAGING AND WE FIND MORE AND MORE OF THESE, IT'S AN EVEN BIGGER QUESTION. SO THIS OUTSTANDING GROUP OF INVESTIGATORS RECRUITED SITES FROM AROUND THE COUNTRY AND THE WORLD. AND PROPOSED A TRYING OF 800 SUBJECTS. AND GOT FUNDED FOUR IT. AND DSMB LOVED IT. AND THEN AS THEY STARTED TO ENROLL THEY REALIZED THAT THEY WEREN'T ENROLLING AS WELL AS THEY THOUGHT. AFTER A YEAR THEY WENT BACK AND SAY WE THINK THAT REALLY A TRIAL OF 400 SUBJECTS WOULD PROBABLY ANSWER THIS QUESTION. CAN WE READVISE OUR SAMPLE SIZE TO 400? WE KEPT ENROLLING. THEY KEPT ENROLLING POORLY, AND WENT BACK AGAIN AND SAID IF WE FOLLOW THEM A LITTLE BIT LONGER, I BET WE CAN ANSWER WITH 250 PATIENTS. AND IRB APPROVED. AND EVENTUALLY THEY HAD 468 PATIENTS AND OH O248 PATIENTS AND FINISHED THE STUDIES. IT SHOWS THAT PEOPLE MANAGED MEDICALLY, WATCHED AND WAITED, DID BETTER THAN SURGERY TO PREVENT THIS. IT'S ALSO DIFFICULT. WHEN PEOPLE DO WELL DEPENDS. UP UNTIL THEY RUPTURE, SO FIVE OR 10 YEARS LATER, WHEN DO THE LINES CROSS? WHEN YOU HAVE SURGERY, A BUNCH OF PEOPLE DO POORLY RIGHT AWAY BUT HOPEFULLY DON'T RUPTURE LATER. WHEN DO THE TWO LINES CROSS? BUT AT LEAST FIVE YEARS, IT LOOKS LIKE IT WAS ALWAYS BETTER TO WAIT AND NOT DO THE SURGERY. THAT WOULD BE A GREAT SOLUTION BY 39 CENTERS ENROLLED ONE PATIENT. 9 CENTERS DIDN'T ENROLL ANYBODY IN THE STUDY. THE STUDIES WERE ALL SELECTED ON THE BASIS OF SHOWING THAT THEY COULD ENROLL AT LEAST 10 PATIENTS PER YEAR. THEY HAD TO DO A RUN IN PERIOD, WHERE THEY SHOWED 10 ELIGIBLE PATIENTS PER YEAR. THAT WAS NECESSARY TO GET IN. ALL THE STUDIES SHOWED THEY HAD ABOUT 50 ELIGIBLE PATIENTS PER YEAR. SO WHAT HAPPENED? WELL, THEY DIDN'T REQUIRE TO KEEP SCREENING LOGS. WE DON'T KNOW EXACTLY WHAT HAPPENED. BUT THERE WAS A VOLUNTARY SCREENING LOG PROCESS WHERE SOME OF THE SITES KEPT SCREENING LOGS OF EVERYBODY, AND MAYBE IF THOSE SITES WERE ILLUSTRATIVE, MAYBE WE CAN LEARN SOMETHING. PROBABLY, THIS IS AN UNDER ESTIMATION OF WHAT HAPPENED. FIRST OF ALL, A BUNCH OF PATIENTS WEREN'T ELIGIBLE. SO EACH OF THESE FIGURES REPRESENTS 100 PATIENT. THEIR SHOULD HAVE BEEN AT LEAST MEETING THE SCREENING CRITERIA, SHOULD HAVE BEEN AT LEAST 4,000 OR SO PATIENTS ALL THESE PATIENTS WERE ELIGIBLE. AND THE QUESTION WAS WHY DIDN'T THESE PATIENTS GET ENROLLED? PROBABLY, TWICE AS MANY OF THESE. MORE LIKE 50 RATHER THAN TEN PER YEAR. PROBABLY ABOUT THIS NUMBER. PROBABLY ABOUT TWICE AS LARGE. THIS IS THE PEOPLE THAT GOT ENROLLED. SO THESE PEOPLE ALL OPTED FOR SURGERY OR THE INVESTIGATORS TOLD THEM THEY NEEDED SURGERY, EVEN THOUGH THEY MET THE ELIGIBILITY CRITERIA. FOR WE DON'T KNOW WHETHER YOU NEED SURGERY OR NOT. THEY WERE TOLD THEY NEEDED SURGERY OR SAID THEY WANT SURGERY, SO THEY CHOOSE NOT TO. THE GREEN PEOPLE SAID THEY DIDN'T WANT SURGERY, NO MATTER WHAT. THEY WEREN'T PUT IN THE TRIAL. A FEW OTHER PEOPLE HAD OTHER REASONS. SO WE ENDED UP WITH THESE 250 PATIENTS. NOW WE'VE GOT AN AMAZING FINDING. IF YOU KNOW THAT EVEN THOUGH YOU'RE NOT NECESSARILY -- IF YOU'RE IN THE QUESTION OFF WHETHER YOU NEED SURGERY AND YOU KNOW YOU NEED SURGERY, YOU SHOULD STILL HAVE SURGERY, APPARENTLY. IF YOU DON'T KNOW, YOU SHOULDN'T. IF YOU ARE REALLY UNSURE YOU WILL YOU'RE LIKE THESE 250 PATIENTS, AND YOU KNOW WHAT TO DO WITH. THE SCIENCE TELLS US VERY LITTLE ABOUT THESE PEOPLE. IF THESE PEOPLE WERE JUST COMPLETELY LIKE THESE PEOPLE, IT COULD BE APPLIED BUT IF THERE IS SOME REASON THAT THESE PEOPLE WERE SUGGESTED TO HAVE SURGERY AND THESE PEOPLE WERE TOLD THEY SHOULDN'T HAVE SURGERY, WE DON'T KNOW ANYTHING ABOUT THAT. THAT'S WHAT WE NEED TO AVOID. SO WE'LL TALK ABOUT THIS, AND THE GOALS ARE TO MANAGE, NOT ELIMINATE BIAS. SO HOW DID THIS HAPPEN? WE GET THE -- THE ROMNEY WOE GET BIAS, IF -- REASON WE GET BIAS, IF WE HAVE ELIGIBLE PATIENTS, TRIAL A ENROLLS THESE PEOPLE AND B THESE PEOPLE, IF THEY'RE NOT ALL THE SAME BUT THERE IS INCREASING SEVERITY, SO THIS TRYING ENROLLS MORE SEVERE PEOPLE, IF YOU PUT THESE PEOPLE WHO ARE OVERLAPPING IN THIS TRIAL, THEN THIS TRIAL WILL BE DEPLETED OF SEVERE PATIENTS. IN THIS TRIAL, THIS TRIAL WILL BE DEPLETED OF LEAST VEER PATIENTS. YOU'LL OF THE BIAS. SO THE UNBIASED ALLOCATION IS TO PUT ONE OF THESE PEOPLE IN THIS TRIAL, ONE IN THAT TRIAL. SOMEHOW. AND THAT'S WHAT YOUR PLAN IS TO DO. IT'S STILL BIASED. THIS POPULATION IS STILL 1 PERSON LOW ON THE PEOPLE, SO IT'S NOT UNBIASED STRATEGY. SO THE ADMINISTRATIVE GOALS ARE WORTH DISCUSSING, WHY WOULD YOU GET INTO THIS SITUATION. >> WHY NOT JUST RUN -- WHY RUN TRIALS WITH COMPETING POPULATIONS IN THE FIRST MACE? THERE ARE GOOD ADMINISTRATIVE REASONS FOR RUNNING MULTIPLE TRIALS. YOUR GOAL IS TO HAVE A TRIAL FOR EVERY PATIENT IDEALLY. YOU'D LIKE EVERY ONE TO BEABLING TO HAVE THE OPPORTUNITY TO PARTICIPATE IN RESEARCH IF HE WANT TO. SO YOU'D LIKE TO BE ABLE TO STUDY EVERYBODY, SO THAT YOU GET THE MAXIMUM BENEFIT. YOU WANT ALSO TO HAVE A PATIENT FOR EVERY TRIAL. YOU WANT THEM TO HAVE MULTIPLE TRIALS BECAUSE IF SOMEBODY GOING TO BE INELIGIBLE AND DROP OUT, YOU WANT TO PUT THEM IN A TRIAL. THAT'S WHERE YOU GET YOUR RESEARCH PRODUCTIVITY. BUT ALSO YOU WANT PREDICTABLE OPERATIONS. YOU WANT TO HAVE AS MANY TRIALS AS YOU CAN SO YOU HAVE A PIPE LIBTHAT'S FULL. THE ONE THING THAT NIH NEVER WANTS TO SEE IS IDLING CLINICAL TRIAL OPERATION. YOU WANT SOMETHING, A SIMPLE IMPLEMENTATION IF AT ALL POSSIBLE. YOU WANT IT WORKABLE. SO THE NOTION HERE IS THAT IF YOU WERE ONLY RUNNING THIS TRIAL, I'D HAVE NOTHING FOR THOSE PATIENTS. AND THAT WOULD BE INENOUGH. SO WITH ONLY A FEW MINUTES LEFT I'LL TALK ON WHAT I THINK ARE THE ETHICAL GOALS AND DISCUSS ISSUES OFF AUTONOMY, FAIRNESS AND VIRTUE. FROM AUTONOMY STANDPOINT, THERE IS REALLY NO RIGHT TO PARTICIPATE. THERE IS NO MORE NEED ON AN ETHICAL BASIS TO GIVE PEOPLE A CHOICE OF ALLOCATION BECAUSE RESEARCH ISN'T A RIGHT OF ITSELF. IT'S AN OPPORTUNITY FOR INDIVIDUALS, AN OPPORTUNITY FOR THE INVESTIGATORS TO DO A SOCIAL GOOD, AND -- BOTH SIDES HAVE TO PLAY BY THE RULES IN THAT CONTRACT. AS LONG AS IT'S HAPPENING, YOU DON'T HAVE ANY OBLIGATION TO PUT SOMEBODY IN TO CREATE THAT OPPORTUNITY FOR PEOPLE. THERE MIGHT BE AN EXCEPTION TO THAT, FOR COMPASSIONATE USE. CERTAIN TIMES WHEN PEOPLE ARE IN DIRE SITUATION AND THE RESEARCH SEEMS LIKE THE ONLY HOPE. THEN WE SHOULD HAVE A SAFETY VALVE AND WE YOU BELIEVELY DO -- WE USUALLY DO. BUT OTHERWISE, I THINK IT'S AKIN TO RANDOMIZATION. AND WE NEED AN ALLOCATION PLAN THAT FOLLOWS RULES SIMILAR TO THE WAY WE STILL RANDOMIZE. FROM A FAIRNESS STANDPOINT, I THINK A NUMBER OF THINGS NEED TO BE CONSIDERED. FAIRNESS NEEDS TO BE -- WHEN WE TALK ABOUT OPPORTUNITIES, SO THESE ARE OPPORTUNITIES, NOT RIGHTS. WE SHOULD MAKE SURE WE'RE FAIRLY ALLOCATING OPPORTUNITIES. AND THAT THERE ARE OPPORTUNITIES THAT SEEM BETTER THAN ONE OR ANOTHER, OR OPPORTUNITIES THAT ARE EQUALLY IMPORTANT. THEN WE CAN STILL OFFER ON THE NATURE OF FAIRNESS, A WAY TO HANDLE THOSE WITHIN THE ALLOCATION PLAN. FAIRNESS SHOULD BE FAIR TO THE TRIAL, AND YOU NEED TO MAKE SURE YOU DON'T SUBVERT THE EFFORTS OF OTHER SCIENCES WHEN YOU DEVELOP YOUR ALLOCATION STRATEGY. EVEN IF IT ISN'T SOMETHING YOU NEED TO DO TO HONOR AUTONOMY, MAYBE A VALUE THAT COMES FROM PAYING ATTENTION TO PATIENT PREFERENCE, AKIN TO WHAT WE ACTUALLY TALKED ABOUT. AS A ALTERNATIVE. THERE ARE TIMES WE TALK ABOUT HAVING PREFERENCES DESIGNS IN CLINICAL TRIALS. WHEN RANDOMIZATION IS NOT GOING TO BE ADEQUATE BY ITSELF. THERE IS A WHOLE BODY OF LITERATURE ON THAT. I THINK THE BENEFIT THAT COMES FROM THAT VIRTUE ALLOWING SOME FORM OF PATIENT PREFERENCE IS PROBABLY MOSTLY GOING TO BE TO AVOID LOSS OF PATIENTS OUT OF THE RESEARCH ENTERPRISE ALL TOGETHER. SO FOR EXAMPLE, IN THE ARUBA TRIAL, IF PEOPLE WERE NOT WILLING TO CONSENT TO THIS ARM AND THEY FOLLOWED OFF HERE, OFFERING AN ALTERNATIVE WITH PATIENT PREFERENCE TO RANDOMIZATION MIGHT HAVE BEEN ABLE TO CAPTURE SOME OF THESE PATIENTS AND BRING THEM INTO A NON RANDOMIZED ARM. THERE ARE WAYS OF CREATING THESE ARMS AND AS LONG AS NOT TOO MANY PEOPLE GO IN, YOU CAN COLLECT INFORMATION ABOUT PATIENTS BASED ON THOSE NON RANDOMIZED ARMS, AND SEE WHETHER PATIENTS, WHETHER THEY EMPOWER FROM THOSE ARMS AS LONG AS THIS GROUP DOESN'T LOOK TOO DIFFERENT FROM THOSE GROUPS. PATIENT REFERENCE CAN BE WORKED IN AND THE BENEFIT WOULD COME IF IT KEEPS THEM FROM DROPPING OFF IN THE END AND GETS MORE PEOPLE IN THE STUDY. THOSE ARE MY THOUGHTS. TRY TO LEAVE ENOUGH TIME TO -- I'D LIKE TO HEAR OTHER PEOPLE'S THOUGHTS. [APPLAUSE] >> THANK YOU VERY MUCH. THAT WAS VERY GOOD. MY QUESTION IS AUDIENTS MENTION PATIENT PREFERENCE. THE WAY JAMIE SET IT UP, THIS IS A QUESTION ABOUT HOW MUCH BURDEN WE'RE PUTTING ON PEOPLE BY ASKING THEM TO MAKE A PREFERENCE OR A CHOICE. COULD YOU SAY ANYTHING ABOUT THAT? IS THERE A WAY THAT SHOULD BE TAKEN INTO CONSIDERATION IN TERMS OF HOW MUCH WEIGHT WE PUT ON PATIENT PREFERENCE? >> SO I THINK THAT DEPENDS A LOT ON THE SPECIFICS OF THE TRIAL. WAYS OF COMMUNICATING. SO IN THE -- I THINK THAT THERE IS A HUGE CHALLENGE IN HOW WE COMMUNICATE WITH PATIENTS AND TRY TO IDENTIFY PATIENT PREFERENCES IN THE ACUTE SETTING WITH REGARD TO CONSENT ALONE. AND WE PARTICIPATE IN A WORKSHOP LAST YEAR. SO JEREMY BROWN ORGANIZED TO TALK ABOUT HOW WE DEAL WITH CONSENT ISSUES IN THE EMERGENCY SETTING, AND INCORPORATE MEANINGFUL PATIENT DECISION MAKICKING HE BE THE CONTEXT WHERE PATIENTS ARE INHERENTLY IMPAIRED. IN THEIR DECISION MAKING. CERTAINLY, THE MORE BURDEN YOU PUT ON A STRESSED DECISION MAKING PROCESS, YOU CAN ANTICIPATE THAT THE DECISION MAKING IS GOING TO BE WORSE AND THERE IS A LOT OF PSYCHOLOGICAL EVIDENCE THAT IF YOU GIVE TOO MANY CHOICES, THEY OPT OUT ALL TOGETHER. SO THIS WOULD MAKE IT MORE DIFFICULT BUT I THINK IT ACTUALLY -- I THINK THERE ARE WAYS OF HANDLING THAT, WHEN IT'S IMPORTANT TO DO SO. AND YOU CAN SIMPLIFY THE PROCESS, THERE ARE WAYS OF APPROACHING PATIENTS THAT ARE IMPAIRED IN MAKING THEIR DECISION MAKING EASIER. BUT THE DEFAULT OF SAYING GOSH, THIS IS HARD TO DO, I CAN'T DECIDE, SO MY DEFAULT DECISION, I'M GOING TO LEAVE IT UP TO THE PATIENT. THAT'S ALWAYS GOOD, RIGHT? THAT PROBABLY IS A FAILURE. A MORLING FAILURE. -- MORAL FAILURE. TO DEFER TO THE IRB OR PATIENT. WE CAN'T DECIDE, WE CAN'T MAKE A PLAN, WE CAN'T AGREE ON WHAT'S BETTER. SO WE'LL CALL THE PATIENT PREFERENCE AND GET OURSELVES OUT OF THE PROBLEM, AND PRETEND IT'S A VIRTUE. THAT'S NOT A VIRTUE. >> ANYBODY ELSE? I HAD A QUESTION FOR YOU, ROB. SO I DON'T KNOW IF THIS IS YOUR VIEW OR WHAT I TAKE AWAY AS A PLAUSIBLE VIEW FROM YOUR TALK. WHICH IS AS I STARTED, I THINK THAT IF I LOOK THROUGH BOTH THE LITERATURE AND CURRENT PRACTICE, THAT WITH RESPECT TO INFORMED CONSENT, CLINICALLY AVAILABLE ALTERNATIVES GET TREATED VERY DIFFERENTLY THAN ALTERNATIVE RESEARCH. POSSIBILITY. SO MOST PEOPLE HAVE THIS VIEW, MOST IRBS HAVE THIS VIEW. I CAN TELL YOU MY IRB HAS THIS VIEW. IF THERE IS A CLINICALLY AVAILABLE ALTERNATIVE, EVEN IF YOU DON'T HAVE ANY REASON -- YOU THINK IT'S JUST AS GOOD, YOU NEED TO TELL PEOPLE. IF IT'S A RESEARCH STUDY THAT MIGHT BE AS GOOD, YOU DON'T NECESSARILY NEED TO TELL PEOPLE ABOUT THAT. THERE MIGHT BE A LITTLE BIT OF VALUE, BUT IT'S NOT A BIG DEAL. MIGHT NOT WEIGH THAT MUCH AGAINST CONSIDERATIONS. SO I WAS WONDYOU ARING IF THAT DIFFERENTIAL TREATMENT ISN'T JUSTIFIED? THAT THERE IS NO REASON TO PREFERENTIALLY DISCLOSE CLINICALLY AVAILABLE OPTIONS VERSES RESEARCH. IT SHOULD DEPEND ON WHETHER THEY'RE BETTER OR WORSE AND WHAT IMPACT IT WILL HAVE ON THEIR ABILITY TO MAKE A GOOD CHOICE AND OUR ABILITY TO COLLECT GOOD DATA? IS THAT FAIR OR IS THAT -- >> NO. I MEAN I DON'T -- IT MIGHT BE FAIR AND THROUGH. I'M NOT SURE -- IT'S NOT WHERE I WAS COMING WITH IT. I WOULD ARGUE THAT THE -- IT IS IMPORTANT TO GIVE PEOPLE THE CLINICAL OPTIONS. RIGHT? I MEAN ACTUALLY I DON'T KNOW THAT WE DO VERY OFTEN, BY THE WAY. I THINK WE OFTEN DO THAT IN A LITTLE BIT OF SHORT -- BECAUSE WE ACTUALLY DO THINK THAT CLINICALLY, THIS IS MY JUDGMENT AS TO WHAT YOU SHOULD DO. YEAH. YOU SHOULD DO THIS, YOU COULD DO THESE THINGS, TOO, DON'T WORRY ABOUT THAT. BUT I THINK EVERYBODY DOES HAVE A RIGHT TO KNOW THEIR CLINICAL CHOICES. THOSE ARE CHOICES. I THINK THAT THERE IS NO OBLIGATION TO -- WE WOULDN'T HAVE TO CREATE A RESEARCH OPPORTUNITY JUST TO CREATE A RESEARCH OPPORTUNITY BECAUSE PEOPLE SHOULD HAVE THAT AS A CHOICE. RIGHT? THE RESEARCH OPPORTUNITY, I THINK, IS OPTIONAL. THE CLINICAL DECISION HAS TO BE MADE, RIGHT? ONE OF THE CHOICES IS TO PARTICIPATE IN RESEARCH. BUT THE RESEARCH IS AN ADON. THE CLINICAL CHOICES, I CAN, YOU SHOULD ALWAYS TELL PEOPLE THE CLINICAL CHOICES THEY NEED TO MAKE. BURG THAT THERE IS ALL -- THE MEANINGFUL DECISION MAKING PROCESS CAN BE TAKING PLACE. BUT I DON'T THINK THAT AWAY NEED TO LOCATED RESEARCH TO THAT SAME THING. MAYBE WE SHOULD FLY YOU DO THE UNIVERSITY OF ARKANSAS TO DO THOSE TRIALS. AT LEAST IN THE ACUTE CARE SETTING. THIS IS WHERE I GET OUT OF MY TURF. YOU GET INTO THE SITUATION WHERE YOU'VE GOT AS YOU MENTIONED EARLIER, A RARE DISEASE AND PEOPLE ARE ROUTINELY FLYING ALL OVER THE WORLD TO FIND THE TRIAL THEY WANT TO PARTICIPATE IN. THAT MIGHT BE A LITTLE BIT DIFFERENT. I DON'T KNOW THAT IT'S THE -- I DON'T WORK IN THAT WORLD BUT I DON'T KNOW THAT IT'S THE OBLIGATION OF EVERY RESEARCHER TO TALK ABOUT EVERY TRIAL GOING ON EVERYWHERE IN THE WORLD THAT YOU MAY OR MAY NOT BE ELIGIBLE FOR. BUT I THINK YOU NEED A PLAN. >> SO LET'S SAY THAT THE INSTITUTION DECIDES TO ALTERNATE WEEKS, FOR EXAMPLE. THE FIRST WEEK OF THE MONTH, TRIAL A, SECOND WEEK, TRIAL B. WOULD YOU, AS A CLINICIAN, WOULD YOU FEEL COMFORTABLE IF PATIENTS COME IN THE FIRST WEEK, AND ASK FOR THE OTHER TRIAL WHICH IS -- WHICH YOU DECIDED TO ALTERNATE? WOULD YOU FEEL COMFORTABLE TO TELL HIM, YOU HAVE THE CHOICE OF THIS TRIAL, OR SOME OTHER CARE BY THIS IS OTHER TRIAL IS NOT OPEN NOW. WOULD YOU BE COMFORTABLE? IF THE INSTITUTION DECIDES TO DO THE STRATEGY? >> SO DEPENDS ON WHAT'S MAKING YOU UNCOMFORTABLE. THERE IS -- I THINK THIS IS WHERE I THINK IT'S KIN TO RANDOMIZATION. WE RAN INTO THIS PROBLEM -- IN A LOT OF WAYS FROM BIOLOGICAL ETHICS, RANDOMIZATION AND EQUIPOISE IS THOUGHT A PROBLEM EITHER. WHY IT'S OKAY TO GIVE SOMEBODY ONE TREATMENT AND SOMEBODY ELSE ANOTHER TREATMENT, IT'S NOT A SOLVED PROBLEM OF THE WE FALL BACK ON EQUIPOISE AS THE POSSIBLE SOLUTION. WHEN PEOPLE ARE PARTICIPATING IN TRIALS, ONE OF THE THINGS WE HAVE TO TELL INVESTIGATORS IS THAT THEY HAVE GOT TO REALLY INTERNALIZE THAT. SO YOU'VE GOT TO REALLY BELIEVE AND BE WILLING TO EXCEPT, EVEN IF YOU HOPE THE THERAPY IS BETTER, IT'S NOT THAT YOU'RE GIVING THE BAD OR THE GOOD EXPERIMENTAL THERAPY WHICH INVESTIGATORS ARE USUALLY THINKING. I REALLY CARE ABOUT THIS PERSON. I HOPE THEY GET THE TREATMENT. MOST TRIALS FAIL, RIGHT? IT'S UNUSUAL FOR THE NEW EXPERIMENTAL THERAPY TO BE BETTER THAN THE CONTROL GROUP. SO MOST OF THE TIME IF YOU'RE A BETTER MAN YOU'LL BET ON PLACEBO. IT'S THE WINNING CARD IN CLINICAL TRIALS. SO IF INVESTIGATORS CAN STEP BACK AND MAKE SURE THEY UNDERSTAND THE REASON THEY'RE DOING RESEARCH IS PAUSE THEY DON'T KNOW WHETHER THIS WORKS, THEY SHOULD ALSO BE ABLE TO KNOW THAT THEY DON'T KNOW WHETHER TRIAL A IS A BETTER TRIAL THAN B. THERE SHABBY WHICH POISE BETWEEN TRIALS. SHOULD BE ABLE TO IN GOOD HEART AND GOOD FAITH, ALLOCATE PEOPLE BETWEEN TRIALS AND IF YOU THINK THAT ONE TRIAL IS THAT MUCH MORE IMPORTANT OR BETTER, MORE LIKELY TO SUCCEED, THEN TRIAL -- THE ALTERNATIVE TRIALS I PROBABLY SHOULDN'T DO THE ALTERNATIVE TRIALS. IF YOU REALLY KNOW THAT AN EXPERIMENTAL THERAPY IS GET YOU ARE THAN PLEASE STAND BY, YOU SHOULDN'T BE DOING THAT TRIAL. IF YOU KNOW TRIAL C IS THE GOOD TRIAL, DON'T DO THE OTHER TRIALS BUT YOU DON'T. AND PEOPLE JUST HAVE TO UNDERSTAND. IF THEY THINK THEY DO, THEY DON'T. SO THEY SHOULD BE ABLE DO COMFORTABLY PUT PATIENTS IN WHATEVER TRIAL THE PLAN SAYS THEY SHOULD. THEY SHOULD BE ABLE 250 RANDOMIZE PATIENTS TO WHATEVER ARM THE TRIAL SAYS AND SHOULD BE ABLE TO DO SO IRON GOOD FAITH AND FEEL GOOD ABOUT IT. >> SO I'M FROM THIN AND I DO RARE DISEASE TRIALS. WE HAVE 3 ONGOING TRIALS THAT HAVE COMPETING -- THAT COMPETE FOR THE SAME SMALL PATIENT POPULATION. THAT HAVE OVERLAP. AND I THINK IT'S INTERESTING THAT THE IRB REQUIRES US TO PUT IN OUR CONSENT ABOUT ALTERNATE ACCEPTED CLINICALLY AVAILABLE THERAPIES. BUT WE ACTUALLY TAKE THE APPROACH THAT IF A PATIENT WEBSITE ELIGIBLE FOR MULTIPLE TRIALS, WE ACTUALLY IN THE SAME TIME WE'RE TELLING THEM ABOUT WHAT THE STANDARD OPTIONS WOULD BE, WE ALSO TELL THEM ABOUT WHATEVER TRIALS THEY WOULD BE ELIGIBLE FOR. HOWEVER, WE DON'T JUST GIVE THEM THE CONSENT AND TELL THEM ABOUT THE TRIALS THAT WAY. BECAUSE OFTEN TIMES THERE ARE THINGS THAT ARE QUITE DIFFERENT ABOUT THE TRIAL THAT MIGHT BE MORE ACCEPTABLE FOR A GIVEN PATIENT. AND SO WE ACTUALLY THOUGHT ABOUT THIS A LOT WHEN WE WERE DECIDING THAT WE WOULD DO THESE TRIALS, AND MADE A LITS OF THE THINGS THAT WOULD BE ADVANTAGES AND DISADVANTAGES OF EACH ONE OF THE TRIALS. THAT'S HOW WE PRESENT IT TO THE PATIENTS. I WILL SAY, THOUGH, THAT I WASN'T GOING -- THAT WASN'T WHAT I WAS -- THAT WAS WHAT I WAS GOING TO SAY BUT LISTENING TO WHAT I SAID, I REALLY DISAGREE. WE SOMETIMES KNOW THAT THE TREATMENT WE'RE GIVING IS BETTER. ESPECIALLY IN RARE DISEASE BUT WE'RE DOING THE TRIAL TO GET THAT DRUG APPROVES. SO I DEFINITELY WANT MY PATIENTS TO GET THE DRUG BECAUSE IT'S GIVEN FOR COMPASSIONATE USE OR IN A PHASE TWO AND I KNOW THAT DRUG WORKS AND I KNOW PLACEBO DOESN'T. SO WE -- AND SOMETIMES THAT'S IMPORTANT TO THE PATIENT. SOMETIMES THE FACT THIS DRUG HAS BEEN IN TRIAL FOR ANOTHER DISEASE AND THERE IS EXPERIENCE FOR TEN YEARS, AND THE OTHER DRUG THAT WE'RE USING IS PLANNED NEW, SOMETIMES A PATIENT MAKES A DECISION ON THAT BASIS. WE TRITO GIVE WHATEVER INFORMATION WE HAVE. >> SO I MEAN I THINK THE FIRST PART OF THAT IS PERFECT. SO THAT'S DEFINITELY TRUE, THAT WHAT YOU HAVE IS A PLAN. AND IT'S A PLAN THAT INCONTEMPLATES THE IMPORTANT CHARACTERISTICS FOR THE SITUATIONS THAT YOU'RE IN THE TRIALS YOU'RE IN. I THINK THAT'S EXACTLY RIGHT. THE LATER PART IS NOT HIGH WORLD BUT I'LL TELL YOU, I WOULD HAVE A LOT OF PROBLEMS IF I WAS IN A PART OF MEDICINE WHERE I HAD TO DO SOMETHING THAT I THOUGHT WAS ALREADY PROVEN, AND I WAS JUST DOING IT FOR REGULATORY PURPOSES. IF THE WAS A TRIAL THAT -- RANDOMIZING PEOPLE TO WHAT I THOUGHT THERE WAS PROVE IN ORDER TO SATISFY A REGULATORY BODY AND THERE IS NO ONE ELSE THAT THINKS THAT WAY, I DON'T THINK I'D BE COMFORTABLE. THAT BEING SAID, I'D ARGUE ALSO THAT IN OUR AREA, I'VE DEFINITELY SEEN FDA BE THE HOLD OUT NON BELIEVER IN A THERAPY THAT EACH ELSE BELIEVES IN, AND THINKS THE TRIALS AREN'T ETHICAL. WE DO IT FOR REGULATORY PURPOSES AND PROVE THAT FDA WAS RIGHT AND THEIR ATTITUDE WAS CORRECT AND THEY KNEW BETTER THAN THE COLLECTIVE WISDOM OF THE WHOLE COMMUNITY. [INAUDIBLE QUESTION] >> I THINK THAT'S RIGHT DR. IS A PUNCH OF TRIAL DESIGNS THAT CAN HELP -- IT'S NOT -- AT LEAST ATTENUATE THAT PROBLEM. THIS ARE A LOT OF ADAPTIVE DESIGNS THAT SHOULD BE USED IN RARE DISEASE WHEN YOU HAVE LIMITED SUBJECTS. HARDER TO ARRANGE BUT IN THE SITUATION YOU'RE TALKING ABOUT, A RARE DISEASE, HAVING THE DEPTH OF WILL ZION -- I THINK -- ZION -- I THINK THEY SHOULD BE INCORPORATED INTO A SINGLE TRIAL THAT COULD -- I UNDERSTAND. [INAUDIBLE QUESTION] >> YEAH. I KNOW. BUT FDA WILL. >> SO I WAS CURIOUS ABOUT THE QUESTION ABOUT WHETHER A TRIAL MIGHT HAVE HIGHER PRIORITY FOR THE FIELD AND WHAT YOU DO WHEN THAT IS THE CASE. AND I'M THOUGHT SURE I FULLY UNDERSTOOD HOW YOUR NETWORK OPERATES, AND YOU PROBABLY SIMPLIFIED IT A LOT. BUT IT SOUNDED LIKE AS TRIALS COME ON THEY HAVE TO NEGOTIATE WITH OTHER TRIALS THAT WERE ALREADY IN PLACE WHICH MIGHT GIVE ADVANTAGE DO A TRIAL THAT STARTED EARLIER. BUT I WAS WONDERING IF YOUR NETWORK EVER HAS TO DEAL WITH A SITUATION. IT'S CLEARLY IMPORTANT AND MAYBE IT'S CLEAR 250 AT LEAST THE NETWORK LEADERSHIP THAT IT'S MORE IMPORTANT THAN THE OTHER TRIALS, AND IF THERE IS A WAY TO CHANGE THE ALLOCATION SCHEME THAT KEEPS EVERY ONE HAPPY. >> SO IT'S THE NORM THAT ONE TRIAL IS MORE IMPORTANT THAN ANOTHER. EVERY TRIAL IS MOST IMPORTANT TO PEOPLE DOING IT AND MOST -- IT'S RARE THAT EVERYBODY AGREES ON WHAT'S MOST IMPORTANT. IT'S COMMON THAT EVERYBODY THINKS THAT SOMETHING IS MORE IMPORTANT. SO WE DO LEAVE IT UP TO THE -- SO WE DO NEGOTIATE IT SITE BY SITE. IT'S NEVER TRUE THIS IS THE MOST IMPORTANT TRIAL BECAUSE EVERYBODY ISING TO A DIFFERENT SET OF TRIALS TO COMES FROM. BUT WE DO HAVE SITES THAT AS PART OF THE PLAN THAT THEY PROPOSE, THEY MIGHT GIVE PRIORITY TO SOME TRIALS OVER OTHERS. SO FOR EXAMPLE, THERE IS A SITE THAT I CAN THINK OF THAT SAYS A TRIAL THAT WE ARE ORGANIZING, THAT WE'RE DOING JUST AT OUR SITE AND A COUPLE OTHER SITES, JUST OURS AND WE'RE THE PI ON ANG NIGRANT TO DO THIS TRIAL, THAT'S ALWAYS GOING TO BE THE FIRST HIGHEST PRIORITY FOR US. AND THEN OTHER NIH FUNDED DISEASED IS THE SECOND PRIORITY. AND DRUG COMPANY TRIALS ARE ALWAYS THE THIRD PRIORITY. THAT'S A SCHEME THAT THEY NEED TO WORK WITH WITHIN THEIR INSTITUTION. THE THAT'S OKAY, AS LONG AS WE MAKE SURE THAT NO TRIAL IS INAPPROPRIATELY OR OVER-HARMED BY THAT STRATEGY. AND EACH OF THE TRIALS SIGNS OFF ON THAT. IF THE PARM SOOTICAL COMPANIES FEEL LIKE THEY GET DISSED AND THEY'RE NOT GOING TO DO IT, THEY CAN DROP THAT SITE. IF THEY FEEL THEY'RE NOT GOING TO GET SUBJECTS THEY MIGHT NOT MIND DOING THAT. MOSTLY, IT'S A HIGH PERFORMING SITE THAT CAN MAKE THAT KIND OF NEGOTIATES. I DON'T THINK THAT IT PUTS SUBSEQUENT TRIALS AT DISADVANTAGES. MUST IN. IF YOU'RE THE ONLY TRIAL YOU GET ALL THE PATIENTS, AND THEN OTHER TRIALS COME TO THAT SITE BECAUSE THEY'RE SO PRODUCTIVE, AND THEY NEED TO START DIVVYING PEOPLE UP A LITTLE BIT, THAT MIGHT BE OKAY. SO THE MOST PART, MOST PEOPLE DON'T WANT TO DO TRIALS THAT HAVE LOTS OF OVERLAP. SO MOST OF THE TIME WHAT THEY'RE TRYING TO DO IS GET TRIALS THAT MAP OUT THEIR WHOLE POPULATION. SO WHAT YOU REALLY WANT TO DO IS MAKE SURE YOU HAVE A TRIAL FOR EVERYBODY. SOMETIMES WHEN YOU'RE TRYING TO HAVE A TRIAL FOR EVERYBODY, THERE IS OVERLAP. BETWEEN THEM. AND SO IT'S REALLY MANAGING THE OVERLAP. WHAT THEY LOOK FOR IS A SET OF TRIALS, THAT'S ONLY GOING TO HAVE SMALL OVERLAPS. WHAT DO WE DO ABOUT THOSE OVERLAPS SOME THOSE ARE THE PARTS THAT ARE NEGOTIATED. >> I WANTED TO MAYBE END, I LOVE YOUR CITATION OF JAMES LINDE EXPERIMENT. THIS GOES BACK 250 AMY'S POINT. SO THE PEOPLE WHO DID WELL, TWO PEOPLE GOT LIMES AND LEMONS, TWO GOT CIDER. WHEN LINDE WAS PROPOSING THE EXPERIMENT, ONE OF THE STANDARD TREATMENTS WAS A COURT OF SEA WATER A DAY. VERY SKEPTICAL OF GIVING A QUARTER OF SEA WATER A DAY AND IT TURNED OUT HE WAS RIGHT. BOTH THOSE PEOPLE DIED. AFTER THEY PARTICIPATED IN HIS EXPERIMENT. SO -- OKAY. WELL, THANKS. LET'S THANK ROB VERY MUCH. [APPLAUSE]