1 00:00:11,440 --> 00:00:13,640 Welcome to the Clinical Center Grand Rounds, 2 00:00:13,640 --> 00:00:17,440 a weekly series of educational lectures for physicians and 3 00:00:17,440 --> 00:00:20,080 health care professionals broadcast from the Clinical 4 00:00:20,080 --> 00:00:23,040 Center at the National Institutes of Health in 5 00:00:23,040 --> 00:00:24,840 Bethesda, MD. 6 00:00:24,840 --> 00:00:28,400 The NIH Clinical Center is the world's largest hospital totally 7 00:00:28,400 --> 00:00:32,080 dedicated to investigational research and leads the global 8 00:00:32,080 --> 00:00:35,040 effort in training today's investigators and discovering 9 00:00:35,040 --> 00:00:37,200 tomorrow's cures. 10 00:00:37,200 --> 00:00:46,400 Learn more by visiting us online at http://clinicalcenter.nih.gov 11 00:00:46,400 --> 00:00:56,800 >>OUR SPEAKER IS GEORGE HRIPCSAK 12 00:00:56,800 --> 00:00:59,800 DIRECTOR OF MEDICAL 13 00:00:59,800 --> 00:01:02,120 INFORMATICS SERVICES FOR 14 00:01:02,120 --> 00:01:03,000 NEW YORK PRESBYTERIAN HOSPITAL 15 00:01:03,000 --> 00:01:05,000 COLUMBIA CAMPUS, HE IS A BOARD 16 00:01:05,000 --> 00:01:07,320 CERTIFIED INTERNIST AND HE LEADS 17 00:01:07,320 --> 00:01:09,120 THE OBSERVATIONAL HEALTH DATA 18 00:01:09,120 --> 00:01:10,480 SCIENCES AND INFORMATICS 19 00:01:10,480 --> 00:01:12,200 COORDINATING CENTER AND 20 00:01:12,200 --> 00:01:13,600 INTERNATIONAL NETWORK WITH 21 00:01:13,600 --> 00:01:14,600 THOUSANDS OF COLLABORATORS. 22 00:01:14,600 --> 00:01:17,360 HE EARNED HIS MEDICAL DEGREE 23 00:01:17,360 --> 00:01:20,400 FROM COLUMBIA MEDICAL SCHOOL AND 24 00:01:20,400 --> 00:01:21,960 MASTERS DEGREE IN BIOSTATISTICS 25 00:01:21,960 --> 00:01:23,920 AND THE SCHOOL OF PUBLIC HEALTH. 26 00:01:23,920 --> 00:01:25,640 HIS RESEARCH FOCUSES ON THE 27 00:01:25,640 --> 00:01:26,720 CLINICAL INFORMATION STORE 28 00:01:26,720 --> 00:01:27,400 INDEED ELECTRONIC HEALTH RECORDS 29 00:01:27,400 --> 00:01:29,640 AND ON THE DEVELOPMENT OF NEXT 30 00:01:29,640 --> 00:01:32,920 GENERATION HEALTH RECORDS 31 00:01:32,920 --> 00:01:33,360 SYSTEMS. 32 00:01:33,360 --> 00:01:35,360 USING NONLINEAR TIME SERIES 33 00:01:35,360 --> 00:01:36,600 ANALYSIS, MACHINE LEARNING, 34 00:01:36,600 --> 00:01:38,480 KNOWLEDGE ENGINEERING AND 35 00:01:38,480 --> 00:01:47,760 NATIONAL LANGUAGE PROCESSING, 36 00:01:47,760 --> 00:01:49,560 DR. HRIPC SAK, THE DOCTOR 37 00:01:49,560 --> 00:01:51,800 CHAIRED THE NATIONAL LIBRARY AND 38 00:01:51,800 --> 00:01:53,200 INFORMATICS REVIEW COMMITTEE, HE 39 00:01:53,200 --> 00:01:54,600 IS AN ELECTED MEMBER OF THE 40 00:01:54,600 --> 00:01:56,880 NATIONAL ACADEMY OF MEDICINE AND 41 00:01:56,880 --> 00:01:59,760 AN ELECTED FELLOW OF THE 42 00:01:59,760 --> 00:02:01,160 AMERICAN COLLEGE OF INFORMATICS 43 00:02:01,160 --> 00:02:02,400 IN THE NEW YORK ACADEMY OF 44 00:02:02,400 --> 00:02:04,800 MEDICINE, HE IS ALSO A MEMBER OF 45 00:02:04,800 --> 00:02:06,400 INTERNATIONAL ACADEMY OF HEALTH 46 00:02:06,400 --> 00:02:07,320 SCIENCES INFORMATICS. 47 00:02:07,320 --> 00:02:11,400 HE HAS PUBLISHED OVER 450 48 00:02:11,400 --> 00:02:12,200 PUBLICATIONS. 49 00:02:12,200 --> 00:02:18,880 THE TITLE OF DR. HRIPSAC'S 50 00:02:18,880 --> 00:02:22,560 PRESENTATION IS GREAT TEACHERS 51 00:02:22,560 --> 00:02:23,840 DRAWING REPRODUCIBLE CONCLUSIONS 52 00:02:23,840 --> 00:02:25,120 IN OBSERVATIONAL CLINICAL DATA, 53 00:02:25,120 --> 00:02:26,880 PLEASE WELCOME ME IN JOINING 54 00:02:26,880 --> 00:02:26,960 TODAY'S GREAT TEACHER DR. HRIPSCAK 55 00:02:26,960 --> 00:02:28,480 THANK YOU SO MUCH, AGAIN IT'S AN 56 00:02:28,480 --> 00:02:30,600 HONOR TO BE ASKED TO DO THIS AND 57 00:02:30,600 --> 00:02:31,920 IT'S SUCH FUN TO TALK ABOUT 58 00:02:31,920 --> 00:02:34,600 THIS, SO AS YOU HEARD ALL I'LL 59 00:02:34,600 --> 00:02:37,400 BE TALKING ABOUT DRAWING 60 00:02:37,400 --> 00:02:38,000 REPRODUCIBLE CONCLUSIONS FROM 61 00:02:38,000 --> 00:02:40,560 CLINICAL DATA, NO CONFLICTS OF 62 00:02:40,560 --> 00:02:42,440 INTEREST OR OFFLABEL USE, THE 63 00:02:42,440 --> 00:02:45,360 OBJECTIVES ARE TO IDENTIFY THE 64 00:02:45,360 --> 00:02:46,880 CHALLENGES TO REPRODUCIBILITY 65 00:02:46,880 --> 00:02:47,600 AND OBSERVATIONAL RESEARCH AND 66 00:02:47,600 --> 00:02:50,320 WHAT APPROACHES WE CAN USE TO 67 00:02:50,320 --> 00:02:50,840 ADDRESS THOSE CHALLENGES. 68 00:02:50,840 --> 00:02:53,120 SO WE ALL KNOW THERE ARE 69 00:02:53,120 --> 00:02:55,920 CHALLENGES, HERE'S AN EXAMPLE OF 70 00:02:55,920 --> 00:02:58,200 2 PAPERS PUBLISHED A MONTH 71 00:02:58,200 --> 00:03:02,520 APART, 2 TOP JOURNALS, SAME 72 00:03:02,520 --> 00:03:04,200 QUESTION, DO FOSTER NURSED 73 00:03:04,200 --> 00:03:05,920 FOCUSED ONINATES CAUSE 74 00:03:05,920 --> 00:03:07,680 ESOPHAGEAL CANCER, 1 SAYS YES, 1 75 00:03:07,680 --> 00:03:09,800 SAYS NO BUT THEY USE THE SAME 76 00:03:09,800 --> 00:03:11,160 DATABASE SO IT'S 2 DIFFERENT 77 00:03:11,160 --> 00:03:12,640 GROUP COMING TO THE OPPOSITE 78 00:03:12,640 --> 00:03:13,680 CONCLUSION AT LEAST TO THE 79 00:03:13,680 --> 00:03:16,120 PUBLIC'S EYE THAT TO THE MORE 80 00:03:16,120 --> 00:03:17,000 SOPHISTICATED READER THEY WOULD 81 00:03:17,000 --> 00:03:18,760 LOOK AND SAY, WELL, THE 82 00:03:18,760 --> 00:03:20,880 CONFIDENCE INTERVAL IS NOT 83 00:03:20,880 --> 00:03:21,760 REALLY THAT DIFFERENT, 1 JUST 84 00:03:21,760 --> 00:03:23,200 MADE AND 1 JUST MISSED THE 85 00:03:23,200 --> 00:03:24,400 THRESHOLD BUT STILL THE PUBLIC 86 00:03:24,400 --> 00:03:25,440 DOESN'T KNOW WHAT TO DO WITH 87 00:03:25,440 --> 00:03:29,680 THIS AND THEY READ THAT COFFEE 88 00:03:29,680 --> 00:03:31,760 CAUSES CANCER 1 YIER AND THE 89 00:03:31,760 --> 00:03:33,040 NEXT YEAR CURES IT AND THEY 90 00:03:33,040 --> 00:03:35,000 DON'T KNOW WHAT TO DO WITH 91 00:03:35,000 --> 00:03:36,400 OBSERVATIONAL RESEARCH. 92 00:03:36,400 --> 00:03:37,960 MORE RECENTLY WE'VE SEEN PAPERS 93 00:03:37,960 --> 00:03:39,280 RETRACTED FROM TOP JOURNALS IN 94 00:03:39,280 --> 00:03:42,120 THE SETTING OF COVID-19, AND A 95 00:03:42,120 --> 00:03:44,440 LOT OF IT ABOUT THE DATA THAT 96 00:03:44,440 --> 00:03:47,080 WAS USED AS A SOURCE. 97 00:03:47,080 --> 00:03:50,280 AND IF WE SEE EDITORIALSOT TOPIC 98 00:03:50,280 --> 00:03:52,400 OF OBSERVATIONAL RESEARCH HERE 99 00:03:52,400 --> 00:03:55,200 BY FRANCIS COLLINS, LOOKING AT 100 00:03:55,200 --> 00:03:56,440 BOTH RANDOMIZED TRIALS ARE KNOWN 101 00:03:56,440 --> 00:03:58,040 TO WORK, AND OBSERVATIONAL 102 00:03:58,040 --> 00:04:02,360 RESEARCH ARE REAL WORLD EVIDENCE 103 00:04:02,360 --> 00:04:08,840 OFTEN LEADS TO DISTRACTING 104 00:04:08,840 --> 00:04:09,600 CONCLUSIONS. 105 00:04:09,600 --> 00:04:10,960 SO THE THESIS OF AND I WILL HAVE 106 00:04:10,960 --> 00:04:12,240 A SLIGHTLY DIFFERENT TOPIC BUT 107 00:04:12,240 --> 00:04:13,880 MOST OF THIS WILL BE ABOUT GRAVE 108 00:04:13,880 --> 00:04:16,400 PROBLEMS AND WHAT I'M CALLING 109 00:04:16,400 --> 00:04:17,880 OBSERVATIONAL RESEARCH, DATABASE 110 00:04:17,880 --> 00:04:18,560 RESEARCH, NONEXPERIMENTAL 111 00:04:18,560 --> 00:04:19,920 RESEARCH, THAT IS GETTING DATA 112 00:04:19,920 --> 00:04:22,040 FROM THESE LARGE DATABASES AND 113 00:04:22,040 --> 00:04:22,520 DRAWING CONCLUSIONS. 114 00:04:22,520 --> 00:04:24,200 BUT I BELIEVE THAT MODERN 115 00:04:24,200 --> 00:04:25,800 METHODS AND DATA HAVE THE 116 00:04:25,800 --> 00:04:29,520 POTENTIAL TO CHANGE THE WAY WE 117 00:04:29,520 --> 00:04:31,960 GENERATE EVIDENCE. 118 00:04:31,960 --> 00:04:32,880 OUR CLINICAL GUIDELINES TODAY WE 119 00:04:32,880 --> 00:04:34,960 TRY TO BASE THEM ON RANDOMIZED 120 00:04:34,960 --> 00:04:35,840 TRIALS BUT WE DON'T HAVE THE 121 00:04:35,840 --> 00:04:38,200 TRIAL SO WE BASE THEM MOSTLY ON 122 00:04:38,200 --> 00:04:39,160 EXPERT OPINION OPINION AND I 123 00:04:39,160 --> 00:04:41,040 WILL GIVE YOU AN EXAMPLE OF THAT 124 00:04:41,040 --> 00:04:43,920 LATER AND WE CAN FILL THOSE GAPS 125 00:04:43,920 --> 00:04:45,120 WITH OBSERVATIONAL RESEARCH SO 126 00:04:45,120 --> 00:04:46,480 IT'S NOT ROLELY A QUESTION OF 127 00:04:46,480 --> 00:04:48,880 WHETHER WE DO RANDOMIZED TRIALS 128 00:04:48,880 --> 00:04:50,560 OR OBSERVATIONAL RESEARCH, IT'S 129 00:04:50,560 --> 00:04:52,480 WHETHER WE DO EXPERT OPINION FOR 130 00:04:52,480 --> 00:04:53,320 OBSERVATIONAL RESEARCH AND HOW 131 00:04:53,320 --> 00:04:55,880 CAN WE FILL IN THE GAP WHERE 132 00:04:55,880 --> 00:04:56,800 RANDOMIZED TRIALS CAN DO THE 133 00:04:56,800 --> 00:04:58,760 TASK THIS, IS A FRAMEWORK BY 134 00:04:58,760 --> 00:04:59,760 SOMEONE NAMED PATRICK RYAN WHO'S 135 00:04:59,760 --> 00:05:02,680 STRONG IN THE EFFORT I WILL BE 136 00:05:02,680 --> 00:05:04,200 TALKING ABOUT AND YOU SEE THE 137 00:05:04,200 --> 00:05:05,280 WORDS ON THE LEFT, DON'T WORRY 138 00:05:05,280 --> 00:05:08,760 TOO MUCH WE HAD TO COME UP WITH 139 00:05:08,760 --> 00:05:09,560 LABELS OF REPEATABLE, 140 00:05:09,560 --> 00:05:10,200 REPRODUCIBLE AND THE THING IS 141 00:05:10,200 --> 00:05:11,520 WHEN YOU DO A STUDY, WHAT DOES 142 00:05:11,520 --> 00:05:14,680 IT MEAN FOR IT TO BE RELIABLE, 143 00:05:14,680 --> 00:05:16,080 LFIRST IF THE SAME RESEARCH 144 00:05:16,080 --> 00:05:17,760 USING THE SAME DATA AND THE SAME 145 00:05:17,760 --> 00:05:18,880 ANALYSIS WILL HOPEFULLY GET THE 146 00:05:18,880 --> 00:05:20,600 SAME ANSWER AND THAT'S THE FIRST 147 00:05:20,600 --> 00:05:21,680 ROW OF THE TABLE, THE SECOND ROW 148 00:05:21,680 --> 00:05:23,600 OF THE TABLE IF A DIFFERENT 149 00:05:23,600 --> 00:05:24,720 RESEARCHER DOES THE SAME STUDY 150 00:05:24,720 --> 00:05:28,120 AND SAME DATA THEY WOULD GET THE 151 00:05:28,120 --> 00:05:28,400 SAME ANSWER. 152 00:05:28,400 --> 00:05:29,600 THE THIRD ROW SAYING IF WE 153 00:05:29,600 --> 00:05:31,120 CHANGE THE DATA TO A DIFFERENT 154 00:05:31,120 --> 00:05:32,520 POPULATION BUT WITH SIMILAR 155 00:05:32,520 --> 00:05:33,800 CHARACTERISTICS WE WOULD HOPE TO 156 00:05:33,800 --> 00:05:35,120 GET A SIMILAR OUTCOME. 157 00:05:35,120 --> 00:05:36,680 AND THEN SLIGHTLY DIFFERENT 158 00:05:36,680 --> 00:05:37,120 TOPIC, IF THE STUDIES 159 00:05:37,120 --> 00:05:38,840 GENERALIZE DIFFERENT POPULATIONS 160 00:05:38,840 --> 00:05:42,480 SO WE'RE TALKING 161 00:05:42,480 --> 00:05:43,760 ABOUT OBSERVATIONAL RESEARCH AND 162 00:05:43,760 --> 00:05:45,960 WE'RE TALKING ABOUT CHALLENGES 163 00:05:45,960 --> 00:05:46,400 TO REPRODUCIBILITY. 164 00:05:46,400 --> 00:05:49,080 SO I'M GOING TO LIST A FEW OF 165 00:05:49,080 --> 00:05:51,120 THOSE CHALLENGES, IT USED TO BE 166 00:05:51,120 --> 00:05:52,640 SAMPLE SIZE WAS 1 OF THEM BUT 167 00:05:52,640 --> 00:05:56,320 REALLY WITH OUR LARGE MODERN 168 00:05:56,320 --> 00:05:57,320 DATABASES, THE STATISTICAL ERROR 169 00:05:57,320 --> 00:06:00,120 SCHUNK SO MUCH, WHAT'S LEFT IS 170 00:06:00,120 --> 00:06:00,960 SYSTEMATIC ERROR AND IN OTHER 171 00:06:00,960 --> 00:06:03,320 WORD FIST I DO A STUDY ON A 172 00:06:03,320 --> 00:06:04,600 MODERN DATABASE WITH A HUNDRED 173 00:06:04,600 --> 00:06:11,560 MILLION DOLLAR PATIENTS IN IT, I 174 00:06:11,560 --> 00:06:12,960 WILL GET SMALL CONFERENCE 175 00:06:12,960 --> 00:06:15,000 INTERVALS, AND RPGHT BIASES THAT 176 00:06:15,000 --> 00:06:16,400 ARE INVOLVED THERE, SO I FALSELY 177 00:06:16,400 --> 00:06:18,160 THINK I'M GOING TO GET AN 178 00:06:18,160 --> 00:06:20,360 ACCURATE ANSWER WHETHER IN FACT, 179 00:06:20,360 --> 00:06:21,840 MY SYSTEMATIC ERROR IS JUST AS 180 00:06:21,840 --> 00:06:23,880 BIG AS IT WAS WHEN I HAD A 181 00:06:23,880 --> 00:06:24,760 SMALLER DAT BASE, SO WHAT'S THE 182 00:06:24,760 --> 00:06:26,560 BIG PROBLEM THERE IS CAUSAL 183 00:06:26,560 --> 00:06:30,800 INFERENCE, WHAT'S THE DIFFERENCE 184 00:06:30,800 --> 00:06:33,840 TBEEN AN ASSOCIATION AND ACTUAL 185 00:06:33,840 --> 00:06:34,920 CAUSE, WITHOUT RANDOMIZATION, 186 00:06:34,920 --> 00:06:36,240 CONFOUNDERS CAN LINK TREATMENTS 187 00:06:36,240 --> 00:06:39,960 TO OUTCOMES THAT DON'T BELONG 188 00:06:39,960 --> 00:06:40,400 THERE. 189 00:06:40,400 --> 00:06:42,440 SO IMAGINE THAT I'M STUDYING 190 00:06:42,440 --> 00:06:45,160 BUTIANYL GAS AND DOES BUTIANYL 191 00:06:45,160 --> 00:06:47,040 GAS CAUSE LUNG CANCER, OKAY? 192 00:06:47,040 --> 00:06:49,160 IF I DO A TODAY WITH A 193 00:06:49,160 --> 00:06:50,920 OBSERVATIONAL DATABASE, I MIGHT 194 00:06:50,920 --> 00:06:54,160 LOOK AND SAY WHO'S CARRYING A 195 00:06:54,160 --> 00:06:56,200 BUTIANYL LIGHTER THAT CONTAINS 196 00:06:56,200 --> 00:06:58,760 BUTIANYL GAS, I FOUND A LOT 197 00:06:58,760 --> 00:07:00,280 GETTING LUNG CANCER, I FOUND A 198 00:07:00,280 --> 00:07:02,760 LINK, THEN I REALIZED A LINK, 199 00:07:02,760 --> 00:07:04,360 SMOKING, SMOKING BITE BE THE 200 00:07:04,360 --> 00:07:05,760 REASON SOMEONE'S CARRYING A 201 00:07:05,760 --> 00:07:07,800 BUTIANYL LIGHTER AND THE SMOKING 202 00:07:07,800 --> 00:07:09,560 MAY CAUSE LUNG CANCER SO THAT'S 203 00:07:09,560 --> 00:07:11,280 A CONFOUNDER AND THAT'S OVER A 204 00:07:11,280 --> 00:07:13,520 LOT OFFER WORK GOES INTO HOW TO 205 00:07:13,520 --> 00:07:15,520 HANDLE THAT WITHOUT 206 00:07:15,520 --> 00:07:16,040 RANDOMIZEITIONA. 207 00:07:16,040 --> 00:07:17,800 ALSO, ANOTHER CHALLENGE IS 208 00:07:17,800 --> 00:07:20,560 MEASURING AN ERROR THAT IS DO 209 00:07:20,560 --> 00:07:22,000 ELECTRONIC HEALTH RECORDS 210 00:07:22,000 --> 00:07:23,760 ACCURATELY REPRESENT THE STATUS 211 00:07:23,760 --> 00:07:25,360 OF THE PATIENT. 212 00:07:25,360 --> 00:07:29,760 THERE'S ALSO DECEPTION, OFTEN 213 00:07:29,760 --> 00:07:30,320 SUBCONSCIOUS P-VALUE HACKING 214 00:07:30,320 --> 00:07:32,240 THAT IS WHEN YOU RUN A STUDY YOU 215 00:07:32,240 --> 00:07:33,760 HAVE AN EXPECTATION OF A RESULT, 216 00:07:33,760 --> 00:07:35,440 YOU DON'T GET THE RESULT, YOU 217 00:07:35,440 --> 00:07:37,000 INVESTIGATE TO SEE WHAT'S WRONG 218 00:07:37,000 --> 00:07:38,000 WITH YOUR STUDY, YOU CHANGE YOUR 219 00:07:38,000 --> 00:07:39,560 STUDY UNTIL YOU GET THE ANSWER 220 00:07:39,560 --> 00:07:40,400 YOU'RE EXPECTING AND THEN YOU 221 00:07:40,400 --> 00:07:40,640 STOP. 222 00:07:40,640 --> 00:07:42,760 IF YOU DO YOUR STUDY AND GET THE 223 00:07:42,760 --> 00:07:43,960 ANSWER YOU WERE EXPECTING THEN 224 00:07:43,960 --> 00:07:44,960 YOU STOP DOING YOUR STUDY AS 225 00:07:44,960 --> 00:07:46,920 SOON AS YOU GET THAT ANSWER, SO 226 00:07:46,920 --> 00:07:48,640 YOU ALWAYS GRAVITATE TOWARDS THE 227 00:07:48,640 --> 00:07:55,320 ANSWER YOU WERE EXPECTING. 228 00:07:55,320 --> 00:07:56,320 AND LAST GENERALIZABILITY DOES 229 00:07:56,320 --> 00:07:59,360 IT APPLY TO MY PATIENTS? 230 00:07:59,360 --> 00:08:00,880 SO, I'M GOING TO TALK ABOUT 231 00:08:00,880 --> 00:08:02,400 SPECIFICALLY ABOUT THE 232 00:08:02,400 --> 00:08:05,280 OBSERVATIONAL HEALTH DATA 233 00:08:05,280 --> 00:08:07,240 SCIENCES AND INFORMATICS OR 234 00:08:07,240 --> 00:08:24,800 ODESSY INITIATIVE, THIS IS AN 235 00:08:24,800 --> 00:08:27,200 INITIATIVE--IT'S A GLOBAL 236 00:08:27,200 --> 00:08:29,600 COMMUNITY, WE HAVE OVER 2000 237 00:08:29,600 --> 00:08:30,480 COLLABORATORS FROM STORY TELLERS 238 00:08:30,480 --> 00:08:31,920 COUNTRYINGS WE HAVE A DATA STATE 239 00:08:31,920 --> 00:08:33,880 LOCAL, BUT IN THAT FEDERATED 240 00:08:33,880 --> 00:08:36,600 DATABASE WE HAVE OVER 241 00:08:36,600 --> 00:08:38,920 800 MILLION UNIQUE PATIENTS AND 242 00:08:38,920 --> 00:08:40,360 300 DATABASES OR 300 SITES SWEE 243 00:08:40,360 --> 00:08:42,360 HAVE OVER 10% OF THE WORLD'S 244 00:08:42,360 --> 00:08:45,560 POPULATION IN THIS DATABASE. 245 00:08:45,560 --> 00:08:48,280 AS SINCE ODESSY WAS CREATED IN 246 00:08:48,280 --> 00:08:50,960 2014, WE PUBLISHED 344 PAPERS, 247 00:08:50,960 --> 00:08:52,760 AND WE'VE HAD SPECIFIC 248 00:08:52,760 --> 00:08:54,600 INFLUENCEOT EUROPEAN MEDICINE'S 249 00:08:54,600 --> 00:08:56,320 AGENCY, AND THE FOOD AND DRUG 250 00:08:56,320 --> 00:08:57,280 ADMINISTRATION FOR EXAMPLE, WITH 251 00:08:57,280 --> 00:09:05,160 THE EVIDENCE THAT WE'VE 252 00:09:05,160 --> 00:09:05,440 PRODUCED. 253 00:09:05,440 --> 00:09:07,720 WHY DO WE NEED SUCH LARGE 254 00:09:07,720 --> 00:09:08,000 DATABASES? 255 00:09:08,000 --> 00:09:10,520 WHEN I DO A STUDY AT COLUMBIA, I 256 00:09:10,520 --> 00:09:12,720 MIGHT HAVE A DATABASE WITH A 257 00:09:12,720 --> 00:09:13,400 MILLION PATIENTS, ISN'T THAT 258 00:09:13,400 --> 00:09:14,800 ENOUGH TO DO MY STUDIES, WELL IF 259 00:09:14,800 --> 00:09:16,160 YOU THINK ABOUT IT, ONCE YOU 260 00:09:16,160 --> 00:09:17,560 PICK A DISEASE, AND YOU PICK A 261 00:09:17,560 --> 00:09:19,000 DRUG AND YOU PICK A SIDE EFFECT, 262 00:09:19,000 --> 00:09:20,760 YOU MAY BE DOWN FROM THAT 263 00:09:20,760 --> 00:09:22,760 1 MILLION TO 10 PATIENTS BUT IF 264 00:09:22,760 --> 00:09:24,400 I HAD STARTED WITH A BILLION 265 00:09:24,400 --> 00:09:26,840 PATIENTS THEN I WOULD BE DOWN TO 266 00:09:26,840 --> 00:09:27,800 10,000, I COULD ACTUALLY CARRY 267 00:09:27,800 --> 00:09:31,480 OUT A STUDY AND DO THE KIND OF 268 00:09:31,480 --> 00:09:37,560 ANALYSIS AND DIAGNOSIS THAT 269 00:09:37,560 --> 00:09:39,080 ODESSY DOES. 270 00:09:39,080 --> 00:09:40,280 SO ODESSY GENERATES 3 KINDS OF 271 00:09:40,280 --> 00:09:42,560 EVIDENCE, WE BREAK IT INTO 272 00:09:42,560 --> 00:09:44,480 CLINICAL CHARACTERIZATION AND 273 00:09:44,480 --> 00:09:45,840 THAT SIMPLE TALLYING COUNTING 274 00:09:45,840 --> 00:09:47,200 HOW OFTEN DO THINGS HAPPEN AND 275 00:09:47,200 --> 00:09:49,360 THAT IS SOMETIMES USEFUL, THE 276 00:09:49,360 --> 00:09:52,200 SECOND 1 IS THE MAIN 1 WE THINK 277 00:09:52,200 --> 00:09:54,120 OF AND THAT'S POPULATION LEVEL 278 00:09:54,120 --> 00:09:55,160 ESTIMATION OR CAUSAL INFERENCE, 279 00:09:55,160 --> 00:10:01,960 SO THIS 1 IS LIKE THIS, DOES 280 00:10:01,960 --> 00:10:04,160 METFORMIN CAUSE LACTIC 281 00:10:04,160 --> 00:10:05,120 ADIDOSEIS, AS OPPOSE TO THE 282 00:10:05,120 --> 00:10:06,320 FIRST 1 WHICH IS HOW MANY PEOPLE 283 00:10:06,320 --> 00:10:09,240 HAVE DIABETES AND THE LAST 1 IS 284 00:10:09,240 --> 00:10:10,000 PATIENT LEVEL PREDICTION, THAT 285 00:10:10,000 --> 00:10:12,880 IS WHAT ARE THE ODDS OF A 286 00:10:12,880 --> 00:10:14,400 PATIENT GETTING A SIDE EFFECT 287 00:10:14,400 --> 00:10:20,960 BASED ON THEIR HISTORY. 288 00:10:20,960 --> 00:10:22,160 ODESSY IS AN OPEN SCIENCE 289 00:10:22,160 --> 00:10:24,600 APPROACH WHICH MEANS EVERYTHING 290 00:10:24,600 --> 00:10:28,560 IS OPEN, OPEN SOURCE SOFTWARE, 291 00:10:28,560 --> 00:10:30,440 EVERYTHING WE DO IS SHARED SO 292 00:10:30,440 --> 00:10:31,640 WHEN WE GENERATE A STUDY, THE 293 00:10:31,640 --> 00:10:33,560 FIRST THING WE DO IS SHARE OUR 294 00:10:33,560 --> 00:10:35,960 PROTOCOL ON THE INTERNET, THEN 295 00:10:35,960 --> 00:10:38,360 WE SHARE OUR SOURCE CODE ON THE 296 00:10:38,360 --> 00:10:39,960 INTERNET, WE NEVER SHARE PATIENT 297 00:10:39,960 --> 00:10:41,120 DATA SO THAT'S THE 1 EXCEPTION, 298 00:10:41,120 --> 00:10:42,360 AS WE GENERATE RESULTS WE PUT 299 00:10:42,360 --> 00:10:44,000 THEM ON THE INTERNET, TOO, WE 300 00:10:44,000 --> 00:10:45,320 FIRST SHOW THE DIAGNOSTICS TO 301 00:10:45,320 --> 00:10:47,080 SEE IF WE PASS OUR OWN 302 00:10:47,080 --> 00:10:48,600 DIAGNOSTICS AND THEN WE SHOW THE 303 00:10:48,600 --> 00:10:51,520 RESULTS, IN THIS WAY, BY FIRST 304 00:10:51,520 --> 00:10:52,440 SHARING THE PROTOCOL SO YOU KNOW 305 00:10:52,440 --> 00:10:54,080 WHAT WE'RE GOING TO DO AND 306 00:10:54,080 --> 00:10:55,760 SECOND BY LETTING EVERYONE ELSE 307 00:10:55,760 --> 00:10:57,240 SEE THE RESULTS AS SOON AS WE 308 00:10:57,240 --> 00:11:01,680 DO, THERE'S NO WAY FOR US TO 309 00:11:01,680 --> 00:11:04,240 P-HACK OR TO CHANGE THE SYSTEM 310 00:11:04,240 --> 00:11:06,160 TO GET THE ANSWERS WE WERE 311 00:11:06,160 --> 00:11:06,920 LOOKING FOR. 312 00:11:06,920 --> 00:11:08,560 >> JUST SO YOU KNOW, THE SLIDES 313 00:11:08,560 --> 00:11:09,640 ARE UP. 314 00:11:09,640 --> 00:11:11,720 >> OH GREAT. 315 00:11:11,720 --> 00:11:12,120 THANK YOU. 316 00:11:12,120 --> 00:11:14,120 NOW HOW DO YOU GET--I'M GOING TO 317 00:11:14,120 --> 00:11:16,760 FLIP BACK BECAUSE I LIKE THIS 318 00:11:16,760 --> 00:11:17,840 GRAPH, SO THAT'S OUR COMMUNITY 319 00:11:17,840 --> 00:11:19,680 RIGHT NOW, I JUST WANTED TO SHOW 320 00:11:19,680 --> 00:11:21,880 IT AS A WORLD GRAPH, STRONG IN 321 00:11:21,880 --> 00:11:24,360 NORTH AMERICA AND IN EUROPE, AND 322 00:11:24,360 --> 00:11:26,400 IN ASIA-PACIFIC AND WORKING ON 323 00:11:26,400 --> 00:11:29,240 SOUTH AMERICA AND AFRICA. 324 00:11:29,240 --> 00:11:32,480 WE HAD 1 PERSON IN ANT ARCTICCA 325 00:11:32,480 --> 00:11:33,880 BRIEFLY AND THE NEXT PERSON DID 326 00:11:33,880 --> 00:11:36,760 NOT RETURN OUR CALLS, IT WASSA 327 00:11:36,760 --> 00:11:39,480 SMALL DATABASE SO WE COULD COVER 328 00:11:39,480 --> 00:11:40,600 THIS. 329 00:11:40,600 --> 00:11:43,720 >> NOW MOVING FORWARD, HOW DOES 330 00:11:43,720 --> 00:11:45,160 ODESSY WORK, IS WELL HOW DO YOU 331 00:11:45,160 --> 00:11:46,440 GET 10% OF THE WORLD'S 332 00:11:46,440 --> 00:11:47,680 POPULATION, THE ANSWER IS YOU 333 00:11:47,680 --> 00:11:49,400 DON'T, YOU LEAVE THE DATA LOCAL, 334 00:11:49,400 --> 00:11:52,320 YOU DO HAVE THEM CONVERTED TO A 335 00:11:52,320 --> 00:11:56,560 COMMON DATA MODEL WHICH IS OMOP 336 00:11:56,560 --> 00:11:57,520 WAS BASICALLY OBSERVATIONAL 337 00:11:57,520 --> 00:11:59,120 WORK, YOU CONVERT YOUR DATA TO 338 00:11:59,120 --> 00:12:01,320 OMOP AND THEN WE SHARE THE 339 00:12:01,320 --> 00:12:04,320 QUESTION WITH YOU, YOU RUN THE 340 00:12:04,320 --> 00:12:05,520 QUESTION LOCALLY, IT GENERATES 341 00:12:05,520 --> 00:12:06,800 RESULTS WHICH YOU SHARE 342 00:12:06,800 --> 00:12:08,760 ESSENTIALLY AND THEN WE 343 00:12:08,760 --> 00:12:09,560 COLLABORATE TOGETHER TO 344 00:12:09,560 --> 00:12:10,600 INTERPRET THOSE RESULTS AND 345 00:12:10,600 --> 00:12:12,680 PUBLISH THEM, AND THAT'S WHY 346 00:12:12,680 --> 00:12:17,520 ODESSY PAPERS TEND TO HAVE A LOT 347 00:12:17,520 --> 00:12:18,800 OF AUTHORS. 348 00:12:18,800 --> 00:12:20,800 HERE'S ANOTHER VIEW OF OW ODESSY 349 00:12:20,800 --> 00:12:23,720 WORKS THAT STRETCHES THE 350 00:12:23,720 --> 00:12:24,040 NETWORK. 351 00:12:24,040 --> 00:12:25,120 EACH NOTE CAN GENERATE QUESTIONS 352 00:12:25,120 --> 00:12:26,880 THAT WILL RUN, NOW WE REALIZE 353 00:12:26,880 --> 00:12:28,400 IT'S A VOLUNTARY NETWORK SO IN 354 00:12:28,400 --> 00:12:30,680 NO STUDIES DO WE ACTUALLY HAVE 355 00:12:30,680 --> 00:12:33,800 800 MILLION PATIENTS, WE WILL 356 00:12:33,800 --> 00:12:37,960 FREQUENTLY GET 300-400 MILLION 357 00:12:37,960 --> 00:12:39,120 POPULATION AS DIFFERENT GROUPS 358 00:12:39,120 --> 00:12:40,560 JOIN IN THE STUDY A LOT. 359 00:12:40,560 --> 00:12:42,640 THEN WE JOIN IN, THAT DON'T 360 00:12:42,640 --> 00:12:43,280 NECESSARILY HAVE COLLABORATION 361 00:12:43,280 --> 00:12:45,240 BUT HELP US IN THE METHOD LOGIC 362 00:12:45,240 --> 00:12:46,520 RESEARCH AND GENERATING TOOLS 363 00:12:46,520 --> 00:12:47,600 AND MOST IMPORTANTLY IN 364 00:12:47,600 --> 00:12:50,760 GENERATING STUDIES FOR US TO 365 00:12:50,760 --> 00:12:51,160 CARRY OUT. 366 00:12:51,160 --> 00:12:52,360 THEN WE RUN THESE NETWORK 367 00:12:52,360 --> 00:12:54,760 STUDIES AS I SAID, COLLECT THE 368 00:12:54,760 --> 00:12:56,480 RESULTS AND DISSEMINATE THE 369 00:12:56,480 --> 00:12:56,760 EVIDENCE. 370 00:12:56,760 --> 00:12:58,640 THIS COMMON DATA MODEL IS VERY 371 00:12:58,640 --> 00:12:59,400 IMPORTANT TO US SO JUST LIKE 372 00:12:59,400 --> 00:13:01,120 WHEN YOU HAVE AN APPLIANCE AND 373 00:13:01,120 --> 00:13:02,280 YOU TRAVEL AROUND THE WORLD AND 374 00:13:02,280 --> 00:13:06,760 TRY TO PLUG IT IN, IT DOESN'T 375 00:13:06,760 --> 00:13:08,760 WORK, YOU COULD FIND WAYS TO PUT 376 00:13:08,760 --> 00:13:11,160 IT IN BUT IF YOU HAVE A COMMON 377 00:13:11,160 --> 00:13:13,480 STANDARD THEN SUDDENLY YOU CAN 378 00:13:13,480 --> 00:13:15,520 USE YOUR APPLIANCE ANYWHERE IN 379 00:13:15,520 --> 00:13:20,960 THE SAME WAY THAT THE MODEL IS 380 00:13:20,960 --> 00:13:30,600 THE STANDARD THAT ALLOWS US TO 381 00:13:30,600 --> 00:13:33,600 SHARE--AS IS FIRST OF ALL A DEEP 382 00:13:33,600 --> 00:13:35,040 INFORMATION MODEL SHOWN HERE SO 383 00:13:35,040 --> 00:13:36,600 THIS IS THE DATA SCHEMA, AS YOU 384 00:13:36,600 --> 00:13:38,520 LOOK AT IT, YOU WILL SEE THINGS 385 00:13:38,520 --> 00:13:41,560 YOU RECOGNIZE, DRUG EXPOSURE, 386 00:13:41,560 --> 00:13:42,160 PROCEDURE, DEVICE, MEASUREMENT 387 00:13:42,160 --> 00:13:46,240 WHICH IS THE LAB TEST, CONDITION 388 00:13:46,240 --> 00:13:50,240 OCCURRENTS DIAGNOSIS AND SO ON, 389 00:13:50,240 --> 00:13:52,040 IT'S INTENDED TO BE ACCESSIBLE 390 00:13:52,040 --> 00:13:52,960 TO NEW RESEARCHERS SO THAT CAN 391 00:13:52,960 --> 00:13:54,040 YOU LOOK AT OUR TABLES AND 392 00:13:54,040 --> 00:13:55,560 FIGURE OUT WHAT IT IS YOU NEED 393 00:13:55,560 --> 00:13:57,720 TO DO, OF COURSE YOU STILL NEED 394 00:13:57,720 --> 00:13:59,400 THE DOCUMENTATION, BUT WE MAKE 395 00:13:59,400 --> 00:14:01,520 IT AVAILABLE TO RESEARCHERS. 396 00:14:01,520 --> 00:14:03,640 THEN WE NEED OF COURSE, 397 00:14:03,640 --> 00:14:04,920 EXTENSIVE VOCABULARY SINCE WE'RE 398 00:14:04,920 --> 00:14:06,040 DOING RESEARCH AROUND THE WORMED 399 00:14:06,040 --> 00:14:10,120 THIS, IS 1 IMAGE OF IT BUT 400 00:14:10,120 --> 00:14:11,560 HERE'S A LISTING OF THE 401 00:14:11,560 --> 00:14:13,320 VOCABULARIES AND YOU NOTE WE 402 00:14:13,320 --> 00:14:16,200 HAVE OVER 9 MILLION CONCEPTS 403 00:14:16,200 --> 00:14:17,320 TAKEN FROM 153 VOCABULARIES 404 00:14:17,320 --> 00:14:18,760 FROM,A ROUND THE WORLD. 405 00:14:18,760 --> 00:14:20,880 THESE THINGS GET CONVERTED TO A 406 00:14:20,880 --> 00:14:22,560 STANDARD WHICH IS PRETTY MUCH 407 00:14:22,560 --> 00:14:25,240 THE FORMER MEANINGFUL USE 3 408 00:14:25,240 --> 00:14:27,480 STANDARD SNOW MED, ARC STORM AND 409 00:14:27,480 --> 00:14:31,760 LOINC, WHICH ARE USED FOR ANY 410 00:14:31,760 --> 00:14:32,320 DATABASE IN ANY COUNTRY. 411 00:14:32,320 --> 00:14:35,200 AND FOR THOSE WHO HAVE DONE DATA 412 00:14:35,200 --> 00:14:35,960 CONVERSION BEFORE KNOW, THE FACT 413 00:14:35,960 --> 00:14:38,720 THAT YOU HAVE A COMMON SCHEMA 414 00:14:38,720 --> 00:14:40,720 AND A COMMON VOCABULARY, YOU 415 00:14:40,720 --> 00:14:42,200 GIVE THE SAME DATA TO 2 416 00:14:42,200 --> 00:14:43,880 DIFFERENT PEOPLE AND THEY WILL 417 00:14:43,880 --> 00:14:45,960 COME UP WITH 2 DIFFERENT WAYS CO 418 00:14:45,960 --> 00:14:47,640 CONVERTING IT TO THAT DATA 419 00:14:47,640 --> 00:14:47,840 MODEL. 420 00:14:47,840 --> 00:14:50,200 SO THE IMPORTANT PART OF THE 421 00:14:50,200 --> 00:14:51,680 ODESSY OF CONVENTIONS THAT 422 00:14:51,680 --> 00:14:52,920 EXPLAIN TO YOU, HOW DO YOU 423 00:14:52,920 --> 00:14:54,480 CONVERT YOUR DATA TO THE MODEL, 424 00:14:54,480 --> 00:14:58,240 FOR EXAMPLE, IF SOMEONE DIES, 425 00:14:58,240 --> 00:15:00,120 WHAT'S THEIR DIAGNOSIS, IS IT 426 00:15:00,120 --> 00:15:01,160 THAT THEIR HEART STOPS, IS IT 427 00:15:01,160 --> 00:15:03,520 THE THING THAT OCCURRED A FEW 428 00:15:03,520 --> 00:15:05,000 MINUTES BEFORE THAT PRECIPITATED 429 00:15:05,000 --> 00:15:06,200 THEIR HEART STOPPING OR THE 430 00:15:06,200 --> 00:15:07,160 CHRONIC DISEASE THAT THEY HAD 431 00:15:07,160 --> 00:15:09,120 AND WHERE DO YOU PUT THOSE 432 00:15:09,120 --> 00:15:09,520 DIFFERENT THINGS? 433 00:15:09,520 --> 00:15:11,560 THOSE ARE ALL IN COWER 434 00:15:11,560 --> 00:15:11,880 CONVENTIONS? 435 00:15:11,880 --> 00:15:13,840 AND THEN WE GIVE YOU AMPLE TOOLS 436 00:15:13,840 --> 00:15:19,360 TO CONVERT YOUR DATA TO THE 437 00:15:19,360 --> 00:15:20,800 COMMON DATA MODEL. 438 00:15:20,800 --> 00:15:21,280 SHOWN HERE. 439 00:15:21,280 --> 00:15:22,560 I WON'T GO THROUGH THEM IN 440 00:15:22,560 --> 00:15:22,840 DETAIL. 441 00:15:22,840 --> 00:15:23,920 DATA QUALITY IS CRITICALLY 442 00:15:23,920 --> 00:15:25,160 IMPORTANT AND WE HAVE A 443 00:15:25,160 --> 00:15:27,400 DASHBOARD WITH THOUSANDS OF 444 00:15:27,400 --> 00:15:29,160 CONCESTENCY CHECKS THAT GO 445 00:15:29,160 --> 00:15:30,560 THROUGH THE TABLES AND MAKE SURE 446 00:15:30,560 --> 00:15:32,840 THAT THE DATA PASSES THOSE 447 00:15:32,840 --> 00:15:33,720 CHECKS, OFTEN PEOPLE TELL US 448 00:15:33,720 --> 00:15:35,200 THAT THE MOST IMPORTANT THING 449 00:15:35,200 --> 00:15:36,760 THAT ODESSY BROUGHT THEM WAS NOT 450 00:15:36,760 --> 00:15:38,160 THE RESEARCH BUT THE FACT THAT 451 00:15:38,160 --> 00:15:40,120 THEY FOUND ALL THOSE PROBLEMS IN 452 00:15:40,120 --> 00:15:41,960 THEIR SOURCE TABLES AND END UP 453 00:15:41,960 --> 00:15:44,080 GOING BACK AND FIXING THEIR 454 00:15:44,080 --> 00:15:46,080 DATABASES AND FRY TO DO THE 455 00:15:46,080 --> 00:15:47,160 CONVERSION TO ODESSY, THEY LEARN 456 00:15:47,160 --> 00:15:51,040 THERE ARE PROBLEMS IN THE DATA 457 00:15:51,040 --> 00:15:51,520 THAT THEY HAVE. 458 00:15:51,520 --> 00:15:53,920 THEN WE GIVE TOOLS TO TRAVERSE 459 00:15:53,920 --> 00:15:56,000 THESE 9 MILLION TERMS IN THE 460 00:15:56,000 --> 00:15:57,120 TERMINOLOGY TO FIND THE 461 00:15:57,120 --> 00:15:59,600 PHENOTYPES THAT IS THE CLINICAL 462 00:15:59,600 --> 00:16:00,440 CONDITIONS YOU'RE LOOKING FOR 463 00:16:00,440 --> 00:16:04,320 AND THEN A TOOL TO BUILD IT 464 00:16:04,320 --> 00:16:04,880 WHICH IS SEMIGRAPHICAL SHOWN 465 00:16:04,880 --> 00:16:07,360 THERE ON THE BOTTOM OF THE 466 00:16:07,360 --> 00:16:07,560 SLIDE. 467 00:16:07,560 --> 00:16:11,040 WE HAVE A WHOLE PROCESS FOR WHAT 468 00:16:11,040 --> 00:16:11,560 WE CALL PHENOTYPING. 469 00:16:11,560 --> 00:16:13,040 NOW WHEN I SAY PHENOTYPING HERE, 470 00:16:13,040 --> 00:16:15,360 I MEAN THE WAY THAT CLINICAL 471 00:16:15,360 --> 00:16:16,800 INFORMATICS HAS STOLEN THE TERM 472 00:16:16,800 --> 00:16:19,240 AND IT MEANS CONVERTING RAW DATA 473 00:16:19,240 --> 00:16:20,640 YOU MIGHT FIND IN THE ELECTRONIC 474 00:16:20,640 --> 00:16:21,320 HELT RECORD OR CLAIMS DAILY 475 00:16:21,320 --> 00:16:22,880 BASIS THEA BASE AND TURN IT INTO 476 00:16:22,880 --> 00:16:25,880 A CLINICAL CONCEPT OF USE FOR 477 00:16:25,880 --> 00:16:26,160 RESEARCH. 478 00:16:26,160 --> 00:16:30,680 AND THERE ARE A NUMBER OF TOOLS 479 00:16:30,680 --> 00:16:32,520 LIKE CD AND ATLAS THAT EXPLOIT 480 00:16:32,520 --> 00:16:34,160 INFORMATION IN OUR NETWORK TO 481 00:16:34,160 --> 00:16:35,440 FIGURE OUT WHAT BEST TO CALL 482 00:16:35,440 --> 00:16:38,880 THINGS AND THEN TOOLS, 483 00:16:38,880 --> 00:16:41,480 DIAGNOSTICS AND FEE EVALUATOR TO 484 00:16:41,480 --> 00:16:42,320 SEMIAUTOMATICALLY REVIEW THOSE 485 00:16:42,320 --> 00:16:44,360 PHENOTYPES AND MEASURE OR AT 486 00:16:44,360 --> 00:16:46,000 LEAST ESTIMATE THEIR SENSITIVITY 487 00:16:46,000 --> 00:16:50,320 AND POSITIVE PREDICTIVE VALUE. 488 00:16:50,320 --> 00:16:52,160 AND THEN WE STORE THOSE 489 00:16:52,160 --> 00:16:53,240 PHENOTYPES FOR FUTURE USE AND 490 00:16:53,240 --> 00:16:55,000 THEN TOOLS FOR VISUALIZING AND 491 00:16:55,000 --> 00:16:58,120 TOOLS TO DESIGN MORE COMPLEX 492 00:16:58,120 --> 00:16:59,160 STUDIES, SOME OF THE EXAMPLES OF 493 00:16:59,160 --> 00:17:02,560 WHICH I'LL SHOW IN A LITTLE BIT. 494 00:17:02,560 --> 00:17:04,000 NOW WHILE IESM TALKING ABOUT THE 495 00:17:04,000 --> 00:17:05,880 COMMON DATA MODEL, I WANT TO 496 00:17:05,880 --> 00:17:09,120 MENTION THE HL7 ODESSY 497 00:17:09,120 --> 00:17:09,560 PARTNERSHIP. 498 00:17:09,560 --> 00:17:12,200 HR7 AND HEALTH LEVEL 7 AND 1 OF 499 00:17:12,200 --> 00:17:15,160 THE MOST IMPORTANT CARE STANDARD 500 00:17:15,160 --> 00:17:16,880 ORGANIZATIONS THAT OVER SEE THE 501 00:17:16,880 --> 00:17:19,120 STANDAS WE USE IN HEALTH RECORDS 502 00:17:19,120 --> 00:17:20,520 AND ON MARCH 1st WE ANNOUNCED 503 00:17:20,520 --> 00:17:23,760 A PARTNERSHIP TO BRING OUR 504 00:17:23,760 --> 00:17:24,680 STANDARDS TOGETHER. 505 00:17:24,680 --> 00:17:25,840 WE'RE STILL IN THE CREATION OF 506 00:17:25,840 --> 00:17:27,480 OUR JOINT WORKING GROUPS AND 507 00:17:27,480 --> 00:17:28,560 SCOPES STARTING WITH EXISTING 508 00:17:28,560 --> 00:17:30,120 WORK WE HAD BEEN DOING WITH 509 00:17:30,120 --> 00:17:33,360 PLACES AT GEORGIA TECH AND IN 510 00:17:33,360 --> 00:17:33,560 EUROPE. 511 00:17:33,560 --> 00:17:35,920 THE RANGE OF WHAT WE MIGHT 512 00:17:35,920 --> 00:17:37,120 ACCOMPLISH IS FIRST MAPPING 513 00:17:37,120 --> 00:17:38,600 BETWEEN THE DAT STANDARDS WHICH 514 00:17:38,600 --> 00:17:40,800 IS A BEAR MINIMUM BECAUSE WE'RE 515 00:17:40,800 --> 00:17:43,000 DOING FURTHER LIKE SHARED 516 00:17:43,000 --> 00:17:47,760 KNOWLEDGE ENGINEERING AND COMMON 517 00:17:47,760 --> 00:17:48,040 STANDARDS. 518 00:17:48,040 --> 00:17:48,760 THE INITIAL WORK--WORKING GROUPS 519 00:17:48,760 --> 00:17:50,480 THAT CAME OUT OF THE FIRST 520 00:17:50,480 --> 00:17:53,840 WORKSHOP ARE SHOWN HERE, DATA 521 00:17:53,840 --> 00:17:56,240 HARMONIZATION BETWEEN HL7'S FIRE 522 00:17:56,240 --> 00:18:01,000 STANDARD AND ODESSY'S OMOP 523 00:18:01,000 --> 00:18:02,440 STANDARD AND THEN DIGITAL 524 00:18:02,440 --> 00:18:03,280 QUALITY MEASURES AND THEN KIND 525 00:18:03,280 --> 00:18:05,040 OF A CATCH ALL FOR THINGS OTHER 526 00:18:05,040 --> 00:18:06,720 PEOPLE WERE INTERESTED IN 527 00:18:06,720 --> 00:18:08,360 REGISTRIES, PREDICTIVE MODELS 528 00:18:08,360 --> 00:18:08,840 AND DECISION SUPPORT. 529 00:18:08,840 --> 00:18:10,960 SO NOW I WANT TO MOVE TO TALKING 530 00:18:10,960 --> 00:18:13,040 ABOUT ODESSY IN ACTION, THAT IS 531 00:18:13,040 --> 00:18:14,480 GENERATING EVIDENCE SINCE THAT 532 00:18:14,480 --> 00:18:15,360 IS OUR PURPOSE. 533 00:18:15,360 --> 00:18:18,040 THIS WAS OUR FIRST LARGE STUDY, 534 00:18:18,040 --> 00:18:22,360 WE DID PUBLISHED IN 2016, ON 535 00:18:22,360 --> 00:18:24,200 TREATMENT PATHWAYS, THAT IS WHAT 536 00:18:24,200 --> 00:18:25,800 DRUGS TO PEOPLE ACTUALLY TAKE? 537 00:18:25,800 --> 00:18:27,280 WE KNOW HOW OFTEN DRUGS ARE SOLD 538 00:18:27,280 --> 00:18:30,520 AND WE KNOW IN SOME ACADEMIC 539 00:18:30,520 --> 00:18:48,720 MEDICAL CENTERS MIGHT REPORT ON 540 00:18:48,720 --> 00:18:52,040 WHAT'S--AND SO FORTH WITH MANY 541 00:18:52,040 --> 00:18:54,160 TAKE STOLEDDERS, THIS STUDY HAD 542 00:18:54,160 --> 00:18:55,600 240 MILLION AND UNDERLYING 543 00:18:55,600 --> 00:19:01,240 POPULATION FROM 5 DIFFERENT 544 00:19:01,240 --> 00:19:02,240 COUNTRIES. 545 00:19:02,240 --> 00:19:04,760 IT'S SHOWN IN THE NATIONAL ACAT 546 00:19:04,760 --> 00:19:06,400 ME OF SCIENCES I'LL SO YOU THE 547 00:19:06,400 --> 00:19:08,840 MAIN EVENT AS A IS CIRCLE, THIS 548 00:19:08,840 --> 00:19:10,760 IS A SUN BURST PLOT, THE INNER 549 00:19:10,760 --> 00:19:12,600 CIRCLE IS THE FIRST DRUG THE 550 00:19:12,600 --> 00:19:14,240 PATIENT TOOK, THE SECOND CIRCLE 551 00:19:14,240 --> 00:19:15,560 IS THE SECOND DRUG AND SO ON AS 552 00:19:15,560 --> 00:19:16,760 YOU MAKE YOUR WAY OUT, THERE 553 00:19:16,760 --> 00:19:18,800 COULD BE AS MANY AS 20 CIRCLES 554 00:19:18,800 --> 00:19:20,920 BUT ONCE IT BECOMES A RARE THING 555 00:19:20,920 --> 00:19:22,360 THEN WE STOP SHOWING IT ON THE 556 00:19:22,360 --> 00:19:25,720 SCREEN BECAUSE THE LINE WOULD BE 557 00:19:25,720 --> 00:19:26,760 TOO SMALL. 558 00:19:26,760 --> 00:19:28,200 WHERE THE CIRCLES STOP GOING, 559 00:19:28,200 --> 00:19:31,280 WHERE I HAVE AN ARROW SAYING 560 00:19:31,280 --> 00:19:33,560 ONLY DRUG, THAT'S SO FOR EXAMPLE 561 00:19:33,560 --> 00:19:36,960 IN THIS SLIDE, 29% OF THE 562 00:19:36,960 --> 00:19:38,640 PATIENTS GOT METFORMIN FOR TYPE 563 00:19:38,640 --> 00:19:41,240 2 DIABETES AS THEIR ONLY DRUG. 564 00:19:41,240 --> 00:19:44,800 SEVENTY-FIVE% OF THE PATIENTS 565 00:19:44,800 --> 00:19:46,720 STARTED ON METFORMIN, 29% DID 566 00:19:46,720 --> 00:19:50,760 ONLY METFORMIN IN A 3 YEAR 567 00:19:50,760 --> 00:19:53,840 PERIOD, AND THEN AND THE SECOND 568 00:19:53,840 --> 00:19:55,600 DRUG, THEY'RE SHOWN HERE WITHOUT 569 00:19:55,600 --> 00:19:56,800 A LOT OF--THERE'S PARTICULARLY 570 00:19:56,800 --> 00:20:02,160 NO PREDOMINANT DRUG AFTER THAT, 571 00:20:02,160 --> 00:20:03,960 LET ME TELL YOU WHAT KNOWLEDGES, 572 00:20:03,960 --> 00:20:09,840 DID THEY SWITCH TO A DRUG? 573 00:20:09,840 --> 00:20:14,080 I'VE MIXED AND SWITCHING DRUGS. 574 00:20:14,080 --> 00:20:15,480 THREE DISEASES TYPE 2 DIABETES, 575 00:20:15,480 --> 00:20:18,240 HYPER TENSION AND DEPRESSION. 576 00:20:18,240 --> 00:20:20,680 THE LEFT MOST COLUMN, YOU SEE 577 00:20:20,680 --> 00:20:23,560 CCACE, THAT'S THE US, THAT'S THE 578 00:20:23,560 --> 00:20:25,200 CPRD, THAT'S THE UK, THE BOTTOM 579 00:20:25,200 --> 00:20:29,720 1 IS JAPAN, IF YOU LOOK AT OUR 580 00:20:29,720 --> 00:20:31,240 DWELVE DATA SETS, METFORMIN IS 581 00:20:31,240 --> 00:20:33,040 USED 75% OF THE TIME, THE BOTTOM 582 00:20:33,040 --> 00:20:35,320 1 JAPAN YOU SEE IT'S USED ONLY 583 00:20:35,320 --> 00:20:36,360 ABOUT 25% OF THE TIME. 584 00:20:36,360 --> 00:20:38,360 WE THOUGHT THAT WAS PROBABLY DUE 585 00:20:38,360 --> 00:20:40,040 TO THE FORMULARY AND HOW THE 586 00:20:40,040 --> 00:20:41,960 GOVERNMENT APPROVED THINGS, IT 587 00:20:41,960 --> 00:20:48,160 TURNS OUT SPEAKING TO A 588 00:20:48,160 --> 00:20:50,360 DIABETOLOGYST IN JAPAN THAT IT'S 589 00:20:50,360 --> 00:20:53,520 BELIEVED THAT JAPAN PATIENTS ARE 590 00:20:53,520 --> 00:20:54,640 SLIGHTLY GENETIC RADIOLY 591 00:20:54,640 --> 00:20:57,600 DIFFERENT AND DON'T GET INSULIN 592 00:20:57,600 --> 00:21:00,160 RESISTANCE AND THEREFORE DON'T 593 00:21:00,160 --> 00:21:01,080 USE METFORMIN DRUGS, THEY STILL 594 00:21:01,080 --> 00:21:02,600 USE IT MORE OFTEN THAN OTHER 595 00:21:02,600 --> 00:21:05,040 DRUGS BUT OTHER DRUGS MORE MORE 596 00:21:05,040 --> 00:21:05,360 PREDOMINANT. 597 00:21:05,360 --> 00:21:07,520 SO IN HYPER TENSION THERE'S 598 00:21:07,520 --> 00:21:08,920 STILL AGREEMENT BUT BUT LESS 599 00:21:08,920 --> 00:21:09,280 AGREEMENT. 600 00:21:09,280 --> 00:21:10,200 BUT THERE'S VERY LITTLE 601 00:21:10,200 --> 00:21:11,280 AGREEMENT WHICH MAKES SENSE, WE 602 00:21:11,280 --> 00:21:12,840 DON'T KNOW WHICH DRUGS SHOULD BE 603 00:21:12,840 --> 00:21:15,240 TARGET TO WHICH PATIENTS AND YOU 604 00:21:15,240 --> 00:21:16,760 SEE EVEN THOSE 3 U.S. DATABASES, 605 00:21:16,760 --> 00:21:19,760 WHICH ARE SHOWN ON THE RIGHT, 606 00:21:19,760 --> 00:21:22,320 THERE'S COMPLETE DISAGREEMENT ON 607 00:21:22,320 --> 00:21:23,360 THE TOP DRUG. 608 00:21:23,360 --> 00:21:24,840 AN INTERESTING FINDING IS THAT 609 00:21:24,840 --> 00:21:27,040 25% OF THE HYPER TENSION 610 00:21:27,040 --> 00:21:30,480 PATIENTS HAVE THE UNIQUE PATH OF 611 00:21:30,480 --> 00:21:31,280 DRUGS DESPITE THIS 612 00:21:31,280 --> 00:21:31,880 250 MILLION-DOLLAR POPULATION, 613 00:21:31,880 --> 00:21:35,360 SO WHAT I'M SAYING IN AN 614 00:21:35,360 --> 00:21:35,960 UNDERLYING POPULATION OF 615 00:21:35,960 --> 00:21:37,040 250 MILLION, YOU TAKE EVERYONE 616 00:21:37,040 --> 00:21:38,480 WITH HYPER TENSION, A COUPLE 617 00:21:38,480 --> 00:21:41,520 MILLION AND 25% OF THEM TOOK A 618 00:21:41,520 --> 00:21:43,000 PATH OF DRUGS, HYPER TENSION 619 00:21:43,000 --> 00:21:45,960 DRUGS THAT MATCHED NO 1 ELSE IN 620 00:21:45,960 --> 00:21:47,600 THE DATABASE AND THAT MEANS THAT 621 00:21:47,600 --> 00:21:49,240 WE'RE DOING VERY PERSONALIZED 622 00:21:49,240 --> 00:21:52,000 TREATMENT, ALTHOUGH NOT VERY 623 00:21:52,000 --> 00:21:52,440 PRECISE. 624 00:21:52,440 --> 00:21:54,080 THIS STUDY SHOWED US JUST ON THE 625 00:21:54,080 --> 00:21:55,720 NETWORK RESEARCH SIDE THAT IT'S 626 00:21:55,720 --> 00:21:57,760 FEASIBLE TO ENCODE THE WORLD 627 00:21:57,760 --> 00:21:58,960 POPULATION OVER 10% NOW, I 628 00:21:58,960 --> 00:22:00,680 SHOULD FIX THAT SLIDE, 629 00:22:00,680 --> 00:22:02,280 GENERATING EVIDENCE IS FEASIBLE, 630 00:22:02,280 --> 00:22:04,040 STACK HOLDERS ARE WILLING TO 631 00:22:04,040 --> 00:22:07,600 SHARE RESULTS AND WE'RE ABLE TO 632 00:22:07,600 --> 00:22:08,840 ACCOMMODATE VAST DIFFERENCES IN 633 00:22:08,840 --> 00:22:11,560 PRIVACY AND REGULATION IN THOSE 634 00:22:11,560 --> 00:22:12,040 COUNTRIES. 635 00:22:12,040 --> 00:22:13,320 SO NOW, THAT'S A 636 00:22:13,320 --> 00:22:14,160 CHARACTERIZATION STUDY, NOW 637 00:22:14,160 --> 00:22:16,040 LET'S DO SOME CAUSAL INFERENCE, 638 00:22:16,040 --> 00:22:17,040 THAT IS LET'S GENERATE EVIDENCE 639 00:22:17,040 --> 00:22:19,400 THAT SAYS, WHAT CAUSES WHAT. 640 00:22:19,400 --> 00:22:21,160 HOW DO WE DO THIS? 641 00:22:21,160 --> 00:22:23,280 HOW DO WE ADDRESS THE CHALLENGES 642 00:22:23,280 --> 00:22:25,160 I SHOWED EARLIER, WELL, THROUGH 643 00:22:25,160 --> 00:22:26,400 OPEN SCIENCE, FIRST ADMIT 644 00:22:26,400 --> 00:22:28,200 THERE'S A PROBLEM IN 645 00:22:28,200 --> 00:22:28,520 RILIABILITY. 646 00:22:28,520 --> 00:22:30,440 STUDY IT SCIENTIFICALLY AND DO 647 00:22:30,440 --> 00:22:33,400 RESEARCH INTO NEW METHODS TO 648 00:22:33,400 --> 00:22:35,200 ADDRESS IT. 649 00:22:35,200 --> 00:22:37,640 ODESSY IN ITS PREDECESSOR OMOP 650 00:22:37,640 --> 00:22:39,840 HAVE TAKEN A SCIENTIFIC APPROACH 651 00:22:39,840 --> 00:22:41,040 TO THIS OBSERVATIONAL RESEARCH, 652 00:22:41,040 --> 00:22:42,680 HERER OLD ARMS THAT PRECEDED IT 653 00:22:42,680 --> 00:22:44,680 BUT I WILL SHOW YOU IN THE NEXT 654 00:22:44,680 --> 00:22:46,840 SLIDES SPECIFICALLY WHAT WE'RE 655 00:22:46,840 --> 00:22:47,480 WORKING ON RECENTLY. 656 00:22:47,480 --> 00:22:52,000 SO THIS IS A KEY--THIS IS A 657 00:22:52,000 --> 00:22:53,680 SLIDE TO SHOW YOU TO DEFINE 658 00:22:53,680 --> 00:22:54,920 WHAT'S GOING TO BE ON THE NEXT 659 00:22:54,920 --> 00:22:58,000 SLIDE, SO ON THE X-AXIS IS THE 660 00:22:58,000 --> 00:22:59,280 HAZARD RATIO, 1 MEANS A DRUG HAS 661 00:22:59,280 --> 00:23:02,760 NO EFFECT, ON THE Y-AXIS IS THE 662 00:23:02,760 --> 00:23:04,120 STANDARD ERROR OF THAT ESTIMATE, 663 00:23:04,120 --> 00:23:06,760 THE IMPORTANT THING HERE IS THAT 664 00:23:06,760 --> 00:23:09,160 WE'RE THE PURPLE TRIANGLE UP 665 00:23:09,160 --> 00:23:11,960 TOP, THAT'S NOT STATISTICALLY 666 00:23:11,960 --> 00:23:12,280 SIGNIFICANT. 667 00:23:12,280 --> 00:23:14,080 THE TRIANGLE ON THE RIGHT, THE 668 00:23:14,080 --> 00:23:16,360 BLUE 1 MEANS IT'S STATISTICALLY 669 00:23:16,360 --> 00:23:17,040 SIGNIFICANT AND HARMFUL FOR THIS 670 00:23:17,040 --> 00:23:21,640 SIDE EFFECT AND ON THE LEFT, I 671 00:23:21,640 --> 00:23:24,440 GUESS IT'S TAN, OR GOLD, THAT'S 672 00:23:24,440 --> 00:23:25,560 STATISTICALLY SIGNIFICANT AND 673 00:23:25,560 --> 00:23:27,720 PROTECTIVE AGAINST THAT SIDE 674 00:23:27,720 --> 00:23:27,960 EFFECT. 675 00:23:27,960 --> 00:23:29,800 AND WHAT WE WILL DO IS PLOT 676 00:23:29,800 --> 00:23:34,080 STUDIES FROM THE LITERATURE ON 677 00:23:34,080 --> 00:23:36,200 THIS AXIS AND SEE HOW MANY ARE 678 00:23:36,200 --> 00:23:37,240 STATISTICALLY SIGNIFICANT OR 679 00:23:37,240 --> 00:23:38,400 NOT. 680 00:23:38,400 --> 00:23:39,040 SO WE PARSE 29,000 STUDIES AND 681 00:23:39,040 --> 00:23:41,360 PLOTTED THEM ON THIS FIGURE AND 682 00:23:41,360 --> 00:23:53,400 WHAT WE SEE-- 683 00:23:53,400 --> 00:23:55,480 AND WHAT'S LACKING IS THE 684 00:23:55,480 --> 00:23:57,840 TRIANGLE IN THE MIDDLE WHICH IS 685 00:23:57,840 --> 00:23:58,520 NOT SIGNIFICANT. 686 00:23:58,520 --> 00:24:00,640 NOW WE KNOW ABOUT PUBLICATION 687 00:24:00,640 --> 00:24:02,200 BIAS WHICH IS WHEN AUTHORS MAY 688 00:24:02,200 --> 00:24:05,200 NOT SUBMIT THE WORK THAT'S NOT 689 00:24:05,200 --> 00:24:06,400 STATISTICALLY SIGNIFICANT AND 690 00:24:06,400 --> 00:24:08,200 EDITORS MAY NOT REVIEW IT AND 691 00:24:08,200 --> 00:24:09,960 REVIEWERS MAY NOT ACCEPT IT AS 692 00:24:09,960 --> 00:24:11,600 UNIMPORTANT IF IT SHOWS NO 693 00:24:11,600 --> 00:24:12,600 RESULT, THOSE NEGATIVE RESULTS 694 00:24:12,600 --> 00:24:14,400 ISSUES THE PROBLEM WITH THIS IS 695 00:24:14,400 --> 00:24:15,960 IF WE DON'T KNOW HOW MANY STUDY 696 00:24:15,960 --> 00:24:17,640 VS BEEN DONE BEFORE THIS 697 00:24:17,640 --> 00:24:19,520 POSITIVE 1 WAS FOUND, THEN WE 698 00:24:19,520 --> 00:24:21,400 CAN'T TELL WHETHER THE STUDY IS 699 00:24:21,400 --> 00:24:23,840 ACCURATE OR NOT SO WHAT IT'S 700 00:24:23,840 --> 00:24:24,800 POSITIVE PREDICTIVE VALUE IS, SO 701 00:24:24,800 --> 00:24:26,800 WE DON'T KNOW THE DENOMINATOR OF 702 00:24:26,800 --> 00:24:27,680 NEGATIVE STUDIES AND THEN WE SEE 703 00:24:27,680 --> 00:24:28,960 SOMETHING ELSE ON THIS FIGURE, 704 00:24:28,960 --> 00:24:31,520 YOU'LL NOTICE THAT IT'S A BIT 705 00:24:31,520 --> 00:24:33,360 HIGHLIGHTED WHERE THOSE ARROWS 706 00:24:33,360 --> 00:24:36,160 SHOW ON THE BOTTOM, RIGHT ON THE 707 00:24:36,160 --> 00:24:37,280 OTHER SIDE OF .05. 708 00:24:37,280 --> 00:24:39,160 I MEAN I SUSPECT THAT WHAT WE'RE 709 00:24:39,160 --> 00:24:41,400 SEEING HERE IS THAT WHEN SOMEONE 710 00:24:41,400 --> 00:24:44,280 GETTINGS A STUDY WITH THE 711 00:24:44,280 --> 00:24:46,800 P-VALUE OF SAY .06 CHRKS IS NOT 712 00:24:46,800 --> 00:24:48,280 STATISTICALLY SIGNIFICANT WHICH 713 00:24:48,280 --> 00:24:49,920 MEANS NO EDITOR OR REVIEWER WILL 714 00:24:49,920 --> 00:24:51,160 BE EXCITED ABOUT IT, YOU TELL 715 00:24:51,160 --> 00:24:52,440 THE POST DOC YOU THINK THEY GOT 716 00:24:52,440 --> 00:24:54,880 IT WRONG AND THAT YOU SHOULD 717 00:24:54,880 --> 00:24:56,280 INCLUDE THIS OTHER VARIABLE OR 718 00:24:56,280 --> 00:24:59,000 USE THIS OTHER METHOD AND 719 00:24:59,000 --> 00:25:02,160 SUDDENLY THE P-VALUE BECOMES .04 720 00:25:02,160 --> 00:25:02,880 AND IT'S STATISTICALLY 721 00:25:02,880 --> 00:25:04,760 SIGNIFICANT AND YOU SAY THAT'S 722 00:25:04,760 --> 00:25:06,280 WHAT I INTEND IN THE FIRST PLACE 723 00:25:06,280 --> 00:25:07,800 AND THAT MUST BE RIGHT WHEN YOU 724 00:25:07,800 --> 00:25:09,720 SENT THAT IN TO BE PUBLISHED AND 725 00:25:09,720 --> 00:25:11,600 THAT'S CALLED THE P-VALUE 726 00:25:11,600 --> 00:25:13,200 TACTSIC, NOW THE WAY I EXPLAINED 727 00:25:13,200 --> 00:25:14,800 IT I MADE IT SOUND DECEPTIVE, 728 00:25:14,800 --> 00:25:18,360 BUT IN FACT, IT HAPPENED SUBTLEY 729 00:25:18,360 --> 00:25:19,200 AND SUBCONSCIOUSLY, SO IT'S 730 00:25:19,200 --> 00:25:24,280 SOMETHING TO WATCH OUT FOR. 731 00:25:24,280 --> 00:25:25,800 SO INDIVIDUALS MAY PRODUCE GOOD 732 00:25:25,800 --> 00:25:27,560 RESEARCH STUDIES BUT IF 733 00:25:27,560 --> 00:25:29,520 AGGREGATE THE MEDICAL RESEARCH 734 00:25:29,520 --> 00:25:30,200 SYSTEM, OBSERVATIONAL RESEARCH 735 00:25:30,200 --> 00:25:33,200 SYSTEM IS KIND OF A DATA 736 00:25:33,200 --> 00:25:34,320 DREDGING MACHINE ONLY PUTTING 737 00:25:34,320 --> 00:25:35,600 OUT THE POSITIVE RESULTS, SO NOW 738 00:25:35,600 --> 00:25:37,400 EACH OF THE FOLLOWING SLIDES 739 00:25:37,400 --> 00:25:39,560 WOULD BE AN HOUR LECTURE SO I 740 00:25:39,560 --> 00:25:42,000 WOULD JUST SUMMARIZE ODESSY'S 741 00:25:42,000 --> 00:25:42,920 APPROACH TO ADDRESS 742 00:25:42,920 --> 00:25:43,320 PREPRODUCIBILITY. 743 00:25:43,320 --> 00:25:45,040 FIRST OF ALL I MENTION 744 00:25:45,040 --> 00:25:53,400 CONFOUNDING EARLIER, EXCUSE 745 00:25:53,400 --> 00:25:54,200 ME--WE DON'T HAVE RANDOMIZATION 746 00:25:54,200 --> 00:25:55,880 AS YOU WOULD HAVE IN A 747 00:25:55,880 --> 00:25:57,040 RANDOMIZED CLINICAL TRIAL SO WE 748 00:25:57,040 --> 00:25:58,600 USE SOMETHING THAT HAS BEEN 749 00:25:58,600 --> 00:26:01,800 AROUND SINCE THE LATE 1970S 750 00:26:01,800 --> 00:26:02,600 PROPENSITY SCORE ADJUSTMENT AND 751 00:26:02,600 --> 00:26:04,400 IT ATTEMPTS TO MAKE THE 2 GROUPS 752 00:26:04,400 --> 00:26:06,560 YOU'RE COMPARING SIMILAR TO EACH 753 00:26:06,560 --> 00:26:08,280 OTHER, AND WHAT WE DO THAT'S A 754 00:26:08,280 --> 00:26:10,200 LITTLE BIT DIFFERENT IS THAT 755 00:26:10,200 --> 00:26:11,280 INSTEAD OF SELECTING WHICH 756 00:26:11,280 --> 00:26:13,600 VARIABLES MIGHT BE DOING THE 757 00:26:13,600 --> 00:26:15,000 CONFOUNDING, IN OTHER WORDS IN 758 00:26:15,000 --> 00:26:17,200 THAT STUDY OF BUTIANYL GAS 759 00:26:17,200 --> 00:26:18,920 VERSUS LUNG CANCER, I CAN THINK 760 00:26:18,920 --> 00:26:20,520 WELL I BETTER CHECK IF THERE'S 761 00:26:20,520 --> 00:26:21,960 SMOKING AND CORRECT FOR THAT IN 762 00:26:21,960 --> 00:26:23,240 MY STUDY, WE DO SOMETHING 763 00:26:23,240 --> 00:26:24,840 SLIGHTLY DIFFERENT WHICH IS WE 764 00:26:24,840 --> 00:26:26,160 ACTUALLY TAKE EVERY VARIABLE 765 00:26:26,160 --> 00:26:27,600 WHICH IS GENERALLY 10S OF 766 00:26:27,600 --> 00:26:29,120 THOUSANDS OF VAB VARIABLES IN 767 00:26:29,120 --> 00:26:30,600 THE ELECTRONIC HEALTH RECORD AND 768 00:26:30,600 --> 00:26:33,600 ADJUST FOR ALL OF THEM. 769 00:26:33,600 --> 00:26:36,840 THERE'S A WHOLE PAPER ON WHY 770 00:26:36,840 --> 00:26:38,400 THAT DOESN'T HURT US IN OTHER 771 00:26:38,400 --> 00:26:40,440 WAYS I DON'T WANT TO GET INTO 772 00:26:40,440 --> 00:26:41,640 THE STATISTICAL INFERENCE, THE 773 00:26:41,640 --> 00:26:42,960 DIAGNOSTICS ARE SHOWN THERE, I 774 00:26:42,960 --> 00:26:44,720 WILL GO INTO DIAGNOSTICS LATER 775 00:26:44,720 --> 00:26:49,160 ON, BUT LET ME JUST SAY, THAT 776 00:26:49,160 --> 00:26:51,400 THE GOOD THING HERE IS WE DON'T 777 00:26:51,400 --> 00:26:53,600 HAVE A HUMAN BEING DECIDING 778 00:26:53,600 --> 00:26:55,320 WHICH 20 SHOULD I CONFOUND FOR 779 00:26:55,320 --> 00:26:58,200 SO WHEN HUMAN BEINGS PREDICT 780 00:26:58,200 --> 00:26:59,440 VARIABLES THEY MISS 1 AND 781 00:26:59,440 --> 00:27:00,600 INCLUDE 1 THAT MAYBE SHOULDN'T 782 00:27:00,600 --> 00:27:03,080 BE THERE WHERE IF YOU TAKE A 783 00:27:03,080 --> 00:27:03,800 SYSTEMATIC APPROACH, THEN WE 784 00:27:03,800 --> 00:27:07,800 TAKE OUT THAT VARIABLE. 785 00:27:07,800 --> 00:27:09,600 BUT WE DON'T TRUST THAT SO THIS 786 00:27:09,600 --> 00:27:11,440 NEXT STEP IS TO USE NEGATIVE 787 00:27:11,440 --> 00:27:12,440 CONTROLS AND YOU HEARD OF 788 00:27:12,440 --> 00:27:13,640 NEGATIVE CONTROLS IN STUDIES 789 00:27:13,640 --> 00:27:15,000 WHERE SOMEONE DOES A STUDY AND 790 00:27:15,000 --> 00:27:16,040 THEN PICK SOMETHING THEY KNOW 791 00:27:16,040 --> 00:27:17,240 AND APPROXIMATE THEY BELIEVE NOT 792 00:27:17,240 --> 00:27:18,680 TO BE TRUE AND MAKE SURE IT CAME 793 00:27:18,680 --> 00:27:22,200 OUT NEGATIVE F. YOU DO 1 OR 2 OF 794 00:27:22,200 --> 00:27:23,400 THOSE, THAT'S ANECDOTAL EVIDENCE 795 00:27:23,400 --> 00:27:27,920 BUT WE TAKE 50-100 NEGATIVE 796 00:27:27,920 --> 00:27:30,440 CONTROLS AND SEE IF NOW UNDER 797 00:27:30,440 --> 00:27:31,920 STATISTICS IF YOU HAVE A--IF 798 00:27:31,920 --> 00:27:34,480 YOU'RE LOOKING FOR A P-VALUE OF 799 00:27:34,480 --> 00:27:36,600 .05, THEN ON AVERAGE 95% OF 800 00:27:36,600 --> 00:27:38,440 THEM, OF YOUR NEGATIVE CONTROL 801 00:27:38,440 --> 00:27:40,960 SHOULD BE NEGATIVE, 5% SHOULD BE 802 00:27:40,960 --> 00:27:41,600 MISTAKENLY POSITIVE. 803 00:27:41,600 --> 00:27:43,760 WHAT WE FIND IN TYPICAL STUDIES 804 00:27:43,760 --> 00:27:47,200 THAT LIKE, 40% OF THEM MAY BE 805 00:27:47,200 --> 00:27:48,720 POSITIVE DUE TO RESIDUAL ERROR 806 00:27:48,720 --> 00:27:51,120 IF WE DON'T DO OUR PROPENSITY 807 00:27:51,120 --> 00:27:54,840 SCORES SO WE DOUBLE CHECK FOR 808 00:27:54,840 --> 00:27:58,680 RESIDUAL CONFOUNDING USING 809 00:27:58,680 --> 00:27:59,680 NEGATIVE CONTROLS. 810 00:27:59,680 --> 00:28:02,000 THEN WE ALWAYS USE MULTIPLE 811 00:28:02,000 --> 00:28:04,720 DATABASES, MULTIPLE LOCATIONS, 812 00:28:04,720 --> 00:28:05,800 MULTIPLE PRACTICE TAPES TO LOOK 813 00:28:05,800 --> 00:28:07,400 FOR CONSISTENCY AND ALSO IF 814 00:28:07,400 --> 00:28:09,000 THOSE DATABASES ARE DIFFERENT 815 00:28:09,000 --> 00:28:11,000 FROM EACH OTHER FOR EXAMPLE IF 816 00:28:11,000 --> 00:28:14,840 OLDER GROUP THAT ADDS TO 817 00:28:14,840 --> 00:28:15,200 GENERALLABILITY. 818 00:28:15,200 --> 00:28:16,600 I ALREADY TALKED ABOUT OPEN 819 00:28:16,600 --> 00:28:20,000 LIST, OPEN LIST ARE A 820 00:28:20,000 --> 00:28:22,800 HYPOTHESIS, CODE, PARAMETERS, 821 00:28:22,800 --> 00:28:24,160 RUNS WE PRESPECIFY THE KNOWN 822 00:28:24,160 --> 00:28:25,360 CODES, WE PUBLISH ALL CODES SO 823 00:28:25,360 --> 00:28:28,080 OTHERS CAN RUN IT, WE PUBLISHED 824 00:28:28,080 --> 00:28:28,920 MASKS RESULTS CHECKED 825 00:28:28,920 --> 00:28:31,880 DIAGNOSTICS AND REVEAL OUR 826 00:28:31,880 --> 00:28:32,120 RESULTS. 827 00:28:32,120 --> 00:28:36,320 AND FIEPALLY WE --FINALLY WE 828 00:28:36,320 --> 00:28:39,520 CARRY OUT A LINE TYPE HYPOTHESIS 829 00:28:39,520 --> 00:28:41,760 AT SCALE THAT IS WE GENERALLY 830 00:28:41,760 --> 00:28:43,160 TEST OUT NOT 1 HYPOTHESIS BUT 831 00:28:43,160 --> 00:28:48,040 SEVERAL AT ONCE, NOT NEGTIARAS 832 00:28:48,040 --> 00:28:49,440 CONTROLS HYPOTH WHERE THERE'S A 833 00:28:49,440 --> 00:28:51,000 POSITIVE ANSWER. 834 00:28:51,000 --> 00:28:53,200 WE DO THIS BY CHECKING 835 00:28:53,200 --> 00:29:00,760 CHARACTERISTICS OF OUR ANALYSIS. 836 00:29:00,760 --> 00:29:02,560 --IF I FIND THAT 98% OF THEM ARE 837 00:29:02,560 --> 00:29:05,000 ALSO POSITIVE, THAT WOULD MAKE 838 00:29:05,000 --> 00:29:05,600 ME WORRY. 839 00:29:05,600 --> 00:29:08,120 AND THEN, AGAIN TO EMPHASIZE 840 00:29:08,120 --> 00:29:09,440 THESE DIAGNOSTICS, THAT IS EVERY 841 00:29:09,440 --> 00:29:10,720 TIME WE MAKE AN ASSERTION THAT 842 00:29:10,720 --> 00:29:13,320 WE THINK THE THING IS BALANCED 843 00:29:13,320 --> 00:29:14,720 WE WANT TO SHOW EVIDENCE THAT 844 00:29:14,720 --> 00:29:17,280 IT'S BALANCED, HERE IS A PLOT OF 845 00:29:17,280 --> 00:29:19,440 BALANCE BETWEEN THE 2 DPROWPS, 846 00:29:19,440 --> 00:29:24,000 SAY, THIS MIGHT BE 60,000 CO 847 00:29:24,000 --> 00:29:28,520 VARIANTS AND WE LOOK TO SEE ON 848 00:29:28,520 --> 00:29:31,000 THE X-AXIS WE LOOK FOR THE 849 00:29:31,000 --> 00:29:32,000 PROENSEL PENSITY ADJUSTMENT, AS 850 00:29:32,000 --> 00:29:35,160 CAN YOU SEE THEY DIFFER BY .02, 851 00:29:35,160 --> 00:29:36,840 THAT'S A HIGH DIFFERENCE IN THE 852 00:29:36,840 --> 00:29:37,880 STANDARDIZED DIFFERENCE OF THE 853 00:29:37,880 --> 00:29:38,600 MEAN. 854 00:29:38,600 --> 00:29:40,480 ON THE Y-AXIS AFTER WE DO OUR 855 00:29:40,480 --> 00:29:42,680 ADJUSTMENT, AFTER WE DO OUR 856 00:29:42,680 --> 00:29:45,200 ADJUSTMENT EVERYTHING'S LESS 857 00:29:45,200 --> 00:29:48,080 THAN .02, .025, AND EVERYTHING'S 858 00:29:48,080 --> 00:29:50,200 UNDER .05 WELL UNDER OUR 859 00:29:50,200 --> 00:29:52,120 THRESHOLD OF .1 WHICH WAS NOT 860 00:29:52,120 --> 00:29:53,600 SET BY US BUT BY THE LITERATURE 861 00:29:53,600 --> 00:29:55,440 SO WE DO OUR PROPENSITY 862 00:29:55,440 --> 00:29:57,360 ADJUSTMENT AND WE DO DIAGNOSTIC 863 00:29:57,360 --> 00:29:58,600 TO DOUBLE CHECK IT DID WHAT WE 864 00:29:58,600 --> 00:30:03,400 EXPECT IT TO DO THAT IS BALANCE 865 00:30:03,400 --> 00:30:04,080 THE 2 GROUPS. 866 00:30:04,080 --> 00:30:23,960 AND THIS IS THE NEG--MAYBE I 867 00:30:23,960 --> 00:30:27,440 SHOULD WIDE JUMPSEN THE 868 00:30:27,440 --> 00:30:29,400 CONFERENCE SO THE RATE OF 869 00:30:29,400 --> 00:30:33,280 FALSE-POSITIVES IS CORRECT. 870 00:30:33,280 --> 00:30:35,600 IF I DO MY--THE ANALYSIS I 871 00:30:35,600 --> 00:30:37,320 SHOWED YOU BEFORE BUT IF I DO IT 872 00:30:37,320 --> 00:30:39,520 NOW USING THE ODESSY METHODS WE 873 00:30:39,520 --> 00:30:40,680 GET THE DIAGRAM SHOWN HERE IN 874 00:30:40,680 --> 00:30:42,240 THE MIDDLE FOR MOST OF THE 875 00:30:42,240 --> 00:30:46,760 RESULTS ARE NEGATIVE WHICH IS 876 00:30:46,760 --> 00:30:48,520 WHAT WE WOULD EXPECT AND 877 00:30:48,520 --> 00:30:52,000 COMPARES ARE POSITIVE, WE EXPECT 878 00:30:52,000 --> 00:30:54,040 5% OF THEM ARE THERE BY CHANCE 879 00:30:54,040 --> 00:30:55,000 ALONE WHICH MEANS 6% ARE REAL 880 00:30:55,000 --> 00:30:55,600 SIDE CENTER FOR EXCELLENCE ON 881 00:30:55,600 --> 00:30:57,480 AGINGS IN THIS CASE THIS WAS A 882 00:30:57,480 --> 00:30:58,520 STUDY OF DEPRESSION DRUGS AND 883 00:30:58,520 --> 00:31:00,680 WHAT ARE THEIR SIDE EFFECTS. 884 00:31:00,680 --> 00:31:04,200 WHAT THAT SAYS IS IN OUR IDEAL 885 00:31:04,200 --> 00:31:05,680 STUDY, 55% OF OUR POSITIVE 886 00:31:05,680 --> 00:31:08,560 RESULTS ARE ACTUALLY POSITIVE, 887 00:31:08,560 --> 00:31:10,200 SO NOW LOOKOT LOWER LEFT WHICH 888 00:31:10,200 --> 00:31:11,520 IS THE LOWER LITERATURE, WHEN 889 00:31:11,520 --> 00:31:13,480 YOU HAVE A NEW OBSERVATIONAL 890 00:31:13,480 --> 00:31:16,240 RESEARCH STUDY THAT SHOWS THE 891 00:31:16,240 --> 00:31:17,400 RESULTS, THE ODDS IT'S CORRECT 892 00:31:17,400 --> 00:31:25,160 ARE PROBABLY WAY LESS THAN 55%. 893 00:31:25,160 --> 00:31:27,080 NOW WHEN WE DO THESE STUDIES, 894 00:31:27,080 --> 00:31:29,240 YOU WILL ASK WELL, WE HAVE SO 895 00:31:29,240 --> 00:31:30,680 MANY HYPOTHESIS, HOW DO YOU 896 00:31:30,680 --> 00:31:33,320 CORRECT FOR THAT, ISN'T THAT 897 00:31:33,320 --> 00:31:33,720 DATA DREDGING. 898 00:31:33,720 --> 00:31:34,520 DATA DREDGE SUGGEST NOT ABOUT 899 00:31:34,520 --> 00:31:36,600 WHAT YOU DO BUT WHAT YOU THROW 900 00:31:36,600 --> 00:31:36,800 AWAY. 901 00:31:36,800 --> 00:31:38,600 THE PROBLEM WITH THE LITERATURE 902 00:31:38,600 --> 00:31:40,240 IS THAT WE'VE THROWN AWAY MOST 903 00:31:40,240 --> 00:31:42,600 OF OUR NEGATIVE STUDIES, WE KEEP 904 00:31:42,600 --> 00:31:43,360 OUR NIGGATIVE STUDIES, WHEN YOU 905 00:31:43,360 --> 00:31:45,440 COME TO US AND TEST A 906 00:31:45,440 --> 00:31:47,320 HYPOTHESIS, WE ASK YOU WHAT ARE 907 00:31:47,320 --> 00:31:48,200 YOUR HYPOTHESIS, WHAT DO YOU 908 00:31:48,200 --> 00:31:50,520 WANT TO TEST AND YOU CORRECT FOR 909 00:31:50,520 --> 00:31:51,800 THOSE HYPOTHESIS, WE NEVER GO IN 910 00:31:51,800 --> 00:31:54,840 OUR DAT BASE AND SAY GIVE ME THE 911 00:31:54,840 --> 00:31:56,360 SMALLEST P-VALUE JUST AS I WOULD 912 00:31:56,360 --> 00:31:57,680 NEVER ASK A STUDENT IF I WERE 913 00:31:57,680 --> 00:32:00,400 NEVER INTERESTED IN A CLINICAL 914 00:32:00,400 --> 00:32:02,400 AREA, I WOULD SAY SCAN 915 00:32:02,400 --> 00:32:05,000 LITERATURE FOR THE SMALLEST 916 00:32:05,000 --> 00:32:07,000 P-VALUES I WOULD SAY HERE'S MY 917 00:32:07,000 --> 00:32:08,520 INFORMATION, WHAT'S YOUR BEST 918 00:32:08,520 --> 00:32:10,760 ESTIMATE, SO WE DO THIS WITH 919 00:32:10,760 --> 00:32:11,800 HYPOTHESIS AND WE DON'T SCOUR 920 00:32:11,800 --> 00:32:14,800 THE RESULTS FOR THINGS THAT ARE 921 00:32:14,800 --> 00:32:15,200 PUBLISHABLE. 922 00:32:15,200 --> 00:32:18,200 WE'VE SUMMARIZED THIS WORK IN 923 00:32:18,200 --> 00:32:20,160 10--WHAT WE CALL LEGEND 924 00:32:20,160 --> 00:32:25,200 PRINCIPLES, LEDGE SEND SHORT FOR 925 00:32:25,200 --> 00:32:28,000 LARGE SCALE EVIDENCE GENERATION 926 00:32:28,000 --> 00:32:29,800 AND EVALUATION ACROSS A NETWORK 927 00:32:29,800 --> 00:32:31,440 OF DATABASES, THIS WILL NOT 928 00:32:31,440 --> 00:32:32,520 DEPEND ON ESTIMATED EFFECTS THAT 929 00:32:32,520 --> 00:32:34,640 MEANS WE WILL NONAPOPTOTIC THE 930 00:32:34,640 --> 00:32:35,400 HAVE PUBLICATION BIAS, WE 931 00:32:35,400 --> 00:32:36,440 PUBLISH ALL OUR STUDIES ON THE 932 00:32:36,440 --> 00:32:38,600 INTERNET EVEN IF THE JOURNALS 933 00:32:38,600 --> 00:32:40,040 DON'T WANT THEM. 934 00:32:40,040 --> 00:32:45,600 WE USE A PRESPECIFIED ANALYTIC 935 00:32:45,600 --> 00:32:45,840 DESIGN. 936 00:32:45,840 --> 00:32:47,400 WE DO CONFOUND AND CONTROL 937 00:32:47,400 --> 00:32:49,400 THROUGH A SYSTEMATIC PROCESS, WE 938 00:32:49,400 --> 00:32:51,400 USE BEST PRACTICES, WE USE 939 00:32:51,400 --> 00:32:53,320 NEGATIVE CONTROL AND POSITIVE 940 00:32:53,320 --> 00:32:57,520 CONTROL QUESTIONS, EVERYTHING IS 941 00:32:57,520 --> 00:32:59,600 OPEN SOURCE WE DON'T EVALUATE 942 00:32:59,600 --> 00:33:00,680 NEW METHODS WHILE TESTING 943 00:33:00,680 --> 00:33:03,440 HYPOTHESIS, WE COME UP WITH 944 00:33:03,440 --> 00:33:05,280 METHODS BUT NOT FOR THE SAKE OF 945 00:33:05,280 --> 00:33:05,880 GENERATING CLINICAL EVIDENCE 946 00:33:05,880 --> 00:33:08,400 THAT WE USE FOR PRACTICE, WE GO 947 00:33:08,400 --> 00:33:10,760 ACROSS MULTIPLE DATABASES AND WE 948 00:33:10,760 --> 00:33:11,600 PROTECT CONFIDENTIALITY. 949 00:33:11,600 --> 00:33:13,440 SO HERE'S AN EXAMPLE OF THE 950 00:33:13,440 --> 00:33:15,840 ODESSY LEDGE ENDED HYPER TENSION 951 00:33:15,840 --> 00:33:16,800 STUDY, HERE'S THE 2017 952 00:33:16,800 --> 00:33:18,800 GUIDELINE, A LITTLE LESS THAN 60 953 00:33:18,800 --> 00:33:20,200 DRUGS, HALF ARE FIRST LINE, HALF 954 00:33:20,200 --> 00:33:22,240 ARE SECOND LINE, NOT MUCH 955 00:33:22,240 --> 00:33:24,240 DIFFERENTIATING AMONG THE FIRST 956 00:33:24,240 --> 00:33:26,080 LINE DRUGS THOUGH, AMONG THE 957 00:33:26,080 --> 00:33:26,720 EVIDENCE THAT'S AVAILABLE. 958 00:33:26,720 --> 00:33:28,000 HERE'S THE EVIDENCE THAT'S 959 00:33:28,000 --> 00:33:29,600 AVAILABLE, SO I TAKE THOSE 960 00:33:29,600 --> 00:33:30,480 61st AND SECOND LINE DRUGS AND 961 00:33:30,480 --> 00:33:34,720 PUT THEM IN A CIRCLE, THOSE ARE 962 00:33:34,720 --> 00:33:36,720 THE DOTS AROUND THE CIRCLE, AND 963 00:33:36,720 --> 00:33:39,520 I LINK THEM WITH ARCs WHETHER 964 00:33:39,520 --> 00:33:40,720 WE HAVE RANDOMIZED TRIAL 965 00:33:40,720 --> 00:33:43,560 EVIDENCE, WHAT YOU CAN SEE A 966 00:33:43,560 --> 00:33:45,000 MOSTLY EMPTY INERN CIRCLE AND 967 00:33:45,000 --> 00:33:45,800 RANDOMIZED TRIAL EVIDENCE IS 968 00:33:45,800 --> 00:33:47,920 JUST A SUBSET WHAT HAVE IS IN 969 00:33:47,920 --> 00:33:49,800 THE GUIDELINE, THE REST IS 970 00:33:49,800 --> 00:33:50,560 EXPERT OPINION, IT'S REASONABLE 971 00:33:50,560 --> 00:33:51,800 OPINION BECAUSE IT SAYS THAT 972 00:33:51,800 --> 00:33:54,120 DRUGS ARE IN A CLASS WILL ACT 973 00:33:54,120 --> 00:33:55,600 SIMILARLY, ALTHOUGH IN SOME 974 00:33:55,600 --> 00:33:58,320 CASES WE DON'T HAVE COMPARISONS 975 00:33:58,320 --> 00:34:10,000 BETWEEN 2 CLASSES NEVER MIND 976 00:34:10,000 --> 00:34:11,520 BETWEEN 2 INDIVIDUALS. 977 00:34:11,520 --> 00:34:13,760 --OUTCOMES WE LOOKEDDA THE BOTH 978 00:34:13,760 --> 00:34:15,200 EFFECTIVENESS AND SAFETY, AND IF 979 00:34:15,200 --> 00:34:17,040 I LOOK AT PAIRS OF DRUGS BECAUSE 980 00:34:17,040 --> 00:34:19,280 REMEMBER A LOT OF TIMES PEOPLE 981 00:34:19,280 --> 00:34:20,680 ARE TAKING DUAL THERAPY SO EVERY 982 00:34:20,680 --> 00:34:22,000 PAIR OF DRUGS AGAINST EVERY PAIR 983 00:34:22,000 --> 00:34:24,440 OF DRUGS AGAINST ALL THESE 984 00:34:24,440 --> 00:34:26,880 OUTCOMES, YOU GET MANY BILLIONS 985 00:34:26,880 --> 00:34:28,320 OF STUDIES, BUT MOST PEOPLE 986 00:34:28,320 --> 00:34:29,360 DON'T TAKE MOST COMBINATIONS SO 987 00:34:29,360 --> 00:34:32,640 IT WAS ABOUT HALF A MILLION 988 00:34:32,640 --> 00:34:33,360 HYPOTHESIS WE TESTED. 989 00:34:33,360 --> 00:34:34,640 AND HERE'S SOME OF THE RESULTS 990 00:34:34,640 --> 00:34:38,800 THAT I WILL SUMMARIZE. 991 00:34:38,800 --> 00:34:41,520 ON THE LEFT IS RANDOMIZED 992 00:34:41,520 --> 00:34:43,680 CLINICAL TRIALS AND IT DIDN'T 993 00:34:43,680 --> 00:34:44,280 SHOW DIFFERENCES AMONG THESE 994 00:34:44,280 --> 00:34:47,440 CLASSES ON THE RIGHT IS OUR 995 00:34:47,440 --> 00:34:47,960 LEGEND, DIRECT EVIDENCE. 996 00:34:47,960 --> 00:34:49,760 ON THE RIGHT IS OUR LEGEND 997 00:34:49,760 --> 00:34:51,400 STUDY, NOW THE KEY HERE IS THAT 998 00:34:51,400 --> 00:34:52,600 IF IT'S GREEN, IT MEANS THE 999 00:34:52,600 --> 00:34:54,880 THING ON THE LEFT IS BETTER AND 1000 00:34:54,880 --> 00:34:56,240 IF IT'S PURPLE, THE THING ON THE 1001 00:34:56,240 --> 00:34:58,000 BOTTOM IS BETTER, BUT WE SEE IS 1002 00:34:58,000 --> 00:34:59,600 THAT EVEN CARDIO SELECTIVE BATTA 1003 00:34:59,600 --> 00:35:04,160 BLOCKERS ARE NOT AS GOOD AS THE 1004 00:35:04,160 --> 00:35:05,840 OTHER CLASSINGS AND OUR U.S. 1005 00:35:05,840 --> 00:35:07,400 GUIDELINE SAYS THAT BETA 1006 00:35:07,400 --> 00:35:08,720 BLOCKERS ARE A SECOND LINE 1007 00:35:08,720 --> 00:35:09,680 TREATMENT SO OUR EVIDENCE GOES 1008 00:35:09,680 --> 00:35:11,880 ALONG WITH THE DECISIONS MADE 1009 00:35:11,880 --> 00:35:13,640 FOR THE U.S. GUIDELINE ALTHOUGH 1010 00:35:13,640 --> 00:35:15,600 I POINTED OUT THE EUROPEAN 1011 00:35:15,600 --> 00:35:17,160 GUIDELINE HAS BETA BLOCKERS, I 1012 00:35:17,160 --> 00:35:21,920 THINK HAS BETA BLOCKERS AS FIRST 1013 00:35:21,920 --> 00:35:22,200 LEVEL DRUGS. 1014 00:35:22,200 --> 00:35:28,160 WE ALSO HAVE THIS EFFECT WHERE 1015 00:35:28,160 --> 00:35:30,320 THIACIDES ARE BETTER THAN MY O 1016 00:35:30,320 --> 00:35:31,080 CARDIAL INFARCTION AND WE'RE 1017 00:35:31,080 --> 00:35:32,800 LOOKING INTO THAT FURTHER. 1018 00:35:32,800 --> 00:35:35,280 THOSE ARE CLASSIC BUT WE LOOK AT 1019 00:35:35,280 --> 00:35:36,280 INDIVIDUAL DRUGS, THEY'RE SHOWN 1020 00:35:36,280 --> 00:35:37,760 HERE, THE FIRST LINE DRUGS ARE 1021 00:35:37,760 --> 00:35:39,040 IN THE TOP HALF AND THEN COME 1022 00:35:39,040 --> 00:35:41,800 THE SECOND LINE DRUGS. 1023 00:35:41,800 --> 00:35:44,440 THE GREEN SQUARE ON THE RIGHT 1024 00:35:44,440 --> 00:35:45,760 SAYS THE FIRST LINE DRUG SEEMS 1025 00:35:45,760 --> 00:35:47,520 TO BE ON AVERAGE BETTER THAN THE 1026 00:35:47,520 --> 00:35:48,400 SECOND LINE DRUG SO IT AGREEN 1027 00:35:48,400 --> 00:35:50,840 CELLSS WITH THE GUIDELINE AND 1028 00:35:50,840 --> 00:35:52,520 THE LOWER LEFT, THAT'S THE UPPER 1029 00:35:52,520 --> 00:35:53,200 RIGHT, THAT'S PURPLE AND HE WILL 1030 00:35:53,200 --> 00:35:55,320 THAT SAYS THE SAME THING BECAUSE 1031 00:35:55,320 --> 00:35:57,920 THE QUAR IS SYMMETRIC. 1032 00:35:57,920 --> 00:36:00,000 ON THE UPPER LEFT YOU SEE THE 1033 00:36:00,000 --> 00:36:01,560 GRAY SQUARE AND THAT'S MOSTLY 1034 00:36:01,560 --> 00:36:02,840 NOT MUCH THERE, THERE'S SOME 1035 00:36:02,840 --> 00:36:08,720 PURPLE AND GREEN THAT'S MOSTLY 1036 00:36:08,720 --> 00:36:09,800 DUE TO CAPTOPRIL, AND THEN ON 1037 00:36:09,800 --> 00:36:12,600 THE BOTTOM RIGHT, WE SEE A BUNCH 1038 00:36:12,600 --> 00:36:19,400 OF PURPLE AND GREEN SAYING AMONG 1039 00:36:19,400 --> 00:36:21,800 THE THIS LARGELY AGREES WITH THE 1040 00:36:21,800 --> 00:36:22,440 EVIDENCE AND THE DECISIONS THAT 1041 00:36:22,440 --> 00:36:25,840 WERE MADE IN THE U.S. GUIDELINE. 1042 00:36:25,840 --> 00:36:30,720 ONE SPECIFIC EXAMPLE TO POINT 1043 00:36:30,720 --> 00:36:34,800 OUT IS DIURETICS HIDE O CHLOR O 1044 00:36:34,800 --> 00:36:35,880 THIOSWRIED IS BEING USED AND 1045 00:36:35,880 --> 00:36:37,400 IT'S A DRUG THAT THE GUIDELINE 1046 00:36:37,400 --> 00:36:39,000 FOR EXAMPLE, ALTHOUGH IT GIVES 1047 00:36:39,000 --> 00:36:43,360 BOTH THE FIRST LINE IT DOES THEY 1048 00:36:43,360 --> 00:36:44,400 THAT CHLORTHALIDONE MAY BE 1049 00:36:44,400 --> 00:36:46,000 PREFERRED BUT WE'RE FINDING IS 1050 00:36:46,000 --> 00:36:49,720 THAT THE 2 DRUGS ARE EQUALLY 1051 00:36:49,720 --> 00:36:51,520 EFFECTIVE, LET ME NOT SAY 1052 00:36:51,520 --> 00:36:53,840 EQUALLY EFFECTIVE, THERE'S NO 1053 00:36:53,840 --> 00:36:55,360 DETECTABLE EFFECTIVENESS AND BIG 1054 00:36:55,360 --> 00:37:01,760 DIFFERENCE IN SAFETY, 3 TIMES 1055 00:37:01,760 --> 00:37:05,640 FOR HYPOHYPOCANNED WHAT AND 1056 00:37:05,640 --> 00:37:06,280 INCLUDING KIDNEY, ELECTROLYTES, 1057 00:37:06,280 --> 00:37:13,080 ET CETERA, WE'VE LOOKED INTO 1058 00:37:13,080 --> 00:37:16,360 THIS DEEPLY, WE'VE SO CTD IS A 1059 00:37:16,360 --> 00:37:17,640 STRONGER DIURETIC AND A LONGER 1060 00:37:17,640 --> 00:37:19,000 LASTING 1 SO IT SHOULD BE 1061 00:37:19,000 --> 00:37:19,720 WORKING BETTER AND IT'S MORE 1062 00:37:19,720 --> 00:37:21,320 EFFECT AND I HAVE THAT'S WHY IT 1063 00:37:21,320 --> 00:37:23,000 HAS MORE SIDE EFFECTS BUT WE'RE 1064 00:37:23,000 --> 00:37:26,720 NOT SEEING THE UPSIDE ON THE 1065 00:37:26,720 --> 00:37:27,400 EFFECTIVENESS. 1066 00:37:27,400 --> 00:37:28,360 THAT'S WHY IT'S--I'M SORRY 1067 00:37:28,360 --> 00:37:30,200 THAT'S WHY IT HAS A MOST SAFETY 1068 00:37:30,200 --> 00:37:31,880 PATTERN BECAUSE IT'S MORE 1069 00:37:31,880 --> 00:37:33,200 EFFECTIVE AS A DIARETINAL 1070 00:37:33,200 --> 00:37:35,560 LOCATIONIC THEREFORE IT CAUSES 1071 00:37:35,560 --> 00:37:36,760 ELECTROLIGHTS TO BE PROBLEMS BUT 1072 00:37:36,760 --> 00:37:38,160 WE'RE NOT SEEING A DIFFERENCE IN 1073 00:37:38,160 --> 00:37:39,520 THE ACUTE MY O CARDIAL 1074 00:37:39,520 --> 00:37:40,320 INFARCTIONS AND THINGS LIKE 1075 00:37:40,320 --> 00:37:44,240 THAT, SO THERE'S A COUPLE 1076 00:37:44,240 --> 00:37:46,200 REASONS, 1 IS THAT, LET ME NOT 1077 00:37:46,200 --> 00:37:48,080 GO INTO TOO MUCH DETAIL BUT IT 1078 00:37:48,080 --> 00:37:49,800 SAYS THAT HALF THE PATIENTS STOP 1079 00:37:49,800 --> 00:37:53,600 WITHIN THE FIRST 2 MONTHS 1080 00:37:53,600 --> 00:37:54,600 PERHAPS BECAUSE IT'S NOT 1081 00:37:54,600 --> 00:37:55,600 COMPLETE CONTROL AND A LITTLE 1082 00:37:55,600 --> 00:37:58,320 LATER THAT KNOW THAT, PEOPLE 1083 00:37:58,320 --> 00:38:00,080 START HIDE HCT SO WE'RE LOOKING 1084 00:38:00,080 --> 00:38:01,160 MORE DEEPLY INTO THIS EFFECT 1085 00:38:01,160 --> 00:38:02,760 ALSO, NOW IF I TAKE THE LEGEND 1086 00:38:02,760 --> 00:38:05,040 STUDY I SHOWED YOU AND I PICK 1087 00:38:05,040 --> 00:38:06,360 OUT THE RCTs THAT DO EXIST 1088 00:38:06,360 --> 00:38:07,640 WHICH IS ABOUT 40 OF THEM AND 30 1089 00:38:07,640 --> 00:38:09,760 OF THEM MATCH OUR STUDIES 10 OF 1090 00:38:09,760 --> 00:38:13,400 THEM LIKE WE COULDN'T MATCH BUT 1091 00:38:13,400 --> 00:38:16,200 OF THOSE 30, OUR CIs OVERLAP 1092 00:38:16,200 --> 00:38:19,080 ON THEM, SO 95% OF THE TIME OUR 1093 00:38:19,080 --> 00:38:20,720 LEGEND STUDY MATCHES OUR CTs 1094 00:38:20,720 --> 00:38:22,600 IN 5 OR 6% OF THE TIME WE DON'T 1095 00:38:22,600 --> 00:38:24,320 BUT OF COURSE BY CHANCE ALONE WE 1096 00:38:24,320 --> 00:38:26,800 WOULD EXPECT NOT TO MATCH ON 5 1097 00:38:26,800 --> 00:38:27,960 OR 6% OF THE TIME. 1098 00:38:27,960 --> 00:38:30,920 SO WE HAVE GOOD CONCORDANCE FOR 1099 00:38:30,920 --> 00:38:33,000 RCTs WITH THE THINGS WE'RE 1100 00:38:33,000 --> 00:38:33,320 OVERLAPPING. 1101 00:38:33,320 --> 00:38:37,240 AND THEN IF YOU LOOK AT THIS 1102 00:38:37,240 --> 00:38:38,600 DIAGRAM, THE ODESSY EVIDENCE 1103 00:38:38,600 --> 00:38:41,040 THAT WAS PRODUCED HELPS FILL IN 1104 00:38:41,040 --> 00:38:45,040 THAT CIRCLE ON THE LEFT AND THE 1105 00:38:45,040 --> 00:38:51,400 RESULTS OF THIS STUDY WERE 1106 00:38:51,400 --> 00:38:54,480 PUBLISHED IN JOURNALS, LANCET, 1107 00:38:54,480 --> 00:38:57,000 JOURNAL OF INTERNAL MEDICINE AND 1108 00:38:57,000 --> 00:38:59,040 JAMIA, JUST CHANGING TOPIC FOR A 1109 00:38:59,040 --> 00:39:00,800 MOMENT TO POINT OUT THAT ODESSY 1110 00:39:00,800 --> 00:39:01,520 WORKS THAT THIRD TYPE OF 1111 00:39:01,520 --> 00:39:03,520 EVIDENCE THAT IS PREDICTIVE 1112 00:39:03,520 --> 00:39:07,720 ALGORITHMS THAT PATIENT LEVEL 1113 00:39:07,720 --> 00:39:08,880 PREDICTION, THE MOST IMPORTANT 1114 00:39:08,880 --> 00:39:28,080 THING THERE FOR ODESSY IS I JUST 1115 00:39:28,080 --> 00:39:29,400 WANT TO HIGHLIGHT A FEW THINGS 1116 00:39:29,400 --> 00:39:30,000 WE DID IN COVID. 1117 00:39:30,000 --> 00:39:32,640 THIS IS THE TIMELINE. 1118 00:39:32,640 --> 00:39:35,440 WE HAVE MANY MILLIONS OF COVID 1119 00:39:35,440 --> 00:39:36,400 RECORDS WE PUBLISHED 41 STUDIES 1120 00:39:36,400 --> 00:39:39,040 IN THE PASTIER ON COVID BUT 1121 00:39:39,040 --> 00:39:40,000 PUBLISHING STUDIES IS 1 THING, 1122 00:39:40,000 --> 00:39:42,440 THE BIGGER QUESTION IS WHAT 1123 00:39:42,440 --> 00:39:42,680 IMPACT. 1124 00:39:42,680 --> 00:39:49,000 HERE'S A STUDY ON THE SAFETY OF 1125 00:39:49,000 --> 00:39:49,440 HYDROXYCHLOROQUINE. 1126 00:39:49,440 --> 00:39:50,000 BASICALLY WHEN 1127 00:39:50,000 --> 00:39:51,080 HYDROXYCHLOROQUINE WAS BEING 1128 00:39:51,080 --> 00:39:53,200 CONSIDERED, USED BY BOTH THE 1129 00:39:53,200 --> 00:39:54,360 EUROPEAN MEDICINE'S AGENCY OR 1130 00:39:54,360 --> 00:39:56,600 EMA WHICH IS SHOWN HERE AND THE 1131 00:39:56,600 --> 00:39:58,680 FDA, THEY'RE LOOKING AT BOTH 1132 00:39:58,680 --> 00:39:59,800 SAFETY AND EFFECTIVENESS AND 1133 00:39:59,800 --> 00:40:01,080 ODESSY STUDY IS 1 OF 2 STUDIES 1134 00:40:01,080 --> 00:40:03,400 THAT THE EUROPE MEDICINE'S 1135 00:40:03,400 --> 00:40:04,200 AGENCY HIGHLIGHTED AS THEIR 1136 00:40:04,200 --> 00:40:06,480 EVIDENCE FOR WHY THEY DECIDED TO 1137 00:40:06,480 --> 00:40:08,480 RETRACT SUPPORT FOR USE OF 1138 00:40:08,480 --> 00:40:10,000 HYDROXYCHLOROQUINE IN THE 1139 00:40:10,000 --> 00:40:10,640 SETTING OF COVID. 1140 00:40:10,640 --> 00:40:13,720 SO THAT WAS A STUDY, THE 1141 00:40:13,720 --> 00:40:14,600 EUROPEAN MEDICINE AGENCY, TENDS 1142 00:40:14,600 --> 00:40:15,720 TO TELL YOU WHAT EVIDENCE THEY 1143 00:40:15,720 --> 00:40:17,920 USE TO MAKE THEIR DECISION, SO I 1144 00:40:17,920 --> 00:40:21,040 CAN POINT TO THINGS WHERE THEY 1145 00:40:21,040 --> 00:40:24,280 HIGHLIGHTED ODESSY EVIDENCE. 1146 00:40:24,280 --> 00:40:28,320 A SECOND 1 IS ACE INHIBITORS AND 1147 00:40:28,320 --> 00:40:29,680 ARMS AND SUSCEPTIBILITY, WE WERE 1148 00:40:29,680 --> 00:40:31,120 WORRIED BECAUSE OF THE 1149 00:40:31,120 --> 00:40:32,320 SIMILARITY OR USE OF THE 1150 00:40:32,320 --> 00:40:34,680 CHANNELS THAT THE ACE--TAKING AN 1151 00:40:34,680 --> 00:40:37,240 ACE INHIBITOR OR AN R, ANTIHYPER 1152 00:40:37,240 --> 00:40:39,600 TENSION DRUG, MIGHT MAKE YOU 1153 00:40:39,600 --> 00:40:43,200 MORE SUSCEPTIBLE TO COVID-19 1154 00:40:43,200 --> 00:40:44,320 INFECTION OR COMPLICATIONS, LIKE 1155 00:40:44,320 --> 00:40:45,960 OTHERS WE DID NOT FIND THAT 1156 00:40:45,960 --> 00:40:46,600 EFFECT. 1157 00:40:46,600 --> 00:40:47,720 BUT WHAT I HIGHLIGHT HERE IN THE 1158 00:40:47,720 --> 00:40:50,600 BOTTOM IS THE EUROPEAN 1159 00:40:50,600 --> 00:40:51,840 MEDICINE'S AGENCY AGAIN, 1160 00:40:51,840 --> 00:40:54,000 POINTING OUT THE MORALES STUDY 1161 00:40:54,000 --> 00:40:57,440 WHICH WAS OURS AND USE IT AS AN 1162 00:40:57,440 --> 00:40:58,000 EXEMPLAR OF REPRODUCIBLE 1163 00:40:58,000 --> 00:40:59,120 RESEARCH SO THAT THE POLICY 1164 00:40:59,120 --> 00:41:00,400 MAKERS ARE LOOKING AT THE WAY 1165 00:41:00,400 --> 00:41:01,400 EVIDENCE IS GENERATED AND SEEING 1166 00:41:01,400 --> 00:41:03,920 THIS AS A TEMPLATE FOR HOW 1167 00:41:03,920 --> 00:41:11,480 RESEARCH SHOULD BE CONDUCTED. 1168 00:41:11,480 --> 00:41:13,200 [INDISCERNIBLE] ARE LARGE 1169 00:41:13,200 --> 00:41:17,320 STUDIES ON DRUG SAFETY AND 1170 00:41:17,320 --> 00:41:19,000 PATIENT CHARACTERISTICS. 1171 00:41:19,000 --> 00:41:22,320 SOMETIME AGO WE SWUNG FULLY INTO 1172 00:41:22,320 --> 00:41:23,600 VACCINE SAFETY RESEARCH, IF YOU 1173 00:41:23,600 --> 00:41:29,440 REMEMBER THIS PAST YEAR IN 1174 00:41:29,440 --> 00:41:31,560 MARCH, ASTRAZENECA, WAS HELD IN 1175 00:41:31,560 --> 00:41:33,840 13 COUNTRIES BECAUSE OF FEAR OF 1176 00:41:33,840 --> 00:41:36,200 BLOOD CLOTS THEN ON 1177 00:41:36,200 --> 00:41:38,360 MARCH 18th, THE EMA DETERMINED 1178 00:41:38,360 --> 00:41:40,000 THAT THE BENEFITS OUTWEIGH THE 1179 00:41:40,000 --> 00:41:41,200 RISK THAT YES THERE'S SOME RISK 1180 00:41:41,200 --> 00:41:43,560 OF CLOT BUT LIFE ON EARTH LOWER 1181 00:41:43,560 --> 00:41:45,440 THAT THE RISK OF COVID. 1182 00:41:45,440 --> 00:41:46,840 WELL THAT DETERMINATION WAS 1183 00:41:46,840 --> 00:41:49,320 BASED ON THE ODESSY DATA FOR THE 1184 00:41:49,320 --> 00:41:52,360 BASE LINE RATES SO THIS WAS 1185 00:41:52,360 --> 00:41:53,880 PUBLISHED LATER IN BMJ, AND WE 1186 00:41:53,880 --> 00:41:57,120 GOT A LETTER FROM THE EMA TO 1187 00:41:57,120 --> 00:41:58,760 PUBLISH THE STUDY SO THAT IT 1188 00:41:58,760 --> 00:42:00,200 WOULD INTEREST INTO THE 1189 00:42:00,200 --> 00:42:00,480 LITERATURE. 1190 00:42:00,480 --> 00:42:02,560 WE ALSO WORK STRONGLY WITH THE 1191 00:42:02,560 --> 00:42:05,800 U.S. FDA, THE CENTER FOR 1192 00:42:05,800 --> 00:42:11,520 BIOLOGICS EVALUATION ON RESEARCH 1193 00:42:11,520 --> 00:42:13,040 OR CBER, AND THEN FUN STUFF 1194 00:42:13,040 --> 00:42:15,040 LOOKING IN CHANGES IN POLICY IN 1195 00:42:15,040 --> 00:42:16,680 HOW IT CORRELATES WITH EVIDENCE, 1196 00:42:16,680 --> 00:42:18,440 THAT IS WHAT WAS PUBLISHED AND 1197 00:42:18,440 --> 00:42:20,680 HOW DID USAGE CHANGE AND HOW DID 1198 00:42:20,680 --> 00:42:22,480 POLICY DECISIONS FROM THE 1199 00:42:22,480 --> 00:42:25,120 AGENCIES IN U.S. AND EUROPE 1200 00:42:25,120 --> 00:42:25,360 CHANGE? 1201 00:42:25,360 --> 00:42:33,000 AND THEN SOME WORK ON LONG COVID 1202 00:42:33,000 --> 00:42:34,840 SHOWING HERE OF COURSE THERE'S 1203 00:42:34,840 --> 00:42:37,640 ALSO SAYING, WELL, IT'S HARD TO 1204 00:42:37,640 --> 00:42:39,200 SAY, WHICH SYMPTOMS OCCUR LONG 1205 00:42:39,200 --> 00:42:40,400 AFTER COVID WERE NOT JUST 1206 00:42:40,400 --> 00:42:41,800 BECAUSE THE PATIENT WAS GOING TO 1207 00:42:41,800 --> 00:42:43,400 GET THOSE THINGS BECAUSE OF 1208 00:42:43,400 --> 00:42:45,600 THEIR HIGHER RISK OF GETTING 1209 00:42:45,600 --> 00:42:46,760 COVID OR HAVING SEVERE COVID 1210 00:42:46,760 --> 00:42:47,760 BECAUSE THEY'RE OLDER ANDIC 1211 00:42:47,760 --> 00:42:49,400 ISER, MAYBE THOSE ARE REALLY 1212 00:42:49,400 --> 00:42:51,040 RELATED TO THEIR OTHER DISEASE, 1213 00:42:51,040 --> 00:42:55,800 SO WHAT WE DID IS WE SAID IS 1214 00:42:55,800 --> 00:42:56,920 WHAT'S LONG INFLUENZA, WHAT'S 1215 00:42:56,920 --> 00:42:58,840 LONG COVID, AND WHAT POPS THEN 1216 00:42:58,840 --> 00:43:01,520 AND THEN YOU GET ALTERED TASTE 1217 00:43:01,520 --> 00:43:05,040 AND SMELL, MY O CARDITIS ACUTE 1218 00:43:05,040 --> 00:43:06,200 KIDNEY INJURY, ALOPECIA AND 1219 00:43:06,200 --> 00:43:08,080 THINGS THAT IT BEINGLY SHOW UP 1220 00:43:08,080 --> 00:43:14,960 IN POST COVID RATHER THAN POST 1221 00:43:14,960 --> 00:43:16,800 OTHER VIRALS. 1222 00:43:16,800 --> 00:43:18,200 THIS WAS PUBLISHED IN TOP 1223 00:43:18,200 --> 00:43:19,080 JOURNALS AROUND THE WORLD. 1224 00:43:19,080 --> 00:43:23,080 WE DID WORK WITH THE NATIONAL 1225 00:43:23,080 --> 00:43:24,040 CANCER INSTITUTE UNDERSTANDING 1226 00:43:24,040 --> 00:43:25,440 THE SEQUENCE, I WILL GO THROUGH 1227 00:43:25,440 --> 00:43:27,320 THIS QUICKLY, IT'S BASICALLY A 1228 00:43:27,320 --> 00:43:29,520 MEASUREMENT STUDY TO SAY COULD 1229 00:43:29,520 --> 00:43:32,800 ODESSY WORK DO WORK IN CANCER MY 1230 00:43:32,800 --> 00:43:34,480 POST DOC WAS WORKING ON THIS, HE 1231 00:43:34,480 --> 00:43:37,040 FIRST REPEAT THAD TREATMENT 1232 00:43:37,040 --> 00:43:38,120 PATHWAYS PAPER I SHOW INDEED 1233 00:43:38,120 --> 00:43:39,560 CANCER AND SHOWED THAT PATIENTS 1234 00:43:39,560 --> 00:43:40,760 WITH CANCER GET A SIMILAR 1235 00:43:40,760 --> 00:43:45,600 SEQUENCE OF DRUGS TO PATIENTS 1236 00:43:45,600 --> 00:43:46,960 WITHOUT CANCER. 1237 00:43:46,960 --> 00:43:48,920 WE SHOWED HIGH SENSITIVITY AND 1238 00:43:48,920 --> 00:43:50,360 SPECIFICITY IN SAYING DO YOU 1239 00:43:50,360 --> 00:43:52,840 HAVE A CANCER DIAGNOSIS AND DID 1240 00:43:52,840 --> 00:43:54,400 YOU GET A PARTICULAR CANCER 1241 00:43:54,400 --> 00:43:56,160 TREATMENT ALSO HIGH SENSITIVITY 1242 00:43:56,160 --> 00:43:56,720 AND SPECIFICITY. 1243 00:43:56,720 --> 00:43:58,520 WE LOOKED AT THINGS LIKE CANCER 1244 00:43:58,520 --> 00:44:02,080 TREATMENTS OVERTIME, THIS IS FOR 1245 00:44:02,080 --> 00:44:03,600 PROSTATE CANCER SHOWING, FOR GET 1246 00:44:03,600 --> 00:44:06,080 ABOUT THE LEFT HAND OF THE 1247 00:44:06,080 --> 00:44:08,920 FIGURE, LOOK THE LEFT HAND, BUT 1248 00:44:08,920 --> 00:44:11,360 ON THE RIGHT HAND PROSECTOMYOSIN 1249 00:44:11,360 --> 00:44:13,560 ME, DECREASING OVER TIME, 1250 00:44:13,560 --> 00:44:14,400 CREATION THERAPY, STAYING ABOUT 1251 00:44:14,400 --> 00:44:16,840 THE SAME OR COMING UP AT THE 1252 00:44:16,840 --> 00:44:18,160 END, HORMONE THERAPY RISING AND 1253 00:44:18,160 --> 00:44:20,360 OTHER FORMS OF CHEMO THERAPY 1254 00:44:20,360 --> 00:44:22,520 FALLING OVER TIME SO OUR ODESSY 1255 00:44:22,520 --> 00:44:26,600 ONCOLOGY WORKING GROUP FORMED IN 1256 00:44:26,600 --> 00:44:37,360 2017 IS EXTENDING--BECAUSE THE 1257 00:44:37,360 --> 00:44:39,720 COMPLEXITY OF THE PHENOTYPE AND 1258 00:44:39,720 --> 00:44:41,000 THE TREATMENT PATHWAYS SO YOU 1259 00:44:41,000 --> 00:44:43,200 NEED A DATA MODEL AND YOU NEED 1260 00:44:43,200 --> 00:44:44,280 ANALYTIC TOOLS THAT CAN HANDLE 1261 00:44:44,280 --> 00:44:45,400 THAT AND THAT'S WHAT THAT 1262 00:44:45,400 --> 00:44:51,120 WORKING GROUP IS WORKING ON. 1263 00:44:51,120 --> 00:44:54,200 AND JUST TO POINT OUT THIS WORK 1264 00:44:54,200 --> 00:44:56,400 ON CANCER IN THE SETTING OF 1265 00:44:56,400 --> 00:44:59,040 COVID-19, WHAT YOU SEE HERE, 1266 00:44:59,040 --> 00:45:00,360 THESE GRAY FIGURURES, I WILL 1267 00:45:00,360 --> 00:45:02,440 JUST SAY QUICKLY THAT THEY LOOK 1268 00:45:02,440 --> 00:45:03,400 PRETTY MUCH SYMMETRIC WHICH 1269 00:45:03,400 --> 00:45:04,720 MEANS IN THE SETTING OF COVID 1270 00:45:04,720 --> 00:45:07,840 AND THE SETTING OF INFLUENZA, 1271 00:45:07,840 --> 00:45:09,000 CANCER PATIENTS ARE NO DIFFERENT 1272 00:45:09,000 --> 00:45:11,240 THAN OTHER PEOPLE IN OTHER WORDS 1273 00:45:11,240 --> 00:45:13,640 IT DOESN'T SEEM THAT COVID IS 1274 00:45:13,640 --> 00:45:14,360 PARTICULARLY HURTING CANCER 1275 00:45:14,360 --> 00:45:20,200 ANYMORE THAN IT HURTS--ANYMORE 1276 00:45:20,200 --> 00:45:20,840 THAN INFLUENZA DOES. 1277 00:45:20,840 --> 00:45:23,200 AND 1 EFFORT THAT IS GETTING 1278 00:45:23,200 --> 00:45:25,360 CLOSER TO CLINICAL CARE IS THE 1279 00:45:25,360 --> 00:45:27,000 DAILY BASIS THEA CONSULT 1280 00:45:27,000 --> 00:45:28,560 SERVICE, THANK YOU TO STANFORD'S 1281 00:45:28,560 --> 00:45:30,400 GREEN BUTTON TEAM FOR INSIGHTS 1282 00:45:30,400 --> 00:45:33,600 WHO PRODUCED A SIMILAR SERVICE 1283 00:45:33,600 --> 00:45:35,840 EARLIER [INDISCERNIBLE] IS THE 1284 00:45:35,840 --> 00:45:37,360 PERSON AT COLUMBIA WHO DID THIS. 1285 00:45:37,360 --> 00:45:39,160 BASICALLY YOU GO ON A TEAM, YOU 1286 00:45:39,160 --> 00:45:41,480 LOOK IF THE LITERATURE, IF IT'S 1287 00:45:41,480 --> 00:45:43,240 NOT THERE, YOU GENERATE IT IN A 1288 00:45:43,240 --> 00:45:45,920 DAY ON THE ODESSY BASIS, SOMEONE 1289 00:45:45,920 --> 00:45:48,160 LIKE A PATIENT WITH PENICILLIN 1290 00:45:48,160 --> 00:45:51,000 HAD ANGIO DEMA AND THE QUESTION 1291 00:45:51,000 --> 00:45:53,880 WAS DOES PENICILLIN TYPICALLY 1292 00:45:53,880 --> 00:45:56,280 CAUSE ANGIO DEMA, THE ANSWER WAS 1293 00:45:56,280 --> 00:45:58,200 NO, BUT WAS IT A SIDE EFFECT 1294 00:45:58,200 --> 00:45:59,880 THEY SHOULD HAVE KNOWN ABOUT? 1295 00:45:59,880 --> 00:46:01,200 TALKING ABOUT ODESSY 1296 00:46:01,200 --> 00:46:04,960 COLLABORATIONS AS WE WORK WITH 1297 00:46:04,960 --> 00:46:06,480 THE FOOD AND DRUG ADMINISTRATION 1298 00:46:06,480 --> 00:46:08,320 CBER AND IT SERVES AS A CONVENER 1299 00:46:08,320 --> 00:46:10,440 FOR ACTIVITIES THERE IN 1300 00:46:10,440 --> 00:46:15,600 BIOLOGICS, AND THE ALL OF US 1301 00:46:15,600 --> 00:46:17,880 RESEARCH PROGRAM AND A MODEL AS 1302 00:46:17,880 --> 00:46:21,080 THE BASIS OF THAT AND WE WORK ON 1303 00:46:21,080 --> 00:46:22,000 PROCESSING THAT DATABASE, 1304 00:46:22,000 --> 00:46:27,800 COLLECTING THE DATA AND MAKING 1305 00:46:27,800 --> 00:46:29,560 IT RELIABLE DATABASE. 1306 00:46:29,560 --> 00:46:31,640 THE NIH'S EMERGE INITIATIVE, 1307 00:46:31,640 --> 00:46:33,480 OMOP BEING A COMMON DATA MODEL 1308 00:46:33,480 --> 00:46:37,200 FOR THAT 1 TOO AND COLUMBIA 1309 00:46:37,200 --> 00:46:38,840 PARTICIPATE NOTHING THAT, AND N3 1310 00:46:38,840 --> 00:46:40,520 C, THE NATIONAL COHORT 1311 00:46:40,520 --> 00:46:41,920 COLLABORATIVE, BEING THE BASIS 1312 00:46:41,920 --> 00:46:45,840 OF THAT, ALSO IN EUROPE, THE 1313 00:46:45,840 --> 00:46:49,160 EATON INITIATIVE, SO THE LARGE 1314 00:46:49,160 --> 00:46:51,000 EUROPEAN NETWORK USING OMOP AND 1315 00:46:51,000 --> 00:46:54,160 FOR EXAMPLE, COUNTRIES LIKE 1316 00:46:54,160 --> 00:46:56,200 KOREA, PICKING OMOP AS THE BASIS 1317 00:46:56,200 --> 00:47:00,840 OF ITS WORK AND DOG A LOT OF 1318 00:47:00,840 --> 00:47:01,400 STUDIES USING IT. 1319 00:47:01,400 --> 00:47:05,400 SO TO SUMMARIZE I WILL SAY THE 1320 00:47:05,400 --> 00:47:07,600 CURRENT OBSERVATIONAL RESEARCH 1321 00:47:07,600 --> 00:47:10,240 IS SUSPECT, POLICIES TO BE SET, 1322 00:47:10,240 --> 00:47:11,640 LARGE SCALE OBSERVATION APPEARS 1323 00:47:11,640 --> 00:47:14,080 TO BE POSSIBLE AND MORE RELIABLE 1324 00:47:14,080 --> 00:47:15,160 THAN CURRENT APPROACHES ISSUES 1325 00:47:15,160 --> 00:47:17,600 USING THOSE THINGS I ALREADY 1326 00:47:17,600 --> 00:47:18,400 OUTLINES WE'RE GENERATING 1327 00:47:18,400 --> 00:47:21,320 EVIDENCE IS PUBLISH NOTHING TOP 1328 00:47:21,320 --> 00:47:22,480 JOURNALS, DATA MODELS BEING USED 1329 00:47:22,480 --> 00:47:26,360 IN U.S. AND DPLOABALLY, WE'RE 1330 00:47:26,360 --> 00:47:29,160 SUCCESSFULLY, COLLABORATING ON 1331 00:47:29,160 --> 00:47:31,600 GENERATING EVIDENCE AND JUST GO 1332 00:47:31,600 --> 00:47:32,200 TO ODESSY.ORGFOR MORE 1333 00:47:32,200 --> 00:47:32,520 INFORMATION. 1334 00:47:32,520 --> 00:47:33,880 I'M GOING TO SKIP THE NEXT 1335 00:47:33,880 --> 00:47:34,600 COUPLE OF SLIDES. 1336 00:47:34,600 --> 00:47:41,440 THAT WAS THERE IN CASE I WANT TO 1337 00:47:41,440 --> 00:47:45,200 SAY THANK YOU A LOT WAS STARTED 1338 00:47:45,200 --> 00:47:48,600 BY RO1 LMOO 16-10 WHICH I HAD 1339 00:47:48,600 --> 00:47:49,840 WELL FOR 2121 YEARS IT WAS 1340 00:47:49,840 --> 00:47:51,760 FUNDED FOR A BIT LONGER AND OUR 1341 00:47:51,760 --> 00:47:53,480 TRAINING GRANT WHICH HAS BEEN 1342 00:47:53,480 --> 00:47:55,400 AROUND FOR 30 YEARS. 1343 00:47:55,400 --> 00:47:57,480 >> IDDK BOTH IN FUNDING AND 1344 00:47:57,480 --> 00:47:58,200 COLLABORATION ON SOME OF THE 1345 00:47:58,200 --> 00:48:00,080 WORK THAT I WASN'T ABLE TO SHOW 1346 00:48:00,080 --> 00:48:04,640 AT THE END, NCI CONTRACTS AND 1347 00:48:04,640 --> 00:48:06,760 NHGRI, AND FDA CBER BEST, IT'S 1348 00:48:06,760 --> 00:48:08,040 BEEN AN HONOR TO DO THIS WORK 1349 00:48:08,040 --> 00:48:10,520 AND IT'S BEEN FUN DOING THIS, 1350 00:48:10,520 --> 00:48:13,400 OUR MOTTO AT COLUMBIA 1351 00:48:13,400 --> 00:48:14,160 BIO-INFORMATICS MODEL IS 1352 00:48:14,160 --> 00:48:15,440 DISCOVER AND IMPACT AND THAT'S 1353 00:48:15,440 --> 00:48:17,560 WHAT WE STRIVE TO ACHIEVE. 1354 00:48:17,560 --> 00:48:19,040 LET ME STOP THERE. 1355 00:48:19,040 --> 00:48:23,240 >> THANK YOU SO MUCH 1356 00:48:23,240 --> 00:48:24,200 DR. HRIPSCAK, IT'S AN INCREDIBLE 1357 00:48:24,200 --> 00:48:25,160 HONOR TO HAVE YOU HERE AND SPEAK 1358 00:48:25,160 --> 00:48:27,160 TO US AND I CERTAINLY LEARNED A 1359 00:48:27,160 --> 00:48:28,360 LOT, I'M SURE EVERYONE IN THE 1360 00:48:28,360 --> 00:48:33,440 AUDIENCE DID AS WELL, I DO WANT 1361 00:48:33,440 --> 00:48:37,040 TO SHARE THAT SINCERE APOLOGIES 1362 00:48:37,040 --> 00:48:39,160 TO YOU AND THE AUDIENCE FOR OUR 1363 00:48:39,160 --> 00:48:39,880 PROBLEM EARLIER. 1364 00:48:39,880 --> 00:48:43,360 IT WAS ACTUALLY A HARDWARE 1365 00:48:43,360 --> 00:48:46,360 GLITCH AND I APPRECIATE IT'S 1366 00:48:46,360 --> 00:48:47,880 PROMPT DETECTION AND TROUBLE 1367 00:48:47,880 --> 00:48:48,920 SHOOTING OF THE ISSUE. 1368 00:48:48,920 --> 00:48:50,520 WE HAVE ABOUT 3 OR 4 MINUTES AND 1369 00:48:50,520 --> 00:48:51,800 I WANT TO LEVERAGE THIS TIME TO 1370 00:48:51,800 --> 00:48:58,360 MAYBE ASK A FEW QUESTIONS. 1371 00:48:58,360 --> 00:49:04,360 FIRST THERE IS THIS PROBLEM OF 1372 00:49:04,360 --> 00:49:06,560 AND OVER AND ABOVE, THE FACT 1373 00:49:06,560 --> 00:49:08,440 THAT PEOPLE LOOK AT 1374 00:49:08,440 --> 00:49:09,520 OBSERVATIONAL STUDIES WITH A 1375 00:49:09,520 --> 00:49:11,840 GRAIN OF SALT BECAUSE OF THE 1376 00:49:11,840 --> 00:49:13,600 LACK OF RANDOMIZATION, ON TOP OF 1377 00:49:13,600 --> 00:49:14,680 THAT, WHEN THERE'S THIS ISSUE 1378 00:49:14,680 --> 00:49:18,640 OF, YOU KNOW WHAT WAS A PRIORITY 1379 00:49:18,640 --> 00:49:19,960 AND WHAT WAS PRESPECIFIED, THERE 1380 00:49:19,960 --> 00:49:24,280 HAS BEEN A PARADIGM SHIFT 1381 00:49:24,280 --> 00:49:25,200 TOWARDS ENCOURAGING 1382 00:49:25,200 --> 00:49:27,720 INVESTIGATORS TO REGISTER THEIR 1383 00:49:27,720 --> 00:49:28,320 OBSERVATIONAL STUDIES. 1384 00:49:28,320 --> 00:49:29,920 WHAT ARE YOUR VIEWS ON THAT 1385 00:49:29,920 --> 00:49:31,320 BECAUSE I THINK THAT THERE'S NO 1386 00:49:31,320 --> 00:49:33,400 HARD AND FAST POLICY, AND DO YOU 1387 00:49:33,400 --> 00:49:47,480 FEEL LIKE THAT HAS HELPFUL AND 1388 00:49:47,480 --> 00:49:48,000 WILL OFFER-- 1389 00:49:48,000 --> 00:49:49,800 >> AND SEE WHAT WE CHANGE IN 1390 00:49:49,800 --> 00:49:51,680 ADDITION WE TRY TO PUT IT IN 1391 00:49:51,680 --> 00:49:53,240 NATIONAL AND INTERNATIONAL REG 1392 00:49:53,240 --> 00:49:54,560 STEES IN THIS EDITION BUT WE 1393 00:49:54,560 --> 00:49:57,640 ALWAYS PUT IT ON GITHUB BUT IT 1394 00:49:57,640 --> 00:49:58,600 ALWAYS MAKES A DIFFERENCE 1395 00:49:58,600 --> 00:50:02,760 BECAUSE IT'S SO HARD NOT TO 1396 00:50:02,760 --> 00:50:04,240 ACCIDENTALLY STEAL RESULTS, LOOK 1397 00:50:04,240 --> 00:50:07,160 IF SOMEONE'S GOING TO BE 1398 00:50:07,160 --> 00:50:07,840 DECEPTIVE, THEY WILL DO 1399 00:50:07,840 --> 00:50:09,840 ANYTHING, WE ARE TRYING TO 1400 00:50:09,840 --> 00:50:12,120 IMPROVE THE QUALITY AND HONESTY 1401 00:50:12,120 --> 00:50:14,720 OF EVIDENCE PRESENTED BY THEM 1402 00:50:14,720 --> 00:50:15,360 AND PRESPECIFYING YOUR PROTOCOL 1403 00:50:15,360 --> 00:50:17,600 IS 1 OF THE SINGLE BIGGEST 1404 00:50:17,600 --> 00:50:20,040 THINGS WE CAN DO EASILY BECAUSE 1405 00:50:20,040 --> 00:50:21,480 WE'RE USED TOKING IT FOR 1406 00:50:21,480 --> 00:50:22,440 RANDOMIZED TRIALS AND WHERE ARE 1407 00:50:22,440 --> 00:50:23,800 WE NOT DOING IT FOR 1408 00:50:23,800 --> 00:50:24,840 OBSERVATIONAL RESEARCH. 1409 00:50:24,840 --> 00:50:26,120 I AGREE IN MY OWN EXPERIENCE I 1410 00:50:26,120 --> 00:50:28,000 FIND IT HELPFUL NOW TO REGISTER 1411 00:50:28,000 --> 00:50:31,000 EVEN OUR OBSERVATIONAL STUDIES, 1412 00:50:31,000 --> 00:50:31,960 AND INITIALLY THERE WAS A 1413 00:50:31,960 --> 00:50:33,400 CONCERN THAT WE MAY NOT BE ABLE 1414 00:50:33,400 --> 00:50:36,840 TO CHANGE THINGS LATER ON, BUT I 1415 00:50:36,840 --> 00:50:38,920 THINK THAT BY DEFAULT THIS IS A 1416 00:50:38,920 --> 00:50:40,560 TYPE OF WORK THAT DOES REQUIRE 1417 00:50:40,560 --> 00:50:42,400 CHANGES AND JUST REQUIRES A 1418 00:50:42,400 --> 00:50:43,600 RATIONAL EXPLANATION AS TO WHY 1419 00:50:43,600 --> 00:50:46,600 THE CHANGE HAD TO BE DONE. 1420 00:50:46,600 --> 00:50:48,120 >> EXACTLY SO WE DO WANT TO 1421 00:50:48,120 --> 00:50:49,040 SUPPORT THOSE CHANGES BUT THEN 1422 00:50:49,040 --> 00:50:51,720 PEOPLE CAN SEE WHAT YOU DID AND 1423 00:50:51,720 --> 00:50:53,800 YOU CAN ASSESS WHY IS WHETHER 1424 00:50:53,800 --> 00:50:57,040 THAT WILL HAVE AN EFFECT ON THE 1425 00:50:57,040 --> 00:50:57,480 TRUSTWORTHINESS. 1426 00:50:57,480 --> 00:50:57,920 >> THAT'S CORRECT. 1427 00:50:57,920 --> 00:50:58,600 >> YOU KNOW THE OTHER QUESTION 1428 00:50:58,600 --> 00:51:00,000 IS YOU ARE ON THE INTRAMURAL 1429 00:51:00,000 --> 00:51:03,480 SIDE BUT YOU'RE AWARE OF THE 1430 00:51:03,480 --> 00:51:04,880 ACTIVITIES THAT OCCUR THAT HERE 1431 00:51:04,880 --> 00:51:07,000 ON THE CAMPUSOT INTRAMURAL SIDE. 1432 00:51:07,000 --> 00:51:10,200 DO YOU HAVE A SENSE OF WHAT THE 1433 00:51:10,200 --> 00:51:14,000 FUTURE SCOPE MIGHT BE FOR THIS 1434 00:51:14,000 --> 00:51:15,680 TYPE OF RESEARCH WITHIN THE 1435 00:51:15,680 --> 00:51:17,440 INTRAMURAL SPACE AND HOW 1436 00:51:17,440 --> 00:51:18,040 INTRAMURAL INVESTIGATORS CAN 1437 00:51:18,040 --> 00:51:19,480 LEVERAGE THIS NOT ONLY FOR THEIR 1438 00:51:19,480 --> 00:51:21,000 OWN STUDIES BUT ALSO 1439 00:51:21,000 --> 00:51:22,640 INDEPENDENTLY DURING THIS WORK? 1440 00:51:22,640 --> 00:51:23,120 >> SURE, THING. 1441 00:51:23,120 --> 00:51:24,560 WELL A COUPLE OF THINGS, SO 1442 00:51:24,560 --> 00:51:26,480 NUMBER 1, YOU KNOW, YOU CAN LOOK 1443 00:51:26,480 --> 00:51:28,640 AT THIS EXTERNAL BODY OF 1444 00:51:28,640 --> 00:51:30,600 OBSERVATIONAL RESEARCH AS BEING 1445 00:51:30,600 --> 00:51:32,360 HYPOTHESIS GENERATE FIGURE THE 1446 00:51:32,360 --> 00:51:33,440 STUDIES YOU DO INTRA MIEWRALLY, 1447 00:51:33,440 --> 00:51:35,440 AS WELL AS HYPOTHESIS TESTING 1448 00:51:35,440 --> 00:51:36,280 THAT ARE GENERATED INTRA 1449 00:51:36,280 --> 00:51:38,160 MIEWRALLY, SO THAT'S GOOD. 1450 00:51:38,160 --> 00:51:39,440 IT MIGHT BE WORTH WHILE FOR THE 1451 00:51:39,440 --> 00:51:41,200 CLINICAL CENTER TO HAVE SOMEONE 1452 00:51:41,200 --> 00:51:43,280 BECOME PART OF ODESSY, THAT IS, 1453 00:51:43,280 --> 00:51:44,760 YOU KNOW TO HAVE A 1454 00:51:44,760 --> 00:51:46,240 COLLABORATION, YOU REALLY NEED 1455 00:51:46,240 --> 00:51:47,800 SOMEONE WHO STANDS BETWEEN AND 1456 00:51:47,800 --> 00:51:49,040 INTERPRETS BETWEEN THE 2 SO 1457 00:51:49,040 --> 00:51:52,200 HAVING SOMEONE AT A POST DOC 1458 00:51:52,200 --> 00:51:53,480 LEVEL, GOES ALL IN AND LEARNS 1459 00:51:53,480 --> 00:51:56,240 ABOUT IT AND DECIDES STUDIES AND 1460 00:51:56,240 --> 00:51:57,480 PUTS THEM FORWARD TO THE NETWORK 1461 00:51:57,480 --> 00:51:59,000 COULD BEING A PATHWAY TO GET 1462 00:51:59,000 --> 00:51:59,600 THINGS DONE. 1463 00:51:59,600 --> 00:52:01,280 NOW THE CLINICAL CENTER COULD 1464 00:52:01,280 --> 00:52:03,240 PUT ITS DATA INTO THE OMOP DATA 1465 00:52:03,240 --> 00:52:04,720 MODAND HE WILL RUN STUDIES AND 1466 00:52:04,720 --> 00:52:07,600 BECOME A NODEOT ODESSY NETWORK, 1467 00:52:07,600 --> 00:52:09,240 REMEMBER WE NEVER SHARE DATA SO 1468 00:52:09,240 --> 00:52:11,760 IT'S NOT A PRIVACY CONCERN. 1469 00:52:11,760 --> 00:52:16,080 BUT YEAH AND THEN, I DIDN'T GET 1470 00:52:16,080 --> 00:52:18,040 A CHANCE TO SHOW TODAY BUT WE 1471 00:52:18,040 --> 00:52:19,680 ARE ALSO BRANCHING INTO THINGS 1472 00:52:19,680 --> 00:52:21,680 THAT INVOLVE PHYSIOLOGY AND SAY 1473 00:52:21,680 --> 00:52:23,480 WORKING WITHIN NIDDK ON THE 1474 00:52:23,480 --> 00:52:25,560 METABOLISM, THAT'S WHERE WE'RE 1475 00:52:25,560 --> 00:52:27,600 BOTH USING OBSERVATIONAL 1476 00:52:27,600 --> 00:52:29,680 RESEARCH AND LOOKING AT 1477 00:52:29,680 --> 00:52:30,800 DEEP--EXCUSE ME PHYSIOLOGIC 1478 00:52:30,800 --> 00:52:32,520 MODELS, AND JOINING THOSE 2 AND 1479 00:52:32,520 --> 00:52:33,480 THAT'S BEEN QUITE FUN. 1480 00:52:33,480 --> 00:52:35,200 I THINK THERE'S A LOT OF 1481 00:52:35,200 --> 00:52:37,440 POTENTIAL THERE IN INTRAMURAL 1482 00:52:37,440 --> 00:52:39,560 RESEARCH. 1483 00:52:39,560 --> 00:52:40,280 >> WONDERFUL. 1484 00:52:40,280 --> 00:52:40,920 WE'VE UNFORTUNATELY RUN OUT OF 1485 00:52:40,920 --> 00:52:41,320 TIME. 1486 00:52:41,320 --> 00:52:44,080 WE HAVE A LOT MORE QUESTIONS 1487 00:52:44,080 --> 00:52:45,200 THAT CAME THROUGH AND GIVEN THE 1488 00:52:45,200 --> 00:52:48,240 FACT THAT WE LOST SOME TIME 1489 00:52:48,240 --> 00:52:49,800 WE'RE UNABLE TO TOUCH ON ALL OF 1490 00:52:49,800 --> 00:52:51,200 THEM, BUT AGAIN, ONCE AGAIN 1491 00:52:51,200 --> 00:52:53,200 THANK YOU SO MUCH FOR YOUR TIME 1492 00:52:53,200 --> 00:52:56,680 AND THANK YOU EVERYONE FOR 1493 00:52:56,680 --> 00:52:56,960 LISTENING. 1494 00:52:56,960 --> 00:52:57,480 >> THANK YOU SAMEER. 1495 00:52:57,480 --> 00:00:00,000 >> HAVE A GOOD DAY, EVERYONE.