1 00:00:04,738 --> 00:00:06,039 >> I WANT TO WELCOME YOU TO THIS 2 00:00:06,106 --> 00:00:09,709 REALLY IMPORTANT WORKSHOP ON 3 00:00:09,776 --> 00:00:10,310 NOVEL APPROACHES TO PREVENTING 4 00:00:10,377 --> 00:00:10,710 PUBLICATION BIAS. 5 00:00:10,777 --> 00:00:12,379 SO I'M THE DIRECTOR OF NINDS, 6 00:00:12,445 --> 00:00:16,549 AND I AM PRIVILEGED TO 7 00:00:16,616 --> 00:00:17,417 WORK WITH SOME REALLY GREAT 8 00:00:17,484 --> 00:00:19,619 PEOPLE IN OUR OFFICE OF RESEARCH 9 00:00:19,686 --> 00:00:22,355 QUALITY, AND THE PURPOSE IS TO 10 00:00:22,422 --> 00:00:24,758 TRY TO MAKE SURE THAT THE 11 00:00:24,824 --> 00:00:28,128 RESEARCH WE FUND ATTAINS THE 12 00:00:28,194 --> 00:00:29,229 HIGHEST QUALITY STANDARDS. 13 00:00:29,295 --> 00:00:32,298 SO I WAS FOR MANY YEARS A 14 00:00:32,365 --> 00:00:33,933 PHYSICIAN, WE SPENT TONS OF TIME 15 00:00:34,000 --> 00:00:36,236 DOING QUALITY ASSURANCE TO MAKE 16 00:00:36,302 --> 00:00:37,871 SURE THAT THINGS DIDN'T GO WRONG 17 00:00:37,937 --> 00:00:40,173 WITH OUR PATIENTS. 18 00:00:40,240 --> 00:00:42,642 WHEN YOU GO ON AN AIRPLANE, 19 00:00:42,709 --> 00:00:43,343 THERE'S LOTS OF QUALITY 20 00:00:43,410 --> 00:00:44,444 ASSURANCE THAT THE DOORS ARE NOT 21 00:00:44,511 --> 00:00:45,979 GOING TO FALL OFF WHILE YOU'RE 22 00:00:46,045 --> 00:00:49,249 IN FLIGHT. 23 00:00:49,315 --> 00:00:50,884 SOMETIMES IT WORKS, BUT WE SPEND 24 00:00:50,950 --> 00:00:54,621 BILLIONS OF DOLLARS ON RESEARCH 25 00:00:54,687 --> 00:00:55,722 AND WE HAVEN'T REALLY THOUGHT 26 00:00:55,789 --> 00:00:56,689 ABOUT THAT EQUIVALENT. 27 00:00:56,756 --> 00:00:59,559 HOW DO WE GET THE BEST QUALITY 28 00:00:59,626 --> 00:01:00,860 OUT OF -- IN OUR RESEARCH, AND 29 00:01:00,927 --> 00:01:02,595 THAT'S WHAT OUR OFFICE IS REALLY 30 00:01:02,662 --> 00:01:02,929 ABOUT. 31 00:01:02,996 --> 00:01:04,364 THERE'S A LOT OF DIFFERENT 32 00:01:04,431 --> 00:01:07,434 THINGS, EDUCATIONAL PROGRAMS, 33 00:01:07,500 --> 00:01:09,502 DEVELOPING GROUPS OF RIGOR 34 00:01:09,569 --> 00:01:11,771 CHAMPIONS TO PROMOTE RIGOR IN 35 00:01:11,838 --> 00:01:14,441 RESEARCH ACROSS THE COUNTRY, 36 00:01:14,507 --> 00:01:15,708 ACTUALLY GAVE OUT PRIZES THIS 37 00:01:15,775 --> 00:01:17,811 YEAR FOR PEOPLE WHO ARE REAL 38 00:01:17,877 --> 00:01:19,112 CHAMPIONS AND DONE REALLY GOOD 39 00:01:19,179 --> 00:01:21,948 WORK IN THEIR AREA. 40 00:01:22,015 --> 00:01:23,149 BUT NOW WE'RE GOING TO BE 41 00:01:23,216 --> 00:01:24,150 TALKING ABOUT A PARTICULAR AREA 42 00:01:24,217 --> 00:01:25,452 TODAY, AND THAT'S PUBLICATION 43 00:01:25,518 --> 00:01:26,186 BIAS. 44 00:01:26,252 --> 00:01:27,420 YOU'LL HEAR MUCH MORE ABOUT IT. 45 00:01:27,487 --> 00:01:29,088 I WAS JUST GOING TO OFFER JUST A 46 00:01:29,155 --> 00:01:30,723 COUPLE OF TAKE-HOME POINTS FOR 47 00:01:30,790 --> 00:01:32,559 YOU TO REMEMBER. 48 00:01:32,625 --> 00:01:36,496 ONE IS THAT WHEN YOU READ A 49 00:01:36,563 --> 00:01:37,697 PUBLICATION, THE REAL CRUX IS IN 50 00:01:37,764 --> 00:01:39,466 THE DATA AND THE METHODS AND 51 00:01:39,532 --> 00:01:40,867 EVERYTHING ELSE IS AN 52 00:01:40,934 --> 00:01:45,572 ADVERTISEMENT. 53 00:01:45,638 --> 00:01:49,075 SO I'M SAYING THAT SOMEWHAT 54 00:01:49,142 --> 00:01:49,943 FACETIOUSLY, BUT IN TRUTH THE 55 00:01:50,009 --> 00:01:51,044 DATA IS THE GOAL. 56 00:01:51,110 --> 00:01:52,679 AND THE PUBLICATION IS BASED ON 57 00:01:52,745 --> 00:01:54,347 THE GOAL BUT ALSO THE STORY. 58 00:01:54,414 --> 00:01:57,784 YOU HAVE TO HAVE THAT BALANCE, 59 00:01:57,851 --> 00:01:59,219 AND OFTENTIMES THAT BALANCE GETS 60 00:01:59,285 --> 00:02:01,120 OUT OF WHACK, WHERE THE STORY 61 00:02:01,187 --> 00:02:06,025 REALLY KIND OF TAKES OVER AND 62 00:02:06,092 --> 00:02:08,428 THEN THERE'S TONS OF DATA THAT 63 00:02:08,495 --> 00:02:10,530 WERE DONE IN PREPARATION AND 64 00:02:10,597 --> 00:02:11,931 THAT WERE NEVER PUBLISHED, AND 65 00:02:11,998 --> 00:02:13,700 WE DON'T KNOW WHAT THOSE THINGS 66 00:02:13,766 --> 00:02:14,267 SHOWED. 67 00:02:14,334 --> 00:02:15,468 I THINK THE FUTURE IS GOING TO 68 00:02:15,535 --> 00:02:16,870 BE DIFFERENT. 69 00:02:16,936 --> 00:02:18,638 THE REASON WE'RE IN THE SPOT 70 00:02:18,705 --> 00:02:20,607 WE'RE IN NOW IS BECAUSE WE RELY 71 00:02:20,673 --> 00:02:25,378 ON A TECHNOLOGY FOR 72 00:02:25,445 --> 00:02:27,146 COMMUNICATING SCIENCE THAT'S 73 00:02:27,213 --> 00:02:28,882 BASED ON THE TECHNOLOGY OF THE 74 00:02:28,948 --> 00:02:29,883 PRINTING PRESS. 75 00:02:29,949 --> 00:02:31,084 AND THAT'S CHANGED. 76 00:02:31,150 --> 00:02:32,952 WE PUBLISH PAPERS, THAT'S BEEN 77 00:02:33,019 --> 00:02:34,053 GOING ON SINCE THE PRINTING 78 00:02:34,120 --> 00:02:34,821 PRESS. 79 00:02:34,888 --> 00:02:36,322 BUT THE FUTURE, I THINK, IS 80 00:02:36,389 --> 00:02:38,525 GOING TO BE DATA AND DATA IS 81 00:02:38,591 --> 00:02:39,626 GOING TO BE THE GOAL, THEN THE 82 00:02:39,692 --> 00:02:41,261 GOAL IS GOING TO BE THESE MINES 83 00:02:41,327 --> 00:02:43,897 WHERE PEOPLE CAN GO AN LOOK AT 84 00:02:43,963 --> 00:02:45,532 DATA AND WHEN YOU WRITE YOUR 85 00:02:45,598 --> 00:02:48,368 STORY ABOUT YOUR DATA, YOU'RE 86 00:02:48,434 --> 00:02:49,235 PROBABLY RIGHT BUT MAYBE NOT 87 00:02:49,302 --> 00:02:50,537 RIGHT AND THERE MIGHT BE 88 00:02:50,603 --> 00:02:51,571 ALTERNATIVE STORIES THAT SOME 89 00:02:51,638 --> 00:02:52,839 PEOPLE COULD GENERATE BASED ON 90 00:02:52,906 --> 00:02:53,306 THE DATA. 91 00:02:53,373 --> 00:02:56,509 AND SO THAT'S WHY IT'S REALLY 92 00:02:56,576 --> 00:02:57,844 IMPORTANT THAT WE FIGURE OUT A 93 00:02:57,911 --> 00:03:00,013 WAY TO GET REALLY GOOD DATA IN 94 00:03:00,079 --> 00:03:01,014 PLACES WHERE PEOPLE CAN LOOK AT 95 00:03:01,080 --> 00:03:03,850 IT AND TRY TO UNDERSTAND WHERE 96 00:03:03,917 --> 00:03:05,585 IT INTERSECTS IN WHAT THEY'RE 97 00:03:05,652 --> 00:03:08,321 THINKING ABOUT AND MAYBE DEVELOP 98 00:03:08,388 --> 00:03:09,522 ALTERNATE STORIES. 99 00:03:09,589 --> 00:03:11,491 SO IT'S REALLY ENCOURAGING THAT 100 00:03:11,558 --> 00:03:15,995 WE HAVE A DISIE VE DIVERSE GROUF 101 00:03:16,062 --> 00:03:17,764 RESEARCHERS FROM A VARIETY OF 102 00:03:17,830 --> 00:03:18,965 COUNTRIES AND SCIENTIFIC FIELDS 103 00:03:19,032 --> 00:03:21,568 HERE, AND I WANT TO JUST REALLY 104 00:03:21,634 --> 00:03:28,341 END OUT BY THANKING ALL FOR 105 00:03:28,408 --> 00:03:29,342 THEIR LEADERSHIP IN THIS SPACE 106 00:03:29,409 --> 00:03:29,709 AT NIH. 107 00:03:29,776 --> 00:03:30,977 AND WITH THAT, I WANT TO 108 00:03:31,044 --> 00:03:32,712 INTRODUCE MY GOOD FRIEND, 109 00:03:32,779 --> 00:03:33,713 DR. LARRY TABAK. 110 00:03:33,780 --> 00:03:36,249 LARRY IS THE DEPUTY DIRECTOR OF 111 00:03:36,316 --> 00:03:38,518 NIH, AND HE'S JUST BEEN A MAJOR 112 00:03:38,585 --> 00:03:40,253 CHAMPION FOR RIGOR AND RESEARCH, 113 00:03:40,320 --> 00:03:41,921 ALONG WITH SAVING THE INSTITUTE 114 00:03:41,988 --> 00:03:45,191 MULTIPLE TIMES, AND HE CAN'T 115 00:03:45,258 --> 00:03:46,859 TALK ABOUT THAT BUT WHEN I 116 00:03:46,926 --> 00:03:50,997 RETIRE, I CAN TELL YOU ABOUT IT. 117 00:03:51,064 --> 00:03:53,066 HE PREVIOUSLY HE WAS THE 118 00:03:53,132 --> 00:03:54,801 DIRECTOR OF THE NATIONAL 119 00:03:54,867 --> 00:03:56,769 INSTITUTES OF DENTAL AND 120 00:03:56,836 --> 00:03:59,639 CRANIOFACIAL RESEARCH, BEFORE 121 00:03:59,706 --> 00:04:00,907 HERE, UNIVERSITY OF ROCHESTER, 122 00:04:00,974 --> 00:04:03,343 PROFESSOR OF BIOCHEMISTRY AND 123 00:04:03,409 --> 00:04:05,211 BIOPHYSICS. 124 00:04:05,278 --> 00:04:06,846 STUDYING GLYCAN BIOLOGY, WHICH 125 00:04:06,913 --> 00:04:08,281 IS REALLY IMPORTANT IN HEALTH OF 126 00:04:08,348 --> 00:04:10,249 ALL THE MUCOSAL SURFACES OF THE 127 00:04:10,316 --> 00:04:10,483 BODY. 128 00:04:10,550 --> 00:04:12,752 IF YOU WANT TO HEAR ABOUT 129 00:04:12,819 --> 00:04:20,526 GLYCANS, MAYBE DO THAT OFFLINE. 130 00:04:20,593 --> 00:04:23,930 >> WELL GOOD MORNING, EVERYBODY. 131 00:04:23,997 --> 00:04:24,797 THANK YOU, WALTER. 132 00:04:24,864 --> 00:04:26,766 SO I DON'T TALK TO MY WIFE 133 00:04:26,833 --> 00:04:30,570 WITHOUT POWERPOINTS, SO YOU HAVE 134 00:04:30,637 --> 00:04:32,505 TO PUT UP MY FIRST SLIDE, OR DO 135 00:04:32,572 --> 00:04:34,007 I JUST CLICK THIS? 136 00:04:34,073 --> 00:04:35,441 THAT DIDN'T WORK. 137 00:04:35,508 --> 00:04:36,442 SO SOMEBODY NEEDS TO PUT MY 138 00:04:36,509 --> 00:04:36,809 SLIDES UP. 139 00:04:36,876 --> 00:04:39,879 DO YOU HAVE MY SLIDES? 140 00:04:39,946 --> 00:04:40,446 NO? 141 00:04:40,513 --> 00:04:42,615 NO ONE HAS MY SLIDES? 142 00:04:42,682 --> 00:04:43,950 BOY, NOW I'M REALLY IN TROUBLE. 143 00:04:44,017 --> 00:04:45,818 NOW IF I GIVE YOU A FLASH DRIVE, 144 00:04:45,885 --> 00:04:47,120 WILL YOU BE ABLE TO FIND MY 145 00:04:47,186 --> 00:04:48,287 SLIDES? 146 00:04:48,354 --> 00:04:51,157 THERE'S THE FLASH DRIVE. 147 00:04:51,224 --> 00:04:54,861 SO, IF I HAD MY SLIDES, THE 148 00:04:54,927 --> 00:04:57,830 FIRST THING THAT I WOULD DO IS I 149 00:04:57,897 --> 00:04:59,999 WOULD GIVE YOU GREETINGS FROM 150 00:05:00,066 --> 00:05:02,702 THE NIH DIRECTOR, DR. MONICA 151 00:05:02,769 --> 00:05:03,002 BERTAGNOLLI. 152 00:05:03,069 --> 00:05:05,705 I THINK YOU ALL KNOW THAT MONICA 153 00:05:05,772 --> 00:05:09,509 WAS APPOINTED AS THE 17TH 154 00:05:09,575 --> 00:05:11,044 DIRECTOR OF THE NATIONAL 155 00:05:11,110 --> 00:05:11,811 INSTITUTES OF HEALTH. 156 00:05:11,878 --> 00:05:13,479 HER SIX-MONTH ANNIVERSARY WAS 157 00:05:13,546 --> 00:05:14,447 LAST WEEK. 158 00:05:14,514 --> 00:05:18,184 OR MAYBE TWO WEEKS AGO NOW. 159 00:05:18,251 --> 00:05:20,687 SHE IS FIRST SURGEON TO BE 160 00:05:20,753 --> 00:05:22,655 APPOINTED AS DIRECTOR OF NIH. 161 00:05:22,722 --> 00:05:23,856 THERE'S ALL SORTS OF 162 00:05:23,923 --> 00:05:25,258 METAPHORICAL THINGS THAT YOU CAN 163 00:05:25,324 --> 00:05:28,561 THINK ABOUT ABOUT THAT, BUT -- 164 00:05:28,628 --> 00:05:29,862 AND SHE IS ONLY THE SECOND WOMAN 165 00:05:29,929 --> 00:05:34,033 TO BE APPOINTED NIH DIRECTOR. 166 00:05:34,100 --> 00:05:35,968 SHE HIT THE GROUND RUNNING. 167 00:05:36,035 --> 00:05:37,203 IT DOESN'T FIT? 168 00:05:37,270 --> 00:05:37,837 WHY? 169 00:05:37,904 --> 00:05:39,706 OKAY, THERE IT IS. 170 00:05:39,772 --> 00:05:41,607 AND PICK WHICHEVER ONE YOU WANT. 171 00:05:41,674 --> 00:05:43,910 ANY ONE OF THE TALKS THAT ARE ON 172 00:05:43,976 --> 00:05:45,511 THAT DRIVE, AND I'LL ADAPT TO 173 00:05:45,578 --> 00:05:47,580 WHAT WE'RE TALKING ABOUT HERE 174 00:05:47,647 --> 00:05:51,150 TODAY. 175 00:05:51,217 --> 00:05:54,253 IT'S LIKE GLYCANS, THEY'RE VERY 176 00:05:54,320 --> 00:05:54,687 ADAPTABLE. 177 00:05:54,754 --> 00:05:55,588 WAS THERE ONE THAT YOU FOUND 178 00:05:55,655 --> 00:06:01,928 THAT MAYBE WOULD WORK? 179 00:06:01,994 --> 00:06:02,995 THAT'S PROBABLY NOT THE ONE YOU 180 00:06:03,062 --> 00:06:05,531 WANT. 181 00:06:05,598 --> 00:06:06,065 ALL RIGHT. 182 00:06:06,132 --> 00:06:07,900 AND IN ANY EVENT, MONICA -- 183 00:06:07,967 --> 00:06:08,367 THANK YOU. 184 00:06:08,434 --> 00:06:12,405 MONICA WANTED TO BE HERE, BUT 185 00:06:12,472 --> 00:06:20,079 UNFORTUNATELY, IS ON TRAVEL. 186 00:06:20,146 --> 00:06:22,048 SO HERE SHE IS BEING SWORN IN. 187 00:06:22,115 --> 00:06:23,449 THE PERSON ON THE FAR LEFT OF 188 00:06:23,516 --> 00:06:26,219 COURSE IS SECRETARY BECERRA. 189 00:06:26,285 --> 00:06:29,155 THE GENTLEMAN IN THE MIDDLE IS 190 00:06:29,222 --> 00:06:30,456 MONICA'S HUSBAND, AL, AND I WAS 191 00:06:30,523 --> 00:06:31,891 PRIVILEGED TO BE THERE DURING 192 00:06:31,958 --> 00:06:33,192 HER SWEARING IN BECAUSE I WANTED 193 00:06:33,259 --> 00:06:34,293 TO MAKE SURE THAT I DIDN'T HAVE 194 00:06:34,360 --> 00:06:35,695 TO BE THE ACTING DIRECTOR 195 00:06:35,762 --> 00:06:38,664 ANYMORE. 196 00:06:38,731 --> 00:06:39,198 AFTER 23 MONTHS. 197 00:06:39,265 --> 00:06:40,733 BUT WHO WAS COUNTING? 198 00:06:40,800 --> 00:06:45,004 OKAY. 199 00:06:45,071 --> 00:06:48,608 SO WALTER REALLY, YOU KNOW, 200 00:06:48,674 --> 00:06:51,210 SUMMARIZED SOME OF WHAT I WAS 201 00:06:51,277 --> 00:06:52,779 GOING TO SAY BUT HE DID IT IN A 202 00:06:52,845 --> 00:06:54,280 MUCH NICER WAY THAN I WOULD 203 00:06:54,347 --> 00:07:00,319 HAVE. 204 00:07:00,386 --> 00:07:02,155 BACK IN 2014, FRANCIS COLLINS 205 00:07:02,221 --> 00:07:04,891 AND I PUBLISHED THIS POSITION 206 00:07:04,957 --> 00:07:09,495 PIECE IN NATURE, AND IT REALLY 207 00:07:09,562 --> 00:07:14,567 OUTLINED WHAT THE ISSUES WERE 208 00:07:14,634 --> 00:07:16,202 AND THE SORT OF BEGINNINGS OF 209 00:07:16,269 --> 00:07:17,537 WHAT WE WERE TRYING TO DO. 210 00:07:17,603 --> 00:07:20,840 NOW WHEN I SAY "WE," I HAVE TO 211 00:07:20,907 --> 00:07:23,109 PAY HOMAGE TO SHAI. 212 00:07:23,176 --> 00:07:24,277 BECAUSE BEFORE THIS BECAME A 213 00:07:24,343 --> 00:07:26,145 THING, SHAI WAS TALKING ABOUT 214 00:07:26,212 --> 00:07:30,516 IT, AND REALLY INFLUENCED ME, 215 00:07:30,583 --> 00:07:31,617 INFLUENCED FRANCIS AND HAS 216 00:07:31,684 --> 00:07:37,824 INFLUSSED THEINFLUENCED THE WHO. 217 00:07:37,890 --> 00:07:40,560 SO SHAI, THANK YOU FOR YOUR 218 00:07:40,626 --> 00:07:41,460 LEADERSHIP, NOT JUST AT NINDS 219 00:07:41,527 --> 00:07:43,196 BUT REALLY ACROSS THE WHOLE 220 00:07:43,262 --> 00:07:44,197 BIOMEDICAL RESEARCH SPACE. 221 00:07:44,263 --> 00:07:47,366 IN THIS PIECE, FRANCIS AND I 222 00:07:47,433 --> 00:07:50,236 WROTE THAT PRE-CLINICAL RESEARCH 223 00:07:50,303 --> 00:07:54,073 IS REALLY SUSCEPTIBILITY TO 224 00:07:54,140 --> 00:07:55,675 REPRODUCIBILITY ISSUES, BUT THEN 225 00:07:55,741 --> 00:08:03,983 EMPHASIZING THAT SOME SOME EAR 226 00:08:04,050 --> 00:08:04,617 REPRODUCIBLE REPORTS ARE 227 00:08:04,684 --> 00:08:06,152 PROBABLY THE RESULT OF 228 00:08:06,219 --> 00:08:08,120 COINCIDENTAL FINDINGS THAT 229 00:08:08,187 --> 00:08:10,122 HAPPEN TO REACH STATISTICAL 230 00:08:10,189 --> 00:08:14,393 SIGNIFICANCE COUPLED WITH BIAS 231 00:08:14,460 --> 00:08:15,528 SO EVEN BACK THEN WE KNEW THIS 232 00:08:15,595 --> 00:08:16,762 WAS A POTENTIAL ISSUE. 233 00:08:16,829 --> 00:08:18,397 SO JUST TO LEVEL SET, AND YOU 234 00:08:18,464 --> 00:08:20,399 ALL KNOW THIS BETTER THAN I DO, 235 00:08:20,466 --> 00:08:23,236 BUT AGAIN JUST TO LEVEL SET, 236 00:08:23,302 --> 00:08:24,871 PUBLICATION BIAS IS A TENDENCY 237 00:08:24,937 --> 00:08:27,340 ON THE PARTS OF EVERYBODY IN THE 238 00:08:27,406 --> 00:08:30,610 WHOLE ECOSYSTEM TO SUBMIT OR 239 00:08:30,676 --> 00:08:31,711 ACCEPT MANUSCRIPTS FOR 240 00:08:31,777 --> 00:08:34,213 PUBLICATION BASED ON WHAT 241 00:08:34,280 --> 00:08:35,748 HAPPENED, SO TO SPEAK. 242 00:08:35,815 --> 00:08:37,750 AND OFTEN, THIS RESULTS IN THE 243 00:08:37,817 --> 00:08:40,453 ABSENCE OF NO RESULTS FROM THE 244 00:08:40,519 --> 00:08:41,053 SCIENTIFIC RECORD. 245 00:08:41,120 --> 00:08:43,422 YOU ONLY KNOW WHAT, QUOTE, 246 00:08:43,489 --> 00:08:44,957 WORKED, YOU NEVER KNOW WHAT, 247 00:08:45,024 --> 00:08:45,825 QUOTE, DIDN'T WORK. 248 00:08:45,892 --> 00:08:47,627 AND THAT, OF COURSE, LEADS TO 249 00:08:47,693 --> 00:08:53,599 THIS INCOMPLETE PICTURE, AND IT 250 00:08:53,666 --> 00:08:54,567 DRAMATICALLY UNDERUTILIZES THE 251 00:08:54,634 --> 00:08:58,070 NIH INVESTMENT IN BIOMEDICAL 252 00:08:58,137 --> 00:09:00,273 RESEARCH, WHICH IS WHY AS AN 253 00:09:00,339 --> 00:09:01,240 AGENCY, WE'RE SO STRONGLY 254 00:09:01,307 --> 00:09:04,644 INTERESTED IN THIS. 255 00:09:04,710 --> 00:09:07,380 SO WHY AM I SHOWING YOU THIS 256 00:09:07,446 --> 00:09:11,651 PAPER ON PERIODONTAL DISEASE? 257 00:09:11,717 --> 00:09:14,253 NOT BECAUSE I'M PAYING HOMAGE TO 258 00:09:14,320 --> 00:09:15,421 NIDCR, WHERE I WAS THE DIRECTOR 259 00:09:15,488 --> 00:09:20,626 FOR A DECADE. 260 00:09:20,693 --> 00:09:22,028 BUT IT'S A PAPER THAT WAS A 261 00:09:22,094 --> 00:09:23,462 LONG, LONG, LONG TIME IN COMING. 262 00:09:23,529 --> 00:09:25,164 SO FOR MANY YEARS, MY COLLEAGUES 263 00:09:25,231 --> 00:09:30,102 IN THE DENTAL INSTITUTE WOULD 264 00:09:30,169 --> 00:09:34,373 FUND SMALL TRIEPS THAT WERE -- L 265 00:09:34,440 --> 00:09:37,843 TRIES TTRIALS THAT WERE UNDERPOD 266 00:09:37,910 --> 00:09:40,947 AND WHEN I ARRIVED AS A 267 00:09:41,013 --> 00:09:42,048 GLYCOPATHOLOGIST I HAD TO ASK 268 00:09:42,114 --> 00:09:43,115 WHY ARE YOU ONLY FUNDING THESE 269 00:09:43,182 --> 00:09:46,218 TRIALS WHICH THE CONCLUSIONARY 270 00:09:46,285 --> 00:09:47,753 SENTENCE OF EVERY PAPER THAT'S 271 00:09:47,820 --> 00:09:49,689 PUBLISHED IS, "AND FUSHT 272 00:09:49,755 --> 00:09:52,425 RESEARCH WILL BE NEEDED TO 273 00:09:52,491 --> 00:09:56,696 DEFINITIVELY -- I WAS TOLD WE 274 00:09:56,762 --> 00:09:58,364 CAN'T AFFORD TO DO DEFINITIVE 275 00:09:58,431 --> 00:10:00,199 TRIALS, THEY'RE VERY EXPENSIVE. 276 00:10:00,266 --> 00:10:03,602 AND I SAID, WELL, I'M ONLY AN 277 00:10:03,669 --> 00:10:06,539 ENZYMOLOGIST, I STUDY 278 00:10:06,605 --> 00:10:07,406 TRANSFERASES BUT I THINK PAYING 279 00:10:07,473 --> 00:10:09,275 FOR THE TRIAL IS MY PROBLEM, NOT 280 00:10:09,342 --> 00:10:11,143 YOURS, LET'S DO SOME DEFINITIVE 281 00:10:11,210 --> 00:10:11,777 TRIALS. 282 00:10:11,844 --> 00:10:13,512 AND THIS WAS ONE OF THE VERY 283 00:10:13,579 --> 00:10:15,081 FIRST THAT THEY TOOK ON. 284 00:10:15,147 --> 00:10:19,885 AT THE TIME, IT WAS RAGING BACK 285 00:10:19,952 --> 00:10:23,155 IN THE MID 2000s, 2006 IS WHEN 286 00:10:23,222 --> 00:10:25,024 THIS WAS PUBLISHED, THAT 287 00:10:25,091 --> 00:10:25,791 PERIODONTAL DISEASE CAUSED 288 00:10:25,858 --> 00:10:28,260 EVERYTHING. 289 00:10:28,327 --> 00:10:32,465 FROM ALZHEIMER'S TO IN THIS CASE 290 00:10:32,531 --> 00:10:34,700 PRETERM BIRTH TO CARDIOVASCULAR 291 00:10:34,767 --> 00:10:35,634 ACCIDENT, ET CETERA, ET CETERA, 292 00:10:35,701 --> 00:10:35,901 ET CETERA. 293 00:10:35,968 --> 00:10:37,403 I MEAN, IT WAS THE CAUSE OF 294 00:10:37,470 --> 00:10:38,804 EVERYTHING. 295 00:10:38,871 --> 00:10:41,774 AND PERIODONTISTS WERE REALLY 296 00:10:41,841 --> 00:10:43,209 EXCITED BECAUSE THEY WERE 297 00:10:43,275 --> 00:10:45,411 CONTRIBUTING TO OVERALL 298 00:10:45,478 --> 00:10:47,346 MAINTENANCE OF SYSTEMIC HEALTH. 299 00:10:47,413 --> 00:10:48,748 NOW, IF YOU FAST FORWARD, THAT'S 300 00:10:48,814 --> 00:10:50,082 ACTUALLY TRUE, BECAUSE IT DOES 301 00:10:50,149 --> 00:10:53,352 REDUCE THE OVERALL BURDEN OF 302 00:10:53,419 --> 00:10:56,422 CHRONIC INFLAMMATION. 303 00:10:56,489 --> 00:10:57,957 BUT SPECIFICALLY, DID 304 00:10:58,024 --> 00:10:59,925 PERIODONTAL DISEASE INDUCE THE 305 00:10:59,992 --> 00:11:02,995 RISK OF PRETERM BIRTH? 306 00:11:03,062 --> 00:11:06,999 SO BRIAN MICHALOW ICZ AND HIS 307 00:11:07,066 --> 00:11:07,700 COLLEAGUES IN MINNESOTA ACTUALLY 308 00:11:07,767 --> 00:11:09,001 DID THE DEFINITIVE TRIAL, AND IF 309 00:11:09,068 --> 00:11:14,473 YOU CAN READ THE CONCLUSIONS, 310 00:11:14,540 --> 00:11:16,308 TREATMENT OF PERIODONTITIS IN 311 00:11:16,375 --> 00:11:17,843 PREGNANT WOMEN IMPROVED 312 00:11:17,910 --> 00:11:19,145 PERIODONTAL DISEASE, THAT'S 313 00:11:19,211 --> 00:11:20,679 GOOD, IT PROVES PERIODONTAL 314 00:11:20,746 --> 00:11:22,648 THERAPY WORKS FOR PERIODONTAL 315 00:11:22,715 --> 00:11:24,750 DISEASE, BUT DOES NOT 316 00:11:24,817 --> 00:11:25,918 SIGNIFICANTLY ALTER PRETERM 317 00:11:25,985 --> 00:11:27,887 BIRTH, LOW BIRTH WEIGHT OR FETAL 318 00:11:27,953 --> 00:11:32,158 GROWTH RESTRICTION. 319 00:11:32,224 --> 00:11:34,360 A NEGATIVE FINDING PUBLISHED IN 320 00:11:34,427 --> 00:11:35,094 THE NEW ENGLAND JOURNAL OF 321 00:11:35,161 --> 00:11:36,328 MEDICINE. 322 00:11:36,395 --> 00:11:38,064 SO THE MESSAGE IS THAT IF YOU DO 323 00:11:38,130 --> 00:11:40,900 A STUDY REALLY WELL, YOU CAN 324 00:11:40,966 --> 00:11:43,636 PUBLISH NULL RESULTS. 325 00:11:43,702 --> 00:11:45,738 BUT THE STUDY'S GOT TO BE DONE 326 00:11:45,805 --> 00:11:46,906 REALLY, REALLY WELL. 327 00:11:46,972 --> 00:11:48,941 NOW, I BECAME A PARIAH IN THE 328 00:11:49,008 --> 00:11:50,309 PERIODONTAL RESEARCH COMMUNITY 329 00:11:50,376 --> 00:11:52,178 TO THIS DAY, I AM NOT INVITED TO 330 00:11:52,244 --> 00:11:55,247 ANY MEETINGS WITH PERIODONTISTS, 331 00:11:55,314 --> 00:11:58,851 OKAY, BUT THAT'S OKAY. 332 00:11:58,918 --> 00:12:00,586 BUT THE POINT OF THIS IS THAT 333 00:12:00,653 --> 00:12:01,754 NULL RESULTS CAN BE PUBLISHED, 334 00:12:01,821 --> 00:12:03,556 THEY CAN BE PUBLISHED IN 335 00:12:03,622 --> 00:12:05,758 OUTSTANDING JOURNALS, BUT IT 336 00:12:05,825 --> 00:12:07,493 ONLY IS IF IT'S DONE IN THE MOST 337 00:12:07,560 --> 00:12:11,564 RIGOROUS WAY POSSIBLE. 338 00:12:11,630 --> 00:12:13,299 AND THAT WAS AN EXPENSIVE BUT 339 00:12:13,365 --> 00:12:15,167 IMPORTANT LESSON THAT AT LEAST 340 00:12:15,234 --> 00:12:16,302 THE DENTAL COMMUNITY LEARNED, 341 00:12:16,368 --> 00:12:17,903 AND I THINK OTHER COMMUNITIES 342 00:12:17,970 --> 00:12:22,475 ARE LEARNING THIS ALONG THE WAY. 343 00:12:22,541 --> 00:12:23,909 NOW THE QUESTION IS, WHEN THIS 344 00:12:23,976 --> 00:12:27,313 GOES TO THE REVIEW BOARD FOR 345 00:12:27,379 --> 00:12:30,349 TENURE AND PROMOTION AT YOUR 346 00:12:30,416 --> 00:12:31,684 INSTITUTION, DOES THIS COUNT 347 00:12:31,750 --> 00:12:33,519 BECAUSE IT'S A NULL RESULT? 348 00:12:33,586 --> 00:12:36,388 NOW, IT'S A GOOD PUBLICATION, 349 00:12:36,455 --> 00:12:37,256 IT'S NEW ENGLAND JOURNAL OF 350 00:12:37,323 --> 00:12:38,023 MEDICINE. 351 00:12:38,090 --> 00:12:39,658 BUT DOES IT COUNT? 352 00:12:39,725 --> 00:12:40,593 BECAUSE WHAT DIFFERENCE HAVE YOU 353 00:12:40,659 --> 00:12:43,496 MADE IN PRACTICE? 354 00:12:43,562 --> 00:12:43,696 RIGHT? 355 00:12:43,762 --> 00:12:45,231 AND THOSE OF YOU WHO SIT ON 356 00:12:45,297 --> 00:12:47,433 THESE TENURE AND PROMOTION 357 00:12:47,500 --> 00:12:49,034 BOARDS, IT WILL BE FASCINATING 358 00:12:49,101 --> 00:12:51,370 WHAT YOU ALL THINK ABOUT THAT -- 359 00:12:51,437 --> 00:12:53,005 JUST AS AN EXAMPLE OBVIOUSLY BUT 360 00:12:53,072 --> 00:12:54,440 THERE ARE MANY, MANY MORE 361 00:12:54,507 --> 00:12:56,408 EXAMPLES LIKE THIS. 362 00:12:56,475 --> 00:12:58,043 NOW, I JUST WANTED TO SHARE WITH 363 00:12:58,110 --> 00:13:00,646 YOU SOME GUIDING PRINCIPLES THAT 364 00:13:00,713 --> 00:13:02,515 DR. BERTAGNOLLI HAS BEEN 365 00:13:02,581 --> 00:13:05,584 ARTICULATING BECAUSE IT REALLY 366 00:13:05,651 --> 00:13:06,552 BEARS ON WHAT YOU ALL ARE GOING 367 00:13:06,619 --> 00:13:08,587 TO BE DOING HERE DURING THIS 368 00:13:08,654 --> 00:13:11,056 CONFERENCE. 369 00:13:11,123 --> 00:13:12,024 MONICA HAS INDICATED THAT OUR 370 00:13:12,091 --> 00:13:13,893 WORK IS NOT FINISHED WHEN WE 371 00:13:13,959 --> 00:13:14,994 DIFFERENT A SCIENTIFIC 372 00:13:15,060 --> 00:13:15,427 DISCOVERY. 373 00:13:15,494 --> 00:13:17,396 THE WORK IS FINISHED WHEN PEOPLE 374 00:13:17,463 --> 00:13:18,597 ARE LIVING LONG AND HEALTHY 375 00:13:18,664 --> 00:13:21,433 LIVES. 376 00:13:21,500 --> 00:13:22,635 SO IT'S NOT JUST THE 377 00:13:22,701 --> 00:13:23,936 PUBLICATION, IT'S HOW THAT IS 378 00:13:24,003 --> 00:13:27,439 TRANSLATED INTO PRACTICE. 379 00:13:27,506 --> 00:13:28,641 THE RESEARCH COMMUNITY IS NOT 380 00:13:28,707 --> 00:13:29,842 ONLY THE LABORATORY, IT'S NOT 381 00:13:29,909 --> 00:13:31,076 ONLY THE CLINIC, BUT IT'S ALSO 382 00:13:31,143 --> 00:13:31,510 THE COMMUNITY. 383 00:13:31,577 --> 00:13:33,245 SHE REALLY WANTS TO STRETCH 384 00:13:33,312 --> 00:13:34,446 THINGS OUT AT THE LEVEL OF WHERE 385 00:13:34,513 --> 00:13:36,849 PEOPLE ARE. 386 00:13:36,916 --> 00:13:39,084 AND FINALLY, PROGRESS IS 387 00:13:39,151 --> 00:13:40,786 ACCELERATED WHEN ADVANCED 388 00:13:40,853 --> 00:13:42,121 SCIENTIFIC METHODS SUCH AS THE 389 00:13:42,188 --> 00:13:44,423 NEW DATA ANALYTICS, ARE APPLIED 390 00:13:44,490 --> 00:13:46,825 TO DATA THAT INCLUDES EVERYONE, 391 00:13:46,892 --> 00:13:48,360 AND WHEN NEW DISCOVERIES ARE 392 00:13:48,427 --> 00:13:51,096 RAPIDLY AND EQUITABLY, EQUITABLY 393 00:13:51,163 --> 00:13:53,699 ADOPTED IN CLINICAL CARE. 394 00:13:53,766 --> 00:13:58,404 NOW EACH ONE OF THESE POINTS 395 00:13:58,470 --> 00:13:59,071 BEARS ON THE CONVERSATION THAT 396 00:13:59,138 --> 00:14:00,272 YOU ALL WILL BE HAVING OVER THE 397 00:14:00,339 --> 00:14:03,642 NECK FEW DAYS, A -- THE NEXT FEW 398 00:14:03,709 --> 00:14:04,977 DAYS, AND AS YOU THINK THROUGH 399 00:14:05,044 --> 00:14:07,780 THE PERPLEXING, CHALLENGING 400 00:14:07,846 --> 00:14:09,448 ISSUES THAT YOU'RE WRESTLING 401 00:14:09,515 --> 00:14:10,983 WITH, SEE HOW THEY ALIGN OR 402 00:14:11,050 --> 00:14:12,518 DON'T WITH SOME OF THE THINGS 403 00:14:12,585 --> 00:14:14,720 THAT THE NEW NIH DIRECTOR HAS 404 00:14:14,787 --> 00:14:19,191 BEEN DISCUSSING. 405 00:14:19,258 --> 00:14:22,928 SO WITH THAT, MY EMAIL ADDRESS 406 00:14:22,995 --> 00:14:24,396 IF YOU WANT TO SEND ME ANY HATE 407 00:14:24,463 --> 00:14:25,431 MAIL. 408 00:14:25,497 --> 00:14:27,299 I APOLOGIZE FOR PARACHUTING IN 409 00:14:27,366 --> 00:14:28,801 AND PARACHUTING OUT, BUT I HAVE 410 00:14:28,867 --> 00:14:30,236 TWO MORE THINGS LIKE THIS I HAVE 411 00:14:30,302 --> 00:14:32,438 TO DO THIS MORNING, BECAUSE 412 00:14:32,504 --> 00:14:33,872 DR. BERTAGNOLLI IS ON TRAVEL, 413 00:14:33,939 --> 00:14:35,140 BUT IT WAS REALLY DELIGHTFUL TO 414 00:14:35,207 --> 00:14:36,408 BE HERE WITH YOU THIS MORNING, 415 00:14:36,475 --> 00:14:39,545 AND I REALLY HOPE THE CONFERENCE 416 00:14:39,612 --> 00:14:41,180 GOALS WELL AND I LOOK FORWARD, 417 00:14:41,247 --> 00:14:42,381 SHAI, TO HEARING WHAT YOU ALL 418 00:14:42,448 --> 00:14:42,915 CAME UP WITH. 419 00:14:42,982 --> 00:14:44,016 SO AGAIN, THANK YOU ALL VERY 420 00:14:44,083 --> 00:14:54,193 MUCH. 421 00:14:55,761 --> 00:14:57,029 >> GOOD MORNING TO YOU ALL. 422 00:14:57,096 --> 00:14:58,364 WOW, THIS IS SO EXCITING. 423 00:14:58,430 --> 00:14:59,331 IT'S BEEN SO LONG IN THE 424 00:14:59,398 --> 00:15:00,532 WAITING. 425 00:15:00,599 --> 00:15:02,401 EVEN THOUGH HE PARACHUTED OUT, I 426 00:15:02,468 --> 00:15:04,903 WOULD LIKE TO THANK DR. TABAK 427 00:15:04,970 --> 00:15:07,206 AGAIN FOR HIS VERY SUPPORTIVE 428 00:15:07,273 --> 00:15:07,640 COMMENTS. 429 00:15:07,706 --> 00:15:09,508 HE'S REALLY BEEN A TRUE CHAMPION 430 00:15:09,575 --> 00:15:12,244 FOR RIGOR AT THE NIH FOR MANY 431 00:15:12,311 --> 00:15:13,979 YEARS, AND HIS EFFORTS HAVE 432 00:15:14,046 --> 00:15:16,949 REALLY INCREASED ATTENTION TO 433 00:15:17,016 --> 00:15:19,785 SCIENTIFIC RIGOR BY NIH. 434 00:15:19,852 --> 00:15:22,588 AS YOU MIGHT HAVE SURMISED, I'M 435 00:15:22,655 --> 00:15:23,722 SHAI SILBERBERG. 436 00:15:23,789 --> 00:15:28,227 I'VE BEEN FOR MANY YEARS 437 00:15:28,294 --> 00:15:29,528 CHAMPIONING RIGOR AND 438 00:15:29,595 --> 00:15:32,498 TRANSPARENCY AT NINDS. 439 00:15:32,564 --> 00:15:34,233 AND THE ISSUE OF PUBLICATION 440 00:15:34,300 --> 00:15:35,401 BIAS HAS BEEN A MAJOR CONCERN OF 441 00:15:35,467 --> 00:15:37,069 THE OFFICE OF RESEARCH QUALITY 442 00:15:37,136 --> 00:15:41,640 AT NINDS FOR QUITE SOME TIME, SO 443 00:15:41,707 --> 00:15:42,975 WE'RE REALLY EXCITED ABOUT THE 444 00:15:43,042 --> 00:15:43,942 DISCUSSION WE'RE GOING TO HAVE 445 00:15:44,009 --> 00:15:47,012 IN THESE NEXT TWO DAYS. 446 00:15:47,079 --> 00:15:48,447 SO I WANT TO START BY THANKING 447 00:15:48,514 --> 00:15:50,282 ALL THE INVITEES WHO TOOK FROM 448 00:15:50,349 --> 00:15:52,484 THEIR PRECIOUS TIME TO COME AND 449 00:15:52,551 --> 00:15:56,855 DEVOTE TO THIS WORKSHOP. 450 00:15:56,922 --> 00:16:00,159 AND YOUR BREADTH OF EXPERTISE, 451 00:16:00,225 --> 00:16:02,761 ACADEMIC DISCIPLINES, AND CAREER 452 00:16:02,828 --> 00:16:04,396 STAGES, I'M SURE WILL GREATLY 453 00:16:04,463 --> 00:16:07,700 ENRICH OUR DISCUSSIONS ABOUT 454 00:16:07,766 --> 00:16:10,369 BIAS AGAINST NULL RESULTS. 455 00:16:10,436 --> 00:16:11,837 WHICH PERMEATE MOST IF NOT ALL 456 00:16:11,904 --> 00:16:14,273 SCIENTIFIC FIELDS. 457 00:16:14,340 --> 00:16:16,442 I ALSO WANT TO THANK NINDS 458 00:16:16,508 --> 00:16:20,713 LEADERSHIP FOR THEIR CONTINUOUS 459 00:16:20,779 --> 00:16:24,116 AND WHOLEHEARTED SUPPORT. 460 00:16:24,183 --> 00:16:27,619 THE NINDS RIGOR WORKING GROUP, 461 00:16:27,686 --> 00:16:30,389 WHICH PROVIDED GREAT ADVICE 462 00:16:30,456 --> 00:16:31,657 DURING THE PLANNING STAGES OF 463 00:16:31,724 --> 00:16:36,462 THIS WORKSHOP, THE LOGISTICS 464 00:16:36,528 --> 00:16:46,305 TEAM, TOYA ROGERS, NINDS, HOLLY 465 00:16:46,372 --> 00:16:47,406 RILEY AND THE REST OF THE TEAM, 466 00:16:47,473 --> 00:16:48,707 AND LAST BUT CERTAINLY NOT 467 00:16:48,774 --> 00:16:51,810 LEAST, DEVON AND MARIAH, WHO DID 468 00:16:51,877 --> 00:16:53,412 MOST OF THE HEAVY LIFTING 469 00:16:53,479 --> 00:16:55,981 LEADING UP TO THIS DAY. 470 00:16:56,048 --> 00:16:57,750 SO I HAVE A DISCLAIMER TO MAKE 471 00:16:57,816 --> 00:16:59,385 BEFORE WE MOVE ON, AND THAT'S 472 00:16:59,451 --> 00:17:01,053 THE OPINIONS THAT THE THREE OF 473 00:17:01,120 --> 00:17:03,155 US WILL LIKELY VOICE ARE NOT 474 00:17:03,222 --> 00:17:05,891 OFFICIAL OPINIONS OF NIH. 475 00:17:05,958 --> 00:17:06,892 THAT'S AN IMPORTANT POINT TO 476 00:17:06,959 --> 00:17:07,726 MAKE. 477 00:17:07,793 --> 00:17:11,130 SO IN A NUTSHELL, THE WORKSHOP 478 00:17:11,196 --> 00:17:12,664 IS STRUCTURED AS FOLLOWS. 479 00:17:12,731 --> 00:17:14,833 WE'RE GOING TO HAVE WHAT I HOPE 480 00:17:14,900 --> 00:17:17,603 WILL BE A VERY EXCITING AND -- 481 00:17:17,669 --> 00:17:19,438 NOT HOPEFUL BUT I'M SURE A VERY 482 00:17:19,505 --> 00:17:21,173 EXCITING KEYNOTE BY EDWARD 483 00:17:21,240 --> 00:17:21,540 MIGUEL. 484 00:17:21,607 --> 00:17:25,043 THIS WILL BE FOLLOWED BY FIVE 485 00:17:25,110 --> 00:17:26,011 SESSIONS WHERE WE WILL DISCUSS 486 00:17:26,078 --> 00:17:27,646 THE CURRENT STATE OF AFFAIRS. 487 00:17:27,713 --> 00:17:30,916 THE FIRST FOUR WILL TAKE PLACE 488 00:17:30,983 --> 00:17:33,886 TODAY, AND THE LAST ONE WILL BE 489 00:17:33,952 --> 00:17:35,087 FIRST ONE TOMORROW MORNING. 490 00:17:35,154 --> 00:17:38,257 THESE SESSIONS WILL BE FOLLOWED 491 00:17:38,323 --> 00:17:39,358 BY WORKING GROUPS. 492 00:17:39,425 --> 00:17:41,527 WE'LL BREAK OUT INTO WORKING 493 00:17:41,593 --> 00:17:42,628 GROUPS, WHERE EACH WORKING GROUP 494 00:17:42,694 --> 00:17:44,263 WILL BE CHARGED WITH ONE GROUP 495 00:17:44,329 --> 00:17:47,566 OF STAKEHOLDERS TO DISCUSS WHAT 496 00:17:47,633 --> 00:17:49,101 THAT STAKEHOLDER COULD POSSIBLY 497 00:17:49,168 --> 00:17:53,238 DO TO HELP REDUCE THE BIAS 498 00:17:53,305 --> 00:17:54,239 AGAINST NULL STUDIES. 499 00:17:54,306 --> 00:17:55,541 THE WORKING GROUPS WILL THEN 500 00:17:55,607 --> 00:17:59,044 REPORT BACK TO THE ASSEMBLY, AND 501 00:17:59,111 --> 00:18:00,579 THAT'S WHEN, AFTER THAT, WHERE 502 00:18:00,646 --> 00:18:02,014 THE RUBBER REALLY HITS THE ROAD 503 00:18:02,080 --> 00:18:03,182 WHERE WE'LL HAVE THIS LAST 504 00:18:03,248 --> 00:18:05,617 SESSION WHERE HOPEFULLY WE'LL BE 505 00:18:05,684 --> 00:18:07,453 ABLE TO IDENTIFY NOVEL 506 00:18:07,519 --> 00:18:09,988 STRATEGIES THAT SIGNIFICANTLY 507 00:18:10,055 --> 00:18:11,190 REDUCE PUBLICATION BIAS. 508 00:18:11,256 --> 00:18:12,724 AND THE HOPE IS THEN TO SHARE 509 00:18:12,791 --> 00:18:15,093 THESE IDEAS WITH THE SCIENTIFIC 510 00:18:15,160 --> 00:18:19,264 COMMUNITY TO ACCELERATE CHANGE. 511 00:18:19,331 --> 00:18:21,366 I'M SHOWING YOU HERE THE 512 00:18:21,433 --> 00:18:23,635 MASTERPIECE BY MATISSE TO MAKE 513 00:18:23,702 --> 00:18:24,736 TWO IMPORTANT POINTS. 514 00:18:24,803 --> 00:18:27,139 ONE IS THAT I'M SURE AS WE WILL 515 00:18:27,206 --> 00:18:28,774 DISCOVER THROUGHOUT THIS 516 00:18:28,841 --> 00:18:31,076 WORKSHOP, NO SINGLE STAKEHOLDER 517 00:18:31,143 --> 00:18:32,344 CAN SOLVE PUBLICATION BIAS ON 518 00:18:32,411 --> 00:18:33,812 THEIR OWN. 519 00:18:33,879 --> 00:18:36,782 IT WILL TAKE A CONCERTED EFFORT. 520 00:18:36,849 --> 00:18:38,217 AND THE SECOND POINT I WANT TO 521 00:18:38,283 --> 00:18:39,318 BRING UP IS THAT THE STRUCTURE 522 00:18:39,384 --> 00:18:40,719 OF THE WORKSHOP IS SUCH THAT 523 00:18:40,786 --> 00:18:43,088 MOST OF THE TIME WILL BE DEVOTED 524 00:18:43,155 --> 00:18:46,291 TO DISCUSSIONS. 525 00:18:46,358 --> 00:18:48,160 AND HOPEFULLY HAND IN HAND, WITH 526 00:18:48,227 --> 00:18:48,927 ALL THE GREAT MINDS WE HAVE 527 00:18:48,994 --> 00:18:51,096 HERE, WE WILL BE ABLE TO 528 00:18:51,163 --> 00:18:53,065 IDENTIFY NOVEL STRATEGIES THAT 529 00:18:53,131 --> 00:18:54,399 COULD SIGNIFICANTLY REDUCE 530 00:18:54,466 --> 00:18:57,302 PUBLICATION BIAS. 531 00:18:57,369 --> 00:18:58,437 BUT FIRST WE HAVE TO DEFINE WHAT 532 00:18:58,504 --> 00:19:01,507 WE MEAN BY PUBLICATIONS. 533 00:19:01,573 --> 00:19:03,909 SO IN THIS REGARD, I WOULD FIRST 534 00:19:03,976 --> 00:19:06,879 LIKE TO PERSONALLY DEDICATE THIS 535 00:19:06,945 --> 00:19:10,682 WORKSHOP TO ROBERT ROSENTHAL. 536 00:19:10,749 --> 00:19:12,117 ROBERT ROSENTHAL PASSED AWAY 537 00:19:12,184 --> 00:19:14,520 EARLIER THIS YEAR, AND HE'S 538 00:19:14,586 --> 00:19:16,054 FAMOUSLY KNOWN FOR COINING THE 539 00:19:16,121 --> 00:19:17,789 TERM "THE FILE DRAWER PROBLEM." 540 00:19:17,856 --> 00:19:20,325 HE WROTE, FOR ANY GIVEN RESEARCH 541 00:19:20,392 --> 00:19:21,527 AREA, ONE CANNOT TELL HOW MANY 542 00:19:21,593 --> 00:19:22,828 STUDIES HAVE BEEN CONDUCTED BUT 543 00:19:22,895 --> 00:19:24,596 NEVER REPORTED. 544 00:19:24,663 --> 00:19:25,998 THE EXTREME VIEW OF THE FILE 545 00:19:26,064 --> 00:19:27,633 DRAWER PROBLEM IS THAT JOURNALS 546 00:19:27,699 --> 00:19:28,834 ARE FILLED WITH THE 5% OF 547 00:19:28,901 --> 00:19:30,769 STUDIES THAT SHOW TYPE 1 ERRORS 548 00:19:30,836 --> 00:19:31,904 WHILE THE FILE DRAWERS ARE 549 00:19:31,970 --> 00:19:33,438 FILLED WITH THE 95% OF STUDIES 550 00:19:33,505 --> 00:19:34,673 THAT SHOW NON-SIGNIFICANT 551 00:19:34,740 --> 00:19:36,041 RESULTS. 552 00:19:36,108 --> 00:19:38,143 CLEARLY, HE HAD A CYNICAL SENSE 553 00:19:38,210 --> 00:19:40,746 OF HUMOR. 554 00:19:40,812 --> 00:19:43,282 BUT YOU CAN SEE HERE I 555 00:19:43,348 --> 00:19:45,551 HIGHLIGHTED THE WORD "STUDIES," 556 00:19:45,617 --> 00:19:46,518 AND THAT'S REALLY IMPORTANT, 557 00:19:46,585 --> 00:19:48,520 BECAUSE HE WAS TALKING ABOUT 558 00:19:48,587 --> 00:19:50,389 STUDIES AND NOT RESULTS, AND 559 00:19:50,455 --> 00:19:52,024 THAT IS IMPORTANT FOR HOW WE'RE 560 00:19:52,090 --> 00:19:54,326 GOING TO DESCRIBE AND THINK 561 00:19:54,393 --> 00:19:55,427 ABOUT PUBLICATION BIAS IN THE 562 00:19:55,494 --> 00:19:58,363 CONTEXT OF THIS WORKSHOP. 563 00:19:58,430 --> 00:19:59,898 TO UNDERSTAND THIS DIFFERENCE, 564 00:19:59,965 --> 00:20:01,333 LET'S LOOK AT THE EARLIEST 565 00:20:01,400 --> 00:20:02,634 ACCOUNT OF WHAT WAS CONSIDERED 566 00:20:02,701 --> 00:20:06,471 TO BE PUBLICATION BIAS. 567 00:20:06,538 --> 00:20:09,174 MARCUS CICERO, IN THE FIRST 568 00:20:09,241 --> 00:20:10,742 CENTURY B.C., DESCRIBED AN EENT 569 00:20:10,809 --> 00:20:13,011 THAT TOOK PLACE 300 YEARS 570 00:20:13,078 --> 00:20:13,345 EARLIER. 571 00:20:13,412 --> 00:20:17,182 AND HE WROTE, DIAGORAS, WHO IS 572 00:20:17,249 --> 00:20:20,919 CALLED THE ATHEIST, BEING AT 573 00:20:20,986 --> 00:20:22,087 SAMOTHRACE, ONE OF HIS FRIENDS 574 00:20:22,154 --> 00:20:23,422 SHOWED HIM SEVERAL PICTURES OF 575 00:20:23,488 --> 00:20:25,157 PEOPLE WHO HAD ENDURED VERY 576 00:20:25,223 --> 00:20:25,958 DANGEROUS STORMS. 577 00:20:26,024 --> 00:20:28,226 SEE, SAYS HE, YOU WHO DENY A 578 00:20:28,293 --> 00:20:29,094 PROVIDENCE, HOW MANY HAVE BEEN 579 00:20:29,161 --> 00:20:30,562 SAVED BY THEIR PRAYERS TO THE 580 00:20:30,629 --> 00:20:31,396 GODS? 581 00:20:31,463 --> 00:20:33,899 AY, SAYS DIAGORAS, I SEE THOSE 582 00:20:33,966 --> 00:20:35,133 WHO WERE SAVED BUT WHERE ARE 583 00:20:35,200 --> 00:20:36,501 THOSE PAINTED WHO WERE 584 00:20:36,568 --> 00:20:37,102 SHIPWRECKED? 585 00:20:37,169 --> 00:20:38,637 SO IS THIS PUBLICATION BIAS? 586 00:20:38,704 --> 00:20:39,504 WELL, IT DEPENDS. 587 00:20:39,571 --> 00:20:42,541 TO UNDERSTAND WHY IT DEPENDS, 588 00:20:42,608 --> 00:20:45,077 LET'S DO AN EXPERIMENT. 589 00:20:45,143 --> 00:20:48,914 LET'S PRETEND THAT WE'RE AN 590 00:20:48,981 --> 00:20:49,781 ANCIENT GREEK SCIENTIST AND WE 591 00:20:49,848 --> 00:20:51,917 WANT TO TEST THE HYPOTHESIS THAT 592 00:20:51,984 --> 00:20:53,285 IF YOU PRAY TO THE GODS, YOU 593 00:20:53,352 --> 00:20:55,487 WILL BE SAVED FROM DANGEROUS 594 00:20:55,554 --> 00:20:57,089 STORMS. 595 00:20:57,155 --> 00:20:59,391 SO THE SCIENTIST SENDS THREE 596 00:20:59,458 --> 00:21:00,826 SHIPS TO SAIL THROUGH A VERY 597 00:21:00,892 --> 00:21:01,927 DANGEROUS STORM AND INSTRUCTS 598 00:21:01,994 --> 00:21:04,262 THE CREW TO PRAY TO THE GODS TO 599 00:21:04,329 --> 00:21:07,065 SAVE THEIR LIVES. 600 00:21:07,132 --> 00:21:08,900 TWO SHIPS MAKE IT ACROSS, ONE 601 00:21:08,967 --> 00:21:10,335 GOES DOWN. 602 00:21:10,402 --> 00:21:11,870 NOW SINCE TWO SHIPS MADE IT 603 00:21:11,937 --> 00:21:14,439 ACROSS, THE SCIENTIST SAYS, OH, 604 00:21:14,506 --> 00:21:15,907 WOW, MY HYPOTHESIS SEEMS TO 605 00:21:15,974 --> 00:21:17,876 WORK, YOU KNOW, THE SHIP WENT 606 00:21:17,943 --> 00:21:18,977 DOWN, MAYBE BECAUSE THEY DIDN'T 607 00:21:19,044 --> 00:21:20,646 PRAY HARD ENOUGH OR MAYBE THEY 608 00:21:20,712 --> 00:21:22,280 FORGOT TO PRAY. 609 00:21:22,347 --> 00:21:24,249 SO LET'S JUST PUBLISH THE 610 00:21:24,316 --> 00:21:26,218 RESULTS THAT SHOW THAT MY 611 00:21:26,284 --> 00:21:27,285 HYPOTHESIS IS CORRECT. 612 00:21:27,352 --> 00:21:29,388 AND HE PUBLISHES IT AS A MURAL 613 00:21:29,454 --> 00:21:31,690 OR MOSAIC IN THE TEMPLE. 614 00:21:31,757 --> 00:21:34,159 THIS IS NOT PUBLICATION BIAS IN 615 00:21:34,226 --> 00:21:36,695 THE CONTEXT OF THIS WORKSHOP. 616 00:21:36,762 --> 00:21:38,463 THIS IS SELECTIVE REPORTING. 617 00:21:38,530 --> 00:21:40,132 OKAY? 618 00:21:40,198 --> 00:21:44,569 BECAUSE NOT ALL THE RESULTS WERE 619 00:21:44,636 --> 00:21:45,003 NULL. 620 00:21:45,070 --> 00:21:47,205 IN CONTRAST, LET'S PRETEND THAT 621 00:21:47,272 --> 00:21:48,073 ALL THREE SHIPS WENT DOWN. 622 00:21:48,140 --> 00:21:51,810 IN THIS CASE, CLEARLY THE 623 00:21:51,877 --> 00:21:52,978 HYPOTHESIS WERE NOT SUPPORTED BY 624 00:21:53,045 --> 00:21:54,546 THE RESULTS OF THE EXPERIMENT. 625 00:21:54,613 --> 00:21:56,815 SO EITHER THE INVESTIGATOR OR 626 00:21:56,882 --> 00:21:58,684 THE TEMPLE CURATORS DECIDED THAT 627 00:21:58,750 --> 00:22:00,452 THESE RESULTS ARE NOT 628 00:22:00,519 --> 00:22:03,488 INTERESTING AND NOT WORTHY OF 629 00:22:03,555 --> 00:22:04,623 PUBLICATION, AND THEY REMAIN 630 00:22:04,690 --> 00:22:05,357 SILENT ABOUT IT. 631 00:22:05,424 --> 00:22:07,092 THIS IS THE KIND OF PUBLICATION 632 00:22:07,159 --> 00:22:08,427 BIAS WE'RE TALKING ABOUT HERE, 633 00:22:08,493 --> 00:22:09,194 OKAY? 634 00:22:09,261 --> 00:22:10,929 WHERE THE ENTIRE STUDY IS NULL. 635 00:22:10,996 --> 00:22:12,130 NOW, AN ENTIRE STUDY COULD BE 636 00:22:12,197 --> 00:22:13,498 EVEN A SINGLE EXPERIMENT, BUT 637 00:22:13,565 --> 00:22:15,767 IT'S THE WHOLE STUDY. 638 00:22:15,834 --> 00:22:17,536 WAS NULL. 639 00:22:17,602 --> 00:22:18,270 OKAY. 640 00:22:18,336 --> 00:22:23,842 SO THAT CONCEPT THAT IT'S 641 00:22:23,909 --> 00:22:25,711 IMPORTANT TO REPORT ALL STUDIES, 642 00:22:25,777 --> 00:22:27,546 NOT ONLY THOSE THAT SUPPORT THE 643 00:22:27,612 --> 00:22:31,717 HYPOTHESIS, IS BY NO MEANS NEW. 644 00:22:31,783 --> 00:22:34,252 SO CLAUDE BERNARD IN 1865 WROTE, 645 00:22:34,319 --> 00:22:36,188 IF A HYPOTHESIS IS NOT VERIFIED 646 00:22:36,254 --> 00:22:38,924 AND DISAPPEARS, THE FACTS WHICH 647 00:22:38,990 --> 00:22:40,358 IT HAS ENABLED US TO FIND ARE 648 00:22:40,425 --> 00:22:43,195 NONT THE LESS ACQUIRED AS 649 00:22:43,261 --> 00:22:43,895 INDESTRUCTIBLE MATERIALS FOR 650 00:22:43,962 --> 00:22:44,730 SCIENCE. 651 00:22:44,796 --> 00:22:48,500 SO HOW OVE ARE INDESTRUCTIBLE 652 00:22:48,567 --> 00:22:51,503 MATERIALS FOR SCIENCE NOT SHA 653 00:22:51,570 --> 00:22:52,204 SHARED? 654 00:22:52,270 --> 00:22:53,238 THAT'S THE TOPIC FOR DISCUSSION 655 00:22:53,305 --> 00:22:57,676 IN THE FIRST SESSION. 656 00:22:57,743 --> 00:22:59,177 SADLY, ERICK TURNER WHO WAS 657 00:22:59,244 --> 00:23:00,612 SUPPOSED TO MODERATE THE SESSION 658 00:23:00,679 --> 00:23:04,449 COULDN'T MAKE IT SO MARIAH AND 659 00:23:04,516 --> 00:23:05,851 DEVON WILL CO-MODERATE THE 660 00:23:05,917 --> 00:23:06,718 SESSION. 661 00:23:06,785 --> 00:23:08,820 AND WE'LL HAVE VERY INTERESTING 662 00:23:08,887 --> 00:23:13,024 DISCUSSIONS, I'M SURE. 663 00:23:13,091 --> 00:23:16,061 WE'LL HEAR SOME DATA -- SOME -- 664 00:23:16,128 --> 00:23:19,264 WHY AM I BLANKING ON THE WORD? 665 00:23:19,331 --> 00:23:21,199 WE'LL GET SOME DATA FROM REAL 666 00:23:21,266 --> 00:23:23,301 EXPERIMENTS AS WELL AS MANY DATA 667 00:23:23,368 --> 00:23:26,104 FROM MODELS, AND THE PROBLEM 668 00:23:26,171 --> 00:23:27,439 WITH MODELS ARE THAT THEY ALL 669 00:23:27,506 --> 00:23:28,940 ARE BASED ON ASSUMPTIONS, AND 670 00:23:29,007 --> 00:23:31,376 WHEN YOU HAVE ASSUMPTIONS, WE'RE 671 00:23:31,443 --> 00:23:32,577 PROBABLY GOING TO HEAR A BROAD 672 00:23:32,644 --> 00:23:35,647 RANGE OF OPINIONS OF WHAT THE 673 00:23:35,714 --> 00:23:36,982 MAGNITUDE OF PUBLICATION BIAS 674 00:23:37,048 --> 00:23:39,251 IS. 675 00:23:39,317 --> 00:23:41,686 I PERSONALLY PREFER GOOD OLD 676 00:23:41,753 --> 00:23:44,756 DATA AS A SOURCE OF INFORMATION. 677 00:23:44,823 --> 00:23:47,025 SO I'M GOING TO SHOW YOU TWO 678 00:23:47,092 --> 00:23:50,095 STUDIES, EMPIRICAL STUDIES 679 00:23:50,162 --> 00:23:51,296 RELATED TO PUBLICATION BIAS. 680 00:23:51,363 --> 00:23:53,698 IN THE FIRST, THEY LOOKED AT 681 00:23:53,765 --> 00:23:57,202 PROGNOSTIC MARKERS, AND THEY 682 00:23:57,269 --> 00:23:58,970 CHECKED OUT TWO DATABASES. 683 00:23:59,037 --> 00:24:00,272 IN THE FIRST DATABASE, THEY HAD 684 00:24:00,338 --> 00:24:02,507 A TOTAL OF 304 ARTICLES AND THEY 685 00:24:02,574 --> 00:24:04,209 FOUND THAT ONLY 4 WERE FULLY 686 00:24:04,276 --> 00:24:05,210 NEGATIVE FOR ALL PRESENTED 687 00:24:05,277 --> 00:24:08,246 RESULTS IN THE ABSTRACT. 688 00:24:08,313 --> 00:24:09,347 A SECOND DATABASE THEY LOOKED AT 689 00:24:09,414 --> 00:24:13,185 HAD A TOTAL OF 1,575 ARTICLES, 690 00:24:13,251 --> 00:24:16,021 AND ONLY 21 OF THEM WERE 691 00:24:16,087 --> 00:24:17,222 NEGATIVE. 692 00:24:17,289 --> 00:24:19,090 THE SECOND STUDY FOCUSED ON 693 00:24:19,157 --> 00:24:21,359 ANIMAL STUDIES OF ACUTE ISCHEMIC 694 00:24:21,426 --> 00:24:25,330 STROKE, AND THEY FOUND THAT ONLY 695 00:24:25,397 --> 00:24:27,899 6, 1.2%, DID NOT REPORT AT LEAST 696 00:24:27,966 --> 00:24:30,569 ONE SIGNIFICANT FINDING. 697 00:24:30,635 --> 00:24:32,103 CLEARLY, OUR WORKING HYPOTHESIS 698 00:24:32,170 --> 00:24:34,272 CAN'T BE CORRECT 98% OF THE 699 00:24:34,339 --> 00:24:34,472 TIME. 700 00:24:34,539 --> 00:24:35,807 I MEAN, IF THAT WAS THE CASE, 701 00:24:35,874 --> 00:24:37,876 WE'RE REALLY NOT DOING 702 00:24:37,943 --> 00:24:40,178 CUTTING-EDGE SCIENCE, RIGHT? 703 00:24:40,245 --> 00:24:43,148 SO THERE SEEMS TO BE, YOU KNOW, 704 00:24:43,215 --> 00:24:46,351 AN ISSUE THAT THERE ARE STUDIES 705 00:24:46,418 --> 00:24:47,752 THAT AREN'T PUBLISHED, MANY OF 706 00:24:47,819 --> 00:24:50,522 THEM PROBABLY, AND THEY CAN HAVE 707 00:24:50,589 --> 00:24:54,192 SEVERE CONSEQUENCES. 708 00:24:54,259 --> 00:24:57,996 ONE SEVERE CONSEQUENCE IS, 709 00:24:58,063 --> 00:25:01,199 ORVETION FINANCIAL. 710 00:25:01,266 --> 00:25:01,499 OF 711 00:25:01,566 --> 00:25:01,933 COURSE, FINANCIAL. 712 00:25:02,000 --> 00:25:03,802 SO WHAT I'M SHOWING YOU HERE IS 713 00:25:03,869 --> 00:25:05,003 THE DISTRIBUTION OF FUNDS BY 714 00:25:05,070 --> 00:25:07,038 EACH OF THE INSTITUTES AT NIH IN 715 00:25:07,105 --> 00:25:12,010 FISCAL YEAR 2023, DEVOTED TO THE 716 00:25:12,077 --> 00:25:13,511 MOST COMMON FUNDING MECHANISM AT 717 00:25:13,578 --> 00:25:16,915 THE NIH, THE R01 MECHANISM. 718 00:25:16,982 --> 00:25:19,784 IN TOTAL, IN FISCAL YEAR '23, 719 00:25:19,851 --> 00:25:21,920 NIH SPENT CLOSE TO $16 BILLION 720 00:25:21,987 --> 00:25:24,256 TO SUPPORT THE R01 MECHANISM. 721 00:25:24,322 --> 00:25:26,658 NOW LET'S BE VERY GENEROUS, AND 722 00:25:26,725 --> 00:25:28,193 ASSUME THAT 80% OF THE TIME WHEN 723 00:25:28,260 --> 00:25:30,262 WE COME UP WITH AN HYPOTHESIS, 724 00:25:30,328 --> 00:25:31,696 WE'RE CORRECT. 725 00:25:31,763 --> 00:25:33,865 AND OF THE 20% WHERE THE RESULTS 726 00:25:33,932 --> 00:25:35,400 ARE NULL, HALF OF THEM, WE DO 727 00:25:35,467 --> 00:25:37,168 NOT PUBLISH, OKAY? 728 00:25:37,235 --> 00:25:39,571 IT MEANS TAKE 10% OF OUR TOTAL 729 00:25:39,638 --> 00:25:40,872 STUDIES AREN'T PUBLISHED. 730 00:25:40,939 --> 00:25:43,408 IN THAT CASE, IT'S NOT VERY 731 00:25:43,475 --> 00:25:46,578 DIFFICULT MATH, CLOSE TO 732 00:25:46,645 --> 00:25:52,918 $1.6 BILLION IN FISCAL YEAR '23 733 00:25:52,984 --> 00:25:55,854 WERE WASTED DUE TO LOSS OF 734 00:25:55,921 --> 00:25:57,923 KNOWLEDGE JUST FROM THE 735 00:25:57,989 --> 00:25:58,256 R01 MECHANISM. 736 00:25:58,323 --> 00:26:00,458 BUT IT GOES WELL BEYOND THE 737 00:26:00,525 --> 00:26:01,126 DIRECT FINANCIAL LOSS. 738 00:26:01,192 --> 00:26:02,427 IF RESULTS ARE NOT SHARED, 739 00:26:02,494 --> 00:26:03,628 STUDIES ARE NOT SHARED, THEN 740 00:26:03,695 --> 00:26:05,363 OTHER SCIENTISTS WILL PROBABLY 741 00:26:05,430 --> 00:26:06,798 REPEAT THOSE EXPERIMENTS, AND 742 00:26:06,865 --> 00:26:07,999 THEY ALSO WON'T SEE ANYTHING SO 743 00:26:08,066 --> 00:26:09,634 THEY WON'T REPORT THAT. 744 00:26:09,701 --> 00:26:12,704 AND THIS CAN GO ON SEVERAL 745 00:26:12,771 --> 00:26:13,371 TIMES. 746 00:26:13,438 --> 00:26:15,206 AND THAT'S TREMENDOUS LOSS OF 747 00:26:15,273 --> 00:26:21,346 TIME, EFFORT AND RESOURCES. 748 00:26:21,413 --> 00:26:23,315 FURTHERMORE, WHEN WE DO 749 00:26:23,381 --> 00:26:24,749 META-ANALYSIS, PUBLICATION BIAS 750 00:26:24,816 --> 00:26:26,251 AND P-HACKING WHICH WE'LL HEAR 751 00:26:26,318 --> 00:26:29,220 ABOUT LATER ON, WILL LEAD TO 752 00:26:29,287 --> 00:26:30,855 MISLEADING OR EXAGGERATED EFFECT 753 00:26:30,922 --> 00:26:31,489 SIZES. 754 00:26:31,556 --> 00:26:33,491 NOW IF EFFECT SIZE IS 755 00:26:33,558 --> 00:26:34,926 EXAGGERATED, FUTURE STUDIES WILL 756 00:26:34,993 --> 00:26:36,561 LIKELY BE UNDERPOWERED BECAUSE 757 00:26:36,628 --> 00:26:39,164 THEY BASE WHAT THEY'RE GOING TO 758 00:26:39,230 --> 00:26:41,132 USE BASED ON THIS FICTITIOUS 759 00:26:41,199 --> 00:26:41,733 OVERESTIMATION OF THE EFFECT 760 00:26:41,800 --> 00:26:42,667 SIZE. 761 00:26:42,734 --> 00:26:45,203 AND THESE UNDERPOWERED STUDIES 762 00:26:45,270 --> 00:26:46,838 NOW COULD LEAD TO LOTS OF FALSE 763 00:26:46,905 --> 00:26:48,907 POSITIVES, AND, THEREFORE, MIGHT 764 00:26:48,974 --> 00:26:50,041 LEAD -- THAT'S FUNNY. 765 00:26:50,108 --> 00:26:51,309 WE'RE STUCK HERE. 766 00:26:51,376 --> 00:26:55,914 THERE WE GO. 767 00:26:55,981 --> 00:26:57,482 TO BASELESS CLINICAL TRIALS. 768 00:26:57,549 --> 00:26:59,217 SO GIVEN THE GRAVITY OF 769 00:26:59,284 --> 00:27:02,053 PUBLICATION BIAS, WHY ARE SO FEW 770 00:27:02,120 --> 00:27:06,558 STUDIES ACTUALLY PUBLISHED? 771 00:27:06,624 --> 00:27:09,094 SO THAT WILL BE THE TOPIC OF 772 00:27:09,160 --> 00:27:10,295 DISCUSSIONS IN SESSIONS 773 00:27:10,362 --> 00:27:11,496 2 THROUGH 5. 774 00:27:11,563 --> 00:27:14,332 THE SECOND SESSION WILL FOCUS 775 00:27:14,399 --> 00:27:15,967 SPECIFICALLY ON DISCUSSIONS 776 00:27:16,034 --> 00:27:19,938 RELATED TO TRADITIONAL 777 00:27:20,005 --> 00:27:21,039 MANUSCRIPTS, AND IN PREPARATION 778 00:27:21,106 --> 00:27:31,649 FOR THIS, MARIAH DID A VERY 779 00:27:32,650 --> 00:27:33,051 INTERESTING ANALYSIS. 780 00:27:33,118 --> 00:27:34,686 WE ARE COMMONLY -- IT'S KIND OF 781 00:27:34,753 --> 00:27:37,088 LIKE YOU KNOW, EVERYONE TALKS 782 00:27:37,155 --> 00:27:38,823 ABOUT THAT THERE ARE JOURNALS 783 00:27:38,890 --> 00:27:40,692 THAT ACCEPT NULL STUDIES. 784 00:27:40,759 --> 00:27:41,793 AND YOU'RE LIKE OH, I DON'T KNOW 785 00:27:41,860 --> 00:27:46,931 WHAT YOU'RE TALKING ABOUT, PLUS 786 00:27:46,998 --> 00:27:50,769 ONE EXCEPTIONAL STUDIES. 787 00:27:50,835 --> 00:27:52,070 SO THE MAJORITY OF THEM DO NOT 788 00:27:52,137 --> 00:27:53,705 TRACK IT AND THE TWO THAT GOT 789 00:27:53,772 --> 00:27:54,806 NUMBERS, AS YOU CAN SEE, THE 790 00:27:54,873 --> 00:27:59,611 FRACTION IS VERY, VERY SMALL. 791 00:27:59,677 --> 00:28:03,348 MARIAH ALSO DID A VERY 792 00:28:03,415 --> 00:28:04,549 INTERESTING ANALYSIS, SHE WENT 793 00:28:04,616 --> 00:28:07,585 THROUGH THE WEB PAGES OF 215 794 00:28:07,652 --> 00:28:10,588 NEUROSCIENCE JOURNALS THAT 795 00:28:10,655 --> 00:28:14,826 PUBLISHED PRE-CLINICAL RESEARCH, 796 00:28:14,893 --> 00:28:16,928 AND HAD AN IMPACT FACTOR OF 2 OR 797 00:28:16,995 --> 00:28:17,162 GREATER. 798 00:28:17,228 --> 00:28:18,563 SHE FOUND THE MAJORITY OF THEM 799 00:28:18,630 --> 00:28:20,965 DON'T SAY ANYTHING ON THE 800 00:28:21,032 --> 00:28:22,467 WEBSITE ABOUT ACCEPTING NULL 801 00:28:22,534 --> 00:28:23,468 STUDIES. 802 00:28:23,535 --> 00:28:25,236 AND THERE ARE 35 THAT DO AND 803 00:28:25,303 --> 00:28:28,206 THEY CAN BE DIVIDED INTO TWO 804 00:28:28,273 --> 00:28:29,607 CATEGORIES, 14 ACCEPTED THEM 805 00:28:29,674 --> 00:28:31,142 UNCONDITIONALLY, THEY JUST SAID 806 00:28:31,209 --> 00:28:32,777 WE ACCEPT NULL STUDIES. 807 00:28:32,844 --> 00:28:35,280 21 OF THEM HAD VARIOUS 808 00:28:35,346 --> 00:28:36,614 CONDITIONS, LIKE YOU CAN ONLY 809 00:28:36,681 --> 00:28:38,016 PUBLISH AS A BRIEF OR SHORT 810 00:28:38,083 --> 00:28:39,784 REPORT, AND SO ON AND SO FORTH, 811 00:28:39,851 --> 00:28:42,954 AND PERHAPS WHAT IS MOST 812 00:28:43,021 --> 00:28:46,124 INTERESTING IS THESE TWO 813 00:28:46,191 --> 00:28:47,592 JOURNALS THAT SAID SINCE THE NET 814 00:28:47,659 --> 00:28:49,727 EFFECT OF NEGATIVE RESULTS IS TO 815 00:28:49,794 --> 00:28:51,362 DISCOURAGE REPETITION, THE 816 00:28:51,429 --> 00:28:52,797 STANDARD FOR ACCEPTANCE AS 817 00:28:52,864 --> 00:28:54,966 NEGATIVE RESULTS WILL BE HIGHLY 818 00:28:55,033 --> 00:28:55,500 DEMANDING. 819 00:28:55,567 --> 00:28:57,068 I MEAN, THIS IS VERY STRANGE IN 820 00:28:57,135 --> 00:29:00,371 MY OPINION. 821 00:29:00,438 --> 00:29:01,773 BECAUSE IT SUGGESTS THAT 822 00:29:01,840 --> 00:29:02,807 STANDARDS FOR ACCEPTING POSITIVE 823 00:29:02,874 --> 00:29:05,176 RESULTS WILL NOT BE HIGHLY 824 00:29:05,243 --> 00:29:05,443 DEMANDING. 825 00:29:05,510 --> 00:29:06,811 SO THAT KIND OF LINKING INTO 826 00:29:06,878 --> 00:29:08,379 WHAT DR. TABAK SAID, YOU KNOW, 827 00:29:08,446 --> 00:29:10,348 IF YOU DO STUDIES REALLY 828 00:29:10,415 --> 00:29:11,783 CAREFULLY, YOU CAN GET INTO -- 829 00:29:11,850 --> 00:29:15,120 YOU KNOW, BUT ALL STUDIES SHOULD 830 00:29:15,186 --> 00:29:16,754 BE DONE VERY CAREFULLY AND ALL 831 00:29:16,821 --> 00:29:18,389 STUDIES SHOULD BE REVIEWED VERY 832 00:29:18,456 --> 00:29:23,428 CAREFULLY, ET CETERA. 833 00:29:23,495 --> 00:29:26,831 SO MARIAH REACHED OUT TO THE 834 00:29:26,898 --> 00:29:28,199 EDITORS OF THESE 35 JOURNALS 835 00:29:28,266 --> 00:29:30,602 THAT MENTIONED NULL RESULTS. 836 00:29:30,668 --> 00:29:31,536 ONLY FOUR PROVIDED NUMBERS ON 837 00:29:31,603 --> 00:29:33,204 HOW MUCH THEY PUBLISH AND AS YOU 838 00:29:33,271 --> 00:29:34,839 CAN SEE, THE NUMBERS ARE VERY 839 00:29:34,906 --> 00:29:37,575 SMALL. 840 00:29:37,642 --> 00:29:39,611 SO PEOPLE HAD -- THE QUESTION 841 00:29:39,677 --> 00:29:41,112 IS, WHY IS THIS THE CASE? 842 00:29:41,179 --> 00:29:42,680 WHY -- I MEAN, IF NULL RESULTS 843 00:29:42,747 --> 00:29:44,883 ARE SO IMPORTANT, WHY ARE SO FEW 844 00:29:44,949 --> 00:29:47,619 STUDIES PUBLISHED? 845 00:29:47,685 --> 00:29:52,223 NULL RESULTS PUBLISHED? 846 00:29:52,290 --> 00:29:55,260 AND ONE POSSIBLE REASON IS THAT 847 00:29:55,326 --> 00:29:56,694 REVIEWERS MIGHT REVIEW THEM MORE 848 00:29:56,761 --> 00:29:58,329 HARSHLY. 849 00:29:58,396 --> 00:30:00,632 AND HERE I'M GOING TO DESCRIBE A 850 00:30:00,698 --> 00:30:03,034 VERY INTERESTING STUDY BY 851 00:30:03,101 --> 00:30:03,801 EMERSON AND COLLEAGUE. 852 00:30:03,868 --> 00:30:07,839 THEY GENERATED TWO VERSIONS OF A 853 00:30:07,906 --> 00:30:09,707 WELL-DESIGNED RANDOMIZED CONTROL 854 00:30:09,774 --> 00:30:12,877 TRIAL, AND THEY ONLY DIFFERED IN 855 00:30:12,944 --> 00:30:14,412 THE OUT COME, WHETHER IT WAS 856 00:30:14,479 --> 00:30:16,281 POSITIVE OR NULL. 857 00:30:16,347 --> 00:30:17,615 AND THEY SUBMITTED TO TWO 858 00:30:17,682 --> 00:30:19,884 JOURNALS WHO WERE PART OF THIS 859 00:30:19,951 --> 00:30:22,086 STUDY, AND THE JOURNALS IN PLACE 860 00:30:22,153 --> 00:30:23,521 SENT THESE OUT TO REVIEWERS TO 861 00:30:23,588 --> 00:30:25,256 REVIEW OBVIOUSLY EACH REVIEWER 862 00:30:25,323 --> 00:30:28,326 GOT JUST ONE OF THE VERSIONS, 863 00:30:28,393 --> 00:30:29,861 AND WHEN THEY TABULATED, THEY 864 00:30:29,928 --> 00:30:31,396 GOT COMMENTS BACK FROM 210 865 00:30:31,462 --> 00:30:32,263 REVIEWERS AND WHEN THEY 866 00:30:32,330 --> 00:30:36,434 TABULATED THEM, YOU CAN SEE THAT 867 00:30:36,501 --> 00:30:38,603 97% RECOMMENDED ACCEPTING THE 868 00:30:38,670 --> 00:30:39,938 MANUSCRIPT FOR PUBLICATION IF 869 00:30:40,004 --> 00:30:42,006 THE RESULTS WERE POSITIVE, WHILE 870 00:30:42,073 --> 00:30:43,741 ONLY 80% DID THAT WHEN THE 871 00:30:43,808 --> 00:30:48,179 RESULTS WERE NULL. 872 00:30:48,246 --> 00:30:51,683 SO WE HAVE AN ISSUE HERE. 873 00:30:51,749 --> 00:30:55,820 SO IN GIVING THAT FEW JOURNALS 874 00:30:55,887 --> 00:30:56,688 ACTUALLY SPECIFICALLY SAY WE 875 00:30:56,754 --> 00:30:58,456 REALLY WOULD LIKE YOU TO SUBMIT 876 00:30:58,523 --> 00:31:00,425 NULL RESULTS AT LEAST IN THE 877 00:31:00,491 --> 00:31:02,193 NEUROSCIENCES, AND THE NOTION, 878 00:31:02,260 --> 00:31:03,728 WHICH IS VERY COMMON, THAT 879 00:31:03,795 --> 00:31:05,930 REVIEWERS WILL TREAT IT MORE 880 00:31:05,997 --> 00:31:08,533 HARSHLY, AS MY EDITORS AS WELL, 881 00:31:08,600 --> 00:31:12,570 IT WOULD NOT BE SURPRISING IF 882 00:31:12,637 --> 00:31:14,038 THE SCIENTIFIC COMMUNITY WOULD 883 00:31:14,105 --> 00:31:17,308 BE RELUCTANT TO GO TO THE 884 00:31:17,375 --> 00:31:18,943 TROUBLE TO WRITE UP PAPERS AND 885 00:31:19,010 --> 00:31:20,144 SUBMIT THEM IF THE STUDY IS 886 00:31:20,211 --> 00:31:20,478 NULL. 887 00:31:20,545 --> 00:31:24,415 SO IN THIS REGARD, THIS IS A 888 00:31:24,482 --> 00:31:26,951 VERY INTERESTING STUDY BY FRANCO 889 00:31:27,018 --> 00:31:28,052 AND COLLEAGUES. 890 00:31:28,119 --> 00:31:31,456 THEY LOOKED AT THE FATE OF 891 00:31:31,522 --> 00:31:32,657 221 SURVEY-BASED STUDIES IN THE 892 00:31:32,724 --> 00:31:38,463 SOCIAL SCIENCES. 893 00:31:38,529 --> 00:31:40,965 I NEED MORE COFFEE. 894 00:31:41,032 --> 00:31:42,900 THAT WERE ALL FUNDED BY THE 895 00:31:42,967 --> 00:31:44,369 NATIONAL SCIENCE FOUNDATION. 896 00:31:44,435 --> 00:31:46,237 AND WHAT THEY FOUND IS THAT IF 897 00:31:46,304 --> 00:31:48,072 THE RESULTS OF THE SURVEY WERE 898 00:31:48,139 --> 00:31:50,041 STRONG AND THEY DEFINE STRONG AS 899 00:31:50,108 --> 00:31:51,576 ALL OR MOST OF THE HYPOTHESES 900 00:31:51,643 --> 00:31:53,611 WERE SUPPORTED BY THE 901 00:31:53,678 --> 00:31:56,214 STATISTICAL TESTS, ONLY 4% WERE 902 00:31:56,281 --> 00:31:58,049 NOT WRITTEN UP BY THE 903 00:31:58,116 --> 00:31:58,716 INVESTIGATORS. 904 00:31:58,783 --> 00:32:01,653 HOWEVER, IF THE RESULTS WERE 905 00:32:01,719 --> 00:32:05,723 NULL, 64% WERE NOT WRITTEN UP. 906 00:32:05,790 --> 00:32:08,993 SO WE CAN SEE HERE, YOU KNOW, 907 00:32:09,060 --> 00:32:11,763 THAT THERE IS AN ISSUE. 908 00:32:11,829 --> 00:32:14,499 AND THIS KIND OF, I GUESS, 909 00:32:14,565 --> 00:32:17,869 LED -- NOT THIS PAPER BUT BY THE 910 00:32:17,935 --> 00:32:26,444 WAY, NEIL MALHOTRA IS THE 911 00:32:26,511 --> 00:32:27,211 CORRESPONDING AUTHOR ON THIS 912 00:32:27,278 --> 00:32:31,149 STUDY AND I'M SURE HE'LL GIVE -- 913 00:32:31,215 --> 00:32:32,450 ON THE STUDY WHEN HE SPEAKS. 914 00:32:32,517 --> 00:32:33,885 BUT THE SCIENTIFIC COMMUNITY 915 00:32:33,951 --> 00:32:35,520 FELT THAT MAYBE WHAT IS NEEDED 916 00:32:35,586 --> 00:32:37,055 ARE DEDICATED JOURNALS TO NULL 917 00:32:37,121 --> 00:32:39,524 RESULTS SO THAT THE REVIEWERS 918 00:32:39,590 --> 00:32:41,426 WILL ALL BE ON THE SAME PAGE, 919 00:32:41,492 --> 00:32:42,460 THEY KNOW THEY'RE GOING TO BE 920 00:32:42,527 --> 00:32:43,961 LOOKING AT NULL STUDY, AND THAT 921 00:32:44,028 --> 00:32:47,131 WILL SOLVE OUR PROBLEM. 922 00:32:47,198 --> 00:32:48,132 AND OF COURSE THE FLIP SIDE OF 923 00:32:48,199 --> 00:32:49,767 THAT IS SAYING OH, NULL RESULTS 924 00:32:49,834 --> 00:32:50,868 SHOULD HAVE THEIR OWN SEPARATE 925 00:32:50,935 --> 00:32:55,139 PLACE IN THE UNIVERSE. 926 00:32:55,206 --> 00:32:56,674 TO OUR KNOWLEDGE, SINCE 2002, 927 00:32:56,741 --> 00:32:58,743 THERE HAVE BEEN EIGHT CREDIBLE 928 00:32:58,810 --> 00:33:02,513 JOURNALS FOR NULL STUDIES. 929 00:33:02,580 --> 00:33:04,449 AND THEY DID NOT FARE VERY WELL. 930 00:33:04,515 --> 00:33:06,184 IF YOU LOOK IN TOTAL, THEY 931 00:33:06,250 --> 00:33:08,519 PUBLISHED JUST 604 ARTICLES 932 00:33:08,586 --> 00:33:10,154 SINCE 2002. 933 00:33:10,221 --> 00:33:14,625 TO PUT THINGS IN PERSPECTIVE, 934 00:33:14,692 --> 00:33:16,594 PNAS ALONE PUBLISHED MORE THAN 935 00:33:16,661 --> 00:33:17,795 77,000 RESEARCH ARTICLES IN THE 936 00:33:17,862 --> 00:33:18,563 SAME TIME PERIOD. 937 00:33:18,629 --> 00:33:20,198 OKAY? 938 00:33:20,264 --> 00:33:22,233 SO CLEARLY, THIS IS NOT THE 939 00:33:22,300 --> 00:33:25,169 SOLUTION. 940 00:33:25,236 --> 00:33:28,106 SO PERHAPS WHAT IS NEEDED -- 941 00:33:28,172 --> 00:33:29,974 WHAT'S THIS? 942 00:33:30,041 --> 00:33:32,276 AM I ADMITTING TOYA HERE? 943 00:33:32,343 --> 00:33:33,277 WHO'S ADMITTING HER? 944 00:33:33,344 --> 00:33:35,480 MIKE? 945 00:33:35,546 --> 00:33:37,014 THANK YOU. 946 00:33:37,081 --> 00:33:39,317 SO CLEARLY, YOU KNOW, 947 00:33:39,384 --> 00:33:40,618 TRADITIONAL JOURNALS DON'T SEEM 948 00:33:40,685 --> 00:33:42,920 TO BE DOING -- MAYBE WE COULD 949 00:33:42,987 --> 00:33:44,322 FOCUS ON NON-TRADITIONAL 950 00:33:44,389 --> 00:33:44,555 FORMATS. 951 00:33:44,622 --> 00:33:45,857 SO THAT'S GOING TO BE THE TOPIC 952 00:33:45,923 --> 00:33:50,495 OF DISCUSSION IN SESSION THREE. 953 00:33:50,561 --> 00:33:52,029 AND WE'LL HEAR ABOUT INTERESTING 954 00:33:52,096 --> 00:33:55,733 DIFFERENT MECHANISMS LIKE 955 00:33:55,800 --> 00:33:57,368 PREPRINTS, LIKE REGISTERED 956 00:33:57,435 --> 00:34:00,204 REPORTS, LIKE MICROPUBLICATIONS, 957 00:34:00,271 --> 00:34:03,274 AND ALSO INTERESTING VENUES LIKE 958 00:34:03,341 --> 00:34:08,746 OCTOPUS AND ARCADIA SCIENCE. 959 00:34:08,813 --> 00:34:10,982 PAUL STERNBERG, WHO'S ON THIS 960 00:34:11,048 --> 00:34:12,884 PANEL, HAD A VERY INTERESTING -- 961 00:34:12,950 --> 00:34:15,119 IS RUNNING A VERY INTERESTING 962 00:34:15,186 --> 00:34:16,788 INITIATIVE WHICH I WANTED TO 963 00:34:16,854 --> 00:34:18,423 BRIEFLY MENTION. 964 00:34:18,489 --> 00:34:20,291 SO IT'S CALLED MICRO PUBLICATION 965 00:34:20,358 --> 00:34:21,259 BIOLOGY. 966 00:34:21,325 --> 00:34:23,127 AND IT PUBLISHES SINGLE 967 00:34:23,194 --> 00:34:24,228 VALIDATED RESULTS THAT INCLUDE 968 00:34:24,295 --> 00:34:27,265 NOVEL FINDINGS, NEGATIVE AND/OR 969 00:34:27,331 --> 00:34:28,132 REPRODUCIBLE RESULTS, AND 970 00:34:28,199 --> 00:34:29,133 RESULTS THAT ARE PERCEIVED TO 971 00:34:29,200 --> 00:34:31,135 LACK HIGH IMPACT. 972 00:34:31,202 --> 00:34:34,238 AND THEY DID EVERYTHING TO MAKE 973 00:34:34,305 --> 00:34:39,010 IT EASY AND I WOULD SAY 974 00:34:39,076 --> 00:34:41,212 WORTHWHILE FOR SCIENTISTS TO 975 00:34:41,279 --> 00:34:43,181 SUBMIT TO MICRO PUBLICATION, SO 976 00:34:43,247 --> 00:34:45,817 THERE'S JUST A NOMINAL FEE FOR 977 00:34:45,883 --> 00:34:46,684 SUBMISSION. 978 00:34:46,751 --> 00:34:48,886 IT'S PEER REVIEWED, IT'S 979 00:34:48,953 --> 00:34:49,987 PUBLISHED ONLINE, IT CAN BE 980 00:34:50,054 --> 00:34:53,491 FOUND ON PUBMED, AND THE DATA IS 981 00:34:53,558 --> 00:34:55,159 INCLUDED, UNDERLYING DATA WHICH, 982 00:34:55,226 --> 00:34:56,561 OF COURSE, ADDS CREDIBILITY TO 983 00:34:56,627 --> 00:34:58,863 THE MICROPUBLICATION. 984 00:34:58,930 --> 00:35:00,731 NEVERTHELESS, FROM OUR ANALYSIS, 985 00:35:00,798 --> 00:35:04,068 FROM THE 808 MICROPUBLICATIONS 986 00:35:04,135 --> 00:35:07,605 THAT HAVE BEEN PUBLISHED, ONLY 987 00:35:07,672 --> 00:35:10,041 54% WERE DESIGNATE BY MICROPUBL 988 00:35:10,107 --> 00:35:12,109 DAITION BIOLOGY AS NEGATIVE 989 00:35:12,176 --> 00:35:13,110 RESULTS. 990 00:35:13,177 --> 00:35:14,378 SO WE CAN HAVE A DISCUSSION 991 00:35:14,445 --> 00:35:15,847 WHETHER THIS IS A LOT OR A 992 00:35:15,913 --> 00:35:18,883 LITTLE. 993 00:35:18,950 --> 00:35:21,018 I THINK GIVEN THE MAGNITUDE OF 994 00:35:21,085 --> 00:35:24,255 THE LIKELIHOOD THAT HYPOTHESIS 995 00:35:24,322 --> 00:35:25,590 WILL NOT BE CORRECT, I WOULD 996 00:35:25,656 --> 00:35:27,024 GUESS THIS NUMBER SHOULD BE 997 00:35:27,091 --> 00:35:28,826 HIGHER. 998 00:35:28,893 --> 00:35:34,565 OKAY. 999 00:35:34,632 --> 00:35:38,736 SO THE NEXT ISSUE I WANT TO 1000 00:35:38,803 --> 00:35:40,505 DISCUSS, WHICH IS SESSION FOUR, 1001 00:35:40,571 --> 00:35:43,140 WILL FOCUS ON INSTITUTIONS. 1002 00:35:43,207 --> 00:35:47,411 BECAUSE INSTITUTIONS CAN ALSO 1003 00:35:47,478 --> 00:35:49,146 CONTRIBUTE AS A NEGATIVE FORCE 1004 00:35:49,213 --> 00:35:50,681 TO SUBMISSION OF NULL RESULTS. 1005 00:35:50,748 --> 00:35:54,852 AND TO EXPLAIN THIS POINT, LET'S 1006 00:35:54,919 --> 00:35:56,087 JUST SAY THE FOLLOWING. 1007 00:35:56,153 --> 00:35:59,257 SO WE KNOW THAT RESEARCH IS -- 1008 00:35:59,323 --> 00:36:01,325 WHAT DRIVES RESEARCH IS FUNDS. 1009 00:36:01,392 --> 00:36:03,094 AND THAT TYPICALLY COMES THROUGH 1010 00:36:03,160 --> 00:36:03,761 GRANT SUPPORT. 1011 00:36:03,828 --> 00:36:05,396 AND IN ORDER TO GET GRANT 1012 00:36:05,463 --> 00:36:07,899 SUPPORT, YOU HAVE TO SHOW THAT 1013 00:36:07,965 --> 00:36:10,535 YOU ARE A PRODUCTIVE SCIENTIST, 1014 00:36:10,601 --> 00:36:12,737 WHICH MEANS THAT YOU PUBLISHED A 1015 00:36:12,803 --> 00:36:15,606 LOT, AND IN THE ERA OF 1016 00:36:15,673 --> 00:36:16,908 BIBLIOMETRICS, THAT YOU HAVE TO 1017 00:36:16,974 --> 00:36:18,776 PUBLISH IN WHAT THEY CALL HIGH 1018 00:36:18,843 --> 00:36:19,644 PROFILE JOURNALS. 1019 00:36:19,710 --> 00:36:21,312 AND THAT MEANS THAT YOU HAVE 1020 00:36:21,379 --> 00:36:23,147 TO -- IT HAS TO HAVE A HIGH 1021 00:36:23,214 --> 00:36:24,382 IMPACT FACTOR, FOR THAT YOUR 1022 00:36:24,448 --> 00:36:26,050 RESULTS HAVE TO BE NOVEL, 1023 00:36:26,117 --> 00:36:27,451 INNOVATIVE AND SIGNIFICANT, AND 1024 00:36:27,518 --> 00:36:29,220 THIS CYCLE OCCURS AT LEAST IN 1025 00:36:29,287 --> 00:36:33,057 THE U.S. EVERY FOUR YEARS OR SO, 1026 00:36:33,124 --> 00:36:34,859 WHICH MEANS THAT IT'S UNLIKELY 1027 00:36:34,926 --> 00:36:36,227 THAT YOU WILL SPEND THE TIME TO 1028 00:36:36,294 --> 00:36:38,029 TRY TO WRITE UP A PAPER WHICH IS 1029 00:36:38,095 --> 00:36:39,363 NULL BECAUSE IT'S UNLIKELY THAT 1030 00:36:39,430 --> 00:36:40,865 IT'S GOING TO MAKE ITS WAY INTO 1031 00:36:40,932 --> 00:36:42,700 A HIGH PROFILE JOURNAL, AND EVEN 1032 00:36:42,767 --> 00:36:44,535 IF IT DOES, YOU DON'T KNOW HOW 1033 00:36:44,602 --> 00:36:45,403 THE INSTITUTIONS ARE GOING TO 1034 00:36:45,469 --> 00:36:47,238 TREAT IT BECAUSE THEY WANT YOU 1035 00:36:47,305 --> 00:36:50,107 TO PUBLISH IN HIGH PROFILE 1036 00:36:50,174 --> 00:36:51,275 JOURNALS, AND IF YOU PUBLISH IT 1037 00:36:51,342 --> 00:36:53,844 IN THE LOWER LEVEL JOURNAL, THEN 1038 00:36:53,911 --> 00:36:57,048 IT WON'T LOOK GOOD ON YOUR 1039 00:36:57,114 --> 00:36:58,583 TENURE AS DR. TABAK MENTIONED, 1040 00:36:58,649 --> 00:37:00,117 ON YOUR TENURE AND PROMOTION 1041 00:37:00,184 --> 00:37:02,386 COMMITTEES. 1042 00:37:02,453 --> 00:37:03,554 SO OF COURSE THIS IS AN 1043 00:37:03,621 --> 00:37:04,422 IMPORTANT ISSUE THAT WE'RE GOING 1044 00:37:04,488 --> 00:37:08,125 TO BE DISCUSSING. 1045 00:37:08,192 --> 00:37:10,661 SO THIS IS SESSION FOUR HERE, 1046 00:37:10,728 --> 00:37:13,497 WHICH WILL DISCUSS NOT ONLY 1047 00:37:13,564 --> 00:37:17,902 WE'LL HEAR ABOUT FROM ERIN 1048 00:37:17,969 --> 00:37:21,839 MCKIERNAN ABOUT ISSUES THAT WHAT 1049 00:37:21,906 --> 00:37:24,275 INSTITUTIONS DON'T DO TO SUPPORT 1050 00:37:24,342 --> 00:37:28,446 NULL STUDIES, BUT WE'LL ALSO 1051 00:37:28,512 --> 00:37:29,513 HAVE LOTS OF INTERESTING 1052 00:37:29,580 --> 00:37:30,548 DISCUSSIONS OF WHAT INSTITUTIONS 1053 00:37:30,615 --> 00:37:32,183 ARE DOING TO SUPPORT NULL 1054 00:37:32,250 --> 00:37:32,617 STUDIES. 1055 00:37:32,683 --> 00:37:33,718 IN SESSION FIVE, WE'LL TAKE A 1056 00:37:33,784 --> 00:37:35,620 MUCH MORE BROADER LOOK ON WHAT 1057 00:37:35,686 --> 00:37:38,222 CAN INCENTIVIZE REPORTING OF 1058 00:37:38,289 --> 00:37:41,726 NULL STUDIES, AND IN THIS 1059 00:37:41,792 --> 00:37:43,394 SESSION, WE'LL BE DISCUSSING 1060 00:37:43,461 --> 00:37:48,099 INITIATIVES SUCH AS THE 1061 00:37:48,165 --> 00:37:48,933 REPRODUCIBILITY NETWORKS THAT 1062 00:37:49,000 --> 00:37:52,603 ARE SPROUTING ACROSS THE WORLD, 1063 00:37:52,670 --> 00:37:58,643 AND VENUES SUCH AS -- SORRY, AND 1064 00:37:58,709 --> 00:38:03,781 OTHER INITIATIVES SUCH AS THE 1065 00:38:03,848 --> 00:38:09,453 PRIZE GIVEN OUT BY THE -- AT 1066 00:38:09,520 --> 00:38:13,290 VETERAN SCIENCES AND ALSO WHAT 1067 00:38:13,357 --> 00:38:14,725 SITE -- IS TRYING TO DO TO 1068 00:38:14,792 --> 00:38:15,526 ADDRESS PUBLICATION BIAS. 1069 00:38:15,593 --> 00:38:16,794 SO IN SUMMARY, WHAT I'VE SHOWED 1070 00:38:16,861 --> 00:38:18,295 YOU IN THIS BRIEF INTRODUCTION 1071 00:38:18,362 --> 00:38:21,899 IS LIMITED DATA SUGGESTING THE 1072 00:38:21,966 --> 00:38:23,267 PUBLICATION BIAS IS A BIG 1073 00:38:23,334 --> 00:38:25,336 PROBLEM THAT SHOULD BE SOLVED. 1074 00:38:25,403 --> 00:38:28,939 AND BACK TO OUR FAMOUS MATISSE 1075 00:38:29,006 --> 00:38:31,375 PICTURE, I THINK IT WILL BE 1076 00:38:31,442 --> 00:38:34,278 CLEAR THAT ALL THE STAKEHOLDERS 1077 00:38:34,345 --> 00:38:38,049 WHO SOMEHOW HAVE TO WORK 1078 00:38:38,115 --> 00:38:39,917 TOGETHER TO ADDRESS PUBLICATION 1079 00:38:39,984 --> 00:38:44,021 BIAS, AND THAT WILL BE THE TASK 1080 00:38:44,088 --> 00:38:45,022 AHEAD OF US IN THE NEXT COUPLE 1081 00:38:45,089 --> 00:38:46,090 OF DAYS AND IT WILL BE VERY 1082 00:38:46,157 --> 00:38:47,091 INTERESTING TO SEE WHAT WE COME 1083 00:38:47,158 --> 00:38:47,658 UP WITH. 1084 00:38:47,725 --> 00:38:48,926 I JUST WANT TO END WITH A LITTLE 1085 00:38:48,993 --> 00:38:50,795 BIT OF LOGISTICS, WHICH PROBABLY 1086 00:38:50,861 --> 00:38:52,697 SHOULD HAVE BEEN THE FIRST 1087 00:38:52,763 --> 00:38:56,834 SLIDE. 1088 00:38:56,901 --> 00:38:58,469 SO FOR COFFEE AND SNACKS AND THE 1089 00:38:58,536 --> 00:38:59,570 LUNCH BOXES FOR THOSE WHO 1090 00:38:59,637 --> 00:39:01,338 ORDERED THEM, WHEN YOU EXIT THE 1091 00:39:01,405 --> 00:39:03,641 ROOM, TAKE A RIGHT, GO DOWN THE 1092 00:39:03,708 --> 00:39:06,077 HALL, AND THAT'S WHERE IT IS. 1093 00:39:06,143 --> 00:39:06,844 RESTROOMS, YOU HAVE TO WALK BACK 1094 00:39:06,911 --> 00:39:08,379 TO WHERE YOU CAME IN AND THE 1095 00:39:08,446 --> 00:39:09,680 RESTROOMS ARE BEHIND THE 1096 00:39:09,747 --> 00:39:10,881 SECURITY DESK. 1097 00:39:10,948 --> 00:39:12,116 THEY'LL TAKE YOUR I.D. EVERY 1098 00:39:12,183 --> 00:39:15,086 TIME YOU GO. 1099 00:39:15,152 --> 00:39:19,890 FOR THOSE WHO DID NOT ORDER A 1100 00:39:19,957 --> 00:39:21,092 LUNCH BOX AND DID NOT SIGN UP 1101 00:39:21,158 --> 00:39:22,393 FOR DINNER, THERE ARE MANY 1102 00:39:22,460 --> 00:39:23,394 RESTAURANTS TO CHOOSE FROM 1103 00:39:23,461 --> 00:39:27,031 ACROSS THE STREET, PIKE AND 1104 00:39:27,098 --> 00:39:27,865 ROSE. 1105 00:39:27,932 --> 00:39:29,200 BUT SINCE YOU ONLY HAVE ONE HOUR 1106 00:39:29,266 --> 00:39:31,168 FOR LUNCH BREAK, WE SUGGEST THAT 1107 00:39:31,235 --> 00:39:32,703 YOU TAKE A TAKEOUT, BRING IT 1108 00:39:32,770 --> 00:39:34,672 BACK HERE AND EAT IT WITH THE 1109 00:39:34,739 --> 00:39:36,807 REST OF US HERE. 1110 00:39:36,874 --> 00:39:39,877 AND LASTLY, SINCE ALL THE 1111 00:39:39,944 --> 00:39:42,279 DISCUSSIONS IN THIS ROOM ARE 1112 00:39:42,346 --> 00:39:43,280 VIDEOCAST, WE KINDLY ASK THAT 1113 00:39:43,347 --> 00:39:47,118 YOU ALL USE MICROPHONES WHEN 1114 00:39:47,184 --> 00:39:47,785 ADDRESSING THE OTHERS. 1115 00:39:47,852 --> 00:39:49,520 WE HAVE TWO HERE IN THE ROOM AND 1116 00:39:49,587 --> 00:39:51,188 I'M SURE WE'LL RUN AROUND AND 1117 00:39:51,255 --> 00:39:53,190 HAND THEM FOR PEOPLE WHO CAN'T 1118 00:39:53,257 --> 00:39:54,492 GET UP BECAUSE THEY STUCK IN THE 1119 00:39:54,558 --> 00:39:57,862 MIDDLE. 1120 00:39:57,928 --> 00:39:59,296 AND PLEASE INTRODUCE YOURSELVES 1121 00:39:59,363 --> 00:40:00,531 WHEN YOU DO SPEAK AT THE 1122 00:40:00,598 --> 00:40:01,132 MICROPHONE. 1123 00:40:01,198 --> 00:40:03,033 SO WITH THAT, THANK YOU FOR YOUR 1124 00:40:03,100 --> 00:40:03,801 ATTENTION, AND I'M LOOKING 1125 00:40:03,868 --> 00:40:14,345 FORWARD TO A GREAT MEETING. 1126 00:40:35,666 --> 00:40:42,473 >> SO IT MY GREAT, GREAT PLEASUE 1127 00:40:42,540 --> 00:40:44,608 TO INVITE EDWARD MIGUEL AS OUR 1128 00:40:44,675 --> 00:40:46,243 KEYNOTE SPEAKER. 1129 00:40:46,310 --> 00:40:48,112 AS DISTINGUISHED PROFESSOR OF 1130 00:40:48,179 --> 00:40:49,346 ECONOMICS, PROFESSOR OF 1131 00:40:49,413 --> 00:40:50,447 ENVIRONMENTAL AND RESOURCE 1132 00:40:50,514 --> 00:40:52,283 ECONOMICS AT UC BERKELEY, HE 1133 00:40:52,349 --> 00:40:53,851 EARNED A BACHELOR OF SCIENCE IN 1134 00:40:53,918 --> 00:40:55,052 BOTH ECONOMICS AND MATHEMATICS 1135 00:40:55,119 --> 00:40:57,154 FROM MIT AND A PH.D. IN 1136 00:40:57,221 --> 00:40:58,255 ECONOMICS FROM HARVARD 1137 00:40:58,322 --> 00:41:00,558 UNIVERSITY. 1138 00:41:00,624 --> 00:41:02,626 ED'S MAIN RESEARCH FOCUS IS 1139 00:41:02,693 --> 00:41:04,061 AFRICAN AMERICAN ECONOMIC 1140 00:41:04,128 --> 00:41:06,897 DEVELOPMENT, INCLUDING WORK ON 1141 00:41:06,964 --> 00:41:08,232 THE ECONOMIC CAUSES AND 1142 00:41:08,299 --> 00:41:09,433 CONSEQUENCES OF VIOLENCE, THE 1143 00:41:09,500 --> 00:41:11,869 IMPACT OF ETHNIC DIVISIONS ON 1144 00:41:11,936 --> 00:41:14,605 LOCAL COLLECTIVE ACTION, AND 1145 00:41:14,672 --> 00:41:16,373 INTERACTIONS BETWEEN HEALTH, 1146 00:41:16,440 --> 00:41:17,474 EDUCATION, ENVIRONMENT, AND 1147 00:41:17,541 --> 00:41:18,876 PRODUCTIVITY FOR THE POOR. 1148 00:41:18,943 --> 00:41:19,543 PRETTY COOL. 1149 00:41:19,610 --> 00:41:20,978 NOW YOU KNOW WHY I HAD TO READ 1150 00:41:21,045 --> 00:41:24,715 IT AND NO, MA NOT MEMORIZE IT. 1151 00:41:24,782 --> 00:41:26,483 IN 2012, HE CO-FOUNDED THE 1152 00:41:26,550 --> 00:41:27,585 BERKELEY INITIATIVE FOR 1153 00:41:27,651 --> 00:41:29,353 TRANSPARENCY IN THE SOCIAL 1154 00:41:29,420 --> 00:41:30,921 SCIENCES TO IMPROVE THE 1155 00:41:30,988 --> 00:41:32,656 CREDIBILITY OF SCIENCE BY 1156 00:41:32,723 --> 00:41:34,625 ADVANCING TRANSPARENT 1157 00:41:34,692 --> 00:41:35,759 REPRODUCIBILITY, ETHICS AND 1158 00:41:35,826 --> 00:41:36,026 RESEARCH. 1159 00:41:36,093 --> 00:41:38,495 WE HAVE A LOT IN COMMON. 1160 00:41:38,562 --> 00:41:40,130 FURTHERMORE, IN 2021, HE AND 1161 00:41:40,197 --> 00:41:42,466 OTHERS PROPOSED A FRAMEWORK TO 1162 00:41:42,533 --> 00:41:43,968 REPORT ALL RESULTS EFFICIENTLY 1163 00:41:44,034 --> 00:41:46,604 IN THE SOCIAL SCIENCES, WHICH 1164 00:41:46,670 --> 00:41:47,504 OBVIOUSLY HAS DIRECT RELEVANCE 1165 00:41:47,571 --> 00:41:50,374 TO OUR WORKSHOP. 1166 00:41:50,441 --> 00:41:52,543 THIS COMPREHENSIVE FRAMEWORK 1167 00:41:52,610 --> 00:41:53,744 DEMONSTRATES THAT PUBLICATION 1168 00:41:53,811 --> 00:41:56,614 BIAS IS NOT UNIQUE TO BIOMEDICAL 1169 00:41:56,680 --> 00:41:58,182 RESEARCH, AND, HENCE, TO 1170 00:41:58,249 --> 00:41:59,683 MITIGATE PUBLICATION BIAS WILL 1171 00:41:59,750 --> 00:42:03,854 LIKELY REQUIRE CULTURAL REFORM. 1172 00:42:03,921 --> 00:42:05,522 THIS BRIEF INTRODUCTION DOES NOT 1173 00:42:05,589 --> 00:42:08,926 DO JUSTICE TO HIS CREDENTIALS, 1174 00:42:08,993 --> 00:42:09,593 ACHIEVEMENTS, BUT FOR THE SAKE 1175 00:42:09,660 --> 00:42:11,695 OF TIME, AND MY IGNORANCE WHEN 1176 00:42:11,762 --> 00:42:14,965 IT COMES TO SOCIAL SCIENCES, I 1177 00:42:15,032 --> 00:42:16,267 WILL STOP HERE AND PLEASE JOIN 1178 00:42:16,333 --> 00:42:22,439 ME IN WELCOMING DR. MIGUEL. 1179 00:42:22,506 --> 00:42:26,610 IS 1180 00:42:26,677 --> 00:42:28,112 >> THANKS SO MUCH FOR HAVING ME. 1181 00:42:28,178 --> 00:42:29,747 IT'S REALLY GREAT TO BE HERE. 1182 00:42:29,813 --> 00:42:31,315 THANKS FOR THE INTRODUCTION AND 1183 00:42:31,382 --> 00:42:32,650 I'M REALLY EXCITED ABOUT THIS 1184 00:42:32,716 --> 00:42:33,417 MEETING, GETTING TO SHARE WHAT 1185 00:42:33,484 --> 00:42:34,852 WE'RE DOING ACROSS FIELDS, WHERE 1186 00:42:34,919 --> 00:42:37,821 I THINK WE'LL FIND A LOT OF 1187 00:42:37,888 --> 00:42:39,456 COMMONALITY, SO AGAIN, I GO BY 1188 00:42:39,523 --> 00:42:41,091 TED, TED MIGUELING FROM 1189 00:42:41,158 --> 00:42:43,761 BERKELEY. 1190 00:42:43,827 --> 00:42:45,296 I'M EXCITED ABOUT WHAT MAY COME 1191 00:42:45,362 --> 00:42:46,997 OUT OF THIS MEETING. 1192 00:42:47,064 --> 00:42:47,932 ABOUT FIVE YEARS A WE HELD A 1193 00:42:47,998 --> 00:42:50,167 KIND OF RELATED MEETING FOR 1194 00:42:50,234 --> 00:42:52,336 SOCIAL SCIENTISTS THAT LED TO 1195 00:42:52,403 --> 00:42:53,337 THE FRAMEWORK THAT SHAI 1196 00:42:53,404 --> 00:42:54,772 MENTIONED AND JUST THE KIND 1197 00:42:54,838 --> 00:42:56,540 OF -- THE ENERGY AND THE 1198 00:42:56,607 --> 00:42:57,508 INTERACTIONS ACROSS FIELDS CAN 1199 00:42:57,574 --> 00:43:00,277 REALLY LEAD TO NEW IDEAS AND 1200 00:43:00,344 --> 00:43:01,178 HOPEFULLY BRING THINGS FORWARD 1201 00:43:01,245 --> 00:43:03,814 FOR THE FIELD. 1202 00:43:03,881 --> 00:43:04,915 ONE OF THE THINGS I'M GOING TO 1203 00:43:04,982 --> 00:43:06,684 TRY DO IS BRING OUT WHAT WE'VE 1204 00:43:06,750 --> 00:43:13,357 LEARNED IN ECONOMICS, TR TRY TO 1205 00:43:13,424 --> 00:43:14,124 CREATE POINT OF CONTACT. 1206 00:43:14,191 --> 00:43:15,392 I'M GOING TO HAVE THREE MAIN 1207 00:43:15,459 --> 00:43:15,726 THEMES. 1208 00:43:15,793 --> 00:43:17,461 SO I'M ACTUALLY GOING TO START 1209 00:43:17,528 --> 00:43:18,262 FROM FIRST PRINCIPLES FROM THE 1210 00:43:18,329 --> 00:43:18,996 BEGINNING OF WHAT WE'RE TRYING 1211 00:43:19,063 --> 00:43:21,098 TO DO AS SCIENTISTS, WHAT IS THE 1212 00:43:21,165 --> 00:43:22,066 SCIENTIFIC ETHOS, WHAT ARE OUR 1213 00:43:22,132 --> 00:43:22,499 GOALS. 1214 00:43:22,566 --> 00:43:23,500 THEN I'LL DISCUSS SOME OF WHAT 1215 00:43:23,567 --> 00:43:24,501 WE'VE BEEN DOING AND TRENDS IN 1216 00:43:24,568 --> 00:43:26,170 THE SOCIAL SCIENCES BEFORE 1217 00:43:26,236 --> 00:43:28,339 ENDING WITH SOME NEW DATA ON 1218 00:43:28,405 --> 00:43:29,540 PUBLICATION BIAS IN ECONOMICS, 1219 00:43:29,606 --> 00:43:31,041 WHICH I THINK IS JUST DIRECTLY 1220 00:43:31,108 --> 00:43:33,844 LINKED TO SOME OF THE IDEAS IN 1221 00:43:33,911 --> 00:43:34,511 THIS WORKSHOP. 1222 00:43:34,578 --> 00:43:36,613 AND I'LL TALK ABOUT THE 1223 00:43:36,680 --> 00:43:37,815 FRAMEWORK THAT SHAI MENTIONED AS 1224 00:43:37,881 --> 00:43:41,485 WELL FOR GOING FORWARD. 1225 00:43:41,552 --> 00:43:42,920 I THINK THERE'S A FEW MINUTES 1226 00:43:42,987 --> 00:43:44,788 FOR QUESTIONS AT THE END SO I'M 1227 00:43:44,855 --> 00:43:46,323 REALLY EXCITED TO ENGAGE ACROSS 1228 00:43:46,390 --> 00:43:48,559 FIELD AND THINK OF COMMONALITY 1229 00:43:48,625 --> 00:43:50,060 AND DIFFERENCES TOO. 1230 00:43:50,127 --> 00:43:51,161 SO LET ME START FROM THE 1231 00:43:51,228 --> 00:43:51,695 BEGINNING AND MAYBE IT'S 1232 00:43:51,762 --> 00:43:52,396 APPROPRIATE TO START THE 1233 00:43:52,463 --> 00:43:53,864 CONFERENCE, YOU KNOW, STEPPING 1234 00:43:53,931 --> 00:43:56,667 BACK A LITTLE BIT. 1235 00:43:56,734 --> 00:43:57,901 SO I'M GOING TO START BY TALKING 1236 00:43:57,968 --> 00:44:00,437 ABOUT WHAT OUR AIMS SHOULD BE AS 1237 00:44:00,504 --> 00:44:01,205 RESEARCHERS, WHAT SHOULD BE 1238 00:44:01,271 --> 00:44:02,840 GUIDING US, WHAT SHOULD OUR 1239 00:44:02,906 --> 00:44:04,141 PRINCIPLES AND VALUES BE. 1240 00:44:04,208 --> 00:44:07,911 ONE OF THE MOST INFLUENTIAL 1241 00:44:07,978 --> 00:44:09,546 ARTICULATIONS OF THE SCIENTIFIC 1242 00:44:09,613 --> 00:44:12,049 ETHOS WAS PUT TOGETHER BY ROBERT 1243 00:44:12,116 --> 00:44:15,285 MERTON, A FAMOUS SOCIOLOGIST OF 1244 00:44:15,352 --> 00:44:17,154 SCIENCE, WHO REALLY THOUGHT 1245 00:44:17,221 --> 00:44:20,190 ABOUT THE NORMS AT INSTITUTIONS 1246 00:44:20,257 --> 00:44:21,325 THAT LEAD SCIENTISTS IN A 1247 00:44:21,392 --> 00:44:22,226 CERTAIN DIRECTION AND THE KIND 1248 00:44:22,292 --> 00:44:24,294 OF VALUES THAT MOTIVATE THEM AND 1249 00:44:24,361 --> 00:44:27,998 HE TALKED ABOUT NORMS AS BINDING 1250 00:44:28,065 --> 00:44:29,900 THAT ARE BOTH EFFICIENT BUT ALSO 1251 00:44:29,967 --> 00:44:32,202 SEEMED TO BE SORT OF NORMATIVELY 1252 00:44:32,269 --> 00:44:33,303 RIGHT OR MORALLY RIGHT. 1253 00:44:33,370 --> 00:44:36,607 THERE WAS SOMETHING DEEPER THAN 1254 00:44:36,673 --> 00:44:38,475 JUST EFFICIENT SCIENTIFIC 1255 00:44:38,542 --> 00:44:39,443 PRODUCTION MOTIVATING 1256 00:44:39,510 --> 00:44:39,777 RESEARCHERS. 1257 00:44:39,843 --> 00:44:40,878 AND I THINK IT'S KIND OF 1258 00:44:40,944 --> 00:44:42,112 INSPIRING TO THINK SOME OF 1259 00:44:42,179 --> 00:44:43,447 THESE PRINCIPLES AND HOW THEY 1260 00:44:43,514 --> 00:44:45,516 LINK UP TO OUR WORK. 1261 00:44:45,582 --> 00:44:47,184 SO WHAT WERE THE FOUR CORE 1262 00:44:47,251 --> 00:44:49,286 VALUES THAT MERTON TALKED ABOUT? 1263 00:44:49,353 --> 00:44:50,487 I'LL VERY QUICKLY GO THROUGH 1264 00:44:50,554 --> 00:44:52,022 THEM IN TURN. 1265 00:44:52,089 --> 00:44:57,294 THEY'RE LISTED HERE: 1266 00:44:57,361 --> 00:45:00,364 UNIVERSALISM, COMMUNALITY, 1267 00:45:00,431 --> 00:45:01,465 DISINTERESTEDNESS AND ORGANIZED 1268 00:45:01,532 --> 00:45:01,765 SKEPTICISM. 1269 00:45:01,832 --> 00:45:03,033 I'LL HAVE A WORD OR TWO TO SAY 1270 00:45:03,100 --> 00:45:04,568 ABOUT EACH OF THESE AND HE AND 1271 00:45:04,635 --> 00:45:05,769 OTHERS HAVE WRITTEN ABOUT THE 1272 00:45:05,836 --> 00:45:07,771 ORIGINS OF THESE NORMS AND HOW 1273 00:45:07,838 --> 00:45:09,373 OFTEN THEY GO BACK CENTURIES 1274 00:45:09,440 --> 00:45:11,075 INTO THE KIND OF CULTURE OF 1275 00:45:11,141 --> 00:45:13,577 SCHOLARS EVEN CENTURIES AGO IN 1276 00:45:13,644 --> 00:45:16,780 EUROPEAN MONASTERIES. 1277 00:45:16,847 --> 00:45:21,819 SO WHAT IS THIS FIRST VALUE OF 1278 00:45:21,885 --> 00:45:23,420 UNIVERSALISM THAT MOTIVATES 1279 00:45:23,487 --> 00:45:24,254 SCIENCE? 1280 00:45:24,321 --> 00:45:25,122 THE FIRST IDEA IS THAT THE 1281 00:45:25,189 --> 00:45:26,657 RESEARCH FINDINGS AT THEIR BASE 1282 00:45:26,723 --> 00:45:27,524 ARE IMPERSONAL. 1283 00:45:27,591 --> 00:45:28,792 THE SCIENCE AND DATA STANDS ON 1284 00:45:28,859 --> 00:45:29,460 ITS OWN. 1285 00:45:29,526 --> 00:45:31,428 IT DOESN'T MATTER WHAT YOUR 1286 00:45:31,495 --> 00:45:33,697 IDENTITY IS, ETHNIC BACKGROUND, 1287 00:45:33,764 --> 00:45:34,431 GENDER IDENTITY, WHERE YOU'RE 1288 00:45:34,498 --> 00:45:35,065 FROM. 1289 00:45:35,132 --> 00:45:37,668 THE MATHEMATICAL PROOF IS THE 1290 00:45:37,734 --> 00:45:39,103 SAME WHOEVER WRITES IT, THE 1291 00:45:39,169 --> 00:45:39,803 SCIENTIFIC DATA SPEAKS 1292 00:45:39,870 --> 00:45:40,671 REGARDLESS OF WHO THE AUTHORS 1293 00:45:40,737 --> 00:45:41,338 ARE. 1294 00:45:41,405 --> 00:45:42,639 SO WE SHOULD BE JUDGING THE 1295 00:45:42,706 --> 00:45:43,874 SCIENCE FOR THE SCIENCE, NOT 1296 00:45:43,941 --> 00:45:45,409 JUST WHO THE PROTAGONIST IS OR 1297 00:45:45,476 --> 00:45:47,945 WHO THE AUTHOR IS. 1298 00:45:48,011 --> 00:45:49,012 AND HERE, THEN, YOU KNOW, 1299 00:45:49,079 --> 00:45:51,482 THERE'S AN IMMEDIATE IMPLICATION 1300 00:45:51,548 --> 00:45:53,550 WHICH IS RESEARCH COMMUNITIES OR 1301 00:45:53,617 --> 00:45:55,185 SOCIETIES THAT REALLY OPEN UP 1302 00:45:55,252 --> 00:45:56,987 THE SCIENTIFIC ENTERPRISE TO A 1303 00:45:57,054 --> 00:45:57,888 BROADER ARRAY OF PEOPLE ARE 1304 00:45:57,955 --> 00:46:01,358 GOING TO MAKE MORE PROGRESS. 1305 00:46:01,425 --> 00:46:02,759 THERE SHOULDN'T JUST BE ONE 1306 00:46:02,826 --> 00:46:04,228 SLICE OF PEOPLE WHO ARE KIND OF 1307 00:46:04,294 --> 00:46:05,129 WELCOMED INTO THE SCIENTIFIC AND 1308 00:46:05,195 --> 00:46:05,762 OTHERS NOD. 1309 00:46:05,829 --> 00:46:08,532 EVERYBODY CAN CONTRIBUTE, 1310 00:46:08,599 --> 00:46:11,168 EVERYBODY'S SCIENCE CAN MOVE THE 1311 00:46:11,235 --> 00:46:11,668 FIELD FORWARD. 1312 00:46:11,735 --> 00:46:13,137 SO ACTUALLY HE MAKES THE LINK 1313 00:46:13,203 --> 00:46:14,671 BETWEEN BEING IN AN OPEN, 1314 00:46:14,738 --> 00:46:15,973 DEMOCRATIC SOCIETY AND 1315 00:46:16,039 --> 00:46:16,807 SCIENTIFIC PROGRESS IN THE LONG 1316 00:46:16,874 --> 00:46:17,074 RUN. 1317 00:46:17,141 --> 00:46:19,743 AND I THINK WHEN WE THINK ABOUT 1318 00:46:19,810 --> 00:46:21,512 OUR COUNTRY HERE IN THE U.S., 1319 00:46:21,578 --> 00:46:22,746 OUR DEMOCRATIC INSTITUTIONS, HOW 1320 00:46:22,813 --> 00:46:24,381 WE'VE TRIED TO MAKE THE 1321 00:46:24,448 --> 00:46:25,582 SCIENTIFIC ENTERPRISE MORE 1322 00:46:25,649 --> 00:46:27,484 INCLUSIVE, YOU KNOW, IMPERFECTLY 1323 00:46:27,551 --> 00:46:28,819 BUT EFFORTS TO DO THAT, THOSE 1324 00:46:28,886 --> 00:46:30,554 ARE THINGS THAT ARE GOING TO 1325 00:46:30,621 --> 00:46:31,421 ACCELERATE THIS SCIENCE GOING 1326 00:46:31,488 --> 00:46:31,889 FORWARD. 1327 00:46:31,955 --> 00:46:36,727 SO THAT'S ONE KEY VALUE, 1328 00:46:36,793 --> 00:46:38,128 UNIVERSALISM. 1329 00:46:38,195 --> 00:46:40,764 THE SECOND ONE IS COMMUNALITY. 1330 00:46:40,831 --> 00:46:42,132 OPEN SHARING OF SCIENCE IS 1331 00:46:42,199 --> 00:46:42,599 CRITICAL. 1332 00:46:42,666 --> 00:46:44,535 WE ARE A COMMUNITY, WE SHARE OUR 1333 00:46:44,601 --> 00:46:45,536 RESULTS WITH EACH OTHER, AND 1334 00:46:45,602 --> 00:46:48,105 THAT'S HOW WE ADVANCE AS A 1335 00:46:48,172 --> 00:46:50,174 COMMUNITY, HIDING OUR RESULTS 1336 00:46:50,240 --> 00:46:51,475 DOESN'T ADVANCE SCIENCE. 1337 00:46:51,542 --> 00:46:52,843 SO HERE WE'RE ALREADY STARTING 1338 00:46:52,910 --> 00:46:54,945 TO GET INTO THE ISSUES OF 1339 00:46:55,012 --> 00:46:56,346 PUBLICATION BIAS, IF SCIENTISTS 1340 00:46:56,413 --> 00:46:59,616 ARE HIDING RESULTS, SELECTIVELY 1341 00:46:59,683 --> 00:47:00,784 REPORTING RESULTS, NOT OPENLY 1342 00:47:00,851 --> 00:47:02,419 SHARING WHAT THEY'RE DOING, IT 1343 00:47:02,486 --> 00:47:05,255 REALLY GOES AGAINST THIS CENTRAL 1344 00:47:05,322 --> 00:47:07,558 NORM OF OPEN SHARING SO THE 1345 00:47:07,624 --> 00:47:08,192 WHOLE COMMUNITY CAN LEARN FROM 1346 00:47:08,258 --> 00:47:08,825 OUR FINDINGS. 1347 00:47:08,892 --> 00:47:10,027 AND IF WE'RE NOT DOING THAT, 1348 00:47:10,093 --> 00:47:12,663 WE'RE REALLY NOT CONSISTENT WITH 1349 00:47:12,729 --> 00:47:13,897 THIS PRINCIPLE. 1350 00:47:13,964 --> 00:47:15,332 ANOTHER KIND OF IMPLICATION HE 1351 00:47:15,399 --> 00:47:19,369 WRITES ABOUT AND OTHERS IS THE 1352 00:47:19,436 --> 00:47:20,504 TENSION, YOU COULD SAY, BETWEEN 1353 00:47:20,571 --> 00:47:22,906 THE GOALS OF SCIENCE AND 1354 00:47:22,973 --> 00:47:23,674 COMMERCIAL GOALS. 1355 00:47:23,740 --> 00:47:25,108 NOW, I KNOW IN BIOMEDICAL 1356 00:47:25,175 --> 00:47:26,610 RESEARCH AND ALSO IN SILICON VA 1357 00:47:26,677 --> 00:47:29,780 LESILICONVALLEY AND THE BAY AREI 1358 00:47:29,846 --> 00:47:31,048 COME FROM, THERE'S A LOT OF 1359 00:47:31,114 --> 00:47:32,149 COMMERCIALIZATION OF RESEARCH 1360 00:47:32,216 --> 00:47:33,217 FINDINGS, BUT ON SOME LEVEL 1361 00:47:33,283 --> 00:47:34,318 THERE'S A TENSION THERE BECAUSE 1362 00:47:34,384 --> 00:47:36,353 THEN IF YOU KIND OF TAKE 1363 00:47:36,420 --> 00:47:38,322 INTELLECTUAL PROPERTY OUT OF THE 1364 00:47:38,388 --> 00:47:40,390 SCIENTIFIC COMMUNITY AND TRY TO 1365 00:47:40,457 --> 00:47:41,692 COMMERCIALIZE IT, MAYBE YOU'RE 1366 00:47:41,758 --> 00:47:43,060 NOT FULLY SHARING ANYMORE. 1367 00:47:43,126 --> 00:47:44,828 YOU'RE KIND OF KEEPING SECRETS 1368 00:47:44,895 --> 00:47:45,896 FROM THE REST OF THE SCIENTIFIC 1369 00:47:45,963 --> 00:47:47,097 COMMUNITY. 1370 00:47:47,164 --> 00:47:51,335 SO THOSE ARE A TENSION. 1371 00:47:51,401 --> 00:47:52,736 THE THIRD VALUE, SCIENTIFIC 1372 00:47:52,803 --> 00:47:55,172 VALUE, IS DISINTERESTEDNESS. 1373 00:47:55,239 --> 00:47:57,174 WE AS RESEARCHERS SHOULD BE 1374 00:47:57,241 --> 00:48:00,010 MOTIVATED BY THE TRUTH. 1375 00:48:00,077 --> 00:48:01,311 WE SHOULDN'T BE THINKING ABOUT 1376 00:48:01,378 --> 00:48:02,779 THE IMPLICATIONS FOR OUR CAREERS 1377 00:48:02,846 --> 00:48:05,949 OR WHAT OTHER PEOPLE THINK ABOUT 1378 00:48:06,016 --> 00:48:07,284 US FROM REPORTING SCIENTIFIC 1379 00:48:07,351 --> 00:48:07,718 TRUTH. 1380 00:48:07,784 --> 00:48:09,019 OUR RESPONSIBILITY IS TO REPORT 1381 00:48:09,086 --> 00:48:10,487 THAT TRUTH TO THE REST OF THE 1382 00:48:10,554 --> 00:48:11,688 COMMUNITY AND DEAL WITH THE 1383 00:48:11,755 --> 00:48:14,858 CONSEQUENCES OF REALITY. 1384 00:48:14,925 --> 00:48:15,726 OF THE 1385 00:48:15,792 --> 00:48:17,527 TRUTH AND NOT OBSESS ABOUT 1386 00:48:17,594 --> 00:48:19,296 MONETARY GOALS OR OUR PERSONAL 1387 00:48:19,363 --> 00:48:20,130 MOTIVATIONS. 1388 00:48:20,197 --> 00:48:23,066 AND YOU KNOW, THIS IS A VERY 1389 00:48:23,133 --> 00:48:23,834 IMPORTANT ISSUE. 1390 00:48:23,900 --> 00:48:25,435 WE ALREADY HEARD IN A COUPLE OF 1391 00:48:25,502 --> 00:48:27,304 THE TALKS, SHAI'S TALK AND THE 1392 00:48:27,371 --> 00:48:29,940 EARLIER TALK, ABOUT HOW CAREER 1393 00:48:30,007 --> 00:48:31,908 MOTIVATIONS AND TENURE REVIEWS 1394 00:48:31,975 --> 00:48:32,809 REALLY FACTOR INTO WHAT 1395 00:48:32,876 --> 00:48:33,777 SCIENTISTS ARE DOING AND A LOT 1396 00:48:33,844 --> 00:48:39,750 OF THE CHOICES THEY MAKE. 1397 00:48:39,816 --> 00:48:41,251 THERE IS A TENSION THERE 1398 00:48:41,318 --> 00:48:42,152 POTENTIALLY WITH 1399 00:48:42,219 --> 00:48:42,686 DISINTERESTEDNESS. 1400 00:48:42,753 --> 00:48:44,254 WE SHOULD BE REPORTING RESULTS 1401 00:48:44,321 --> 00:48:45,789 REGARDLESS OF THE CONSEQUENCES. 1402 00:48:45,856 --> 00:48:46,990 EVEN IF THERE'S A NULL RESULT, 1403 00:48:47,057 --> 00:48:48,091 IT'S IMPORTANT FOR SCIENCE AND 1404 00:48:48,158 --> 00:48:50,227 THE COMMUNITY TO KNOW THIS, AND 1405 00:48:50,294 --> 00:48:53,297 THAT'S REALLY OUR RESPONSIBILITY 1406 00:48:53,363 --> 00:48:55,899 AS SCIENTISTS. 1407 00:48:55,966 --> 00:48:57,367 MERTON, AS A SOCIOLOGIST, WAS 1408 00:48:57,434 --> 00:48:58,468 THINKING ABOUT THE SOCIAL 1409 00:48:58,535 --> 00:49:00,203 STRUCTURES AND NORMS THAT REALLY 1410 00:49:00,270 --> 00:49:03,040 COULD PUSH PEOPLE TOWARDS THIS 1411 00:49:03,106 --> 00:49:03,974 ETHOS, AND I THINK THAT'S 1412 00:49:04,041 --> 00:49:04,908 SOMETHING VERY IMPORTANT TO 1413 00:49:04,975 --> 00:49:06,076 THINK ABOUT, AND WE'RE GOING TO 1414 00:49:06,143 --> 00:49:07,744 COME BACK TO, I THINK, IN BOTH 1415 00:49:07,811 --> 00:49:09,379 DAYS OVER AND OVER AGAIN, HOW 1416 00:49:09,446 --> 00:49:10,814 CAN STRUCTURES AND INCENTIVES 1417 00:49:10,881 --> 00:49:13,383 AND INSTITUTIONS BE SET UP TO 1418 00:49:13,450 --> 00:49:14,818 MAKE SURE SCIENTISTS REALLY LIVE 1419 00:49:14,885 --> 00:49:18,322 UP TO THIS VALUE. 1420 00:49:18,388 --> 00:49:20,691 THE FOURTH SCIENTIFIC PILLAR IS 1421 00:49:20,757 --> 00:49:21,792 ORGANIZED SKEPTICISM. 1422 00:49:21,858 --> 00:49:24,161 THE ABILITY TO VERIFY DATA, 1423 00:49:24,227 --> 00:49:25,595 VERIFY RESULTS, IS REALLY 1424 00:49:25,662 --> 00:49:27,397 CRITICAL FOR SCIENCE TO ADVANCE. 1425 00:49:27,464 --> 00:49:29,266 WE CAN'T JUST SAY OH, TRUST ME, 1426 00:49:29,333 --> 00:49:31,001 TRUST US, AGAIN, IF THERE'S A 1427 00:49:31,068 --> 00:49:31,835 MATHEMATICAL PROOF, OTHER PEOPLE 1428 00:49:31,902 --> 00:49:33,337 HAVE TO WORK THROUGH IT. 1429 00:49:33,403 --> 00:49:35,005 IF THERE'S DATA OUT THERE, 1430 00:49:35,072 --> 00:49:36,306 OTHERS NEED TO BE ABLE TO 1431 00:49:36,373 --> 00:49:37,674 REPRODUCE THE RESULTS, REPLICATE 1432 00:49:37,741 --> 00:49:39,176 THE RESULTS, CHECK THE RESULTS. 1433 00:49:39,242 --> 00:49:41,712 THAT'S REALLY CENTRAL TO WHO WE 1434 00:49:41,778 --> 00:49:43,313 ARE AS SCIENTISTS, AND ONE OF 1435 00:49:43,380 --> 00:49:45,482 THE THINGS THAT SETS US APART 1436 00:49:45,549 --> 00:49:46,717 FROM MANY OTHER FIELDS. 1437 00:49:46,783 --> 00:49:49,686 SO AGAIN, THIS OPENNESS, OPEN 1438 00:49:49,753 --> 00:49:51,455 SHARING, ABILITY FOR OTHERS TO 1439 00:49:51,521 --> 00:49:52,622 REALLY INTERROGATE RESULTS IS 1440 00:49:52,689 --> 00:49:55,258 CENTRAL TO PROGRESS OF OUR WHOLE 1441 00:49:55,325 --> 00:49:55,992 ENTERPRISE. 1442 00:49:56,059 --> 00:49:57,060 ALL RIGHT. 1443 00:49:57,127 --> 00:49:58,929 SO THESE FOUNDATIONAL SCIENTIFIC 1444 00:49:58,995 --> 00:50:02,099 NORMS, I FIND THEM INSPIRING. 1445 00:50:02,165 --> 00:50:03,600 I THINK IT'S EXCITING TO THINK 1446 00:50:03,667 --> 00:50:06,870 THAT WE'RE PART OF A PROCESS AS 1447 00:50:06,937 --> 00:50:08,305 RESEARCHERS ACROSS FIELDS THAT 1448 00:50:08,372 --> 00:50:12,109 GOES BACK CENT REE CENTURIES THS 1449 00:50:12,175 --> 00:50:13,377 VALUES WE CAN BELIEVE IN AND 1450 00:50:13,443 --> 00:50:14,811 VALUES THAT CAN ADVANCE THE 1451 00:50:14,878 --> 00:50:16,613 WHOLE COMMUNITY, BUT CENTRAL TO 1452 00:50:16,680 --> 00:50:19,249 THIS WHOLE ARTICULATION OF THE 1453 00:50:19,316 --> 00:50:23,520 SCIENTIFIC ETHOS IS OPEN 1454 00:50:23,587 --> 00:50:24,321 COMMUNICATION AND COMPLETE 1455 00:50:24,388 --> 00:50:25,021 SHARING OF FINDINGS. 1456 00:50:25,088 --> 00:50:26,723 IF WE'RE NOT DOING THAT, WE'RE 1457 00:50:26,790 --> 00:50:28,158 NOT LIVING UP TO THESE VALUES 1458 00:50:28,225 --> 00:50:29,426 AND WE'RE SLOWING DOWN SCIENCE. 1459 00:50:29,493 --> 00:50:30,961 WE HEARD THE VERY GOOD POINTS 1460 00:50:31,027 --> 00:50:32,696 ABOUT WE'RE WASTING RESEARCH 1461 00:50:32,763 --> 00:50:33,930 FUNDS POTENTIALLY IF WE'RE NOT 1462 00:50:33,997 --> 00:50:34,731 DOING IT. 1463 00:50:34,798 --> 00:50:36,032 MAYBE WE'RE LEADING OTHERS TO 1464 00:50:36,099 --> 00:50:38,435 TRY TO REPLICATE RESULTS OR 1465 00:50:38,502 --> 00:50:39,436 EXTEND RESULTS THAT AREN'T 1466 00:50:39,503 --> 00:50:40,670 REALLY SOLID TO BEGIN WITH. 1467 00:50:40,737 --> 00:50:42,672 WE COULD BE LEADING SCIENCE INTO 1468 00:50:42,739 --> 00:50:43,540 DEAD ENDS. 1469 00:50:43,607 --> 00:50:46,209 SO OPEN SHARING IS CENTRAL, NOT 1470 00:50:46,276 --> 00:50:48,211 JUST FOR US BUT FOR THE WHOLE 1471 00:50:48,278 --> 00:50:51,148 COMMUNITY. 1472 00:50:51,214 --> 00:50:52,549 SO THEN THE QUESTION IS, HOW 1473 00:50:52,616 --> 00:50:53,917 CLOSELY DO WE AS RESEARCHERS 1474 00:50:53,984 --> 00:50:56,887 LIVE UP TO THESE VALUES TODAY? 1475 00:50:56,953 --> 00:50:58,755 IN THE BIOMEDICAL SCIENCES, IN 1476 00:50:58,822 --> 00:51:01,324 THE SOCIAL SCIENCES WHERE I COME 1477 00:51:01,391 --> 00:51:05,162 FROM, THE DATA THAT SHAI JUST 1478 00:51:05,228 --> 00:51:06,897 SHOWED US OF HOW THE EXTENT OF 1479 00:51:06,963 --> 00:51:08,698 THE PUBLICATION BIAS AND LACK OF 1480 00:51:08,765 --> 00:51:09,900 NULL RESULTS IN THE LITERATURE 1481 00:51:09,966 --> 00:51:13,437 SUGGEST THERE IS A GAP BETWEEN 1482 00:51:13,503 --> 00:51:14,871 THESE VALUE AND OUR PRACTICES, 1483 00:51:14,938 --> 00:51:16,406 BUT SOME PEOPLE HAVE TRIED TO 1484 00:51:16,473 --> 00:51:18,708 QUANTIFY THESE SO I'LL BRIEFLY 1485 00:51:18,775 --> 00:51:20,377 MENTION ONE STUDY, ANDERSON ET 1486 00:51:20,444 --> 00:51:21,378 AL., AND I HAVE THE REFERENCES 1487 00:51:21,445 --> 00:51:23,480 FOR THOSE WHO ARE INTERESTED, 1488 00:51:23,547 --> 00:51:24,781 STUDIED THE DEGREE TO WHICH 1489 00:51:24,848 --> 00:51:26,383 PEOPLE SUBSCRIBE TO THESE NORMS, 1490 00:51:26,450 --> 00:51:28,118 THESE MERTONIAN NORMS. 1491 00:51:28,185 --> 00:51:29,453 DO WE AS SCIENTISTS SORT OF 1492 00:51:29,519 --> 00:51:31,154 BELIEVE IN THESE THINGS IN THESE 1493 00:51:31,221 --> 00:51:31,488 VALUES? 1494 00:51:31,555 --> 00:51:34,090 AND THEN ALSO, SURVEYED THEM ON 1495 00:51:34,157 --> 00:51:36,526 HOW THEY VIEW THEIR RESEARCH 1496 00:51:36,593 --> 00:51:36,927 COMMUNITIES. 1497 00:51:36,993 --> 00:51:38,895 HOW CLOSE IS BEHAVIOR TO THESE 1498 00:51:38,962 --> 00:51:39,963 NORMS, AND ACTUALLY IT'S BASED 1499 00:51:40,030 --> 00:51:43,400 ON A SAMPLE OF R01 RECIPIENTS 1500 00:51:43,467 --> 00:51:46,136 FROM NIH, SO THERE WERE A LOT OF 1501 00:51:46,203 --> 00:51:47,671 BIOMEDICAL SCIENCES, SOME 1502 00:51:47,737 --> 00:51:49,172 DEMOGRAPHERS AND SOCIAL 1503 00:51:49,239 --> 00:51:50,040 SCIENTISTS IN THE SAMPLE AS 1504 00:51:50,106 --> 00:51:51,141 WELL, AND THE KEY FINDING HERE, 1505 00:51:51,208 --> 00:51:53,210 AND I'LL SHOW YOU A FIGURE NEXT, 1506 00:51:53,276 --> 00:51:57,013 IS THERE'S ACTUALLY VERY HIGH 1507 00:51:57,080 --> 00:51:58,648 RATES OF ATTACHMENT TO THESE 1508 00:51:58,715 --> 00:51:59,616 VALUES AMONG SCIENTISTS. 1509 00:51:59,683 --> 00:52:01,218 WE WANT TO LIVE UP TO THESE 1510 00:52:01,284 --> 00:52:01,551 VALUES. 1511 00:52:01,618 --> 00:52:02,552 WE BELIEVE IN THEM. 1512 00:52:02,619 --> 00:52:03,520 WE THINK THEY'RE IMPORTANT. 1513 00:52:03,587 --> 00:52:04,521 WE THINK THEY'RE THE RIGHT WAY 1514 00:52:04,588 --> 00:52:04,955 FORWARD. 1515 00:52:05,021 --> 00:52:08,191 BUT PEOPLE ARE DISI DISILLUSIOND 1516 00:52:08,258 --> 00:52:09,292 BECAUSE THEY FEEL LIKE OUR 1517 00:52:09,359 --> 00:52:10,160 COMMUNITIES HAVEN'T ATTAINED 1518 00:52:10,227 --> 00:52:12,462 THESE VALUES, THAT OUR PRACTICES 1519 00:52:12,529 --> 00:52:13,497 AREN'T LIVING UP TO THEM, SO 1520 00:52:13,563 --> 00:52:15,665 WHAT THEY CALL NORMATIVE 1521 00:52:15,732 --> 00:52:16,666 DISSIDENCE, THIS GAP BETWEEN 1522 00:52:16,733 --> 00:52:19,035 NORMS AND PRACTICE. 1523 00:52:19,102 --> 00:52:20,871 LET ME JUST SHOW YOU SOME OF 1524 00:52:20,937 --> 00:52:21,438 THEIR DATA. 1525 00:52:21,505 --> 00:52:23,139 WHAT THEY HAVE HERE IS BOTH 1526 00:52:23,206 --> 00:52:24,241 AMONG MID CAREER AND EARLY 1527 00:52:24,307 --> 00:52:25,775 CAREER SCHOLARS, SUBSCRIPTION TO 1528 00:52:25,842 --> 00:52:26,409 THESE VALUES. 1529 00:52:26,476 --> 00:52:28,044 SO THEY ASKED A WHOLE SERIES OF 1530 00:52:28,111 --> 00:52:29,880 SURVEY QUESTIONS TO SEE IF 1531 00:52:29,946 --> 00:52:32,282 PEOPLE BELIEVED IN OPENNESS OR 1532 00:52:32,349 --> 00:52:34,017 SECRECY, SHARING OR NOT SHARING, 1533 00:52:34,084 --> 00:52:35,619 ET CETERA. 1534 00:52:35,685 --> 00:52:37,754 AND THE LIGHT GREY MEANS YOU 1535 00:52:37,821 --> 00:52:39,756 VERY MUCH ARE ATTACHED TO THESE 1536 00:52:39,823 --> 00:52:40,490 VALUES. 1537 00:52:40,557 --> 00:52:41,791 THE MIDDLE GREY MEANS YOU'RE 1538 00:52:41,858 --> 00:52:44,327 SOMEWHAT ATTACHED TO THEM, THE 1539 00:52:44,394 --> 00:52:45,529 DARK COLOR MEANS UROPOSED TO 1540 00:52:45,595 --> 00:52:46,663 THEM. 1541 00:52:46,730 --> 00:52:48,098 90-SOMETHING PERCENT OF SCHOLARS 1542 00:52:48,164 --> 00:52:49,966 THAT WERE VAIRED SAY THESE ARE 1543 00:52:50,033 --> 00:52:51,668 THE RIGHT VALUES, WE BELIEVE IN 1544 00:52:51,735 --> 00:52:53,103 THIS, WE THINK SHARING, OPEN 1545 00:52:53,169 --> 00:52:54,304 COMMUNICATION IS THE RIGHT WAY 1546 00:52:54,371 --> 00:52:54,738 FORWARD. 1547 00:52:54,804 --> 00:52:57,541 SO THAT'S EXCITING AND IT MEANS 1548 00:52:57,607 --> 00:52:59,943 THIS VALUE SYSTEM IS ALIVE AND 1549 00:53:00,010 --> 00:53:01,244 WELL, BUT WHEN YOU JUMP DOWN TO 1550 00:53:01,311 --> 00:53:03,280 THE BOTTOM PANEL AND PEOPLE ARE 1551 00:53:03,346 --> 00:53:04,915 ASKED, WHAT DO YOU THINK THE 1552 00:53:04,981 --> 00:53:07,884 BEHAVIOR OF COLLEAGUES IN YOUR 1553 00:53:07,951 --> 00:53:09,185 FIELD, HOW DOES THAT LOOK, ONLY 1554 00:53:09,252 --> 00:53:10,887 ABOUT 10% OF PEOPLE THINK THEIR 1555 00:53:10,954 --> 00:53:12,989 COLLEAGUES ARE LIVING TO THESE 1556 00:53:13,056 --> 00:53:13,290 VALUES. 1557 00:53:13,356 --> 00:53:14,724 SO THERE'S THIS HUGE GAP BETWEEN 1558 00:53:14,791 --> 00:53:15,759 WHAT PEOPLE WANT TO DO AND WHAT 1559 00:53:15,825 --> 00:53:17,627 WE THINK WE SHOULD BE DOING AS 1560 00:53:17,694 --> 00:53:19,162 SCIENTISTS, AND HOW THEY FEEL 1561 00:53:19,229 --> 00:53:20,430 THEIR COLLEAGUES ARE ACTUALLY 1562 00:53:20,497 --> 00:53:20,997 BEHAVING. 1563 00:53:21,064 --> 00:53:22,532 SO I THINK THIS IS THE KIND OF 1564 00:53:22,599 --> 00:53:24,167 GAP THAT COULD MAYBE EXPLAIN 1565 00:53:24,234 --> 00:53:26,636 SOME OF THE PATTERNS WE'VE SEEN, 1566 00:53:26,703 --> 00:53:27,938 AND THEN IT KIND OF BEGS THE 1567 00:53:28,004 --> 00:53:30,140 QUESTION, HOW CAN WE BRING OUR 1568 00:53:30,206 --> 00:53:31,675 PRACTICES BACK IN LINE WITH OUR 1569 00:53:31,741 --> 00:53:32,509 VALUES? 1570 00:53:32,576 --> 00:53:34,177 AND I THINK DEALING WITH 1571 00:53:34,244 --> 00:53:35,111 PUBLICATION BIAS AND PREVENTING 1572 00:53:35,178 --> 00:53:36,813 IT AND REALLY THE GOALS OF THIS 1573 00:53:36,880 --> 00:53:38,448 WORKSHOP ARE REALLY AN IMPORTANT 1574 00:53:38,515 --> 00:53:46,222 STEP IN THAT DIRECTION. 1575 00:53:46,289 --> 00:53:47,524 SO THAT'S THE KIND OF I THINK 1576 00:53:47,591 --> 00:53:49,693 HIGH LEVEL ABOUT WHY WE'RE HERE, 1577 00:53:49,759 --> 00:53:51,227 WHY THIS MATTERS, WHAT OUR 1578 00:53:51,294 --> 00:53:51,761 VALUES SHOULD BE. 1579 00:53:51,828 --> 00:53:53,496 I WANT TO NOW TAKE A TURN INTO 1580 00:53:53,563 --> 00:53:55,665 THE SOCIAL SCIENCES AND IN 1581 00:53:55,732 --> 00:53:56,399 PARTICULAR ECONOMICS. 1582 00:53:56,466 --> 00:53:57,834 MY OWN FIELD, JUST TO GIVE A 1583 00:53:57,901 --> 00:53:59,469 LITTLE BIT OF BACK STORY AS TO 1584 00:53:59,536 --> 00:54:00,737 OUR WORK -- I THINK THERE'S A 1585 00:54:00,804 --> 00:54:02,372 FEW OTHER SOCIAL SCIENTISTS IN 1586 00:54:02,439 --> 00:54:02,939 THE FIELD. 1587 00:54:03,006 --> 00:54:05,609 I SEE NEIL BACK THERE AND I'VE 1588 00:54:05,675 --> 00:54:06,810 BEEN AT A FEW OF THESE MEETINGS 1589 00:54:06,876 --> 00:54:07,811 THROUGH THE YEARS. 1590 00:54:07,877 --> 00:54:09,112 MY OWN WORK IN THIS FIELD 1591 00:54:09,179 --> 00:54:11,448 STARTED WHEN I WAS DOING WORK IN 1592 00:54:11,514 --> 00:54:13,116 WEST AFRICA AND SIERRA LEONE 1593 00:54:13,183 --> 00:54:14,651 RIGHT AFTER THE END OF THE CIVIL 1594 00:54:14,718 --> 00:54:14,884 WAR. 1595 00:54:14,951 --> 00:54:16,786 SO THE CIVIL WAR ENDED IN 2002, 1596 00:54:16,853 --> 00:54:22,025 AND IN 2004, 5, WE WERE WORKING 1597 00:54:22,092 --> 00:54:23,293 TOGETHER, MY TEAM AND BERKELEY 1598 00:54:23,360 --> 00:54:24,861 WITH THE WORLD BANK AND THE 1599 00:54:24,928 --> 00:54:26,830 GOVERNMENT OF SIERRA LEONE ON 1600 00:54:26,896 --> 00:54:27,464 SOME RECONSTRUCTION EFFORTS. 1601 00:54:27,530 --> 00:54:29,299 IT WAS A PRETTY EXCITING TIME TO 1602 00:54:29,366 --> 00:54:29,733 BE THERE. 1603 00:54:29,799 --> 00:54:31,701 THERE WAS A LOT OF HOPE AND 1604 00:54:31,768 --> 00:54:32,969 OPTIMISM THAT THERE COULD BE 1605 00:54:33,036 --> 00:54:33,837 ECONOMIC REFORMS, POLITICAL 1606 00:54:33,903 --> 00:54:35,372 REFORMS THAT COULD LEAD SIERRA 1607 00:54:35,438 --> 00:54:36,706 LEONE FORWARD AND REALLY KIND OF 1608 00:54:36,773 --> 00:54:37,507 PREVENT THE RETURN TO CIVIL WAR. 1609 00:54:37,574 --> 00:54:38,742 SO THE STAKES WERE VERY HIGH. 1610 00:54:38,808 --> 00:54:40,043 AND I THINK FOR EVERYBODY HERE 1611 00:54:40,110 --> 00:54:41,878 WHO'S DOING RESEARCH ON 1612 00:54:41,945 --> 00:54:43,480 NEUROSCIENCE, THE STAKES ARE 1613 00:54:43,546 --> 00:54:43,947 HIGH. 1614 00:54:44,014 --> 00:54:45,915 OTHER PARTS OF BIOMEDICAL 1615 00:54:45,982 --> 00:54:46,349 RESEARCH. 1616 00:54:46,416 --> 00:54:47,217 CERTAINLY IN THIS CASE WHEN 1617 00:54:47,283 --> 00:54:48,818 WE'RE TALKING ABOUT WAR AND 1618 00:54:48,885 --> 00:54:50,553 PEACE, SUPER HIGH STAKES TO 1619 00:54:50,620 --> 00:54:51,755 THESE REFORMS. 1620 00:54:51,821 --> 00:54:53,156 THE REFORM THAT THE GOVERNMENT 1621 00:54:53,223 --> 00:54:55,458 WITH FUNDING FROM THE WORLD BANK 1622 00:54:55,525 --> 00:54:59,829 WAS ADVOCATING WAS CALLED GOBIFO 1623 00:54:59,896 --> 00:55:02,298 WHICH IN CREOLE, THE LANGUAGE IN 1624 00:55:02,365 --> 00:55:03,967 SIERRA LEONE MEANS MOVE AHEAD, 1625 00:55:04,034 --> 00:55:05,769 MOVE FORWARD, WAS A LOCAL 1626 00:55:05,835 --> 00:55:07,737 GOVERNMENT REFORM THAT TRIED TO 1627 00:55:07,804 --> 00:55:09,572 IMPROVE REPRESENTATIVENESS OF 1628 00:55:09,639 --> 00:55:10,473 LOCAL GOVERNMENT COUNCILS. 1629 00:55:10,540 --> 00:55:12,509 SO TRADITIONALLY, HEREDITARY 1630 00:55:12,575 --> 00:55:14,010 CHIEFS HAD A LOT OF AUTHORITY IN 1631 00:55:14,077 --> 00:55:17,247 RURAL AREAS IN SIERRA LEONE AND 1632 00:55:17,313 --> 00:55:18,581 THERE WAS A FEAR THAT THAT MAY 1633 00:55:18,648 --> 00:55:19,783 HAVE CONTRIBUTED TO POLITICAL 1634 00:55:19,849 --> 00:55:20,717 INSTABILITY AND CONFLICT AND THE 1635 00:55:20,784 --> 00:55:21,818 HOPE WAS AFTER THE WAR, THERE 1636 00:55:21,885 --> 00:55:24,154 COULD BE ELECTED LOCAL COUNCILS 1637 00:55:24,220 --> 00:55:25,722 INCLUDING BRINGING WOMEN ON THE 1638 00:55:25,789 --> 00:55:27,090 COUNCILS, MEMBERS OF 1639 00:55:27,157 --> 00:55:28,158 MARGINALIZED GROUPS ON TO 1640 00:55:28,224 --> 00:55:29,592 COUNCILS, NOT JUST THESE 1641 00:55:29,659 --> 00:55:31,327 HEREDITARY LEADERS TO LEAD TO 1642 00:55:31,394 --> 00:55:32,562 MORE DEMOCRATIC LOCAL GOVERNANCE 1643 00:55:32,629 --> 00:55:33,463 AND HAVE BETTER OUTCOMES. 1644 00:55:33,530 --> 00:55:35,899 SO A PRETTY EXCITING REFORM. 1645 00:55:35,965 --> 00:55:37,200 BUT ALSO CONTROVERSIAL. 1646 00:55:37,267 --> 00:55:39,469 IT WAS VERY POLITICALLY CHARGED. 1647 00:55:39,536 --> 00:55:41,204 THE CHIEFS WEREN'T THRILLED BY 1648 00:55:41,271 --> 00:55:42,839 THIS. 1649 00:55:42,906 --> 00:55:45,909 AND ALSO, IN THE ACADEMIC 1650 00:55:45,975 --> 00:55:47,544 LITERATURE, THE NOTION OF HOW 1651 00:55:47,610 --> 00:55:49,179 FEASIBLE IT EVEN WAS TO REFORM 1652 00:55:49,245 --> 00:55:50,146 LOCAL GOVERNMENT INSTITUTIONS 1653 00:55:50,213 --> 00:55:51,681 WAS VERY CONTESTED. 1654 00:55:51,748 --> 00:55:53,116 VERY CONTROVERSIAL. 1655 00:55:53,183 --> 00:55:55,085 SO OUR TEAM WHEN WE WERE WORKING 1656 00:55:55,151 --> 00:55:56,886 ON IT DECIDED TO DO SOMETHING 1657 00:55:56,953 --> 00:56:00,757 THAT WAS INSPIRED ACTUALLY BY 1658 00:56:00,824 --> 00:56:01,858 WHAT HAPPENS IN CLINICAL TRIALS, 1659 00:56:01,925 --> 00:56:04,327 WE DECIDED TO PREREGISTER OUR 1660 00:56:04,394 --> 00:56:05,895 ANALYSIS BEFORE CARRYING OUT 1661 00:56:05,962 --> 00:56:07,664 THIS RCT OF LOCAL GOVERNMENT 1662 00:56:07,731 --> 00:56:08,898 REFORM WHICH TOOK PLACE ACROSS 1663 00:56:08,965 --> 00:56:10,867 HUNDREDS OF VILLAGES IN SIERRA 1664 00:56:10,934 --> 00:56:11,301 LEONE. 1665 00:56:11,367 --> 00:56:12,802 WE WROTE DOWN AN ANALYSIS PLAN, 1666 00:56:12,869 --> 00:56:14,771 WE WROTE DOWN THE HYPOTHESES 1667 00:56:14,838 --> 00:56:16,306 THAT MOTIVATED US, WHAT OUR OUT 1668 00:56:16,372 --> 00:56:18,241 COME MEASURES WOULD BE, WHAT OUR 1669 00:56:18,308 --> 00:56:19,109 STATISTICAL TESTS WOULD BE, AND 1670 00:56:19,175 --> 00:56:20,710 WE ACTUALLY SHARED THEM WITH OUR 1671 00:56:20,777 --> 00:56:22,345 GOVERNMENT AND WORLD BANK 1672 00:56:22,412 --> 00:56:23,513 PARTNERS AND SAID, HEY, ARE WE 1673 00:56:23,580 --> 00:56:24,414 ON THE SAME PAGE? 1674 00:56:24,481 --> 00:56:25,548 ARE THESE THE RIGHT OUTCOMES 1675 00:56:25,615 --> 00:56:26,916 THAT WE SHOULD BE LOOKING AT OR 1676 00:56:26,983 --> 00:56:27,450 NOT? 1677 00:56:27,517 --> 00:56:29,319 AND WE BUILT CONSENSUS, WE 1678 00:56:29,385 --> 00:56:31,387 MODIFIED OUR HYPOTHESES AND GOT 1679 00:56:31,454 --> 00:56:32,255 EVERYBODY TO AGREE. 1680 00:56:32,322 --> 00:56:33,456 OUR FEAR WAS IF WE FOUND 1681 00:56:33,523 --> 00:56:35,325 SOMETHING THAT WAS NOT 1682 00:56:35,391 --> 00:56:36,059 POLITICALLY PALATABLE, THERE 1683 00:56:36,126 --> 00:56:37,694 WOULD BE PRESSURE PLACED ON US 1684 00:56:37,761 --> 00:56:40,196 TO CHANGE RESULTS, NOT PUBLISH 1685 00:56:40,263 --> 00:56:42,966 RESULTS, AND THIS IS SOMETHING 1686 00:56:43,032 --> 00:56:44,834 THAT HAPPENED IN SOCIAL SCIENCE 1687 00:56:44,901 --> 00:56:46,136 RESEARCH, ESPECIALLY IF YOU'RE 1688 00:56:46,202 --> 00:56:47,437 STUDYING GOVERNMENT POLICY. 1689 00:56:47,504 --> 00:56:48,605 SO THAT WAS OUR INNOVATION. 1690 00:56:48,671 --> 00:56:51,975 THIS WAS ONE OF THE FIRST COUPLE 1691 00:56:52,041 --> 00:56:52,742 FREE ANALYSIS PLANS IN 1692 00:56:52,809 --> 00:56:53,076 ECONOMICS. 1693 00:56:53,143 --> 00:56:54,844 IT WAS SOMETHING A LITTLE 1694 00:56:54,911 --> 00:56:58,448 STRANGE AT THE TIME. 1695 00:56:58,515 --> 00:56:59,883 BUT THAT WE BELIEVED IN AND IT 1696 00:56:59,949 --> 00:57:00,817 TURNED INTO SOMETHING OF A 1697 00:57:00,884 --> 00:57:02,118 MOVEMENT IN THE SOCIAL SCIENCES. 1698 00:57:02,185 --> 00:57:03,753 SO OVER TIME IN ECONOMICS, IN 1699 00:57:03,820 --> 00:57:05,522 POLITICAL SCIENCE AND OTHER 1700 00:57:05,588 --> 00:57:07,590 FIELDS, IT'S ACTUALLY AS I'LL 1701 00:57:07,657 --> 00:57:09,692 SHOW YOU BECOME THE NORM WHEN 1702 00:57:09,759 --> 00:57:12,595 OUR FIELD RCTs ARE RUN, 1703 00:57:12,662 --> 00:57:14,464 RANDOMIZED CONTROLLED TRIALS, TO 1704 00:57:14,531 --> 00:57:16,065 WRITE PRE-ANALYSIS PLANS AND 1705 00:57:16,132 --> 00:57:16,466 PRE-REGISTER THEM. 1706 00:57:16,533 --> 00:57:17,801 SO IT WAS AN EXCITING PROCESS. 1707 00:57:17,867 --> 00:57:19,202 I WILL SAY JUST ANOTHER ONE 1708 00:57:19,269 --> 00:57:23,273 SECOND ON THIS STUDY, WE'VE 1709 00:57:23,339 --> 00:57:24,240 BASICALLY FOUND NULL RESULTS OF 1710 00:57:24,307 --> 00:57:24,774 THIS REFORM. 1711 00:57:24,841 --> 00:57:26,543 WE BASICALLY FOUND THIS ATTEMPT 1712 00:57:26,609 --> 00:57:29,813 TO IMPROVE INCLUSION, HAVE 1713 00:57:29,879 --> 00:57:30,980 GREATER LOCAL PUBLIC GOODS, ET 1714 00:57:31,047 --> 00:57:31,981 CETERA, THERE WAS NO IMPACT OF 1715 00:57:32,048 --> 00:57:32,982 THIS REFORM. 1716 00:57:33,049 --> 00:57:35,218 WE HAVE A NEGATIVE RESULT, A 1717 00:57:35,285 --> 00:57:35,985 NULL PAPER. 1718 00:57:36,052 --> 00:57:37,520 I THINK IT ONLY REALLY GOT 1719 00:57:37,587 --> 00:57:38,755 PUBLISHED BECAUSE WE WERE WELL 1720 00:57:38,822 --> 00:57:41,324 POWERED AND HAD THIS 1721 00:57:41,391 --> 00:57:41,724 PRE-REGISTRATION. 1722 00:57:41,791 --> 00:57:43,059 SO IT GETS BACK TO THE POINT OF 1723 00:57:43,126 --> 00:57:44,060 DIRECTOR TABAK THAT CERTAIN 1724 00:57:44,127 --> 00:57:45,461 TYPES OF NULL STUDIES ARE 1725 00:57:45,528 --> 00:57:48,431 GETTING PUBLISHED, OUR PARTNER 1726 00:57:48,498 --> 00:57:49,399 IN THE GOVERNMENT WERE NOT 1727 00:57:49,465 --> 00:57:50,400 EXCITED ABOUT THAT RESULT, BUT 1728 00:57:50,466 --> 00:57:52,535 WE'D ALL AGREED ON THE ANALYSIS 1729 00:57:52,602 --> 00:57:54,204 BEFOREHAND THAT KIND OF SAVED 1730 00:57:54,270 --> 00:57:54,537 US. 1731 00:57:54,604 --> 00:57:55,805 WHAT ENDED UP HAPPENING TO THE 1732 00:57:55,872 --> 00:57:57,006 COUNCILS IS ACTUALLY THE 1733 00:57:57,073 --> 00:57:57,774 TRADITIONAL CHIEFS TOOK OVER THE 1734 00:57:57,841 --> 00:57:58,341 COUNCILS. 1735 00:57:58,408 --> 00:58:00,143 THEY HAD THEIR OWN ALLIES 1736 00:58:00,210 --> 00:58:02,011 BASICALLY GET ELECTED AND THEY 1737 00:58:02,078 --> 00:58:03,446 CONTINUED TO CONTROL THE COUNCIL 1738 00:58:03,513 --> 00:58:04,547 SO NOTHING CHANGED WITH THE 1739 00:58:04,614 --> 00:58:05,081 REFORM. 1740 00:58:05,148 --> 00:58:06,249 BUT WE WERE ABLE TO PUBLISH 1741 00:58:06,316 --> 00:58:09,986 THAT, I THINK, IN PART BECAUSE 1742 00:58:10,053 --> 00:58:11,087 OF PRE-REGISTRATION. 1743 00:58:11,154 --> 00:58:11,287 OKAY. 1744 00:58:11,354 --> 00:58:12,589 SO IN THE LAST 20 YEARS, THINGS 1745 00:58:12,655 --> 00:58:13,857 HAVE CHANGED A LOT IN OUR FIELD. 1746 00:58:13,923 --> 00:58:15,291 I THINK IN OTHER FIELDS AS WELL, 1747 00:58:15,358 --> 00:58:17,894 THERE'S MUCH MORE AWARENESS 1748 00:58:17,961 --> 00:58:19,095 ABOUT THE IMPORTANCE OF OPEN 1749 00:58:19,162 --> 00:58:21,397 SCIENCE AND THE PROBLEMS OF 1750 00:58:21,464 --> 00:58:23,700 PUBLICATION BIAS. 1751 00:58:23,766 --> 00:58:24,868 SO THINGS HAVE CHANGED. 1752 00:58:24,934 --> 00:58:26,402 LET ME JUST SHOW YOU SOME OF THE 1753 00:58:26,469 --> 00:58:28,805 CHANGES THAT HAVE OCCURRED IN 1754 00:58:28,872 --> 00:58:31,074 ECONOMICS, AND MY OWN FIELD. 1755 00:58:31,140 --> 00:58:32,942 SHORTLY -- A FEW YEARS AFTER WE 1756 00:58:33,009 --> 00:58:33,910 PREREGISTERED OUR STUDY, THERE 1757 00:58:33,977 --> 00:58:35,144 WASN'T EVEN A REGISTRY WHEN WE 1758 00:58:35,211 --> 00:58:35,545 DID IT. 1759 00:58:35,612 --> 00:58:37,513 WE SENT OUR PRE-ANALYSIS PLANS 1760 00:58:37,580 --> 00:58:39,382 TO CERTAIN PEOPLE TO TIME STAMP 1761 00:58:39,449 --> 00:58:40,650 IN PUBLIC PLACES BASICALLY, 1762 00:58:40,717 --> 00:58:43,052 BECAUSE THERE WAS NO REGISTRY. 1763 00:58:43,119 --> 00:58:43,519 THE AMERICAN ECONOMIC 1764 00:58:43,586 --> 00:58:45,455 ASSOCIATION CREATED A REGISTRY, 1765 00:58:45,521 --> 00:58:47,090 AND AS I MENTIONED, IT'S BECOME 1766 00:58:47,156 --> 00:58:49,926 A NORM IN THEIR EXPERIMENTS TO 1767 00:58:49,993 --> 00:58:52,528 REGISTER, SO WE WERE INSPIRED BY 1768 00:58:52,595 --> 00:58:52,962 CLINICALTRIALS.GOV. 1769 00:58:53,029 --> 00:58:54,397 AND YOU CAN SEE THE KIND OF 1770 00:58:54,464 --> 00:58:55,064 CUMULATIVE NUMBER. 1771 00:58:55,131 --> 00:58:57,567 NOW THERE HAVE BEEN 8500 1772 00:58:57,634 --> 00:58:58,468 REGISTERED EXPERIMENTS IN 1773 00:58:58,534 --> 00:59:00,036 ECONOMICS OVER THE LAST DECADE 1774 00:59:00,103 --> 00:59:03,539 OR SO, AND THE PER-QUARTER 1775 00:59:03,606 --> 00:59:04,240 REGISTRATIONS JUST CONTINUE TO 1776 00:59:04,307 --> 00:59:04,540 CLIMB. 1777 00:59:04,607 --> 00:59:07,010 SO THIS IS SOMETHING THAT'S 1778 00:59:07,076 --> 00:59:08,878 REALLY TAKEN OFF, AND NORMS 1779 00:59:08,945 --> 00:59:09,279 SHIFTED RAPIDLY. 1780 00:59:09,345 --> 00:59:12,715 THERE WAS NO REQUIREMENT TO 1781 00:59:12,782 --> 00:59:13,182 REGISTER. 1782 00:59:13,249 --> 00:59:14,217 BUT PEOPLE UNDERSTOOD THE VALUE 1783 00:59:14,284 --> 00:59:15,652 OF IT AND NORMS SHIFTED WITHIN A 1784 00:59:15,718 --> 00:59:18,454 FEW YEARS IN OUR FIELD. 1785 00:59:18,521 --> 00:59:19,956 SOMETHING SIMILAR HAS HAPPENED 1786 00:59:20,023 --> 00:59:20,957 WITH EXPERIMENTS IN POLITICAL 1787 00:59:21,024 --> 00:59:23,426 SCIENCE AND PSYCHOLOGY, WHERE 1788 00:59:23,493 --> 00:59:24,127 PREREGISTRATION HAS BECOME A 1789 00:59:24,193 --> 00:59:24,594 NORM. 1790 00:59:24,661 --> 00:59:25,461 SO A LOT OF CHANGES IN OUR 1791 00:59:25,528 --> 00:59:25,762 FIELD. 1792 00:59:25,828 --> 00:59:27,597 AND OUR HOPE AT THE TIME WAS 1793 00:59:27,664 --> 00:59:28,965 THAT PREREGISTRATION COULD HELP 1794 00:59:29,032 --> 00:59:30,033 AVOID PUBLICATION BIAS. 1795 00:59:30,099 --> 00:59:32,135 NOW THAT YOUR STUDY IS 1796 00:59:32,201 --> 00:59:33,202 REGISTERED AND PEOPLE CAN SEE 1797 00:59:33,269 --> 00:59:35,171 WHAT YOUR HYPOTHESES ARE, 1798 00:59:35,238 --> 00:59:36,239 THERE'S GOING TO BE MORE 1799 00:59:36,306 --> 00:59:37,206 ACCOUNTABILITY AND PRESSURE TO 1800 00:59:37,273 --> 00:59:37,874 PUBLISH THOSE RESULTS. 1801 00:59:37,941 --> 00:59:39,275 THAT WAS OUR WORKING HYPOTHESIS. 1802 00:59:39,342 --> 00:59:40,543 I'LL HAVE A FEW WORDS TO SAY 1803 00:59:40,610 --> 00:59:43,546 ABOUT THAT IN A FEW SLIDES. 1804 00:59:43,613 --> 00:59:45,315 WHAT ELSE HAS CHANGED, THIS IS 1805 00:59:45,381 --> 00:59:48,651 FRI A 'NOFROM ANOTHER STUDY WITH 1806 00:59:48,718 --> 00:59:50,320 CO-AUTHORS WE PUBLISHED LAST 1807 00:59:50,386 --> 00:59:50,520 YEAR. 1808 00:59:50,586 --> 00:59:52,055 SORRY THE FONT IS KIND OF SMALL. 1809 00:59:52,121 --> 00:59:54,357 WE SURVEYED THOUSANDS OF 1810 00:59:54,424 --> 00:59:55,992 SCHOLARS IN ECONOMICS, POLITICAL 1811 00:59:56,059 --> 00:59:57,727 SCIENCE, PSYCHOLOGY AND 1812 00:59:57,794 --> 00:59:57,994 SOCIOLOGY. 1813 00:59:58,061 --> 00:59:59,796 IN THIS FIGURE, WE LOOKED AT 1814 00:59:59,862 --> 01:00:00,863 SCHOLARS WHO HAD BEEN ACTIVE IN 1815 01:00:00,930 --> 01:00:02,365 THE FIELD FOR AT LEAST ABOUT A 1816 01:00:02,432 --> 01:00:04,200 DECADE, AND WE ASKED THEM WHEN 1817 01:00:04,267 --> 01:00:06,269 IF EVER THEY HAD ADOPTED CERTAIN 1818 01:00:06,336 --> 01:00:08,738 OPEN SCIENCE PRACTICES LIKE 1819 01:00:08,805 --> 01:00:10,239 SHARING THEIR OWN DATA, 1820 01:00:10,306 --> 01:00:11,674 PREREGISTERING, AND YOU SEE 1821 01:00:11,741 --> 01:00:13,009 THESE UPWARD TRENDS OVER TIME. 1822 01:00:13,076 --> 01:00:16,746 THERE'S BEEN BIG SHIFTS IN 1823 01:00:16,813 --> 01:00:17,747 BEHAVIOR, PREREGISTRATION, DATA 1824 01:00:17,814 --> 01:00:19,816 SHARING, POSTING INSTRUMENTS, 1825 01:00:19,882 --> 01:00:21,784 AND AGAIN, SOME STUFF THAT WAS 1826 01:00:21,851 --> 01:00:26,589 RARE 15 YEARS AGO OR 20 YEARS 1827 01:00:26,656 --> 01:00:29,092 AGO HAS BECOME VERY COMMON. 1828 01:00:29,158 --> 01:00:30,626 NEIL WILL BE HAPPY TO HEAR THAT 1829 01:00:30,693 --> 01:00:31,561 POLITICAL SCIENTISTS ACTUALLY 1830 01:00:31,627 --> 01:00:32,729 HAVE THE HIGHEST RATES OF 1831 01:00:32,795 --> 01:00:35,131 ADOPTION EVEN A BIT HIGHER THAN 1832 01:00:35,198 --> 01:00:35,865 ECONOMISTS HERE. 1833 01:00:35,932 --> 01:00:38,301 AND FOR BIOMEDICAL AUDIENCE, 1834 01:00:38,368 --> 01:00:39,469 DATA SHARING HAS REALLY TAKEN 1835 01:00:39,535 --> 01:00:39,969 OFF. 1836 01:00:40,036 --> 01:00:42,405 THERE'S A LOT OF BARRIERS TO 1837 01:00:42,472 --> 01:00:43,473 SHARING INDIVIDUAL DATA IN THE 1838 01:00:43,539 --> 01:00:44,907 HEALTH FIELD AND OBVIOUSLY 1839 01:00:44,974 --> 01:00:46,342 THERE'S HIPAA CONCERNS, BUT IN 1840 01:00:46,409 --> 01:00:47,643 ECONOMICS AND POLITICAL SCIENCE 1841 01:00:47,710 --> 01:00:50,380 JOURNALS NOW, IT'S REALLY 1842 01:00:50,446 --> 01:00:52,015 STANDARD TO SHARE REPLICATION 1843 01:00:52,081 --> 01:00:53,850 DATA AND CODE, IT'S BECOME 1844 01:00:53,916 --> 01:00:56,686 STANDARD. 1845 01:00:56,753 --> 01:00:57,987 AND OTHER INNOVATIONS, I 1846 01:00:58,054 --> 01:00:59,689 MENTIONED STUDY PREREGISTRATION, 1847 01:00:59,756 --> 01:01:01,557 SHARING OF DATA CODE AND 1848 01:01:01,624 --> 01:01:01,958 MATERIALS. 1849 01:01:02,025 --> 01:01:03,993 THERE ARE NEW JOURNAL REVIEW 1850 01:01:04,060 --> 01:01:04,594 PRACTICES, REGISTERED REPORTS 1851 01:01:04,660 --> 01:01:05,328 HAVE TAKEN OFF. 1852 01:01:05,395 --> 01:01:07,730 THERE'S BEEN SOME GREAT 1853 01:01:07,797 --> 01:01:08,798 INNOVATION ON THE JOURNAL SIDE. 1854 01:01:08,865 --> 01:01:10,333 A LOT OF JOURNALS NOW HAVE WHAT 1855 01:01:10,400 --> 01:01:12,969 ARE CALLED DATA EDITORS, █WHERE 1856 01:01:13,036 --> 01:01:14,404 THE DATA EDITOR TEAM DOUBLE 1857 01:01:14,470 --> 01:01:18,674 CHECKS ALL STATISTICAL CODE AND 1858 01:01:18,741 --> 01:01:20,910 DATA BEFORE PUBLICATION TO 1859 01:01:20,977 --> 01:01:21,477 VERIFY THE ESTIMATES IN THE 1860 01:01:21,544 --> 01:01:22,245 PAPER ACTUALLY HOLD. 1861 01:01:22,311 --> 01:01:24,414 SO THERE'S THAT DEGREE OF LIKE 1862 01:01:24,480 --> 01:01:26,282 COLLABORATION BETWEEN THE 1863 01:01:26,349 --> 01:01:29,285 AUTHORS AND THE JOURNAL, THE 1864 01:01:29,352 --> 01:01:29,952 AMERICAN ECONOMIC ASSOCIATION 1865 01:01:30,019 --> 01:01:34,457 WAS ONE OF THE FIRST TO DO THAT. 1866 01:01:34,524 --> 01:01:35,391 AMERICAN JOURNAL OF POLITICAL 1867 01:01:35,458 --> 01:01:36,526 SCIENCE DID IT AS WELL. 1868 01:01:36,592 --> 01:01:37,827 THERE'S BEEN A LOT MORE META 1869 01:01:37,894 --> 01:01:38,728 SCIENTIFIC WORK IN OUR FIELDS 1870 01:01:38,795 --> 01:01:40,763 AND A BUNCH OF NEW ORGANIZATIONS 1871 01:01:40,830 --> 01:01:42,632 HAVE BEEN CREATED TO TRY TO 1872 01:01:42,698 --> 01:01:44,834 INSTITUTIONALIZE CHANGE AND 1873 01:01:44,901 --> 01:01:46,035 PROMOTE IT. 1874 01:01:46,102 --> 01:01:47,570 INCLUDING OUR GROUP AT BERKELEY, 1875 01:01:47,637 --> 01:01:48,938 BITS, THE CENTER FOR OPEN 1876 01:01:49,005 --> 01:01:50,740 SCIENCE AND THE INSTITUTE FOR 1877 01:01:50,807 --> 01:01:51,374 REPLICATION AMONG OTHERS. 1878 01:01:51,441 --> 01:01:53,209 SO IT'S BEEN A VERY EXCITING 1879 01:01:53,276 --> 01:01:54,644 TIME PERIOD AND IN MANY WAYS, 1880 01:01:54,710 --> 01:01:56,012 WE'VE BEEN OPTIMISTIC, WE'VE 1881 01:01:56,079 --> 01:01:58,247 SEEN A LOT OF CHANGES THAT ARE 1882 01:01:58,314 --> 01:01:59,749 POSITIVE. 1883 01:01:59,816 --> 01:02:01,717 WHAT I WANT TO DO NEXT IS TALK 1884 01:02:01,784 --> 01:02:02,919 ABOUT REMAINING CHALLENGES 1885 01:02:02,985 --> 01:02:04,353 BECAUSE THERE'S A LOT OF THEM. 1886 01:02:04,420 --> 01:02:06,089 AS WE'VE COLLECTED MORE DATA, WE 1887 01:02:06,155 --> 01:02:07,623 REALIZE THAT EVEN THOUGH WE'VE 1888 01:02:07,690 --> 01:02:09,058 TAKEN THREE OR FOUR STEPS 1889 01:02:09,125 --> 01:02:10,159 FORWARD IN THE SOCIAL SCIENCES, 1890 01:02:10,226 --> 01:02:11,761 WE STILL HAVE A LONG WAY TO GO. 1891 01:02:11,828 --> 01:02:13,229 SO MAYBE THERE'S SOME PARALLELS 1892 01:02:13,296 --> 01:02:17,900 THERE TO BIOMEDICAL RESEARCH. 1893 01:02:17,967 --> 01:02:19,735 SO ONE BIG CHALLENGE IS EVEN 1894 01:02:19,802 --> 01:02:21,170 THOUGH A LOT OF NEW TOOLS HAVE 1895 01:02:21,237 --> 01:02:25,741 BEEN ADOPTED IN EXPERIMENTAL 1896 01:02:25,808 --> 01:02:27,276 RESEARCH, THOSE CARRYING OUT 1897 01:02:27,343 --> 01:02:29,345 RANDOMIZED CONTROL TRIALS OR 1898 01:02:29,412 --> 01:02:30,246 EXPERIMENTAL LABORATORY TRIALS 1899 01:02:30,313 --> 01:02:31,747 LIKE PSYCHOLOGY TRIALS, THERE'S 1900 01:02:31,814 --> 01:02:33,082 BEEN A LOT LESS PROGRESS IN WHAT 1901 01:02:33,149 --> 01:02:35,918 WE CALL OBSERVATIONAL RESEARCH. 1902 01:02:35,985 --> 01:02:37,353 SO DATA NOT BASED ON 1903 01:02:37,420 --> 01:02:38,621 EXPERIMENTS. 1904 01:02:38,688 --> 01:02:40,590 DATA THAT'S BASED ON EXISTING 1905 01:02:40,656 --> 01:02:42,191 ADMINISTRATIVE RECORDS OR SURVEY 1906 01:02:42,258 --> 01:02:42,592 DATA. 1907 01:02:42,658 --> 01:02:44,560 THERE'S SOME PARALLELS HERE, I 1908 01:02:44,627 --> 01:02:46,062 THINK, BETWEEN THE KIND OF 1909 01:02:46,129 --> 01:02:47,396 PRE-CLINICAL LABORATORY PHASE OF 1910 01:02:47,463 --> 01:02:49,031 RESEARCH AND MAYBE THE CLINICAL 1911 01:02:49,098 --> 01:02:52,101 PHASE IN MEDICAL RESEARCH. 1912 01:02:52,168 --> 01:02:54,170 VERY OFTEN THE OBSERVATIONAL 1913 01:02:54,237 --> 01:02:55,705 RESEARCH THEN INSPIRES 1914 01:02:55,771 --> 01:02:57,974 RANDOMIZED CONTROL TRIALS. 1915 01:02:58,040 --> 01:02:59,575 TO TEST CERTAIN IDEAS. 1916 01:02:59,642 --> 01:03:01,344 BUT YOU KNOW, EVEN IN OUR OWN 1917 01:03:01,410 --> 01:03:06,182 DATA, THE RATE OF 1918 01:03:06,249 --> 01:03:07,650 PREREGISTRATION, DATA SHARING, 1919 01:03:07,717 --> 01:03:09,118 ET CETERA, IS LOWER IN 1920 01:03:09,185 --> 01:03:11,187 OBSERVATIONAL RESEARCH. 1921 01:03:11,254 --> 01:03:11,721 NON-EXPERIMENTAL RESEARCH. 1922 01:03:11,787 --> 01:03:14,357 SO THAT'S A BIG GAP, AND IT'S 1923 01:03:14,423 --> 01:03:15,791 HARDER TO DO TO ADOPT A LOT OF 1924 01:03:15,858 --> 01:03:17,293 THESE TOOLS IN OBSERVATIONAL 1925 01:03:17,360 --> 01:03:17,827 RESEARCH. 1926 01:03:17,894 --> 01:03:19,795 IT'S HARDER, FOR INSTANCE, TO 1927 01:03:19,862 --> 01:03:20,730 PRE-REGISTER STUDIES THAT ARE 1928 01:03:20,796 --> 01:03:22,965 NOT BASED ON DATA THAT HASN'T 1929 01:03:23,032 --> 01:03:23,733 YET BEEN COLLECTED. 1930 01:03:23,799 --> 01:03:25,468 WHEN THE DATA IS ALREADY OUT 1931 01:03:25,535 --> 01:03:29,739 THERE, PUBLICLY AVAILABLE 1932 01:03:29,805 --> 01:03:31,641 ADMINISTRATIVE DATA, WHEN 1933 01:03:31,707 --> 01:03:32,642 EXACTLY DO YOU PREREGISTER? 1934 01:03:32,708 --> 01:03:33,843 THERE'S BEEN A LOT OF DEBATE 1935 01:03:33,910 --> 01:03:34,710 ABOUT THAT IN THE SOCIAL 1936 01:03:34,777 --> 01:03:35,978 SCIENCES AND A LOT OF PUSH BACK 1937 01:03:36,045 --> 01:03:37,446 SO THAT'S REALLY AN OPEN ISSUE. 1938 01:03:37,513 --> 01:03:38,981 ANOTHER OPEN ISSUE OR JUST 1939 01:03:39,048 --> 01:03:40,383 QUESTION IS GOSH, WE'VE ADOPTED 1940 01:03:40,449 --> 01:03:41,384 ALL THESE PRACTICES IN THE LAST 1941 01:03:41,450 --> 01:03:42,451 15 OR 20 YEARS, I SHOWED YOU 1942 01:03:42,518 --> 01:03:43,653 THAT FIGURE WITH ALL THOSE 1943 01:03:43,719 --> 01:03:45,688 CURVES GOING UPWARDS, HAVE WE 1944 01:03:45,755 --> 01:03:47,056 DEALT WITH PUBLICATION BIAS? 1945 01:03:47,123 --> 01:03:50,560 HAVE WE CRACKED THE CODE OR NOT? 1946 01:03:50,626 --> 01:03:51,861 AND -- OR IS THERE STILL A LOT 1947 01:03:51,928 --> 01:03:53,296 OF PUBLICATION BIAS AND 1948 01:03:53,362 --> 01:03:57,667 SELECTIVE REPORTING? 1949 01:03:57,733 --> 01:03:59,535 WE DON'T HAVE DEFINITIVE ANSWERS 1950 01:03:59,602 --> 01:04:00,836 TO THIS BUT THERE'S SOME NEW 1951 01:04:00,903 --> 01:04:02,471 DATA AND IN FACT MY GROUP AT 1952 01:04:02,538 --> 01:04:03,873 BERKELEY IS JUST STARTING TO TRY 1953 01:04:03,940 --> 01:04:05,174 TO QUANTIFY THIS IN ECONOMICS 1954 01:04:05,241 --> 01:04:06,709 AND WE HAVE SOME KIND OF 1955 01:04:06,776 --> 01:04:07,476 DISCOURAGING RESULTS. 1956 01:04:07,543 --> 01:04:08,911 MAYBE THIS IS JUST A GOOD SEGUE 1957 01:04:08,978 --> 01:04:10,179 INTO THIS WORKSHOP. 1958 01:04:10,246 --> 01:04:14,250 THERE'S' THERE'S A LOT MORE TO . 1959 01:04:14,317 --> 01:04:20,590 WE HAVE A GREAT TEAM LED BY MY 1960 01:04:20,656 --> 01:04:26,162 COLLEAGUE, DR. DELA GUARDIA. 1961 01:04:26,229 --> 01:04:27,797 WE DECIDED TO GO BACK TO OUR 1962 01:04:27,863 --> 01:04:30,366 EARLY QUESTION, HOW MUCH DOES 1963 01:04:30,433 --> 01:04:31,968 REGISTRATION DEAL WITH 1964 01:04:32,034 --> 01:04:32,368 PUBLICATION BIAS? 1965 01:04:32,435 --> 01:04:33,836 THAT WAS OUR OPEN WITH THE STUDY 1966 01:04:33,903 --> 01:04:34,704 IN SIERRA LEONE. 1967 01:04:34,770 --> 01:04:35,938 NOW THE REGISTRY IN ECONOMICS 1968 01:04:36,005 --> 01:04:37,006 HAS BEEN AROUND FOR OVER A 1969 01:04:37,073 --> 01:04:38,207 DECADE SO WE CAN ACTUALLY START 1970 01:04:38,274 --> 01:04:39,842 LOOKING AT THIS. 1971 01:04:39,909 --> 01:04:41,444 SO WE WENT BACK TO THE STUDY 1972 01:04:41,510 --> 01:04:43,980 REGISTRY AND STARTED CODING UP 1973 01:04:44,046 --> 01:04:44,847 STUDIES STARTING WITH THOSE 1974 01:04:44,914 --> 01:04:47,450 REGISTERED IN 2015, 2016, 2017, 1975 01:04:47,516 --> 01:04:48,551 2018, WE'RE KIND OF WORKING OUR 1976 01:04:48,618 --> 01:04:49,785 WAY THROUGH THOSE EARLY STUDIES 1977 01:04:49,852 --> 01:04:50,319 ON THE REGISTRY. 1978 01:04:50,386 --> 01:04:52,488 A LOT OF TIME HAS PASSED SINCE 1979 01:04:52,555 --> 01:04:53,823 THOSE STUDIES WERE REGISTERED. 1980 01:04:53,889 --> 01:04:55,658 PLENTY OF TIME TO KIND OF GET 1981 01:04:55,725 --> 01:04:57,059 MANUSCRIPTS OUT AND GET THINGS 1982 01:04:57,126 --> 01:04:57,426 PUBLISHED. 1983 01:04:57,493 --> 01:04:58,961 AND WHAT WE'RE DOING IS GOING 1984 01:04:59,028 --> 01:05:00,796 THROUGH THOSE REGISTRIES, AND 1985 01:05:00,863 --> 01:05:02,765 CODING UP THE PRIMARY HYPOTHESES 1986 01:05:02,832 --> 01:05:03,766 THAT THEY LIST. 1987 01:05:03,833 --> 01:05:05,601 SO IN THEIR PLAN, THEY'RE LIKE, 1988 01:05:05,668 --> 01:05:07,136 HERE ARE OUR PRIMARY HYPOTHESES, 1989 01:05:07,203 --> 01:05:09,305 OUR MAIN OUTCOMES, OUR MAIN 1990 01:05:09,372 --> 01:05:11,307 MODELS, AND WE'VE ACTUALLY CODED 1991 01:05:11,374 --> 01:05:12,174 UP 136 REGISTRATIONS SO FAR. 1992 01:05:12,241 --> 01:05:14,910 THIS IS A LOT OF WORK BECAUSE 1993 01:05:14,977 --> 01:05:16,879 THERE'S NOT STANDARDIZED 1994 01:05:16,946 --> 01:05:17,847 PRE-ANALYSIS PLAN FORMATS 1995 01:05:17,913 --> 01:05:18,948 ACTUALLY IN ECONOMICS, AND 1996 01:05:19,015 --> 01:05:21,250 THERE'S A LOT OF JUDGMENT CALLS, 1997 01:05:21,317 --> 01:05:22,251 AUTHORS AREN'T ALWAYS CLEAR, SO 1998 01:05:22,318 --> 01:05:25,388 WE HAVE AN AMAZING TEAM THAT'S 1999 01:05:25,454 --> 01:05:29,558 DOING THIS CODING, AND THEN 2000 01:05:29,625 --> 01:05:31,294 WE'RE FOLLOWING UP THESE AUTHORS 2001 01:05:31,360 --> 01:05:32,928 IN THESE STUDIES TO SEE WHAT GOT 2002 01:05:32,995 --> 01:05:35,197 PUBLISHED NOW, EIGHT, NINE, 2003 01:05:35,264 --> 01:05:37,400 SEVEN YEARS LATER. 2004 01:05:37,466 --> 01:05:38,634 COULD BE A JOURNAL PUBLICATION, 2005 01:05:38,701 --> 01:05:39,935 IT COULD BE A PRE-PRINT WORKING 2006 01:05:40,002 --> 01:05:41,037 PAPER, IT COULD BE A REPORT FOR 2007 01:05:41,103 --> 01:05:42,138 A GOVERNMENT. 2008 01:05:42,204 --> 01:05:43,105 WE'RE JUST TRYING TO FIND IT. 2009 01:05:43,172 --> 01:05:45,641 IS THIS RESULT IN THE PUBLIC 2010 01:05:45,708 --> 01:05:46,309 DOMAIN? 2011 01:05:46,375 --> 01:05:49,779 SO REGISTRATIONS IN ECONOMICS ON 2012 01:05:49,845 --> 01:05:51,447 AVERAGE HAVE ABOUT 17 PRIMARY 2013 01:05:51,514 --> 01:05:51,747 HYPOTHESES. 2014 01:05:51,814 --> 01:05:53,416 YOU MIGHT SAY WOW, THAT'S A LOT, 2015 01:05:53,482 --> 01:05:55,885 BUT IF A TRIAL HAS LIKE THREE 2016 01:05:55,951 --> 01:05:58,220 ARMS, WHICH IS PRETTY TYPICAL 2017 01:05:58,287 --> 01:06:00,856 FOR ECONOMIC IS STUDIES, THERE 2018 01:06:00,923 --> 01:06:02,291 COULD BE DIFFERENT VERSIONS OF A 2019 01:06:02,358 --> 01:06:07,730 TREATMENT AND THERE COULD BE 2020 01:06:07,797 --> 01:06:09,699 THREE MAIN OUTCOMES OR END 2021 01:06:09,765 --> 01:06:11,233 POINTS AND YOU THROW IN A COUPLE 2022 01:06:11,300 --> 01:06:13,436 OF SUBGROUP ANALYSES AND YOU'RE 2023 01:06:13,502 --> 01:06:16,405 THERE, KIND OF 18 HYPOTHESES, SO 2024 01:06:16,472 --> 01:06:17,173 THAT'S THE AVERAGE. 2025 01:06:17,239 --> 01:06:18,607 MEDIAN IS 8. 2026 01:06:18,674 --> 01:06:22,645 SO, SO FAR WE'VE REGISTERED 2400 2027 01:06:22,712 --> 01:06:23,045 HYPOTHESES. 2028 01:06:23,112 --> 01:06:24,113 HOW MANY OF THESE ARE PUBLICLY 2029 01:06:24,180 --> 01:06:24,413 AVAILABLE? 2030 01:06:24,480 --> 01:06:25,581 KIND OF A SIMPLE QUESTION. 2031 01:06:25,648 --> 01:06:26,882 CAN WE FIND THESE? 2032 01:06:26,949 --> 01:06:27,983 THESE THINGS WERE REGISTERED 2033 01:06:28,050 --> 01:06:28,651 ALMOST 10 YEARS. 2034 01:06:28,718 --> 01:06:29,618 CAN WE ACTUALLY FIND THESE 2035 01:06:29,685 --> 01:06:29,952 ESTIMATES? 2036 01:06:30,019 --> 01:06:31,387 IT ACTUALLY GOES BACK TO NEIL'S 2037 01:06:31,454 --> 01:06:32,421 WORK AND IT GOES BACK TO THE 2038 01:06:32,488 --> 01:06:34,523 FIGURES THAT SHAI WAS SHOWING, 2039 01:06:34,590 --> 01:06:35,725 LIKE HOW MUCH STUFF IS 2040 01:06:35,791 --> 01:06:36,258 DISAPPEARING FROM THE 2041 01:06:36,325 --> 01:06:39,128 LITERATURE? 2042 01:06:39,195 --> 01:06:40,396 I ALSO WANT TO SAY I THINK THIS 2043 01:06:40,463 --> 01:06:42,732 IS A BEST CASE SCENARIO. 2044 01:06:42,798 --> 01:06:43,933 THESE ARE EXPERIMENTAL 2045 01:06:43,999 --> 01:06:45,000 ESTIMATES, THEY'RE 2046 01:06:45,067 --> 01:06:45,801 PREREGISTERED, THERE'S SOME 2047 01:06:45,868 --> 01:06:46,302 ACCOUNTABILITY. 2048 01:06:46,369 --> 01:06:47,436 PROBABLY THE SHARE OF HYPOTHESIS 2049 01:06:47,503 --> 01:06:48,838 TESTS THAT ARE SHOWING UP IN 2050 01:06:48,904 --> 01:06:50,373 OBSERVATIONAL RESEARCH IS 2051 01:06:50,439 --> 01:06:51,374 LIKE -- I'M ALMOST SURE IT'S 2052 01:06:51,440 --> 01:06:52,575 LOWER, I DON'T HAVE DATA, BUT 2053 01:06:52,641 --> 01:06:53,976 THIS IS PROBABLY A BEST CASE 2054 01:06:54,043 --> 01:06:54,810 SCENARIO. 2055 01:06:54,877 --> 01:06:58,147 SO HAVE WE SOLVED THE PROBLEM? 2056 01:06:58,214 --> 01:06:59,548 AGAIN THIS IS PRELIMINARY DATA. 2057 01:06:59,615 --> 01:07:10,126 WE'VE ONLY COLLECTED -- WE'RE 2058 01:07:14,330 --> 01:07:15,865 GOING TO DO ABOUT 400 STUDIES, 2059 01:07:15,931 --> 01:07:18,501 WE'VE DONE ABOUT A THIRD. 2060 01:07:18,567 --> 01:07:21,537 THIS IS TENTATIVE, DO CITE, BUT 2061 01:07:21,604 --> 01:07:22,638 I WANTED TO SHARE WITH THIS 2062 01:07:22,705 --> 01:07:23,139 GROUP. 2063 01:07:23,205 --> 01:07:25,775 SO FAR WHEN WE LOOK EIGHT YEARS 2064 01:07:25,841 --> 01:07:27,209 LATER, WE CAN ACTUALLY FIND SOME 2065 01:07:27,276 --> 01:07:28,310 DATA FOR 79% OF THE 2066 01:07:28,377 --> 01:07:28,644 REGISTRATIONS. 2067 01:07:28,711 --> 01:07:30,012 I DON'T KNOW IF YOU SEE THAT AS 2068 01:07:30,079 --> 01:07:30,713 GOOD OR BAD. 2069 01:07:30,780 --> 01:07:31,914 IT'S DEFINITELY NOT 100. 2070 01:07:31,981 --> 01:07:34,083 IT'S A LITTLE HIGHER THAN I WAS 2071 01:07:34,150 --> 01:07:34,650 EXPECTING. 2072 01:07:34,717 --> 01:07:36,819 BUT THAT'S STILL LIKE ALMOST A 2073 01:07:36,886 --> 01:07:38,354 QUARTER OF THESE REGISTRATIONS 2074 01:07:38,421 --> 01:07:39,555 JUST, LIKE -- THERE'S NO DATA 2075 01:07:39,622 --> 01:07:40,890 THAT WE COULD -- THERE'S NO 2076 01:07:40,956 --> 01:07:43,292 PUBLICLY AVAILABLE -- NO 2077 01:07:43,359 --> 01:07:44,126 PUBLICATIONS, NO RESULTS THAT WE 2078 01:07:44,193 --> 01:07:44,593 COULD FIND. 2079 01:07:44,660 --> 01:07:45,461 WE'RE INVESTIGATING THROUGH 2080 01:07:45,528 --> 01:07:46,662 SURVEYS AND OTHER RESEARCH HOW 2081 01:07:46,729 --> 01:07:48,097 MANY OF THESE ARE JUST LIKE 2082 01:07:48,164 --> 01:07:50,266 TOTALLY NON-REPORTED VERSUS 2083 01:07:50,332 --> 01:07:51,901 MAYBE THE TRIAL NEVER HAPPENED. 2084 01:07:51,967 --> 01:07:53,335 SO SOME OF THIS COULD BE THEY 2085 01:07:53,402 --> 01:07:54,770 REGISTERED IT AND THEN THEIR 2086 01:07:54,837 --> 01:07:55,871 FUNDING DEPARTMENT COME THROUGH 2087 01:07:55,938 --> 01:07:58,707 OR THEIR GOVERNMENT PARTNER SHUT 2088 01:07:58,774 --> 01:07:59,542 THEM DOWN. 2089 01:07:59,608 --> 01:08:00,476 SO WE'RE TRYING TO SORT THAT 2090 01:08:00,543 --> 01:08:00,810 OUT. 2091 01:08:00,876 --> 01:08:02,211 SO WE COULD BE GENEROUS AND SAY 2092 01:08:02,278 --> 01:08:03,412 WELL, MAYBE THIS IS JUST -- 2093 01:08:03,479 --> 01:08:04,847 THESE WERE STUDIES THAT DIDN'T 2094 01:08:04,914 --> 01:08:06,282 OCCUR, BUT PROBABLY A LOT OF 2095 01:08:06,348 --> 01:08:08,551 THEM, NOTHING GOT REPORTED. 2096 01:08:08,617 --> 01:08:10,085 SO IN THE KIND OF TERMINOLOGY 2097 01:08:10,152 --> 01:08:11,954 SHAI WAS USING, THIS WOULD BE 2098 01:08:12,021 --> 01:08:13,155 LIKE PUBLICATION BIAS. 2099 01:08:13,222 --> 01:08:14,490 NOTHING CAME OUT OF THESE 2100 01:08:14,557 --> 01:08:14,790 STUDIES. 2101 01:08:14,857 --> 01:08:16,025 WE'RE TRYING TO FIGURE OUT NOW 2102 01:08:16,091 --> 01:08:16,759 WHY. 2103 01:08:16,826 --> 01:08:18,093 THEN WE DIG INTO STUDIES, SO 2104 01:08:18,160 --> 01:08:19,595 THIS IS RELATED TO SELECTIVE 2105 01:08:19,662 --> 01:08:21,163 REPORTING AND PUBLICATION BIAS, 2106 01:08:21,230 --> 01:08:26,068 SOME COMBINATION, AS TO HOW MANY 2107 01:08:26,135 --> 01:08:27,803 OF THESE ESTIMATES, THESE 2400 2108 01:08:27,870 --> 01:08:29,038 HYPOTHESES, CAN WE FIND RESULTS 2109 01:08:29,104 --> 01:08:30,105 FOR IN THE PUBLISHED LITERATURE 2110 01:08:30,172 --> 01:08:31,907 IN THIS DATA SO FAR? 2111 01:08:31,974 --> 01:08:34,643 WE CAN FINE ABOUT HALF. 2112 01:08:34,710 --> 01:08:35,744 FIND ABOUT HALF. 2113 01:08:35,811 --> 01:08:37,279 SO 79% OF STUDIES ARE OUT THERE, 2114 01:08:37,346 --> 01:08:39,348 BUT NOT ALL THE PRIMARY 2115 01:08:39,415 --> 01:08:40,783 HYPOTHESES AND THE REGISTRATION 2116 01:08:40,850 --> 01:08:41,984 GET REPORTED IN THE PAPER. 2117 01:08:42,051 --> 01:08:43,953 AND THIS IS SOMETHING THAT'S 2118 01:08:44,019 --> 01:08:44,720 VERY COMMON. 2119 01:08:44,787 --> 01:08:46,789 THERE'S LIKE THIS SLIPPAGE WHERE 2120 01:08:46,856 --> 01:08:48,657 ONLY A SUBSET OF OUTCOMES ARE 2121 01:08:48,724 --> 01:08:50,726 REPORTED, A SUBSET OF 2122 01:08:50,793 --> 01:08:51,560 TREATMENTS. 2123 01:08:51,627 --> 01:08:52,828 YOU MIGHT SAY, WELL, EVENTUALLY 2124 01:08:52,895 --> 01:08:53,729 MAYBE THEY'LL PUBLISH SOMETHING 2125 01:08:53,796 --> 01:08:54,964 THAT REPORTS THESE ESTIMATES, 2126 01:08:55,030 --> 01:08:56,265 BUT AGAIN, THIS IS LIKE EIGHT OR 2127 01:08:56,332 --> 01:08:56,966 NINE YEARS LATER. 2128 01:08:57,032 --> 01:08:58,267 SO WE'RE NOT SURE HOW MUCH FLOW 2129 01:08:58,334 --> 01:08:59,802 OF NEW PUBLISHED STUDIES THERE'S 2130 01:08:59,869 --> 01:09:00,803 GOING TO BE FROM THESE 2131 01:09:00,870 --> 01:09:05,074 REGISTRATIONS. 2132 01:09:05,140 --> 01:09:07,676 IF WE ONLY LOOK AT THAT 79% OF 2133 01:09:07,743 --> 01:09:08,677 STUDIES WHERE WE FOUND 2134 01:09:08,744 --> 01:09:12,414 SOMETHING, THEN ABOUT 72% OF 2135 01:09:12,481 --> 01:09:13,182 HYPOTHESES GET REPORTED. 2136 01:09:13,249 --> 01:09:15,251 SO SOME OF THIS IS JUST PURE 2137 01:09:15,317 --> 01:09:16,352 PUBLICATION BUY A WE DON'T SEE 2138 01:09:16,418 --> 01:09:17,019 THE PUBLICATION. 2139 01:09:17,086 --> 01:09:18,554 SOME OF IT'S SELECTIVE REPORTING 2140 01:09:18,621 --> 01:09:19,622 WITHIN THOSE STUDIES. 2141 01:09:19,688 --> 01:09:23,692 BUT ABOUT 57% OF ESTIMATES ARE 2142 01:09:23,759 --> 01:09:25,828 FOUND. 2143 01:09:25,895 --> 01:09:27,763 THEN OUR THIRD FINDING, AND I'LL 2144 01:09:27,830 --> 01:09:29,164 JUST LEAVE IT THERE, I'LL START 2145 01:09:29,231 --> 01:09:31,267 WRAPPING IT UP, SO WE WENT TO 2146 01:09:31,333 --> 01:09:32,101 THOSE PUBLICATIONS. 2147 01:09:32,167 --> 01:09:34,003 WE HAD REGISTERED THESE 2148 01:09:34,069 --> 01:09:35,838 HYPOTHESES AND MADE SOME 2149 01:09:35,905 --> 01:09:36,939 JUDGMENT CALLS BASED ON THE 2150 01:09:37,006 --> 01:09:40,576 RULES AS TO HOW CLOSE. 2151 01:09:40,643 --> 01:09:41,877 SO EVEN WHEN THERE'S A RESULT 2152 01:09:41,944 --> 01:09:43,512 REPORTED FOR A GIVEN HYPOTHESIS, 2153 01:09:43,579 --> 01:09:45,047 MANY TIMES IT DEVIATES PRETTY 2154 01:09:45,114 --> 01:09:46,148 SHARPLY FROM THE ORIGINAL TASK. 2155 01:09:46,215 --> 01:09:47,683 MAYBE A DIFFERENT STATISTICAL 2156 01:09:47,750 --> 01:09:48,884 TASK, A DIFFERENT MEASURE. 2157 01:09:48,951 --> 01:09:50,619 SO IF WE USE EVEN A KIND OF 2158 01:09:50,686 --> 01:09:51,520 GENEROUS DEFINITION OF BEING 2159 01:09:51,587 --> 01:09:55,891 CONSISTENT WITH THE ORIGINAL TAS 2160 01:09:55,958 --> 01:09:58,627 TASK, ONLY ABOUT 35% OF THE 2161 01:09:58,694 --> 01:09:59,295 ORIGINAL HYPOTHESES WERE FOUND 2162 01:09:59,361 --> 01:10:00,729 IN A WAY THAT WE JUDGE TO BE 2163 01:10:00,796 --> 01:10:01,864 CONSISTENT WITH THE ORIGINAL 2164 01:10:01,931 --> 01:10:02,264 REGISTRATION. 2165 01:10:02,331 --> 01:10:04,433 SO THIS NUMBER DOESN'T LOOK AS 2166 01:10:04,500 --> 01:10:04,967 GOOD. 2167 01:10:05,034 --> 01:10:06,201 THIS LOOKS LIKE MORE LIKE MAYBE 2168 01:10:06,268 --> 01:10:07,169 ONLY ABOUT A THIRD OF WHAT 2169 01:10:07,236 --> 01:10:08,704 PEOPLE SAID THEY WOULD DO ALMOST 2170 01:10:08,771 --> 01:10:10,039 A DECADE AGO IS ACTUALLY SHOWING 2171 01:10:10,105 --> 01:10:11,140 UP IN THE PUBLISHED LITERATURE. 2172 01:10:11,206 --> 01:10:12,575 THAT IS A LOT OF MISSING RESULTS 2173 01:10:12,641 --> 01:10:13,842 THAT WE MIGHT WANT TO SEE. 2174 01:10:13,909 --> 01:10:16,478 SO THIS IS, AGAIN, EARLY DATA, 2175 01:10:16,545 --> 01:10:17,513 WE'RE STILL PUTTING THIS 2176 01:10:17,580 --> 01:10:18,080 TOGETHER. 2177 01:10:18,147 --> 01:10:19,315 IT BUILDS ON SOME OF THE STUDIES 2178 01:10:19,381 --> 01:10:23,118 WE'VE BEEN HEARING WIT ABOUT, BT 2179 01:10:23,185 --> 01:10:24,286 WE'RE HOPING THIS WILL GIVE YOU 2180 01:10:24,353 --> 01:10:27,156 A MORE QUANTITATIVE TAKE ON 2181 01:10:27,222 --> 01:10:28,390 PUBLICATION BIAS AND SELECTIVE 2182 01:10:28,457 --> 01:10:29,491 REPORTING IN ECONOMICS DESPITE 2183 01:10:29,558 --> 01:10:31,694 ALL THE PROGRESS WE'VE MADE, SO 2184 01:10:31,760 --> 01:10:32,595 THERE'S A LOT WE STILL NEED TO 2185 01:10:32,661 --> 01:10:33,762 DO. 2186 01:10:33,829 --> 01:10:34,997 SO LET ME JUST WRAP UP WITH KIND 2187 01:10:35,064 --> 01:10:35,864 OF A WAY FORWARD. 2188 01:10:35,931 --> 01:10:37,032 AT THE CONFERENCE WE RAN ABOUT 2189 01:10:37,099 --> 01:10:38,467 FIVE YEARS AGO WHEN WE BROUGHT 2190 01:10:38,534 --> 01:10:39,602 TOGETHER ECONOMISTS, 2191 01:10:39,668 --> 01:10:40,502 PSYCHOLOGISTS, POLITICAL 2192 01:10:40,569 --> 01:10:41,437 SCIENTISTS AND OTHERS, WE TRIED 2193 01:10:41,503 --> 01:10:42,571 TO THINK THROUGH HOW TO IMPROVE 2194 01:10:42,638 --> 01:10:43,439 THIS KIND OF REPORTING. 2195 01:10:43,505 --> 01:10:44,640 WE WERE ALREADY CONCERNED THAT 2196 01:10:44,707 --> 01:10:45,941 THERE WOULDN'T BE GREAT 2197 01:10:46,008 --> 01:10:47,710 REPORTING FROM THE REGISTRY. 2198 01:10:47,776 --> 01:10:49,445 AND TRIED TO THINK THROUGH HOW 2199 01:10:49,511 --> 01:10:50,746 THE ECOSYSTEM FITS TOGETHER. 2200 01:10:50,813 --> 01:10:52,181 THAT'S WHY I'M PARTICULARLY 2201 01:10:52,247 --> 01:10:53,716 EXCITED ABOUT THIS MEETING, THE 2202 01:10:53,782 --> 01:10:54,683 DIFFERENT SESSIONS REALLY TALK 2203 01:10:54,750 --> 01:10:58,887 ABOUT DIFFERENT PIECES IN THE 2204 01:10:58,954 --> 01:10:59,521 PUZZLE. 2205 01:10:59,588 --> 01:11:00,889 PEOPLE START WITH A PROPOSAL, 2206 01:11:00,956 --> 01:11:02,291 THEY HAVE TO GET FUNDING, MAYBE 2207 01:11:02,358 --> 01:11:04,026 GO THROUGH IRB, CARRY OUT THE 2208 01:11:04,093 --> 01:11:06,228 DATA COLLECTION AND ANALYSIS, 2209 01:11:06,295 --> 01:11:10,265 AND IF YOU LACK LOOK AT THAT RE 2210 01:11:10,332 --> 01:11:12,501 ON TOP, THERE ARE A LOT OF 2211 01:11:12,568 --> 01:11:13,569 HYPOTHESES AND FINDINGS THAT 2212 01:11:13,636 --> 01:11:14,336 JUST DON'T GET REPORTED. 2213 01:11:14,403 --> 01:11:16,305 THAT'S THE PROBLEM, MAYBE TWO 2214 01:11:16,372 --> 01:11:17,640 THIRDS OF HYPOTHESES ARE 2215 01:11:17,706 --> 01:11:18,073 DISAPPEAR. 2216 01:11:18,140 --> 01:11:19,274 WHAT WE PROPOSED HERE WAS A 2217 01:11:19,341 --> 01:11:21,343 SYSTEM TO TRY TO STREAMLINE THE 2218 01:11:21,410 --> 01:11:23,646 REPORTING OF PREREGISTERED 2219 01:11:23,712 --> 01:11:25,080 HYPOTHESES BACK INTO THE PUBLIC 2220 01:11:25,147 --> 01:11:26,181 REGISTRY. 2221 01:11:26,248 --> 01:11:27,483 SO WE'VE ACTUALLY BEEN TALKING 2222 01:11:27,549 --> 01:11:31,920 TO FOLKS IN THE ASI A.A. REGISTY 2223 01:11:31,987 --> 01:11:33,288 ABOUT WAYS TO FACILITATE THAT. 2224 01:11:33,355 --> 01:11:34,623 OUR TEAM HAS BEEN DEVELOPING A 2225 01:11:34,690 --> 01:11:35,591 STREAMLINED TEMPLATE TO 2226 01:11:35,658 --> 01:11:36,692 CREATE -- IT'S NOT QUITE A PAPER 2227 01:11:36,759 --> 01:11:38,260 BUT JUST TO REPORT THOSE 2228 01:11:38,327 --> 01:11:39,695 HYPOTHESES IN A STANDARDIZED 2229 01:11:39,762 --> 01:11:42,097 STREAMLINED WAY SO THEY APPEAR 2230 01:11:42,164 --> 01:11:44,299 SOMEWHERE ON THE REGISTRY. 2231 01:11:44,366 --> 01:11:46,168 THAT'S THE ORANGE LINE THERE. 2232 01:11:46,235 --> 01:11:47,369 SO WE SEE THAT AS EFFICIENT 2233 01:11:47,436 --> 01:11:49,104 REPORTING TO REALLY REDUCE THE 2234 01:11:49,171 --> 01:11:50,539 KOTION TO RESEARCHERS OF DOING 2235 01:11:50,606 --> 01:11:52,274 IT, BUT JUST TO MAKE SURE 2236 01:11:52,341 --> 01:11:53,475 RESULTS GET SHARED WITH A 2237 01:11:53,542 --> 01:11:54,910 SCIENTIFIC COMMUNITY, BUT 2238 01:11:54,977 --> 01:11:55,944 JOURNALS AND FUNDING AGENCIES 2239 01:11:56,011 --> 01:11:56,879 AND OTHERS CAN PLAY A ROLE 2240 01:11:56,945 --> 01:11:58,747 BECAUSE THEY CAN ASK TO SEE YOUR 2241 01:11:58,814 --> 01:11:59,581 COMPLETE REPORTING BEFORE THEY 2242 01:11:59,648 --> 01:12:00,616 GIVE YOU FUNDING FOR THE NEXT 2243 01:12:00,683 --> 01:12:01,383 ROUND. 2244 01:12:01,450 --> 01:12:03,118 THEY COULD ASK TO SEE YOUR 2245 01:12:03,185 --> 01:12:04,653 COMPLETE REPORTING OF THAT TRIAL 2246 01:12:04,720 --> 01:12:06,755 BEFORE THEY PUBLISH YOUR PAPER. 2247 01:12:06,822 --> 01:12:08,157 AND AGAIN, IF IN IS DESIGNED IN 2248 01:12:08,223 --> 01:12:10,559 A WAY THAT ISN'T TOO ONEROUS, IT 2249 01:12:10,626 --> 01:12:13,128 CAN GIVE INCENTIVES TO PEOPLE TO 2250 01:12:13,195 --> 01:12:14,430 REPORT THEIR RESULTS, AND THEN 2251 01:12:14,496 --> 01:12:16,865 ALLOW OTHERS TO REPRODUCE, 2252 01:12:16,932 --> 01:12:19,168 REPLICATE AND KIND OF EXTEND THE 2253 01:12:19,234 --> 01:12:19,635 FINDINGS. 2254 01:12:19,702 --> 01:12:21,270 SO THIS WAS OUR PROPOSAL, WHICH 2255 01:12:21,336 --> 01:12:23,338 AGAIN THERE MAY BE SOME 2256 01:12:23,405 --> 01:12:24,339 PARALLELS TO IN BIOMEDICAL 2257 01:12:24,406 --> 01:12:25,040 RESEARCH. 2258 01:12:25,107 --> 01:12:25,741 SO LET ME JUST WRAP UP THERE 2259 01:12:25,808 --> 01:12:28,143 BECAUSE I THINK I'M ABOUT AT 2260 01:12:28,210 --> 01:12:32,014 TIME AND THEN I'M REALLY EXCITED 2261 01:12:32,081 --> 01:12:34,083 TO DISCUSS FURTHER AND HAVE 2262 01:12:34,149 --> 01:12:34,717 QUESTIONS. 2263 01:12:34,783 --> 01:12:36,585 AGAIN, I'M VERY HAPPY TO BE WITH 2264 01:12:36,652 --> 01:12:37,152 THIS GROUP. 2265 01:12:37,219 --> 01:12:39,354 I THINK OUR PERSPECTIVES WILL 2266 01:12:39,421 --> 01:12:40,456 DIFFER BETWEEN THE SOCIAL 2267 01:12:40,522 --> 01:12:41,757 SCIENCES AND BIOMEDICAL 2268 01:12:41,824 --> 01:12:43,092 RESEARCH, BUT AT THE SAME TIME, 2269 01:12:43,158 --> 01:12:44,326 THERE'S ACTUALLY, I THINK, A LOT 2270 01:12:44,393 --> 01:12:45,828 OF POINTS OF CONTACT AND I'VE 2271 01:12:45,894 --> 01:12:47,896 CERTAINLY ALWAYS LEARNED A LOT 2272 01:12:47,963 --> 01:12:48,464 ACROSS FIELDS. 2273 01:12:48,530 --> 01:12:49,665 I WANT TO COME BACK TO SOMETHING 2274 01:12:49,732 --> 01:12:50,332 THAT SHAI MENTIONED. 2275 01:12:50,399 --> 01:12:51,734 THIS IS SOMETHING I'M A BIG 2276 01:12:51,800 --> 01:12:54,369 BELIEVER IN, WHICH IS MORE THAN 2277 01:12:54,436 --> 01:12:55,804 JUST IMPLEMENTING A CERTAIN 2278 01:12:55,871 --> 01:12:56,739 REFORM OR A CERTAIN POLICY 2279 01:12:56,805 --> 01:12:57,039 CHANGE. 2280 01:12:57,106 --> 01:12:58,474 THOSE ARE IMPORTANT. 2281 01:12:58,540 --> 01:13:00,743 WE REALLY HAVE TO CONTINUALLY 2282 01:13:00,809 --> 01:13:05,547 STRIVE FOR CULTURAL CHANGE IN 2283 01:13:05,614 --> 01:13:05,981 OUR FIELDS. 2284 01:13:06,048 --> 01:13:07,549 SO IT KIND OF SHOCKS ME, EVEN 2285 01:13:07,616 --> 01:13:10,919 WHEN I'M ADVISING AN UNDERGRAD 2286 01:13:10,986 --> 01:13:13,655 THESIS PROJECT, UNDERGRADS HAVE 2287 01:13:13,722 --> 01:13:14,990 ALREADY INTERNALIZED A LOT OF 2288 01:13:15,057 --> 01:13:17,826 NORMS AND IDEAS THAT MAY NOT BE 2289 01:13:17,893 --> 01:13:18,827 THE BEST. 2290 01:13:18,894 --> 01:13:20,162 SO I'M SUPERVISING EVEN THIS 2291 01:13:20,229 --> 01:13:22,231 YEAR A SENIOR HONORS THESIS AT 2292 01:13:22,297 --> 01:13:23,532 BERKELEY, AND STUDENT -- SHE'S 2293 01:13:23,599 --> 01:13:24,833 RUNNING HER ANALYSIS. 2294 01:13:24,900 --> 01:13:25,734 IT'S A REALLY INTERESTING 2295 01:13:25,801 --> 01:13:26,935 PROJECT TRYING TO UNDERSTAND 2296 01:13:27,002 --> 01:13:29,037 CRIME IN BRAZIL AND HOW ECONOMIC 2297 01:13:29,104 --> 01:13:30,539 CONDITIONS MAY BE CONTRIBUTING 2298 01:13:30,606 --> 01:13:33,308 TO HIGH CRIME RATES IN CERTAIN 2299 01:13:33,375 --> 01:13:34,510 AREAS, AND SHE PRESENTS HER 2300 01:13:34,576 --> 01:13:36,812 RESULTS TO ME AND SAYS, YOU 2301 01:13:36,879 --> 01:13:40,649 KNOW, THIS FINDING ISN'T QUITE 2302 01:13:40,716 --> 01:13:42,184 SIGNIFICANT AT 95% CONFIDENCE 2303 01:13:42,251 --> 01:13:43,452 AND ALREADY SHE'S LIKE TORMENTED 2304 01:13:43,519 --> 01:13:43,852 BY THIS. 2305 01:13:43,919 --> 01:13:45,420 AND I'M LIKE, WHERE DID THIS 2306 01:13:45,487 --> 01:13:46,155 COME FROM? 2307 01:13:46,221 --> 01:13:49,324 HOW IS THIS SUCH A STRONG, YOU 2308 01:13:49,391 --> 01:13:50,325 KNOW, FEELING AMONG A 2309 01:13:50,392 --> 01:13:51,960 20-YEAR-OLD WHO'S JUST DOING HER 2310 01:13:52,027 --> 01:13:53,629 FIRST RESEARCH PROJECT? 2311 01:13:53,695 --> 01:13:54,963 BUT THOSE FEELINGS ARE OUT 2312 01:13:55,030 --> 01:13:56,932 THERE, AND I THINK IN ALL OF OUR 2313 01:13:56,999 --> 01:13:58,901 WORK, WE HAVE TO ALMOST LIKE 2314 01:13:58,967 --> 01:14:03,405 DE-PROGRAM OUR STUDENTS, OUR 2315 01:14:03,472 --> 01:14:04,673 ADVISEES, FOLKS IN OUR CLASSES 2316 01:14:04,740 --> 01:14:06,275 TO EMPHASIZE THE IMPORTANCE OF 2317 01:14:06,341 --> 01:14:09,578 THE QUESTION, STRENGTH OF THE 2318 01:14:09,645 --> 01:14:12,514 DESIGN, STATISTICAL POWER, THE 2319 01:14:12,581 --> 01:14:18,420 STATISTICAL RIGOR AS WELL, AND 2320 01:14:18,487 --> 01:14:19,621 REALLY THE QUALITY OF THE DATA 2321 01:14:19,688 --> 01:14:21,056 RATHER THAN THE SIGN OF THE 2322 01:14:21,123 --> 01:14:22,191 FINDING, RATHER THAN THE 2323 01:14:22,257 --> 01:14:24,827 STATISTICAL SIGNIFICANCE. 2324 01:14:24,893 --> 01:14:26,662 THOSE THINGS HAVE TO BE 2325 01:14:26,728 --> 01:14:27,329 REPORTED, THEY'RE GOING TO 2326 01:14:27,396 --> 01:14:28,564 MATTER FOR INTERPRETATION, BUT 2327 01:14:28,630 --> 01:14:29,765 THE QUALITY OF THE STUDY 2328 01:14:29,832 --> 01:14:32,067 SHOULDN'T LIVE OR DIE BASED ON 2329 01:14:32,134 --> 01:14:32,601 THOSE THINGS. 2330 01:14:32,668 --> 01:14:34,036 SO I THINK WE ALL HAVE A LOT OF 2331 01:14:34,102 --> 01:14:35,370 WORK TO DO. 2332 01:14:35,437 --> 01:14:37,239 AT BITSS, THE BERKELEY 2333 01:14:37,306 --> 01:14:39,474 INITIATIVE FOR TRANSPARENCY IN 2334 01:14:39,541 --> 01:14:41,176 THE SOCIAL SCIENCES, FROM THE 2335 01:14:41,243 --> 01:14:42,177 BEGINNING WHAT WE'VE DONE IS 2336 01:14:42,244 --> 01:14:43,478 CARRY OUT TRAINING COURSES FOR 2337 01:14:43,545 --> 01:14:44,913 GRADUATE STUDENTS AND POSTDOCS 2338 01:14:44,980 --> 01:14:47,983 IN THE SOCIAL SCIENCES TO TEACH 2339 01:14:48,050 --> 01:14:49,318 THEM THESE VALUES, THE 2340 01:14:49,384 --> 01:14:50,619 SCIENTIFIC ETHOS AND THE TOOLS 2341 01:14:50,686 --> 01:14:52,321 THEY CAN USE TO DO MORE 2342 01:14:52,387 --> 01:14:53,589 REPRODUCIBLE SOCIAL SCIENCE AND 2343 01:14:53,655 --> 01:14:54,690 OUR HOPE IS THERE WILL JUST BE 2344 01:14:54,756 --> 01:14:56,792 MORE AND MORE YOUNG CADRES OF 2345 01:14:56,859 --> 01:14:58,994 SCHOLARS WHO SEE SCIENCE, YOU 2346 01:14:59,061 --> 01:15:00,829 KNOW, IN THIS TRADITIONAL WAY, 2347 01:15:00,896 --> 01:15:02,097 YOU KNOW, THAT'S REALLY 2348 01:15:02,164 --> 01:15:04,900 CONSISTENT WITH THE SCIENTIFIC 2349 01:15:04,967 --> 01:15:06,001 ETHOS RATHER THAN ALWAYS 2350 01:15:06,068 --> 01:15:07,636 OBSESSING OVER SIGNIFICANCE AND 2351 01:15:07,703 --> 01:15:11,073 NOT THE QUALITY OF THE SCIENCE. 2352 01:15:11,139 --> 01:15:12,474 SO LET ME JUST LEAVE IT THERE 2353 01:15:12,541 --> 01:15:13,475 AND I LOOK FORWARD TO ANY 2354 01:15:13,542 --> 01:15:13,842 QUESTIONS. 2355 01:15:13,909 --> 01:15:24,086 [APPLAUSE] 2356 01:15:30,359 --> 01:15:37,032 >> HOLD IT, THE MIC, THE MIC. 2357 01:15:37,099 --> 01:15:38,533 >> THE MOST CONTROVERSIAL PART, 2358 01:15:38,600 --> 01:15:40,335 WHICH IS OBSERVATIONAL STUDIES, 2359 01:15:40,402 --> 01:15:44,172 AND WHEN DO YOU REGISTER, SO 2360 01:15:44,239 --> 01:15:46,241 COULD YOU KIND OF TELL US ABOUT 2361 01:15:46,308 --> 01:15:49,878 THE CONTROVERSY AND HOW YOU FALL 2362 01:15:49,945 --> 01:15:53,282 IN THAT RANGE OF POSSIBLE 2363 01:15:53,348 --> 01:15:53,548 SOLUTIONS? 2364 01:15:53,615 --> 01:15:55,250 >> SO IN ECONOMICS AND POLITICAL 2365 01:15:55,317 --> 01:15:57,653 SCIENCE, THERE'S BEEN A LOT OF 2366 01:15:57,719 --> 01:15:59,187 PUSHBACK TO REGISTERING 2367 01:15:59,254 --> 01:16:00,522 OBSERVATIONAL STUDIES. 2368 01:16:00,589 --> 01:16:01,723 WHAT WE'VE DONE IN SOME OF OUR 2369 01:16:01,790 --> 01:16:02,925 OWN OBSERVATIONAL RESEARCH AND 2370 01:16:02,991 --> 01:16:05,661 OTHERS HAVE DONE IS START 2371 01:16:05,727 --> 01:16:08,196 REGISTERING ANALYTICAL DECISIONS 2372 01:16:08,263 --> 01:16:09,498 THAT WE'RE MAKING ALONG THE WAY. 2373 01:16:09,564 --> 01:16:10,832 I THINK THE CONCERN THAT SOME 2374 01:16:10,899 --> 01:16:12,367 PEOPLE HAVE, AGAIN, IS THAT DID 2375 01:16:12,434 --> 01:16:13,368 THEY LOOK AT THE DATA FIRST 2376 01:16:13,435 --> 01:16:14,469 BEFORE THEY MADE THOSE CHOICES? 2377 01:16:14,536 --> 01:16:16,004 IT'S REALLY HARD TO RULE THOSE 2378 01:16:16,071 --> 01:16:16,338 THINGS OUT. 2379 01:16:16,405 --> 01:16:18,740 WHAT WE'VE DONE AS A TEAM, I'M 2380 01:16:18,807 --> 01:16:20,409 WORKING ON A TEAM NOW WITH FIVE 2381 01:16:20,475 --> 01:16:23,278 CO-AUTHORS, WE ALL BASICALLY 2382 01:16:23,345 --> 01:16:24,579 CREATE A PREANALYSIS PLAN AND 2383 01:16:24,646 --> 01:16:25,714 SIGN OFF TO SAY LOOK, THIS IS 2384 01:16:25,781 --> 01:16:26,949 THE DATA WE'VE LOOKED AT, THIS 2385 01:16:27,015 --> 01:16:29,184 IS THE DATA WE HAVEN'T LOOKED AT 2386 01:16:29,251 --> 01:16:30,285 YET, THESE ARE THE ASSUMPTIONS 2387 01:16:30,352 --> 01:16:31,520 WE'RE GOING TO MAKE IN OUR 2388 01:16:31,586 --> 01:16:32,955 MODELING, AND WE REGISTER THAT. 2389 01:16:33,021 --> 01:16:35,891 PEOPLE CAN USE THAT AS THEY 2390 01:16:35,958 --> 01:16:37,993 WILL, BUT WE'RE TRYING TO CHANGE 2391 01:16:38,060 --> 01:16:39,428 NORMS THERE AS WELL. 2392 01:16:39,494 --> 01:16:40,429 THERE'S WHOLE FIELDS OF 2393 01:16:40,495 --> 01:16:42,397 ECONOMICS, ESPECIALLY LIKE MACRO 2394 01:16:42,464 --> 01:16:43,532 ECONOMICS, WHICH USES A LOT OF 2395 01:16:43,598 --> 01:16:45,233 BIG PUBLICLY AVAILABLE AGGREGATE 2396 01:16:45,300 --> 01:16:47,436 DATA WHERE THERE HASN'T BEEN ANY 2397 01:16:47,502 --> 01:16:49,104 PRE ANALYSIS PLANS AT ALL. 2398 01:16:49,171 --> 01:16:50,305 AND THAT'S A FIELD THAT WE'RE 2399 01:16:50,372 --> 01:16:51,306 TRYING TO PUSH FOR CHANGE. 2400 01:16:51,373 --> 01:16:54,476 SO I'M WORKING ON A DEVELOPMENT 2401 01:16:54,543 --> 01:16:56,678 MACRO PROJECT WHERE WE DID 2402 01:16:56,745 --> 01:16:58,113 REGISTER A PREANALYSIS PLAN AND 2403 01:16:58,180 --> 01:16:59,047 ACTUALLY SOME MACRO ECONOMISTS 2404 01:16:59,114 --> 01:17:00,015 ARE STARTING TO THINK ABOUT HOW 2405 01:17:00,082 --> 01:17:00,849 THEY CAN DO IT AS WELL. 2406 01:17:00,916 --> 01:17:02,084 SO I DON'T KNOW HOW QUICK CHANGE 2407 01:17:02,150 --> 01:17:03,618 WILL BE THERE, BUT THE CONCERN 2408 01:17:03,685 --> 01:17:04,720 IN THE FIELD IS PEOPLE ARE GOING 2409 01:17:04,786 --> 01:17:06,688 TO KIND OF LOOK AT THE DATA, 2410 01:17:06,755 --> 01:17:08,023 MANIPULATE THINGS AND THEN GO 2411 01:17:08,090 --> 01:17:11,660 BACK AND WRITE A PLAN AS IF THEY 2412 01:17:11,727 --> 01:17:13,195 HAD -- PRETENDING THEY DIDN'T 2413 01:17:13,261 --> 01:17:13,729 LOOK AT THE DATA. 2414 01:17:13,795 --> 01:17:15,030 I THINK THAT'S THE KIND OF 2415 01:17:15,097 --> 01:17:18,133 ETHICAL -- YOU KNOW, ETHICAL 2416 01:17:18,200 --> 01:17:18,567 CONCERN. 2417 01:17:18,633 --> 01:17:20,002 THERE'S STILL MANY OTHER OPEN 2418 01:17:20,068 --> 01:17:21,203 SCIENCE PRACTICES THAT FOLKS 2419 01:17:21,269 --> 01:17:22,537 DOING OBSERVATIONAL 2420 01:17:22,604 --> 01:17:23,138 NON-EXPERIMENTAL RESEARCH CAN 2421 01:17:23,205 --> 01:17:23,405 DO. 2422 01:17:23,472 --> 01:17:25,374 THEY CAN SHARE THEIR DATA, YOU 2423 01:17:25,440 --> 01:17:26,475 KNOW, FOR INSTANCE, THEY CAN 2424 01:17:26,541 --> 01:17:27,843 HELP OTHERS REPLICATE THEIR 2425 01:17:27,909 --> 01:17:29,478 RESULTS, SO I DO THINK THERE ARE 2426 01:17:29,544 --> 01:17:30,145 WAYS FORWARD. 2427 01:17:30,212 --> 01:17:31,480 BUT THAT'S BEEN A BIG STICKING 2428 01:17:31,546 --> 01:17:32,447 POINT FOR US, AND AGAIN, THERE 2429 01:17:32,514 --> 01:17:34,016 MAY BE SOME PARALLELS WITH 2430 01:17:34,082 --> 01:17:35,851 LABORATORY OR PRE-CLINICAL 2431 01:17:35,917 --> 01:17:39,488 RESEARCH RELATIVE TO TRIALS AND 2432 01:17:39,554 --> 01:17:40,856 BIOMEDICAL -- 2433 01:17:40,922 --> 01:17:44,192 >> HI. 2434 01:17:44,259 --> 01:17:48,997 -- BERNARD, THANK YOU FOR THIS 2435 01:17:49,064 --> 01:17:49,431 FASCINATING TALK. 2436 01:17:49,498 --> 01:17:51,233 WE WOULD LOVE TO HEAR HOW IN 2437 01:17:51,299 --> 01:17:55,804 YOUR COMMUNITY, YOU MANAGE THESE 2438 01:17:55,871 --> 01:17:58,974 CONSCIOUSNESS IN ADOPTING THIS 2439 01:17:59,041 --> 01:18:01,943 FORMAT, AND PRODUCING REGISTERED 2440 01:18:02,010 --> 01:18:03,712 REPORTS AND DATA SHARING AND 2441 01:18:03,779 --> 01:18:05,147 WHERE IT CAME FROM, WHETHER 2442 01:18:05,213 --> 01:18:06,348 THERE WAS AN INCENTIVE OR 2443 01:18:06,415 --> 01:18:08,083 ANYTHING. 2444 01:18:08,150 --> 01:18:12,287 BUT MY QUESTION IS A BIT TRICKY. 2445 01:18:12,354 --> 01:18:17,325 IT'S ABOUT NULL RESULT. 2446 01:18:17,392 --> 01:18:18,293 I DISAPPROVE THE USE OF THIS 2447 01:18:18,360 --> 01:18:18,693 TERM. 2448 01:18:18,760 --> 01:18:20,128 BECAUSE THIS HAS A VERY STRONG 2449 01:18:20,195 --> 01:18:24,933 MEANING. 2450 01:18:25,000 --> 01:18:28,537 SO NOT FINDING STATISTICAL 2451 01:18:28,603 --> 01:18:30,906 SIGNIFICANCE IN NEARLY 80% OF 2452 01:18:30,972 --> 01:18:33,508 THE PAPERS, I CAN TAKE ANY PAPER 2453 01:18:33,575 --> 01:18:34,810 PUBLISHED YESTERDAY IN NATURE 2454 01:18:34,876 --> 01:18:36,511 AND SCIENCE AND I WILL SHOW YOU 2455 01:18:36,578 --> 01:18:39,047 THAT THEY AFFIRM THE NULL, 2456 01:18:39,114 --> 01:18:40,882 AFFIRMING THE NULL IS A COMMON 2457 01:18:40,949 --> 01:18:41,783 MISTAKE IN STATISTICS. 2458 01:18:41,850 --> 01:18:43,085 NOW TO AFFIRM THE NULL, YOU HAVE 2459 01:18:43,151 --> 01:18:44,753 TO DO A DIFFERENT TYPE OF 2460 01:18:44,820 --> 01:18:45,420 ANALYSIS. 2461 01:18:45,487 --> 01:18:47,389 SO NOT FINDING STATISTICAL 2462 01:18:47,456 --> 01:18:48,356 SIGNIFICANCE, WHICH WE WANT TO 2463 01:18:48,423 --> 01:18:51,660 BE REPORTED, HAS NOTHING TO DO 2464 01:18:51,726 --> 01:18:52,661 WITH AFFIRMING THE NULL, BECAUSE 2465 01:18:52,727 --> 01:18:55,597 YOU NEED TO DO OTHER EXPERIMENTS 2466 01:18:55,664 --> 01:18:56,264 FROM THE NULL. 2467 01:18:56,331 --> 01:18:57,466 SO DID YOU SOLVE THIS PROBLEM IN 2468 01:18:57,532 --> 01:18:59,134 YOUR FIELD? 2469 01:18:59,201 --> 01:19:02,437 BECAUSE THAT'S PERHAPS THE MOST 2470 01:19:02,504 --> 01:19:05,073 COMMON MISTAKE OF MISUSE OF 2471 01:19:05,140 --> 01:19:09,444 STATISTICS, AT LEAST IN 2472 01:19:09,511 --> 01:19:09,778 NEUROSCIENCE. 2473 01:19:09,845 --> 01:19:10,912 >> SO THERE WERE TWO PARTS 2474 01:19:10,979 --> 01:19:11,746 THERE. 2475 01:19:11,813 --> 01:19:13,648 THE FIRST WAS YOU MENTIONED HOW 2476 01:19:13,715 --> 01:19:14,816 DID WE GAIN PROGRESS IN OUR 2477 01:19:14,883 --> 01:19:15,150 COMMUNITY. 2478 01:19:15,217 --> 01:19:17,052 LET ME SAY A WORD ABOUT THAT AND 2479 01:19:17,119 --> 01:19:19,154 THEN GET TO THE KINDS OF ISSUES 2480 01:19:19,221 --> 01:19:20,522 OF THE TYPES OF HYPOTHESIS 2481 01:19:20,589 --> 01:19:21,690 TESTING THAT WE'RE DOING. 2482 01:19:21,756 --> 01:19:22,858 I THINK ECONOMICS IS AN 2483 01:19:22,924 --> 01:19:23,758 INTERESTING CASE. 2484 01:19:23,825 --> 01:19:25,827 POLITICAL SCIENCE HAS SOME 2485 01:19:25,894 --> 01:19:26,928 PARALLELS, THEY'RE FIELDS THAT 2486 01:19:26,995 --> 01:19:28,029 OFTEN COMMUNICATE CLOSELY, THOSE 2487 01:19:28,096 --> 01:19:29,131 TWO FIELDS. 2488 01:19:29,197 --> 01:19:30,332 THE EARLY PROGRESS ACTUALLY IN 2489 01:19:30,398 --> 01:19:31,933 ECONOMICS WAS MADE BY THE 2490 01:19:32,000 --> 01:19:32,534 AMERICAN ECONOMIC ASSOCIATION. 2491 01:19:32,601 --> 01:19:35,737 SO THE BIG KIND OF SOCIETY HAD 2492 01:19:35,804 --> 01:19:37,139 LEADERSHIP THAT EVEN 20 YEARS 2493 01:19:37,205 --> 01:19:39,674 AGO PUSHED FOR SHARING OF DATA 2494 01:19:39,741 --> 01:19:40,909 AND CODE. 2495 01:19:40,976 --> 01:19:43,211 SO IN 2005, THE AEA IMPLEMENTED 2496 01:19:43,278 --> 01:19:45,080 THIS POLICY, ACTUALLY SOMEONE 2497 01:19:45,147 --> 01:19:46,181 VERY FAMOUS WAS THE PRESIDENT OF 2498 01:19:46,248 --> 01:19:48,283 THE AEA, BEN BERNANKE, WHO ENDED 2499 01:19:48,350 --> 01:19:52,988 UP BEING, YOU KNOW, THE HEAD OF 2500 01:19:53,054 --> 01:19:55,657 THE FEDERAL RESERVE WAS A 2501 01:19:55,724 --> 01:19:56,658 PROFESSOR AT PRINCE TOLL AT THAT 2502 01:19:56,725 --> 01:19:56,958 TIME. 2503 01:19:57,025 --> 01:19:58,160 THERE WERE A COUPLE OF 2504 01:19:58,226 --> 01:20:00,262 CONTROVERSIES OF 2505 01:20:00,328 --> 01:20:01,062 NON-REPLICABILITY AND YOU KNOW 2506 01:20:01,129 --> 01:20:02,063 DATA NOT BEING SHARED. 2507 01:20:02,130 --> 01:20:06,535 HE SAID ALL AEA JOURNALS WILL 2508 01:20:06,601 --> 01:20:08,003 MANDATE DATA ON CODE SHARING NOW 2509 01:20:08,069 --> 01:20:09,304 IT ACTUALLY ENDED UP TAKING 2510 01:20:09,371 --> 01:20:10,939 SEVERAL YEARS TO TRULY BE 2511 01:20:11,006 --> 01:20:13,241 IMPLEMENTED BUT IT SET A 2512 01:20:13,308 --> 01:20:15,343 STANDARD AND OTHER JOURNALS 2513 01:20:15,410 --> 01:20:17,646 FOLLOWED AND TRULY IT'S ALMOST 2514 01:20:17,712 --> 01:20:19,714 UNIVERSAL NOW. 2515 01:20:19,781 --> 01:20:21,016 AT THE SAME TIME, YOU MENTIONED 2516 01:20:21,082 --> 01:20:21,816 REGISTERED REPORTS REALLY CAME 2517 01:20:21,883 --> 01:20:23,818 FROM THE COMMUNITY SO OUR GROUP 2518 01:20:23,885 --> 01:20:26,488 AT BITSS LEARNED ABOUT 2519 01:20:26,555 --> 01:20:28,323 REGISTERED REPORTS IN OTHER 2520 01:20:28,390 --> 01:20:29,090 FIELDS, THERE WAS NONE IN 2521 01:20:29,157 --> 01:20:30,192 ECONOMICS OR POLITICAL SCIENCE, 2522 01:20:30,258 --> 01:20:31,560 AND WE BROUGHT THE IDEA TO MY 2523 01:20:31,626 --> 01:20:33,628 OWN FIELD'S FIELD JOURNAL, THE 2524 01:20:33,695 --> 01:20:34,729 JOURNAL OF DEVELOPMENTAL 2525 01:20:34,796 --> 01:20:35,030 ECONOMICS. 2526 01:20:35,096 --> 01:20:36,231 THE EDITORS WERE COLLEAGUES AND 2527 01:20:36,298 --> 01:20:37,832 FRIENDS OF MINE, AND WE 2528 01:20:37,899 --> 01:20:40,001 CONVINCED THEM, TRY A PILOT WITH 2529 01:20:40,068 --> 01:20:40,669 REGISTERED REPORTS. 2530 01:20:40,735 --> 01:20:42,137 WE'LL SUPPORT YOU WITH STAFFING 2531 01:20:42,204 --> 01:20:43,772 FROM OUR INITIATIVE. 2532 01:20:43,838 --> 01:20:45,106 AND THEY DID IT AND THERE WAS A 2533 01:20:45,173 --> 01:20:48,176 LOT OF INTEREST IN REGISTERED 2534 01:20:48,243 --> 01:20:48,777 REPORTS. 2535 01:20:48,843 --> 01:20:49,744 LOTS OF SUBMISSIONS. 2536 01:20:49,811 --> 01:20:50,745 YOUNG SCHOLARS IN PARTICULAR 2537 01:20:50,812 --> 01:20:52,280 WERE EXCITED ABOUT THIS NEW 2538 01:20:52,347 --> 01:20:53,715 FORMAT, SO THEN THEY 2539 01:20:53,782 --> 01:20:55,250 INSTITUTIONALIZED IT AND OTHER 2540 01:20:55,317 --> 01:20:56,918 ECON JOURNALS HAVE BEEN COPYING. 2541 01:20:56,985 --> 01:20:58,186 SO I THINK IT'S BEEN A 2542 01:20:58,253 --> 01:21:00,355 COMBINATION OF TOP-DOWN SOCIETY 2543 01:21:00,422 --> 01:21:04,392 WORK, BUT THEN ACTUALLY BOTTOM 2544 01:21:04,459 --> 01:21:05,060 UP SCHOLARLY WORK. 2545 01:21:05,126 --> 01:21:07,696 ON THE SECOND POINT, THERE ISN'T 2546 01:21:07,762 --> 01:21:09,064 MUCH TO SAY BECAUSE WE CONTINUE 2547 01:21:09,130 --> 01:21:10,999 TO DO TRADITIONAL HYPOTHESIS 2548 01:21:11,066 --> 01:21:14,169 TESTING AND SO THE KIND OF 2549 01:21:14,236 --> 01:21:15,370 DIFFERENT SORTS OF TESTS TO 2550 01:21:15,437 --> 01:21:17,472 AFFIRM THE NULL AND REALLY FOCUS 2551 01:21:17,539 --> 01:21:19,140 ON IS IT A ZERO AND WHAT TEST DO 2552 01:21:19,207 --> 01:21:20,342 WE WANT TO DO AROUND 2553 01:21:20,408 --> 01:21:21,343 UNDERSTANDING HOW PRECISE THOSE 2554 01:21:21,409 --> 01:21:25,213 ESTIMATES ARE, ET CETERA, THERE 2555 01:21:25,280 --> 01:21:25,880 ISN'T SO MUCH PROGRESS. 2556 01:21:25,947 --> 01:21:27,515 SO I THINK THAT'S AN AREA WHERE 2557 01:21:27,582 --> 01:21:28,917 MORE WORK CAN BE DONE AND MAYBE 2558 01:21:28,984 --> 01:21:30,185 THERE'S PARALLELS AGAIN BETWEEN 2559 01:21:30,252 --> 01:21:31,152 OUR FIELDS OR NON-PROGRESS 2560 01:21:31,219 --> 01:21:34,689 BETWEEN OUR FIELDS. 2561 01:21:34,756 --> 01:21:38,827 >> HI. 2562 01:21:38,893 --> 01:21:40,562 -- TANZA, UNIVERSITY OF 2563 01:21:40,629 --> 01:21:44,432 FLORIDA. 2564 01:21:44,499 --> 01:21:47,936 MALU TANSY, UNIVERSITY OF 2565 01:21:48,003 --> 01:21:49,904 FLORIDA, OF NATURE PARTNER 2566 01:21:49,971 --> 01:21:50,972 JOURNAL, PARKINSON'S DISEASE. 2567 01:21:51,039 --> 01:21:51,806 A COUPLE OF THINGS ON THE 2568 01:21:51,873 --> 01:21:54,109 FUNDERS. 2569 01:21:54,175 --> 01:21:55,977 SOME OF THE FUNDERS LIKE MICHAEL 2570 01:21:56,044 --> 01:21:57,646 J. FOX FOUNDATION, ALZHEIMER'S 2571 01:21:57,712 --> 01:22:00,248 ASSOCIATION IN OUR BIOMEDICAL 2572 01:22:00,315 --> 01:22:02,584 FIELD, ARE NOW REQUIRING OPEN 2573 01:22:02,651 --> 01:22:03,685 SCIENCE. 2574 01:22:03,752 --> 01:22:05,587 SO IT IS PRETTY ONEROUS, THOUGH. 2575 01:22:05,654 --> 01:22:09,124 YOU HAVE TO GIVE OUR IDs AND 2576 01:22:09,190 --> 01:22:10,925 YOU HAVE TO PUT EVERYTHING IN 2577 01:22:10,992 --> 01:22:14,296 CERTAIN WORK SPACES, SO NODO, ET 2578 01:22:14,362 --> 01:22:15,697 CETERA. 2579 01:22:15,764 --> 01:22:19,501 IT'S A LOT OF LOT OF LOT OF WORK 2580 01:22:19,567 --> 01:22:21,569 FOR THE FIRST AUTHOR, 2581 01:22:21,636 --> 01:22:22,370 CORRESPONDING AUTHOR. 2582 01:22:22,437 --> 01:22:26,207 BUT I DO THINK THAT THE ONLY WAY 2583 01:22:26,274 --> 01:22:28,410 TO GET CREDIT FOR THAT 2584 01:22:28,476 --> 01:22:30,378 PUBLICATION BY THE FUNDER IS TO 2585 01:22:30,445 --> 01:22:32,514 BE IN COMPLIANCE, SO THEY RUN 2586 01:22:32,580 --> 01:22:34,983 COMPLIANCE REPORTS ON THAT. 2587 01:22:35,050 --> 01:22:37,419 AND THIS IS PART OF THE ALIGNING 2588 01:22:37,485 --> 01:22:42,590 SCIENCE ACROSS PARKINSON'S ASAP, 2589 01:22:42,657 --> 01:22:43,892 MICHAEL J. FOX. 2590 01:22:43,958 --> 01:22:45,427 SO AS A P.I., IT HAS GIVEN US A 2591 01:22:45,493 --> 01:22:46,428 LOT OF HEADACHES BUT I THINK IN 2592 01:22:46,494 --> 01:22:47,762 THE END, THIS IS GOING TO 2593 01:22:47,829 --> 01:22:51,299 GUARANTEE THAT YOU DO HAVE OPEN 2594 01:22:51,366 --> 01:22:52,834 SCIENCE AND YOU HAVE NO OTHER 2595 01:22:52,901 --> 01:22:53,935 CHOICE BUT TO DO IT OR YOU WILL 2596 01:22:54,002 --> 01:22:57,672 NOT GET FUNDING, YOU KNOW, 2597 01:22:57,739 --> 01:22:59,207 RETROSPECTIVELY TO YOUR 2598 01:22:59,274 --> 01:23:00,975 INSTITUTION AND PROSPECTIVELY. 2599 01:23:01,042 --> 01:23:02,410 WHAT I WAS GOING TO SAY IS IN 2600 01:23:02,477 --> 01:23:04,379 TERMS OF THE RIGOR AND 2601 01:23:04,446 --> 01:23:05,347 REPRODUCIBILITY COURSES THAT OUR 2602 01:23:05,413 --> 01:23:07,949 STUDENTS HAVE TO TAKE, THIS 2603 01:23:08,016 --> 01:23:08,950 SHOULD BE A PART OF IT. 2604 01:23:09,017 --> 01:23:10,418 THERE SHOULD BE A MODULE ON 2605 01:23:10,485 --> 01:23:13,288 FIRST OF ALL JUST HOW TO PROPOSE 2606 01:23:13,355 --> 01:23:13,688 A HYPOTHESIS. 2607 01:23:13,755 --> 01:23:16,858 I HAVE A BIG PET PEEVE IN 2608 01:23:16,925 --> 01:23:18,159 BIOLOGY BECAUSE WHEN I WAS A 2609 01:23:18,226 --> 01:23:23,031 GRADUATE STUDENT, LEON AVERY AT 2610 01:23:23,098 --> 01:23:27,402 U.T. SOUTHWESTERN TAUGHT US THE 2611 01:23:27,469 --> 01:23:30,572 HYPOTHESIS THAT THE STATEMENT IS 2612 01:23:30,638 --> 01:23:31,706 ABOUT BIOLOGY, IT'S NOT THE DRUG 2613 01:23:31,773 --> 01:23:34,309 THAT IS GOING TO CURE OR REDUCE 2614 01:23:34,376 --> 01:23:34,743 HYPERTENSION. 2615 01:23:34,809 --> 01:23:35,977 THAT IS SOMETHING THAT IS 2616 01:23:36,044 --> 01:23:37,112 PREDICTED IF THE HYPOTHESIS IS 2617 01:23:37,178 --> 01:23:37,912 CORRECT AND YOU DO SOMETHING TO 2618 01:23:37,979 --> 01:23:39,080 TEST IT. 2619 01:23:39,147 --> 01:23:49,691 I THINK WE HAVE TO -- WE HAVE TO 2620 01:23:52,761 --> 01:23:57,165 GO BACK TO BASICS -- HOW TO 2621 01:23:57,232 --> 01:23:58,933 GENERATE HYPOTHESES BECAUSE IF 2622 01:23:59,000 --> 01:24:01,136 THEY DON'T KNOW HOW TO DO THAT, 2623 01:24:01,202 --> 01:24:04,773 I DON'T THINK THEY EVEN KNOW 2624 01:24:04,839 --> 01:24:08,977 WHAT -- THE NULL OR -- 2625 01:24:09,043 --> 01:24:10,779 >> THANK YOU FOR THAT. 2626 01:24:10,845 --> 01:24:13,581 I THINK ON THE FIRST POINT ABOUT 2627 01:24:13,648 --> 01:24:14,783 FUNDERS, MANDATING THINGS, I 2628 01:24:14,849 --> 01:24:15,984 MEAN, IT SOUNDS LIKE IT'S BEEN 2629 01:24:16,050 --> 01:24:17,819 EFFECTIVE IN YOUR COMMUNITY. 2630 01:24:17,886 --> 01:24:18,987 AGAIN, IT'S THE LIFE BLOOD OF 2631 01:24:19,053 --> 01:24:23,057 OUR WORK, IS FUNDING, SO I THINK 2632 01:24:23,124 --> 01:24:24,692 FUNDERS HAVE A LOT OF POWER AND 2633 01:24:24,759 --> 01:24:25,760 THAT'S WHERE I THINK SPEAKING AT 2634 01:24:25,827 --> 01:24:26,828 NIH IS VERY INTERESTING BECAUSE 2635 01:24:26,895 --> 01:24:28,129 THEY'RE REALLY A BIG FUNDER. 2636 01:24:28,196 --> 01:24:29,264 EVEN IN DEVELOPMENT ECONOMICS, A 2637 01:24:29,330 --> 01:24:30,899 LOT OF MY RESEARCH IN KENYA IS 2638 01:24:30,965 --> 01:24:32,967 FUNDED BY NIA AND NICHD. 2639 01:24:33,034 --> 01:24:35,203 WE CARRY OUT LONG-TERM 2640 01:24:35,270 --> 01:24:36,171 LONGITUDAL DATA COLLECTION AND 2641 01:24:36,237 --> 01:24:36,838 THEY'RE INTERESTED IN THOSE 2642 01:24:36,905 --> 01:24:39,574 HEALTH MEASURES. 2643 01:24:39,641 --> 01:24:41,810 SO I DO THINK THEY'RE VERY 2644 01:24:41,876 --> 01:24:42,377 INFLUENTIAL. 2645 01:24:42,444 --> 01:24:43,678 THIS WORK TAKES TIME, AND I 2646 01:24:43,745 --> 01:24:46,848 THINK YOUR POINT ABOUT, YOU KN 2647 01:24:46,915 --> 01:24:49,083 KNOW, HOW CAN WE IMPLEMENT OPEN 2648 01:24:49,150 --> 01:24:50,718 SCIENCE PRACTICES TO REDUCE 2649 01:24:50,785 --> 01:24:51,719 PUBLICATION BIAS AND HAVE OTHER 2650 01:24:51,786 --> 01:24:53,721 BENEFITS IN WAYS THAT ARE LESS 2651 01:24:53,788 --> 01:24:55,390 ONEROUS IS A HUGE AGENDA POINT. 2652 01:24:55,457 --> 01:24:58,893 I'VE WORKED SO OFTEN FOR MONTHS 2653 01:24:58,960 --> 01:24:59,861 PREPARING DATA TO BE PUBLISHED 2654 01:24:59,928 --> 01:25:00,829 BECAUSE THAT'S THE STANDARD IN 2655 01:25:00,895 --> 01:25:01,129 OUR FIELD. 2656 01:25:01,196 --> 01:25:02,864 IT TAKES A LOT OF WORK AND 2657 01:25:02,931 --> 01:25:03,198 THOUGHT. 2658 01:25:03,264 --> 01:25:04,799 IT HAS A LOT OF BENEFITS BECAUSE 2659 01:25:04,866 --> 01:25:06,201 THEN A LOT OF OTHER PEOPLE USE 2660 01:25:06,267 --> 01:25:07,735 THAT DATA, AND THEN THE TOTAL 2661 01:25:07,802 --> 01:25:08,870 IRONY AND THE FUNNY THING ABOUT 2662 01:25:08,937 --> 01:25:10,171 IT IS ALL THAT WORK WE DO RIGHT 2663 01:25:10,238 --> 01:25:12,540 AFTER WE FINISH DATA COLLECTION 2664 01:25:12,607 --> 01:25:13,875 MAKES IT SO MUCH EASIER FOR ME 2665 01:25:13,942 --> 01:25:15,376 TO USE MY OWN DATA TWO YEARS 2666 01:25:15,443 --> 01:25:16,444 LATER. 2667 01:25:16,511 --> 01:25:17,579 BECAUSE IF I DON'T PUT 2668 01:25:17,645 --> 01:25:19,147 EVERYTHING IN AND DOCUMENT IT 2669 01:25:19,214 --> 01:25:21,883 AND POST IT, I END UP USING THE 2670 01:25:21,950 --> 01:25:23,084 DATA NOT THAT'S LIKE IN OUR 2671 01:25:23,151 --> 01:25:23,952 SECURE SERVERS. 2672 01:25:24,018 --> 01:25:25,987 I JUST USE THE DATA THAT'S LIKE 2673 01:25:26,054 --> 01:25:27,622 PUBLICLY POSTED, BECAUSE THAT, I 2674 01:25:27,689 --> 01:25:29,257 KNOW WE REALLY CLEANED AND 2675 01:25:29,324 --> 01:25:29,824 ORGANIZED. 2676 01:25:29,891 --> 01:25:30,959 SO ACTUALLY YOU'RE BENEFITING 2677 01:25:31,025 --> 01:25:33,461 YOUR FUTURE SELF IN SOME WAYS. 2678 01:25:33,528 --> 01:25:36,464 EVEN WITH PREANALYSIS PLANS AND 2679 01:25:36,531 --> 01:25:37,332 PREREGISTRATION IN ECONOMICS, 2680 01:25:37,398 --> 01:25:38,766 THE INITIAL PUSHBACK WAS, OH, 2681 01:25:38,833 --> 01:25:40,435 THIS TAKES A LOT OF TIME, IT 2682 01:25:40,502 --> 01:25:41,469 TAKES YOU A MONTH TO WRITE ONE 2683 01:25:41,536 --> 01:25:43,505 OF THESE PLANS, BUT IT SAVES US 2684 01:25:43,571 --> 01:25:44,639 SO MUCH TIME ON THE BACK END 2685 01:25:44,706 --> 01:25:46,508 WHEN WE HAVE THE DATA. 2686 01:25:46,574 --> 01:25:48,042 SO THERE REALLY IS THIS KIND OF 2687 01:25:48,109 --> 01:25:49,677 FUTURE SELF VERSUS CURRENT SELF, 2688 01:25:49,744 --> 01:25:53,715 YOU KNOW,IS, ISSUE AROUND A LOTF 2689 01:25:53,781 --> 01:25:53,882 IT. 2690 01:25:53,948 --> 01:25:55,416 YOU TALKED ABOUT STUDENTS AND 2691 01:25:55,483 --> 01:25:55,850 TRAINING STUDENTS. 2692 01:25:55,917 --> 01:25:57,519 WE FACE A LOT OF SIMILAR ISSUES, 2693 01:25:57,585 --> 01:26:00,622 THERE'S A LOT OF DEBATE IN 2694 01:26:00,688 --> 01:26:02,457 ECONOMICS ABOUT HOW CLOSE SHOULD 2695 01:26:02,524 --> 01:26:06,094 WE BE LINKING OUR HYPOTHESES TO 2696 01:26:06,160 --> 01:26:08,496 CORE THEORY VERSUS SOME FIELD 2697 01:26:08,563 --> 01:26:11,032 TESTS THAT ARE KIND OF VERY 2698 01:26:11,099 --> 01:26:12,467 SPECIFIC ABOUT A PARTICULAR 2699 01:26:12,534 --> 01:26:12,800 INTERVENTION. 2700 01:26:12,867 --> 01:26:13,601 THERE'S ALWAYS THIS KIND OF BACK 2701 01:26:13,668 --> 01:26:14,802 AND FORTH TO DO A MAPPING 2702 01:26:14,869 --> 01:26:16,337 BETWEEN OUR CORE THEORIES AND 2703 01:26:16,404 --> 01:26:18,006 THE TEST, SO AGAIN IT FEELS LIKE 2704 01:26:18,072 --> 01:26:19,007 THERE'S A LOT OF PARALLELS 2705 01:26:19,073 --> 01:26:23,311 BETWEEN OUR FIELDS. 2706 01:26:23,378 --> 01:26:26,014 >> HE WAS BEFORE BUT HE HAS NO 2707 01:26:26,080 --> 01:26:27,715 MICROPHONE SO SHOULD I SEIZE THE 2708 01:26:27,782 --> 01:26:30,385 DAY? 2709 01:26:30,451 --> 01:26:33,788 TED, AMERICAN SOCIETY FOR 2710 01:26:33,855 --> 01:26:35,390 MICROBIOLOGY, GREAT TALK, YOURS 2711 01:26:35,456 --> 01:26:37,125 AND SHAI, THE MARGINAL UTILITY 2712 01:26:37,191 --> 01:26:39,093 OF YOUR COST GREATLY ECK SIDES 2713 01:26:39,160 --> 01:26:41,095 THE MARGINAL COSTS OF THE 2714 01:26:41,162 --> 01:26:42,397 ADMISSION TICKET, IT'S FAN TASTE 2715 01:26:42,463 --> 01:26:42,597 TICK. 2716 01:26:42,664 --> 01:26:44,499 >> THANK YOU FOR PUTTING IT IN 2717 01:26:44,566 --> 01:26:45,433 LANGUAGE I UNDERSTAND. 2718 01:26:45,500 --> 01:26:47,068 >> QUICK QUESTION. 2719 01:26:47,135 --> 01:26:49,270 THIS INSTITUTE, NINDS AND NIH, 2720 01:26:49,337 --> 01:26:52,307 HAVE DONE A STUDY DOCTOR -- 2721 01:26:52,373 --> 01:26:53,508 WHERE'S WALTER -- MANY YEARS AGO 2722 01:26:53,575 --> 01:26:55,443 ON THE SPLIT BETWEEN BASIC 2723 01:26:55,510 --> 01:26:57,545 SCIENCE AND CLINICAL SCIENCE, 2724 01:26:57,612 --> 01:26:59,113 AND GREAT THAT YOU REFER TO THE 2725 01:26:59,180 --> 01:27:01,716 BIOMEDICAL SCIENCES FOR 2726 01:27:01,783 --> 01:27:03,251 CLINICALTRIALS.GOV THAT MAKES 2727 01:27:03,318 --> 01:27:03,818 PERFECT SENSE. 2728 01:27:03,885 --> 01:27:06,888 THE QUESTION IS THAT THE NIH AND 2729 01:27:06,955 --> 01:27:08,423 NINDS FUNDS A LARGE AMOUNT OF 2730 01:27:08,489 --> 01:27:10,525 SCIENCE WHICH IS NOT CLINICAL. 2731 01:27:10,592 --> 01:27:12,961 AND FOR WHICH DEFINING THE 2732 01:27:13,027 --> 01:27:14,495 HYPOTHESIS, LET ALONE 2733 01:27:14,562 --> 01:27:16,798 REGISTERING IT, MAKES IT VERY 2734 01:27:16,864 --> 01:27:18,499 COMPLICATED AND ON TOP, I WOULD 2735 01:27:18,566 --> 01:27:19,801 ARGUE -- AND I'M VERY ON YOUR 2736 01:27:19,867 --> 01:27:21,536 PAGE, I'M NOT CRITICIZING YOU, 2737 01:27:21,603 --> 01:27:24,372 BUT THE VALUE OF ASKING 2738 01:27:24,439 --> 01:27:26,374 QUESTIONS IS ACTUALLY VERY HARD, 2739 01:27:26,441 --> 01:27:29,877 YOU KNOW, PROBABLY TO PUT INTO A 2740 01:27:29,944 --> 01:27:30,845 REGISTER, BUT IT IS EQUALLY 2741 01:27:30,912 --> 01:27:32,647 IMPORTANT TO HAVE THE OUTCOMES 2742 01:27:32,714 --> 01:27:33,982 THAT YOU'RE ALLUDING TO. 2743 01:27:34,048 --> 01:27:36,918 IN ANY SUGGESTIONS? 2744 01:27:36,985 --> 01:27:38,186 >> IT'S A FUNDAMENTAL TENSION. 2745 01:27:38,252 --> 01:27:39,354 I THINK THIS GETS BACK TO THE 2746 01:27:39,420 --> 01:27:40,154 VERY GOOD QUESTION BEFORE. 2747 01:27:40,221 --> 01:27:41,856 I THINK ONE OF THE VALUES OF 2748 01:27:41,923 --> 01:27:43,257 REGISTRATION AT LEAST IS WE KNOW 2749 01:27:43,324 --> 01:27:44,792 WHETHER SOMETHING IS EXPLORATORY 2750 01:27:44,859 --> 01:27:45,893 OR CONFIRMATORY, AND I THINK 2751 01:27:45,960 --> 01:27:47,261 THAT IN ITSELF IS REALLY 2752 01:27:47,328 --> 01:27:49,130 VALUABLE. 2753 01:27:49,197 --> 01:27:51,499 WHEN WE STARTED PROMOTING 2754 01:27:51,566 --> 01:27:52,533 PREREGISTRATION IN ECONOMICS, WE 2755 01:27:52,600 --> 01:27:54,068 WERE FACING THIS ISSUE, AGAIN, 2756 01:27:54,135 --> 01:27:56,270 ARE YOU SAYING THAT BECAUSE WE 2757 01:27:56,337 --> 01:27:58,706 CAN PREREGISTER SOME TYPE OF 2758 01:27:58,773 --> 01:27:59,907 STUDIES, OTHER TYPE OF WORK 2759 01:27:59,974 --> 01:28:01,342 WON'T HAVE VALUE? 2760 01:28:01,409 --> 01:28:03,411 ABSOLUTELY NOT. 2761 01:28:03,478 --> 01:28:06,080 IN FACT IN MOST, THERE ARE 2762 01:28:06,147 --> 01:28:06,948 ADDITIONAL ANALYSES WE DIDN'T 2763 01:28:07,015 --> 01:28:07,181 PLAN. 2764 01:28:07,248 --> 01:28:08,383 THEY JUST NEED TO BE SORT OF 2765 01:28:08,449 --> 01:28:10,385 FLAGGED FOR WHAT THEY ARE. 2766 01:28:10,451 --> 01:28:17,191 THEY'RE MORE EX EXPLORATORY RATR 2767 01:28:17,258 --> 01:28:20,361 THAN CONFIRMATORY. 2768 01:28:20,428 --> 01:28:23,031 SO EXPERIMENTING WITH 2769 01:28:23,097 --> 01:28:23,798 PREREGISTER OBSERVATIONAL WORK, 2770 01:28:23,865 --> 01:28:25,199 WHICH I THINK COULD BE THE WAY 2771 01:28:25,266 --> 01:28:25,667 FORWARD. 2772 01:28:25,733 --> 01:28:27,969 IT'S NOT PERFECT, I UNDERSTAND 2773 01:28:28,036 --> 01:28:29,203 THERE'S ISSUES WITH THE TIME BUG 2774 01:28:29,270 --> 01:28:30,872 WHY NOT TRY AND SEE WHAT NORMS 2775 01:28:30,938 --> 01:28:31,539 DO DEVELOP IN THE COMMUNITY. 2776 01:28:31,606 --> 01:28:33,741 SO AGAIN, IT MAY TAKE SEVERAL 2777 01:28:33,808 --> 01:28:34,642 YEARS, BUT MAYBE FOR CERTAIN 2778 01:28:34,709 --> 01:28:35,943 TYPES OF LABORATORY WORK, 2779 01:28:36,010 --> 01:28:36,944 PRE-CLINICAL WORK, THERE REALLY 2780 01:28:37,011 --> 01:28:38,946 ARE WAYS FORWARD. 2781 01:28:39,013 --> 01:28:39,814 SO I'LL GIVE YOU ANOTHER 2782 01:28:39,881 --> 01:28:40,314 EXAMPLE. 2783 01:28:40,381 --> 01:28:42,016 EVEN WITH NON-EXPERIMENTAL WORK 2784 01:28:42,083 --> 01:28:43,151 NOW, SOMETIMES IN THE SOCIAL 2785 01:28:43,217 --> 01:28:44,452 SCIENCES, PEOPLE ARE 2786 01:28:44,519 --> 01:28:46,054 PREREGISTERING ANALYSES BEFORE 2787 01:28:46,120 --> 01:28:46,854 DATA GETS RELEASED. 2788 01:28:46,921 --> 01:28:48,389 SO IT DOESN'T HAVE TO BE 2789 01:28:48,456 --> 01:28:49,090 EXPERIMENTAL WORK, BUT IF 2790 01:28:49,157 --> 01:28:51,192 THERE'S A NEW CENSUS ROUND THAT 2791 01:28:51,259 --> 01:28:53,861 ISN'T BEING RELEASED, IT'S 2792 01:28:53,928 --> 01:28:54,429 PROSPECTIVE, YOUR WORK, YOU 2793 01:28:54,495 --> 01:28:55,596 HAVEN'T SEEN THE DATA, YOU CAN 2794 01:28:55,663 --> 01:28:56,497 REGISTER ANALYSES. 2795 01:28:56,564 --> 01:28:57,965 MAYBE THERE'S OTHER TYPES OF 2796 01:28:58,032 --> 01:28:59,033 WORK, PRE-CLINICAL WORK THAT HAS 2797 01:28:59,100 --> 01:29:00,468 THAT SIMILAR TIMING WHERE IT'S 2798 01:29:00,535 --> 01:29:04,205 REALLY CREDIBLE THAT THE KIND OF 2799 01:29:04,272 --> 01:29:05,873 HYPOTHESIS REGISTRATION CAME 2800 01:29:05,940 --> 01:29:07,175 BEFORE DATA APPEARED, BUT OTHER 2801 01:29:07,241 --> 01:29:08,710 WORK COULD BE PURELY EXPLORATORY 2802 01:29:08,776 --> 01:29:10,378 AND KIND OF DATA MINING 2803 01:29:10,445 --> 01:29:11,813 EXERCISES IN THE GOOD WAY, AND 2804 01:29:11,879 --> 01:29:14,115 THAT'S FINE AS LONG AS WE 2805 01:29:14,182 --> 01:29:14,882 UNDERSTAND THAT THAT'S WHAT IT 2806 01:29:14,949 --> 01:29:18,286 IS. 2807 01:29:18,352 --> 01:29:22,156 >> HI, STEVEN CARRY, IMPERIAL 2808 01:29:22,223 --> 01:29:24,292 COLLEGE, RESEARCH -- ON RESEARCH 2809 01:29:24,358 --> 01:29:27,028 INSTITUTE AND FORMALLY -- I HAVE 2810 01:29:27,095 --> 01:29:28,096 A RHETORICAL QUESTION AND AN 2811 01:29:28,162 --> 01:29:28,996 ACTUAL QUESTION. 2812 01:29:29,063 --> 01:29:31,065 MY RHETORICAL QUESTION REFERS TO 2813 01:29:31,132 --> 01:29:31,833 YOUR LAST SLIDE. 2814 01:29:31,899 --> 01:29:33,301 WE ARE WE LOOKING TO TRAINING 2815 01:29:33,367 --> 01:29:34,402 THE NEXT GENERATION OF 2816 01:29:34,469 --> 01:29:35,369 SCIENTISTS TO DEAL WITH THIS? 2817 01:29:35,436 --> 01:29:37,038 WHY AREN'T WE FOCUSING ON THE 2818 01:29:37,105 --> 01:29:38,606 CURRENT GENERATION OF SCIENTISTS 2819 01:29:38,673 --> 01:29:39,807 AND GETTING THEM TO GET THEIR 2820 01:29:39,874 --> 01:29:41,876 ACT TOGETHER? 2821 01:29:41,943 --> 01:29:44,312 MY ACTUAL QUESTION IS AROUND -- 2822 01:29:44,378 --> 01:29:45,780 YOU SHARED THIS IMPRESSIVE 2823 01:29:45,847 --> 01:29:47,081 EXPONENTIAL RISE OF REGISTERED 2824 01:29:47,148 --> 01:29:48,649 STUDIES IN ECONOMICS OVER THE 2825 01:29:48,716 --> 01:29:49,851 LAST DECADE. 2826 01:29:49,917 --> 01:29:51,519 WHAT PERCENTAGE OF PAPERS 2827 01:29:51,586 --> 01:29:54,455 PUBLISHED IN ECONOMICS ARE 2828 01:29:54,522 --> 01:29:54,789 PREREGISTERED? 2829 01:29:54,856 --> 01:29:57,692 >> GREAT QUESTION. 2830 01:29:57,759 --> 01:30:00,762 SO WHY THE NEXT GENERATION? 2831 01:30:00,828 --> 01:30:01,829 WE MATTER TOO! 2832 01:30:01,896 --> 01:30:03,931 THOSE OF US WHO ARE NOT THE NEXT 2833 01:30:03,998 --> 01:30:05,166 GENERATION MATTER, YES, AND WE 2834 01:30:05,233 --> 01:30:07,368 WANT TO ALL -- WE WANT TO CHANGE 2835 01:30:07,435 --> 01:30:08,269 PRACTICES AND CERTAINLY THERE'S 2836 01:30:08,336 --> 01:30:11,239 A LOT OF THAT WORK GOING ON. 2837 01:30:11,305 --> 01:30:12,673 I THINK THE FEELING WITH THE 2838 01:30:12,740 --> 01:30:13,741 NEXT GENERATION IS MAYBE THE 2839 01:30:13,808 --> 01:30:15,276 NORMS AREN'T AS SOLIDIFIED. 2840 01:30:15,343 --> 01:30:16,644 THEY'RE LEARNING HOW TO DO 2841 01:30:16,711 --> 01:30:17,145 SCIENCE. 2842 01:30:17,211 --> 01:30:20,581 AND THAT COULD BE A PATH 2843 01:30:20,648 --> 01:30:20,915 FORWARD. 2844 01:30:20,982 --> 01:30:22,583 AGAIN, IF THE NEXT GENERATION'S 2845 01:30:22,650 --> 01:30:23,918 NORMS CHANGES, THAT WILL JUST 2846 01:30:23,985 --> 01:30:25,586 SOLIDIFY THE CHANGE AS WELL. 2847 01:30:25,653 --> 01:30:27,088 WE'RE HEARING FROM A LOT OF THE 2848 01:30:27,155 --> 01:30:28,990 TRAINEES, THE POSTDOCS AND 2849 01:30:29,056 --> 01:30:29,991 GRADUATE STUDENTS, THEY GO TO 2850 01:30:30,057 --> 01:30:31,526 THE BITSS TRAINING COURSES AND 2851 01:30:31,592 --> 01:30:33,528 THEN THEY GO BACK TO THEIR LABS 2852 01:30:33,594 --> 01:30:34,395 AND THEIR RESEARCH TEAMS AND 2853 01:30:34,462 --> 01:30:35,596 THEY'RE THE ONES PUSHING THROUGH 2854 01:30:35,663 --> 01:30:36,597 CHANGES IN THEIR GROUP. 2855 01:30:36,664 --> 01:30:37,865 SO THEY COULD BE VERY EFFECTIVE 2856 01:30:37,932 --> 01:30:38,399 AGENTS OF CHANGE. 2857 01:30:38,466 --> 01:30:40,168 BUT I THINK IT'S NEVER TOO LATE. 2858 01:30:40,234 --> 01:30:42,003 IT'S NEVER TOO LATE TO CHANGE. 2859 01:30:42,069 --> 01:30:44,906 IN TERMS OF PREREGISTERED 2860 01:30:44,972 --> 01:30:46,274 STUDIES, IN ECONOMICS JOURNALS, 2861 01:30:46,340 --> 01:30:47,175 THERE WAS JUST RECENT 2862 01:30:47,241 --> 01:30:48,509 PUBLICATIONS ON THIS, SOMETHING 2863 01:30:48,576 --> 01:30:51,379 LIKE ONLY 25% OF STUDIES ARE 2864 01:30:51,445 --> 01:30:53,047 EXPERIMENTAL. 2865 01:30:53,114 --> 01:30:55,750 75% ARE OBSERVATIONAL WORK. 2866 01:30:55,817 --> 01:30:58,452 NON-EXPERIMENTAL WORK. 2867 01:30:58,519 --> 01:30:59,620 SO AMONG OBSERVATIONAL WORK, 2868 01:30:59,687 --> 01:31:00,855 THERE'S NOT MUCH 2869 01:31:00,922 --> 01:31:01,489 PREREGISTRATION. 2870 01:31:01,556 --> 01:31:03,124 AMONG EXPERIMENTAL WORK, THE 2871 01:31:03,191 --> 01:31:04,525 PROPORTION OF PREREGISTERED 2872 01:31:04,592 --> 01:31:06,060 STUDIES HAS INCREASED 2873 01:31:06,127 --> 01:31:07,028 DRAMATICALLY, AND ACTUALLY IN 2874 01:31:07,094 --> 01:31:09,630 RECENT YEARS, NOW THE AEA HAS 2875 01:31:09,697 --> 01:31:10,798 MANDATED THAT EXPERIMENTAL 2876 01:31:10,865 --> 01:31:12,667 STUDIES PUBLISHED IN AEA 2877 01:31:12,733 --> 01:31:15,436 JOURNALS NEED TO BE PREREJ 2878 01:31:15,503 --> 01:31:16,771 PREREGISTERED ON THE AEA 2879 01:31:16,838 --> 01:31:17,405 REGISTRY. 2880 01:31:17,471 --> 01:31:19,140 SO THEY DIDN'T START WITH THAT. 2881 01:31:19,207 --> 01:31:20,441 THE NORM ACTUALLY CHANGED FIRST 2882 01:31:20,508 --> 01:31:21,843 AND THAT TAKEOFF YOU SAW HAPPEN, 2883 01:31:21,909 --> 01:31:23,144 AND THEN ONCE THE NORM WAS 2884 01:31:23,211 --> 01:31:24,345 ACCEPTED, THEY KIND OF CODIFIED 2885 01:31:24,412 --> 01:31:25,112 IT. 2886 01:31:25,179 --> 01:31:27,715 SO I THINK HIGH SHARES NOW ARE 2887 01:31:27,782 --> 01:31:30,184 BEING PREREGISTERED, AMONG THE 2888 01:31:30,251 --> 01:31:31,719 EXPERIMENTAL STUDIES. 2889 01:31:31,786 --> 01:31:33,354 >> WELL, THANK YOU ALL FOR THIS 2890 01:31:33,421 --> 01:31:34,322 VERY INTERESTING DISCUSSION. 2891 01:31:34,388 --> 01:31:36,390 THANKS, TED, FOR A FASCINATING 2892 01:31:36,457 --> 01:31:36,924 TALK. 2893 01:31:36,991 --> 01:31:38,226 WE'LL TAKE A BREAK NOW. 2894 01:31:38,292 --> 01:31:40,494 WE'LL RECONVENE AT 10:30, AND 2895 01:31:40,561 --> 01:31:41,395 LET'S THANK TED AGAIN. 2896 01:31:41,462 --> 01:31:48,236 [APPLAUSE] 2897 01:31:48,302 --> 01:31:51,105 >> HI, EVERYONE. 2898 01:31:51,172 --> 01:31:55,643 I'M DEVON CRAWFORD, PROGRAM 2899 01:31:55,710 --> 01:31:57,178 DIRECTOR IN THE NINDS OFFICE OF 2900 01:31:57,245 --> 01:31:57,712 RESEARCH QUALITY. 2901 01:31:57,778 --> 01:32:01,349 AT THE HEARD AT THE LAST MINUTE, 2902 01:32:01,415 --> 01:32:02,783 ERICK TURNER WAS NO LONGER ABLE 2903 01:32:02,850 --> 01:32:03,985 TO ATTEND THE MEETING SO I AM 2904 01:32:04,051 --> 01:32:05,887 SORT OF FILLING IN AS THE 2905 01:32:05,953 --> 01:32:08,289 GENERAL MODERATOR. 2906 01:32:08,356 --> 01:32:11,392 AND THIS SESSION IS REALLY 2907 01:32:11,459 --> 01:32:13,294 ABOUT -- IT'S CALLED QUANTIFYING 2908 01:32:13,361 --> 01:32:15,129 PUBLICATION BIAS, BUT IT'S 2909 01:32:15,196 --> 01:32:17,398 ASKING HOW DO WE QUANTIFY 2910 01:32:17,465 --> 01:32:18,733 PUBLICATION BIAS, HOW DO WE KNOW 2911 01:32:18,799 --> 01:32:20,234 IF WE'RE FIXING PUBLICATION 2912 01:32:20,301 --> 01:32:21,836 BIAS, HOW BAD IS PUBLICATION 2913 01:32:21,903 --> 01:32:22,904 BIAS, AND HOW DO WE THINK ABOUT 2914 01:32:22,970 --> 01:32:25,973 IT IN THIS WAY? 2915 01:32:26,040 --> 01:32:30,912 AND IN ORDER TO START OFF THIS 2916 01:32:30,978 --> 01:32:35,883 SESSION, MARIAH HOY, PROJECT 2917 01:32:35,950 --> 01:32:36,751 MANAGER IN THE NINDS OFFICE OF 2918 01:32:36,817 --> 01:32:37,752 RESEARCH QUALITY, BECAUSE WE 2919 01:32:37,818 --> 01:32:38,953 FELT THAT ERICK'S CONTRIBUTION 2920 01:32:39,020 --> 01:32:41,289 WAS REALLY IMPORTANT SHE'S GOING 2921 01:32:41,355 --> 01:32:42,723 TO BE PRESENTING SOME OF HIS 2922 01:32:42,790 --> 01:32:48,729 DATA TO START US OFF, SO MARIAH. 2923 01:32:48,796 --> 01:32:48,996 >> GREAT. 2924 01:32:49,063 --> 01:32:49,997 THANK YOU. 2925 01:32:50,064 --> 01:32:52,633 SO LIKE DEVON SAID, MY NAME IS 2926 01:32:52,700 --> 01:32:54,335 MARIAH HOY, I'M ALSO AN OFFICE 2927 01:32:54,402 --> 01:32:55,202 OF RESEARCH QUALITY AND I'M 2928 01:32:55,269 --> 01:32:57,071 GOING TO BE SHARING SOME OF 2929 01:32:57,138 --> 01:32:59,273 ERICK'S EMPIRICAL DATA THAT I 2930 01:32:59,340 --> 01:33:00,942 THINK JUST REALLY NICELY 2931 01:33:01,008 --> 01:33:02,043 DEMONSTRATES HOW PUBLICATION 2932 01:33:02,109 --> 01:33:04,879 BIAS CAN REALLY DISTORT REALITY. 2933 01:33:04,946 --> 01:33:07,715 SO THIS DATA CAME FROM HIS TIME 2934 01:33:07,782 --> 01:33:11,218 AS AN FDA REVIEWER BETWEEN 1998 2935 01:33:11,285 --> 01:33:12,386 AND 2001. 2936 01:33:12,453 --> 01:33:14,755 HE'S NOW A FACULTY MEMBER AT 2937 01:33:14,822 --> 01:33:18,492 OHSU, AND HE'S BEEN THERE SINCE 2938 01:33:18,559 --> 01:33:20,628 2005, BUT REMAINS REALLY 2939 01:33:20,695 --> 01:33:22,730 INTERESTED IN THE TOPIC OF 2940 01:33:22,797 --> 01:33:24,131 PUBLICATION BIAS AND OUR EFFORTS 2941 01:33:24,198 --> 01:33:29,737 TO ESTIMATE IT. 2942 01:33:29,804 --> 01:33:32,073 SO DURING HIS TIME AS AN FDA 2943 01:33:32,139 --> 01:33:33,274 REVIEWER, ERICK BEGAN TO NOTICE 2944 01:33:33,341 --> 01:33:35,042 THAT WHAT WAS REPRESENTED IN THE 2945 01:33:35,109 --> 01:33:37,979 PUBLISHED LITERATURE WAS VERY 2946 01:33:38,045 --> 01:33:40,348 DIFFERENT IN TERMS OF EFFICACY 2947 01:33:40,414 --> 01:33:43,317 OF ANTI-DEPRESSANTS, WAS VERY 2948 01:33:43,384 --> 01:33:44,185 DIFFERENT THAN WHAT THE FDA 2949 01:33:44,251 --> 01:33:45,453 COULD SEE WHEN THEY HAD THE 2950 01:33:45,519 --> 01:33:47,221 OUTCOMES OF THESE CLINICAL 2951 01:33:47,288 --> 01:33:49,190 TRIALS BEING REPORTED TO THEM. 2952 01:33:49,256 --> 01:33:53,027 SO IN THIS SEMINAL PAPER, ERICK 2953 01:33:53,094 --> 01:33:55,029 AND COLLEAGUES SHOW HOW 2954 01:33:55,096 --> 01:33:57,331 PUBLICATION BIAS CAN REALLY 2955 01:33:57,398 --> 01:34:01,168 DISTORT OUR UNDERSTANDING OF A 2956 01:34:01,235 --> 01:34:04,438 SCIENTIFIC FIELD, AND 2957 01:34:04,505 --> 01:34:05,806 PARTICULARLY OUR INTERPRETATION 2958 01:34:05,873 --> 01:34:07,241 OF EFFICACY OF VARIOUS 2959 01:34:07,308 --> 01:34:09,210 ANTI-DEPRESSANTS. 2960 01:34:09,276 --> 01:34:19,754 SO IN TOTAL, THEY LOOKED AT 2961 01:34:25,292 --> 01:34:27,294 74 FDA REGISTERED STUDIES. 2962 01:34:27,361 --> 01:34:29,296 THESE CLINICAL TRIALS WERE 2963 01:34:29,363 --> 01:34:32,900 CONDUCTED BETWEEN 1987 AND 2006. 2964 01:34:32,967 --> 01:34:43,344 THE MEDIAN BEING 1996. 2965 01:34:44,478 --> 01:34:45,646 SO THEY LOOKED AT THE OUTCOMES 2966 01:34:45,713 --> 01:34:48,382 OF THESE 74 FDA REGISTERED 2967 01:34:48,449 --> 01:34:49,984 STUDIES, AND THEY DID A REALLY 2968 01:34:50,051 --> 01:34:51,185 INTERESTING ANALYSIS. 2969 01:34:51,252 --> 01:34:54,288 SO THEY FOLLOWED UP ON WHAT WAS 2970 01:34:54,355 --> 01:34:56,057 THE JOURNAL VIEW, SO HOW ARE 2971 01:34:56,123 --> 01:34:58,793 THESE OUTCOMES REPORTED IN 2972 01:34:58,859 --> 01:35:00,761 JOURNALS, AND THEN COMPARED THAT 2973 01:35:00,828 --> 01:35:03,397 TO WHAT THEY SAW AS FDA 2974 01:35:03,464 --> 01:35:09,170 REVIEWERS. 2975 01:35:09,236 --> 01:35:10,137 OKAY. 2976 01:35:10,204 --> 01:35:12,973 SO HERE I'M SHOWING YOU THE 2977 01:35:13,040 --> 01:35:15,843 JOURNAL VIEW FOR THE OUTCOMES OF 2978 01:35:15,910 --> 01:35:18,412 THESE 74 FDA REGISTERED STUDIES. 2979 01:35:18,479 --> 01:35:21,248 SO AS YOU CAN SEE, THE 2980 01:35:21,315 --> 01:35:23,984 OVERWHELMING MAJORITY OF THEM 2981 01:35:24,051 --> 01:35:28,122 REPORTED POSITIVE FINDINGS. 2982 01:35:28,189 --> 01:35:29,957 SO 51 OF THE STUDYS WERE 2983 01:35:30,024 --> 01:35:31,659 PUBLISHED IN TRADITIONAL 2984 01:35:31,725 --> 01:35:33,127 JOURNALS. 2985 01:35:33,194 --> 01:35:35,930 48 OF THOSE REPORTED POSITIVE 2986 01:35:35,996 --> 01:35:38,666 FINDINGS. 2987 01:35:38,732 --> 01:35:39,533 3 REPORTED NEGATIVE FINDINGS. 2988 01:35:39,600 --> 01:35:42,837 THE WAY THAT I'M PR REPRESENTING 2989 01:35:42,903 --> 01:35:44,672 THIS IS EACH STUDY IS A BOX, AND 2990 01:35:44,738 --> 01:35:46,173 THEN BELOW IT, THE VARIOUS DRUGS 2991 01:35:46,240 --> 01:35:50,845 THAT THEY WERE TESTING. 2992 01:35:50,911 --> 01:35:51,178 ARE 2993 01:35:51,245 --> 01:35:56,884 REPRESENTED BE HE LO. 2994 01:35:56,951 --> 01:35:57,718 >> [INAUDIBLE] 2995 01:35:57,785 --> 01:35:59,487 >> OH, THAT'S JUST THE NUMBER OF 2996 01:35:59,553 --> 01:36:00,087 THE STUDY. 2997 01:36:00,154 --> 01:36:01,288 THERE WERE 74 TOTAL. 2998 01:36:01,355 --> 01:36:02,723 SO IT'S NOT GOING TO BE 2999 01:36:02,790 --> 01:36:04,458 FOLLOWING IN NUMERICAL ORDER 3000 01:36:04,525 --> 01:36:05,726 BECAUSE NOT ALL OF THE STUDIES 3001 01:36:05,793 --> 01:36:06,861 WERE PUBLISHED IN TRADITIONAL 3002 01:36:06,927 --> 01:36:17,104 JOURNALS. 3003 01:36:23,144 --> 01:36:25,212 HOWEVER, AS YOU CAN SEE, THE 3004 01:36:25,279 --> 01:36:26,881 FDA'S VIEW OF THEIR ASSESSMENT 3005 01:36:26,947 --> 01:36:29,216 OF THE OUTCOMES OF THESE 3006 01:36:29,283 --> 01:36:30,417 CLINICAL TRIALS WAS VERY 3007 01:36:30,484 --> 01:36:30,951 DIFFERENT. 3008 01:36:31,018 --> 01:36:34,755 SO THEY ACTUALLY FOUND THAT 38 3009 01:36:34,822 --> 01:36:38,259 OF THOSE 74 STUDIES HAD NEGATIVE 3010 01:36:38,325 --> 01:36:42,663 OUTCOMES. 3011 01:36:42,730 --> 01:36:44,398 AND -- OR HAD POSITIVE OUTCOMES, 3012 01:36:44,465 --> 01:36:48,169 AND THEN 36 OF THEM HAD EITHER 3013 01:36:48,235 --> 01:36:50,938 NEGATIVE OR QUESTIONABLE OR 3014 01:36:51,005 --> 01:36:52,006 INCONCLUSIVE. 3015 01:36:52,072 --> 01:36:54,208 AND SO YOU CAN SEE THAT THEY 3016 01:36:54,275 --> 01:36:56,744 ACTUALLY FOUND THAT ABOUT 50% OF 3017 01:36:56,810 --> 01:36:58,045 THE TRIALS ARE ACTUALLY 3018 01:36:58,112 --> 01:36:59,413 NEGATIVE. 3019 01:36:59,480 --> 01:37:01,482 SO A VERY DIFFERENT VIEW FROM 3020 01:37:01,549 --> 01:37:02,883 WHAT'S PUT OUT IN THE PUBLISHED 3021 01:37:02,950 --> 01:37:04,418 LITERATURE. 3022 01:37:04,485 --> 01:37:06,587 AND THEN SO WHEN YOU LOOK AT 3023 01:37:06,654 --> 01:37:09,423 THOSE 36 STUDIES THAT WERE 3024 01:37:09,490 --> 01:37:11,892 EITHER NEGATIVE, WHICH WAS 24 OF 3025 01:37:11,959 --> 01:37:13,761 THEM, OR QUESTIONABLE OR 3026 01:37:13,827 --> 01:37:16,063 INCONCLUSIVE, WHICH WAS 12 OF 3027 01:37:16,130 --> 01:37:17,431 THEM, SO THREE OF THOSE HAD 3028 01:37:17,498 --> 01:37:19,300 ACTUALLY BEEN PUBLISHED IN 3029 01:37:19,366 --> 01:37:20,501 TRADITIONAL JOURNALS AS 3030 01:37:20,568 --> 01:37:24,972 NEGATIVE. 3031 01:37:25,039 --> 01:37:26,807 22 OF THEM WERE NEVER PUBLISHED, 3032 01:37:26,874 --> 01:37:28,242 BUT THEN 11 STUDIES WERE 3033 01:37:28,309 --> 01:37:30,077 ACTUALLY PUBLISHED IN JOURNALS 3034 01:37:30,144 --> 01:37:33,247 TO HAVE POSITIVE FINDINGS AND 3035 01:37:33,314 --> 01:37:34,548 THAT DIRECTLY CONFLICTED WITH 3036 01:37:34,615 --> 01:37:36,083 THE CONCLUSIONS MADE BY THE FDA. 3037 01:37:36,150 --> 01:37:38,352 AND SO THIS IDEA THAT PEOPLE TRY 3038 01:37:38,419 --> 01:37:41,722 TO PUT A POSITIVE SPIN ON 3039 01:37:41,789 --> 01:37:44,692 SOMETHING AND PUT THAT OUT 3040 01:37:44,758 --> 01:37:45,993 THERE. 3041 01:37:46,060 --> 01:37:47,561 SO I JUST THINK THIS IS LIKE ONE 3042 01:37:47,628 --> 01:37:49,430 OF THE BEST EXAMPLES EMPIRICALLY 3043 01:37:49,496 --> 01:37:51,432 THAT WE HAVE IN KIND OF THE 3044 01:37:51,498 --> 01:37:54,235 BIOMEDICAL SPACE SHOWING HOW 3045 01:37:54,301 --> 01:37:55,202 PUBLICATION BIAS CAN JUST REALLY 3046 01:37:55,269 --> 01:37:58,072 DISTORT OUR ABILITY TO INTERPRET 3047 01:37:58,138 --> 01:37:59,273 THE EVIDENCE IN A SCIENTIFIC 3048 01:37:59,340 --> 01:38:00,741 FIELD, AND THAT'S BECAUSE WE 3049 01:38:00,808 --> 01:38:02,610 DON'T HAVE ALL OF THE DATA, ALL 3050 01:38:02,676 --> 01:38:06,113 OF THOSE BLACK BOXES, WE 3051 01:38:06,180 --> 01:38:07,081 TYPICALLY DON'T HAVE. 3052 01:38:07,147 --> 01:38:09,250 3% OF THOSE WERE PUBLISHED OR 3053 01:38:09,316 --> 01:38:10,684 3 OF THOSE WERE PUBLISHED IN 3054 01:38:10,751 --> 01:38:11,885 TRADITIONAL JOURNALS. 3055 01:38:11,952 --> 01:38:15,556 AND I THINK THIS IS A REALLY BIG 3056 01:38:15,623 --> 01:38:16,657 PROBLEM THAT WE HAVE TO TACKLE, 3057 01:38:16,724 --> 01:38:17,625 THAT WE HAVE TO DO SOMETHING 3058 01:38:17,691 --> 01:38:20,361 ABOUT. 3059 01:38:20,427 --> 01:38:22,529 AND I THINK WE'LL SEE FROM OUR 3060 01:38:22,596 --> 01:38:28,869 NEXT SPEAKER, THIS IS IS NOT 3061 01:38:28,936 --> 01:38:30,738 JUST A PROBLEM WE SEE IN 3062 01:38:30,804 --> 01:38:33,307 BIOMEDICINE, AS WE SAW IN SHAI 3063 01:38:33,374 --> 01:38:35,609 AND TED'S TALKS, THIS IS A 3064 01:38:35,676 --> 01:38:36,610 PROBLEM THAT AFFECTS ALMOST ALL 3065 01:38:36,677 --> 01:38:37,044 RESEARCH FIELDS. 3066 01:38:37,111 --> 01:38:43,651 SO WITH THAT, I WILL LET NEIL -- 3067 01:38:43,717 --> 01:38:48,055 >> NEIL MA HOE TRA IS FROM 3068 01:38:48,122 --> 01:38:49,456 STANFORD UNIVERSITY. 3069 01:38:49,523 --> 01:38:50,724 >> I APPRECIATE THE INVITATION 3070 01:38:50,791 --> 01:38:53,494 TO JOIN A GROUP OF SCIENTISTS 3071 01:38:53,560 --> 01:38:55,029 AND NEUROSCIENCES, BIOLOGICAL 3072 01:38:55,095 --> 01:38:56,130 SCIENCES AND HOPEFULLY WE CAN 3073 01:38:56,196 --> 01:39:04,371 LEARN FROM EACH OTHER. 3074 01:39:04,438 --> 01:39:05,906 BASICALLY WHAT I'VE BEEN DOING 3075 01:39:05,973 --> 01:39:07,775 FOR MAYBE A DECADE IS TRYING TO 3076 01:39:07,841 --> 01:39:09,610 USE STUDY REGISTRIES TO MEASURE 3077 01:39:09,677 --> 01:39:10,377 PUBLICATION BIAS. 3078 01:39:10,444 --> 01:39:11,578 IN MANY WAYS, I THINK THIS IS 3079 01:39:11,645 --> 01:39:12,913 THE BEST WAY TO QUANTIFY HOW BAD 3080 01:39:12,980 --> 01:39:13,514 THE PROBLEM IS. 3081 01:39:13,580 --> 01:39:15,582 SO TED AT THE END OF HIS TALK 3082 01:39:15,649 --> 01:39:17,117 USED A VERY SIMILAR METHOD WHERE 3083 01:39:17,184 --> 01:39:18,619 YOU TRY AND TAKE A COHORT OF 3084 01:39:18,686 --> 01:39:20,254 STUDIES AND FOLLOW THEM THROUGH 3085 01:39:20,321 --> 01:39:21,922 THE PUBLICATION PROCESS, AND 3086 01:39:21,989 --> 01:39:23,857 THIS MAKES A LOT LESS 3087 01:39:23,924 --> 01:39:25,592 ASSUMPTIONS THAN USING THINGS 3088 01:39:25,659 --> 01:39:26,894 LIKE FUNNEL PLOTS OR TRIM AND 3089 01:39:26,960 --> 01:39:28,595 FILL, AND SO I THINK IF YOU LOOK 3090 01:39:28,662 --> 01:39:29,963 AT THE MOST CUTTING EDGE 3091 01:39:30,030 --> 01:39:30,931 RESEARCH ON THIS IS KIND OF 3092 01:39:30,998 --> 01:39:34,201 MOVING IN THIS DIRECTION. 3093 01:39:34,268 --> 01:39:35,803 THE DEPUTY DIRECTOR HAD THE 3094 01:39:35,869 --> 01:39:36,804 DICKERSON DEFINITION OF 3095 01:39:36,870 --> 01:39:38,005 PUBLICATION BIAS, WHICH I HAVE 3096 01:39:38,072 --> 01:39:39,740 USED BUT I'M NOT ACTUALLY A HUGE 3097 01:39:39,807 --> 01:39:40,974 FAN OF, SO THIS IS KIND OF THE 3098 01:39:41,041 --> 01:39:42,376 DEFINITION THAT I LIKE USING, 3099 01:39:42,443 --> 01:39:44,545 WHICH IS THAT PUBLICATION BIAS 3100 01:39:44,611 --> 01:39:46,180 OCCURS WHEN THE STATISTICAL 3101 01:39:46,246 --> 01:39:48,482 SIGNIFICANCE OF FINDINGS 3102 01:39:48,549 --> 01:39:49,583 INFLUENCES PUBLICATION 3103 01:39:49,650 --> 01:39:50,784 PROBABILITY, IMPORTANTLY 3104 01:39:50,851 --> 01:39:51,852 CONDITIONAL ON THE QUALITY OF 3105 01:39:51,919 --> 01:39:53,053 STUDY DESIGN. 3106 01:39:53,120 --> 01:39:55,589 SO THIS CONDITION IS IMPORTANT 3107 01:39:55,656 --> 01:39:57,324 BECAUSE ON AVERAGE, NULL STUDIES 3108 01:39:57,391 --> 01:39:59,593 ARE GOING TO BE LOWER QUALITY 3109 01:39:59,660 --> 01:40:00,728 THAN POSITIVE STUDIES. 3110 01:40:00,794 --> 01:40:03,230 AND THAT'S FOR MANY REASONS. 3111 01:40:03,297 --> 01:40:04,631 SO ONE, THEY'RE MORE LIKELY TO 3112 01:40:04,698 --> 01:40:06,266 HAVE MEASUREMENT ERROR IN THEIR 3113 01:40:06,333 --> 01:40:07,801 VARIABLES, THEY'RE GOING TO BE 3114 01:40:07,868 --> 01:40:10,104 MORE LIKELY TO HAVE LOW 3115 01:40:10,170 --> 01:40:11,405 STATISTICAL POWER, SO WHAT WE 3116 01:40:11,472 --> 01:40:12,806 ACTUALLY MEAN BY PUBLICATION 3117 01:40:12,873 --> 01:40:15,042 BIAS IS ONCE YOU CONDITION ON 3118 01:40:15,109 --> 01:40:17,010 THINGS LIKE PROPER MEASUREMENT, 3119 01:40:17,077 --> 01:40:20,080 GOOD DESIGN, STATISTICAL POWER, 3120 01:40:20,147 --> 01:40:23,250 THEN DOES THE SIGNIFICANCE OF 3121 01:40:23,317 --> 01:40:23,584 THE RESULTS. 3122 01:40:23,650 --> 01:40:25,419 SO THAT'S WHY STUDY REGISTRIES 3123 01:40:25,486 --> 01:40:26,720 ARE AN IMPORTANT WAY TO DO THIS. 3124 01:40:26,787 --> 01:40:28,889 SO WHAT WE DID IS WE USED THE 3125 01:40:28,956 --> 01:40:29,957 TIMESHARE EXPERIMENTS PROGRAMMED 3126 01:40:30,023 --> 01:40:30,691 IN THE SOCIAL SCIENCES. 3127 01:40:30,758 --> 01:40:32,526 THIS IS AN NSF-FUNDED PROGRAM 3128 01:40:32,593 --> 01:40:34,261 WHERE RESEARCHERS HAVE TO APPLY 3129 01:40:34,328 --> 01:40:37,765 TO OBTAIN SPACE ON VERY 3130 01:40:37,831 --> 01:40:39,400 EXPENSIVE, HIGH QUALITY 3131 01:40:39,466 --> 01:40:41,268 PROBABILITY SURVEYS TO RUN 3132 01:40:41,335 --> 01:40:42,302 RANDOMIZED EXPERIMENTS. 3133 01:40:42,369 --> 01:40:44,271 SO WHY DOES THIS COHORT SOLVE 3134 01:40:44,338 --> 01:40:46,140 SOME OF THE PROBLEMS THAT ALLOWS 3135 01:40:46,206 --> 01:40:47,875 US TO GET AT THIS DEFINITION? 3136 01:40:47,941 --> 01:40:49,543 SO ONE, YOU ACTUALLY HAVE A 3137 01:40:49,610 --> 01:40:50,944 SAMPLE OF COMPLETED STUDIES THAT 3138 01:40:51,011 --> 01:40:52,813 ARE OBSERVED PRIOR TO 3139 01:40:52,880 --> 01:40:55,416 PUBLICATION BIAS. 3140 01:40:55,482 --> 01:40:57,050 SO UNLIKE TED'S REGISTRY WHERE 3141 01:40:57,117 --> 01:40:58,352 WE MIGHT NOT KNOW WHETHER SOME 3142 01:40:58,419 --> 01:41:01,755 OF THESE THINGS WERE ACTUALLY 3143 01:41:01,822 --> 01:41:04,124 COLLECTED THE DATA, WITH TESTS 3144 01:41:04,191 --> 01:41:06,093 WE KNOW ALL OF THE DATA HAS BEEN 3145 01:41:06,160 --> 01:41:07,327 COLLECTED AND ALL OF THE DATA IS 3146 01:41:07,394 --> 01:41:08,362 POSTED ONLINE, AND BECAUSE YOU 3147 01:41:08,429 --> 01:41:09,797 HAVE TO APPLY, YOU DON'T JUST 3148 01:41:09,863 --> 01:41:11,665 HAVE TO HAVE LIKE A FACE AND 3149 01:41:11,732 --> 01:41:14,201 POST YOUR PREREGISTRATION PLAN. 3150 01:41:14,268 --> 01:41:15,836 THERE HYPOTHETICALLY IS A 3151 01:41:15,903 --> 01:41:16,703 MEANINGFUL QUALITY THRESHOLD 3152 01:41:16,770 --> 01:41:17,504 THAT'S BEEN ESTABLISHED SUCH 3153 01:41:17,571 --> 01:41:20,174 THAT THIS BOLDED LANGUAGE IS 3154 01:41:20,240 --> 01:41:21,074 MEANINGFUL, THAT WE HAVE SOME 3155 01:41:21,141 --> 01:41:22,910 CONDITION OF QUALITY, THAT HAS 3156 01:41:22,976 --> 01:41:24,344 BEEN PREVETTED. 3157 01:41:24,411 --> 01:41:26,313 SO SHAI ALREADY SHOWED THIS, BUT 3158 01:41:26,380 --> 01:41:29,016 WE'LL KIND OF SHOW IT AGAIN. 3159 01:41:29,082 --> 01:41:30,784 WHICH IS, ESSENTIALLY WHAT WE 3160 01:41:30,851 --> 01:41:32,853 FOUND IS THAT IF YOU KIND OF 3161 01:41:32,920 --> 01:41:35,989 COMPARE STRONG TO NULL STUDIES, 3162 01:41:36,056 --> 01:41:37,324 THERE WAS A MASSIVE DIFFERENCE, 3163 01:41:37,391 --> 01:41:40,894 THIS RATIO OF 65 TO 5 IN TERMS 3164 01:41:40,961 --> 01:41:42,129 OF NULL STUDIES NEVER BEING 3165 01:41:42,196 --> 01:41:43,430 WRITTEN UP AND STRONG STUDIES 3166 01:41:43,497 --> 01:41:44,531 BEING WRITTEN, AND YOU ALSO SEE 3167 01:41:44,598 --> 01:41:45,933 IN TERMS OF PUBLISHED PAPERS 3168 01:41:45,999 --> 01:41:48,302 THIS KIND OF PROBLEMATIC RATIO. 3169 01:41:48,368 --> 01:41:50,838 NOW ALL OF THESE DATA REALLY 3170 01:41:50,904 --> 01:41:52,039 COME FROM WHAT YOU WOULD CALL 3171 01:41:52,105 --> 01:41:55,075 THE BEFORE OPEN SCIENCE ERA FROM 3172 01:41:55,142 --> 01:41:57,811 2002 TO 2012. 3173 01:41:57,878 --> 01:41:59,980 SO I'M GOING TO TO SHOW YOU SOME 3174 01:42:00,047 --> 01:42:06,353 STUFF AFTER THE OPEN SCIENCE E 3175 01:42:06,420 --> 01:42:08,522 ERA, YOU CAN SEE POST OPEN 3176 01:42:08,589 --> 01:42:10,157 SCIENCE ERA IS CHARACTERIZED BY 3177 01:42:10,224 --> 01:42:11,458 MANY THINGS, SO ONE IS LIKE 3178 01:42:11,525 --> 01:42:13,427 STUFF THAT TED HAS DONE LIKE THE 3179 01:42:13,494 --> 01:42:15,429 BITSS PROGRAM, BUT I THINK THE 3180 01:42:15,496 --> 01:42:16,563 REAL CAUSE OF IT WAS A PAPER 3181 01:42:16,630 --> 01:42:18,866 PUBLISHED IN THE LATE 2000s BY 3182 01:42:18,932 --> 01:42:20,367 PSYCHOLOGIST DARRELL BEMM 3183 01:42:20,434 --> 01:42:21,602 PURPORTING TO SHOW THE EFFECT OF 3184 01:42:21,668 --> 01:42:23,337 ESP OR EXTRA SENSORY PERCEPTION. 3185 01:42:23,403 --> 01:42:24,738 AND ESSENTIALLY I THINK WHAT 3186 01:42:24,805 --> 01:42:26,740 THIS PAPER DID IS IT LIKE THIS 3187 01:42:26,807 --> 01:42:27,808 DEFINING EVENT THAT SAID WHAT IS 3188 01:42:27,875 --> 01:42:29,877 GOING ON HERE, AND IT LED TO A 3189 01:42:29,943 --> 01:42:33,280 BUNCH OF RESEARCH IN THE SPACE 3190 01:42:33,347 --> 01:42:38,952 INCLUDING URI SIMONSOHON'S WORK. 3191 01:42:39,019 --> 01:42:40,587 THIS IS ACTUALLY A QUESTION FROM 3192 01:42:40,654 --> 01:42:41,922 THE S.A.T. WHERE THEY ACTUALLY 3193 01:42:41,989 --> 01:42:42,990 TALKED ABOUT THIS PAPER. 3194 01:42:43,056 --> 01:42:46,193 SO IT'S KIND OF FUNNY WHERE THEY 3195 01:42:46,260 --> 01:42:49,897 SAY BASED ON THIS PASSAGE, WHICH 3196 01:42:49,963 --> 01:42:52,065 OF THESE FOLLOWING HYPOTHETICAL 3197 01:42:52,132 --> 01:42:53,800 SITUATIONS WOULD MALHOTRA 3198 01:42:53,867 --> 01:42:54,268 STRONGLY OBJECT TO? 3199 01:42:54,334 --> 01:42:55,669 I WOULD SAY THERE'S SUPPOSED TO 3200 01:42:55,736 --> 01:42:57,204 BE ONE ANSWER TO THIS, BUT I 3201 01:42:57,271 --> 01:42:59,606 OBJECT TO MANY OF THESE THINGS. 3202 01:42:59,673 --> 01:43:01,575 SO I WOULD OBJECT THAT IF A 3203 01:43:01,642 --> 01:43:02,442 RESEARCH TEAM EXCLUDES THE 3204 01:43:02,509 --> 01:43:04,311 PORTION OF DATA THAT PRODUCED 3205 01:43:04,378 --> 01:43:07,247 NULL RESULTS, WHICH IS SELECT 3206 01:43:07,314 --> 01:43:08,348 REPORTING, I WOULD ALSO OBJECT 3207 01:43:08,415 --> 01:43:09,650 TO A RESEARCH TEAM UNKNOWINGLY 3208 01:43:09,716 --> 01:43:11,151 REPEATING A STUDY THAT PRODUCED 3209 01:43:11,218 --> 01:43:12,085 NULL RESULTS FOR ANOTHER 3210 01:43:12,152 --> 01:43:13,921 RESEARCH TEAM. 3211 01:43:13,987 --> 01:43:16,857 AND THEN FINALLY, RESEARCH TEAM 3212 01:43:16,924 --> 01:43:18,392 PERFORMS A FOLLOW-UP STUDY THAT 3213 01:43:18,458 --> 01:43:19,493 EXPANDS ON THE SCOPE OF AN 3214 01:43:19,560 --> 01:43:20,928 INITIAL STUDY THAT PRODUCES NULL 3215 01:43:20,994 --> 01:43:21,795 RESULTS. 3216 01:43:21,862 --> 01:43:25,933 I THINK ALL THERE'S THREE OF THE 3217 01:43:25,999 --> 01:43:27,234 THINGS ARE REALLY BAD. 3218 01:43:27,301 --> 01:43:28,869 I'M NOT SURE WHAT THAT SAYS 3219 01:43:28,936 --> 01:43:30,304 ABOUT THE S.A.T. WHEN I CAN'T 3220 01:43:30,370 --> 01:43:31,071 EVEN ANSWER THE QUESTION ABOUT 3221 01:43:31,138 --> 01:43:33,707 MY OWN VIEWS CORRECTLY. 3222 01:43:33,774 --> 01:43:37,978 SO THREE RESEARCHERS, MONIZ, 3223 01:43:38,045 --> 01:43:41,748 DRUKMAN AND FRIES RECENTLY 3224 01:43:41,815 --> 01:43:42,416 REPLICATED THE ORIGINAL SCIENCE 3225 01:43:42,482 --> 01:43:45,586 PAPER WE WROTE IN 2014. 3226 01:43:45,652 --> 01:43:46,587 SO THEY'RE THE ONES WHO 3227 01:43:46,653 --> 01:43:47,788 ADMINISTER THIS PROGRAM FOR THE 3228 01:43:47,854 --> 01:43:48,088 NSF. 3229 01:43:48,155 --> 01:43:49,289 THE IMPORTANT THING THEY HAD WAS 3230 01:43:49,356 --> 01:43:50,824 THEY NOT ONLY HAVE THE PEOPLE 3231 01:43:50,891 --> 01:43:52,225 WHO GOT THESE GRANTS, THEY ALSO 3232 01:43:52,292 --> 01:43:53,427 HAVE THE PEOPLE THEY REJECTED 3233 01:43:53,493 --> 01:43:54,428 FOR THESE GRANTS. 3234 01:43:54,494 --> 01:43:55,629 SO THAT'S AN INTERESTING 3235 01:43:55,696 --> 01:43:56,396 COMPARISON. 3236 01:43:56,463 --> 01:43:58,465 NOW THIS ISN'T LIKE SOME 3237 01:43:58,532 --> 01:43:59,266 REGRESSION DISCONTINUITY DESIGN 3238 01:43:59,333 --> 01:44:01,101 OR SOMETHING, SO -- BUT YOU 3239 01:44:01,168 --> 01:44:02,402 COULD BASICALLY KIND OF COMPARE 3240 01:44:02,469 --> 01:44:05,472 TO SEE HOW THESE PUBLICATIONS 3241 01:44:05,539 --> 01:44:06,907 ASSESS, AND YOU CAN ACTUALLY SEE 3242 01:44:06,974 --> 01:44:08,275 THAT THINGS HAVE GOTTEN A LOT 3243 01:44:08,342 --> 01:44:08,642 BETTER. 3244 01:44:08,709 --> 01:44:11,278 SO IF YOU LOOK AT THAT TOP GAP 3245 01:44:11,345 --> 01:44:12,913 BETWEEN SIGNIFICANT AND NOT 3246 01:44:12,980 --> 01:44:15,315 SIGNIFICANT, IT'S NOT THE 64 TO 3247 01:44:15,382 --> 01:44:16,817 4 RATIO FROM THE PRE-OPEN 3248 01:44:16,883 --> 01:44:17,517 SCIENCE ERA. 3249 01:44:17,584 --> 01:44:19,353 IT'S ONLY ABOUT 28 TO 8. 3250 01:44:19,419 --> 01:44:20,554 SO IT'S SORT OF IN THE EYE OF 3251 01:44:20,621 --> 01:44:22,089 THE BEHOLDER LIKE THAT'S STILL 3252 01:44:22,155 --> 01:44:23,890 NOT PERFECT, SO THE QUESTION IS, 3253 01:44:23,957 --> 01:44:25,292 IS THAT ARE WE GOING TO JUST 3254 01:44:25,359 --> 01:44:27,194 EXPECT THINGS TO GET BETTER OR 3255 01:44:27,260 --> 01:44:28,962 IS THIS LAST MILE GOING TO 3256 01:44:29,029 --> 01:44:29,863 REQUIRE A LOT OF THE STUFF WE'RE 3257 01:44:29,930 --> 01:44:31,598 GOING TO BE TALKING ABOUT AT 3258 01:44:31,665 --> 01:44:33,100 THIS CONFERENCE. 3259 01:44:33,166 --> 01:44:34,001 SO ESSENTIALLY WHAT YOU CAN DO 3260 01:44:34,067 --> 01:44:35,302 IS KIND OF MAKE THESE 3261 01:44:35,369 --> 01:44:36,169 COMPARISONS. 3262 01:44:36,236 --> 01:44:38,105 THIS ISN'T A PERFECT DESIGN BUT 3263 01:44:38,171 --> 01:44:39,439 YOU CAN ESSENTIALLY SAY LET'S 3264 01:44:39,506 --> 01:44:40,540 LOOK AT THE ACCEPTED STUDIES 3265 01:44:40,607 --> 01:44:41,875 BEFORE OPEN SCIENCE AND AFTER 3266 01:44:41,942 --> 01:44:42,209 OPEN SCIENCE. 3267 01:44:42,275 --> 01:44:44,778 YOU CAN SAY WHAT EXPLAINS THAT 3268 01:44:44,845 --> 01:44:45,679 DIFFERENCE IS CULTURAL CHANGE. 3269 01:44:45,746 --> 01:44:47,681 AND THEN YOU CAN LOOK AT THE 3270 01:44:47,748 --> 01:44:49,816 MODERN AFTER OPEN SCIENCE ERA 3271 01:44:49,883 --> 01:44:51,284 AND COMPARE THE ACCEPTED AND 3272 01:44:51,351 --> 01:44:52,252 REJECTED STUDIES. 3273 01:44:52,319 --> 01:44:54,021 SO THESE TEST STUDIES, THEY KNOW 3274 01:44:54,087 --> 01:44:55,422 THEY'RE BEING WATCHED, THEY'RE 3275 01:44:55,489 --> 01:44:56,723 REGISTERED, WHEREAS THE REJECTED 3276 01:44:56,790 --> 01:44:58,158 STUDY, IT'S STILL THE WILD WEST, 3277 01:44:58,225 --> 01:44:59,026 THEY CAN KIND OF DO WHATEVER 3278 01:44:59,092 --> 01:44:59,359 THEY WANT. 3279 01:44:59,426 --> 01:45:00,460 SO YOU CAN SAY IF YOU'RE SUBJECT 3280 01:45:00,527 --> 01:45:01,995 TO THESE INSTITUTIONS, WHAT 3281 01:45:02,062 --> 01:45:04,064 ACTUALLY AFFECTS THE RESULTS? 3282 01:45:04,131 --> 01:45:06,533 SO YOU CAN SEE THAT FOR EXAMPLE, 3283 01:45:06,600 --> 01:45:09,102 OF THE NOT WRITTEN STATUS, 3284 01:45:09,169 --> 01:45:11,405 BEFORE OPEN SCIENCE AND AFTER 3285 01:45:11,471 --> 01:45:12,372 OPEN SCIENCE, I'VE SHOWN THIS, 3286 01:45:12,439 --> 01:45:15,108 YOU SEE THIS MASSIVE CHANGE, 3287 01:45:15,175 --> 01:45:16,410 THIS 60.2 PERCENTAGE POINT GAP 3288 01:45:16,476 --> 01:45:18,412 HAS NARROWED TO ABOUT 20.1 3289 01:45:18,478 --> 01:45:19,279 PERCENTAGE POINTS AND THE 3290 01:45:19,346 --> 01:45:20,714 DIFFERENCE BETWEEN SIGNIFICANCE 3291 01:45:20,781 --> 01:45:22,115 AND NOT SIGNIFICANCE, NOT BEING 3292 01:45:22,182 --> 01:45:22,683 WRITTEN. 3293 01:45:22,749 --> 01:45:23,283 BUT ON THE OTHER HAND, IF YOU 3294 01:45:23,350 --> 01:45:28,588 LOOK HE ILOOK AT THE INSTITUTIOS 3295 01:45:28,655 --> 01:45:31,425 CHANGE, IT'S A LOT SMALLER. 3296 01:45:31,491 --> 01:45:37,764 SO THERE'S A 38.7% BEGAN IN THE 3297 01:45:37,831 --> 01:45:41,034 REJECTED TEST STUDIES, AND -- 3298 01:45:41,101 --> 01:45:41,935 YOU ALSO SEE THE CHANGE THERE 3299 01:45:42,002 --> 01:45:44,237 BUT IF YOU LOOK AT THE 3300 01:45:44,304 --> 01:45:46,273 INSTITUTIONAL CHANGE, IT'S A LOT 3301 01:45:46,339 --> 01:45:46,940 MORE MODEST. 3302 01:45:47,007 --> 01:45:48,809 SO WHAT DOES REGISTRATION DO, IT 3303 01:45:48,875 --> 01:45:50,677 ONLY DECREASES THE GAP BY ABOUT 3304 01:45:50,744 --> 01:45:51,645 8 PERCENTAGE POINTS. 3305 01:45:51,712 --> 01:45:54,047 THERE'S ACTUALLY AN INCREASE IN 3306 01:45:54,114 --> 01:45:55,482 THE PUBLICATION STATUS IF YOU 3307 01:45:55,549 --> 01:45:56,450 ACTUALLY HAVE REGISTRATION. 3308 01:45:56,516 --> 01:45:57,984 SO WHAT DOES THIS SHOW? 3309 01:45:58,051 --> 01:45:59,286 IT SHOWS THAT LIKE MOST OF WHAT 3310 01:45:59,352 --> 01:46:02,889 WE'VE GOTTEN IS THROUGH CULTURAL 3311 01:46:02,956 --> 01:46:03,990 CHANGE POTENTIALLY, AND MAYBE 3312 01:46:04,057 --> 01:46:05,192 NOT AS MUCH ON INSTITUTIONAL 3313 01:46:05,258 --> 01:46:06,526 CHANGE. 3314 01:46:06,593 --> 01:46:08,161 SO THAT'S NOT TO SAY THAT THE 3315 01:46:08,228 --> 01:46:09,463 LAST MILE IS NOT GOING TO BE 3316 01:46:09,529 --> 01:46:11,331 DONE THROUGH INSTITUTIONS, BUT 3317 01:46:11,398 --> 01:46:13,166 MAYBE WE HAVE TO FOCUS A LITTLE 3318 01:46:13,233 --> 01:46:15,035 BIT MORE ON KIND OF HOW THE 3319 01:46:15,102 --> 01:46:17,537 OVERALL CULTURE IS DONE. 3320 01:46:17,604 --> 01:46:18,772 NOW EVEN THOUGH I'M IN THE 3321 01:46:18,839 --> 01:46:21,508 BUSINESS SCHOOL, I DON'T 3322 01:46:21,575 --> 01:46:23,043 ACTUALLY STUDY THIS STUFF BY 3323 01:46:23,110 --> 01:46:24,010 ORGANIZATIONAL RESEARCHERS DO 3324 01:46:24,077 --> 01:46:25,445 STUDY THINGS LIKE HOW DO YOU 3325 01:46:25,512 --> 01:46:26,246 ACHIEVE CULTURAL CHANGE IN 3326 01:46:26,313 --> 01:46:27,114 ORGANIZATIONS WITHOUT 3327 01:46:27,180 --> 01:46:28,348 INSTITUTIONS, AND SO THERE IS 3328 01:46:28,415 --> 01:46:29,516 THEORY MIND THIS, AND MAYBE 3329 01:46:29,583 --> 01:46:30,550 THOSE PEOPLE SHOULD BE INCLUDED 3330 01:46:30,617 --> 01:46:32,352 IN FUTURE MEETINGS, NOT JUST 3331 01:46:32,419 --> 01:46:33,653 META SCIENTISTS AND 3332 01:46:33,720 --> 01:46:34,121 STATISTICIANS. 3333 01:46:34,187 --> 01:46:35,956 AND THEN THE FINAL THING I'LL 3334 01:46:36,022 --> 01:46:36,957 SAY IS THAT SOMETIMES LIKE 3335 01:46:37,023 --> 01:46:38,692 RANDOM EVENTS CAN BE VERY 3336 01:46:38,759 --> 01:46:39,459 POWERFUL. 3337 01:46:39,526 --> 01:46:42,062 SO IT'S KIND OF WEIRD THAT A 3338 01:46:42,129 --> 01:46:44,364 STUDY ABOUT ESP IS THE REASON 3339 01:46:44,431 --> 01:46:45,665 WHY SOCIAL SCIENCE STARTED 3340 01:46:45,732 --> 01:46:47,234 GETTING BETTER AT THIS STUFF, 3341 01:46:47,300 --> 01:46:48,769 BUT IT MIGHT BE THAT FUTURE 3342 01:46:48,835 --> 01:46:49,970 THINGS AND RANDOM EVENTS LIKE 3343 01:46:50,036 --> 01:46:54,207 THAT HAVE TO BE THE NUDGES THAT 3344 01:46:54,274 --> 01:46:55,442 CREATE SOME OF THESE CULTURAL 3345 01:46:55,509 --> 01:46:59,045 CHANGES. 3346 01:46:59,112 --> 01:47:00,580 SO THIS IS INTERESTING AND 3347 01:47:00,647 --> 01:47:01,314 LOOKING FORWARD TO THE 3348 01:47:01,381 --> 01:47:01,615 DISCUSSION. 3349 01:47:01,681 --> 01:47:06,286 >> NEXT WE WILL BE HEARING SOME 3350 01:47:06,353 --> 01:47:07,487 REPORTS FROM ARIELLE 3351 01:47:07,554 --> 01:47:15,195 MARKS-ANGLIN FROM MERCK. 3352 01:47:15,262 --> 01:47:22,502 >> GOOD MORNING. 3353 01:47:22,569 --> 01:47:25,338 I'M ARIELLE MARKS-ANGLIN, SENIOR 3354 01:47:25,405 --> 01:47:26,606 SCIENTIST IN REAL WORLD EVIDENCE 3355 01:47:26,673 --> 01:47:27,073 AT MERCK. 3356 01:47:27,140 --> 01:47:28,942 MY BACKGROUND TRAINING IS IN 3357 01:47:29,009 --> 01:47:30,577 BIOSTATISTICS, AND I'M HERE TO 3358 01:47:30,644 --> 01:47:33,547 GIVE A HISTORICAL AND 3359 01:47:33,613 --> 01:47:35,515 METHODOLOGIC PERSPECTIVE FOR 3360 01:47:35,582 --> 01:47:36,716 QUANTIFYING AND DETECTING 3361 01:47:36,783 --> 01:47:37,818 PUBLICATION BIAS, AND JUST A 3362 01:47:37,884 --> 01:47:39,653 QUICK DISCLAIMER THAT THE VIEWS 3363 01:47:39,719 --> 01:47:41,655 AND OPINIONS I EXPRESS HERE ARE 3364 01:47:41,721 --> 01:47:42,956 MY OWN AND DON'T NECESSARILY 3365 01:47:43,023 --> 01:47:46,293 REFLECT THE VIEWS OF MY EMPLO 3366 01:47:46,359 --> 01:47:50,297 EMPLOYER. 3367 01:47:50,363 --> 01:47:52,165 SO PUBLICATION BIAS, I WOULD 3368 01:47:52,232 --> 01:47:53,667 ARGUE AT LEAST AS WE STUDY IT 3369 01:47:53,733 --> 01:48:00,674 TODAY, IS REALLY A TALE AS OLD 3370 01:48:00,740 --> 01:48:02,742 AS NULL HYPOTHESIS SIGNIFICANCE 3371 01:48:02,809 --> 01:48:02,976 TESTING. 3372 01:48:03,043 --> 01:48:05,478 IN THE EARLY 100s, RONALD 3373 01:48:05,545 --> 01:48:07,447 FISHER PIERSON AND NAYMAN TRIED 3374 01:48:07,514 --> 01:48:10,050 TO MAKE MATHEMATICAL STATISTICS 3375 01:48:10,116 --> 01:48:11,718 MORE ACCESSIBLE TO RESEARCHERS, 3376 01:48:11,785 --> 01:48:14,020 SO THEY DEVELOPED, NOT TOGETHER, 3377 01:48:14,087 --> 01:48:16,656 VERY SEPARATELY, THEY DEVELOPED 3378 01:48:16,723 --> 01:48:19,392 A HYPOTHESIS TETHER TESTING 3379 01:48:19,459 --> 01:48:20,560 FRAME WORKS AND INTERTRE 3380 01:48:20,627 --> 01:48:21,728 PAITIONS OF THE P VALUE THAT CAN 3381 01:48:21,795 --> 01:48:23,396 BE USED MORE WIDELY BY 3382 01:48:23,463 --> 01:48:25,599 RESEARCHERS IN DIFFERENT FIELDS. 3383 01:48:25,665 --> 01:48:30,270 AND FISHER WAS PARTICULARLY 3384 01:48:30,337 --> 01:48:32,639 INSTRUMENTAL IN CATEGORIZING THE 3385 01:48:32,706 --> 01:48:35,642 SIGNIFICANCE OF STUDIES BASED ON 3386 01:48:35,709 --> 01:48:39,246 THAT .05 SIGNIFICANCE THRESHOLD. 3387 01:48:39,312 --> 01:48:40,881 THEN FROM THERE, ESPECIALLY IN 3388 01:48:40,947 --> 01:48:42,616 THE FIELD OF PSYCHOLOGY BUT ALSO 3389 01:48:42,682 --> 01:48:46,753 FOOD SCIENCE AND OTHER RESEARCH 3390 01:48:46,820 --> 01:48:48,154 AREAS, THE RESEARCHERS ADOPTED 3391 01:48:48,221 --> 01:48:49,556 THE MODERN FORM OF NULL 3392 01:48:49,623 --> 01:48:52,792 HYPOTHESIS SIGNIFICANCE TESTING, 3393 01:48:52,859 --> 01:48:54,427 AND PROFESSIONAL JOURNALS AT THE 3394 01:48:54,494 --> 01:48:55,829 TIME EVEN REQUIRED SIGNIFICANT P 3395 01:48:55,896 --> 01:48:57,163 VALUES FOR DISSEMINATION. 3396 01:48:57,230 --> 01:49:00,734 SO FAST FORWARD TO AS EARLY AS 3397 01:49:00,800 --> 01:49:02,869 1959, WHEN WE HAVE META-ANALYSIS 3398 01:49:02,936 --> 01:49:03,904 AND EVIDENCE SYNTHESIS STARTING 3399 01:49:03,970 --> 01:49:05,372 TO COME INTO THE RESEARCH 3400 01:49:05,438 --> 01:49:09,209 SPHERE, WE SEE EARLY INDICATIONS 3401 01:49:09,276 --> 01:49:10,944 OF SELECTIVE PUBLICATION BASED 3402 01:49:11,011 --> 01:49:16,650 ON THE SIGNIFICANCE OF RESULTS. 3403 01:49:16,716 --> 01:49:19,719 WHAT THIS HAS LED TO IS A 3404 01:49:19,786 --> 01:49:20,820 DECADES-LONG MISSING DATA 3405 01:49:20,887 --> 01:49:21,087 CHALLENGE. 3406 01:49:21,154 --> 01:49:24,224 HERE WE SEE A META-ANALYSIS OF 3407 01:49:24,291 --> 01:49:25,292 PSYCHOTHERAPY STUDIES SPANNING 3408 01:49:25,358 --> 01:49:27,427 DECADES -- I BELIEVE THE EARLY 3409 01:49:27,494 --> 01:49:28,261 STUDY INCLUDED IN THIS WAS IN 3410 01:49:28,328 --> 01:49:30,664 THE 60s, BUT WHAT YOU SEE HERE 3411 01:49:30,730 --> 01:49:31,865 ARE FUNNEL PLOTS AND THE IDEA IS 3412 01:49:31,932 --> 01:49:33,566 THAT IN THE ABSENCE OF 3413 01:49:33,633 --> 01:49:37,003 PUBLICATION BIAS, THE X AXIS 3414 01:49:37,070 --> 01:49:38,772 SHOWS THE EFFECT SIZE AND YOU 3415 01:49:38,838 --> 01:49:40,507 WOULD EXPECT AN EVEN 3416 01:49:40,573 --> 01:49:44,244 DISTRIBUTION OF STUDIES OF -- 3417 01:49:44,311 --> 01:49:45,879 AROUND WHAT IS LIKELY THE TRUE 3418 01:49:45,946 --> 01:49:46,746 EFFECT SIZE. 3419 01:49:46,813 --> 01:49:49,015 BUT HERE WE SEE A SKEWNESS 3420 01:49:49,082 --> 01:49:51,618 TOWARDS MORE EXTREME STUDIES, 3421 01:49:51,685 --> 01:49:54,321 EXTREME EFFECTS, PARTICULARLY 3422 01:49:54,387 --> 01:49:55,855 FOR STUDYS THAT HAVE SMALLER 3423 01:49:55,922 --> 01:49:59,793 SAMPLE SIZES. 3424 01:49:59,859 --> 01:50:01,127 AND SO WHAT DO WE DO WHEN WE 3425 01:50:01,194 --> 01:50:02,529 HAVE A MISSING DATA CHALLENGE? 3426 01:50:02,595 --> 01:50:04,931 HOW DO YOU QUANTIFY THE UNKNOWN? 3427 01:50:04,998 --> 01:50:07,667 AND IN STATISTICS, TYPICALLY 3428 01:50:07,734 --> 01:50:09,869 THAT MEANS MAKING ASSUMPTIONS. 3429 01:50:09,936 --> 01:50:11,805 WE CAN DRAW FROM PATTERNS WE SEE 3430 01:50:11,871 --> 01:50:13,039 IN THE OBSERVED DATA. 3431 01:50:13,106 --> 01:50:14,607 WE CAN FORMALIZE THE KNOWLEDGE 3432 01:50:14,674 --> 01:50:16,676 AND BELIEFS WE HAVE ABOUT THE 3433 01:50:16,743 --> 01:50:18,545 PUBLICATION PROCESS. 3434 01:50:18,611 --> 01:50:19,546 AND WE'D ALSO LIKE TO ACCOUNT 3435 01:50:19,612 --> 01:50:20,714 FOR OTHER PROCESSES THAT MAY 3436 01:50:20,780 --> 01:50:25,752 LEAD TO THE OBSERVED DATA. 3437 01:50:25,819 --> 01:50:29,055 SO I DID A HISTORICAL REVIEW OF 3438 01:50:29,122 --> 01:50:31,558 THESE METHODS AND I WOULD ARGUE 3439 01:50:31,624 --> 01:50:33,460 THAT YOU CAN CLASSIFY THE 3440 01:50:33,526 --> 01:50:34,961 QUANTITATIVE METHODS FOR 3441 01:50:35,028 --> 01:50:37,797 PUBLICATION BIAS INTO TWO BROAD 3442 01:50:37,864 --> 01:50:38,264 CLASSES. 3443 01:50:38,331 --> 01:50:41,634 ONE OF WHICH IS GRAPH BASED. 3444 01:50:41,701 --> 01:50:44,037 AND WHICH WE INCLUDE THIS -- 3445 01:50:44,104 --> 01:50:45,805 THAT INCLUDES THIS FUNNEL PLOD I 3446 01:50:45,872 --> 01:50:48,541 SHOWED PREVIOUSLY. 3447 01:50:48,608 --> 01:50:49,909 AND THE CENTRAL ASSUMPTION FOR 3448 01:50:49,976 --> 01:50:53,680 THAT IS THAT -- IS AN ASSUMPTION 3449 01:50:53,747 --> 01:50:56,316 CALL SYMMETRY, WHERE YOU ASSUME 3450 01:50:56,383 --> 01:50:57,784 A RANDOM SYMMETRIC DISTRIBUTION 3451 01:50:57,851 --> 01:50:59,386 OF STUDY EFFECT SIZES AROUND THE 3452 01:50:59,452 --> 01:51:01,454 TRUE EFFECTS. 3453 01:51:01,521 --> 01:51:04,190 AND THAT'S WHERE YOU GET METHODS 3454 01:51:04,257 --> 01:51:06,726 SUCH AS EGGERS TEST, TRIM AND 3455 01:51:06,793 --> 01:51:08,028 FILL, METHODS THAT YOU MAY HAVE 3456 01:51:08,094 --> 01:51:10,964 HEARD OF AND EVEN USED WIDELY IN 3457 01:51:11,031 --> 01:51:12,699 META-ANALYSES OR SYSTEMATIC 3458 01:51:12,766 --> 01:51:13,033 REVIEWS. 3459 01:51:13,099 --> 01:51:14,134 AND THEY'RE VERY BENEFICIAL 3460 01:51:14,200 --> 01:51:15,869 BECAUSE THEY'RE INTUITIVE AND 3461 01:51:15,935 --> 01:51:16,302 USER-FRIENDLY. 3462 01:51:16,369 --> 01:51:17,037 THEY'VE BEEN AROUND FOR A WHILE 3463 01:51:17,103 --> 01:51:20,440 SO WE HAVE VALIDATED CODE 3464 01:51:20,507 --> 01:51:21,641 PACKAGES AVAILABLE. 3465 01:51:21,708 --> 01:51:25,278 HOWEVER, THEY ARE PRONE TO 3466 01:51:25,345 --> 01:51:26,846 MISINTERPRETATION BECAUSE 3467 01:51:26,913 --> 01:51:28,248 PUBLICATION BIAS IS NOT THE ONLY 3468 01:51:28,314 --> 01:51:31,084 SOURCE OF ASYMMETRY YOU MIGHT 3469 01:51:31,151 --> 01:51:32,519 FIND IN A FUNNEL PLOT. 3470 01:51:32,585 --> 01:51:35,688 AND I THINK KNE NEIL ALLUDED TO 3471 01:51:35,755 --> 01:51:36,923 THIS, BUT DEPENDING ON STUDY 3472 01:51:36,990 --> 01:51:38,658 QUALITY OR HETEROGENEITY IN YOUR 3473 01:51:38,725 --> 01:51:40,727 STUDY DESIGNS, YOU CAN ALSO GET 3474 01:51:40,794 --> 01:51:41,761 DIFFERENT EFFECTS WHEN YOU'RE 3475 01:51:41,828 --> 01:51:43,163 TRYING TO ANSWER THE SAME 3476 01:51:43,229 --> 01:51:46,332 RESEARCH QUESTION. 3477 01:51:46,399 --> 01:51:47,300 SO ONE CHALLENGE IN THESE 3478 01:51:47,367 --> 01:51:49,069 METHODS IS HOW TO DISTINGUISH 3479 01:51:49,135 --> 01:51:50,303 BETWEEN PUBLICATION BIAS AND 3480 01:51:50,370 --> 01:51:52,005 OTHER SOURCES OF WHAT WE CALL 3481 01:51:52,072 --> 01:51:53,440 SMALL STUDY EFFECTS OR 3482 01:51:53,506 --> 01:51:55,075 ASYMMETRY. 3483 01:51:55,141 --> 01:51:56,509 ON THE OTHER HAND, WE HAVE 3484 01:51:56,576 --> 01:51:59,779 SELECTION MODELS, IN WHICH YOU 3485 01:51:59,846 --> 01:52:01,748 USE MATHEMATICAL MODELS TO 3486 01:52:01,815 --> 01:52:04,184 OUTLINE THE ASSUMPTIONS ABOUT 3487 01:52:04,250 --> 01:52:05,251 THE PUBLICATION PROCESS AND THEN 3488 01:52:05,318 --> 01:52:07,020 TRY TO ESTIMATE AN EFFECT BASED 3489 01:52:07,087 --> 01:52:11,391 ON THOSE MODELS. 3490 01:52:11,458 --> 01:52:13,059 AND THEY'RE VERY FLEXIBLE, YOU 3491 01:52:13,126 --> 01:52:14,494 CAN DEFINE MORE COMPLEX 3492 01:52:14,561 --> 01:52:15,695 PROCESSES, SO AT THE CRUDEST 3493 01:52:15,762 --> 01:52:18,031 LEVEL, YOU CAN ASSUME THAT 3494 01:52:18,098 --> 01:52:20,400 STUDIES WITH A P VALUELESS THAN 3495 01:52:20,467 --> 01:52:21,734 .05 ARE PUBLISHED WITH A 3496 01:52:21,801 --> 01:52:23,169 PROBABILITY OF 1. 3497 01:52:23,236 --> 01:52:25,038 AND IF THEY'RE GREATER THAN .05, 3498 01:52:25,105 --> 01:52:26,239 THEY'RE PUBLISHED WITH A 3499 01:52:26,306 --> 01:52:27,574 PROBABILITY OF ZERO. 3500 01:52:27,640 --> 01:52:31,377 AND OVER TIME, STATISTICIANS 3501 01:52:31,444 --> 01:52:33,680 HAVE DEVELOPED MORE COMPLEX 3502 01:52:33,746 --> 01:52:35,682 MODELS, THEY'VE TRIED TO EXTEND 3503 01:52:35,748 --> 01:52:36,749 TO MULTIVARIABLE SETTINGS, AND 3504 01:52:36,816 --> 01:52:38,952 THEY HELP TO EASILY DISTINGUISH 3505 01:52:39,018 --> 01:52:39,919 BETWEEN PUBLICATION BIAS AND 3506 01:52:39,986 --> 01:52:42,222 OTHER SOURCES OF ASYMMETRY OR 3507 01:52:42,288 --> 01:52:44,324 SMALL STUDY EFFECTS. 3508 01:52:44,390 --> 01:52:45,425 HOWEVER, BECAUSE THESE ARE 3509 01:52:45,492 --> 01:52:49,028 COMPLEX MODELS, AND 3510 01:52:49,095 --> 01:52:49,662 META-ANALYSES OFTEN HAVE VERY 3511 01:52:49,729 --> 01:52:53,099 SMALL SAMPLE SIZES, YOU CAN RUN 3512 01:52:53,166 --> 01:52:55,168 INTO COMPUTATION ISSUES, 3513 01:52:55,235 --> 01:52:57,237 IDENTIFIABILITY ISSUES. 3514 01:52:57,303 --> 01:53:01,174 THEY DON'T OFTEN -- THEY'RE NOT 3515 01:53:01,241 --> 01:53:03,143 OFTEN ACCOMPANIED BY CODE 3516 01:53:03,209 --> 01:53:04,577 PACKAGES SO THEY'RE LESS 3517 01:53:04,644 --> 01:53:06,012 ACCESSIBLE, SO YOU TEND TO NEED 3518 01:53:06,079 --> 01:53:07,113 A DEEPER KNOWLEDGE OF STATISTICS 3519 01:53:07,180 --> 01:53:08,982 IN ORDER TO IMPLEMENT THEM. 3520 01:53:09,048 --> 01:53:12,919 AND SO THERE'S NOT AN IDEAL 3521 01:53:12,986 --> 01:53:13,419 SOLUTION. 3522 01:53:13,486 --> 01:53:15,021 I WILL SKIP OVER THIS NEXT SLIDE 3523 01:53:15,088 --> 01:53:17,624 DUE TO TIME, BUT IF YOU'RE 3524 01:53:17,690 --> 01:53:18,825 INTERESTED, I DO INCLUDE IN THE 3525 01:53:18,892 --> 01:53:20,894 PAPER A TIMELINE OF THE 3526 01:53:20,960 --> 01:53:23,763 METHODOLOGIC DEVELOPMENTS, BUT . 3527 01:53:23,830 --> 01:53:25,298 WHAT YOU CAN GATHER FROM THIS IS 3528 01:53:25,365 --> 01:53:26,266 WE ARE CONTINUOUSLY ITERATING 3529 01:53:26,332 --> 01:53:27,800 AND TRYING TO IMPROVE UPON 3530 01:53:27,867 --> 01:53:30,904 METHODS FOR QUANTIFYING, 3531 01:53:30,970 --> 01:53:33,206 DETECTING, AND CORRECTING FOR 3532 01:53:33,273 --> 01:53:36,042 PUBLICATION BIAS, AS WE CONTINUE 3533 01:53:36,109 --> 01:53:41,314 TO IDENTIFY NEW COMPLICATIONS OR 3534 01:53:41,381 --> 01:53:42,382 NUANCES IN THE MISSING DATA 3535 01:53:42,448 --> 01:53:44,484 PROCESS. 3536 01:53:44,551 --> 01:53:45,785 SO I WOULD SAY THAT KEY 3537 01:53:45,852 --> 01:53:46,553 CHALLENGES FOR ANY 3538 01:53:46,619 --> 01:53:47,887 QUANTIFICATION OR DETECTION 3539 01:53:47,954 --> 01:53:51,057 METHOD FROM AT -- AT LEAST FOR 3540 01:53:51,124 --> 01:53:52,258 STATISTICAL METHODS IS THE 3541 01:53:52,325 --> 01:53:53,259 VALIDITY OF THE ASSUMPTIONS 3542 01:53:53,326 --> 01:53:55,128 MADE, HOW TO ISOLATE PUBLICATION 3543 01:53:55,195 --> 01:53:59,332 BIAS FROM OTHER SOUR SOURCES OF 3544 01:53:59,399 --> 01:54:00,400 ASYMMETRY, THE INTERPRETABILITY 3545 01:54:00,466 --> 01:54:01,434 OF YOUR APPROACH, HOW WELL YOU 3546 01:54:01,501 --> 01:54:02,602 CAN ESTIMATE THE PARAMETERS OF 3547 01:54:02,669 --> 01:54:04,804 YOUR MODEL, AND ALSO THE 3548 01:54:04,871 --> 01:54:08,208 COMPUTATIONAL FEASIBILITY. 3549 01:54:08,274 --> 01:54:10,076 I WOULD PARTICULARLY LIKE TO 3550 01:54:10,143 --> 01:54:12,478 HIGHLIGHT THESE TWO BECAUSE IF 3551 01:54:12,545 --> 01:54:13,913 WE'RE TRYING TO MAKE THIS MORE 3552 01:54:13,980 --> 01:54:16,316 ACCESSIBLE TO OTHERS, 3553 01:54:16,382 --> 01:54:17,750 COMPUTATIONAL FEASIBILITY IS A 3554 01:54:17,817 --> 01:54:18,418 MUST. 3555 01:54:18,484 --> 01:54:28,928 I THINK OUR NEXT SPEE SPEAK UR,, 3556 01:54:28,995 --> 01:54:30,330 HAS DONE EXCELLENT WORK IN THIS 3557 01:54:30,396 --> 01:54:32,799 AREA WITH HIS METHODOLOGIC 3558 01:54:32,865 --> 01:54:33,533 DEVELOPMENTS. 3559 01:54:33,600 --> 01:54:35,068 BUT IT'S IMPORTANT TO KNOW THAT 3560 01:54:35,134 --> 01:54:36,402 THESE ARE VERY AD HOC 3561 01:54:36,469 --> 01:54:36,836 APPROACHES. 3562 01:54:36,903 --> 01:54:38,271 THEY CAN HELP US IDENTIFY THE 3563 01:54:38,338 --> 01:54:39,806 PROBLEM, DO SENSITIVITY ANALYSES 3564 01:54:39,872 --> 01:54:42,175 BUT THEY'RE BY NO MEANS A 3565 01:54:42,242 --> 01:54:42,875 SUFFICIENT SOLUTION, SO I'M GLAD 3566 01:54:42,942 --> 01:54:44,310 THIS WORKSHOP IS TALKING ABOUT 3567 01:54:44,377 --> 01:54:45,144 PREVENTION BECAUSE THAT'S 3568 01:54:45,211 --> 01:54:45,878 ULTIMATELY WHERE THE SOLUTION 3569 01:54:45,945 --> 01:54:49,415 NEEDS TO COME FROM. 3570 01:54:49,482 --> 01:54:52,285 AND IN MY REDUCTIONIST 3571 01:54:52,352 --> 01:54:54,988 PERSPECTIVE, I SEE THAT THERE IS 3572 01:54:55,054 --> 01:54:56,856 A DESIRE FOR ACTIONABLE RESULTS 3573 01:54:56,923 --> 01:54:59,292 FROM RESEARCH TO ADVANCE THE 3574 01:54:59,359 --> 01:55:00,493 FIELD. 3575 01:55:00,560 --> 01:55:02,795 WE INVEST OUR TIME, WE INVEST 3576 01:55:02,862 --> 01:55:04,664 FUNDING INTO STUDIES BECAUSE WE 3577 01:55:04,731 --> 01:55:06,232 HOPE TO ADVANCE A CERTAIN AREA 3578 01:55:06,299 --> 01:55:09,168 OF KNOWLEDGE. 3579 01:55:09,235 --> 01:55:10,770 AND I ULTIMATELY THINK THAT'S A 3580 01:55:10,837 --> 01:55:11,571 GOOD DESIRE. 3581 01:55:11,638 --> 01:55:14,841 WE ALSO DESIRE GREATER RIGOR AND 3582 01:55:14,907 --> 01:55:15,975 OBJECTIVITY IN EXPERIMENTAL 3583 01:55:16,042 --> 01:55:17,043 STUDIES. 3584 01:55:17,110 --> 01:55:19,812 THAT'S WHY THESE EARLIER 3585 01:55:19,879 --> 01:55:22,515 STATISTICIANS, FISHER, NEYMAN 3586 01:55:22,582 --> 01:55:24,584 AND PEARSON, TRIED TO DEVELOP AN 3587 01:55:24,651 --> 01:55:28,655 EASY TO USE RULE FOR CLASSIFYING 3588 01:55:28,721 --> 01:55:29,522 EVIDENCE. 3589 01:55:29,589 --> 01:55:30,757 HOWEVER, WE'VE BECOME TOO 3590 01:55:30,823 --> 01:55:32,392 RELIANT ON IT. 3591 01:55:32,458 --> 01:55:34,127 WE DETERMINE WHETHER A STUDY 3592 01:55:34,193 --> 01:55:36,763 STANDS OR FALLS BASED ON WHETHER 3593 01:55:36,829 --> 01:55:37,964 YOU REJECT THE NULL BASED ON 3594 01:55:38,031 --> 01:55:40,266 WHETHER YOU HAVE THE P VALUE 3595 01:55:40,333 --> 01:55:41,601 SURPASSES OR GOES BELOW THIS 3596 01:55:41,668 --> 01:55:45,004 THRESHOLD OF .05, WHICH WAS 3597 01:55:45,071 --> 01:55:45,972 REALLY ONLY CONVENIENTLY 3598 01:55:46,039 --> 01:55:48,808 SELECTED BY FISHER BECAUSE IT 3599 01:55:48,875 --> 01:55:51,344 CORRESPONDS TO TWO STANDARD 3600 01:55:51,411 --> 01:55:52,312 DEVIATIONS FROM THE MEAN ON A 3601 01:55:52,378 --> 01:55:55,281 BELL CURVE. 3602 01:55:55,348 --> 01:55:58,685 I WOULD ENCOURAGE US TO CONSIDER 3603 01:55:58,751 --> 01:56:00,520 PERHAPS CAN WE RE-EVALUATE HOW 3604 01:56:00,586 --> 01:56:03,723 WE DEFINE EVIDENCE, OR TO TED'S 3605 01:56:03,790 --> 01:56:07,994 TALK EARLIER, HOW WE DEFINE 3606 01:56:08,061 --> 01:56:08,861 VALUE OF A STUDY. 3607 01:56:08,928 --> 01:56:10,963 THERE ARE OTHER MEASURES AND 3608 01:56:11,030 --> 01:56:13,032 FRAME WORKS APART FROM NULL 3609 01:56:13,099 --> 01:56:15,601 HYPOTHESIS TESTING, APART FROM P 3610 01:56:15,668 --> 01:56:17,337 VALUES, OR EVEN IF STICKING WITH 3611 01:56:17,403 --> 01:56:19,439 P VALUES AND CONFIDENCE 3612 01:56:19,505 --> 01:56:23,009 INTERVALS CAN WE MOVE AWAY FROM 3613 01:56:23,076 --> 01:56:26,546 HONING IN ON THIS .05 THRESHOLD 3614 01:56:26,612 --> 01:56:28,181 AND CLASSIFYING STUDIES ABOVE 3615 01:56:28,247 --> 01:56:29,916 THAT AS NULL STUDIES, WHICH I'M 3616 01:56:29,982 --> 01:56:31,484 GLAD SOMEONE POINTED OUT EARLIER 3617 01:56:31,551 --> 01:56:35,421 IS A BIT OF A PROBLEMATIC TERM. 3618 01:56:35,488 --> 01:56:37,056 SO THAT'S MY PERSPECTIVE WHERE 3619 01:56:37,123 --> 01:56:37,924 I'M COMING FROM. 3620 01:56:37,990 --> 01:56:44,163 THANK YOU. 3621 01:56:44,230 --> 01:56:46,799 >> THANK YOU SO MUCH. 3622 01:56:46,866 --> 01:56:51,504 SO NEXT, URI SIMONSOHON FROM 3623 01:56:51,571 --> 01:56:53,806 RAMON LLULL UNIVERSITY FROM 3624 01:56:53,873 --> 01:56:55,641 SPAIN AND YOU MAY ALSO KNOW HIM 3625 01:56:55,708 --> 01:56:56,209 FROM THE BLOG. 3626 01:56:56,275 --> 01:56:58,845 HE WAS UNABLE TO COME IN PERSON 3627 01:56:58,911 --> 01:57:00,246 BECAUSE OF A CONFLICT, BUT HE 3628 01:57:00,313 --> 01:57:01,614 DID SUBMIT A VIDEO AND WE WILL 3629 01:57:01,681 --> 01:57:05,118 BE WATCHING THAT NOW. 3630 01:57:05,184 --> 01:57:05,752 >> HI. 3631 01:57:05,818 --> 01:57:06,919 BECAUSE I COULDN'T MAKE THE 3632 01:57:06,986 --> 01:57:07,720 CONFERENCE, THEY ASKED ME TO 3633 01:57:07,787 --> 01:57:09,389 MAKE A FIVE-MINUTE VIDEO, WHICH 3634 01:57:09,455 --> 01:57:10,890 I'LL DO AND I'LL TALK ABOUT 3635 01:57:10,957 --> 01:57:12,291 THREE PERSPECTIVES ON 3636 01:57:12,358 --> 01:57:17,096 PUBLICATION BIAS I THINK MAY BE 3637 01:57:17,163 --> 01:57:20,600 SOMEWHAT NON-MAJORITY VIEWS AND 3638 01:57:20,666 --> 01:57:22,268 SO MAY BE BIGGER VALUE SHOWING 3639 01:57:22,335 --> 01:57:22,568 THOSE. 3640 01:57:22,635 --> 01:57:24,771 THE FIRST ONE IF YOU HAD TO 3641 01:57:24,837 --> 01:57:25,838 CHOOSE BETWEEN -- 3642 01:57:25,905 --> 01:57:26,572 >> TURN IT UP. 3643 01:57:26,639 --> 01:57:29,809 >> AND I SEE LITTLE BENEFIT IN 3644 01:57:29,876 --> 01:57:31,310 ADDRESSING PUBLICATION BIAS 3645 01:57:31,377 --> 01:57:32,645 WITHOUT ALSO ADDRESSING P 3646 01:57:32,712 --> 01:57:33,780 HACKING. 3647 01:57:33,846 --> 01:57:34,580 SO LET ME CONTRAST THE 3648 01:57:34,647 --> 01:57:36,282 DEFINITION OF THE TWO. 3649 01:57:36,349 --> 01:57:38,885 PUBLICATION BIAS IS WHEN WE 3650 01:57:38,951 --> 01:57:40,319 SELECTED THE REPORT IN ENTIRE 3651 01:57:40,386 --> 01:57:41,053 STUDIES THAT WORK IN A 3652 01:57:41,120 --> 01:57:41,988 PARTICULAR WAY, FOR EXAMPLE, IF 3653 01:57:42,054 --> 01:57:46,259 THERE WERE STATISTIC -- P 3654 01:57:46,325 --> 01:57:48,861 HACKING, SELECTIVE REPORTING OF 3655 01:57:48,928 --> 01:57:49,962 PREFERRED ANALYSIS. 3656 01:57:50,029 --> 01:57:51,397 SO CONSEQUENCES OF HAVING MORE 3657 01:57:51,464 --> 01:57:53,132 STUDIES BEING AVAILABLE BUT NOT 3658 01:57:53,199 --> 01:57:54,600 MAKING SURE THE ANALYSIS OF 3659 01:57:54,667 --> 01:57:56,002 THOSE STUDIES WERE THE ONES THAT 3660 01:57:56,068 --> 01:58:00,640 WERE INTENDED OR COMPLEMENTARY, 3661 01:58:00,706 --> 01:58:01,808 THE LEVEL OF BIAS IN THE 3662 01:58:01,874 --> 01:58:04,110 LITERATURE COULD EASILY INCREASE 3663 01:58:04,177 --> 01:58:11,317 OR STAY THE SAME BECAUSE -- TH 3664 01:58:11,384 --> 01:58:13,953 THE -- IT DOESN'T MEAN -- YOU 3665 01:58:14,020 --> 01:58:17,056 CAN EASILY P-HACK A STUDY 3666 01:58:17,123 --> 01:58:18,991 SUCCESSFULLY LET'S SAY TO 3667 01:58:19,058 --> 01:58:21,961 PUBLISH P VALUES, DOESN'T MATTER 3668 01:58:22,028 --> 01:58:25,031 THE -- SIZE, FROM .4 TO .3, SO 3669 01:58:25,097 --> 01:58:27,133 STILL NOT SIGNIFICANT. 3670 01:58:27,200 --> 01:58:28,568 IF YOU FORCE ME TO REVEAL THEM, 3671 01:58:28,634 --> 01:58:30,670 I SHOW YOU THE .3 BASED ON BIAS 3672 01:58:30,736 --> 01:58:32,138 ESTIMATE, DIVIDE THE CORRECT -- 3673 01:58:32,205 --> 01:58:35,908 OF THE EFFECT SIZE -- .4 AND 3674 01:58:35,975 --> 01:58:37,109 SOME -- OBSERVATION THAT'S BIAS 3675 01:58:37,176 --> 01:58:39,078 AND THE EXISTING -- STUDIES THAT 3676 01:58:39,145 --> 01:58:42,048 ARE PUBLISHED REMAINS THERE. 3677 01:58:42,114 --> 01:58:44,684 IT'S ALSO VERY HARD AS I'LL 3678 01:58:44,750 --> 01:58:47,820 ARGUE LATER TO -- PUBLICATION 3679 01:58:47,887 --> 01:58:49,489 BIAS WHILE IT'S VERY EASY TO 3680 01:58:49,555 --> 01:58:50,990 ADDRESS P-HACKING. 3681 01:58:51,057 --> 01:58:54,994 ON THE OTHER HAND, IF YOU REDUCE 3682 01:58:55,061 --> 01:58:57,763 P-HACKING AND -- PUBLICATION BUY 3683 01:58:57,830 --> 01:58:59,966 A THE BIAS ESTIMATES THAT ARE IN 3684 01:59:00,032 --> 01:59:03,202 THE LITERATURE WILL -- MANY WILL 3685 01:59:03,269 --> 01:59:05,738 DISAPPEAR BECAUSE WITHOUT 3686 01:59:05,805 --> 01:59:06,806 P-HACKING -- REPORTED. 3687 01:59:06,873 --> 01:59:08,274 NOW YOU HAVE TO WORRY ABOUT IS 3688 01:59:08,341 --> 01:59:10,243 WHETHER THE ONES THAT ARE 3689 01:59:10,309 --> 01:59:13,613 BIASED-FREE ARE SIMPLY BIASED 3690 01:59:13,679 --> 01:59:14,747 BECAUSE 1 IN 20 STUDIES ARE 3691 01:59:14,814 --> 01:59:16,916 WORKING BY CHANCE BUT IT'S HARD 3692 01:59:16,983 --> 01:59:24,156 TO HAVE A SUCCESSFUL KEER WHERER 3693 01:59:24,223 --> 01:59:26,526 WHERE 1 IN 20 STUDIES WORK. 3694 01:59:26,592 --> 01:59:29,161 -- MORE OF THE STUDIES BEING 3695 01:59:29,228 --> 01:59:30,696 PUBLISHED BECAUSE OF THE KIND OF 3696 01:59:30,763 --> 01:59:35,101 STUDIES THAT ARE BEING RUN. 3697 01:59:35,167 --> 01:59:36,435 THE SECOND POINT, AND THIS 3698 01:59:36,502 --> 01:59:38,070 RELATES TO THE FIRST ONE, IS 3699 01:59:38,137 --> 01:59:39,572 THAT PREREGISTRATION IS BETTER 3700 01:59:39,639 --> 01:59:43,209 AND MORE BENEFICIAL THAN 3701 01:59:43,276 --> 01:59:44,510 REGISTRATION. 3702 01:59:44,577 --> 01:59:47,480 -- MADE AN IMPACT IN MEDICAL 3703 01:59:47,547 --> 01:59:50,650 RESEARCH INCLUDED IN CLINICAL 3704 01:59:50,716 --> 01:59:52,518 TRIALS, CLINICAL TRIALS ARE 3705 01:59:52,585 --> 01:59:53,719 GOVERNING, SO PREREGISTRATION IS 3706 01:59:53,786 --> 01:59:58,090 WHEN YOU THINK OF THE PERFECT -- 3707 01:59:58,157 --> 02:00:01,160 YOU PERFECTLY STIPULATE HOW 3708 02:00:01,227 --> 02:00:03,262 EVERY ANALYSIS WE CONDUCTED, 3709 02:00:03,329 --> 02:00:04,697 EVERY ASPECT -- HOW LONG IT WILL 3710 02:00:04,764 --> 02:00:06,432 LAST AND SO ON. 3711 02:00:06,499 --> 02:00:07,934 BUT IN REALITY, NOTHING IS 3712 02:00:08,000 --> 02:00:08,200 PERFECT. 3713 02:00:08,267 --> 02:00:11,170 SO YOU HAVE IMPERFECT 3714 02:00:11,237 --> 02:00:11,971 PREREGISTRATION, YOU STILL HAVE 3715 02:00:12,038 --> 02:00:14,674 A MEANINGFUL REDUCTION OF 3716 02:00:14,740 --> 02:00:15,641 P-HACKING, EVERYTHING YOU 3717 02:00:15,708 --> 02:00:17,076 STIPULATED YOU DON'T P-HACK AND 3718 02:00:17,143 --> 02:00:18,911 EVERYTHING YOU DON'T STIPULATE, 3719 02:00:18,978 --> 02:00:21,147 IT IS EXPLICITLY NOT STIPULATED 3720 02:00:21,213 --> 02:00:22,782 SO THE READER CAN IDENTIFY 3721 02:00:22,848 --> 02:00:24,750 THINGS THAT WERE AND WERE NOT 3722 02:00:24,817 --> 02:00:26,419 P-HACKED SO IT'S HIGHLY USEFUL 3723 02:00:26,485 --> 02:00:29,689 EVEN IF IMPERFECT. 3724 02:00:29,755 --> 02:00:31,357 NOW REGISTRATION, REGISTRATION 3725 02:00:31,424 --> 02:00:33,492 IS WHEN YOU STIPULATE AHEAD OF 3726 02:00:33,559 --> 02:00:36,596 TIME THAT YOU WILL RUN A STUDY 3727 02:00:36,662 --> 02:00:44,837 SO PEOPLE KNOW THAT YOU EAX AM D 3728 02:00:44,904 --> 02:00:47,907 THE IMPACT OF THIS PARTICULAR 3729 02:00:47,974 --> 02:00:49,842 TREATMENT, WHY DO WE WANT THAT, 3730 02:00:49,909 --> 02:00:51,844 WHY DO WE WANT PEOPLE TO TELL 3731 02:00:51,911 --> 02:00:53,479 THAWS AHEAD OF TIME, BECAUSE OF 3732 02:00:53,546 --> 02:00:55,247 PUBLICATION BIAS, WE VEN TIRE 3733 02:00:55,314 --> 02:00:56,782 STUDIES MISSING, BUT FOR THAT TO 3734 02:00:56,849 --> 02:01:00,620 BE USEFUL, DOWN THE LINE 3735 02:01:00,686 --> 02:01:01,921 SOMEBODY IS GOING TO BE FINDING 3736 02:01:01,988 --> 02:01:03,456 ALL THOSE STUDIES THAT WERE RUN 3737 02:01:03,522 --> 02:01:06,292 AND WERE PUBLISHED BR FOW -- 3738 02:01:06,359 --> 02:01:07,393 ANALYST TO DO ANYTHING WITH THE 3739 02:01:07,460 --> 02:01:08,527 REGISTRATION, THEY NEED TO KNOW 3740 02:01:08,594 --> 02:01:10,196 NOT JUST THAT YOU PLANNED TO RUN 3741 02:01:10,262 --> 02:01:11,797 THE STUDY BUT ALSO WHETHER YOU 3742 02:01:11,864 --> 02:01:14,367 ACTUALLY RUN IT, HOW YOU 3743 02:01:14,433 --> 02:01:16,002 EXECUTED IT AND WHAT THE RESULTS 3744 02:01:16,068 --> 02:01:17,770 ARE, AND THAT HAPPENS VERY 3745 02:01:17,837 --> 02:01:19,505 RARELY. 3746 02:01:19,572 --> 02:01:22,575 IT HAPPENS VERY RARELY IN A STU 3747 02:01:22,642 --> 02:01:24,877 DAY THAT FAILS AND IS NOT 3748 02:01:24,944 --> 02:01:26,512 PUBLISHED, THE AUTHOR NEVER GOES 3749 02:01:26,579 --> 02:01:27,213 THROUGH THE TROUBLE OF SHOWING 3750 02:01:27,279 --> 02:01:28,648 ALL THE DETAILS OF THE STUDY. 3751 02:01:28,714 --> 02:01:30,383 SO THAT MEANS HAVING PERFECT 3752 02:01:30,449 --> 02:01:31,250 REGISTRATION WHERE PEOPLE 3753 02:01:31,317 --> 02:01:33,486 RESEARCH STUDIES AND -- METHODS, 3754 02:01:33,552 --> 02:01:35,354 THIS HAS ALMOST NO VALUE IN 3755 02:01:35,421 --> 02:01:38,858 CONTRAST TO IMPERFECT 3756 02:01:38,924 --> 02:01:39,692 PREREGISTRATION WHICH HAS HIGH 3757 02:01:39,759 --> 02:01:40,426 VALUE. 3758 02:01:40,493 --> 02:01:41,727 SO IF YOU REALLY NEED A 3759 02:01:41,794 --> 02:01:42,695 REGISTRY, IF YOU REALLY NEED 3760 02:01:42,762 --> 02:01:45,765 PEOPLE TO TELL YOU AHEAD OF TIME 3761 02:01:45,831 --> 02:01:48,100 WHICH STUDIES THEY RUN BECAUSE 3762 02:01:48,167 --> 02:01:49,335 THEY NEED TO KNOW THE OUTCOME, 3763 02:01:49,402 --> 02:01:53,005 WHAT YOU REALLY NEED IS -- THE 3764 02:01:53,072 --> 02:01:53,406 RORLTS. 3765 02:01:53,472 --> 02:01:55,041 PUBLICATION PROCESS WHERE PEOPLE 3766 02:01:55,107 --> 02:01:56,942 WILL PRE-COMMIT TO MAKING THE 3767 02:01:57,009 --> 02:01:57,877 RESULTS PUBLIC, REGARDLESS OF 3768 02:01:57,943 --> 02:01:59,245 THE OUTCOME. 3769 02:01:59,311 --> 02:02:00,846 THAT COULD BE IN THE WAY THEY 3770 02:02:00,913 --> 02:02:02,715 CURRENTLY WORK, WHICH IS A 3771 02:02:02,782 --> 02:02:04,583 JOURNAL HAS A COMMITMENT TO THE 3772 02:02:04,650 --> 02:02:06,585 AUTHORS TO PUBLISH IT BUT IT 3773 02:02:06,652 --> 02:02:08,988 COULD ALSO BE SOMETHING LIKE 3774 02:02:09,055 --> 02:02:10,856 CLINICALTRIALS.GOV REQUIRES THAT 3775 02:02:10,923 --> 02:02:13,259 AUTHORS -- THE INFORMATION, FOR 3776 02:02:13,325 --> 02:02:14,060 EXAMPLE, PREVENTING THEM FROM 3777 02:02:14,126 --> 02:02:16,228 CONDUCTING NEW REGISTRIES -- NEW 3778 02:02:16,295 --> 02:02:18,531 REGISTRATIONS IF THEY DON'T 3779 02:02:18,597 --> 02:02:19,799 HAVE -- THIS IS VERY IMPORTANT 3780 02:02:19,865 --> 02:02:20,766 AS FAR AS I KNOW HAS NOT BEEN 3781 02:02:20,833 --> 02:02:23,069 REALLY ADDRESSED. 3782 02:02:23,135 --> 02:02:24,937 AND THE LAST THING I'LL SAY IS 3783 02:02:25,004 --> 02:02:26,572 THAT META-ANALYSIS REALLY CANNOT 3784 02:02:26,639 --> 02:02:28,574 FIX PUBLICATION GU BIAS. 3785 02:02:28,641 --> 02:02:30,109 SO THE HOPE IS TO HAVE A REJ 3786 02:02:30,176 --> 02:02:31,343 INDUSTRIAL TO FIX IT. 3787 02:02:31,410 --> 02:02:32,211 META-ANALYSIS CANNOT REALLY 3788 02:02:32,278 --> 02:02:34,780 DELIVER FOR A FEW REASONS. 3789 02:02:34,847 --> 02:02:36,182 FIRST, META-ANALYSIS EXACERBATES 3790 02:02:36,248 --> 02:02:37,817 BIAS FROM P-HACKING IF EVERY 3791 02:02:37,883 --> 02:02:42,555 STUDY HAS A RI LITTLE BIT OF 3792 02:02:42,621 --> 02:02:46,592 P-HACKING THE -- IT'S VERY 3793 02:02:46,659 --> 02:02:48,461 DIFFICULT GIVEN -- STUDIES AND 3794 02:02:48,527 --> 02:02:50,463 LACK OF DETAIL IN THE ORIGINAL, 3795 02:02:50,529 --> 02:02:54,500 QUALITY CONTROL, AND THEY 3796 02:02:54,567 --> 02:02:56,135 INEVITABLY ADD LOW QUALITY 3797 02:02:56,202 --> 02:02:57,403 STUDIES THAT WEREN'T PUBLISHED 3798 02:02:57,470 --> 02:02:58,904 PERHAPS BECAUSE OF NO RESULTS, 3799 02:02:58,971 --> 02:03:00,005 BECAUSE PERHAPS BECAUSE OF 3800 02:03:00,072 --> 02:03:02,608 QUALITY CONCERNS. 3801 02:03:02,675 --> 02:03:05,811 THIS IS -- REVIEW, ALMOST ANY 3802 02:03:05,878 --> 02:03:07,046 SERIOUS META AL LIST WILL KNOW 3803 02:03:07,113 --> 02:03:09,782 THAT SMALL STUDY BIAS ISN'T 3804 02:03:09,849 --> 02:03:11,417 PUBLICATION BIAS, SO THE 3805 02:03:11,484 --> 02:03:13,185 FINAL -- OTHER TECHNIQUES AREN'T 3806 02:03:13,252 --> 02:03:14,820 REALLY DETECTING PUBLICATION 3807 02:03:14,887 --> 02:03:19,425 BIAS -- VERY NOISY TOOLS THAT 3808 02:03:19,492 --> 02:03:22,061 HAS A LOT OF WISHFUL THINKING -- 3809 02:03:22,128 --> 02:03:23,062 PUBLICATION BIAS DETECTION. 3810 02:03:23,129 --> 02:03:25,364 THEN MAIN TOOLS LIKE TRIM-AND 3811 02:03:25,431 --> 02:03:29,435 TRIM-AND-FILL, THEY HAVEN'T 3812 02:03:29,502 --> 02:03:30,970 REALLY BEEN VALIDATED, DOESN'T 3813 02:03:31,036 --> 02:03:32,371 WORK AT ALL FOR PUBLICATION 3814 02:03:32,438 --> 02:03:32,571 BIAS. 3815 02:03:32,638 --> 02:03:35,407 SO IN SUMMARY, P-HACKING IS A 3816 02:03:35,474 --> 02:03:36,575 BIGGER PROBLEM THAN PUBLICATION 3817 02:03:36,642 --> 02:03:39,445 BIAS AND IS EASIER TO ADDRESS. 3818 02:03:39,512 --> 02:03:42,214 PUBLICATION BIAS CANNOT BE FIXED 3819 02:03:42,281 --> 02:03:44,450 WITH REGISTRY, META-ANALYSIS, IF 3820 02:03:44,517 --> 02:03:45,785 ONE MUST MIX IT, YOU NEED A 3821 02:03:45,851 --> 02:03:52,792 REGISTERED REPORT. 3822 02:03:52,858 --> 02:03:53,526 >> ALL RIGHT. 3823 02:03:53,592 --> 02:03:56,128 NEXT WE WILL HAVE SOME BRIEF 3824 02:03:56,195 --> 02:03:58,764 REMARKS FROM LAUREN VON 3825 02:03:58,831 --> 02:04:03,502 KLINGGRAEFF FROM AW AUGUSTA 3826 02:04:03,569 --> 02:04:12,378 UNIVERSITY. 3827 02:04:12,444 --> 02:04:13,879 >> WE HAD SOME GUIDING QUESTIONS 3828 02:04:13,946 --> 02:04:16,048 ABOUT THINGS WE CAN SPEAK TO 3829 02:04:16,115 --> 02:04:16,882 DURING SESSION ONE, AND THE 3830 02:04:16,949 --> 02:04:18,350 FIRST QUESTION BEING WHY OR 3831 02:04:18,417 --> 02:04:21,387 WHETHER IT IS IMPORTANT TO 3832 02:04:21,453 --> 02:04:23,489 ADDRESS PUBLICATION BIAS OR 3833 02:04:23,556 --> 02:04:24,023 QUANTIFY PUBLICATION BIAS. 3834 02:04:24,089 --> 02:04:26,559 AND I WOULD ARGUE THAT IN MY 3835 02:04:26,625 --> 02:04:28,761 FIELD, IT IS VERY IMPORTANT TO 3836 02:04:28,828 --> 02:04:30,229 QUANTIFY PUBLICATION BIAS AND TO 3837 02:04:30,296 --> 02:04:31,230 KEEP AN EYE ON IT. 3838 02:04:31,297 --> 02:04:33,399 SO I WORK IN THE FIELD OF 3839 02:04:33,465 --> 02:04:35,167 POPULATION LEVEL HEALTH BEHAVIOR 3840 02:04:35,234 --> 02:04:36,502 INTERVENTIONS, AND WHEN WE GO TO 3841 02:04:36,569 --> 02:04:40,439 DESIGN AN INTERVENTION TO HELP 3842 02:04:40,506 --> 02:04:42,308 PEOPLE STOP SMOKING OR EAT MORE 3843 02:04:42,374 --> 02:04:44,844 FRUITS AND VEGETABLES WE USUALLY 3844 02:04:44,910 --> 02:04:46,278 FOLLOW MOD DELLS LIKE THIS, THE 3845 02:04:46,345 --> 02:04:50,449 NIH STAGE MODEL OR NHLBI ORBIT 3846 02:04:50,516 --> 02:04:52,418 MODEL WHERE YOU DO A PRELIMINARY 3847 02:04:52,484 --> 02:04:53,986 FEASIBILITY TEST TO SEE WHETHER 3848 02:04:54,053 --> 02:04:55,554 THAT INTERVENTION HAS ANYTHING 3849 02:04:55,621 --> 02:04:57,323 WORTH PURSUING BEFORE YOU GO ON 3850 02:04:57,389 --> 02:04:59,258 AND RE-TEST IT ITTIVELY IN 3851 02:04:59,325 --> 02:05:01,894 HOPEFULLY LARGER SAMPLE SIZES. 3852 02:05:01,961 --> 02:05:03,195 THE ISSUE BEING THAT WITH THESE 3853 02:05:03,262 --> 02:05:04,430 EARLY SMALL STUDIES WE KNOW 3854 02:05:04,496 --> 02:05:06,532 THEY'RE UNDERPOWERED AND WE KNOW 3855 02:05:06,599 --> 02:05:08,400 EARLY STUDIES ARE LIKELY TO HAVE 3856 02:05:08,467 --> 02:05:09,602 LARGER EFFECT SIZES THAN THEY 3857 02:05:09,668 --> 02:05:11,337 MIGHT BE OTHERWISE IF THEY WERE 3858 02:05:11,403 --> 02:05:12,571 AT A DIFFERENT POINT IN THE 3859 02:05:12,638 --> 02:05:13,172 DEVELOPMENTAL PROCESS. 3860 02:05:13,239 --> 02:05:14,707 SO IF YOU TAKE THAT AND COMBINE 3861 02:05:14,773 --> 02:05:16,008 IT WITH PUBLICATION BIAS, WHAT 3862 02:05:16,075 --> 02:05:17,009 YOU COULD HAVE IS A SITUATION 3863 02:05:17,076 --> 02:05:19,345 WHERE YOU ARE PROMOTING AND 3864 02:05:19,411 --> 02:05:20,646 POSSIBLY INVESTING MORE INTO 3865 02:05:20,713 --> 02:05:21,447 THESE INTERVENTIONS THAT MAYBE 3866 02:05:21,513 --> 02:05:22,982 ARE BASELESS AND WE SHOULDN'T BE 3867 02:05:23,048 --> 02:05:23,949 TESTING IN THE FIRST PLACE 3868 02:05:24,016 --> 02:05:25,951 BECAUSE THAT POSITIVE RESULT IS 3869 02:05:26,018 --> 02:05:29,955 JUST COMING FROM A TYPE I ERROR. 3870 02:05:30,022 --> 02:05:32,124 SO THEN THE QUESTION IS HOW DO 3871 02:05:32,191 --> 02:05:33,259 WE QUANTIFY WHETHER OR NOT THIS 3872 02:05:33,325 --> 02:05:34,660 IS HAPPENING IN PRELIMINARY 3873 02:05:34,727 --> 02:05:36,528 BEHAVIORAL INTERVENTIONS? 3874 02:05:36,595 --> 02:05:40,499 WE DID A STUDY AND IT'S RIGHT 3875 02:05:40,566 --> 02:05:41,700 THERE, WHERE WE WENT AND JUST 3876 02:05:41,767 --> 02:05:44,436 LIKE YOU CAN LOOK AT THE FDA 3877 02:05:44,503 --> 02:05:46,872 REGISTRIES OR STUDY REGISTRIES, 3878 02:05:46,939 --> 02:05:47,840 PREREGISTRIES, YOU CAN LOOK AT 3879 02:05:47,907 --> 02:05:49,275 ABSTRACTS THAT ARE PRESENTED AT 3880 02:05:49,341 --> 02:05:50,442 CONFERENCES. 3881 02:05:50,509 --> 02:05:52,845 SO WE WENT AND LOOKED AT TWO 3882 02:05:52,912 --> 02:05:54,046 CONFERENCES, THE INTERNATIONAL 3883 02:05:54,113 --> 02:05:55,047 SOCIETY OF BEHAVIORAL NUTRITION 3884 02:05:55,114 --> 02:05:56,649 AND PHYSICAL ACTIVITY AND THE 3885 02:05:56,715 --> 02:05:57,816 SOCIETY OF BEHAVIORAL MEDICINE, 3886 02:05:57,883 --> 02:05:59,084 WHICH ARE TWO LARGE CONFERENCES 3887 02:05:59,151 --> 02:05:59,985 IN OUR FIELD. 3888 02:06:00,052 --> 02:06:02,354 WE LOOKED THROUGH ABOUT 18,000 3889 02:06:02,421 --> 02:06:05,624 CONFERENCE ABSTRACTS AND OF 3890 02:06:05,691 --> 02:06:06,692 THOSE, 800 WERE PRELIMINARY 3891 02:06:06,759 --> 02:06:08,394 BEHAVIORAL INTERVENTIONS. 3892 02:06:08,460 --> 02:06:09,795 WE THEN WENT THROUGH AND CODED 3893 02:06:09,862 --> 02:06:11,096 THOSE ABSTRACTS FOR LOTS OF 3894 02:06:11,163 --> 02:06:13,198 VARIABLES, BUT THE MAIN ONES 3895 02:06:13,265 --> 02:06:15,301 BEING WERE THEY RANDOMIZED 3896 02:06:15,367 --> 02:06:16,302 CONTROL TRIALS, WHAT WAS THEIR 3897 02:06:16,368 --> 02:06:18,270 SAMPLE SIZE, AND DID THEY HAVE 3898 02:06:18,337 --> 02:06:19,605 SOME FORM OF SIGNIFICANCE, AND 3899 02:06:19,672 --> 02:06:22,608 THIS IS SORT OF A MORE GENERAL 3900 02:06:22,675 --> 02:06:23,676 DEFINITION OF SIGNIFICANCE THAT 3901 02:06:23,742 --> 02:06:25,311 YOU MIGHT THINK, BEING THAT 3902 02:06:25,377 --> 02:06:26,946 ABSTRACTS ARE SHORT. 3903 02:06:27,012 --> 02:06:29,415 IF THE AUTHORS OF THE ABSTRACT 3904 02:06:29,481 --> 02:06:30,849 SAID THEY HAVE SOMETHING 3905 02:06:30,916 --> 02:06:32,017 SIGNIFICANT AND THEY'RE RESULTS, 3906 02:06:32,084 --> 02:06:32,818 THAT'S WHAT WE TOOK. 3907 02:06:32,885 --> 02:06:34,286 WE DIDN'T REQUIRE THEM TO HAVE A 3908 02:06:34,353 --> 02:06:36,789 P VALUE OR CONFIDENCE INTERVAL. 3909 02:06:36,855 --> 02:06:39,425 WE THEN TRACE THOSE FORWARD IN 3910 02:06:39,491 --> 02:06:40,693 TIME TO SEE IF THEY HAVE EVER 3911 02:06:40,759 --> 02:06:42,628 BEEN PUBLISHED AS FULL LENGTH 3912 02:06:42,695 --> 02:06:43,929 MANUSCRIPTS AND WHAT WE 3913 02:06:43,996 --> 02:06:45,564 DISCOVERED WAS THAT 50% OF ALL 3914 02:06:45,631 --> 02:06:46,165 PRELIMINARY BEHAVIORAL 3915 02:06:46,231 --> 02:06:47,833 INTERVENTIONS IN OUR FIELD GO 3916 02:06:47,900 --> 02:06:50,769 UNPUBLISHED, WHICH IS PRETTY 3917 02:06:50,836 --> 02:06:51,236 WEIGHTY. 3918 02:06:51,303 --> 02:06:52,471 WHEN WE WENT TO GO TEST WHETHER 3919 02:06:52,538 --> 02:06:54,640 WE SAW ANY INDICATION OF 3920 02:06:54,707 --> 02:06:55,841 PUBLICATION BIAS FAVORING 3921 02:06:55,908 --> 02:06:57,943 STATISTICAL SIGNIFICANCE, WHAT 3922 02:06:58,010 --> 02:07:01,246 WE FOUND IS THAT INITIALLY IT 3923 02:07:01,313 --> 02:07:02,681 APPEARS BUT ONCE WE CONTROLLED 3924 02:07:02,748 --> 02:07:04,583 FOR WHETHER STUDIES WERE RCT AND 3925 02:07:04,650 --> 02:07:05,584 WHAT THEIR SAMPLE SIZE WAS, THAT 3926 02:07:05,651 --> 02:07:09,188 THAT EVIDENCE OF PUBLICATION 3927 02:07:09,254 --> 02:07:11,423 BIAS -- AND STATISTICAL 3928 02:07:11,490 --> 02:07:12,191 SIGNIFICANCE DISAPPEARED SO I 3929 02:07:12,257 --> 02:07:14,360 THINK THAT KIND OF GOES BACK TO 3930 02:07:14,426 --> 02:07:15,260 THIS IDEA THAT IF YOU CONTROL 3931 02:07:15,327 --> 02:07:19,665 FOR STUDY QUALITY, PERHAPS 3932 02:07:19,732 --> 02:07:20,332 STATISTICAL SIGNIFICANT ISN'T 3933 02:07:20,399 --> 02:07:21,533 PLAYING QUITE AS BIG OF A ROLE 3934 02:07:21,600 --> 02:07:23,769 IN PUBLICATION BIAS. 3935 02:07:23,836 --> 02:07:24,737 SO THAT'S JUST ONE STUDY. 3936 02:07:24,803 --> 02:07:27,172 IF YOU GO AND YOU LOOK AT THE 3937 02:07:27,239 --> 02:07:27,773 LARGEST COLLECTION OF STUDIES 3938 02:07:27,840 --> 02:07:29,174 THAT USE THIS METHOD OF 3939 02:07:29,241 --> 02:07:30,275 CONFERENCE ABSTRACTS FOLLOW 3940 02:07:30,342 --> 02:07:33,178 FORWARD TO STUDY THE PUBLICATION 3941 02:07:33,245 --> 02:07:36,348 PROCESS, YOU'LL FIND THIS 3942 02:07:36,415 --> 02:07:38,650 REVIEW -- THIS META-ANALYSIS IN 3943 02:07:38,717 --> 02:07:39,184 SYSTEMATIC REVIEW. 3944 02:07:39,251 --> 02:07:42,187 SO THEY HAVE 461 STUDIES IN HERE 3945 02:07:42,254 --> 02:07:44,456 THAT USE THAT ABSTRACT FOLLOW 3946 02:07:44,523 --> 02:07:44,923 FORWARD PROCESS. 3947 02:07:44,990 --> 02:07:48,127 OF THOSE, ONLY 64 ACCOUNT FOR 3948 02:07:48,193 --> 02:07:48,994 WHETHER STATISTICAL SIGNIFICANCE 3949 02:07:49,061 --> 02:07:52,264 PLAYS A ROLE. 3950 02:07:52,331 --> 02:07:54,233 THEN IF YOU LOOK AND YOU SAY 3951 02:07:54,299 --> 02:07:56,502 DOES STATISTICAL SIGNIFICANCE 3952 02:07:56,568 --> 02:07:58,337 PREDICT AN INCREASED RISK OF 3953 02:07:58,404 --> 02:07:59,805 PUBLICATION, THE ANSWER IS YES. 3954 02:07:59,872 --> 02:08:01,473 BUT THEN IF YOU GO AND ONLY LOOK 3955 02:08:01,540 --> 02:08:04,610 AT RCTs AND QUASI EXPERIMENTAL 3956 02:08:04,676 --> 02:08:05,677 STUDIES, YOU SEE A LITTLE BIT OF 3957 02:08:05,744 --> 02:08:06,445 A DROP. 3958 02:08:06,512 --> 02:08:07,546 UNFORTUNATELY THEY DON'T HAVE 3959 02:08:07,613 --> 02:08:09,581 ANY SUBANALYSES THAT LOOK AT 3960 02:08:09,648 --> 02:08:10,983 SAMPLE SIZE AND STUDY DESIGN 3961 02:08:11,050 --> 02:08:12,484 ALONG WITH SIGNIFICANCE. 3962 02:08:12,551 --> 02:08:14,253 SO THE QUESTION BECOMES, ARE 3963 02:08:14,319 --> 02:08:15,454 THESE GOOD ESTIMATES OF 3964 02:08:15,521 --> 02:08:16,321 PUBLICATION BIAS? 3965 02:08:16,388 --> 02:08:18,390 AND I SAY YES, I DO THINK THE 3966 02:08:18,457 --> 02:08:19,691 PUBLICATION BIAS EXISTS, BUT I 3967 02:08:19,758 --> 02:08:22,127 THINK WE HAVE TO TAKE IT IN PART 3968 02:08:22,194 --> 02:08:23,896 WITH STUDY DESIGN, AND MY 3969 02:08:23,962 --> 02:08:25,764 ARGUMENT FOR THAT BEING THAT OUR 3970 02:08:25,831 --> 02:08:27,166 STUDY AND OTHER STUDIES THAT USE 3971 02:08:27,232 --> 02:08:28,967 THIS METHOD, WE GO ON AND WE 3972 02:08:29,034 --> 02:08:30,002 INTERVIEW OR SURVEY THE AUTHORS 3973 02:08:30,069 --> 02:08:31,637 AND SAY, HEY, WHY DIDN'T YOU 3974 02:08:31,703 --> 02:08:33,172 PUBLISH YOUR STUDY? 3975 02:08:33,238 --> 02:08:34,506 AND CONSISTENTLY AND I CAN 3976 02:08:34,573 --> 02:08:35,707 PROVIDE CITATIONS FOR THESE, WE 3977 02:08:35,774 --> 02:08:39,011 HAVE THE SAME ANSWER, SO AS A 3978 02:08:39,078 --> 02:08:40,012 RESEARCHER, WHEN YOU'VE GOT YOUR 3979 02:08:40,079 --> 02:08:41,780 RESULTS, YOU'RE GOING TO SIT 3980 02:08:41,847 --> 02:08:43,449 THERE AND THINK, WELL, HOW 3981 02:08:43,515 --> 02:08:45,084 DIFFICULT IS THIS GOING TO BE TO 3982 02:08:45,150 --> 02:08:46,085 PUBLISH, AND CONSIDERING ALL THE 3983 02:08:46,151 --> 02:08:47,052 OTHER RESPONSIBILITIES I HAVE, 3984 02:08:47,119 --> 02:08:49,188 IS IT WORTH THE EFFORT? 3985 02:08:49,254 --> 02:08:50,823 AND I THINK THAT IF YOU JUST 3986 02:08:50,889 --> 02:08:53,125 THINK ABOUT STATISTICAL 3987 02:08:53,192 --> 02:08:54,493 SIGNIFICANCE, THAT'S NOT THE 3988 02:08:54,560 --> 02:08:54,793 FULL STORY. 3989 02:08:54,860 --> 02:08:56,261 IF I'VE JUST DONE A SMALL SINGLE 3990 02:08:56,328 --> 02:08:58,864 ARM STUDY THAT WAS NULL OR SIG 3991 02:08:58,931 --> 02:09:00,666 KARNGHTS CONVINCING SOMEONE THAT 3992 02:09:00,732 --> 02:09:02,167 THERE'S MEANING IN THAT DATA IS 3993 02:09:02,234 --> 02:09:03,268 MUCH MORE DIFFICULT, BUT IF I 3994 02:09:03,335 --> 02:09:04,937 DID A LARGE RANDOMIZED CONTROL 3995 02:09:05,003 --> 02:09:07,039 TRIAL AND I DIDN'T HAVE 3996 02:09:07,106 --> 02:09:08,240 SIGNIFICANCE, I STILL COULD 3997 02:09:08,307 --> 02:09:09,108 CONVINCE SOMEONE THERE'S MEANING 3998 02:09:09,174 --> 02:09:10,976 IN THAT AND HOPEFULLY GET IT 3999 02:09:11,043 --> 02:09:11,243 PUBLISHED. 4000 02:09:11,310 --> 02:09:12,978 SO I THINK THAT WHEN WE TAKE 4001 02:09:13,045 --> 02:09:14,746 INTO ACCOUNT STATISTICAL 4002 02:09:14,813 --> 02:09:16,381 SIGNIFICANCE AND PUBLICATION 4003 02:09:16,448 --> 02:09:17,216 BIAS, WE ALSO NEED TO THINK 4004 02:09:17,282 --> 02:09:17,950 ABOUT STUDY QUALITY. 4005 02:09:18,016 --> 02:09:24,323 THAT'S ALL I HAVE. 4006 02:09:24,389 --> 02:09:25,991 >> THANK YOU VERY MUCH. 4007 02:09:26,058 --> 02:09:36,835 AND LAST BUT ABC LEET NOT LEASTT 4008 02:09:38,003 --> 02:09:39,171 LEAST IS VIRGINIA 4009 02:09:39,238 --> 02:09:42,641 ARECHAVALA-GOMEZA FROM 4010 02:09:42,708 --> 02:09:45,611 BIOBIZKAIA. 4011 02:09:45,677 --> 02:09:54,086 >>SO IN MY REAL I'VE, I AM A 4012 02:09:54,153 --> 02:09:55,721 RESEARCHER THAT HAS MADE HER 4013 02:09:55,787 --> 02:09:57,956 CAREER INTO QUANTIFYING 4014 02:09:58,023 --> 02:09:58,590 PROTEINS. 4015 02:09:58,657 --> 02:10:01,360 SO I DON'T HAVE ANY DATA HERE 4016 02:10:01,426 --> 02:10:09,334 TODAY, SO I FEEL A BIT LIKE A 4017 02:10:09,401 --> 02:10:11,403 SAMPLE OF AN INVESTIGATOR,, BUT 4018 02:10:11,470 --> 02:10:12,538 I'M GOING TO TAKE THOSE FIVE 4019 02:10:12,604 --> 02:10:14,206 MINUTES TO TELL YOU ABOUT HOW 4020 02:10:14,273 --> 02:10:15,841 HAPPY I AM TO BE HERE, BECAUSE I 4021 02:10:15,908 --> 02:10:17,209 THINK BEING ABLE TO PUBLISH 4022 02:10:17,276 --> 02:10:18,310 NEGATIVE RESULTS IS SOMETHING WE 4023 02:10:18,377 --> 02:10:22,447 SHOULD ALL STRIVE TO. 4024 02:10:22,514 --> 02:10:24,850 SO I'M HERE AS A RESEARCHER AND 4025 02:10:24,917 --> 02:10:25,884 I'M ONE OF THOSE RESEARCHERS 4026 02:10:25,951 --> 02:10:27,219 THAT IS EXCELLENT, AND I THINK 4027 02:10:27,286 --> 02:10:29,087 ALL OF US HERE ARE EXCELLENT. 4028 02:10:29,154 --> 02:10:31,757 BECAUSE RIGHT NOW, YOU CANNOT 4029 02:10:31,823 --> 02:10:34,259 NOT BE EXCELLENT TO DO RESEARCH. 4030 02:10:34,326 --> 02:10:35,794 IT'S EXCELLENT SCIENCE HERE, 4031 02:10:35,861 --> 02:10:36,995 EXCELLENT SCIENCE THERE. 4032 02:10:37,062 --> 02:10:38,597 AND IT'S IMPOSSIBLE TO DO IT 4033 02:10:38,664 --> 02:10:40,866 OTHERWISE. 4034 02:10:40,933 --> 02:10:41,867 SO WE'RE EXCELLENT. 4035 02:10:41,934 --> 02:10:43,602 AND I THINK WE HAVE TO STRIVE TO 4036 02:10:43,669 --> 02:10:47,039 BE PROFICIENT. 4037 02:10:47,105 --> 02:10:47,673 THAT'S PRETTY MUCH WHAT WE CAN 4038 02:10:47,739 --> 02:10:49,441 TO. 4039 02:10:49,508 --> 02:10:49,975 DO. 4040 02:10:50,042 --> 02:10:51,310 I DON'T THINK WE CAN ALL BE 4041 02:10:51,376 --> 02:10:57,049 EXCELLENT, I DON -- THIS IS 4042 02:10:57,115 --> 02:10:57,916 IMPOSSIBLE TO LIVE UP TO THAT 4043 02:10:57,983 --> 02:10:58,250 IDEA. 4044 02:10:58,317 --> 02:10:59,685 AND I THINK WE ALL STARTED DOING 4045 02:10:59,751 --> 02:11:00,819 RESEARCH BECAUSE WE WANTED 4046 02:11:00,886 --> 02:11:01,720 ANSWERS AND WE WANT THE WORLD TO 4047 02:11:01,787 --> 02:11:02,621 BE A BETTER PLACE. 4048 02:11:02,688 --> 02:11:03,855 IF YOU THINK ABOUT THIS AS A 4049 02:11:03,922 --> 02:11:04,823 COMMUNITY BEING IN THE MIDDLE OF 4050 02:11:04,890 --> 02:11:06,725 THIS MAZE, WHICH THERE IS NO 4051 02:11:06,792 --> 02:11:09,895 LIGHT, WE KIND OF SET OUT 4052 02:11:09,962 --> 02:11:11,630 TOGETHER TO SEE WHICH IS THE 4053 02:11:11,697 --> 02:11:13,131 RIGHT WAY FORWARD, AND THEN WE 4054 02:11:13,198 --> 02:11:13,899 RECONVENE BECAUSE THIS IS A 4055 02:11:13,966 --> 02:11:16,835 LARGE PLACE AND WE NEED TO TELL, 4056 02:11:16,902 --> 02:11:17,869 OKAY, THIS WAY IS GOING THERE 4057 02:11:17,936 --> 02:11:20,072 AND THEN WE NEED TO MAP IT AND 4058 02:11:20,138 --> 02:11:21,406 LEAVE THE TORCHES ON SO 4059 02:11:21,473 --> 02:11:22,708 EVERYBODY KNOWS WHERE WE HAVE TO 4060 02:11:22,774 --> 02:11:24,343 GO AND HOW TO GET THERE. 4061 02:11:24,409 --> 02:11:26,311 AND THAT INFORMATION IS 4062 02:11:26,378 --> 02:11:27,179 VALUABLE, WHETHER IT'S THE RIGHT 4063 02:11:27,246 --> 02:11:28,981 WAY OR THE WRONG WAY. 4064 02:11:29,047 --> 02:11:29,581 AND I THINK THAT'S SOMETHING 4065 02:11:29,648 --> 02:11:33,819 THAT WE TEND TO FORGET. 4066 02:11:33,885 --> 02:11:39,925 SO I WAS A BIT DISTRACTED WHEN I 4067 02:11:39,992 --> 02:11:41,493 READ THIS, IT WAS A PAPER YEARS 4068 02:11:41,560 --> 02:11:44,029 BACK SAYING THAT -- CONFIRMING 4069 02:11:44,096 --> 02:11:47,499 THE HYPOTHESIS THAT INCREASING 4070 02:11:47,566 --> 02:11:48,500 NOW 85% OF THE PUBLISHED RESULTS 4071 02:11:48,567 --> 02:11:49,167 WERE POSITIVE RESULTS. 4072 02:11:49,234 --> 02:11:53,105 SO I DID A VERY UNSCIENTIFIC 4073 02:11:53,171 --> 02:11:56,108 STUDY ON BIOARCHIVE, AND I JUST 4074 02:11:56,174 --> 02:11:58,510 CHECKED, BECAUSE THEY HAPPEN TO 4075 02:11:58,577 --> 02:11:59,978 HAVE CONTRADICTORY, CONFIRMATORY 4076 02:12:00,045 --> 02:12:02,848 AND NEW RESULTS, SO IT WAS A 4077 02:12:02,914 --> 02:12:04,283 PRETTY EASY THING TO DO, IN 10 4078 02:12:04,349 --> 02:12:04,516 MINUTES. 4079 02:12:04,583 --> 02:12:08,453 SO YOU SEE TONS OF DATA, AND 4080 02:12:08,520 --> 02:12:09,554 THEN YOU SEE THAT ACTUALLY 4081 02:12:09,621 --> 02:12:11,089 CONFIRMATORY AND CONTRADICTORY 4082 02:12:11,156 --> 02:12:12,124 RESULTS DO INCREASE OVER THE 4083 02:12:12,190 --> 02:12:16,461 YEARS, BUT WHEN YOU SEE THE NEW 4084 02:12:16,528 --> 02:12:17,262 RESULTS, IT'S A VERY SMALL 4085 02:12:17,329 --> 02:12:19,331 AMOUNT. 4086 02:12:19,398 --> 02:12:22,668 AND IT'S ONLY AROUND .3 OR 1% OF 4087 02:12:22,734 --> 02:12:23,335 ALL THE DATA. 4088 02:12:23,402 --> 02:12:24,770 WHICH IS NOT ENOUGH. 4089 02:12:24,836 --> 02:12:26,204 OBVIOUSLY WE DON'T KNOW THE 4090 02:12:26,271 --> 02:12:28,707 RESULT WHEN WE START DOING SOME 4091 02:12:28,774 --> 02:12:31,810 EXPERIMENTS, IT'S NOT 85% OF THE 4092 02:12:31,877 --> 02:12:32,678 TIME WE KNOW ALREADY WHAT'S 4093 02:12:32,744 --> 02:12:33,011 HAPPENING. 4094 02:12:33,078 --> 02:12:35,447 SO THAT COMES BACK TO 4095 02:12:35,514 --> 02:12:36,081 REPRODUCIBILITY. 4096 02:12:36,148 --> 02:12:38,317 I THINK PROBABLY MANY OF YOU 4097 02:12:38,383 --> 02:12:39,718 HAVE READ THIS PAPER ON WHY 4098 02:12:39,785 --> 02:12:41,153 THINGS ARE NOT -- WE ARE NOT 4099 02:12:41,219 --> 02:12:42,654 ABLE TO REPRODUCE THINGS THAT 4100 02:12:42,721 --> 02:12:43,522 OTHER PEOPLE DID, NOT EVEN 4101 02:12:43,588 --> 02:12:45,490 THINGS THAT WE DID IN OUR OWN 4102 02:12:45,557 --> 02:12:48,827 LABS, WHICH IS A PAIN AND IT'S A 4103 02:12:48,894 --> 02:12:54,199 BIG PROBLEM. 4104 02:12:54,266 --> 02:12:55,600 WE THINK ABOUT HONESTY BUT IT'S 4105 02:12:55,667 --> 02:12:56,468 NOT JUST THAT. 4106 02:12:56,535 --> 02:12:58,270 WE DID A -- WE'RE NOT SHARING 4107 02:12:58,337 --> 02:12:59,738 ALL THE ANSWERS THAT WE ARE 4108 02:12:59,805 --> 02:13:04,843 GETTING WHEN WE DO RESEARCH. 4109 02:13:04,910 --> 02:13:07,012 SORRY, THIS IS VERY ANNOYING, I 4110 02:13:07,079 --> 02:13:08,080 DON'T KNOW IF YOU CAN STOP IT 4111 02:13:08,146 --> 02:13:12,884 BECAUSE IT'S JUST GOING FURTH 4112 02:13:12,951 --> 02:13:20,792 FURTHER -- I THINK IT'S PROBABLY 4113 02:13:20,859 --> 02:13:29,468 ON AUTOMATIC AT SOME POINT. 4114 02:13:29,534 --> 02:13:32,637 SO I'VE BEEN THE CHAIR OF A 4115 02:13:32,704 --> 02:13:35,273 NETWORK OF EUROPEAN RESEARCHERS 4116 02:13:35,340 --> 02:13:37,976 THAT IS -- THAT WORKS ON NUCLEIC 4117 02:13:38,043 --> 02:13:40,712 ACIDS THAT ARE -- AND IT'S A 4118 02:13:40,779 --> 02:13:42,013 QUITE VARIED GROUP OF PEOPLE. 4119 02:13:42,080 --> 02:13:44,049 AND WE DID MAKE A SMALL SURVEY 4120 02:13:44,116 --> 02:13:47,352 AMONGST OUR MEMBERS, AND WE 4121 02:13:47,419 --> 02:13:49,187 REPORTED THOSE RESULTS, AND 4122 02:13:49,254 --> 02:13:50,689 BASICALLY VERY SIMILAR TO WHAT 4123 02:13:50,756 --> 02:13:52,324 THE PREVIOUS SPEAKER SAID, WE 4124 02:13:52,391 --> 02:13:55,060 HAD THIS ISSUE OF NOT HAVING 4125 02:13:55,127 --> 02:13:56,828 TIME OR DIFFICULTIES IN 4126 02:13:56,895 --> 02:13:58,196 PUBLISHING ARE NOT BEING VALUED. 4127 02:13:58,263 --> 02:13:59,464 AND I THINK THE MAIN PROBLEM IS 4128 02:13:59,531 --> 02:14:01,800 THAT WE ARE NOT GIVING THE SAME 4129 02:14:01,867 --> 02:14:03,001 VALUE TO GOOD RESULTS AND BAD 4130 02:14:03,068 --> 02:14:03,235 RESULTS. 4131 02:14:03,301 --> 02:14:06,605 AND I THINK THAT IS AN ERROR. 4132 02:14:06,671 --> 02:14:08,440 BECAUSE IF YOU ARE FINDING A WAY 4133 02:14:08,507 --> 02:14:10,108 TO -- IF YOU'RE FINDING A WAY 4134 02:14:10,175 --> 02:14:14,413 OUT OF THAT MAZE, IT'S USEFUL TO 4135 02:14:14,479 --> 02:14:16,181 YOU TO KNOW IF THIS IS THE RIGHT 4136 02:14:16,248 --> 02:14:17,983 WAY OR THIS IS A DE DEAD END. 4137 02:14:18,049 --> 02:14:19,851 ACTUALLY IF YOU'RE USING GOOGLE 4138 02:14:19,918 --> 02:14:21,119 MAPS TO GET SOMEWHERE, YOU DON'T 4139 02:14:21,186 --> 02:14:22,320 WANT TO GET INTO A DEAD END SO 4140 02:14:22,387 --> 02:14:23,622 YOU DO WANT THAT DATA AVAILABLE 4141 02:14:23,688 --> 02:14:27,993 FOR ALL OF US. 4142 02:14:28,059 --> 02:14:38,570 SO WHY SHOULD WE BE DOING IT? 4143 02:14:46,678 --> 02:14:50,849 I PROMISE I'M NOT DOING IT. 4144 02:14:50,916 --> 02:14:53,752 SO WITHIN THAT NETWORK, WE DID 4145 02:14:53,819 --> 02:14:56,688 PUBLISH -- I'VE BEEN LUCKY 4146 02:14:56,755 --> 02:14:58,323 ENOUGH TO BE INVOLVED IN TWO 4147 02:14:58,390 --> 02:14:59,925 SPECIAL JOURNALS DEDICATED 4148 02:14:59,991 --> 02:15:03,028 SPECIFICALLY TO NEGATIVE RESU 4149 02:15:03,094 --> 02:15:03,662 RESULTS. 4150 02:15:03,728 --> 02:15:06,832 ONE OF THEM WAS ON NEUROMUSCULAR 4151 02:15:06,898 --> 02:15:08,366 DISORDERS, THE OTHER WAS ON 4152 02:15:08,433 --> 02:15:09,601 NUCLEIC ACIDS BECAUSE MY 4153 02:15:09,668 --> 02:15:12,370 RESEARCH MATCHES BOTH OF THOSE. 4154 02:15:12,437 --> 02:15:15,273 WE WERE YO LUCKY TO DO THAT. 4155 02:15:15,340 --> 02:15:16,842 I THINK I WANT TO USE AN EXAMPLE 4156 02:15:16,908 --> 02:15:17,809 JUST TO SHOW HOW IMPORTANT THIS 4157 02:15:17,876 --> 02:15:18,577 IS TO US. 4158 02:15:18,643 --> 02:15:20,111 WE'VE BEEN TALKING ABOUT NOT 4159 02:15:20,178 --> 02:15:21,346 PUBLISHING RESULTS OF CLINICAL 4160 02:15:21,413 --> 02:15:23,448 TRIALS AND THAT IS REALLY BAD, 4161 02:15:23,515 --> 02:15:25,217 BUT SOMETIMES THE PRE-CLINICAL 4162 02:15:25,283 --> 02:15:26,785 WORK, WHICH IS MUCH MORE WORK 4163 02:15:26,852 --> 02:15:28,320 THAN THAT, IS ALSO VERY 4164 02:15:28,386 --> 02:15:32,057 IMPORTANT. 4165 02:15:32,123 --> 02:15:33,225 THIS ABOUT REPEATING SOMETHING 4166 02:15:33,291 --> 02:15:35,026 AND REPLICATING SOMETHING THAT'S 4167 02:15:35,093 --> 02:15:35,660 ALREADY BEEN DONE IS SOMETHING 4168 02:15:35,727 --> 02:15:38,163 THAT YOU DON'T USUALLY GET MONEY 4169 02:15:38,230 --> 02:15:40,532 OR FUNDING TO DO BECAUSE IT'S 4170 02:15:40,599 --> 02:15:41,933 NOT -- IT'S NOT NOVEL BUT IT'S 4171 02:15:42,000 --> 02:15:42,868 REALLY IMPORTANT. 4172 02:15:42,934 --> 02:15:44,102 THIS IS A STUDY THAT WAS 4173 02:15:44,169 --> 02:15:45,837 PUBLISHED YEARS AGO ON A DRUG 4174 02:15:45,904 --> 02:15:51,643 THAT COULD IMPROVE RARE DISEASE. 4175 02:15:51,710 --> 02:15:53,912 THAT WAS AN INTERESTING RESULT. 4176 02:15:53,979 --> 02:15:55,447 AND THESE PEOPLE WENT TO GET 4177 02:15:55,514 --> 02:15:56,982 FUNDING FOR A CLINICAL TRIAL ON 4178 02:15:57,048 --> 02:15:59,484 THIS PARTICULAR DRUG. 4179 02:15:59,551 --> 02:16:02,854 THIS IS A NEUROMUSCULAR 4180 02:16:02,921 --> 02:16:04,923 DISORDER, DUCHENNE MUSCULAR 4181 02:16:04,990 --> 02:16:05,757 DYSTROPHY, AND THOSE PATIENTS 4182 02:16:05,824 --> 02:16:06,525 HAVE NO TIME. 4183 02:16:06,591 --> 02:16:07,526 SO WHEN YOU HAVE NO TIME, WHAT 4184 02:16:07,592 --> 02:16:08,627 YOU HAVE TO DO IS DO THINGS 4185 02:16:08,693 --> 02:16:08,960 RIGHT. 4186 02:16:09,027 --> 02:16:10,562 SO WHAT THEY DID WAS TO FUND 4187 02:16:10,629 --> 02:16:11,863 ANOTHER COUPLE OF RESEARCH 4188 02:16:11,930 --> 02:16:13,398 GROUPS TO REPLICATE THAT DATA. 4189 02:16:13,465 --> 02:16:15,233 AND THAT IS SOMETHING THAT YOU 4190 02:16:15,300 --> 02:16:16,201 DON'T GET USUALLY. 4191 02:16:16,268 --> 02:16:18,069 BUT THEY DID. 4192 02:16:18,136 --> 02:16:19,037 ACTUALLY IN THE ORIGINAL PAPER, 4193 02:16:19,104 --> 02:16:21,239 YOU HAD AN INTERESTING RESPONSE. 4194 02:16:21,306 --> 02:16:23,308 AND THEY DID SEE THAT AS WELL IN 4195 02:16:23,375 --> 02:16:24,276 ANOTHER OF THE LABS. 4196 02:16:24,342 --> 02:16:25,810 BUT ACTUALLY WHEN THEY COMPARE 4197 02:16:25,877 --> 02:16:28,313 THAT TO THE WILD TYPE, IT WAS 4198 02:16:28,380 --> 02:16:29,147 NOT AS RELEVANT. 4199 02:16:29,214 --> 02:16:31,917 THERE WAS A P VALUE THERE. 4200 02:16:31,983 --> 02:16:33,285 BUT THAT MAKES NOT SUCH A BIG 4201 02:16:33,351 --> 02:16:34,185 DIFFERENCE FOR THE MICE. 4202 02:16:34,252 --> 02:16:35,320 THERE WAS LOTS OF INTERESTING 4203 02:16:35,387 --> 02:16:36,855 DATA IN THE ORIGINAL PAPER, 4204 02:16:36,922 --> 02:16:39,257 DON'T THINK THAT I'M CRITICIZING 4205 02:16:39,324 --> 02:16:40,458 THEM, BUT THERE WAS MORE 4206 02:16:40,525 --> 02:16:41,860 RESEARCH NEEDED BEFORE RUNNING 4207 02:16:41,927 --> 02:16:44,930 INTO A CLINICAL TRIAL. 4208 02:16:44,996 --> 02:16:45,964 WHEN YOU SEE WHERE THOSE THINGS 4209 02:16:46,031 --> 02:16:47,933 WERE PUBLISHED, THE ORIGINAL 4210 02:16:47,999 --> 02:16:52,003 DATA WAS PUBLISHED IN A HIGH 4211 02:16:52,070 --> 02:16:53,438 IMPACT JOURNAL AND THE OTHER WAS 4212 02:16:53,505 --> 02:16:54,339 PUBLISHED IN SOMETHING THAT AT 4213 02:16:54,406 --> 02:16:55,941 THE TIME HAD NO IMPACT FACTOR. 4214 02:16:56,007 --> 02:16:57,542 THAT MEANS FOR THOSE WHO DID THE 4215 02:16:57,609 --> 02:16:58,877 RESEARCH, ONE OF THEM, YOU KNOW, 4216 02:16:58,944 --> 02:17:00,679 MADE A BIG IMPACT IN THEIR 4217 02:17:00,745 --> 02:17:02,647 CAREERS, THE OTHER ONE BASICALLY 4218 02:17:02,714 --> 02:17:05,050 WAS VALUED AT ZERO IN MANY OF 4219 02:17:05,116 --> 02:17:06,051 THOSE ASSESSMENTS. 4220 02:17:06,117 --> 02:17:07,352 SO I THINK SOMETIMES THE VALUE 4221 02:17:07,419 --> 02:17:10,188 THAT WE GIVE THE RESULTS 4222 02:17:10,255 --> 02:17:11,389 DEPENDING ON WHERE THEY 4223 02:17:11,456 --> 02:17:12,490 PUBLISHED OR HOW THEY'RE 4224 02:17:12,557 --> 02:17:13,892 PUBLISHED AND HOW IMPACTFUL THAT 4225 02:17:13,959 --> 02:17:15,327 IS IS SOMETHING THAT WE SHOULD 4226 02:17:15,393 --> 02:17:17,729 TRY TO GET RID. 4227 02:17:17,796 --> 02:17:19,864 BECAUSE WE CANNOT AFFORD IT. 4228 02:17:19,931 --> 02:17:21,633 SO I THINK THAT'S A BIT OF THE 4229 02:17:21,700 --> 02:17:25,003 ELEPHANT IN THE ROOM. 4230 02:17:25,070 --> 02:17:27,539 WE NEED TO CHANGE THE VALUE OF 4231 02:17:27,606 --> 02:17:29,374 THAT, BECAUSE I THINK IT'S 4232 02:17:29,441 --> 02:17:30,909 VALUABLE TO KNOW WHEN SOMETHING 4233 02:17:30,976 --> 02:17:32,410 IS RIGHT AND WHEN SOMETHING IS 4234 02:17:32,477 --> 02:17:33,278 NOT CORRECT. 4235 02:17:33,345 --> 02:17:36,448 SO I THINK WE NEED TO CHANGE 4236 02:17:36,514 --> 02:17:38,850 THAT IN OUR OWN CAREERS AND ALSO 4237 02:17:38,917 --> 02:17:40,819 WHEN WE ARE TRYING TO CONVINCE 4238 02:17:40,885 --> 02:17:43,722 THE NEXT GENERATIONS INTO WHAT 4239 02:17:43,788 --> 02:17:45,056 IS -- WHAT THEY'RE DOING, AND WE 4240 02:17:45,123 --> 02:17:50,895 NEED TO BE PROFICIENT AND NOT -- 4241 02:17:50,962 --> 02:17:56,868 THANK YOU VERY MUCH. 4242 02:17:56,935 --> 02:17:57,569 >> ALL RIGHT. 4243 02:17:57,636 --> 02:17:59,004 SO THE FORMAT OF THESE SESSIONS 4244 02:17:59,070 --> 02:18:00,538 IS MEANT TO BE PRIMARILY 4245 02:18:00,605 --> 02:18:02,474 DISCUSSION, SO AT THIS POINT, I 4246 02:18:02,540 --> 02:18:03,775 DON'T KNOW IF WE NEED TO HAND 4247 02:18:03,842 --> 02:18:05,010 OUT MICROPHONES OR IF PEOPLE 4248 02:18:05,076 --> 02:18:07,078 WOULD LIKE TO STAND UP AND SPEAK 4249 02:18:07,145 --> 02:18:08,880 AT THE MICROPHONE. 4250 02:18:08,947 --> 02:18:11,716 ASK QUESTIONS OR JUST START A 4251 02:18:11,783 --> 02:18:14,819 DISCUSSION. 4252 02:18:14,886 --> 02:18:16,221 BUT WHILE PEOPLE ARE DOING THAT, 4253 02:18:16,287 --> 02:18:17,789 MAYBE I CAN ASK, IT SEEMS LIKE 4254 02:18:17,856 --> 02:18:21,960 WE DON'T NECESSARILY HAVE A 4255 02:18:22,027 --> 02:18:23,828 DEFINITIVE WAY TO MEASURE 4256 02:18:23,895 --> 02:18:25,530 PUBLICATION BIAS, AND SO WHAT DO 4257 02:18:25,597 --> 02:18:28,266 YOU ALL THINK IS THE BEST PATH 4258 02:18:28,333 --> 02:18:29,467 FORWARD TO SEE WHETHER OR NOT 4259 02:18:29,534 --> 02:18:39,978 WE'RE FIXING THE PROBLEM? 4260 02:18:48,653 --> 02:18:50,755 >> I THINK IT WOULD HAVE TO BE A 4261 02:18:50,822 --> 02:18:54,225 MULTIPRONGED APPROACH WHERE YOU 4262 02:18:54,292 --> 02:18:55,760 INCLUDE STUDIES WHERE YOU'RE 4263 02:18:55,827 --> 02:18:57,629 LOOKING AT PUBLISHED RESULTS AND 4264 02:18:57,696 --> 02:19:01,199 COMPARING THEM TO RESULTS FROM 4265 02:19:01,266 --> 02:19:02,600 REGISTRIES OR ABSTRACTS OR OTHER 4266 02:19:02,667 --> 02:19:06,438 SOURCES, AND OVER TIME, SEE 4267 02:19:06,504 --> 02:19:10,008 WHETHER THOSE VARIOUS SOURCES 4268 02:19:10,075 --> 02:19:13,945 CONTINUE TO ALIGN OR IMPROVE IN 4269 02:19:14,012 --> 02:19:16,247 ALIGNMENT, AND YOU DON'T SEE 4270 02:19:16,314 --> 02:19:17,515 INCREASED NULL RESULTS PUBLISHED 4271 02:19:17,582 --> 02:19:19,684 COMPARED TO OTHER SOURCES. 4272 02:19:19,751 --> 02:19:22,420 I THINK THAT CAN BE SUPPLEMENTED 4273 02:19:22,487 --> 02:19:26,758 BY META-ANALYSES THAT USE 4274 02:19:26,825 --> 02:19:28,059 QUANTIFICATION OR DETECTION 4275 02:19:28,126 --> 02:19:30,261 METHODS AND TO SEE IF THOSE 4276 02:19:30,328 --> 02:19:33,765 REDUCE OVER TIME, I DON'T THINK 4277 02:19:33,832 --> 02:19:35,433 THERE'S ANY UNFORTUNATELY ONE 4278 02:19:35,500 --> 02:19:38,069 WAY TO DO IT. 4279 02:19:38,136 --> 02:19:40,939 BUT I THINK SOMETHING THAT IS 4280 02:19:41,005 --> 02:19:43,007 PERHAPS AT REGULAR INTERVALS 4281 02:19:43,074 --> 02:19:46,177 CONSISTENTLY MEASURED REPEAT 4282 02:19:46,244 --> 02:19:47,345 REPEATEDLY, I THINK WOULD BE 4283 02:19:47,412 --> 02:19:53,384 STRONG EVIDENCE OVER TIME. 4284 02:19:53,451 --> 02:19:55,620 >> HI, MY NAME IS LAURA SMITH 4285 02:19:55,687 --> 02:19:57,789 FROM TEXAS A & M UNIVERSITY. 4286 02:19:57,856 --> 02:19:59,357 I WANTED TO GIVE A LITTLE BIT OF 4287 02:19:59,424 --> 02:20:01,025 PUSH BACK AND ASK A BIGGER 4288 02:20:01,092 --> 02:20:01,359 QUESTION HERE. 4289 02:20:01,426 --> 02:20:03,661 SO I HEARD SOME INDICATIONS 4290 02:20:03,728 --> 02:20:05,530 MAYBE FROM DR. MALHOTRA THAT 4291 02:20:05,597 --> 02:20:07,665 NULL STUDIES OR NEGATIVE RESULTS 4292 02:20:07,732 --> 02:20:11,236 MIGHT BE THE RESULT OF 4293 02:20:11,302 --> 02:20:14,439 MISMANAGED RESEARCH OR SOME 4294 02:20:14,506 --> 02:20:15,874 MISTAKES IN THE RESEARCH, AND 4295 02:20:15,940 --> 02:20:17,675 THAT'S MORE LIKELY TO PRODUCE 4296 02:20:17,742 --> 02:20:19,310 NEGATIVE RESULTS. 4297 02:20:19,377 --> 02:20:22,046 AND I JUST WANT TO BRING A WORD 4298 02:20:22,113 --> 02:20:24,249 OF CAUTION HERE THAT THAT BRINGS 4299 02:20:24,315 --> 02:20:26,050 SOME SHAME TO SCIENTISTS FOR 4300 02:20:26,117 --> 02:20:28,686 PRODUCING NEGATIVE RESULTS, AND 4301 02:20:28,753 --> 02:20:30,655 THERE'S OTHER REASONS THAT 4302 02:20:30,722 --> 02:20:32,290 SCIENTISTS DON'T WANT TO OH 4303 02:20:32,357 --> 02:20:36,194 CAORCAN'T PUBLISH NEGATIVE RESUS 4304 02:20:36,261 --> 02:20:37,929 SO I THINK WE HAVE TO BE CAREFUL 4305 02:20:37,996 --> 02:20:38,997 BECAUSE WHENEVER A SCIENTIST 4306 02:20:39,063 --> 02:20:40,064 ACTUALLY IS TRYING TO GET THEIR 4307 02:20:40,131 --> 02:20:41,166 WORK PUBLISHED AND THEY'RE 4308 02:20:41,232 --> 02:20:42,200 LOOKING AT THE INCENTIVE 4309 02:20:42,267 --> 02:20:44,335 STRUCTURE OF WHAT THEY'RE DOING, 4310 02:20:44,402 --> 02:20:45,603 THEY'RE PRESSED INTO DOING 4311 02:20:45,670 --> 02:20:47,038 THINGS LIKE URI WAS TALKING 4312 02:20:47,105 --> 02:20:48,873 ABOUT, P-HACKING, ET CETERA. 4313 02:20:48,940 --> 02:20:50,175 SO I THINK, YOU KNOW, WE'RE NOT 4314 02:20:50,241 --> 02:20:53,878 ONLY LOOKING AT THE ISSUE OF 4315 02:20:53,945 --> 02:20:55,046 PUBLISHING NON-SIGNIFICANT 4316 02:20:55,113 --> 02:20:56,881 VERSUS SIGNIFICANT, BUT WE'VE 4317 02:20:56,948 --> 02:20:57,849 GOT A BIGGER PROBLEM. 4318 02:20:57,916 --> 02:20:59,284 WE DON'T EVEN SEE MOST OF THE 4319 02:20:59,350 --> 02:21:00,585 NEGATIVE STUDIES, AND THAT'S 4320 02:21:00,652 --> 02:21:02,754 MAYBE BECAUSE OF SHAME AND 4321 02:21:02,821 --> 02:21:04,823 INABILITY TO PUBLISH. 4322 02:21:04,889 --> 02:21:06,558 SO JUST COMMENTS, PLEASE. 4323 02:21:06,624 --> 02:21:07,392 THANKS. 4324 02:21:07,458 --> 02:21:08,393 >> OKAY. 4325 02:21:08,459 --> 02:21:09,160 I'LL RESPOND TO THAT. 4326 02:21:09,227 --> 02:21:12,197 SO IF I MEANT IT NORMTIVELY, I 4327 02:21:12,263 --> 02:21:13,097 DIDN'T MEAN IT AT ALL. 4328 02:21:13,164 --> 02:21:14,799 IT WAS A POSITIVE STATEMENT. 4329 02:21:14,866 --> 02:21:17,969 WHICH IS THAT JUST 4330 02:21:18,036 --> 02:21:20,271 STATISTICALLY, YOU CANNOT ASSESS 4331 02:21:20,338 --> 02:21:23,474 PUBLICATION BIAS SIMPLY BY 4332 02:21:23,541 --> 02:21:25,476 COUNTING HOW MANY PAPERS ARE 4333 02:21:25,543 --> 02:21:27,745 SIGNIFICANT VERSUS 4334 02:21:27,812 --> 02:21:28,246 NON-SIGNIFICANT. 4335 02:21:28,313 --> 02:21:30,048 BECAUSE THERE'S A LOT OF 4336 02:21:30,114 --> 02:21:31,249 UNOBSERVED HETEROGENEITY BETWEEN 4337 02:21:31,316 --> 02:21:32,450 SIGNIFICANT AND NON-SIGNIFICANT 4338 02:21:32,517 --> 02:21:33,451 STUDIES. 4339 02:21:33,518 --> 02:21:34,719 SO JUST THE PROPER WAY TO 4340 02:21:34,786 --> 02:21:35,920 MEASURE IT, NOT FROM A NORMATIVE 4341 02:21:35,987 --> 02:21:37,789 STATEMENT BUT JUST A POSITIVE 4342 02:21:37,856 --> 02:21:39,190 STATEMENT, IS YOU HAVE TO 4343 02:21:39,257 --> 02:21:40,692 CONDITION ON ALL THAT STUFF. 4344 02:21:40,758 --> 02:21:43,094 SO JUST TWO EXAMPLES. 4345 02:21:43,161 --> 02:21:44,629 WHICH IS SMALLER STUDIES ARE 4346 02:21:44,696 --> 02:21:47,198 GOING TO HAVE LESS SIGNIFICANT 4347 02:21:47,265 --> 02:21:47,765 RESULTS. 4348 02:21:47,832 --> 02:21:48,967 SO ONE THING YOU'D WANT TO 4349 02:21:49,033 --> 02:21:50,602 CONDITION ON IS WHAT IS THE 4350 02:21:50,668 --> 02:21:51,970 STATISTICAL POWER OF THESE 4351 02:21:52,036 --> 02:21:52,203 STUDIES. 4352 02:21:52,270 --> 02:21:53,671 SO IT WOULD BE A BAD THING IF 4353 02:21:53,738 --> 02:21:55,840 EQUALLY POWERED STUDIES, THE 4354 02:21:55,907 --> 02:21:57,175 SIGNIFICANT ONES, ARE THE ONLY 4355 02:21:57,242 --> 02:21:58,409 ONES GETTING PUBLISHED. 4356 02:21:58,476 --> 02:22:00,478 IT WOULD NOT NECESSARILY BE A 4357 02:22:00,545 --> 02:22:01,779 PROBLEM IF SMALLER STUDIES ARE 4358 02:22:01,846 --> 02:22:03,181 NOT BEING PUBLISHED AND LARGER 4359 02:22:03,248 --> 02:22:03,915 STUDIES ARE. 4360 02:22:03,982 --> 02:22:05,149 ANOTHER EXAMPLE IS MEASUREMENT 4361 02:22:05,216 --> 02:22:06,818 ERROR AND THE VARIABLES. 4362 02:22:06,885 --> 02:22:07,919 SO WE KNOW THAT IF YOU HAVE 4363 02:22:07,986 --> 02:22:10,221 MEASUREMENT ERROR IN THE 4364 02:22:10,288 --> 02:22:11,089 VARIABLES, THE ESTIMATES CAN BE 4365 02:22:11,155 --> 02:22:15,660 BUYS BYE-BYE ASSED TOWARDS ZERO. 4366 02:22:15,727 --> 02:22:18,463 SO I DON'T THINK IT'S A PROBLEM 4367 02:22:18,529 --> 02:22:20,298 WITH IF HE -- BUT CONDITIONAL ON 4368 02:22:20,365 --> 02:22:21,733 EQUIVALENT MEASUREMENT ERROR, WE 4369 02:22:21,799 --> 02:22:23,534 SHOULD NOT BE PUBLISHING 4370 02:22:23,601 --> 02:22:24,335 SIGNIFICANT VERSUS NOT 4371 02:22:24,402 --> 02:22:25,003 SIGNIFICANT STUDIES. 4372 02:22:25,069 --> 02:22:26,404 SO HOPEFULLY THAT CLARIFIES IT 4373 02:22:26,471 --> 02:22:28,439 AND TO SAY YES, THERE'S PLENTY 4374 02:22:28,506 --> 02:22:29,440 OF GOOD STUDIES THAT HAVE NULL 4375 02:22:29,507 --> 02:22:31,109 RESULTS, THERE'S PLENTY OF BAD 4376 02:22:31,175 --> 02:22:31,943 STUDIES. 4377 02:22:32,010 --> 02:22:33,077 THERE'S ALSO BAD STUDIES THAT 4378 02:22:33,144 --> 02:22:33,745 HAVE POSITIVE RESULTS BUT WE 4379 02:22:33,811 --> 02:22:35,046 WANT TO TRY TO CONDITION ON 4380 02:22:35,113 --> 02:22:35,947 THESE THINGS IF YOU WANT TO 4381 02:22:36,014 --> 02:22:36,748 ASSESS THE PROBLEM. 4382 02:22:36,814 --> 02:22:40,051 >> I JUST WANT TO -- ADD A 4383 02:22:40,118 --> 02:22:40,818 COMMENT TO THAT. 4384 02:22:40,885 --> 02:22:44,656 SO AT LEAST IN YOUR SCIENCE IN 4385 02:22:44,722 --> 02:22:48,026 PRE-CLINICAL RESEARCH, THE POWER 4386 02:22:48,092 --> 02:22:52,063 IS SOMEWHERE BETWEEN 8 AND 30%. 4387 02:22:52,130 --> 02:22:52,597 OKAY? 4388 02:22:52,664 --> 02:22:55,633 IN THE LITERATURE. 4389 02:22:55,700 --> 02:22:57,035 SO THE NOTION THAT AT LEAST IN 4390 02:22:57,101 --> 02:23:01,239 THE PRE-CLINICAL WORLD, THERE'S 4391 02:23:01,306 --> 02:23:02,807 THIS BIG DIFFERENCE IN HOW THE 4392 02:23:02,874 --> 02:23:08,446 STUDIES ARE POWERED, IF THE 4393 02:23:08,513 --> 02:23:09,981 RESULTS ARE POSITIVE OR NULL 4394 02:23:10,048 --> 02:23:14,218 DOESN'T SEEM TO EXIST. 4395 02:23:14,285 --> 02:23:18,690 SO WE'RE GOING TO -- PROBABLY BY 4396 02:23:18,756 --> 02:23:19,991 PRIMARILY DEAL HERE NOT WITH 4397 02:23:20,058 --> 02:23:21,526 STUDIES THAT ARE LARGE AND HAD 4398 02:23:21,592 --> 02:23:25,530 TO GO THROUGH AN IRB AND GET 4399 02:23:25,596 --> 02:23:26,731 APPROVAL, WE'RE GOING TO BE 4400 02:23:26,798 --> 02:23:28,833 TALKING MAINLY AS WE HEARD 4401 02:23:28,900 --> 02:23:31,536 ALREADY ABOUT THE PRE-CLINICAL 4402 02:23:31,602 --> 02:23:32,603 UNIVERSE WHERE MOST OF THE 4403 02:23:32,670 --> 02:23:37,575 STUDIES ARE SMALL, AND YET THEY 4404 02:23:37,642 --> 02:23:38,609 PUBLISHED THE ONES THAT REACH 4405 02:23:38,676 --> 02:23:41,446 THIS MAGICAL P VALUE, AND THE 4406 02:23:41,512 --> 02:23:45,149 ONES THAT DON'T, I DON'T SEE THE 4407 02:23:45,216 --> 02:23:46,250 LIGHT -- IN MY OPINION THERE'S 4408 02:23:46,317 --> 02:23:47,285 REALLY NO DIFFERENCE BETWEEN THE 4409 02:23:47,352 --> 02:23:50,588 TWO STARS THE TRUST THAT YOU 4410 02:23:50,655 --> 02:23:52,056 COULD PUT STATISTICALLY ON THE 4411 02:23:52,123 --> 02:23:57,261 RESULTS. 4412 02:23:57,328 --> 02:23:59,263 >> SO I HAVE TWO QUESTIONS, THE 4413 02:23:59,330 --> 02:24:02,400 FIRST IS JUST CHEEKY AND 4414 02:24:02,467 --> 02:24:04,469 DOESN'T -- IS THERE PUBLICATION 4415 02:24:04,535 --> 02:24:06,104 BIAS IN PUBLICATION BIAS 4416 02:24:06,170 --> 02:24:09,140 PUBLICATIONS? 4417 02:24:09,207 --> 02:24:10,408 THE SECOND QUESTION I HAVE, IT 4418 02:24:10,475 --> 02:24:14,078 RELATES TO THIS TABLE, WHICH I 4419 02:24:14,145 --> 02:24:17,115 WAS THINKING MAYBE, NEIL, OR 4420 02:24:17,181 --> 02:24:19,650 ANYONE UP HERE MIGHT BE ABLE TO 4421 02:24:19,717 --> 02:24:21,919 ANSWER, WHICH IS THE PERSEVERING 4422 02:24:21,986 --> 02:24:22,820 RESEARCHERS, THE ONES WHO DIDN'T 4423 02:24:22,887 --> 02:24:28,159 GET FUNDED ORIGINALLY, ARE 4424 02:24:28,226 --> 02:24:32,130 THEY -- IS THERE SOME POSITIVE 4425 02:24:32,196 --> 02:24:35,767 RESULTS HERE IN THE SENSE OF IS 4426 02:24:35,833 --> 02:24:39,103 THERE INDICATION IN THIS TABLE 4427 02:24:39,170 --> 02:24:40,405 THAT THE PERSEVERING RESEARCHERS 4428 02:24:40,471 --> 02:24:43,374 ARE PUBLISHING MORE NULL RESU 4429 02:24:43,441 --> 02:24:43,608 RESULTS? 4430 02:24:43,674 --> 02:24:44,942 I REALLY -- WE DIDN'T GET ENOUGH 4431 02:24:45,009 --> 02:24:46,177 TIME WITH THIS, MAYBE YOU CAN 4432 02:24:46,244 --> 02:24:48,846 EXPLAIN TO US A LITTLE MORE. 4433 02:24:48,913 --> 02:24:52,383 >> SURE, WE'LL LOOK AT IT A 4434 02:24:52,450 --> 02:24:53,184 LITTLE MORE SLOWLY. 4435 02:24:53,251 --> 02:24:54,585 SO LET'S LOOK AT THE BOTTOM ROW 4436 02:24:54,652 --> 02:24:55,920 IS WHAT YOU'RE MOST INTERESTED 4437 02:24:55,987 --> 02:24:57,155 IN, I GUESS; IS THAT CORRECT? 4438 02:24:57,221 --> 02:24:57,688 >> THE TOTAL. 4439 02:24:57,755 --> 02:24:58,990 >> THE TOTAL, YEAHMENT SO I 4440 02:24:59,057 --> 02:25:01,492 THINK THE RELEVANT THING IS FOR 4441 02:25:01,559 --> 02:25:02,860 THE PURPOSE OF THIS CONFERENCE 4442 02:25:02,927 --> 02:25:03,227 IS LIKE THE GAP. 4443 02:25:03,294 --> 02:25:03,995 SO IF YOU LOOK AT THE 4444 02:25:04,062 --> 02:25:05,430 PERSEVERING RESEARCHERS AND 4445 02:25:05,496 --> 02:25:06,898 THEIR PUBLICATION STATUS, THEIR 4446 02:25:06,964 --> 02:25:08,800 GAP IS ABOUT LIKE 20 PERCENTAGE 4447 02:25:08,866 --> 02:25:10,001 POINTS, WHEREAS IT'S 30 4448 02:25:10,068 --> 02:25:10,768 PERCENTAGE POINTS FOR THE OTHER 4449 02:25:10,835 --> 02:25:13,171 GROUP. 4450 02:25:13,237 --> 02:25:14,472 SO I BRIEFLY MENTIONED THIS AT 4451 02:25:14,539 --> 02:25:14,772 THE END. 4452 02:25:14,839 --> 02:25:17,308 THIS IS THE OPPOSITE DIRECTION 4453 02:25:17,375 --> 02:25:18,843 OF WHAT YOU WOULD EXPECT IF YOU 4454 02:25:18,910 --> 02:25:19,911 THINK THE INSTITUTION MATTERS. 4455 02:25:19,977 --> 02:25:21,546 IF THE INSTITUTION IS MATTERING, 4456 02:25:21,612 --> 02:25:23,815 YOU WOULD EXPECT THE GAP TO BE 4457 02:25:23,881 --> 02:25:28,786 LOWER ON THE LEFT-HAND SIDE. 4458 02:25:28,853 --> 02:25:30,188 SO I THINK THIS IS SOMETHING WE 4459 02:25:30,254 --> 02:25:30,955 SHOULD THINK ABOUT, WHICH IS 4460 02:25:31,022 --> 02:25:32,423 MAYBE THE INSTITUTIONS ARE NOT 4461 02:25:32,490 --> 02:25:33,791 AS POWERFUL AS WE THINK AT WHAT 4462 02:25:33,858 --> 02:25:35,026 IS DRIVING THE RESULT. 4463 02:25:35,093 --> 02:25:36,828 I'M NOT SAYING THAT'S FOR SURE 4464 02:25:36,894 --> 02:25:37,628 BUT IT'S A QUESTION WE MIGHT 4465 02:25:37,695 --> 02:25:38,729 WANT TO ASK. 4466 02:25:38,796 --> 02:25:41,365 BECAUSE YOU'RE CONDITIONING ON 4467 02:25:41,432 --> 02:25:42,667 CULTURE IN THIS ANALYSIS, 4468 02:25:42,733 --> 02:25:43,935 THEY'RE ALL OPERATING IN THE 4469 02:25:44,001 --> 02:25:45,503 SAME OVERALL POST OPEN SCIENCE 4470 02:25:45,570 --> 02:25:47,004 CULTURE, BUT ONE IS SUBJECT TO 4471 02:25:47,071 --> 02:25:47,638 INSTITUTIONS AND ONE IS NOT. 4472 02:25:47,705 --> 02:25:54,812 >> THANK YOU SO MUCH. 4473 02:25:54,879 --> 02:25:57,048 >> I HAVE A QUESTION THAT KIND 4474 02:25:57,115 --> 02:25:59,984 OF SPEAKS TO -- SORRY, I'M THE 4475 02:26:00,051 --> 02:26:01,085 EDITOR-IN-CHIEF OF THE JOURNAL 4476 02:26:01,152 --> 02:26:01,786 PLUS ONE. 4477 02:26:01,853 --> 02:26:02,954 IT SPEAKS TO THE QUESTIONS THAT 4478 02:26:03,020 --> 02:26:04,155 WE'RE GOING TO BE HEARING OVER 4479 02:26:04,222 --> 02:26:05,656 THE NEXT DAY OR TWO. 4480 02:26:05,723 --> 02:26:07,058 SORRY. 4481 02:26:07,125 --> 02:26:09,460 I HAVE TO SHRINK DOWN. 4482 02:26:09,527 --> 02:26:13,030 SO YOU MENTIONED THE PROBLEMS 4483 02:26:13,097 --> 02:26:14,232 WITH -- SORRY -- PUBLISHING 4484 02:26:14,298 --> 02:26:16,868 SMALL STUDIES THAT AREN'T 4485 02:26:16,934 --> 02:26:18,503 SUITABLY POWERED, BUT I JUST 4486 02:26:18,569 --> 02:26:19,937 WONDERED IF I COULD GET YOUR 4487 02:26:20,004 --> 02:26:21,139 VIEW ON THE VALUE OF RELEASING 4488 02:26:21,205 --> 02:26:22,773 THAT DATA INTO THE RESEARCH 4489 02:26:22,840 --> 02:26:23,841 ECOSYSTEM FOR OTHERS TO RE-USE 4490 02:26:23,908 --> 02:26:25,042 TO BUILD UPON. 4491 02:26:25,109 --> 02:26:28,179 IS THERE VALUE IN DATA THAT MAY 4492 02:26:28,246 --> 02:26:30,014 NOT BE STATISTICALLY SIGNIFICANT 4493 02:26:30,081 --> 02:26:31,549 ON ITS OWN BUT COULD FORM THE 4494 02:26:31,616 --> 02:26:36,621 BASIS OF FUTURE RESEARCH? 4495 02:26:36,687 --> 02:26:39,757 >> I THINK IT WILL BE GREAT, 4496 02:26:39,824 --> 02:26:40,525 SOMETIMES YOU DON'T HAVE THE 4497 02:26:40,591 --> 02:26:46,430 RESOURCES TO MAKE A LARGER -- 4498 02:26:46,497 --> 02:26:47,598 STUDY BECAUSE YOU DON'T HAVE 4499 02:26:47,665 --> 02:26:48,699 NUSS MICE, YOU DON'T HAVE NUSS 4500 02:26:48,766 --> 02:26:50,401 CELLS OR YOU DON'T HAVE ENOUGH 4501 02:26:50,468 --> 02:26:54,172 MONEY TO RUN IT. 4502 02:26:54,238 --> 02:26:57,375 I THINK IF I COULD SPECULATE AND 4503 02:26:57,441 --> 02:26:58,976 KIND OF DREAM OUT LOUD, I THINK 4504 02:26:59,043 --> 02:27:00,278 MORE THAN OPEN SCIENCE, I THINK 4505 02:27:00,344 --> 02:27:02,180 IF YOU COULD HAVE OPEN LAB 4506 02:27:02,246 --> 02:27:06,417 BOOKS, THAT WOULD BE FANTASTIC. 4507 02:27:06,484 --> 02:27:07,618 BUT OBVIOUSLY TO BE ABLE TO DO 4508 02:27:07,685 --> 02:27:10,354 THAT, YOU NEED SOME KIND OF A 4509 02:27:10,421 --> 02:27:11,622 STRUCTURE TO LAB BOOKS THAT WE 4510 02:27:11,689 --> 02:27:12,290 ALL SHARED. 4511 02:27:12,356 --> 02:27:13,925 SO I THINK THERE COULD BE A WAY 4512 02:27:13,991 --> 02:27:15,993 THAT TO MAKE SURE WHAT WE DO 4513 02:27:16,060 --> 02:27:19,497 DOESN'T STAY, YOU KNOW, IN THE 4514 02:27:19,564 --> 02:27:21,532 LAB UNPUBLISHED FOR MANY YEARS, 4515 02:27:21,599 --> 02:27:23,301 BECAUSE THERE ARE LOTS OF VIABLE 4516 02:27:23,367 --> 02:27:26,437 DATA THERE THAT WE'RE NOT TAKING 4517 02:27:26,504 --> 02:27:29,440 ADVANTAGE. 4518 02:27:29,507 --> 02:27:31,676 >> THE HYPOTHESIS WAS THAT IT 4519 02:27:31,742 --> 02:27:33,144 WAS SUFFICIENTLY POWERED BUT IT 4520 02:27:33,211 --> 02:27:34,712 WAS AN INCORRECT HYPOTHESIS. 4521 02:27:34,779 --> 02:27:35,379 THANKS. 4522 02:27:35,446 --> 02:27:36,681 I JUST HAVE A QUESTION ABOUT -- 4523 02:27:36,747 --> 02:27:38,449 OR COMMENT ABOUT WHAT ABOUT 4524 02:27:38,516 --> 02:27:40,284 STARTING FROM FUNDED GRANTS? 4525 02:27:40,351 --> 02:27:41,919 OR OTHER FUNDED SOURCES? 4526 02:27:41,986 --> 02:27:44,555 AND USING THAT TO TRACK WHAT 4527 02:27:44,622 --> 02:27:47,391 STUDIES ARE BEING PROPOSED? 4528 02:27:47,458 --> 02:27:47,825 RIGHT? 4529 02:27:47,892 --> 02:27:49,026 SORT OF THE OUTCOMES RESEARCH, 4530 02:27:49,093 --> 02:27:52,530 AND I DON'T KNOW HOW NIH HAN HAS 4531 02:27:52,597 --> 02:27:54,966 THAT OR ANYBODY DOES, AS ANOTHER 4532 02:27:55,032 --> 02:27:56,400 SOURCE OF THAT INFORMATION. 4533 02:27:56,467 --> 02:27:58,569 >> SO WE'VE BEEN TRYING TO DO 4534 02:27:58,636 --> 02:28:00,404 THAT IN MY RESEARCH GROUP. 4535 02:28:00,471 --> 02:28:03,908 THE TROUBLE IS THAT WE WANT SORT 4536 02:28:03,975 --> 02:28:07,812 OF THE -- SO WE'VE BEEN TRYING 4537 02:28:07,878 --> 02:28:09,447 DO THAT IN OUR RESEARCH GROUP. 4538 02:28:09,513 --> 02:28:10,314 THE TROUBLE IS THAT WE REALLY 4539 02:28:10,381 --> 02:28:11,949 WERE WANTING TO GET THE 4540 02:28:12,016 --> 02:28:13,384 PRELIMINARY DATA THAT WAS 4541 02:28:13,451 --> 02:28:14,352 PRESENTED IN THE GRANTS AND AT 4542 02:28:14,418 --> 02:28:15,853 LEAST FOR THE NIH, YOU HAVE TO 4543 02:28:15,920 --> 02:28:17,388 DO A FREEDOM OF INFORMATION ACT 4544 02:28:17,455 --> 02:28:20,224 TO GET EVERY SINGLE ONE OF THOSE 4545 02:28:20,291 --> 02:28:21,025 APPLICATIONS IF YOU WANTED TO 4546 02:28:21,092 --> 02:28:22,360 SORT OF FOLLOW THAT PRELIMINARY 4547 02:28:22,426 --> 02:28:28,499 DATA FORWARD. 4548 02:28:28,566 --> 02:28:29,934 WE JUST RAN INTO CAPACITY 4549 02:28:30,001 --> 02:28:30,835 ISSUES. 4550 02:28:30,901 --> 02:28:33,537 >> LET ME JUST CHIME IN, AS THE 4551 02:28:33,604 --> 02:28:36,674 GOVERNMENT HERE. 4552 02:28:36,741 --> 02:28:37,141 [LAUGHTER] 4553 02:28:37,208 --> 02:28:39,744 YOU'RE ALL UNDER ARREST. 4554 02:28:39,810 --> 02:28:43,347 [LAUGHTER] 4555 02:28:43,414 --> 02:28:44,915 SO AS YOU KNOW, THESE ARE GRANTS 4556 02:28:44,982 --> 02:28:48,719 AND NOT CONTRACTS. 4557 02:28:48,786 --> 02:28:52,890 AND AS A RESULT, CERTAINLY YOU 4558 02:28:52,957 --> 02:28:54,058 HAVE SPECIFIC AIMS BUT THEY CAN 4559 02:28:54,125 --> 02:28:55,192 CHANGE AND YOU CAN FOLLOW YOUR 4560 02:28:55,259 --> 02:28:57,261 HEART AND YOU CAN MAKE ALL -- 4561 02:28:57,328 --> 02:29:00,798 YOU KNOW, SO IT'S ESSENTIALLY 4562 02:29:00,865 --> 02:29:02,166 IMPOSSIBLE TO KIND OF -- TO 4563 02:29:02,233 --> 02:29:03,467 RELATE NECESSARILY WHAT ENDS UP 4564 02:29:03,534 --> 02:29:06,203 WITH WHAT YOU WROTE IN YOUR 4565 02:29:06,270 --> 02:29:09,840 APPLICATION. 4566 02:29:09,907 --> 02:29:14,412 >> YEAH, GRANTS ARE NOT EXACTLY 4567 02:29:14,478 --> 02:29:14,779 PREREGISTRATION. 4568 02:29:14,845 --> 02:29:17,948 >> HI, MY NAME IS NID. 4569 02:29:18,015 --> 02:29:19,250 BALA, UNIVERSITY OF CALIFORNIA 4570 02:29:19,317 --> 02:29:20,017 SANTA CRUZ. 4571 02:29:20,084 --> 02:29:22,386 I REALLY APPRECIATED VIRGINIA'S 4572 02:29:22,453 --> 02:29:23,688 POINT ABOUT EXCELLENCE AND THE 4573 02:29:23,754 --> 02:29:25,089 WAY THAT WE'VE WEAPONIZED THE 4574 02:29:25,156 --> 02:29:27,658 WORD EXCELLENCE. 4575 02:29:27,725 --> 02:29:29,293 AND I APPRECIATED YOUR CONTRAST 4576 02:29:29,360 --> 02:29:31,662 WITH PROFICIENT OR COMPETENCE. 4577 02:29:31,729 --> 02:29:33,331 AND I THINK I WOULD LIKE TO 4578 02:29:33,397 --> 02:29:33,998 INTRODUCE TWO ADDITIONAL WORDS 4579 02:29:34,065 --> 02:29:35,166 THAT MIGHT BE USEFUL FOR US TO 4580 02:29:35,232 --> 02:29:36,600 THINK ABOUT FOR THE NEXT TWO 4581 02:29:36,667 --> 02:29:39,103 DAYS, WHICH IS INCENTIVES, I 4582 02:29:39,170 --> 02:29:40,738 THINK SOMETIMES WE USE 4583 02:29:40,805 --> 02:29:42,139 INCENTIVES AS AN EXCUSE BUT 4584 02:29:42,206 --> 02:29:43,908 THEY'RE ALSO INCREDIBLY 4585 02:29:43,974 --> 02:29:44,675 POWERFUL. 4586 02:29:44,742 --> 02:29:46,711 AND THEN STATUS. 4587 02:29:46,777 --> 02:29:48,012 I THINK THAT WE FAIL TO 4588 02:29:48,079 --> 02:29:48,779 RECOGNIZE THE IMPORTANT ROLE 4589 02:29:48,846 --> 02:29:54,685 THAT IS IT TUS PLAY STATUS PLACE 4590 02:29:54,752 --> 02:29:56,187 WOULD LIKE THEIR WORK TO BE 4591 02:29:56,253 --> 02:29:57,054 RECOGNIZED AND THAT OFTEN IS 4592 02:29:57,121 --> 02:29:58,489 EQUATED WITH STATUS IN BAD AND 4593 02:29:58,556 --> 02:30:02,626 GOOD WAYS. 4594 02:30:02,693 --> 02:30:04,261 AND BRINGING TOGETHER THOSE 4595 02:30:04,328 --> 02:30:05,329 IDEAS, I THINK ONE THING I'D 4596 02:30:05,396 --> 02:30:06,430 ALSO LIKE TO INTRODUCE IS THIS 4597 02:30:06,497 --> 02:30:07,865 IDEA THAT WHEN WE TALK ABOUT 4598 02:30:07,932 --> 02:30:10,801 INCENTIVES FOR SCIENCE, AND WE 4599 02:30:10,868 --> 02:30:12,670 TALK ABOUT AN ALTERNATIVE TO 4600 02:30:12,737 --> 02:30:14,071 EXCELLENCE, WHICH IS AN 4601 02:30:14,138 --> 02:30:16,273 INCREDIBLY VAGUE TERM, THAT WE 4602 02:30:16,340 --> 02:30:17,641 SHOULD THINK ABOUT SCIENCE AS 4603 02:30:17,708 --> 02:30:19,377 BEING CAPACITY-BUILDING. 4604 02:30:19,443 --> 02:30:19,944 RIGHT? 4605 02:30:20,010 --> 02:30:22,146 THAT WE WANT PEOPLE TO BUILD ON 4606 02:30:22,213 --> 02:30:24,949 OUR WORK, AND THAT MEANS, 4607 02:30:25,015 --> 02:30:27,485 NATURALLY, THAT WE SHOULD BE 4608 02:30:27,551 --> 02:30:29,387 PUBLISHING BOTH POSITIVE AND 4609 02:30:29,453 --> 02:30:31,956 NEGATIVE RESULTS FROM OUR WORK 4610 02:30:32,022 --> 02:30:34,225 BECAUSE THAT NEGATIVE RESULT -- 4611 02:30:34,291 --> 02:30:35,559 THOSE NEGATIVE RESULTS WILL 4612 02:30:35,626 --> 02:30:36,660 PROVIDE THE CAPACITY FOR THE 4613 02:30:36,727 --> 02:30:39,597 PEOPLE TO MAKE THE NEXT JUMPS IN 4614 02:30:39,663 --> 02:30:41,132 LOGIC AND TECHNIQUES AND 4615 02:30:41,198 --> 02:30:42,299 TECHNOLOGY THAT MIGHT BE 4616 02:30:42,366 --> 02:30:43,300 NECESSARY WITHOUT HAVING TO 4617 02:30:43,367 --> 02:30:45,836 REPEAT THAT. 4618 02:30:45,903 --> 02:30:47,138 SO I WOULD JUST LIKE TO 4619 02:30:47,204 --> 02:30:48,339 INTRODUCE THOSE CONCEPTS AS WE 4620 02:30:48,406 --> 02:30:49,106 CONTINUE TO HAVE THIS 4621 02:30:49,173 --> 02:30:52,710 CONVERSATION. 4622 02:30:52,777 --> 02:30:53,444 >> THANK YOU. 4623 02:30:53,511 --> 02:30:55,146 I THINK THAT REALLY IS PROBABLY 4624 02:30:55,212 --> 02:30:56,614 WHAT I WANTED TO SAY WITH OTHER 4625 02:30:56,680 --> 02:30:57,848 WORDS. 4626 02:30:57,915 --> 02:30:59,784 I THINK WHEN YOU HAVE SOMEONE 4627 02:30:59,850 --> 02:31:01,419 THAT SAVES THE DAY, THAT PERSON 4628 02:31:01,485 --> 02:31:03,154 TAKES THE MERIT, AND SOMETIMES 4629 02:31:03,220 --> 02:31:05,022 THERE'S BEEN MANY OTHER PEOPLE 4630 02:31:05,089 --> 02:31:08,426 BEHIND THAT PERSON THAT 4631 02:31:08,492 --> 02:31:10,161 SOMETIMES DICTATE THE WRONG TURN 4632 02:31:10,227 --> 02:31:11,562 SO THAT PERSON COULD TAKE THE 4633 02:31:11,629 --> 02:31:13,631 RIGHT ONE. 4634 02:31:13,697 --> 02:31:17,601 AND WE DON'T -- FOR PEOPLE THAT 4635 02:31:17,668 --> 02:31:18,769 DIDN'T GET FIRST. 4636 02:31:18,836 --> 02:31:20,004 BUT IN SCIENCE, I THINK WE 4637 02:31:20,070 --> 02:31:21,605 SHOULD FOCUS ON DOING THE WORK 4638 02:31:21,672 --> 02:31:23,140 PROPERLY, AND NOT DOING THINGS 4639 02:31:23,207 --> 02:31:24,341 THAT ARE GOING TO BE ON THE 4640 02:31:24,408 --> 02:31:27,745 COVER OF A MAGAZINE. 4641 02:31:27,812 --> 02:31:29,146 BUT I THINK IT'S THE TYPICAL 4642 02:31:29,213 --> 02:31:31,015 THING, IT'S MORE LIKE A SOCIETAL 4643 02:31:31,081 --> 02:31:32,516 THING, PROBABLY NOT THE SCIENCE, 4644 02:31:32,583 --> 02:31:35,052 WE DON'T WANT TO BE THE, YOU 4645 02:31:35,119 --> 02:31:37,221 KNOW, LABORIOUS -- AND DOING 4646 02:31:37,288 --> 02:31:38,389 THINGS. 4647 02:31:38,456 --> 02:31:40,024 WE'D RATHER BE THE FAMOUS SINGER 4648 02:31:40,090 --> 02:31:41,659 OR RESEARCHER, BUT RESEARCH 4649 02:31:41,725 --> 02:31:44,795 SHOULD BE A BIT BORING AND JUST 4650 02:31:44,862 --> 02:31:46,864 METICULOUS, I THINK. 4651 02:31:46,931 --> 02:31:49,366 BUT WE DON'T SELL THAT. 4652 02:31:49,433 --> 02:31:51,035 >> I'D ALSO JUST LIKE TO ADD TO 4653 02:31:51,101 --> 02:31:52,670 THAT AND I THINK ALSO IN 4654 02:31:52,736 --> 02:31:54,405 RESPONSE TO AN EARLIER QUESTION 4655 02:31:54,472 --> 02:31:55,840 ON THE VALUE OF THE NULL 4656 02:31:55,906 --> 02:31:59,076 STUDIES, I THINK AS SCIENTISTS 4657 02:31:59,143 --> 02:32:00,744 WE CAN SOMETIMES HAVE A 4658 02:32:00,811 --> 02:32:03,147 SHORT-TERM VIEW OF THE IMPACT OF 4659 02:32:03,214 --> 02:32:04,348 OUR RESULTS, AND I THINK WE 4660 02:32:04,415 --> 02:32:08,052 SHOULD REMEMBER THAT RARELY ARE 4661 02:32:08,118 --> 02:32:11,889 DECISIONS SUCH AS POLICY OR 4662 02:32:11,956 --> 02:32:13,324 CLINICAL GUIDELINES THAT ARE 4663 02:32:13,390 --> 02:32:14,792 PUBLISHED, RARELY ARE THEY BASED 4664 02:32:14,859 --> 02:32:17,461 OFF OF ONE SINGLE STUDY, IT'S 4665 02:32:17,528 --> 02:32:18,996 USUALLY AFTER A PROCESS OF 4666 02:32:19,063 --> 02:32:27,304 EVIDENCE SYNTHESIS, SYSTEMATIC 4667 02:32:27,371 --> 02:32:29,206 REVIEWS OF META-ANALYSIS THAT WE 4668 02:32:29,273 --> 02:32:31,876 LOOK AT THE ENTIRE EVIDENCE AND 4669 02:32:31,942 --> 02:32:33,544 THEN DETERMINE OKAY -- I THINK 4670 02:32:33,611 --> 02:32:35,946 AS RESEARCHERS, WE SHOULD KEEP 4671 02:32:36,013 --> 02:32:37,248 THAT BROAD VIEW THAT WE ARE VERY 4672 02:32:37,314 --> 02:32:38,883 MUCH CONTRIBUTING TO A BODY OF 4673 02:32:38,949 --> 02:32:39,149 RESEARCH. 4674 02:32:39,216 --> 02:32:40,718 IT'S NOT JUST OUR OWN WORK AND 4675 02:32:40,784 --> 02:32:42,620 OUR OWN SINGLE RESULT THAT'S 4676 02:32:42,686 --> 02:32:45,222 GOING TO DETERMINE EVERYTHING. 4677 02:32:45,289 --> 02:32:50,461 AND I THINK ALSO, I MEAN, I'M 4678 02:32:50,528 --> 02:32:52,129 NOT GETTING INTO STATISTICS 4679 02:32:52,196 --> 02:32:54,598 HEAVY HERE, BUT PATIENT 4680 02:32:54,665 --> 02:32:57,134 STATISTICS ARE BECOMING 4681 02:32:57,201 --> 02:32:58,769 INCREASINGLY COMMON AND 4682 02:32:58,836 --> 02:33:01,305 ENCOURAGED TO USE IN THE FIELD, 4683 02:33:01,372 --> 02:33:04,675 AND A HUGE PART OF PAY SHAN 4684 02:33:04,742 --> 02:33:06,243 STATISTICS IS HAVING PRIOR 4685 02:33:06,310 --> 02:33:07,945 EVIDENCE, PRIOR KNOWLEDGE TO 4686 02:33:08,012 --> 02:33:11,649 HELP INFORM -- TO INFORM YOUR 4687 02:33:11,715 --> 02:33:11,916 ANALYSIS. 4688 02:33:11,982 --> 02:33:13,317 IF YOU BASE YOUR PRIOR BELIEFS 4689 02:33:13,384 --> 02:33:14,818 ON A BIASED EVIDENCE BASE, 4690 02:33:14,885 --> 02:33:16,253 THAT'S NOT GOING TO HELP YOUR 4691 02:33:16,320 --> 02:33:18,989 FUTURE STUDIES EITHER. 4692 02:33:19,056 --> 02:33:19,790 SO IT IS VERY MUCH IMPORTANT 4693 02:33:19,857 --> 02:33:22,893 THAT WE HAVE AN ACCURATE PICTURE 4694 02:33:22,960 --> 02:33:27,932 OF WHAT IS OUT THERE BOTH FOR 4695 02:33:27,998 --> 02:33:29,333 DECISION-MAKING FOR FUTURE 4696 02:33:29,400 --> 02:33:30,634 ANALYSIS, FOR EVIDENCE 4697 02:33:30,701 --> 02:33:36,674 SYNTHESIS, IT'S VERY IMPORTANT. 4698 02:33:36,740 --> 02:33:37,641 >> THANKS, EVERYONE. 4699 02:33:37,708 --> 02:33:40,277 I'M THE CO-FOUNDER AND CSO OF 4700 02:33:40,344 --> 02:33:40,678 ARCADIAN SCIENCE. 4701 02:33:40,744 --> 02:33:42,546 I WANTED TO MAYBE TALK OUT OF 4702 02:33:42,613 --> 02:33:44,181 BOTH SIDES OF MY MOUTH AND 4703 02:33:44,248 --> 02:33:44,815 HOPEFULLY SYNTHESIZE SOMETHING 4704 02:33:44,882 --> 02:33:46,283 THAT WE HAVEN'T YET TOUCHED ON, 4705 02:33:46,350 --> 02:33:48,886 WHICH IS THAT I COMPLETELY TAKE 4706 02:33:48,953 --> 02:33:52,189 THE POINT OF JUST ACTUALLY PR 4707 02:33:52,256 --> 02:33:54,191 PRECONDITIONING RESULT ON SOME 4708 02:33:54,258 --> 02:33:55,593 SORT OF SHARED ASSUMPTIONS ABOUT 4709 02:33:55,659 --> 02:33:57,394 RIGOR OF METHODOLOGY AND THINGS 4710 02:33:57,461 --> 02:33:57,995 LIKE THAT. 4711 02:33:58,062 --> 02:34:00,731 OBVIOUSLY YOU CAN IMAGINE 4712 02:34:00,798 --> 02:34:02,333 OMITTING HALF OF THE RELEVANT 4713 02:34:02,399 --> 02:34:03,133 COMPONENTS AND REACTION AND NOT 4714 02:34:03,200 --> 02:34:04,335 GETTING SOMETHING TO WORK AND 4715 02:34:04,401 --> 02:34:05,035 THAT'S NOT NECESSARILY SOMETHING 4716 02:34:05,102 --> 02:34:07,037 THAT'S USEFUL TO PEOPLE. 4717 02:34:07,104 --> 02:34:14,578 NOW THAT SAID, THE IDEA OF HAV 4718 02:34:14,645 --> 02:34:15,679 HAVING -- MAYBE TO SPEAK TO THE 4719 02:34:15,746 --> 02:34:16,981 QUESTION ABOUT DOES IT HELP TO 4720 02:34:17,047 --> 02:34:18,515 HAVE EITHER SMALLER OR 4721 02:34:18,582 --> 02:34:19,617 UNDERPOWERED OR OTHER TYPES OF 4722 02:34:19,683 --> 02:34:21,151 STUDIES OR EVEN INFORMATION 4723 02:34:21,218 --> 02:34:23,654 ABOUT INCOMPLETE WORK OUT THERE, 4724 02:34:23,721 --> 02:34:25,623 SUGGESTING THAT THERE ARE OTHER 4725 02:34:25,689 --> 02:34:28,792 USES FOR SOME OF THIS NEGATIVE 4726 02:34:28,859 --> 02:34:30,227 DATA BEYOND JUST SORT OF 4727 02:34:30,294 --> 02:34:31,862 CONFIRMING THE KNOWLEDGE, MAKING 4728 02:34:31,929 --> 02:34:36,000 SURE THAT WE CAN -- WHAT WE ARE 4729 02:34:36,066 --> 02:34:37,334 LOOKING AT THAT INFORMATION AS, 4730 02:34:37,401 --> 02:34:38,769 IS IT TRUTH, ARE WE LOOKING TO 4731 02:34:38,836 --> 02:34:40,704 GET INSIGHTS OR SOMEBODY ELSE'S 4732 02:34:40,771 --> 02:34:42,006 ASSESSMENT OF THAT OUT, OR ARE 4733 02:34:42,072 --> 02:34:43,207 WE SAYING OKAY, WE KNOW THIS 4734 02:34:43,273 --> 02:34:44,642 DIDN'T WORK AND NOW OTHERS KNOW 4735 02:34:44,708 --> 02:34:47,344 WHAT ELSE TO TRY, AND YOU KNOW, 4736 02:34:47,411 --> 02:34:49,747 TRYING TO NOT BASE OUR 4737 02:34:49,813 --> 02:34:52,149 ASSUMPTIONS ON IF WE START TO 4738 02:34:52,216 --> 02:34:53,784 RELEASE MORE NULL DATA, MORE 4739 02:34:53,851 --> 02:34:55,152 NEGATIVE DATA, HAVING PEOPLE 4740 02:34:55,219 --> 02:34:56,854 UNDERSTAND HOW TO INTERPRET THAT 4741 02:34:56,920 --> 02:34:58,055 INFORMATION, EVEN IF YOU WERE TO 4742 02:34:58,122 --> 02:35:00,491 BE EXTREMELY RIGOROUS IN YOUR 4743 02:35:00,557 --> 02:35:01,558 METHODOLOGY, RIGHT, AND THEN PUT 4744 02:35:01,625 --> 02:35:04,395 OUT A NULL RESULT, TECHNOLOGY IS 4745 02:35:04,461 --> 02:35:07,865 ALWAYS INCREASING, WE ALWAYS ARE 4746 02:35:07,931 --> 02:35:08,799 ACCELERATED OUR RESOLUTION, 4747 02:35:08,866 --> 02:35:11,402 WE'RE REDUCING OUR MEASUREMENT 4748 02:35:11,468 --> 02:35:13,037 ERROR. WHAT WE MIGHT FIND 10 4749 02:35:13,103 --> 02:35:14,605 YEARS FROM NOW MIGHT BE 4750 02:35:14,672 --> 02:35:15,806 DIFFERENT FROM WHAT WE FIND OT 4751 02:35:15,873 --> 02:35:17,341 EVEN IF IT WAS VERY RIGOROUSLY 4752 02:35:17,408 --> 02:35:18,308 DONE BEFORE JUST BECAUSE THE 4753 02:35:18,375 --> 02:35:21,045 NATURE OF OUR REALITY IS 4754 02:35:21,111 --> 02:35:21,311 CHANGING. 4755 02:35:21,378 --> 02:35:22,346 SO I THINK THE POINT I'M 4756 02:35:22,413 --> 02:35:23,447 SUGGESTING IS MAYBE THE WAY WE 4757 02:35:23,514 --> 02:35:24,415 INTERPRET OR THE WAY WE TEACH 4758 02:35:24,481 --> 02:35:26,216 OTHERS TO INTERPRET THOSE NULL 4759 02:35:26,283 --> 02:35:27,951 RESULTS IS JUST AS IMPORTANT AS 4760 02:35:28,018 --> 02:35:31,188 WHAT IT IS THAT WE PUT OUT THE 4761 02:35:31,255 --> 02:35:36,927 THERE. 4762 02:35:36,994 --> 02:35:38,662 >> I THINK THIS IS A VERY 4763 02:35:38,729 --> 02:35:39,763 IMPORTANT POINT AND I ALSO THINK 4764 02:35:39,830 --> 02:35:41,865 THIS TIES IT BIT INTO 4765 02:35:41,932 --> 02:35:42,933 REPRODUCIBILITY ISSUES OR WHY 4766 02:35:43,000 --> 02:35:44,702 PEOPLE DON'T ATTEMPT TO 4767 02:35:44,768 --> 02:35:45,803 REPLICATE STUDIES THAT HAVE BEEN 4768 02:35:45,869 --> 02:35:47,438 DONE BEFORE, ESPECIALLY IF IT 4769 02:35:47,504 --> 02:35:48,872 HAD A NON-SIGNIFICANT RESULT. 4770 02:35:48,939 --> 02:35:52,342 WE THINK OF STUDIES WHERE THEY 4771 02:35:52,409 --> 02:35:54,678 DIDN'T SAY OKAY, THIS QUESTION 4772 02:35:54,745 --> 02:35:56,380 HAS ALREADY BEEN ANSWERED BASED 4773 02:35:56,447 --> 02:35:58,449 ON THIS ONE STUDY CONDUCTED IN 4774 02:35:58,515 --> 02:36:00,984 THIS WAY AND GAVE THIS RESULT, 4775 02:36:01,051 --> 02:36:02,186 BUT THAT'S NOT THE CASE. 4776 02:36:02,252 --> 02:36:03,954 STUDIES SHOULD BE REPLICATED 4777 02:36:04,021 --> 02:36:07,124 BECAUSE REALLY ALL A P VALUE IS 4778 02:36:07,191 --> 02:36:08,992 SAYING THAT THIS IS THE 4779 02:36:09,059 --> 02:36:10,728 PROBABILITY THAT YOUR STUDY 4780 02:36:10,794 --> 02:36:14,098 CONDUCTED IN THIS WAY WITH THIS 4781 02:36:14,164 --> 02:36:16,100 DATA WOULD PRODUCE THAT RESULT, 4782 02:36:16,166 --> 02:36:17,468 GIVEN THAT THE NULL HYPOTHESIS 4783 02:36:17,534 --> 02:36:18,168 IS TRUE. 4784 02:36:18,235 --> 02:36:19,770 IF YOU CONDUCT THE STUDY ON A 4785 02:36:19,837 --> 02:36:21,105 DIFFERENT SAMPLE OR WITH 4786 02:36:21,171 --> 02:36:24,508 SLIGHTLY DIFFERENT CONDITIONS, 4787 02:36:24,575 --> 02:36:27,644 OR EVEN SIMILAR CONDITIONS, DUE 4788 02:36:27,711 --> 02:36:29,847 TO CHANCE, YOU CAN END UP WITH A 4789 02:36:29,913 --> 02:36:30,814 DIFFERENT RESULT. 4790 02:36:30,881 --> 02:36:31,682 SO IT IS VERY IMPORTANT TO 4791 02:36:31,749 --> 02:36:33,016 ATTEMPT TO REPLICATE AND TRY 4792 02:36:33,083 --> 02:36:34,318 DIFFERENT STUDIES, NOT JUST 4793 02:36:34,384 --> 02:36:35,853 ASSUME THAT WHAT'S ALREADY BEEN 4794 02:36:35,919 --> 02:36:40,124 DONE HAS TOLD THE WHOLE TRUTH. 4795 02:36:40,190 --> 02:36:43,594 >> HI, GIDA SWAMY DUKE 4796 02:36:43,660 --> 02:36:43,894 UNIVERSITY. 4797 02:36:43,961 --> 02:36:45,129 I WANTED TO MAKE TWO COMMENTS. 4798 02:36:45,195 --> 02:36:46,764 ONE, I NE WE HAVE A DIFFERENT 4799 02:36:46,830 --> 02:36:48,966 SESSION ON SORT OF 4800 02:36:49,032 --> 02:36:49,533 NON-TRADITIONAL PUBLICATION 4801 02:36:49,600 --> 02:36:51,802 ROUTE BUT JUST TO PICK UP ON THE 4802 02:36:51,869 --> 02:36:55,405 ANALYSIS BETWEEN -- ON THE 4803 02:36:55,472 --> 02:36:56,507 ANTIDEPRESSANT DATA. 4804 02:36:56,573 --> 02:36:57,474 I'M ASSUMING THAT FDA LOOKED AT 4805 02:36:57,541 --> 02:37:00,110 THAT OR THAT WAS LOOKED AT FROM 4806 02:37:00,177 --> 02:37:00,878 CLINICALTRIALS.GOV REGISTRATION, 4807 02:37:00,944 --> 02:37:03,914 AND SO WITH THE IMPROVEMENTS IN 4808 02:37:03,981 --> 02:37:05,349 COMPLIANCE WITH 4809 02:37:05,415 --> 02:37:05,983 CLINICALTRIALS.GOV REQUIREMENTS 4810 02:37:06,049 --> 02:37:08,919 AT LEAST IN THE UNITED STATES, 4811 02:37:08,986 --> 02:37:10,654 WE HAVE A LOT OF DATA THERE ON 4812 02:37:10,721 --> 02:37:11,655 RESULTS THAT CLEARLY IS NOT 4813 02:37:11,722 --> 02:37:14,258 GETTING OUT, RIGHT, IN A SUMMARY 4814 02:37:14,324 --> 02:37:17,094 FORMAT, IN A PRESS RELEASE, IN 4815 02:37:17,161 --> 02:37:20,063 THE WAY THAT WE WANT TO CONVEY 4816 02:37:20,130 --> 02:37:21,598 THAT INFORMATION. 4817 02:37:21,665 --> 02:37:23,000 SO IT'S REALLY ONLY IF YOU'RE 4818 02:37:23,066 --> 02:37:24,101 GOING TO DESIGN A STUDY THAT YOU 4819 02:37:24,168 --> 02:37:25,836 MIGHT GO AND LOOK AT 4820 02:37:25,903 --> 02:37:26,737 CLINICALTRIALS.GOV TO FIGURE OUT 4821 02:37:26,804 --> 02:37:27,805 WHAT'S OUT THERE AND WHAT HAS 4822 02:37:27,871 --> 02:37:28,138 BEEN DONE. 4823 02:37:28,205 --> 02:37:30,107 SO I THINK THAT WITH LOTS OF 4824 02:37:30,174 --> 02:37:31,108 THINGS COMING ON BOARD, WE 4825 02:37:31,175 --> 02:37:33,010 SHOULD REALLY BE CONSIDERING HOW 4826 02:37:33,076 --> 02:37:36,213 TO GET THAT INTO A BETTER 4827 02:37:36,280 --> 02:37:38,949 MAINSTREAM FORMAT THAT IS 4828 02:37:39,016 --> 02:37:40,951 SOMETHING THAT THE LAY PUBLIC 4829 02:37:41,018 --> 02:37:42,986 CAN UNDERSTAND AND TO BE HONEST, 4830 02:37:43,053 --> 02:37:44,221 EVEN SCIENTISTS WHO MAYBE AREN'T 4831 02:37:44,288 --> 02:37:47,157 DESIGNING THE TRIALS FROM AN EPI 4832 02:37:47,224 --> 02:37:48,458 STANDPOINT CAN UNDERSTAND AS 4833 02:37:48,525 --> 02:37:49,993 WELL BECAUSE RIGHT NOW THE 4834 02:37:50,060 --> 02:37:52,963 TABULAR FORMAT AND SO FORTH IS 4835 02:37:53,030 --> 02:37:53,964 CUMBERSOME TO FIGURE OUT BUT 4836 02:37:54,031 --> 02:37:55,432 IT'S THERE AND IT'S INFORMATION 4837 02:37:55,499 --> 02:37:56,600 WE REALLY SHOULD BE CONVEYING 4838 02:37:56,667 --> 02:37:57,201 AND USING. 4839 02:37:57,267 --> 02:37:58,702 ONE OTHER THING I WANTED TO MAKE 4840 02:37:58,769 --> 02:38:00,304 COMMENT THAT SORT OF PICKS UP ON 4841 02:38:00,370 --> 02:38:01,505 THE LAST DISCUSSION WAS ABOUT 4842 02:38:01,572 --> 02:38:04,541 PUBLISHING THINGS THAT I THINK 4843 02:38:04,608 --> 02:38:06,543 PEOPLE FEEL ARE REDUNDANT. 4844 02:38:06,610 --> 02:38:07,611 SO REPLICATION IS THE RIGHT 4845 02:38:07,678 --> 02:38:09,146 WORD, BUT WHEN PEOPLE SEE THAT 4846 02:38:09,213 --> 02:38:10,814 THEY'VE ALREADY BEEN SCOOPED AND 4847 02:38:10,881 --> 02:38:12,182 SOMETHING'S BEEN PUBLISHED, THEY 4848 02:38:12,249 --> 02:38:14,718 SORT OF LOSE FEITH AND THAT 4849 02:38:14,785 --> 02:38:15,819 ANYONE WILL ACTUALLY PICK IT UP 4850 02:38:15,886 --> 02:38:17,120 OR TAKE IT OR AS A MATTER OF 4851 02:38:17,187 --> 02:38:19,122 FACT, IF IT SHOWS SOMETHING THAT 4852 02:38:19,189 --> 02:38:20,991 THERE'S IS NOT SIGNIFICANT AND 4853 02:38:21,058 --> 02:38:22,726 THE OTHER ONE THAT'S PUBLISHED 4854 02:38:22,793 --> 02:38:23,627 IS, THEN IT BECOMES WHAT DOES 4855 02:38:23,694 --> 02:38:24,161 THAT MEAN? 4856 02:38:24,228 --> 02:38:25,495 I'D ARGUE IT SHOULD ALL BE OUT 4857 02:38:25,562 --> 02:38:27,231 THERE, RIGHT, THAT'S THE WHOLE 4858 02:38:27,297 --> 02:38:29,733 BENEFIT OF PEER REVIEW AND FOR 4859 02:38:29,800 --> 02:38:30,801 LOOKING AT THIS ACROSS THE 4860 02:38:30,868 --> 02:38:31,001 BOARD. 4861 02:38:31,068 --> 02:38:32,236 SO I DO THINK THAT'S ANOTHER 4862 02:38:32,302 --> 02:38:33,837 AREA THAT COMES INTO PLAY, IS 4863 02:38:33,904 --> 02:38:36,473 THAT PEOPLE SIMPLY DON'T PUBLISH 4864 02:38:36,540 --> 02:38:37,474 BECAUSE SOMEBODY ALREADY 4865 02:38:37,541 --> 02:38:38,442 PUBLISHED IT IN THE TIME COURSE 4866 02:38:38,508 --> 02:38:41,111 THEY WERE DOING THEIR WORK AS 4867 02:38:41,178 --> 02:38:41,812 WELL. 4868 02:38:41,879 --> 02:38:43,614 >> I DIDN'T WHAT TO MAKE A 4869 02:38:43,680 --> 02:38:46,250 SLIGHT COMMENT ON THE 4870 02:38:46,316 --> 02:38:47,251 CLINICALTRIALS.GOV EXEUN, AND IF 4871 02:38:47,317 --> 02:38:53,156 YOU THINK ABOUT IT, WHAT SH AT 4872 02:38:53,223 --> 02:38:54,391 LEAST NINDS FUNDS, CLINICAL 4873 02:38:54,458 --> 02:38:55,492 TRIALS ARE NOT THE MAJORITY OF 4874 02:38:55,559 --> 02:38:56,193 THAT. 4875 02:38:56,260 --> 02:38:57,494 SO IMAGINE HOW MUCH WORK WE HAVE 4876 02:38:57,561 --> 02:38:59,796 TO DO IN TERMS OF THE 4877 02:38:59,863 --> 02:39:00,264 PRE-CLINICAL WORK. 4878 02:39:00,330 --> 02:39:01,665 A LOT OF THE ANIMAL STUDIES 4879 02:39:01,732 --> 02:39:04,835 WE'RE DOING, IN VITRO STUDIES, 4880 02:39:04,902 --> 02:39:08,772 COMPUTATIONAL STUDIES, SO EVEN 4881 02:39:08,839 --> 02:39:09,973 IF CLINICALTRIALS.GOV WERE 4882 02:39:10,040 --> 02:39:10,974 PERFECT, WE WOULD STILL HAVE A 4883 02:39:11,041 --> 02:39:13,944 LONG WAY TO GO. 4884 02:39:14,011 --> 02:39:16,213 >> I ALSO WANTED TO SAY, WE WERE 4885 02:39:16,280 --> 02:39:18,615 LUCKY WHEN WE WERE DOING A 4886 02:39:18,682 --> 02:39:19,616 SPECIAL ISSUE -- RESULTS OF 4887 02:39:19,683 --> 02:39:21,218 BEING ABLE TO PUBLISH A CLINICAL 4888 02:39:21,285 --> 02:39:22,352 TRIAL AS WELL, AND I THINK IT 4889 02:39:22,419 --> 02:39:25,188 WAS VERY COMMENDABLE THAT THEY 4890 02:39:25,255 --> 02:39:27,057 DID MAKE A HUGE EFFORT TO 4891 02:39:27,124 --> 02:39:28,358 PUBLISH THAT EVEN THOUGH THEY 4892 02:39:28,425 --> 02:39:30,327 KNEW IT WAS NEGATIVE DATA. 4893 02:39:30,394 --> 02:39:32,296 AND IT WAS A SMALL JOURNAL. 4894 02:39:32,362 --> 02:39:34,398 AND I THINK IT MAY HAVE TO DO 4895 02:39:34,464 --> 02:39:37,968 WITH THE FACT THAT IT WAS RARE 4896 02:39:38,035 --> 02:39:40,504 DISEASE AND THERE ARE A LOT OF 4897 02:39:40,570 --> 02:39:42,339 STAKEHOLDERS AND THEY ARE 4898 02:39:42,406 --> 02:39:43,573 PUSHING AND IN MANY CASES 4899 02:39:43,640 --> 02:39:44,441 SPONSORING SOME OF THOSE TRIALS. 4900 02:39:44,508 --> 02:39:47,744 SO I THINK SPHIEMS SOMETIMES WE 4901 02:39:47,811 --> 02:39:49,046 ARE STAKEHOLDERS, BUT THEY HAVE, 4902 02:39:49,112 --> 02:39:54,484 YOU KNOW, THEIR OWN AGENDAS, 4903 02:39:54,551 --> 02:39:55,118 IT'S IMPORTANT SOMETIMES TO KNOW 4904 02:39:55,185 --> 02:39:56,687 WHO IS BEHIND A CLINICAL TRIAL, 4905 02:39:56,753 --> 02:39:59,323 AND THAT USUALLY MAKES IT MUCH 4906 02:39:59,389 --> 02:40:02,359 MORE ACCESSIBLE. 4907 02:40:02,426 --> 02:40:07,130 AND ALSO ON THE POINT OF -- IS 4908 02:40:07,197 --> 02:40:09,833 THE JOURNAL THE LAST DESTINATION 4909 02:40:09,900 --> 02:40:10,467 FOR SOMETHING? 4910 02:40:10,534 --> 02:40:14,972 IF THOSE DATA ON 4911 02:40:15,038 --> 02:40:15,439 CLINICALTRIALS.GOV WAS 4912 02:40:15,505 --> 02:40:16,073 ACCESSIBLE IN THE WAY THAT IT 4913 02:40:16,139 --> 02:40:19,676 SHOULD BE, PROBABLY AS THEY SAID 4914 02:40:19,743 --> 02:40:20,644 BEFORE, DATA AND METHODS ARE 4915 02:40:20,711 --> 02:40:22,612 WHAT ARE YOU'RE INTERESTED IN, 4916 02:40:22,679 --> 02:40:26,750 THE REST IS MARKET BEING, SO MAO 4917 02:40:26,817 --> 02:40:31,588 PROBABLY WE COULD DO WITHOUT THE 4918 02:40:31,655 --> 02:40:36,093 FINAL JOURNAL -- 4919 02:40:36,159 --> 02:40:38,495 >> MALUDE TAN SEE, UNIVERSITY OF 4920 02:40:38,562 --> 02:40:39,363 FLORIDA. 4921 02:40:39,429 --> 02:40:39,963 I'M GOING TO REFLECT THE VIEWS 4922 02:40:40,030 --> 02:40:40,998 OF SOME OF THE PEOPLE THAT 4923 02:40:41,064 --> 02:40:42,432 REPLIED TO MY POST ON SOCIAL 4924 02:40:42,499 --> 02:40:44,668 MEDIA ABOUT THIS BIAS 4925 02:40:44,735 --> 02:40:45,602 CONFERENCE. 4926 02:40:45,669 --> 02:40:46,169 WORKSHOP. 4927 02:40:46,236 --> 02:40:48,538 AND ONE OF THEM IS THAT IT'S 4928 02:40:48,605 --> 02:40:50,707 BEEN THE EXPERIENCE OF SEVERAL 4929 02:40:50,774 --> 02:40:52,009 COLLEAGUES OF MINE IN THE 4930 02:40:52,075 --> 02:40:54,444 BIOMEDICAL RESEARCH THAT 4931 02:40:54,511 --> 02:40:57,948 SOMETIMES TO DISPUTE OR REJECT 4932 02:40:58,015 --> 02:40:59,583 OR PUBLISH CONTRADICTORY 4933 02:40:59,649 --> 02:41:01,985 FINDINGS TO WHAT HAS BEEN 4934 02:41:02,052 --> 02:41:04,054 PUBLISHED BY SOMEONE WITH 4935 02:41:04,121 --> 02:41:06,156 STATUS, IT TAKES SEVERAL LABS TO 4936 02:41:06,223 --> 02:41:09,092 DO IT TOGETHER. 4937 02:41:09,159 --> 02:41:12,295 AND I THINK -- I MEAN, I'M ALL 4938 02:41:12,362 --> 02:41:13,930 FOR MULTIDISCIPLINARY TEAM 4939 02:41:13,997 --> 02:41:15,032 SCIENCE, AND SOMETIMES THAT'S 4940 02:41:15,098 --> 02:41:17,634 WHAT IT TAKES BECAUSE A SINGLE 4941 02:41:17,701 --> 02:41:23,006 LAB MAY NOT HAVE THE, I GUESS, 4942 02:41:23,073 --> 02:41:24,207 CREDIBILITY ON THEIR OWN TO 4943 02:41:24,274 --> 02:41:25,075 DISPUTE SOMETHING THAT WAS 4944 02:41:25,142 --> 02:41:28,512 PUBLISHED IN A MAJOR JOURNAL. 4945 02:41:28,578 --> 02:41:32,416 SO I THINK EVEN THEN, WHEN THAT 4946 02:41:32,482 --> 02:41:34,051 HAPPENS, THERE'S ONE PARTICULAR 4947 02:41:34,117 --> 02:41:35,485 INSTANCE IN WHICH THERE'S NOW 4948 02:41:35,552 --> 02:41:36,686 BEEN SEVERAL PUBLICATIONS SAYING 4949 02:41:36,753 --> 02:41:43,260 THIS IS NOT THE CASE, THAT DOGMA 4950 02:41:43,326 --> 02:41:46,663 PERSISTS IN THE FIELD. 4951 02:41:46,730 --> 02:41:47,297 DESPITE THE FACT THAT YOU TOOK 4952 02:41:47,364 --> 02:41:49,833 THE TROUBLE TO PUBLISH THE 4953 02:41:49,900 --> 02:41:52,102 CONTRADICTORY FINDINGS BY 10 4954 02:41:52,169 --> 02:41:53,570 DIFFERENT LABS IN ONE PAPER. 4955 02:41:53,637 --> 02:41:58,475 SO HOW DO WE GET BEYOND THE NEED 4956 02:41:58,542 --> 02:42:03,613 TO CONFIRM OR DISPUTE IF THE 4957 02:42:03,680 --> 02:42:05,782 FIELD IS STILL SO STUCK ON 4958 02:42:05,849 --> 02:42:07,851 GIVING CREDIBILITY TO IDEAS OF 4959 02:42:07,918 --> 02:42:11,688 PEOPLE WITH STATUS THAT'S NOT 4960 02:42:11,755 --> 02:42:19,362 DATA-DRIVEN? 4961 02:42:19,429 --> 02:42:21,231 ANY THOUGHTS? 4962 02:42:21,298 --> 02:42:22,632 >> I DON'T HAVE AN ANSWER FOR 4963 02:42:22,699 --> 02:42:22,833 THAT. 4964 02:42:22,899 --> 02:42:25,602 I THINK THAT'S LIFE IN GENERAL. 4965 02:42:25,669 --> 02:42:27,137 I DON'T THINK IT'S ONLY -- I 4966 02:42:27,204 --> 02:42:29,005 DON'T THINK IT'S ONLY SCIENCE. 4967 02:42:29,072 --> 02:42:31,508 I THINK IT HAPPENS PRETTY MUCH 4968 02:42:31,575 --> 02:42:32,375 EVERYWHERE, AND I WISH IT 4969 02:42:32,442 --> 02:42:37,547 DIDN'T, BUT IT DOES. 4970 02:42:37,614 --> 02:42:39,316 >> I'LL JUST ADD, I FIND THAT 4971 02:42:39,382 --> 02:42:40,317 COMMENT SURPRISING COMING FROM 4972 02:42:40,383 --> 02:42:42,652 OUTSIDE THE BIOLOGICAL SCIENCES, 4973 02:42:42,719 --> 02:42:43,753 BECAUSE I GUESS PEOPLE IN THE 4974 02:42:43,820 --> 02:42:45,689 SOCIAL SCIENCES ARE LIKE, WELL, 4975 02:42:45,755 --> 02:42:46,890 WE DO STUFF THAT'S LOW STAKES 4976 02:42:46,957 --> 02:42:48,425 AND IT'S VERY HARD TO ASSESS THE 4977 02:42:48,492 --> 02:42:49,192 TRUTH. 4978 02:42:49,259 --> 02:42:50,927 BUT IN THE SCIENCES, EVERYTHING 4979 02:42:50,994 --> 02:42:52,996 IS PERFECT BECAUSE IF A DRUG 4980 02:42:53,063 --> 02:42:54,331 DOESN'T WORK, YOU HAVE THE BIAS 4981 02:42:54,397 --> 02:42:55,232 OF REALITY. 4982 02:42:55,298 --> 02:42:57,400 WHICH IS LIKE, HUMANS WILL 4983 02:42:57,467 --> 02:42:58,401 REALIZE THAT IT'S ACTUALLY 4984 02:42:58,468 --> 02:43:00,303 CAUSING PROBLEMS OR NON-EFFICACY 4985 02:43:00,370 --> 02:43:02,038 AND THEN LIKE THE REALITY OF THE 4986 02:43:02,105 --> 02:43:03,440 WORLD WILL CORRECT FOR ALL THESE 4987 02:43:03,507 --> 02:43:04,674 BIASES AND STATUSES. 4988 02:43:04,741 --> 02:43:07,277 SO YOUR COMMENT WAS SURPRISING 4989 02:43:07,344 --> 02:43:08,278 BECAUSE YOU KNOW, IT'S NOT WHAT 4990 02:43:08,345 --> 02:43:10,447 I WOULD HAVE EXPECTED, AND THEN 4991 02:43:10,514 --> 02:43:11,748 I JUST POINT THIS OUT WHICH IS A 4992 02:43:11,815 --> 02:43:12,849 LOT OF THE COMMENTS PEOPLE HAVE 4993 02:43:12,916 --> 02:43:16,786 BEEN MAKING ARE SOCIOLOGICAL 4994 02:43:16,853 --> 02:43:17,687 THEORIES. 4995 02:43:17,754 --> 02:43:21,958 EVEN TED MIGUEL OPENED WITH 4996 02:43:22,025 --> 02:43:24,361 ROBERT MERTON'S BELIEFS AS A 4997 02:43:24,427 --> 02:43:24,661 SOCIOLOGIST. 4998 02:43:24,728 --> 02:43:28,098 SO I DON'T KNOW HOW MANY 4999 02:43:28,165 --> 02:43:28,765 SOCIOLOGISTS ARE IN THE ROOM 5000 02:43:28,832 --> 02:43:30,600 RIGHT NOW BUT ACTUALLY KIND OF 5001 02:43:30,667 --> 02:43:32,135 SURPRISING IF YOU WANT TO SOLVE 5002 02:43:32,202 --> 02:43:32,936 A SOCIOLOGICAL PROBLEM, YOU HAVE 5003 02:43:33,003 --> 02:43:34,337 NO EXPERTS IN SOCIOLOGY IN THE 5004 02:43:34,404 --> 02:43:34,538 ROOM. 5005 02:43:34,604 --> 02:43:36,439 >> I WAS GOING TO SAY THAT 5006 02:43:36,506 --> 02:43:38,842 ACTUALLY ONE OF THE REASONS WE 5007 02:43:38,909 --> 02:43:40,477 HAVE A LOT OF PEOPLE FROM NOT 5008 02:43:40,544 --> 02:43:42,445 JUST THE BIOMEDICAL SCIENCES 5009 02:43:42,512 --> 02:43:44,381 HERE TODAY IS BECAUSE THIS IS 5010 02:43:44,447 --> 02:43:46,750 ACTUALLY SUCH A BROAD ISSUE, AND 5011 02:43:46,816 --> 02:43:48,585 THAT THE BIOMEDICAL SCIENCES ARE 5012 02:43:48,652 --> 02:43:50,353 NOT PERFECT AND WE NEED 5013 02:43:50,420 --> 02:43:52,222 PERSPECTIVES FROM OUTSIDE OF 5014 02:43:52,289 --> 02:43:53,089 BIOMEDICAL RESEARCH TO SOLVE 5015 02:43:53,156 --> 02:43:57,360 THIS PROBLEM. 5016 02:43:57,427 --> 02:44:00,130 >> I'M RACHEL MILES, VIRGINIA 5017 02:44:00,197 --> 02:44:01,097 TECH UNIVERSITY LIBRARIES. 5018 02:44:01,164 --> 02:44:02,632 I SPECIALIZE IN RESEARCH IMPACT 5019 02:44:02,699 --> 02:44:04,467 AND ASSESSMENT. 5020 02:44:04,534 --> 02:44:05,535 AND MY QUESTION MIGHT BE A 5021 02:44:05,602 --> 02:44:06,770 LITTLE BIT ABSTRACT BUT IT KIND 5022 02:44:06,836 --> 02:44:08,438 OF GOES OFF WHAT THE LAST 5023 02:44:08,505 --> 02:44:09,773 QUESTION WAS ASKING ABOUT 5024 02:44:09,839 --> 02:44:10,974 CULTURE AND DOGMAS. 5025 02:44:11,041 --> 02:44:14,477 AND I WONDER IF ANY OF YOU HAVE 5026 02:44:14,544 --> 02:44:16,646 HAD AN OPPORTUNITY OR MAYBE A 5027 02:44:16,713 --> 02:44:18,882 WAY OF FIGURING OUT A WAY TO 5028 02:44:18,949 --> 02:44:20,283 QUANTIFY DOGMAS OR CULTURE IN 5029 02:44:20,350 --> 02:44:21,384 YOUR RESEARCH. 5030 02:44:21,451 --> 02:44:22,852 AND IF THAT'S POSSIBLE BECAUSE I 5031 02:44:22,919 --> 02:44:25,789 KNOW WE CAN ALL SAY WE SHOULD BE 5032 02:44:25,855 --> 02:44:26,990 DOING THIS OR SHOULD BE DOING 5033 02:44:27,057 --> 02:44:28,525 THAT BUT IT'S EASIER SAID THAN 5034 02:44:28,592 --> 02:44:30,026 DONE TO SAY THAT WE SHOULD BE 5035 02:44:30,093 --> 02:44:30,827 DOING THESE THINGS BECAUSE I 5036 02:44:30,894 --> 02:44:32,362 WORK IN THE REALM OF 5037 02:44:32,429 --> 02:44:33,697 INSTITUTIONAL CHANGE, AND I 5038 02:44:33,763 --> 02:44:35,232 BELIEVE THAT NEIL'S RESEARCH 5039 02:44:35,298 --> 02:44:38,868 SHOWED THAT IBS TEUTIONAL CHANGE 5040 02:44:38,935 --> 02:44:40,704 IS LESS LIKELY TO OCCUR AROUND 5041 02:44:40,770 --> 02:44:41,838 CULTURAL CHANGE HAS TO HAPPEN 5042 02:44:41,905 --> 02:44:42,439 FIRST. 5043 02:44:42,505 --> 02:44:44,207 AND I AM A TRUE BELIEVER IN THAT 5044 02:44:44,274 --> 02:44:47,310 AND THAT'S WHY I KIND OF WORK IN 5045 02:44:47,377 --> 02:44:48,812 THIS REALM OF LOOKING AT 5046 02:44:48,878 --> 02:44:50,247 RESEARCH ASSESSMENT REFORM AND 5047 02:44:50,313 --> 02:44:51,615 THIS ALL GOES BACK TO CULTURAL 5048 02:44:51,681 --> 02:44:53,550 CHANGE AND IF YOU CAN LOOK AT -- 5049 02:44:53,617 --> 02:44:55,619 IF WE CAN ZOOM OUT AT A MACRO 5050 02:44:55,685 --> 02:44:58,154 LEVEL AND KIND OF LOOK AT ROBERT 5051 02:44:58,221 --> 02:44:59,556 MERTON'S -- THAT KIND OF MATTHEW 5052 02:44:59,623 --> 02:45:01,558 EFFECT, AND LOOK AT THIS FROM A 5053 02:45:01,625 --> 02:45:03,226 MACRO LEVEL IN THINKING ABOUT 5054 02:45:03,293 --> 02:45:04,527 FROM A SOCIOLOGICAL PERSPECTIVE 5055 02:45:04,594 --> 02:45:06,463 OF HOW WE CAN QUANTIFY THESE 5056 02:45:06,529 --> 02:45:10,767 CULTURAL ASPECTS AND THESE 5057 02:45:10,834 --> 02:45:11,701 DOGMAS, CAN WE ACTUALLY TARGET 5058 02:45:11,768 --> 02:45:13,803 SOME OF THE CULTURAL ASPECTS AND 5059 02:45:13,870 --> 02:45:16,573 MAYBE FIGURE OUT WHERE WE NEED 5060 02:45:16,640 --> 02:45:17,907 TO CHANGE SPECIFICALLY, RATHER 5061 02:45:17,974 --> 02:45:19,309 THAN JUST SAYING WE NEED TO 5062 02:45:19,376 --> 02:45:20,410 CHANGE, WHERE DO WE NEED TO 5063 02:45:20,477 --> 02:45:23,813 CHANGE. 5064 02:45:23,880 --> 02:45:27,083 >> JUST ADDING TO THAT, IS THERE 5065 02:45:27,150 --> 02:45:29,386 A WAY ALSO TO TAKE INTO ACCOUNT 5066 02:45:29,452 --> 02:45:31,921 CULTURAL DIFFERENCES ACROSS 5067 02:45:31,988 --> 02:45:33,223 CONTINENTS, BECAUSE THE WAY THE 5068 02:45:33,290 --> 02:45:35,125 WESTERN WORLD IS DOING SCIENCE 5069 02:45:35,191 --> 02:45:36,559 IS NOT THE SAME AS THE RUSSIAN 5070 02:45:36,626 --> 02:45:38,762 WORLD IS DOING SCIENCE, AND THE 5071 02:45:38,828 --> 02:45:48,038 ASIAN WORLD IS DOING SCIENCE. 5072 02:45:48,104 --> 02:45:51,408 >> THE SOCIAL CHANGES, I CAN 5073 02:45:51,474 --> 02:45:54,377 ONLY TELL YOU WHAT WE TRIED TO 5074 02:45:54,444 --> 02:45:56,146 DO IN THE NETWORK I WAS 5075 02:45:56,212 --> 02:45:56,579 CHAIRING. 5076 02:45:56,646 --> 02:46:01,184 SO WE DID INCLUDE IN NATURE -- 5077 02:46:01,251 --> 02:46:02,919 MEETINGS WE HAD A SECTION ON 5078 02:46:02,986 --> 02:46:04,154 NEGATIVE RESULTS, WE TRIED TO 5079 02:46:04,220 --> 02:46:06,656 INVOLVE ALL THE YOUNGER 5080 02:46:06,723 --> 02:46:07,991 RESEARCHERS TO PRESENT THEIR 5081 02:46:08,058 --> 02:46:11,094 DATA, EVEN IF IT WAS, YOU KNOW, 5082 02:46:11,161 --> 02:46:15,899 WE TRIED TO MAKE IT NON- -- 5083 02:46:15,965 --> 02:46:17,834 TRIED NOT TO BE TOO HARSH ON 5084 02:46:17,901 --> 02:46:19,169 THEM, BASICALLY BECAUSE I THINK 5085 02:46:19,235 --> 02:46:20,236 THE MAIN IDEA IS THAT IF YOU 5086 02:46:20,303 --> 02:46:21,271 HAVE THE RIGHT CONTROLS AND THEN 5087 02:46:21,338 --> 02:46:23,873 YOU HAVE THE RIGHT DATA, YOU 5088 02:46:23,940 --> 02:46:26,109 DON'T NEED THE RIGHT RESULTS. 5089 02:46:26,176 --> 02:46:27,844 YOU JUST NEED TO SHARE THEM WITH 5090 02:46:27,911 --> 02:46:28,478 EVERYBODY ELSE. 5091 02:46:28,545 --> 02:46:32,449 SO WE INCLUDED SESSIONS ON 5092 02:46:32,515 --> 02:46:34,317 NEGATIVE DATA IN ONE OF OUR 5093 02:46:34,384 --> 02:46:36,619 MEETINGS AND WE ALSO ORGANIZED 5094 02:46:36,686 --> 02:46:38,154 WEBINARS AND WE ESPECIALLY WORK 5095 02:46:38,221 --> 02:46:39,923 ON NEGATIVE OR CONTRADICTORY 5096 02:46:39,989 --> 02:46:40,256 RESULTS. 5097 02:46:40,323 --> 02:46:42,859 AND I THINK THERE WERE QUITE A 5098 02:46:42,926 --> 02:46:43,893 LOT OF THINGS THAT CAME OUT OF 5099 02:46:43,960 --> 02:46:45,061 IT AND WE ARE QUITE HAPPY 5100 02:46:45,128 --> 02:46:47,397 BECAUSE I THINK IN MANY OF THE 5101 02:46:47,464 --> 02:46:48,698 RESEARCHERS THAT HAD COME 5102 02:46:48,765 --> 02:46:51,101 THROUGH THAT, I THINK THEY HAVE 5103 02:46:51,167 --> 02:46:52,736 A DIFFERENT VIEW AND AT LEAST 5104 02:46:52,802 --> 02:46:54,170 IT'S SOMETHING EMBEDDED IN THEIR 5105 02:46:54,237 --> 02:46:55,405 MINDS. 5106 02:46:55,472 --> 02:46:57,474 SO HOPEFULLY WE'LL BE ABLE TO 5107 02:46:57,540 --> 02:47:00,543 MAKE IT -- I THINK IT'S 5108 02:47:00,610 --> 02:47:03,613 HIGHLIGHTED, GIVEN A BIT -- IT'S 5109 02:47:03,680 --> 02:47:05,248 A WAY TO GO FORWARD. 5110 02:47:05,315 --> 02:47:09,953 BUT IT'S JUST ONE WAY. 5111 02:47:10,019 --> 02:47:11,921 >> THANK YOU. 5112 02:47:11,988 --> 02:47:16,292 SO I TOTALLY AGREE, THIS IS A 5113 02:47:16,359 --> 02:47:17,394 SOCIAL CULTURAL BEHAVIORAL 5114 02:47:17,460 --> 02:47:20,563 PROBLEM WITH MAJOR INSTITUTIONS, 5115 02:47:20,630 --> 02:47:23,767 PUBLISHERS, FUNDERS, ACADEMIC 5116 02:47:23,833 --> 02:47:27,036 INSTITUTIONS, AS THE 5117 02:47:27,103 --> 02:47:27,604 THREE-LEGGED STOOL OF THE 5118 02:47:27,670 --> 02:47:28,905 PROBLEM, SO WE CAN ALL 5119 02:47:28,972 --> 02:47:29,706 CONTRIBUTE TO THE PROBLEM OR THE 5120 02:47:29,773 --> 02:47:31,341 SOLUTION. 5121 02:47:31,408 --> 02:47:32,876 TO THE POINT, I THINK 5122 02:47:32,942 --> 02:47:34,644 REPLICATION RESEARCH IS, TO 5123 02:47:34,711 --> 02:47:37,947 START WITH, NOT ATTEMPTED OFTEN 5124 02:47:38,014 --> 02:47:40,984 BECAUSE NOBODY IN ACADEMIA WAS 5125 02:47:41,050 --> 02:47:43,953 PROMOTED THAT TENURE TRACK FOR 5126 02:47:44,020 --> 02:47:45,622 DOING REPLICATION OR 5127 02:47:45,688 --> 02:47:46,256 REPRODUCIBLE RESEARCH. 5128 02:47:46,322 --> 02:47:48,358 THEN WHEN IT IS ATTEMPTED, THERE 5129 02:47:48,425 --> 02:47:50,960 IS THAT PRIMACY OF PUBLISHING IN 5130 02:47:51,027 --> 02:47:52,629 THE FANCY PAPER AND PUBLISHING 5131 02:47:52,695 --> 02:47:56,466 FIRST, AND SORT OF THE BURDEN OF 5132 02:47:56,533 --> 02:48:00,270 EVIDENCE OR -- TO ACCEPTING THAT 5133 02:48:00,336 --> 02:48:01,738 THERE IS CONTROVERSY AND THAT 5134 02:48:01,805 --> 02:48:03,273 THE DATA -- THE CONCLUSIONS ARE 5135 02:48:03,339 --> 02:48:06,609 NOT SUPPORTED BY NEW DATA. 5136 02:48:06,676 --> 02:48:08,178 BUT THE THIRD ONE TO THE POINT 5137 02:48:08,244 --> 02:48:11,047 OF THE COLLEAGUE FROM PLUS ONE 5138 02:48:11,114 --> 02:48:16,920 IS THAT, YES, STUDIES ARE 5139 02:48:16,986 --> 02:48:18,021 IMPERFECT WHEN THEY'RE SMALL, 5140 02:48:18,087 --> 02:48:19,222 WHETHER THEY HAVE POSITIVE 5141 02:48:19,289 --> 02:48:24,494 CONCLUSIONS OR IPT PRE TAITIONSS 5142 02:48:24,561 --> 02:48:25,762 OR NEGATIVE INTERPRETATIONS, 5143 02:48:25,829 --> 02:48:27,263 THEY COULD BE USEFUL EXCEPT 5144 02:48:27,330 --> 02:48:28,565 THEY'RE NOT, AND THE REASON IS 5145 02:48:28,631 --> 02:48:31,768 THAT THE PROVENANCE OF THE 5146 02:48:31,835 --> 02:48:34,704 FINDINGS FROM THE PRIMARY DATA 5147 02:48:34,771 --> 02:48:38,107 METHODS, ET CETERA, IS VERY 5148 02:48:38,174 --> 02:48:38,808 OPAQUE. 5149 02:48:38,875 --> 02:48:42,679 NOT ONLY ANIMAL MODEL, RESEARCH, 5150 02:48:42,745 --> 02:48:44,047 WHERE IT'S ALMOST -- NEVER YOU 5151 02:48:44,113 --> 02:48:45,615 GET THE PRIMARY DATA, SO THE 5152 02:48:45,682 --> 02:48:46,916 IDEA OF OPEN NOTEBOOKS AND THIS 5153 02:48:46,983 --> 02:48:48,651 IS WHERE BOTH PUBLISHERS AND 5154 02:48:48,718 --> 02:48:50,186 PRINTERS CAN REALLY CONTRIBUTE, 5155 02:48:50,253 --> 02:48:51,354 BUT ALSO INCREASINGLY THE BIG 5156 02:48:51,421 --> 02:48:53,957 DATA RESEARCH, WHERE THERE IS A 5157 02:48:54,023 --> 02:48:57,393 LOT OF NEW FINDINGS, FAST 5158 02:48:57,460 --> 02:49:00,463 FINDINGS, WHERE OFTEN THE DATA 5159 02:49:00,530 --> 02:49:01,664 AVAILABILITY SECTION, EVEN IN 5160 02:49:01,731 --> 02:49:03,833 FANCY JOURNALS, IS KIND OF EMPTY 5161 02:49:03,900 --> 02:49:05,535 OR IT TAKES YOU -- PROVIDES A 5162 02:49:05,602 --> 02:49:08,938 LINK THAT LEADS TO NOWHERE. 5163 02:49:09,005 --> 02:49:10,673 FOR WHERE THE DATA IS. 5164 02:49:10,740 --> 02:49:14,110 SO ANYWAY, IT'S GREAT TO BE PART 5165 02:49:14,177 --> 02:49:14,777 OF THE DISCUSSION. 5166 02:49:14,844 --> 02:49:17,714 I SHOULD ALSO SAY FROM A -- I 5167 02:49:17,780 --> 02:49:20,483 DON'T KNOW IF I SAID THAT, BUT 5168 02:49:20,550 --> 02:49:24,087 ONLY FOSSAE T TO SAY THAT WE HAN 5169 02:49:24,153 --> 02:49:26,222 ANALYSIS IN TERMS OF THE QUALITY 5170 02:49:26,289 --> 02:49:29,058 OF THE RIGOR OF ANIMAL MODEL 5171 02:49:29,125 --> 02:49:35,498 EFFICACY STUDIES COMPARING -- 5172 02:49:35,565 --> 02:49:37,033 AND NOT SO HIGH JOURNALS, NOT 5173 02:49:37,100 --> 02:49:38,134 MUCH DIFFERENCE, SPEAKING TO 5174 02:49:38,201 --> 02:49:39,335 THAT AGAIN. 5175 02:49:39,402 --> 02:49:41,638 THERE ARE SOME REAL CULTURAL 5176 02:49:41,704 --> 02:49:43,740 BARRIERS IN INERTIA. 5177 02:49:43,806 --> 02:49:45,074 LASTLY ON THE ACADEMIC 5178 02:49:45,141 --> 02:49:46,809 INSTITUTION SIDE, AS WELL AS THE 5179 02:49:46,876 --> 02:49:49,345 FUNDER SIDES, A GREAT 5180 02:49:49,412 --> 02:49:51,814 OPPORTUNITY TO IMPROVE AND BUILD 5181 02:49:51,881 --> 02:49:54,183 NEW TRAINING PROGRAMS, AND JUST 5182 02:49:54,250 --> 02:49:55,485 TEACH THE SCIENTIFIC METHOD. 5183 02:49:55,552 --> 02:49:58,354 IT FEELS LIKE I DON'T THINK 5184 02:49:58,421 --> 02:50:00,323 THERE IS A SUBJECT CALLED THAT. 5185 02:50:00,390 --> 02:50:02,525 IT SHOULD START IN HIGH SCHOOL 5186 02:50:02,592 --> 02:50:04,527 OR BEFORE AND CONTINUE THROUGH 5187 02:50:04,594 --> 02:50:11,434 UNDERGRAD, GRAD AND FOREVER. 5188 02:50:11,501 --> 02:50:15,905 >> OKAY, CAN I -- OH, YOU WANT 5189 02:50:15,972 --> 02:50:16,806 TO ANSWER? 5190 02:50:16,873 --> 02:50:17,006 SORRY. 5191 02:50:17,073 --> 02:50:18,908 >> MY BRAIN IS JUST -- THE 5192 02:50:18,975 --> 02:50:21,844 QUESTION ABOUT HOW TO MEASURE OR 5193 02:50:21,911 --> 02:50:23,746 ASSESS CULTURAL CHANGE, I WAS 5194 02:50:23,813 --> 02:50:25,048 JUST BRAINSTORMING THINKING 5195 02:50:25,114 --> 02:50:27,784 PERHAPS THAT COULD BE AN 5196 02:50:27,850 --> 02:50:29,319 OPPORTUNITY FOR A.I. OR NATURAL 5197 02:50:29,385 --> 02:50:33,056 LANGUAGE PROCESSING TO SCOUR 5198 02:50:33,122 --> 02:50:35,491 ARTICLES, MOST RECENT ARTICLES 5199 02:50:35,558 --> 02:50:36,559 COMPARED TO PREVIOUSLY AND SEE 5200 02:50:36,626 --> 02:50:39,996 IF THERE ARE ANY DIFFERENCES IN 5201 02:50:40,063 --> 02:50:41,564 HOW RESEARCHERS DISCUSS THEIR 5202 02:50:41,631 --> 02:50:43,066 RESULTS OR PRESENT THEM OR ARGUE 5203 02:50:43,132 --> 02:50:44,801 FOR THE STRENGTH OF THEIR 5204 02:50:44,867 --> 02:50:46,169 CONCLUSIONS. 5205 02:50:46,235 --> 02:50:47,570 JUST -- I DON'T KNOW, JUST 5206 02:50:47,637 --> 02:50:48,571 BRAINSTORMING AND THOUGHT I'D 5207 02:50:48,638 --> 02:50:50,540 THROW THAT OUT THERE AS A 5208 02:50:50,607 --> 02:50:51,941 POSSIBILITY FOR USING NOVEL 5209 02:50:52,008 --> 02:50:56,813 METHODS. 5210 02:50:56,879 --> 02:50:57,814 >> OKAY. 5211 02:50:57,880 --> 02:50:59,415 IT'S NEARLY LUNCHTIME SO MAYBE I 5212 02:50:59,482 --> 02:51:02,619 CAN THROW NOT A ROCK BUT A 5213 02:51:02,685 --> 02:51:05,254 BOULDER INTO THE POND. 5214 02:51:05,321 --> 02:51:07,123 CAN I DO THAT? 5215 02:51:07,190 --> 02:51:07,657 OKAY. 5216 02:51:07,724 --> 02:51:10,627 I'M GOING TO -- LET'S DO A 5217 02:51:10,693 --> 02:51:12,729 LITTLE BIT OF PHILOSOPHY OF 5218 02:51:12,795 --> 02:51:13,162 SCIENCE. 5219 02:51:13,229 --> 02:51:15,031 YOU'LL SEE WHERE I'M GETTING AT. 5220 02:51:15,098 --> 02:51:18,835 SO I'M COMING FROM THEORETICAL 5221 02:51:18,901 --> 02:51:19,636 PHYSICS. 5222 02:51:19,702 --> 02:51:20,737 THEORETICAL PHYSICS IS MATH, AS 5223 02:51:20,803 --> 02:51:21,504 WE JUST HEARD. 5224 02:51:21,571 --> 02:51:23,706 MATH, THEY DO NOT LIE. 5225 02:51:23,773 --> 02:51:24,907 WHEN YOU WANT TO DEMONSTRATE 5226 02:51:24,974 --> 02:51:25,908 SOMETHING, IT'S EITHER TRUE OR 5227 02:51:25,975 --> 02:51:26,776 FALSE. 5228 02:51:26,843 --> 02:51:30,413 THERE'S NO OTHER ALTERNATIVE. 5229 02:51:30,480 --> 02:51:32,148 BECAUSE IT FOLLOWS A SET OF 5230 02:51:32,215 --> 02:51:34,217 RULES AND AXIOMA. 5231 02:51:34,283 --> 02:51:38,521 IN BIOLOGY, WE DO NONT DO NOT Y 5232 02:51:38,588 --> 02:51:39,722 THEORY TO GUIDE US SO WE DON'T 5233 02:51:39,789 --> 02:51:41,090 KNOW WHAT WE'RE TALKING ABOUT, 5234 02:51:41,157 --> 02:51:41,591 THAT'S THE FIRST THING. 5235 02:51:41,658 --> 02:51:42,959 BUT THAT'S THE REALITIMENT 5236 02:51:43,026 --> 02:51:45,428 RIGHT? 5237 02:51:45,495 --> 02:51:45,795 REALITY, 5238 02:51:45,862 --> 02:51:46,963 OTHERWISE WE WOULD HAVE SOLVED 5239 02:51:47,030 --> 02:51:47,897 THE PROBLEM OF LIFE A LONG TIME 5240 02:51:47,964 --> 02:51:48,531 AGO. 5241 02:51:48,598 --> 02:51:51,534 WE STILL DO NOT UNDERSTAND HOW A 5242 02:51:51,601 --> 02:51:55,705 SINGLE TROA CARE YOT IS WORKING. 5243 02:51:55,772 --> 02:51:57,006 WE HAVE NO CLUE. 5244 02:51:57,073 --> 02:51:59,542 NOW, WE KNOW THAT THERE IS A 5245 02:51:59,609 --> 02:52:02,311 PROBLEM WITH REPLICATION, NULL 5246 02:52:02,378 --> 02:52:03,279 STUDIES, AND THE PROBLEM WITH 5247 02:52:03,346 --> 02:52:04,480 THE P VALUE. 5248 02:52:04,547 --> 02:52:05,348 THAT'S FOR SURE. 5249 02:52:05,415 --> 02:52:07,417 THIS IS A PROBLEM. 5250 02:52:07,483 --> 02:52:08,818 BUT I'M GOING TO ARGUE THAT IT 5251 02:52:08,885 --> 02:52:12,455 IS ALSO AN ILL-POSED PROBLEM, 5252 02:52:12,522 --> 02:52:14,991 LIKE IN MATH, ILL-POSED. 5253 02:52:15,058 --> 02:52:16,092 ILL-POSED MEANS THERE IS NO 5254 02:52:16,159 --> 02:52:16,626 SOLUTION TO IT. 5255 02:52:16,693 --> 02:52:19,896 AND FOR A SCIENTIFIC REASON. 5256 02:52:19,962 --> 02:52:25,735 SO YOU MENTIONED, SHAI, -- WHAT 5257 02:52:25,802 --> 02:52:30,239 IS CALLED HOMEOSTASIS, MEANS 5258 02:52:30,306 --> 02:52:31,407 THAT EVERYTHING IS IN 5259 02:52:31,474 --> 02:52:32,542 RELATIONSHIP WITH EVERYTHING, 5260 02:52:32,608 --> 02:52:35,845 AND MAINTAINS A KIND OF 5261 02:52:35,912 --> 02:52:36,846 EQUILIBRIUM. 5262 02:52:36,913 --> 02:52:39,015 BUT YOU DIDN'T FIND THAT, BUT 5263 02:52:39,082 --> 02:52:43,086 THIS LED LATER TO THE CONCEPT OF 5264 02:52:43,152 --> 02:52:44,787 DEGENRE SEE, THE MULTIPLE 5265 02:52:44,854 --> 02:52:51,761 SOLUTION TO THE SAME PROBLEM. 5266 02:52:51,828 --> 02:52:53,963 PROOF, YOU CAN REMOVE 70% OF THE 5267 02:52:54,030 --> 02:52:55,598 JEANS OF E. COLI AND NOTHING 5268 02:52:55,665 --> 02:52:55,965 HAPPENS. 5269 02:52:56,032 --> 02:52:57,600 THERE ARE REDUB DAN SEES AND YOU 5270 02:52:57,667 --> 02:53:03,639 CAN FIND SOLUTIONS ONE BY ONE, 5271 02:53:03,706 --> 02:53:04,507 NOT ALL OF THEM AT THE SAME 5272 02:53:04,574 --> 02:53:06,342 TIME, OF COURSE, BUT ONE BY ONE. 5273 02:53:06,409 --> 02:53:08,478 YOU SLOW DOWN METABOLISM, IT'S A 5274 02:53:08,544 --> 02:53:10,646 FAMOUS PAPER, TWO PAPERS. 5275 02:53:10,713 --> 02:53:11,414 PAPER SHOWING 5276 02:53:11,481 --> 02:53:11,881 THAT. 5277 02:53:11,948 --> 02:53:13,082 SO WHAT DOES IT MEAN? 5278 02:53:13,149 --> 02:53:15,618 IT MEANS THAT WHATEVER PARAMETER 5279 02:53:15,685 --> 02:53:17,587 YOU MEASURE IN A BIOLOGICAL 5280 02:53:17,653 --> 02:53:21,524 SAMPLE, YOU'RE GOING TO FIND A 5281 02:53:21,591 --> 02:53:24,627 LARGE SPECTRUM OF VALUES IN MANY 5282 02:53:24,694 --> 02:53:26,362 PARAMETERS. 5283 02:53:26,429 --> 02:53:28,898 THIS HAS BEEN FANTASTICALLY 5284 02:53:28,965 --> 02:53:29,699 DEMONSTRATED. 5285 02:53:29,766 --> 02:53:31,300 SO WHATEVER YOU MEASURE, YOU 5286 02:53:31,367 --> 02:53:32,301 HAVE A HUGE DISTRIBUTION. 5287 02:53:32,368 --> 02:53:34,270 NOW IF YOU PICK UP FIVE OR 10 5288 02:53:34,337 --> 02:53:36,005 SAMPLES FROM THIS HUGE 5289 02:53:36,072 --> 02:53:38,107 DISTRIBUTION, YOU MAY HAVE 5290 02:53:38,174 --> 02:53:40,743 SOMETHING WITH AN AVERAGE VALUE 5291 02:53:40,810 --> 02:53:42,845 IN THE STANDARD -- OF DOMAIN. 5292 02:53:42,912 --> 02:53:45,014 YOU DO SOME CHANGE, YOU MAY FIND 5293 02:53:45,081 --> 02:53:47,116 SOMETHING DIFFERENT OR NOT. 5294 02:53:47,183 --> 02:53:48,985 NOW, WHY DO WE HAVE SO MUCH 5295 02:53:49,051 --> 02:53:51,287 VARIABILITY IN CLINICAL TRIALS? 5296 02:53:51,354 --> 02:53:53,356 BECAUSE EACH INDIVIDUAL IS 5297 02:53:53,422 --> 02:53:55,591 UNIQUE, AND DENYING THAT, IT'S 5298 02:53:55,658 --> 02:53:57,026 DENYING BIOLOGY. 5299 02:53:57,093 --> 02:54:00,296 SO A DRUG WILL WORK IN A CERTAIN 5300 02:54:00,363 --> 02:54:01,264 NUMBER OF INDIVIDUALS WHERE THE 5301 02:54:01,330 --> 02:54:02,298 SET OF PARAMETERS IS SUCH THAT 5302 02:54:02,365 --> 02:54:04,400 THE DRUG IS TARGETING THEM, AND 5303 02:54:04,467 --> 02:54:06,469 NOT OTHERS. 5304 02:54:06,536 --> 02:54:08,905 THE PROBLEM THAT WE HAVE IS 5305 02:54:08,971 --> 02:54:09,806 TRYING TO FIND SOMETHING THAT 5306 02:54:09,872 --> 02:54:13,309 WORKS FOR EVERYTHING. 5307 02:54:13,376 --> 02:54:14,977 BUT THAT IS DENYING BIOLOGY, 5308 02:54:15,044 --> 02:54:18,915 BECAUSE BIOLOGY IS VARIABILITY. 5309 02:54:18,981 --> 02:54:21,417 AND THAT ALLOWED US TO EVOLVE 5310 02:54:21,484 --> 02:54:22,185 AND TO SURVIVE. 5311 02:54:22,251 --> 02:54:24,120 SO THE FACT THAT WE FIND STUDIES 5312 02:54:24,187 --> 02:54:25,188 WITH P VALUES WHICH ARE 5313 02:54:25,254 --> 02:54:26,722 DIFFERENT, SOMETIMES 5314 02:54:26,789 --> 02:54:27,790 SIGNIFICANT, SOMETIMES NOT 5315 02:54:27,857 --> 02:54:30,459 SIGNIFICANT, MOST OF IT FOR ME 5316 02:54:30,526 --> 02:54:31,694 REFLECTS THE VARIABILITY OF 5317 02:54:31,761 --> 02:54:32,995 LIFE. 5318 02:54:33,062 --> 02:54:37,366 AND NOT A MISTAKE IN THE DESIGN 5319 02:54:37,433 --> 02:54:42,471 OF THE EXPERIMENT, OF THE ME 5320 02:54:42,538 --> 02:54:43,439 MEASUREMENTS WHICH ARE NOT IN 5321 02:54:43,506 --> 02:54:44,207 THE SAME WAY. 5322 02:54:44,273 --> 02:54:44,974 IT'S BOUND TO BE VARIABLE. 5323 02:54:45,041 --> 02:54:46,742 SO PART OF WHAT EXPLAINS THIS 5324 02:54:46,809 --> 02:54:48,044 VARIABILITY OR THIS CRISIS IS 5325 02:54:48,110 --> 02:54:48,711 JUST BIOLOGY. 5326 02:54:48,778 --> 02:54:50,880 I'M NOT SAYING THAT EVERYTHING 5327 02:54:50,947 --> 02:54:51,447 EXPLAINS IT. 5328 02:54:51,514 --> 02:54:53,783 I'M JUST GIVING YOU SOME KIND OF 5329 02:54:53,850 --> 02:54:55,284 BROADER PERSPECTIVE THAT 5330 02:54:55,351 --> 02:54:57,386 PROVIDES A DIFFERENT APPROACH OF 5331 02:54:57,453 --> 02:54:59,989 THIS PROBLEM. 5332 02:55:00,056 --> 02:55:02,892 YOU CAN READ MY EDITORIAL ON 5333 02:55:02,959 --> 02:55:03,993 DEGENERACY THAT THEY PUBLISHED 5334 02:55:04,060 --> 02:55:06,729 IN NOVEMBER. 5335 02:55:06,796 --> 02:55:08,064 THAT WAS A BRIEF SUMMARY. 5336 02:55:08,130 --> 02:55:10,433 IT'S MORE COMPLEX THAN SAYING 5337 02:55:10,499 --> 02:55:12,835 IT'S TRUE OR NOT TRUE OR THERE 5338 02:55:12,902 --> 02:55:14,570 ARE PROBLEMS AND NOT PROBLEMS. 5339 02:55:14,637 --> 02:55:16,839 WE ARE DEALING WITH BIOLOGY AND 5340 02:55:16,906 --> 02:55:19,041 NOT WITH MATH. 5341 02:55:19,108 --> 02:55:21,544 DO YOU KNOW WHEN P VALUES 5342 02:55:21,611 --> 02:55:22,745 WERE -- WHERE STATISTICS ARE 5343 02:55:22,812 --> 02:55:24,814 COMING FROM? 5344 02:55:24,881 --> 02:55:29,352 IT'S ALREADY FROM BABYLON TIMES 5345 02:55:29,418 --> 02:55:30,686 AND EGYPTIAN TIMES WHEN THEY ARE 5346 02:55:30,753 --> 02:55:32,188 TRYING TO FIGURE OUT, PREDICT 5347 02:55:32,255 --> 02:55:35,858 THE TRAJECTORIES IN THE SKY TO 5348 02:55:35,925 --> 02:55:37,827 PREDICT AUTUMN, FALL, WINTER 5349 02:55:37,894 --> 02:55:39,362 HARVEST, AND EVERYTHING. 5350 02:55:39,428 --> 02:55:41,764 THESE ARE BASED ON PHYSICS. 5351 02:55:41,831 --> 02:55:43,866 YOU DO STATISTICS BECAUSE YOU 5352 02:55:43,933 --> 02:55:45,701 HAVE VARIABILITY IN YOUR 5353 02:55:45,768 --> 02:55:48,104 MEASUREMENT, AND YOU DESCRIBE 5354 02:55:48,170 --> 02:55:49,772 THIS UNCERTAINTY WITH 5355 02:55:49,839 --> 02:55:50,439 STATISTICS. 5356 02:55:50,506 --> 02:55:52,208 BUT IN BIOLOGY, THERE WAS NO 5357 02:55:52,275 --> 02:55:52,842 CERTAINTY. 5358 02:55:52,909 --> 02:55:56,145 IT'S ONLY VARIABILITY. 5359 02:55:56,212 --> 02:56:02,518 SO DO STATI STATISTICS REALLY AO 5360 02:56:02,585 --> 02:56:02,752 BIOLOGY? 5361 02:56:02,818 --> 02:56:04,854 >> I WOULD SAY THE AMERICAN 5362 02:56:04,921 --> 02:56:05,421 STATISTICAL ASSOCIATION HAS 5363 02:56:05,488 --> 02:56:07,189 DEFINITELY PUT OUT COMMENTARIES 5364 02:56:07,256 --> 02:56:09,125 ABOUT QUESTIONING WHETHER OR NOT 5365 02:56:09,191 --> 02:56:10,559 NULL HYPOTHESIS SIGNIFICANCE 5366 02:56:10,626 --> 02:56:12,061 TESTING IS EVEN APPROPRIATE AND 5367 02:56:12,128 --> 02:56:13,429 WHETHER OR NOT WE SHOULD BE 5368 02:56:13,496 --> 02:56:15,598 MAKING DI DICHOTOMOUS DECISIONS, 5369 02:56:15,665 --> 02:56:19,268 FOR EXAMPLE, AT P.05 BECAUSE OF 5370 02:56:19,335 --> 02:56:19,869 BIOLOGICAL VARIABILITY AND CAN 5371 02:56:19,936 --> 02:56:22,238 YOU REALLY MAKE A YES/NO ANSWER 5372 02:56:22,305 --> 02:56:25,207 WHEN THERE'S, YOU KNOW, A HUGE 5373 02:56:25,274 --> 02:56:27,076 VARIABILITY OF OUTCOMES AND SO 5374 02:56:27,143 --> 02:56:29,812 THAT IS AN OPEN QUESTION. 5375 02:56:29,879 --> 02:56:30,446 WERE YOU NEXT? 5376 02:56:30,513 --> 02:56:32,481 >> YEAH. 5377 02:56:32,548 --> 02:56:36,118 SO FISHER WAS -- A LOT OF THE 5378 02:56:36,185 --> 02:56:37,853 MODERN STATISTICS CAME OUT OF 5379 02:56:37,920 --> 02:56:40,456 GENETICS LIKE FISHER, BUT I TO 5380 02:56:40,523 --> 02:56:42,959 AGREE WITH THIS POINT FROM A 5381 02:56:43,025 --> 02:56:45,361 PRACTICAL PERSPECTIVE, I'M 5382 02:56:45,428 --> 02:56:46,696 PAUL -- CAL TECH. 5383 02:56:46,762 --> 02:56:48,698 THERE'S PRACTICAL REASONS TO 5384 02:56:48,764 --> 02:56:52,969 REPORT EVEN STUDIES THAT ARE NOT 5385 02:56:53,035 --> 02:56:55,738 COMING TO APPROPRIATE SUPPORTIVE 5386 02:56:55,805 --> 02:56:57,373 HYPOTHESIS BECAUSE THE 5387 02:56:57,440 --> 02:56:58,741 TECHNOLOGY AND INTERPRETATION 5388 02:56:58,808 --> 02:56:58,975 CHANGES. 5389 02:56:59,041 --> 02:57:00,443 FOR EXAMPLE, IF YOU'RE USING 5390 02:57:00,509 --> 02:57:02,345 PRE-CLINICAL STUDY WITH 5391 02:57:02,411 --> 02:57:03,779 BIOMARKERS, THE MEANING OF THOSE 5392 02:57:03,846 --> 02:57:04,647 BIOMARKERS OFTEN CHANGES BECAUSE 5393 02:57:04,714 --> 02:57:05,748 WE THINK WE'RE MEASURING 5394 02:57:05,815 --> 02:57:07,116 SOMETHING BUT WE'RE NOT ACTUALLY 5395 02:57:07,183 --> 02:57:09,185 DOING THAT. 5396 02:57:09,251 --> 02:57:10,753 ANOTHER MORE POINTIAN EXAMPLE, 5397 02:57:10,820 --> 02:57:12,021 THERE'S THOUSANDS OR TENS OF 5398 02:57:12,088 --> 02:57:13,723 THOUSANDS OF STUDIES THAT USE 5399 02:57:13,789 --> 02:57:15,458 GENE ONTOLOGY TO INTERPRET THEM 5400 02:57:15,524 --> 02:57:20,162 AND AS A PI OF THE GENE ONTOLOGY 5401 02:57:20,229 --> 02:57:22,064 CONSORTIUM I CAN TELL YOU THE 5402 02:57:22,131 --> 02:57:23,165 ONTOLOGY CHANGES SO THE ACTUAL 5403 02:57:23,232 --> 02:57:24,934 WAY THAT WE'RE DESCRIBING THINGS 5404 02:57:25,001 --> 02:57:26,802 CHANGES IN TIME AND IF YOU 5405 02:57:26,869 --> 02:57:27,603 REANALYZE DATA FROM 10 YEARS AGO 5406 02:57:27,670 --> 02:57:29,005 THAT WASN'T SIGNIFICANT OR 5407 02:57:29,071 --> 02:57:31,540 MEANINGFUL, IT NOW OFTEN BECOMES 5408 02:57:31,607 --> 02:57:33,209 MEANINGFUL OR VICE VERSA SO YOU 5409 02:57:33,275 --> 02:57:34,643 WANT TO DO THAT AND REPORT THOSE 5410 02:57:34,710 --> 02:57:35,845 AND AGAIN TO THE POINT OF 5411 02:57:35,911 --> 02:57:37,380 STRATIFICATION AS ANOTHER 5412 02:57:37,446 --> 02:57:38,280 EXAMPLE, YOU DON'T KNOW WHAT THE 5413 02:57:38,347 --> 02:57:39,715 STRATIFICATION IS AT THIS POINT, 5414 02:57:39,782 --> 02:57:41,150 BUT YOU MIGHT LATER ON. 5415 02:57:41,217 --> 02:57:42,952 AND IN THE LONG TERM, THE 5416 02:57:43,019 --> 02:57:44,587 JOURNAL IMPACT FACTOR IS LESS 5417 02:57:44,653 --> 02:57:46,088 IMPORTANT THAN WORKING IT UP IN 5418 02:57:46,155 --> 02:57:47,757 A DATABASE EVEN IN PUBMED OR A 5419 02:57:47,823 --> 02:57:52,294 KNOWLEDGE BASE. 5420 02:57:52,361 --> 02:57:55,231 >> I HAVE A VERY SHORT POINT. 5421 02:57:55,297 --> 02:57:56,332 THERE WERE SOME REALLY 5422 02:57:56,399 --> 02:57:58,801 INTERESTING POINTS, I THINK 5423 02:57:58,868 --> 02:58:00,469 RACHEL MADE THIS PINTPOINT AND 5424 02:58:00,536 --> 02:58:01,537 SOME OTHERS ABOUT DOGMA AND 5425 02:58:01,604 --> 02:58:01,837 STATUS. 5426 02:58:01,904 --> 02:58:03,639 I DON'T KNOW IF FOLKS ARE 5427 02:58:03,706 --> 02:58:04,974 FAMILIAR, IF SAYING SIMILAR 5428 02:58:05,041 --> 02:58:07,376 HAPPENING IN BIOMEDICINE BUT IN 5429 02:58:07,443 --> 02:58:09,378 THE SOCIAL SCIENCES THERE'S THIS 5430 02:58:09,445 --> 02:58:10,246 EFFORT OF GATHERING PREDICTIONS 5431 02:58:10,312 --> 02:58:12,114 AND FORECASTS, I DON'T KNOW IF 5432 02:58:12,181 --> 02:58:13,549 THEY'RE SAYING SIMILAR IN 5433 02:58:13,616 --> 02:58:15,117 BIOMEDICINE, BUT IT'S BEEN 5434 02:58:15,184 --> 02:58:16,485 PROVED TO BE REALLY USEFUL TO 5435 02:58:16,552 --> 02:58:17,753 QUANTIFY PEOPLE'S EXPECTATIONS 5436 02:58:17,820 --> 02:58:18,687 IN THE FIELD ABOUT WHAT THE 5437 02:58:18,754 --> 02:58:20,056 EFFECT OF A CERTAIN TREATMENT 5438 02:58:20,122 --> 02:58:20,856 WILL BE AND THEN YOU COMPARE 5439 02:58:20,923 --> 02:58:21,557 THAT TO THE ESTIMATE. 5440 02:58:21,624 --> 02:58:23,259 SO WE'RE NOT ALWAYS COMPARING 5441 02:58:23,325 --> 02:58:26,062 AGAINST THE NULL ANYMORE, WE'RE 5442 02:58:26,128 --> 02:58:27,463 COMPARING TO THE PRIOR JUST TO 5443 02:58:27,530 --> 02:58:30,466 GET TO THE KIND OF BAYESIAN 5444 02:58:30,533 --> 02:58:32,001 POINT, AND IT GETS YOU TO THE 5445 02:58:32,068 --> 02:58:33,202 STRENGTH AND UNIFORMITY OF 5446 02:58:33,269 --> 02:58:34,737 PEOPLE'S DOGMA, LIKE WHAT WAS 5447 02:58:34,804 --> 02:58:36,439 THE EXPECTATION OF EFFECT OF A 5448 02:58:36,505 --> 02:58:36,806 CERTAIN TREAT M. 5449 02:58:36,872 --> 02:58:37,940 SO THERE'S SOMETHING CALLED 5450 02:58:38,007 --> 02:58:39,141 SOCIAL SCIENCE PREDICTION 5451 02:58:39,208 --> 02:58:41,377 PLATFORM, SSPP AND SOME OTHER 5452 02:58:41,444 --> 02:58:42,611 EFFORTS TO TRY TO CROWD SOURCE 5453 02:58:42,678 --> 02:58:44,213 OR GET EXPERT PREDICTIONS. 5454 02:58:44,280 --> 02:58:46,916 I DON'T KNOW AS MUCH ABOUT 5455 02:58:46,982 --> 02:58:48,684 BIOMEDICINE BUT AT LEAST OUR 5456 02:58:48,751 --> 02:58:49,852 FIELD, THAT'S NOW. 5457 02:58:49,919 --> 02:58:51,954 >> ONE ANSWER HERE, IT'S THE 5458 02:58:52,021 --> 02:58:53,622 STOCK MARKET. 5459 02:58:53,689 --> 02:58:57,460 >> THANKS, SHAI. 5460 02:58:57,526 --> 02:58:58,761 YEAH, SO THAT ACTUALLY BRINGS US 5461 02:58:58,828 --> 02:59:00,563 UP TO -- UNLESS YOU HAD ONE 5462 02:59:00,629 --> 02:59:02,098 QUICK COMMENT YOU'D LIKE TO 5463 02:59:02,164 --> 02:59:02,298 MAKE? 5464 02:59:02,364 --> 02:59:02,765 >> I HAVE A QUESTION. 5465 02:59:02,832 --> 02:59:03,899 >> GO FOR IT, GO FOR IT. 5466 02:59:03,966 --> 02:59:04,733 >> OH, YES. 5467 02:59:04,800 --> 02:59:06,402 I JUST KIND OF HAVE MORE OF LIKE 5468 02:59:06,469 --> 02:59:07,736 A METHODOLOGICAL QUESTION. 5469 02:59:07,803 --> 02:59:11,173 MY NAME IS CHRISTINE LU, POSTDOC 5470 02:59:11,240 --> 02:59:11,841 AT UCSF. 5471 02:59:11,907 --> 02:59:13,676 I WAS A BIT MORE CURIOUS ABOUT 5472 02:59:13,742 --> 02:59:15,311 HOW YOU ACTUALLY QUANTIFY THE 5473 02:59:15,377 --> 02:59:16,679 PUBLICATION BUY A ESPECIALLY 5474 02:59:16,745 --> 02:59:20,082 WITH THIS RELATIONSHIP TO RIGOR 5475 02:59:20,149 --> 02:59:21,917 AND DESIGNING A GOOD EXPERIMENT, 5476 02:59:21,984 --> 02:59:23,185 YOU WANT PERHAPS A POSITIVE 5477 02:59:23,252 --> 02:59:24,386 CONTROL IN WHICH YOU MIGHT 5478 02:59:24,453 --> 02:59:25,588 EXPECT SORT OF STATISTICAL 5479 02:59:25,654 --> 02:59:26,155 SIGNIFICANCE. 5480 02:59:26,222 --> 02:59:27,022 SO JUST TO UNDERSTAND A LITTLE 5481 02:59:27,089 --> 02:59:30,559 BIT MORE ABOUT WHETHER NULL 5482 02:59:30,626 --> 02:59:33,295 RESULTS OR NEGATIVE RESULTS OR 5483 02:59:33,362 --> 02:59:34,964 REJECTING THE NULL HYPOTHESIS, 5484 02:59:35,030 --> 02:59:36,499 LIKE HOW DOES THIS PLAY INTO 5485 02:59:36,565 --> 02:59:38,067 WHAT YOU GUYS HAVE PUBLISHED, 5486 02:59:38,134 --> 02:59:40,469 HOW YOU TAKE INTO ACCOUNT THIS 5487 02:59:40,536 --> 02:59:41,370 RELATIONSHIP BETWEEN MAYBE 5488 02:59:41,437 --> 02:59:43,005 PEOPLE DESIGNING EXPERIMENTS 5489 02:59:43,072 --> 02:59:45,541 THAT WILL HAVE THE LITTLE 5490 02:59:45,608 --> 02:59:47,209 ASTERISK BUT THAT THEN IS IN 5491 02:59:47,276 --> 02:59:49,011 FAVORITE OF NULL HYPOTHESIS, 5492 02:59:49,078 --> 02:59:50,646 WHERE DOES THIS ALL FIT INTO 5493 02:59:50,713 --> 02:59:51,714 THIS FRAMEWORK OF PUBLICATION 5494 02:59:51,780 --> 02:59:54,817 BIAS? 5495 02:59:54,884 --> 02:59:58,220 >> I MEAN, MAYBE PEOPLE HAVE 5496 02:59:58,287 --> 03:00:00,523 DIFFERENT VIEWS ON THIS, BUT MY 5497 03:00:00,589 --> 03:00:02,424 VIEW OF WHAT PUBLICATION BIAS IS 5498 03:00:02,491 --> 03:00:04,793 IS IT'S THE CAUSAL EFFECT OF THE 5499 03:00:04,860 --> 03:00:06,896 SIGNIFICANCE OF THE RESULT ON 5500 03:00:06,962 --> 03:00:07,930 PUBLICATION, OKAY? 5501 03:00:07,997 --> 03:00:09,698 AND MY VIEW IS THAT'S A VERY 5502 03:00:09,765 --> 03:00:12,234 HARD THING TO ACTUALLY QUANTIFY, 5503 03:00:12,301 --> 03:00:15,204 BECAUSE WE DON'T HAVE RANDOM 5504 03:00:15,271 --> 03:00:16,305 ASSIGNMENT OF THE SIGNIFICANCE 5505 03:00:16,372 --> 03:00:16,872 OF THE EFFECT. 5506 03:00:16,939 --> 03:00:18,107 SHAI PRESENTED A STUDY WHICH I 5507 03:00:18,174 --> 03:00:19,508 THINK WOULD BE THE CLOSEST STUDY 5508 03:00:19,575 --> 03:00:21,410 THAT YOU WOULD WANT ON THAT, 5509 03:00:21,477 --> 03:00:23,712 WHICH IS YOU HAVE THE SAME PAPER 5510 03:00:23,779 --> 03:00:26,448 AND YOU KIND OF LIKE FAKE OR ARE 5511 03:00:26,515 --> 03:00:28,050 A LITTLE DECEPTIVE ABOUT WHAT 5512 03:00:28,117 --> 03:00:29,318 THE RESULTS ARE, BUT YOU KEEP 5513 03:00:29,385 --> 03:00:30,519 EVERYTHING ELSE IN THE STUDY THE 5514 03:00:30,586 --> 03:00:31,053 SAME. 5515 03:00:31,120 --> 03:00:32,888 YOU SEND IT OUT TO THE COMMUNITY 5516 03:00:32,955 --> 03:00:34,323 AND SEE HOW THE REVIEWERS REACT, 5517 03:00:34,390 --> 03:00:36,692 AND THAT STUDY SHOWED ABOUT A 20 5518 03:00:36,759 --> 03:00:38,360 PERCENTAGE POINT EFFECT, 17 5519 03:00:38,427 --> 03:00:38,761 PERCENTAGE POINT. 5520 03:00:38,827 --> 03:00:41,330 SO THE QUESTION IS, WITHOUT A 5521 03:00:41,397 --> 03:00:42,965 POTENTIALLY DECEPTIVE STUDY OR 5522 03:00:43,032 --> 03:00:45,534 WHERE YOU HAVE THE CONFEDERATES 5523 03:00:45,601 --> 03:00:46,635 BEING THE JOURNAL EDITORS, HOW 5524 03:00:46,702 --> 03:00:48,504 DO YOU TAKE THE BROAD SAMPLE OF 5525 03:00:48,571 --> 03:00:50,873 STUDIES IN ORDER TO USE THAT 5526 03:00:50,940 --> 03:00:52,074 RESEARCH DESIGN, THE LOGIC OF 5527 03:00:52,141 --> 03:00:54,276 IT, TO GET WHAT THE CAUSAL 5528 03:00:54,343 --> 03:00:57,746 EFFECT OF THE SIZE OF THE EFFECT 5529 03:00:57,813 --> 03:00:58,881 WITH THE SIGNIFICANCE IS. 5530 03:00:58,948 --> 03:00:59,648 THAT'S KIND OF MY VIEW OF WHAT 5531 03:00:59,715 --> 03:00:59,949 IT IS. 5532 03:01:00,015 --> 03:01:03,319 >> I WOULD ALSO LIKE TO ADD TO 5533 03:01:03,385 --> 03:01:04,987 THAT ON THE DIFFICULTY OF 5534 03:01:05,054 --> 03:01:05,854 QUANTIFYING PUBLICATION BIAS. 5535 03:01:05,921 --> 03:01:09,024 IT CAN'T SIMPLY BE THROUGH THE 5536 03:01:09,091 --> 03:01:10,326 NUMBER OF NON-SIGNIFICANT 5537 03:01:10,392 --> 03:01:11,293 STUDIES THAT ARE PUBLISHED, 5538 03:01:11,360 --> 03:01:13,028 BECAUSE SIGNIFICANCE, YOU CAN 5539 03:01:13,095 --> 03:01:14,630 REACH STATISTICAL SIGNIFICANCE 5540 03:01:14,697 --> 03:01:16,832 BY HAVING A LARGE ENOUGH STUDY 5541 03:01:16,899 --> 03:01:20,002 OR BY THE TRUE EFFECTUALLY BEING 5542 03:01:20,069 --> 03:01:20,269 LARGENESS. 5543 03:01:20,336 --> 03:01:22,438 IF YOU HAVE A TRUE EFFECT SIZE 5544 03:01:22,504 --> 03:01:24,506 THAT IS LARGE, YOU KNOW, LESS 5545 03:01:24,573 --> 03:01:26,542 THAN 5% OF THE TIME YOU PROBABLY 5546 03:01:26,609 --> 03:01:28,944 WILL GET A NULL RESULT. 5547 03:01:29,011 --> 03:01:31,013 OR A NON-SIGNIFICANT RESULT BY 5548 03:01:31,080 --> 03:01:32,548 DEFINITION. 5549 03:01:32,615 --> 03:01:34,950 AND SO IT WOULD BE EXPECTED THAT 5550 03:01:35,017 --> 03:01:37,653 FOR THAT QUESTION IN THAT AREA 5551 03:01:37,720 --> 03:01:38,420 FOR A CERTAIN TYPE OF STUDY, 5552 03:01:38,487 --> 03:01:40,789 YOU'RE GOING TO HAVE VERY, VERY 5553 03:01:40,856 --> 03:01:42,691 FEW NON-SIGNIFICANT RESULTS OUT 5554 03:01:42,758 --> 03:01:43,225 THERE. 5555 03:01:43,292 --> 03:01:45,794 THAT'S NOT PUBLICATION BIAS, 5556 03:01:45,861 --> 03:01:48,197 THAT'S JUST THE TRUE EFFECTUALLY 5557 03:01:48,264 --> 03:01:48,931 BEING LARGE. 5558 03:01:48,998 --> 03:01:51,433 SO IT IS VERY DIFFICULT TO 5559 03:01:51,500 --> 03:01:52,901 QUANTIFY AS HE SAID, AND I THINK 5560 03:01:52,968 --> 03:01:57,606 A KEY POINT HE MADE ABOUT STUDY 5561 03:01:57,673 --> 03:02:00,009 QUALITY, PERHAPS I WOULDN'T SAY 5562 03:02:00,075 --> 03:02:01,377 STUDY QUALITY, MAYBE STUDY 5563 03:02:01,443 --> 03:02:03,879 DESIGN AND POPULATION. 5564 03:02:03,946 --> 03:02:05,848 I THINK YOU NEED TO REALLY TRY 5565 03:02:05,914 --> 03:02:08,217 TO COMPARE APPLES TO APPLES TO 5566 03:02:08,284 --> 03:02:10,486 GET AN IDEA OF PUBLICATION BIAS, 5567 03:02:10,552 --> 03:02:12,588 COMPARING STUDIES THAT ARE 5568 03:02:12,655 --> 03:02:14,723 FUNDAMENTALLY DIFFERENT IN THEIR 5569 03:02:14,790 --> 03:02:17,993 SAMPLE OR DESIGN, IT'S NOT A 5570 03:02:18,060 --> 03:02:21,297 TRUE ACCURATE COMPARISON TO SAY 5571 03:02:21,363 --> 03:02:23,532 THAT ONE IS PREFERENTIALLY 5572 03:02:23,599 --> 03:02:24,633 PUBLISHED OVER A 'NOTHER BASED 5573 03:02:24,700 --> 03:02:25,834 ON THE SIGNIFICANCE. 5574 03:02:25,901 --> 03:02:27,903 >> WELL, THANK YOU VERY MUCH TO 5575 03:02:27,970 --> 03:02:29,071 OUR PANELISTS. 5576 03:02:29,138 --> 03:02:33,876 THIS WAS A GREAT OPENING SESSION 5577 03:02:33,942 --> 03:02:36,945 FOR -- AND THIS BRINGS US TO 5578 03:02:37,012 --> 03:02:38,113 LUNCHTIME, AND SO THOSE OF YOU 5579 03:02:38,180 --> 03:02:39,882 WHO ORDERED BOX LUNCHES, THEY 5580 03:02:39,948 --> 03:02:41,350 WILL BE AVAILABLE DOWN THE HALL 5581 03:02:41,417 --> 03:02:43,652 BY THE KITCHENETTE WHERE THE 5582 03:02:43,719 --> 03:02:45,354 COFFEE WAS. 5583 03:02:45,421 --> 03:02:46,989 AND WE WILL MEET BACK HERE AND 5584 03:02:47,056 --> 03:02:47,823 START AGAIN AT 1:00. 5585 03:02:47,890 --> 03:02:50,192 THANK YOU. 5586 03:02:50,259 --> 03:02:53,162 >> SESSION 2 IS ABOUT REPORTING 5587 03:02:53,228 --> 03:02:54,663 NULL STUDIES AND BARRIERS WE MAY 5588 03:02:54,730 --> 03:02:58,867 FACE, NOT ONLY IN THE PUBLISHING 5589 03:02:58,934 --> 03:03:04,873 WORLD BUT ALSO AS SCIENTISTS. 5590 03:03:04,940 --> 03:03:05,607 I'M CRISTTOPHE BERNARD, 5591 03:03:05,674 --> 03:03:07,309 NEUROSCIENTIST WORKING IN THE 5592 03:03:07,376 --> 03:03:11,880 FRENCH NIH, ALSO THE FUNDER AND 5593 03:03:11,947 --> 03:03:13,749 EDITOR IN CHIEF OF NEURO. 5594 03:03:13,816 --> 03:03:18,354 WE'LL GIVE YOU A FEW WORDS ABOUT 5595 03:03:18,420 --> 03:03:20,522 IT, I INTRODUCED FROM THE 5596 03:03:20,589 --> 03:03:23,726 BEGINNING THE POSSIBILITY TO 5597 03:03:23,792 --> 03:03:24,960 PUBLISH NEGATIVE RESULTS, 5598 03:03:25,027 --> 03:03:27,296 BECAUSE THINKING IT WAS VERY 5599 03:03:27,363 --> 03:03:28,630 IMPORTANT FOR THE COMMUNITY TO 5600 03:03:28,697 --> 03:03:30,499 HAVE ACCESS TO THAT KIND OF 5601 03:03:30,566 --> 03:03:33,202 KNOWLEDGE IN ORDER NOT TO 5602 03:03:33,268 --> 03:03:37,606 REPRODUCE THE SAME EXPERIMENTS 5603 03:03:37,673 --> 03:03:39,508 LEADING TO NEGATIVE RESULTS. 5604 03:03:39,575 --> 03:03:42,311 IT'S BEEN TOUGH AT THE BEGINNING 5605 03:03:42,378 --> 03:03:44,146 TO HAVE PEOPLE SUBMIT TO THE 5606 03:03:44,213 --> 03:03:46,448 JOURNAL FOR THAT BECAUSE THEY 5607 03:03:46,515 --> 03:03:50,519 ARE STILL UNAWARE OF THIS 5608 03:03:50,586 --> 03:03:50,819 POSSIBILITY. 5609 03:03:50,886 --> 03:03:54,423 I THINK IT'S A KIND OF CRIME 5610 03:03:54,490 --> 03:03:55,524 AGAINST THE POSTDOCS AND Ph.D. 5611 03:03:55,591 --> 03:03:58,694 STUDENTS WHO DID THE WORK, AND 5612 03:03:58,761 --> 03:04:00,763 ALL OF YOU HAD WITH TWO OR THREE 5613 03:04:00,829 --> 03:04:01,764 YEARS OF WORK DON'T GET 5614 03:04:01,830 --> 03:04:04,733 PUBLISHED, THE WORK IS NOT 5615 03:04:04,800 --> 03:04:05,934 ACKNOWLEDGED BECAUSE NOBODY 5616 03:04:06,001 --> 03:04:10,639 WANTS TO PUBLIC THESE THINGS. 5617 03:04:10,706 --> 03:04:12,508 SO, HAVING THE POSSIBILITY TO 5618 03:04:12,574 --> 03:04:13,809 PUBLISH NEGATIVE RESULTS IS VERY 5619 03:04:13,876 --> 03:04:16,779 IMPORTANT, YOU'LL HEAR ALSO 5620 03:04:16,845 --> 03:04:18,280 EXAMPLES FROM EXPERIMENTAL 5621 03:04:18,347 --> 03:04:22,117 NEUROLOGY AND PLOS ONE, WE'LL 5622 03:04:22,184 --> 03:04:22,951 INTRODUCE THESE POSSIBILITIES, 5623 03:04:23,018 --> 03:04:25,087 THERE'S I THINK A BARRIER AND WE 5624 03:04:25,154 --> 03:04:26,955 CAN DISCUSS THAT AMONG THE 5625 03:04:27,022 --> 03:04:29,691 SCIENTISTS NOT TO PUBLISH NULL 5626 03:04:29,758 --> 03:04:30,793 RESULTS, THERE MAY BE REASONS 5627 03:04:30,859 --> 03:04:31,527 FOR THAT. 5628 03:04:31,593 --> 03:04:33,962 AND WE'D LIKE TO KNOW HOW TO 5629 03:04:34,029 --> 03:04:38,434 INCREASE THE NUMBER OF 5630 03:04:38,500 --> 03:04:38,734 SUBMISSIONS. 5631 03:04:38,801 --> 03:04:42,805 THE OTHER THING THAT IS ALSO 5632 03:04:42,871 --> 03:04:44,673 IMPORTANT FOR NEGATIVE NULL 5633 03:04:44,740 --> 03:04:49,278 STUDIES OR NEGATIVE STUDIES IS 5634 03:04:49,344 --> 03:04:51,013 THAT IT'S REALLY INSTRUMENTAL I 5635 03:04:51,079 --> 03:04:53,315 THINK FOR PEOPLE TO KNOW ABOUT 5636 03:04:53,382 --> 03:04:53,649 THAT. 5637 03:04:53,715 --> 03:04:56,251 AND WE HEARD THIS MORNING THAT 5638 03:04:56,318 --> 03:04:58,921 WHO IS GOING TO ACKNOWLEDGE 5639 03:04:58,987 --> 03:04:59,888 THAT, RIGHT? 5640 03:04:59,955 --> 03:05:02,391 SO HOW DO YOU VALUE THIS WORK 5641 03:05:02,458 --> 03:05:06,562 THAT HAS BEEN DONE AND 5642 03:05:06,628 --> 03:05:08,764 SUPPOSEDLY DOESN'T MAKE SCIENCE 5643 03:05:08,831 --> 03:05:11,633 PROGRESS BUT SCIENCE ALSO 5644 03:05:11,700 --> 03:05:13,035 PROGRESSES WITH ERRORS, DOESN'T 5645 03:05:13,101 --> 03:05:14,002 ONLY PROGRESS WITH POSITIVE 5646 03:05:14,069 --> 03:05:14,236 RESULTS. 5647 03:05:14,303 --> 03:05:16,738 IF YOU LOOK AT THE HISTORY OF 5648 03:05:16,805 --> 03:05:18,073 SCIENCE OUTSIDE BIOLOGY IT HAS 5649 03:05:18,140 --> 03:05:20,476 PROGRESSED WITH ERRORS, NOT ONLY 5650 03:05:20,542 --> 03:05:22,544 WITH MAJOR DISCOVERIES. 5651 03:05:22,611 --> 03:05:22,778 OKAY. 5652 03:05:22,845 --> 03:05:24,847 THERE IS ANOTHER THING THAT 5653 03:05:24,913 --> 03:05:27,683 PREVENTS THE PUBLICATION OF NULL 5654 03:05:27,749 --> 03:05:29,651 STUDIES, TAKES SOME GUTS FOR A 5655 03:05:29,718 --> 03:05:33,922 JOURNAL TO PUBLISH BECAUSE THESE 5656 03:05:33,989 --> 03:05:37,092 STUDIES ARE USUALLY NOT CITED. 5657 03:05:37,159 --> 03:05:39,595 IF YOU'RE NOT CITED, YOUR IMPACT 5658 03:05:39,661 --> 03:05:42,030 FACTOR IS GOING DOWN, MOST 5659 03:05:42,097 --> 03:05:44,533 PEOPLE PLAY THE GAME, YOU WON'T 5660 03:05:44,600 --> 03:05:48,570 GET PAPERS SUBMITTED TO YOUR 5661 03:05:48,637 --> 03:05:49,137 JOURNAL. 5662 03:05:49,204 --> 03:05:54,576 SO, YES, I INSTRUCT THE 5663 03:05:54,643 --> 03:05:55,377 REVIEWERS AT NEURO, THAT WE'RE 5664 03:05:55,444 --> 03:05:58,080 HERE TO SERVE THE COMMUNITY. 5665 03:05:58,146 --> 03:06:00,382 IF A NULL RESULT IS IMPORTANT 5666 03:06:00,449 --> 03:06:01,917 FOR THE COMMUNITY IT MUST BE 5667 03:06:01,984 --> 03:06:03,552 PUBLISHED, NO MATTER WHAT. 5668 03:06:03,619 --> 03:06:04,319 OUR IMPACT FACTOR WILL DECREASE, 5669 03:06:04,386 --> 03:06:05,320 I DON'T CARE. 5670 03:06:05,387 --> 03:06:06,655 I DON'T BELIEVE IN IMPACT 5671 03:06:06,722 --> 03:06:07,189 FACTOR. 5672 03:06:07,256 --> 03:06:08,390 I DON'T MENTION IT ON THE 5673 03:06:08,457 --> 03:06:08,924 WEBSITE. 5674 03:06:08,991 --> 03:06:10,359 YOU CAN FIND IT IF YOU WANT, 5675 03:06:10,425 --> 03:06:11,593 IT'S THERE, BUT IT'S NOT 5676 03:06:11,660 --> 03:06:13,428 ADVERTISED BECAUSE IT'S A WRONG 5677 03:06:13,495 --> 03:06:14,296 METRIC ANYWAY. 5678 03:06:14,363 --> 03:06:16,832 AND I WOULD LIKE TO HEAR ABOUT 5679 03:06:16,899 --> 03:06:18,700 EXPERIMENTAL NEUROLOGY IN PLOS 5680 03:06:18,767 --> 03:06:20,335 ONE BECAUSE THEY DO THE SAME 5681 03:06:20,402 --> 03:06:20,569 THING. 5682 03:06:20,636 --> 03:06:22,304 WE ACCEPT IT. 5683 03:06:22,371 --> 03:06:26,241 YOU HAVE TO REALIZE THAT FOR A 5684 03:06:26,308 --> 03:06:28,210 JOURNAL, THIS IS BAD THING. 5685 03:06:28,277 --> 03:06:30,846 BUT IF YOUR GOAL IS TO SERVE THE 5686 03:06:30,913 --> 03:06:34,016 COMMUNITY, THEN YOU'RE DOING IT. 5687 03:06:34,082 --> 03:06:34,683 OKAY. 5688 03:06:34,750 --> 03:06:39,054 SO, THIS IS WHAT WE'RE DOING, IN 5689 03:06:39,121 --> 03:06:41,690 EUROPE WE HAVE ALSO REGISTRY 5690 03:06:41,757 --> 03:06:43,025 REPORTS, CONFIRMATION STUDIES, 5691 03:06:43,091 --> 03:06:45,761 WE ALSO PUBLISH CONFIRMATION 5692 03:06:45,827 --> 03:06:47,629 STUDIES, ALL THESE THINGS, THEY 5693 03:06:47,696 --> 03:06:51,333 ARE NOT CITED, RIGHT? 5694 03:06:51,400 --> 03:06:53,835 ANYWAY, THE ORDER FIRST WE'LL 5695 03:06:53,902 --> 03:06:55,671 HEAR FROM CHRIS, WHO INTRODUCED 5696 03:06:55,737 --> 03:06:58,674 THE REGISTRY REPORT BUT IS GOING 5697 03:06:58,740 --> 03:07:01,243 TO SHOW US SOME INITIATIVES, AND 5698 03:07:01,310 --> 03:07:03,478 NOVEL TOOLS THAT WOULD BE 5699 03:07:03,545 --> 03:07:05,213 HELPFUL FOR THE COMMUNITY. 5700 03:07:05,280 --> 03:07:08,050 THEN EMILY IS EDITOR IN CHIEF OF 5701 03:07:08,116 --> 03:07:10,052 PLOS ONE, WILL COVER ALSO WHAT 5702 03:07:10,118 --> 03:07:13,322 THEY DO AT THE JOURNAL. 5703 03:07:13,388 --> 03:07:15,290 WE DISCUSSED ALREADY AND 5704 03:07:15,357 --> 03:07:19,428 BASICALLY SHARE THE SAME 5705 03:07:19,494 --> 03:07:19,761 PHILOSOPHY. 5706 03:07:19,828 --> 03:07:23,265 KIF WILL PRESENT A CASE STUDY 5707 03:07:23,332 --> 03:07:25,133 BASED ON HIS OWN EXPERIENCE WITH 5708 03:07:25,200 --> 03:07:25,534 NEGATIVE RESULTS. 5709 03:07:25,601 --> 03:07:30,405 AND I THINK THAT YOU WILL SEE TO 5710 03:07:30,472 --> 03:07:32,708 GET THE POSITION, PUBLISHING A 5711 03:07:32,774 --> 03:07:35,877 NEGATIVE STUDY. 5712 03:07:35,944 --> 03:07:38,213 SO IT'S LIKE HELPFUL 5713 03:07:38,280 --> 03:07:38,513 REALIZATION. 5714 03:07:38,580 --> 03:07:41,950 AND THEN JOSE WHO IS GOING TO 5715 03:07:42,017 --> 03:07:42,918 PRESENT EXPERIMENTAL NEUROLOGY 5716 03:07:42,985 --> 03:07:44,252 WHERE THEY DO THAT KIND OF THING 5717 03:07:44,319 --> 03:07:46,955 IS GOING TO TALK MORE ABOUT THE 5718 03:07:47,022 --> 03:07:57,232 SAME TOPIC. 5719 03:07:58,467 --> 03:08:01,870 AND FINALLY LAURA FROM TEXAS 5720 03:08:01,937 --> 03:08:05,040 AMU, WHICH IS MEXICO ANYWAY, 5721 03:08:05,107 --> 03:08:08,944 WILL TALK ALSO ABOUT REAL LIFE 5722 03:08:09,011 --> 03:08:10,746 EXPERIENCE AND PUBLICATION BIAS 5723 03:08:10,812 --> 03:08:13,482 AND THE DIFFICULTY OF BEING A 5724 03:08:13,548 --> 03:08:24,159 NEUROSCIENTIST IN THE PRESENT W. 5725 03:08:26,294 --> 03:08:27,062 FIRST, CHRIS. 5726 03:08:27,129 --> 03:08:27,696 >> THANK YOU. 5727 03:08:27,763 --> 03:08:29,264 THIS IS THE ONLY SLIDE I NEED, 5728 03:08:29,331 --> 03:08:32,768 IF YOU WANT TO FIX PUBLICATION 5729 03:08:32,834 --> 03:08:35,370 BIAS USE THESE. 5730 03:08:35,437 --> 03:08:36,705 WE CAN TALK AROUND THE ISSUE. 5731 03:08:36,772 --> 03:08:38,674 IT WAS MENTIONED IN THE PREVIOUS 5732 03:08:38,740 --> 03:08:40,075 TALK WE HAVE MEETINGS EVERY FIVE 5733 03:08:40,142 --> 03:08:40,976 YEARS AND NOT MUCH CHANGES. 5734 03:08:41,043 --> 03:08:42,844 IF YOU WANT TO FIX PUBLICATION 5735 03:08:42,911 --> 03:08:45,113 BIAS, IF YOU WANT TO SEE MORE 5736 03:08:45,180 --> 03:08:47,416 NULL RESULTS IN LITERATURE, STOP 5737 03:08:47,482 --> 03:08:48,850 DECIDING WHAT GETS INTO 5738 03:08:48,917 --> 03:08:49,985 LITERATURE BASED ON RESULTS. 5739 03:08:50,052 --> 03:08:51,586 THE ONLY WAY TO DO THAT AT THE 5740 03:08:51,653 --> 03:08:53,121 MOMENT IS TO USE REGISTERED 5741 03:08:53,188 --> 03:08:53,588 REPORTS. 5742 03:08:53,655 --> 03:08:55,957 I DON'T NEED TO INTRODUCE THIS 5743 03:08:56,024 --> 03:08:58,827 OBVIOUSLY TO THIS AUDIENCE, BUT 5744 03:08:58,894 --> 03:09:01,863 THE TWO KEY INGREDIENTS OF 5745 03:09:01,930 --> 03:09:03,065 REGISTERED REPORT ARE PRE-STUDY 5746 03:09:03,131 --> 03:09:04,733 EVALUATION BEFORE RESULTS ARE 5747 03:09:04,800 --> 03:09:07,069 KNOWN AND THEN IN PRINCIPLE 5748 03:09:07,135 --> 03:09:07,936 ACCEPTANCE REGARDLESS OF 5749 03:09:08,003 --> 03:09:09,037 OUTCOME. 5750 03:09:09,104 --> 03:09:12,140 IF YOU DO THIS, AS WE'VE DONE, 5751 03:09:12,207 --> 03:09:13,575 WE CREATED REGISTERED REPORTS 5752 03:09:13,642 --> 03:09:20,482 ABOUT TEN YEARS AGO, AND WE'VE 5753 03:09:20,549 --> 03:09:22,017 SEEN IT INCREASE TO 350 5754 03:09:22,084 --> 03:09:25,053 JOURNALS, WE'VE SEEN GOOD 5755 03:09:25,120 --> 03:09:26,388 IMPACT, REGISTERED REPORTS ARE 5756 03:09:26,455 --> 03:09:27,055 POSSIBLE WITH CAREER 5757 03:09:27,122 --> 03:09:28,457 RESEARCHERS, IMPORTANT FOR OPEN 5758 03:09:28,523 --> 03:09:29,658 SCIENCE INITIATIVE TO BE POPULAR 5759 03:09:29,725 --> 03:09:31,093 WITH THE RESEARCHERS WHO ARE 5760 03:09:31,159 --> 03:09:32,627 DOING THE RESEARCH MOST OF THE 5761 03:09:32,694 --> 03:09:35,330 TIME, WHICH IS GOOD NEWS, WE'RE 5762 03:09:35,397 --> 03:09:36,765 FINDING THAT REGISTERED REPORTS 5763 03:09:36,832 --> 03:09:39,501 ARE MUCH MORE LIKELY TO 5764 03:09:39,568 --> 03:09:42,003 DISCONFIRM PREDICTIONS, YOU'VE 5765 03:09:42,070 --> 03:09:44,239 GOT A 5 TO 10 INCREASE CHANCE OF 5766 03:09:44,306 --> 03:09:45,507 DISCONFIRMING YOUR HYPOTHESIS IF 5767 03:09:45,574 --> 03:09:47,042 YOU DO A REGISTERED REPORT 5768 03:09:47,109 --> 03:09:48,243 COMPARED TO A REGULAR ARTICLE. 5769 03:09:48,310 --> 03:09:51,313 WE'RE FINDING THEY ARE MORE 5770 03:09:51,379 --> 03:09:52,514 COMPUTATIONALLY REPRODUCIBLE 5771 03:09:52,581 --> 03:09:53,849 THAN REGULAR ARTICLES, THEY ARE 5772 03:09:53,915 --> 03:09:55,150 RATED HIGHER IN QUALITY BY 5773 03:09:55,217 --> 03:09:58,620 EXPERTS IN THE FIELD COMPARED TO 5774 03:09:58,687 --> 03:09:59,187 REGULAR ARTICLES. 5775 03:09:59,254 --> 03:10:02,791 AND ALSO TO TOUCH ON YOUR POINT 5776 03:10:02,858 --> 03:10:04,559 ABOUT CITATION, FOR REGULAR 5777 03:10:04,626 --> 03:10:05,293 RESEARCH ARTICLES STUDIES 5778 03:10:05,360 --> 03:10:06,962 REPORTING NEGATIVE RESULTS ARE 5779 03:10:07,028 --> 03:10:08,930 CITED LESS THAN THOSE REPORTING 5780 03:10:08,997 --> 03:10:09,798 POSITIVE RESULTS. 5781 03:10:09,865 --> 03:10:11,733 WE FIND THAT TRENDS DISAPPEARS 5782 03:10:11,800 --> 03:10:13,368 INTERESTINGLY FOR REGISTERED 5783 03:10:13,435 --> 03:10:13,602 REPORTS. 5784 03:10:13,668 --> 03:10:15,403 THEY ARE CITED ON AVERAGE THE 5785 03:10:15,470 --> 03:10:17,472 SAME AS REGULAR ARTICLES IN THE 5786 03:10:17,539 --> 03:10:17,773 LITERATURE. 5787 03:10:17,839 --> 03:10:19,341 SO THAT ALL LOOKS GREAT. 5788 03:10:19,407 --> 03:10:20,942 IT'S GREAT, GOOD NEWS, 5789 03:10:21,009 --> 03:10:22,377 REGISTERED REPORTS SEEM TO BE 5790 03:10:22,444 --> 03:10:23,245 WORKING AS INTENDED. 5791 03:10:23,311 --> 03:10:25,347 WHAT I WANT TO TALK ABOUT IS THE 5792 03:10:25,413 --> 03:10:26,748 FUTURE, JUST FOR A COUPLE 5793 03:10:26,815 --> 03:10:30,719 MINUTES I WANT TO FLOAT TWO VERY 5794 03:10:30,786 --> 03:10:31,586 IMPORTANT CURRENT AND EMERGING 5795 03:10:31,653 --> 03:10:32,354 FUTURE STEPS. 5796 03:10:32,420 --> 03:10:38,693 THE FIRST OF THESE IS COMMUNITY 5797 03:10:38,760 --> 03:10:39,895 GOVERNED PRE-PRINT REVIEW WHICH 5798 03:10:39,961 --> 03:10:44,065 I THINK WILL BE THE FUTURE 5799 03:10:44,132 --> 03:10:45,634 BECAUSE OF EFFICIENCY SAVINGS. 5800 03:10:45,700 --> 03:10:46,968 WE'RE REGISTERED PEER COMMUNITY 5801 03:10:47,035 --> 03:10:48,003 AND REGISTERED REPORTS A FEW 5802 03:10:48,069 --> 03:10:51,506 YEARS AGO IN WHICH IT'S A FREE 5803 03:10:51,573 --> 03:10:52,474 NON-COMMERCIAL PLATFORM THAT 5804 03:10:52,541 --> 03:10:54,209 MANAGES THE PEER REVIEW OF STAGE 5805 03:10:54,276 --> 03:10:59,147 1 AND STAGE 2 REGISTERED REPORT 5806 03:10:59,214 --> 03:10:59,447 PRE-PRINTS. 5807 03:10:59,514 --> 03:11:02,450 BASED UPON EVALUATIONS OF THE 5808 03:11:02,517 --> 03:11:03,318 PRE-PRINTS WHICH ARE MADE 5809 03:11:03,385 --> 03:11:06,154 PUBLIC, WE HAVE A FLEET OF PCI 5810 03:11:06,221 --> 03:11:07,923 FRIENDLY JOURNALS WHICH COMMIT 5811 03:11:07,989 --> 03:11:09,257 TO ACCEPTING THOSE 5812 03:11:09,324 --> 03:11:10,325 RECOMMENDATIONS WITHOUT FURTHER 5813 03:11:10,392 --> 03:11:12,961 PEER REVIEW, WHICH MAKES -- IT'S 5814 03:11:13,028 --> 03:11:14,563 A CLEARINGHOUSE FOR CENTRALIZING 5815 03:11:14,629 --> 03:11:18,667 THE PEER REVIEW OF REGISTERED 5816 03:11:18,733 --> 03:11:22,137 REPORTS, CONVERGES REGISTERED 5817 03:11:22,204 --> 03:11:24,339 REPORTS EXPERTISE IN ONE PLACE, 5818 03:11:24,406 --> 03:11:26,174 SUBMISSIONS ACROSS THE WHOLE OF 5819 03:11:26,241 --> 03:11:26,942 SCIENCES, NOT JUST NEUROSCIENCE 5820 03:11:27,008 --> 03:11:29,377 BUT IN THE HUMANITIES AND SOCIAL 5821 03:11:29,444 --> 03:11:31,313 SCIENCES AND PHYSICAL SCIENCES. 5822 03:11:31,379 --> 03:11:32,981 AND IT PLACES CONTROL OF PEER 5823 03:11:33,048 --> 03:11:33,949 REVIEW AND DECISION MAKING IN 5824 03:11:34,015 --> 03:11:36,017 THE HANDS OF THE SCIENTIFIC 5825 03:11:36,084 --> 03:11:36,484 COMMUNITY RATHER THAN 5826 03:11:36,551 --> 03:11:37,986 PUBLISHERS, WHICH I THINK IS A 5827 03:11:38,053 --> 03:11:40,755 VERY IMPORTANT STEP. 5828 03:11:40,822 --> 03:11:43,692 IT ALSO TO LEAD TO MY NEXT POINT 5829 03:11:43,758 --> 03:11:44,526 MAKES IT EASIER FOR REGISTERED 5830 03:11:44,593 --> 03:11:46,962 REPORTS TO WORK IN TANDEM WITH 5831 03:11:47,028 --> 03:11:47,963 PUBLIC FUNDERS. 5832 03:11:48,029 --> 03:11:50,165 PUBLIC FUNDERS OFTEN HAVE LEGAL 5833 03:11:50,232 --> 03:11:52,100 RESTRICTIONS ON WORKING TOO 5834 03:11:52,167 --> 03:11:53,568 CLOSELY WITH PARTICULARLY 5835 03:11:53,635 --> 03:11:54,970 PROFIT-MAKING PUBLISHERS. 5836 03:11:55,036 --> 03:11:56,271 ON REGISTERED REPORTS 5837 03:11:56,338 --> 03:11:57,239 PARTNERSHIPS. 5838 03:11:57,305 --> 03:11:58,673 THIS IS ANOTHER VERY CRUCIAL 5839 03:11:58,740 --> 03:11:59,341 FUTURE STEP. 5840 03:11:59,407 --> 03:12:01,309 SO ONE OF THE CRITICISMS OF THE 5841 03:12:01,376 --> 03:12:02,944 REGISTERED REPORTS INITIATIVE IT 5842 03:12:03,011 --> 03:12:04,279 DOUBLES UP ON PRE-STUDY 5843 03:12:04,346 --> 03:12:04,779 EVALUATION. 5844 03:12:04,846 --> 03:12:06,681 WE ALREADY HAVE A LOT OF PHASES 5845 03:12:06,748 --> 03:12:08,316 OF PRE-STUDY REVIEW THAT WE GO 5846 03:12:08,383 --> 03:12:13,788 TO WHEN WE'RE APPLYING FOR A 5847 03:12:13,855 --> 03:12:16,424 GRANT GETTING ETHICS OR 5848 03:12:16,491 --> 03:12:17,392 REGULATORY APPROVAL, WE 5849 03:12:17,459 --> 03:12:19,060 INTEGRATE STAGE 1 PEER REVIEW 5850 03:12:19,127 --> 03:12:20,161 WITH GRANT REVIEW AND PERFORM AT 5851 03:12:20,228 --> 03:12:23,298 THE SAME TIME, A PROVEN MODEL 5852 03:12:23,365 --> 03:12:26,167 THAT'S WORKED BETWEEN PLOS ONE 5853 03:12:26,234 --> 03:12:28,136 AND THE FOUNDATION THAT EMILY 5854 03:12:28,203 --> 03:12:30,305 WILL TALK ABOUT AN EXAMPLE WHEN 5855 03:12:30,372 --> 03:12:31,973 SHE HAS HER WORDS SHORTLY. 5856 03:12:32,040 --> 03:12:33,141 I WANT TO HIGHLIGHT VERY 5857 03:12:33,208 --> 03:12:34,910 IMPORTANT FUTURE STEP. 5858 03:12:34,976 --> 03:12:37,445 WE NEED FUNDERS LIKE THE NIH TO 5859 03:12:37,512 --> 03:12:38,413 BE OFFERING REGISTERED REPORTS 5860 03:12:38,480 --> 03:12:40,482 GRANTS AS A STANDARD TRACK THAT 5861 03:12:40,548 --> 03:12:49,157 RESEARCHERS CAN USE AND WE'LL 5862 03:12:49,224 --> 03:12:57,132 SEE MASSIVE UPTAKE. 5863 03:12:57,198 --> 03:12:58,600 THANK YOU. 5864 03:12:58,667 --> 03:13:01,870 >> I DON'T HAVE SLIDES AND I'VE 5865 03:13:01,937 --> 03:13:04,372 HAD ENOUGH CAFFEINE TO SPEAK AS 5866 03:13:04,439 --> 03:13:05,473 TWICE SPY. 5867 03:13:05,540 --> 03:13:07,676 I'M THE EDITOR IN CHIEF OF THE 5868 03:13:07,742 --> 03:13:10,045 JOURNAL PLOS ONE, HEAD OF 5869 03:13:10,111 --> 03:13:12,280 EDITORIAL BOARD SERVICES AT 5870 03:13:12,347 --> 03:13:13,615 PLOS, I HAVE ULTIMATE 5871 03:13:13,682 --> 03:13:15,216 RESPONSIBILITY FOR OUR BOARD OF 5872 03:13:15,283 --> 03:13:17,285 ABOUT 10,000 ACADEMIC EDITORS 5873 03:13:17,352 --> 03:13:19,354 AND MAKING SURE WE'RE MAKING 5874 03:13:19,421 --> 03:13:20,689 CONSISTENT DECISIONS ACROSS THE 5875 03:13:20,755 --> 03:13:22,891 PORTFOLIO AND EVERYONE IS 5876 03:13:22,958 --> 03:13:23,658 ADHERING TO PUBLICATION 5877 03:13:23,725 --> 03:13:23,925 CRITERIA. 5878 03:13:23,992 --> 03:13:25,827 TO PICK UP ON THE POINT CHRIS 5879 03:13:25,894 --> 03:13:26,828 MADE ABOUT REGISTERED REPORTS 5880 03:13:26,895 --> 03:13:28,897 ONE OF THE THINGS THAT WE'RE 5881 03:13:28,964 --> 03:13:33,501 TRYING AT PLOS IS WAYS TO MAKE 5882 03:13:33,568 --> 03:13:36,438 THE PUBLICATION PROCESS EASIER 5883 03:13:36,504 --> 03:13:40,175 FOR AUTHORS AND MORE 5884 03:13:40,241 --> 03:13:40,475 TRANSPARENT. 5885 03:13:40,542 --> 03:13:42,577 WE WORK IN CONCERT WITH FUNDING 5886 03:13:42,644 --> 03:13:44,279 AGENCIES, ONE WE'VE DONE FOR 5887 03:13:44,346 --> 03:13:46,982 YEARS, IN THE U.S., ONE WE'VE 5888 03:13:47,048 --> 03:13:50,385 MORE RECENTLY STARTED WITH, 5889 03:13:50,452 --> 03:13:56,057 CANCER RESEARCH U.K., BOTH FUND 5890 03:13:56,124 --> 03:13:58,259 CANCER RESEARCH. 5891 03:13:58,326 --> 03:14:01,529 THERE ARE HURDLES TO AUTHORS, IT 5892 03:14:01,596 --> 03:14:04,032 TAKES MORE TIME, THERE'S A DELAY 5893 03:14:04,099 --> 03:14:09,070 BETWEEN THE TIME THE GRANT IS 5894 03:14:09,137 --> 03:14:10,505 ACCEPTED AND THEN START 5895 03:14:10,572 --> 03:14:13,041 RECRUITING PARTICIPANTS BECAUSE 5896 03:14:13,108 --> 03:14:14,809 THEY HAVE TO GET THEIR PROTOCOL 5897 03:14:14,876 --> 03:14:16,177 PEER REVIEWED AND PUBLISHED. 5898 03:14:16,244 --> 03:14:17,645 WE'RE WORKING WITH THE COMMUNITY 5899 03:14:17,712 --> 03:14:19,581 TO LOWER THE BARRIERS, TO GET 5900 03:14:19,647 --> 03:14:21,449 ADDITIONAL FUNDERS TO SIGN UP, 5901 03:14:21,516 --> 03:14:22,550 TO NORMALIZE THIS BECAUSE I 5902 03:14:22,617 --> 03:14:26,187 THINK THE END RESULT IS ONE 5903 03:14:26,254 --> 03:14:27,922 THAT'S MORE STRAIGHTFORWARD, A 5904 03:14:27,989 --> 03:14:29,124 CLEAR PATH FROM SUBMITTING 5905 03:14:29,190 --> 03:14:30,325 FUNDING PROPOSAL TO HAVING 5906 03:14:30,392 --> 03:14:32,627 RESEARCH FUNDED TO HAVING REVIEW 5907 03:14:32,694 --> 03:14:34,929 OF THE PROJECT, TO COMPLETING 5908 03:14:34,996 --> 03:14:36,898 THE STUDY TO THEN PUBLISH IN THE 5909 03:14:36,965 --> 03:14:39,501 RESULTS, IN A VERY TRANSPARENT 5910 03:14:39,567 --> 03:14:39,968 AND OPEN WAY. 5911 03:14:40,035 --> 03:14:42,804 I THINK THIS IS A REALLY 5912 03:14:42,871 --> 03:14:44,906 EXCITING OPPORTUNITY FOR FURTHER 5913 03:14:44,973 --> 03:14:45,673 INVESTIGATION, FURTHER 5914 03:14:45,740 --> 03:14:46,207 EXPERIMENTATION. 5915 03:14:46,274 --> 03:14:47,742 BUT I ALSO WANTED TO TALK ABOUT 5916 03:14:47,809 --> 03:14:54,115 THE GOAL OF THE SESSION. 5917 03:14:54,182 --> 03:14:56,251 WHAT ARE BARRIERS AND HOW DO WE 5918 03:14:56,317 --> 03:15:01,623 ENCOURAGE DISSEMINATION OF NULL 5919 03:15:01,689 --> 03:15:02,857 AND NEGATIVE RESULTS? 5920 03:15:02,924 --> 03:15:05,026 ONE IS AROUND PERCEPTION. 5921 03:15:05,093 --> 03:15:06,227 THERE IS THIS OVERARCHING 5922 03:15:06,294 --> 03:15:07,629 CONSIDERATION IN THE COMMUNITY 5923 03:15:07,695 --> 03:15:09,431 THAT A NULL OR NEGATIVE RESULTS 5924 03:15:09,497 --> 03:15:10,832 SOMEHOW MEANS LOWER QUALITY BUT 5925 03:15:10,899 --> 03:15:13,001 IF YOU'VE DONE THE STUDY 5926 03:15:13,068 --> 03:15:14,736 PROPERLY YOU WOULD HAVE SEEN 5927 03:15:14,803 --> 03:15:16,504 POSITIVE ASSOCIATION, CONFIRMED 5928 03:15:16,571 --> 03:15:17,605 YOUR HYPOTHESIS. 5929 03:15:17,672 --> 03:15:18,373 IT'S ALSO PROBLEMATIC WHEN WE 5930 03:15:18,440 --> 03:15:21,309 HAVE AUTHORS WHO SET OUT TO 5931 03:15:21,376 --> 03:15:22,610 CONFIRM A HYPOTHESIS RATHER THAN 5932 03:15:22,677 --> 03:15:27,082 TEST A HIGH POTENTIAL APPROACHES 5933 03:15:27,148 --> 03:15:29,684 SITS, PRE-REGISTRATION IS A WAY 5934 03:15:29,751 --> 03:15:30,418 AROUND THIS. 5935 03:15:30,485 --> 03:15:32,487 I WANT TO BE CLEAR THAT ONE OF 5936 03:15:32,554 --> 03:15:33,721 THE PROBLEMS WITH NULL AND 5937 03:15:33,788 --> 03:15:35,256 NEGATIVE RESULTS IS PEOPLE LIKE 5938 03:15:35,323 --> 03:15:38,893 ME, JOURNAL EDITORS. 5939 03:15:38,960 --> 03:15:40,228 SO WHEN WE'RE SELECTING FOR 5940 03:15:40,295 --> 03:15:42,197 STUDIES WITH AN EXCITING OR NEW 5941 03:15:42,263 --> 03:15:44,232 OR IMPORTANT RESULT WE'RE 5942 03:15:44,299 --> 03:15:47,335 EXHIBITING TO THE PROBLEM. 5943 03:15:47,402 --> 03:15:48,970 I WAS AN EDITOR FOR NATURE CELL 5944 03:15:49,037 --> 03:15:50,505 BIOLOGY AND I CONTRIBUTED TO THE 5945 03:15:50,572 --> 03:15:53,241 PROBLEM BY BEING PART OF THE 5946 03:15:53,308 --> 03:15:54,976 SELECTED JOURNALS LOOKING FOR 5947 03:15:55,043 --> 03:15:56,444 NEW INTERESTING NOVEL RESULTS 5948 03:15:56,511 --> 03:15:57,846 HIGHLY CITED. 5949 03:15:57,912 --> 03:15:59,481 BUT NOW IN MY CURRENT POSITION, 5950 03:15:59,547 --> 03:16:03,218 WE DON'T CARE ABOUT THE IMPACT 5951 03:16:03,284 --> 03:16:03,451 FACTOR. 5952 03:16:03,518 --> 03:16:05,720 WE TAKE ALL RESEARCH THAT IS 5953 03:16:05,787 --> 03:16:07,355 PERFORMED TO HIGH TECHNICAL AND 5954 03:16:07,422 --> 03:16:08,890 ETHICAL STANDARDS AND WE'LL 5955 03:16:08,957 --> 03:16:10,625 PUBLISH IT REGARDLESS OF THE 5956 03:16:10,692 --> 03:16:11,092 OUTCOME. 5957 03:16:11,159 --> 03:16:15,497 THAT'S A REALLY WONDERFUL 5958 03:16:15,563 --> 03:16:18,733 POSITION TO BE IN. 5959 03:16:18,800 --> 03:16:20,502 EDITORS WITH DO A LOT TO HELP 5960 03:16:20,568 --> 03:16:21,936 THE COMMUNITY ADDRESSING THE 5961 03:16:22,003 --> 03:16:23,238 PROBLEM BY ENSURING WE'RE 5962 03:16:23,304 --> 03:16:24,105 EVALUATING RESEARCH ON INTEREST 5963 03:16:24,172 --> 03:16:25,406 OF THE QUESTION, RIGHT? 5964 03:16:25,473 --> 03:16:28,042 HOW INTERESTING IS THAT 5965 03:16:28,109 --> 03:16:28,710 QUESTION, NOW HOW INTERESTING 5966 03:16:28,776 --> 03:16:30,345 ARE THE DATA. 5967 03:16:30,411 --> 03:16:31,546 THAT'S A FUNDAMENTAL CHANGE IN 5968 03:16:31,613 --> 03:16:33,081 TERMS OF HOW WE'RE EVALUATING 5969 03:16:33,148 --> 03:16:33,515 RESEARCH. 5970 03:16:33,581 --> 03:16:34,849 I THINK THE JOURNALS NEED TO 5971 03:16:34,916 --> 03:16:37,118 ENSURE THEY ARE SELECTING FOR 5972 03:16:37,185 --> 03:16:45,226 RIGOR, SO TECHNICAL AND ETHICAL 5973 03:16:45,293 --> 03:16:45,426 RIGOR. 5974 03:16:45,493 --> 03:16:48,396 THESE IDEAS OF NEW WAYS OF 5975 03:16:48,463 --> 03:16:49,831 WORKING WITH OUR COMMUNITIES 5976 03:16:49,898 --> 03:16:51,132 WITH PARTNERSHIPS ARE ONE, 5977 03:16:51,199 --> 03:16:53,134 ADDITIONAL INITIATIVES IN THE 5978 03:16:53,201 --> 03:16:54,435 COMING YEARS TO MAKE SURE WE'RE 5979 03:16:54,502 --> 03:16:56,504 MAKING IT EASY FOR AUTHORS, THE 5980 03:16:56,571 --> 03:16:57,372 DEFAULT OPTION, THE EASY OPTION 5981 03:16:57,438 --> 03:17:00,208 IS TO BE OPEN, TO BE 5982 03:17:00,275 --> 03:17:01,876 TRANSPARENT, IS TO PUBLISH THE 5983 03:17:01,943 --> 03:17:03,845 FULL SET OF DATA REGARDLESS OF 5984 03:17:03,912 --> 03:17:04,913 THE DIRECTIONALITY OF THE 5985 03:17:04,979 --> 03:17:05,146 OUTCOME. 5986 03:17:05,213 --> 03:17:07,448 I LOOK FORWARD TO DISCUSSING 5987 03:17:07,515 --> 03:17:12,720 THIS IN MORE DETAIL. 5988 03:17:12,787 --> 03:17:15,823 THANKS. 5989 03:17:18,893 --> 03:17:22,764 >> WONDERFUL TO BE PART OF THIS 5990 03:17:22,830 --> 03:17:23,965 PANEL. 5991 03:17:24,032 --> 03:17:29,437 MY NAME IS KIF LIAKETH-ALI, JUST 5992 03:17:29,504 --> 03:17:30,738 COMPLETE MY POSTDOC AT STANFORD 5993 03:17:30,805 --> 03:17:31,272 UNIVERSITY. 5994 03:17:31,339 --> 03:17:38,713 I STARTED MY NEW JOB AS 5995 03:17:38,780 --> 03:17:44,452 LECTURER, ASSISTANT PROFESSOR, 5996 03:17:44,519 --> 03:17:46,554 UNIVERSITY OF SOUTHAMPTON THREE 5997 03:17:46,621 --> 03:17:51,259 WEEKS AGO. 5998 03:17:51,326 --> 03:17:52,126 SLIGHTLY DIFFERENT COMPONENT 5999 03:17:52,193 --> 03:17:55,096 WANTED TO SHARE MY EXPERIENCE 6000 03:17:55,163 --> 03:17:56,297 PUBLISHING NEGATIVE RESULTS AS 6001 03:17:56,364 --> 03:17:58,800 SOMEONE DURING EARLY IN THEIR 6002 03:17:58,866 --> 03:18:00,001 CAREER IN MOLECULAR 6003 03:18:00,068 --> 03:18:01,102 NEUROSCIENCE. 6004 03:18:01,169 --> 03:18:08,676 SO, WE AT THE LAB STUDY HOW 6005 03:18:08,743 --> 03:18:11,279 NEURONS COMMUNICATE, A 6006 03:18:11,346 --> 03:18:12,714 FUNDAMENTAL UNIT OF INFORMATION 6007 03:18:12,780 --> 03:18:14,916 PROCESSING IN THE BRAIN. 6008 03:18:14,983 --> 03:18:18,686 WE PARTICULARLY ARE INTERESTED 6009 03:18:18,753 --> 03:18:21,356 IN HOW LIGANDS COMMUNICATE WITH 6010 03:18:21,422 --> 03:18:23,091 EACH OTHER, THE TWO IMPORTANT 6011 03:18:23,157 --> 03:18:26,928 PROTEINS IN THE SYNAPSE THAT 6012 03:18:26,995 --> 03:18:30,331 ALLOWS US, YOU CAN NAME ANY 6013 03:18:30,398 --> 03:18:33,034 FUNCTION, DAY-TO-DAY FUNCTIONS, 6014 03:18:33,101 --> 03:18:36,971 THAT IS IMPORTANT AND THESE 6015 03:18:37,038 --> 03:18:40,608 PROTEINS ARE KNOWN TO HAVE 6016 03:18:40,675 --> 03:18:43,211 MUTATED, SO THIS IS INTERESTING 6017 03:18:43,278 --> 03:18:45,847 QUESTION TO ASK HOW THEY 6018 03:18:45,913 --> 03:18:50,118 FUNDAMENTALLY FUNCTION IN THE 6019 03:18:50,184 --> 03:18:51,419 SYNAPSES. 6020 03:18:51,486 --> 03:18:52,720 WHEN I STARTED MY POSTDOC SIX 6021 03:18:52,787 --> 03:18:56,457 YEARS AGO, BEFORE I GO TO THE 6022 03:18:56,524 --> 03:18:58,526 ACTUAL CASE STUDY I WANT TO 6023 03:18:58,593 --> 03:19:00,962 BRIEFLY INTRODUCE THE ACTUAL 6024 03:19:01,029 --> 03:19:03,131 ASPECT OF HOW, YOU KNOW, 6025 03:19:03,197 --> 03:19:05,233 ALTERNATIVE SPLICING WHICH HAS 6026 03:19:05,300 --> 03:19:08,503 BEEN THE FOCUS OF OUR QUESTION, 6027 03:19:08,569 --> 03:19:12,774 SO IT CAN -- IT'S A VERY 6028 03:19:12,840 --> 03:19:13,975 FUNDAMENTAL MECHANISM BY WHICH 6029 03:19:14,042 --> 03:19:17,045 GENES CAN CHANGE THE OUTCOME OF 6030 03:19:17,111 --> 03:19:19,013 FUNCTION, SO IN A SENTENCE HOW 6031 03:19:19,080 --> 03:19:21,582 TO CREATE A DIFFERENT MEANING BY 6032 03:19:21,649 --> 03:19:24,185 SCRAMBLING THE WORDS. 6033 03:19:24,252 --> 03:19:27,221 SIMILARLY, THE GENES CAN CHANGE 6034 03:19:27,288 --> 03:19:31,059 THEIR FUNCTION BASED ON WHAT 6035 03:19:31,125 --> 03:19:34,462 SEQUENCE THEY INSERT. 6036 03:19:34,529 --> 03:19:39,467 NOW, THIS VERY SPECIFIC 6037 03:19:39,534 --> 03:19:42,637 MECHANISM, VERY FUNDAMENTAL IN 6038 03:19:42,704 --> 03:19:43,838 EUKARYOTE. 6039 03:19:43,905 --> 03:19:48,242 THE ACTIVITY, THE NEURONAL 6040 03:19:48,309 --> 03:19:49,977 INDUCED ACTIVITY SPLICING HAS 6041 03:19:50,044 --> 03:19:52,513 WIDE IMPLICATIONS NOT ONLY TO 6042 03:19:52,580 --> 03:19:53,381 FUNDAMENTAL BIOLOGY, ALSO 6043 03:19:53,448 --> 03:19:55,016 VARIOUS DISEASES. 6044 03:19:55,083 --> 03:19:58,186 WHEN I STARTED POSTDOC POSITION 6045 03:19:58,252 --> 03:20:02,557 IN THE LAB, THIS PAPER PUBLISHED 6046 03:20:02,623 --> 03:20:05,293 IN MARCH 2017 SHOWING REALLY, 6047 03:20:05,360 --> 03:20:08,696 REALLY INTERESTING RESULTS WHERE 6048 03:20:08,763 --> 03:20:11,099 YOU CAN CHANGE THE GENE PROTEIN 6049 03:20:11,165 --> 03:20:13,835 COMPONENT BY PUTTING THE MOUSE 6050 03:20:13,901 --> 03:20:18,973 IN A BEHAVIORAL PARADIGM OR ANY 6051 03:20:19,040 --> 03:20:19,307 BEHAVIORAL. 6052 03:20:19,374 --> 03:20:22,043 JUST INDUCING THE NEURONAL 6053 03:20:22,110 --> 03:20:22,310 ACTIVITY. 6054 03:20:22,377 --> 03:20:26,080 NOW, I WAS SO FASCINATED BECAUSE 6055 03:20:26,147 --> 03:20:28,616 I REALLY WANTED TO CONTRIBUTE A 6056 03:20:28,683 --> 03:20:30,485 VERY KEY DISCOVERY TO THE 6057 03:20:30,551 --> 03:20:32,353 COMMUNITY, SO THAT'S THE REASON 6058 03:20:32,420 --> 03:20:35,523 I STARTED POSTDOC IN THIS LAB. 6059 03:20:35,590 --> 03:20:40,461 AND I STARTED THIS POSTDOC WITH 6060 03:20:40,528 --> 03:20:42,497 THE TWO-YEAR LIMITED FELLOWSHIP 6061 03:20:42,563 --> 03:20:43,631 FROM EUROPEAN MOLECULAR BIOLOGY 6062 03:20:43,698 --> 03:20:47,468 ORGANIZATION WHICH MEANS MY TIME 6063 03:20:47,535 --> 03:20:48,669 WAS LIMITED. 6064 03:20:48,736 --> 03:20:51,873 HAVING A VERY AMBITIOUS PLANS, 6065 03:20:51,939 --> 03:20:58,613 ONLY TO PUBLIC THIS NEGATIVE 6066 03:20:58,679 --> 03:21:01,249 RESULT, THE TITLE SOUNDS LIKE 6067 03:21:01,315 --> 03:21:03,551 REVIEW BUT IT'S EXPERIMENTAL 6068 03:21:03,618 --> 03:21:05,653 PAPER WE PROVE IT IS WRONG, NOT 6069 03:21:05,720 --> 03:21:10,458 ENTIRELY WRONG BUT MAJOR 6070 03:21:10,525 --> 03:21:14,162 COMPONENT THEY USE, EXPERIMENTAL 6071 03:21:14,228 --> 03:21:15,363 BASIC COMPONENT IS FLAWED. 6072 03:21:15,430 --> 03:21:19,867 NOW, WHAT WE FOUND IS BASICALLY 6073 03:21:19,934 --> 03:21:21,502 A REPEAT FUNDAMENTAL PARADIGM 6074 03:21:21,569 --> 03:21:27,175 USED IN MANY NEUROSCIENCE LABS 6075 03:21:27,241 --> 03:21:31,446 ESPECIALLY AT THE CELL BIOLOGY 6076 03:21:31,512 --> 03:21:34,515 COMPONENT, TO MANIPULATE NEURAL 6077 03:21:34,582 --> 03:21:37,785 ACTIVITY EXPOSING TO CONDITIONS, 6078 03:21:37,852 --> 03:21:39,654 BASICALLY HAMMERING THE CELLS, 6079 03:21:39,720 --> 03:21:41,389 WHEN YOU INDUCE ACTIVITY 6080 03:21:41,456 --> 03:21:49,497 CREATING EPILEPSY, CELLS DIE, 6081 03:21:49,564 --> 03:21:53,901 NOT ALL CELLS, IT INDUCES 6082 03:21:53,968 --> 03:21:54,535 SPECIFIC POPULATIONS THAT LEADS 6083 03:21:54,602 --> 03:21:55,670 TO CELL DEATH. 6084 03:21:55,736 --> 03:21:59,240 IN A PHYSIOLOGY CONDITION YOU 6085 03:21:59,307 --> 03:22:02,310 HAVE NEURONS, TWO MAJOR CELL 6086 03:22:02,376 --> 03:22:06,280 POPULATIONS IN THE BRAIN, WHEN 6087 03:22:06,347 --> 03:22:12,487 YOU INDUCE CONDITIONS THAT LEADS 6088 03:22:12,553 --> 03:22:15,456 TO NEURONAL CELL DEATH, 6089 03:22:15,523 --> 03:22:17,425 ASTROCYTES PRESERVED, LEADS TO 6090 03:22:17,492 --> 03:22:22,997 SKEWING OF RESULTS ESPECIALLY 6091 03:22:23,064 --> 03:22:23,931 MOLECULAR FOCUSED, DIFFERENT 6092 03:22:23,998 --> 03:22:26,067 ALTERNATE SPLICING PATTERNS. 6093 03:22:26,133 --> 03:22:31,105 YOU COULD BUILD A STORY OUT OF 6094 03:22:31,172 --> 03:22:31,973 IT. 6095 03:22:32,039 --> 03:22:35,676 LIMITATIONS CREATIVE ENOUGH, YOU 6096 03:22:35,743 --> 03:22:37,879 CAN BUILD A STORY THAT'S 6097 03:22:37,945 --> 03:22:38,846 COMPLETELY FLAWED. 6098 03:22:38,913 --> 03:22:41,816 TO SUMMARIZE WHAT WE'VE FOUND IN 6099 03:22:41,883 --> 03:22:52,360 THIS CASE STUDY, IT'S VERY 6100 03:22:56,497 --> 03:22:58,032 SPECIFIC TO COMMUNITY, MOSTLY 6101 03:22:58,099 --> 03:23:01,435 WORKING ON THIS EXPERIMENTAL 6102 03:23:01,502 --> 03:23:01,702 PARADIGMS. 6103 03:23:01,769 --> 03:23:05,239 SO, OUR STUDY OR PAPER 6104 03:23:05,306 --> 03:23:06,841 UNDERSCORED VULNERABILITY OF 6105 03:23:06,908 --> 03:23:09,076 COMMON PROCEDURES, FUNDAMENTAL 6106 03:23:09,143 --> 03:23:13,681 IN VITRO TECHNIQUE, AND ALSO 6107 03:23:13,748 --> 03:23:15,416 EMPHASIZED PREVALENCE OF 6108 03:23:15,483 --> 03:23:17,485 EXPERIMENTAL OPTI-PLEX. 6109 03:23:17,552 --> 03:23:24,592 THERE'S TONS OF DATA, OTHER WORK 6110 03:23:24,659 --> 03:23:27,995 WAS VALIDATED BY OTHER GROUPS. 6111 03:23:28,062 --> 03:23:29,530 SO, FINALLY, IT ALSO UNDERSCORES 6112 03:23:29,597 --> 03:23:32,900 THE VALUE OF PUBLISHING NEGATIVE 6113 03:23:32,967 --> 03:23:34,902 RESULTS TO REFINE EXPERIMENTAL 6114 03:23:34,969 --> 03:23:37,738 PROTOCOL TO ADVANCE SCIENCE. 6115 03:23:37,805 --> 03:23:40,508 WHEN WE PUBLISHED SEVERAL MONTHS 6116 03:23:40,575 --> 03:23:43,010 LATER, WE GOT A HEARTENING IMAGE 6117 03:23:43,077 --> 03:23:48,182 FROM SOMEBODY WE DIDN'T 6118 03:23:48,249 --> 03:23:50,885 COLLABORATE WITH OR KNOW, SAYING 6119 03:23:50,952 --> 03:23:52,954 THAT THEIR GROUP ALSO HAVE SEEN 6120 03:23:53,020 --> 03:23:55,156 EXACTLY THE SAME RESULTS, BUT 6121 03:23:55,222 --> 03:23:56,457 THEY FAILED TO PUBLISH IT 6122 03:23:56,524 --> 03:23:58,926 BECAUSE THEY WERE TRYING TO 6123 03:23:58,993 --> 03:24:00,628 CHASE OFF THE ORIGINAL JOURNAL 6124 03:24:00,695 --> 03:24:05,666 WHO PUBLISHED THIS PAPER AND 6125 03:24:05,733 --> 03:24:07,735 THEN ULTIMATELY REJECTED. 6126 03:24:07,802 --> 03:24:09,937 THAT WAS THE BARRIER, BECAUSE WE 6127 03:24:10,004 --> 03:24:11,906 DON'T GO THAT ROUTE BECAUSE SOME 6128 03:24:11,973 --> 03:24:14,742 WARNED ME THAT ARE YOU READY TO 6129 03:24:14,809 --> 03:24:16,177 SPEND ANOTHER TWO YEARS WITH 6130 03:24:16,243 --> 03:24:23,084 HUNDREDS OFE MAILS BACK AND 6131 03:24:23,150 --> 03:24:23,417 FORTH? 6132 03:24:23,484 --> 03:24:28,522 WE JUST WANT THIS PAPER TO BE 6133 03:24:28,589 --> 03:24:28,689 OUT. 6134 03:24:28,756 --> 03:24:30,057 SO, EVENTUALLY WORKING WITH 6135 03:24:30,124 --> 03:24:33,260 GILES, WE MANAGED TO PUBLISH OR 6136 03:24:33,327 --> 03:24:37,531 MANAGED TO HELP OR PUBLISH THEIR 6137 03:24:37,598 --> 03:24:39,300 PAPER LATER. 6138 03:24:39,367 --> 03:24:43,004 NONE OF OUR PUBLICATION PAPERS 6139 03:24:43,070 --> 03:24:44,438 NOT CITED THE WAY ORIGINAL 6140 03:24:44,505 --> 03:24:45,973 PAPERS STILL BEING CITED TODAY. 6141 03:24:46,040 --> 03:24:49,143 A PAPER I'M REVIEWING RIGHT NOW 6142 03:24:49,210 --> 03:24:51,312 CITING THE ORIGINAL PAPER 6143 03:24:51,379 --> 03:24:53,948 IGNORING COMPLETELY WHAT WE DID 6144 03:24:54,015 --> 03:24:59,120 SHOWING NEGATIVE RESULTS, IT'S 6145 03:24:59,186 --> 03:25:02,490 COMPLETELY MISSING IMPORTANT 6146 03:25:02,556 --> 03:25:03,090 SCIENCE. 6147 03:25:03,157 --> 03:25:03,991 EVEN DISHEARTENING, ANY GENE 6148 03:25:04,058 --> 03:25:04,992 FUNCTION INVOLVED IN A DISEASE 6149 03:25:05,059 --> 03:25:09,163 AND WHERE IT IS LEADING TO. 6150 03:25:09,230 --> 03:25:11,132 NOW, WHEN I STARTED POSTDOC AT 6151 03:25:11,198 --> 03:25:12,900 STANFORD, I CAME FROM THE U.K., 6152 03:25:12,967 --> 03:25:17,471 WHERE I HAD MY Ph.D., AND MY 6153 03:25:17,538 --> 03:25:23,377 Ph.D. SUPERVISOR GAVE ME AN 6154 03:25:23,444 --> 03:25:25,579 ADVICE WHICH WAS AN EXPERIMENT, 6155 03:25:25,646 --> 03:25:34,321 SHE SAID, IF YOUR EXPERIMENT IS 6156 03:25:34,388 --> 03:25:36,190 NOT COMPLETE, NOBODY WILL 6157 03:25:36,257 --> 03:25:37,825 PUBLISH, NO MATTER NEGATIVE OR 6158 03:25:37,892 --> 03:25:39,226 POSITIVE RESULTS. 6159 03:25:39,293 --> 03:25:43,931 I SACRIFICED MY -- I WOULDN'T 6160 03:25:43,998 --> 03:25:44,899 SAY SACRIFICED, TOOK ADVENTUROUS 6161 03:25:44,965 --> 03:25:48,436 ENCOUNTER, LAST TWO YEARS OF MY 6162 03:25:48,502 --> 03:25:48,936 POSTDOC. 6163 03:25:49,003 --> 03:25:53,140 THAT ACTUALLY ALMOST KILLED MY 6164 03:25:53,207 --> 03:25:53,441 CAREER. 6165 03:25:53,507 --> 03:25:56,177 BUT MY RESILIENCE AND SUPPORT 6166 03:25:56,243 --> 03:25:58,979 FROM MY POSTDOC ADVISER MADE ME 6167 03:25:59,046 --> 03:26:00,381 GO TO THAT. 6168 03:26:00,448 --> 03:26:02,149 AND WHAT I WANT TO DISCUSS TODAY 6169 03:26:02,216 --> 03:26:06,654 IS WHAT ARE THE ACTUAL BARRIERS 6170 03:26:06,721 --> 03:26:08,723 IN PUBLISHING NEGATIVE RESULTS 6171 03:26:08,789 --> 03:26:11,425 AS SOMEONE FOR THE VERY EARLY IN 6172 03:26:11,492 --> 03:26:15,162 THEIR CAREER, WHAT ARE THE 6173 03:26:15,229 --> 03:26:16,163 SUPPORTS IN PLACE? 6174 03:26:16,230 --> 03:26:18,966 WE'VE SEEN VARIETY OF SUPPORT IN 6175 03:26:19,033 --> 03:26:20,601 SPECIFICALLY TO SPECIFIC TO MY 6176 03:26:20,668 --> 03:26:25,339 CASE STUDY FROM P.I., FUNDERS, 6177 03:26:25,406 --> 03:26:27,208 WHAT ARE INCENTIVES, I WAS 6178 03:26:27,274 --> 03:26:30,611 FORTUNATE TO HAVE SOME FORM OF 6179 03:26:30,678 --> 03:26:33,247 RECOGNITION FOR THIS CASE STUDY. 6180 03:26:33,314 --> 03:26:35,049 BUT THE QUESTION IS DO ALL 6181 03:26:35,116 --> 03:26:43,090 PEOPLE WHO STRIVE TO DO THIS 6182 03:26:43,157 --> 03:26:43,891 HAVE THAT SUPPORT? 6183 03:26:43,958 --> 03:26:45,326 I'M HAPPY TO DISCUSS THIS IN 6184 03:26:45,392 --> 03:26:45,826 THIS PANEL. 6185 03:26:45,893 --> 03:26:48,395 THANK YOU. 6186 03:26:51,198 --> 03:26:51,732 >> HELLO, EVERYBODY. 6187 03:26:51,799 --> 03:26:54,668 I DON'T HAVE SLIDES. 6188 03:26:54,735 --> 03:26:57,505 I'M JOSE MERINO, EDITOR IN CHIEF 6189 03:26:57,571 --> 03:26:59,273 OF NEUROLOGY, THE JOURNAL OF THE 6190 03:26:59,340 --> 03:27:02,109 AMERICAN ACADEMY OF NEUROLOGY, 6191 03:27:02,176 --> 03:27:05,613 AND THE FAMILY OF JOURNALS UNDER 6192 03:27:05,679 --> 03:27:08,582 INFLAMMATION, ONE ON GENETICS, 6193 03:27:08,649 --> 03:27:09,316 CLINICAL PRACTICE, EDUCATION. 6194 03:27:09,383 --> 03:27:13,154 TO GIVE A LITTLE BIT OF 6195 03:27:13,220 --> 03:27:15,222 BACKGROUND, JUST FOR NEUROLOGY, 6196 03:27:15,289 --> 03:27:16,991 WE FOCUS ON CLINICAL RESEARCH, 6197 03:27:17,057 --> 03:27:20,928 WE DON'T PUBLISH BASIC STUDIES, 6198 03:27:20,995 --> 03:27:21,796 ANIMAL STUDIES, IT'S CLINICAL 6199 03:27:21,862 --> 03:27:23,130 RESEARCH ONLY. 6200 03:27:23,197 --> 03:27:25,299 WE GOT ABOUT 4,000 SUBMISSIONS A 6201 03:27:25,366 --> 03:27:27,301 YEAR AND REJECT HALF OF THEM AND 6202 03:27:27,368 --> 03:27:29,537 PUBLISH ABOUT 10% OF THE 6203 03:27:29,603 --> 03:27:30,304 SUBMISSIONS THAT WE HAVE, SO 6204 03:27:30,371 --> 03:27:32,406 THAT GIVES YOU AN IDEA OF MY 6205 03:27:32,473 --> 03:27:35,810 COMMENTS WILL COME FROM THAT 6206 03:27:35,876 --> 03:27:37,011 PERSPECTIVE. 6207 03:27:37,077 --> 03:27:40,848 IN 2018 WE ACTUALLY STARTED AN 6208 03:27:40,915 --> 03:27:44,251 ACTIVE INITIATIVE TO PROMOTE THE 6209 03:27:44,318 --> 03:27:47,221 SUBMISSIONS OF WELL CONDUCTED 6210 03:27:47,288 --> 03:27:48,589 NEGATIVE, INCONCLUSIVE OR 6211 03:27:48,656 --> 03:27:50,057 REPRODUCIBLE REPORTS, INITIALLY 6212 03:27:50,124 --> 03:27:51,892 AS A SEPARATE MANUSCRIPT 6213 03:27:51,959 --> 03:27:52,159 CATEGORY. 6214 03:27:52,226 --> 03:27:53,994 PEOPLE WOULD SUBMIT TO THE NULL 6215 03:27:54,061 --> 03:27:54,528 CATEGORY. 6216 03:27:54,595 --> 03:27:56,197 WHEN I BECAME EDITOR IN CHIEF I 6217 03:27:56,263 --> 03:27:57,698 SAID THAT DOESN'T MAKE SENSE TO 6218 03:27:57,765 --> 03:28:00,000 ME BECAUSE WE DON'T WANT TO 6219 03:28:00,067 --> 03:28:01,635 SIGNAL A DIFFERENT CATEGORY. 6220 03:28:01,702 --> 03:28:02,436 IT'S RESEARCH IS RESEARCH, 6221 03:28:02,503 --> 03:28:06,106 WHATEVER IT IS, IT SHOULD BE. 6222 03:28:06,173 --> 03:28:09,643 WE DID AWAY WITH THAT MANUSCRIPT 6223 03:28:09,710 --> 03:28:12,680 CATEGORY, LABELED THEM AS NULL 6224 03:28:12,746 --> 03:28:13,781 RESULTS COLLECTION BUT SUBMITTED 6225 03:28:13,848 --> 03:28:14,982 AS REGULAR RESEARCH ARTICLES. 6226 03:28:15,049 --> 03:28:16,851 WE DO STATE IN OUR INSTRUCTIONS 6227 03:28:16,917 --> 03:28:18,619 FOR AUTHORS IF THE QUESTION IS 6228 03:28:18,686 --> 03:28:20,154 IMPORTANT, THE METHODS ARE 6229 03:28:20,221 --> 03:28:22,990 SOUND, AND THE REPORTING IS 6230 03:28:23,057 --> 03:28:24,225 TRANSPARENT, FULL CONSIDERATION 6231 03:28:24,291 --> 03:28:25,426 IS GIVEN TO ALL PAPERS 6232 03:28:25,492 --> 03:28:27,061 REGARDLESS OF RESULT. 6233 03:28:27,127 --> 03:28:28,395 WE'RE ACTUALLY VERY PROMINENTLY 6234 03:28:28,462 --> 03:28:29,897 SIGNALING THE FACT THAT WE'RE 6235 03:28:29,964 --> 03:28:34,168 INTERESTED IN NULL STUDIES AND 6236 03:28:34,235 --> 03:28:36,036 THAT REPORT SO-CALLED NEGATIVE 6237 03:28:36,103 --> 03:28:37,905 REPORTS, THAT'S NOT AN IDEAL 6238 03:28:37,972 --> 03:28:40,307 TERM BUT WE'LL LEAVE IT AT THAT. 6239 03:28:40,374 --> 03:28:41,942 WE DON'T ACTUALLY TRACK HOW MANY 6240 03:28:42,009 --> 03:28:43,344 HAVE BEEN SUBMITTED ABOUT WE DO 6241 03:28:43,410 --> 03:28:45,546 KNOW HOW MANY HAVE BEEN 6242 03:28:45,613 --> 03:28:45,813 PUBLISHED. 6243 03:28:45,880 --> 03:28:48,816 ABOUT 40 OVER THE PAST FIVE 6244 03:28:48,883 --> 03:28:49,116 YEARS. 6245 03:28:49,183 --> 03:28:51,151 THERE'S BEEN MORE THAT NULL 6246 03:28:51,218 --> 03:28:52,786 STUDIES THAT HAVE BEEN 6247 03:28:52,853 --> 03:28:53,354 PUBLISHED. 6248 03:28:53,420 --> 03:28:55,756 IT'S JUST THEY HAVEN'T BEEN 6249 03:28:55,823 --> 03:28:57,057 LABELED AS SUCH. 6250 03:28:57,124 --> 03:28:59,994 WE FIND THOSE WE'VE LABELED HAVE 6251 03:29:00,060 --> 03:29:10,604 THE SAME NUMBER OF AVERAGE SIGH 6252 03:29:17,878 --> 03:29:18,612 CITATIONS, SAME ALT-METRICS, 6253 03:29:18,679 --> 03:29:20,347 DELAY ON PICKING UP ON THESE 6254 03:29:20,414 --> 03:29:20,748 INSTANCES. 6255 03:29:20,814 --> 03:29:24,084 I WANTED TO TALK ABOUT SOME OF 6256 03:29:24,151 --> 03:29:25,619 THE BARRIERS WE FOUND, SOME 6257 03:29:25,686 --> 03:29:26,153 LESSONS. 6258 03:29:26,220 --> 03:29:27,922 WHAT IS A NULL STUDY? 6259 03:29:27,988 --> 03:29:30,424 I FIND THAT'S SOMETIMES WE HAVE 6260 03:29:30,491 --> 03:29:32,293 DIFFICULTY DEFINING EXACTLY. 6261 03:29:32,359 --> 03:29:34,795 WE OFTEN GET PAPERS, WITH A 6262 03:29:34,862 --> 03:29:36,897 CLINICAL TRIAL, YOU HAVE A 6263 03:29:36,964 --> 03:29:37,631 REGISTERED PRIMARY OUTCOME AND 6264 03:29:37,698 --> 03:29:39,934 THEN THIS IS THE PRIMARY 6265 03:29:40,000 --> 03:29:40,968 OUTCOME, THERE WAS AN EFFECT, 6266 03:29:41,035 --> 03:29:43,337 THERE WAS NO EFFECT. 6267 03:29:43,404 --> 03:29:44,538 IT'S DIFFICULT IN OBSERVATIONAL 6268 03:29:44,605 --> 03:29:47,508 STUDIES PEOPLE SAY WE'RE LOOKING 6269 03:29:47,574 --> 03:29:49,810 AT THE BIOMARKERS FOR A GIVEN 6270 03:29:49,877 --> 03:29:50,878 OUTCOME IN PATIENTS WITH 6271 03:29:50,945 --> 03:29:51,845 ALZHEIMER'S, WE FOLLOWED THEM, 6272 03:29:51,912 --> 03:29:55,049 NO ASSOCIATION WITH SOME OF 6273 03:29:55,115 --> 03:29:57,017 THEM, DO YOU CALL THAT NULL OR 6274 03:29:57,084 --> 03:29:58,319 NOT NULL STUDY, THOSE ARE THE 6275 03:29:58,385 --> 03:30:01,055 ONES WHERE WE GET INTO DIFFICULT 6276 03:30:01,121 --> 03:30:01,388 AREA. 6277 03:30:01,455 --> 03:30:05,092 THAT LEADS TO ANOTHER THING TO 6278 03:30:05,159 --> 03:30:06,627 COMMENT ABOUT. 6279 03:30:06,694 --> 03:30:11,231 WHAT I'M FINDING IS AUTHORS HAVE 6280 03:30:11,298 --> 03:30:12,466 DIFFICULTY WITH FRAMING STUDIES 6281 03:30:12,533 --> 03:30:13,867 AS NULL STUDIES. 6282 03:30:13,934 --> 03:30:16,503 SO IN THE EXAMPLE, FOR EXAMPLE, 6283 03:30:16,570 --> 03:30:17,972 THE VARIOUS BIOMARKERS 6284 03:30:18,038 --> 03:30:19,540 ASSOCIATED WITH OUTCOME, THE 6285 03:30:19,606 --> 03:30:23,811 TITLE WILL SAY FACTOR X IS 6286 03:30:23,877 --> 03:30:25,446 ASSOCIATED, AND THAT'S WHAT THEY 6287 03:30:25,512 --> 03:30:27,848 HIGHLIGHT IN THE CONCLUSION, IN 6288 03:30:27,915 --> 03:30:29,717 THE FINDINGS, EVEN THOUGH AT THE 6289 03:30:29,783 --> 03:30:30,818 BEGINNING LOOKING AT SIX 6290 03:30:30,884 --> 03:30:32,786 FACTORS, FOUND THIS ONE 6291 03:30:32,853 --> 03:30:33,687 POSITIVE, ASKING PEOPLE DON'T 6292 03:30:33,754 --> 03:30:34,888 ANNOUNCE RESULTS, TELL US WHAT 6293 03:30:34,955 --> 03:30:36,557 THE QUESTION WAS AND THE TITLE 6294 03:30:36,623 --> 03:30:37,758 AND THEN YOU HAVE TO HIGHLIGHT 6295 03:30:37,825 --> 03:30:39,560 THE FACT THAT IT'S NOT. 6296 03:30:39,626 --> 03:30:41,662 WE ALSO HAVE THAT ISSUE WITH 6297 03:30:41,729 --> 03:30:41,929 TRIALS. 6298 03:30:41,996 --> 03:30:43,497 AS I MENTIONED, FOR US IT'S EASY 6299 03:30:43,564 --> 03:30:45,933 TO GO TO clinicaltrials.gov OR 6300 03:30:46,000 --> 03:30:48,535 ANY OF THE OTHER W.H.O. 6301 03:30:48,602 --> 03:30:50,504 REGISTRIES WHICH WE ACCEPT AND 6302 03:30:50,571 --> 03:30:53,140 SEE IF WHAT'S BEEN REPORTED IS A 6303 03:30:53,207 --> 03:30:54,908 PRIMARY OUTCOME OR NOT, WE'RE 6304 03:30:54,975 --> 03:30:56,076 GETTING IN EARLIER PHASE 6305 03:30:56,143 --> 03:30:58,812 CLINICAL TRIALS REPORTS WILL 6306 03:30:58,879 --> 03:31:00,180 HIGHLIGHT SECONDARY OUTCOMES AND 6307 03:31:00,247 --> 03:31:01,448 NOT THE PRIMARY OUTCOME. 6308 03:31:01,515 --> 03:31:04,184 WHAT WE TELL AUTHORS, YOU HAVE 6309 03:31:04,251 --> 03:31:06,053 TO REFRAME THE PAPER, TELL US 6310 03:31:06,120 --> 03:31:08,889 ABOUT 9 PRIMARY OUTCOME. 6311 03:31:08,956 --> 03:31:12,393 WE DON'T CARE THE RESULT WAS 6312 03:31:12,459 --> 03:31:13,694 NULL BUT YOU'RE REPORTING A 6313 03:31:13,761 --> 03:31:16,897 FINDING WHICH WE THINK IS 6314 03:31:16,964 --> 03:31:18,532 IMPORTANT BECAUSE WE THINK THE 6315 03:31:18,599 --> 03:31:20,501 RESEARCH QUESTION IS IMPORTANT. 6316 03:31:20,567 --> 03:31:23,003 THE OTHER ONE HAS TO DO WITH HOW 6317 03:31:23,070 --> 03:31:25,639 I BEGAN, I THINK, AS I 6318 03:31:25,706 --> 03:31:27,708 MENTIONED, ANNOUNCING THE 6319 03:31:27,775 --> 03:31:29,810 RESULTS, HIGHLIGHTING THE OTHER 6320 03:31:29,877 --> 03:31:30,677 SECONDARY OUTCOMES. 6321 03:31:30,744 --> 03:31:34,281 AND THEN I ALSO FIND THAT 6322 03:31:34,348 --> 03:31:36,383 REVIEWERS HAVE A DIFFICULTY WITH 6323 03:31:36,450 --> 03:31:38,318 NULL STUDIES. 6324 03:31:38,385 --> 03:31:40,654 SO WE OFTEN GET IN THE COMMENTS, 6325 03:31:40,721 --> 03:31:42,489 THIS IS A GREAT QUESTION, THIS 6326 03:31:42,556 --> 03:31:44,324 IS A GREAT STUDY, IT'S VERY WELL 6327 03:31:44,391 --> 03:31:46,660 DONE, BUT WE DON'T THINK IT'S OF 6328 03:31:46,727 --> 03:31:47,861 THE CALIBER FOR NEUROLOGY 6329 03:31:47,928 --> 03:31:50,364 BECAUSE THE RESULTS ARE 6330 03:31:50,431 --> 03:31:50,864 NEGATIVE. 6331 03:31:50,931 --> 03:31:52,332 OBVIOUSLY WE'RE NOT ACTING ON 6332 03:31:52,399 --> 03:31:53,367 THOSE COMMENTS. 6333 03:31:53,434 --> 03:31:55,936 WE ALWAYS SAY OUR ROLE IS NOT TO 6334 03:31:56,003 --> 03:31:58,272 ASK HOW MANY POINTS REVIEWERS 6335 03:31:58,338 --> 03:31:59,773 GAVE, IF ABOVE A CERTAIN GOAL WE 6336 03:31:59,840 --> 03:32:01,875 PUBLISH, IF NOT WE DON'T. 6337 03:32:01,942 --> 03:32:03,510 WE TAKE COMMENTS OF REVIEWERS 6338 03:32:03,577 --> 03:32:05,245 AND MAKE OUR OWN JUDGMENT BASED 6339 03:32:05,312 --> 03:32:05,779 ON THOSE. 6340 03:32:05,846 --> 03:32:08,649 SO WE TEND TO IGNORE THAT. 6341 03:32:08,715 --> 03:32:12,252 BUT THE POINT IS WE'RE HAVING 6342 03:32:12,319 --> 03:32:15,756 DIFFICULTY IDENTIFYING SOME NULL 6343 03:32:15,823 --> 03:32:16,523 STUDIES. 6344 03:32:16,590 --> 03:32:18,158 TWO, AUTHORS ARE RESISTANT TO 6345 03:32:18,225 --> 03:32:28,001 IDEA OF SUBMITTING NULL RESULTS, 6346 03:32:28,068 --> 03:32:28,902 REVIEWERS ARE RESISTANT. 6347 03:32:28,969 --> 03:32:30,437 MAYBE THAT'S SOMETHING WE CAN 6348 03:32:30,504 --> 03:32:31,405 TALK ABOUT IN THIS PANEL 6349 03:32:31,472 --> 03:32:39,713 SESSION. 6350 03:32:39,780 --> 03:32:42,349 >> HI, I'M LAURA SMITH, 6351 03:32:42,416 --> 03:32:45,185 NEUROSCIENCE DEPARTMENT, TEXAS 6352 03:32:45,252 --> 03:32:45,953 A & M UNIVERSITY, ASSISTANT 6353 03:32:46,019 --> 03:32:46,653 PROFESSOR. 6354 03:32:46,720 --> 03:32:48,689 AND SO I BRING A FEW 6355 03:32:48,755 --> 03:32:51,625 PERSPECTIVES, I THINK, TO THIS 6356 03:32:51,692 --> 03:32:52,659 PANEL. 6357 03:32:52,726 --> 03:32:53,727 THE FIRST IS, WELL, LET ME TELL 6358 03:32:53,794 --> 03:32:55,362 YOU ABOUT MY BACKGROUND. 6359 03:32:55,429 --> 03:32:57,564 I COME FROM A PSYCHOLOGY 6360 03:32:57,631 --> 03:33:00,534 DEPARTMENT, GRADUATE TRAINING 6361 03:33:00,601 --> 03:33:02,703 WAS IN BIOPSYCHOLOGY, BUT I WAS 6362 03:33:02,769 --> 03:33:05,639 DEEPLY STEEPED IN THE ETHOS OF 6363 03:33:05,706 --> 03:33:07,708 PROPER SCIENCE. 6364 03:33:07,774 --> 03:33:08,709 PROPER RESEARCH METHODS, 6365 03:33:08,775 --> 03:33:09,476 STATISTICAL METHODS, ET CETERA. 6366 03:33:09,543 --> 03:33:15,215 AS I WENT OUT INTO THE WORLD AND 6367 03:33:15,282 --> 03:33:17,151 TO REAL SCIENCE, WHAT I FOUND 6368 03:33:17,217 --> 03:33:19,319 WAS THAT THE THINGS I HAD 6369 03:33:19,386 --> 03:33:21,088 LEARNED, THE IDEALS I HAD 6370 03:33:21,155 --> 03:33:23,590 LEARNED ABOUT SCIENCE, WEREN'T 6371 03:33:23,657 --> 03:33:25,559 FITTING WHAT I WAS SEEING, WHAT 6372 03:33:25,626 --> 03:33:27,661 I WAS BEING TRAINED TO BE IN THE 6373 03:33:27,728 --> 03:33:29,163 REAL WORLD OF SCIENCE. 6374 03:33:29,229 --> 03:33:31,798 SO, HOW YOU BE A SUCCESSFUL 6375 03:33:31,865 --> 03:33:33,433 SCIENTIST WAS NOT LINING UP WITH 6376 03:33:33,500 --> 03:33:37,371 HOW YOU SHOULD BE A SCIENTIST. 6377 03:33:37,437 --> 03:33:39,339 SO, THAT CREATED A LOT OF 6378 03:33:39,406 --> 03:33:43,710 DISSONANCE FOR ME. 6379 03:33:43,777 --> 03:33:44,478 PERSPECTIVES I BRING, FIRST IN 6380 03:33:44,545 --> 03:33:47,981 THE FIELD I'M IN NOW, FRAGILE X 6381 03:33:48,048 --> 03:33:49,983 SYNDROME, WHICH IS A SMALL 6382 03:33:50,050 --> 03:33:51,051 CONDITION, A CONDITION THAT 6383 03:33:51,118 --> 03:33:52,386 AFFECTS A SMALL NUMBER OF 6384 03:33:52,452 --> 03:33:54,454 PEOPLE, A SMALL COMMUNITY OF 6385 03:33:54,521 --> 03:33:56,523 RESEARCHERS, BUT TIGHTLY KNIT 6386 03:33:56,590 --> 03:33:59,059 AND WITH SOME VERY INFLUENTIAL 6387 03:33:59,126 --> 03:33:59,860 LABS INVOLVED. 6388 03:33:59,927 --> 03:34:01,762 AND THIS GAVE ME AN INTERESTING 6389 03:34:01,828 --> 03:34:04,498 PERSPECTIVE BECAUSE WE HAD A 6390 03:34:04,565 --> 03:34:05,666 RECENT FAILURE OF CLINICAL 6391 03:34:05,732 --> 03:34:08,802 TRIALS, THAT WERE BASED ON VERY 6392 03:34:08,869 --> 03:34:11,405 STRONG PRE-CLINICAL WORK. 6393 03:34:11,471 --> 03:34:12,940 AND WE'RE A SMALL COMMUNITY BUT 6394 03:34:13,006 --> 03:34:14,041 SHOOK OUR COMMUNITY. 6395 03:34:14,107 --> 03:34:16,009 WE HAD FAITH IN THESE STUDIES. 6396 03:34:16,076 --> 03:34:17,844 WHEN THEY DIDN'T WORK OUT THIS 6397 03:34:17,911 --> 03:34:19,279 WAS A HUGE DISAPPOINTMENT, CLOSE 6398 03:34:19,346 --> 03:34:21,515 KNIT WITH PARENTS OF KIDS WHO 6399 03:34:21,582 --> 03:34:23,317 HAVE FRAGILE X, ET CETERA. 6400 03:34:23,383 --> 03:34:27,287 SO THIS REALLY HIT HOME. 6401 03:34:27,354 --> 03:34:29,623 IT ALSO MADE ME REFLECT ON THE 6402 03:34:29,690 --> 03:34:30,424 DISSONANCE THAT I HAD BEEN 6403 03:34:30,490 --> 03:34:32,292 FEELING ABOUT HOW TO BE A 6404 03:34:32,359 --> 03:34:33,060 SCIENTIST. 6405 03:34:33,126 --> 03:34:34,494 I WAS ASKED TO DO -- BECAUSE I'M 6406 03:34:34,561 --> 03:34:37,231 IN THIS FIELD WAS ASKED TO DO A 6407 03:34:37,297 --> 03:34:38,232 COMMENTARY A PUBLICATION BIAS. 6408 03:34:38,298 --> 03:34:39,099 I DID THAT. 6409 03:34:39,166 --> 03:34:41,501 AND I THINK THAT'S ONE OF THE 6410 03:34:41,568 --> 03:34:45,339 REASONS I'M HERE TODAY AT THIS 6411 03:34:45,405 --> 03:34:45,772 PANEL. 6412 03:34:45,839 --> 03:34:48,175 THE OTHER PERSPECTIVE I BRING IS 6413 03:34:48,242 --> 03:34:49,309 BEING AN ASSISTANT PROFESSOR 6414 03:34:49,376 --> 03:34:51,078 TRYING TO SURVIVE IN THE WORLD 6415 03:34:51,144 --> 03:34:55,315 OF RESEARCH THAT WE HAVE 6416 03:34:55,382 --> 03:34:56,250 CREATED. 6417 03:34:56,316 --> 03:34:57,517 SO, I DO PRE-CLINICAL WORK. 6418 03:34:57,584 --> 03:34:58,952 IF ANYONE FEELS LIKE THEY WANT 6419 03:34:59,019 --> 03:35:00,254 TO ASK QUESTIONS REGARDING THAT 6420 03:35:00,320 --> 03:35:03,624 AND HOW IT IS TO TRANSLATE YOUR 6421 03:35:03,690 --> 03:35:05,225 PRE-CLINICAL WORK TO CLINICAL 6422 03:35:05,292 --> 03:35:08,662 WORK, HOW HARD IT IS TO -- WHAT 6423 03:35:08,729 --> 03:35:10,864 IS INCENTIVE STRUCTURE WE'RE 6424 03:35:10,931 --> 03:35:11,632 FUNCTIONING UNDER, WHY IS IT 6425 03:35:11,698 --> 03:35:14,134 HARD TO BE HONEST IN YOUR 6426 03:35:14,201 --> 03:35:15,569 SCIENCE, I'M WILLING TO TALK 6427 03:35:15,636 --> 03:35:19,306 ABOUT ALL OF THOSE THINGS. 6428 03:35:19,373 --> 03:35:21,475 AND YEAH, I THINK THE LAST THING 6429 03:35:21,541 --> 03:35:23,877 I WANT TO SAY, SCIENTISTS HAVE 6430 03:35:23,944 --> 03:35:27,047 REALLY HEARD THE RULES OF THE 6431 03:35:27,114 --> 03:35:28,148 GAME LOUD AND CLEAR. 6432 03:35:28,215 --> 03:35:29,683 WE UNDERSTAND WHAT IT TAKES TO 6433 03:35:29,750 --> 03:35:31,184 BE A SUCCESSFUL SCIENTIST. 6434 03:35:31,251 --> 03:35:32,619 THIS IS REALLY A SYSTEMIC 6435 03:35:32,686 --> 03:35:33,720 PROBLEM. 6436 03:35:33,787 --> 03:35:35,589 IF WE DON'T CHANGE THE SYSTEM, 6437 03:35:35,656 --> 03:35:39,059 THEN WE'RE NOT GOING TO CHANGE 6438 03:35:39,126 --> 03:35:41,261 WHAT'S HAPPENING. 6439 03:35:41,328 --> 03:35:45,932 IT'S NOT JUST AS P-HACKING, IN 6440 03:35:45,999 --> 03:35:49,369 THE FIRST PANEL, THERE ARE OTHER 6441 03:35:49,436 --> 03:35:50,137 PROBLEMS, COLLECTING DATA, 6442 03:35:50,203 --> 03:35:51,204 ANALYZING IT. 6443 03:35:51,271 --> 03:35:52,539 THOSE ARE THE PERSPECTIVES I 6444 03:35:52,606 --> 03:35:52,839 BRING. 6445 03:35:52,906 --> 03:35:55,275 I'M EXCITED TO BE HERE. 6446 03:35:55,342 --> 03:35:58,512 SO I'LL CONTINUE OVER HERE. 6447 03:36:09,990 --> 03:36:12,125 >> LET'S OPEN THE CONVERSATION 6448 03:36:12,192 --> 03:36:13,460 FOR DISCUSSION, IF YOU HAVE 6449 03:36:13,527 --> 03:36:15,595 QUESTIONS. 6450 03:36:27,207 --> 03:36:29,943 >> IS IT ON NOW? 6451 03:36:30,010 --> 03:36:30,277 OKAY. 6452 03:36:30,344 --> 03:36:33,113 HI, I'M SAMUEL, I'M A TEAM 6453 03:36:33,180 --> 03:36:37,651 LEADER, SENIOR EDITOR OF BNC, 6454 03:36:37,718 --> 03:36:39,486 PART OF SPRINGER NATURE, I'M 6455 03:36:39,553 --> 03:36:41,254 ALSO JET LAGGED, BEAR WITH ME AS 6456 03:36:41,321 --> 03:36:45,625 I TRY TO COMPOSE A QUESTION. 6457 03:36:45,692 --> 03:36:48,595 THE BNC PROMOTES NEGATIVE DATA 6458 03:36:48,662 --> 03:36:50,230 AND NULL STUDIES BUT WE STRUGGLE 6459 03:36:50,297 --> 03:36:53,066 TO GET MANY SENT TO US. 6460 03:36:53,133 --> 03:36:54,634 FILLING MISSING DATA POINT 6461 03:36:54,701 --> 03:36:56,903 RAISED IN THE PREVIOUS, 4% OF 6462 03:36:56,970 --> 03:36:58,872 ALL OUR PAPERS. 6463 03:36:58,939 --> 03:37:01,308 AND I FIND EVEN THOUGH I PUSH 6464 03:37:01,375 --> 03:37:02,809 QUITE HARD ABOUT ONCE A YEAR FOR 6465 03:37:02,876 --> 03:37:04,344 IT TO HAPPEN WE GET CAUGHT IN 6466 03:37:04,411 --> 03:37:06,980 THE SAME CYCLE OVER AND OVER 6467 03:37:07,047 --> 03:37:09,583 AGAIN, SOME AUTHORS DO SUBMIT TO 6468 03:37:09,649 --> 03:37:12,452 US, NEARLY ALL GET REJECTED. 6469 03:37:12,519 --> 03:37:14,654 I THINK FOR TWO MAIN REASONS. 6470 03:37:14,721 --> 03:37:17,257 ONE, THERE'S A BIAS AGAINST 6471 03:37:17,324 --> 03:37:18,925 THEM, REVIEWERS ARE HARSHER ON 6472 03:37:18,992 --> 03:37:19,993 NULL OR NEGATIVE STUDIES, NOT 6473 03:37:20,060 --> 03:37:22,729 CERTAIN WHAT TO DO WITH THEM. 6474 03:37:22,796 --> 03:37:25,932 EVEN THOUGH MY JOURNAL IS A 6475 03:37:25,999 --> 03:37:27,567 SOUND SCIENCE ONLY JOURNAL. 6476 03:37:27,634 --> 03:37:29,836 WE DON'T JUDGE ON NOVELTY. 6477 03:37:29,903 --> 03:37:31,571 IT'S REALLY HARD TO DETERMINE 6478 03:37:31,638 --> 03:37:33,140 THE QUALITY OF A NEGATIVE OR 6479 03:37:33,206 --> 03:37:33,907 NULL STUDY. 6480 03:37:33,974 --> 03:37:36,743 IT'S EASY TO SAY WHY SOMETHING 6481 03:37:36,810 --> 03:37:38,612 IS AND WHY IT'S GOOD, MUCH 6482 03:37:38,678 --> 03:37:40,947 HARDER TO SAY WHY IT ISN'T. 6483 03:37:41,014 --> 03:37:42,949 THIS MEANS YOU REJECT A LOT, SAY 6484 03:37:43,016 --> 03:37:45,652 NO TO A LOT, A LOT OF AUTHORS 6485 03:37:45,719 --> 03:37:48,155 GET REJECTED AND DECIDE NOT TO 6486 03:37:48,221 --> 03:37:50,223 SUBMIT AGAIN, NOT TO REALLY TRY, 6487 03:37:50,290 --> 03:37:51,825 YOU GET EVEN LESS VISIBILITY, NO 6488 03:37:51,892 --> 03:37:53,827 ONE ELSE SEEMS TO BE PUBLISHING 6489 03:37:53,894 --> 03:37:55,128 ABOUT THIS, SO WHY SHOULD WE? 6490 03:37:55,195 --> 03:37:57,130 AND YOU GET CAUGHT IN A CYCLE 6491 03:37:57,197 --> 03:38:01,468 WHERE NOTHING HAPPENS OR GETS 6492 03:38:01,535 --> 03:38:02,002 BETTER. 6493 03:38:02,068 --> 03:38:03,336 I WONDER IF ANYONE HAS THOUGHTS 6494 03:38:03,403 --> 03:38:05,639 HOW WE CAN IMPROVE THIS AS 6495 03:38:05,705 --> 03:38:07,474 EDITORS AND PUBLISHERS. 6496 03:38:07,541 --> 03:38:09,976 THANK YOU. 6497 03:38:14,147 --> 03:38:15,315 >> I FEEL LIKE THAT'S SOMETHING 6498 03:38:15,382 --> 03:38:18,885 I'M HAPPY TO SPEAK TO YOU, BUT 6499 03:38:18,952 --> 03:38:20,954 WITH A DIFFERENT PERSPECTIVE. 6500 03:38:21,021 --> 03:38:22,823 PART HAS TO DO WITH EXPECTATIONS 6501 03:38:22,889 --> 03:38:25,559 IN THE COMMUNITY, RIGHT? 6502 03:38:25,625 --> 03:38:28,829 SO WE'RE EVALUATING RESEARCH 6503 03:38:28,895 --> 03:38:32,699 BASED ON HOW EXPERIMENT WAS 6504 03:38:32,766 --> 03:38:36,803 CONDUCTED, WHAT THE QUESTIONS 6505 03:38:36,870 --> 03:38:37,838 NOT RESULTS, REREGISTRATION 6506 03:38:37,904 --> 03:38:38,805 ADDRESSES THIS, BEFORE YOU EVEN 6507 03:38:38,872 --> 03:38:41,107 HAVE DATA, BEFORE YOU KNOW THE 6508 03:38:41,174 --> 03:38:42,609 DIRECTION OF THE RESULTS. 6509 03:38:42,676 --> 03:38:44,811 BUT I THINK A LOT OF THIS HAS TO 6510 03:38:44,878 --> 03:38:47,013 DO WITH BETTER TRAINING, 6511 03:38:47,080 --> 03:38:48,315 MANAGEMENT AND OVERSIGHT OF 6512 03:38:48,381 --> 03:38:51,351 ACADEMIC EDITORS WHO WILL BE 6513 03:38:51,418 --> 03:38:53,019 HANDLING THOSE PAPERS TO HELP 6514 03:38:53,086 --> 03:38:57,057 THEM, TO SUPPORT THEM IN 6515 03:38:57,123 --> 03:38:57,958 EVALUATING RESEARCH OBJECTIVELY. 6516 03:38:58,024 --> 03:38:59,793 SO THERE'S A BIG TRAINING AND 6517 03:38:59,860 --> 03:39:01,661 EDUCATION PIECE THAT COMES IN 6518 03:39:01,728 --> 03:39:05,799 BUT IT'S ESSENTIALLY ACADEMIC 6519 03:39:05,866 --> 03:39:07,667 EDITORS ARE RESEARCHERS. 6520 03:39:07,734 --> 03:39:09,870 EVERYONE IS AUTHOR, RESEARCHER, 6521 03:39:09,936 --> 03:39:10,103 EDITOR. 6522 03:39:10,170 --> 03:39:11,605 SO I THINK, YOU KNOW, HAVING 6523 03:39:11,671 --> 03:39:13,373 THAT CYCLE OF PUBLISHING 6524 03:39:13,440 --> 03:39:15,675 NEGATIVE RESULTS, HAVING A FAIR 6525 03:39:15,742 --> 03:39:17,644 REVIEW PROCESS, SEEING IT IN 6526 03:39:17,711 --> 03:39:20,046 PRINT AND THEN APPLYING YOUR OWN 6527 03:39:20,113 --> 03:39:21,047 TREATMENT THROUGH THAT CYCLE 6528 03:39:21,114 --> 03:39:23,550 WHEN YOU'RE REVIEWING AND 6529 03:39:23,617 --> 03:39:24,985 HANDLING PAPERS IS ONE WAY TO 6530 03:39:25,051 --> 03:39:28,321 BEGIN TO ADDRESS THIS. 6531 03:39:31,625 --> 03:39:35,195 >> THIS MAY BE A CONTROVERSIAL 6532 03:39:35,262 --> 03:39:35,428 COMMENT. 6533 03:39:35,495 --> 03:39:37,564 I DON'T THINK YOU'RE EVER GOING 6534 03:39:37,631 --> 03:39:41,601 TO FIX PUBLICATION BIAS WITH 6535 03:39:41,668 --> 03:39:42,903 INITIATIVES THAT CALL FOR 6536 03:39:42,969 --> 03:39:43,403 NEGATIVE RESULTS. 6537 03:39:43,470 --> 03:39:45,005 YOU'RE NEVER GOING TO GET IT. 6538 03:39:45,071 --> 03:39:47,541 TOO MANY SELECTIVE PRESSURES ON 6539 03:39:47,607 --> 03:39:49,276 EVERYONE TO REJECT, TURN THEM 6540 03:39:49,342 --> 03:39:51,244 POSITIVE, IT'S NEVER GOING TO 6541 03:39:51,311 --> 03:39:51,578 WORK. 6542 03:39:51,645 --> 03:39:54,080 IT'S JUST A BAD IDEA. 6543 03:39:54,147 --> 03:39:57,584 IT'S MUCH BETTER TO BLIND YOUR 6544 03:39:57,651 --> 03:40:02,923 REVIEW SYSTEM RESULTS. 6545 03:40:02,989 --> 03:40:04,457 WHETHER REGISTERED REPORT OR 6546 03:40:04,524 --> 03:40:06,560 SIMPLY PEER REVIEWING REGULAR 6547 03:40:06,626 --> 03:40:07,594 ARTICLES WITH RESULTS REDACTED 6548 03:40:07,661 --> 03:40:10,397 UNTIL YOU'VE DONE A PROPER 6549 03:40:10,463 --> 03:40:12,465 EVALUATION OF POLICY. 6550 03:40:12,532 --> 03:40:14,668 I DON'T THINK NULL RESULTS 6551 03:40:14,734 --> 03:40:16,536 INITIATIVES ARE GOING TO FIX THE 6552 03:40:16,603 --> 03:40:18,138 PROBLEM. 6553 03:40:18,204 --> 03:40:19,239 THE PROBLEM IS RESULTS-BASED 6554 03:40:19,306 --> 03:40:19,773 EVALUATION. 6555 03:40:19,839 --> 03:40:23,276 YOU DON'T FIX THAT BY DOING MORE 6556 03:40:23,343 --> 03:40:24,010 RESULTS-BASED EVALUATION. 6557 03:40:24,077 --> 03:40:26,012 >> I'M GLAD YOU SAID THAT. 6558 03:40:26,079 --> 03:40:30,083 ANOTHER JOURNAL IN THE BMC, WE 6559 03:40:30,150 --> 03:40:31,718 TRIED NO RESULTS REVIEW, IT HAS 6560 03:40:31,785 --> 03:40:32,986 CHALLENGES IN THE MIDDLE OF 6561 03:40:33,053 --> 03:40:34,888 COVID, NOT A LOT OF THINGS 6562 03:40:34,955 --> 03:40:36,656 WORKED AS YOU WANT DURING COVID. 6563 03:40:36,723 --> 03:40:38,091 BUT WE HAD ALMOST EXACTLY THE 6564 03:40:38,158 --> 03:40:40,160 SAME PROBLEMS WHICH IS REVIEWERS 6565 03:40:40,226 --> 03:40:43,663 STRUGGLED TO KIND OF GET TO GRIP 6566 03:40:43,730 --> 03:40:45,165 HOW TO ASSESS A MANUSCRIPT OF 6567 03:40:45,231 --> 03:40:45,865 BAD RESULTS. 6568 03:40:45,932 --> 03:40:47,801 THERE WAS A LOT OF BIAS. 6569 03:40:47,867 --> 03:40:49,369 AGAIN, I THINK TO PICK UP ON 6570 03:40:49,436 --> 03:40:52,872 WHAT YOU SAID, EMILY, IT'S A LOT 6571 03:40:52,939 --> 03:40:56,042 OF LEARNING REQUIRED BEFORE 6572 03:40:56,109 --> 03:40:58,345 EVERYONE BEFORE AN INITIATIVE 6573 03:40:58,411 --> 03:41:01,114 WILL WORK. 6574 03:41:01,181 --> 03:41:02,415 ANYTHING NEW INSTANTLY GETS 6575 03:41:02,482 --> 03:41:04,150 PUSHED AGAINST, HOW BEST TO 6576 03:41:04,217 --> 03:41:05,251 ADDRESS THAT ACROSS PUBLISHERS 6577 03:41:05,318 --> 03:41:07,754 AND AUTHORS AND FUNDERS AND 6578 03:41:07,821 --> 03:41:08,622 REGULATORS IS THE REASON WE'RE 6579 03:41:08,688 --> 03:41:13,126 HERE BUT THERE IS NO SIMPLE 6580 03:41:13,193 --> 03:41:14,327 SOLUTION ALTHOUGH I LIKE THE 6581 03:41:14,394 --> 03:41:16,396 SOUND OF YOURS BETTER THAN MINE 6582 03:41:16,463 --> 03:41:17,497 AT THE MOMENT. 6583 03:41:17,564 --> 03:41:21,534 >> I HAVE VERY DIFFERENT 6584 03:41:21,601 --> 03:41:21,868 EXPERIENCE. 6585 03:41:21,935 --> 03:41:22,502 WE INTRODUCED PUBLISHING 6586 03:41:22,569 --> 03:41:24,270 NEGATIVE RESULTS FROM THE VERY 6587 03:41:24,337 --> 03:41:26,473 BEGINNING SO WE PUBLISHED QUITE 6588 03:41:26,539 --> 03:41:28,108 A FEW NOW. 6589 03:41:28,174 --> 03:41:31,945 AND I NEVER SAW A PROBLEM LIKE 6590 03:41:32,012 --> 03:41:32,278 THAT. 6591 03:41:32,345 --> 03:41:35,882 SO, THE WAY WE DO IT REALLY 6592 03:41:35,949 --> 03:41:39,719 DIFFERENT SO WE HAVE A 6593 03:41:39,786 --> 03:41:40,954 CONSULTATION PROCESS. 6594 03:41:41,021 --> 03:41:46,926 THE REVIEWERS KNOW THEIR NAMES 6595 03:41:46,993 --> 03:41:48,261 READ EACH OTHER'S REVIEWS TO 6596 03:41:48,328 --> 03:41:49,796 ENSURE THAT ONE MESSAGE IS 6597 03:41:49,863 --> 03:41:51,097 CONVEYED TO THE OTHERS SO THEY 6598 03:41:51,164 --> 03:41:54,934 ALL TALK TO EACH OTHER. 6599 03:41:55,001 --> 03:41:57,337 IF THERE'S A MISCONCEPTION FROM 6600 03:41:57,404 --> 03:41:58,772 ONE REVIEWER THAT SHOULDN'T 6601 03:41:58,838 --> 03:41:59,472 PUBLISH NEGATIVE RESULTS, 6602 03:41:59,539 --> 03:42:01,374 DOESN'T GO FURTHER THAN THAT, 6603 03:42:01,441 --> 03:42:04,878 IT'S THE JOB OF THE EDITOR TO 6604 03:42:04,944 --> 03:42:07,313 ASK THEM IS THIS A MESSAGE THAT 6605 03:42:07,380 --> 03:42:09,382 IS USEFUL TO THE COMMUNITY, 6606 03:42:09,449 --> 03:42:12,018 THAT'S THE ONLY QUESTION YOU'RE 6607 03:42:12,085 --> 03:42:13,553 BEING ASKED, IF THE EXPERIMENTS 6608 03:42:13,620 --> 03:42:15,388 ARE DONE PROPERLY DOESN'T MATTER 6609 03:42:15,455 --> 03:42:17,724 WHAT THE RESULT IS EXACTLY, WHAT 6610 03:42:17,791 --> 03:42:21,094 EMILY SAID, BUT HAVING THIS 6611 03:42:21,161 --> 03:42:26,900 CONSULTATION PROCESS BECAUSE WE 6612 03:42:26,966 --> 03:42:27,500 ALSO PUBLISH REVIEWS. 6613 03:42:27,567 --> 03:42:28,868 I DON'T KNOW WHETHER IT'S 6614 03:42:28,935 --> 03:42:32,038 WORKING IN EUROPE BECAUSE OF 6615 03:42:32,105 --> 03:42:33,707 THAT, I DON'T KNOW, BUT THAT'S 6616 03:42:33,773 --> 03:42:35,108 THE MAIN DIFFERENCE I SEE WITH 6617 03:42:35,175 --> 03:42:36,443 OTHER JOURNALS BECAUSE WE'RE 6618 03:42:36,509 --> 03:42:39,145 FULLY TRANSPARENT. 6619 03:42:55,128 --> 03:43:04,370 >> I HAVE A SPECIFIC QUESTION. 6620 03:43:04,437 --> 03:43:08,975 [ GARBLED AUDIO ] YOU SPOKE 6621 03:43:09,042 --> 03:43:09,843 ABOUT PRE-PRINT COMMUNITY 6622 03:43:09,909 --> 03:43:10,276 REVIEW. 6623 03:43:10,343 --> 03:43:12,912 COULD YOU PROVIDE MORE DETAIL ON 6624 03:43:12,979 --> 03:43:15,749 THAT BECAUSE THAT SOUNDED 6625 03:43:15,815 --> 03:43:17,484 PROVOCATIVE. 6626 03:43:17,550 --> 03:43:19,686 >> SURE, YEAH. 6627 03:43:19,753 --> 03:43:20,520 ABSOLUTELY. 6628 03:43:20,587 --> 03:43:22,422 THE COMMUNITY AND REGISTERED 6629 03:43:22,489 --> 03:43:23,957 REPORTS INITIATIVE IS ONE 6630 03:43:24,023 --> 03:43:26,493 COMMUNITY WITHIN THE BROADER PCI 6631 03:43:26,559 --> 03:43:26,793 FAMILY. 6632 03:43:26,860 --> 03:43:32,265 SO PEER COMMUNITY IS A 6633 03:43:32,332 --> 03:43:33,466 PRE-PRINT-BASED NON-COMMERCIAL 6634 03:43:33,533 --> 03:43:34,567 PLATFORM FOR REVIEWING 6635 03:43:34,634 --> 03:43:36,102 PRE-PRINTS SO IT'S EXISTED FOR A 6636 03:43:36,169 --> 03:43:38,071 FEW YEARS NOW IN DIFFERENT 6637 03:43:38,138 --> 03:43:39,139 FIELDS. 6638 03:43:39,205 --> 03:43:42,041 THE IDEA IS THAT RATHER THAN 6639 03:43:42,108 --> 03:43:42,909 AUTHORS GETTING PEER-REVIEWED 6640 03:43:42,976 --> 03:43:44,978 JOURNALS WHY CAN'T WE DO PEER 6641 03:43:45,044 --> 03:43:48,381 REVIEW OF PRE-PRINTS, THIS IS AN 6642 03:43:48,448 --> 03:43:49,149 IMPORTANT -- WE'VE ALWAYS DONE 6643 03:43:49,215 --> 03:43:50,250 PEER REVIEW OURSELVES AS AN 6644 03:43:50,316 --> 03:43:51,217 ACADEMIC COMMUNITY. 6645 03:43:51,284 --> 03:43:53,853 WE MAKE UP ALL THE REVIEWERS, 6646 03:43:53,920 --> 03:43:57,357 VIRTUALLY ALL THE EDITORS AT 6647 03:43:57,423 --> 03:43:58,458 MOST JOURNALS. 6648 03:43:58,525 --> 03:44:00,426 WE'RE PERFECTLY CAPABLE OF DOING 6649 03:44:00,493 --> 03:44:00,860 THIS OURSELVES. 6650 03:44:00,927 --> 03:44:02,061 AND THEN THAT GIVES THE 6651 03:44:02,128 --> 03:44:04,898 COMMUNITY A LOT MORE CONTROL 6652 03:44:04,964 --> 03:44:07,100 OVER THE ARTICLES, WHAT PROBLEMS 6653 03:44:07,167 --> 03:44:08,334 WITH OPEN ACCESS BECAUSE 6654 03:44:08,401 --> 03:44:09,402 PRE-PRINTS HAVE BEEN PUBLICLY 6655 03:44:09,469 --> 03:44:13,773 AVAILABLE FOR EVERYBODY TO READ, 6656 03:44:13,840 --> 03:44:15,542 DEMOCRATIZE SCIENCE, IT'S CHEAP 6657 03:44:15,608 --> 03:44:16,976 AND COST EFFECTIVE. 6658 03:44:17,043 --> 03:44:19,045 WE CREATED A SEPARATE PCI FOR 6659 03:44:19,112 --> 03:44:20,580 REGISTERED REPORTS SPECIFICALLY 6660 03:44:20,647 --> 03:44:24,384 BECAUSE WE WANTED TO CREATE THE 6661 03:44:24,450 --> 03:44:26,719 SAME COMMUNITY MECHANISM FOR 6662 03:44:26,786 --> 03:44:27,487 REGISTERED REPORTS, SO 6663 03:44:27,554 --> 03:44:29,856 REGISTERED REPORTS EVALUATION 6664 03:44:29,923 --> 03:44:31,291 DONE ABOVE AND BEYOND JOURNALS 6665 03:44:31,357 --> 03:44:35,128 BY THE COMMUNITY, BY EXPERTS IN 6666 03:44:35,195 --> 03:44:41,167 REGISTERED REPORTS AT THE 6667 03:44:41,234 --> 03:44:42,468 PRE-PRINT REVIEWS, MADE PUBLIC 6668 03:44:42,535 --> 03:44:43,236 IN CASE OF POSITIVE 6669 03:44:43,303 --> 03:44:43,837 RECOMMENDATION. 6670 03:44:43,903 --> 03:44:48,474 WE HAVE A SET OF I THINK 36 6671 03:44:48,541 --> 03:44:50,777 PCI-FRIENDLY JOURNALS THAT 6672 03:44:50,844 --> 03:44:55,148 COMMIT TO ACCEPTING THE REVIEWS 6673 03:44:55,215 --> 03:44:55,815 AND RECOMMENDATIONS WITHOUT 6674 03:44:55,882 --> 03:44:58,551 FURTHER PEER REVIEW. 6675 03:44:58,618 --> 03:45:00,320 THESE JOURNALS ESSENTIALLY ALLOW 6676 03:45:00,386 --> 03:45:03,823 THE PCI REVIEW PROCESS TO 6677 03:45:03,890 --> 03:45:04,958 REPLACE THE REGISTERED REPORTS 6678 03:45:05,024 --> 03:45:06,326 THAT WOULD HAVE HAPPENED HAD 6679 03:45:06,392 --> 03:45:07,527 AUTHORS GONE TO THAT JOURNAL. 6680 03:45:07,594 --> 03:45:08,628 NOW, THIS IS REALLY GOOD FOR 6681 03:45:08,695 --> 03:45:08,995 EVERYBODY. 6682 03:45:09,062 --> 03:45:11,030 THIS IS GOOD FOR THE COMMUNITY 6683 03:45:11,097 --> 03:45:12,899 BECAUSE IT GIVES AUTHORS THE 6684 03:45:12,966 --> 03:45:14,000 OPPORTUNITY TO CHOOSE THEIR 6685 03:45:14,067 --> 03:45:16,603 JOURNAL BASED ON THE AUDIENCE 6686 03:45:16,669 --> 03:45:18,371 THEY ARE LOOKING FOR, BASED ON 6687 03:45:18,438 --> 03:45:19,305 ANY OTHER PARTICULAR FACTOR THEY 6688 03:45:19,372 --> 03:45:21,875 MAY HAVE, IT GIVES THEM A LOT OF 6689 03:45:21,941 --> 03:45:24,043 CONTROL OR POWER OVER THEIR 6690 03:45:24,110 --> 03:45:25,111 ARTICLE. 6691 03:45:25,178 --> 03:45:27,213 IT MAKES THE -- CENTRALIZED THE 6692 03:45:27,280 --> 03:45:29,649 PROCESS IN A VERY EFFICIENT WAY, 6693 03:45:29,716 --> 03:45:33,019 ONE REVIEW PROCESS CAN OPEN THE 6694 03:45:33,086 --> 03:45:34,687 DOOR TO MULTIPLE JOURNALS. 6695 03:45:34,754 --> 03:45:36,222 BECAUSE WE CONTROL THE PLATFORM, 6696 03:45:36,289 --> 03:45:39,692 WE CAN INNOVATE IN WAYS THAT 6697 03:45:39,759 --> 03:45:41,261 PUBLISHERS ARE UNABLE OR 6698 03:45:41,327 --> 03:45:43,529 UNWILLING TO DO. 6699 03:45:43,596 --> 03:45:45,899 WE IMPLEMENTED A SCHEDULED 6700 03:45:45,965 --> 03:45:47,600 REVIEW PROCESS, STAGE 1 REVIEW 6701 03:45:47,667 --> 03:45:51,437 TIME, A BARRIER FOR MANY AUTHORS 6702 03:45:51,504 --> 03:45:54,574 WAITING TO GO PRE-STUDY 6703 03:45:54,641 --> 03:46:00,380 EVALUATION, WE CAN PLAN AHEAD, A 6704 03:46:00,446 --> 03:46:03,449 PERIOD OF SEVEN DAYS, ONE TO 6705 03:46:03,516 --> 03:46:05,852 TWO MONTHS FROM SUBMISSION, 6706 03:46:05,919 --> 03:46:07,620 ACCELERATED TO A MATTER OF A FEW 6707 03:46:07,687 --> 03:46:09,022 WEEKS BY PLANNING THINGS IN 6708 03:46:09,088 --> 03:46:09,255 ADVANCE. 6709 03:46:09,322 --> 03:46:10,890 YOU THINK IT'S A SIMPLE 6710 03:46:10,957 --> 03:46:13,059 INNOVATION BUT ONE NO PUBLISHER 6711 03:46:13,126 --> 03:46:21,401 HAS ACCOMPLISHED. 6712 03:46:21,467 --> 03:46:22,201 WE'VE IMPLEMENTED PROGRAMMATIC 6713 03:46:22,268 --> 03:46:23,603 REPORTS, BASED ON MULTIPLE STAGE 6714 03:46:23,670 --> 03:46:24,570 2 REGISTERED REPORTS WILL BE 6715 03:46:24,637 --> 03:46:26,673 ACCEPTED IN ADVANCE. 6716 03:46:26,739 --> 03:46:28,207 SO ONE PROGRAMMATIC LINE OF 6717 03:46:28,274 --> 03:46:30,076 RESEARCH CAN LEAD TO MULTIPLE 6718 03:46:30,143 --> 03:46:34,013 REGISTERED REPORTS THROUGH ONE 6719 03:46:34,080 --> 03:46:36,049 CENTRALIZED REVIEW PROCESS. 6720 03:46:36,115 --> 03:46:37,550 IT'S COMMUNITY DRIVEN, FREE FOR 6721 03:46:37,617 --> 03:46:40,820 EVERYBODY, FREE TO READ, IT'S 6722 03:46:40,887 --> 03:46:43,122 OPEN. 6723 03:46:43,189 --> 03:46:44,123 AND IT'S INNOVATIVE, SOMETHING 6724 03:46:44,190 --> 03:46:45,591 WE CAN ALL SUPPORT IN DIFFERENT 6725 03:46:45,658 --> 03:46:47,493 WAYS, WHETHER YOU'RE A JOURNAL 6726 03:46:47,560 --> 03:46:48,594 EDITOR, JOIN AS RECOMMENDER, 6727 03:46:48,661 --> 03:46:55,601 WHETHER YOU'RE AN AUTHOR OR 6728 03:46:55,668 --> 03:46:56,169 REVIEW. 6729 03:46:56,235 --> 03:46:57,003 >> HELLO. 6730 03:46:57,070 --> 03:47:00,406 I'M A PROGRAM MANAGER AT 6731 03:47:00,473 --> 03:47:01,207 GLADSTONE. 6732 03:47:01,274 --> 03:47:04,911 NOT LONG BEFORE UP TILL LAST 6733 03:47:04,978 --> 03:47:07,747 YEAR WAS A POSTDOC, KIF'S STORY 6734 03:47:07,814 --> 03:47:08,114 RESONATED. 6735 03:47:08,181 --> 03:47:09,382 BEFORE I LEFT MY POSTDOC I 6736 03:47:09,449 --> 03:47:18,024 PUBLISHED A STORY ON NEGATIVE 6737 03:47:18,091 --> 03:47:20,093 RESULTS, MY VISOR WAS SUPPORTIVE 6738 03:47:20,159 --> 03:47:21,761 BUT NOT SURE IF EVERYONE IS 6739 03:47:21,828 --> 03:47:23,629 SUPPORTIVE OF HAVING STUDENTS OR 6740 03:47:23,696 --> 03:47:26,399 POSTDOCS OR WHOEVER PUBLIC 6741 03:47:26,466 --> 03:47:28,234 NEGATIVE RESULTS AND I THINK WE 6742 03:47:28,301 --> 03:47:30,837 HAVE ALSO SEEN THERE WAS RECENT 6743 03:47:30,903 --> 03:47:34,807 DATA THAT THE LENGTH OF POSTDOC 6744 03:47:34,874 --> 03:47:37,243 OR GRADUATE TIME INCREASED 6745 03:47:37,310 --> 03:47:38,511 SUBSTANTIALLY IN THE PAST YEARS. 6746 03:47:38,578 --> 03:47:40,480 THEY ARE PRODUCING DATA ABOUT 6747 03:47:40,546 --> 03:47:43,016 MAYBE IT'S NOT COALESCING INTO 6748 03:47:43,082 --> 03:47:49,288 HIGH IMPACT STORY OR WHATEVER. 6749 03:47:49,355 --> 03:47:51,324 COMING FROM JOURNALS OR REVIEW 6750 03:47:51,391 --> 03:47:54,727 SYSTEM, I WANT TO UNDERSTAND NOT 6751 03:47:54,794 --> 03:47:58,131 BIAS, IN TERMS OF PUBLISHING 6752 03:47:58,197 --> 03:47:59,432 NEGATIVE DATA LIKE PUBLICATION 6753 03:47:59,499 --> 03:48:02,702 BIAS IN GENERAL WE ALWAYS SORT 6754 03:48:02,769 --> 03:48:05,438 OF SAY LIKE SO-AND-SO ET AL. 6755 03:48:05,505 --> 03:48:06,773 PUBLISHED THIS, IT'S PEOPLE 6756 03:48:06,839 --> 03:48:09,275 FIRST, MAYBE SCIENCE LATER. 6757 03:48:09,342 --> 03:48:13,446 BUT LIKE CHANGING DYNAMIC TO 6758 03:48:13,513 --> 03:48:15,481 TALK ABOUT SCIENCE BEING IN 6759 03:48:15,548 --> 03:48:16,516 PERSON, I DON'T KNOW HOW MUCH IT 6760 03:48:16,582 --> 03:48:21,120 IS POSSIBLE TO BE IN PERSON BUT 6761 03:48:21,187 --> 03:48:22,422 MAYBE IN REGARDS TO SCIENTIFIC 6762 03:48:22,488 --> 03:48:24,190 DATA AND HOW IN TERMS OF THE 6763 03:48:24,257 --> 03:48:25,491 JOURNALS OR REVIEWERS WHICH ARE 6764 03:48:25,558 --> 03:48:26,759 ASSOCIATED WITH JOURNAL HOW CAN 6765 03:48:26,826 --> 03:48:30,430 THAT BE LIKE IS IT POSSIBLE TO 6766 03:48:30,496 --> 03:48:37,537 DO SCIENCE FIRST, PEOPLE LATER, 6767 03:48:37,603 --> 03:48:47,580 SOMETHING ON THOSE LINES? 6768 03:48:47,647 --> 03:48:49,549 >> TRICKY QUESTION GETTING AT 6769 03:48:49,615 --> 03:48:50,983 CREDIT, RECOGNIZING HOW WE 6770 03:48:51,050 --> 03:48:54,520 REWARD THE WORK IT'S VERY MUCH A 6771 03:48:54,587 --> 03:48:55,688 PERSON-DRIVEN THING. 6772 03:48:55,755 --> 03:48:58,791 THERE ARE WAYS WE CAN EXPLORE 6773 03:48:58,858 --> 03:49:05,465 GETTING CREDIT FOR DIFFERENT 6774 03:49:05,531 --> 03:49:08,301 PARTS OF THE RESEARCH QUESTION 6775 03:49:08,367 --> 03:49:10,503 BUT TRYING TO TEASE OUT CREDIT 6776 03:49:10,570 --> 03:49:13,773 IS A DIFFICULT PROPOSITION. 6777 03:49:13,840 --> 03:49:15,074 >> YEAH, I AGREE. 6778 03:49:15,141 --> 03:49:15,942 THAT'S THE CHALLENGE. 6779 03:49:16,008 --> 03:49:20,546 REALLY YOU SHOULD HAVE JUST THE 6780 03:49:20,613 --> 03:49:20,947 RESULT. 6781 03:49:21,013 --> 03:49:25,251 IN A WAY, YOU DON'T WANT TO 6782 03:49:25,318 --> 03:49:26,686 DOWNPLAY THE CONTRIBUTION OF 6783 03:49:26,752 --> 03:49:27,186 INDIVIDUALS. 6784 03:49:27,253 --> 03:49:30,623 >> I MEANT MORE TOWARDS THE SORT 6785 03:49:30,690 --> 03:49:34,026 OF LIKE PROCESS OF REVIEW LIKE 6786 03:49:34,093 --> 03:49:35,128 JOURNALS, GRANTS OR WHATEVER, IF 6787 03:49:35,194 --> 03:49:37,096 YOU KNOW THE PERSON WHO HAS THE 6788 03:49:37,163 --> 03:49:41,000 PAPER LIKE IN TERMS OF KIF THAT 6789 03:49:41,067 --> 03:49:42,568 PAPER IS MORE CITED, NATURE 6790 03:49:42,635 --> 03:49:45,938 NEUROSCIENCE MAYBE COMING FROM 6791 03:49:46,005 --> 03:49:48,007 YOU HAVE THE STATUS, IN TERMS OF 6792 03:49:48,074 --> 03:49:51,511 REVIEW PROCESS IN GENERAL DOES 6793 03:49:51,577 --> 03:49:53,379 THAT MATTER WHEN SOMEBODY IS 6794 03:49:53,446 --> 03:49:54,814 TRYING TO NEGATIVE RESULTS 6795 03:49:54,881 --> 03:49:57,650 CONTRADICTING LIKE NOT AS 6796 03:49:57,717 --> 03:49:58,718 SUPPORTIVE OF ANOTHER REPUTED 6797 03:49:58,784 --> 03:50:03,022 SCIENTIST IN TERMS OF THE BIAS 6798 03:50:03,089 --> 03:50:03,890 IN REVIEWING? 6799 03:50:03,956 --> 03:50:08,728 >> YEAH, THERE'S A BIAS, RIGHT? 6800 03:50:08,794 --> 03:50:10,663 WE HAD THAT IN THE LAST SESSION, 6801 03:50:10,730 --> 03:50:11,797 WHERE YOU'RE RECOGNIZING IT 6802 03:50:11,864 --> 03:50:15,301 TAKES A LOT OF EVIDENCE TO GO 6803 03:50:15,368 --> 03:50:16,402 AGAINST THE STATUS QUO. 6804 03:50:16,469 --> 03:50:17,937 AND THERE'S A LITTLE -- THERE'S 6805 03:50:18,004 --> 03:50:21,407 CERTAINLY A BIAS IF YOU'RE A 6806 03:50:21,474 --> 03:50:26,579 POSTDOC AND HAVE A FAMOUS PERSON 6807 03:50:26,646 --> 03:50:28,648 THERE IS WRONG, IT'S VERY 6808 03:50:28,714 --> 03:50:28,915 DIFFICULT. 6809 03:50:28,981 --> 03:50:30,816 THERE'S SOME FEAR YOU MAY HAVE, 6810 03:50:30,883 --> 03:50:35,321 IT MAY BE THE FAMOUS PERSON 6811 03:50:35,388 --> 03:50:37,490 REVIEWING THE PAPER, THAT TOPIC 6812 03:50:37,557 --> 03:50:46,365 AREA, SO THERE'S SOME BIAS 6813 03:50:46,432 --> 03:50:47,567 THERE. 6814 03:50:47,633 --> 03:50:50,203 >> THAT DIDN'T HELP. 6815 03:50:50,269 --> 03:50:51,737 YEAH, EXACTLY. 6816 03:50:51,804 --> 03:50:56,075 THE THING IS IT'S MORE -- IT'S 6817 03:50:56,142 --> 03:50:56,409 DISHEARTENING. 6818 03:50:56,475 --> 03:51:00,446 SINCE THE PUBLICATION OF OUR 6819 03:51:00,513 --> 03:51:02,215 PAPER IN 2021, LOOKING FOR 6820 03:51:02,281 --> 03:51:05,618 CITATIONS OF THE ORIGINAL PAPER, 6821 03:51:05,685 --> 03:51:08,321 HAD MORE THAN 40 CITATIONS, ALL 6822 03:51:08,387 --> 03:51:11,524 IN THE CONTEXT OF WHAT THEY HAVE 6823 03:51:11,591 --> 03:51:16,462 SHOWN IS RIGHT. 6824 03:51:16,529 --> 03:51:20,166 THAT MEANS mRNA CAN'T UNDERGO 6825 03:51:20,233 --> 03:51:20,700 SPLICING. 6826 03:51:20,766 --> 03:51:25,638 OUR PAPER CITED SIX TIMES, AND 6827 03:51:25,705 --> 03:51:29,875 THREE IS COMPLETELY DIFFERENT 6828 03:51:29,942 --> 03:51:30,176 CONTEXT. 6829 03:51:30,243 --> 03:51:31,344 PAPER WAS CITED FOR SOMETHING 6830 03:51:31,410 --> 03:51:32,411 ELSE BASICALLY. 6831 03:51:32,478 --> 03:51:36,349 IN MY OPINION I THINK THE VENUE 6832 03:51:36,415 --> 03:51:38,384 IDEALLY YOU WANT TO CHALLENGE 6833 03:51:38,451 --> 03:51:45,992 SOME STUDIES WHERE IT IS 6834 03:51:46,058 --> 03:51:48,160 ORIGINALLY PUBLISHED, WHICH 6835 03:51:48,227 --> 03:51:50,363 MISERABLY FAILED, WE WERE LUCKY 6836 03:51:50,429 --> 03:51:52,999 NOT TO RESULT IN THAT GROUP, 6837 03:51:53,065 --> 03:51:55,835 DIDN'T WASTE TIME. 6838 03:51:55,901 --> 03:52:01,307 I THINK IF WE PUBLISHED IN THAT 6839 03:52:01,374 --> 03:52:02,942 EXACT JOURNAL PROBABLY WE WOULD 6840 03:52:03,009 --> 03:52:05,911 GET SOME TENSION BUT THERE'S NO 6841 03:52:05,978 --> 03:52:07,980 BARRIER IN PUBLISHING NEGATIVE 6842 03:52:08,047 --> 03:52:09,515 RESULTS IN A WAY THAT HUNDREDS 6843 03:52:09,582 --> 03:52:11,384 OF VENUES THAT YOU CAN PUT IT 6844 03:52:11,450 --> 03:52:12,451 IN, RIGHT? 6845 03:52:12,518 --> 03:52:18,391 YOU CAN PUBLISH, YOU CAN ALSO 6846 03:52:18,457 --> 03:52:21,227 PUT IT IN -- THE PROBLEM IS NOT 6847 03:52:21,294 --> 03:52:25,398 PUBLISHING, IT'S IN THE 6848 03:52:25,464 --> 03:52:25,731 VISIBILITY. 6849 03:52:25,798 --> 03:52:26,932 NO MATTER WHERE IT IS COMING 6850 03:52:26,999 --> 03:52:30,069 FROM, EVEN A BIG LAB OR VISIBLE 6851 03:52:30,136 --> 03:52:31,203 LAB. 6852 03:52:31,270 --> 03:52:32,838 AND TALKING ABOUT CRITICS, I 6853 03:52:32,905 --> 03:52:36,542 ALSO KNOW FROM OTHER THINGS LIKE 6854 03:52:36,609 --> 03:52:38,844 ASTRONOMY AND PHYSICS, THEY 6855 03:52:38,911 --> 03:52:44,984 DON'T CARE ABOUT THE AUTHOR 6856 03:52:45,051 --> 03:52:48,487 ORDER, IT'S ALL ABOUT WHAT THE 6857 03:52:48,554 --> 03:52:49,188 SCIENCE THEY CONTRIBUTE TO. 6858 03:52:49,255 --> 03:52:51,223 I DON'T KNOW HOW MANY HUNDRED 6859 03:52:51,290 --> 03:52:53,492 YEARS IT WILL TAKE FOR US. 6860 03:52:53,559 --> 03:52:56,362 THAT I THINK IN THAT UTOPIAN 6861 03:52:56,429 --> 03:52:58,197 WORLD PEOPLE WILL SAY THE 6862 03:52:58,264 --> 03:53:01,534 SCIENCE FIRST RATHER THAN WHO 6863 03:53:01,600 --> 03:53:04,103 DID IT. 6864 03:53:04,170 --> 03:53:04,870 >> THANK YOU. 6865 03:53:04,937 --> 03:53:09,375 >> YOU ASKED MANY QUESTIONS BUT 6866 03:53:09,442 --> 03:53:13,412 IN SOME FIELDS, THEY WANT TO 6867 03:53:13,479 --> 03:53:14,847 ACKNOWLEDGE THE PRODUCTION OF 6868 03:53:14,914 --> 03:53:17,049 DATA, AND THEY ARE TRYING TO 6869 03:53:17,116 --> 03:53:23,289 INVENT A WAY BUT THAT'S BEYOND 6870 03:53:23,356 --> 03:53:24,256 JOURNALS, RIGHT? 6871 03:53:24,323 --> 03:53:26,425 ACQUIRED BY POSTDOCS, EVEN 6872 03:53:26,492 --> 03:53:27,860 ENGINEERS OR WHATEVER, THEY ARE 6873 03:53:27,927 --> 03:53:29,395 LABELED OTHER NAME, AND WHEN 6874 03:53:29,462 --> 03:53:34,567 THEY ARE CITED, USED BY OTHERS, 6875 03:53:34,633 --> 03:53:38,704 THAT IS BECOMING SOMETHING LIKE 6876 03:53:38,771 --> 03:53:39,805 A TRUE SCIENTIFIC PREDICTION, 6877 03:53:39,872 --> 03:53:41,874 THEY ARE THINKING ABOUT THAT 6878 03:53:41,941 --> 03:53:43,042 KIND OF APPROACH. 6879 03:53:43,109 --> 03:53:45,711 NOW, TO ADDRESS THE QUESTION OF 6880 03:53:45,778 --> 03:53:49,315 PUBLISHING NEGATIVE RESULTS OR 6881 03:53:49,382 --> 03:53:50,850 GOING AGAINST DOGMA, BECAUSE 6882 03:53:50,916 --> 03:53:52,618 YOU'RE FIGHTING A BIG NAME, FOR 6883 03:53:52,685 --> 03:53:57,757 EXAMPLE, CONTRADICTING A BIG 6884 03:53:57,823 --> 03:53:59,592 NAME, IT'S NOT AN YOU SHALL 6885 03:53:59,658 --> 03:54:03,195 BECAUSE WE'RE USING DOUBLE 6886 03:54:03,262 --> 03:54:03,529 BLIND. 6887 03:54:03,596 --> 03:54:05,464 IF THOMAS SENT THE PAPER TO 6888 03:54:05,531 --> 03:54:06,732 NEURO, THE REVIEWERS DON'T KNOW 6889 03:54:06,799 --> 03:54:08,934 WHETHER IT'S COMING FROM THOMAS 6890 03:54:09,001 --> 03:54:11,237 LAB OR SOMEBODY THEY HAVE NEVER 6891 03:54:11,303 --> 03:54:12,805 HEARD BEFORE. 6892 03:54:12,872 --> 03:54:15,408 SO, THAT REMOVES THAT PART OF 6893 03:54:15,474 --> 03:54:16,242 BIAS. 6894 03:54:16,308 --> 03:54:22,982 WHICH IS ALSO ANOTHER PROCESS 6895 03:54:23,048 --> 03:54:24,183 THAT SIMPLIFIES THINGS FOR US. 6896 03:54:24,250 --> 03:54:25,551 THAT'S WHY MAYBE IT'S WORKING. 6897 03:54:25,618 --> 03:54:26,218 YOU DON'T KNOW. 6898 03:54:26,285 --> 03:54:28,454 SO IF YOU DON'T KNOW, MAYBE IT'S 6899 03:54:28,521 --> 03:54:31,557 A BIG NAME. 6900 03:54:34,460 --> 03:54:35,428 >> ALEX FREEMAN, PUBLISHING 6901 03:54:35,494 --> 03:54:35,861 PLATFORM. 6902 03:54:35,928 --> 03:54:37,730 A COUPLE OF YOU MENTIONED 6903 03:54:37,797 --> 03:54:38,931 PROBLEMS OF HAVING REVIEWERS 6904 03:54:38,998 --> 03:54:40,800 COMMENTING ON THE FACT THAT THEY 6905 03:54:40,866 --> 03:54:43,002 DIDN'T THINK THAT NEGATIVE OR 6906 03:54:43,068 --> 03:54:44,637 NULL RESULTS SHOULD BE 6907 03:54:44,703 --> 03:54:44,904 PUBLISHED. 6908 03:54:44,970 --> 03:54:49,008 I WONDERED IF YOU COULD REFLECT 6909 03:54:49,074 --> 03:54:51,343 ON HOW MUCH GUIDANCE YOU GIVE TO 6910 03:54:51,410 --> 03:54:53,179 REVIEWERS AND HOW MUCH 6911 03:54:53,245 --> 03:54:54,380 STRUCTURED REVIEW YOU ASK AND 6912 03:54:54,447 --> 03:54:57,917 WHETHER IT MAKES A DIFFERENCE IF 6913 03:54:57,983 --> 03:54:59,118 REVIEWS WILL BE PUBLISHED OPENLY 6914 03:54:59,185 --> 03:55:02,388 AS WELL. 6915 03:55:02,455 --> 03:55:07,993 >> SO, IN TERMS OF THE GUIDANCE, 6916 03:55:08,060 --> 03:55:08,928 WE PROVIDE THEM, I DON'T KNOW 6917 03:55:08,994 --> 03:55:11,163 HOW MUCH THEY READ LETTERS IN 6918 03:55:11,230 --> 03:55:12,465 TERMS OF INSTRUCTIONS, RIGHT? 6919 03:55:12,531 --> 03:55:13,966 WHERE WE SAY WE'RE INTERESTED 6920 03:55:14,033 --> 03:55:16,402 MORE IN THE QUESTIONS THAN THE 6921 03:55:16,469 --> 03:55:18,370 RESULTS, SO ON. 6922 03:55:18,437 --> 03:55:21,540 OFTEN, THIS COMES -- IT'S NOT 6923 03:55:21,607 --> 03:55:24,176 SUITABLE FOR JOURNAL, 6924 03:55:24,243 --> 03:55:26,345 CONFIDENTIAL, SO IT'S NOT -- 6925 03:55:26,412 --> 03:55:27,880 NOW, WE JUST ARE STARTING A 6926 03:55:27,947 --> 03:55:30,616 PILOT PROJECT AT THE MOMENT FOR 6927 03:55:30,683 --> 03:55:31,817 POSTING OPEN REVIEWS ON THE 6928 03:55:31,884 --> 03:55:36,422 JOURNAL, SO WE'LL SEE EFFECT OF 6929 03:55:36,489 --> 03:55:36,622 THAT. 6930 03:55:36,689 --> 03:55:38,491 WE HAD A LOT OF PUSHBACK FROM 6931 03:55:38,557 --> 03:55:39,658 THE COMMUNITY, THAT'S ANOTHER 6932 03:55:39,725 --> 03:55:40,993 TOPIC, IN TERMS OF MAKING IT 6933 03:55:41,060 --> 03:55:41,961 FULLY OPEN. 6934 03:55:42,027 --> 03:55:46,365 I WAS EDITOR AT THE U.S. EDITOR 6935 03:55:46,432 --> 03:55:49,201 FOR BMJ FOR SEVERAL YEARS, A 6936 03:55:49,268 --> 03:55:50,102 FULLY OPEN PROCESS, EVERYBODY 6937 03:55:50,169 --> 03:55:51,737 KNOWS WHO EVERYBODY IS. 6938 03:55:51,804 --> 03:55:53,138 AND THAT WORKED. 6939 03:55:53,205 --> 03:55:56,542 THAT WORKED WELL. 6940 03:55:56,609 --> 03:55:58,844 AND THEN YOU WOULD HAVE THAT 6941 03:55:58,911 --> 03:55:59,712 COMMENT, NOT SUITABLE, WE WOULD 6942 03:55:59,778 --> 03:56:02,014 TAKE IT OUT IF THAT COMMENT WAS 6943 03:56:02,081 --> 03:56:03,549 THERE FROM WHAT'S PUT OUT THERE 6944 03:56:03,616 --> 03:56:04,650 BECAUSE THAT'S NOT FOR THE 6945 03:56:04,717 --> 03:56:05,851 REVIEWER TO SAY. 6946 03:56:05,918 --> 03:56:09,455 THE REVIEWERS IS JUST TO COMMENT 6947 03:56:09,522 --> 03:56:10,856 ON THE QUESTION, AND METHODS AND 6948 03:56:10,923 --> 03:56:14,260 SO ON, NOT SO MUCH IN TERMS OF 6949 03:56:14,326 --> 03:56:15,294 WHAT GOES IN. 6950 03:56:15,361 --> 03:56:16,161 THAT MAY CHANGE THINGS. 6951 03:56:16,228 --> 03:56:19,532 I DON'T KNOW IF HAD YOU THAT 6952 03:56:19,598 --> 03:56:20,699 EXPERIENCE WITH PLOS OR -- 6953 03:56:20,766 --> 03:56:23,035 >> YEAH, I THINK REVIEWERS 6954 03:56:23,102 --> 03:56:24,503 ARE -- WE DON'T HAVE A DIRECT 6955 03:56:24,570 --> 03:56:25,871 RELATIONSHIP WITH THEM, RIGHT? 6956 03:56:25,938 --> 03:56:30,042 THEY ARE SELECTED BY ACADEMIC 6957 03:56:30,109 --> 03:56:30,943 EDITORS. 6958 03:56:31,010 --> 03:56:32,912 WE RELY ON ACADEMIC EDITORS BUT 6959 03:56:32,978 --> 03:56:36,949 WE HAVE A BOARD OF 10,000ED 6960 03:56:37,016 --> 03:56:38,250 CORES, GETTING THEM TO ALL 6961 03:56:38,317 --> 03:56:40,352 FOLLOW THE SAME CRITERIA IS 6962 03:56:40,419 --> 03:56:42,421 REALLY TRICKY, THERE'S A BIG 6963 03:56:42,488 --> 03:56:43,289 EDUCATION PIECE AND BIG SUPPORT 6964 03:56:43,355 --> 03:56:49,094 PIECE THAT HAS TO COME INTO PLAY 6965 03:56:49,161 --> 03:56:49,662 HERE. 6966 03:56:49,728 --> 03:56:50,329 AUTHORS, REVIEWERS, EDITORS, 6967 03:56:50,396 --> 03:56:53,032 READERS, WE UNDERSTAND THIS FROM 6968 03:56:53,098 --> 03:56:55,334 PERSPECTIVE OF RESEARCHERS, HOW 6969 03:56:55,401 --> 03:56:56,735 FRUSTRATING NOT TO PUBLISH 6970 03:56:56,802 --> 03:56:57,836 RESULTS BUT THERE'S A DISCONNECT 6971 03:56:57,903 --> 03:57:07,613 WHEN WE GET THESE PAPERS AS 6972 03:57:07,680 --> 03:57:08,347 REVIEWERS AND AUTHORS, THIS 6973 03:57:08,414 --> 03:57:09,114 CAN'T BE PUBLISHED. 6974 03:57:09,181 --> 03:57:09,682 TOUGH NUT TO CRACK. 6975 03:57:09,748 --> 03:57:12,284 I DON'T KNOW THE ANSWER TO THAT. 6976 03:57:12,351 --> 03:57:15,921 >> JUST TO ADD, I THINK 6977 03:57:15,988 --> 03:57:17,556 SOMETIMES REVIEWERS APPLY A 6978 03:57:17,623 --> 03:57:18,657 DIFFERENT LENS TO METHODOLOGY 6979 03:57:18,724 --> 03:57:19,758 WHEN THERE'S ANYTHING RESULTS IN 6980 03:57:19,825 --> 03:57:23,028 FRONT OF THEM. 6981 03:57:23,095 --> 03:57:25,664 THEY MAY SAY WE SHOULD REJECT 6982 03:57:25,731 --> 03:57:26,732 BECAUSE RESULTS ARE NULL, THEY 6983 03:57:26,799 --> 03:57:29,134 WILL GO ON THE HUNT FOR THINGS 6984 03:57:29,201 --> 03:57:31,570 THAT MIGHT BE WRONG WITH 6985 03:57:31,637 --> 03:57:33,572 METHODS, IN A WAY THEY WOULD NOT 6986 03:57:33,639 --> 03:57:36,809 DO, ALL OF US ARE PRONE TO 6987 03:57:36,875 --> 03:57:37,910 CONFIRMATION BIAS, IT'S THE JOB 6988 03:57:37,977 --> 03:57:42,648 OF THE EDITOR TO KEEP A CLOSE 6989 03:57:42,715 --> 03:57:45,517 EYE AS MUCH AS POSSIBLE, BUT 6990 03:57:45,584 --> 03:57:47,786 IT'S AN IMPOSSIBLE TASK. 6991 03:57:47,853 --> 03:57:53,926 IF YOU'RE AN EDITOR, DETAILED 6992 03:57:53,993 --> 03:57:56,862 INFORMATION, YOU MAY FEEL 6993 03:57:56,929 --> 03:57:57,529 OBLIGATED TO RESPOND BY 6994 03:57:57,596 --> 03:57:58,631 REJECTING THAT PAPER. 6995 03:57:58,697 --> 03:58:00,165 IF THOSE COMMENTS WERE NOT MADE 6996 03:58:00,232 --> 03:58:01,367 BECAUSE THE RESULTS WERE 6997 03:58:01,433 --> 03:58:03,235 POSITIVE, YOU WOULD NOT FEEL THE 6998 03:58:03,302 --> 03:58:04,103 SAME. 6999 03:58:04,169 --> 03:58:05,971 SO I THINK EDITORS ARE GOING TO 7000 03:58:06,038 --> 03:58:08,374 FALL INTO THE SAME BLACK HOLE, 7001 03:58:08,440 --> 03:58:11,110 AS IT WERE. 7002 03:58:11,176 --> 03:58:12,645 I FOUND HAVING EDITED REGISTERED 7003 03:58:12,711 --> 03:58:13,846 REPORTS AND REGULAR ARTICLES, 7004 03:58:13,912 --> 03:58:15,214 THAT REGISTERED REPORTS ARE THE 7005 03:58:15,280 --> 03:58:15,648 ONLY SOLUTION. 7006 03:58:15,714 --> 03:58:18,651 YOU HAVE TO TAKE AWAY THE 7007 03:58:18,717 --> 03:58:18,984 RESULTS. 7008 03:58:19,051 --> 03:58:20,619 EVEN THEN, OCCASIONALLY AT STAGE 7009 03:58:20,686 --> 03:58:22,488 2, A REVIEWER WILL SAY IT'S A 7010 03:58:22,554 --> 03:58:24,223 SHAME THE RESULTS WERE NEGATIVE, 7011 03:58:24,289 --> 03:58:28,761 AND THEY WILL ASK FOR A LOT OF 7012 03:58:28,827 --> 03:58:30,396 EXTRA ANALYSES. 7013 03:58:30,462 --> 03:58:31,797 WITH A REGISTERED REPORT YOU 7014 03:58:31,864 --> 03:58:35,934 CAN'T SHIFT THE GOAL POSTS BUT 7015 03:58:36,001 --> 03:58:37,503 YOU CAN WITH A REGULAR ARTICLE. 7016 03:58:37,569 --> 03:58:40,105 WE HAVE TO BE MINDFUL OF THE 7017 03:58:40,172 --> 03:58:41,206 COGNITIVE LIMITATIONS OF EDITORS 7018 03:58:41,273 --> 03:58:45,344 AND REVIEWERS AND BUILD SYSTEMS 7019 03:58:45,411 --> 03:58:49,114 LIKE REGISTERED REPORTS WHICH 7020 03:58:49,181 --> 03:58:51,750 PREVENT BIASES FROM HAVING AN 7021 03:58:51,817 --> 03:58:53,152 EFFECT. 7022 03:58:53,218 --> 03:58:53,752 >> NEIL, STANFORD. 7023 03:58:53,819 --> 03:58:55,220 I'LL DIRECT MY QUESTION TO 7024 03:58:55,287 --> 03:58:56,755 LAURA, BUT IF YOU DON'T WANT TO 7025 03:58:56,822 --> 03:58:57,956 ANSWER THAT'S FINE. 7026 03:58:58,023 --> 03:59:00,392 CASE STUDY WAS ABOUT A 7027 03:59:00,459 --> 03:59:04,129 REPLICATION AND CRITIQUE FROM 7028 03:59:04,196 --> 03:59:05,698 KIF. 7029 03:59:05,764 --> 03:59:06,765 THERE'S EMERGING LITERATURE ON 7030 03:59:06,832 --> 03:59:08,067 NULL HACKING THAT SUGGESTS YOU 7031 03:59:08,133 --> 03:59:09,168 HAVE THE OPPOSITE BIAS THAT 7032 03:59:09,234 --> 03:59:12,137 WE'RE DISCUSSING WHEN IT COMES 7033 03:59:12,204 --> 03:59:13,038 TO REPLICATION AND CRITIQUED 7034 03:59:13,105 --> 03:59:13,338 PAPERS. 7035 03:59:13,405 --> 03:59:16,075 IT'S A LOT EASIER TO PUBLISH A 7036 03:59:16,141 --> 03:59:17,710 REPLICATION THAT SAYS SOMEONE'S 7037 03:59:17,776 --> 03:59:19,144 WRONG THAN PUBLISHING A 7038 03:59:19,211 --> 03:59:20,212 REPLICATION CRITIQUE THAT SAYS 7039 03:59:20,279 --> 03:59:21,413 SOMEONE IS RIGHT BUT ALL OF THIS 7040 03:59:21,480 --> 03:59:24,183 IS REALLY IMPORTANT FOR SCIENCE 7041 03:59:24,249 --> 03:59:25,718 TO SEE EVERYTHING. 7042 03:59:25,784 --> 03:59:26,985 IF YOU'RE BAYESIAN YOU WANT TO 7043 03:59:27,052 --> 03:59:28,120 KNOW IF SOMEONE DID SOMETHING 7044 03:59:28,187 --> 03:59:29,655 BEFORE THAT WAS CORRECT, SO WHAT 7045 03:59:29,722 --> 03:59:32,491 KIND OF INSTITUTIONS DO WE NEED 7046 03:59:32,558 --> 03:59:34,460 TO ENCOURAGE REPLICATIONS THAT 7047 03:59:34,526 --> 03:59:35,828 DON'T OVERTURN PRIOR RESEARCH 7048 03:59:35,894 --> 03:59:39,598 BUT SUPPORT IT? 7049 03:59:39,665 --> 03:59:41,567 >> I'M NOT SURE ABOUT 7050 03:59:41,633 --> 03:59:41,900 INSTITUTIONS. 7051 03:59:41,967 --> 03:59:43,535 I THINK WE NEED TO CHANGE THE 7052 03:59:43,602 --> 03:59:46,171 CULTURE SURROUNDING IS IT OKAY 7053 03:59:46,238 --> 03:59:48,474 TO REPLICATE AND TO, YOU KNOW, 7054 03:59:48,540 --> 03:59:49,675 DISAGREE OR AGREE, AND I AGREE 7055 03:59:49,742 --> 03:59:51,343 WITH YOUR COMMENT THAT IT'S 7056 03:59:51,410 --> 03:59:54,847 EASIER TO OVERTURN. 7057 03:59:54,913 --> 03:59:56,615 CERTAINLY EASIER BECAUSE 7058 03:59:56,682 --> 03:59:57,483 METHODOLOGIES DIFFER BETWEEN 7059 03:59:57,549 --> 04:00:03,155 LABS IN MANY CASES, IT'S EASIER 7060 04:00:03,222 --> 04:00:04,790 TO FRAME A QUESTION WE'LL NOT 7061 04:00:04,857 --> 04:00:06,658 SUPPORT WHAT'S ALREADY BEEN 7062 04:00:06,725 --> 04:00:07,092 SHOWN. 7063 04:00:07,159 --> 04:00:09,628 I'M NOT SURE IF I HAVE BEYOND 7064 04:00:09,695 --> 04:00:12,131 THAT A GOOD -- I DON'T THINK 7065 04:00:12,197 --> 04:00:13,766 THIS IS A BIG PROBLEM IN MY 7066 04:00:13,832 --> 04:00:15,400 UNDERSTANDING THAT PEOPLE ARE 7067 04:00:15,467 --> 04:00:17,603 OUT THERE, YOU KNOW, REPLICATING 7068 04:00:17,669 --> 04:00:22,207 AND TRYING TO BRING DOWN OTHER 7069 04:00:22,274 --> 04:00:25,711 LABS, NOT REPLICATING, I SHOULD 7070 04:00:25,778 --> 04:00:25,978 SAY. 7071 04:00:26,044 --> 04:00:30,849 IS THIS A BIG PROBLEM? 7072 04:00:30,916 --> 04:00:31,717 >> IF YOU'RE REPLICATING, YOU'LL 7073 04:00:31,784 --> 04:00:32,918 GO IN THE OTHER DIRECTION. 7074 04:00:32,985 --> 04:00:35,587 THE HARDEST THING TO PUBLISH IS 7075 04:00:35,654 --> 04:00:36,655 REPLICATING THAT SHOWS SOMEONE 7076 04:00:36,722 --> 04:00:37,456 IS RIGHT. 7077 04:00:37,523 --> 04:00:40,292 HOW DO WE GET THOSE PAPERS 7078 04:00:40,359 --> 04:00:40,759 PUBLISHED? 7079 04:00:40,826 --> 04:00:43,362 >> YOU MEAN ACTUAL PUBLICATION 7080 04:00:43,428 --> 04:00:44,229 PROCESS ITSELF? 7081 04:00:44,296 --> 04:00:46,398 YEAH, YEAH. 7082 04:00:46,465 --> 04:00:47,299 I THINK OUR WHOLE INCENTIVE 7083 04:00:47,366 --> 04:00:48,834 STRUCTURE NEEDS TO BE DIFFERENT. 7084 04:00:48,901 --> 04:00:52,204 WE NEED TO VALUE DIFFERENT 7085 04:00:52,271 --> 04:00:56,241 THINGS, INCLUDING SUPPORT, 7086 04:00:56,308 --> 04:00:59,211 EMERGING EVIDENCE SOMETHING IS 7087 04:00:59,278 --> 04:01:02,614 TRUE OR NOT. 7088 04:01:02,681 --> 04:01:03,582 THAT'S WHAT I HAVE TO SAY. 7089 04:01:03,649 --> 04:01:08,287 >> I WANT TO COMMENT ON THAT. 7090 04:01:08,353 --> 04:01:11,490 AS A BASIC RESEARCHER, IF YOU DO 7091 04:01:11,557 --> 04:01:12,224 THINGS CAREFULLY BEFORE YOU 7092 04:01:12,291 --> 04:01:13,892 START SOMETHING YOU REPEAT THE 7093 04:01:13,959 --> 04:01:15,527 EXPERIMENT TO SEE IS ANY 7094 04:01:15,594 --> 04:01:16,528 FOUNDATION FOR WHAT YOU'RE 7095 04:01:16,595 --> 04:01:20,232 TRYING TO DO, OKAY? 7096 04:01:20,299 --> 04:01:22,401 NOW, IF IT SUPPORTS, YOU KEEP ON 7097 04:01:22,467 --> 04:01:25,137 WITH YOUR RESEARCH AND ADVANCE 7098 04:01:25,204 --> 04:01:25,671 SCIENCE. 7099 04:01:25,737 --> 04:01:28,640 IF IT'S NULL YOU PUT IT IN THE 7100 04:01:28,707 --> 04:01:32,344 FILE DRAWER ACCOUNT , YOU DON'T 7101 04:01:32,411 --> 04:01:34,346 WANT TO GET INTO CONFRONTATION 7102 04:01:34,413 --> 04:01:37,616 WITH WHOEVER PUBLISHED THE 7103 04:01:37,683 --> 04:01:37,816 PAPER. 7104 04:01:37,883 --> 04:01:39,484 REPRODUCIBILITY, I DON'T LIKE 7105 04:01:39,551 --> 04:01:42,554 THE TERM AT ALL, BUT BUT 7106 04:01:42,621 --> 04:01:44,056 REPRODUCIBILITY IN THE SENSE YOU 7107 04:01:44,122 --> 04:01:46,592 TAKE A BIG STUDY AND REPEAT IT, 7108 04:01:46,658 --> 04:01:48,060 THAT'S VERY DIFFICULT TO DO 7109 04:01:48,126 --> 04:01:49,561 BECAUSE THAT TAKES A LOT OF 7110 04:01:49,628 --> 04:01:54,066 FUNDS, GOOD LUCK GETTING FUNDS 7111 04:01:54,132 --> 04:01:55,167 TO DO THAT. 7112 04:01:55,234 --> 04:01:57,236 THAT'S A MAJOR ISSUE. 7113 04:01:57,302 --> 04:01:57,769 >> NEIL? 7114 04:01:57,836 --> 04:01:58,470 >> INTERESTING QUESTION. 7115 04:01:58,537 --> 04:02:01,073 I REALLY LIKE IT. 7116 04:02:01,139 --> 04:02:04,676 IT'S NOT EASY TO DISPROVE 7117 04:02:04,743 --> 04:02:06,645 SOMEONE IS WRONG ESPECIALLY WHEN 7118 04:02:06,712 --> 04:02:09,715 PUBLISHED IN A TOP TIER JOURNAL. 7119 04:02:09,781 --> 04:02:12,184 WHEN I STARTED THE EXPERIMENT, 7120 04:02:12,251 --> 04:02:15,721 WHEN I SAW WHAT I SAW, I DIDN'T 7121 04:02:15,787 --> 04:02:16,722 BELIEVE IT. 7122 04:02:16,788 --> 04:02:18,257 I'M NOT AN EXPERT, JUST STARTING 7123 04:02:18,323 --> 04:02:19,791 MY POSTDOC CAREER. 7124 04:02:19,858 --> 04:02:23,295 TWO, THAT I COULD BE WRONG. 7125 04:02:23,362 --> 04:02:25,364 SO IT INVOLVED MULTIPLE 7126 04:02:25,430 --> 04:02:28,000 DIFFERENT EXPERIMENTS TO SUPPORT 7127 04:02:28,066 --> 04:02:28,967 THIS OBSERVATION. 7128 04:02:29,034 --> 04:02:33,805 I WOULD SAY IT'S EQUALLY 7129 04:02:33,872 --> 04:02:36,742 DIFFICULT TO DISPROVE SOMEONE IS 7130 04:02:36,808 --> 04:02:37,276 WRONG. 7131 04:02:37,342 --> 04:02:39,177 TO ANSWER YOUR QUESTION 7132 04:02:39,244 --> 04:02:43,548 BASICALLY IF SOMEONE IS RIGHT, 7133 04:02:43,615 --> 04:02:44,650 ESPECIALLY IN BASIC SCIENCE, 7134 04:02:44,716 --> 04:02:47,386 RECITE THEM IN A WAY WE'RE ABLE 7135 04:02:47,452 --> 04:02:48,687 TO REPRODUCE THEIR RESULTS, IN A 7136 04:02:48,754 --> 04:02:51,323 WAY WE'RE ABLE TO USE THEIR 7137 04:02:51,390 --> 04:02:52,190 TECHNIQUE OR TOOL. 7138 04:02:52,257 --> 04:02:56,561 THAT IN A WAY IS KIND OF 7139 04:02:56,628 --> 04:02:58,563 CORRECTING WHAT THEY HAVE. 7140 04:02:58,630 --> 04:03:01,533 IT'S NOT SEPARATE STORY LINE 7141 04:03:01,600 --> 04:03:04,803 THAT WE WANTED TO PUBLISH, YOU 7142 04:03:04,870 --> 04:03:15,414 KNOW, SO-AND-SO TOOL IS AMAZING. 7143 04:03:17,883 --> 04:03:21,119 >> CAN I CHALLENGE NIH AT THAT 7144 04:03:21,186 --> 04:03:21,553 POINT? 7145 04:03:21,620 --> 04:03:24,957 WE HEARD ABOUT THE PROBLEM WITH 7146 04:03:25,023 --> 04:03:28,460 PUBLISHING NULL RESULTS, AND 7147 04:03:28,527 --> 04:03:29,995 ALSO REPLICATIONS. 7148 04:03:30,062 --> 04:03:30,896 JUST SOME PUBLIC REPLICATIONS, 7149 04:03:30,963 --> 04:03:33,932 SEE WHAT HAPPENS. 7150 04:03:33,999 --> 04:03:35,233 WE DO PUBLISH REPLICATION 7151 04:03:35,300 --> 04:03:35,600 STUDIES. 7152 04:03:35,667 --> 04:03:36,335 WE VALUE THEM. 7153 04:03:36,401 --> 04:03:39,638 IT'S IMPORTANT FOR THE FIELD. 7154 04:03:39,705 --> 04:03:44,443 NOW, P.I.s MAY NOT WANT TO 7155 04:03:44,509 --> 04:03:47,412 PUBLISH NEGATIVE RESULTS, 7156 04:03:47,479 --> 04:03:48,513 REPLICATION STUDIES. 7157 04:03:48,580 --> 04:03:51,016 IN EUROPE, BECAUSE IT COSTS 7158 04:03:51,083 --> 04:03:56,388 MONEY, BUT IN THE U.S. YOUR LABS 7159 04:03:56,455 --> 04:03:59,458 ARE RICH. 7160 04:03:59,524 --> 04:04:02,294 IN EUROPE, SOUTH AMERICA, WE GET 7161 04:04:02,361 --> 04:04:04,830 MUCH LESS FUNDING, OR AFRICA AND 7162 04:04:04,896 --> 04:04:05,163 ASIA. 7163 04:04:05,230 --> 04:04:07,566 NOW, PUBLISHING SOMETHING IS A 7164 04:04:07,632 --> 04:04:11,169 LOT OF OUR BUDGET. 7165 04:04:11,236 --> 04:04:13,672 NOW, THE NIH IS NOT ENCOURAGING 7166 04:04:13,739 --> 04:04:16,308 THE PUBLICATION OF NEGATIVE 7167 04:04:16,375 --> 04:04:18,076 RESULTS AND REPLICATION BY 7168 04:04:18,143 --> 04:04:20,145 SUBSIDIZING DIRECTLY FROM THE 7169 04:04:20,212 --> 04:04:21,913 GRANT, OUTSIDE THE GRANT, BUT 7170 04:04:21,980 --> 04:04:23,348 THE PUBLICATION OF THE RESULTS? 7171 04:04:23,415 --> 04:04:25,851 AND THAT WOULD BE A VERY GOOD 7172 04:04:25,917 --> 04:04:28,253 INCENTIVE FOR THE LABS TO 7173 04:04:28,320 --> 04:04:33,859 PUBLISH THEIR NEGATIVE STUDIES 7174 04:04:33,925 --> 04:04:37,896 AND REPLICATION STUDIES. 7175 04:04:37,963 --> 04:04:38,864 I'M SERIOUS. 7176 04:04:38,930 --> 04:04:41,733 >> I CAN'T SPEAK FOR NIH. 7177 04:04:41,800 --> 04:04:44,369 BUT WHAT I CAN SAY IS, YOU KNOW, 7178 04:04:44,436 --> 04:04:51,376 I DON'T KNOW WHY THERE'S NO MORE 7179 04:04:51,443 --> 04:04:53,845 ENCOURAGEMENT FOR NULL STUDIES. 7180 04:04:53,912 --> 04:04:55,380 REPLICATION, I CAN TELL YOU THE 7181 04:04:55,447 --> 04:04:59,051 ISSUE, IF YOU'RE TRYING TO 7182 04:04:59,117 --> 04:04:59,484 REPRODUCE RESULTS. 7183 04:04:59,551 --> 04:05:04,589 AND THAT IS THAT YES, NIH HAS A 7184 04:05:04,656 --> 04:05:06,925 A LOT OF FUNDS, ALSO MANY MORE 7185 04:05:06,992 --> 04:05:07,659 INVESTIGATORS THAN IN EUROPE. 7186 04:05:07,726 --> 04:05:09,394 AT THE END OF THE DAY IT BOILS 7187 04:05:09,461 --> 04:05:13,799 DOWN TO THE SAME ISSUES, AND 7188 04:05:13,865 --> 04:05:17,169 EVERY STUDY THAT YOU SAY, YOU 7189 04:05:17,235 --> 04:05:18,937 KNOW, WE -- LET'S PRETEND FOR 7190 04:05:19,004 --> 04:05:20,872 ARGUMENT THAT YOU HAVE TO 7191 04:05:20,939 --> 04:05:21,540 REPRODUCE EVERYTHING. 7192 04:05:21,606 --> 04:05:25,710 SO EVERYTHING HAS TO BE 7193 04:05:25,777 --> 04:05:26,211 REPRODUCED. 7194 04:05:26,278 --> 04:05:27,712 THAT MEANS HALF THE FUNDS WILL 7195 04:05:27,779 --> 04:05:30,015 BE GONE FOR NEW RESEARCH, THERE 7196 04:05:30,082 --> 04:05:31,716 IS WHAT THERE IS. 7197 04:05:31,783 --> 04:05:37,022 AND SO YOU HAVE TO MAKE A 7198 04:05:37,089 --> 04:05:38,957 DECISION WHAT MEETS THE BAR THAT 7199 04:05:39,024 --> 04:05:41,359 YOU SAY SHOULD BE REPRODUCED? 7200 04:05:41,426 --> 04:05:42,260 THERE ARE ONGOING DISCUSSIONS 7201 04:05:42,327 --> 04:05:42,928 THE NIH. 7202 04:05:42,994 --> 04:05:46,731 YOU KNOW, WHAT THAT MIGHT BE. 7203 04:05:46,798 --> 04:05:48,934 THERE HASN'T BEEN DECISIONS, NOT 7204 04:05:49,000 --> 04:05:49,801 PART OF THIS CONVERSATION, BUT 7205 04:05:49,868 --> 04:05:52,537 AS FAR AS NULL RESULTS I FULLY 7206 04:05:52,604 --> 04:05:57,876 AGREE, YOU KNOW, THERE NEEDS TO 7207 04:05:57,943 --> 04:05:59,144 BE MORE. 7208 04:05:59,211 --> 04:06:00,245 >> PAUL SUNDBERG. 7209 04:06:00,312 --> 04:06:04,783 IF YOU DECREASE PUBLICATION 7210 04:06:04,850 --> 04:06:06,151 COSTS, BIOLOGY HAS LOW COST, YOU 7211 04:06:06,218 --> 04:06:07,519 KNOW, WOULD ALLOW MORE OF THAT 7212 04:06:07,586 --> 04:06:09,955 BUT PEOPLE STILL HAVE TO SPEND 7213 04:06:10,021 --> 04:06:11,056 THEIR TIME, RIGHT? 7214 04:06:11,123 --> 04:06:17,996 A LOT OF THESE AS WE HEARD, THE 7215 04:06:18,063 --> 04:06:20,432 FIRST THING YOU DO IS REPLICATE, 7216 04:06:20,499 --> 04:06:22,067 KEY PARTS OF THE STUDY, YOU'RE 7217 04:06:22,134 --> 04:06:23,702 NOT SPENDING TWO YEARS DOING 7218 04:06:23,768 --> 04:06:23,902 THAT. 7219 04:06:23,969 --> 04:06:25,237 YOU'RE SPENDING, I DON'T KNOW, 7220 04:06:25,303 --> 04:06:28,306 LET'S SAY A MONTH OR TWO WEEKS, 7221 04:06:28,373 --> 04:06:29,074 DEPENDING ON YOUR PLATFORM, 7222 04:06:29,141 --> 04:06:30,942 SMALL WITH RESPECT TO YOUR 7223 04:06:31,009 --> 04:06:31,243 TIMELINES. 7224 04:06:31,309 --> 04:06:32,577 THEN YOU CAN PUBLISH THAT. 7225 04:06:32,644 --> 04:06:36,515 IF YOU LOOK AT HOW MANY PAPERS 7226 04:06:36,581 --> 04:06:38,049 PEOPLE PUBLISH MOST PEOPLE DON'T 7227 04:06:38,116 --> 04:06:44,389 EVEN PUBLISH ONE PAPER A YEAR, 7228 04:06:44,456 --> 04:06:45,390 THERE'S LOTS OF CAPACITY TO 7229 04:06:45,457 --> 04:06:46,925 PUBLISH MORE. 7230 04:06:46,992 --> 04:06:48,793 WE HAVE TO MAKE IT EASIER AND 7231 04:06:48,860 --> 04:06:50,095 THEY WILL INCENTIVIZE IT. 7232 04:06:50,162 --> 04:06:51,863 THE COST, IN THE LIMIT, THAT'S A 7233 04:06:51,930 --> 04:06:52,097 PROBLEM. 7234 04:06:52,164 --> 04:06:56,001 RIGHT NOW I DON'T THINK THAT'S 7235 04:06:56,067 --> 04:06:56,868 THE LIMIT. 7236 04:06:56,935 --> 04:06:59,838 >> FOLLOWING ON THAT, IT'S ABOUT 7237 04:06:59,905 --> 04:07:02,574 HOW TO MAKE THE PUBLICATION MORE 7238 04:07:02,641 --> 04:07:06,111 DYNAMIC, IT TAKES A LONG TIME TO 7239 04:07:06,178 --> 04:07:07,946 GET FROM EXPERIMENT TO 7240 04:07:08,013 --> 04:07:09,281 PUBLICATION BUT ALSO FOLLOWING 7241 04:07:09,347 --> 04:07:14,085 ON THE OTHER COLLEAGUE'S 7242 04:07:14,152 --> 04:07:16,288 QUESTIONS ABOUT PEER REVIEW, AS 7243 04:07:16,354 --> 04:07:18,390 AN EARLY CAREER RESEARCHERS MY 7244 04:07:18,456 --> 04:07:19,991 COLLEAGUES, I HEAR THEM TALKING 7245 04:07:20,058 --> 04:07:21,760 ALL THE TIME LIKE WHY DO WE HAVE 7246 04:07:21,826 --> 04:07:23,962 TO DO ALL THIS WORK? 7247 04:07:24,029 --> 04:07:26,031 AND THEY SEE IT AS A CHORE THAT 7248 04:07:26,097 --> 04:07:28,200 THEY JUST WANT TO GET OUT OF THE 7249 04:07:28,266 --> 04:07:30,835 WAY, BUT I CAN SEE THAT IT'S 7250 04:07:30,902 --> 04:07:32,137 CENTRAL FOR EVERYTHING. 7251 04:07:32,204 --> 04:07:37,309 SO, HOW DO WE CHANGE THAT? 7252 04:07:37,375 --> 04:07:38,710 HOW DO WE ACKNOWLEDGE PEER 7253 04:07:38,777 --> 04:07:40,679 REVIEW IS AS IMPORTANT AS BEING 7254 04:07:40,745 --> 04:07:43,048 AN AUTHOR ON A PUBLICATION? 7255 04:07:43,114 --> 04:07:47,953 THANK YOU. 7256 04:07:50,855 --> 04:07:53,825 >> CHANGING THE WAY THAT THESE 7257 04:07:53,892 --> 04:07:55,794 CONTRIBUTIONS ARE VALUED, SO 7258 04:07:55,860 --> 04:07:57,862 MAKING SURE REVIEWERS HAVE THE 7259 04:07:57,929 --> 04:07:59,631 OPPORTUNITY TO DISCLOSE THEIR 7260 04:07:59,698 --> 04:08:01,032 IDENTITIES, REVIEWS WILL BE 7261 04:08:01,099 --> 04:08:02,334 PUBLISHED, THAT THEY CAN RECEIVE 7262 04:08:02,400 --> 04:08:04,536 CREDIT FOR THAT WORK, AND THAT 7263 04:08:04,603 --> 04:08:06,504 INSTITUTIONS AND UNIVERSITIES 7264 04:08:06,571 --> 04:08:08,039 ACTUALLY TAKE THIS ACTIVITY 7265 04:08:08,106 --> 04:08:10,442 SERIOUSLY AND SET ASIDE TIME IN 7266 04:08:10,508 --> 04:08:15,247 PEOPLE'S DAYS TO REVIEW 7267 04:08:15,313 --> 04:08:16,448 MANUSCRIPTS, SET ASIDE RESOURCES 7268 04:08:16,514 --> 04:08:17,449 FOR TRAINING, THAT'S THE ONLY 7269 04:08:17,515 --> 04:08:26,858 WAY WE CAN START TO MAKE HEADWAY 7270 04:08:26,925 --> 04:08:27,626 ON THAT. 7271 04:08:27,692 --> 04:08:28,693 >> ONE THE MOST INTERESTING 7272 04:08:28,760 --> 04:08:30,262 EXPERIENCES I HAD WHEN WE FIRST 7273 04:08:30,328 --> 04:08:31,429 LAUNCHED THE INITIATIVE WAS 7274 04:08:31,496 --> 04:08:34,165 NUMBER OF REVIEWERS WHO WOULD 7275 04:08:34,232 --> 04:08:35,700 WRITE TO US SPONTANEOUSLY, I'VE 7276 04:08:35,767 --> 04:08:37,035 NEVER SEEN THIS BEFORE, TO SAY 7277 04:08:37,102 --> 04:08:38,436 HOW MUCH THEY ENJOYED BEING PART 7278 04:08:38,503 --> 04:08:40,405 OF THIS PROCESS. 7279 04:08:40,472 --> 04:08:43,074 WHEN DID A REVIEWERRER EVERY SAY 7280 04:08:43,141 --> 04:08:44,776 THEY ENJOYED PEER REVIEW? 7281 04:08:44,843 --> 04:08:46,778 AND YOU CAN SEE WHY THAT IS, 7282 04:08:46,845 --> 04:08:47,212 RIGHT? 7283 04:08:47,279 --> 04:08:51,916 THEY HAVE A CHANCE TO CONTRIBUTE 7284 04:08:51,983 --> 04:08:53,218 CONSTRUCTIVELY TO A PROJECT THEY 7285 04:08:53,285 --> 04:08:54,853 ARE NOT INVOLVED IN WHEN THE 7286 04:08:54,919 --> 04:08:56,921 COMMENT CAN BE USED TO MAKE THE 7287 04:08:56,988 --> 04:08:58,023 RESEARCH BETTER OR AVOID 7288 04:08:58,089 --> 04:08:58,523 PROBLEMS. 7289 04:08:58,590 --> 04:09:03,161 I THINK THE REGISTERED REPORT, 7290 04:09:03,228 --> 04:09:06,464 HAS ANYBODY IN THE ROOM REVIEWED 7291 04:09:06,531 --> 04:09:07,399 A REGISTERED REPORT? 7292 04:09:07,465 --> 04:09:08,333 RIGHT, SMALL NUMBER. 7293 04:09:08,400 --> 04:09:09,768 I HOPE YOU FIND WHAT I FIND WHEN 7294 04:09:09,834 --> 04:09:11,303 YOU DO IT. 7295 04:09:11,369 --> 04:09:13,605 YOU DO FEEL LIKE YOU'RE A LOT 7296 04:09:13,672 --> 04:09:16,107 MORE VALUABLE TO THE PROCESS, 7297 04:09:16,174 --> 04:09:18,943 AND I THINK THERE'S A GREATER 7298 04:09:19,010 --> 04:09:20,712 SENSE OF SATISFACTION, 7299 04:09:20,779 --> 04:09:25,850 PARTICULARLY WHEN YOU ALMOST -- 7300 04:09:25,917 --> 04:09:28,153 YOU ARE WITH THE AUTHORS ALL THE 7301 04:09:28,219 --> 04:09:30,689 WAY SO I'VE HAD LONGSTANDING 7302 04:09:30,755 --> 04:09:35,060 PROJECTS WHERE I'VE EDITED, 7303 04:09:35,126 --> 04:09:36,361 THERE'S ALL SORTS OF STUFF THAT 7304 04:09:36,428 --> 04:09:37,929 HAPPENS OF THE YOU'RE THERE WITH 7305 04:09:37,996 --> 04:09:39,230 THEM ON THIS JOURNEY. 7306 04:09:39,297 --> 04:09:42,634 THAT CAN CHANGE THE REVIEW 7307 04:09:42,701 --> 04:09:45,937 PROCESS FROM BEING COMBATIVE 7308 04:09:46,004 --> 04:09:47,439 TRIAL OR INQUISITION OF FLAW 7309 04:09:47,505 --> 04:09:49,407 FINDING INTO SOMETHING MORE 7310 04:09:49,474 --> 04:09:56,748 CONSTRUCTIVE AN COLLEGIAL. 7311 04:09:56,815 --> 04:09:58,383 AND ULTIMATELY POSITIVE. 7312 04:09:58,450 --> 04:10:00,485 >> ALEX RAMIREZ, UNIVERSITY OF 7313 04:10:00,552 --> 04:10:01,553 PENNSYLVANIA, POSTBAC SCHOLAR, 7314 04:10:01,619 --> 04:10:04,923 HAVEN'T STARTED GRAD SCHOOL YET. 7315 04:10:04,989 --> 04:10:06,157 I JUST WANTED TO GET SOME 7316 04:10:06,224 --> 04:10:08,126 COMMENTS OR THOUGHTS FROM YOU 7317 04:10:08,193 --> 04:10:10,295 ALL OR EVEN TIPS FOR US REALLY 7318 04:10:10,362 --> 04:10:12,163 EARLY CAREER SCIENTISTS ON HOW 7319 04:10:12,230 --> 04:10:13,131 TO NAVIGATE WHEN YOUR PROJECT 7320 04:10:13,198 --> 04:10:16,101 GOES IN A DIRECTION OF NEGATIVE 7321 04:10:16,167 --> 04:10:18,169 OR NULL RESULTS, AND FACING 7322 04:10:18,236 --> 04:10:20,939 BARRIERS SUCH AS GRADUATION 7323 04:10:21,005 --> 04:10:23,208 REQUIREMENTS, P.I.'S 7324 04:10:23,274 --> 04:10:25,076 EXPECTATIONS, IF YOU'RE A 7325 04:10:25,143 --> 04:10:27,512 POSTDOC, JOB REQUIREMENTS 7326 04:10:27,579 --> 04:10:28,279 GETTING PAPERS OUT IN 7327 04:10:28,346 --> 04:10:30,682 HIGH-IMPACT JOURNALS, DO YOU 7328 04:10:30,749 --> 04:10:33,518 HAVE ANY COMMENTS ON LIKE HOW TO 7329 04:10:33,585 --> 04:10:35,053 NAVIGATE THAT IN THAT SPACE WHEN 7330 04:10:35,120 --> 04:10:40,525 YOU'RE EARLY ON IN YOUR CAREER? 7331 04:10:40,592 --> 04:10:43,361 >> I WILL TRY TO TACKLE THIS 7332 04:10:43,428 --> 04:10:43,895 QUESTION. 7333 04:10:43,962 --> 04:10:46,331 I'M AFRAID I DON'T HAVE ANY GOOD 7334 04:10:46,398 --> 04:10:48,500 ANSWERS FOR YOU AT THIS STAGE, 7335 04:10:48,566 --> 04:10:51,936 UNTIL WE CHANGE OUR CULTURAL 7336 04:10:52,003 --> 04:10:53,271 EXPECTATIONS, I WOULD ENCOURAGE 7337 04:10:53,338 --> 04:10:54,973 YOU TO PRESS FOR DOING THE RIGHT 7338 04:10:55,039 --> 04:10:58,042 THING, WHICH IS PUBLISHING YOUR 7339 04:10:58,109 --> 04:10:58,777 NULL REPORTS, WHATEVER RESULTS 7340 04:10:58,843 --> 04:11:01,413 IN THE STATE THEY ARE IN, MAKING 7341 04:11:01,479 --> 04:11:02,547 SURE YOU'RE RIGOROUS IN YOUR 7342 04:11:02,614 --> 04:11:06,251 DESIGN PROCESS TO THAT POINT AND 7343 04:11:06,317 --> 04:11:07,886 GETTING OTHER OPINIONS OF WHAT 7344 04:11:07,952 --> 04:11:10,722 THAT SHOULD ENTAIL BEFORE YOU 7345 04:11:10,789 --> 04:11:13,258 EMBARK, IF POSSIBLE. 7346 04:11:13,324 --> 04:11:14,659 AND IF YOU'VE ALREADY BEEN 7347 04:11:14,726 --> 04:11:16,861 THROUGH THOSE STEPS THEN GO GET 7348 04:11:16,928 --> 04:11:18,263 OPINIONS ABOUT YOUR RESULTS AND 7349 04:11:18,329 --> 04:11:22,233 WHAT TO DO FROM THERE. 7350 04:11:22,300 --> 04:11:24,502 IF YOU HAVE A PARTICULARLY 7351 04:11:24,569 --> 04:11:27,572 DIFFICULT P.I. WHO REALLY IS 7352 04:11:27,639 --> 04:11:29,741 PUSHING FOR, YOU KNOW, LET'S 7353 04:11:29,808 --> 04:11:33,044 JUST SAY MISJUDGMENTS OF HOW YOU 7354 04:11:33,111 --> 04:11:35,013 SHOULD DO YOUR DATA ANALYSIS, 7355 04:11:35,079 --> 04:11:36,648 YOU SHOULD GO GET MORE HELP FROM 7356 04:11:36,714 --> 04:11:37,982 SOMEONE ELSE WHO CAN HELP STAND 7357 04:11:38,049 --> 04:11:39,083 UP FOR YOURSELF BECAUSE YOU'RE 7358 04:11:39,150 --> 04:11:40,919 NOT IN A POSITION TO STAND UP 7359 04:11:40,985 --> 04:11:45,423 FOR YOURSELF VERY WELL IN YOUR 7360 04:11:45,490 --> 04:11:45,824 ROLE. 7361 04:11:45,890 --> 04:11:46,157 YOU CAN TRY. 7362 04:11:46,224 --> 04:11:49,861 >> BUT YOU SHOULD HAVE A MENTOR, 7363 04:11:49,928 --> 04:11:51,296 RIGHT? 7364 04:11:51,362 --> 04:11:55,033 OUR STUDENTS HAVE MENTORS, 7365 04:11:55,099 --> 04:11:56,868 EXTERNAL MENTORS. 7366 04:11:56,935 --> 04:11:58,536 THESE EXTERNAL PEOPLE CAN REALLY 7367 04:11:58,603 --> 04:12:00,405 ACT IF THEY NEED TO AND YOU CAN 7368 04:12:00,472 --> 04:12:02,006 TALK TO THEM FREELY. 7369 04:12:02,073 --> 04:12:05,643 I DON'T KNOW WHETHER YOU HAVE 7370 04:12:05,710 --> 04:12:07,612 THIS MENTORSHIP IN THE U.S. 7371 04:12:07,679 --> 04:12:10,114 >> SHE JUST GOT HER P.I. 7372 04:12:10,181 --> 04:12:15,053 >> I WAS JUST SAYING IF HER P.I. 7373 04:12:15,119 --> 04:12:18,990 IS AGAINST. 7374 04:12:19,057 --> 04:12:19,491 >> SHE CAN REQUEST -- 7375 04:12:19,557 --> 04:12:22,494 >> THEY ARE ONE AND THE SAME 7376 04:12:22,560 --> 04:12:27,966 USUALLY. 7377 04:12:28,032 --> 04:12:28,833 THESE ARE ONE AND THE SAME IN 7378 04:12:28,900 --> 04:12:30,034 THE U.S. 7379 04:12:30,101 --> 04:12:31,469 USUALLY. 7380 04:12:31,536 --> 04:12:32,670 YOU MIGHT HAVE MENTORS OUTSIDE 7381 04:12:32,737 --> 04:12:34,105 THE P.I. OF THE LAB ABOUT THE 7382 04:12:34,172 --> 04:12:37,141 P.I. IS THE MENTOR, RIGHT? 7383 04:12:37,208 --> 04:12:38,877 YES. 7384 04:12:38,943 --> 04:12:40,745 >> SORRY, MY JET LAG AND CAN'T 7385 04:12:40,812 --> 04:12:43,948 FIND THE WORD IN ENGLISH, LIKE 7386 04:12:44,015 --> 04:12:46,217 THE GODFATHER, SOMEONE WHO IS 7387 04:12:46,284 --> 04:12:48,319 EXTERNAL, AND TO WHOM YOU CAN 7388 04:12:48,386 --> 04:12:52,857 REFER TO AND ASK ADVICE OUTSIDE 7389 04:12:52,924 --> 04:12:55,126 THE LAB. 7390 04:12:55,193 --> 04:13:00,899 SORRY. 7391 04:13:00,965 --> 04:13:05,470 >> YEAH, ADVISER. 7392 04:13:05,537 --> 04:13:09,040 >> CAN I ADD TO THAT QUESTION? 7393 04:13:09,107 --> 04:13:10,141 IN MY EXPERIENCE I THINK THE 7394 04:13:10,208 --> 04:13:15,346 IMMEDIATE HELP I HAD WHEN I WAS, 7395 04:13:15,413 --> 04:13:16,114 YOU KNOW, LACKING MOTIVATION 7396 04:13:16,180 --> 04:13:17,849 PAST TWO YEARS OF MY POSTDOC, 7397 04:13:17,916 --> 04:13:20,585 WHEN I WAS SEEING NEGATIVE 7398 04:13:20,652 --> 04:13:21,886 RESULTS, THAT RESULTS THAT IS 7399 04:13:21,953 --> 04:13:27,625 NOT GOING TO GET ME A BIG PAPER, 7400 04:13:27,692 --> 04:13:29,360 OBVIOUSLY WHEN THE SHOWED THE 7401 04:13:29,427 --> 04:13:30,061 PAPER, RESULTS, PEOPLE SAID THIS 7402 04:13:30,128 --> 04:13:31,996 IS NOT GOING TO MAKE YOU FAMOUS. 7403 04:13:32,063 --> 04:13:35,633 THIS IS NOT GOING TO GET YOU A 7404 04:13:35,700 --> 04:13:35,833 JOB. 7405 04:13:35,900 --> 04:13:41,205 BUT I ALSO HAD COLLEAGUES WHO 7406 04:13:41,272 --> 04:13:43,608 REALLY ENCOURAGED ME, WHO MADE 7407 04:13:43,675 --> 04:13:47,045 ME STAND FOR THE TRUTH BECAUSE 7408 04:13:47,111 --> 04:13:49,881 THIS IS -- IF I HAD TO FOLLOW 7409 04:13:49,948 --> 04:13:53,251 THE SIMILAR ROUTE AS THE 7410 04:13:53,318 --> 04:13:55,453 ORIGINAL PAPER HAS SHOWED, AS I 7411 04:13:55,520 --> 04:13:57,755 SAID, ALL YOU NEED IS A LITTLE 7412 04:13:57,822 --> 04:13:59,090 BIT OF CREATIVITY, SEA OF DATA 7413 04:13:59,157 --> 04:14:00,925 IN FRONT OF YOU. 7414 04:14:00,992 --> 04:14:05,730 JUST NEED TO KNOW HOW TO PACKAGE 7415 04:14:05,797 --> 04:14:05,897 IT. 7416 04:14:05,964 --> 04:14:08,466 BUT IMMEDIATE HELP IN TERMS OF 7417 04:14:08,533 --> 04:14:10,234 LIKE DOING RIGHT THING THAT YOU 7418 04:14:10,301 --> 04:14:13,304 DO IS BASICALLY FROM YOUR 7419 04:14:13,371 --> 04:14:13,871 COLLEAGUE. 7420 04:14:13,938 --> 04:14:14,939 SO IDENTIFY LIKE-MINDED 7421 04:14:15,006 --> 04:14:16,174 COLLEAGUES, NOT TELLING IF YOU 7422 04:14:16,240 --> 04:14:20,645 HAVE TEN LAB MEMBERS THEY WILL 7423 04:14:20,712 --> 04:14:25,016 ALL BE ALIGNING, UNFORTUNATELY 7424 04:14:25,083 --> 04:14:27,518 EVERYBODY HAS THEIR OWN AGENDA, 7425 04:14:27,585 --> 04:14:33,324 THEIR OWN PLANS FOR THEIR 7426 04:14:33,391 --> 04:14:33,791 CAREER. 7427 04:14:33,858 --> 04:14:35,860 EVERYBODY WANTS TO ALIGN THE 7428 04:14:35,927 --> 04:14:37,362 SAME VALUE BUT UNFORTUNATELY WE 7429 04:14:37,428 --> 04:14:39,130 DON'T TALK INSIDE THE LAB. 7430 04:14:39,197 --> 04:14:41,199 YOU NEED TO HAVE LIKE-MINDED 7431 04:14:41,265 --> 04:14:41,466 PEOPLE. 7432 04:14:41,532 --> 04:14:43,735 YOU NEED TO TALK TO. 7433 04:14:43,801 --> 04:14:46,137 EVEN BEFORE P.I. 7434 04:14:46,204 --> 04:14:48,673 OBVIOUSLY HAVING SUPPORTIVE P.I. 7435 04:14:48,740 --> 04:14:50,174 IS VERY, VERY IMPORTANT, SO IN 7436 04:14:50,241 --> 04:14:54,012 YOUR STAIN OF YOUR CAREER I 7437 04:14:54,078 --> 04:14:55,747 THINK REALLY RIGHT TIME TO 7438 04:14:55,813 --> 04:15:03,621 IDENTIFY LABS THAT DOES THINGS 7439 04:15:03,688 --> 04:15:03,821 RIGHT. 7440 04:15:03,888 --> 04:15:04,722 >> HI, UNIVERSITY OF CALIFORNIA 7441 04:15:04,789 --> 04:15:05,523 SANTA CRUZ. 7442 04:15:05,590 --> 04:15:07,492 I WANT TO EMPHASIZE A COUPLE 7443 04:15:07,558 --> 04:15:09,661 THINGS I'M HEARING WHICH ARE 7444 04:15:09,727 --> 04:15:10,895 SUPER IMPORTANT, WHICH ARE WHAT 7445 04:15:10,962 --> 04:15:19,203 ARE THE ENSIGN TEFFS EN-- 7446 04:15:19,270 --> 04:15:19,737 INCENTIVES AND BARRIERS, 7447 04:15:19,804 --> 04:15:23,374 INCENTIVE TO LOWER THE BARRIERS, 7448 04:15:23,441 --> 04:15:24,342 PUBLICATION BARRIERS. 7449 04:15:24,409 --> 04:15:28,946 I WANT TO POINT OUT MICRO PUB 7450 04:15:29,013 --> 04:15:31,049 PROVIDES EXPLICIT GUIDANCE HOW 7451 04:15:31,115 --> 04:15:33,117 TO REVIEW PAPERS, WHETHER OR NOT 7452 04:15:33,184 --> 04:15:34,318 ADDITIONAL EXPERIMENTS ARE 7453 04:15:34,385 --> 04:15:35,186 REQUIRED, IF ADDITIONAL 7454 04:15:35,253 --> 04:15:38,389 EXPERIMENTS ARE NOT REQUIRED 7455 04:15:38,456 --> 04:15:40,558 WHAT LANGUAGE NEEDS TO CHANGE? 7456 04:15:40,625 --> 04:15:43,861 I THINK THE VARIABILITY IN 7457 04:15:43,928 --> 04:15:47,999 ACADEMIC EDITORS, REVIEWERS, 7458 04:15:48,066 --> 04:15:50,201 THERE NEEDS TO BE 7459 04:15:50,268 --> 04:15:51,302 STANDARDIZATION TO LOWER 7460 04:15:51,369 --> 04:15:52,704 PUBLICATION. 7461 04:15:52,770 --> 04:15:54,572 I THINK FUNDING CAN BE A 7462 04:15:54,639 --> 04:15:55,540 POWERFUL INCENTIVE. 7463 04:15:55,606 --> 04:15:58,676 THERE'S A MECHANISM CALLED THE 7464 04:15:58,743 --> 04:16:01,579 R03 WHICH PROVIDES ONE YEAR 7465 04:16:01,646 --> 04:16:03,915 FUNDING FOR LAB OR P.I. TO SORT 7466 04:16:03,981 --> 04:16:06,651 OF FINISH UP A PROJECT OR DO A 7467 04:16:06,718 --> 04:16:08,386 SMALL SCALE SET OF PROJECTS, SO 7468 04:16:08,453 --> 04:16:09,587 THAT COULD BE SOMETHING THAT 7469 04:16:09,654 --> 04:16:12,090 COULD ON A SHORT TERM BASIS 7470 04:16:12,156 --> 04:16:14,759 INCENTIVIZE WHAT COULD BE THE 7471 04:16:14,826 --> 04:16:19,330 PUBLICATION OF NULL RESULTS, AND 7472 04:16:19,397 --> 04:16:22,533 POTENTIALLY ADDRESS LOWERING OF 7473 04:16:22,600 --> 04:16:25,369 PUBLICATION BARRIERS AS WELL. 7474 04:16:25,436 --> 04:16:26,704 REGISTRATION AND 7475 04:16:26,771 --> 04:16:28,005 PRE-REGISTRATION KEEPS COMING UP 7476 04:16:28,072 --> 04:16:29,307 AS A SOLUTION. 7477 04:16:29,373 --> 04:16:31,075 I'M FRUSTRATED BECAUSE I DON'T 7478 04:16:31,142 --> 04:16:32,944 SEE HOW THAT'S APPLICABLE TO 7479 04:16:33,010 --> 04:16:34,912 OBSERVATIONAL STUDIES WHICH IS 7480 04:16:34,979 --> 04:16:37,849 OFTEN WHAT STARTS OUT A LOT OF 7481 04:16:37,915 --> 04:16:40,485 OUR BIOLOGICAL RESEARCH. 7482 04:16:40,551 --> 04:16:41,886 AND SO I THINK WHAT I WOULD 7483 04:16:41,953 --> 04:16:45,323 APPRECIATE OVER THE NEXT TWO 7484 04:16:45,389 --> 04:16:48,359 DAYS IS WHAT A VERSION OF 7485 04:16:48,426 --> 04:16:50,962 PRE-REGISTRATION COULD LOOK LIKE 7486 04:16:51,028 --> 04:16:52,830 FOR OBSERVATIONAL STUDIES. 7487 04:16:52,897 --> 04:16:55,066 ONE THING I MIGHT PRESENT AS 7488 04:16:55,133 --> 04:16:57,335 OPTION IDEA OF STANDARDIZING OR 7489 04:16:57,401 --> 04:16:58,369 EMPHASIZING EXPERIMENTAL DESIGN 7490 04:16:58,436 --> 04:17:00,104 BEFORE YOU START YOUR 7491 04:17:00,171 --> 04:17:01,739 EXPERIMENTS, AND THAT BEING A 7492 04:17:01,806 --> 04:17:03,007 CRITICAL PART OF TRAINING. 7493 04:17:03,074 --> 04:17:06,043 THAT'S A NEW CLASS WE STARTED AT 7494 04:17:06,110 --> 04:17:13,151 U.K. SANTA -- UC SANTA CRUZ, 7495 04:17:13,217 --> 04:17:15,119 GRAD TRAINING, THAT MIGHT BE 7496 04:17:15,186 --> 04:17:17,288 BETTER SUITED FOR THE 7497 04:17:17,355 --> 04:17:19,390 OBSERVATIONAL BIOLOGICAL SCIENCE 7498 04:17:19,457 --> 04:17:21,692 THAN PRE-REGISTRATION MIGHT BE. 7499 04:17:21,759 --> 04:17:24,095 >> YEAH, THANK YOU. 7500 04:17:24,162 --> 04:17:26,831 ALL GREAT POINTS. 7501 04:17:26,898 --> 04:17:28,166 FIRST OF ALL, CONCERNING 7502 04:17:28,232 --> 04:17:29,467 STANDARDIZATION OF REVIEW 7503 04:17:29,534 --> 04:17:30,802 CRITERIA WE TAKE SERIOUSLY WITH 7504 04:17:30,868 --> 04:17:32,103 REGISTERED REPORTS. 7505 04:17:32,170 --> 04:17:38,342 WE HAVE SIX STAGE 1 AND STAGE 2 7506 04:17:38,409 --> 04:17:41,279 CRITERIA WHICH MUST BE MET TO 7507 04:17:41,345 --> 04:17:43,047 ACHIEVE STAGE 2 ACCEPTANCE, WE 7508 04:17:43,114 --> 04:17:47,418 ASK ASK REVIEWERS. 7509 04:17:47,485 --> 04:17:49,520 IF THEY GO BEYOND THEM, AND THEY 7510 04:17:49,587 --> 04:17:51,589 DO, IT'S THE JOB OF RECOMMENDER 7511 04:17:51,656 --> 04:17:52,924 OR EDITOR TO MAKE SURE 7512 04:17:52,990 --> 04:17:53,825 SUBMISSION IS ONLY EVALUATED 7513 04:17:53,891 --> 04:17:56,294 WITHIN THE SCOPE OF THE 7514 04:17:56,360 --> 04:17:57,662 CRITERIA. 7515 04:17:57,728 --> 04:18:00,998 AND RECOMMENDED AT PCI 7516 04:18:01,065 --> 04:18:02,333 REGISTERED REPORTS DO AN EXAM 7517 04:18:02,400 --> 04:18:04,836 BEFORE THEY HANDLE THE FIRST 7518 04:18:04,902 --> 04:18:08,973 SUBMISSION, TAKES AN HOUR, 95%, 7519 04:18:09,040 --> 04:18:10,308 REACTIVATES ONLY EXAM ANXIETY IN 7520 04:18:10,374 --> 04:18:11,509 PEOPLE LONG PAST DOING THAT, 7521 04:18:11,576 --> 04:18:13,578 WHICH IS KIND OF FUN. 7522 04:18:13,644 --> 04:18:17,415 ON THE SECOND POINT WE GET A LOT 7523 04:18:17,481 --> 04:18:18,749 OF OBSERVATIONAL STUDIES 7524 04:18:18,816 --> 04:18:21,352 SUBMITTED TO ROYAL SOCIETY OPEN 7525 04:18:21,419 --> 04:18:24,322 SCIENCE OR TO PCIRR. 7526 04:18:24,388 --> 04:18:27,925 AND I THINK IN GENERAL THE 7527 04:18:27,992 --> 04:18:28,893 FORMAT, REGISTERED REPORTS 7528 04:18:28,960 --> 04:18:30,094 FORMAT CAN BE VERSATILE. 7529 04:18:30,161 --> 04:18:32,029 IT'S ALWAYS UP TO THE COMMUNITY 7530 04:18:32,096 --> 04:18:34,432 TO SHAPE IT ACCORDING TO WHAT IT 7531 04:18:34,498 --> 04:18:36,467 NEEDS TO LOOK LIKE. 7532 04:18:36,534 --> 04:18:37,468 YOU'VE GOT THAT PRE-STUDY 7533 04:18:37,535 --> 04:18:40,104 EVALUATION AND THEN YOU'VE GOT 7534 04:18:40,171 --> 04:18:41,072 IN PRINCIPLES ACCEPTANCE 7535 04:18:41,138 --> 04:18:42,073 REGARDLESS OF RESULTS. 7536 04:18:42,139 --> 04:18:50,147 PROVIDED YOU CAN DEFINE A PLAN, 7537 04:18:50,214 --> 04:18:51,082 THAT SOMEONE ELSE CAN EVALUATE, 7538 04:18:51,148 --> 04:18:53,117 PROVIDED CAN YOU ARTICULATE THAT 7539 04:18:53,184 --> 04:18:55,186 IT'S SUITABLE IN ANY AREA. 7540 04:18:55,253 --> 04:18:56,721 WE GET SUBMISSIONS ACROSS THE 7541 04:18:56,787 --> 04:19:01,659 BOARD, SUBMISSIONS AND MORE IN 7542 04:19:01,726 --> 04:19:04,695 DRAMA STUDIES, NANOSCIENCE, 7543 04:19:04,762 --> 04:19:05,229 NEUROSCIENCE, EVERYTHING. 7544 04:19:05,296 --> 04:19:06,063 I THINK I'D BE INTERESTED TO 7545 04:19:06,130 --> 04:19:07,899 TALK TO YOU LATER ABOUT THE 7546 04:19:07,965 --> 04:19:09,767 CHALLENGES YOU FACE WITH 7547 04:19:09,834 --> 04:19:10,835 OBSERVATIONAL BIOLOGICAL 7548 04:19:10,902 --> 04:19:12,937 RESEARCH BUT I'M SURE THEY ARE 7549 04:19:13,004 --> 04:19:22,013 SURMOUNTABLE TO MAKE REGISTERED 7550 04:19:22,079 --> 04:19:23,581 REPORTS FORMATS COMPATIBLE. 7551 04:19:23,648 --> 04:19:24,982 >> THERE'S A NICE PIECE WRITTEN, 7552 04:19:25,049 --> 04:19:31,255 I HAVEN'T RED THAT IN PLOS 7553 04:19:31,322 --> 04:19:31,789 BIOLOGY. 7554 04:19:31,856 --> 04:19:32,657 PRE-REGISTRATION FOR EXPLORATORY 7555 04:19:32,723 --> 04:19:35,192 SCIENCE, I NEED TO READ THAT. 7556 04:19:35,259 --> 04:19:37,161 >> YEAH, AND THAT'S GREAT. 7557 04:19:37,228 --> 04:19:39,664 WE CAN -- YOU CAN -- A 7558 04:19:39,730 --> 04:19:41,198 REGISTERED REPORT CAN BE 7559 04:19:41,265 --> 04:19:41,465 ANYTHING. 7560 04:19:41,532 --> 04:19:42,833 THE MORE DETAILED IT IS, 7561 04:19:42,900 --> 04:19:45,670 OBVIOUSLY THE BETTER. 7562 04:19:45,736 --> 04:19:47,438 ONE OF THE CHALLENGES YOU DON'T 7563 04:19:47,505 --> 04:19:49,073 QUITE NOW WHAT YOU'RE GOING TO 7564 04:19:49,140 --> 04:19:50,174 BE DOING UNTIL IT'S DONE. 7565 04:19:50,241 --> 04:19:52,476 AND THAT'S FINE. 7566 04:19:52,543 --> 04:19:54,145 THERE'S ALWAYS A PLACE OUTSIDE 7567 04:19:54,211 --> 04:19:55,246 THE REGISTERED REPORTS FORMAT. 7568 04:19:55,313 --> 04:19:58,182 FOR A LOT OF WORK STILL 7569 04:19:58,249 --> 04:19:59,951 OBSERVATIONAL AND HAS AN 7570 04:20:00,017 --> 04:20:01,585 EXPLORER TO ANGLE, THERE'S STILL 7571 04:20:01,652 --> 04:20:04,221 A VERY DETAILED PLAN THAT GOES 7572 04:20:04,288 --> 04:20:05,856 INTO IMPLEMENTING THE RESEARCH. 7573 04:20:05,923 --> 04:20:10,661 EVEN IF THERE AREN'T STRONG 7574 04:20:10,728 --> 04:20:11,162 HYPOTHESES. 7575 04:20:11,228 --> 04:20:12,296 I'VE HANDLED SUBMISSIONS AT 7576 04:20:12,363 --> 04:20:13,731 ROYAL SOCIETY OPEN SCIENCE WHERE 7577 04:20:13,798 --> 04:20:16,000 IT WAS PURE OBSERVATION WITH NO 7578 04:20:16,067 --> 04:20:18,536 HYPOTHESES, AND YET A CLEAR 7579 04:20:18,602 --> 04:20:20,504 QUESTION, A VERY CLEAR ANALYSIS 7580 04:20:20,571 --> 04:20:25,876 PLAN AND CLEAR STATEMENT HOW 7581 04:20:25,943 --> 04:20:29,280 THEY WOULD INTERPRET OUTCOMES IS 7582 04:20:29,347 --> 04:20:30,247 IMPORTANT TO AVOID INTERPRETIVE 7583 04:20:30,314 --> 04:20:30,614 BIAS. 7584 04:20:30,681 --> 04:20:32,416 YOU CAN MAKE THE FORMAT WORK FOR 7585 04:20:32,483 --> 04:20:34,418 ANYTHING IF YOU WANT TO. 7586 04:20:34,485 --> 04:20:38,456 >> JUST TO BRIEFLY ADD TO THAT, 7587 04:20:38,522 --> 04:20:40,324 YOU KNOW, GRANTS REVIEW 7588 04:20:40,391 --> 04:20:43,728 COMMITTEES ARE IN SOME CASES 7589 04:20:43,794 --> 04:20:45,029 REVIEWING OBSERVATIONAL STUDIES. 7590 04:20:45,096 --> 04:20:46,897 THERE'S PRECEDENT OF HOW YOU'RE 7591 04:20:46,964 --> 04:20:48,866 GOING TO REVIEW A STUDY PLAN 7592 04:20:48,933 --> 04:20:51,068 WITHOUT DATA, YOU KNOW, THAT 7593 04:20:51,135 --> 04:20:52,069 WORKS ACROSS STUDY TYPES AS 7594 04:20:52,136 --> 04:20:55,873 WELL. 7595 04:20:55,940 --> 04:20:58,943 >> WHEN YOU'RE PLANNING FOR THAT 7596 04:20:59,010 --> 04:21:01,779 KIND OF REGISTERED REPORT YOU 7597 04:21:01,846 --> 04:21:05,383 CAN HAVE A DECISION TREE TRYING 7598 04:21:05,449 --> 04:21:06,717 TO IMAGINE POSSIBLE OUTCOME AND 7599 04:21:06,784 --> 04:21:08,386 THEN YOU TAKE A DECISION AND 7600 04:21:08,452 --> 04:21:10,654 TAKE A BRANCH AND A BRANCH, SO 7601 04:21:10,721 --> 04:21:12,189 YOU DON'T HAVE TO EXPLORE ALL 7602 04:21:12,256 --> 04:21:13,924 THE BRANCHES BECAUSE AT SOME 7603 04:21:13,991 --> 04:21:15,026 POINT OF YOUR OBSERVATION YOU 7604 04:21:15,092 --> 04:21:18,095 WILL HAVE TO MAKE CHOICES BUT IF 7605 04:21:18,162 --> 04:21:20,631 YOU ALREADY INTRODUCE THE 7606 04:21:20,698 --> 04:21:22,266 POSSIBILITIES THAT YOU THINK MAY 7607 04:21:22,333 --> 04:21:24,368 BE APPLICABLE TO YOUR PROJECT, 7608 04:21:24,435 --> 04:21:26,537 THEN IT IS REVIEWED LIKE THIS, 7609 04:21:26,604 --> 04:21:28,272 OF COURSE YOU NEVER KNOW WHERE A 7610 04:21:28,339 --> 04:21:30,875 PROJECT IS GOING TO GO BUT THEN 7611 04:21:30,941 --> 04:21:34,412 IF YOU DIVERT TOO MUCH FROM THE 7612 04:21:34,478 --> 04:21:35,946 REGISTERED, THAT'S OUR RULE, THE 7613 04:21:36,013 --> 04:21:38,149 PEOPLE BECOMES LIKE A NORMAL 7614 04:21:38,215 --> 04:21:40,785 PAPER BECAUSE IT WAS NOT 7615 04:21:40,851 --> 04:21:41,318 EVALUATED AS SUCH. 7616 04:21:41,385 --> 04:21:43,187 IF YOU TRY TO IMAGINE 7617 04:21:43,254 --> 04:21:44,155 EVERYTHING, THEN YOU'RE FINE. 7618 04:21:44,221 --> 04:21:45,689 >> YOU CAN ALSO HAVE NO ANALYSIS 7619 04:21:45,756 --> 04:21:46,357 PLAN AT ALL. 7620 04:21:46,424 --> 04:21:48,659 THIS IS GOING TO BLOW YOUR MIND. 7621 04:21:48,726 --> 04:21:50,861 YOU CAN HAVE A BLIND ANALYST, 7622 04:21:50,928 --> 04:21:51,162 RIGHT? 7623 04:21:51,228 --> 04:21:57,068 I'VE HANDLED A FEW REGISTERED 7624 04:21:57,134 --> 04:21:58,035 REPORTS INSTEAD OF SPECIFYING 7625 04:21:58,102 --> 04:22:01,038 THEY SAID WE'LL HAVE BLINDED 7626 04:22:01,105 --> 04:22:03,340 ANALYST THAT DOESN'T KNOW THE 7627 04:22:03,407 --> 04:22:04,975 DATA. 7628 04:22:05,042 --> 04:22:07,945 THEY HAVE LISTED VARIABLES, AND 7629 04:22:08,012 --> 04:22:08,712 THEY WILL IMPLEMENT ANALYSIS ON 7630 04:22:08,779 --> 04:22:11,449 THE DATA IN A WAY THAT IS 7631 04:22:11,515 --> 04:22:12,883 COMPLETELY PROTECTED AGAINST 7632 04:22:12,950 --> 04:22:13,250 BIAS. 7633 04:22:13,317 --> 04:22:15,386 IT'S DONE SO WITHOUT DETAILED 7634 04:22:15,453 --> 04:22:15,886 PRE-ANALYSIS PLAN. 7635 04:22:15,953 --> 04:22:19,190 WHICH I THINK IS A FASCINATING 7636 04:22:19,256 --> 04:22:25,663 APPROACH, QUITE POPULAR IN 7637 04:22:25,729 --> 04:22:26,063 PHYSICS. 7638 04:22:26,130 --> 04:22:27,531 >> I WANT TO CONTINUE THE THEME 7639 04:22:27,598 --> 04:22:28,065 ON INCENTIVES. 7640 04:22:28,132 --> 04:22:29,834 THE MORE I LISTEN TO TALKS AND 7641 04:22:29,900 --> 04:22:31,802 Q&A THE MORE I THINK THIS IS 7642 04:22:31,869 --> 04:22:35,539 JUST AS MUCH ABOUT CREDIT-WISE 7643 04:22:35,606 --> 04:22:37,708 AS PUBLICATION BIAS. 7644 04:22:37,775 --> 04:22:39,343 MAYBE WE MANAGE TO PUBLISH ALL 7645 04:22:39,410 --> 04:22:40,644 THESE NULL STUDIES. 7646 04:22:40,711 --> 04:22:45,683 IT COULD STILL LEAD TO PUBLISH 7647 04:22:45,749 --> 04:22:48,085 AND PERISH PARADIGM. 7648 04:22:48,152 --> 04:22:49,420 IT'S DISCOUNTED IF YOU PUBLISH 7649 04:22:49,487 --> 04:22:50,621 NULL STUDIES IN THE JOURNAL NOT 7650 04:22:50,688 --> 04:22:55,993 HIGH ON THE FOOD CHAIN. 7651 04:22:56,060 --> 04:23:03,067 I WANTED TO RUN AND IDEA PAST 7652 04:23:03,134 --> 04:23:04,568 YOU, WHETHER WE CAN INCENT -- 7653 04:23:04,635 --> 04:23:06,303 SCIENTISTS MAY NOT BE INTERESTED 7654 04:23:06,370 --> 04:23:08,372 IN PUBLISHING THESE STUDIES SO 7655 04:23:08,439 --> 04:23:15,579 WHAT IF WE COULD INCENTIVIZE 7656 04:23:15,646 --> 04:23:17,348 PUBLISHING NULL STUDIES, THROUGH 7657 04:23:17,414 --> 04:23:19,416 TRAINING AND MENTORING, WE 7658 04:23:19,483 --> 04:23:20,718 EXPECT -- I'M NOT TALKING ABOUT 7659 04:23:20,784 --> 04:23:23,687 WHAT WE HEARD EARLIER, OH, TRAIN 7660 04:23:23,754 --> 04:23:25,623 THE NEXT GENERATION, I'M TALKING 7661 04:23:25,689 --> 04:23:26,824 ABOUT EXPECTING FROM LAB HEADS 7662 04:23:26,891 --> 04:23:30,127 THAT PART OF TRAINING AND 7663 04:23:30,194 --> 04:23:31,428 MENTORSHIP IS THAT TRAINEES ARE 7664 04:23:31,495 --> 04:23:32,763 PREPARED FOR THE NEXT CAREER 7665 04:23:32,830 --> 04:23:37,368 PHASE, WHATEVER THEY WANT TO DO, 7666 04:23:37,434 --> 04:23:39,470 BUT THAT INCLUDES OUTPUT FROM 7667 04:23:39,537 --> 04:23:40,337 TRAINING PERIOD. 7668 04:23:40,404 --> 04:23:43,507 IF THAT IS AN EXPECTATION, AND 7669 04:23:43,574 --> 04:23:45,476 WE ACTUALLY CHECK IT, WE ASK FOR 7670 04:23:45,543 --> 04:23:47,978 IT, IF EVERY TRAINEE HAS TO COME 7671 04:23:48,045 --> 04:23:51,715 OUT WITH SOMETHING, THAT MAY 7672 04:23:51,782 --> 04:23:52,683 HELP TO INCENTIVIZE PUBLISHING 7673 04:23:52,750 --> 04:23:55,653 THEIR PROJECT AS WELL BECAUSE 7674 04:23:55,719 --> 04:23:58,155 THE PUBLISH OR PERISH PARADIGM 7675 04:23:58,222 --> 04:24:00,457 WORKS VERY OFTEN IN BIG LABS 7676 04:24:00,524 --> 04:24:05,296 WHERE YOU HAVE SOME SUCCESSFUL 7677 04:24:05,362 --> 04:24:07,998 TRAINEES AND THE REST -- IF WE 7678 04:24:08,065 --> 04:24:11,535 EXPECT ALL TRAINEES HAVE A SHOT 7679 04:24:11,602 --> 04:24:17,208 AT SUCCESS, THAT THEY PRODUCE 7680 04:24:17,274 --> 04:24:19,276 SOMETHING, AN EXPERIMENT IS ONLY 7681 04:24:19,343 --> 04:24:20,811 DONE WHEN IT'S PUBLISHED, EVERY 7682 04:24:20,878 --> 04:24:22,346 TRAINING SHOULD LEAD WITH 7683 04:24:22,413 --> 04:24:23,681 SOMETHING THAT IS PUBLISHED, 7684 04:24:23,747 --> 04:24:26,717 THAT COULD BE AN INCENTIVE FOR 7685 04:24:26,784 --> 04:24:27,218 SCIENTISTS. 7686 04:24:27,284 --> 04:24:28,252 >> I THINK IT'S AN INTERESTING 7687 04:24:28,319 --> 04:24:28,719 IDEA. 7688 04:24:28,786 --> 04:24:30,087 I JUST DON'T THINK IT WILL WORK 7689 04:24:30,154 --> 04:24:32,756 BECAUSE I DON'T THINK WE CAN -- 7690 04:24:32,823 --> 04:24:35,793 WHO IS THIS "WE" DOING THIS? 7691 04:24:35,859 --> 04:24:37,161 >> OH, SORRY. 7692 04:24:37,228 --> 04:24:38,162 INSTITUTIONS AND FUNDERS. 7693 04:24:38,229 --> 04:24:39,063 >> RIGHT, RIGHT. 7694 04:24:39,129 --> 04:24:41,498 >> IT WILL BE INSTITUTIONS AND 7695 04:24:41,565 --> 04:24:42,933 FUNDERS, THAT EXPECTATION. 7696 04:24:43,000 --> 04:24:45,970 WE'RE LOOKING AT YOUR TRAINING 7697 04:24:46,036 --> 04:24:48,672 AND MENTORING EFFORTS. 7698 04:24:55,679 --> 04:25:05,222 (INAUDIBLE) 7699 04:25:05,289 --> 04:25:07,391 >> USE THE MICROPHONES. 7700 04:25:07,458 --> 04:25:09,026 >> ISN'T THAT JUST SUPPORTING 7701 04:25:09,093 --> 04:25:10,160 PUBLISH OR PERISH CULTURE? 7702 04:25:10,227 --> 04:25:11,662 EVERYBODY HAS TO PUBLISH? 7703 04:25:11,729 --> 04:25:13,330 THIS GOES WITH THE IDEAS, WHAT 7704 04:25:13,397 --> 04:25:15,833 WE NEED IS LESS RESEARCH, WE 7705 04:25:15,899 --> 04:25:17,134 JUST NEED BETTER RESEARCH. 7706 04:25:17,201 --> 04:25:21,205 AND THAT'S A CHALLENGE I FOUND, 7707 04:25:21,272 --> 04:25:23,274 ALSO RESIDENCY PROGRAM WHO IS 7708 04:25:23,340 --> 04:25:24,908 WANT EVERY NEUROLOGY RESIDENT TO 7709 04:25:24,975 --> 04:25:26,877 PUBLISH A STUDY, MOST WILL NEVER 7710 04:25:26,944 --> 04:25:28,212 PUBLISH ANYTHING AGAIN, MAYBE 7711 04:25:28,279 --> 04:25:29,546 THEY SHOULDN'T BE FOCUSED ON 7712 04:25:29,613 --> 04:25:29,913 THAT. 7713 04:25:29,980 --> 04:25:33,350 WE SHOULD FOCUS ON GOOD QUALITY 7714 04:25:33,417 --> 04:25:35,119 RESEARCH RATHER THAN JUST KEEP 7715 04:25:35,185 --> 04:25:39,256 PROMOTING THIS PUBLISH OR PERISH 7716 04:25:39,323 --> 04:25:39,490 IDEA. 7717 04:25:39,556 --> 04:25:42,226 >> I THINK IT'S NOT ABOUT THE 7718 04:25:42,293 --> 04:25:43,460 QUANTITY NUMBER OF PUBLICATIONS, 7719 04:25:43,527 --> 04:25:45,496 LIKE IT'S ABOUT RESPONSIBILITY. 7720 04:25:45,562 --> 04:25:48,232 WHEN I AM USING EXPLICITLY THE 7721 04:25:48,299 --> 04:25:56,140 PUBLIC MONEY, I'M RESPONSIBLE TO 7722 04:25:56,206 --> 04:25:58,776 DISSEMINATE THAT. 7723 04:25:58,842 --> 04:25:59,843 THE RECENT ARTICLE, THERE'S A 7724 04:25:59,910 --> 04:26:02,379 LOT OF JUNK OUT THERE. 7725 04:26:02,446 --> 04:26:05,115 BUT OF COURSE PEOPLE WOULD ALSO 7726 04:26:05,182 --> 04:26:07,618 ARGUE YOU'RE GIVING ME THIS 7727 04:26:07,685 --> 04:26:11,989 MONEY, AND THAT EXPERIMENT DOES 7728 04:26:12,056 --> 04:26:16,593 NOT GO IN MY FILE DRAWER. 7729 04:26:16,660 --> 04:26:22,066 THE PAPER WE PRESENTED, IF WE 7730 04:26:22,132 --> 04:26:24,368 DIDN'T WANT TO SHOW THE RESULTS, 7731 04:26:24,435 --> 04:26:26,003 THERE WOULD BE CONTINUATION OF 7732 04:26:26,070 --> 04:26:31,508 PUBLICATIONS THAT WOULD CONTINUE 7733 04:26:31,575 --> 04:26:33,344 THAT. 7734 04:26:33,410 --> 04:26:35,713 YOU'RE CONTRIBUTING TO THAT. 7735 04:26:35,779 --> 04:26:37,815 >> I AGREE. 7736 04:26:37,881 --> 04:26:39,583 THAT'S THE IDEA BEHIND THE NEW 7737 04:26:39,650 --> 04:26:40,351 MANDATES, RIGHT? 7738 04:26:40,417 --> 04:26:42,086 IF YOUR WORK IS FUNDED BY NIH IT 7739 04:26:42,152 --> 04:26:45,022 HAS TO BE PUBLISHED IN A WAY 7740 04:26:45,089 --> 04:26:47,691 THAT'S OPEN ACCESS, AVAILABLE 7741 04:26:47,758 --> 04:26:48,559 IMMEDIATELY FOR EVERYBODY 7742 04:26:48,625 --> 04:26:50,427 BECAUSE YOU'RE A STEWARD FOR THE 7743 04:26:50,494 --> 04:26:52,830 TAXPAYER'S MONEY AND YOU HAVE TO 7744 04:26:52,896 --> 04:26:54,465 MAKE YOUR RESULTS PUBLIC. 7745 04:26:54,531 --> 04:26:56,767 >> THIS IS I THINK WE ALL HAD 7746 04:26:56,834 --> 04:27:00,704 SIMILAR LINES OF THOUGHT HERE 7747 04:27:00,771 --> 04:27:05,209 BUT THE WHOLE IDEA OF 7748 04:27:05,275 --> 04:27:06,410 PRE-REGISTRATION AND 7749 04:27:06,477 --> 04:27:08,579 REGISTRATION IS CLEAR, BUT I DO 7750 04:27:08,645 --> 04:27:10,881 SEE, AND I HAVE A ROLE IN 7751 04:27:10,948 --> 04:27:12,316 RESEARCH QUALITY, INTEGRITY AT 7752 04:27:12,383 --> 04:27:18,288 MY INSTITUTION AS WELL, I SEE A 7753 04:27:18,355 --> 04:27:20,257 STARK DIFFERENCE IN CLINICAL 7754 04:27:20,324 --> 04:27:22,559 RESEARCH OR PRE-CLINICAL, WE 7755 04:27:22,626 --> 04:27:23,961 DON'T SEE STUDENTS, TRAINEES, 7756 04:27:24,027 --> 04:27:26,597 GRAD STUDENTS, WHAT NOT, AND 7757 04:27:26,663 --> 04:27:27,164 MENTORS INVOLVED IN 7758 04:27:27,231 --> 04:27:27,798 PRE-REGISTRATION, REGISTRATION 7759 04:27:27,865 --> 04:27:28,098 EARLY ON. 7760 04:27:28,165 --> 04:27:30,534 ONE OF THE THINGS THAT COULD 7761 04:27:30,601 --> 04:27:32,536 HELP WITH IS ALSO THIS 7762 04:27:32,603 --> 04:27:33,604 NEVER-ENDING PLAN TO PUBLISH. 7763 04:27:33,670 --> 04:27:36,573 YOU SEE THIS GO ON, ON ONE MORE 7764 04:27:36,640 --> 04:27:37,975 EXPERIMENT, WE NEED TO SOLVE 7765 04:27:38,041 --> 04:27:39,076 THIS OTHER QUESTION, OH, 7766 04:27:39,143 --> 04:27:40,411 REVIEWERS ARE GOING ASK FOR 7767 04:27:40,477 --> 04:27:42,246 THIS, IT KEEPS GOING. 7768 04:27:42,312 --> 04:27:46,283 IT IS A SERIOUS DETRIMENT TO 7769 04:27:46,350 --> 04:27:48,485 TRAINEES, NOT ABLE TO GET WORK 7770 04:27:48,552 --> 04:27:51,555 PUBLISH, HELD TO THEIR 7771 04:27:51,622 --> 04:27:53,323 MENTOR/P.I. WANTS THEIR PAPER IN 7772 04:27:53,390 --> 04:27:54,324 "CELL" AND "NATURE." 7773 04:27:54,391 --> 04:27:56,493 NOT EVERYBODY IS GOING TO GET IN 7774 04:27:56,560 --> 04:27:59,229 "CELL" OR "NATURE," OTHERWISE WE 7775 04:27:59,296 --> 04:28:00,431 WOULDN'T HAVE THIS MANY OTHER 7776 04:28:00,497 --> 04:28:02,533 PUBLISHERS HERE, SOMEBODY IS 7777 04:28:02,599 --> 04:28:04,501 PUBLISHING IN THEIR JOURNALS. 7778 04:28:04,568 --> 04:28:07,104 GETTING SOMETHING OUT ON WHAT 7779 04:28:07,171 --> 04:28:10,741 HAS BEEN SAID HERE PUTTING FORTH 7780 04:28:10,808 --> 04:28:12,476 WHAT THEIR BASIS IS, 7781 04:28:12,543 --> 04:28:13,110 PRE-REGISTRATION, SOMETHING 7782 04:28:13,177 --> 04:28:15,345 THESE COMES INTO A DIFFERENT 7783 04:28:15,412 --> 04:28:16,313 FORMAT EVEN IF AT THE 7784 04:28:16,380 --> 04:28:20,250 INSTITUTION TO HAVE A 7785 04:28:20,317 --> 04:28:20,551 REPOSITORY. 7786 04:28:20,617 --> 04:28:22,886 SOMETHING THAT COULD GIVE THEM A 7787 04:28:22,953 --> 04:28:24,721 PROTECTION AND BE USED TO LOOK 7788 04:28:24,788 --> 04:28:25,823 FORWARD TO EXPERIMENTAL DESIGN, 7789 04:28:25,889 --> 04:28:27,458 WHAT SHOULD BE PUT FORWARD. 7790 04:28:27,524 --> 04:28:30,327 I OTHER THING I WANT TO SAY, 7791 04:28:30,394 --> 04:28:33,831 MANY HERE I CAN SAY THIS AS A 7792 04:28:33,897 --> 04:28:37,134 TENURED ENDOWED PROFESSOR, THERE 7793 04:28:37,201 --> 04:28:38,869 ARE SO MANY MYTHS ABOUT 7794 04:28:38,936 --> 04:28:40,304 PROMOTION AND TENURE. 7795 04:28:40,370 --> 04:28:44,141 WHEN I WAS ASSISTANT PROFESSOR 7796 04:28:44,208 --> 04:28:46,343 YOUR GUIDE SAID HAVE 15 7797 04:28:46,410 --> 04:28:46,677 PUBLICATIONS. 7798 04:28:46,743 --> 04:28:47,845 YOU NEED DOUBLE. 7799 04:28:47,911 --> 04:28:51,782 IT SAYS 15, WHY DO I NEED 30? 7800 04:28:51,849 --> 04:28:53,083 IF YOU PUBLIC NEGATIVE RESULTS 7801 04:28:53,150 --> 04:28:54,952 YOU'RE NOT GOING TO GET 7802 04:28:55,018 --> 04:28:55,419 PROMOTED. 7803 04:28:55,486 --> 04:28:57,921 IT MIGHT BE TRY. 7804 04:28:57,988 --> 04:29:00,090 PROBABLY IT'S IN BETWEEN, YOU'RE 7805 04:29:00,157 --> 04:29:02,392 ALSO GOING TO HAVE, IF HAVE YOU 7806 04:29:02,459 --> 04:29:03,594 TOP NOTCH PUBLICATIONS IN 7807 04:29:03,660 --> 04:29:05,028 "CELL," YOU HAVE TO HAVE 7808 04:29:05,095 --> 04:29:06,563 NEGATIVE RESULTS ALONG THE LINE, 7809 04:29:06,630 --> 04:29:07,631 OR SOMETHING THAT DIDN'T WORK. 7810 04:29:07,698 --> 04:29:09,066 IF YOU'RE TELLING ME YOU'RE 7811 04:29:09,132 --> 04:29:10,634 GOING TO WAIT UNTIL FOUR YEARS 7812 04:29:10,701 --> 04:29:12,269 AND PUBLISH NOTHING IN BETWEEN, 7813 04:29:12,336 --> 04:29:15,539 ONLY COME OUT WITH THAT, THIS IS 7814 04:29:15,606 --> 04:29:19,476 SOMETHING IN HERE IS NOT RIGHT, 7815 04:29:19,543 --> 04:29:21,678 THIS IS NOT JUST PUBLISHERS, 7816 04:29:21,745 --> 04:29:23,213 INSTITUTION, NOT JUST FUNDERS. 7817 04:29:23,280 --> 04:29:24,114 IT'S ACROSS THE BOARD WE 7818 04:29:24,181 --> 04:29:26,250 CONTINUE A LOT OF THIS FOLKLORE. 7819 04:29:26,316 --> 04:29:28,785 WHEN YOU DIG IN, SOME IS JUST 7820 04:29:28,852 --> 04:29:30,420 BIAS AND MYTHS WE CARRY ON. 7821 04:29:30,487 --> 04:29:35,459 WE HAVE TO DO MYTH BUSTING ALONG 7822 04:29:35,526 --> 04:29:39,930 THE WAY TOO. 7823 04:29:39,997 --> 04:29:41,031 >> GREAT COMMENT. 7824 04:29:41,098 --> 04:29:43,033 WITH A REGISTERED REPORT YOU'LL 7825 04:29:43,100 --> 04:29:46,503 NEVER BE EXPECTED TO DO EXTRA 7826 04:29:46,570 --> 04:29:48,739 EXPERIMENT AT STAGE 2. 7827 04:29:48,805 --> 04:29:52,409 AUTHORS BUY UNCERTAINTY, MUCH 7828 04:29:52,476 --> 04:29:54,711 HAS BEEN WRITTEN, REGISTERED 7829 04:29:54,778 --> 04:29:56,580 REPORTS PROCESS MAKES THE 7830 04:29:56,647 --> 04:29:58,215 TIMELINE MORE PREDICTABLE. 7831 04:29:58,282 --> 04:30:02,719 BECAUSE YOU KNOW IF YOU FOLLOW 7832 04:30:02,786 --> 04:30:05,122 YOUR PLAN, YOU'LL GET ACCEPTED. 7833 04:30:05,188 --> 04:30:05,589 REVIEWERS SAY ANOTHER 7834 04:30:05,656 --> 04:30:10,294 EXPERIMENT, YOU WOULD BE GREAT. 7835 04:30:10,360 --> 04:30:11,028 SURE, IT'S GREAT. 7836 04:30:11,094 --> 04:30:16,934 THERE HAS TO BE AN INFLUENCE. 7837 04:30:17,000 --> 04:30:24,608 IT'S HUGELY EMPOWERING FOR 7838 04:30:24,675 --> 04:30:25,375 CAREER SCIENTISTS. 7839 04:30:25,442 --> 04:30:27,010 >> I WORRY WITH REGISTERED 7840 04:30:27,077 --> 04:30:28,845 REPORTS, I'M FOR THEM, A BIG 7841 04:30:28,912 --> 04:30:31,481 ANSWER TO WHAT WE'RE FACING 7842 04:30:31,548 --> 04:30:32,249 HERE. 7843 04:30:32,316 --> 04:30:34,551 BUT IT'S A BARRIER, ESPECIALLY 7844 04:30:34,618 --> 04:30:35,752 FOR EARLY CAREER INVESTIGATORS, 7845 04:30:35,819 --> 04:30:37,087 IT TAKES TIME. 7846 04:30:37,154 --> 04:30:39,189 IT'S A HUGE TIME COMMITMENT, AND 7847 04:30:39,256 --> 04:30:41,224 COMMITMENTS SEEM TO BE RACKING 7848 04:30:41,291 --> 04:30:44,528 UP WITH COMPLIANCE FOR ANIMAL 7849 04:30:44,595 --> 04:30:52,202 CARE AND OTHER, YOU KNOW, FOR 7850 04:30:52,269 --> 04:31:00,611 HUMAN RESEARCH, ET CETERA. 7851 04:31:00,677 --> 04:31:02,145 NOW YOU'RE SAYING I CAN'T BE 7852 04:31:02,212 --> 04:31:03,680 ASKED FOR EXTRA EXPERIMENTS, 7853 04:31:03,747 --> 04:31:07,985 THAT ISN'T GOING TO DRAG ON, 7854 04:31:08,051 --> 04:31:09,519 THAT'S HELPFUL. 7855 04:31:09,586 --> 04:31:12,155 >> ON TIME ISSUE YOU HAVE TO GO 7856 04:31:12,222 --> 04:31:13,357 THROUGH STAGE 1 REVIEW PROCESS 7857 04:31:13,423 --> 04:31:15,125 BUT YOU SAVE TIME ON THE OTHER 7858 04:31:15,192 --> 04:31:15,292 END. 7859 04:31:15,359 --> 04:31:18,295 YOU'RE NOT GOING TO BE CHASING 7860 04:31:18,362 --> 04:31:19,229 JOURNALS, ONE AFTER ANOTHER, 7861 04:31:19,296 --> 04:31:20,797 AFTER YOU'VE RUN OUT OF FUNDING 7862 04:31:20,864 --> 04:31:22,666 AND YOU'RE TRYING TO PUBLISH 7863 04:31:22,733 --> 04:31:23,300 YOUR WORK. 7864 04:31:23,367 --> 04:31:24,935 THERE IS TIME SPENT AT THE 7865 04:31:25,002 --> 04:31:26,603 BEGINNING, WE PUT A LOT OF 7866 04:31:26,670 --> 04:31:28,338 EFFORT INTO SHORTENING THAT EVEN 7867 04:31:28,405 --> 04:31:32,943 MORE, PARTICULARLY WITH THE 7868 04:31:33,010 --> 04:31:34,277 SCHEDULED REVIEW TRACK. 7869 04:31:34,344 --> 04:31:36,113 WITH THE SCHEDULED REVIEW TRACK 7870 04:31:36,179 --> 04:31:38,315 WE START LINING UP REVIEW 7871 04:31:38,382 --> 04:31:39,850 PROCESS WHILE YOU'RE STILL 7872 04:31:39,916 --> 04:31:40,951 WRITING STAGE 1 MANUSCRIPTS. 7873 04:31:41,018 --> 04:31:43,787 WE DO THAT BASED ON A ONE-PAGE 7874 04:31:43,854 --> 04:31:46,289 PRO FORMA THAT YOU SUBMIT AS 7875 04:31:46,356 --> 04:31:47,057 AUTHOR. 7876 04:31:47,124 --> 04:31:49,226 BASED UPON THAT SNAPSHOT WE 7877 04:31:49,292 --> 04:31:53,897 RECRUIT REVIEWERS IN ADVANCE SO 7878 04:31:53,964 --> 04:31:57,768 WHEN YOU SUBMIT ON THE DUE DATE, 7879 04:31:57,834 --> 04:31:58,869 REVIEWERS ARE BOOKED IN, READY 7880 04:31:58,935 --> 04:32:01,405 TO GO, THE FIRST STAGE OF STAGE 7881 04:32:01,471 --> 04:32:02,939 1 REVIEW TAKES ON AVERAGE 18 7882 04:32:03,006 --> 04:32:09,112 DAYS. 7883 04:32:14,584 --> 04:32:18,021 >> I'VE LISTENED WITH A GROWING 7884 04:32:18,088 --> 04:32:19,690 SENSE OF FRUSTRATIONS, I HOPE WE 7885 04:32:19,756 --> 04:32:22,292 CAN LOB A BOULDER. 7886 04:32:22,359 --> 04:32:26,863 THIS IS ABOUT PUBLISHING, WE 7887 04:32:26,930 --> 04:32:28,532 HEARD SYSTEM OF MANUSCRIPTS HAS 7888 04:32:28,598 --> 04:32:31,268 GIVEN RISE IN THE FIRST PLACE. 7889 04:32:31,334 --> 04:32:32,703 I THINK THE PREMISE IS WRONG. 7890 04:32:32,769 --> 04:32:34,671 JUST TO RESPOND TO A COUPLE 7891 04:32:34,738 --> 04:32:35,839 THINGS I'VE HEARD IN THE 7892 04:32:35,906 --> 04:32:37,574 DISCUSSION, THE THING WE'RE 7893 04:32:37,641 --> 04:32:39,576 MISSING, I HAVE MISSED IN THIS 7894 04:32:39,643 --> 04:32:42,212 CONVERSATION IS RECOGNITION OF 7895 04:32:42,279 --> 04:32:43,213 HUMAN NATURE. 7896 04:32:43,280 --> 04:32:45,382 SO WE HEARD PEER REVIEW IS 7897 04:32:45,449 --> 04:32:46,483 IMPORTANT TO GIVE CREDIT, 7898 04:32:46,550 --> 04:32:48,018 ABSOLUTELY IT'S IMPORTANT TO 7899 04:32:48,085 --> 04:32:49,553 GIVE CREDIT FOR PEER REVIEW. 7900 04:32:49,619 --> 04:32:51,188 NO ONE IS GOING TO HIRE YOU 7901 04:32:51,254 --> 04:32:53,590 BECAUSE YOU'RE GREAT AT PEER 7902 04:32:53,657 --> 04:32:54,791 REVIEW, OKAY? 7903 04:32:54,858 --> 04:32:56,493 WE HEARD THERE'S CAPACITY FOR 7904 04:32:56,560 --> 04:32:57,627 WRITING NEGATIVE RESULTS IN THE 7905 04:32:57,694 --> 04:32:58,028 SYSTEM. 7906 04:32:58,095 --> 04:32:58,962 YOU SAID THIS. 7907 04:32:59,029 --> 04:33:00,497 WHERE IS THAT CAPACITY? 7908 04:33:00,564 --> 04:33:03,533 EVERYBODY I KNOW WORKING IN 7909 04:33:03,600 --> 04:33:04,735 ACADEMIA IS OVERLOADED WITH 7910 04:33:04,801 --> 04:33:05,836 WORK. 7911 04:33:05,902 --> 04:33:08,238 WE HAVE TO THINK ABOUT THAT 7912 04:33:08,305 --> 04:33:09,239 ASPECT. 7913 04:33:09,306 --> 04:33:10,874 AND NEGATIVE RESULTS, ABSOLUTELY 7914 04:33:10,941 --> 04:33:12,309 NEGATIVE RESULTS ARE IMPORTANT 7915 04:33:12,375 --> 04:33:14,711 BUT THEY ARE JUST NOT AS 7916 04:33:14,778 --> 04:33:16,780 EXCITING AS POSITIVE RESULTS. 7917 04:33:16,847 --> 04:33:18,548 OR DISCOVERING SOMETHING NEW, 7918 04:33:18,615 --> 04:33:20,817 NOBODY EVER RAN HOME TO SAY, 7919 04:33:20,884 --> 04:33:23,053 HEY, I DISCOVERED MY HYPOTHESIS 7920 04:33:23,120 --> 04:33:24,621 IS WRONG TODAY. 7921 04:33:24,688 --> 04:33:24,821 OKAY? 7922 04:33:24,888 --> 04:33:26,523 SO, YOU KNOW, WE ABSOLUTELY NEED 7923 04:33:26,590 --> 04:33:33,930 TO MAKE IT EASIER, I'M A FAN OF 7924 04:33:33,997 --> 04:33:34,698 PRE-REGISTERED REPORTS, THE 7925 04:33:34,765 --> 04:33:36,366 INNOVATION THAT MAKES A 7926 04:33:36,433 --> 04:33:37,400 DIFFERENCE, YET TO INFILTRATE 7927 04:33:37,467 --> 04:33:39,402 MANY FIELDS AND I THINK THERE 7928 04:33:39,469 --> 04:33:40,704 ARE STRUCTURAL ISSUES AROUND 7929 04:33:40,771 --> 04:33:43,907 THAT BUT I DO THINK WE NEED TO 7930 04:33:43,974 --> 04:33:45,876 INTRODUCE AN ELEMENT OF 7931 04:33:45,942 --> 04:33:47,010 COMPULSION OR OBLIGATION BECAUSE 7932 04:33:47,077 --> 04:33:51,014 YOU ARE NOT EVER GOING TO CREATE 7933 04:33:51,081 --> 04:33:53,083 A POSITIVE INCENTIVE TO PUBLISH 7934 04:33:53,150 --> 04:33:56,353 NULL RESULTS IN MY VIEW. 7935 04:33:58,855 --> 04:34:00,657 >> WHAT WE HAVE MISSING IN OUR 7936 04:34:00,724 --> 04:34:03,627 PANEL IS THE VIEW FROM THE 7937 04:34:03,693 --> 04:34:05,162 FUNDERS, OR THE PEOPLE WHO ARE 7938 04:34:05,228 --> 04:34:07,097 ON THE AWARD AND RECOGNITION 7939 04:34:07,164 --> 04:34:07,397 COMMITTEES. 7940 04:34:07,464 --> 04:34:09,733 SO THOSE ARE -- I CAN DO SO MUCH 7941 04:34:09,800 --> 04:34:11,835 AS A JOURNAL EDITOR, HAVE 7942 04:34:11,902 --> 04:34:13,036 POLICIES, MAKE IT EASIER, REDUCE 7943 04:34:13,103 --> 04:34:15,338 BARRIERS AS FAR AS WE CAN. 7944 04:34:15,405 --> 04:34:17,607 THAT'S AT THE END STAGE. 7945 04:34:17,674 --> 04:34:19,576 IT WOULD BE EXCITING TO HEAR 7946 04:34:19,643 --> 04:34:21,611 WHEN RESEARCH IS FIRST CONCEIVED 7947 04:34:21,678 --> 04:34:23,213 AND WHEN PEOPLE ARE WRITING UP 7948 04:34:23,280 --> 04:34:27,350 THE RESULTS HOW WE'RE GOING TO 7949 04:34:27,417 --> 04:34:28,919 DO THAT. 7950 04:34:28,985 --> 04:34:30,320 >> THAT'S AS GOOD OF 7951 04:34:30,387 --> 04:34:32,956 ADVERTISEMENT I CAN THINK OF FOR 7952 04:34:33,023 --> 04:34:34,357 LATER SESSIONS LATER TODAY AND 7953 04:34:34,424 --> 04:34:36,459 TOMORROW AND WORKING GROUPS. 7954 04:34:36,526 --> 04:34:38,995 WE LOOK FORWARD TO MORE 7955 04:34:39,062 --> 04:34:39,262 SESSIONS. 7956 04:34:39,329 --> 04:34:41,832 RIGHT NOW WE'RE ON BREAK, WE'LL 7957 04:34:41,898 --> 04:34:44,000 START AGAIN AT 2:45. 7958 04:34:44,067 --> 04:34:47,671 THANK YOU. 7959 04:34:47,737 --> 04:34:50,874 >> WELCOME BACK FROM THE BREAK, 7960 04:34:50,941 --> 04:34:57,514 THIS SESSION IS ABOUT 7961 04:34:57,581 --> 04:34:58,615 NON-TRADITIONAL FORMATS. 7962 04:34:58,682 --> 04:35:00,817 I'M ALEX FREEMAN, DIRECTOR OF 7963 04:35:00,884 --> 04:35:04,187 OCTOPUS. 7964 04:35:04,254 --> 04:35:07,057 RATHER THAN EXPLAINING, I PUT 7965 04:35:07,123 --> 04:35:08,358 LEAFLETS AROUND THE ROOM. 7966 04:35:08,425 --> 04:35:10,227 OBVIOUSLY ONE WAY TO GET RID OF 7967 04:35:10,293 --> 04:35:15,532 PUBLICATION BIAS IS GET RID OF 7968 04:35:15,599 --> 04:35:16,967 GATE KEEPERS, JOURNAL EDITORS. 7969 04:35:17,033 --> 04:35:18,768 AS WE'VE HEARD, THAT'S NOT THE 7970 04:35:18,835 --> 04:35:19,436 SOLUTION TO EVERYTHING BECAUSE 7971 04:35:19,502 --> 04:35:22,606 THEY ARE NOT THE ONES WHO ARE 7972 04:35:22,672 --> 04:35:24,040 CAUSING THIS PUBLICATION BIAS. 7973 04:35:24,107 --> 04:35:25,775 BUT I THINK EVERYBODY IN THIS 7974 04:35:25,842 --> 04:35:28,178 PANEL IS TALKING ABOUT THINGS 7975 04:35:28,245 --> 04:35:30,513 THAT DON'T INVOLVE BOTH INITIAL 7976 04:35:30,580 --> 04:35:31,681 GATE KEEPERS BUT ALSO WE'RE 7977 04:35:31,748 --> 04:35:38,922 TALKING ABOUT HOW TO CHANGE THE 7978 04:35:38,989 --> 04:35:41,558 MIND SETS OF RESEARCHERS WHEN WE 7979 04:35:41,625 --> 04:35:45,395 THINK ABOUT WHAT PUBLICATION IS 7980 04:35:45,462 --> 04:35:49,866 IS, WHO IS IT FOR, WHAT IS IT 7981 04:35:49,933 --> 04:35:51,701 FOR? 7982 04:35:51,768 --> 04:35:53,270 PROGRAMS LIKE OCTOPUS FIT 7983 04:35:53,336 --> 04:35:56,239 ALONGSIDE, TO DO SOMETHING 7984 04:35:56,306 --> 04:35:57,540 DIFFERENT, THAT IS TO 7985 04:35:57,607 --> 04:36:01,711 INCENTIVIZE BEST RESEARCH 7986 04:36:01,778 --> 04:36:04,748 PRACTICES ABOVE EVERYTHING ELSE. 7987 04:36:04,814 --> 04:36:13,189 OCTOPUS IS FUNDED BY UKRI AND 7988 04:36:13,256 --> 04:36:16,226 SHARES MANY FEATURES WE'LL TALK 7989 04:36:16,293 --> 04:36:18,094 ABOUT ACROSS THE PLATFORMS LIKE 7990 04:36:18,161 --> 04:36:19,195 HAVING IMMEDIATE PUBLICATIONS 7991 04:36:19,262 --> 04:36:22,399 WITHOUT THE GATE KEEPERS, SO 7992 04:36:22,465 --> 04:36:25,235 IT'S QUICK AND EASY. 7993 04:36:25,302 --> 04:36:26,403 HAS OPEN POST PUBLICATION PEER 7994 04:36:26,469 --> 04:36:27,537 REVIEW WHICH WE'LL TALK ABOUT 7995 04:36:27,604 --> 04:36:29,272 PROS AND CONS OF THAT. 7996 04:36:29,339 --> 04:36:31,808 IT HAS SMALLER UNITS OF 7997 04:36:31,875 --> 04:36:32,909 PUBLICATION SO IT DOESN'T 7998 04:36:32,976 --> 04:36:33,877 PUBLISH PAPERS. 7999 04:36:33,944 --> 04:36:37,914 AGAIN WE'LL BE TALKING ABOUT 8000 04:36:37,981 --> 04:36:38,548 MICROPUBLICATIONS BUT OCTOPUS'S 8001 04:36:38,615 --> 04:36:41,184 PUBLICATIONS I SAY IT'S A BIT 8002 04:36:41,251 --> 04:36:43,086 LIKE TAKING REGISTERED REPORTS 8003 04:36:43,153 --> 04:36:45,822 TO ULTIMATE CONCLUSION, WHICH IS 8004 04:36:45,889 --> 04:36:48,024 BREAKING UP THE PROCESS NOT THUS 8005 04:36:48,091 --> 04:36:53,697 BEFORE AND AFTER RESULTS BUT 8006 04:36:53,763 --> 04:36:56,766 DIFFERENT STAGES, EACH CAN BE 8007 04:36:56,833 --> 04:36:58,101 PUBLISHED, PEER REVIEWED, TREATS 8008 04:36:58,168 --> 04:37:02,672 PEER REVIEWS AS PUBLICATION IN 8009 04:37:02,739 --> 04:37:03,540 THEIR OWN RIGHT. 8010 04:37:03,606 --> 04:37:05,942 WE'RE GOING TO TALK ABOUT 8011 04:37:06,009 --> 04:37:07,811 METRICS OF QUALITY INSTEAD OF 8012 04:37:07,877 --> 04:37:10,013 IMPACT AND POPULARITY. 8013 04:37:10,080 --> 04:37:11,314 ABOUT AVOIDING STORY TELLING, 8014 04:37:11,381 --> 04:37:13,717 THINGS THAT CAN DRIVE 8015 04:37:13,783 --> 04:37:16,252 QUESTIONABLE RESEARCH PRACTICES, 8016 04:37:16,319 --> 04:37:19,756 IN THE DRIVE FOR POSITIVE 8017 04:37:19,823 --> 04:37:19,990 RESULTS. 8018 04:37:20,056 --> 04:37:21,925 BECAUSE GOOD STORY TELLING ISN'T 8019 04:37:21,992 --> 04:37:25,228 GOOD SCIENCE, SO WE WANT TO LOOK 8020 04:37:25,295 --> 04:37:27,297 AT HOW PUBLISHING PLATFORM CAN 8021 04:37:27,364 --> 04:37:29,132 FOCUS ON GOOD SCIENCE AND WHAT'S 8022 04:37:29,199 --> 04:37:32,836 IMPORTANT TO THE COMMUNITY, 8023 04:37:32,902 --> 04:37:34,971 RESEARCHERS, TO SOCIETY. 8024 04:37:35,038 --> 04:37:36,139 AND I THINK FUNDAMENTAL TO IT 8025 04:37:36,206 --> 04:37:38,608 REALLY TO ALL OF THESE PLATFORMS 8026 04:37:38,675 --> 04:37:41,511 IS RETHINKING WHAT WE'RE DOING 8027 04:37:41,578 --> 04:37:44,714 WITH PUBLICATIONS, AND THINKING 8028 04:37:44,781 --> 04:37:46,216 INSTEAD OF BEING INDIVIDUALS 8029 04:37:46,282 --> 04:37:49,719 THINKING ABOUT CAREERS, CAREER 8030 04:37:49,786 --> 04:37:50,687 STAGES, GETTING THOSE BOXES TO 8031 04:37:50,754 --> 04:37:54,124 WHERE YOU WANT TO BE IS TO THINK 8032 04:37:54,190 --> 04:37:55,225 COLLABORATIVELY, WE KNOW WHEN WE 8033 04:37:55,291 --> 04:37:58,828 START OUT IN SCIENCE WE WANT TO 8034 04:37:58,895 --> 04:38:01,431 BE ABLE TO CONTRIBUTE POSITIVITY 8035 04:38:01,498 --> 04:38:02,866 TO SOCIETY, FINDING OUT ABOUT 8036 04:38:02,932 --> 04:38:04,067 THE WORLD AND HELPING PEOPLE. 8037 04:38:04,134 --> 04:38:06,903 TO DO THAT IN A COLLABORATIVE 8038 04:38:06,970 --> 04:38:09,539 WAY TO PUBLISH INDIVIDUAL -- 8039 04:38:09,606 --> 04:38:10,740 PUBLISH AN IDEA, SOMEBODY CAN 8040 04:38:10,807 --> 04:38:12,275 PUBLISH A PROTOCOL HOW TO TEST 8041 04:38:12,342 --> 04:38:14,577 THAT IDEA, THREE OTHER PEOPLE 8042 04:38:14,644 --> 04:38:16,346 COULD COLLECT DATA ACCORDING TO 8043 04:38:16,413 --> 04:38:17,981 THOSE PROTOCOLS AND THEN LINK 8044 04:38:18,048 --> 04:38:19,616 THOSE TO PROTOCOLS. 8045 04:38:19,682 --> 04:38:21,584 SO THINKING IN A COLLABORATIVE 8046 04:38:21,651 --> 04:38:22,786 WAY WHERE PEOPLE AREN'T THINKING 8047 04:38:22,852 --> 04:38:24,954 ABOUT WHAT ARE THE FINDINGS AND 8048 04:38:25,021 --> 04:38:27,257 IMPACTS BUT INSTEAD THINKING 8049 04:38:27,323 --> 04:38:29,359 ABOUT WHAT CAN I DO TO 8050 04:38:29,426 --> 04:38:31,327 CONTRIBUTE TO OVERALL PICTURE OF 8051 04:38:31,394 --> 04:38:33,296 KNOWLEDGE GENERATION. 8052 04:38:33,363 --> 04:38:35,165 AND THE FINDINGS ARE LIKE AN 8053 04:38:35,231 --> 04:38:36,800 EMERGENT PROPERTY OUT OF THIS. 8054 04:38:36,866 --> 04:38:38,568 THESE ARE PRINCIPLES THAT I 8055 04:38:38,635 --> 04:38:41,304 THOUGHT ABOUT WHEN I DESIGNED 8056 04:38:41,371 --> 04:38:43,273 OCTOPUS, HAPPY TO TALK FURTHER 8057 04:38:43,339 --> 04:38:45,775 ABOUT, ALSO PRINCIPLES THAT I 8058 04:38:45,842 --> 04:38:48,511 THINK ARE IN COMMON ACROSS A LOT 8059 04:38:48,578 --> 04:38:49,479 OF AUTHENTIC PUBLISH PLATFORMS 8060 04:38:49,546 --> 04:38:55,952 WE'RE GOING TO TALK ABOUT, 8061 04:38:56,019 --> 04:38:57,720 NON-TRADITIONAL FORMATS. 8062 04:38:57,787 --> 04:39:03,626 FIRST SPEAKER HAS PRE-RECORDED 8063 04:39:03,693 --> 04:39:14,237 HER TALK, TALKING ABOUT ARCADIA, 8064 04:39:15,238 --> 04:39:18,374 TRADEOFFS WITH TOOLS, 8065 04:39:18,441 --> 04:39:19,375 MICROPUBLICATION BIOLOGY, 8066 04:39:19,442 --> 04:39:22,112 ARCADIA SCIENCE, EXPERIENCES 8067 04:39:22,178 --> 04:39:25,014 FROM THEIR NEW FORMATS, AND 8068 04:39:25,081 --> 04:39:25,982 SUZANNA FROM THE NATIONAL 8069 04:39:26,049 --> 04:39:27,317 INSTITUTE ON AGING TALKING ABOUT 8070 04:39:27,383 --> 04:39:30,186 THEIR EXPERIENCE, HOW THEY ARE 8071 04:39:30,253 --> 04:39:31,855 ENCOURAGING FORMATS AND PROS AND 8072 04:39:31,921 --> 04:39:32,288 CONS. 8073 04:39:32,355 --> 04:39:36,326 HOPEFULLY WE CAN START WITH 8074 04:39:36,392 --> 04:39:39,929 JOY'S RECORDING FIRST. 8075 04:39:39,996 --> 04:39:41,231 >> GOOD MORNING, EVERYBODY. 8076 04:39:41,297 --> 04:39:42,899 GOOD AFTERNOON, GOOD EVENING, 8077 04:39:42,966 --> 04:39:43,766 WHEREVER YOU ARE. 8078 04:39:43,833 --> 04:39:46,503 FOR THOSE IN THE U.S., GOOD 8079 04:39:46,569 --> 04:39:46,836 MORNING. 8080 04:39:46,903 --> 04:39:48,238 IT'S UNFORTUNATE I COULD NOT 8081 04:39:48,304 --> 04:39:52,509 MAKE IT TO THE NIH MEETING BUT 8082 04:39:52,575 --> 04:39:54,711 THANK YOU FOR ALLOWING ME TO 8083 04:39:54,777 --> 04:39:56,813 SHARE THIS RECORDING ON MY VIEWS 8084 04:39:56,880 --> 04:39:59,849 ON THE PUBLISHING OF NULL 8085 04:39:59,916 --> 04:40:02,819 RESULTS, ESPECIALLY WITH 8086 04:40:02,886 --> 04:40:03,353 PRE-PRINTS. 8087 04:40:03,419 --> 04:40:04,787 I'M EXECUTIVE DIRECTOR OF 8088 04:40:04,854 --> 04:40:05,822 TRAINING CENTER IN 8089 04:40:05,889 --> 04:40:07,056 COMMUNICATIONS, WE'RE BASED AT 8090 04:40:07,123 --> 04:40:14,063 THE UNIVERSITY OF NAIROBI IN 8091 04:40:14,130 --> 04:40:15,298 KENYA, WE SUPPORT RESEARCHERS ON 8092 04:40:15,365 --> 04:40:19,002 HOW THEY CAN IMPROVE THEIR 8093 04:40:19,068 --> 04:40:20,570 RESEARCH. 8094 04:40:20,637 --> 04:40:25,675 THE POINT OF TODAY'S MEETING, 8095 04:40:25,742 --> 04:40:28,945 HOW DO PRE-PRINT PLAY 8096 04:40:29,012 --> 04:40:31,114 SIGNIFICANT ROLE IN REPORTING 8097 04:40:31,181 --> 04:40:32,348 NULL RESULTS IN AFRICA. 8098 04:40:32,415 --> 04:40:34,450 WITH THE RIGHT OF PRE-PRINT 8099 04:40:34,517 --> 04:40:36,719 COVERS WE HAVE AFRICA ARCHIVE, 8100 04:40:36,786 --> 04:40:39,923 WHICH IS A CONTINENTAL PRE-PRINT 8101 04:40:39,989 --> 04:40:46,129 PLATFORM, WE ALSO SEE 8102 04:40:46,196 --> 04:40:50,400 UNIVERSITIES PRODUCING PRE-PRINT 8103 04:40:50,466 --> 04:40:54,571 COVERS, A PRE-PRINT SERVER, 8104 04:40:54,637 --> 04:40:55,772 PRETORIA, AND ONE FOR ADDIS 8105 04:40:55,838 --> 04:40:56,539 ABABA UNIVERSITY. 8106 04:40:56,606 --> 04:41:01,644 WITH THIS RISE OF PRE-PRINTS, A 8107 04:41:01,711 --> 04:41:04,514 WAY TO LOOK WITH PHENOMENON OF 8108 04:41:04,581 --> 04:41:06,049 GROWTH OF PRE-PRINTS AND HOW 8109 04:41:06,115 --> 04:41:08,585 THEY ARE CONTRIBUTING IN THE 8110 04:41:08,651 --> 04:41:10,853 REPORTING OF NULL RESULTS. 8111 04:41:10,920 --> 04:41:15,458 FOR OBVIOUS REASONS RAPID 8112 04:41:15,525 --> 04:41:17,760 DISSEMINATION OF INFORMATION, 8113 04:41:17,827 --> 04:41:21,898 REDUCING PUBLICATION BIAS, AND 8114 04:41:21,965 --> 04:41:23,766 PUBLISHING NULL RESULTS IN 8115 04:41:23,833 --> 04:41:29,806 PRE-PRINT MAKES IT EASY FOR 8116 04:41:29,872 --> 04:41:32,075 COLLABORATION AND INCREASED 8117 04:41:32,141 --> 04:41:33,076 TRANSPARENCY. 8118 04:41:33,142 --> 04:41:34,644 INCREASED TRANSPARENCY YOU'RE 8119 04:41:34,711 --> 04:41:38,014 LOOKING AT POSTING NULL RESULTS 8120 04:41:38,081 --> 04:41:39,649 FOR RESEARCH WHICH ALLOW 8121 04:41:39,716 --> 04:41:41,951 RESEARCHERS TO SEE THE FULL 8122 04:41:42,018 --> 04:41:43,519 SPECTRUM OF FINDINGS RELATED TO 8123 04:41:43,586 --> 04:41:44,520 A PARTICULAR TOPIC, INCLUDING 8124 04:41:44,587 --> 04:41:46,656 THOSE THAT MAY NOT HAVE BEEN 8125 04:41:46,723 --> 04:41:47,991 PUBLISHED IN PEER-REVIEWED 8126 04:41:48,057 --> 04:41:49,592 JOURNALS, DUE TO PUBLICATION 8127 04:41:49,659 --> 04:41:49,859 BIAS. 8128 04:41:49,926 --> 04:41:55,531 SO THE RIGHT OF PRE-PRINTS, 8129 04:41:55,598 --> 04:41:57,000 CONTINENTAL OR INSTITUTIONAL, 8130 04:41:57,066 --> 04:42:01,004 ARE AIDING IN MITIGATING THROUGH 8131 04:42:01,070 --> 04:42:02,305 INCREASED TRANSPARENCY BY 8132 04:42:02,372 --> 04:42:05,408 PUBLISHING OF NULL RESULTS. 8133 04:42:05,475 --> 04:42:07,243 THIS TRANSPARENCY FOCUS, A MORE 8134 04:42:07,310 --> 04:42:09,746 COMPLETE UNDERSTANDING OF THE 8135 04:42:09,812 --> 04:42:11,614 SCIENTIFIC EVIDENCE BASE AND CAN 8136 04:42:11,681 --> 04:42:13,483 HELP PREVENT RECOGNITION OF 8137 04:42:13,549 --> 04:42:15,118 MISLEADING OR INCOMPLETE 8138 04:42:15,184 --> 04:42:17,220 INFORMATION SO LET'S LOOK FROM 8139 04:42:17,287 --> 04:42:20,623 AFRICAN PERSPECTIVE, A COMPLETE 8140 04:42:20,690 --> 04:42:31,100 PICTURE OF THE RESEARCH. 8141 04:42:31,801 --> 04:42:32,035 RESEARCH. 8142 04:42:32,101 --> 04:42:34,470 WE REALLY ACKNOWLEDGE POWER OF 8143 04:42:34,537 --> 04:42:36,906 PRE-PRINTS IN CONTRIBUTING TO 8144 04:42:36,973 --> 04:42:39,108 THE TRANSPARENCY AND VISIBILITY 8145 04:42:39,175 --> 04:42:42,378 OF NULL RESULTS AND MITIGATING 8146 04:42:42,445 --> 04:42:43,913 SKEWED PUBLICATION BIAS BASED ON 8147 04:42:43,980 --> 04:42:52,588 OUTCOMES OUT OF AFRICA. 8148 04:42:52,655 --> 04:42:54,991 THEN SPECIFIC DATA, THE TRENDS 8149 04:42:55,058 --> 04:42:56,959 AFRICAN RESEARCHERS PUBLISHING 8150 04:42:57,026 --> 04:43:00,997 NULL RESULTS, SEVERAL 8151 04:43:01,064 --> 04:43:02,565 INITIATIVES IN PROMOTING 8152 04:43:02,632 --> 04:43:05,635 TRANSPARENCY AND RESEARCH 8153 04:43:05,702 --> 04:43:05,968 TRANSPARENCY. 8154 04:43:06,035 --> 04:43:07,570 WHICH COULD POTENTIALLY LEAD TO 8155 04:43:07,637 --> 04:43:13,176 AN INCREASE IN PUBLICATION OF 8156 04:43:13,242 --> 04:43:14,677 NULL RESULTS. 8157 04:43:14,744 --> 04:43:18,848 PLATFORMS ADVOCATE FOR OPEN 8158 04:43:18,915 --> 04:43:21,250 ACCESS TO PROMOTE COLLABORATION 8159 04:43:21,317 --> 04:43:23,453 AMONG AFRICAN SCIENTISTS AND 8160 04:43:23,519 --> 04:43:24,320 RESEARCHERS. 8161 04:43:24,387 --> 04:43:26,422 AND WE HAVE INITIATIVES LIKE THE 8162 04:43:26,489 --> 04:43:27,557 COALITION OF AFRICAN RESEARCH 8163 04:43:27,623 --> 04:43:30,226 AND INNOVATION WHERE THEY ARE 8164 04:43:30,293 --> 04:43:32,128 WORKING TO STRENGTHEN CAPACITY 8165 04:43:32,195 --> 04:43:35,598 TO PROMOTE ACROSS THE CONTINENT. 8166 04:43:35,665 --> 04:43:37,233 WITH TRAINING CENTER IN 8167 04:43:37,300 --> 04:43:38,901 COMMUNICATION WE'RE ADVOCATING 8168 04:43:38,968 --> 04:43:42,171 FOR THIS ADOPTION OF OPEN STANCE 8169 04:43:42,238 --> 04:43:43,806 PRINCIPLES, WHETHER IT'S 8170 04:43:43,873 --> 04:43:44,907 REGIONAL, NATIONAL, 8171 04:43:44,974 --> 04:43:48,745 INSTITUTIONAL, AND THIS WILL AID 8172 04:43:48,811 --> 04:43:50,146 IN HAVING INSTITUTIONAL SYSTEMS 8173 04:43:50,213 --> 04:43:52,348 SET IN PLACE WHEREBY WHEN IT 8174 04:43:52,415 --> 04:43:56,386 COMES TO PUBLISHING, YOU HAVE 8175 04:43:56,452 --> 04:43:58,254 ALSO NULL RESULTS, BEYOND JUST 8176 04:43:58,321 --> 04:43:59,322 POSITIVE RESULTS THAT COME OUT 8177 04:43:59,389 --> 04:43:59,622 OF RESEARCH. 8178 04:43:59,689 --> 04:44:02,525 AT THE END OF THE DAY WE'RE 8179 04:44:02,592 --> 04:44:04,827 LOOKING AT TRANSPARENCY, TO GET 8180 04:44:04,894 --> 04:44:06,796 PUBLICATION BIAS, NOT ONLY THE 8181 04:44:06,863 --> 04:44:07,563 RISE OF PRE-PRINT REPOSITORY, 8182 04:44:07,630 --> 04:44:09,532 JUST THEY ARE ALSO INITIATIVES 8183 04:44:09,599 --> 04:44:12,935 THAT ARE TRYING TO PROMOTE 8184 04:44:13,002 --> 04:44:17,206 ADOPTION OF OPEN SCIENCE WITHIN 8185 04:44:17,273 --> 04:44:18,508 THE CONTINENT, WILL HAVE IMPACT 8186 04:44:18,574 --> 04:44:21,244 ON PUBLICATION OF NULL RESULTS, 8187 04:44:21,310 --> 04:44:22,111 BECAUSE OF DEMAND FOR 8188 04:44:22,178 --> 04:44:23,746 TRANSPARENCY AND DEMAND FOR 8189 04:44:23,813 --> 04:44:24,680 REDUCTION OF PUBLICATION BIAS 8190 04:44:24,747 --> 04:44:25,848 COMING OUT OF THE AFRICA. 8191 04:44:25,915 --> 04:44:27,183 THANK YOU SO MUCH FOR LISTENING 8192 04:44:27,250 --> 04:44:28,117 TO ME. 8193 04:44:28,184 --> 04:44:32,088 I HOPE YOU ENJOY THE REST OF 8194 04:44:32,155 --> 04:44:32,722 YOUR WATCHING. 8195 04:44:32,789 --> 04:44:35,525 GOOD-BYE. 8196 04:44:49,605 --> 04:44:51,974 >> THANK YOU FOR INVITING ME. 8197 04:44:52,041 --> 04:44:58,614 REALLY DELIGHTED TO BE HERE. 8198 04:44:58,681 --> 04:45:02,985 AS OF LAST MONTH, ASSISTANT 8199 04:45:03,052 --> 04:45:04,554 PROFESSOR AT SCHOOL OF MEDICINE, 8200 04:45:04,620 --> 04:45:11,994 DURING MY POSTDOCTORAL WORK 8201 04:45:12,061 --> 04:45:16,699 GAME INTERESTED IN MENTORSHIP, 8202 04:45:16,766 --> 04:45:18,468 IMPROVING FACULTY HIRING AND 8203 04:45:18,534 --> 04:45:21,270 PUBLISHING, AND SO I HAVE BEEN 8204 04:45:21,337 --> 04:45:22,605 INVOLVED WITH A FEW COMMUNITIES, 8205 04:45:22,672 --> 04:45:28,644 TODAY I HELP TO SHARE WITH YOU 8206 04:45:28,711 --> 04:45:29,645 MORE EARLY CAREER PERSPECTIVES 8207 04:45:29,712 --> 04:45:33,649 AND COMMUNITIES OF EARLY CAREER 8208 04:45:33,716 --> 04:45:36,752 RESEARCHERS PERSPECTIVE ON TWO 8209 04:45:36,819 --> 04:45:38,287 ASPECTS, ONE IS THEIR ROLE IN 8210 04:45:38,354 --> 04:45:41,557 OPEN SCIENCE AND IN PUBLISHING 8211 04:45:41,624 --> 04:45:43,626 ALL THEIR FINDINGS AT THE SPEED 8212 04:45:43,693 --> 04:45:44,627 OF RESEARCH, SECOND ON THE VALUE 8213 04:45:44,694 --> 04:45:53,135 OF THIS FOR EARLY CAREER 8214 04:45:53,202 --> 04:45:58,174 RESEARCHERS. 8215 04:45:58,241 --> 04:46:01,143 MY FIRST INVOLVEMENT, WE PUT 8216 04:46:01,210 --> 04:46:04,447 FORWARD A PERSPECTIVE ON HOW 8217 04:46:04,514 --> 04:46:05,848 PRE-PRINTS ARE ACTUALLY MORE 8218 04:46:05,915 --> 04:46:08,784 THAN JUST ALLOWING US TO 8219 04:46:08,851 --> 04:46:10,219 DISSEMINATE RESEARCH FAST. 8220 04:46:10,286 --> 04:46:12,955 THEY ARE SUPPORTING A PLETHORA 8221 04:46:13,022 --> 04:46:15,258 OF REALLY POSITIVE ASPECTS IN 8222 04:46:15,324 --> 04:46:19,662 OUR CAREERS AND RESEARCH LIVES. 8223 04:46:19,729 --> 04:46:28,204 OVER THE PAST FEW YEARS, ALL OF 8224 04:46:28,271 --> 04:46:29,505 THE STAKEHOLDERS, FUNDERS HAVE 8225 04:46:29,572 --> 04:46:31,807 APPRECIATED THIS APPROACH AND 8226 04:46:31,874 --> 04:46:37,413 HELPED RESEARCHERS IMPROVE THEIR 8227 04:46:37,480 --> 04:46:40,683 CAREERS IN MANY WAYS BY HOSTING 8228 04:46:40,750 --> 04:46:41,017 PRE-PRINTS. 8229 04:46:41,083 --> 04:46:45,488 SO THE SECOND COMMUNITY THAT I 8230 04:46:45,555 --> 04:46:50,760 HAVE BEEN INVOLVED WITH HAS BEEN 8231 04:46:50,826 --> 04:46:52,595 THE LARGE COMMUNITY OF 7,000 8232 04:46:52,662 --> 04:46:54,764 POSTDOCS, FOR THE PAST SEVEN 8233 04:46:54,830 --> 04:46:57,967 YEARS WE'VE HAD A LOT OF GOOD 8234 04:46:58,034 --> 04:47:00,603 DISCUSSIONS, THE COMMUNITY IS 8235 04:47:00,670 --> 04:47:04,206 FOCUSED ON THE LATE STAGE OF 8236 04:47:04,273 --> 04:47:07,376 CAREER, WHICH IS FOR POSTDOCS 8237 04:47:07,443 --> 04:47:08,711 INTERESTED IN ACADEMIC CAREER, 8238 04:47:08,778 --> 04:47:11,547 APPLYING FOR FACULTY POSITION 8239 04:47:11,614 --> 04:47:13,716 AND THESE ARE SOME OF THE 8240 04:47:13,783 --> 04:47:16,886 HIGHLIGHTS OF THE CONVERSATIONS 8241 04:47:16,953 --> 04:47:18,554 ON PUBLISHING DIFFERENT TYPES OF 8242 04:47:18,621 --> 04:47:21,357 FINDINGS AND PUBLISHING THEM AS 8243 04:47:21,424 --> 04:47:24,126 PRE-PRINTS, SO QUITE A FEW OF 8244 04:47:24,193 --> 04:47:25,962 THEM THINK THAT EVERYTHING 8245 04:47:26,028 --> 04:47:28,364 SHOULD BE PRE-PRINTED TO SHOW 8246 04:47:28,431 --> 04:47:29,365 PROGRESS, AND THERE ARE EXAMPLES 8247 04:47:29,432 --> 04:47:32,201 OF PEOPLE WHO HAVE BEEN ABLE TO 8248 04:47:32,268 --> 04:47:35,805 FIND FACULTY POSITIONS AND GET 8249 04:47:35,871 --> 04:47:37,773 INTERVIEWS IN THE JOB MARKET 8250 04:47:37,840 --> 04:47:40,042 BECAUSE THEY HAVE PRE-PRINTS, 8251 04:47:40,109 --> 04:47:42,044 BECAUSE POST DOCTOR CAN BE VERY 8252 04:47:42,111 --> 04:47:44,880 SHORT, ALSO QUESTIONS ON LISTING 8253 04:47:44,947 --> 04:47:46,749 PRE-PRINTS ALONG OTHER 8254 04:47:46,816 --> 04:47:49,051 PUBLICATIONS AND MANY THINK THAT 8255 04:47:49,118 --> 04:47:51,887 PRE-PRINTS ARE THE BEST 8256 04:47:51,954 --> 04:47:53,756 REPRESENTATION OF THEIR WORK, 8257 04:47:53,823 --> 04:47:54,824 ACTUALLY. 8258 04:47:54,890 --> 04:47:59,428 THERE ARE EVEN QUESTIONS AND 8259 04:47:59,495 --> 04:48:00,763 DISCUSSIONS ON WHETHER RESEARCH 8260 04:48:00,830 --> 04:48:02,498 FINDINGS SUCH AS THOSE THAT ARE 8261 04:48:02,565 --> 04:48:03,699 PRESENT AT CONFERENCES WHICH 8262 04:48:03,766 --> 04:48:05,768 TEND TO BE SHORTER, CAN BE 8263 04:48:05,835 --> 04:48:09,405 PRE-PRINTED, AND SHOWN AS 8264 04:48:09,472 --> 04:48:11,907 EVIDENCE OF PRODUCTIVITY, SO 8265 04:48:11,974 --> 04:48:14,644 EARLY CAREER RESEARCHERS SUCH AS 8266 04:48:14,710 --> 04:48:15,845 POSTDOCTORAL RESEARCHERS ARE 8267 04:48:15,911 --> 04:48:19,048 VERY INTERESTED IN PUBLISHING 8268 04:48:19,115 --> 04:48:20,616 THEIR FINDINGS. 8269 04:48:20,683 --> 04:48:26,322 NOW, I WAS ALSO INVOLVED IN 8270 04:48:26,389 --> 04:48:28,924 STUDY WITH FELLOW POSTDOCS ON 8271 04:48:28,991 --> 04:48:30,860 THE FUTURE PI SLACK PLATFORM. 8272 04:48:30,926 --> 04:48:34,597 WE LOOKED AT A FEW HUNDRED 8273 04:48:34,664 --> 04:48:35,765 POSTDOC, NOT NECESSARILY ON THE 8274 04:48:35,831 --> 04:48:40,403 PLATFORM BUT YOU THERE A -- 8275 04:48:40,469 --> 04:48:42,938 THROUGH A SURVEY AND FOUND OVER 8276 04:48:43,005 --> 04:48:45,775 55% OF THEM HAD ACTUALLY POSTED 8277 04:48:45,841 --> 04:48:46,208 PRE-PRINTS. 8278 04:48:46,275 --> 04:48:49,311 THERE ARE ALSO OTHER STUDIES, 8279 04:48:49,378 --> 04:48:52,014 FOR EXAMPLE, ON THE RIGHT WHERE 8280 04:48:52,081 --> 04:48:58,354 IN THE FIELD OF ECOLOGY LOOKING 8281 04:48:58,421 --> 04:49:01,624 AT 500 PRE-PRINTS FOUND IT IS 8282 04:49:01,691 --> 04:49:02,525 POSITIVELY ASSOCIATED WITH EARLY 8283 04:49:02,591 --> 04:49:03,926 CAREER STATUS IN ECOLOGY. 8284 04:49:03,993 --> 04:49:08,431 ON THE LEFT OF THAT FIGURE A, IT 8285 04:49:08,497 --> 04:49:10,433 SHOWS PUBLICATIONS THAT ARE 8286 04:49:10,499 --> 04:49:12,101 PRE-PRINTED WITH EARLY CAREER 8287 04:49:12,168 --> 04:49:13,502 RESEARCHER AS FIRST AUTHOR. 8288 04:49:13,569 --> 04:49:17,540 ON THE RIGHT EARLY CAREER 8289 04:49:17,606 --> 04:49:19,842 RESEARCHER AS CORRESPONDING 8290 04:49:19,909 --> 04:49:20,276 AUTHOR. 8291 04:49:20,342 --> 04:49:24,880 YOU CAN SEE WHEN EARLY CAREER 8292 04:49:24,947 --> 04:49:25,981 RESEARCHERS HAVE THE POWER AND 8293 04:49:26,048 --> 04:49:32,555 BETTER CONTROL OVER THE FATE OF 8294 04:49:32,621 --> 04:49:35,057 THE PUBLICATION, THEY ARE VERY 8295 04:49:35,124 --> 04:49:37,026 MUCH INTERESTED IN POSTING 8296 04:49:37,093 --> 04:49:39,528 PRE-PRINTS AS OPPOSED TO WHEN 8297 04:49:39,595 --> 04:49:48,170 THEY ARE PRE-PRINTING AS A FIRST 8298 04:49:48,237 --> 04:49:48,537 AUTHOR. 8299 04:49:48,604 --> 04:49:49,839 IN THE SURVEY WE RAN THESE 8300 04:49:49,905 --> 04:49:52,007 FIGURES ARE FROM THAT PAPER, 8301 04:49:52,074 --> 04:49:53,342 THROUGH QUALITATIVE RESPONSES WE 8302 04:49:53,409 --> 04:49:58,347 FOUND PRE-PRINTING AS YOU CAN 8303 04:49:58,414 --> 04:50:00,850 SEE IN THE WORD CLOUD ASSOCIATED 8304 04:50:00,916 --> 04:50:02,618 WITH THE APPLICANT'S 8305 04:50:02,685 --> 04:50:03,619 PERSPECTIVE, WITH POSITIVE 8306 04:50:03,686 --> 04:50:05,588 OUTCOME, FACULTY JOB MARKET, AND 8307 04:50:05,654 --> 04:50:07,757 ON THE RIGHT A SAMPLE FOR THE 8308 04:50:07,823 --> 04:50:08,858 SEARCH COMMITTEE MEMBERS FOR 8309 04:50:08,924 --> 04:50:10,359 VARIOUS MODES. 8310 04:50:10,426 --> 04:50:13,863 YOU CAN SEE ABOUT 60% OF THEM 8311 04:50:13,929 --> 04:50:15,965 APPRECIATED PRE-PRINTS LISTED ON 8312 04:50:16,031 --> 04:50:19,034 THE C.V. OF THE JOB APPLICANTS. 8313 04:50:19,101 --> 04:50:21,003 FINALLY, THERE ARE MULTIPLE 8314 04:50:21,070 --> 04:50:24,273 STUDIES THAT SHOW THAT 8315 04:50:24,340 --> 04:50:26,475 PRE-PRINTING IS ASSOCIATED WITH 8316 04:50:26,542 --> 04:50:28,344 BETTER CITATIONS AND ATTENTION 8317 04:50:28,411 --> 04:50:35,117 TO THE FINDINGS, AND ALSO THIS 8318 04:50:35,184 --> 04:50:37,086 STUDY BY FRASEE FOUND THEY GET A 8319 04:50:37,153 --> 04:50:39,488 LOT OF SOCIAL MEDIA ATTENTION 8320 04:50:39,555 --> 04:50:43,192 AND FOUND THAT YOUNGER AUTHORS 8321 04:50:43,259 --> 04:50:47,730 TEND TO SUBMIT MORE OF THE 8322 04:50:47,797 --> 04:50:49,231 bioRxiv PRE-PRINTS AND 8323 04:50:49,298 --> 04:50:51,967 bioRxiv DEPOSITED PAPERS, SO 8324 04:50:52,034 --> 04:50:56,238 IT MAY IN FACT THAT EARLY CAREER 8325 04:50:56,305 --> 04:50:57,473 RESEARCHERS REALLY EMBRACE OPEN 8326 04:50:57,540 --> 04:51:02,478 SCIENCE AND ACTUALLY DRIVE IT. 8327 04:51:02,545 --> 04:51:04,446 I HOPE WE CONTINUE DISCUSSIONS 8328 04:51:04,513 --> 04:51:07,950 ON THE ROLE OF EARLY CAREER 8329 04:51:08,017 --> 04:51:09,151 RESEARCHES AND STAKEHOLDERS IN 8330 04:51:09,218 --> 04:51:09,685 THIS. 8331 04:51:09,752 --> 04:51:17,760 THANKS. 8332 04:51:22,765 --> 04:51:25,801 >> I'M PROFESSOR OF BUSINESS AND 8333 04:51:25,868 --> 04:51:28,404 MARKETING AT COLUMBIA, THE AREA 8334 04:51:28,470 --> 04:51:29,839 I SPECIALIZE IN IS CONSUMER 8335 04:51:29,905 --> 04:51:33,609 RESOURCES, IN THE AREA OF SOCIAL 8336 04:51:33,676 --> 04:51:36,645 SCIENCES, BULK HAS TO DO, FOR 8337 04:51:36,712 --> 04:51:37,646 EXAMPLE, WITH WHAT MAKES 8338 04:51:37,713 --> 04:51:39,915 CONSUMERS MORE LIKELY TO ADOPT 8339 04:51:39,982 --> 04:51:41,984 NEW TECHNOLOGIES, WHAT MADE THEM 8340 04:51:42,051 --> 04:51:43,719 RESPOND MORE TO CERTAIN MESSAGES 8341 04:51:43,786 --> 04:51:45,387 OR OTHER MESSAGES, AND SO ON AND 8342 04:51:45,454 --> 04:51:45,821 SO FORTH. 8343 04:51:45,888 --> 04:51:48,691 MOST OF THE WORK WE DO IS 8344 04:51:48,757 --> 04:51:50,526 EXPERIMENTAL, WE DO USE OTHER 8345 04:51:50,593 --> 04:51:52,294 TYPE OF PARADIGMS AS WELL. 8346 04:51:52,361 --> 04:51:54,930 IF I WAS TO CHARACTERIZE THE 8347 04:51:54,997 --> 04:51:57,333 TYPE OF RESEARCH THAT WE DO, 8348 04:51:57,399 --> 04:51:59,435 WE'VE HEARD SEVERAL TIMES TODAY 8349 04:51:59,501 --> 04:52:04,807 SOME KIND OF A CONTINUUM BETWEEN 8350 04:52:04,874 --> 04:52:07,443 CLINICAL AND PRE-CLINICAL OR 8351 04:52:07,509 --> 04:52:09,078 EXPLORATORY VERSUS CONFIRMATORY, 8352 04:52:09,144 --> 04:52:12,715 WE SORT OF HAVE THE SENSE OF 8353 04:52:12,781 --> 04:52:14,783 WHAT THAT MEANS. 8354 04:52:14,850 --> 04:52:16,218 EVEN THOUGH WITHIN OUR FIELD 8355 04:52:16,285 --> 04:52:18,387 MANY PEOPLE IN THE FIELD WOULD 8356 04:52:18,454 --> 04:52:19,922 NOT AGREE WITH MY 8357 04:52:19,989 --> 04:52:21,891 CHARACTERIZATION BUT WE TENDING 8358 04:52:21,957 --> 04:52:23,092 TO EXPLORATORY IN OTHER 8359 04:52:23,158 --> 04:52:30,232 DISCIPLINES THAN WE THINK WE 8360 04:52:30,299 --> 04:52:30,566 ARE. 8361 04:52:30,633 --> 04:52:33,836 SO, A NUMBER OF YEARS AGO I WAS 8362 04:52:33,903 --> 04:52:35,471 ASKED TO REFLECT, I WAS 8363 04:52:35,537 --> 04:52:40,009 PRESIDENT OF SOCIETY FOR 8364 04:52:40,075 --> 04:52:42,444 CONSUMER PSYCHOLOGY, AND WE HAD 8365 04:52:42,511 --> 04:52:43,512 REPRODUCIBILITY CRISIS, AND AT 8366 04:52:43,579 --> 04:52:44,580 THAT TIME AS PRESIDENT OF THE 8367 04:52:44,647 --> 04:52:46,682 SOCIETY I WAS ABLE TO PUSH FOR 8368 04:52:46,749 --> 04:52:48,851 MORE OPEN SCIENCE PRACTICES 8369 04:52:48,918 --> 04:52:50,286 BEING IMPLEMENTED IN THE 8370 04:52:50,352 --> 04:52:52,021 PRIMARY -- ONE OF THE PRIMARY 8371 04:52:52,087 --> 04:52:53,656 JOURNALS IN OUR FIELD WE WERE 8372 04:52:53,722 --> 04:52:57,259 ABLE TO CONTROL, BUT THE LAST 8373 04:52:57,326 --> 04:52:59,628 FEW YEARS LATER, I WAS ASKED TO 8374 04:52:59,695 --> 04:53:02,197 THINK ABOUT, OKAY, WHAT DO YOU 8375 04:53:02,264 --> 04:53:03,299 THINK ABOUT PRE-REGISTRATION? 8376 04:53:03,365 --> 04:53:07,569 THIS IS A TOPIC THAT HAS BEEN 8377 04:53:07,636 --> 04:53:11,073 MENTIONED A NUMBER OF TIMES 8378 04:53:11,140 --> 04:53:12,374 WITHIN THE TWO-DAY SESSION, AND 8379 04:53:12,441 --> 04:53:16,545 AS I WAS THINKING ABOUT IT, THE 8380 04:53:16,612 --> 04:53:17,646 THINKING ABOUT THE BIG PICTURE 8381 04:53:17,713 --> 04:53:19,615 WHAT IS GOOD FOR THE FIELD, NOT 8382 04:53:19,682 --> 04:53:21,116 JUST NECESSARILY WHAT IS GOOD 8383 04:53:21,183 --> 04:53:22,751 FOR THE RESEARCHER AND SO ON AND 8384 04:53:22,818 --> 04:53:23,452 SO FORTH. 8385 04:53:23,519 --> 04:53:28,023 AS I THOUGHT ABOUT THIS, I ENDED 8386 04:53:28,090 --> 04:53:31,427 UP HAVING A FEW OF PERHAPS LESS 8387 04:53:31,493 --> 04:53:33,395 GUNG HO, PERHAPS MORE SKEPTICAL 8388 04:53:33,462 --> 04:53:35,664 OF THE PROMISES OF 8389 04:53:35,731 --> 04:53:36,231 PRE-REGISTRATION, EVERYTHING 8390 04:53:36,298 --> 04:53:38,634 ELSE EQUAL TO USE A PHRASE THAT 8391 04:53:38,701 --> 04:53:42,905 NEIL USED, CONDITIONAL ON THE 8392 04:53:42,972 --> 04:53:43,806 RESEARCH BEING EQUAL, 8393 04:53:43,872 --> 04:53:44,840 PRE-REGISTRATION APPLIES. 8394 04:53:44,907 --> 04:53:46,642 THE QUESTION IS ONCE YOU THINK 8395 04:53:46,709 --> 04:53:49,478 ABOUT IMPLEMENTING AS A POLICY 8396 04:53:49,545 --> 04:53:52,014 OR MANDATE POLICY IN NEW 8397 04:53:52,081 --> 04:53:54,283 SCIENTIFIC PRACTICE OR NORM, 8398 04:53:54,350 --> 04:53:55,150 WHAT ARE SYSTEMIC CONSEQUENCES 8399 04:53:55,217 --> 04:54:01,724 OF A CHANGE IN POLICY. 8400 04:54:01,790 --> 04:54:03,258 IN AN ARTICLE IN CONSUMER 8401 04:54:03,325 --> 04:54:06,862 PSYCHOLOGY I TRIED TO STRUCTURE 8402 04:54:06,929 --> 04:54:09,264 A LITTLE BIT WHAT IS THE BIG 8403 04:54:09,331 --> 04:54:10,599 PICTURE OF SCIENCE WHAT WE'RE 8404 04:54:10,666 --> 04:54:12,868 TRYING TO DO HERE, AND THE WAY 8405 04:54:12,935 --> 04:54:15,738 I'M THINKING ABOUT IT, OKAY, 8406 04:54:15,804 --> 04:54:16,372 ESSENTIALLY PRE-REGISTRATION IS 8407 04:54:16,438 --> 04:54:17,172 JUST A TOOL. 8408 04:54:17,239 --> 04:54:21,543 THERE'S A LOT OF GOOD SCIENCE 8409 04:54:21,610 --> 04:54:26,148 TOOLS, DATA ANALYSIS, PEER 8410 04:54:26,215 --> 04:54:32,154 REVIEW PROCESS, WHAT DO YOU CALL 8411 04:54:32,221 --> 04:54:34,456 IT, PRE-REPORT? 8412 04:54:34,523 --> 04:54:36,525 AND PRE-PRINTS, THESE ARE ALL 8413 04:54:36,592 --> 04:54:37,026 TOOLS. 8414 04:54:37,092 --> 04:54:39,061 THESE ARE JUST TOOLS. 8415 04:54:39,128 --> 04:54:41,130 AND THE TOOLS ONLY VALUABLE TO 8416 04:54:41,196 --> 04:54:44,933 THE EXTENT THEY HELP US FULL 8417 04:54:45,000 --> 04:54:46,935 FELL A SERIES OF GOALS WE'RE 8418 04:54:47,002 --> 04:54:51,740 TRYING TO ACHIEVE COLLECTIVELY. 8419 04:54:51,807 --> 04:54:53,475 THE GOALS, WE WANT TO CONTROL 8420 04:54:53,542 --> 04:54:55,878 FOR TYPE 1 ERROR, THAT'S PART OF 8421 04:54:55,944 --> 04:54:57,312 THE REASON PEOPLE WERE CONCERNED 8422 04:54:57,379 --> 04:55:01,250 ABOUT P-HACKING AND SO ON, BUT 8423 04:55:01,316 --> 04:55:02,451 ALSO TALKED ABOUT FACTUAL 8424 04:55:02,518 --> 04:55:05,387 ERRORS, I DON'T WANT TO CREATE A 8425 04:55:05,454 --> 04:55:07,189 SYSTEM TWO AREAS INFLATED 8426 04:55:07,256 --> 04:55:09,124 BECAUSE YOU TRIED TO CONTROL 8427 04:55:09,191 --> 04:55:10,659 TYPE 1 ERROR, AND OTHER THINGS 8428 04:55:10,726 --> 04:55:13,195 THAT MATTER AS WELL IN MY FIELD, 8429 04:55:13,262 --> 04:55:14,396 MORE APPLIED, MOST OF THE CON 8430 04:55:14,463 --> 04:55:19,868 THE CITY CONSTITUENTS ARE 8431 04:55:19,935 --> 04:55:21,070 BUSINESSING, PEOPLE WHO COULD 8432 04:55:21,136 --> 04:55:23,138 POTENTIALLY USE WHAT WE HAVE, 8433 04:55:23,205 --> 04:55:26,408 THERE ARE ISSUES OF WHAT WE CALL 8434 04:55:26,475 --> 04:55:27,509 RELEVANCE, ARE YOU STUDYING 8435 04:55:27,576 --> 04:55:28,944 RESEARCH QUESTION THAT'S 8436 04:55:29,011 --> 04:55:30,746 SOMETHING THAT BUSINESSES, FOR 8437 04:55:30,813 --> 04:55:32,347 EXAMPLE, POLICYMAKERS FIND TO BE 8438 04:55:32,414 --> 04:55:33,449 IMPORTANT, THIS IS REAL 8439 04:55:33,515 --> 04:55:36,452 QUESTION, NOT SOMETHING TRIVIAL. 8440 04:55:36,518 --> 04:55:37,553 THERE'S AN ISSUE OF 8441 04:55:37,619 --> 04:55:40,089 CROSS-TRANSPARENCY WHICH OPEN 8442 04:55:40,155 --> 04:55:42,491 SCIENCE HELPS IN THAT, 8443 04:55:42,558 --> 04:55:43,492 GENERALIZABILITY, A CRITERIA, 8444 04:55:43,559 --> 04:55:45,661 ARE WE DISCOVERING SOMETHING WE 8445 04:55:45,727 --> 04:55:46,762 DIDN'T KNOW ABOUT? 8446 04:55:46,829 --> 04:55:52,000 IT'S IMPORTANT WHEN YOU THINK 8447 04:55:52,067 --> 04:55:53,435 ABOUT THIS KIND OF SYSTEM 8448 04:55:53,502 --> 04:55:54,536 CHANGE, POLICY CHANGE YOU WANT 8449 04:55:54,603 --> 04:55:57,806 TO IMPLEMENT, WHAT ARE THE TOTAL 8450 04:55:57,873 --> 04:55:59,475 EFFECT OF HAVING THESE NEW 8451 04:55:59,541 --> 04:56:02,511 POLICIES IN PLACE. 8452 04:56:02,578 --> 04:56:06,882 AND THIS IS WHERE -- I DO THINK 8453 04:56:06,949 --> 04:56:08,350 THIS WILL INFORM THE BACKGROUND 8454 04:56:08,417 --> 04:56:09,985 OF THE DISCUSSION, I THOUGHT 8455 04:56:10,052 --> 04:56:14,022 ABOUT, OKAY, WE ALSO HAVE 8456 04:56:14,089 --> 04:56:15,858 POTENTIALLY SOME DOWNSTREAM 8457 04:56:15,924 --> 04:56:16,558 UNINTENDED ADVERSE CONSEQUENCES. 8458 04:56:16,625 --> 04:56:18,594 ONE OF THE MAIN ONES I WAS 8459 04:56:18,660 --> 04:56:20,496 CONCERNED ABOUT AS THE PERSON 8460 04:56:20,562 --> 04:56:23,866 WHO IS BEING IN MY OWN FOR A 8461 04:56:23,932 --> 04:56:25,734 LONG TIME, PRE-REGISTRATION IF 8462 04:56:25,801 --> 04:56:27,803 IT BECAME A MANDATE WOULD CHANGE 8463 04:56:27,870 --> 04:56:29,571 THE NATURE OF THE RESEARCH 8464 04:56:29,638 --> 04:56:33,842 QUESTIONS THAT PEOPLE STARTED 8465 04:56:33,909 --> 04:56:34,176 INVESTIGATING. 8466 04:56:34,243 --> 04:56:35,811 INSTEAD OF ADDRESSING QUESTIONS 8467 04:56:35,878 --> 04:56:38,514 MOST USEFUL, MOST IMPORTANT, 8468 04:56:38,580 --> 04:56:41,817 MOST CRITICAL, THEY WILL MOVE 8469 04:56:41,884 --> 04:56:43,485 THE RESEARCH AGENDA TO THINGS 8470 04:56:43,552 --> 04:56:45,654 WHERE THEY EXPECT IT TO BE ABLE 8471 04:56:45,721 --> 04:56:46,622 TO FIND SUPPORT FOR WHATEVER 8472 04:56:46,688 --> 04:56:51,460 THEY TRY TO TEST AND COULD DO 8473 04:56:51,527 --> 04:56:53,862 THAT ON THE (INAUDIBLE) COST. 8474 04:56:53,929 --> 04:56:55,497 QUANTITATIVE ANALYSIS, OVER THE 8475 04:56:55,564 --> 04:56:57,466 LAST FEW YEARS OBSERVED RESEARCH 8476 04:56:57,533 --> 04:57:00,536 PUBLISHED IN MY DISCIPLINE WHERE 8477 04:57:00,602 --> 04:57:09,411 A LOT OF RESEARCH BEING DONE 8478 04:57:09,478 --> 04:57:10,279 ONLINE USING MECHANICAL 8479 04:57:10,345 --> 04:57:13,448 SURROGATES, RELATIVELY CHEAP TO 8480 04:57:13,515 --> 04:57:23,292 RUN, TO RERUN, SO ON AND SO 8481 04:57:23,358 --> 04:57:24,593 FORTH. 8482 04:57:24,660 --> 04:57:28,230 IF WE CAN PRE-REGISTER WHERE WE 8483 04:57:28,297 --> 04:57:30,098 ONLY END UP WITH EASY TO RUN 8484 04:57:30,165 --> 04:57:31,033 STUDIES THAT DON'T NECESSARILY 8485 04:57:31,099 --> 04:57:34,036 TEACH US ANYTHING, THAT WAS THE 8486 04:57:34,102 --> 04:57:37,873 KIND OF THING NOW, SO THAT'S 8487 04:57:37,940 --> 04:57:38,807 WHERE CONDITIONAL ON RESEARCH 8488 04:57:38,874 --> 04:57:43,345 BEING THE SAME IN TERMS OF 8489 04:57:43,412 --> 04:57:43,979 QUALITY, PRE-REGISTRATION HELPS 8490 04:57:44,046 --> 04:57:47,683 BUT YOU HAVE TO ALSO SAY ARE WE 8491 04:57:47,749 --> 04:57:49,251 CREATING SYSTEMS WHERE THERE ARE 8492 04:57:49,318 --> 04:57:52,321 COSTS THAT WERE NOT ANTICIPATED. 8493 04:57:52,387 --> 04:57:58,126 ANOTHER COST I FORESAW WAS 8494 04:57:58,193 --> 04:57:59,661 HAPPENING, PEOPLE READING 8495 04:57:59,728 --> 04:58:00,729 PRE-REGISTRATION AS HUERISTIC. 8496 04:58:00,796 --> 04:58:06,668 IT'S FOR RESEARCH THAT MUST BE 8497 04:58:06,735 --> 04:58:09,171 GOOD, IF YOU STOP PRE-REGISTER 8498 04:58:09,238 --> 04:58:11,139 MUST BE BAD, IN FACT YOU WANT TO 8499 04:58:11,206 --> 04:58:13,542 MAKE SURE YOU EVALUATE THE 8500 04:58:13,609 --> 04:58:15,911 RESEARCH IN ITS TOTALITY, THAT'S 8501 04:58:15,978 --> 04:58:17,646 THE SAME KIND OF THINKING YOU 8502 04:58:17,713 --> 04:58:22,084 NEED TO APPLY WHEN YOU INTERPRET 8503 04:58:22,150 --> 04:58:24,386 SIGNIFICANCE, NOT BECAUSE IT'S 8504 04:58:24,453 --> 04:58:26,722 SIGNIFICANT BELOW THE .05 LEVEL, 8505 04:58:26,788 --> 04:58:29,124 OR ABOVE IT'S BAD, THE SAME 8506 04:58:29,191 --> 04:58:31,093 THING HAPPENS, DICHOTOMOUS 8507 04:58:31,159 --> 04:58:32,094 THINKING USING SIMPLE 8508 04:58:32,160 --> 04:58:34,930 HUERISTICS, GOOD OR BAD 8509 04:58:34,997 --> 04:58:36,898 RESEARCH, HELPFUL, AND WHATEVER 8510 04:58:36,965 --> 04:58:38,533 PRACTICES WE COLLECTIVELY TRY TO 8511 04:58:38,600 --> 04:58:42,471 DEVICE HAS TO THINK ABOUT THE 8512 04:58:42,537 --> 04:58:44,139 BIG PICTURE, BALANCING ALL THE 8513 04:58:44,206 --> 04:58:45,474 GOALS THAT WE'RE COLLECTIVELY 8514 04:58:45,540 --> 04:58:52,681 PURSUING AND INTERESTED IN. 8515 04:58:52,748 --> 04:58:58,487 THANK YOU. 8516 04:58:58,553 --> 04:58:59,221 >> GREAT. 8517 04:58:59,288 --> 04:58:59,588 HELLO. 8518 04:58:59,655 --> 04:59:00,756 THANKS FOR INVITING ME TO 8519 04:59:00,822 --> 04:59:02,391 PARTICIPATE IN THIS INTERESTING 8520 04:59:02,457 --> 04:59:02,958 WORKSHOP. 8521 04:59:03,025 --> 04:59:04,826 I LOOK FORWARD TO SEEING THINGS 8522 04:59:04,893 --> 04:59:08,430 CONVERGE TOMORROW TO SOME 8523 04:59:08,497 --> 04:59:13,135 POSITIVE RESULTS. 8524 04:59:13,201 --> 04:59:14,336 WELL-DEFINED NEGATIVE RESULTS. 8525 04:59:14,403 --> 04:59:17,072 IN THE KIND OF MECHANISTIC BASIC 8526 04:59:17,139 --> 04:59:20,542 WORLD I LIVE IN, MOLECULAR CELL 8527 04:59:20,609 --> 04:59:22,411 BIOLOGY, NEURAL CIRCUITS, ET 8528 04:59:22,477 --> 04:59:24,346 CETERA, WE HAVE A TYPICAL 8529 04:59:24,413 --> 04:59:26,281 STUDIES, A LOT OF EXPERIMENTS, 8530 04:59:26,348 --> 04:59:28,684 AND SOME ARE POSITIVE AND SOME 8531 04:59:28,750 --> 04:59:29,551 ARE NEGATIVE. 8532 04:59:29,618 --> 04:59:31,953 SO EVEN IN ONE FIGURE THERE 8533 04:59:32,020 --> 04:59:35,991 MIGHT BE A TEST OF 30 GENES, TWO 8534 04:59:36,058 --> 04:59:37,426 OF WHICH GIVE RESULTS THAT ARE 8535 04:59:37,492 --> 04:59:40,295 POSITIVE AND FOLLOWED UP ON, THE 8536 04:59:40,362 --> 04:59:43,332 OTHER ONES ARE LEFT, THOSE ARE 8537 04:59:43,398 --> 04:59:44,966 NEGATIVE RESULTS. 8538 04:59:45,033 --> 04:59:46,301 WE STILL COUNT THAT AS A PAPER 8539 04:59:46,368 --> 04:59:47,302 WITH POSITIVE RESULTS. 8540 04:59:47,369 --> 04:59:50,672 FROM THE BIG DATA PERSPECTIVE, 8541 04:59:50,739 --> 04:59:53,742 I'M A PROPONENT OF THE IDEA THAT 8542 04:59:53,809 --> 04:59:57,446 AGGREGATION OF MANY SMALL 8543 04:59:57,512 --> 04:59:58,146 EXPERIMENTS, AGGREGATED PROPERLY 8544 04:59:58,213 --> 04:59:59,214 IN THE KNOWLEDGE BASE WHICH I 8545 04:59:59,281 --> 05:00:01,049 SPEND A LOT OF TIME DOING, 8546 05:00:01,116 --> 05:00:04,086 BECOMES BIG DATA, YOU CAN USE 8547 05:00:04,152 --> 05:00:05,987 MACHINE LEARNING AND A.I. TO 8548 05:00:06,054 --> 05:00:07,856 COMPUTE THAT INFORMATION. 8549 05:00:07,923 --> 05:00:09,825 WE WANT TO AGGREGATE AS MUCH 8550 05:00:09,891 --> 05:00:10,992 POSITIVE AND NEGATIVE DATA. 8551 05:00:11,059 --> 05:00:16,365 SO, THIS IS ONE OF THE 8552 05:00:16,431 --> 05:00:21,770 MOTIVATIONS, WE REALIZED ABOUT 8553 05:00:21,837 --> 05:00:22,871 HALF THE EXPERIMENTAL RESULTS, 8554 05:00:22,938 --> 05:00:26,041 IN SOMEONE'S Ph.D. THESIS, 8555 05:00:26,108 --> 05:00:27,576 WERE NEVER PUBLISHED. 8556 05:00:27,642 --> 05:00:32,481 IT'S FILE DRAWER PROBLEM WE 8557 05:00:32,547 --> 05:00:33,348 HEARD ABOUT. 8558 05:00:33,415 --> 05:00:35,083 ACCUMULATING THOSE IS A GOOD 8559 05:00:35,150 --> 05:00:36,985 VALUE, THE EXPERIENCE HAS BEEN 8560 05:00:37,052 --> 05:00:37,652 GOOD. 8561 05:00:37,719 --> 05:00:45,761 WE EVEN HAD WHAT I CALL VECRs, 8562 05:00:45,827 --> 05:00:48,764 EARLY CAREER RESEARCHERS, HIGH 8563 05:00:48,830 --> 05:00:49,631 SCHOOL STUDENTS, UNDERGRADUATES, 8564 05:00:49,698 --> 05:00:51,333 SOMETIMES COMPLETE ON THEIR OWN 8565 05:00:51,400 --> 05:00:54,503 AND PUBLISH IN PEER-REVIEWED 8566 05:00:54,569 --> 05:00:55,137 FORMAT. 8567 05:00:55,203 --> 05:01:00,842 ONE CRITERIA FOR 8568 05:01:00,909 --> 05:01:01,410 MICRO-PUBLICATION BIOLOGY, 8569 05:01:01,476 --> 05:01:02,911 TRYING TO BALANCE OUT 8570 05:01:02,978 --> 05:01:03,912 TRADITIONAL THINKING ABOUT 8571 05:01:03,979 --> 05:01:06,681 PUBLICATION AND USING HUMAN 8572 05:01:06,748 --> 05:01:07,983 NATURE, WHAT'S CURRENTLY THERE, 8573 05:01:08,049 --> 05:01:10,185 PEOPLE WANT CREDIT AND WANT PEER 8574 05:01:10,252 --> 05:01:12,020 REVIEW, THEY WANT UNITS OF A 8575 05:01:12,087 --> 05:01:17,292 CERTAIN SIZE, I WANT IT TO BE 8576 05:01:17,359 --> 05:01:17,626 MICRO. 8577 05:01:17,692 --> 05:01:19,928 MICRO PUBLICATIONS ARE DECI OR 8578 05:01:19,995 --> 05:01:22,097 MILLI, BUT NOT MICRO. 8579 05:01:22,164 --> 05:01:23,298 IT'S A DIFFERENT ISSUE. 8580 05:01:23,365 --> 05:01:25,700 SO THAT WE CAN CAPTURE THAT. 8581 05:01:25,767 --> 05:01:27,669 SO THE IDEA IS JUST TO REALLY 8582 05:01:27,736 --> 05:01:29,638 CONTROL THE PLATFORM AND MAKE IT 8583 05:01:29,704 --> 05:01:32,274 EASIER FOR PEOPLE TO GET OUT, 8584 05:01:32,340 --> 05:01:34,543 WHAT WE'RE DOING IS SINGLE 8585 05:01:34,609 --> 05:01:35,444 FIGURE PUBLICATIONS. 8586 05:01:35,510 --> 05:01:37,612 NOW, WE HAD HOPED THAT THESE 8587 05:01:37,679 --> 05:01:41,783 WOULD BE REALLY SIMPLE AND ONE 8588 05:01:41,850 --> 05:01:42,083 EXPERIMENT. 8589 05:01:42,150 --> 05:01:44,753 TURNS OUT PEOPLE WANT TO PUT 8590 05:01:44,820 --> 05:01:46,488 MORE IN THERE. 8591 05:01:46,555 --> 05:01:48,356 THAT'S JUST -- IT'S OKAY. 8592 05:01:48,423 --> 05:01:50,225 IT MAKES IT HARDER TO REACH THE 8593 05:01:50,292 --> 05:01:54,029 OTHER GOAL TO TRY TO CAPTURE ALL 8594 05:01:54,095 --> 05:01:57,632 THAT INFORMATION WITH ONTOLOGY 8595 05:01:57,699 --> 05:01:58,633 DIRECT INSERTIONS, CONNECTED 8596 05:01:58,700 --> 05:01:59,601 INSERTIONS, THAT'S LESS 8597 05:01:59,668 --> 05:02:00,268 SUCCESSFUL. 8598 05:02:00,335 --> 05:02:02,437 IN TERMS OF GETTING RESEARCH OUT 8599 05:02:02,504 --> 05:02:04,706 IT'S BEEN SUCCESSFUL. 8600 05:02:04,773 --> 05:02:07,943 MY PERSONAL FIELD, C. ELEGANS, A 8601 05:02:08,009 --> 05:02:10,779 WORM THAT'S THE DARLING OF LAB 8602 05:02:10,846 --> 05:02:15,050 SCIENTISTS, I THINK IT'S 8603 05:02:15,116 --> 05:02:16,351 STATISTICALLY SIGNIFICANT, I'M 8604 05:02:16,418 --> 05:02:18,119 P-HACKING, 5% MORE ARE 8605 05:02:18,186 --> 05:02:19,187 PUBLISHED, WE'RE GETTING EXCESS 8606 05:02:19,254 --> 05:02:20,522 PUBLICATIONS LAST FOUR YEARS 8607 05:02:20,589 --> 05:02:21,189 BECAUSE OF THIS. 8608 05:02:21,256 --> 05:02:22,390 THAT'S THE RESULTS THAT WOULDN'T 8609 05:02:22,457 --> 05:02:24,559 BE THERE. 8610 05:02:24,626 --> 05:02:25,360 THAT'S A SUCCESS. 8611 05:02:25,427 --> 05:02:28,497 I THINK THAT'S THE MAIN POINT, 8612 05:02:28,563 --> 05:02:30,031 LITTLE DATA BECOMES BIG DATA. 8613 05:02:30,098 --> 05:02:31,766 NEGATIVE RESULTS ARE IMPORTANT 8614 05:02:31,833 --> 05:02:33,535 BUT THERE'S ACTUALLY A GRANULAR 8615 05:02:33,602 --> 05:02:35,837 LEVEL STUDIES HAVE POSITIVE AND 8616 05:02:35,904 --> 05:02:37,272 NEGATIVE ONES, AND THEY ARE 8617 05:02:37,339 --> 05:02:39,007 REALLY IMPORTANT TO PULL OUT AND 8618 05:02:39,074 --> 05:02:48,383 TO GET INTO PUBLIC DOMAINS. 8619 05:02:52,787 --> 05:02:54,689 >> HI, EVERYONE. 8620 05:02:54,756 --> 05:02:56,024 I'M CO-FOUNDER AND CHIEF 8621 05:02:56,091 --> 05:02:57,225 SCIENTIFIC OFFICER OF ARCADIA 8622 05:02:57,292 --> 05:03:01,162 SCIENCE, HEAD OF OPEN SCIENCE AT 8623 05:03:01,229 --> 05:03:02,564 A NON-PROFIT WHICH I WON'T TALK 8624 05:03:02,631 --> 05:03:06,835 ABOUT TODAY BUT WE'RE FUNDING 8625 05:03:06,902 --> 05:03:11,439 BRAND NEW ZIRTA EXPERIMENTS IN 8626 05:03:11,506 --> 05:03:13,408 PUBLISHING, IF ANYONE HAS CRAZY 8627 05:03:13,475 --> 05:03:16,378 IDEAS TALK TO ME LATER. 8628 05:03:16,444 --> 05:03:23,051 IT'S A SCIENCE COMPANY, TRYING 8629 05:03:23,118 --> 05:03:25,020 TO RETHINK HOW WE DO SCIENCE, 8630 05:03:25,086 --> 05:03:27,656 HOW WE SUPPORT AND FUND SCIENCE, 8631 05:03:27,722 --> 05:03:29,491 HOW WE COMMUNICATE SCIENCE. 8632 05:03:29,558 --> 05:03:31,259 IT'S THE LARGER BROADER GOAL FOR 8633 05:03:31,326 --> 05:03:31,459 US. 8634 05:03:31,526 --> 05:03:33,895 AND ONE OF THE THEMES I WANTED 8635 05:03:33,962 --> 05:03:36,665 TO BRING UP, A KEY ISSUE FOR US, 8636 05:03:36,731 --> 05:03:38,567 IS SOMETHING THAT I WANTED TO 8637 05:03:38,633 --> 05:03:40,569 CHALLENGE THE NARRATIVE AND 8638 05:03:40,635 --> 05:03:41,970 FUNCTIONS AROUND SOMETHING THAT 8639 05:03:42,037 --> 05:03:44,072 MAYBE TED SAID EARLIER, WE'RE A 8640 05:03:44,139 --> 05:03:45,473 FOR-PROFIT COMPANY AND WE'RE 8641 05:03:45,540 --> 05:03:46,875 ALSO AN OPEN SCIENCE 8642 05:03:46,942 --> 05:03:47,309 ORGANIZATION. 8643 05:03:47,375 --> 05:03:50,278 WE ARE TRYING TO PUSH THE NEEDLE 8644 05:03:50,345 --> 05:03:51,379 AND CHALLENGE THIS IDEA THAT IS 8645 05:03:51,446 --> 05:03:54,349 IT THE CASE THAT THESE ARE LIKE 8646 05:03:54,416 --> 05:03:57,185 ALWAYS INTENTIONED OR WHAT WE 8647 05:03:57,252 --> 05:04:01,323 BELIEVE IS STRONG SYNERGY WITH 8648 05:04:01,389 --> 05:04:02,223 ONE ANOTHER. 8649 05:04:02,290 --> 05:04:04,092 WE'RE ALSO TRYING TO PUSH 8650 05:04:04,159 --> 05:04:05,927 AGAINST -- ALL OUR WORK, THE 8651 05:04:05,994 --> 05:04:08,563 SCIENCE IS DONE IN HOUSE, 8652 05:04:08,630 --> 05:04:11,566 PUBLISHED ON OUR OWN PLATFORM 8653 05:04:11,633 --> 05:04:12,934 OUTSIDE ACADEMIC JOURNAL SYSTEM. 8654 05:04:13,001 --> 05:04:16,338 IT GETS BACK TO SOMETHING 8655 05:04:16,404 --> 05:04:18,740 ALEXANDER SAID, WHY WE GOT INTO 8656 05:04:18,807 --> 05:04:20,041 THIS FOR COLLECTIVE SCIENTIFIC 8657 05:04:20,108 --> 05:04:21,810 BENEFIT, REGARDLESS OF ANY 8658 05:04:21,876 --> 05:04:22,911 SECTOR YOU'RE TALKING ABOUT, 8659 05:04:22,978 --> 05:04:26,381 SORT OF A LOT OF TIMES WE'RE NOT 8660 05:04:26,448 --> 05:04:27,916 PUBLISHING IN ACADEMIA BECAUSE 8661 05:04:27,983 --> 05:04:31,519 WE'RE HOLDING ON TO THINGS FOR 8662 05:04:31,586 --> 05:04:32,821 CREDIT OR REWARDS AND IN THE 8663 05:04:32,887 --> 05:04:35,023 PRIVATE SECTOR MIGHT BE TO 8664 05:04:35,090 --> 05:04:37,425 PROTECT PROFITS, WE LIKE TO 8665 05:04:37,492 --> 05:04:38,760 CHALLENGE THIS ASSUMPTION, CAN 8666 05:04:38,827 --> 05:04:40,729 WE DO THIS OUTSIDE ASSUMPTIONS 8667 05:04:40,795 --> 05:04:43,565 AND PUSH THE NEEDLE AND GET 8668 05:04:43,632 --> 05:04:44,899 EVERYONE TO RELEASE MORE SCIENCE 8669 05:04:44,966 --> 05:04:48,670 BECAUSE WE ALL NEED IT FOR 8670 05:04:48,737 --> 05:04:58,046 ACCELERATING ADVANCEMENTS FOR 8671 05:04:58,113 --> 05:04:59,247 ALL OF SOCIETY. 8672 05:04:59,314 --> 05:05:01,182 WE'RE USING A PUBLIC OPEN 8673 05:05:01,249 --> 05:05:02,717 PLATFORM BUILD BY KNOWLEDGE 8674 05:05:02,784 --> 05:05:04,686 FUTURES GROUP OUT OF M.I.T. 8675 05:05:04,753 --> 05:05:05,487 I PICKED THIS BECAUSE IT WAS 8676 05:05:05,553 --> 05:05:06,454 SOMETHING THAT COULD BE 8677 05:05:06,521 --> 05:05:07,422 AVAILABLE TO OTHERS SO WHEN 8678 05:05:07,489 --> 05:05:09,824 PEOPLE SAY HOW DO WE DO WHAT YOU 8679 05:05:09,891 --> 05:05:11,693 DO, IF IT WERE TO CATCH ON, WE 8680 05:05:11,760 --> 05:05:15,864 CAN POINT THEM AND GIVE THEM A 8681 05:05:15,930 --> 05:05:16,131 SOLUTION. 8682 05:05:16,197 --> 05:05:18,166 WE TRY TO PUBLISH EARLY AND 8683 05:05:18,233 --> 05:05:20,468 OFTEN IN A MODULAR WAY. 8684 05:05:20,535 --> 05:05:21,136 POST-PUBLICATION REVIEW MEANS WE 8685 05:05:21,202 --> 05:05:24,606 WANT ALL OF THE FEEDBACK ON IT 8686 05:05:24,673 --> 05:05:25,840 TO BE PUBLIC. 8687 05:05:25,907 --> 05:05:27,809 IN FACT IF SOMEONE E-MAILS US 8688 05:05:27,876 --> 05:05:29,678 PRIVATELY WE SAY WOULD YOU MIND 8689 05:05:29,744 --> 05:05:31,079 PUTTING THAT PUBLICLY AVAILABLE 8690 05:05:31,146 --> 05:05:33,615 BECAUSE IT BENEFITS THE READER, 8691 05:05:33,682 --> 05:05:35,483 NOT JUST US AS AUTHORS. 8692 05:05:35,550 --> 05:05:38,987 WHILE WE WANT PUBLIC FEED BACK, 8693 05:05:39,054 --> 05:05:41,756 WE'RE NOT DOING PRE-PUBLICATION 8694 05:05:41,823 --> 05:05:44,292 REVIEW, EVEN THOUGH IT IS 8695 05:05:44,359 --> 05:05:45,226 DIFFICULT, REQUIRES PRO-ACTIVE, 8696 05:05:45,293 --> 05:05:47,762 YOU KNOW, SEEKING TO GET THAT 8697 05:05:47,829 --> 05:05:49,297 PUBLIC FEEDBACK, YOU KNOW, THAT 8698 05:05:49,364 --> 05:05:50,598 FEEDBACK, WHAT PEOPLE THINK WHEN 8699 05:05:50,665 --> 05:05:52,067 THEY READ SOMETHING, THE MOMENT 8700 05:05:52,133 --> 05:05:56,304 IT DROPS, IS NOT AS IMPORTANT TO 8701 05:05:56,371 --> 05:05:59,040 US AS WE USE, STANDING TEST OF 8702 05:05:59,107 --> 05:06:01,443 TIME, ORTHOGONAL, THE REALLY 8703 05:06:01,509 --> 05:06:02,444 MEASURES THAT TELL US WHETHER OR 8704 05:06:02,510 --> 05:06:04,179 NOT SCIENCE IS RIGOROUS AND HIGH 8705 05:06:04,245 --> 05:06:05,280 QUALITY, THE SUBSTANCE OF THAT 8706 05:06:05,346 --> 05:06:06,815 IS SO MUCH MORE IMPORTANT THAN 8707 05:06:06,881 --> 05:06:07,682 APPEARANCE OF THAT. 8708 05:06:07,749 --> 05:06:09,117 SO EVEN THOUGH WE WANT THESE 8709 05:06:09,184 --> 05:06:11,086 THINGS TO BE GOOD PRACTICES, OUT 8710 05:06:11,152 --> 05:06:13,922 IN THE OPEN, THAT'S NOT THE 8711 05:06:13,988 --> 05:06:16,991 BE-ALL, END-ALL, WE'LL HAVE TO 8712 05:06:17,058 --> 05:06:18,426 TAKE DATA, METADATA, PROTOCOLS, 8713 05:06:18,493 --> 05:06:21,029 CODE, ALL OF THOSE THINGS 8714 05:06:21,096 --> 05:06:22,564 SERIOUSLY, 100% OF WHAT WE DROP 8715 05:06:22,630 --> 05:06:25,533 ON OUR PLATFORM HAS ALL OF THOSE 8716 05:06:25,600 --> 05:06:27,569 THINGS AVAILABLE THE MOMENT IT'S 8717 05:06:27,635 --> 05:06:28,369 AVAILABLE PUBLICLY. 8718 05:06:28,436 --> 05:06:31,306 AND SO TO GIVE SOME IDEA HOW 8719 05:06:31,372 --> 05:06:34,175 THAT'S GONE, IN THE LAST TWO 8720 05:06:34,242 --> 05:06:37,011 YEARS WE'VE HAD OUR PUBLICATION 8721 05:06:37,078 --> 05:06:39,514 PLATFORM ONLINE WE'VE PUBLISHED 8722 05:06:39,581 --> 05:06:40,782 46 DIFFERENT RESEARCH ARTIFACTS, 8723 05:06:40,849 --> 05:06:43,985 TWO PER MONTH, RIGHT NOW ABOUT 8724 05:06:44,052 --> 05:06:45,086 40 SCIENTISTS, STARTING WITH 8725 05:06:45,153 --> 05:06:47,088 MUCH FEWER SO GIVES YOU AN IDEA 8726 05:06:47,155 --> 05:06:49,691 OF THE CLIP WE'RE ABLE TO 8727 05:06:49,758 --> 05:06:51,359 RELEASE A AN IDEA, DATASET, 8728 05:06:51,426 --> 05:06:51,626 WHATEVER. 8729 05:06:51,693 --> 05:06:53,294 IT COULD BE ANY TYPE OF SHAPE, 8730 05:06:53,361 --> 05:06:55,497 REALLY TRYING TO MOVE THE NEEDLE 8731 05:06:55,563 --> 05:06:55,730 EARLIER. 8732 05:06:55,797 --> 05:06:58,566 JUST TO GET TO THE SORT OF HUMAN 8733 05:06:58,633 --> 05:07:00,635 NATURE PROBLEM, AS YOU'LL SEE WE 8734 05:07:00,702 --> 05:07:02,403 TRY TO PULL OUT LITERALLY ALL 8735 05:07:02,470 --> 05:07:04,272 THE STOPS THAT PREVENT PEOPLE 8736 05:07:04,339 --> 05:07:06,875 FROM SHARING RESEARCH, PROVIDE 8737 05:07:06,941 --> 05:07:07,709 SUPPORT AND INFRASTRUCTURE AND 8738 05:07:07,776 --> 05:07:08,843 REQUIRE THEM TO SHARE 8739 05:07:08,910 --> 05:07:12,247 EVERYTHING, EVEN SO WE HAVE A 8740 05:07:12,313 --> 05:07:13,314 LOT OF PERFECTIONIST SCIENTISTS 8741 05:07:13,381 --> 05:07:14,983 THAT WANT TO DO ONE MORE THING, 8742 05:07:15,049 --> 05:07:16,751 THEY WANT IT TO BE REALLY GOOD. 8743 05:07:16,818 --> 05:07:17,819 IT'S NOT DONE YET. 8744 05:07:17,886 --> 05:07:21,890 IT IS A THING WE CAN TEND WITH, 8745 05:07:21,956 --> 05:07:23,858 WE SIT OUTSIDE THE SYSTEM, TAKE 8746 05:07:23,925 --> 05:07:25,627 AWAY THE BARRIERS, THERE'S STILL 8747 05:07:25,693 --> 05:07:27,228 THIS PROBLEM WORTH MENTIONING. 8748 05:07:27,295 --> 05:07:29,197 FOR US, YOU KNOW, WE'RE DOING 8749 05:07:29,264 --> 05:07:30,398 THIS, OF COURSE IT'S A MISSION 8750 05:07:30,465 --> 05:07:33,368 FOR US TO MAKE SURE WE'RE 8751 05:07:33,434 --> 05:07:34,469 ADVANCING THIS SCIENCE 8752 05:07:34,536 --> 05:07:36,404 COLLECTIVELY BUT IT'S ALSO 8753 05:07:36,471 --> 05:07:36,905 PRAGMATIC. 8754 05:07:36,971 --> 05:07:40,575 WE DO THINK THAT OPEN SCIENCE IS 8755 05:07:40,642 --> 05:07:42,110 BETTER SCIENCE, FASTEST WAY, IF 8756 05:07:42,177 --> 05:07:44,412 WE WANT TO GO FROM BASIC TO 8757 05:07:44,479 --> 05:07:45,079 COMMERCIALIZATION QUICKLY I 8758 05:07:45,146 --> 05:07:47,248 WOULD DO IT IN THE OPEN, GOOD 8759 05:07:47,315 --> 05:07:48,683 IDEAS WHEREVER THEY LIVE, NOT 8760 05:07:48,750 --> 05:07:49,884 JUST IN OUR BUILDING. 8761 05:07:49,951 --> 05:07:52,387 ONE OF THE THINGS I MENTIONED, 8762 05:07:52,453 --> 05:07:53,922 WE MANDATE, WE SAY, WE EXPECT 8763 05:07:53,988 --> 05:07:55,890 ALL OF OUR SCIENTISTS TO SHARE 8764 05:07:55,957 --> 05:07:59,394 THEIR SCIENCE, EVERY PROJECT WE 8765 05:07:59,460 --> 05:08:03,097 DO IS BUILT INTO THE WORKFLOWS, 8766 05:08:03,164 --> 05:08:04,199 KICK OFF DISCUSSING WITH 8767 05:08:04,265 --> 05:08:05,533 PUBLICATIONS TEAM THE PLAN FOR 8768 05:08:05,600 --> 05:08:08,269 GETTING THIS OUT UP FRONT. 8769 05:08:08,336 --> 05:08:10,138 AND THAT INCLUDES SHARING THINGS 8770 05:08:10,205 --> 05:08:11,973 THAT DIDN'T WORK OR SHARING 8771 05:08:12,040 --> 05:08:14,442 THINGS THAT WE MAY DECIDE NOT TO 8772 05:08:14,509 --> 05:08:14,709 PURSUE. 8773 05:08:14,776 --> 05:08:17,212 AND SO WE HAVE A SECTION ON OUR 8774 05:08:17,278 --> 05:08:19,214 PLATFORM THAT'S CALLED THE ICE 8775 05:08:19,280 --> 05:08:21,850 BOX, THINGS WE'RE NOT PURSUING, 8776 05:08:21,916 --> 05:08:24,052 SO THAT'S ONE CASE IN WHICH WE 8777 05:08:24,118 --> 05:08:26,287 ARE JUST SAYING WE'RE NOT GOING 8778 05:08:26,354 --> 05:08:28,957 TO DO THIS, IT DIDN'T WORK. 8779 05:08:29,023 --> 05:08:30,892 IT'S NOT BECAUSE SCIENCE WAS 8780 05:08:30,959 --> 05:08:31,226 BAD. 8781 05:08:31,292 --> 05:08:32,560 FOR WHATEVER STRATEGIC REASON WE 8782 05:08:32,627 --> 05:08:33,962 DECIDED NOT TO PURSUE IT OR 8783 05:08:34,028 --> 05:08:37,031 COULD BE IT DIDN'T WORK AND OUR 8784 05:08:37,098 --> 05:08:38,900 ASSUMPTIONS WERE WRONG, WE TRIED 8785 05:08:38,967 --> 05:08:40,435 TO BE TRANSPARENT SO IT'S CLEAR 8786 05:08:40,501 --> 05:08:42,403 TO EVERYONE WHAT COULD BE THE 8787 05:08:42,470 --> 05:08:44,606 SORT OF REASONS WE DIDN'T MOVE 8788 05:08:44,672 --> 05:08:45,039 FORWARD. 8789 05:08:45,106 --> 05:08:47,208 WE ALSO WANT TO SHARE CLEARLY, 8790 05:08:47,275 --> 05:08:48,676 YOU KNOW, WHEN WE DO PUBLISH 8791 05:08:48,743 --> 05:08:50,945 SOMETHING EVEN IF IT IS A 8792 05:08:51,012 --> 05:08:54,115 POSITIVE RESULT WE ASK QUESTIONS 8793 05:08:54,182 --> 05:08:56,417 OF THE AUDIENCE AND REALLY 8794 05:08:56,484 --> 05:08:57,852 SUGGEST FUTURE DIRECTIONS 8795 05:08:57,919 --> 05:08:58,820 BECAUSE OFTEN THE ASSUMPTION 8796 05:08:58,887 --> 05:09:00,355 WHEN YOU PUBLISH SOMETHING IS 8797 05:09:00,421 --> 05:09:01,623 THAT YOU'RE PLANTING YOUR FLAG. 8798 05:09:01,689 --> 05:09:03,725 WHAT I DON'T WANT TO HAPPEN IS 8799 05:09:03,791 --> 05:09:05,260 PEOPLE TO SAY, OH, ARCADIA IS 8800 05:09:05,326 --> 05:09:07,695 DOING THIS WORK IN THIS AREA AND 8801 05:09:07,762 --> 05:09:09,430 THEY HAVE RESOURCES, AND A LOT 8802 05:09:09,497 --> 05:09:10,832 OF PEOPLE, THEREFORE WE SHOULD 8803 05:09:10,899 --> 05:09:11,733 STEER CLEAR. 8804 05:09:11,799 --> 05:09:13,701 NO, WE WANT YOU TO WORK OUT. 8805 05:09:13,768 --> 05:09:15,069 WE WANT EVERYONE, IT'S GOING TO 8806 05:09:15,136 --> 05:09:17,739 TAKE ALL OF US TO GET THERE. 8807 05:09:17,805 --> 05:09:19,173 WHEN WE PUBLISH SOMETHING, ALL 8808 05:09:19,240 --> 05:09:21,042 OF OUR SCIENCE GETS BETTER 8809 05:09:21,109 --> 05:09:22,577 FASTER, IT'S VERY -- WE'RE 8810 05:09:22,644 --> 05:09:24,112 TRYING TO BE AS TRANSPARENT AS 8811 05:09:24,178 --> 05:09:25,580 POSSIBLE WITH WHAT WE'RE DOING, 8812 05:09:25,647 --> 05:09:28,716 WHY, WE WANT TO DO THIS 8813 05:09:28,783 --> 05:09:29,150 TOGETHER. 8814 05:09:29,217 --> 05:09:32,553 AND SO ONE THING I WANTED TO 8815 05:09:32,620 --> 05:09:34,389 MENTION ABOUT DOING THINGS THIS 8816 05:09:34,455 --> 05:09:40,862 WAY IS THAT, YOU KNOW, WHEN YOU 8817 05:09:40,929 --> 05:09:43,097 START CHANGING ASSUMPTIONS, WE 8818 05:09:43,164 --> 05:09:44,465 EXPECT OUR SCIENTISTS TO PUBLISH 8819 05:09:44,532 --> 05:09:49,704 YOUR RESEARCH NO MATTER OUTCOME 8820 05:09:49,771 --> 05:09:51,706 AND MAKE SURE RESEARCH ARTIFACTS 8821 05:09:51,773 --> 05:09:54,676 ARE AVAILABLE, SIMILAR TO A 8822 05:09:54,742 --> 05:09:55,877 SITUATION, OSTP MEMO, FOR 8823 05:09:55,944 --> 05:09:57,612 EXAMPLE, YOU HEAR WHEN PEOPLE 8824 05:09:57,679 --> 05:10:00,248 SAY, OKAY, NOW YOU'RE REQUIRING 8825 05:10:00,315 --> 05:10:01,983 DATA SHARING, METADATA SHARING, 8826 05:10:02,050 --> 05:10:04,919 YOU HEAR CRIES ABOUT UNFUNDED 8827 05:10:04,986 --> 05:10:06,587 MANDATES, WHAT THAT IS IS AN 8828 05:10:06,654 --> 05:10:09,090 ARTIFACT OF THIS MOMENT IN TIME 8829 05:10:09,157 --> 05:10:10,758 WHERE WE GO FROM NOT EXPECTING 8830 05:10:10,825 --> 05:10:12,493 THAT DATA RELEASE TO EXPECTING 8831 05:10:12,560 --> 05:10:12,961 IT. 8832 05:10:13,027 --> 05:10:14,696 IF YOU HAVE THAT ASSUMPTION UP 8833 05:10:14,762 --> 05:10:17,332 FRONT THIS IS HOW WE DO SCIENCE, 8834 05:10:17,398 --> 05:10:20,401 IN A MINUTE THAT BECOMES 8835 05:10:20,468 --> 05:10:22,136 BASELINE ASSUMPTION, YOU NOW 8836 05:10:22,203 --> 05:10:25,273 CHANGE YOUR ENTIRE SCIENTIFIC 8837 05:10:25,340 --> 05:10:25,974 WORK FLOW. 8838 05:10:26,040 --> 05:10:28,476 IF YOU DON'T WORK, WORK, WORK, 8839 05:10:28,543 --> 05:10:29,844 AND WRITE, WRITE, WRITE, AND 8840 05:10:29,911 --> 05:10:32,347 HOPE IT'S WORTH IT TO PUBLISH, 8841 05:10:32,413 --> 05:10:33,614 IT'S DIFFICULT DEPENDING ON 8842 05:10:33,681 --> 05:10:33,948 RESULT. 8843 05:10:34,015 --> 05:10:36,451 PEOPLE ARE NOT RELEASING IT. 8844 05:10:36,517 --> 05:10:39,754 IF THE PRIMARY ASSUMPTION IS 8845 05:10:39,821 --> 05:10:41,055 YOU'RE RELEASING DATA ARTIFACTS 8846 05:10:41,122 --> 05:10:42,590 YOU NOW DO SOMETHING THAT 8847 05:10:42,657 --> 05:10:45,426 EVERYBODY DOES, TRY TO MAKE YOUR 8848 05:10:45,493 --> 05:10:47,962 LIFE EASIER. 8849 05:10:48,029 --> 05:10:49,263 YOU SAY WHAT SUCKS AT THE END WE 8850 05:10:49,330 --> 05:10:50,231 HAVE TO COLLECT THE INFORMATION 8851 05:10:50,298 --> 05:10:52,433 AND FIGURE OUT WHAT THE HELL WE 8852 05:10:52,500 --> 05:10:56,604 DID AND INSTEAD YOU SAY WAIT A 8853 05:10:56,671 --> 05:10:58,906 SECOND, AS WE GO COLLECT -- 8854 05:10:58,973 --> 05:11:00,775 BETTER DATA HYGIENE, COLLECT 8855 05:11:00,842 --> 05:11:02,210 ARTIFACTS, DO CODE REVIEW ALONG 8856 05:11:02,276 --> 05:11:04,045 THE WAY, ALL THOSE OTHER THINGS. 8857 05:11:04,112 --> 05:11:06,347 NOW IT'S BUILT IN BECAUSE IT'S 8858 05:11:06,414 --> 05:11:06,748 EASIER. 8859 05:11:06,814 --> 05:11:09,083 BECAUSE WHEN THAT'S OUR BASELINE 8860 05:11:09,150 --> 05:11:11,719 ASSUMPTION IT'S EASIER TO HAVE 8861 05:11:11,786 --> 05:11:13,688 BETTER DATA PRACTICES AND FLIP 8862 05:11:13,755 --> 05:11:16,124 IT OPEN. 8863 05:11:16,190 --> 05:11:17,325 NO MATTER THE RESULTS. 8864 05:11:17,392 --> 05:11:19,827 WHICH GETS BACK AT OUR ABILITY 8865 05:11:19,894 --> 05:11:22,230 TO SHARE THINGS IN AN EASIER 8866 05:11:22,296 --> 05:11:22,397 WAY. 8867 05:11:22,463 --> 05:11:25,566 THAT'S THE POINT I WANTED TO 8868 05:11:25,633 --> 05:11:25,767 MAKE. 8869 05:11:25,833 --> 05:11:30,338 WE CAN CHANGE THE WAIT WE -- 8870 05:11:30,405 --> 05:11:33,274 CHANGE THE WAY WE OPERATE. 8871 05:11:33,341 --> 05:11:34,976 WHEN YOU TAKE AWAY BOTTLENECKS 8872 05:11:35,043 --> 05:11:36,577 YOU LEARN THE REAL BOTTLENECKS. 8873 05:11:36,644 --> 05:11:37,912 THAT'S SOMETHING THAT'S HELPFUL 8874 05:11:37,979 --> 05:11:46,387 THAT WE LEARNED. 8875 05:11:52,827 --> 05:11:54,595 >> THANK YOU FOR THE SUPER 8876 05:11:54,662 --> 05:11:59,167 INSPIRING PRESENTATION. 8877 05:11:59,233 --> 05:12:00,034 THANK YOU ALL. 8878 05:12:00,101 --> 05:12:03,638 IT'S GREAT TO BE IN THE ROOM 8879 05:12:03,704 --> 05:12:05,373 WITH SO MANY CHANGEMAKERS. 8880 05:12:05,440 --> 05:12:09,644 I WAS INVITED TO SHARE NATIONAL 8881 05:12:09,710 --> 05:12:11,612 INSTITUTE ON AGING APPROACH TO 8882 05:12:11,679 --> 05:12:14,382 THE PROBLEM DU JOUR. 8883 05:12:14,449 --> 05:12:17,418 SO NIA IS LEADING FUNDING AGENCY 8884 05:12:17,485 --> 05:12:18,619 FOR ALZHEIMER'S RESEARCH AND 8885 05:12:18,686 --> 05:12:19,821 RELATED DEMENTIA RESEARCH. 8886 05:12:19,887 --> 05:12:23,558 EVERY FEW YEARS WE GO THROUGH A 8887 05:12:23,624 --> 05:12:25,093 STRATEGIC PLANNING EXERCISE, 8888 05:12:25,159 --> 05:12:26,928 CONVENE A LARGE MEETING. 8889 05:12:26,994 --> 05:12:29,697 WE HAD FOUR SUMMITS TO DATE. 8890 05:12:29,764 --> 05:12:34,068 YOU CAN SEE HALF OF THE KEY 8891 05:12:34,135 --> 05:12:36,704 RECOMMENDATIONS FROM THESE 8892 05:12:36,771 --> 05:12:39,807 SUMMITS SPEAK ABOUT THE NEED TO 8893 05:12:39,874 --> 05:12:44,011 CHANGE INCENTIVES, BRING OPEN 8894 05:12:44,078 --> 05:12:46,647 PRACTICES, AND IMPROVE RESEARCH 8895 05:12:46,714 --> 05:12:54,956 RIGOR, REPRODUCIBILITY, 8896 05:12:55,022 --> 05:12:55,823 TRANSLATABILITY. 8897 05:12:55,890 --> 05:12:58,459 SO, HERE IS, AGAIN, THE MAJOR 8898 05:12:58,526 --> 05:12:59,760 STAKEHOLDERS CONTRIBUTING TO THE 8899 05:12:59,827 --> 05:13:02,530 CURRENT SYSTEM OR TRYING TO 8900 05:13:02,597 --> 05:13:04,732 CHANGE THE SYSTEM FROM WITHIN 8901 05:13:04,799 --> 05:13:09,637 BUT ALSO INCREASINGLY FROM 8902 05:13:09,704 --> 05:13:10,271 OUTSIDE. 8903 05:13:10,338 --> 05:13:12,039 SO THE PUBLISHERS, 8904 05:13:12,106 --> 05:13:13,541 NON-TRADITIONAL PUBLISHERS, 8905 05:13:13,608 --> 05:13:16,811 FUNDING AGENCIES, SMALL AND 8906 05:13:16,878 --> 05:13:19,147 LARGE, ACADEMIC INSTITUTIONS, AT 8907 05:13:19,213 --> 05:13:20,581 THE CENTER INDIVIDUALS WITH 8908 05:13:20,648 --> 05:13:29,524 THEIR OWN AGENCY AND 8909 05:13:29,590 --> 05:13:29,991 RESPONSIBILITY. 8910 05:13:30,057 --> 05:13:34,996 AND SO SOME YEARS AGO, I THINK 8911 05:13:35,062 --> 05:13:40,134 2016, WE AT THE NIA LAUNCHED 8912 05:13:40,201 --> 05:13:41,469 ALZHEIMER'S DATABASE, A 8913 05:13:41,536 --> 05:13:43,337 KNOWLEDGE BASE THAT OUR GOAL WAS 8914 05:13:43,404 --> 05:13:47,241 TO SHINE THE LIGHT ON THE SCOPE 8915 05:13:47,308 --> 05:13:51,746 OF THE PROBLEM OF RIGOR AND 8916 05:13:51,812 --> 05:13:59,854 REPRODUCIBILITY OR LACK THEREOF 8917 05:13:59,921 --> 05:14:02,356 ANIMAL MODELS, AND THE TOOL TO 8918 05:14:02,423 --> 05:14:03,991 CONTRIBUTE TO SOLVING THE 8919 05:14:04,058 --> 05:14:05,626 PROBLEM, INCLUDING PUBLICATION 8920 05:14:05,693 --> 05:14:06,093 BIAS. 8921 05:14:06,160 --> 05:14:09,897 TO DATE WE HAVE CURATED OVER 8922 05:14:09,964 --> 05:14:13,634 1400 PUBLISHED STUDIES, OVER THE 8923 05:14:13,701 --> 05:14:14,835 SPAN OF 1996 TO 2023. 8924 05:14:14,902 --> 05:14:16,904 HERE IT GIVES YOU A VERY QUICK 8925 05:14:16,971 --> 05:14:21,509 LINE OF SIGHT INTO MANY ASPECTS 8926 05:14:21,576 --> 05:14:23,144 OF THESE PUBLISHED WORKS 8927 05:14:23,211 --> 05:14:25,246 INCLUDING VERY GRAN ALREADY 8928 05:14:25,313 --> 05:14:26,581 ASSESSMENT OF EXPERIMENTAL 8929 05:14:26,647 --> 05:14:27,848 DESIGN, INFORMALLY WHAT THIS 8930 05:14:27,915 --> 05:14:31,385 DISCUSSION, IT'S ALSO A PLATFORM 8931 05:14:31,452 --> 05:14:33,254 FOR CREATING CITEABLE REPORTS 8932 05:14:33,321 --> 05:14:36,524 AND PRE-PRINTS OF UNPUBLISHED 8933 05:14:36,591 --> 05:14:37,858 STUDIES INCLUDING STUDIES WITH 8934 05:14:37,925 --> 05:14:40,595 NEGATIVE FINDINGS, WORKING ON 8935 05:14:40,661 --> 05:14:45,066 BRINGING FUNCTIONALITY TO OFFER 8936 05:14:45,132 --> 05:14:46,033 PRE-REGISTRATION TO RESEARCH 8937 05:14:46,100 --> 05:14:46,300 COMMUNITY. 8938 05:14:46,367 --> 05:14:48,803 HERE ONE OF THE KEY FEATURES, 8939 05:14:48,869 --> 05:14:54,508 THERE IS THIS RIGOR REPORT CARD 8940 05:14:54,575 --> 05:14:56,244 FOR EVERY CURATED STUDY, WHERE 8941 05:14:56,310 --> 05:15:00,081 USE IT TO MON STORE RIGOR AND 8942 05:15:00,147 --> 05:15:01,949 STUDY DESIGN, CURATORS ASK THE 8943 05:15:02,016 --> 05:15:03,150 QUESTION, IS THE FOLLOWING 8944 05:15:03,217 --> 05:15:04,785 INFORMATION REPORTED IN THE 8945 05:15:04,852 --> 05:15:05,453 STUDY. 8946 05:15:05,519 --> 05:15:07,221 THESE ARE 24 ELEMENTS OF 8947 05:15:07,288 --> 05:15:10,258 RIGOROUS STUDY DESIGN FOR 8948 05:15:10,324 --> 05:15:13,227 TRANSLATIONAL RESEARCH STUDIES, 8949 05:15:13,294 --> 05:15:16,364 TESTING COMPOUNDS, NINE OF THESE 8950 05:15:16,430 --> 05:15:20,568 ARE ARE ACTUALLY ELEMENTS OF 8951 05:15:20,635 --> 05:15:22,536 JUST BASIC RIGOROUS STUDY 8952 05:15:22,603 --> 05:15:24,805 DESIGN. 8953 05:15:24,872 --> 05:15:29,410 ON MY RIGHT SIDE HERE, OUT OF 8954 05:15:29,477 --> 05:15:30,845 1200 STUDIES THERE'S REALLY LOW 8955 05:15:30,911 --> 05:15:33,681 PERCENTAGE OF REPORTING ON MANY 8956 05:15:33,748 --> 05:15:36,083 OF THE ELEMENTS OF STUDY, 8957 05:15:36,150 --> 05:15:39,387 RIGOROUS STUDY DESIGN, INCLUDING 8958 05:15:39,453 --> 05:15:49,363 THESE IN RED, THE BASIC ELEMENTS 8959 05:15:49,430 --> 05:15:53,301 OF SCIENTIFIC RESEARCH RIGOR. 8960 05:15:53,367 --> 05:15:57,805 A COUPLE WORDS AND DOES POLICY 8961 05:15:57,872 --> 05:16:02,610 MATTER, SO IN 2011 AND 2015 NIH 8962 05:16:02,677 --> 05:16:04,245 ISSUED TWO NOTICES, ONE 8963 05:16:04,312 --> 05:16:05,379 REQUIRING TRANSPARENCY REPORTING 8964 05:16:05,446 --> 05:16:06,447 OF FINANCIAL CONFLICT OF 8965 05:16:06,514 --> 05:16:07,782 INTEREST, THE OTHER REQUIRING 8966 05:16:07,848 --> 05:16:13,321 CONSIDERATION OF SEX AS A 8967 05:16:13,387 --> 05:16:14,088 BIOLOGICAL VARIABLE. 8968 05:16:14,155 --> 05:16:16,490 WHAT IS THE IMPACT? 8969 05:16:16,557 --> 05:16:20,761 WHEN WE SCAN TRENDS BY FIVE-YEAR 8970 05:16:20,828 --> 05:16:21,862 INCREMENTS, IN NINE BASIC 8971 05:16:21,929 --> 05:16:23,964 ELEMENTS OF RIGOR THERE'S A 8972 05:16:24,031 --> 05:16:27,234 POSITIVE SIGNIFICANT UPTICK ON 8973 05:16:27,301 --> 05:16:30,071 THESE TWO ELEMENTS FOR WHICH NIH 8974 05:16:30,137 --> 05:16:32,273 AT THAT INFLECTION POINT FOR 8975 05:16:32,340 --> 05:16:34,241 WHICH NIH ISSUED POLICY. 8976 05:16:34,308 --> 05:16:39,380 POLICIES DO MATTER. 8977 05:16:39,447 --> 05:16:41,248 AND THEY MATTER -- HERE I WANT 8978 05:16:41,315 --> 05:16:42,750 TO POINT OUT WHEN WE TALK ABOUT 8979 05:16:42,817 --> 05:16:45,186 SEX AS A BIOLOGICAL VARIABLE, 8980 05:16:45,252 --> 05:16:48,823 THAT IS REQUIRED TO BE REPORTED. 8981 05:16:48,889 --> 05:16:50,858 BUT WHAT IS NOT REQUIRED IS FOR 8982 05:16:50,925 --> 05:16:53,394 STUDIES TO BE BALANCED FOR SEX. 8983 05:16:53,461 --> 05:16:55,796 AND THE STUDIES ARE SERIOUSLY 8984 05:16:55,863 --> 05:16:58,966 LACKING IN THIS DOMAIN. 8985 05:16:59,033 --> 05:17:01,168 WHICH IS PARTICULARLY DAMNING 8986 05:17:01,235 --> 05:17:03,371 KNOWING THAT ALZHEIMER'S LIKE 8987 05:17:03,437 --> 05:17:05,439 MANY COMPLEX DISORDERS, THE 8988 05:17:05,506 --> 05:17:07,408 MOLECULAR ETIOLOGY OF THE 8989 05:17:07,475 --> 05:17:17,885 DISEASE IS PROFOUNDLY SEXUALLY 8990 05:17:17,952 --> 05:17:18,152 DIMORPHIC 8991 05:17:18,219 --> 05:17:21,188 HOW IT WORKS ACROSS SEXES, MANY 8992 05:17:21,255 --> 05:17:24,992 TRANSLATIONAL OUTCOMES. 8993 05:17:25,059 --> 05:17:27,294 WHAT WE'VE STARTED TO DO, 8994 05:17:27,361 --> 05:17:29,497 OBVIOUSLY WE'RE HOPING THAT 8995 05:17:29,563 --> 05:17:30,698 GIVEN INCREASED CONVERSATION 8996 05:17:30,765 --> 05:17:33,667 EMPHASIS ON RIGOR AND 8997 05:17:33,734 --> 05:17:34,735 REPRODUCIBILITY AND PUBLICATION 8998 05:17:34,802 --> 05:17:36,804 BIAS THERE'S MORE CHANGING 8999 05:17:36,871 --> 05:17:39,240 POLICY BUT IN THE MEANWHILE WE 9000 05:17:39,306 --> 05:17:42,710 START TO EMBED REQUIREMENT FOR 9001 05:17:42,777 --> 05:17:44,044 RIGOROUS STUDY DESIGN AND 9002 05:17:44,111 --> 05:17:46,347 TRANSPARENT REPORTING IN OUR 9003 05:17:46,414 --> 05:17:47,214 FUNDING INITIATIVES, 9004 05:17:47,281 --> 05:17:50,151 PARTICULARLY THOSE THAT ARE 9005 05:17:50,217 --> 05:17:51,385 TRANSLATIONAL, TARGETED, WHERE 9006 05:17:51,452 --> 05:17:54,522 THERE'S A LOT AT STAKE, A LOT IS 9007 05:17:54,588 --> 05:17:56,724 PLACED ON OUTCOME OF THOSE 9008 05:17:56,791 --> 05:17:57,458 STUDIES. 9009 05:17:57,525 --> 05:18:01,529 SO, WE HOPE THAT IS, AGAIN, WE 9010 05:18:01,595 --> 05:18:03,831 COMBINE CARROTS AND STICKS 9011 05:18:03,898 --> 05:18:05,800 APPROACH, WE PROVIDE TONIGHTS 9012 05:18:05,866 --> 05:18:06,901 FOR INVESTIGATORS TO REALLY HAVE 9013 05:18:06,967 --> 05:18:10,104 A PLACE TO SHARE DATA, HAVE A 9014 05:18:10,171 --> 05:18:13,808 TEAM TO SUPPORT THEM, WE EXPECT 9015 05:18:13,874 --> 05:18:16,477 THEY SHARE THE PROVENANCE OF 9016 05:18:16,544 --> 05:18:21,315 THEIR FINDINGS REGARDLESS OF THE 9017 05:18:21,382 --> 05:18:25,486 OUTCOME OF THE STUDY. 9018 05:18:25,553 --> 05:18:27,121 SO, SUPPORTS NOT JUST ACTUAL 9019 05:18:27,188 --> 05:18:28,222 RESEARCHERS IN ACADEMIA AND 9020 05:18:28,289 --> 05:18:30,758 INDUSTRY, IT'S A QUICK WAY TO 9021 05:18:30,825 --> 05:18:32,293 SURVEY WHAT'S BEEN PUBLISHED AND 9022 05:18:32,359 --> 05:18:34,795 RIGOR AND DECIDE WHETHER YOU 9023 05:18:34,862 --> 05:18:37,865 WOULD LIKE TO REPEAT A STUDY OR 9024 05:18:37,932 --> 05:18:40,701 REPEAT AND DO IT BETTER. 9025 05:18:40,768 --> 05:18:42,236 DATA SCIENTISTS CAN CONDUCT MET 9026 05:18:42,303 --> 05:18:44,538 ANNUAL SITS, LEARN FROM DIG AS A 9027 05:18:44,605 --> 05:18:47,341 GATED KNOWLEDGE, AND AGENCIES 9028 05:18:47,408 --> 05:18:54,315 ARE PARTNER, A NUMBER OF 9029 05:18:54,381 --> 05:18:57,351 NON-PROFITS, WITH SHARED 9030 05:18:57,418 --> 05:18:57,785 MISSION. 9031 05:18:57,852 --> 05:19:00,087 A TOOL FOR REQUIREMENTS FOR 9032 05:19:00,154 --> 05:19:01,622 RIGOROUS STUDY DESIGN. 9033 05:19:01,689 --> 05:19:06,560 AGAIN, THIS IS AVAILABLE TO ALL 9034 05:19:06,627 --> 05:19:09,196 ENGAGED IN ALZHEIMER'S RESEARCH, 9035 05:19:09,263 --> 05:19:13,234 REGISTRATION IS FREE. 9036 05:19:13,300 --> 05:19:17,171 MY COLLEAGUES ON THE LEADING 9037 05:19:17,238 --> 05:19:20,274 EDGE ARE HERE AND I HOPE YOU 9038 05:19:20,341 --> 05:19:22,643 ENGAGE WITH THEM OVER THE COURSE 9039 05:19:22,710 --> 05:19:23,844 OF TODAY AND TOMORROW. 9040 05:19:23,911 --> 05:19:27,982 I DO WANT TO END THAT WE HAVE -- 9041 05:19:28,048 --> 05:19:29,950 IT'S NOT ONE SOLUTION TO THIS 9042 05:19:30,017 --> 05:19:35,322 BIG PROBLEM, WE'RE APPLYING A 9043 05:19:35,389 --> 05:19:36,090 MULTI-PRONGED APPROACH, 9044 05:19:36,156 --> 05:19:36,657 PARTICULARLY BECAUSE ANIMAL 9045 05:19:36,724 --> 05:19:38,158 MODELS RESEARCH IS NOT THE ONLY 9046 05:19:38,225 --> 05:19:38,926 RESEARCH WE FUND. 9047 05:19:38,993 --> 05:19:42,563 WE LEAN ON OPEN SCIENCE FOR THE 9048 05:19:42,630 --> 05:19:46,400 LAST TWO DECADES TO REALLY USE 9049 05:19:46,467 --> 05:19:49,336 IT TO IMPROVE RIGOR AND 9050 05:19:49,403 --> 05:19:49,703 REPRODUCIBILITY. 9051 05:19:49,770 --> 05:19:51,972 HERE TO ILLUSTRATE IN THE 9052 05:19:52,039 --> 05:19:52,940 PARTICULAR TRANSLATIONAL DOMAIN 9053 05:19:53,007 --> 05:19:54,041 WHERE WE BUILT LARGE PROGRAMS 9054 05:19:54,108 --> 05:20:00,281 THAT ARE BIG TEAM SCIENCE 9055 05:20:00,347 --> 05:20:02,783 EFFORTS CONSORTIA, ET CETERA, 9056 05:20:02,850 --> 05:20:04,652 EVERYTHING IS FROM THE DISCOVERY 9057 05:20:04,718 --> 05:20:07,321 TO MODERN TOOL DEVELOPMENT TO 9058 05:20:07,388 --> 05:20:09,123 CLINICAL TRIALS. 9059 05:20:09,189 --> 05:20:12,426 IT IS THE ONLY WAY TO DO FORWARD 9060 05:20:12,493 --> 05:20:13,294 AND REVERSE TRANSLATION, TO 9061 05:20:13,360 --> 05:20:16,697 LEARN AS FAST AS WE NEED TO 9062 05:20:16,764 --> 05:20:18,432 LEARN TO DO SOMETHING INVOLVED 9063 05:20:18,499 --> 05:20:22,269 WITH THIS TERRIBLE DISORDER. 9064 05:20:22,336 --> 05:20:32,012 WE DO THAT, WE BUILD ENTIRE 9065 05:20:32,079 --> 05:20:34,081 ECOSYSTEM, NOT JUST ACROSS LARGE 9066 05:20:34,148 --> 05:20:36,050 CENTERS DEPICTED HERE BUT 9067 05:20:36,116 --> 05:20:36,550 INCREASINGLY SO ACROSS 9068 05:20:36,617 --> 05:20:37,918 INDIVIDUAL GRANTS. 9069 05:20:37,985 --> 05:20:39,987 THESE ARE JUST SOME OF THE OPEN 9070 05:20:40,054 --> 05:20:41,589 SCIENCE PROGRAMS THAT ARE 9071 05:20:41,655 --> 05:20:44,692 FOCUSING ON BEING PRECISION 9072 05:20:44,758 --> 05:20:46,226 MEDICINE PRINCIPLES AND 9073 05:20:46,293 --> 05:20:49,063 PRACTICES IN DISCOVERY AND DRUG 9074 05:20:49,129 --> 05:20:50,931 DEVELOPMENT THROUGH A 9075 05:20:50,998 --> 05:20:53,000 CENTRALIZED DATA INFRASTRUCTURE 9076 05:20:53,067 --> 05:20:54,868 WHERE INVESTIGATORS CAN REALLY 9077 05:20:54,935 --> 05:20:57,938 SHARE AN ENORMOUS DIVERSITY OF 9078 05:20:58,005 --> 05:21:08,415 DATA TYPES WHERE THERE'S A TEAM, 9079 05:21:08,482 --> 05:21:18,859 SUPPORTING CURATION. 9080 05:21:20,361 --> 05:21:22,329 WE CONTINUE TO EVOLVE AND MAKE 9081 05:21:22,396 --> 05:21:24,298 IT BETTER. 9082 05:21:24,365 --> 05:21:26,834 IT'S NEVER PERFECT. 9083 05:21:26,900 --> 05:21:32,840 AND ALSO IN ADDITION TO 9084 05:21:32,906 --> 05:21:33,707 REPOSITORY OFFERING FULL 9085 05:21:33,774 --> 05:21:35,242 PROVENANCE OF DATA WE HAVE A 9086 05:21:35,309 --> 05:21:40,381 NUMBER OF OPEN SOURCE PLATFORMS 9087 05:21:40,447 --> 05:21:42,616 WHERE THINGS LIKE NEW NOMINATED 9088 05:21:42,683 --> 05:21:44,118 TARGETS DISCOVERED THROUGH BIG 9089 05:21:44,184 --> 05:21:47,821 DATA APPROACHES CAN SHALL PUSHED 9090 05:21:47,888 --> 05:21:49,123 OUT PRE-PUBLICATION WITH THE 9091 05:21:49,189 --> 05:21:50,557 PROVENANCE OF WHAT'S BEHIND 9092 05:21:50,624 --> 05:21:50,758 THEM. 9093 05:21:50,824 --> 05:21:54,395 WE'RE NOT SAYING THIS IS THE 9094 05:21:54,461 --> 05:21:55,162 BEST TARGET. 9095 05:21:55,229 --> 05:21:58,866 ALL WE'RE DOING IS SUR VASTING 9096 05:21:58,932 --> 05:22:01,502 THE BURDEN OF EVIDENCE, LETTING 9097 05:22:01,568 --> 05:22:06,106 THE COMMUNITY DECIDE. 9098 05:22:06,173 --> 05:22:07,474 THANK YOU AND LOOK FORWARD TO 9099 05:22:07,541 --> 05:22:15,816 THE DISCUSSION. 9100 05:22:17,584 --> 05:22:20,788 >> FEEL FREE TO JUMP TO THE 9101 05:22:20,854 --> 05:22:21,555 MICROPHONES TO START THIS 9102 05:22:21,622 --> 05:22:31,265 DISCUSSION. 9103 05:22:31,331 --> 05:22:36,203 >> ONE OF THE QUESTIONS WHEN IT 9104 05:22:36,270 --> 05:22:37,971 COMES TO PRE-REGISTRATIONS, AT 9105 05:22:38,038 --> 05:22:43,110 LEAST IN THE WORLD OF BIOMEDICAL 9106 05:22:43,177 --> 05:22:45,646 SCIENCES, WHEN THESE EVOLVE, ARE 9107 05:22:45,713 --> 05:22:46,780 THERE ISSUES RELATED TO 9108 05:22:46,847 --> 05:22:49,249 INTELLECTUAL PROPERTY THAT THIS 9109 05:22:49,316 --> 05:22:50,984 IS NOW IN THE -- PRIOR ART, AND 9110 05:22:51,051 --> 05:22:59,960 HOW ARE YOU TRYING TO MESSAGE 9111 05:23:00,027 --> 05:23:02,162 AND MANAGE THAT? 9112 05:23:02,229 --> 05:23:04,231 >> IT IS -- PUBLIC EXPOSURE 9113 05:23:04,298 --> 05:23:05,766 WOULD PREVENT YOU FROM FILING 9114 05:23:05,833 --> 05:23:06,633 AFTER THE FACT. 9115 05:23:06,700 --> 05:23:09,803 SO THERE ARE LOTS OF SOLUTIONS 9116 05:23:09,870 --> 05:23:10,804 AROUND THIS. 9117 05:23:10,871 --> 05:23:13,540 ONE IS, YOU KNOW, OF COURSE YOU 9118 05:23:13,607 --> 05:23:17,344 COULD FILE PROVISIONAL AND 9119 05:23:17,411 --> 05:23:19,346 IMMEDIATELY PUBLISH THE IDEAS TO 9120 05:23:19,413 --> 05:23:21,415 GET THEM OUT THERE. 9121 05:23:21,482 --> 05:23:25,919 WE REELEDDED FOR US OUR GOAL FOR 9122 05:23:25,986 --> 05:23:27,321 COMMERCIALIZATION IS VERY MUCH 9123 05:23:27,387 --> 05:23:31,492 TO BUILD A MODEL OF 9124 05:23:31,558 --> 05:23:32,693 SUSTAINABILITY TO SO OTHER 9125 05:23:32,760 --> 05:23:34,661 PEOPLE CAN START ORGANIZATION 9126 05:23:34,728 --> 05:23:36,964 IT'S TO DO SCIENCE IN DIFFERENT 9127 05:23:37,030 --> 05:23:37,164 WAYS. 9128 05:23:37,231 --> 05:23:40,901 WE DON'T WANT TO JUST TAKE OUR 9129 05:23:40,968 --> 05:23:44,171 INVESTMENT AND RUN IT TO ZERO. 9130 05:23:44,238 --> 05:23:45,372 IT'S DIFFICULT TO COMMERCIALIZE 9131 05:23:45,439 --> 05:23:46,373 SOME THINGS. 9132 05:23:46,440 --> 05:23:56,884 WE HAVE A NOT MAXIMAL USE. 9133 05:24:02,990 --> 05:24:05,726 INSTEAD OF SAYING THIS IS A 9134 05:24:05,793 --> 05:24:07,928 THING, YOU KNOW, IT'S NOT LIKE A 9135 05:24:07,995 --> 05:24:09,329 DEFENSIVE POSITION. 9136 05:24:09,396 --> 05:24:10,764 THE VAST MAJORITY OF THINGS WE 9137 05:24:10,831 --> 05:24:13,367 DO ARE NOT GOING TO BE THAT 9138 05:24:13,433 --> 05:24:17,337 NUGGET THAT'S GOING TO RESULT IN 9139 05:24:17,404 --> 05:24:17,571 REVENUE. 9140 05:24:17,638 --> 05:24:18,639 THE VAST MAJORITY IS MUCH MORE 9141 05:24:18,705 --> 05:24:29,249 USEFUL TO THE UNIVERSE THAN US. 9142 05:24:30,150 --> 05:24:32,219 IF ACADEMIC INSTITUTIONS GET A 9143 05:24:32,286 --> 05:24:35,155 WHIFF IT MIGHT BE OF VALUE, MOST 9144 05:24:35,222 --> 05:24:36,790 ARE UPSTREAM, WE TIE THE WHOLE 9145 05:24:36,857 --> 05:24:41,295 THING UP IN KNOTS FOR NO REASON. 9146 05:24:41,361 --> 05:24:43,564 CAN WE UPSTREAM FROM A LONG 9147 05:24:43,630 --> 05:24:44,565 COMMERCIALIZATION STRATEGY THAT 9148 05:24:44,631 --> 05:24:46,200 HIGHLIGHTS WHAT OR IS NOT 9149 05:24:46,266 --> 05:24:47,501 SOMETHING WE PURSUE? 9150 05:24:47,568 --> 05:24:49,837 IF IT'S LAYING THE GROUND WORK, 9151 05:24:49,903 --> 05:24:51,238 GETTING INFORMATION, THAT'S MORE 9152 05:24:51,305 --> 05:24:53,240 USEFUL, IT'S A MORE SELECTIVE 9153 05:24:53,307 --> 05:24:57,611 STRATEGY AND THEN EVEN THEN IF 9154 05:24:57,678 --> 05:24:59,780 IT IS AMONGST THINGS, THERE'S 9155 05:24:59,847 --> 05:25:02,416 STILL WAYS, TRY AND DO THAT IN A 9156 05:25:02,482 --> 05:25:12,993 DIFFERENT ORDER, IT CAN HAPPEN 9157 05:25:14,828 --> 05:25:18,398 FAST. 9158 05:25:18,465 --> 05:25:21,635 >> GINA POE, UCLA. 9159 05:25:21,702 --> 05:25:22,903 THIS IS ABOUT MICROPUBLICATIONS, 9160 05:25:22,970 --> 05:25:27,574 WHAT YOU'RE DOING AT THE 9161 05:25:27,641 --> 05:25:29,910 ALZHEIMER'S DISEASE RESEARCH, 9162 05:25:29,977 --> 05:25:33,180 YOU WERE SAYING, DR. STERNBERG, 9163 05:25:33,247 --> 05:25:35,716 THE MICROPUBLICATION VENUE HAS 9164 05:25:35,782 --> 05:25:39,219 RAISE IT'S ABOUT 5% OUTPUT OF C. 9165 05:25:39,286 --> 05:25:42,723 ELEGANS, IS THAT WHAT YOU'RE 9166 05:25:42,789 --> 05:25:44,258 SAYING? 9167 05:25:44,324 --> 05:25:46,526 >> YES, SO FAR. 9168 05:25:46,593 --> 05:25:49,062 WE STARTED OUT FIVE OR SIX YEARS 9169 05:25:49,129 --> 05:25:54,034 AGO, TAKE OFF, WATCHING IT 9170 05:25:54,101 --> 05:25:55,102 PROPAGATE WITHOUT ADVERTISING. 9171 05:25:55,168 --> 05:25:55,569 >> IT'S GOING. 9172 05:25:55,636 --> 05:25:56,637 >> IT'S GOING. 9173 05:25:56,703 --> 05:25:57,504 >> IT'S WORKING. 9174 05:25:57,571 --> 05:25:58,505 >> I'M SURPRISED IT'S ALREADY 9175 05:25:58,572 --> 05:25:59,172 THAT MUCH. 9176 05:25:59,239 --> 05:26:02,576 >> YEAH, SO I WAS WONDERING IF 9177 05:26:02,643 --> 05:26:13,186 MAYBE PERHAPS WE CAN ADOPT THE 9178 05:26:19,760 --> 05:26:22,930 ALT-P PORTAL, MAKE 9179 05:26:22,996 --> 05:26:25,098 MICROPUBLICATIONS MICRO, AGAIN, 9180 05:26:25,165 --> 05:26:28,302 INSTEAD OF MILLI OR DECI, 9181 05:26:28,368 --> 05:26:33,440 PUBLISH ONE PANEL WITH FULL 9182 05:26:33,507 --> 05:26:35,309 METHODS AND THAT'S STATISTICAL 9183 05:26:35,375 --> 05:26:43,951 ANALYSES BUT NO INTRODUCTION. 9184 05:26:44,017 --> 05:26:46,687 IF WE DID THAT WOULD IT SPEED 9185 05:26:46,753 --> 05:26:49,289 RESEARCH EVEN FASTER THAN IT'S 9186 05:26:49,356 --> 05:26:50,057 GOING NOW? 9187 05:26:50,123 --> 05:26:57,097 AND SO THAT'S MY QUESTION. 9188 05:26:57,164 --> 05:26:58,632 >> USEFUL THING, PEOPLE HAVE 9189 05:26:58,699 --> 05:27:00,167 METHODS OR REAGENTS IF THEY WERE 9190 05:27:00,233 --> 05:27:01,601 GOING TO WAIT FOR A FULL PAPER 9191 05:27:01,668 --> 05:27:03,637 IT'S NOT GOING TO HAPPEN. 9192 05:27:03,704 --> 05:27:06,640 THAT'S REALLY HELPING DRIVE 9193 05:27:06,707 --> 05:27:08,108 ETHICS FORWARD. 9194 05:27:08,175 --> 05:27:11,545 AND IT'S GOING TO TAKE MORE 9195 05:27:11,611 --> 05:27:13,080 ONE-ON-ONE CONVINCING OF PEOPLE 9196 05:27:13,146 --> 05:27:17,918 THAT IT'S WORTH IT, RIGHT? 9197 05:27:17,985 --> 05:27:23,256 AND THAT'S GRADUALLY GOING. 9198 05:27:23,323 --> 05:27:24,324 BUT GO BACK -- 9199 05:27:24,391 --> 05:27:25,792 >> THAT'S GOING FORWARD. 9200 05:27:25,859 --> 05:27:28,295 WHAT YOU'RE DOING IS GOING 9201 05:27:28,362 --> 05:27:29,062 BACKWARD THROUGH PUBLICATIONS 9202 05:27:29,129 --> 05:27:30,797 THAT HAVE ALREADY BEEN OUT 9203 05:27:30,864 --> 05:27:31,398 THERE. 9204 05:27:31,465 --> 05:27:37,371 DO YOU RUN INTO PROBLEMS WITH 9205 05:27:37,437 --> 05:27:39,239 PUBLISHERS AND COPYRIGHT FOR THE 9206 05:27:39,306 --> 05:27:44,044 ARTICLES WHEN YOU BREAK IT UP 9207 05:27:44,111 --> 05:27:47,347 INTO INDIVIDUAL UNITS? 9208 05:27:47,414 --> 05:27:53,787 >> WELL, I MEAN, WE PROVIDE THE 9209 05:27:53,854 --> 05:27:55,522 CITATION TO PubMed. 9210 05:27:55,589 --> 05:27:59,259 WE'RE JUST -- WE COLLECT GRAPHIC 9211 05:27:59,326 --> 05:28:00,227 INFORMATION, CONFLICTS OF 9212 05:28:00,293 --> 05:28:02,229 INTEREST, ET CETERA. 9213 05:28:02,295 --> 05:28:03,997 WE EXTRACT CERTAIN ELEMENTS. 9214 05:28:04,064 --> 05:28:05,499 WE LINK TO, FOR EXAMPLE, 9215 05:28:05,565 --> 05:28:14,674 CLINICAL TRIALS, TO OTHER 9216 05:28:14,741 --> 05:28:15,575 TRANSLATIONAL COMPONENTS RELATED 9217 05:28:15,642 --> 05:28:17,611 TO THE EVOLUTION OF THE 9218 05:28:17,677 --> 05:28:18,645 COMPOUND, AGGREGATE MORE THAN 9219 05:28:18,712 --> 05:28:21,248 WHAT'S IN THE PAPER. 9220 05:28:21,314 --> 05:28:23,216 FOR THE PAPER, CURATORS ASK THE 9221 05:28:23,283 --> 05:28:24,885 QUESTION, HAS THIS BEEN 9222 05:28:24,951 --> 05:28:25,185 REPORTED? 9223 05:28:25,252 --> 05:28:27,921 WE'RE NOT SAYING, OH, THIS IS A 9224 05:28:27,988 --> 05:28:28,588 BAD PAPER. 9225 05:28:28,655 --> 05:28:30,123 OH, WE DON'T LIKE IT. 9226 05:28:30,190 --> 05:28:32,526 OR LOOK AT THIS. 9227 05:28:32,592 --> 05:28:34,361 WE SIMPLY -- FULL DISCLOSURE, 9228 05:28:34,428 --> 05:28:37,330 SOME OF OUR OWN PAPERS ARE 9229 05:28:37,397 --> 05:28:37,631 THERE. 9230 05:28:37,697 --> 05:28:40,700 THEY HAVE QUITE A LOT OF RED. 9231 05:28:40,767 --> 05:28:45,739 SO WE'RE NOT -- AND SO THIS IS 9232 05:28:45,806 --> 05:28:48,809 JUST AN ENTRY POINT INTO THAT 9233 05:28:48,875 --> 05:28:51,011 MANUSCRIPT, A QUICK WAY FOR THE 9234 05:28:51,078 --> 05:28:53,847 COMMUNITY TO SURVEY IS IT 9235 05:28:53,914 --> 05:28:55,582 SEARCHABLE BY COMPOUND, BY 9236 05:28:55,649 --> 05:28:57,451 ANIMAL MODEL, BY TARGET, ET 9237 05:28:57,517 --> 05:28:59,086 CETERA, ET CETERA. 9238 05:28:59,152 --> 05:29:03,356 AND SO IT'S REALLY KIND OF 9239 05:29:03,423 --> 05:29:06,326 PORTFOLIO ANALYSIS AT YOUR 9240 05:29:06,393 --> 05:29:06,626 FINGERTIPS. 9241 05:29:06,693 --> 05:29:08,562 IT SAVES TIME TO INVESTIGATORS 9242 05:29:08,628 --> 05:29:15,368 WHEN THEY WANT TO START IN 9243 05:29:15,435 --> 05:29:17,037 ALZHEIMER'S RESEARCH, WE LIKE TO 9244 05:29:17,104 --> 05:29:18,738 KNOW WHAT'S OUT THERE, WHO ARE 9245 05:29:18,805 --> 05:29:20,774 THE TARGETS. 9246 05:29:20,841 --> 05:29:24,244 WE HAVEN'T RUN INTO ANY ISSUES. 9247 05:29:24,311 --> 05:29:28,415 >> WHAT I'M ASKING, CAN THE NIH, 9248 05:29:28,482 --> 05:29:30,817 FOR EXAMPLE, BREAK UP PAST 9249 05:29:30,884 --> 05:29:31,384 PUBLICATIONS INTO 9250 05:29:31,451 --> 05:29:32,786 MICROPUBLICATIONS IN THE SENSE 9251 05:29:32,853 --> 05:29:35,655 OF JUST SAY WHAT THE METHODS 9252 05:29:35,722 --> 05:29:37,824 WERE, THESE ARE THE AUTHORS, SO 9253 05:29:37,891 --> 05:29:39,025 TAKE ALL THE PAST PUBLICATIONS 9254 05:29:39,092 --> 05:29:41,561 THAT HAVE BEEN FUNDED BY THE NIH 9255 05:29:41,628 --> 05:29:43,063 THAT ARE NOW ALREADY ON 9256 05:29:43,130 --> 05:29:45,265 PubMed AND BREAK IT UP SO 9257 05:29:45,332 --> 05:29:50,203 PEOPLE CAN SEARCH FOR THINGS 9258 05:29:50,270 --> 05:29:50,770 WITHOUT NEEDING THE PERHAPS 9259 05:29:50,837 --> 05:29:55,142 BIASED STORY THAT GOES WITH IT. 9260 05:29:55,208 --> 05:29:56,309 >> FOR CURATED KNOWLEDGE BASES, 9261 05:29:56,376 --> 05:29:57,978 WHICH ARE NOT AVAILABLE FOR 9262 05:29:58,044 --> 05:30:01,815 HUMANS BUT FOR THE MODEL 9263 05:30:01,882 --> 05:30:08,155 ORGANISMS, MOUSE, RAT, FLY, 9264 05:30:08,221 --> 05:30:08,788 WORMSES THERE'S GRANULAR 9265 05:30:08,855 --> 05:30:10,657 INFORMATION THAT CAN BE LOCATED 9266 05:30:10,724 --> 05:30:11,658 AND SEARCHED AND COMPARED, EVEN 9267 05:30:11,725 --> 05:30:14,294 THOUGH THE PAPER IS NOT ABOUT 9268 05:30:14,361 --> 05:30:15,362 REPLICATE, RIGHT? 9269 05:30:15,428 --> 05:30:17,030 THAT'S VALUABLE AT A GRANULAR 9270 05:30:17,097 --> 05:30:18,431 LEVEL AND IS MAYBE THE RIGHT WAY 9271 05:30:18,498 --> 05:30:20,934 TO DO IT WITH CONTROLLED 9272 05:30:21,001 --> 05:30:31,545 VOCABULARIES SO METADATA IS JUST 9273 05:30:33,013 --> 05:30:33,113 NAILED. 9274 05:30:33,180 --> 05:30:33,547 THAT'S USEFUL. 9275 05:30:33,613 --> 05:30:36,950 >> THE FACT WE HAVE THE REPORT 9276 05:30:37,017 --> 05:30:40,887 CARDS AND OUTCOMES MEASURED, 9277 05:30:40,954 --> 05:30:43,056 THAT GIVES YOU THE PART THAT 9278 05:30:43,123 --> 05:30:44,491 MATTERS TO INVESTIGATORS. 9279 05:30:44,558 --> 05:30:48,995 ONCE THEY SEE THAT, THEY TEND TO 9280 05:30:49,062 --> 05:30:50,730 BE MESMERIZED BY THE NARRATIVE 9281 05:30:50,797 --> 05:30:54,134 AND JUST -- OR TAKE A CLOSER 9282 05:30:54,201 --> 05:30:55,302 LOOK. 9283 05:30:55,368 --> 05:31:00,707 STILL A FASCINATING STORY, GREAT 9284 05:31:00,774 --> 05:31:02,642 COMPOUND, BUT GIVES THEM 9285 05:31:02,709 --> 05:31:07,781 SKEPTICAL VIEW OF THAT 9286 05:31:07,847 --> 05:31:08,682 PARTICULAR PAPER. 9287 05:31:08,748 --> 05:31:15,689 >> CAN I ADD A COMMENT FROM NIH? 9288 05:31:15,755 --> 05:31:17,657 SO, DOING THAT TAKES A LOT OF 9289 05:31:17,724 --> 05:31:18,959 RESOURCES AND TIME. 9290 05:31:19,025 --> 05:31:20,827 THIS IS MANUALLY GOING OVER 9291 05:31:20,894 --> 05:31:23,797 EVERY PAPER, IT'S VERY, VERY 9292 05:31:23,863 --> 05:31:24,064 DIFFICULT. 9293 05:31:24,130 --> 05:31:26,499 THERE ARE NOW TOOLS TO TRY TO 9294 05:31:26,566 --> 05:31:28,168 AUTOMATE THESE THINGS. 9295 05:31:28,235 --> 05:31:29,769 THEY ARE NOT PERFECT. 9296 05:31:29,836 --> 05:31:31,238 AND THEY CERTAINLY ARE NOT AT 9297 05:31:31,304 --> 05:31:33,406 THE LEVEL WHERE THEY BREAK IT 9298 05:31:33,473 --> 05:31:34,608 DOWN PER EXPERIMENT. 9299 05:31:34,674 --> 05:31:37,143 IT'S LIKE PER PAPER. 9300 05:31:37,210 --> 05:31:38,445 THEY MENTIONED BLINDING IN THE 9301 05:31:38,511 --> 05:31:41,047 PAPER, WE'LL GIVE THEM CREDIT 9302 05:31:41,114 --> 05:31:41,715 FOR THAT. 9303 05:31:41,781 --> 05:31:43,683 IF FOR ANY ONE EXPERIMENT THEY 9304 05:31:43,750 --> 05:31:44,784 MENTION BLINDING THEY GET CREDIT 9305 05:31:44,851 --> 05:31:46,419 FOR THAT. 9306 05:31:46,486 --> 05:31:48,054 IT'S NOT PER EXPERIMENT AND SO 9307 05:31:48,121 --> 05:31:48,888 ON. 9308 05:31:48,955 --> 05:31:53,193 I THINK WE'RE FAR AWAY, SO FAR 9309 05:31:53,260 --> 05:31:57,797 AWAY FROM A.I. BEING ABLE TO 9310 05:31:57,864 --> 05:31:59,099 HELP US WITH THAT. 9311 05:31:59,165 --> 05:32:03,270 >> WE'RE ALWAYS LOOKING FOR WAYS 9312 05:32:03,336 --> 05:32:11,711 TO IMPROVE AND HAVE MACHINE MEET 9313 05:32:11,778 --> 05:32:13,079 HUMAN, YEAH, SYNERGY. 9314 05:32:13,146 --> 05:32:14,314 >> I THINK MAYBE THIS IS 9315 05:32:14,381 --> 05:32:15,749 SOMETHING WHICH HAS BEEN ALLUDED 9316 05:32:15,815 --> 05:32:18,151 IN THE LAST PANEL DISCUSSION 9317 05:32:18,218 --> 05:32:21,855 ABOUT FUNDING, TYING INTO SOME 9318 05:32:21,921 --> 05:32:22,389 NON-TRADITIONAL VENUES FOR 9319 05:32:22,455 --> 05:32:24,391 DISSEMINATION, FOR INSTANCE WE 9320 05:32:24,457 --> 05:32:25,592 TALKED ABOUT PRE-PRINT. 9321 05:32:25,659 --> 05:32:26,926 PRE-PRINTS HAVE BEEN AROUND BUT 9322 05:32:26,993 --> 05:32:29,095 REALLY TOOK OFF SINCE THE 9323 05:32:29,162 --> 05:32:29,863 PANDEMIC AND THE NIH SAID THEY 9324 05:32:29,929 --> 05:32:32,932 WILL BE PART OF THE REVIEW 9325 05:32:32,999 --> 05:32:34,768 PROCESS, WHO KNOWS IF YOU CAN 9326 05:32:34,834 --> 05:32:36,536 ADD, THAT'S WHEN PEOPLE STARTED 9327 05:32:36,603 --> 05:32:37,837 TAKING SERIOUSLY PEOPLE WHO 9328 05:32:37,904 --> 05:32:40,707 DIDN'T EVEN KNOW ABOUT 9329 05:32:40,774 --> 05:32:41,174 PRE-PRINTS. 9330 05:32:41,241 --> 05:32:43,310 YOU'RE NOT TIED TO AN EXTERNAL 9331 05:32:43,376 --> 05:32:45,078 FUNDING SOURCE SO YOU HAVE THESE 9332 05:32:45,145 --> 05:32:48,114 THINGS TO DEVELOP SOMETHING IN 9333 05:32:48,181 --> 05:32:50,216 HOUSE, REALLY SHAPE NOT JUST 9334 05:32:50,283 --> 05:32:51,518 PUBLICATION PROCESS BUT ALSO 9335 05:32:51,584 --> 05:32:52,986 SCIENTIFIC PROCESS ACCORDING TO 9336 05:32:53,053 --> 05:32:55,121 HOW YOU WANT. 9337 05:32:55,188 --> 05:32:56,656 SIMILARLY THE NIA TALKED ABOUT 9338 05:32:56,723 --> 05:33:01,394 THIS RESOURCE, YOU MAKE IT 9339 05:33:01,461 --> 05:33:04,964 MANDATORY FOR RESEARCHERS TO DO 9340 05:33:05,031 --> 05:33:06,666 THIS, YOU REQUIRE THEM TO -- SEX 9341 05:33:06,733 --> 05:33:08,468 AS A BIOLOGICAL VARIABLE OR 9342 05:33:08,535 --> 05:33:11,338 WHATEVER, THERE ARE MANDATED 9343 05:33:11,404 --> 05:33:12,539 REQUIREMENTS, ALLUDED TO, LIKE 9344 05:33:12,605 --> 05:33:16,042 IF THESE KIND OF THINGS WE 9345 05:33:16,109 --> 05:33:18,678 REALLY WANT RESEARCHERS TO 9346 05:33:18,745 --> 05:33:20,113 SHARE, THEY HAVE TO HAVE 9347 05:33:20,180 --> 05:33:21,514 MANDATES WHICH COME FROM FUNDING 9348 05:33:21,581 --> 05:33:23,383 SOURCES BUT AT THE SAME TIME 9349 05:33:23,450 --> 05:33:26,319 IT'S ALSO RAISING THE PROBLEM OF 9350 05:33:26,386 --> 05:33:28,188 THE BURDEN, SO MAYBE TYING WITH 9351 05:33:28,254 --> 05:33:31,257 VENUES OR TECHNOLOGY TO MAKE IT 9352 05:33:31,324 --> 05:33:33,993 EASIER FOR RESEARCHERS, DONE IN 9353 05:33:34,060 --> 05:33:36,062 A MANDATED FASHION SO IT'S A 9354 05:33:36,129 --> 05:33:36,863 REQUIREMENT, RESEARCHERS ALSO 9355 05:33:36,930 --> 05:33:38,331 GET THE CREDIT BUT ALSO GET THE 9356 05:33:38,398 --> 05:33:41,368 SUPPORT TO DO ALL OF THIS. 9357 05:33:47,607 --> 05:33:48,441 >> IT'S NOT ONE THING. 9358 05:33:48,508 --> 05:33:50,643 IT HAS TO BE ALL THESE THINGS. 9359 05:33:50,710 --> 05:33:53,680 I HEAR A LOT OF TIMES IF 9360 05:33:53,747 --> 05:33:54,280 EVERYONE WOULD REQUIRE 9361 05:33:54,347 --> 05:33:55,749 EVERYTHING THIS IS WHAT WOULD 9362 05:33:55,815 --> 05:33:56,149 HAPPEN. 9363 05:33:56,216 --> 05:33:57,283 YOU KNOW WHAT HAPPENS IF YOU SIT 9364 05:33:57,350 --> 05:33:59,152 IN THE ROOMS, NO ONE IS IN 9365 05:33:59,219 --> 05:34:01,554 SUPPORT OF ANY MANDATE FOR WHICH 9366 05:34:01,621 --> 05:34:02,689 THERE'S NOT COMMUNITY APPETITE 9367 05:34:02,756 --> 05:34:07,994 OR THEY HAVE NEVER SEEN OR DONE 9368 05:34:08,061 --> 05:34:08,361 IT. 9369 05:34:08,428 --> 05:34:10,864 I TEND TO TAKE A BOTTOM-UP VIEW, 9370 05:34:10,930 --> 05:34:12,532 HAVING THINGS EXIST IN WAYS THAT 9371 05:34:12,599 --> 05:34:14,467 NO ONE IS CERTAIN OF YET OR 9372 05:34:14,534 --> 05:34:18,405 WHATEVER, THINGS THAT DON'T 9373 05:34:18,471 --> 05:34:20,140 SCALE YET, TRYING TO DO 9374 05:34:20,206 --> 05:34:21,274 SOMETHING, SOME NEW EXISTS, IT 9375 05:34:21,341 --> 05:34:23,543 CAN MOVE YOU OVER TO WINDOW, 9376 05:34:23,610 --> 05:34:25,311 CHANGES THE NORMS, MAKES PEOPLE 9377 05:34:25,378 --> 05:34:26,179 THINK DIFFERENTLY, STARTS TO 9378 05:34:26,246 --> 05:34:28,481 CATCH ON. 9379 05:34:28,548 --> 05:34:30,016 NOW THERE'S A DISCUSSION -- 9380 05:34:30,083 --> 05:34:32,519 THERE'S NO SUCH THING AS A COLD 9381 05:34:32,585 --> 05:34:32,986 MANDATE. 9382 05:34:33,052 --> 05:34:34,421 NOW I REQUIRE OF EVERYONE. 9383 05:34:34,487 --> 05:34:35,822 IT'S NOT A THING. 9384 05:34:35,889 --> 05:34:36,723 IT'S NEVER HAPPENED BEFORE BUT 9385 05:34:36,790 --> 05:34:38,458 THESE ARE THE SORTS OF THINGS 9386 05:34:38,525 --> 05:34:40,427 WHEN YOU THINK ABOUT ALL OF US 9387 05:34:40,493 --> 05:34:41,394 SOMEWHERE ALONG THIS 9388 05:34:41,461 --> 05:34:42,195 DISTRIBUTION THERE'S THE LEADING 9389 05:34:42,262 --> 05:34:45,465 EDGE THINGS THAT CAN MOVE THE 9390 05:34:45,532 --> 05:34:48,201 NEEDLE, THEN MANDATES BRING UP 9391 05:34:48,268 --> 05:34:49,936 THE REAR, ALL AT DIFFERENT PARTS 9392 05:34:50,003 --> 05:34:51,137 OF DISTRIBUTION, THEY ALL NEED 9393 05:34:51,204 --> 05:34:52,338 TO BE THERE. 9394 05:34:52,405 --> 05:34:58,144 IT'S NOT LIKE ONE SOLUTION OR 9395 05:34:58,211 --> 05:34:59,145 THE OTHER. 9396 05:34:59,212 --> 05:35:02,348 >> I POINT OUT NIH HAS MANDATED 9397 05:35:02,415 --> 05:35:03,750 THESE DATA MANAGEMENT AND 9398 05:35:03,817 --> 05:35:04,551 SHARING PLANS. 9399 05:35:04,617 --> 05:35:07,253 NOW, THE CHALLENGE RIGHT NOW IS 9400 05:35:07,320 --> 05:35:09,222 THAT NOBODY KNOWS WHAT THAT -- 9401 05:35:09,289 --> 05:35:11,291 WHAT SHOULD BE DONE. 9402 05:35:11,357 --> 05:35:11,624 RIGHT? 9403 05:35:11,691 --> 05:35:13,159 SO, IT'S A GOOD OPPORTUNITY TO 9404 05:35:13,226 --> 05:35:17,864 THINK ABOUT WHAT WE REALLY WANT 9405 05:35:17,931 --> 05:35:19,466 TO DO, AND TO PROVE THE 9406 05:35:19,532 --> 05:35:21,267 PRINCIPLE BEFORE IT CAN BE 9407 05:35:21,334 --> 05:35:25,638 REALLY MANDATED IN A STRICT WAY. 9408 05:35:25,705 --> 05:35:27,707 SO WE NEED THESE EXPERIMENTS. 9409 05:35:27,774 --> 05:35:29,576 THE GOAL, THE SPIRIT OF THAT 9410 05:35:29,642 --> 05:35:32,011 MANDATE, IS TO MAKE THE DATA 9411 05:35:32,078 --> 05:35:32,745 FAIR, INCLUDING NEGATIVE 9412 05:35:32,812 --> 05:35:33,079 RESULTS. 9413 05:35:33,146 --> 05:35:34,814 JUST WE HAVE TO FIGURE OUT HOW 9414 05:35:34,881 --> 05:35:38,985 TO DO IT. 9415 05:35:39,052 --> 05:35:41,287 >> JUST TO ANSWER THAT FROM MY 9416 05:35:41,354 --> 05:35:43,556 PERSPECTIVE, I FIND WHEN I TALK 9417 05:35:43,623 --> 05:35:46,059 TO FUNDERS ABOUT ALL OF THESE 9418 05:35:46,125 --> 05:35:50,029 SORTS OF THINGS THEY ARE VERY 9419 05:35:50,096 --> 05:35:51,231 INTERESTED IN SEEING PRINCIPLES, 9420 05:35:51,297 --> 05:35:54,534 SOME HAVE MORE AT STAKE THAN 9421 05:35:54,601 --> 05:35:56,102 ANYBODY IN THE WHOLE ECOSYSTEM 9422 05:35:56,169 --> 05:35:56,936 GETTING GOOD SCIENCE. 9423 05:35:57,003 --> 05:36:00,540 THEY ARE THE ONES PAYING FOR IT. 9424 05:36:00,607 --> 05:36:08,848 I FIND FUNDERS ARE VERY OPEN TO 9425 05:36:08,915 --> 05:36:09,616 CHANGING PROCESS, ALBEIT SLOWLY, 9426 05:36:09,682 --> 05:36:11,351 DIFFERENT WAYS OF DOING THINGS, 9427 05:36:11,417 --> 05:36:12,986 BUT IT'S A STEP-BY-STEP THING, 9428 05:36:13,052 --> 05:36:14,754 MAKING IT POSSIBLE AND EASY AND 9429 05:36:14,821 --> 05:36:15,421 THEN ENCOURAGING IT. 9430 05:36:15,488 --> 05:36:18,892 WE'VE GOT TO THE STAGE WHERE 9431 05:36:18,958 --> 05:36:20,760 FUNDERS ARE ENCOURAGING IT AND 9432 05:36:20,827 --> 05:36:22,629 THERE WILL BE MANDATES COMING IN 9433 05:36:22,695 --> 05:36:22,929 AFTER THAT. 9434 05:36:22,996 --> 05:36:25,932 I THINK IT'S VERY EXCITING TIME, 9435 05:36:25,999 --> 05:36:28,001 WHERE THINGS ARE CHANGING AFTER 9436 05:36:28,067 --> 05:36:33,940 DECADES OF PEOPLE JUST MOANING. 9437 05:36:34,007 --> 05:36:35,675 >> ANOTHER COMMENT TO THAT 9438 05:36:35,742 --> 05:36:38,311 POINT, TIMING IS EVERYTHING. 9439 05:36:38,378 --> 05:36:40,179 THE ADVENT OF PRE-PRINTS, BUT 9440 05:36:40,246 --> 05:36:42,782 THEN NIH ALLOWING PRE-PRINTS TO 9441 05:36:42,849 --> 05:36:43,883 COME INTO THE BIOSKETCH, MADE IT 9442 05:36:43,950 --> 05:36:46,052 POSSIBLE FOR US TO SAY, OH, 9443 05:36:46,119 --> 05:36:48,254 LET'S ADD A PRE-PRINT FUNCTION. 9444 05:36:48,321 --> 05:36:49,989 WE'LL MAKE IT EASY, WE'RE NOT 9445 05:36:50,056 --> 05:36:51,524 SAYING YOU MUST HAVE HAVE A 9446 05:36:51,591 --> 05:36:53,192 PRE-PRINT WITH US, HAVE IT 9447 05:36:53,259 --> 05:36:54,260 ANYWHERE YOU WANT, BUT HERE IT 9448 05:36:54,327 --> 05:36:56,996 IS IF YOU CAN'T FIND ANOTHER 9449 05:36:57,063 --> 05:36:58,197 PLACE. 9450 05:36:58,264 --> 05:37:00,600 FOR THE DATA SHARING, BECAUSE 9451 05:37:00,667 --> 05:37:04,771 WHEN WE DECIDED TO REALLY BUILD 9452 05:37:04,837 --> 05:37:05,672 ROBUST OPEN SCIENCE 9453 05:37:05,738 --> 05:37:06,539 EXPECTATIONS, PRACTICES FOR 9454 05:37:06,606 --> 05:37:08,741 DISCOVERY RESEARCH, BIG DATA 9455 05:37:08,808 --> 05:37:10,610 SCIENCE, WE HAD TO CREATE -- WE 9456 05:37:10,677 --> 05:37:12,779 CAN'T SAY IT'S GOOD TO WALK BUT 9457 05:37:12,845 --> 05:37:14,113 NO SIDEWALKS. 9458 05:37:14,180 --> 05:37:17,817 YOU HAVE TO HELP -- ONE THING 9459 05:37:17,884 --> 05:37:20,453 NOT ONLY TO PROVIDE, MAKE THE 9460 05:37:20,520 --> 05:37:22,555 RIGHT CHOICE, THE EASIER CHOICE, 9461 05:37:22,622 --> 05:37:24,390 ALSO TO SHOW SUCCESS BECAUSE IF 9462 05:37:24,457 --> 05:37:26,926 YOU DON'T SHOW TWO YEARS, THIS 9463 05:37:26,993 --> 05:37:31,297 IS WORKING, THE COMMUNITY IS 9464 05:37:31,364 --> 05:37:32,699 BENEFITING, PARTICULARLY YOUNG 9465 05:37:32,765 --> 05:37:34,133 INVESTIGATE WORSE, I WAS 9466 05:37:34,200 --> 05:37:36,869 HEARTENED BY THE PRESENTATION 9467 05:37:36,936 --> 05:37:38,404 THAT YOUNG INVESTIGATORS 9468 05:37:38,471 --> 05:37:41,808 CONTRARY TO MYTHS OR POPULAR 9469 05:37:41,874 --> 05:37:43,643 BELIEF, THEY THRIVE -- OUR 9470 05:37:43,710 --> 05:37:46,179 EXPERIENCE, THEY THRIVE WITHIN 9471 05:37:46,245 --> 05:37:47,146 AN OPEN SCIENCE ECOSYSTEM. 9472 05:37:47,213 --> 05:37:51,517 THEY ARE THE NEW LEADERS WITH SO 9473 05:37:51,584 --> 05:37:53,820 MUCH MORE OPPORTUNITY TO LEAD, 9474 05:37:53,886 --> 05:37:57,090 OF A DIFFERENT GENERATION, TO 9475 05:37:57,156 --> 05:37:58,424 THEM SUPPORTING THINGS, SITTING 9476 05:37:58,491 --> 05:38:07,467 ON SOMETHING QUIETLY IS 9477 05:38:07,533 --> 05:38:10,436 SOMETHING NOT APPEALING. 9478 05:38:13,172 --> 05:38:16,609 >> ALSO MAKING IT EASIER, 2017, 9479 05:38:16,676 --> 05:38:20,013 ALLOWING PRE-PRINTS, BUT ALSO 9480 05:38:20,079 --> 05:38:26,052 COVID PRE-PRINTS, INDEX IN 9481 05:38:26,119 --> 05:38:26,786 PubMed, NIH PRE-PRINTS, 9482 05:38:26,853 --> 05:38:28,321 DISCOVERABILITY IS KEY TO MAKE 9483 05:38:28,388 --> 05:38:30,323 THINGS WE NEED TO HAPPEN TO 9484 05:38:30,390 --> 05:38:32,058 UTILIZE ALL OF THIS INFORMATION 9485 05:38:32,125 --> 05:38:35,128 IS LIKE MAKING IT EASY AND 9486 05:38:35,194 --> 05:38:37,296 HAVING THAT INFRASTRUCTURE 9487 05:38:37,363 --> 05:38:39,599 EXIST, USING THOSE SOLUTIONS TO 9488 05:38:39,666 --> 05:38:43,002 DEMONSTRATE THAT THEY ARE WORK 9489 05:38:43,069 --> 05:38:48,741 AND ARE EASY. 9490 05:38:50,076 --> 05:38:52,845 >> WE'D LIKE TO COME BACK TO THE 9491 05:38:52,912 --> 05:38:54,414 NEW FORMATS, THE TOPIC OF 9492 05:38:54,480 --> 05:38:56,582 INITIATIVES FOR OPEN SCIENCE. 9493 05:38:56,649 --> 05:38:59,986 JUST AS A DISCLAIMER, I'M A FULL 9494 05:39:00,053 --> 05:39:02,488 PROPONENT OF OPEN SCIENCE. 9495 05:39:02,555 --> 05:39:04,357 I LIKE PRE-PRINTS. 9496 05:39:04,424 --> 05:39:07,193 PUBLISHING EVERYTHING ON THE 9497 05:39:07,260 --> 05:39:08,628 ARCHIVE, bioRxiv. 9498 05:39:08,695 --> 05:39:10,029 BUT THERE IS A PROBLEM. 9499 05:39:10,096 --> 05:39:11,898 THAT WORKS PERFECTLY IN PHYSICS. 9500 05:39:11,964 --> 05:39:14,634 WHERE IT'S EASY TO CHECK THE 9501 05:39:14,701 --> 05:39:16,502 MATH OF SOMEBODY FOR 9502 05:39:16,569 --> 05:39:18,004 SIMULATIONS, NOW WE'RE DEALING 9503 05:39:18,071 --> 05:39:20,239 WITH BIOLOGY. 9504 05:39:20,306 --> 05:39:21,207 YOU CAN'T VERIFY THINGS EXCEPT 9505 05:39:21,274 --> 05:39:23,176 IF YOU DO YOURSELF THE 9506 05:39:23,242 --> 05:39:24,143 EXPERIMENTS. 9507 05:39:24,210 --> 05:39:27,013 IF YOU START TO DO OPEN SCIENCE 9508 05:39:27,080 --> 05:39:28,881 AND PUBLISH YOUR RESULTS, WHO IS 9509 05:39:28,948 --> 05:39:32,051 GOING TO SAY WHETHER WHAT 9510 05:39:32,118 --> 05:39:33,486 YOU'RE SAYING IS CORRECT OR NOT? 9511 05:39:33,553 --> 05:39:35,455 IT COULD BE A POSSIBILITY, HAVE 9512 05:39:35,521 --> 05:39:37,757 YOU NO WAY TO CHECK WHETHER IT'S 9513 05:39:37,824 --> 05:39:39,926 TRUE OR NOT TRUE. 9514 05:39:39,992 --> 05:39:42,128 SINCE WE'RE DEALING WITH BIOLOGY 9515 05:39:42,195 --> 05:39:45,198 AND THERE ARE PATIENTS, ALSO IN 9516 05:39:45,264 --> 05:39:45,865 THE POPULATION, WITH 9517 05:39:45,932 --> 05:39:48,401 EXPECTATIONS FOR THEIR KIDS TO 9518 05:39:48,468 --> 05:39:53,639 BE CURED OF A SPECIFIC DISEASE, 9519 05:39:53,706 --> 05:39:55,374 AND SOMEBODY'S INVENTING 9520 05:39:55,441 --> 05:39:56,375 SOMETHING PLAUSIBLE, THAT COULD 9521 05:39:56,442 --> 05:39:57,677 GENERATE MASSIVE DAMAGE. 9522 05:39:57,744 --> 05:40:00,947 I HAVEN'T SEEN ANYTHING IN THE 9523 05:40:01,013 --> 05:40:02,482 FIELD ARGUING OR DISCUSSING 9524 05:40:02,548 --> 05:40:03,816 ABOUT THIS POSSIBILITY. 9525 05:40:03,883 --> 05:40:06,352 I MEAN, THAT COULD BE FOLLOWING 9526 05:40:06,419 --> 05:40:09,288 YOUR BRIGHTEST MIND IN THE 9527 05:40:09,355 --> 05:40:11,390 UNITED STATES, SOMEBODY DOING 9528 05:40:11,457 --> 05:40:13,259 THE EXPERIMENT OF INJECTING 9529 05:40:13,326 --> 05:40:16,863 BELIEVE TO CLEAN THE BRAIN FROM 9530 05:40:16,929 --> 05:40:18,698 AMYLOID DEPOSITS, INSTEAD OF 9531 05:40:18,765 --> 05:40:20,700 TRYING TO CURE COVID, IT SOUNDS 9532 05:40:20,767 --> 05:40:23,102 STUPID BUT THAT COULD HAPPEN. 9533 05:40:23,169 --> 05:40:26,372 WELL, IT DID HAPPEN. 9534 05:40:26,439 --> 05:40:30,309 YOU REMEMBER "THE LANCET" STUDY? 9535 05:40:30,376 --> 05:40:31,978 TWELVE YEARS UNTIL RETRACTION, 9536 05:40:32,044 --> 05:40:32,712 THAT WAS PUBLISHED, PEER 9537 05:40:32,779 --> 05:40:33,913 REVIEWED. 9538 05:40:33,980 --> 05:40:35,014 HOW MANY CHILDREN WERE KILLED 9539 05:40:35,081 --> 05:40:37,316 BECAUSE THEY WERE NOT VACCINATED 9540 05:40:37,383 --> 05:40:37,850 AND GOT MEASLES? 9541 05:40:37,917 --> 05:40:39,752 THE U.K. DID A SURVEY. 9542 05:40:39,819 --> 05:40:41,654 THAT WAS PLAUSIBLE. 9543 05:40:41,721 --> 05:40:44,757 NOW, THAT GENERATED A LOT OF 9544 05:40:44,824 --> 05:40:45,958 DAMAGE. 9545 05:40:46,025 --> 05:40:49,562 WE HAD ALSO THIS EXPERIENCE WITH 9546 05:40:49,629 --> 05:40:50,129 GENDER DYSPHORIA, SOMEBODY 9547 05:40:50,196 --> 05:40:52,064 PUBLISHED A PAPER IN OUR 9548 05:40:52,131 --> 05:40:54,801 JOURNAL, A SMALL SENTENCE AT THE 9549 05:40:54,867 --> 05:40:57,637 END SAYING THERE ARE BIOMARKERS 9550 05:40:57,703 --> 05:40:58,938 OF GENDER DYSPHORIA, THEREFORE 9551 05:40:59,005 --> 05:41:02,575 THAT COULD BE AN AVENUE OF 9552 05:41:02,642 --> 05:41:08,481 RESEARCH TO FIND TREATMENT 9553 05:41:08,548 --> 05:41:10,550 AGAINST GENDER DYSPHORIA. 9554 05:41:10,616 --> 05:41:12,518 IT'S WHEN THE GENDER HAVE YOU AT 9555 05:41:12,585 --> 05:41:14,387 BIRTH IS NOT WHAT YOU FEEL LIKE 9556 05:41:14,453 --> 05:41:15,488 YOU ARE. 9557 05:41:15,555 --> 05:41:19,425 YOU HAVE MALE SEX BUT FEEL LIKE 9558 05:41:19,492 --> 05:41:20,293 A WOMAN. 9559 05:41:20,359 --> 05:41:22,061 SO EXACTLY A COUPLE MINUTES 9560 05:41:22,128 --> 05:41:26,165 LATER THE PAPER WAS PUBLISHED, 9561 05:41:26,232 --> 05:41:27,200 RELIGIOUS OR WHATEVER PEOPLE IN 9562 05:41:27,266 --> 05:41:31,237 THE UNITED STATES, THEY POSTED 9563 05:41:31,304 --> 05:41:32,872 BLOGS EVERYWHERE, AND A ON 9564 05:41:32,939 --> 05:41:35,508 TWITTER AT THE TIME. 9565 05:41:35,575 --> 05:41:39,245 A SCIENTIST PROVES GENDER 9566 05:41:39,312 --> 05:41:41,647 DYSPHORIA IS A DISEASE THAT CAN 9567 05:41:41,714 --> 05:41:43,850 BE CURED SO WE SHOULD PROHIBIT 9568 05:41:43,916 --> 05:41:46,352 AT THE STATE LEVEL TRANSGENDER 9569 05:41:46,419 --> 05:41:46,819 SURGERY. 9570 05:41:46,886 --> 05:41:51,390 BILLIONS OF SECONDS LATER THE 9571 05:41:51,457 --> 05:41:52,825 LGBTQ COMMUNITY REACTED 9572 05:41:52,892 --> 05:41:58,698 STRONGLY, THAT CREATED MASSIVE 9573 05:41:58,764 --> 05:41:59,232 UPHEAVAL. 9574 05:41:59,298 --> 05:42:00,700 THAT WAS REVIEWED BY THE 9575 05:42:00,766 --> 05:42:00,933 JOURNAL. 9576 05:42:01,000 --> 05:42:02,134 WE JUST MISSED IT. 9577 05:42:02,201 --> 05:42:06,305 NOW YOU DO OPEN SCIENCE, YOU 9578 05:42:06,372 --> 05:42:08,608 HAVE PEOPLE WHO ARE GOING 9579 05:42:08,674 --> 05:42:10,643 THROUGH THINGS LIKE THAT, HOW DO 9580 05:42:10,710 --> 05:42:12,078 YOU PREVENT THAT? 9581 05:42:12,144 --> 05:42:13,746 WE'RE DEALING WITH BIOLOGY, 9582 05:42:13,813 --> 05:42:15,481 WE'RE DEALING WITH DISEASES, SO 9583 05:42:15,548 --> 05:42:17,350 SOMEBODY CLAIMS ALL OF A SUDDEN 9584 05:42:17,416 --> 05:42:20,186 THEY HAVE FOUND A CURE, A 9585 05:42:20,253 --> 05:42:25,992 DISEASE CURE FOR COVID? 9586 05:42:26,058 --> 05:42:28,427 ALZHEIMER'S OR PARKINSON'S OR 9587 05:42:28,494 --> 05:42:28,928 EPILEPSY? 9588 05:42:28,995 --> 05:42:30,563 WE'RE NOT LIVING IN A FANTASY 9589 05:42:30,630 --> 05:42:30,796 WORLD. 9590 05:42:30,863 --> 05:42:34,300 WE'RE DEALING WITH HUMAN NATURE. 9591 05:42:34,367 --> 05:42:37,236 >> YOU'VE JUST MADE THE BEST 9592 05:42:37,303 --> 05:42:39,772 ARGUMENT FOR WORKING IN OPEN 9593 05:42:39,839 --> 05:42:41,307 PRACTICE BECAUSE THE WHOLE POINT 9594 05:42:41,374 --> 05:42:43,376 THINGS LIKE PEER REVIEWS ARE 9595 05:42:43,442 --> 05:42:45,978 OPEN, EVERYBODY CAN READ THEM, 9596 05:42:46,045 --> 05:42:47,747 ANYBODY CAN POST PEER REVIEW, 9597 05:42:47,813 --> 05:42:50,149 YOU CAN READ OPINIONS OF OTHER 9598 05:42:50,216 --> 05:42:50,383 PEOPLE. 9599 05:42:50,449 --> 05:42:52,451 ON OCTOPUS WE HAVE A RED 9600 05:42:52,518 --> 05:42:54,220 FLAGGING SYSTEM SO A READER CAN 9601 05:42:54,287 --> 05:42:57,690 RAISE A RED FLAG IF THEY THINK 9602 05:42:57,757 --> 05:42:58,758 IT'S IMPORTANT, SERIOUS CONCERN. 9603 05:42:58,824 --> 05:43:00,426 ALL THESE THINGS ARE OPEN, 9604 05:43:00,493 --> 05:43:03,195 TRANSPART, THE DATA IS OPEN AND 9605 05:43:03,262 --> 05:43:03,896 TRANSPARENT. 9606 05:43:03,963 --> 05:43:04,830 OTHER PEOPLE CAN LOOK. 9607 05:43:04,897 --> 05:43:06,565 IT MAY BE DIFFICULT FOR OTHER 9608 05:43:06,632 --> 05:43:08,434 PEOPLE TO COLLECT NEW DATA BUT 9609 05:43:08,501 --> 05:43:11,270 THE PROTOCOL IS THERE SO PEOPLE 9610 05:43:11,337 --> 05:43:11,871 CAN. 9611 05:43:11,938 --> 05:43:13,339 THEY CAN INVESTIGATE THE 9612 05:43:13,406 --> 05:43:15,107 VARIATION AROUND THE BIOLOGICAL 9613 05:43:15,174 --> 05:43:19,011 VARIATION AROUND THE VARIABLES, 9614 05:43:19,078 --> 05:43:21,213 COLLECTING DATA ACCORDING TO THE 9615 05:43:21,280 --> 05:43:21,881 PROTOCOLS. 9616 05:43:21,948 --> 05:43:23,182 THE WAY WE'RE THINKING OF 9617 05:43:23,249 --> 05:43:24,617 WORKING IN THIS TRANSPARENT AND 9618 05:43:24,684 --> 05:43:25,851 OPEN WAY, DOING IT QUICKLY, 9619 05:43:25,918 --> 05:43:27,687 DOING IT IN A WAY THAT EVERYBODY 9620 05:43:27,753 --> 05:43:31,857 CAN SEE WHAT'S BEEN DONE, 9621 05:43:31,924 --> 05:43:32,658 REPLICATED, REPRODUCE IT, 9622 05:43:32,725 --> 05:43:35,194 REANALYZE IT, THE BEST WAY TO 9623 05:43:35,261 --> 05:43:37,263 GUARD AGAINST THE PROBLEMS. 9624 05:43:37,330 --> 05:43:41,200 IT'S THE PEER REVIEWED BINARY 9625 05:43:41,267 --> 05:43:43,035 VIEW, PEER REVIEW THEREFORE IT'S 9626 05:43:43,102 --> 05:43:44,904 TRUE, NOT PEER REVIEW THEREFORE 9627 05:43:44,971 --> 05:43:47,406 IT'S FALSE, THAT'S CAUSING 9628 05:43:47,473 --> 05:43:47,740 PROBLEMS. 9629 05:43:47,807 --> 05:43:49,075 >> I'M SORRY, THIS IS A 9630 05:43:49,141 --> 05:43:51,711 FANTASTIC WORLD. 9631 05:43:51,777 --> 05:43:55,081 WHAT HAPPENED TO "THE LANCET" 9632 05:43:55,147 --> 05:43:56,115 STORY? 9633 05:43:56,182 --> 05:43:58,484 >> HE WASN'T PUBLISHING IN AN 9634 05:43:58,551 --> 05:43:59,452 OPEN WORLD. 9635 05:43:59,518 --> 05:43:59,752 >> SORRY? 9636 05:43:59,819 --> 05:44:01,120 YES, BUT IT WAS THEN 9637 05:44:01,187 --> 05:44:02,521 DEMONSTRATED AS WRONG. 9638 05:44:02,588 --> 05:44:05,257 IT WAS RETRACTED. 9639 05:44:05,324 --> 05:44:08,094 AND THEN HE'S STILL ACTIVE, ALL 9640 05:44:08,160 --> 05:44:09,261 THESE FAMILIES ARE AUTISTIC 9641 05:44:09,328 --> 05:44:11,263 CHILDREN, THEY STILL LISTEN TO 9642 05:44:11,330 --> 05:44:13,899 HIM, THEY BELIEVE HE'S RIGHT AND 9643 05:44:13,966 --> 05:44:19,205 THAT THE VACCINE IS THE REASON 9644 05:44:19,271 --> 05:44:20,039 FOR THAT. 9645 05:44:20,106 --> 05:44:20,806 [CROSSTALK] 9646 05:44:20,873 --> 05:44:22,541 >> ONCE THE DAMAGE IS DONE, IT'S 9647 05:44:22,608 --> 05:44:22,775 DONE. 9648 05:44:22,842 --> 05:44:24,844 >> THAT'S THE WORLD WE'RE TRYING 9649 05:44:24,910 --> 05:44:27,246 TO GET AWAY FROM WITH NEW 9650 05:44:27,313 --> 05:44:29,115 SYSTEMS WHERE PER REVIEW IS 9651 05:44:29,181 --> 05:44:30,783 OPEN, TRANSPARENT, NOT DONE IN A 9652 05:44:30,850 --> 05:44:31,951 BLACK BOX BEHIND THE SCENES 9653 05:44:32,018 --> 05:44:34,587 PUBLISH AND PEOPLE SAY THEREFORE 9654 05:44:34,653 --> 05:44:35,454 IT'S PUBLISHED IN A 9655 05:44:35,521 --> 05:44:36,022 PEER-REVIEWED JOURNAL EVEN 9656 05:44:36,088 --> 05:44:38,290 THOUGH WE CAN'T SEE THE DATA WE 9657 05:44:38,357 --> 05:44:39,425 CAN'T REANALYZE IT, WE BELIEVE 9658 05:44:39,492 --> 05:44:41,594 THAT'S THE TRUTH. 9659 05:44:41,660 --> 05:44:42,695 THAT'S A DANGEROUS WORLD. 9660 05:44:42,762 --> 05:44:49,802 >> I WOULD LIKE TO PLAY DEVIL'S 9661 05:44:49,869 --> 05:44:50,069 ADVOCATE. 9662 05:44:50,136 --> 05:44:51,670 JUST OPEN THE NEWSPAPER. 9663 05:44:51,737 --> 05:44:53,205 FORGET A STUDY. 9664 05:44:53,272 --> 05:44:57,376 LOOK AT ANY NEWSPAPER, DEPENDS 9665 05:44:57,443 --> 05:44:59,879 WHAT OUTLET YOU LISTEN TO OR 9666 05:44:59,945 --> 05:45:02,381 LOOK AT, YOU'LL HEAR WHATEVER 9667 05:45:02,448 --> 05:45:03,582 THE STORY IS, COMPLETELY 9668 05:45:03,649 --> 05:45:07,219 DIFFERENT INTERPRETATIONS OF THE 9669 05:45:07,286 --> 05:45:07,753 STORY. 9670 05:45:07,820 --> 05:45:09,588 THAT CAN GET DANGEROUS. 9671 05:45:09,655 --> 05:45:12,024 I AGREE THINGS CAN GET VERY 9672 05:45:12,091 --> 05:45:15,194 DANGEROUS, EVEN IF WE'RE 9673 05:45:15,261 --> 05:45:17,496 OPTIMISTIC AND SAY, OH, WE'RE 9674 05:45:17,563 --> 05:45:18,397 THE SCIENTIFIC COMMUNITY. 9675 05:45:18,464 --> 05:45:20,366 I'M WORRIED NOT ABOUT THE 9676 05:45:20,433 --> 05:45:21,333 SCIENTIFIC COMMUNITY BUT THE 9677 05:45:21,400 --> 05:45:27,139 OUTSIDE WORLD, WHATTER -- WHAT 9678 05:45:27,206 --> 05:45:29,975 THEY ARE GOING TO MAKE OF THIS. 9679 05:45:30,042 --> 05:45:31,277 >> THE CHALLENGE IS IF YOU SEND 9680 05:45:31,343 --> 05:45:32,545 SOMETHING TO A JOURNAL AND NOT 9681 05:45:32,611 --> 05:45:34,880 ONLY DO WE NOT HAVE BENEFITS OF 9682 05:45:34,947 --> 05:45:36,882 ALL THE OPEN INFORMATION TO 9683 05:45:36,949 --> 05:45:38,250 VERIFY THINGS, OPEN DISCUSSION 9684 05:45:38,317 --> 05:45:40,252 TO ACTUALLY SEE EXPERTS WEIGH IN 9685 05:45:40,319 --> 05:45:41,487 ON SOMETHING, WHICH IS ALSO KEY 9686 05:45:41,554 --> 05:45:43,456 AND WHY IT'S IMPORTANT TO HAVE 9687 05:45:43,522 --> 05:45:45,558 READERS HAVE ACCESS TO THAT 9688 05:45:45,624 --> 05:45:47,426 INFORMATION, NOT JUST A SECRET 9689 05:45:47,493 --> 05:45:50,996 CONVERSATION BETWEEN AUTHOR AND 9690 05:45:51,063 --> 05:45:52,198 REVIEWERS, BETTER TO HAVE THAT 9691 05:45:52,264 --> 05:45:54,633 BUT OFTEN YOUR PAPER THAT MIGHT 9692 05:45:54,700 --> 05:45:56,936 BE NONSENSE, WHATEVER OPINION 9693 05:45:57,002 --> 05:45:59,004 GETS REJECTED, GOES SOMEWHERE 9694 05:45:59,071 --> 05:46:00,005 ELSE, PUBLISHED AS IS, MOVES 9695 05:46:00,072 --> 05:46:01,240 DOWN THE CHAIN UNTIL SOMEONE 9696 05:46:01,307 --> 05:46:02,608 WILL HELP IT. 9697 05:46:02,675 --> 05:46:04,910 NOW THAT APPEARANCE OF TRUTH IS 9698 05:46:04,977 --> 05:46:06,112 MORE DANGEROUS. 9699 05:46:06,178 --> 05:46:07,213 THE BEST THING PRE-PRINTS DID 9700 05:46:07,279 --> 05:46:13,586 WAS FORCE US TO QUESTION 9701 05:46:13,652 --> 05:46:14,353 EVERYTHING. 9702 05:46:14,420 --> 05:46:17,289 >> NOT TO FORGET SCIENCE NOT NOT 9703 05:46:17,356 --> 05:46:18,624 HAPPEN AT NIH OR INSTITUTIONS, 9704 05:46:18,691 --> 05:46:21,560 IT HAPPENS IN SOCIETY. 9705 05:46:21,627 --> 05:46:26,699 IT'S ALWAYS A CONVERSATION IN 9706 05:46:26,765 --> 05:46:26,932 SOCIETY. 9707 05:46:26,999 --> 05:46:28,901 SOMETIMES WE SUFFER A SWOLLEN 9708 05:46:28,968 --> 05:46:33,372 HEAD AND THINK WE'RE ABOVE THE 9709 05:46:33,439 --> 05:46:34,273 CITIZENRY, THESE ARE STUPID 9710 05:46:34,340 --> 05:46:36,108 PEOPLE WHO ARE GOING TO 9711 05:46:36,175 --> 05:46:37,409 INTERPRET, MISINTERPRET. 9712 05:46:37,476 --> 05:46:41,046 THAT IS PART OF THE CONVERSATION 9713 05:46:41,113 --> 05:46:42,581 AROUND RESEARCH, AND MORE 9714 05:46:42,648 --> 05:46:46,285 TRANSPARENCY AND MORE EDUCATION, 9715 05:46:46,352 --> 05:46:50,122 MORE CONVERSATION, I THINK, WE 9716 05:46:50,189 --> 05:46:52,091 CANNOT WALL OFF SOME 9717 05:46:52,158 --> 05:46:54,126 INCONVENIENT DISCOVERY. 9718 05:46:54,193 --> 05:46:58,297 WE JUST HAVE TO MAKE SURE THAT 9719 05:46:58,364 --> 05:47:01,367 THE VERACITY OF THAT FINDING CAN 9720 05:47:01,433 --> 05:47:03,669 BE TESTED BY OTHERS IN THE 9721 05:47:03,736 --> 05:47:06,071 SCIENTIFIC COMMUNITY AND THAT 9722 05:47:06,138 --> 05:47:08,040 WE'RE HONEST AND CAN EXPLAIN 9723 05:47:08,107 --> 05:47:12,311 DIFFICULT SUBJECTS AND HAVE A 9724 05:47:12,378 --> 05:47:13,078 CONVERSATION, SOCIETAL 9725 05:47:13,145 --> 05:47:15,281 CONVERSATION, WHETHER SOCIAL 9726 05:47:15,347 --> 05:47:21,787 MEDIA, NEWSPAPERS, ET CETERA. 9727 05:47:21,854 --> 05:47:25,324 >> A LOT OF TIMES THEY DON'T 9728 05:47:25,391 --> 05:47:26,959 INTERVIEW THE AUTHORS OR OTHER 9729 05:47:27,026 --> 05:47:29,061 EXPERTS TO SEE WHAT THE STORY 9730 05:47:29,128 --> 05:47:32,765 ACTUALLY IS OR WHAT THE LIMITS 9731 05:47:32,831 --> 05:47:34,633 OF THE DATA ARE. 9732 05:47:34,700 --> 05:47:36,135 HOW MUCH IS EXAGGERATED IN 9733 05:47:36,202 --> 05:47:37,136 DISCUSSIONS, SO ON. 9734 05:47:37,203 --> 05:47:39,672 THERE ARE MULTIPLE CASES, COVID 9735 05:47:39,738 --> 05:47:41,974 OR NOT, WHERE PRE-PRINTS OR PEER 9736 05:47:42,041 --> 05:47:42,741 REVIEWED ARTICLES WERE REPORTED 9737 05:47:42,808 --> 05:47:47,146 IN THE WAY THAT WAS NOT 9738 05:47:47,213 --> 05:47:48,514 ACCURATE. 9739 05:47:48,581 --> 05:47:50,816 AND SO BECAUSE SCIENCE IS NOT 9740 05:47:50,883 --> 05:47:54,320 REALLY DONE OR REPORTED IN A 9741 05:47:54,386 --> 05:47:55,521 VACUUM I THINK WOULD BE 9742 05:47:55,588 --> 05:47:57,489 IMPORTANT FOR JOURNALISTS TO DO 9743 05:47:57,556 --> 05:48:00,092 THEIR JOB, A BIT BETTER TOO. 9744 05:48:00,159 --> 05:48:03,696 >> IF I MAY ADD ON, THERE'S 9745 05:48:03,762 --> 05:48:05,064 RESPONSE FROM RESEARCHERS 9746 05:48:05,130 --> 05:48:07,099 THEMSELVES HAS TO DO WITH 9747 05:48:07,166 --> 05:48:08,534 INCENTIVES. 9748 05:48:08,601 --> 05:48:11,503 TO THE EXTENT RESEARCHERS ARE 9749 05:48:11,570 --> 05:48:13,005 ENCOURAGED TO HAVE IMPACT, 9750 05:48:13,072 --> 05:48:14,540 DEMONSTRATE IMPACT IN 9751 05:48:14,607 --> 05:48:16,642 INSTITUTIONS, THEY TEND TO TALK 9752 05:48:16,709 --> 05:48:17,943 MORE TO MEDIA, SOMETHING PEOPLE 9753 05:48:18,010 --> 05:48:24,316 PUT ON THEIR C.V., WHEN THEY 9754 05:48:24,383 --> 05:48:26,919 SPEAK TO THE MEDIA NOT AS 9755 05:48:26,986 --> 05:48:28,120 CAREFUL, YOU SEE EXAGGERATED 9756 05:48:28,187 --> 05:48:34,627 CLAIMS BEING MADE THAT GET 9757 05:48:34,693 --> 05:48:42,268 FURTHER DISTORTED BY THE MEDIA. 9758 05:48:42,334 --> 05:48:44,436 >> HOW DOES THE NON-TRADITIONAL 9759 05:48:44,503 --> 05:48:52,378 VENUES DEFER FROM PRE-PRINT 9760 05:48:52,444 --> 05:48:52,678 SERVICE? 9761 05:48:54,313 --> 05:48:56,982 >> MICRO PUBLICATION IS PEER 9762 05:48:57,049 --> 05:48:57,383 REVIEWED. 9763 05:48:57,449 --> 05:49:06,759 PRE-PRINT CENTERS ARE NOT 9764 05:49:06,825 --> 05:49:07,660 REVIEWED. 9765 05:49:07,726 --> 05:49:09,028 >> OCTOPUS IS NOT. 9766 05:49:09,094 --> 05:49:10,663 >> A LOT OF PEOPLE ASK ME WHY WE 9767 05:49:10,729 --> 05:49:12,298 DIDN'T DO THIS. 9768 05:49:12,364 --> 05:49:13,932 TWO PRIMARY REASONS, WITH YOU 9769 05:49:13,999 --> 05:49:15,834 WE'RE VERY INTERESTED IN REUSE 9770 05:49:15,901 --> 05:49:18,237 OF THE DATA AND UTILITY OF THE 9771 05:49:18,304 --> 05:49:21,640 DATA, WE DIDN'T WANT TO BE STUCK 9772 05:49:21,707 --> 05:49:25,010 IN A SINGLE FORMAT LIKE PDF, 9773 05:49:25,077 --> 05:49:27,713 CONVERT TO XML, WE WANTED 9774 05:49:27,780 --> 05:49:29,581 ABILITY FOR FEMUR DIRECTLY 9775 05:49:29,648 --> 05:49:32,217 INTERACT AND REDISPLAY, PLAY 9776 05:49:32,284 --> 05:49:36,155 WITH DATA FORMATS BECAUSE WE 9777 05:49:36,221 --> 05:49:40,092 DON'T DO PRE-PUBLICATION OR PEER 9778 05:49:40,159 --> 05:49:41,527 REVIEW WE WANTED TO MAKE SURE 9779 05:49:41,593 --> 05:49:50,069 PEOPLE COULD USE THE DATA, NOT 9780 05:49:50,135 --> 05:49:55,607 LIKE AN EXCEL SPREADSHEET. 9781 05:49:55,674 --> 05:50:00,479 WE ON PURPOSE DON'T HAVE A 9782 05:50:00,546 --> 05:50:01,647 HIERARCHICAL BYLINE, SOMETHING 9783 05:50:01,714 --> 05:50:06,719 WE HAD TOE DO IF WE GO ON 9784 05:50:06,785 --> 05:50:09,822 PRE-PRINT OFFICE, WE USE CREDIT 9785 05:50:09,888 --> 05:50:13,058 TAXONOMY, WHO DID WHAT? 9786 05:50:13,125 --> 05:50:14,493 IT'S IN ALPHABETICAL ORDER ON 9787 05:50:14,560 --> 05:50:17,229 THE BOTTOM OF OUR PUBLICATION. 9788 05:50:17,296 --> 05:50:18,330 IT'S IMPORTANT BECAUSE 9789 05:50:18,397 --> 05:50:19,231 AUTHORSHIP AND CONTRIBUTORSHIP 9790 05:50:19,298 --> 05:50:21,867 IS NOT JUST ABOUT CREDIT. 9791 05:50:21,934 --> 05:50:23,502 IT'S ALSO ABOUT RESPONSIBILITY, 9792 05:50:23,569 --> 05:50:25,204 VERY IMPORTANT IT'S CLEAR WHO IS 9793 05:50:25,270 --> 05:50:28,073 RESPONSIBLE FOR WHAT, IN A 9794 05:50:28,140 --> 05:50:30,809 MODULAR WAY, WE HAVE 9795 05:50:30,876 --> 05:50:32,244 COLLABORATIVE RESEARCH, WE 9796 05:50:32,311 --> 05:50:33,679 WANTED TO GO OUTSIDE, FIRST 9797 05:50:33,746 --> 05:50:37,149 AUTHOR, LAST AUTHOR, MY PROJECT, 9798 05:50:37,216 --> 05:50:39,785 YOUR PROJECT. 9799 05:50:39,852 --> 05:50:43,155 I DON'T GET MAXIMUM CREDIT. 9800 05:50:43,222 --> 05:50:44,590 IT ACCELERATED OUR SCIENCE, TWO 9801 05:50:44,656 --> 05:50:47,593 REASONS WE DECIDED TO NOT OPT 9802 05:50:47,659 --> 05:50:50,195 FOR PRE-PRINT, I WANT THE AUTHOR 9803 05:50:50,262 --> 05:50:52,030 AGENCY OF PEOPLE POST WHEN 9804 05:50:52,097 --> 05:50:54,166 SCIENCE IS READY, THAT WAS KEY, 9805 05:50:54,233 --> 05:50:59,004 IN THE SPIRIT OF THAT WE'RE 9806 05:50:59,071 --> 05:51:00,305 STILL DISCOVERABLE, ALL THOSE 9807 05:51:00,372 --> 05:51:02,941 OTHER THINGS BUT THESE TWO 9808 05:51:03,008 --> 05:51:04,543 FEATURES ARE WHY WE DID NOT WANT 9809 05:51:04,610 --> 05:51:06,412 TO DO A TRADITIONAL PRE-PRINT. 9810 05:51:06,478 --> 05:51:09,047 >> THE OTHER QUESTION, HOW DO 9811 05:51:09,114 --> 05:51:12,785 THE FUNDERS RECOGNIZE THESE 9812 05:51:12,851 --> 05:51:13,852 PLATFORMS? 9813 05:51:13,919 --> 05:51:15,621 FOR EXAMPLE, FROM THE EARLY 9814 05:51:15,687 --> 05:51:20,192 CAREER RESEARCH POINT OF VIEW, 9815 05:51:20,259 --> 05:51:23,061 WHEN Ph.D. HAVE PAPER, WHEN I 9816 05:51:23,128 --> 05:51:25,030 STARTED MY POSTDOC OBLIGATIONS, 9817 05:51:25,097 --> 05:51:28,233 FUNDERS IN EUROPE WERE 9818 05:51:28,300 --> 05:51:30,536 EXPLICITLY ASKED FOR A PAPER 9819 05:51:30,602 --> 05:51:33,005 RESULTS FROM YOUR Ph.D. 9820 05:51:33,071 --> 05:51:33,539 STUDIES. 9821 05:51:33,605 --> 05:51:36,208 NOW THEY CHANGED EVEN IF YOU 9822 05:51:36,275 --> 05:51:38,243 HAVE A PRE-PRINT THAT'S ENOUGH 9823 05:51:38,310 --> 05:51:40,879 TO BE QUALIFIED FOR APPLICATION 9824 05:51:40,946 --> 05:51:42,214 OF POSTDOCTORAL FELLOWSHIPS, FOR 9825 05:51:42,281 --> 05:51:43,182 EXAMPLE. 9826 05:51:43,248 --> 05:51:45,484 IT'S VERY IMPORTANT. 9827 05:51:45,551 --> 05:51:48,587 SO bioRxiv HAS BEEN WELL 9828 05:51:48,654 --> 05:51:50,489 RECOGNIZED IN THAT ASPECT. 9829 05:51:50,556 --> 05:51:52,024 DOES YOUR PLATFORM ALSO 9830 05:51:52,090 --> 05:51:53,659 RECOGNIZED BY FUNDERS WHEN IT 9831 05:51:53,725 --> 05:51:54,660 COMES TO CAREER PROGRESSION? 9832 05:51:54,726 --> 05:51:56,228 >> I CAN'T SPEAK FOR INDIVIDUAL 9833 05:51:56,295 --> 05:51:59,765 FUNDERS BUT IN EVERY WAY THAT'S 9834 05:51:59,832 --> 05:52:02,434 MEANINGFUL WE HAVE THE SAME 9835 05:52:02,501 --> 05:52:04,102 TYPES OF, YOU KNOW, LIKE 9836 05:52:04,169 --> 05:52:05,604 PROPERTIES AS A PRE-PRINT 9837 05:52:05,671 --> 05:52:06,004 SERVER. 9838 05:52:06,071 --> 05:52:07,706 WE HAVE EXTRA BELLS AND WHISTLES 9839 05:52:07,773 --> 05:52:11,443 ON TOP OF THAT BUT WE MEET THE 9840 05:52:11,510 --> 05:52:13,111 SAME CRITERIA, SO THEORETICALLY 9841 05:52:13,178 --> 05:52:18,550 A LOT OF PLATFORMS TO BE 9842 05:52:18,617 --> 05:52:20,652 COMPLIANT, YEAH. 9843 05:52:20,719 --> 05:52:23,355 >> NOTHING TO ADD EXCEPT EXACTLY 9844 05:52:23,422 --> 05:52:26,625 THE SAME, YEP. 9845 05:52:26,692 --> 05:52:27,793 >> OKAY. 9846 05:52:27,860 --> 05:52:30,996 MY QUESTION IS FOR YOU. 9847 05:52:31,063 --> 05:52:33,665 YOU TALKED ABOUT IN YOUR STUDIES 9848 05:52:33,732 --> 05:52:35,968 YOU FOUND YOUNGER EARLY STAGE 9849 05:52:36,034 --> 05:52:37,636 SCIENTISTS MORE ENTHUSIASTIC AND 9850 05:52:37,703 --> 05:52:43,008 WILLING TO EMBRACE OPEN SCIENCE, 9851 05:52:43,075 --> 05:52:45,577 PARTICULARLY YOU TALKED ABOUT 9852 05:52:45,644 --> 05:52:45,844 POSTDOCS. 9853 05:52:45,911 --> 05:52:47,646 WAS THAT A MORE BROAD STUDY AND 9854 05:52:47,713 --> 05:52:51,383 DID YOU IN YOUR STUDY FIND THAT 9855 05:52:51,450 --> 05:52:55,220 MORE MID-LEVEL AND SENIOR LEVEL 9856 05:52:55,287 --> 05:52:59,391 PEOPLE, ARE THEY INDIFFERENT OR 9857 05:52:59,458 --> 05:53:00,692 RESISTANT OR SLOW ADAPTERS? 9858 05:53:00,759 --> 05:53:02,528 EVEN IN YOUNGER COHORT IS THERE 9859 05:53:02,594 --> 05:53:05,497 A DIFFERENCE IN BIOMEDICAL 9860 05:53:05,564 --> 05:53:08,934 VERSUS HOW YOU FOUND ECOLOGY AND 9861 05:53:09,001 --> 05:53:09,501 EVOLUTION? 9862 05:53:09,568 --> 05:53:10,302 >> DEFINITELY, THANK YOU. 9863 05:53:10,369 --> 05:53:13,372 STUDY ON THE FACULTY JOB MARKET 9864 05:53:13,438 --> 05:53:16,642 WAS FOR RESEARCHERS IN EUROPE 9865 05:53:16,708 --> 05:53:18,176 AND U.S. 9866 05:53:18,243 --> 05:53:19,311 BUT MAJORITY OF RESPONDENTS WERE 9867 05:53:19,378 --> 05:53:20,913 IN THE U.S. 9868 05:53:20,979 --> 05:53:24,082 AND SO IT WASN'T -- I WASN'T AT 9869 05:53:24,149 --> 05:53:27,586 U-CONN, I WAS AT JOHNS HOPKINS, 9870 05:53:27,653 --> 05:53:31,089 IT WASN'T LIMITED TO JOHNS 9871 05:53:31,156 --> 05:53:31,857 HOPKINS. 9872 05:53:31,924 --> 05:53:33,125 IT'S TRUE EARLY CAREER 9873 05:53:33,191 --> 05:53:34,626 RESEARCHERS ARE INTERESTED IN 9874 05:53:34,693 --> 05:53:39,097 PRE-PRINTING BUT IT IS ALSO TRUE 9875 05:53:39,164 --> 05:53:41,033 THAT SOME ARE INTERESTED IN 9876 05:53:41,099 --> 05:53:41,767 PUBLISHING IN HIGH-IMPACT 9877 05:53:41,833 --> 05:53:42,134 JOURNALS. 9878 05:53:42,200 --> 05:53:45,304 I HAVE TALKED TO FACULTY WHO 9879 05:53:45,370 --> 05:53:47,773 TELL ME STUDENTS ASK THEM TO 9880 05:53:47,839 --> 05:53:50,475 PUBLISH, TO SEND MANUSCRIPT TO 9881 05:53:50,542 --> 05:53:53,312 "CELL" OR "NATURE," SO THEIR 9882 05:53:53,378 --> 05:53:57,349 CAREER STRUCTURE, WE RAN THE 9883 05:53:57,416 --> 05:53:59,217 SURVEY TO REALLY BUST SOME 9884 05:53:59,284 --> 05:53:59,551 MYTHS. 9885 05:53:59,618 --> 05:54:04,156 WE SHOWED THERE ARE PEOPLE THAT 9886 05:54:04,222 --> 05:54:06,592 DIDN'T HAVE "SCIENCE" OR 9887 05:54:06,658 --> 05:54:07,993 "NATURE" ARTICLES AND STILL GOT 9888 05:54:08,060 --> 05:54:10,596 FACULTY POSITIONS. 9889 05:54:10,662 --> 05:54:12,698 WE ALSO FOUND 7% INCREASED 9890 05:54:12,764 --> 05:54:13,565 CHANCE OF GETTING FACULTY 9891 05:54:13,632 --> 05:54:21,106 POSITION IF YOU HAVE THOSE 9892 05:54:21,173 --> 05:54:25,644 ARTICLES. 9893 05:54:25,711 --> 05:54:28,647 IT'S A HETEROGENOUS COMMUNITY OF 9894 05:54:28,714 --> 05:54:30,315 RESEARCHERS, SOME DO PRE-PRINT A 9895 05:54:30,382 --> 05:54:30,682 LOT. 9896 05:54:30,749 --> 05:54:33,585 THEY GO FOR THAT, FOR THE 9897 05:54:33,652 --> 05:54:36,088 FACULTY JOB WITH THAT ONLY ON 9898 05:54:36,154 --> 05:54:37,222 THEIR C.V. 9899 05:54:37,289 --> 05:54:41,660 I WOULD BE VERY INTERESTED IN A 9900 05:54:41,727 --> 05:54:51,670 WORKSHOP LIKE THIS FOR IMPROVING 9901 05:54:51,737 --> 05:54:52,037 FACULTY HIRING. 9902 05:54:52,104 --> 05:54:53,238 THERE ARE A LOT OF ISSUES IN 9903 05:54:53,305 --> 05:54:54,473 FACULTY HIRING. 9904 05:54:54,539 --> 05:54:56,775 THERE ARE DISCUSSIONS ON THE 9905 05:54:56,842 --> 05:54:59,978 FUTURE P.I. COMMUNITY WHERE 9906 05:55:00,045 --> 05:55:00,712 POSTDOCS MENTION THEIR 9907 05:55:00,779 --> 05:55:07,386 COLLABORATORS OR P.I.s ARE THE 9908 05:55:07,452 --> 05:55:08,820 BARRIER FOR PRE-PRINTING. 9909 05:55:08,887 --> 05:55:11,156 THAT'S WHY I THINK -- AND THIS 9910 05:55:11,223 --> 05:55:14,326 IS FIELD DEPENDENT TOO. 9911 05:55:14,393 --> 05:55:15,627 IN ROBOTICS, PHYSICS, 9912 05:55:15,694 --> 05:55:17,496 MATHEMATICS, IT'S VERY 9913 05:55:17,562 --> 05:55:18,163 ESTABLISHED IN THE COMMUNITY 9914 05:55:18,230 --> 05:55:20,832 PEOPLE COME IN THE MORNING AND 9915 05:55:20,899 --> 05:55:22,134 READ ARCHIVE. 9916 05:55:22,200 --> 05:55:23,268 THIS HAS NOT HAPPENED IN 9917 05:55:23,335 --> 05:55:23,702 BIOMEDICAL FIELD. 9918 05:55:23,769 --> 05:55:29,841 I THINK IF YOU LOOK AT THE DATA 9919 05:55:29,908 --> 05:55:32,144 FROM FOUNDERS OF bioRxiv AND 9920 05:55:32,210 --> 05:55:34,513 NET ARCHIVE, IT'S RISING BUT A 9921 05:55:34,579 --> 05:55:36,948 STEEPENED PERCENT AFTER ALMOST 9922 05:55:37,015 --> 05:55:40,118 TEN YEARS, ONLY 10% OF ALL THE 9923 05:55:40,185 --> 05:55:44,589 SCIENTIFIC OUTPUT WHICH IS OVER 9924 05:55:44,656 --> 05:55:48,026 A MILLION A YEAR, SCIENTIFIC, IT 9925 05:55:48,093 --> 05:55:51,563 GOES TO BIOMEDICAL PRE-PRINT 9926 05:55:51,630 --> 05:55:52,764 SERVER. 9927 05:55:52,831 --> 05:55:56,234 I THINK IT'S TAKING FAR SLOWER 9928 05:55:56,301 --> 05:55:58,870 TO GET THERE IN BIOMEDICAL 9929 05:55:58,937 --> 05:56:01,540 SCIENCES AS OPPOSED TO PHYSICIST 9930 05:56:01,606 --> 05:56:02,808 AND MATHEMATICIANS. 9931 05:56:02,874 --> 05:56:11,583 I THINK THERE ARE PHYSICISTS AND 9932 05:56:11,650 --> 05:56:15,053 MAT MATHEMATICIANS THAT SUBMIT 9933 05:56:15,120 --> 05:56:16,154 BUT COMPARING DIFFERENCES WOULD 9934 05:56:16,221 --> 05:56:18,056 BE VERY HELPFUL IN INCREASING 9935 05:56:18,123 --> 05:56:21,526 THE UPTAKE OF PRE-PRINTS. 9936 05:56:21,593 --> 05:56:23,128 I ALSO THINK THERE'S MORE OF 9937 05:56:23,195 --> 05:56:24,696 MAYBE THIS CONVERSATIONS IN 9938 05:56:24,763 --> 05:56:28,233 TERMS OF WORKSHOPS OR 9939 05:56:28,300 --> 05:56:29,367 CONFERENCES, ON INTRODUCING 9940 05:56:29,434 --> 05:56:34,106 PRE-PRINTS MAY HELP P.I., 9941 05:56:34,172 --> 05:56:37,142 FACULTY, WHO HAVE THE CONTROL OF 9942 05:56:37,209 --> 05:56:38,243 THE CORRESPONDING AUTHOR TO 9943 05:56:38,310 --> 05:56:40,212 UNDERSTAND IT'S OKAY TO 9944 05:56:40,278 --> 05:56:45,050 PRE-PRINT AND YOU DON'T 9945 05:56:45,117 --> 05:56:46,518 REALLY -- ALMOST ALL JOURNALS 9946 05:56:46,585 --> 05:56:47,786 ACCEPT PRE-PRINTS, THERE'S NO 9947 05:56:47,853 --> 05:56:49,454 BARRIER TO PUBLISHING IN A 9948 05:56:49,521 --> 05:56:50,856 JOURNAL WHEN YOU PRE-PRINT. 9949 05:56:50,922 --> 05:56:54,192 I THINK TO THE POINT THAT YOU 9950 05:56:54,259 --> 05:56:57,529 ALSO MENTIONED, MANDATES WOULD 9951 05:56:57,596 --> 05:57:04,369 BE HELPFUL IN PRE-PRINTING SO WE 9952 05:57:04,436 --> 05:57:05,637 GET THERE. 9953 05:57:05,704 --> 05:57:07,506 MICHAEL J. FOX FOUNDATION 9954 05:57:07,572 --> 05:57:09,641 MANDATED PRE-PRINTS, IT'S A 9955 05:57:09,708 --> 05:57:12,677 SMALL FOUNDATION BUT I THINK NIH 9956 05:57:12,744 --> 05:57:14,179 OR WELLCOME TRUST MANDATES 9957 05:57:14,246 --> 05:57:20,285 PRE-PRINTS, INSTEAD OF JUST 9958 05:57:20,352 --> 05:57:24,256 ENCOURAGING IT, WOULD BE REALLY 9959 05:57:24,322 --> 05:57:24,756 HELPFUL. 9960 05:57:24,823 --> 05:57:26,424 JUST AS MUCH LEVERAGE IN SOMEONE 9961 05:57:26,491 --> 05:57:31,429 ELSE'S LACKS FOR DECISIONS IN 9962 05:57:31,496 --> 05:57:32,964 PUBLISHING, THEY OBVIOUSLY, YOU 9963 05:57:33,031 --> 05:57:34,766 KNOW, VOICE INTEREST IN 9964 05:57:34,833 --> 05:57:37,369 PRE-PRINTING BUT AT THE END DAY 9965 05:57:37,435 --> 05:57:39,638 THE DECISION BY FACULTY MAY BE 9966 05:57:39,704 --> 05:57:43,341 SOMETHING DIFFERENT. 9967 05:57:43,408 --> 05:57:46,545 SO, HAVING MORE DISCUSSIONS, YOU 9968 05:57:46,611 --> 05:57:51,449 KNOW, REALLY COMMUNITY-WIDE 9969 05:57:51,516 --> 05:57:53,385 DISCUSSIONS WOULD BE HELPFUL. 9970 05:57:53,451 --> 05:57:58,990 >> LET ME JUMP IN, A QUESTION ON 9971 05:57:59,057 --> 05:58:00,392 THE PRE-PRINT FRONT. 9972 05:58:00,458 --> 05:58:02,460 PEOPLE ARE CONCERNED ABOUT 9973 05:58:02,527 --> 05:58:04,162 PUTTING OUT THEIR WORK BEFORE 9974 05:58:04,229 --> 05:58:05,630 FORMAL PUBLICATION BECAUSE THEY 9975 05:58:05,697 --> 05:58:07,966 ARE AFRAID SOMEONE ELSE IS GOING 9976 05:58:08,033 --> 05:58:11,269 TO SCOOP THEIR IDEA AND BEAT 9977 05:58:11,336 --> 05:58:17,375 THEM TO THE PUBLICATION FASTER? 9978 05:58:17,442 --> 05:58:21,780 >> I WAS GOING TO ASK THAT, 9979 05:58:21,847 --> 05:58:23,181 RELATED TO THAT. 9980 05:58:23,248 --> 05:58:24,583 >> YES, IT'S DEFINITELY A THING. 9981 05:58:24,649 --> 05:58:27,586 PEOPLE HAVE THIS CONCERN. 9982 05:58:27,652 --> 05:58:29,654 I THINK A LOT OF EDUCATION IS A 9983 05:58:29,721 --> 05:58:30,488 BIG PIECE OF THIS. 9984 05:58:30,555 --> 05:58:33,325 A LOT OF PEOPLE HAVE FEAR OF 9985 05:58:33,391 --> 05:58:33,925 PRE-PRINTS THAT DON'T TOTALLY 9986 05:58:33,992 --> 05:58:35,327 UNDERSTAND WHAT THEY ARE OR 9987 05:58:35,393 --> 05:58:39,130 WHAT'S POSSIBLE OR WHAT IT MEANS 9988 05:58:39,197 --> 05:58:42,968 OR I THINK OFTEN THE ARGUMENT 9989 05:58:43,034 --> 05:58:45,570 I'VE MOST BEEN ABLE TO COMBAT 9990 05:58:45,637 --> 05:58:47,539 THIS IDEA WITH IS THAT I THINK 9991 05:58:47,606 --> 05:58:48,974 IT'S PRETTY WELL ACCEPTED AND 9992 05:58:49,040 --> 05:58:50,709 PEOPLE HAVE NO PROBLEMS SHARING 9993 05:58:50,775 --> 05:58:53,245 THEIR EARLY RESEARCH AT 9994 05:58:53,311 --> 05:58:57,616 CONFERENCES WITH THEIR LIKE 9995 05:58:57,682 --> 05:58:58,917 TIGHTEST STRONGEST COMPETITORS, 9996 05:58:58,984 --> 05:59:00,919 AND THAT WHERE THERE'S NO SORT 9997 05:59:00,986 --> 05:59:03,622 OF CLAIM OF PRIORITY, THERE'S NO 9998 05:59:03,688 --> 05:59:05,490 TIME STAMP THERE THAT COULD 9999 05:59:05,557 --> 05:59:07,392 SIGNAL PRIORITY, SO IN MANY 10000 05:59:07,459 --> 05:59:08,894 CASES THE SAFER THING TO DO THAN 10001 05:59:08,960 --> 05:59:11,529 WHAT THEY ARE ALREADY DOING IS 10002 05:59:11,596 --> 05:59:12,931 TO PRE-PRINT THEIR WORK SO BUT 10003 05:59:12,998 --> 05:59:15,000 IN MANY CASES THEY JUST DON'T 10004 05:59:15,066 --> 05:59:19,838 KNOW WHAT IT MEANS AND SO A BIG 10005 05:59:19,905 --> 05:59:22,240 BARRIER FOR TRYING TO GET 10006 05:59:22,307 --> 05:59:23,441 ADOPTION THIS WAY, SOMETIMES IT 10007 05:59:23,508 --> 05:59:24,876 FEELS LIKE EDUCATION AND HOLDING 10008 05:59:24,943 --> 05:59:26,611 SOMEONE'S HAND SO THEY CAN 10009 05:59:26,678 --> 05:59:28,580 UNDERSTAND WHAT DO THE LICENSES 10010 05:59:28,647 --> 05:59:31,082 MEAN, ALL OF THE ASPECTS THAT GO 10011 05:59:31,149 --> 05:59:34,419 INTO THIS AND WHAT ARE THE 10012 05:59:34,486 --> 05:59:35,954 DOWNSTREAM IMPLICATIONS, IT'S 10013 05:59:36,021 --> 05:59:37,055 TOTALLY NOT MUTUALLY EXCLUSIVE 10014 05:59:37,122 --> 05:59:39,224 WITH JOURNALS AND ALL THESE 10015 05:59:39,291 --> 05:59:41,626 THINGS, IT'S LIKE OPEN 10016 05:59:41,693 --> 05:59:44,496 QUESTIONS, PEOPLE ARE NEW AND 10017 05:59:44,562 --> 05:59:47,132 DON'T REALLY KNOW. 10018 05:59:47,198 --> 05:59:48,033 RESOURCES, INFINITE COLLECTION 10019 05:59:48,099 --> 05:59:50,035 OF RESOURCES ABOUT ALL VARIOUS 10020 05:59:50,101 --> 05:59:53,571 ASPECTS SO I THINK THAT'S A BIG 10021 05:59:53,638 --> 05:59:53,805 PIECE. 10022 05:59:53,872 --> 05:59:57,709 >> SCOOPING IS ONE OF THE 10023 05:59:57,776 --> 06:00:00,779 BIGGEST PROBLEMS PEOPLE SAY 10024 06:00:00,845 --> 06:00:02,948 STOPS THEM FROM PUBLISHING, THEY 10025 06:00:03,014 --> 06:00:04,482 MIGHT PUBLISH A HYPOTHESIS 10026 06:00:04,549 --> 06:00:06,484 BEFORE PROTOCOL BEFORE THEY 10027 06:00:06,551 --> 06:00:07,619 PUBLISH RESULTS. 10028 06:00:07,686 --> 06:00:10,188 ACTUALLY IT'S THE OPPOSITE. 10029 06:00:10,255 --> 06:00:11,923 IF YOU'RE HOLDING ON OR TALKING 10030 06:00:11,990 --> 06:00:15,994 ABOUT IT AT CONFERENCES WITHOUT 10031 06:00:16,061 --> 06:00:17,529 HAVING PUBLISHED, YOU COULD 10032 06:00:17,595 --> 06:00:18,997 POTENTIALLY GET SCOOPED. 10033 06:00:19,064 --> 06:00:20,398 ONCE IT'S THERE IT'S LIKE A 10034 06:00:20,465 --> 06:00:25,270 PATENT OFFICE WITH YOUR NAME ON 10035 06:00:25,337 --> 06:00:27,572 THE DATA, YOU CAN CITE IF 10036 06:00:27,639 --> 06:00:28,573 SOMEBODY COLLECTS DATA AND 10037 06:00:28,640 --> 06:00:32,477 PROVES YOU WERE RIGHT, THE YOUR 10038 06:00:32,544 --> 06:00:32,877 IDEA. 10039 06:00:32,944 --> 06:00:36,314 >> I THINK THE GOOD FEATURE OF 10040 06:00:36,381 --> 06:00:37,415 THESE COMPLICATED PAPERS, IF 10041 06:00:37,482 --> 06:00:39,284 SOMEONE WANTS TO SCOOP YOU AND 10042 06:00:39,351 --> 06:00:41,886 SPEND THREE YEARS DOING THAT, 10043 06:00:41,953 --> 06:00:43,588 YOU KNOW, GO FOR IT. 10044 06:00:43,655 --> 06:00:46,925 BUT THERE ARE CASES WHETHER 10045 06:00:46,992 --> 06:00:48,026 OBVIOUSLY IT'S VERY QUICK AND 10046 06:00:48,093 --> 06:00:52,430 YOU SHOULDN'T SCREW AROUND WITH 10047 06:00:52,497 --> 06:00:52,797 IT. 10048 06:00:52,864 --> 06:00:55,800 >> IT HAS TO DO MORE WITH THE 10049 06:00:55,867 --> 06:00:57,669 JOURNAL'S POSITION, HOW IN 10050 06:00:57,736 --> 06:00:58,636 DIFFERENT FIELDS YOU CREDIT WHO 10051 06:00:58,703 --> 06:01:01,606 IS THE ONE WHO IS CREDITED FOR 10052 06:01:01,673 --> 06:01:04,209 ORIGINATING THE IDEA, WHO GOT IT 10053 06:01:04,275 --> 06:01:05,543 PUBLISHED IN A OFFICIAL JOURNAL 10054 06:01:05,610 --> 06:01:14,853 OR GOT IT FIRST IN A PRE-PRINT? 10055 06:01:14,919 --> 06:01:18,323 >> PRE-PRINTS ARE NOW IN 10056 06:01:18,390 --> 06:01:18,556 PubMed. 10057 06:01:18,623 --> 06:01:21,926 SOMEONE WHO IS AN HONEST 10058 06:01:21,993 --> 06:01:24,562 RESEARCHER PROPERLY CITING CAN 10059 06:01:24,629 --> 06:01:25,397 CERTAINLY ACKNOWLEDGE. 10060 06:01:25,463 --> 06:01:27,632 WE'RE SEEING AS A PRE-PRINT 10061 06:01:27,699 --> 06:01:29,067 COMES OUT GOOD DATA 10062 06:01:29,134 --> 06:01:34,406 AVAILABILITY, PROVENANCE OF 10063 06:01:34,472 --> 06:01:36,307 FINDINGS, ANOTHER PRE-PRINT 10064 06:01:36,374 --> 06:01:38,276 COMES OUT, QUESTIONING IT, WE 10065 06:01:38,343 --> 06:01:40,678 ANALYZE AND FIND SOMETHING ELSE, 10066 06:01:40,745 --> 06:01:42,547 HOW SCIENCE SHOULD BE. 10067 06:01:42,614 --> 06:01:44,682 YES AND KNOW, TIME STAMP FOR 10068 06:01:44,749 --> 06:01:46,818 COMPLEX STUDIES, 2 1/2 YEARS TO 10069 06:01:46,885 --> 06:01:47,552 PUBLISH. 10070 06:01:47,619 --> 06:01:51,956 YOUNG INVESTIGATORS AND FOR THE 10071 06:01:52,023 --> 06:01:52,290 LEADING P.I.s. 10072 06:01:52,357 --> 06:01:53,925 THEY REALLY DON'T WANT TO RISK 10073 06:01:53,992 --> 06:01:58,897 JUST GOING DARK AND JUST REALLY 10074 06:01:58,963 --> 06:02:03,701 BEING SCOOPED BY SOMEBODY. 10075 06:02:03,768 --> 06:02:03,968 >> OKAY. 10076 06:02:04,035 --> 06:02:06,137 THE SCOOP WAS MY QUESTION. 10077 06:02:06,204 --> 06:02:10,675 I'LL ASK ANOTHER ONE. 10078 06:02:13,044 --> 06:02:16,114 YEAH, I AGREE WITH THE BENEFITS 10079 06:02:16,181 --> 06:02:17,582 OF THE PRE-PRINTS, I HAVE TO 10080 06:02:17,649 --> 06:02:19,951 ADMIT I DIDN'T KNOW THE 10081 06:02:20,018 --> 06:02:21,653 DIFFERENT PLATFORM, WHETHER THEY 10082 06:02:21,719 --> 06:02:23,955 ARE ALL ON PubMed, FOR 10083 06:02:24,022 --> 06:02:24,956 INSTANCE. 10084 06:02:25,023 --> 06:02:27,659 I KNOW SOME ARE PEER REVIEWED, 10085 06:02:27,725 --> 06:02:32,730 SOME ARE NOT. 10086 06:02:32,797 --> 06:02:34,666 IT DOESN'T GUARANTEE IT'S A 10087 06:02:34,732 --> 06:02:36,968 VALUED STUDY BUT IT'S A 10088 06:02:37,035 --> 06:02:37,836 CHECKPOINT. 10089 06:02:37,902 --> 06:02:39,771 PAPERS ARE REVIEWED, REVISED, 10090 06:02:39,838 --> 06:02:40,038 REJECTED. 10091 06:02:40,105 --> 06:02:42,207 THE WORK HAS BEEN REVIEWED. 10092 06:02:42,273 --> 06:02:45,844 IN THE CASE OF bioRxiv IT'S 10093 06:02:45,910 --> 06:02:46,044 NOT. 10094 06:02:46,111 --> 06:02:49,147 SO HOW DO WE SORT OF PREVENT 10095 06:02:49,214 --> 06:02:50,315 MISUSE OF THIS CONCEPT? 10096 06:02:50,381 --> 06:02:57,288 IT CAN BE USED, YOU KNOW, YOU 10097 06:02:57,355 --> 06:03:00,859 WANT TO AMPLIFY MY C.V., AND I 10098 06:03:00,925 --> 06:03:03,862 LOOK GOOD FOR AN APPLICATION. 10099 06:03:03,928 --> 06:03:05,864 DO WE KNOW FOR INSTANCE WHAT 10100 06:03:05,930 --> 06:03:08,800 PERCENTAGE OF THE PAPERS THAT 10101 06:03:08,867 --> 06:03:12,303 SHALL SUBMITTED TO bioRxiv 10102 06:03:12,370 --> 06:03:15,807 TRANSITION INTO A PEER-REVIEWED 10103 06:03:15,874 --> 06:03:16,608 JOURNAL. 10104 06:03:16,674 --> 06:03:16,875 >> 70%. 10105 06:03:16,941 --> 06:03:18,109 >> AND 70% THAT DON'T, WHAT IS 10106 06:03:18,176 --> 06:03:22,046 THE MAIN RECENT THEY DON'T MAKE 10107 06:03:22,113 --> 06:03:22,680 IT? 10108 06:03:22,747 --> 06:03:26,117 >> FROM WHAT I'VE READ I THINK 10109 06:03:26,184 --> 06:03:29,487 THERE WAS A PRE-PRINT ON 10110 06:03:29,554 --> 06:03:32,991 bioRxiv ON THE FIRST FIVE 10111 06:03:33,057 --> 06:03:35,093 YEARS OF THE SYSTEM, BY THE 10112 06:03:35,160 --> 06:03:37,262 FOUNDERS, AND THEY FOUND 30% OF 10113 06:03:37,328 --> 06:03:43,168 PRE-PRINTS DON'T TRANSITION TO 10114 06:03:43,234 --> 06:03:44,702 AN ARTICLE, THEIR REASONS WERE 10115 06:03:44,769 --> 06:03:48,506 THAT THEY MAY HAVE BEEN JUST 10116 06:03:48,573 --> 06:03:50,241 SMALLER STUDIES OR THINGS IN 10117 06:03:50,308 --> 06:03:52,243 PROGRESS OR STUDENT MAY HAVE 10118 06:03:52,310 --> 06:03:54,212 GRADUATED AND WORK WAS NOT 10119 06:03:54,279 --> 06:03:56,281 COMPLETED OR THAT THE GROUP JUST 10120 06:03:56,347 --> 06:03:57,415 DECIDED THEY DIDN'T WANT TO 10121 06:03:57,482 --> 06:04:00,185 PURSUE IT TO A JOURNAL BECAUSE A 10122 06:04:00,251 --> 06:04:03,321 LOT OF TIMES JOURNALS WANT A 10123 06:04:03,388 --> 06:04:04,389 STORY. 10124 06:04:04,455 --> 06:04:06,524 SOME SORT OF FRAMEWORK THAT 10125 06:04:06,591 --> 06:04:08,159 DESCRIBES EVERYTHING ABOUT THE 10126 06:04:08,226 --> 06:04:09,194 FINDINGS, AND IF THE RESEARCH 10127 06:04:09,260 --> 06:04:11,329 GROUP IS NOT YET THERE TO PUT IT 10128 06:04:11,396 --> 06:04:13,765 OUT THERE THEY ALSO HAVEN'T DONE 10129 06:04:13,831 --> 06:04:15,366 A VERY LONG-TERM STUDY I THINK 10130 06:04:15,433 --> 06:04:17,769 TO SEE IF THEY EVENTUALLY GET 10131 06:04:17,835 --> 06:04:18,036 PUBLISHED. 10132 06:04:18,102 --> 06:04:19,971 THEY ALSO DIDN'T DO A STUDY TO 10133 06:04:20,038 --> 06:04:24,642 SEE IF IT WAS PUBLISHED AS PART 10134 06:04:24,709 --> 06:04:25,243 OF ANOTHER STUDY. 10135 06:04:25,310 --> 06:04:28,846 A LOT OF THESE THINGS CAN'T BE 10136 06:04:28,913 --> 06:04:30,915 TRACKED IN AN AUTOMATED WAY 10137 06:04:30,982 --> 06:04:31,649 UNFORTUNATELY. 10138 06:04:31,716 --> 06:04:37,121 ALSO A LOT OF THINGS THAT YOU 10139 06:04:37,188 --> 06:04:38,923 THINK ARE VERY KNEW OR MAY NOT 10140 06:04:38,990 --> 06:04:40,858 BE READY TO BE PUBLISHED, AT 10141 06:04:40,925 --> 06:04:42,961 SOME POINT GET PUBLISHED IN 10142 06:04:43,027 --> 06:04:43,728 JOURNALS ANYWAY. 10143 06:04:43,795 --> 06:04:46,664 SO I THINK PRE-PRINTS AT LEAST 10144 06:04:46,731 --> 06:04:49,234 GET MORE EYES FOR PEER REVIEW AS 10145 06:04:49,300 --> 06:04:50,935 OPPOSED TO JOURNAL REVIEW WHERE 10146 06:04:51,002 --> 06:04:53,504 WE EVEN DON'T KNOW WHO IS 10147 06:04:53,571 --> 06:04:55,139 REVIEWING PAPER, TWO TO THREE 10148 06:04:55,206 --> 06:04:59,077 PEOPLE THAT ARE DOING IT AT 10149 06:04:59,143 --> 06:04:59,811 JOURNALS. 10150 06:04:59,877 --> 06:05:00,612 >> THAT'S A GREAT POINT. 10151 06:05:00,678 --> 06:05:03,881 I WANT TO MAKE A PLUG FOR MY 10152 06:05:03,948 --> 06:05:04,782 FAVORITE PRE-PRINT EVER THAT 10153 06:05:04,849 --> 06:05:07,085 WILL NOT APPEAR IN A JOURNAL, 10154 06:05:07,151 --> 06:05:09,287 EPIC TAKEDOWN OF IMPACT FACTOR, 10155 06:05:09,354 --> 06:05:12,490 THE AUTHOR IS HERE. 10156 06:05:12,557 --> 06:05:14,192 YEAH, STRONGLY CAUTION AGAINST 10157 06:05:14,259 --> 06:05:14,959 MAKING ASSUMPTIONS ABOUT THINGS 10158 06:05:15,026 --> 06:05:18,263 THAT DON'T MAKE IT INTO A 10159 06:05:18,329 --> 06:05:19,697 JOURNAL IT COULD BE ANY NUMBER 10160 06:05:19,764 --> 06:05:23,301 OF REASONS, AND IF WE TAKE THIS 10161 06:05:23,368 --> 06:05:25,303 SERIOUSLY THAT IT'S ON US TO 10162 06:05:25,370 --> 06:05:27,438 MAKE SURE THAT'S PUBLIC, IF WE 10163 06:05:27,505 --> 06:05:28,940 GET PUBLIC FUNDING, AND 10164 06:05:29,007 --> 06:05:30,074 SUGGESTIONS FROM EARLIER THAT 10165 06:05:30,141 --> 06:05:31,175 STUDENTS SHOULD HAVE TO, YOU 10166 06:05:31,242 --> 06:05:32,610 KNOW, MAKE THEIR WORK PUBLIC, 10167 06:05:32,677 --> 06:05:35,713 WHETHER OR NOT ALL THE TIME WHAT 10168 06:05:35,780 --> 06:05:36,414 WOULD HAPPEN. 10169 06:05:36,481 --> 06:05:37,749 PEOPLE DO OTHER THINGS AND MAYBE 10170 06:05:37,815 --> 06:05:39,584 NO ONE ELSE IS THERE TO FINISH 10171 06:05:39,651 --> 06:05:39,751 IT. 10172 06:05:39,817 --> 06:05:41,052 THERE'S ALL SORTS OF REASONS 10173 06:05:41,119 --> 06:05:42,287 THAT HAVE NOTHING TO DO WITH THE 10174 06:05:42,353 --> 06:05:44,756 QUALITY OF THE WORK SO THAT'S 10175 06:05:44,822 --> 06:05:46,190 IMPORTANT TO REMEMBER THERE'S A 10176 06:05:46,257 --> 06:05:47,292 LOT OF REASONS CERTAIN THINGS 10177 06:05:47,358 --> 06:05:48,993 DON'T MAKE IT THROUGH AND WE 10178 06:05:49,060 --> 06:05:50,461 DON'T KNOW ULTIMATE OUTCOMES BUT 10179 06:05:50,528 --> 06:05:52,997 I HOPE MORE TAKE THE PATH THAT 10180 06:05:53,064 --> 06:05:53,197 -- 10181 06:05:53,264 --> 06:05:55,900 >> MY POINT IS HOW DO WE KEEP 10182 06:05:55,967 --> 06:05:56,634 OURSELVES HONEST AND USE 10183 06:05:56,701 --> 06:05:58,770 PLATFORMS FOR WHAT THEY ARE AND 10184 06:05:58,836 --> 06:05:59,704 DON'T MISUSE THEM. 10185 06:05:59,771 --> 06:06:01,939 THAT WAS THE POINT. 10186 06:06:02,006 --> 06:06:03,574 >> I THINK YOU SHOULD NOT TAKE 10187 06:06:03,641 --> 06:06:05,209 ANYTHING THAT'S PUBLISHED IN A 10188 06:06:05,276 --> 06:06:06,878 JOURNAL FOR GRANTED ON QUALITY 10189 06:06:06,944 --> 06:06:07,512 EITHER. 10190 06:06:07,578 --> 06:06:09,047 ANYBODY WHO IS LOOKING DOWN THE 10191 06:06:09,113 --> 06:06:10,715 LIST OF PUBLICATIONS AND GOING, 10192 06:06:10,782 --> 06:06:13,117 OH, ONE, TWO, THREE, FOUR, FIVE, 10193 06:06:13,184 --> 06:06:14,752 YEP, WHATEVER, NOT LOOKING AT 10194 06:06:14,819 --> 06:06:16,954 THOSE PUBLICATIONS IS NOT DOING 10195 06:06:17,021 --> 06:06:18,589 THEIR JOB PROPERLY HIRING AND 10196 06:06:18,656 --> 06:06:18,923 PROMOTION. 10197 06:06:18,990 --> 06:06:20,992 >> WE'RE AT TIME FOR THIS 10198 06:06:21,059 --> 06:06:22,026 SESSION. 10199 06:06:22,093 --> 06:06:26,798 THANK YOU SO MUCH. 10200 06:06:26,864 --> 06:06:28,366 WE HAVE ANOTHER SHORT BREAK AND 10201 06:06:28,433 --> 06:06:34,205 WILL MEET BACK AT 4:30. 10202 06:06:34,272 --> 06:06:35,540 >> IT'S 4:30, WE WANT TO GET 10203 06:06:35,606 --> 06:06:36,507 STARTED ON TIME. 10204 06:06:36,574 --> 06:06:41,312 IT'S END OF DAY, I UNDERSTAND IF 10205 06:06:41,379 --> 06:06:43,748 YOU'RE FATIGUED. 10206 06:06:43,815 --> 06:06:49,520 THIS WILL HAVE VERY FEW SLIDES. 10207 06:06:49,587 --> 06:06:51,089 MOSTLY OPEN CONVERSATION, EACH 10208 06:06:51,155 --> 06:06:52,590 PERSON WILL TALK ABOUT THEIR 10209 06:06:52,657 --> 06:06:57,195 EXPERTISE AREA AND WE'LL HAVE A 10210 06:06:57,261 --> 06:06:58,963 VERY OPEN DISCUSSION LIKE WE'VE 10211 06:06:59,030 --> 06:07:00,965 BEEN HAVING ALL DAY. 10212 06:07:01,032 --> 06:07:02,233 THIS AFTERNOON, THIS LAST ONE, 10213 06:07:02,300 --> 06:07:05,636 WE'RE GOING TO TALK ABOUT 10214 06:07:05,703 --> 06:07:07,972 PROBLEMS TODAY AND INSTITUTIONAL 10215 06:07:08,039 --> 06:07:10,141 EFFORTS TO INCENTIVIZE REPORTING 10216 06:07:10,208 --> 06:07:11,776 OF NULL STUDIES, TALKING MORE 10217 06:07:11,843 --> 06:07:12,844 ABOUT INSTITUTIONS THIS 10218 06:07:12,910 --> 06:07:14,912 AFTERNOON, THE PROGRESS, THE 10219 06:07:14,979 --> 06:07:16,581 BARRIERS, I'M THE MODERATOR, 10220 06:07:16,647 --> 06:07:20,651 GINA POE, PROFESSOR AT UCLA, 10221 06:07:20,718 --> 06:07:21,619 NEUROSCIENTIST AND JUST BECAME 10222 06:07:21,686 --> 06:07:28,326 TWO WEEKS AGO DIRECTOR OF BRAIN 10223 06:07:28,393 --> 06:07:28,760 RESEARCH INSTITUTE. 10224 06:07:28,826 --> 06:07:30,128 THANK YOU. 10225 06:07:30,194 --> 06:07:34,565 I WANT TO BRING TO THE JOB NEW 10226 06:07:34,632 --> 06:07:37,535 STANDARDS OF RIGOR AND 10227 06:07:37,602 --> 06:07:38,603 REPRODUCIBILITY AND OPEN 10228 06:07:38,669 --> 06:07:39,504 PUBLICATIONS. 10229 06:07:39,570 --> 06:07:40,438 I'M REALLY GOING TO TAKE A LOT 10230 06:07:40,505 --> 06:07:42,006 OF THINGS I'VE LEARNED TOGETHER 10231 06:07:42,073 --> 06:07:44,175 AND I HOPE TO LEARN TOMORROW 10232 06:07:44,242 --> 06:07:46,778 BACK TO UCLA AND VERY EXCITED 10233 06:07:46,844 --> 06:07:47,111 ABOUT THIS. 10234 06:07:47,178 --> 06:07:48,212 ALL RIGHT. 10235 06:07:48,279 --> 06:07:52,350 SO OUR PANELISTS TODAY ARE 10236 06:07:52,417 --> 06:07:55,787 ROBERT NOBLES, AT EMORY 10237 06:07:55,853 --> 06:07:56,087 UNIVERSITY. 10238 06:07:56,154 --> 06:08:03,561 HE WAS AT TIMOTHY -- TENNESSEE 10239 06:08:03,628 --> 06:08:04,896 STUDIES AT RISK YOUTH. 10240 06:08:04,962 --> 06:08:08,800 AND HE'S A VICE PRESIDENT FOR 10241 06:08:08,866 --> 06:08:11,469 RESEARCH AT EMORY, A TOP 20 10242 06:08:11,536 --> 06:08:19,644 UNIVERSITY IN THE NATION, HE 10243 06:08:19,710 --> 06:08:22,380 WILL TALKING ABOUT INSTITUTIONAL 10244 06:08:22,447 --> 06:08:22,847 INCENTIVE. 10245 06:08:22,914 --> 06:08:24,515 RACHEL MILES, VIRGINIA TECH FOR 10246 06:08:24,582 --> 06:08:26,250 SIX YEARS, SHE IS GOING TO TALK 10247 06:08:26,317 --> 06:08:31,489 ABOUT AND HAS BEEN STUDYING 10248 06:08:31,556 --> 06:08:36,894 RESEARCH IMPACT AND ASSESSMENTS, 10249 06:08:36,961 --> 06:08:39,864 SHE'S A LIBRARY SCIENCES 10250 06:08:39,931 --> 06:08:42,166 PERSON, RESEARCHING SCHOLARLY 10251 06:08:42,233 --> 06:08:42,900 COMMUNICATIONS, DR. RENE 10252 06:08:42,967 --> 06:08:44,435 BERNARD, NOT WITH US TODAY 10253 06:08:44,502 --> 06:08:45,970 BECAUSE THEY CANCELED HIS FLIGHT 10254 06:08:46,037 --> 06:08:49,474 FROM EUROPE, SO HE'S ON HIS WAY, 10255 06:08:49,540 --> 06:08:51,676 IN FLIGHT RIGHT NOW, HE'S 10256 06:08:51,742 --> 06:08:57,882 PROBABLY -- HE LANDED AT 4:30, 10257 06:08:57,949 --> 06:08:59,083 HE'S RUSHING HERE AND PROBABLY 10258 06:08:59,150 --> 06:09:00,952 WON'T MAKE IT BY THE END OF THE 10259 06:09:01,018 --> 06:09:01,219 MESSAGE. 10260 06:09:01,285 --> 06:09:07,725 WE HAVE A RECORDED MESSAGE 10261 06:09:07,792 --> 06:09:16,868 BECAUSE PLANNED HEAD, AT A 10262 06:09:16,934 --> 06:09:19,904 UNIVERSITY IN BERLIN, AND 10263 06:09:19,971 --> 06:09:20,905 CHARITE, TALKING NOT LIVE, BUT 10264 06:09:20,972 --> 06:09:25,810 YOU CAN ALSO TALK TO HIM OVER 10265 06:09:25,877 --> 06:09:31,282 DINNER OR TOMORROW. 10266 06:09:31,349 --> 06:09:41,859 WE HAVE ERIN MCKIERNAN, FROM 10267 06:09:44,795 --> 06:09:46,864 NATIONAL ACADEMY UNIVERSITY, 10268 06:09:46,931 --> 06:09:49,300 BIOMEDICAL PHYSICS, PROMOTION 10269 06:09:49,367 --> 06:09:50,601 AND TENURED DOCUMENTS IN THE 10270 06:09:50,668 --> 06:09:54,639 U.S. AND CANADA, WHAT'S IN THESE 10271 06:09:54,705 --> 06:09:55,973 DOCUMENTS. 10272 06:09:56,040 --> 06:09:57,808 WE HAVE SRIRAMA RAO, VIRGINIA 10273 06:09:57,875 --> 06:09:58,476 COMMONWEALTH UNIVERSITY, A 10274 06:09:58,543 --> 06:10:04,115 LEADER AT MULTIPLE INSTITUTIONS, 10275 06:10:04,181 --> 06:10:05,917 HIGH LEVEL PERSPECTIVE ABOUT 10276 06:10:05,983 --> 06:10:07,718 INSTITUTIONAL BARRIERS, HONING 10277 06:10:07,785 --> 06:10:09,186 DOWN ON FACULTY FEEDBACK AND 10278 06:10:09,253 --> 06:10:14,892 REALITY OF THE PUBLICATION 10279 06:10:14,959 --> 06:10:15,526 WORLD. 10280 06:10:15,593 --> 06:10:17,929 EACH SPEAKER HAS FIVE MINUTES, 10281 06:10:17,995 --> 06:10:19,797 ONE HAS SLIDES, ONE WILL BE A 10282 06:10:19,864 --> 06:10:21,799 RECORDING, AND WE'LL OPEN IT UP 10283 06:10:21,866 --> 06:10:22,733 FOR QUESTIONS. 10284 06:10:22,800 --> 06:10:25,536 THANK YOU VERY MUCH. 10285 06:10:34,845 --> 06:10:37,381 >> I'LL MAKE SURE I STAY UNDER 10286 06:10:37,448 --> 06:10:39,584 FIVE MINUTES, WITH MY PHONE 10287 06:10:39,650 --> 06:10:40,151 WATCH. 10288 06:10:40,217 --> 06:10:47,825 ROBERT NOBLE, ENVIRONMENT FOR 10289 06:10:47,892 --> 06:10:48,659 RESEARCH, AGAIN AT EMORY. 10290 06:10:48,726 --> 06:10:51,963 I'LL START WITH A QUICK STORY. 10291 06:10:52,029 --> 06:10:54,732 I'VE BEEN LISTENING TO YOU ALL 10292 06:10:54,799 --> 06:10:55,666 MORNING, WHICH IS -- REALLY 10293 06:10:55,733 --> 06:10:59,303 SITTING IN THE BACK TRYING TO BE 10294 06:10:59,370 --> 06:11:00,071 QUIET. 10295 06:11:00,137 --> 06:11:01,038 WHICH ISN'T REALLY EASY. 10296 06:11:01,105 --> 06:11:05,142 BUT LET ME TELL YOU MY MAIN 10297 06:11:05,209 --> 06:11:07,211 TAKEAWAY FROM TODAY. 10298 06:11:07,278 --> 06:11:09,280 MANY TIMES I DON'T STAND IN 10299 06:11:09,347 --> 06:11:11,549 FRONT OF AN AUDIENCE THAT WANT 10300 06:11:11,616 --> 06:11:13,985 TO DO THE RIGHT THING. 10301 06:11:14,051 --> 06:11:17,254 ON THE OTHER SIDE, I INVESTIGATE 10302 06:11:17,321 --> 06:11:19,323 AND I MEDIATE PROBLEMS THAT 10303 06:11:19,390 --> 06:11:20,358 HAPPEN AT THE INSTITUTIONN MY 10304 06:11:20,424 --> 06:11:22,393 ROLE AS VICE PRESIDENT FOR 10305 06:11:22,460 --> 06:11:23,327 RESEARCH. 10306 06:11:23,394 --> 06:11:25,763 I PROMOTE AND REALLY SERVE AS 10307 06:11:25,830 --> 06:11:28,532 CONSCIOUSNESS OF WHAT IS RIGHT, 10308 06:11:28,599 --> 06:11:33,204 BUT MY AUDIENCES TYPICALLY DON'T 10309 06:11:33,270 --> 06:11:37,608 KNOW OR WOULD RATHER GO REALLY 10310 06:11:37,675 --> 06:11:39,110 FAST. 10311 06:11:39,176 --> 06:11:41,012 SO THIS PROSPECT OF -- . 10312 06:11:41,078 --> 06:11:42,613 >> OR CHOOSE NOT TO KNOW. 10313 06:11:42,680 --> 06:11:45,016 >> OR CHOOSE NOT TO KNOW, VERY 10314 06:11:45,082 --> 06:11:46,350 TRUE. 10315 06:11:46,417 --> 06:11:47,251 VERY TRUE. 10316 06:11:47,318 --> 06:11:51,188 SO, THIS PROSPECT OF PUBLICATION 10317 06:11:51,255 --> 06:11:52,957 BIAS, IT MADE ME REALLY THINK 10318 06:11:53,024 --> 06:11:57,194 AND ONE OF THE RULES OF THIS 10319 06:11:57,261 --> 06:11:59,597 SESSION WHICH IS VERY DIFFICULT, 10320 06:11:59,664 --> 06:12:01,132 DON'T JUMP INTO SOLUTIONS 10321 06:12:01,198 --> 06:12:02,500 BECAUSE SOLUTIONS ARE FOR 10322 06:12:02,566 --> 06:12:03,734 TOMORROW, ONLY TALK ABOUT THE 10323 06:12:03,801 --> 06:12:04,669 PROBLEMS. 10324 06:12:04,735 --> 06:12:07,605 THAT'S NOT VERY TYPICAL. 10325 06:12:07,672 --> 06:12:08,639 SO LET ME HIGHLIGHT, I THINK 10326 06:12:08,706 --> 06:12:10,341 WHAT I WANT TO DO FOR YOU IN THE 10327 06:12:10,408 --> 06:12:13,377 FIVE MINUTES I HAVE IS REALLY 10328 06:12:13,444 --> 06:12:14,045 HELP YOU UNDERSTAND THE 10329 06:12:14,111 --> 06:12:16,213 COMPLEXITIES OF THE INSTITUTION 10330 06:12:16,280 --> 06:12:18,549 AND HOW WE INCENTIVIZE OUR 10331 06:12:18,616 --> 06:12:18,783 FACULTY. 10332 06:12:18,849 --> 06:12:20,718 BUT THERE'S SOMETHING THAT I 10333 06:12:20,785 --> 06:12:22,153 HEARD EARLIER, EVEN WHEN 10334 06:12:22,219 --> 06:12:24,488 LAWRENCE WAS GIVING THE KEYNOTE, 10335 06:12:24,555 --> 06:12:30,261 AND HE ASKED THE QUESTION, I 10336 06:12:30,327 --> 06:12:32,563 THOUGHT IT WAS INTERESTING. 10337 06:12:32,630 --> 06:12:36,333 HE ASKED DOES PUBLISHING NULL 10338 06:12:36,400 --> 06:12:38,069 RESULTS IN THE NEW ENGLAND 10339 06:12:38,135 --> 06:12:42,406 JOURNAL OF MEDICINE COUNT FOR 10340 06:12:42,473 --> 06:12:44,875 TENURE PROMOTION? 10341 06:12:44,942 --> 06:12:47,645 I TOLD MYSELF TO BE QUIET. 10342 06:12:47,712 --> 06:12:48,846 THE ANSWER IS YES. 10343 06:12:48,913 --> 06:12:51,048 TO MAKE SURE WE'RE CLEAR, WHAT 10344 06:12:51,115 --> 06:12:53,984 WE DO AT INSTITUTIONS IS A 10345 06:12:54,051 --> 06:12:56,654 NUMBER OF THINGS TO INCENTIVIZE 10346 06:12:56,721 --> 06:12:58,189 THE CONDUCT OF RESEARCH, THE 10347 06:12:58,255 --> 06:12:59,190 SEEKING OF RESEARCH. 10348 06:12:59,256 --> 06:13:01,959 I'LL WALK YOU THROUGH THAT. 10349 06:13:02,026 --> 06:13:03,494 WHAT WE MEASURE, WHAT WE MEASURE 10350 06:13:03,561 --> 06:13:05,229 IN TERMS OF FACULTY 10351 06:13:05,296 --> 06:13:07,131 PRODUCTIVITY, ESPECIALLY FROM 10352 06:13:07,198 --> 06:13:09,300 RESEARCH OFFICE, IS TWO THINGS. 10353 06:13:09,366 --> 06:13:12,770 NUMBER OF AWARDS, NUMBER OF 10354 06:13:12,837 --> 06:13:13,404 PUBLICATIONS. 10355 06:13:13,471 --> 06:13:16,073 IF IT'S PUBLISHED IN NEW ENGLAND 10356 06:13:16,140 --> 06:13:19,410 JOURNAL, IT COUNTS AS A 10357 06:13:19,477 --> 06:13:20,144 PUBLICATION, HIGH-IMPACT 10358 06:13:20,211 --> 06:13:21,912 JOURNAL, PERIOD. 10359 06:13:21,979 --> 06:13:24,081 AND WHEN IT BECOMES REVIEWED WE 10360 06:13:24,148 --> 06:13:28,786 DON'T DIG -- TRUST ME, I DO NOT 10361 06:13:28,853 --> 06:13:31,188 READ EVERY ARTICLE THEY CITE. 10362 06:13:31,255 --> 06:13:34,525 WE CONFIRM IT WAS PUBLISH AND 10363 06:13:34,592 --> 06:13:34,959 CITED. 10364 06:13:35,025 --> 06:13:38,729 AND SO WE DON'T DIG INTO IF IT 10365 06:13:38,796 --> 06:13:42,566 WAS NULL RESULTS, POSITIVE 10366 06:13:42,633 --> 06:13:44,435 RESULTS, NEGATIVE RESULTS. 10367 06:13:44,502 --> 06:13:49,673 WE TRUST THIS PACT WITH THE 10368 06:13:49,740 --> 06:13:51,008 COMMUNITY AND PROCESS OF PEER 10369 06:13:51,075 --> 06:13:51,742 REVIEW. 10370 06:13:51,809 --> 06:13:54,478 I THINK ABOUT PUBLICATION BIAS, 10371 06:13:54,545 --> 06:13:56,614 AS DEFINED, BECAUSE I THINK WHEN 10372 06:13:56,680 --> 06:13:57,481 I STARTED TEACHING EPIDEMIOLOGY 10373 06:13:57,548 --> 06:13:59,383 A LONG TIME AGO TO GRADUATE 10374 06:13:59,450 --> 06:14:04,054 STUDENTS, I ALWAYS START WITH 10375 06:14:04,121 --> 06:14:06,757 THE DEFINITION, WHAT WE'VE BEEN 10376 06:14:06,824 --> 06:14:09,527 TALKING ABOUT IS THIS SELECTIVE 10377 06:14:09,593 --> 06:14:10,394 PUBLICATION OF RESEARCH STUDIES 10378 06:14:10,461 --> 06:14:14,198 BASED ON THE RESULTS FOR WHICH 10379 06:14:14,265 --> 06:14:15,332 WE HAVE FOUND, POSITIVE OR 10380 06:14:15,399 --> 06:14:15,699 NEGATIVE. 10381 06:14:15,766 --> 06:14:16,200 IT'S FAIR. 10382 06:14:16,267 --> 06:14:19,403 ONE OF THE THINGS THAT HAPPENED 10383 06:14:19,470 --> 06:14:24,074 TODAY IN SOME OF THE DISCUSSIONS 10384 06:14:24,141 --> 06:14:29,413 WAS THAT WE MADE THIS THIS 10385 06:14:29,480 --> 06:14:35,019 REALLY SYNONYMOUS JUMP THAT 10386 06:14:35,085 --> 06:14:37,755 PUBLICATION BIAS EQUALS 10387 06:14:37,822 --> 06:14:39,156 RESEARCHER COLLECTIVE REPORTING. 10388 06:14:39,223 --> 06:14:42,359 AND I NEED TO PUSH BACK, I THINK 10389 06:14:42,426 --> 06:14:44,228 WE'VE PUT A LOT OF STUCK THAT 10390 06:14:44,295 --> 06:14:47,731 IT'S UP TO THE RESEARCHER AND 10391 06:14:47,798 --> 06:14:49,266 INVESTIGATOR, ON IF THE 10392 06:14:49,333 --> 06:14:51,135 COMMUNITY CARES ABOUT THE 10393 06:14:51,202 --> 06:14:55,072 RESULTS FOR WHICH THEY FIND. 10394 06:14:55,139 --> 06:14:57,842 I DON'T REALLY THINK, IT'S LESS 10395 06:14:57,908 --> 06:14:58,676 CONSCIOUS THAN THAT. 10396 06:14:58,742 --> 06:15:01,345 I THINK HOW WE'VE BEEN 10397 06:15:01,412 --> 06:15:02,446 CONDITIONED OVER TIME DICTATES 10398 06:15:02,513 --> 06:15:06,283 WHAT WE END UP SHARING WITH THE 10399 06:15:06,350 --> 06:15:12,056 WORLD, NOT THER ON THE -- NOT 10400 06:15:12,122 --> 06:15:13,123 THE OTHER WAY AROUND. 10401 06:15:13,190 --> 06:15:16,327 I WONDER IF I SHOULD HIDE THESE 10402 06:15:16,393 --> 06:15:17,528 RESULTS IF THERE WAS INCENTIVE 10403 06:15:17,595 --> 06:15:20,431 TO PROVIDE RESULTS IT WOULD BE 10404 06:15:20,497 --> 06:15:23,100 PROVIDED, GUARANTEED. 10405 06:15:23,167 --> 06:15:25,302 AND SO I THINK -- OH, YEAH, I'M 10406 06:15:25,369 --> 06:15:26,937 GOING OVER MY ONE MINUTE, I'M 10407 06:15:27,004 --> 06:15:30,241 GOING TO LET YOU KNOW. 10408 06:15:30,307 --> 06:15:35,079 COUNT THIS AS SEVEN OR EIGHT 10409 06:15:35,145 --> 06:15:36,914 MINUTES, SOMEBODY MISSED THEIR 10410 06:15:36,981 --> 06:15:37,147 FLIGHT. 10411 06:15:37,214 --> 06:15:38,782 REALLY QUICK. 10412 06:15:38,849 --> 06:15:44,021 WHAT DO ACADEMIC INSTITUTIONS DO 10413 06:15:44,088 --> 06:15:45,456 TO INCENTIVIZE RESEARCH BY 10414 06:15:45,522 --> 06:15:47,424 GIVING FACULTY MEMBERS TIME. 10415 06:15:47,491 --> 06:15:51,362 AND DO UNDERSTAND THE WAY WE 10416 06:15:51,428 --> 06:15:53,364 CALCULATE TEACHING IN CLASS IS 10417 06:15:53,430 --> 06:15:55,399 25% OF THE TIME, AT SOME 10418 06:15:55,466 --> 06:15:59,536 INSTITUTIONS THE FACULTY LOAD IS 10419 06:15:59,603 --> 06:15:59,770 TWO-TWO. 10420 06:15:59,837 --> 06:16:01,105 OR THREE-THREE. 10421 06:16:01,171 --> 06:16:02,273 AND THERE'S SOME INSTITUTIONS 10422 06:16:02,339 --> 06:16:05,042 OUT OF THE 4,000 INSTITUTIONS 10423 06:16:05,109 --> 06:16:08,078 IN THE U.S. THAT HAVE A 10424 06:16:08,145 --> 06:16:10,814 FOUR-FOUR TEACHING LOAD WHICH 10425 06:16:10,881 --> 06:16:11,949 THEY DON'T INCENTIVIZE THROUGH 10426 06:16:12,016 --> 06:16:13,150 TIME, THE OPPORTUNITY FOR 10427 06:16:13,217 --> 06:16:15,019 FACULTY TO CARRY OUT RESEARCH. 10428 06:16:15,085 --> 06:16:19,123 THEY HAVE TO USE EXTRA TIME FOR 10429 06:16:19,189 --> 06:16:24,194 THAT. 10430 06:16:24,261 --> 06:16:30,067 WE IS INCENTIVIZE BY PROVIDES 10431 06:16:30,134 --> 06:16:31,201 INFRASTRUCTURE, CARRYING OUT 10432 06:16:31,268 --> 06:16:33,370 RESEARCH, BUT THE PROSPECT OF 10433 06:16:33,437 --> 06:16:35,873 DOING THAT WHEN WE GO A STEP 10434 06:16:35,940 --> 06:16:38,509 FURTHER, WE INCENTIVIZE THE 10435 06:16:38,575 --> 06:16:40,678 OPPORTUNITY TO SEEK FUNDING. 10436 06:16:40,744 --> 06:16:41,378 WHICH MEANS WHAT? 10437 06:16:41,445 --> 06:16:44,315 IT MEANS THAT YOU HAVE TO PITCH 10438 06:16:44,381 --> 06:16:47,685 AN IDEA AND SOMEBODY CARES 10439 06:16:47,751 --> 06:16:49,353 ENOUGH TO GIVE YOU RESOURCES SO 10440 06:16:49,420 --> 06:16:55,492 YOU CAN THEN CARRY OUT YOUR 10441 06:16:55,559 --> 06:16:57,594 IDEAS, INNOVATION AND DRIVE THE 10442 06:16:57,661 --> 06:16:59,096 CREATION OF NEW IDEAS. 10443 06:16:59,163 --> 06:17:03,701 SO WE PROVIDE SEED FUNDING. 10444 06:17:03,767 --> 06:17:05,102 WE OFTENTIMES SUPPORT PILOT 10445 06:17:05,169 --> 06:17:07,071 PROJECTS, WE PROVIDE FUNDING FOR 10446 06:17:07,137 --> 06:17:10,941 THAT WITH PROSPECT THEY WILL 10447 06:17:11,008 --> 06:17:13,877 SEEK EXTERNAL FUNDING LATER. 10448 06:17:13,944 --> 06:17:15,212 WE FUND DATABASES WHERE THEY CAN 10449 06:17:15,279 --> 06:17:17,514 USE TO SEEK FUNDING AND FUND 10450 06:17:17,581 --> 06:17:19,817 RESEARCH DEVELOPMENT OFFICERS AT 10451 06:17:19,883 --> 06:17:21,218 OUR INSTITUTION TO HELP NEW 10452 06:17:21,285 --> 06:17:26,357 SOURCE, WHERE TO GO FISHING IN 10453 06:17:26,423 --> 06:17:28,092 THE POND. 10454 06:17:28,158 --> 06:17:29,093 WE INCENTIVIZE ETHICAL CONDUCT 10455 06:17:29,159 --> 06:17:33,464 OF RESEARCH. 10456 06:17:33,530 --> 06:17:35,599 WE HAVE OVERSIGHT COMMITTEES 10457 06:17:35,666 --> 06:17:38,635 THAT I SUPPORT, YOU CAN GO DOWN 10458 06:17:38,702 --> 06:17:41,805 THE LIST, BUT WE INCENTIVIZE 10459 06:17:41,872 --> 06:17:42,539 THROUGH COMPLIANCE. 10460 06:17:42,606 --> 06:17:45,075 PEOPLE ASK IF IT'S CARROTS OR 10461 06:17:45,142 --> 06:17:45,776 STICKS. 10462 06:17:45,843 --> 06:17:47,144 COMPLIANCE IS AN INCENTIVE 10463 06:17:47,211 --> 06:17:49,446 BECAUSE NON-COMPLIANCE IS A 10464 06:17:49,513 --> 06:17:50,014 DISINCENTIVE. 10465 06:17:50,080 --> 06:17:53,517 SO WE HELP OUR FACULTY BY 10466 06:17:53,584 --> 06:17:54,885 HELPING THEM UNDERSTAND HOW TO 10467 06:17:54,952 --> 06:17:56,053 FIND FUNDING, WHY TO DO 10468 06:17:56,120 --> 06:18:00,491 RESEARCH, HOW TO DO IT 10469 06:18:00,557 --> 06:18:00,924 APPROPRIATELY. 10470 06:18:00,991 --> 06:18:03,560 WE INCENTIVIZE THE PUBLICATION 10471 06:18:03,627 --> 06:18:06,530 OF RESULTS. 10472 06:18:06,597 --> 06:18:08,799 AND THE PUBLICATION OF RESULTS 10473 06:18:08,866 --> 06:18:10,834 HAPPENS FOR A COUPLE OF REASONS. 10474 06:18:10,901 --> 06:18:15,105 ONE IS THAT THERE'S A MANDATE TO 10475 06:18:15,172 --> 06:18:16,974 SHARE RESULTS, FOR SOME OF OUR 10476 06:18:17,041 --> 06:18:18,909 SPONSORS, AND FOR OTHERS IT'S 10477 06:18:18,976 --> 06:18:19,309 NOT. 10478 06:18:19,376 --> 06:18:20,511 IT'S ENCOURAGED. 10479 06:18:20,577 --> 06:18:23,914 BUT THERE'S AN EXPECTATION. 10480 06:18:23,981 --> 06:18:26,817 IMPLICIT AND EXPLICIT. 10481 06:18:26,884 --> 06:18:27,818 THAT RESULTS ARE SHARED. 10482 06:18:27,885 --> 06:18:29,186 I THINK THE CHALLENGE IS RESULTS 10483 06:18:29,253 --> 06:18:31,922 YOU SHARE HAVE TO BE ACCEPTED BY 10484 06:18:31,989 --> 06:18:32,623 YOUR PEERS. 10485 06:18:32,689 --> 06:18:36,093 AND THEN THAT'S A DIFFERENT -- 10486 06:18:36,160 --> 06:18:37,594 THAT'S THE INCENTIVE STRUCTURE 10487 06:18:37,661 --> 06:18:39,163 FOR THE UNIVERSITY IS THERE BUT 10488 06:18:39,229 --> 06:18:41,965 BARRIERS WHICH WE WERE ASKED 10489 06:18:42,032 --> 06:18:43,634 ABOUT, THE BARRIERS REALLY FLOW 10490 06:18:43,700 --> 06:18:45,069 INTO THIS OTHER COMPONENT OF 10491 06:18:45,135 --> 06:18:46,703 HAVING A RELATIONSHIP WITH THE 10492 06:18:46,770 --> 06:18:49,540 COMMUNITY FOR WHICH THAT WE ARE 10493 06:18:49,606 --> 06:18:51,175 SCIENTISTS WITHIN. 10494 06:18:51,241 --> 06:18:54,211 SO THEN WE HAVE PEER INCENTIVES. 10495 06:18:54,278 --> 06:18:56,847 DO THEY CARE ABOUT WHAT YOU'RE 10496 06:18:56,914 --> 06:18:57,548 WRITING? 10497 06:18:57,614 --> 06:19:00,484 IF THEY DON'T, WHOEVER REVIEWS 10498 06:19:00,551 --> 06:19:01,919 IT, THEY WON'T PUBLISH IT. 10499 06:19:01,985 --> 06:19:04,021 THEY WON'T TALK BIT. 10500 06:19:04,088 --> 06:19:06,723 THEY WON'T CITE IT. 10501 06:19:06,790 --> 06:19:08,625 AND SO WHAT WE'VE BEEN 10502 06:19:08,692 --> 06:19:14,364 CONDITIONED TO DO IS FIND VALUE 10503 06:19:14,431 --> 06:19:19,837 IN POSITIVE RESULTS. 10504 06:19:19,903 --> 06:19:20,737 COLLECTIVELY. 10505 06:19:20,804 --> 06:19:21,738 SPONSOR INCENTIVES, POSITIVE 10506 06:19:21,805 --> 06:19:25,142 RESULTS LEAD TO MORE GRANT 10507 06:19:25,209 --> 06:19:25,409 FUNDING. 10508 06:19:25,476 --> 06:19:26,510 NULL RESULTS DON'T. 10509 06:19:26,577 --> 06:19:29,246 WE'VE BEEN CONDITIONED TO FIND 10510 06:19:29,313 --> 06:19:34,618 POSITIVE RESULTS. 10511 06:19:34,685 --> 06:19:35,586 JOURNAL INCENTIVES. 10512 06:19:35,652 --> 06:19:37,187 WHEN THEY REVIEW AND SEE IT IT'S 10513 06:19:37,254 --> 06:19:38,388 THE IMPACT SAME THING. 10514 06:19:38,455 --> 06:19:40,124 TELL US YOUR STORY OF HOW YOU'RE 10515 06:19:40,190 --> 06:19:41,625 MARKETING THE INFORMATION AND 10516 06:19:41,692 --> 06:19:43,260 WHY WE SHOULD CARE. 10517 06:19:43,327 --> 06:19:45,229 ONCE YOU MARKET IT, THEN WE'LL 10518 06:19:45,295 --> 06:19:46,330 PUBLISH IT. 10519 06:19:46,396 --> 06:19:51,768 BECAUSE OUR READERS WILL CARE. 10520 06:19:51,835 --> 06:19:53,437 SCIENTISTS ARE NOT TRAINED HOW 10521 06:19:53,504 --> 06:19:57,274 TO MARKET THEIR RESEARCH OUTPUT. 10522 06:19:57,341 --> 06:19:59,343 SO, SOME DO WELL. 10523 06:19:59,409 --> 06:20:01,211 SOME DO NOT. 10524 06:20:01,278 --> 06:20:06,450 AND SO THIS PACT WITH COMMUNITY 10525 06:20:06,517 --> 06:20:09,086 SETS THE STAGE OF OUR FACULTY, 10526 06:20:09,153 --> 06:20:10,287 OUR RESEARCHERS, TRYING TO 10527 06:20:10,354 --> 06:20:15,559 FIGURE OUT HOW TO NAVIGATE AND 10528 06:20:15,626 --> 06:20:17,027 BE SUCCESSFUL, GO THROUGH TENURE 10529 06:20:17,094 --> 06:20:17,427 PROCESS. 10530 06:20:17,494 --> 06:20:18,529 OVER TIME WHEN GRADUATE STUDENTS 10531 06:20:18,595 --> 06:20:22,966 ASK THE QUESTION HOW DO I GET MY 10532 06:20:23,033 --> 06:20:25,402 FACULTY AND ADVISORS TO SUPPORT 10533 06:20:25,469 --> 06:20:26,737 ME PUBLISHING NULL RESULTS, WE 10534 06:20:26,803 --> 06:20:31,175 HAVE TO CHANGE AN ENTIRE 10535 06:20:31,241 --> 06:20:32,409 ENVIRONMENT BECAUSE THE SHORTCUT 10536 06:20:32,476 --> 06:20:35,245 WHEN WE'RE BUSY AND WE'RE MOVING 10537 06:20:35,312 --> 06:20:36,880 FAST IS DON'T FOCUS ON THAT, 10538 06:20:36,947 --> 06:20:39,283 FOCUS ON SOMETHING THAT YOU'RE 10539 06:20:39,349 --> 06:20:40,851 GOING TO GET NOTORIETY AND 10540 06:20:40,918 --> 06:20:41,251 VISIBILITY. 10541 06:20:41,318 --> 06:20:43,120 BUT THAT'S THE WRONG ANSWER 10542 06:20:43,187 --> 06:20:44,421 TRUTHFULLY. 10543 06:20:44,488 --> 06:20:45,522 IT'S THE WRONG ANSWER. 10544 06:20:45,589 --> 06:20:49,059 BUT OVER TIME WE'VE BEEN 10545 06:20:49,126 --> 06:20:50,394 CONDITIONED, THE ENVIRONMENT HAS 10546 06:20:50,460 --> 06:20:52,629 CONDITIONED US, ONLY LOOK FOR 10547 06:20:52,696 --> 06:20:53,297 POSITIVE THINGS. 10548 06:20:53,363 --> 06:20:59,269 SO WHEN WE TALK ABOUT 10549 06:20:59,336 --> 06:21:01,405 PUBLICATION BIAS, I THINK IT 10550 06:21:01,471 --> 06:21:07,444 GOES INTO THE NOTION OF THE WAY 10551 06:21:07,511 --> 06:21:10,948 THAT WE SEEK INFORMATION. 10552 06:21:11,014 --> 06:21:13,450 WE REALLY ONLY WANT TO BE TOLD 10553 06:21:13,517 --> 06:21:16,353 THAT WHAT YOU SOUGHT IS WHAT YOU 10554 06:21:16,420 --> 06:21:16,853 FOUND. 10555 06:21:16,920 --> 06:21:18,755 IN SOMEBODY TELLS YOU 10556 06:21:18,822 --> 06:21:19,523 ALTERNATIVE, YOU'RE NOT 10557 06:21:19,590 --> 06:21:20,123 NECESSARILY HAPPY. 10558 06:21:20,190 --> 06:21:23,327 AND I CAN ONLY THINK OF ONE 10559 06:21:23,393 --> 06:21:27,064 REASON, THIS IS THE CYNICAL 10560 06:21:27,130 --> 06:21:31,835 COMPONENT, THAT INDIVIDUALS 10561 06:21:31,902 --> 06:21:32,703 WOULD INTENTIONALLY PUSH 10562 06:21:32,769 --> 06:21:34,104 THEMSELVES TO FIND NULL RESULTS 10563 06:21:34,171 --> 06:21:35,505 OF THINGS PUBLISHED IN A 10564 06:21:35,572 --> 06:21:36,273 POSITIVE FASHION. 10565 06:21:36,340 --> 06:21:39,343 AND THAT IS IF THE FACULTY 10566 06:21:39,409 --> 06:21:40,444 MEMBER OR RESEARCHER SETS 10567 06:21:40,510 --> 06:21:42,946 THEMSELVES OUT TO BE A MYTH 10568 06:21:43,013 --> 06:21:46,583 BUSTER THAT ALL THESE PEOPLE ARE 10569 06:21:46,650 --> 06:21:48,619 CITING X, Y,Z, AND I'M GOING TO 10570 06:21:48,685 --> 06:21:49,853 PROVE THEM WRONG BECAUSE I CAN'T 10571 06:21:49,920 --> 06:21:51,488 DO THE SAME THING. 10572 06:21:51,555 --> 06:21:53,290 AND AS THE FORMER POSTDOC WHO 10573 06:21:53,357 --> 06:21:57,094 WENT OVER TO U.K., CAME FROM 10574 06:21:57,160 --> 06:21:58,362 STANFORD, AS HE STATED, IT WOULD 10575 06:21:58,428 --> 06:22:00,163 TAKE A LOT OF COURAGE TO STAND 10576 06:22:00,230 --> 06:22:04,334 UP TO GIANTS IN YOUR FIELD WHEN 10577 06:22:04,401 --> 06:22:06,136 YOU'RE JUST STARTING OUT. 10578 06:22:06,203 --> 06:22:08,238 SO THEN THE ENVIRONMENT THEN 10579 06:22:08,305 --> 06:22:10,641 AGAIN CONDITIONS US TO BE QUIET 10580 06:22:10,707 --> 06:22:14,144 ABOUT THE THINGS FOR WHICH THAT 10581 06:22:14,211 --> 06:22:15,445 WE CAN'T PUSH DIALOGUE ON 10582 06:22:15,512 --> 06:22:18,282 BECAUSE IT WOULD NOT BE WELCOMED 10583 06:22:18,348 --> 06:22:22,352 IN OUR PEER GROUPS, AND THE 10584 06:22:22,419 --> 06:22:32,629 COMMUNITY. 10585 06:22:45,409 --> 06:22:48,178 >> THANK YOU. 10586 06:22:48,245 --> 06:22:50,514 I'M RACHEL, I'M THE -- WORK IN 10587 06:22:50,580 --> 06:22:52,249 UNIVERSITY LIBRARIES AT VIRGINIA 10588 06:22:52,316 --> 06:22:53,317 TECH, RESEARCH IMPACT 10589 06:22:53,383 --> 06:22:53,617 COORDINATOR. 10590 06:22:53,684 --> 06:22:55,285 AND I'M GOING TO TALK ABOUT SOME 10591 06:22:55,352 --> 06:22:57,587 EFFORTS THAT WE HAVE THAT I'VE 10592 06:22:57,654 --> 06:22:59,122 STARTED AT MY OWN INSTITUTION 10593 06:22:59,189 --> 06:23:01,291 AND THESE GO BACK FIVE YEARS TO 10594 06:23:01,358 --> 06:23:03,960 WHEN I FIRST STARTED OUT AT THE 10595 06:23:04,027 --> 06:23:05,362 INSTITUTION, I CAME FROM KANSAS 10596 06:23:05,429 --> 06:23:07,331 STATE UNIVERSITY BEFORE I'VE 10597 06:23:07,397 --> 06:23:09,232 BEEN WORKING ON SCHOLARLY 10598 06:23:09,299 --> 06:23:13,337 COMMUNICATION FOR PAST NINE 10599 06:23:13,403 --> 06:23:14,171 YEARS OR SO. 10600 06:23:14,237 --> 06:23:15,906 AND SO STARTING WHEN I FIRST 10601 06:23:15,972 --> 06:23:19,476 CAME IN, I STARTED IN 2018, THEN 10602 06:23:19,543 --> 06:23:20,877 IN 2019 THERE WAS A SUBCOMMITTEE 10603 06:23:20,944 --> 06:23:22,612 FORMED UNDER THE FACULTY SENATE 10604 06:23:22,679 --> 06:23:24,281 SO I'VE BEEN INVOLVED WITH 10605 06:23:24,348 --> 06:23:25,949 FACULTY SENATE SINCE I STARTED 10606 06:23:26,016 --> 06:23:27,517 AT VIRGINIA TECH. 10607 06:23:27,584 --> 06:23:30,187 AND THAT SUBCOMMITTEE WAS 10608 06:23:30,253 --> 06:23:31,555 FOCUSED ON REALLY CONDUCTING A 10609 06:23:31,621 --> 06:23:33,490 SURVEY AND ALSO TRYING TO MAKE 10610 06:23:33,557 --> 06:23:35,492 RECOMMENDATIONS BASED ON THE 10611 06:23:35,559 --> 06:23:37,094 RESULTS OF THAT SURVEY TO LOOK 10612 06:23:37,160 --> 06:23:40,764 AT SALARIES BUT ALSO LOOK AT 10613 06:23:40,831 --> 06:23:44,301 EVALUATION OF FACULTY RESEARCH, 10614 06:23:44,368 --> 06:23:45,602 SO PERCEPTION ON THE FACULTY 10615 06:23:45,669 --> 06:23:46,203 LEVEL. 10616 06:23:46,269 --> 06:23:48,505 WE HAD A HIGH SURVEY RESPONSE 10617 06:23:48,572 --> 06:23:50,507 BECAUSE OF THAT SALARY COMPONENT 10618 06:23:50,574 --> 06:23:51,241 IN THERE. 10619 06:23:51,308 --> 06:23:53,777 SO FACULTY ARE ALWAYS GOING TO 10620 06:23:53,844 --> 06:23:56,913 LOOK FOR AN EXCUSE TO EXPLAIN 10621 06:23:56,980 --> 06:23:58,648 ABOUT THEIR SALARIES, BUT WE 10622 06:23:58,715 --> 06:24:00,617 HAVE A HIGH SURVEY RESPONSE 10623 06:24:00,684 --> 06:24:02,152 RATE, AROUND 10%, AND 10624 06:24:02,219 --> 06:24:04,221 REPRESENTATION FROM EACH COLLEGE 10625 06:24:04,287 --> 06:24:05,989 INCLUDING UNIVERSITY LIBRARIES, 10626 06:24:06,056 --> 06:24:07,324 WE WANTED TO ASSESS PERCEPTIONS 10627 06:24:07,391 --> 06:24:10,127 AND ALSO WANTED TO PRESENT 10628 06:24:10,193 --> 06:24:12,229 OUTCOMES TO THE PRESIDENT OF THE 10629 06:24:12,295 --> 06:24:13,463 FACULTY SENATE WOULD PRESENT 10630 06:24:13,530 --> 06:24:16,166 OUTCOMES TO THE BOARD OF 10631 06:24:16,233 --> 06:24:17,801 VISITORS, OUR GOVERNING BOARD, 10632 06:24:17,868 --> 06:24:19,669 BY THE REPORT WHICH WAS DUE JUNE 10633 06:24:19,736 --> 06:24:20,003 3. 10634 06:24:20,070 --> 06:24:23,940 SO THIS IS GOING BACK ALMOST 10635 06:24:24,007 --> 06:24:25,475 EXACTLY FIVE YEARS. 10636 06:24:25,542 --> 06:24:27,244 SO, THIS IS JUST SOME OF THE 10637 06:24:27,310 --> 06:24:29,446 RESULTS I WANT TO SHOW YOU, THAT 10638 06:24:29,513 --> 06:24:31,615 WE'RE LOOKING AT THE METRICS 10639 06:24:31,681 --> 06:24:32,816 THAT THEY ARE USING, WHY WE'RE 10640 06:24:32,883 --> 06:24:33,250 USING THEM. 10641 06:24:33,316 --> 06:24:34,718 WE ASKED THEM WHAT DO YOU USE 10642 06:24:34,785 --> 06:24:38,422 AND WHY DO YOU WANT TO USE IT. 10643 06:24:38,488 --> 06:24:39,823 WE WOULD SAY, FOR EXAMPLE, MOST 10644 06:24:39,890 --> 06:24:42,759 OF THEM WERE USING NUMBER OF 10645 06:24:42,826 --> 06:24:43,393 PUBLICATIONS, JOURNAL METRICS, 10646 06:24:43,460 --> 06:24:45,629 BUT THEY WERE MORE INCLINED TO 10647 06:24:45,695 --> 06:24:51,868 USE NUMBER OF PUBLICATIONS 10648 06:24:51,935 --> 06:24:57,441 JOURNAL METRICS, JOURNAL 10649 06:24:57,507 --> 06:25:00,277 ACCEPTANCE RATES, WANTING TO USE 10650 06:25:00,343 --> 06:25:02,145 METRICS, ATTENDANCE NUMBERS, 10651 06:25:02,212 --> 06:25:06,183 SURPRISINGLY GRANT AWARD AMOUNT 10652 06:25:06,249 --> 06:25:07,551 AND AWARDS RECOGNITIONS, HONORS, 10653 06:25:07,617 --> 06:25:08,585 JOURNAL REPUTATION, MORE TO DO 10654 06:25:08,652 --> 06:25:10,554 WITH WHAT IS REPUTATION IN YOUR 10655 06:25:10,620 --> 06:25:12,422 FIELD, IS IT WELL KNOWN IN YOUR 10656 06:25:12,489 --> 06:25:13,523 FIELD, AND WE FOUND THAT WAS 10657 06:25:13,590 --> 06:25:15,058 MORE SO FOR PERSONAL REASONS. 10658 06:25:15,125 --> 06:25:19,396 THEY WANTED TO USE THAT METRIC. 10659 06:25:22,666 --> 06:25:25,001 WE ASKED THEM HOW THEY PERCEIVED 10660 06:25:25,068 --> 06:25:26,203 THE FAIRNESS OF RESEARCH 10661 06:25:26,269 --> 06:25:27,537 ASSESSMENT AT THE DEPARTMENT, 10662 06:25:27,604 --> 06:25:29,306 THE COLLEGE OR ADMINISTRATIVE 10663 06:25:29,372 --> 06:25:31,608 UNIT AND UNIVERSITY LEVEL. 10664 06:25:31,675 --> 06:25:34,010 OVERALL WE FOUND THAT IN GENERAL 10665 06:25:34,077 --> 06:25:35,512 THEY PERCEIVED THE DEPARTMENT 10666 06:25:35,579 --> 06:25:37,481 LEVEL TO BE FAIRER THAN THE 10667 06:25:37,547 --> 06:25:38,381 COLLEGE LEVEL AND THE COLLEGE 10668 06:25:38,448 --> 06:25:40,984 LEVEL TO BE FAIRER THAN THE 10669 06:25:41,051 --> 06:25:42,419 UNIVERSITY LEVEL. 10670 06:25:42,486 --> 06:25:46,256 HIGHER UP ASSESSMENT LEVEL WENT, 10671 06:25:46,323 --> 06:25:47,591 LESS THEY TRUSTED ASSESSMENT. 10672 06:25:47,657 --> 06:25:50,327 THE LESS THEY TRUSTED THAT, AS 10673 06:25:50,393 --> 06:25:52,028 IT WENT UP. 10674 06:25:52,095 --> 06:25:57,434 WE HAVE QUALITATIVE RESULTS, 10675 06:25:57,501 --> 06:25:58,768 ESPECIALLY FROM FACULTY IN 10676 06:25:58,835 --> 06:26:00,871 LIBERAL ARTS AND HUMAN SCIENCES, 10677 06:26:00,937 --> 06:26:03,607 COLLEGE OF LIBERAL ARTS AND 10678 06:26:03,673 --> 06:26:08,845 HUMAN SCIENCES WERE FRUSTRATED 10679 06:26:08,912 --> 06:26:10,480 BECAUSE THEY NEEDED MORE 10680 06:26:10,547 --> 06:26:12,148 COUNTING, THAT'S WHAT'S VALUED 10681 06:26:12,215 --> 06:26:15,652 MORE AND THEY GOT FRUSTRATED, 10682 06:26:15,719 --> 06:26:18,355 THERE WAS A LOT OF PLEASE STOP 10683 06:26:18,421 --> 06:26:21,391 COUNTING COMMENTS AND REMARKS. 10684 06:26:21,458 --> 06:26:32,002 THERE WAS ALSO -- WE ASKED ABOUT 10685 06:26:33,003 --> 06:26:34,638 PERCEPTION ON TIME COMMITMENT, 10686 06:26:34,704 --> 06:26:38,241 WENT OVER 100%, LET THEM 10687 06:26:38,308 --> 06:26:38,808 SELF-SELECT. 10688 06:26:38,875 --> 06:26:40,677 REPORTED ACTUAL, IT WAS 10689 06:26:40,744 --> 06:26:42,012 SIGNIFICANT HOW MUCH THEY WERE 10690 06:26:42,078 --> 06:26:43,547 CONTRIBUTING TOWARDS SERVICE AND 10691 06:26:43,613 --> 06:26:46,082 THERE WAS A LOT OF QUALITATIVE 10692 06:26:46,149 --> 06:26:49,986 RESPONSES RELATED TO HOW MUCH 10693 06:26:50,053 --> 06:26:53,857 THEY WERE NOT ABLE TO CONTRIBUTE 10694 06:26:53,924 --> 06:26:54,758 TO TEACHING, TOWARDS RESEARCH 10695 06:26:54,824 --> 06:26:57,127 BECAUSE OF ALL THE WORKLOAD THEY 10696 06:26:57,193 --> 06:27:07,737 HAD TOWARD AS -- TOWARD SERVICE 10697 06:27:07,971 --> 06:27:09,806 AND ADMINISTRATIVE DUTIES. 10698 06:27:09,873 --> 06:27:11,908 WE ALSO ASKED THEM SENTIMENT AND 10699 06:27:11,975 --> 06:27:19,749 CONCERNS FOR USE OF BUDGET 10700 06:27:19,816 --> 06:27:22,552 MODEL, THAT WE HAD JUST SWITCHED 10701 06:27:22,619 --> 06:27:23,720 TO IN 2018. 10702 06:27:23,787 --> 06:27:27,424 AND ASKED ABOUT WHAT ARE YOUR 10703 06:27:27,490 --> 06:27:29,392 SENTIMENTS TOWARDS WHAT IF 10704 06:27:29,459 --> 06:27:30,961 PUBLICATIONS AND CITATIONS ARE 10705 06:27:31,027 --> 06:27:32,829 INCLUDED, AND OVERALL IT WAS 10706 06:27:32,896 --> 06:27:35,098 NEGATIVE SENTIMENT OR IT WAS 10707 06:27:35,165 --> 06:27:39,135 MIXED OR NOT ENOUGH KNOWLEDGE TO 10708 06:27:39,202 --> 06:27:43,340 EVALUATE TO BE INCLUDED IN THE 10709 06:27:43,406 --> 06:27:44,374 PIBB MODEL. 10710 06:27:44,441 --> 06:27:45,609 IT WAS OVERALL NEGATIVE, PLEASE 10711 06:27:45,675 --> 06:27:47,110 DON'T DO THIS, PLEASE DON'T DO 10712 06:27:47,177 --> 06:27:47,344 THIS. 10713 06:27:47,410 --> 06:27:50,513 WE HAD A LOT OF THAT. 10714 06:27:50,580 --> 06:27:52,215 SO WE HAD A LOT OF 10715 06:27:52,282 --> 06:27:54,918 RECOMMENDATIONS BASED ON ALL OF 10716 06:27:54,985 --> 06:27:57,187 THIS, AND ONE HIGHLIGHTED IS THE 10717 06:27:57,253 --> 06:28:00,156 ONE THAT WE ENDED UP COMING TO 10718 06:28:00,223 --> 06:28:01,257 THE CONCLUSION OF DOING BECAUSE 10719 06:28:01,324 --> 06:28:02,759 WE HAVE TO START SOMEWHERE. 10720 06:28:02,826 --> 06:28:04,060 THIS IS WHERE WE STARTED. 10721 06:28:04,127 --> 06:28:05,862 THIS IS WHERE OTHER INSTITUTIONS 10722 06:28:05,929 --> 06:28:06,062 START. 10723 06:28:06,129 --> 06:28:07,731 AND MANY OF YOU MIGHT BE 10724 06:28:07,797 --> 06:28:11,201 FAMILIAR WITH THIS TYPE OF 10725 06:28:11,267 --> 06:28:12,402 RECOMMENDATION THAT YOU DEVELOP 10726 06:28:12,469 --> 06:28:15,905 A BRIEF -- I SAY BRIEF BUT IT'S 10727 06:28:15,972 --> 06:28:19,909 NOT AS BRIEF AS IT COULD BE. 10728 06:28:19,976 --> 06:28:23,013 BUT DEPARTMENT LEVEL DRIVEN BUT 10729 06:28:23,079 --> 06:28:24,247 UNIVERSITY-WIDE INCLUSIVE AND 10730 06:28:24,314 --> 06:28:24,981 CAREFULLY WRITTEN RESPONSIBLE 10731 06:28:25,048 --> 06:28:26,650 RESEARCH ASSESSMENT STATEMENT OF 10732 06:28:26,716 --> 06:28:28,752 PRINCIPLES TO SUPPORT AND DRIVE 10733 06:28:28,818 --> 06:28:29,386 RESEARCH PRODUCTION. 10734 06:28:29,452 --> 06:28:34,891 SO THIS IS WHAT WE FOCUSED ON. 10735 06:28:34,958 --> 06:28:36,726 THE UNIVERSITY SHOULD PROVIDE 10736 06:28:36,793 --> 06:28:41,998 RESOURCES TO REDUCE TEACHING 10737 06:28:42,065 --> 06:28:43,433 SERVICE AND EXPECTATIONS, WE'RE 10738 06:28:43,500 --> 06:28:44,968 FOCUSING ON, I'M ONE OF THE 10739 06:28:45,035 --> 06:28:47,537 SUPPORTS FOR THE ELEMENT SYSTEM, 10740 06:28:47,604 --> 06:28:48,238 RESEARCH INFORMATION MANAGEMENT 10741 06:28:48,304 --> 06:28:49,873 SYSTEM, TRUST ME I'M THE KEY 10742 06:28:49,939 --> 06:28:51,975 PERP THAT GETS TO HEAR ALL THE 10743 06:28:52,042 --> 06:28:53,143 COMPLAINT ABOUT THAT SYSTEM. 10744 06:28:53,209 --> 06:28:55,345 I GET TO ALSO TRY TO PROVIDE 10745 06:28:55,412 --> 06:28:57,313 MORE SUPPORT FOR THAT SYSTEM AS 10746 06:28:57,380 --> 06:28:57,514 WELL. 10747 06:28:57,580 --> 06:28:59,249 AND SO THAT'S ALSO SOMETHING 10748 06:28:59,315 --> 06:29:01,584 WE'RE WORKING ON. 10749 06:29:01,651 --> 06:29:03,753 WE'RE ALSO CONSIDERING FACULTY 10750 06:29:03,820 --> 06:29:08,658 RESEARCH LIAISONS BETWEEN 10751 06:29:08,725 --> 06:29:10,026 ADMINISTRATION AND FACULTY. 10752 06:29:10,093 --> 06:29:12,128 I'M THINKING WE NEED TO REVISIT 10753 06:29:12,195 --> 06:29:14,164 THESE THINGS, SO IT'S AMAZING TO 10754 06:29:14,230 --> 06:29:16,599 LOOK BACK AT OUR WORK FROM FIVE 10755 06:29:16,666 --> 06:29:17,333 YEARS AGO. 10756 06:29:17,400 --> 06:29:19,069 SO, YEAH, A LOT OF THESE THINGS 10757 06:29:19,135 --> 06:29:23,339 AS YOU CAN SEE UP HERE ARE VERY 10758 06:29:23,406 --> 06:29:25,508 MUCH WHERE WE SHOULD BE FOCUSING 10759 06:29:25,575 --> 06:29:27,143 BUT WE HAVE TO FOCUS ON ONE 10760 06:29:27,210 --> 06:29:29,112 THING AT A TIME BECAUSE I DO 10761 06:29:29,179 --> 06:29:31,214 BELIEVE IN QUALITY OVER 10762 06:29:31,281 --> 06:29:32,182 QUANTITY. 10763 06:29:32,248 --> 06:29:36,019 WITH THE DEPARTMENT LEVEL 10764 06:29:36,086 --> 06:29:37,153 RECOMMENDATIONS, THEY ARE DOING 10765 06:29:37,220 --> 06:29:45,995 THINGS AT FACULTY AFFAIRS. 10766 06:29:46,062 --> 06:29:46,730 ALMOST DONE. 10767 06:29:46,796 --> 06:29:48,198 MAKING THINGS CLEAR IN THE 10768 06:29:48,264 --> 06:29:49,199 GUIDELINES, OUTSIDE MY AREA BUT 10769 06:29:49,265 --> 06:29:52,769 THEY ARE DOING THINGS. 10770 06:29:52,836 --> 06:29:58,708 AVOID PUTTING OVERLY BURDENSOME 10771 06:29:58,775 --> 06:29:59,542 AND UNREALISTIC EXPECTATIONS ON 10772 06:29:59,609 --> 06:30:00,343 FACULTY. 10773 06:30:00,410 --> 06:30:02,378 WHAT DOES THIS HAVE TO DO WITH 10774 06:30:02,445 --> 06:30:03,813 PUBLICATION BIAS? 10775 06:30:03,880 --> 06:30:05,782 THIS IS OUR PROGRESS AND 10776 06:30:05,849 --> 06:30:06,049 TIMELINE. 10777 06:30:06,116 --> 06:30:07,317 WE HAD ANOTHER SUBCOMMITTEE THAT 10778 06:30:07,383 --> 06:30:11,054 NORMANDY TO -- FORMED TO 10779 06:30:11,121 --> 06:30:12,789 ADDRESS ALL THE AREAS TO ADDRESS 10780 06:30:12,856 --> 06:30:14,324 THIS PROBLEM WITH TIME 10781 06:30:14,390 --> 06:30:15,091 ALLOCATIONS, WE ADDRESSED 10782 06:30:15,158 --> 06:30:17,460 TEACHING SERVICE AND RESEARCH TO 10783 06:30:17,527 --> 06:30:18,495 MAKE FURTHER RECOMMENDATIONS. 10784 06:30:18,561 --> 06:30:20,530 ANOTHER TWO YEARS TO FORM A TASK 10785 06:30:20,597 --> 06:30:23,967 FORCE BECAUSE I HAD TO BE, LIKE, 10786 06:30:24,033 --> 06:30:25,502 WELL, IS ANYTHING HAPPENING? 10787 06:30:25,568 --> 06:30:26,903 NO, NOTHING WAS HAPPENING. 10788 06:30:26,970 --> 06:30:31,741 I SAID LET'S MAKE SOMETHING 10789 06:30:31,808 --> 06:30:32,408 HAPPEN. 10790 06:30:32,475 --> 06:30:34,477 IN 2022 FORMED A TASK FORCE TO 10791 06:30:34,544 --> 06:30:35,545 DRAFT OUR OWN STATEMENT. 10792 06:30:35,612 --> 06:30:38,081 WE DID DRAFT OUR OWN STATEMENT, 10793 06:30:38,148 --> 06:30:40,483 2023, THE STATEMENT WAS VOTED 10794 06:30:40,550 --> 06:30:43,887 ON, ENDORSED BY FACULTY SENATE. 10795 06:30:43,953 --> 06:30:46,256 THIS YEAR WE'VE CRAFTED A 10796 06:30:46,322 --> 06:30:47,590 RESOLUTION THROUGH UNIVERSITY 10797 06:30:47,657 --> 06:30:49,893 GOVERNANCE, JUST ENDORSED BY 10798 06:30:49,959 --> 06:30:51,628 UNIVERSITY COUNCIL AND PRESIDENT 10799 06:30:51,694 --> 06:30:53,263 MAY 6, IT'S AVAILABLE ON THE 10800 06:30:53,329 --> 06:30:53,530 INTERNET. 10801 06:30:53,596 --> 06:30:55,465 YOU CAN GO TO THAT STATEMENT AND 10802 06:30:55,532 --> 06:31:00,403 READ IT. 10803 06:31:00,470 --> 06:31:02,605 BASEDDED ON THE LIEDEN 10804 06:31:02,672 --> 06:31:03,473 MANIFESTO. 10805 06:31:03,540 --> 06:31:06,743 WE CAN'T JUST HAVE IT DORMANT ON 10806 06:31:06,810 --> 06:31:07,544 THE INTERNET. 10807 06:31:07,610 --> 06:31:09,512 WE NEED TO COMMUNICATE TO 10808 06:31:09,579 --> 06:31:10,680 UNIVERSITY LEADERS, DEANS GROUP, 10809 06:31:10,747 --> 06:31:12,048 DEPARTMENT HEADS, ASSOCIATE 10810 06:31:12,115 --> 06:31:12,982 DEANS FOR RESEARCH. 10811 06:31:13,049 --> 06:31:17,487 THE STATEMENT IS NOT POLICY. 10812 06:31:17,554 --> 06:31:18,321 IT'S ASPIRATIONAL, EDUCATIONAL 10813 06:31:18,388 --> 06:31:19,923 GUIDANCE TOOL, DOES NOT 10814 06:31:19,989 --> 06:31:22,091 SPECIFICALLY MENTION NULL 10815 06:31:22,158 --> 06:31:22,358 STUDIES. 10816 06:31:22,425 --> 06:31:24,828 IT WAS MENTIONED IN THE REPORT. 10817 06:31:24,894 --> 06:31:28,164 WHEN I WAS WRITING THE SLIDES WE 10818 06:31:28,231 --> 06:31:29,766 DIDN'T MENTION IT IN THE NOTES. 10819 06:31:29,833 --> 06:31:34,470 WE HAD A HUGE FOCUS ON 10820 06:31:34,537 --> 06:31:36,005 INTERDISCIPLINARY AND 10821 06:31:36,072 --> 06:31:37,540 TRANSDISCIPLINARY RESEARCH, A 10822 06:31:37,607 --> 06:31:42,278 HUGE PROJECT WITH INCENTIVIZING 10823 06:31:42,345 --> 06:31:44,013 NULL STUDIES AND INCENTIVIZING 10824 06:31:44,080 --> 06:31:46,216 RESEARCH AS WELL, A PROBLEM AT 10825 06:31:46,282 --> 06:31:51,921 OUR INSTITUTION AS WELL. 10826 06:31:51,988 --> 06:31:53,523 THE PIBB MODEL INCORPORATION IT. 10827 06:31:53,590 --> 06:31:55,325 IT'S A SMALL PORTION OF THE 10828 06:31:55,391 --> 06:31:58,995 BUDGET MODEL BUT STILL, TO ME, A 10829 06:31:59,062 --> 06:32:00,563 PROBLEM AND OUR INCENTIVE BASED 10830 06:32:00,630 --> 06:32:04,167 MODEL. 10831 06:32:04,234 --> 06:32:05,368 THE UNIVERSITY IS TRYING TO 10832 06:32:05,435 --> 06:32:07,136 BECOME A MEMBER OF AAU, SOME 10833 06:32:07,203 --> 06:32:08,638 DOES GET AWAY FROM THAT BUT I 10834 06:32:08,705 --> 06:32:10,940 THINK IF WE WORK IN CONJUNCTION 10835 06:32:11,007 --> 06:32:14,777 WITH COMMUNICATING AND WORKING 10836 06:32:14,844 --> 06:32:16,279 WITH UNIVERSITY LEADERS THROUGH 10837 06:32:16,346 --> 06:32:17,680 WHAT'S CALLED GLOBAL DISTINCTION 10838 06:32:17,747 --> 06:32:20,550 INITIATIVE I THINK THAT WE CAN 10839 06:32:20,617 --> 06:32:23,086 KIND OF BLEND BOTH OF THE GOALS, 10840 06:32:23,152 --> 06:32:25,722 THE STRATEGIES TOGETHER TO HAVE 10841 06:32:25,788 --> 06:32:26,923 THE BEST OUTCOME. 10842 06:32:26,990 --> 06:32:31,761 SO IT'S REALLY IMPORTANT TO HAVE 10843 06:32:31,828 --> 06:32:34,030 FACULTY VOICES ON KEY COMMITTEES 10844 06:32:34,097 --> 06:32:37,967 WHILE WE MOVE THESE FORWARD. 10845 06:32:38,034 --> 06:32:40,970 SO THEY ARE CLOSELY RELATED TO 10846 06:32:41,037 --> 06:32:42,472 PUBLICATION BIAS, I WON'T GO 10847 06:32:42,538 --> 06:32:45,208 OVER THIS SLIDE IN TOTAL BUT I 10848 06:32:45,275 --> 06:32:48,978 REALLY WANT TO EMPHASIZE THAT 10849 06:32:49,045 --> 06:32:50,880 THIS IS -- THE INCENTIVE SYSTEM 10850 06:32:50,947 --> 06:32:52,615 IS RELATED TO THIS AND WE DID 10851 06:32:52,682 --> 06:32:54,250 HAVE A NULL STUDIES HIGHLIGHTED 10852 06:32:54,317 --> 06:32:56,920 IN OUR REPORT AND EDUCATING THE 10853 06:32:56,986 --> 06:32:58,421 COMMUNITY ON THIS STATEMENT OF 10854 06:32:58,488 --> 06:33:02,125 PRINCIPLES IS A KEY FIRST STEP 10855 06:33:02,191 --> 06:33:03,626 MOVING TOWARDS A RESPONSIBLE 10856 06:33:03,693 --> 06:33:04,894 RESEARCH CULTURE. 10857 06:33:04,961 --> 06:33:10,333 THANK YOU. 10858 06:33:12,602 --> 06:33:14,537 >> HELLO, I'M RENE BERNARD, 10859 06:33:14,604 --> 06:33:23,513 BERLIN INSTITUTE OF HEALTH. 10860 06:33:44,667 --> 06:33:47,603 I'M AFFILIATED WITH THE CENTER 10861 06:33:47,670 --> 06:33:52,075 FOR RESPONSIBLE RESEARCH, BERLIN 10862 06:33:52,141 --> 06:33:54,243 INTUITION FOR HEALTH, 10863 06:33:54,310 --> 06:33:56,646 COORDINATOR AT NEUROCURE, MY 10864 06:33:56,713 --> 06:33:59,949 AFFILIATIONS ARE PART OF THE 10865 06:34:00,016 --> 06:34:01,384 CHARITE MEDICINE BERLIN, ONE OF 10866 06:34:01,451 --> 06:34:02,919 THE LARGEST MEDICAL CAMPUSES IN 10867 06:34:02,986 --> 06:34:03,152 EUROPE. 10868 06:34:03,219 --> 06:34:07,724 I'M SORRY I CAN'T BE THERE IN 10869 06:34:07,790 --> 06:34:13,162 PERSON BECAUSE MY FLIGHT WAS 10870 06:34:13,229 --> 06:34:15,798 CANCELED, I SHOULD BE THERE IN 10871 06:34:15,865 --> 06:34:19,502 PERSON TONIGHT TO MEET YOU ALL. 10872 06:34:21,671 --> 06:34:26,342 LET ME BRIEFLY TELL YOU WHAT MY 10873 06:34:26,409 --> 06:34:28,011 INSTITUTION HAS DONE TO INCREASE 10874 06:34:28,077 --> 06:34:31,114 AWARENESS FOR NULL PUBLICATION 10875 06:34:31,180 --> 06:34:31,547 DATA. 10876 06:34:31,614 --> 06:34:36,486 NUMBER ONE, SINCE 2017 THE QUEST 10877 06:34:36,552 --> 06:34:41,424 CENTER OFFERS 1,000 URONULL 10878 06:34:41,491 --> 06:34:47,163 PRIZE TO FIRST, LAST, 10879 06:34:47,230 --> 06:34:49,465 CORRESPONDING AUTHORS FROM OUR 10880 06:34:49,532 --> 06:34:52,068 INSTITUTION, NO BUDGET LIMIT, 10881 06:34:52,135 --> 06:34:53,803 ONCE APPROVED AWARDEES RECEIVED 10882 06:34:53,870 --> 06:34:54,837 A LETTER, NULL PUBLICATION 10883 06:34:54,904 --> 06:34:56,539 DISPLAYED ON THE QUEST WEBSITE. 10884 06:34:56,606 --> 06:35:01,244 WITH A CLEAR AND DEFINED PROCESS 10885 06:35:01,310 --> 06:35:01,944 FOR THIS, FOR INSTANCE 10886 06:35:02,011 --> 06:35:04,414 HYPOTHESIS NEEDS TO BE CLEARLY 10887 06:35:04,480 --> 06:35:07,417 DEFINED AND NULL FINDING TO BE 10888 06:35:07,483 --> 06:35:09,018 REPORTED IN ABSTRACT AND 10889 06:35:09,085 --> 06:35:09,585 RESULTS. 10890 06:35:09,652 --> 06:35:14,724 SO FAR WE'VE GIVEN OUT 46 10891 06:35:14,791 --> 06:35:15,725 AWARDS, 8 APPLICATIONS DENIED, 10892 06:35:15,792 --> 06:35:18,528 THE REASON WAS OFTEN IT WAS 10893 06:35:18,594 --> 06:35:21,497 PURELY EXPLORATORY OR MIX OF 10894 06:35:21,564 --> 06:35:23,900 POSITIVE AND NEGATIVE DATA, AND 10895 06:35:23,966 --> 06:35:28,571 BEFORE WE SEND OUT THESE DENIALS 10896 06:35:28,638 --> 06:35:33,576 WE ALSO HAVE TO APPLY 10897 06:35:33,643 --> 06:35:34,777 PRINCIPLES, SOMEBODY LOOKS OVER 10898 06:35:34,844 --> 06:35:36,546 THIS AGAIN. 10899 06:35:36,612 --> 06:35:39,282 WE HAVE AN AWARENESS TOOL FOR 10900 06:35:39,348 --> 06:35:42,051 FILE DRAWER DATA, IT'S CALLED 10901 06:35:42,118 --> 06:35:42,452 FIDDLE. 10902 06:35:42,518 --> 06:35:45,855 THAT GOES BEYOND JUST NULL DATA, 10903 06:35:45,922 --> 06:35:47,490 BUT THE TOOL REPRESENTS 10904 06:35:47,557 --> 06:35:49,358 ALTERNATIVE OUTLETS AND 10905 06:35:49,425 --> 06:35:51,394 STRATEGIES TO MAKE UNPUBLISHED 10906 06:35:51,461 --> 06:35:55,932 DATA PUBLISHABLE WITH A PROPER 10907 06:35:55,998 --> 06:35:56,699 RECOGNITION. 10908 06:35:56,766 --> 06:36:01,170 WE ALSO PUBLISHED AN ACCOMPANY 10909 06:36:01,237 --> 06:36:02,305 PAPER, HIGHLIGHT QUALITY 10910 06:36:02,371 --> 06:36:03,806 PREREQUISITES AND WHAT ENOUGH OF 10911 06:36:03,873 --> 06:36:06,008 INFORMATION SHOULD BE INCLUDED 10912 06:36:06,075 --> 06:36:07,510 WHEN PUBLISHING NEGATIVE DATA 10913 06:36:07,577 --> 06:36:11,781 ALSO TO PROVIDE STUDIES OF 10914 06:36:11,848 --> 06:36:13,015 QUESTIONABLE QUALITY TO SURFACE. 10915 06:36:13,082 --> 06:36:15,518 SO IT WAS NICE TO SEE THAT THE 10916 06:36:15,585 --> 06:36:18,221 AWARDS WE STARTED GOT ALSO 10917 06:36:18,287 --> 06:36:23,426 ADOPTED BY OTHERS, FOR INSTANCE 10918 06:36:23,493 --> 06:36:29,632 BY ASAP BIO, NEGATIVE RESULTS, 10919 06:36:29,699 --> 06:36:32,135 BY PRE-CLINICAL DATA FORUM WHICH 10920 06:36:32,201 --> 06:36:35,571 GIVES OUT NEGATIVE DATA PRIZE IN 10921 06:36:35,638 --> 06:36:36,472 PRE-CLINICAL NEUROSCIENCE. 10922 06:36:36,539 --> 06:36:40,943 THE FIRST THING WE STARTED IS TO 10923 06:36:41,010 --> 06:36:44,347 INCLUDE EXPECTATIONS TO PUBLISH 10924 06:36:44,413 --> 06:36:46,215 NEGATIVE RESULTS IN OUR 10925 06:36:46,282 --> 06:36:47,850 INTRAMURAL FUNDING AGREEMENTS 10926 06:36:47,917 --> 06:36:50,586 AND APPLICANTS FOR INSTANCE IN 10927 06:36:50,653 --> 06:36:54,190 THE COST OF EXCELLENCE NEUROCURE 10928 06:36:54,257 --> 06:36:57,827 WHERE I WORK, WE COMBINE WITH 10929 06:36:57,894 --> 06:37:00,763 OFFER TO INTERACT WITH 10930 06:37:00,830 --> 06:37:04,367 APPLICANTS AND TO COUNSEL THEM. 10931 06:37:04,433 --> 06:37:07,637 OF COURSE, THERE ARE STILL 10932 06:37:07,703 --> 06:37:11,574 BARRIERS, ONE OF THEM IS LIMITED 10933 06:37:11,641 --> 06:37:12,275 VISIBILITY. 10934 06:37:12,341 --> 06:37:15,545 I WOULD SAY PROBABLY NOT 10935 06:37:15,611 --> 06:37:16,312 EVERYONE, OUR INSTITUTIONS, 10936 06:37:16,379 --> 06:37:18,915 KNOWS ABOUT THESE THREE THINGS 10937 06:37:18,981 --> 06:37:23,186 THAT I JUST TALKED ABOUT, AND 10938 06:37:23,252 --> 06:37:24,287 OUTSIDE OF INSTITUTION FROM 10939 06:37:24,353 --> 06:37:28,791 BERLIN, SO ALSO OUR CALL SERVICE 10940 06:37:28,858 --> 06:37:30,193 25 P.I.s IS SMALL SO 10941 06:37:30,259 --> 06:37:33,863 VISIBILITY NEEDS TO BE 10942 06:37:33,930 --> 06:37:34,263 INCREASED. 10943 06:37:34,330 --> 06:37:36,032 SECOND, NULL STUDIES ARE NOT 10944 06:37:36,098 --> 06:37:38,000 LABELED AS SUCH, WHICH IS GOOD 10945 06:37:38,067 --> 06:37:40,303 BECAUSE THEY ARE STILL TRUE 10946 06:37:40,369 --> 06:37:41,671 RESEARCH RESULTS, BUT AT THE 10947 06:37:41,737 --> 06:37:43,940 SAME TIME IT MAKES IT REALLY 10948 06:37:44,006 --> 06:37:46,576 HARD TO QUANTIFY THEM OR 10949 06:37:46,642 --> 06:37:49,245 VISUALIZE TRENDS AND THAT'S 10950 06:37:49,312 --> 06:37:51,380 EXTREMELY DIFFICULT TO FOLLOW UP 10951 06:37:51,447 --> 06:37:52,782 AND UNTIL WE HAVE SOMETHING WE 10952 06:37:52,848 --> 06:37:56,285 WILL NEVER FIND OUT WHAT THE 10953 06:37:56,352 --> 06:38:04,260 TRUE NUMBER OF NULL PUBLICATIONS 10954 06:38:04,327 --> 06:38:04,627 IS. 10955 06:38:04,694 --> 06:38:08,798 STUDY PRE-REGISTRATION IS LIKE A 10956 06:38:08,864 --> 06:38:12,868 REMEDY FOR OUTCOME SWITCHING, IT 10957 06:38:12,935 --> 06:38:13,469 IS EXTREMELY UNPOPULAR. 10958 06:38:13,536 --> 06:38:16,072 WE HAVE A PRIZE FOR THIS, IT'S 10959 06:38:16,138 --> 06:38:18,374 ALMOST NEVER, EVER REQUESTED. 10960 06:38:18,441 --> 06:38:23,546 SO, WE NEED TO FIND A WAY TO 10961 06:38:23,613 --> 06:38:25,915 MAKE SCIENTISTS TO ADOPT THIS 10962 06:38:25,982 --> 06:38:28,084 AND MAYBE THEN WE GET ALSO 10963 06:38:28,150 --> 06:38:33,689 CLOSER TO THE REAL NUMBER OF 10964 06:38:33,756 --> 06:38:34,757 NEGATIVE DATA PUBLISHED 10965 06:38:34,824 --> 06:38:35,024 WORLDWIDE. 10966 06:38:35,091 --> 06:38:37,126 AND THE FOURTH POINT IS OF 10967 06:38:37,193 --> 06:38:39,562 COURSE COMMUNITY NEEDS TO DO 10968 06:38:39,629 --> 06:38:42,365 MORE THAN JUST HANDING OUT 10969 06:38:42,431 --> 06:38:42,965 MONEY. 10970 06:38:43,032 --> 06:38:45,735 WE NEED ANNUAL AWARENESS DAYS OR 10971 06:38:45,801 --> 06:38:48,871 WEEKS, HACKATHONS WHERE PEOPLE 10972 06:38:48,938 --> 06:38:49,672 COME TOGETHER AND PUBLIC 10973 06:38:49,739 --> 06:38:52,708 NEGATIVE DATA OR BRING THEM TO 10974 06:38:52,775 --> 06:38:54,343 THE NEXT STEP, EXPERIENCE 10975 06:38:54,410 --> 06:38:57,013 SHARING OF PEOPLE THAT HAVE 10976 06:38:57,079 --> 06:38:59,515 TROUBLE GETTING THEIR DATA 10977 06:38:59,582 --> 06:39:01,817 PUBLISHED AND SHARED WITHIN THE 10978 06:39:01,884 --> 06:39:12,428 COMMUNITY AND ALSO THE TOPIC OF 10979 06:39:14,497 --> 06:39:15,631 INTEGRATED IN BIOMEDICAL 10980 06:39:15,698 --> 06:39:16,399 CURRICULA. 10981 06:39:16,465 --> 06:39:18,501 THANK YOU FOR THE INVITATION. 10982 06:39:18,567 --> 06:39:20,269 I'M EXCITED TO BE PART OF THIS. 10983 06:39:20,336 --> 06:39:22,772 WHAT WE TRY TO ACHIEVE IS 10984 06:39:22,838 --> 06:39:27,176 NOTHING SHORT OF GRAND CHANGE, 10985 06:39:27,243 --> 06:39:30,012 SCIENTISTS HAVE A LOT TO SAY 10986 06:39:30,079 --> 06:39:31,914 ABOUT THIS, THE GREAT TOOLS 10987 06:39:31,981 --> 06:39:34,417 THAT'S BEEN DEVELOPED, THE 10988 06:39:34,483 --> 06:39:35,918 BEHAVIORAL CHANGE WHILE, MATCH 10989 06:39:35,985 --> 06:39:38,220 SOURCES OF BEHAVIOR WITH 10990 06:39:38,287 --> 06:39:38,988 POSSIBLE INTERVENTIONS AND 10991 06:39:39,055 --> 06:39:41,824 POLICIES AND MAYBE TOMORROW WE 10992 06:39:41,891 --> 06:39:44,327 COULD ADOPT THIS TO CREATE 10993 06:39:44,393 --> 06:39:46,262 BEHAVIORAL CHANGE WHEEL FOR 10994 06:39:46,329 --> 06:39:47,129 PREVENTING PUBLICATION BIAS. 10995 06:39:47,196 --> 06:39:49,231 ALL RIGHT. 10996 06:39:49,298 --> 06:39:49,865 THAT'S IT. 10997 06:39:49,932 --> 06:39:51,400 AND THANK YOU VERY MUCH AGAIN. 10998 06:39:51,467 --> 06:39:56,572 BYE. 10999 06:40:02,311 --> 06:40:03,979 >> HI THERE. 11000 06:40:04,046 --> 06:40:05,514 IT'S LATE, I'VE GIVEN GINA 11001 06:40:05,581 --> 06:40:08,984 PERMISSION TO KICK ME IF I GO 11002 06:40:09,051 --> 06:40:09,852 OVER TIME. 11003 06:40:09,919 --> 06:40:12,822 WE WERE GIVEN THE INSTRUCTION TO 11004 06:40:12,888 --> 06:40:14,924 FOCUS TODAY ON PROBLEMS, RATHER 11005 06:40:14,990 --> 06:40:15,558 THAN SOLUTIONS. 11006 06:40:15,624 --> 06:40:18,661 WHICH IS PERFECT FOR ME BECAUSE 11007 06:40:18,728 --> 06:40:23,032 ALL I HAVE ARE DATA THAT SHOW 11008 06:40:23,099 --> 06:40:23,766 THE PROBLEMS. 11009 06:40:23,833 --> 06:40:25,835 AND GINA SET ME UP NICELY WHEN 11010 06:40:25,901 --> 06:40:27,503 SHE SAID THERE ARE MYTHS AROUND 11011 06:40:27,570 --> 06:40:31,006 PROMOTION AND TEN TENURE. 11012 06:40:31,073 --> 06:40:33,943 THAT'S VERY TRUE. 11013 06:40:34,009 --> 06:40:35,845 SOMETHING WE REALIZE AROUND 11014 06:40:35,911 --> 06:40:37,813 2016, 2017, MY COLLEAGUES AND I 11015 06:40:37,880 --> 06:40:39,682 WHEN TALKING ABOUT INCENTIVES 11016 06:40:39,749 --> 06:40:40,883 AND IMPROVING INCENTIVES, WE 11017 06:40:40,950 --> 06:40:43,252 REALIZED WE DIDN'T REALLY HAVE A 11018 06:40:43,319 --> 06:40:46,055 GOOD IDEA OF WHAT WAS IN 11019 06:40:46,122 --> 06:40:47,556 PROMOTION AND TENURE DOCUMENTS. 11020 06:40:47,623 --> 06:40:49,158 A LOT OF EVALUATIONS HAPPEN 11021 06:40:49,225 --> 06:40:51,660 BEHIND CLOSE DOORS, THOSE ARE 11022 06:40:51,727 --> 06:40:52,628 PRIVILEGED CONVERSATIONS. 11023 06:40:52,695 --> 06:40:54,363 AND WE THINK WE HAVE A LOT OF 11024 06:40:54,430 --> 06:40:55,831 IDEAS ABOUT WHAT IS GETTING 11025 06:40:55,898 --> 06:40:56,699 EVALUATED AND HOW. 11026 06:40:56,766 --> 06:40:58,968 BUT WE WANTED TO KNOW WHAT WAS 11027 06:40:59,034 --> 06:41:01,504 GOING ON. 11028 06:41:01,570 --> 06:41:03,205 SO, WE DECIDED TO COLLECT. 11029 06:41:03,272 --> 06:41:06,842 IT WAS A REPRESENTATIVE SAMPLE 11030 06:41:06,909 --> 06:41:07,676 FROM 129 UNIVERSITIES, OVER 300 11031 06:41:07,743 --> 06:41:12,014 ACADEMIC UNITS IN THE U.S. AND 11032 06:41:12,081 --> 06:41:12,381 CANADA. 11033 06:41:12,448 --> 06:41:14,283 WE COLLECTED 850 DOCUMENTS THAT 11034 06:41:14,350 --> 06:41:16,252 HAD TO DO WITH PROMOTION AND 11035 06:41:16,318 --> 06:41:17,453 TENURE PROCESS. 11036 06:41:17,520 --> 06:41:19,088 WE WENT THROUGH AND ANALYZED 11037 06:41:19,155 --> 06:41:19,288 THOSE. 11038 06:41:19,355 --> 06:41:20,856 I DON'T THINK ANY OF THE NUMBERS 11039 06:41:20,923 --> 06:41:23,359 THAT I'M GOING TO GIVE TODAY 11040 06:41:23,426 --> 06:41:24,894 WILL SURPRISE ANYONE BUT I THINK 11041 06:41:24,960 --> 06:41:26,762 IT'S GOOD TO HAVE THOSE NUMBERS. 11042 06:41:26,829 --> 06:41:28,864 AND I'LL TALK IN A MOMENT ABOUT 11043 06:41:28,931 --> 06:41:31,333 HOW IT RELATES TO PUBLICATION 11044 06:41:31,400 --> 06:41:31,667 BIAS. 11045 06:41:31,734 --> 06:41:33,302 SO, AGAIN, THIS WILL NOT 11046 06:41:33,369 --> 06:41:36,705 SURPRISE ANYBODY, WE FOUND A 11047 06:41:36,772 --> 06:41:38,407 PROMINENT USE OF BIBLIOMETRICS 11048 06:41:38,474 --> 06:41:40,075 LIKE JOURNAL IMPACT FACTOR AND 11049 06:41:40,142 --> 06:41:41,177 CITATION COUNTS. 11050 06:41:41,243 --> 06:41:45,648 IF WE BROKE IT DOWN BY, FOR 11051 06:41:45,714 --> 06:41:46,882 EXAMPLE, RESEARCH INTENSIVE 11052 06:41:46,949 --> 06:41:49,919 R-TYPE INSTITUTIONS, AS MANY AS 11053 06:41:49,985 --> 06:41:51,520 40% WERE USING JOURNAL IMPACT 11054 06:41:51,587 --> 06:41:56,525 FACTOR IN PROMOTION AND TENURE 11055 06:41:56,592 --> 06:42:03,399 GUIDELINES, PROBABLY AND 11056 06:42:03,466 --> 06:42:04,934 UNDERESTIMATION, LOOKING FOR 11057 06:42:05,000 --> 06:42:06,702 HIGHLY RECOGNIZED JOURNAL, SO 11058 06:42:06,769 --> 06:42:08,571 40% USING THAT JOURNAL IMPACT 11059 06:42:08,637 --> 06:42:08,804 FACTOR. 11060 06:42:08,871 --> 06:42:11,106 MORE WORRYING IS A LOT OF THOSE 11061 06:42:11,173 --> 06:42:12,675 INSTITUTIONS WERE USING JOURNAL 11062 06:42:12,741 --> 06:42:13,876 IMPACT FACTOR AS PROXY MEASURE 11063 06:42:13,943 --> 06:42:16,245 FOR QUALITY EVEN THOUGH WE HAVE 11064 06:42:16,312 --> 06:42:19,014 EVIDENCE TO THE CONTRARY JOURNAL 11065 06:42:19,081 --> 06:42:20,282 IMPACT FACTOR MEASURES QUALITY. 11066 06:42:20,349 --> 06:42:23,752 A LOT OF THEM WERE FOCUSING ON 11067 06:42:23,819 --> 06:42:25,754 PUBLICATION COUNTS, HOW MANY 11068 06:42:25,821 --> 06:42:27,389 ARTICLES, HOW MANY PRODUCTS. 11069 06:42:27,456 --> 06:42:29,225 MY DEPARTMENT IN PARTICULAR HAS 11070 06:42:29,291 --> 06:42:31,126 PUT A VERY CLEAR MARK ON IF YOU 11071 06:42:31,193 --> 06:42:32,862 WANT THIS PROMOTION LEVEL YOU 11072 06:42:32,928 --> 06:42:34,730 NEED TEN, IF YOU WANT THIS ONE 11073 06:42:34,797 --> 06:42:37,700 YOU NEED TWENTY SO WE'RE GOING 11074 06:42:37,766 --> 06:42:39,134 BACK, IN MY VIEW. 11075 06:42:39,201 --> 06:42:41,270 ANOTHER THING WE DID WAS WE 11076 06:42:41,337 --> 06:42:42,505 SURVEYED FACULTY, WE DIDN'T JUST 11077 06:42:42,571 --> 06:42:44,807 WANT TO KNOW HOW OFTEN THINGS 11078 06:42:44,874 --> 06:42:46,208 WERE MENTIONED BUT WHAT DID THE 11079 06:42:46,275 --> 06:42:48,310 FACT AT THE SAME INSTITUTIONS 11080 06:42:48,377 --> 06:42:50,713 THINK ABOUT HOW THEY WERE BEING 11081 06:42:50,779 --> 06:42:50,913 USED. 11082 06:42:50,980 --> 06:42:52,448 IT WAS MENTIONED EARLIER TODAY 11083 06:42:52,515 --> 06:42:53,649 THE DIFFERENCE BETWEEN WHAT YOU 11084 06:42:53,716 --> 06:42:54,850 THINK USUAL COLLEAGUES ARE DOING 11085 06:42:54,917 --> 06:42:57,920 AND WHAT YOU'RE ACTUALLY DOING. 11086 06:42:57,987 --> 06:43:00,556 WHEN WE SURVEYED PEOPLE THEY 11087 06:43:00,623 --> 06:43:03,325 SAID, WELL, I'M WORRYING ABOUT 11088 06:43:03,392 --> 06:43:08,197 READERSHIP BUT MY COLLEAGUES ARE 11089 06:43:08,264 --> 06:43:11,333 WORRIED ABOUT JOURNAL IMPACT 11090 06:43:11,400 --> 06:43:16,872 FACTOR AND PRESTIGE. 11091 06:43:16,939 --> 06:43:18,474 NON-TENURED WERE MOST WORRIED. 11092 06:43:18,541 --> 06:43:21,377 THEY ARE WORRIED ABOUT JOURNAL 11093 06:43:21,443 --> 06:43:26,215 IMPACT FACTOR AND PRESTIGE, 11094 06:43:26,282 --> 06:43:27,483 GOING INTO THEIR EVALUATION. 11095 06:43:27,550 --> 06:43:31,020 WE LOOK A LOOK AT 11096 06:43:31,086 --> 06:43:31,954 NON-TRADITIONAL OUTPUTS. 11097 06:43:32,021 --> 06:43:34,056 WHAT ELSE IS IN THERE? 11098 06:43:34,123 --> 06:43:35,591 AND UNFORTUNATELY WE FOUND THAT 11099 06:43:35,658 --> 06:43:38,193 UP TO 95% OF THESE INSTITUTIONS 11100 06:43:38,260 --> 06:43:42,598 ARE MENTIONING TRADITION 11101 06:43:42,665 --> 06:43:45,134 OUTPUTS, VERY FEW MENTIONING 11102 06:43:45,200 --> 06:43:45,534 NON-TRADITIONAL. 11103 06:43:45,601 --> 06:43:47,102 FOR PRE-PRINTS FOR R-TYPE IT'S 11104 06:43:47,169 --> 06:43:47,536 20%. 11105 06:43:47,603 --> 06:43:51,674 WHEN YOU LOOK AT M TYPE AND 11106 06:43:51,740 --> 06:43:54,109 B-TYPE, IT'S AS LOW AS 5% 11107 06:43:54,176 --> 06:43:55,311 MENTIONING PRE-PRINTS IN WORKING 11108 06:43:55,377 --> 06:43:56,378 PAPERS. 11109 06:43:56,445 --> 06:43:57,413 WHY IS THAT IMPORTANT? 11110 06:43:57,479 --> 06:43:59,782 WHAT DOES HAVE IT TO DO WITH 11111 06:43:59,848 --> 06:44:00,883 PUBLICATION BIAS? 11112 06:44:00,950 --> 06:44:03,419 THESE THINGS ARE PUSHING FACULTY 11113 06:44:03,485 --> 06:44:07,823 TOWARDS JOURNALS THAT ARE LESS 11114 06:44:07,890 --> 06:44:09,625 LIKELY TO ACCEPT NULL RESULTS, 11115 06:44:09,692 --> 06:44:11,193 AND TWO, SOME OF MIGHT HAVE OWN 11116 06:44:11,260 --> 06:44:20,235 RESULTS ENDED UP AND STATE IN A 11117 06:44:20,302 --> 06:44:21,670 PRE-PRINT. 11118 06:44:21,737 --> 06:44:23,539 MINE COULDN'T GET PUBLISHED, I 11119 06:44:23,606 --> 06:44:24,974 PREFERRED SHARING RATHER THAN 11120 06:44:25,040 --> 06:44:26,041 FILE DRAWER IT. 11121 06:44:26,108 --> 06:44:28,210 IF IT DOESN'T GET RECOGNIZED, I 11122 06:44:28,277 --> 06:44:30,212 DON'T GET CREDIT. 11123 06:44:30,279 --> 06:44:32,414 AND WE KNOW THIS IS HAPPENING. 11124 06:44:32,481 --> 06:44:34,883 WE ACTUALLY HAVE DATA NOW 11125 06:44:34,950 --> 06:44:37,953 SHOWING HOW THIS SYSTEM IS 11126 06:44:38,020 --> 06:44:39,421 PUSHING RESEARCHERS AWAY FROM 11127 06:44:39,488 --> 06:44:40,990 PUBLISHING NULL RESULTS AND 11128 06:44:41,056 --> 06:44:43,592 TOWARDS PRESTIGE AND JOURNAL 11129 06:44:43,659 --> 06:44:44,526 IMPACT FACTOR, ALL THESE OTHER 11130 06:44:44,593 --> 06:44:50,099 THINGS THAT ARE NOT GOOD 11131 06:44:50,165 --> 06:44:53,168 INDICATOR AND NOT EMBODYING 11132 06:44:53,235 --> 06:44:55,270 VALUES. 11133 06:44:55,337 --> 06:44:56,905 ACCELERATING KNOWLEDGE, PUBLIC 11134 06:44:56,972 --> 06:44:59,842 GOOD, OUR CRITERIA ARE IN DIRECT 11135 06:44:59,908 --> 06:45:01,710 CONTRAST TO THOSE. 11136 06:45:01,777 --> 06:45:03,345 THAT'S THE MESSAGE THAT I WANT 11137 06:45:03,412 --> 06:45:05,481 TO PUT OUT THERE. 11138 06:45:05,547 --> 06:45:06,815 THOSE ARE THE BARRIERS. 11139 06:45:06,882 --> 06:45:09,351 HOW DO WE GET PAST THAT? 11140 06:45:09,418 --> 06:45:11,153 WE NEED TO REFOCUS, GET AWAY 11141 06:45:11,220 --> 06:45:12,755 FROM PRODUCTS AND COUNTS AND GO 11142 06:45:12,821 --> 06:45:14,490 BACK TO WHAT ARE THE ACTUAL 11143 06:45:14,556 --> 06:45:15,858 VALUES, WHAT ARE THE THINGS WE 11144 06:45:15,924 --> 06:45:17,559 WANT TO SEE, WHAT ARE THE 11145 06:45:17,626 --> 06:45:18,661 BEHAVIORS WE WANT TO SEE FROM 11146 06:45:18,727 --> 06:45:19,962 OUR FACULTY BECAUSE UNTIL WE DO 11147 06:45:20,029 --> 06:45:25,167 THAT WE'RE GOING TO CONTINUE 11148 06:45:25,234 --> 06:45:28,937 FOCUSING ON BIBLIOMETRICS AND 11149 06:45:29,004 --> 06:45:34,343 COUNTS, AND ALL OF THAT IS 11150 06:45:34,410 --> 06:45:35,310 GOING TO BE INCENTIVIZED. 11151 06:45:35,377 --> 06:45:37,279 >> CAN YOU HEAR ME IF I SPEAK 11152 06:45:37,346 --> 06:45:38,247 FROM HERE? 11153 06:45:38,313 --> 06:45:38,681 OKAY. 11154 06:45:38,747 --> 06:45:43,252 JUST TO SHAKE IT UP. 11155 06:45:43,318 --> 06:45:47,423 SO, I WANT TO THANK THE FOLKS 11156 06:45:47,489 --> 06:45:52,127 FOR INVITING ME TO BE PART OF 11157 06:45:52,194 --> 06:45:53,228 THIS VITAL DISCUSSION. 11158 06:45:53,295 --> 06:45:57,499 AND I WILL BE PROVIDING A 11159 06:45:57,566 --> 06:46:00,135 PERSPECTIVE OF MIGHT HAVE ROLE 11160 06:46:00,202 --> 06:46:00,836 AS AN INSTITUTIONAL OFFICIAL 11161 06:46:00,903 --> 06:46:04,840 WHICH IS THAT OF A VICE 11162 06:46:04,907 --> 06:46:07,176 PRESIDENT FOR RESEARCH AT 11163 06:46:07,242 --> 06:46:10,112 VIRGINIA EQUALITY UNIVERSITY, AS 11164 06:46:10,179 --> 06:46:11,847 ROBERT VERY ELEGANTLY TALKED, A 11165 06:46:11,914 --> 06:46:16,652 LARGE PART OF OUR ROLE IN THIS 11166 06:46:16,719 --> 06:46:18,987 CAPACITY IS TO REALLY FACILITATE 11167 06:46:19,054 --> 06:46:20,923 AND PROMOTE THE RESEARCH THAT IS 11168 06:46:20,989 --> 06:46:25,294 DONE BY OUR FACULTY AND STUDENTS 11169 06:46:25,360 --> 06:46:27,262 AND STAFF ACROSS VARIOUS 11170 06:46:27,329 --> 06:46:27,596 DISCIPLINES. 11171 06:46:27,663 --> 06:46:30,432 BUT IN MY COMMENTS TODAY 11172 06:46:30,499 --> 06:46:32,735 PROBABLY I WILL PEPPER SOME OF 11173 06:46:32,801 --> 06:46:34,737 MY STATEMENTS WITH MY PAST 11174 06:46:34,803 --> 06:46:37,139 EXPERIENCES AS BEING A 11175 06:46:37,206 --> 06:46:39,675 RESEARCHER AS WELL OVER 30 YEARS 11176 06:46:39,742 --> 06:46:41,310 PREVIOUSLY AT UNIVERSITY OF 11177 06:46:41,376 --> 06:46:47,750 MINNESOTA, PRIOR TO THAT IN 11178 06:46:47,816 --> 06:46:51,053 CALIFORNIA, IN SAN DIEGO. 11179 06:46:51,120 --> 06:46:52,654 SO WHEN WE TALK ABOUT WHAT IS IT 11180 06:46:52,721 --> 06:46:55,758 WE WANT TO PROMOTE AT THE 11181 06:46:55,824 --> 06:46:56,859 HIGHEST LEVEL, OBVIOUSLY WE WANT 11182 06:46:56,925 --> 06:47:00,763 TO SUPPORT THE MOST HIGHEST 11183 06:47:00,829 --> 06:47:01,463 QUALITY RESEARCH THAT HAS THE 11184 06:47:01,530 --> 06:47:03,232 MOST IMPACT BECAUSE AFTER ALL 11185 06:47:03,298 --> 06:47:06,668 SEVERAL OF US ARE PUBLICLY 11186 06:47:06,735 --> 06:47:07,436 FUNDED AND STATE FUNDED 11187 06:47:07,503 --> 06:47:08,470 INSTITUTIONS, SO THIS IS PUBLIC 11188 06:47:08,537 --> 06:47:09,838 TRUST THAT IS IMPORTANT. 11189 06:47:09,905 --> 06:47:11,874 BY THE WAY WE HAVEN'T TALKED 11190 06:47:11,940 --> 06:47:13,709 ABOUT IT, IN THE UNITED STATES 11191 06:47:13,776 --> 06:47:16,044 AND ELSEWHERE THE PUBLIC 11192 06:47:16,111 --> 06:47:17,513 SENTIMENT AND HIGHER ED IS NOT 11193 06:47:17,579 --> 06:47:19,181 WHERE IT NEEDS TO BE. 11194 06:47:19,248 --> 06:47:21,617 AND I THINK ONE REDEEMING 11195 06:47:21,683 --> 06:47:23,886 FEATURE IS THAT INSTITUTIONS OF 11196 06:47:23,952 --> 06:47:26,555 HIGHER ED THAT DO RESEARCH HAVE 11197 06:47:26,622 --> 06:47:29,024 A GREATER VALUE IN THE PUBLIC 11198 06:47:29,091 --> 06:47:29,224 SPACE. 11199 06:47:29,291 --> 06:47:32,628 SO, I THINK IT IS INCREDIBLY 11200 06:47:32,694 --> 06:47:35,497 IMPORTANT WE NOT ONLY FACILITATE 11201 06:47:35,564 --> 06:47:36,999 AND CONDUCT HIGHEST QUALITY 11202 06:47:37,065 --> 06:47:38,534 RESEARCH THAT'S MOST IMPACTFUL 11203 06:47:38,600 --> 06:47:40,269 BUT HAS BEEN STATED SEVERAL 11204 06:47:40,335 --> 06:47:42,604 TIMES TODAY NEEDS TO BE DONE 11205 06:47:42,671 --> 06:47:44,773 WITH RIGOR AND REPRODUCIBILITY. 11206 06:47:44,840 --> 06:47:49,278 AND WITH TRANSPARENCY. 11207 06:47:49,344 --> 06:47:50,813 AND ETHICALLY DONE, REALLY VITAL 11208 06:47:50,879 --> 06:47:52,447 ELEMENTS TO IT. 11209 06:47:52,514 --> 06:47:56,151 NOW, IF YOU ASK WHAT WE AS 11210 06:47:56,218 --> 06:47:57,019 RESEARCH ADMINISTRATORS ARE 11211 06:47:57,085 --> 06:48:02,524 DOING TO PROMOTE THIS WHOLE 11212 06:48:02,591 --> 06:48:05,594 CONCEPT OF THE HYPOTHESIS TODAY 11213 06:48:05,661 --> 06:48:07,229 THAT NULL STUDIES, WE HAVE 11214 06:48:07,296 --> 06:48:09,331 CHALLENGES, BARRIERS, BUT WE'RE 11215 06:48:09,398 --> 06:48:11,366 MAKING PROGRESS, BOTH OF US WILL 11216 06:48:11,433 --> 06:48:15,971 BE THE FIRST TO ADMIT AT AN 11217 06:48:16,038 --> 06:48:18,173 INSTITUTIONAL LEVEL THAT'S NOT 11218 06:48:18,240 --> 06:48:21,043 THE FIRST THING COMING TO OUR 11219 06:48:21,109 --> 06:48:22,144 ROLES AND RESPONSIBILITIES TO 11220 06:48:22,211 --> 06:48:25,080 PUSH AND PROMOTE. 11221 06:48:25,147 --> 06:48:27,149 FOR ALL THE REASONS ERIN TALKED 11222 06:48:27,216 --> 06:48:27,449 ABOUT. 11223 06:48:27,516 --> 06:48:30,219 IT'S NOT TRUE WE'RE NOT DOING 11224 06:48:30,285 --> 06:48:31,553 THINGS ADMINISTRATIVELY IN 11225 06:48:31,620 --> 06:48:33,956 MULTIPLE DIFFERENT LEVELS AT OUR 11226 06:48:34,022 --> 06:48:36,992 INSTITUTION, FOR EXAMPLE, WE 11227 06:48:37,059 --> 06:48:38,861 HAVE THIS DATA SCIENCE LAB THAT 11228 06:48:38,927 --> 06:48:42,531 PROMOTES ALL THE THINGS THAT YOU 11229 06:48:42,598 --> 06:48:47,436 GUYS TALKED ABOUT, 11230 06:48:47,502 --> 06:48:49,171 PRE-REGISTRATIONS, REGISTERED 11231 06:48:49,238 --> 06:48:50,939 REPORTS, VARIETY OF TRAINING, 11232 06:48:51,006 --> 06:48:52,474 RESPONSIBLE CONDUCT OF RESEARCH, 11233 06:48:52,541 --> 06:48:55,677 GRANT ACADEMIES, THERE ARE ALL 11234 06:48:55,744 --> 06:48:57,946 THESE THINGS TAUGHT AND WE'RE 11235 06:48:58,013 --> 06:49:00,916 PLEASED WITH THE NINDS BECAUSE 11236 06:49:00,983 --> 06:49:02,451 OUR UNIVERSITY WAS PART OF THE 11237 06:49:02,517 --> 06:49:07,322 INAUGURAL GROUP OF THE FIRST 11238 06:49:07,389 --> 06:49:07,923 FIVE RIGOR CHAMPIONSHIP PRIZE, 11239 06:49:07,990 --> 06:49:09,424 THANK YOU. 11240 06:49:09,491 --> 06:49:11,326 SO IT'S GOING TO BE AN IMPORTANT 11241 06:49:11,393 --> 06:49:13,595 INAUGURAL GROUP TO LOOK AT THIS. 11242 06:49:13,662 --> 06:49:15,864 BUT IT'S NOT JUST THAT IT'S 11243 06:49:15,931 --> 06:49:16,765 THESE FIVE INSTITUTIONS. 11244 06:49:16,832 --> 06:49:19,501 IT'S ALL OF US WHO CARE ABOUT 11245 06:49:19,568 --> 06:49:24,206 AND I THINK FUNDAMENTALLY ALL OF 11246 06:49:24,273 --> 06:49:26,608 US UNDERSTAND AND APPRECIATEED 11247 06:49:26,675 --> 06:49:28,277 BENEFITS OF BROADENING, I 11248 06:49:28,343 --> 06:49:35,584 SUSPECT THE STANDARDS OF WHAT 11249 06:49:35,651 --> 06:49:36,885 SHOULD QUALIFY IN TERMS OF WHAT 11250 06:49:36,952 --> 06:49:38,420 SHOULD BE PUBLISHED. 11251 06:49:38,487 --> 06:49:44,259 WE UNDERSTAND BY NOT PUBLISHING 11252 06:49:44,326 --> 06:49:44,993 NULL STUDIES, WE CREATE 11253 06:49:45,060 --> 06:49:46,128 SIGNIFICANT GAP IN KNOWLEDGE 11254 06:49:46,194 --> 06:49:47,629 BECAUSE THAT NEVER GETS THERE. 11255 06:49:47,696 --> 06:49:56,138 THAT'S BEEN STATED MULTIPLE 11256 06:49:56,204 --> 06:49:56,338 TIMES. 11257 06:49:56,405 --> 06:49:57,105 INSTITUTIONALLY AND FEDERALLY, 11258 06:49:57,172 --> 06:49:59,508 THE AMOUNT OF RESOURCES WE'RE 11259 06:49:59,574 --> 06:50:01,877 WASTING BY REPEATING STUDIES 11260 06:50:01,944 --> 06:50:04,146 THAT ONLY COULD HAVE DONE US A 11261 06:50:04,212 --> 06:50:07,082 LOT OF GOOD HAVE THEY BEEN 11262 06:50:07,149 --> 06:50:08,016 PUBLISHED. 11263 06:50:08,083 --> 06:50:09,818 YOU'RE A YOUNG RESEARCHER 11264 06:50:09,885 --> 06:50:12,554 STARTING ON A NEW RESEARCH LINE, 11265 06:50:12,621 --> 06:50:13,755 WOULDN'T YOU BENEFIT IF MUCH OF 11266 06:50:13,822 --> 06:50:16,458 THE INFORMATION HAD BEEN IN THE 11267 06:50:16,525 --> 06:50:17,159 PUBLIC DOMAIN ALREADY? 11268 06:50:17,225 --> 06:50:19,962 AND THEN YOU COULD HAVE CHARTED 11269 06:50:20,028 --> 06:50:23,999 YOUR COURSE ELSEWHERE. 11270 06:50:24,066 --> 06:50:25,467 IN TALKING TO FACULTY AND IN 11271 06:50:25,534 --> 06:50:28,837 TALKING TO A NUMBER OF PEOPLE IN 11272 06:50:28,904 --> 06:50:30,839 TERMS OF WHERE WE ARE AND I 11273 06:50:30,906 --> 06:50:35,544 THINK CULTURALLY WE'RE NOT WHERE 11274 06:50:35,610 --> 06:50:38,113 WE NEED TO BE, AND WHAT ARE THE 11275 06:50:38,180 --> 06:50:41,450 CHALLENGES, AND I THINK YOU ALL 11276 06:50:41,516 --> 06:50:42,217 HAVE SUMMARIZED FOUR 11277 06:50:42,284 --> 06:50:44,553 FUNDAMENTALS, I WANT TO REPEAT 11278 06:50:44,619 --> 06:50:44,753 THAT. 11279 06:50:44,820 --> 06:50:46,355 ONE IS THE PUBLICATION BIAS AND 11280 06:50:46,421 --> 06:50:47,756 YOU'VE TALKED ABOUT IT SO I 11281 06:50:47,823 --> 06:50:48,890 WON'T SAY ANYTHING MORE ABOUT 11282 06:50:48,957 --> 06:50:49,124 IT. 11283 06:50:49,191 --> 06:50:52,194 I THINK THE LAST OF INCENTIVES 11284 06:50:52,260 --> 06:50:55,364 IS SOMETHING THAT IS STRIKING, 11285 06:50:55,430 --> 06:50:59,401 AT THE LEVEL OF FUNDING AGENCIES 11286 06:50:59,468 --> 06:51:00,635 AND ORGANIZATIONS, INCLUDING 11287 06:51:00,702 --> 06:51:02,237 NIH, I'LL SAY THAT HERE. 11288 06:51:02,304 --> 06:51:03,372 AND OTHER AGENCIES. 11289 06:51:03,438 --> 06:51:05,841 AND I THINK THAT NEEDS TO 11290 06:51:05,907 --> 06:51:06,842 CHANGE, HOW THE REVIEW PROCESS 11291 06:51:06,908 --> 06:51:09,244 IS AND ALL OF THAT. 11292 06:51:09,311 --> 06:51:11,079 AND I WOULD SAY THE 11293 06:51:11,146 --> 06:51:11,880 METHODOLOGICAL CHALLENGES THAT 11294 06:51:11,947 --> 06:51:13,749 SOME OF YOU TALKED ABOUT, THOSE 11295 06:51:13,815 --> 06:51:16,218 ARE IMPORTANT. 11296 06:51:16,284 --> 06:51:18,120 ARE THESE NULL STUDIES TRULY 11297 06:51:18,186 --> 06:51:24,359 NULL STUDIES OR ARE THEY ALSO AN 11298 06:51:24,426 --> 06:51:25,460 OUTCOME OF POTENTIALLY ERRORS? 11299 06:51:25,527 --> 06:51:28,597 THERE ARE CONSEQUENCES TO SOME 11300 06:51:28,663 --> 06:51:30,198 OF THAT. 11301 06:51:30,265 --> 06:51:35,037 FINALLY I THINK WHAT YOU WERE 11302 06:51:35,103 --> 06:51:36,905 REFERRING TO, IF YOU DON'T 11303 06:51:36,972 --> 06:51:38,740 PUBLISH YOU PERISH, HOW DO YOU 11304 06:51:38,807 --> 06:51:41,510 DEAL WITH THAT? 11305 06:51:41,576 --> 06:51:45,013 SO, A LOT WAS ASKED. 11306 06:51:45,080 --> 06:51:46,982 IN FACT WHEN THE NIH AND OTHERS 11307 06:51:47,049 --> 06:51:48,683 WERE ASKING ROBERT AND I TO 11308 06:51:48,750 --> 06:51:52,087 JOIN, WHAT IS IT THAT WE AS 11309 06:51:52,154 --> 06:51:53,955 RESEARCH ADMINISTRATORS ARE 11310 06:51:54,022 --> 06:51:55,957 DOING TO HELP AND FACILITATE 11311 06:51:56,024 --> 06:51:58,427 WITH THIS PROCESS, LUCKILY THIS 11312 06:51:58,493 --> 06:51:59,761 IS ABOUT BARRIERS, WE WON'T TALK 11313 06:51:59,828 --> 06:52:04,032 ABOUT SOLUTIONS RIGHT NOW. 11314 06:52:04,099 --> 06:52:04,966 THAT'S FOR TOMORROW. 11315 06:52:05,033 --> 06:52:06,668 BUT WE NEED TO BE HONEST ABOUT 11316 06:52:06,735 --> 06:52:06,968 IT. 11317 06:52:07,035 --> 06:52:08,103 IT'S NOT THAT THAT'S THE FIRST 11318 06:52:08,170 --> 06:52:11,473 THING THAT COMES TO MIND WHEN WE 11319 06:52:11,540 --> 06:52:15,644 WAKE UP EACH MORNING, HOUR JOBS 11320 06:52:15,710 --> 06:52:17,379 HAVE OTHER ELEMENTS TO IT BUT 11321 06:52:17,446 --> 06:52:20,849 IT'S NOT AT ALL LOST ON US. 11322 06:52:20,916 --> 06:52:25,287 BUT I THINK FUND MENTAL AS ERIN 11323 06:52:25,353 --> 06:52:27,255 WAS TALKING AS MUCH AS WE VALUE 11324 06:52:27,322 --> 06:52:30,559 IMPORTANCE OF ALL OF THIS, IF 11325 06:52:30,625 --> 06:52:32,761 JOURNALS ARE NOT ACTUALLY 11326 06:52:32,828 --> 06:52:34,796 PUBLISHING NULL STUDIES THESE 11327 06:52:34,863 --> 06:52:35,897 IMPACTFUL JOURNALS THAT MAKE A 11328 06:52:35,964 --> 06:52:38,233 DIFFERENCE IN THE PROMOTION AND 11329 06:52:38,300 --> 06:52:40,936 TENURE OF FACULTY, IT IS GOING 11330 06:52:41,002 --> 06:52:43,572 TO BE A CONTINUING CHALLENGE FOR 11331 06:52:43,638 --> 06:52:45,307 US AS INSTITUTIONS TO DO 11332 06:52:45,373 --> 06:52:48,977 SOMETHING ABOUT IT. 11333 06:52:49,044 --> 06:52:51,012 THAT HAS TO CHANGE. 11334 06:52:51,079 --> 06:52:54,749 I THINK I WOULD CONFESS AND SAY 11335 06:52:54,816 --> 06:52:58,286 THAT THERE IS -- BY NOT 11336 06:52:58,353 --> 06:53:01,089 ENCOURAGING PUBLICATION OF NULL 11337 06:53:01,156 --> 06:53:02,224 RESULTS, WE ARE DISINCENTIVIZING 11338 06:53:02,290 --> 06:53:03,358 OUR SELF WITHOUT REALIZING IT. 11339 06:53:03,425 --> 06:53:07,195 THERE NEEDS TO BE IN THAT SENSE 11340 06:53:07,262 --> 06:53:08,997 A COMPLETE OVERHAUL HOW WE LOOK 11341 06:53:09,064 --> 06:53:11,600 AT HOW RESEARCH IS REWARDED AND 11342 06:53:11,666 --> 06:53:11,833 VALUED. 11343 06:53:11,900 --> 06:53:13,802 AND I THINK IT IS REALLY 11344 06:53:13,869 --> 06:53:16,738 FUNDAMENTAL TO A LOT OF WHAT WE 11345 06:53:16,805 --> 06:53:19,474 DO, AND SO I WOULD SAY THAT THE 11346 06:53:19,541 --> 06:53:20,742 CHALLENGES THAT WE HAVE, 11347 06:53:20,809 --> 06:53:25,380 BARRIERS THAT WE HAVE, ARE TRULY 11348 06:53:25,447 --> 06:53:26,848 AT A SYSTEMIC LEVEL. 11349 06:53:26,915 --> 06:53:31,386 I MEAN, FIRST AND FOREMOST, THE 11350 06:53:31,453 --> 06:53:33,788 AGENCIES AND ORGANIZATIONS THAT 11351 06:53:33,855 --> 06:53:35,023 ARE FUNDING, THAT HAS TO CHANGE 11352 06:53:35,090 --> 06:53:36,625 WHERE THAT IS VALUED. 11353 06:53:36,691 --> 06:53:37,826 IF THAT DOESN'T CHANGE WE'RE 11354 06:53:37,893 --> 06:53:40,462 STILL AT THE SAME PLACE. 11355 06:53:40,529 --> 06:53:42,230 THE JOURNALS, A LOT WAS TALKED 11356 06:53:42,297 --> 06:53:44,166 ABOUT THE OPPORTUNITIES ARE 11357 06:53:44,232 --> 06:53:46,134 THERE BUT AS HAS BEEN SAID THE 11358 06:53:46,201 --> 06:53:47,569 IMPACT JOURNAL CONCEPT HAS NOT 11359 06:53:47,636 --> 06:53:48,570 GONE AWAY. 11360 06:53:48,637 --> 06:53:50,772 AND THEN OF COURSE AT 11361 06:53:50,839 --> 06:53:55,977 INSTITUTIONAL LEVEL, HOW ARE WE 11362 06:53:56,044 --> 06:53:58,146 GOING TO RECOGNIZE, SUPPORT, 11363 06:53:58,213 --> 06:54:02,350 REWORD ALL RESEARCH NO MATTER 11364 06:54:02,417 --> 06:54:03,018 WHERE THESE JOURNALS ARE AND AS 11365 06:54:03,084 --> 06:54:05,187 LONG AS THEY ARE DONE ETHICALLY 11366 06:54:05,253 --> 06:54:12,160 AND THEY CONTINUE TO ADD VALUE. 11367 06:54:12,227 --> 06:54:12,994 IN CLOSING, I WANT TO SAY I 11368 06:54:13,061 --> 06:54:23,071 DON'T THINK THERE'S ANYBODY HERE 11369 06:54:23,138 --> 06:54:23,738 WHO FUNDAMENTALLY DISAGREES WE 11370 06:54:23,805 --> 06:54:26,441 NEED TO DO MORE TO PROMOTE HOW 11371 06:54:26,508 --> 06:54:31,680 FACULTY CAN PROMOTE NULL STUDIES 11372 06:54:31,746 --> 06:54:34,849 BUT THIS CONCEPT OF PROMOTION 11373 06:54:34,916 --> 06:54:36,418 AND TENURE AND EMPHASIS ON HIGH 11374 06:54:36,484 --> 06:54:46,928 IMPACT JOURNALS IS A HUGE 11375 06:54:51,099 --> 06:54:52,267 BARRIER NOBEL LAWYERATES WILL 11376 06:54:52,334 --> 06:54:53,268 TELL YOU SEVERAL FEEL IF THEY 11377 06:54:53,335 --> 06:55:00,575 WERE IN THE SAME CLIMATE TODAY 11378 06:55:00,642 --> 06:55:02,978 THEY WOULD NOT HAVE THE SAME 11379 06:55:03,044 --> 06:55:03,211 TENURE. 11380 06:55:03,278 --> 06:55:05,180 WE HAVE TO THINK ABOUT THIS. 11381 06:55:05,247 --> 06:55:10,885 THE LAST THING I'LL SAY, WE HAVE 11382 06:55:10,952 --> 06:55:12,921 TO BE CAREFUL ABOUT IMPACT OF 11383 06:55:12,988 --> 06:55:13,655 NULL FINDINGS. 11384 06:55:13,722 --> 06:55:16,925 I'M NOT SAYING WHERE ERRORS ARE 11385 06:55:16,992 --> 06:55:18,493 DONE BUT NULL STUDIES PUBLISHED 11386 06:55:18,560 --> 06:55:28,303 BASED ON CERTAINLY FUNDAMENTALS, 11387 06:55:28,370 --> 06:55:29,204 AGENCIES CAN FEEL THERE'S NO 11388 06:55:29,271 --> 06:55:31,906 REASON TO SUPPORT THIS. 11389 06:55:31,973 --> 06:55:32,807 JOURNALS TAKE THE ATTITUDE 11390 06:55:32,874 --> 06:55:35,577 THERE'S NO REASON TO PUBLISH THE 11391 06:55:35,644 --> 06:55:39,347 RESULTS, LATER MORE IN DEPTH 11392 06:55:39,414 --> 06:55:41,283 STUDIES BY SOMEONE OPENED UP, IT 11393 06:55:41,349 --> 06:55:49,424 OPENS UP A NEW AREA OF WORKING 11394 06:55:49,491 --> 06:55:52,127 YOU ABOUT PEOPLE HAVE LOST TIME, 11395 06:55:52,193 --> 06:55:53,695 CAREERS JEOPARDIZED, PEOPLE 11396 06:55:53,762 --> 06:55:58,033 FORCED TO DO OTHER THINGS. 11397 06:55:58,099 --> 06:56:05,140 WE NEED TO DO MORE, SO LOOKING 11398 06:56:05,206 --> 06:56:06,241 FORWARD TO YOUR QUESTIONS AND 11399 06:56:06,308 --> 06:56:07,676 DISCUSSION TOMORROW WHERE WE 11400 06:56:07,742 --> 06:56:12,914 TALK MORE ABOUT SOLUTIONS. 11401 06:56:12,981 --> 06:56:14,883 THANK YOU. 11402 06:56:14,949 --> 06:56:19,821 >> WE SEE HIGH LEVEL 11403 06:56:19,888 --> 06:56:21,256 ADMINISTRATIVE PEOPLE CAN REALLY 11404 06:56:21,323 --> 06:56:21,589 TALK. 11405 06:56:21,656 --> 06:56:25,060 WE'VE DONE A GOOD JOB PAINTING 11406 06:56:25,126 --> 06:56:27,529 THE PROBLEMS, I'D LOVE TO HEAR 11407 06:56:27,595 --> 06:56:38,006 YOUR QUESTIONS AND NOT. 11408 06:56:41,176 --> 06:56:43,111 LET'S BEGIN. 11409 06:56:43,178 --> 06:56:48,983 >> ARE INDIVIDUAL NULL NEGATIVE 11410 06:56:49,050 --> 06:56:49,684 STUDIES CITED. 11411 06:56:49,751 --> 06:56:50,585 JOURNAL IMPACT FACTOR, 11412 06:56:50,652 --> 06:56:53,188 DISCUSSION FOR 20 YEARS IS THAT 11413 06:56:53,254 --> 06:56:55,857 ARTICLE CITED IN THE FIELD 11414 06:56:55,924 --> 06:56:57,459 CONFECTION, MORE NUANCED WAY OF 11415 06:56:57,525 --> 06:56:58,360 LOOKING AT IT. 11416 06:56:58,426 --> 06:57:02,530 I DIDN'T SEE THAT IN YOUR 11417 06:57:02,597 --> 06:57:02,764 SLIDES. 11418 06:57:02,831 --> 06:57:04,766 >> IN TERMS OF TENURE AND 11419 06:57:04,833 --> 06:57:06,201 PROMOTION DOCUMENTS? 11420 06:57:06,267 --> 06:57:08,737 YEAH, THEY DO FOCUS ON CITATION 11421 06:57:08,803 --> 06:57:15,744 COUNTS, NOT JUST JOURNAL LEVEL 11422 06:57:15,810 --> 06:57:16,344 METRICS. 11423 06:57:16,411 --> 06:57:18,179 I SUPPOSE THAT WOULD BE INCLUDED 11424 06:57:18,246 --> 06:57:18,947 IN THAT. 11425 06:57:19,013 --> 06:57:20,849 THERE WAS A QUESTION ABOUT IF 11426 06:57:20,915 --> 06:57:27,756 YOU GET A NULL STUDY PUBLISHED 11427 06:57:27,822 --> 06:57:29,724 DOES IT COUNT. 11428 06:57:29,791 --> 06:57:32,127 IT'S A PUBLICATION BUT ARTICLE 11429 06:57:32,193 --> 06:57:34,429 LEVEL METRICS, NOT CITED, YOU 11430 06:57:34,496 --> 06:57:35,230 MIGHT GET -- 11431 06:57:35,296 --> 06:57:40,001 >> IF IT DOESN'T GET CITED. 11432 06:57:40,068 --> 06:57:40,769 >> YES. 11433 06:57:40,835 --> 06:57:42,737 >> I DON'T KNOW WHETHER PEOPLE 11434 06:57:42,804 --> 06:57:43,204 CITE NEGATIVE STUDIES. 11435 06:57:43,271 --> 06:57:44,172 >> SOMEBODY MENTIONED THERE 11436 06:57:44,239 --> 06:57:47,575 MIGHT BE SOME STUDIES SHOWING 11437 06:57:47,642 --> 06:57:50,178 THEY GET CITED LESS. 11438 06:57:50,245 --> 06:57:52,580 >> ESPECIALLY IN CLINICAL TRIALS 11439 06:57:52,647 --> 06:58:01,156 NEGATIVE STUDIES ARE OFTEN 11440 06:58:01,222 --> 06:58:01,456 CITED. 11441 06:58:01,523 --> 06:58:02,991 >> UC SANTA CRUZ. 11442 06:58:03,057 --> 06:58:06,261 I'M INTERESTED IN THIS QUESTION 11443 06:58:06,327 --> 06:58:07,462 ABOUT INCENTIVIZING AT THE 11444 06:58:07,529 --> 06:58:08,396 INSTITUTIONAL LEVEL, BECAUSE I'M 11445 06:58:08,463 --> 06:58:10,365 A SCIENTIST WHO GOES THROUGH 11446 06:58:10,432 --> 06:58:15,270 TENURE AND PROMOTION, I'M ALSO A 11447 06:58:15,336 --> 06:58:16,104 FACULTY EQUITY ADVOCATE AT 11448 06:58:16,171 --> 06:58:17,238 UNIVERSITY OF CALIFORNIA SANTA 11449 06:58:17,305 --> 06:58:19,441 SANTA CRUZ, TRYING TO MANGE 11450 06:58:19,507 --> 06:58:23,044 REVIEW MORE EQUITABLE AFTER 11451 06:58:23,111 --> 06:58:27,549 INITIALLY FOCUSING ON HIRING. 11452 06:58:27,615 --> 06:58:31,953 IN THIS ANALYSIS, ANALYSIS ABOUT 11453 06:58:32,020 --> 06:58:33,588 INDIVIDUALS RESPONDED TO THEIR 11454 06:58:33,655 --> 06:58:34,989 ASSESSMENT DURING TENURE AND 11455 06:58:35,056 --> 06:58:36,057 PROMOTION YOU MENTIONED SOME 11456 06:58:36,124 --> 06:58:38,827 PEOPLE SAW IT AS LESS THAN FAIR, 11457 06:58:38,893 --> 06:58:40,695 MORE UNFAIR, MORE UNFAIR AS IT 11458 06:58:40,762 --> 06:58:42,831 WENT UP THE ACADEMIC LATTER. 11459 06:58:42,897 --> 06:58:45,366 WAS THERE ANY -- YOU MENTIONED 11460 06:58:45,433 --> 06:58:52,073 WOMEN WERE DOING MORE SERVICE, 11461 06:58:52,140 --> 06:58:53,608 WAS THERE DEMOGRAPHIC DATA, MORE 11462 06:58:53,675 --> 06:58:55,677 UNFAIR OR MORE FAIR DURING THAT 11463 06:58:55,743 --> 06:58:56,711 PROCESS, BECAUSE ONE THING THAT 11464 06:58:56,778 --> 06:58:59,180 WE HAVE TO THINK ABOUT WHEN 11465 06:58:59,247 --> 06:59:01,583 TALKING ABOUT HOW TO ASSESS NULL 11466 06:59:01,649 --> 06:59:03,551 RESULTS AND SCIENCE IN GENERAL 11467 06:59:03,618 --> 06:59:10,058 IS THAT OFTEN THAT HAPPENS AT 11468 06:59:10,124 --> 06:59:12,193 HAPPEN HAZARD LEVEL, OFTEN 11469 06:59:12,260 --> 06:59:14,095 INEQUITABLY WHERE PEOPLE FROM 11470 06:59:14,162 --> 06:59:15,196 HISTORICALLY MARGINALIZED GROUPS 11471 06:59:15,263 --> 06:59:18,199 ARE BEARING THE BRUNT OF BEING 11472 06:59:18,266 --> 06:59:20,401 INNOVATORS AND NOT BEING 11473 06:59:20,468 --> 06:59:21,169 RECOGNIZED FOR THAT INNOVATION 11474 06:59:21,236 --> 06:59:23,571 AND SO I'D BE INTERESTED IF ANY 11475 06:59:23,638 --> 06:59:26,908 OF THE WORK THAT WAS REFERENCED 11476 06:59:26,975 --> 06:59:36,718 HERE TOOK INTO ACCOUNT FACULTY 11477 06:59:36,784 --> 06:59:38,019 DEMOGRAPHICS. 11478 06:59:38,086 --> 06:59:43,358 >> THAT WAS UNIVERSITY LEVEL, WE 11479 06:59:43,424 --> 06:59:45,560 CAN GENERALIZE ACROSS GENERAL 11480 06:59:45,627 --> 06:59:46,027 POPULATION. 11481 06:59:46,094 --> 06:59:48,129 BIGGEST GENDER DIFFERENCE WAS 11482 06:59:48,196 --> 06:59:50,298 SERVICE DIFFERENTIAL THERE, AND 11483 06:59:50,365 --> 06:59:54,869 WE DIDN'T SEE A SIGNIFICANT 11484 06:59:54,936 --> 06:59:59,807 DIFFERENCE BETWEEN THE PERCEIVED 11485 06:59:59,874 --> 07:00:01,743 I GUESS DIFFERENCES BETWEEN WHAT 11486 07:00:01,809 --> 07:00:03,411 WAS DEEMED FAIR BETWEEN 11487 07:00:03,478 --> 07:00:04,379 DEPARTMENT REVEL, UNIVERSITY 11488 07:00:04,445 --> 07:00:07,649 LEVEL, COLLEGE LEVEL OF 11489 07:00:07,715 --> 07:00:07,949 ASSESSMENT. 11490 07:00:08,016 --> 07:00:10,418 WE ENDED UP NOT REPORTING. 11491 07:00:10,485 --> 07:00:12,153 WE DIDN'T FOCUS ON DISPARITIES 11492 07:00:12,220 --> 07:00:14,422 BETWEEN GENDER AND/OR RACE, 11493 07:00:14,489 --> 07:00:19,827 ESPECIALLY WITH RACE, WE HAD SO 11494 07:00:19,894 --> 07:00:22,330 FEW RESPONDENTS, WE'RE A 11495 07:00:22,397 --> 07:00:23,097 PREDOMINANTLY WHITE INSTITUTION, 11496 07:00:23,164 --> 07:00:24,399 UNFORTUNATELY, WE COULDN'T BRING 11497 07:00:24,465 --> 07:00:25,967 OUT THAT DATA WITHOUT BEING, YOU 11498 07:00:26,034 --> 07:00:27,035 KNOW, PRIVACY CONCERNS AND 11499 07:00:27,101 --> 07:00:28,903 THINGS OF THAT NATURE. 11500 07:00:28,970 --> 07:00:33,908 SO I WILL SAY THAT FROM A 11501 07:00:33,975 --> 07:00:36,344 GENERALIZED, ERIN MAY HAVE MORE 11502 07:00:36,411 --> 07:00:37,512 INFORMATION, THERE'S A REALLY -- 11503 07:00:37,579 --> 07:00:40,982 THERE ARE SOME GOOD PAPERS ABOUT 11504 07:00:41,049 --> 07:00:45,453 THIS, WHERE THERE'S A LOT OF 11505 07:00:45,520 --> 07:00:47,689 ISSUES EQUITY IN SCIENTIFIC 11506 07:00:47,755 --> 07:00:49,657 PAPERS, I THINK IT'S CASSIDY'S 11507 07:00:49,724 --> 07:00:50,825 WORK, SHE DOES A LOT OF GOOD 11508 07:00:50,892 --> 07:00:51,859 PAPERS, I DON'T KNOW IF YOU 11509 07:00:51,926 --> 07:00:52,794 HEARD OF HER WORK. 11510 07:00:52,860 --> 07:00:55,830 SHE DOES A LOT OF GOOD RESEARCH 11511 07:00:55,897 --> 07:00:59,534 HOW EVEN IF YOU FIND THERE'S 11512 07:00:59,601 --> 07:01:01,536 EQUITY IN SCIENCE THERE'S 11513 07:01:01,603 --> 07:01:02,470 STILL -- PARITY IN SCIENCE, 11514 07:01:02,537 --> 07:01:04,739 STILL PROBLEMS WITH EQUITY. 11515 07:01:04,806 --> 07:01:07,775 JUST BECAUSE YOU MAYBE HAVEN'T 11516 07:01:07,842 --> 07:01:09,844 MET THAT THRESHOLD DOESN'T MEAN 11517 07:01:09,911 --> 07:01:11,746 THERE'S ACTUAL EQUITY, AND WE 11518 07:01:11,813 --> 07:01:13,247 STILL HAVEN'T GOTTEN INTO PARITY 11519 07:01:13,314 --> 07:01:14,015 YET. 11520 07:01:14,082 --> 07:01:15,316 THAT'S GOING TO TAKE 11521 07:01:15,383 --> 07:01:15,617 GENERATIONS. 11522 07:01:15,683 --> 07:01:19,520 DO YOU HAVE ANYTHING TO ADD TO 11523 07:01:19,587 --> 07:01:19,921 THAT? 11524 07:01:19,988 --> 07:01:21,255 >> MAYBE A CUSPEL RESULTS 11525 07:01:21,322 --> 07:01:25,293 BECAUSE WE DID DO SOME ANALYSIS, 11526 07:01:25,360 --> 07:01:27,128 PARTICULARLY IN SURVEY STUDY. 11527 07:01:27,195 --> 07:01:29,464 WE WERE ONLY LOOKING AT 11528 07:01:29,530 --> 07:01:33,301 DECISIONS IN TERMS OF HOW THEY 11529 07:01:33,368 --> 07:01:35,136 DECIDE WHERE TO PUBLISH. 11530 07:01:35,203 --> 07:01:36,237 I BELIEVE A SIGNIFICANT EFFECT 11531 07:01:36,304 --> 07:01:39,140 IN TERMS OF WOMEN PUBLISHING 11532 07:01:39,207 --> 07:01:41,275 LESS ACROSS THE STUDY. 11533 07:01:41,342 --> 07:01:43,678 THAT COULD BE BECAUSE THEY HAD A 11534 07:01:43,745 --> 07:01:44,412 LARGER SERVICE COMMITMENT AND 11535 07:01:44,479 --> 07:01:46,748 THERE ARE QUITE A FEW STUDIES AS 11536 07:01:46,814 --> 07:01:49,717 YOU SAY DO SHOW THAT SERVICE 11537 07:01:49,784 --> 07:01:51,352 BURDEN COMES MORE HEAVILY ON 11538 07:01:51,419 --> 07:01:52,453 WOMEN AND UNDERREPRESENTED 11539 07:01:52,520 --> 07:01:52,920 GROUPS. 11540 07:01:52,987 --> 07:01:55,490 RELATED TO THAT IN ANOTHER ONE 11541 07:01:55,556 --> 07:01:58,126 OF OUR STUDIES WE FOUND SERVICE 11542 07:01:58,192 --> 07:01:59,927 IS WHERE THE PUBLIC DIMENSIONS 11543 07:01:59,994 --> 07:02:02,296 OF THE WORK WAS GETTING DUMPED, 11544 07:02:02,363 --> 07:02:05,166 IT'S ALSO THE MOST UPDATE VALUED 11545 07:02:05,233 --> 07:02:07,101 CATEGORY IN PROMOTION AND 11546 07:02:07,168 --> 07:02:07,435 TENURE. 11547 07:02:07,502 --> 07:02:08,736 YOU KNOW, THAT CATEGORY WHERE 11548 07:02:08,803 --> 07:02:11,105 YOU'RE PUTTING ALL THE THINGS 11549 07:02:11,172 --> 07:02:15,777 THAT SUPPOSEDLY ARE HOW YOUR 11550 07:02:15,843 --> 07:02:17,645 INSTITUTION EMBODIES VALUE AND 11551 07:02:17,712 --> 07:02:20,348 MEETS MISSION STATEMENTS ABOUT 11552 07:02:20,415 --> 07:02:21,249 DIVERSITY, INCLUSIVITY, PUBLIC 11553 07:02:21,315 --> 07:02:22,650 OUTREACH GETTING DUMPED IN THE 11554 07:02:22,717 --> 07:02:25,853 LEAST VALUED CATEGORY, WHERE YOU 11555 07:02:25,920 --> 07:02:28,589 ALSO HAVE PRIMARILY WOMEN AND 11556 07:02:28,656 --> 07:02:33,394 MINORITIES TAKING THE BRUNT. 11557 07:02:33,461 --> 07:02:35,263 >> THIS IS INTERESTING. 11558 07:02:35,329 --> 07:02:37,265 BEING AT A DISCUSSION WORKSHOP 11559 07:02:37,331 --> 07:02:39,634 THAT WE TALK ABOUT UNPUBLISHED 11560 07:02:39,701 --> 07:02:41,269 RESULTS, I'VE BEEN AT A NUMBER 11561 07:02:41,335 --> 07:02:42,937 OF INSTITUTIONS, WE LOOK AT THIS 11562 07:02:43,004 --> 07:02:43,404 DATA. 11563 07:02:43,471 --> 07:02:45,907 WE DON'T PUBLISH THE DATA. 11564 07:02:45,973 --> 07:02:46,874 ALL RIGHT. 11565 07:02:46,941 --> 07:02:49,210 SO WE DO NOT PUBLISH THE DATA. 11566 07:02:49,277 --> 07:02:53,347 WHAT I WOULD SAY IS THAT IT IS 11567 07:02:53,414 --> 07:02:56,084 VERY TRUE, AND I WROTE AN 11568 07:02:56,150 --> 07:02:57,285 EDITORIAL TO VICE PRESIDENTS OF 11569 07:02:57,351 --> 07:02:59,187 RESEARCH THAT WE NEED TO BE VERY 11570 07:02:59,253 --> 07:03:01,989 CAUTIOUS, IN THE WAY WE SOLICIT 11571 07:03:02,056 --> 07:03:04,525 INDIVIDUALS TO PERFORM INTERNAL 11572 07:03:04,592 --> 07:03:06,060 SERVICE OBLIGATIONS EVEN FOR 11573 07:03:06,127 --> 07:03:12,633 RESEARCH BECAUSE WE FIND 11574 07:03:12,700 --> 07:03:13,668 OURSELVES HEAVILY WEIGHTED WOMEN 11575 07:03:13,735 --> 07:03:16,237 WITH D.E.I. INITIATIVES IN 11576 07:03:16,304 --> 07:03:21,476 GENERAL, HEAVILY WEIGHTED ON 11577 07:03:21,542 --> 07:03:21,809 MINORITIES. 11578 07:03:21,876 --> 07:03:23,678 IT'S INTERESTING THAT FULL 11579 07:03:23,745 --> 07:03:25,079 PROFESSORS TYPICALLY HAVE A 11580 07:03:25,146 --> 07:03:29,283 BIGGER VOICE SO WE'RE HEAVILY 11581 07:03:29,350 --> 07:03:30,985 WEIGHTED IN THE ASSISTANT AND 11582 07:03:31,052 --> 07:03:32,353 ASSOCIATE PROFESSORS IN SERVICE 11583 07:03:32,420 --> 07:03:33,621 OBLIGATIONS FOR INSTITUTION, 11584 07:03:33,688 --> 07:03:39,894 THOSE ARE SERVICE OBLIGATIONS 11585 07:03:39,961 --> 07:03:42,964 THAT HAVE LESSER VALUE AND WE 11586 07:03:43,030 --> 07:03:43,731 PROBABLY SHOULD THINK 11587 07:03:43,798 --> 07:03:45,166 HOLISTICALLY ABOUT THAT. 11588 07:03:45,233 --> 07:03:47,568 AND WHEN WE DO, WHEN I HAVE IT 11589 07:03:47,635 --> 07:03:49,337 AT THE INSTITUTIONS THAT I'VE 11590 07:03:49,403 --> 07:03:54,375 BEEN AT, WE'VE CHANGE THE THE 11591 07:03:54,442 --> 07:03:56,511 DYNAMIC, THE AGREEMENT WAS WITH 11592 07:03:56,577 --> 07:03:59,147 THE DEPARTMENT, HAVING SOME 11593 07:03:59,213 --> 07:04:05,520 SENIOR VICE CHAIRS OF RESEARCH 11594 07:04:05,586 --> 07:04:07,455 DEANS, PERFORM ACTIVITIES SO 11595 07:04:07,522 --> 07:04:08,756 THEIR DEPARTMENT CAN CONTINUE 11596 07:04:08,823 --> 07:04:11,893 INSTEAD OF PUTTING THE WEIGHT OF 11597 07:04:11,959 --> 07:04:14,061 RUNNING AN INSTITUTION ON THE 11598 07:04:14,128 --> 07:04:15,897 BACKS OF INDIVIDUALS THAT NEED 11599 07:04:15,963 --> 07:04:19,200 TO GET PROMOTED THROUGH THE 11600 07:04:19,267 --> 07:04:19,600 INSTITUTION. 11601 07:04:19,667 --> 07:04:24,172 SO IT'S A VERY REAL PHENOMENON. 11602 07:04:24,238 --> 07:04:24,972 THERE'S NO INTERNAL REPORTS TO 11603 07:04:25,039 --> 07:04:29,777 SHARE BUT I CAN TELL YOU IT'S 11604 07:04:29,844 --> 07:04:29,977 TRUE. 11605 07:04:30,044 --> 07:04:31,646 >> YEAH, ONE QUICK THING TOO. 11606 07:04:31,712 --> 07:04:36,651 A LOT OF TIME WE'RE GETTING -- 11607 07:04:36,717 --> 07:04:38,553 IN MY CASE, I SUSPECT OTHER 11608 07:04:38,619 --> 07:04:41,522 FOLKS, WHEN YOU'RE DOING 11609 07:04:41,589 --> 07:04:42,790 RESEARCH WITH UNDERGRADS YOU GET 11610 07:04:42,857 --> 07:04:47,962 NO RESULTS, DUMPED IN THE 11611 07:04:48,029 --> 07:04:49,597 MENTORING CATEGORY, WHICH GETS 11612 07:04:49,664 --> 07:04:50,898 DUMPED IN SERVICE, UNDERVALUED. 11613 07:04:50,965 --> 07:04:52,600 IF WE LINK BACK TO PUBLICATION 11614 07:04:52,667 --> 07:04:56,237 BIAS IS LOT IS FALLING INTO THAT 11615 07:04:56,304 --> 07:05:01,175 UNDERVALUED GROUP AND SERVICE AS 11616 07:05:01,242 --> 07:05:01,742 WELL. 11617 07:05:01,809 --> 07:05:02,510 >> UNIVERSITY OF FLORIDA. 11618 07:05:02,577 --> 07:05:05,213 A FEW THINGS ABOUT THE DISPARITY 11619 07:05:05,279 --> 07:05:06,981 IN SORT OF NUMBER OF 11620 07:05:07,048 --> 07:05:08,416 PUBLICATIONS FOR MEN, FOR WOMEN, 11621 07:05:08,482 --> 07:05:11,385 WE ALL KNOW DURING THE PANDEMIC 11622 07:05:11,452 --> 07:05:14,722 WOMEN AND PEOPLE OF COLOR AND 11623 07:05:14,789 --> 07:05:18,259 CAREGIVERS FOR CHILDREN AND 11624 07:05:18,326 --> 07:05:21,195 PARENTS WERE DISPROPORTIONATELY 11625 07:05:21,262 --> 07:05:22,063 AFFECTED. 11626 07:05:22,129 --> 07:05:25,466 EVERYBODY GOT A YEAR OF COVID 11627 07:05:25,533 --> 07:05:27,101 FOR TENURE DELAY SOME OTHER 11628 07:05:27,168 --> 07:05:28,135 THINGS WERE NOT CONSIDERED, AND 11629 07:05:28,202 --> 07:05:30,404 THAT'S A BIG PROBLEM. 11630 07:05:30,471 --> 07:05:31,606 ONE BIG PROBLEM INSTITUTIONS 11631 07:05:31,672 --> 07:05:36,210 HAVE, I WAS AT EMORY FOR TEN 11632 07:05:36,277 --> 07:05:38,179 YEARS, U.C. SOUTHWESTERN, 11633 07:05:38,246 --> 07:05:41,148 UNIVERSITY OF FLORIDA FOR FIVE 11634 07:05:41,215 --> 07:05:45,419 YEARS, EVERYTHING REALLY DOES 11635 07:05:45,486 --> 07:05:46,621 REVOLVE AROUND NUMBERS. 11636 07:05:46,687 --> 07:05:48,456 I GOT A SPACE SURVEY TO MAKE 11637 07:05:48,522 --> 07:05:49,824 SURE THE SPACE I WAS SITTING IN, 11638 07:05:49,891 --> 07:05:51,325 THE HOW MANY PAPERS HAVE YOU 11639 07:05:51,392 --> 07:05:53,594 PUBLISHED IN THE LAST THREE 11640 07:05:53,661 --> 07:05:55,363 YEARS, HOW MANY GRANTS DO YOU 11641 07:05:55,429 --> 07:05:59,567 HAVE, HOW MANY GRANTS HAVE YOU 11642 07:05:59,634 --> 07:06:00,668 SUBMITTED THAT GOT FUNDED, HOW 11643 07:06:00,735 --> 07:06:02,069 MANY GRANTS HAVE YOU SUBMITTED 11644 07:06:02,136 --> 07:06:05,039 THAT DIDN'T GET FUNDED. 11645 07:06:05,106 --> 07:06:06,207 NO QUESTIONS ABOUT HOW MANY 11646 07:06:06,274 --> 07:06:08,309 PEOPLE DO YOU HAVE SITTING IN 11647 07:06:08,376 --> 07:06:14,682 THAT SPACE, NO REAL QUESTIONS 11648 07:06:14,749 --> 07:06:15,983 ABOUT TEACHING, OR SERVICE. 11649 07:06:16,050 --> 07:06:17,218 I GUESS YOUR SPACE ALLOCATION 11650 07:06:17,285 --> 07:06:18,920 HAS NOTHING TO DO WITH THAT, AND 11651 07:06:18,986 --> 07:06:20,054 IT SHOULD. 11652 07:06:20,121 --> 07:06:22,023 THAT'S A HUGE PROBLEM. 11653 07:06:22,089 --> 07:06:25,293 THE OTHER PROBLEM IS THE H-INDEX 11654 07:06:25,359 --> 07:06:27,061 THAT IS USED FOR PROMOTION AND 11655 07:06:27,128 --> 07:06:29,764 TENURE, I SAT ON THE TENURE AND 11656 07:06:29,830 --> 07:06:33,467 PROMOTION COMMITTEE AT EMORY AND 11657 07:06:33,534 --> 07:06:36,203 YEARS AT FLORIDA, THAT IS REALLY 11658 07:06:36,270 --> 07:06:37,238 ARTIFICIAL. 11659 07:06:37,305 --> 07:06:39,206 BECAUSE THERE ARE A LOT OF 11660 07:06:39,273 --> 07:06:43,044 PEOPLE WHO GET ON A PUBLICATION, 11661 07:06:43,110 --> 07:06:49,784 ONE THAT COMES TO MIND IS 11662 07:06:49,850 --> 07:06:51,852 AUTOPHAGY METHODS, THOUSANDS OF 11663 07:06:51,919 --> 07:06:53,821 AUTHORS, CITED ALMOST EVERY DAY, 11664 07:06:53,888 --> 07:06:56,791 YEAH, A THOUSAND TIMES, THE 11665 07:06:56,857 --> 07:06:57,658 IMPACT FACTOR IS INFLATED FOR 11666 07:06:57,725 --> 07:07:00,094 THE THOUSAND PEOPLE ON THERE FOR 11667 07:07:00,161 --> 07:07:02,196 THEIR CONTRIBUTION. 11668 07:07:02,263 --> 07:07:02,697 RIGHT? 11669 07:07:02,763 --> 07:07:04,999 THE OTHER THING, H-INDEX IS 11670 07:07:05,066 --> 07:07:05,666 ABOUT TIME, RIGHT? 11671 07:07:05,733 --> 07:07:08,135 THERE HAS TO BE ENOUGH TIME TO 11672 07:07:08,202 --> 07:07:09,303 SOMEONE TO CITE YOUR PAPERS. 11673 07:07:09,370 --> 07:07:11,472 IF YOU PUT THEM IN THE WRONG 11674 07:07:11,539 --> 07:07:14,542 JOURNAL WHERE YOUR READERSHIP IS 11675 07:07:14,608 --> 07:07:18,045 NOT NO ONE IS GOING TO FIND 11676 07:07:18,112 --> 07:07:18,913 THEM. 11677 07:07:18,980 --> 07:07:20,348 bioRxiv AND OTHER PRE-PRINTS 11678 07:07:20,414 --> 07:07:21,983 THEY MAY COUNT FOR A POSTDOC 11679 07:07:22,049 --> 07:07:26,554 GOING OUT FOR A JOB INTERVIEW 11680 07:07:26,620 --> 07:07:29,090 BECAUSE IT'S A DELIVERABLE, BUT 11681 07:07:29,156 --> 07:07:31,759 NOBODY COUNTS THEM FOR ACTUAL 11682 07:07:31,826 --> 07:07:33,027 PEER-REVIEWED PUBLICATIONS, 11683 07:07:33,094 --> 07:07:33,561 RIGHT? 11684 07:07:33,627 --> 07:07:36,464 THAT IS ALSO A BIG PROBLEM. 11685 07:07:36,530 --> 07:07:37,898 BECAUSE THAT'S A DISINCENTIVE TO 11686 07:07:37,965 --> 07:07:41,669 PUT THEM OUT THERE. 11687 07:07:41,736 --> 07:07:43,671 I SUSPECT THE REGISTERED, YOU 11688 07:07:43,738 --> 07:07:44,705 KNOW, PRE-REGISTRY WILL BE THE 11689 07:07:44,772 --> 07:07:46,841 SAME ALTHOUGH I LOVE IT, AND I'M 11690 07:07:46,907 --> 07:07:49,310 GOING TO PUSH FOR NATURE TO 11691 07:07:49,377 --> 07:07:53,581 START CONSIDERING IT. 11692 07:07:53,647 --> 07:07:56,817 >> THANK YOU, MALA. 11693 07:07:56,884 --> 07:08:03,157 >> I AGREE WITH YOU. 11694 07:08:03,224 --> 07:08:08,429 I WAS GOING TO SAY I'M ELOQUENT 11695 07:08:08,496 --> 07:08:09,530 IN DESCRIBING WHAT THE 11696 07:08:09,597 --> 07:08:10,865 CHALLENGES ARE, THIS WHOLE ISSUE 11697 07:08:10,931 --> 07:08:14,435 OF SPACE AND HOW IT'S QUANTIFIED 11698 07:08:14,502 --> 07:08:16,637 AND QUANTITATED, AGAIN, IF YOU 11699 07:08:16,704 --> 07:08:18,739 LOOK AT A MORE COMPREHENSIVE 11700 07:08:18,806 --> 07:08:22,443 UNIVERSITY SYSTEM THAT HAS ARTS 11701 07:08:22,510 --> 07:08:23,811 AND SOCIAL SCIENCE VERSUS 11702 07:08:23,878 --> 07:08:25,246 BIOMEDICAL SCIENCES, YOU HAVE 11703 07:08:25,312 --> 07:08:27,415 DIFFERENT YARDSTICKS, IN TERMS 11704 07:08:27,481 --> 07:08:30,785 OF HOW WE MEASURE RESEARCH AND 11705 07:08:30,851 --> 07:08:31,118 SCHOLARSHIP. 11706 07:08:31,185 --> 07:08:35,723 WHAT IS DONE, EXPECTED IN SOCIAL 11707 07:08:35,790 --> 07:08:37,258 SCIENCES, IS ENTIRELY DIFFERENT 11708 07:08:37,324 --> 07:08:39,226 THAN THEY WOULD CRINGE IF YOU 11709 07:08:39,293 --> 07:08:40,728 USED THE H-INDEX AS A VALUE 11710 07:08:40,795 --> 07:08:41,629 FACTOR THERE, RIGHT? 11711 07:08:41,695 --> 07:08:43,464 SO IN THE BIOMEDICAL SCIENCES 11712 07:08:43,531 --> 07:08:45,866 AND MEDICAL SCHOOLS AND SUCH, 11713 07:08:45,933 --> 07:08:49,270 YES, ABSOLUTELY, THEY ARE ALL -- 11714 07:08:49,336 --> 07:08:52,239 IT'S A VERY NUMBERS-DRIVEN GAME. 11715 07:08:52,306 --> 07:08:54,408 IT'S NOT THAT WE NEED TO DO 11716 07:08:54,475 --> 07:08:55,843 AWAY, BUT I THINK WE NEED TO 11717 07:08:55,910 --> 07:09:00,548 ALSO LOOK AT SOME OF THE FACTORS 11718 07:09:00,614 --> 07:09:03,484 THAT YOU'RE TALKING ABOUT. 11719 07:09:03,551 --> 07:09:05,119 >> I JUST WANT TO -- WITH A 11720 07:09:05,186 --> 07:09:08,222 GROUP OF COLLEAGUES WE DID A 11721 07:09:08,289 --> 07:09:10,057 STUDY, GLOBAL STUDY OF RESEARCH 11722 07:09:10,124 --> 07:09:11,692 ASSESSMENT ALSO. 11723 07:09:11,759 --> 07:09:14,095 WE FOUND THAT THE EVALUATION 11724 07:09:14,161 --> 07:09:15,896 BETWEEN THESE DISCIPLINES WAS 11725 07:09:15,963 --> 07:09:16,831 THE SAME. 11726 07:09:16,897 --> 07:09:20,801 IT DIDN'T CHANGE AT ALL. 11727 07:09:20,868 --> 07:09:21,902 >> NUMBER OF PAPERS? 11728 07:09:21,969 --> 07:09:24,305 >> SPACE OR PAPERS? 11729 07:09:24,371 --> 07:09:28,442 YOU'RE TALKING LABS OR PAPERS? 11730 07:09:28,509 --> 07:09:29,210 >> IT'S EVALUATION FOR 11731 07:09:29,276 --> 07:09:29,510 PROMOTION. 11732 07:09:29,577 --> 07:09:31,846 >> SHE WAS TALKING ABOUT LAB 11733 07:09:31,912 --> 07:09:33,714 STAYS EVALUATION, I WAS 11734 07:09:33,781 --> 07:09:35,216 COMMENTING ON THAT PART, BUT 11735 07:09:35,282 --> 07:09:37,418 YOU'RE RIGHT. 11736 07:09:37,485 --> 07:09:39,220 >> WE DID ANALYZE DIFFERENT 11737 07:09:39,286 --> 07:09:40,988 ACADEMIC DISCIPLINES, BROKE IT 11738 07:09:41,055 --> 07:09:43,591 DOWN BY LIFE SCIENCES OR 11739 07:09:43,657 --> 07:09:46,861 PHYSICAL SCIENCE, MATHEMATICAL 11740 07:09:46,927 --> 07:09:48,662 SCIENCE DEPARTMENTS, 11741 07:09:48,729 --> 07:09:49,196 MULTI-DISCIPLINARY UNITS. 11742 07:09:49,263 --> 07:09:50,364 THERE WERE DIFFERENCES BUT IT 11743 07:09:50,431 --> 07:09:52,099 DEPENDED WHAT WE LOOKED AT. 11744 07:09:52,166 --> 07:09:57,371 SO FOR EXAMPLE WHEN WE WERE 11745 07:09:57,438 --> 07:09:58,672 LOOKING AT THE NON-TRADITIONAL 11746 07:09:58,739 --> 07:10:03,511 OUTPUTS, SOME IN THE HUMANITIES, 11747 07:10:03,577 --> 07:10:04,812 THEY WOULD MENTION ARTS THAT 11748 07:10:04,879 --> 07:10:08,215 DIDN'T COME UP ANYWHERE IN THE 11749 07:10:08,282 --> 07:10:09,650 LIFE SCIENCES, LIFE SCIENCE HAD 11750 07:10:09,717 --> 07:10:12,353 A STRONGER FOCUS ON PARTICULAR 11751 07:10:12,419 --> 07:10:13,554 PUBLICATION COUNTS OR THINGS, SO 11752 07:10:13,621 --> 07:10:15,589 WE DID SEE SOME DIFFERENCE IN 11753 07:10:15,656 --> 07:10:17,658 TERMS OF HOW THEY WERE BREAKING 11754 07:10:17,725 --> 07:10:20,361 DOWN OR WHAT ADDITIONAL 11755 07:10:20,427 --> 07:10:21,462 NON-TRADITIONAL OUTPUTS THEY 11756 07:10:21,529 --> 07:10:24,198 INCLUDED IN THEIR PROMOTION AND 11757 07:10:24,265 --> 07:10:25,332 TENURE DOCUMENTS. 11758 07:10:25,399 --> 07:10:29,970 >> REAL QUICK, LET ME RESPOND TO 11759 07:10:30,037 --> 07:10:30,437 THIS. 11760 07:10:30,504 --> 07:10:34,241 YOU'VE BEEN AT RESEARCH 11761 07:10:34,308 --> 07:10:35,376 INTENSIVE INSTITUTIONS. 11762 07:10:35,442 --> 07:10:40,014 THE CHALLENGE IS THAT WE MEASURE 11763 07:10:40,080 --> 07:10:46,487 OUTPUTS AND IMPACT. 11764 07:10:46,554 --> 07:10:50,591 COMMON DENOMINATOR COMES FROM 11765 07:10:50,658 --> 07:10:52,893 MULTIPLE DATA SOURCES. 11766 07:10:52,960 --> 07:10:55,829 SO, THE WAY CONCEPTUALLY THAT 11767 07:10:55,896 --> 07:10:57,898 INSTITUTIONS ARE THINKING ABOUT 11768 07:10:57,965 --> 07:11:00,634 OUR FACULTY FOR RESEARCH 11769 07:11:00,701 --> 07:11:01,602 INTENSIVE INSTITUTIONS, ARE LIKE 11770 07:11:01,669 --> 07:11:02,636 MANY NON-PROFITS. 11771 07:11:02,703 --> 07:11:04,638 YOU'RE THE EXECUTIVE DIRECTOR OF 11772 07:11:04,705 --> 07:11:07,641 YOUR OWN NON-PROFIT ORGANIZATION 11773 07:11:07,708 --> 07:11:11,145 WITHIN OUR INSTITUTION, AND WE 11774 07:11:11,212 --> 07:11:12,680 VALUE -- THE INSTITUTIONS VALET 11775 07:11:12,746 --> 07:11:13,280 WARDS. 11776 07:11:13,347 --> 07:11:14,782 SO YOU'RE PAYING FOR SPACE 11777 07:11:14,848 --> 07:11:16,951 THROUGH YOUR WORK. 11778 07:11:17,017 --> 07:11:18,953 IF YOU DON'T GENERATE, IT'S 11779 07:11:19,019 --> 07:11:21,121 UNFORTUNATE, BUT IT'S TRUE. 11780 07:11:21,188 --> 07:11:26,260 IN THESE INSTITUTIONS IF YOU'RE 11781 07:11:26,327 --> 07:11:27,795 NOT GENERATING RESOURCES THAT WE 11782 07:11:27,861 --> 07:11:29,630 USE TO JUSTIFY THE SPACE, THEY 11783 07:11:29,697 --> 07:11:32,032 GIVE THE SPACE AWAY TO PEOPLE 11784 07:11:32,099 --> 07:11:33,033 WHO GENERATE MORE RESOURCES. 11785 07:11:33,100 --> 07:11:35,936 BUT IF YOU GO TO A LIBERAL ARTS 11786 07:11:36,003 --> 07:11:37,204 INSTITUTION OR SOME EMERGING 11787 07:11:37,271 --> 07:11:38,372 RESEARCH INSTITUTIONS THEY DON'T 11788 07:11:38,439 --> 07:11:40,941 THINK ABOUT IT THE SAME WAY. 11789 07:11:41,008 --> 07:11:43,577 AND SO THE ISSUE, PARADOX THAT 11790 07:11:43,644 --> 07:11:45,479 WE'RE IN, INSTITUTIONS THAT 11791 07:11:45,546 --> 07:11:49,183 YOU'VE BEEN AT ALL OVER 500 11792 07:11:49,250 --> 07:11:51,685 MILLION, CLOSE TO ONE BILLION IN 11793 07:11:51,752 --> 07:11:54,355 RESEARCH AWARDS ON ANNUAL BASIS, 11794 07:11:54,421 --> 07:11:55,656 THEY HAVE CONDITIONED THEMSELF 11795 07:11:55,723 --> 07:11:56,857 TO OVERVALUE THAT. 11796 07:11:56,924 --> 07:12:00,661 AND NOT VALUE JUST THE HEART OF 11797 07:12:00,728 --> 07:12:02,129 THE SCIENCE OR OTHER ACTIVITIES. 11798 07:12:02,196 --> 07:12:04,832 SO THEY PROBABLY HAVE THAT SAME 11799 07:12:04,898 --> 07:12:06,834 DEBATE ON THOSE CAMPUSES FOR 11800 07:12:06,900 --> 07:12:08,035 ARTS AND HUMANITIES AND OTHER 11801 07:12:08,102 --> 07:12:09,136 GROUPS NOT GETTING THE SPACE 11802 07:12:09,203 --> 07:12:09,937 THEY NEED. 11803 07:12:10,004 --> 07:12:12,039 >> BUT THE IMPACT CAN BE 11804 07:12:12,106 --> 07:12:13,807 MEASURED IN DIFFERENT WAYS. 11805 07:12:13,874 --> 07:12:16,710 WHERE ARE MY TRAINEES, ARE THEY 11806 07:12:16,777 --> 07:12:17,411 MAKING AN IMPACT? 11807 07:12:17,478 --> 07:12:20,247 WHAT BOARDS DO I SIT ON? 11808 07:12:20,314 --> 07:12:21,649 NOBODY ASKS THOSE QUESTIONS. 11809 07:12:21,715 --> 07:12:23,517 >> YEAH, I WANT TO MAKE ONE 11810 07:12:23,584 --> 07:12:25,085 COMMENT ABOUT THAT. 11811 07:12:25,152 --> 07:12:28,122 WHICH IS WHAT WOULD HAPPEN TO 11812 07:12:28,188 --> 07:12:31,525 ALL OF THESE INSTITUTIONS, R 1 11813 07:12:31,592 --> 07:12:34,695 INSTITUTIONS, IF INDIRECT COST 11814 07:12:34,762 --> 07:12:40,000 RETURN WAS 10%? 11815 07:12:40,067 --> 07:12:40,167 15%? 11816 07:12:40,234 --> 07:12:41,502 WOULD WE BE CARRYING AS MUCH 11817 07:12:41,568 --> 07:12:43,437 ABOUT THESE GRANTS IF IT WAS 11818 07:12:43,504 --> 07:12:46,173 SUCH A SMALL AMOUNT, WHERE OUR 11819 07:12:46,240 --> 07:12:49,209 WEIGHT OF PUBLICATIONS, WOULD IT 11820 07:12:49,276 --> 07:12:51,412 BE DIFFERENT IF IT WERE THAT? 11821 07:12:51,478 --> 07:12:56,884 THERE IS SCARILY OR EXCITINGLY A 11822 07:12:56,950 --> 07:13:00,354 CALL FOR MUCH LESS 11823 07:13:00,421 --> 07:13:04,958 INSTITUTIONAL -- IBC, YEAH, 11824 07:13:05,025 --> 07:13:07,194 I.C. -- INCORRECT COST, ICR, 11825 07:13:07,261 --> 07:13:07,594 YEAH. 11826 07:13:07,661 --> 07:13:11,198 IT MIGHT -- THAT MIGHT BE THE 11827 07:13:11,265 --> 07:13:11,498 REVOLUTION. 11828 07:13:11,565 --> 07:13:14,501 >> WAIT, WAIT, WE'RE NOT 11829 07:13:14,568 --> 07:13:15,369 ADVOCATING FOR THAT. 11830 07:13:15,436 --> 07:13:16,136 >> I THINK SOMETHING SHE'S 11831 07:13:16,203 --> 07:13:19,206 TRYING SAY, WHAT WOULD HAPPEN IF 11832 07:13:19,273 --> 07:13:19,640 WE STARTED MEASURING 11833 07:13:19,707 --> 07:13:21,608 INSTITUTIONS IN TERMS OF WHETHER 11834 07:13:21,675 --> 07:13:23,010 THEY ARE MEETING THEIR MISSION 11835 07:13:23,077 --> 07:13:24,345 STATEMENTS BECAUSE WE DON'T 11836 07:13:24,411 --> 07:13:26,213 MEASURE THAT. 11837 07:13:26,280 --> 07:13:27,281 ARE YOU ACTUALLY GENERATING 11838 07:13:27,348 --> 07:13:28,982 PUBLIC GOOD? 11839 07:13:29,049 --> 07:13:33,420 ARE YOU MEETING YOUR LOCAL 11840 07:13:33,487 --> 07:13:33,921 COMMUNITY'S NEEDS? 11841 07:13:33,987 --> 07:13:35,289 WE DON'T MEASURE THAT. 11842 07:13:35,356 --> 07:13:37,224 I THINK THAT'S THE POINT, LIKE 11843 07:13:37,291 --> 07:13:39,927 WHAT IS YOUR MEASURE OF IMPACT? 11844 07:13:39,993 --> 07:13:41,562 HOW DOES THAT MATCH UP? 11845 07:13:41,628 --> 07:13:43,797 THERE ARE SOME INSTITUTIONS WHO 11846 07:13:43,864 --> 07:13:46,066 WILL REMAIN NAMELESS THAT DON'T 11847 07:13:46,133 --> 07:13:47,468 ALLOW INVESTIGATORS TO APPLY FOR 11848 07:13:47,534 --> 07:13:50,571 GRANTS THAT CARRY NO INDIRECTS. 11849 07:13:50,637 --> 07:13:53,107 >> ONE THING I WOULD SAY AS A 11850 07:13:53,173 --> 07:13:57,644 SOLUTION, THAT'S FOR TOMORROW, I 11851 07:13:57,711 --> 07:13:59,213 WOULD SAY THIS. 11852 07:13:59,279 --> 07:14:00,080 INSTITUTIONS CYCLE THROUGH 11853 07:14:00,147 --> 07:14:00,781 LEADERSHIP. 11854 07:14:00,848 --> 07:14:03,784 WHAT THEY DON'T HAVE IS IS A 11855 07:14:03,851 --> 07:14:06,954 PLAYBOOK ON HOW TO THINK ABOUT 11856 07:14:07,020 --> 07:14:07,721 MEASURING IMPACT. 11857 07:14:07,788 --> 07:14:09,590 IF YOU WANT TO MAKE A DIFFERENCE 11858 07:14:09,656 --> 07:14:14,194 IT NEEDS TO BE A COLLECTIVE, 11859 07:14:14,261 --> 07:14:15,996 WRITE A PAPER, AND INFORMATION 11860 07:14:16,063 --> 07:14:18,499 THAT GOES TO ADMINISTRATION ON 11861 07:14:18,565 --> 07:14:20,367 HOW TO MEASURE RESEARCH 11862 07:14:20,434 --> 07:14:22,336 APPROPRIATELY, AND THEN THEY 11863 07:14:22,403 --> 07:14:25,038 WILL ADOPT IT. 11864 07:14:25,839 --> 07:14:28,208 WE'RE 11865 07:14:28,275 --> 07:14:31,512 >> WE'RE GOING TO THE NEXT 11866 07:14:31,578 --> 07:14:31,879 QUESTION. 11867 07:14:31,945 --> 07:14:32,212 >> I THINK . 11868 07:14:32,279 --> 07:14:33,347 >> WE'RE GOING TO THE NEXT 11869 07:14:33,414 --> 07:14:35,516 QUESTION, WE DON'T HAVE MUCH 11870 07:14:35,582 --> 07:14:36,383 MORE TIME. 11871 07:14:36,450 --> 07:14:37,384 >> THANK YOU. 11872 07:14:37,451 --> 07:14:40,521 I HAVE A QUESTION MOSTLY FOR 11873 07:14:40,587 --> 07:14:42,356 RACHEL AND ERIN, BOTH HAVE DONE 11874 07:14:42,423 --> 07:14:44,958 SURVEYS ABOUT FACULTY HOW THEY 11875 07:14:45,025 --> 07:14:46,527 FEEL ABOUT THE TENURE PROMOTION 11876 07:14:46,593 --> 07:14:47,227 CRITERIA, RIGHT? 11877 07:14:47,294 --> 07:14:50,831 TO WHAT EXTENT DO YOU FEEL LIKE 11878 07:14:50,898 --> 07:14:51,999 CONCERNS RAISED ARE UNIVERSAL, 11879 07:14:52,065 --> 07:14:55,269 HUMAN CONCERNS ABOUT HOW THEY 11880 07:14:55,335 --> 07:14:57,271 ARE BEING JUDGED AND SPECIFIC TO 11881 07:14:57,337 --> 07:15:01,842 YOUR UNIVERSITY OR TO YOUR 11882 07:15:01,909 --> 07:15:02,943 GEOGRAPHIC REGION? 11883 07:15:03,010 --> 07:15:04,611 >> IT'S VERY UNIVERSAL. 11884 07:15:04,678 --> 07:15:12,920 ERIN CAN SPEAK MORE, BEING 11885 07:15:12,986 --> 07:15:13,487 GENERALIZED, 800 DOCUMENTS 11886 07:15:13,554 --> 07:15:15,756 ACROSS U.S. AND CANADA. 11887 07:15:15,823 --> 07:15:17,791 I DIDN'T MENTION WE DID A 11888 07:15:17,858 --> 07:15:19,893 NATIONAL STUDY FOCUSED ON 11889 07:15:19,960 --> 07:15:21,195 ACADEMIC LIBRARIANS, AND WE 11890 07:15:21,261 --> 07:15:23,063 ASKED THEM LOTS OF QUESTIONS BUT 11891 07:15:23,130 --> 07:15:25,466 IT WAS FOCUSED ON RESEARCH 11892 07:15:25,532 --> 07:15:29,069 METRICS AND ASSESSMENT AND WE 11893 07:15:29,136 --> 07:15:31,939 ASKED THEM TWO QUESTIONS, ABOUT 11894 07:15:32,005 --> 07:15:33,006 CONSULTATIONS WITH FACULTY, SO 11895 07:15:33,073 --> 07:15:34,908 WE ASKED THEM, YOU KNOW, WHAT 11896 07:15:34,975 --> 07:15:36,210 METRICS DO YOU DISCUSS WITH 11897 07:15:36,276 --> 07:15:38,245 FACULTY AND HOW OFTEN DO YOU 11898 07:15:38,312 --> 07:15:40,080 DISCUSS THOSE ON A LIKERT SCALE. 11899 07:15:40,147 --> 07:15:42,282 SO WHEN WE WERE ASKING THEM HOW 11900 07:15:42,349 --> 07:15:44,618 OFTEN DO YOU DISCUSS THESE 11901 07:15:44,685 --> 07:15:47,254 METRICS ON A LIKERT SCALE, WHEN 11902 07:15:47,321 --> 07:15:50,123 IT COMES TO PUBLISHING ISSUES, 11903 07:15:50,190 --> 07:15:51,325 YOU KNOW, THERE WAS NOT THAT 11904 07:15:51,391 --> 07:15:52,092 MUCH INTEREST. 11905 07:15:52,159 --> 07:15:57,130 WHEN WE ASKED HOW OFT OFTEN DO 11906 07:15:57,197 --> 07:16:00,434 YOU DISCUSS THESE WITH TENURE 11907 07:16:00,501 --> 07:16:04,605 AND GRANTS, IT'S A 4 OR 5 ON THE 11908 07:16:04,671 --> 07:16:06,507 LIKERT SCALE WITH STATISTICAL 11909 07:16:06,573 --> 07:16:08,041 DIFFERENCE BETWEEN THE ANSWERS, 11910 07:16:08,108 --> 07:16:13,847 AND I WASN'T EVEN LOOKING, 11911 07:16:13,914 --> 07:16:15,249 REALIZED THERE WAS A DIFFERENCE 11912 07:16:15,315 --> 07:16:16,116 WHEN DOING A POSTER. 11913 07:16:16,183 --> 07:16:17,084 THIS IS UNIVERSAL. 11914 07:16:17,150 --> 07:16:18,519 THAT WAS A NATIONAL STUDY. 11915 07:16:18,585 --> 07:16:20,287 THIS HAS BEEN FOUND IN OTHER 11916 07:16:20,354 --> 07:16:21,688 SURVEY RESULTS AS WELL. 11917 07:16:21,755 --> 07:16:24,124 I THINK THAT THE PRESSURES BEING 11918 07:16:24,191 --> 07:16:26,627 PUT ON ACADEMICS ARE COMING FROM 11919 07:16:26,693 --> 07:16:28,028 EXTERNAL FACTORS, NOT REALLY -- 11920 07:16:28,095 --> 07:16:29,963 THIS IS NOT AN INTERNAL DRIVING 11921 07:16:30,030 --> 07:16:31,598 THING, THEY ARE NOT SAYING, OH, 11922 07:16:31,665 --> 07:16:34,301 YEAH, I CARE ABOUT THE JOURNAL 11923 07:16:34,368 --> 07:16:36,036 IMPACT FACTOR. 11924 07:16:36,103 --> 07:16:36,803 THEY ARE REALLY CARING I 11925 07:16:36,870 --> 07:16:38,438 THINK -- I THINK THEY ARE 11926 07:16:38,505 --> 07:16:41,174 TRAINED TO FIND -- TRAINED AS 11927 07:16:41,241 --> 07:16:44,578 SCIENTISTS AND THEN HAVING 11928 07:16:44,645 --> 07:16:46,747 EXTERNAL PRESSURES TO BE 11929 07:16:46,813 --> 07:16:48,615 SUPERSTARS IN THEIR FIELD AND 11930 07:16:48,682 --> 07:16:51,118 PRESENT THEMSELVES IN A WAY AND 11931 07:16:51,184 --> 07:16:57,057 BEHAVIOR CHANGES AS A RESULT. 11932 07:16:57,124 --> 07:16:59,493 >> YEAH, SO I'LL PUT A TAYLOR 11933 07:16:59,560 --> 07:17:00,594 SWIFT REFERENCE IN HERE. 11934 07:17:00,661 --> 07:17:02,963 WE WERE THE PROBLEM. 11935 07:17:03,030 --> 07:17:04,398 IT'S US. 11936 07:17:04,464 --> 07:17:05,999 SO, EVERYBODY'S TALKING ABOUT, 11937 07:17:06,066 --> 07:17:09,703 WELL, I VALUE THIS BUT THEY 11938 07:17:09,770 --> 07:17:11,939 VALUE. 11939 07:17:12,005 --> 07:17:12,940 WHO IS "THEY"? 11940 07:17:13,006 --> 07:17:14,441 THEY ARE YOUR COLLEAGUES, RIGHT? 11941 07:17:14,508 --> 07:17:16,743 WE NEED TO BE TALKING MORE TO 11942 07:17:16,810 --> 07:17:18,845 ONE ANOTHER BECAUSE IT'S THIS 11943 07:17:18,912 --> 07:17:20,814 GROUP OF FACULTY AND WE ALL 11944 07:17:20,881 --> 07:17:22,883 THINK WE'RE -- WE ALL WANT TO 11945 07:17:22,950 --> 07:17:24,318 VALUE, YOU KNOW, READERSHIP AND 11946 07:17:24,384 --> 07:17:25,752 AUDIENCE AND ARE WE GETTING TO 11947 07:17:25,819 --> 07:17:28,288 THE RIGHT PEOPLE BUT WE THINK 11948 07:17:28,355 --> 07:17:29,723 THE OTHER FOLKS -- THAT HAPPENS 11949 07:17:29,790 --> 07:17:32,092 WHEN YOU HAVE THESE EVALUATIONS 11950 07:17:32,159 --> 07:17:33,126 HAPPEN BEHIND CLOSED DOORS, 11951 07:17:33,193 --> 07:17:34,428 WHERE NOBODY WANTS TO TALK ABOUT 11952 07:17:34,494 --> 07:17:37,464 WHAT'S GOING ON, HOW PEOPLE ARE 11953 07:17:37,531 --> 07:17:37,731 EVALUATED. 11954 07:17:37,798 --> 07:17:39,900 WE NEED TO OPEN THAT UP. 11955 07:17:39,967 --> 07:17:43,403 THAT FEELING IS UNIVERSAL, 11956 07:17:43,470 --> 07:17:43,637 RIGHT? 11957 07:17:43,704 --> 07:17:45,939 I'LL SAY WE ANALYZE THESE 11958 07:17:46,006 --> 07:17:47,240 DOCUMENTS, SURVEYED FACULTY 11959 07:17:47,307 --> 07:17:48,408 ACROSS DIFFERENT INSTITUTIONS, 11960 07:17:48,475 --> 07:17:50,744 IT WASN'T JUST THE R TYPES, IT 11961 07:17:50,811 --> 07:17:54,581 WAS M TYPE AND B TYPE 11962 07:17:54,648 --> 07:17:56,450 INSTITUTIONS WHERE SUPPOSEDLY 11963 07:17:56,516 --> 07:17:58,418 THE FOCUS IS DIFFERENT, TEACHING 11964 07:17:58,485 --> 07:17:59,519 LOAD, RESEARCH INTENSIVE IS 11965 07:17:59,586 --> 07:18:01,021 DIFFERENT, STILL SAYING MY 11966 07:18:01,088 --> 07:18:03,657 COLLEAGUES ARE TOO FOCUSED ON 11967 07:18:03,724 --> 07:18:06,093 PRESTIGE AND TOO FOCUSED ON 11968 07:18:06,159 --> 07:18:07,160 JOURNAL LABEL AND JOURNAL 11969 07:18:07,227 --> 07:18:13,166 PACKAGE AND THINGS LIKE THAT. 11970 07:18:13,233 --> 07:18:16,570 I THINK THOSE THINGS DID GO 11971 07:18:16,637 --> 07:18:21,942 ACROSS TYPE WHEN MISSIONS AND 11972 07:18:22,009 --> 07:18:25,012 RESEARCH INSTITUTIONS WERE 11973 07:18:25,078 --> 07:18:25,545 DIFFERENT. 11974 07:18:25,612 --> 07:18:25,912 >> LAURA SMITH. 11975 07:18:25,979 --> 07:18:27,848 I THINK WE'RE GETTING INTO THIS 11976 07:18:27,914 --> 07:18:28,081 NOW. 11977 07:18:28,148 --> 07:18:30,050 AS A PERSON GOING THROUGH TENURE 11978 07:18:30,117 --> 07:18:31,018 APPLICATION PROCESS I THOUGHT I 11979 07:18:31,084 --> 07:18:33,854 WOULD BRING UP THINGS I WAS 11980 07:18:33,920 --> 07:18:35,756 ASKED SPECIFICALLY ABOUT MY 11981 07:18:35,822 --> 07:18:37,691 IMPACTS ON THE FIELD. 11982 07:18:37,758 --> 07:18:41,061 HOW HAVE I MOVED MY FIELD, YOU 11983 07:18:41,128 --> 07:18:42,362 KNOW, ON A NATIONAL, 11984 07:18:42,429 --> 07:18:44,297 INTERNATIONAL LEVEL, AND THINGS 11985 07:18:44,364 --> 07:18:47,334 YOU CAN USE AS EVIDENCE FOR 11986 07:18:47,401 --> 07:18:55,909 THAT, H-INDEX OF COURSE, MOMENTA 11987 07:18:55,976 --> 07:19:02,449 SCORES, YOU CAN -- SOCIAL MEDIA, 11988 07:19:02,516 --> 07:19:07,087 GET iCITE, I GUESS I'M JUST 11989 07:19:07,154 --> 07:19:09,256 WONDERING, IF I'M SUCCESSFUL 11990 07:19:09,322 --> 07:19:10,157 PUBLISHING NULL PUBLICATIONS, 11991 07:19:10,223 --> 07:19:11,992 THAT'S NOT GOING TO CHANGE MY 11992 07:19:12,059 --> 07:19:14,928 IMPACT ON THE FIELD, I DOUBT. 11993 07:19:14,995 --> 07:19:17,397 SHOULD WE BE ASKING ABOUT 11994 07:19:17,464 --> 07:19:22,302 IMPACTS AND IF WE SHOULD, THEN 11995 07:19:22,369 --> 07:19:24,471 WHAT SHOULD IMPACT BE? 11996 07:19:24,538 --> 07:19:26,006 >> IT'S A VERY INTERESTING 11997 07:19:26,073 --> 07:19:26,640 QUESTION. 11998 07:19:26,707 --> 07:19:29,843 AGAIN, AS WE TALK IMPACT IN 11999 07:19:29,910 --> 07:19:34,614 DIFFERENT FIELDS, GOING TO BE 12000 07:19:34,681 --> 07:19:35,215 EXTRAORDINARILY DIFFERENT, IF 12001 07:19:35,282 --> 07:19:36,750 YOU'RE A CLINICIAN IMPACT IS ONE 12002 07:19:36,817 --> 07:19:37,050 THING. 12003 07:19:37,117 --> 07:19:39,686 IF YOU'RE AN ARTS RESEARCHER, 12004 07:19:39,753 --> 07:19:42,456 IT'S VERY DIFFERENT. 12005 07:19:42,522 --> 07:19:43,623 IF YOU'RE IN EDUCATION, IMPACT 12006 07:19:43,690 --> 07:19:44,291 IS DIFFERENT. 12007 07:19:44,357 --> 07:19:54,234 EACH OF THOSE HAS TO BE 12008 07:19:54,301 --> 07:19:56,002 EVALUATED ON THEIR OWN MERITS 12009 07:19:56,069 --> 07:19:58,505 AND HAS TO BE FACTOR IN WHERE 12010 07:19:58,572 --> 07:20:00,974 WE'RE LOOKING AT WHAT DOES 12011 07:20:01,041 --> 07:20:02,008 COMMUNITY IMPACT LOOK LIKE, NO 12012 07:20:02,075 --> 07:20:04,111 MATTER WHERE WE'RE DOING. 12013 07:20:04,177 --> 07:20:07,013 SO, I THINK INCREASINGLY WE ARE 12014 07:20:07,080 --> 07:20:08,648 ASKING THESE SORTS OF QUESTIONS 12015 07:20:08,715 --> 07:20:12,486 HOW DO WE MEASURE IMPACT AND ONE 12016 07:20:12,552 --> 07:20:14,788 WAY WE FEEL THAT WE 12017 07:20:14,855 --> 07:20:16,022 ADMINISTRATIVELY HAVE A LEVER 12018 07:20:16,089 --> 07:20:19,259 WHERE WE CAN ASK THESE RIGHT 12019 07:20:19,326 --> 07:20:20,794 KINDS OF QUESTIONS, THE INTERNAL 12020 07:20:20,861 --> 07:20:23,697 FUNDING WE PROVIDE OUR FACULTY. 12021 07:20:23,764 --> 07:20:27,267 WE'RE NOT JUST SAYING IT ONLY 12022 07:20:27,334 --> 07:20:28,702 LEADS TO EXTRAMURAL FUNDING, 12023 07:20:28,769 --> 07:20:30,337 YES, IT'S AN IMPORTANT CRITERION 12024 07:20:30,403 --> 07:20:33,373 FOR ALL THE REASONS OF INDIRECT 12025 07:20:33,440 --> 07:20:37,277 COST BUT WE'RE ALSO SAYING IF WE 12026 07:20:37,344 --> 07:20:39,346 ARE TO SUPPORT RESEARCH IN 12027 07:20:39,412 --> 07:20:42,048 MULTIPLE AREAS WE TRULY HAVE TO 12028 07:20:42,115 --> 07:20:43,350 MEASURE IMPACT OF WHAT IT MEANS 12029 07:20:43,416 --> 07:20:45,852 IN EACH OF THE DISCIPLINES AND 12030 07:20:45,919 --> 07:20:49,756 NOT USE JUST H-INDEX AND JUST 12031 07:20:49,823 --> 07:20:52,325 THOSE KIND OF CRITERIA ACROSS 12032 07:20:52,392 --> 07:20:54,394 THE BOARD BECAUSE IF WE DO THAT 12033 07:20:54,461 --> 07:20:56,730 WE ENDS UP WITH THESE 12034 07:20:56,797 --> 07:20:57,230 SITUATIONS. 12035 07:20:57,297 --> 07:21:04,171 >> THANK YOU. 12036 07:21:04,237 --> 07:21:08,208 I WONDER IF YOU REPRESENT 12037 07:21:08,275 --> 07:21:09,676 SIGNATORIES OF DORA? 12038 07:21:09,743 --> 07:21:10,944 >> DECLARATION OF RESEARCH 12039 07:21:11,011 --> 07:21:11,278 ASSESSMENT. 12040 07:21:11,344 --> 07:21:15,081 WE DECIDED TO GO IN A DIFFERENT 12041 07:21:15,148 --> 07:21:17,517 DIRECTION DID OUR OWN STATEMENT 12042 07:21:17,584 --> 07:21:22,522 ON RESPONSIBLE RESEARCH METRICS. 12043 07:21:22,589 --> 07:21:23,223 >> MINE IS NOT. 12044 07:21:23,290 --> 07:21:24,791 >> I HAVE TO LOOK IT UP. 12045 07:21:24,858 --> 07:21:26,593 >> THAT'S PART OF MY QUESTION. 12046 07:21:26,660 --> 07:21:27,894 I'M A POSTDOC RIGHT NOW. 12047 07:21:27,961 --> 07:21:31,898 I JUST MOVED TO THE U.S. FROM 12048 07:21:31,965 --> 07:21:32,766 EUROPE. 12049 07:21:32,833 --> 07:21:34,601 AND IN ALMOST EVERY POSTDOC TALK 12050 07:21:34,668 --> 07:21:36,236 OR CONFERENCE I WENT IN EUROPE 12051 07:21:36,303 --> 07:21:39,773 THEY KIND OF WERE AWARE OF DORA, 12052 07:21:39,840 --> 07:21:41,474 WHICH WAS CREATED HERE IN THE 12053 07:21:41,541 --> 07:21:41,975 U.S. 12054 07:21:42,042 --> 07:21:44,377 BUT SINCE I'VE COME HERE TO THE 12055 07:21:44,444 --> 07:21:48,081 U.S. VERY FEW PEOPLE KNOW ABOUT 12056 07:21:48,148 --> 07:21:48,281 DORA. 12057 07:21:48,348 --> 07:21:52,118 I WONDER WHY AND HOW CAN WE TALK 12058 07:21:52,185 --> 07:21:54,721 THIS? 12059 07:21:54,788 --> 07:21:56,022 >> IT'S A GOOD QUESTION. 12060 07:21:56,089 --> 07:21:57,123 OH, RIGHT THERE. 12061 07:21:57,190 --> 07:21:58,625 YOU'VE GOT THE PERFECT PERSON 12062 07:21:58,692 --> 07:22:00,193 RIGHT BEHIND YOU TO ANSWER THAT. 12063 07:22:00,260 --> 07:22:04,130 I'LL JUST SAY A LOT OF 12064 07:22:04,197 --> 07:22:05,031 INSTITUTIONS IN LATIN AMERICA 12065 07:22:05,098 --> 07:22:06,766 HAVE NOT SIGNED O THERE'S A 12066 07:22:06,833 --> 07:22:08,168 GENERAL MOVEMENT TOWARDS THAT. 12067 07:22:08,235 --> 07:22:10,804 IN THE EVALUATION SYSTEMS WE'RE 12068 07:22:10,871 --> 07:22:13,473 SEEING FOLKS GOING BACKWARDS, 12069 07:22:13,540 --> 07:22:14,641 ADDING IN IMPACT FACTOR. 12070 07:22:14,708 --> 07:22:16,543 NO, WE'RE SUPPOSED TO GO THE 12071 07:22:16,610 --> 07:22:17,344 OTHER WAY. 12072 07:22:17,410 --> 07:22:19,880 THE OTHER PROBLEM, THIS IS NOT 12073 07:22:19,946 --> 07:22:21,214 TO DISCREDIT ANY DECLARATION, 12074 07:22:21,281 --> 07:22:23,250 BUT I THINK IF YOUR INSTITUTION 12075 07:22:23,316 --> 07:22:25,318 SIGNS THAT NEEDS TO BE BACKED UP 12076 07:22:25,385 --> 07:22:26,553 WITH HERE ARE ACTIONS WE'RE 12077 07:22:26,620 --> 07:22:28,421 GOING TO TAKE FOLLOWING OUR 12078 07:22:28,488 --> 07:22:28,822 SIGNATURE, RIGHT? 12079 07:22:28,889 --> 07:22:30,390 BECAUSE WE HAVE SEEN EXAMPLES 12080 07:22:30,457 --> 07:22:32,225 WHERE INSTITUTIONS HAVE SIGNED 12081 07:22:32,292 --> 07:22:33,827 AND THEN THEIR CRITERIA CONTINUE 12082 07:22:33,894 --> 07:22:37,297 TO INCLUDE IMPACT FACTOR OR DO 12083 07:22:37,364 --> 07:22:38,932 THINGS IN DIRECT CONTRADICTION. 12084 07:22:38,999 --> 07:22:40,800 STEPHEN, I'LL HAND IT TO YOU. 12085 07:22:40,867 --> 07:22:42,335 >> I'LL CHIP IN BRIEFLY. 12086 07:22:42,402 --> 07:22:43,837 THERE ARE HUGE CULTURAL 12087 07:22:43,904 --> 07:22:46,239 DIFFERENCES BETWEEN EUROPE AND 12088 07:22:46,306 --> 07:22:47,274 THE UNITED STATES. 12089 07:22:47,340 --> 07:22:49,376 AND AS FORMER CHAIR OF DORA I 12090 07:22:49,442 --> 07:22:50,844 DON'T CARE WHETHER PEOPLE SIGN 12091 07:22:50,911 --> 07:22:51,878 IT OR NOT. 12092 07:22:51,945 --> 07:22:55,015 WHAT I CARE ABOUT IS HOW THEY GO 12093 07:22:55,081 --> 07:22:55,682 ABOUT RESEARCH ASSESSMENT AND SO 12094 07:22:55,749 --> 07:22:57,651 IF YOU HAVE YOUR OWN STATEMENT 12095 07:22:57,717 --> 07:23:00,053 AND YOU THINK IT'S EVERY BIT AS 12096 07:23:00,120 --> 07:23:01,688 GOOD AS DORAS, BULLY FOR YOU. 12097 07:23:01,755 --> 07:23:03,657 THAT'S FINE BY ME AS LONG AS YOU 12098 07:23:03,723 --> 07:23:06,726 LIVE UP TO THE IDEAS. 12099 07:23:06,793 --> 07:23:10,563 THAT'S A WHOLE OTHER 12100 07:23:10,630 --> 07:23:11,097 CONVERSATION. 12101 07:23:11,164 --> 07:23:13,266 >> I AGREE, TRYING TO, IT'S NOT 12102 07:23:13,333 --> 07:23:14,134 ACTIVELY TALKED AND DISCUSSED 12103 07:23:14,200 --> 07:23:16,169 AND SO I THINK WE NEED TO 12104 07:23:16,236 --> 07:23:17,771 ELEVATE THAT. 12105 07:23:17,837 --> 07:23:22,309 >> THE LAST TIME I LOOKED, IN 12106 07:23:22,375 --> 07:23:25,345 THE U.K. THERE WERE 84 I THINK 12107 07:23:25,412 --> 07:23:28,415 INSTITUTIONS THAT SIGNED ON TO 12108 07:23:28,481 --> 07:23:29,149 DORA. 12109 07:23:29,215 --> 07:23:39,759 IN THE GREAT UNITED STATES, TWO. 12110 07:23:41,728 --> 07:23:41,861 >> UPenn. 12111 07:23:41,928 --> 07:23:44,497 I HAVE TWO QUICK THINGS. 12112 07:23:44,564 --> 07:23:45,799 FIRST, JUST A COMMENT ON THE -- 12113 07:23:45,865 --> 07:23:50,270 WHAT YOU BROUGHT UP IN YOUR TALK 12114 07:23:50,337 --> 07:23:51,705 ABOUT SERVICE, AND DIFFERENCES, 12115 07:23:51,771 --> 07:23:54,207 BETWEEN MEN AND WOMEN, AND 12116 07:23:54,274 --> 07:23:55,442 PERCEIVED AMONG SERVICE DONE, 12117 07:23:55,508 --> 07:23:58,244 THERE'S ACTUALLY A LOT OF DATA 12118 07:23:58,311 --> 07:24:00,480 ON THIS FROM UNIVERSITY OF 12119 07:24:00,547 --> 07:24:02,649 PITTSBURGH, IT'S SOMETHING THAT 12120 07:24:02,716 --> 07:24:05,018 SHE HAS CALLED NON-PROMOTABLE 12121 07:24:05,085 --> 07:24:06,586 WORK AND THERE'S A SEVERE GENDER 12122 07:24:06,653 --> 07:24:07,921 DIFFERENCE BETWEEN MEN AND 12123 07:24:07,988 --> 07:24:11,124 WOMEN, WOMEN TAKING ON MORE 12124 07:24:11,191 --> 07:24:13,893 NON-PROMOTABLE WORK, MOSTLY 12125 07:24:13,960 --> 07:24:14,894 SERVICE TYPE WORK. 12126 07:24:14,961 --> 07:24:16,463 AND SO FOR THOSE INTERESTED AND 12127 07:24:16,529 --> 07:24:17,897 I THINK IT'S IMPORTANT JUST TO 12128 07:24:17,964 --> 07:24:21,368 BE AWARE OF THAT DATA, THAT IT 12129 07:24:21,434 --> 07:24:23,803 EXISTS, SHE LOOKS AT CAUSES OF 12130 07:24:23,870 --> 07:24:24,337 THIS TOO. 12131 07:24:24,404 --> 07:24:26,740 SO JUST SOMETHING TO KEEP IN 12132 07:24:26,806 --> 07:24:27,073 MIND. 12133 07:24:27,140 --> 07:24:30,777 THE OTHER THING IS WE'VE BEEN 12134 07:24:30,844 --> 07:24:32,579 TALKING ABOUT FACULTY BUT THE 12135 07:24:32,645 --> 07:24:33,980 ONES CONDUCTING THE RESEARCH ARE 12136 07:24:34,047 --> 07:24:35,382 TRAINEES MOST OF THE TIME. 12137 07:24:35,448 --> 07:24:38,218 AND SO I THINK IT'S IMPORTANT TO 12138 07:24:38,284 --> 07:24:39,853 THINK ABOUT THAT AND LIKE HOW 12139 07:24:39,919 --> 07:24:42,589 ARE THESE EXPECTATIONS BEING PUT 12140 07:24:42,655 --> 07:24:44,324 ON FACULTY, TRICKLING DOWN TO 12141 07:24:44,391 --> 07:24:45,792 TRAINEES AND HOW WILL THAT 12142 07:24:45,859 --> 07:24:48,194 AFFECT THEM GOING THROUGH 12143 07:24:48,261 --> 07:24:50,063 TRAINING GETTING DOCTORATES, 12144 07:24:50,130 --> 07:24:55,635 BEING POSTDOC, FINDING A FACULTY 12145 07:24:55,702 --> 07:24:57,003 POSITION LATER. 12146 07:24:57,070 --> 07:24:57,837 >> EXCELLENT POINT. 12147 07:24:57,904 --> 07:25:03,810 IT'S SOMETHING WE NEED TO TRAIN 12148 07:25:03,877 --> 07:25:05,512 THE NEXT GENERATION OR ACADEMIA 12149 07:25:05,578 --> 07:25:07,313 PERCEIVES ONE FUNERAL AT A TIME. 12150 07:25:07,380 --> 07:25:08,314 IT'S NOT THE CASE. 12151 07:25:08,381 --> 07:25:13,019 THE PROBLEM IS IF THE FACULTY 12152 07:25:13,086 --> 07:25:14,754 MEMBERS ARE TRAINING STUDENTS TO 12153 07:25:14,821 --> 07:25:16,623 THINK ABOUT PRESTIGE AND IMPACT 12154 07:25:16,689 --> 07:25:18,491 FACTOR AND PUBLICATION AND NOT 12155 07:25:18,558 --> 07:25:19,592 PUBLISHING NULL RESULTS, YOU'RE 12156 07:25:19,659 --> 07:25:22,629 GOING TO END UP WITH THAT 12157 07:25:22,695 --> 07:25:23,296 PROBLEM PROPAGATING AND 12158 07:25:23,363 --> 07:25:27,434 CONTINUING TO PROPAGATE THROUGH 12159 07:25:27,500 --> 07:25:28,201 THE SYSTEM. 12160 07:25:28,268 --> 07:25:29,636 THEY ARE BEING TRAINED BY THE 12161 07:25:29,702 --> 07:25:31,971 VERY PEOPLE THAT THINK THAT BAY. 12162 07:25:32,038 --> 07:25:33,339 IF WE'RE GIVING PROMOTION AND 12163 07:25:33,406 --> 07:25:35,108 TENURE TO THOSE PEOPLE WE'RE 12164 07:25:35,175 --> 07:25:36,876 FILTERING OUT FOLKS THAT MIGHT 12165 07:25:36,943 --> 07:25:37,877 THINK DIFFERENTLY, YOU KNOW, 12166 07:25:37,944 --> 07:25:48,421 VERY EARLY ON IN THE PROCESS 12167 07:25:51,458 --> 07:25:55,161 URGES. 12168 07:25:55,228 --> 07:25:57,163 >> A COMMENTER EARLY ON, THE 12169 07:25:57,230 --> 07:26:07,640 YOUNGER POSTDOCS, PEOPLE 12170 07:26:13,213 --> 07:26:14,080 PROMOTING OPEN SCIENCE, THIS IS 12171 07:26:14,147 --> 07:26:15,715 GOING TO TAKE SOME TIME. 12172 07:26:15,782 --> 07:26:18,384 >> EVEN LONGER IF OUR 12173 07:26:18,451 --> 07:26:25,658 INSTITUTIONS ARE USING H-INDEXES 12174 07:26:25,725 --> 07:26:28,428 IN PROMOTION CRITERIA. 12175 07:26:28,495 --> 07:26:32,165 THAT WILL GRIND OUT THE IDEALISM 12176 07:26:32,232 --> 07:26:34,334 AND NOBILITY WE FIND IN THE 12177 07:26:34,400 --> 07:26:34,968 YOUNGER GENERATION BECAUSE THEY 12178 07:26:35,034 --> 07:26:36,669 WILL GET SUCKED INTO THE SYSTEM 12179 07:26:36,736 --> 07:26:39,739 AND HERE IN 20 YEARS TIME HAVING 12180 07:26:39,806 --> 07:26:42,976 THE SAME CONVERSATION WE'RE 12181 07:26:43,042 --> 07:26:44,644 HAVING TODAY. 12182 07:26:44,711 --> 07:26:46,713 IF YOU'RE AN INSTITUTION USING 12183 07:26:46,779 --> 07:26:48,114 H-INDEX OR PLAY THE UNIVERSITY 12184 07:26:48,181 --> 07:26:50,483 RANKING GAME I HOPE YOU'RE NOT 12185 07:26:50,550 --> 07:26:51,885 CONSIDERING YOURSELF FIT TO 12186 07:26:51,951 --> 07:26:55,388 TEACH YOUR STUDENTS ABOUT 12187 07:26:55,455 --> 07:26:57,123 CRITICAL THINKING. 12188 07:26:57,190 --> 07:26:59,959 YOU NEED TO STEP UP TO THE PLATE 12189 07:27:00,026 --> 07:27:01,594 AND LOOK AT HOW FLAWED THESE 12190 07:27:01,661 --> 07:27:02,629 MECHANISMS ARE. 12191 07:27:02,695 --> 07:27:05,431 >> I THINK THE OBSESSION WITH 12192 07:27:05,498 --> 07:27:09,302 INSTITUTIONAL RANKINGS IS A BIG 12193 07:27:09,369 --> 07:27:12,906 PART OF THE PROBLEM. 12194 07:27:12,972 --> 07:27:15,408 >> THEN SECEDE, PULL OUT. 12195 07:27:15,475 --> 07:27:19,812 >> ALL THESE WORLD LEADING 12196 07:27:19,879 --> 07:27:21,614 INSTITUTIONS, SELF-PROCLAIMED, 12197 07:27:21,681 --> 07:27:22,315 PLAY THE RANKING GAMES. 12198 07:27:22,382 --> 07:27:28,087 >> THEY ARE ON THE REVIEW 12199 07:27:28,154 --> 07:27:28,888 COMMITTEE TOO. 12200 07:27:28,955 --> 07:27:30,023 >> H-INDEX ARE NOT THE SAME. 12201 07:27:30,089 --> 07:27:31,724 LET ME SAY THIS. 12202 07:27:31,791 --> 07:27:34,594 AT OUR LEVEL WE DO NOT ESTABLISH 12203 07:27:34,661 --> 07:27:35,895 THE TENURE PROMOTION GUIDELINES 12204 07:27:35,962 --> 07:27:36,930 FOR OUR INSTITUTION. 12205 07:27:36,996 --> 07:27:38,031 THOSE HAPPEN WITHIN THE 12206 07:27:38,097 --> 07:27:38,865 DEPARTMENT BY PEERS WITHIN THAT 12207 07:27:38,932 --> 07:27:40,833 DEPARTMENT WHERE THEY COME UP 12208 07:27:40,900 --> 07:27:43,570 WITH CRITERIA AND SUBMIT IT 12209 07:27:43,636 --> 07:27:44,370 THROUGH INSTITUTION. 12210 07:27:44,437 --> 07:27:48,374 SO, IT IS NOT US WHO ARE 12211 07:27:48,441 --> 07:27:51,377 PROMOTING H-INDEXES FROM 12212 07:27:51,444 --> 07:27:52,045 INSTITUTIONAL LEVEL. 12213 07:27:52,111 --> 07:27:54,747 THIS GOES BACK TO WHAT I SAID. 12214 07:27:54,814 --> 07:27:55,748 PEERS, EVEN WITHIN YOUR 12215 07:27:55,815 --> 07:27:59,352 DEPARTMENT, AND THE DEPARTMENT 12216 07:27:59,419 --> 07:28:01,521 CHAIR, INCENTIVIZE WHAT THEY 12217 07:28:01,588 --> 07:28:02,622 DETERMINE TO BE IMPORTANT BASED 12218 07:28:02,689 --> 07:28:05,391 ON WHAT'S GOING ON IN THE 12219 07:28:05,458 --> 07:28:05,692 ENVIRONMENT. 12220 07:28:05,758 --> 07:28:09,329 IT IS NOT -- I MEAN, I DON'T 12221 07:28:09,395 --> 07:28:10,630 THINK YOU ESTABLISH PROMOTION. 12222 07:28:10,697 --> 07:28:13,700 WE DON'T DO THAT. 12223 07:28:13,766 --> 07:28:15,768 IT HAPPENS SOMEWHERE ELSE, SO 12224 07:28:15,835 --> 07:28:17,704 THE FOCUS, THE FOCUS, AND THE 12225 07:28:17,770 --> 07:28:19,205 DIRECTION, IS REALLY TO HELP 12226 07:28:19,272 --> 07:28:27,347 DEPARTMENT AND PEERS UNDERSTAND 12227 07:28:27,413 --> 07:28:29,249 HOW TO JUDGE THEMSELVES. 12228 07:28:29,315 --> 07:28:30,016 >> BUT CAN INFLUENCE 12229 07:28:30,083 --> 07:28:32,085 DEPARTMENTS, CAN'T THEY? 12230 07:28:32,151 --> 07:28:32,986 I DON'T KNOW. 12231 07:28:33,052 --> 07:28:36,689 >> I CAN PUSH BACK. 12232 07:28:36,756 --> 07:28:38,625 THIS IS PERFECT TIMING. 12233 07:28:38,691 --> 07:28:40,960 WHEN WE RUN WORKSHOPS WITH 12234 07:28:41,027 --> 07:28:43,263 ACADEMICS OR WITH UNIVERSITY 12235 07:28:43,329 --> 07:28:44,130 LEADERSHIP, THE UNIVERSITY 12236 07:28:44,197 --> 07:28:45,031 LEADERSHIP SAYS IT'S THE FACULTY 12237 07:28:45,098 --> 07:28:46,766 THAT DECIDES. 12238 07:28:46,833 --> 07:28:48,768 THEY HAVE TO DO THE REFORM. 12239 07:28:48,835 --> 07:28:50,803 FACULTY SAY I CAN'T DO THIS 12240 07:28:50,870 --> 07:28:52,972 WITHOUT SUPPORT FROM LEADERSHIP. 12241 07:28:53,039 --> 07:28:54,307 SOMETHING IS FAILING IN THE 12242 07:28:54,374 --> 07:28:55,074 COMMUNICATION THERE. 12243 07:28:55,141 --> 07:28:56,276 LIKE AGAIN WE HAVE TO TALK TO 12244 07:28:56,342 --> 07:28:57,143 ONE ANOTHER. 12245 07:28:57,210 --> 07:28:59,145 WE HAVE TO GET THOSE PEOPLE IN 12246 07:28:59,212 --> 07:29:01,414 THE SAME ROOM BECAUSE I REALIZE 12247 07:29:01,481 --> 07:29:02,815 IT DOESN'T HAPPEN AT THE 12248 07:29:02,882 --> 07:29:05,051 DEPARTMENT LEVEL BECAUSE WE DO 12249 07:29:05,118 --> 07:29:07,353 HAVE DISCIPLINARY NORMS AND 12250 07:29:07,420 --> 07:29:08,821 STANDARDS BUT THERE NEEDS TO BE 12251 07:29:08,888 --> 07:29:10,189 AGREEMENT FROM LEADERSHIP TO SAY 12252 07:29:10,256 --> 07:29:12,058 TO FACULTY I'M EMPOWERING YOU TO 12253 07:29:12,125 --> 07:29:13,693 GO FORWARD WITH THIS REFORM. 12254 07:29:13,760 --> 07:29:15,094 WE NEED BOTH. 12255 07:29:15,161 --> 07:29:15,895 >> SHE'S RIGHT. 12256 07:29:15,962 --> 07:29:19,532 AND SO ROBERT AND I IN OUR ROLES 12257 07:29:19,599 --> 07:29:22,335 AS VPs FOR RESEARCH DON'T 12258 07:29:22,402 --> 07:29:24,003 MANAGE ACADEMIC AFFAIRS. 12259 07:29:24,070 --> 07:29:25,605 WE CAN'T NECESSARILY TALK TO THE 12260 07:29:25,672 --> 07:29:27,940 SPECIFICS BUT OUR PROVOST AND 12261 07:29:28,007 --> 07:29:29,442 ACADEMIC OFFICERS DO, AT VCU 12262 07:29:29,509 --> 07:29:31,477 THIS ISSUE HAS COME UP. 12263 07:29:31,544 --> 07:29:35,581 SO LED BY THE PROVOST'S OFFICE, 12264 07:29:35,648 --> 07:29:37,550 THEY HAVE DONE AN ENTIRE 12265 07:29:37,617 --> 07:29:38,418 ACADEMIC TRANSFORMATION WHICH 12266 07:29:38,484 --> 07:29:43,923 WAS LED THROUGH A FACULTY AND 12267 07:29:43,990 --> 07:29:44,724 ADMINISTRATIVE INTEGRATION THAT 12268 07:29:44,791 --> 07:29:46,559 NEEDS TO BE DEPLOYED. 12269 07:29:46,626 --> 07:29:47,960 MAYBE I'LL BE BACK A YEAR FROM 12270 07:29:48,027 --> 07:29:49,162 NOW AND REPORT HOW THAT'S 12271 07:29:49,228 --> 07:29:51,130 LOOKING BUT THESE ARE EXACTLY 12272 07:29:51,197 --> 07:29:52,298 THE ISSUES, YEAH. 12273 07:29:52,365 --> 07:29:58,438 >> SO, THANKS FOR SETTING ME UP. 12274 07:29:58,504 --> 07:30:00,073 MIKE DOUGHERTY, DEPARTMENT 12275 07:30:00,139 --> 07:30:02,275 CHAIR, MID-LEVEL MANAGEMENT, 12276 07:30:02,342 --> 07:30:09,282 MATTER TO NOBODY BUT EVERYBODY 12277 07:30:09,349 --> 07:30:10,183 SIMULTANEOUSLY. 12278 07:30:10,249 --> 07:30:10,717 REFORMING ACADEMIC SYSTEM, 12279 07:30:10,783 --> 07:30:11,718 INCENTIVE SYSTEM WITHIN MY 12280 07:30:11,784 --> 07:30:16,522 DEPARTMENT, THIS HAS BEEN FROM 12281 07:30:16,589 --> 07:30:18,157 HIRING UP THROUGH TENURE 12282 07:30:18,224 --> 07:30:19,826 REVIEWS, PROMOTION, ONE THING I 12283 07:30:19,892 --> 07:30:22,261 LEARNED EARLY ON FACULTY THOUGHT 12284 07:30:22,328 --> 07:30:23,730 THEY COULDN'T DO THINGS THEY 12285 07:30:23,796 --> 07:30:25,465 REALLY HAVE A LOT OF POWER OVER. 12286 07:30:25,531 --> 07:30:27,667 SO I WOULD GO TO MY FACULTY AND 12287 07:30:27,734 --> 07:30:29,769 SAY WE NEED TO REFORM THESE 12288 07:30:29,836 --> 07:30:30,002 SYSTEMS. 12289 07:30:30,069 --> 07:30:33,139 THEY SAID WELL, ARE WE ALLOWED? 12290 07:30:33,206 --> 07:30:35,241 I JUST TALKED TO ASSOCIATE DEAN 12291 07:30:35,308 --> 07:30:36,242 FOR FACULTY AFFAIRS, YEAH, WE'RE 12292 07:30:36,309 --> 07:30:38,745 THE ONES WHO DECIDE. 12293 07:30:38,811 --> 07:30:44,183 WE DID THINGS LIKE NOW HAVE 12294 07:30:44,250 --> 07:30:47,920 PROMOTION AND TENURE GUIDELINES, 12295 07:30:47,987 --> 07:30:49,756 RESEARCH QUALITY AND RIGOR. 12296 07:30:49,822 --> 07:30:51,958 WE ELIMINATED ALL THOSE METRICS 12297 07:30:52,024 --> 07:30:53,359 PEOPLE ARE COMPLAINING ABOUT, WE 12298 07:30:53,426 --> 07:30:58,197 NO LONGER LOOK AT IMPACT FACTOR 12299 07:30:58,264 --> 07:30:59,599 CITATION COUNTS, NOW EXCLUDED, 12300 07:30:59,665 --> 07:31:01,134 SPECIFICALLY MENTIONED WE DO NOT 12301 07:31:01,200 --> 07:31:03,369 USE THESE BECAUSE THERE'S NO 12302 07:31:03,436 --> 07:31:06,205 DATA TO SUPPORT THEIR USE. 12303 07:31:06,272 --> 07:31:07,707 IT'S INCREDIBLY LIBERATING FROM 12304 07:31:07,774 --> 07:31:10,209 THE FACULTY SIDE TO KNOW, HEY, I 12305 07:31:10,276 --> 07:31:11,778 DON'T HAVE TO PURSUE NATURE, 12306 07:31:11,844 --> 07:31:15,915 DON'T HAVE TO PURSUE SCIENCE, I 12307 07:31:15,982 --> 07:31:19,085 CAN PUBLISH WITH MY SCIENCE BEST 12308 07:31:19,152 --> 07:31:21,721 FITS, DIFFERENT DEPENDING ON 12309 07:31:21,788 --> 07:31:22,255 SUBDISCIPLINE. 12310 07:31:22,321 --> 07:31:23,122 NOT EVERYBODY IS DOING WORK 12311 07:31:23,189 --> 07:31:27,093 THAT'S USEFUL FOR THE GENERAL 12312 07:31:27,160 --> 07:31:28,027 SCIENCE OR NATURE AUDIENCE, 12313 07:31:28,094 --> 07:31:29,796 MIGHT NEED TO GO TO A MORE 12314 07:31:29,862 --> 07:31:31,097 SPECIALIZED JOURNAL. 12315 07:31:31,164 --> 07:31:31,697 THAT'S OKAY. 12316 07:31:31,764 --> 07:31:33,099 THERE'S A DIVERSITY OF DIFFERENT 12317 07:31:33,166 --> 07:31:34,434 TIMES OF RESEARCH PEOPLE DO THAT 12318 07:31:34,500 --> 07:31:36,302 NEED TO GO TO OUTLETS AND WE 12319 07:31:36,369 --> 07:31:37,403 SHOULD BE EMPOWERING PEOPLE TO 12320 07:31:37,470 --> 07:31:40,740 PUT THE WORK WHERE IT'S GOING TO 12321 07:31:40,807 --> 07:31:43,943 HAVE VISIBILITY FOR COMMUNITIES 12322 07:31:44,010 --> 07:31:47,280 THEY SERVE, WHETHER IT'S, YOU 12323 07:31:47,346 --> 07:31:49,048 KNOW, VERY SPECIFIC COMMUNITY 12324 07:31:49,115 --> 07:31:50,149 THAT NEEDS A SPECIFIC SOLUTION 12325 07:31:50,216 --> 07:31:53,019 THAT'S GOOD FOR THEM BUT MAY NOT 12326 07:31:53,085 --> 07:31:54,754 APPLY TO ANYBODY ELSE OR MAYBE A 12327 07:31:54,821 --> 07:31:55,855 BASIC SCIENCE SOLUTION. 12328 07:31:55,922 --> 07:31:58,458 THE OTHER THING I'VE LEARNED 12329 07:31:58,524 --> 07:31:59,892 FROM EARLY CAREER PERSPECTIVE 12330 07:31:59,959 --> 07:32:01,761 THIS HAS BECOME A RECRUITMENT 12331 07:32:01,828 --> 07:32:02,161 TOOL. 12332 07:32:02,228 --> 07:32:04,630 EVERYBODY COMES INTO MY OFFICE, 12333 07:32:04,697 --> 07:32:05,932 RECRUITING FACULTY, THEY SAY SO 12334 07:32:05,998 --> 07:32:07,700 I HEAR YOU'VE REFORMED YOUR 12335 07:32:07,767 --> 07:32:09,735 TENURE AND PROMOTION, TELL ME 12336 07:32:09,802 --> 07:32:10,303 ABOUT THIS. 12337 07:32:10,369 --> 07:32:12,038 AND THEY ARE ON BOARD WITH A LOT 12338 07:32:12,104 --> 07:32:13,739 OF THESE THINGS WE'VE BEEN 12339 07:32:13,806 --> 07:32:15,441 TALKING ABOUT HERE, REALLY WANT 12340 07:32:15,508 --> 07:32:16,976 TO SEE REFORM EFFORTS HAPPEN. 12341 07:32:17,043 --> 07:32:20,580 THIS IS A GREAT WORKSHOP TO HELP 12342 07:32:20,646 --> 07:32:21,681 PUSH THROUGH. 12343 07:32:21,747 --> 07:32:22,815 IT'S BIGGER THAN PUBLICATION 12344 07:32:22,882 --> 07:32:24,951 BIAS, THAT'S A HUGE PART OF IT, 12345 07:32:25,017 --> 07:32:27,487 AND SO I THINK THIS IS THE RIGHT 12346 07:32:27,553 --> 07:32:32,992 FORUM TO BE ADDRESSING ISSUES. 12347 07:32:33,059 --> 07:32:33,860 TACKLING AT SYSTEMIC LEVEL, BUT 12348 07:32:33,926 --> 07:32:36,362 ONE THING THAT COMES UP WHEN I 12349 07:32:36,429 --> 07:32:38,331 PRESENT ON THIS IS ARE YOU 12350 07:32:38,397 --> 07:32:39,866 PUTTING YOUR GRADUATE STUDENTS 12351 07:32:39,932 --> 07:32:41,934 AT DISADVANTAGE? 12352 07:32:42,001 --> 07:32:44,036 BECAUSE THEY LOOK AT THE SYSTEM. 12353 07:32:44,103 --> 07:32:45,972 WE'RE TRAINING STUDENTS, PUTTING 12354 07:32:46,038 --> 07:32:47,673 THEM IN A SYSTEM WHERE WE'RE 12355 07:32:47,740 --> 07:32:48,741 INCENTIVIZING THEM TO DO THE 12356 07:32:48,808 --> 07:32:50,176 WORK THAT MATTERS TO THEM, 12357 07:32:50,243 --> 07:32:52,879 PUBLISH IN A PLACE THAT MATTERS 12358 07:32:52,945 --> 07:32:54,514 TO THEM, AND THEN THE QUESTION 12359 07:32:54,580 --> 07:32:56,616 IS, WELL, DOES THAT PUT THEM AT 12360 07:32:56,682 --> 07:32:57,383 DISADVANTAGE? 12361 07:32:57,450 --> 07:33:00,152 TO ME THAT'S A PERCEPTION 12362 07:33:00,219 --> 07:33:02,889 PROBLEM ABOUT THE SYSTEM. 12363 07:33:02,955 --> 07:33:04,624 THE SYSTEM THEY ARE GOING TO BE 12364 07:33:04,690 --> 07:33:06,626 IN, AND SOME OF THEM SEE THAT 12365 07:33:06,692 --> 07:33:07,693 SYSTEM AND THINK I DON'T WANT TO 12366 07:33:07,760 --> 07:33:08,761 BE PART OF IT. 12367 07:33:08,828 --> 07:33:10,630 THEY OPT OUT. 12368 07:33:10,696 --> 07:33:12,465 THAT'S A HUGE PROBLEM. 12369 07:33:12,532 --> 07:33:14,200 WE HAVE PUBLICATION BIAS, 12370 07:33:14,267 --> 07:33:15,868 SELECTING CERTAIN THINGS TO BE 12371 07:33:15,935 --> 07:33:16,969 PUBLISHED, SELECTING CERTAIN 12372 07:33:17,036 --> 07:33:21,040 PEOPLE TO BE PART OF THAT 12373 07:33:21,107 --> 07:33:21,340 SYSTEM. 12374 07:33:21,407 --> 07:33:23,876 >> THANK YOU, I'M NOT GOING TO 12375 07:33:23,943 --> 07:33:28,414 ASK THE PANEL TO RESPOND BECAUSE 12376 07:33:28,481 --> 07:33:29,782 WE'RE HAVE FOUR MINUTES. 12377 07:33:29,849 --> 07:33:36,956 >> I JUST WANT TO PUT A PLUG IN. 12378 07:33:37,023 --> 07:33:39,191 I JUST PUBLISHED A PAPER, WE'RE 12379 07:33:39,258 --> 07:33:42,662 DOING EXACTLY WHAT YOU 12380 07:33:42,728 --> 07:33:44,330 SUGGESTED, WE MEASURE FORGET 12381 07:33:44,397 --> 07:33:48,367 WHERE YOU PUBLISHED, HOW MANY 12382 07:33:48,434 --> 07:33:50,236 CITATIONS IN ABSOLUTE SENSE, BUT 12383 07:33:50,303 --> 07:33:52,138 WE'D LIKE TO TABULATE WHEREVER 12384 07:33:52,204 --> 07:33:54,640 YOU PUBLISHED, SEE HOW WELL YOU 12385 07:33:54,707 --> 07:33:55,708 ARE FARING RELATIVE TO OTHER 12386 07:33:55,775 --> 07:33:58,611 PEOPLE IN THE SAME JOURNAL, SAME 12387 07:33:58,678 --> 07:34:01,213 YEAR, AND METRIC CALLED THE 12388 07:34:01,280 --> 07:34:02,214 P-INDEX, PROVIDES A VERY 12389 07:34:02,281 --> 07:34:05,084 DIFFERENT WAY OF LOOKING AT WHAT 12390 07:34:05,151 --> 07:34:06,419 WE CALL SOFT LEADERSHIP, SO THAT 12391 07:34:06,485 --> 07:34:08,421 YOU STILL WANT TO BE ABLE TO 12392 07:34:08,487 --> 07:34:11,257 MEASURE SOME KIND OF QUALITY 12393 07:34:11,324 --> 07:34:13,559 INDEX BUT STILL RETAIN THE 12394 07:34:13,626 --> 07:34:15,761 FLEXIBILITY, THE FREEDOM FOR 12395 07:34:15,828 --> 07:34:19,098 RESEARCHER TO NOT BE BOUND BY 12396 07:34:19,165 --> 07:34:22,234 H-INDEX OR WHATEVER THE JOURNAL 12397 07:34:22,301 --> 07:34:25,404 THAT MAXIMIZES THAT CITATION. 12398 07:34:25,471 --> 07:34:27,239 IN OUR DATASET, H-INDEX IS THE 12399 07:34:27,306 --> 07:34:30,309 CORRELATION WITH THE NUMBER OF 12400 07:34:30,376 --> 07:34:35,281 PUBLICATION, .93, NUMBER OF 12401 07:34:35,348 --> 07:34:38,184 CITATIONS, NUMBER OF 12402 07:34:38,250 --> 07:34:39,952 PUBLICATIONS, . 89, NOT ADDING 12403 07:34:40,019 --> 07:34:43,222 INFORMATION OVER AND ABOVE 12404 07:34:43,289 --> 07:34:46,325 PRODUCTIVITY, METRICS WE HAVE 12405 07:34:46,392 --> 07:34:47,727 PROVIDE POTENTIAL COMPLEMENT. 12406 07:34:47,793 --> 07:34:49,228 THE OTHER POINT THAT WE HAVEN'T 12407 07:34:49,295 --> 07:34:53,099 TALKED ABOUT GOING BACK TO THE 12408 07:34:53,165 --> 07:34:56,402 NULL EFFECT, I DO THINK THAT WE 12409 07:34:56,469 --> 07:35:01,273 NEED TO BE PRECISE. 12410 07:35:01,340 --> 07:35:03,809 THERE'S A WHOLE RANGE OF THINGS 12411 07:35:03,876 --> 07:35:09,548 THAT COULD PRODUCE NULL RESULTS. 12412 07:35:09,615 --> 07:35:10,883 SOME NULL RESULTS HAVE MORE 12413 07:35:10,950 --> 07:35:11,951 INFORMATION THAN OTHERS. 12414 07:35:12,018 --> 07:35:15,221 IF YOU PUBLISH A SILLY STUDY, 12415 07:35:15,287 --> 07:35:19,291 THAT NOBODY CARESB I -- CARES 12416 07:35:19,358 --> 07:35:20,860 ABOUT, I DON'T THINK WE'RE 12417 07:35:20,926 --> 07:35:24,497 HELPING THE FIELD. 12418 07:35:24,563 --> 07:35:26,365 EARLIER A PAPER DEBUNKED A 12419 07:35:26,432 --> 07:35:27,933 COMMON INTERPRETATION OF 12420 07:35:28,000 --> 07:35:35,274 MOLECULAR EFFECT, IF YOU CAN -- 12421 07:35:35,341 --> 07:35:37,777 WE HAVE TO BE MORE PRECISE. 12422 07:35:37,843 --> 07:35:38,577 >> THANK YOU. 12423 07:35:38,644 --> 07:35:41,080 WE'RE GOING TO TRY ONE MORE 12424 07:35:41,147 --> 07:35:41,547 QUESTION. 12425 07:35:41,614 --> 07:35:44,383 >> THIS IS A QUESTION FOR ERIN 12426 07:35:44,450 --> 07:35:44,717 MAYBE. 12427 07:35:44,784 --> 07:35:47,019 I'M WONDERING, I LOVE YOUR 12428 07:35:47,086 --> 07:35:50,322 ANALYSES OF WHAT GOES INTO THESE 12429 07:35:50,389 --> 07:35:51,290 DOCUMENTS AND EVERYTHING. 12430 07:35:51,357 --> 07:35:54,460 THIS WHOLE THING IS LIKE A GAME 12431 07:35:54,527 --> 07:35:58,264 OF PROXY AND METRICS, METRIC 12432 07:35:58,330 --> 07:36:00,232 BECOMES A GOAL, WHETHER OR NOT 12433 07:36:00,299 --> 07:36:02,401 THERE ARE ASPECTS IF YOU HAVE 12434 07:36:02,468 --> 07:36:07,907 DATA ON ASPECT FOR WHICH THERE'S 12435 07:36:07,973 --> 07:36:09,608 MORE DISCRETION, YOU COUNT ON 12436 07:36:09,675 --> 07:36:10,876 EXTERNAL REVIEWERS TO WEIGH IN 12437 07:36:10,943 --> 07:36:12,912 ON THOSE THINGS BUT SELECT THOSE 12438 07:36:12,978 --> 07:36:14,313 EXTERNAL REVIEWERS AND DEPENDING 12439 07:36:14,380 --> 07:36:15,181 WHO YOU SELECT THERE'S 12440 07:36:15,247 --> 07:36:16,615 DISCRETION THERE, WHO YOU SELECT 12441 07:36:16,682 --> 07:36:18,484 AND WHAT THEY VALUE, WHAT THEY 12442 07:36:18,551 --> 07:36:21,554 SAY IS IMPACTFUL, GOING TO HAVE 12443 07:36:21,620 --> 07:36:23,689 A BIG EFFECT HERE. 12444 07:36:23,756 --> 07:36:26,125 CAN THERE BE SOME -- ESPECIALLY 12445 07:36:26,192 --> 07:36:29,295 IT'S ALWAYS SOMEBODY ELSE'S 12446 07:36:29,361 --> 07:36:30,830 FAULT, IT'S UP TO SOMEONE TO 12447 07:36:30,896 --> 07:36:32,431 DECIDE WHO TO ASK, PEOPLE THAT 12448 07:36:32,498 --> 07:36:33,499 MIGHT VALUE ONE THING OR 12449 07:36:33,566 --> 07:36:36,102 ANOTHER, DO YOU HAVE ANY DATA ON 12450 07:36:36,168 --> 07:36:39,739 LIKE THE STANDARDIZATION OF LIKE 12451 07:36:39,805 --> 07:36:40,506 PRACTICES FOR EXTERNAL 12452 07:36:40,573 --> 07:36:40,906 REVIEWERS? 12453 07:36:40,973 --> 07:36:41,173 >> YES. 12454 07:36:41,240 --> 07:36:42,708 IT'S A GOOD QUESTION. 12455 07:36:42,775 --> 07:36:44,877 I MEAN, WE DIDN'T LOOK 12456 07:36:44,944 --> 07:36:46,412 SPECIFICALLY AT EXTERNAL 12457 07:36:46,479 --> 07:36:49,048 REVIEWER COMPONENT LIKE WHAT THE 12458 07:36:49,115 --> 07:36:50,015 TENURE AND PROMOTION DOCUMENTS 12459 07:36:50,082 --> 07:36:51,917 WERE SAYING ABOUT EXTERNAL 12460 07:36:51,984 --> 07:36:52,184 REVIEWERS. 12461 07:36:52,251 --> 07:36:54,620 I WILL SAY TOO THAT I THINK 12462 07:36:54,687 --> 07:36:56,188 THERE'S AN ASSUMPTION ALL 12463 07:36:56,255 --> 07:36:57,123 UNIVERSITIES HAVE EXTERNAL 12464 07:36:57,189 --> 07:36:58,858 REVIEW, MY UNIVERSITY DOESN'T. 12465 07:36:58,924 --> 07:37:00,760 SO IT'S NOT PART OF THE 12466 07:37:00,826 --> 07:37:02,495 PROMOTION AND TENURE PROCESS. 12467 07:37:02,561 --> 07:37:04,597 I THINK THAT'S ANOTHER THING. 12468 07:37:04,663 --> 07:37:06,565 I THINK YOUR QUESTION IS 12469 07:37:06,632 --> 07:37:07,767 PROBABLY MORE BROADER ABOUT LIKE 12470 07:37:07,833 --> 07:37:12,471 HOW MUCH WIGGLE ROOM IS ACTUALLY 12471 07:37:12,538 --> 07:37:18,611 IN THERE. 12472 07:37:18,677 --> 07:37:20,679 IT'S TRUE THERE'S WHAT'S WRONG, 12473 07:37:20,746 --> 07:37:23,482 HOW IT'S INTERPRETED 12474 07:37:23,549 --> 07:37:28,020 INTERPRETED AND CLOSED DOOR 12475 07:37:28,087 --> 07:37:30,189 DISCUSSIONS, NOT ONE-TO-ONE. 12476 07:37:30,256 --> 07:37:32,758 WE DID SEE INTENTIONAL VAGUENESS 12477 07:37:32,825 --> 07:37:34,493 MEANT TO PROVIDE EXACTLY THAT. 12478 07:37:34,560 --> 07:37:37,930 WE DID ANALYSIS, THE TERM 12479 07:37:37,997 --> 07:37:38,631 COLLEGIALITY WHICH I HATE, 12480 07:37:38,697 --> 07:37:42,368 PRECISELY BECAUSE IT IS USED IN 12481 07:37:42,434 --> 07:37:44,069 THAT IF YOU'RE GETTING ALONG, IT 12482 07:37:44,136 --> 07:37:45,738 CAN BE USED IN YOUR FAVOR AND 12483 07:37:45,805 --> 07:37:50,309 ALSO AGAINST YOU IN REALLY 12484 07:37:50,376 --> 07:37:51,010 BIASED WAYS. 12485 07:37:51,076 --> 07:37:52,745 THERE'S A LOT OF INTENTIONAL 12486 07:37:52,812 --> 07:37:53,979 VAGUENESS IN THE DOCUMENTS. 12487 07:37:54,046 --> 07:37:55,815 WE DON'T HAVE THE COMPLETE 12488 07:37:55,881 --> 07:37:59,752 PICTURE BY LOOKING AT THE DOCS 12489 07:37:59,819 --> 07:38:03,189 BUT IT GETS US CLOSER. 12490 07:38:03,255 --> 07:38:06,625 >> IT'S PAST 6:00. 12491 07:38:06,692 --> 07:38:07,993 >> WE'LL WRAP THIS DISCUSSION. 12492 07:38:08,060 --> 07:38:14,099 THANK YOU TO THE MODERATOR AND 12493 07:38:14,166 PANELISTS.