1 00:00:05,305 --> 00:00:06,906 >> WELCOME, EVERYONE, AND 2 00:00:06,973 --> 00:00:08,274 THANK YOU FOR JOINING US 3 00:00:08,341 --> 00:00:10,009 THIS AFTERNOON. 4 00:00:10,076 --> 00:00:11,511 I'M LANA YEGANOVA, I'M 5 00:00:11,578 --> 00:00:12,746 YOUR HOST FOR TODAY'S 6 00:00:12,812 --> 00:00:15,081 EVENT ORGANIZED BY 7 00:00:15,148 --> 00:00:17,350 METROLANGUAGE PROCESSING 8 00:00:17,417 --> 00:00:18,385 AND SPECIAL INTEREST GROUP 9 00:00:18,451 --> 00:00:20,186 HERE AT NIH. 10 00:00:20,253 --> 00:00:22,021 WE'RE EXTREMELY EXCITED TO 11 00:00:22,088 --> 00:00:23,289 HAVE TODAY DR. TRISTAN 12 00:00:23,356 --> 00:00:26,326 NAUMANN JOIN US FOR THE 13 00:00:26,393 --> 00:00:26,659 SEMINAR. 14 00:00:26,726 --> 00:00:27,761 BEFORE I INTRODUCE OUR 15 00:00:27,827 --> 00:00:29,896 WONDERFUL SPEAKER, I WOULD 16 00:00:29,963 --> 00:00:31,965 LIKE TO MENTION THAT AS 17 00:00:32,031 --> 00:00:33,967 USUAL WE'LL HAVE A FEW 18 00:00:34,033 --> 00:00:35,468 MINUTES AT END OF THE TALK 19 00:00:35,535 --> 00:00:37,237 FOR QUESTIONS AND ANSWERS. 20 00:00:37,303 --> 00:00:38,671 PLEASE SEND YOUR QUESTIONS 21 00:00:38,738 --> 00:00:43,143 IN BY THIS E-MAIL TO ME. 22 00:00:43,209 --> 00:00:51,017 THIS IS WHERE YOU GET 23 00:00:51,084 --> 00:00:51,785 NOTIFICATIONS FROM. 24 00:00:51,851 --> 00:00:52,419 YOU WOULD LIKE TO 25 00:00:52,485 --> 00:00:54,120 INTRODUCE OUR SPEAKER. 26 00:00:54,187 --> 00:00:57,457 WE ARE EXTREMELY EXCITED 27 00:00:57,524 --> 00:01:00,126 TO DOCTOR DR. TRISTAN 28 00:01:00,193 --> 00:01:04,264 NAUMANN, HE'S A PRINCIPAL 29 00:01:04,330 --> 00:01:04,764 RESEARCHER AT MICROSOFT 30 00:01:04,831 --> 00:01:07,367 RESEARCH'S HEALTH FUTURES. 31 00:01:07,434 --> 00:01:09,335 HIS RESEARCH FOCUSES ON 32 00:01:09,402 --> 00:01:11,171 PROBLEMS OF INTERSECTION 33 00:01:11,237 --> 00:01:14,107 OF MACHINE LEARNING AND 34 00:01:14,174 --> 00:01:14,507 HEALTH EXPLORING 35 00:01:14,574 --> 00:01:15,475 RELATIONSHIP IN COMPLEX 36 00:01:15,542 --> 00:01:17,811 AND STRUCTURED HEALTH DATA 37 00:01:17,877 --> 00:01:19,045 USING NATURAL LANGUAGE 38 00:01:19,112 --> 00:01:21,314 PROCESSING AND 39 00:01:21,381 --> 00:01:21,781 UNSUPERVISED LEARNING. 40 00:01:21,848 --> 00:01:23,817 HE VALUES SUPPORTING BRO 41 00:01:23,883 --> 00:01:26,152 BROADER MACHINE LEARNING 42 00:01:26,219 --> 00:01:28,254 COMMUNITY THROUGH ACADEMIC 43 00:01:28,321 --> 00:01:29,422 SERVICE AND HAS SERVED AS 44 00:01:29,489 --> 00:01:31,157 GENERAL CHAIR AND A 45 00:01:31,224 --> 00:01:32,859 VARIETY OF OTHER ROLES AT 46 00:01:32,926 --> 00:01:34,928 THE NEUROINFORMATION 47 00:01:34,994 --> 00:01:37,497 PROCESSING SYSTEMS 48 00:01:37,564 --> 00:01:40,800 CONFERENCE, HLI, ON HEALTH 49 00:01:40,867 --> 00:01:44,170 INFERENCE AND LEARNING. 50 00:01:44,237 --> 00:01:46,406 WITH THAT, TRISTAN, THE 51 00:01:46,473 --> 00:01:47,574 FLOOR IS YOURS. 52 00:01:47,640 --> 00:01:52,111 >> YEAH, THANK YOU, I 53 00:01:52,178 --> 00:01:54,547 REALLY APPRECIATE THE KIND 54 00:01:54,614 --> 00:01:55,682 INTRODUCTION AND I THINK 55 00:01:55,748 --> 00:01:57,984 IN THE SAME LIGHT AS IT 56 00:01:58,051 --> 00:02:00,920 WAS MENTIONED I ENJOY 57 00:02:00,987 --> 00:02:01,354 SUPPORTING BROADER 58 00:02:01,421 --> 00:02:03,189 ACADEMIC COMMUNITY, I CAN 59 00:02:03,256 --> 00:02:04,624 THINK OF FEW OTHER PLACES 60 00:02:04,691 --> 00:02:06,759 THAT SUPPORTED 61 00:02:06,826 --> 00:02:08,328 INTERSEC 62 00:02:08,394 --> 00:02:08,962 INTERSECTIONAL COMMUNITY 63 00:02:09,028 --> 00:02:10,897 AND I'M DELIGHTED TO BE A 64 00:02:10,964 --> 00:02:14,300 PART OF THIS MORE THAN 65 00:02:14,367 --> 00:02:16,035 NIH, I MEAN, I THINK FROM 66 00:02:16,102 --> 00:02:18,071 MY OWN HISTORY, I CAN SAY, 67 00:02:18,137 --> 00:02:20,273 I WOULD NOT BE HERE 68 00:02:20,340 --> 00:02:21,541 WITHOUT SUPPORT OF NIH. 69 00:02:21,608 --> 00:02:23,877 I WAS ON RESEARCH TRAINING 70 00:02:23,943 --> 00:02:25,478 GRANT DURING MY GRADUATE 71 00:02:25,545 --> 00:02:27,747 STUDIES AND SO MANY OF US 72 00:02:27,814 --> 00:02:28,948 REALLY WERE, I'M REALLY 73 00:02:29,015 --> 00:02:30,650 QUITE HONORED TO BE HERE 74 00:02:30,717 --> 00:02:32,385 TODAY AND DELIGHTED TO 75 00:02:32,452 --> 00:02:34,087 TELL YOU A BIT ABOUT THE 76 00:02:34,153 --> 00:02:35,588 WORK GOING ON IN OUR 77 00:02:35,655 --> 00:02:38,758 GROUP, REAL-WORLD EVIDENCE 78 00:02:38,825 --> 00:02:41,427 AT MICROSOFT RESEARCH 79 00:02:41,494 --> 00:02:42,795 HEALTH FUTURES AND HOW IT 80 00:02:42,862 --> 00:02:45,031 IS SHIFTING IN THE AGE OF 81 00:02:45,098 --> 00:02:47,233 L 82 00:02:47,300 --> 00:02:47,433 LLM'S. 83 00:02:47,500 --> 00:02:49,235 LET ME GET STARTED AND 84 00:02:49,302 --> 00:02:50,904 PROVIDE AN IDEA HOW DID I 85 00:02:50,970 --> 00:02:52,505 GET INTO THIS SPACE. 86 00:02:52,572 --> 00:02:54,841 AS I SAID, I STARTED OFF 87 00:02:54,908 --> 00:02:55,875 AS A GRADUATE STUDENT WITH 88 00:02:55,942 --> 00:02:58,511 THIS IDEA THAT NLP WOULD 89 00:02:58,578 --> 00:03:00,146 BE KEY TO UNLOCKING THE 90 00:03:00,213 --> 00:03:01,681 FUTURE OF MEDICINE FROM 91 00:03:01,748 --> 00:03:04,417 CLINICAL TEXT AND THE 92 00:03:04,484 --> 00:03:05,818 INTUITION WAS THIS NOTION 93 00:03:05,885 --> 00:03:07,754 FOR A LOT OF THINGS IN 94 00:03:07,820 --> 00:03:08,988 MEDICINE, MAYBE WE KNOW 95 00:03:09,055 --> 00:03:11,024 THAT HEART RATE IS A VERY 96 00:03:11,090 --> 00:03:12,959 GOOD INDICATOR OF 97 00:03:13,026 --> 00:03:14,494 SOMETHING LIKE 98 00:03:14,561 --> 00:03:14,994 CARDIOVASCULAR HEALTH. 99 00:03:15,061 --> 00:03:17,230 WE COLLECTIVELY AS SOCIETY 100 00:03:17,297 --> 00:03:18,665 PUT FORWARD AN EFFORT TO 101 00:03:18,731 --> 00:03:19,866 MEASURE THAT AND 102 00:03:19,933 --> 00:03:21,200 INSTRUMENT WAYS OF MEASURE 103 00:03:21,267 --> 00:03:22,669 THAT WITH HIGH FREQUENCY, 104 00:03:22,735 --> 00:03:24,137 HIGH FIDELITY. 105 00:03:24,203 --> 00:03:25,872 ALL THE OTHER THINGS WHERE 106 00:03:25,939 --> 00:03:27,574 WE THINK ARE IMPORTANT, WE 107 00:03:27,640 --> 00:03:29,075 DON'T KNOW THEY ARE 108 00:03:29,142 --> 00:03:30,176 IMPORTANT FOR FUTURE 109 00:03:30,243 --> 00:03:31,210 VERSION OF OURSELVES OR 110 00:03:31,277 --> 00:03:32,712 SOMEBODY ELSE IN THE CARE 111 00:03:32,779 --> 00:03:35,515 CHAIN, ALL THAT COULD GET 112 00:03:35,582 --> 00:03:36,649 DUMPED INTO THE NOTES 113 00:03:36,716 --> 00:03:38,151 SOMEWHERE. 114 00:03:38,217 --> 00:03:39,118 INTUITION WAS THIS IDEA IF 115 00:03:39,185 --> 00:03:42,188 WE WANT TO EXPAND OUR 116 00:03:42,255 --> 00:03:43,623 CLINICAL CAPACITY MAYBE WE 117 00:03:43,690 --> 00:03:45,124 SHOULD BE LOOKING TO SOME 118 00:03:45,191 --> 00:03:47,193 THINGS IN THE NOTES. 119 00:03:47,260 --> 00:03:49,529 IN SORT OF SAME LINE OF 120 00:03:49,596 --> 00:03:53,700 THINKING, WE PROBABLY HAVE 121 00:03:53,766 --> 00:03:55,768 MANY DIFFERENT TYPES OF 122 00:03:55,835 --> 00:03:57,303 DATA ABOUT NOTES AND WHILE 123 00:03:57,370 --> 00:03:59,872 I WAS REALLY FOCUSED ON 124 00:03:59,939 --> 00:04:03,643 NATURAL LANGUAGE AND TEXT 125 00:04:03,710 --> 00:04:04,711 AND ENJOY THAT AREA, IT 126 00:04:04,777 --> 00:04:06,713 WOULD BE MAYBE MISLEADING 127 00:04:06,779 --> 00:04:08,681 TO FOCUS ON ONE ASPECT OF 128 00:04:08,748 --> 00:04:10,516 A PATIENTS RECORD OR 129 00:04:10,583 --> 00:04:12,218 COLLECTIVE THINGS THEY 130 00:04:12,285 --> 00:04:12,485 GENERATE. 131 00:04:12,552 --> 00:04:13,753 WHAT WE WANT TO SEE IN IT 132 00:04:13,820 --> 00:04:15,622 IS FUTURE IS NOTION OF 133 00:04:15,688 --> 00:04:18,224 MULTI MODAL MODELS, IF 134 00:04:18,291 --> 00:04:21,027 TEXT WAS IDEA THE BASIS 135 00:04:21,094 --> 00:04:22,595 FOR UNLOCKING FUTURE OF 136 00:04:22,662 --> 00:04:23,997 MEDICINE, MULTI MODAL 137 00:04:24,063 --> 00:04:25,531 WOULD BE BASIS FOR 138 00:04:25,598 --> 00:04:26,332 UNLOCKING NEW 139 00:04:26,399 --> 00:04:27,400 OPPORTUNITIES IN MEDICINE, 140 00:04:27,467 --> 00:04:28,267 AS WELL. 141 00:04:28,334 --> 00:04:29,769 OF COURSE, YOU KNOW, AS WE 142 00:04:29,836 --> 00:04:32,005 ALL MAYBE HAVE TOP OF 143 00:04:32,071 --> 00:04:33,439 MIND, THERE MAY BE TWO 144 00:04:33,506 --> 00:04:34,941 THINGS WE CARE ABOUT, THE 145 00:04:35,008 --> 00:04:36,609 FIRST ONE IS OF COURSE, 146 00:04:36,676 --> 00:04:39,312 DOES IT ACTUALLY WORK? 147 00:04:39,379 --> 00:04:41,347 AND IS IT SAFE TO USE? 148 00:04:41,414 --> 00:04:41,614 RIGHT. 149 00:04:41,681 --> 00:04:42,849 YOU KNOW, IF WE THINK 150 00:04:42,915 --> 00:04:46,486 ABOUT WHAT REAL WORLD DATA 151 00:04:46,552 --> 00:04:48,287 OFFERS US, IT CAN PROVIDE 152 00:04:48,354 --> 00:04:50,623 CRUCIAL EVIDENCE THAT IS 153 00:04:50,690 --> 00:04:53,426 COMPLIMENTARY TO EXISTING 154 00:04:53,493 --> 00:04:55,061 EVIDENCE GENERATION IN 155 00:04:55,128 --> 00:04:56,796 MEDICINE INFORM THIS SAME 156 00:04:56,863 --> 00:04:57,930 VEIN, WHETHER OR NOT IT IS 157 00:04:57,997 --> 00:05:00,266 SAFE, WE MIGHT REALLY LOOK 158 00:05:00,333 --> 00:05:01,934 TO SOLUTION SPECIFICALLY 159 00:05:02,001 --> 00:05:03,536 THAT HELP MITIGATE RISK 160 00:05:03,603 --> 00:05:05,571 AND CONTINUOUSLY IMPROVE 161 00:05:05,638 --> 00:05:06,673 AND I WANT TO ACKNOWLEDGE 162 00:05:06,739 --> 00:05:08,408 THAT I THINK IF I WERE 163 00:05:08,474 --> 00:05:09,876 GIVING THIS SIMILAR TALK A 164 00:05:09,942 --> 00:05:11,077 YEAR AGO, I WOULD PROBABLY 165 00:05:11,144 --> 00:05:12,378 BE TALKING ABOUT VERY 166 00:05:12,445 --> 00:05:13,880 DIFFERENT THINGS, MAYBE 167 00:05:13,946 --> 00:05:15,915 SOME OF THEM OVERLAPPING, 168 00:05:15,982 --> 00:05:17,650 MANY OF THEM VERY 169 00:05:17,717 --> 00:05:18,818 DIFFERENT, STARTING A YEAR 170 00:05:18,885 --> 00:05:20,253 AGO, WE HAD ESPECIALLY IN 171 00:05:20,319 --> 00:05:23,756 THIS NLP SPACE THIS CALL 172 00:05:23,823 --> 00:05:28,261 TO START -- HOW WE 173 00:05:28,327 --> 00:05:29,629 APPROACH MACHINE LEARNING 174 00:05:29,696 --> 00:05:30,263 AND ARTIFICIAL 175 00:05:30,329 --> 00:05:31,631 INTELLIGENCE FOR HEALTH. 176 00:05:31,698 --> 00:05:38,404 NOT SURPRISING THAT GPT 177 00:05:38,471 --> 00:05:39,806 HAD IMPACT IT DID. 178 00:05:39,872 --> 00:05:41,107 I THINK SPECIFICALLY ONE 179 00:05:41,174 --> 00:05:42,575 OF THE MOST IMPORTANT 180 00:05:42,642 --> 00:05:44,277 ASPECT OF RELEASE OF THIS 181 00:05:44,343 --> 00:05:48,548 WAS ASSESS IBLT, A LOT OF 182 00:05:48,614 --> 00:05:49,415 CLINICIANS I WORK WITH 183 00:05:49,482 --> 00:05:51,150 WERE REACHING OUT, HEY, 184 00:05:51,217 --> 00:05:52,652 HAVE YOU TRIED THIS, LOOK 185 00:05:52,719 --> 00:05:55,421 AT THESE ANECDOTAL POINTS 186 00:05:55,488 --> 00:05:57,790 OF EVIDENCE IT IS DOING 187 00:05:57,857 --> 00:05:58,958 INCREDIBLY WELL ON CERTAIN 188 00:05:59,025 --> 00:05:59,826 TYPES OF THINGS. 189 00:05:59,892 --> 00:06:01,060 HERE ARE EXAMPLES I THINK 190 00:06:01,127 --> 00:06:01,928 IT ISSIL FAING. 191 00:06:01,994 --> 00:06:03,863 I THINK INITIAL ENGAGEMENT 192 00:06:03,930 --> 00:06:05,732 WAS MORE EXCITING THAN 193 00:06:05,798 --> 00:06:06,933 MANY OTHER THINGS I'VE 194 00:06:06,999 --> 00:06:10,470 SEEN IN RECENT TIMES 195 00:06:10,536 --> 00:06:10,970 TRYING TO MOVE MACHINE 196 00:06:11,037 --> 00:06:12,772 LEARNING INTO PRACTICE AND 197 00:06:12,839 --> 00:06:14,273 MEDICINE AND WE WERE 198 00:06:14,340 --> 00:06:16,075 SEEING IMMEDIATE DEMANFOR 199 00:06:16,142 --> 00:06:17,443 HOW CAN WE USE THIS AND 200 00:06:17,510 --> 00:06:19,345 THAT IS EXCITING PLACE TO 201 00:06:19,412 --> 00:06:19,712 SEE IT. 202 00:06:19,779 --> 00:06:21,881 YOU KNOW, MAYBE 203 00:06:21,948 --> 00:06:22,915 UNSURPRISINGLY, A LOT OF 204 00:06:22,982 --> 00:06:24,517 INITIAL STUDIES, ONE I 205 00:06:24,584 --> 00:06:26,185 LIKE FROM A YEAR AGO, 206 00:06:26,252 --> 00:06:27,553 LOOKING AT HEY, LIKE HOW 207 00:06:27,620 --> 00:06:30,189 DO SOME OF THESE TOOLS 208 00:06:30,256 --> 00:06:33,025 COMPARE TO HUMANS AND SORT 209 00:06:33,092 --> 00:06:36,729 OF SIMILAR SETTINGS AND I 210 00:06:36,796 --> 00:06:38,164 THOUGHT THIS WAS 211 00:06:38,231 --> 00:06:39,665 SURPRISING ONE IN PART. 212 00:06:39,732 --> 00:06:41,167 IF YOU ASKED ME 213 00:06:41,234 --> 00:06:42,368 BEFOREHAND, I WOULD NOT 214 00:06:42,435 --> 00:06:44,103 GUESS THIS WAS THE CASE, 215 00:06:44,170 --> 00:06:45,138 MAYBE A TOOL WOULD 216 00:06:45,204 --> 00:06:48,074 ACTUALLY GET HIGHER PR 217 00:06:48,141 --> 00:06:49,075 PREVALENCE OF PERCEIVED TO 218 00:06:49,142 --> 00:06:51,844 BE HIGH-QUALITY RESPONDS 219 00:06:51,911 --> 00:06:53,479 THAN A HUMAN WOULD. 220 00:06:53,546 --> 00:06:55,782 IT MAKES SENCE IN VARIETY 221 00:06:55,848 --> 00:06:57,683 OF CONTEXT, HUMANS ARE 222 00:06:57,750 --> 00:07:00,253 TIME BOUND AND TOOLS ARE 223 00:07:00,319 --> 00:07:00,953 NOT. 224 00:07:01,020 --> 00:07:01,988 SIMILARLY, NOW IS TIME 225 00:07:02,054 --> 00:07:03,222 MORE THAN EVER WE NEED TO 226 00:07:03,289 --> 00:07:05,191 BE THINKING ABOUT SOME OF 227 00:07:05,258 --> 00:07:06,292 THESE TOPICS, THERE HAVE 228 00:07:06,359 --> 00:07:07,960 BEEN MANY ANNOUNCEMENTS 229 00:07:08,027 --> 00:07:10,530 THAT INDICATED PREVALENCE 230 00:07:10,596 --> 00:07:12,031 THEY ARE HOLDING WITHIN 231 00:07:12,098 --> 00:07:13,299 THIS BROADER SPACE FCHL WE 232 00:07:13,366 --> 00:07:16,235 LOOK BACK TO HIMMS EARLIER 233 00:07:16,302 --> 00:07:21,774 THIS YEAR, PRIMARY EHR 234 00:07:21,841 --> 00:07:22,608 VENDORS ABOUT "HOW WE SEE 235 00:07:22,675 --> 00:07:24,644 IT" TOOLS HAVE BEEN USED 236 00:07:24,710 --> 00:07:26,879 TO GENERATE MANY, MANY, 237 00:07:26,946 --> 00:07:29,749 MANY MESSAGE DRAFTS BEING 238 00:07:29,816 --> 00:07:31,350 REVIEWED BY HUMANS, AS 239 00:07:31,417 --> 00:07:32,518 WELL, IT IS CLEAR SOME 240 00:07:32,585 --> 00:07:33,920 TOOLING IS MAKING ITS WAY 241 00:07:33,986 --> 00:07:35,621 INTO THIS SPACE. 242 00:07:35,688 --> 00:07:39,025 I THINK ALSO, WHEN WE 243 00:07:39,091 --> 00:07:40,126 THINK ABOUT THIS SPACE, WE 244 00:07:40,193 --> 00:07:42,161 THINK ABOUT SOME OF THE 245 00:07:42,228 --> 00:07:43,529 CONCERNS THAT WE MIGHT 246 00:07:43,596 --> 00:07:43,830 HOLD. 247 00:07:43,896 --> 00:07:46,332 I THINK THERE IS SORT OF 248 00:07:46,399 --> 00:07:47,700 EMERGING LINE OF WORK 249 00:07:47,767 --> 00:07:48,734 HIGHLIGHTING IN ADDITION 250 00:07:48,801 --> 00:07:51,604 TO SOME CONCERNS, REALLY 251 00:07:51,671 --> 00:07:52,171 INCREDIBLE OPPORTUNITIES, 252 00:07:52,238 --> 00:07:53,005 IN PARTICULAR, WITH 253 00:07:53,072 --> 00:07:56,108 RESPECT TO IMPROVING 254 00:07:56,175 --> 00:07:57,543 DETECTION OF BIAS, 255 00:07:57,610 --> 00:07:59,846 CREATION OF SORT OF 256 00:07:59,912 --> 00:08:02,215 DATASET THAT HAVE EQUITY 257 00:08:02,281 --> 00:08:03,149 RELEVANT INFORMATION. 258 00:08:03,216 --> 00:08:10,056 I LIKE THIS WORK FROM P 259 00:08:10,122 --> 00:08:12,758 PIERSON, ETAL, LOOKING AT 260 00:08:12,825 --> 00:08:15,194 HOW TO ACTUALLY GENERATE 261 00:08:15,261 --> 00:08:16,562 DATA WE WISH WE HAD RATHER 262 00:08:16,629 --> 00:08:19,131 THAN MAYBE DATA WE HAVE 263 00:08:19,198 --> 00:08:19,832 CURRENTLY AND WE'RE 264 00:08:19,899 --> 00:08:20,333 LEARNING FROM. 265 00:08:20,399 --> 00:08:21,801 I THINK WHEN WE THINK 266 00:08:21,868 --> 00:08:23,469 ABOUT THIS PROBABLY IN 267 00:08:23,536 --> 00:08:25,671 CONTEXT OF REAL-WORLD 268 00:08:25,738 --> 00:08:27,707 EVIDENCE, THERE IS THIS 269 00:08:27,773 --> 00:08:28,307 BROADER CONSIDERATION 270 00:08:28,374 --> 00:08:30,376 AROUND HOW WE MIGHT BE 271 00:08:30,443 --> 00:08:33,946 ABLE TO USE SOME DATA THAT 272 00:08:34,013 --> 00:08:36,082 IS NATURALLY COLLECTED TO 273 00:08:36,148 --> 00:08:37,316 ACCOMMODATE EXISTING 274 00:08:37,383 --> 00:08:38,918 METHOD OF EVIDENCE 275 00:08:38,985 --> 00:08:39,785 GENERATION IN MEDICINE AND 276 00:08:39,852 --> 00:08:41,821 ONE MAY BE MOTIVATING 277 00:08:41,888 --> 00:08:47,493 EXAMPLES I PRESENT UP 278 00:08:47,560 --> 00:08:49,262 FRONT BEFORE DIVING IN, 279 00:08:49,328 --> 00:08:51,097 SPECIFICALLY IF WE LOOK AT 280 00:08:51,163 --> 00:08:53,165 ONCOLOGY SPACE, WE SEE 281 00:08:53,232 --> 00:08:58,371 THIS SORT OF PARADIGM THAT 282 00:08:58,437 --> 00:08:59,872 DOESN'T MAKE A LOT OF 283 00:08:59,939 --> 00:09:02,708 SENCE AT FIRST GLANCE. 284 00:09:02,775 --> 00:09:04,043 IF WE THINK ABOUT THE 285 00:09:04,110 --> 00:09:06,512 NUMBER OF PATIENTS ENROLL 286 00:09:06,579 --> 00:09:08,247 IN CLINICAL TRIAL, THERE 287 00:09:08,314 --> 00:09:11,217 IS DEMAND FOR MORE 288 00:09:11,284 --> 00:09:12,818 ENROLLMENT, WE SEE SO MANY 289 00:09:12,885 --> 00:09:14,053 TRIAL FAILURES. 290 00:09:14,120 --> 00:09:15,888 ON THE SAME SIDE, WE SEE 291 00:09:15,955 --> 00:09:17,857 THIS OTHER SIDE OF THE 292 00:09:17,924 --> 00:09:19,158 EQUATION THAT IS LIKE WE 293 00:09:19,225 --> 00:09:22,795 HAVE FAILING TRAILS, WE 294 00:09:22,862 --> 00:09:24,130 DON'T HAVE ENOUGH PATIENTS 295 00:09:24,196 --> 00:09:25,498 TO BE IN TRIALS. 296 00:09:25,564 --> 00:09:27,099 MAYBE WE CAN START TO 297 00:09:27,166 --> 00:09:27,934 CLOSE THAT GAP. 298 00:09:28,000 --> 00:09:31,003 OUR GROUP, COMING FROM NLP 299 00:09:31,070 --> 00:09:32,271 BACKGROUND LOOK TO HOW 300 00:09:32,338 --> 00:09:34,407 MIGHT WE USE LARGE 301 00:09:34,473 --> 00:09:35,174 LANGUAGE MODELS TO REALLY 302 00:09:35,241 --> 00:09:36,242 QUICKLY PROVIDE A 303 00:09:36,309 --> 00:09:38,144 MECHANISM FOR UNIVERSAL 304 00:09:38,210 --> 00:09:39,312 STRUCTURING OF A PATIENT 305 00:09:39,378 --> 00:09:41,113 RECORD AND TRY TO UNLOCK 306 00:09:41,180 --> 00:09:42,548 SOME OF THAT VALUE. 307 00:09:42,615 --> 00:09:44,650 MAYBE I'LL PRESENT SOME 308 00:09:44,717 --> 00:09:45,952 INITIAL WORK WE'VE DONE IN 309 00:09:46,018 --> 00:09:47,520 THIS SPACE WE BROUGHT TO 310 00:09:47,586 --> 00:09:49,255 MACHINE LEARNING FOR 311 00:09:49,322 --> 00:09:51,123 HEALTHCARE LAST YEAR 312 00:09:51,190 --> 00:09:52,658 AROUND SPECIFICALLY LIKE A 313 00:09:52,725 --> 00:09:53,893 CLINICAL TRIAL FROM 314 00:09:53,960 --> 00:09:58,264 CLINICAL TRIALS.GOV, 315 00:09:58,331 --> 00:09:59,065 DIFFERENT ELIGIBILITY 316 00:09:59,131 --> 00:09:59,565 CRITERIA. 317 00:09:59,632 --> 00:10:02,702 ONE THING AS COMPUTER 318 00:10:02,768 --> 00:10:03,402 SCIENTIST, CAN WE TURN 319 00:10:03,469 --> 00:10:05,037 THIS INTO A LOGIC PROBLEM 320 00:10:05,104 --> 00:10:06,439 OR LOGIC STATEMENT? 321 00:10:06,505 --> 00:10:09,742 WE LOOKED TO LLMS TO SAY 322 00:10:09,809 --> 00:10:11,577 CAN WE ACTUALLY TRY TO 323 00:10:11,644 --> 00:10:15,748 CREATE SORT OF NORMAL 324 00:10:15,815 --> 00:10:16,916 LOGIC STATEMENTS, LOGIC 325 00:10:16,983 --> 00:10:19,485 STATEMENTS FORMED FROM 326 00:10:19,552 --> 00:10:21,120 ANDS, ORS OR NOTES IN A 327 00:10:21,187 --> 00:10:22,621 VERY SPECIFIC MANNER FROM 328 00:10:22,688 --> 00:10:25,658 INPURT FROM CLINICAL 329 00:10:25,725 --> 00:10:26,659 TRIALS.GOV, TO CONSTRUCT A 330 00:10:26,726 --> 00:10:27,927 QUICK MATCHING TREE LIKE 331 00:10:27,994 --> 00:10:28,260 THIS. 332 00:10:28,327 --> 00:10:30,329 AND WHAT WE WERE QUITE 333 00:10:30,396 --> 00:10:31,731 SURPRISED TO FIND, THE 334 00:10:31,797 --> 00:10:34,767 THERE -- YOU CAN DO THIS 335 00:10:34,834 --> 00:10:35,901 USING SOME TOOLS OUT OF 336 00:10:35,968 --> 00:10:38,371 THE BOX AND OUT OF THE BOX 337 00:10:38,437 --> 00:10:41,941 AND YOU CAN DO IT AT A 338 00:10:42,008 --> 00:10:43,676 PERFORMANCE LEVEL THAT IS 339 00:10:43,743 --> 00:10:45,077 COMPARABLE OR BEATS 340 00:10:45,144 --> 00:10:46,178 ENTIRELY TOOLS 341 00:10:46,245 --> 00:10:47,646 SPECIFICALLY DESIGNED FOR 342 00:10:47,713 --> 00:10:48,247 THIS TASK. 343 00:10:48,314 --> 00:10:50,249 TO SORT OF MOTIVATE WHERE 344 00:10:50,316 --> 00:10:51,717 YOU MIGHT IMAGINE I'M 345 00:10:51,784 --> 00:10:53,085 GOING WITH THIS, ON THE 346 00:10:53,152 --> 00:10:54,020 OTHER SIDE, WE HAVE 347 00:10:54,086 --> 00:10:57,423 PATIENTS WHO GENERATE T 348 00:10:57,490 --> 00:10:58,424 TREMENDOUS AMOUNTS OF 349 00:10:58,491 --> 00:10:59,525 DATA, AS WELL. 350 00:10:59,592 --> 00:11:02,028 I SHOULD CLARIFY UP FRONT, 351 00:11:02,094 --> 00:11:05,031 NO REAL PATIENT DATA IS 352 00:11:05,097 --> 00:11:07,233 SHOWN, THESE ARE EXAMPLES 353 00:11:07,299 --> 00:11:08,401 SYNTHETIC IN NATURE. 354 00:11:08,467 --> 00:11:10,736 IF YOU THINK ABOUT DATA 355 00:11:10,803 --> 00:11:12,038 PATIENT GENERATES MIGHT 356 00:11:12,104 --> 00:11:13,973 GENERATE TONS OF TEXT DATA 357 00:11:14,040 --> 00:11:16,709 ACROSS DIFFERENT TYPES OF 358 00:11:16,776 --> 00:11:16,909 NOTES. 359 00:11:16,976 --> 00:11:19,211 HERE IS COUPLE OF EXAMPLES 360 00:11:19,278 --> 00:11:20,646 SYNTHETIC IN NATURE, YOU 361 00:11:20,713 --> 00:11:22,181 MIGHT IMAGINE TRYING TO 362 00:11:22,248 --> 00:11:23,916 FIND SOME ELIGIBILITY 363 00:11:23,983 --> 00:11:25,051 CRITERIA ACROSS. 364 00:11:25,117 --> 00:11:27,019 SOME EARLY WORK IN THIS 365 00:11:27,086 --> 00:11:27,987 SPACE LOOKED AT CAN WE 366 00:11:28,054 --> 00:11:30,056 CREATE SPECIFICALLY MODELS 367 00:11:30,122 --> 00:11:33,859 AROUND THIS TYPE OF 368 00:11:33,926 --> 00:11:36,162 INFORMATION EXTRACTION, IT 369 00:11:36,228 --> 00:11:37,830 WAS TUMOR STAGING KRI 370 00:11:37,897 --> 00:11:40,232 TERIO, TNM SPECIFICALLY, 371 00:11:40,299 --> 00:11:42,835 IT SEEMS LIKE YES, WE CAN 372 00:11:42,902 --> 00:11:44,003 EXTRACT INFORMATION VERY 373 00:11:44,070 --> 00:11:45,237 QUICKLY AND MAKE IT 374 00:11:45,304 --> 00:11:46,972 ACCESSIBLE TO CLINICIANS 375 00:11:47,039 --> 00:11:49,809 OR MEDICAL EXTRACTORS OR 376 00:11:49,875 --> 00:11:51,777 WHOEVER ELSE NEEDS THIS 377 00:11:51,844 --> 00:11:52,445 INFORMATION AND WAY THAT 378 00:11:52,511 --> 00:11:54,313 IS TRANS FORMATIVE TO WORK 379 00:11:54,380 --> 00:11:55,381 THEY ARE DOING. 380 00:11:55,448 --> 00:11:56,916 YOU CAN IMAGINE PERHAPS 381 00:11:56,982 --> 00:11:59,952 HOW YOU MIGHT ON THE ONE 382 00:12:00,019 --> 00:12:01,520 HAND BE ABLE TO DO THE 383 00:12:01,587 --> 00:12:03,155 TRIALS, MIGHT BETTER 384 00:12:03,222 --> 00:12:03,989 MOTIVATE MATCHING PROCESS 385 00:12:04,056 --> 00:12:05,991 IF YOU ARE ABLE TO EXTRACT 386 00:12:06,058 --> 00:12:07,193 SAME INFORMATION FROM 387 00:12:07,259 --> 00:12:08,227 PATIENT DATA. 388 00:12:08,294 --> 00:12:10,329 AND YOU KNOW, I THINK FROM 389 00:12:10,396 --> 00:12:12,231 THESE PRELIMINARY RESULTS, 390 00:12:12,298 --> 00:12:13,666 THEY SEEM RATHER 391 00:12:13,732 --> 00:12:15,134 PROMISING, IT IS A NICE 392 00:12:15,201 --> 00:12:16,268 IDEA YOU CAN TAKE 393 00:12:16,335 --> 00:12:17,136 SOMETHING LIKE CRITERIA 394 00:12:17,203 --> 00:12:18,404 WE'RE LOOKING FOR AND 395 00:12:18,471 --> 00:12:19,405 QUICKLY IDENTIFY THAT 396 00:12:19,472 --> 00:12:23,476 INFORMATION IN LITERATURE, 397 00:12:23,542 --> 00:12:24,944 PATIENT RECORD AND OTHER 398 00:12:25,010 --> 00:12:26,078 PLACES BECAUSE FOR 399 00:12:26,145 --> 00:12:28,147 EXAMPLE, GPT OR LARGE 400 00:12:28,214 --> 00:12:29,882 LANGUAGE MODEL THESE DAYS 401 00:12:29,949 --> 00:12:31,250 MIGHT BE ABLE TO QUICKLY 402 00:12:31,317 --> 00:12:32,151 IDENTIFY THIS INFORMATION 403 00:12:32,218 --> 00:12:34,220 IN WAY THAT WAS REQUIRED 404 00:12:34,286 --> 00:12:36,489 TO HAVE HUMAN INTERVENTION 405 00:12:36,555 --> 00:12:36,789 PREVIOUSLY. 406 00:12:36,856 --> 00:12:38,491 WE SORT OF AGAIN MOCKED UP 407 00:12:38,557 --> 00:12:40,893 A FEW THINGS AROUND HOW 408 00:12:40,960 --> 00:12:42,428 YOU MIGHT BE ABLE TO 409 00:12:42,495 --> 00:12:45,598 BETTER TRIAGE SOME OF THE 410 00:12:45,664 --> 00:12:46,966 PATIENT DATA AND IDENTIFY 411 00:12:47,032 --> 00:12:48,167 BASED ON THINGS THAT SHOW 412 00:12:48,234 --> 00:12:50,469 UP IN THE RECORD, MAYBE 413 00:12:50,536 --> 00:12:52,071 TRIALS THEY WOULD BE 414 00:12:52,138 --> 00:12:52,404 ELIGIBLE FOR. 415 00:12:52,471 --> 00:12:53,973 I THOUGHT ACTUALLY I'D 416 00:12:54,039 --> 00:12:56,142 TAKE A MOMENT HERE TO SORT 417 00:12:56,208 --> 00:12:57,510 OF QUICKLY COVER FROMECK 418 00:12:57,576 --> 00:12:59,011 IT CAL STANDPOINT HOW DID 419 00:12:59,078 --> 00:13:00,846 WE GET WHERE WE ARE TODAY. 420 00:13:00,913 --> 00:13:02,681 THIS HAS BEEN A RATHER 421 00:13:02,748 --> 00:13:04,250 LARGE SHIFT, I'VE BEEN 422 00:13:04,316 --> 00:13:05,651 LIVING FROM THE NLP SIDE 423 00:13:05,718 --> 00:13:07,186 OF THINGS, I DON'T WANT TO 424 00:13:07,253 --> 00:13:08,988 ASSUME EVERYBODY ELSE HAS 425 00:13:09,054 --> 00:13:10,156 BEEN DOING THE SAME 6789 426 00:13:10,222 --> 00:13:12,158 MAYBE I'LL MENTION AT HIGH 427 00:13:12,224 --> 00:13:14,760 LEVEL, THERE HAS BEEN 428 00:13:14,827 --> 00:13:15,628 CONSOLIDATION WE SEE 429 00:13:15,694 --> 00:13:21,867 ACROSS AI AND LLP, AS 430 00:13:21,934 --> 00:13:23,369 WELL. 431 00:13:23,435 --> 00:13:25,404 ARCHITECTURE TRANSFORMED 432 00:13:25,471 --> 00:13:27,206 ARCHITECTURE JOIN ENCODER 433 00:13:27,273 --> 00:13:30,009 AND ENCODER NETWORK AND 434 00:13:30,075 --> 00:13:30,943 ENCO 435 00:13:31,010 --> 00:13:31,844 ENCODER PORTION IS 436 00:13:31,911 --> 00:13:32,845 RELATIVELY SIMILAR TO 437 00:13:32,912 --> 00:13:36,615 THINGS WE HAD SEEN AND 438 00:13:36,682 --> 00:13:38,517 ADDS THIS PIECE OF 439 00:13:38,584 --> 00:13:39,318 SELF-ATTENTION, AS WELL. 440 00:13:39,385 --> 00:13:41,020 ONE THING WITH COMBINATION 441 00:13:41,086 --> 00:13:42,121 OF SELF-ATTENTION AND 442 00:13:42,188 --> 00:13:45,624 NUMBER OF OTHER ADVANCES 443 00:13:45,691 --> 00:13:47,092 WAS THIS NOTION THAT 444 00:13:47,159 --> 00:13:50,729 CERTAIN WORDS IN A 445 00:13:50,796 --> 00:13:52,331 SEQUENCE RELATE TO OTHER 446 00:13:52,398 --> 00:13:55,534 WORDS IN THE SAME 447 00:13:55,601 --> 00:13:55,801 SEQUENCE. 448 00:13:55,868 --> 00:13:58,237 IN TURNS YOU CAN IMAGINE 449 00:13:58,304 --> 00:13:59,638 CONSTRAINT THE ON 450 00:13:59,705 --> 00:14:00,706 DIRECTIONALITY AND THAT 451 00:14:00,773 --> 00:14:02,208 OPENED BROAD VARIETY OF 452 00:14:02,274 --> 00:14:03,242 DIFFERENT TYPE OF 453 00:14:03,309 --> 00:14:05,644 ARCHITECTURE THAT BIRD OR 454 00:14:05,711 --> 00:14:09,381 GPT OR ELMO, SESAME STREET 455 00:14:09,448 --> 00:14:10,616 CHARACTERS THAT APPEAR IN 456 00:14:10,683 --> 00:14:12,251 THE NAMES. 457 00:14:12,318 --> 00:14:13,519 MORE GENERALLY, ACROSS ALL 458 00:14:13,586 --> 00:14:15,821 OF THEM, SELF-ATTENTION 459 00:14:15,888 --> 00:14:17,823 PROVIDES THIS FORM EVER 460 00:14:17,890 --> 00:14:19,758 CONTEXTUALIZATION THAT IS 461 00:14:19,825 --> 00:14:21,493 POWERFUL IN A NUMBER OF 462 00:14:21,560 --> 00:14:23,229 SETTINGS AND GIVES US IF 463 00:14:23,295 --> 00:14:24,763 WE THINK ABOUT TRADITIONAL 464 00:14:24,830 --> 00:14:27,166 MASS MODEL I'LL TALK ABOUT 465 00:14:27,233 --> 00:14:30,402 LATER, GIVES BROADER 466 00:14:30,469 --> 00:14:31,570 CONTEXT WINDOW. 467 00:14:31,637 --> 00:14:34,373 -- MENTION BEYOND USE IN 468 00:14:34,440 --> 00:14:36,242 TEXT, WHAT WAS NICE TO 469 00:14:36,308 --> 00:14:39,712 SEE, NOT ONLY USEFUL FOR 470 00:14:39,778 --> 00:14:41,480 TEXT, YOU CAN USE THIS FOR 471 00:14:41,547 --> 00:14:43,649 OTHER PLATFORMS, THINGS IN 472 00:14:43,716 --> 00:14:44,183 VISION. 473 00:14:44,250 --> 00:14:45,684 YOU CAN USE IT 474 00:14:45,751 --> 00:14:48,254 INCREASINGLY FOR VARIOUS 475 00:14:48,320 --> 00:14:50,256 ASPECTS IN MOLECULAR 476 00:14:50,322 --> 00:14:51,857 SIMULATION AND MOLECULAR 477 00:14:51,924 --> 00:14:52,591 DESIGN, AS WELL. 478 00:14:52,658 --> 00:14:54,360 AND I WANT TO SAY, THE 479 00:14:54,426 --> 00:14:56,862 OTHER THING THAT HAS BEEN 480 00:14:56,929 --> 00:14:58,264 SORT OF KEY AND SHOWN UP 481 00:14:58,330 --> 00:15:01,233 IN NLP FOR A LONG TIME, 482 00:15:01,300 --> 00:15:02,735 NEWER TO THE SPACE, 483 00:15:02,801 --> 00:15:04,737 ESPECIALLY IN MEDICINE, WE 484 00:15:04,803 --> 00:15:07,206 HAVE THIS SIMILAR 485 00:15:07,273 --> 00:15:08,707 CONUNDRUM, TO NLP, WE HAVE 486 00:15:08,774 --> 00:15:10,409 LABEL DATA, WE DON'T HAVE 487 00:15:10,476 --> 00:15:11,577 THAT MUCH AND WE WANT TO 488 00:15:11,644 --> 00:15:14,413 BUILD THIS APPLICATION 489 00:15:14,480 --> 00:15:14,613 MODEL. 490 00:15:14,680 --> 00:15:15,948 AT THE SAME TIME, WE 491 00:15:16,015 --> 00:15:17,249 PROBABLY HAVE AS WE DO 492 00:15:17,316 --> 00:15:19,118 OFTEN IN MEDICINE, YOU 493 00:15:19,184 --> 00:15:20,452 KNOW, A HUGE AMOUNT OF 494 00:15:20,519 --> 00:15:22,521 DOMAIN KNOWLEDGE THAT TASK 495 00:15:22,588 --> 00:15:23,956 SPECIFIC AND WE TYPICALLY 496 00:15:24,023 --> 00:15:26,325 HAVE HUGE AMOUNT UNLABELED 497 00:15:26,392 --> 00:15:29,395 DATA THAT IS TEST 498 00:15:29,461 --> 00:15:29,662 AGNOSTIC. 499 00:15:29,728 --> 00:15:36,835 ONE ADVANCE IN THIS SENSE 500 00:15:36,902 --> 00:15:38,337 HOW DO WE BUILD ON THE 501 00:15:38,404 --> 00:15:39,371 OTHER SIDE? 502 00:15:39,438 --> 00:15:40,472 WALK THROUGH EXAMPLE OF 503 00:15:40,539 --> 00:15:43,175 HOW THIS WAS DONE IN NLP 504 00:15:43,242 --> 00:15:45,077 SPACE SPECIFICALLY 505 00:15:45,144 --> 00:15:46,445 REALIZATION WE HAVE A TON 506 00:15:46,512 --> 00:15:48,547 OF UNLABELED TEXT AND CAN 507 00:15:48,614 --> 00:15:51,417 MAKE UP OUR OWN TRAINING 508 00:15:51,483 --> 00:15:52,384 TASKS WITH THIS. 509 00:15:52,451 --> 00:15:54,820 SPECIFIC TASK WAS THIS MAS 510 00:15:54,887 --> 00:15:56,322 LANGUAGE OBJECTIVE WHERE 511 00:15:56,388 --> 00:15:58,157 YOU EFFECTIVELY SAY LET'S 512 00:15:58,223 --> 00:16:01,927 COVER UP SOME WORDS M THIS 513 00:16:01,994 --> 00:16:03,629 TEXT, GIVEN ONE MASKED 514 00:16:03,696 --> 00:16:05,497 WORD, WHAT ARE THE OTHER 515 00:16:05,564 --> 00:16:06,699 MASKED WORDS? 516 00:16:06,765 --> 00:16:08,801 IF YOU CONSTRAIN THAT 517 00:16:08,867 --> 00:16:10,836 PROCESS TO HAVING THE NEXT 518 00:16:10,903 --> 00:16:13,739 WORD ONLY EVER BE THIS 519 00:16:13,806 --> 00:16:16,475 GPT-STYLE NEXT-WORD 520 00:16:16,542 --> 00:16:17,910 PREDICTION, THIS IS LIKE 521 00:16:17,976 --> 00:16:19,611 AUTO AGGRESSIVE FORM OF 522 00:16:19,678 --> 00:16:21,480 WHAT I JUST DESCRIBED, YOU 523 00:16:21,547 --> 00:16:23,182 GET AN INTERESTING THING 524 00:16:23,248 --> 00:16:24,650 WHERE WE END UP WITH 525 00:16:24,717 --> 00:16:25,851 MODELS WE HAVE TODAY. 526 00:16:25,918 --> 00:16:27,019 THAT IS MAYBE ANOTHER 527 00:16:27,086 --> 00:16:28,554 SHIFT THAT I'LL DETAIL A 528 00:16:28,620 --> 00:16:30,389 LITTLE BIT MORE WAS WE HAD 529 00:16:30,456 --> 00:16:32,991 THIS SHIFT SORT OF FROM 530 00:16:33,058 --> 00:16:34,360 SPECIFICALLY MASKED 531 00:16:34,426 --> 00:16:36,261 LANGUAGE MODEL OR LANGUAGE 532 00:16:36,328 --> 00:16:37,863 MODEL IN GENERAL AT A 533 00:16:37,930 --> 00:16:38,864 DIFFERENT SCALE TO HOW CAN 534 00:16:38,931 --> 00:16:41,266 WE ACTUALLY USE PROMPTS 535 00:16:41,333 --> 00:16:44,002 AND HAVE THIS GENERALIVITY 536 00:16:44,069 --> 00:16:46,772 FORM OF GENERATIVE AI TO 537 00:16:46,839 --> 00:16:49,274 SUPPORT SOME OF OUR TASKS 538 00:16:49,341 --> 00:16:52,778 LIKE TRANSFORMATION AL TASK 539 00:16:52,845 --> 00:16:54,012 MOVING FROM WHAT WE HAD 540 00:16:54,079 --> 00:16:55,347 BEFORE TO WHAT WE HAVE 541 00:16:55,414 --> 00:16:55,514 NOW. 542 00:16:55,581 --> 00:16:58,617 A LOT OF WORK HAS BEEN DIN 543 00:16:58,684 --> 00:17:00,319 SINCE THEN ON SPECIFIC 544 00:17:00,386 --> 00:17:03,222 FORMS OF SUPERVISED FINE 545 00:17:03,288 --> 00:17:04,957 TUNING, REINFORCEMENT 546 00:17:05,023 --> 00:17:06,325 LEARNING FROM HUMAN 547 00:17:06,392 --> 00:17:11,029 FEEDBACK, PREFERENCE 548 00:17:11,096 --> 00:17:12,030 OPTIMIZATION, CALL OUT 549 00:17:12,097 --> 00:17:13,332 THINGS IN THE SPACE THAT 550 00:17:13,399 --> 00:17:15,968 CONTRIBUTE THAT 551 00:17:16,034 --> 00:17:16,368 TRATRANSFORMATION. 552 00:17:16,435 --> 00:17:17,636 TWO THINGS THAT HAVE 553 00:17:17,703 --> 00:17:22,141 DRIVEN THIS, ONE, T 554 00:17:22,207 --> 00:17:24,243 TREMENDOUS GROWTH INNED 555 00:17:24,309 --> 00:17:25,978 MOEL SIZE, THIS NEEDS TO 556 00:17:26,044 --> 00:17:27,846 BE UPDATED, REINFORCEMENT 557 00:17:27,913 --> 00:17:29,348 LEARNING ADVANCE IN MODEL 558 00:17:29,415 --> 00:17:30,349 SPACE, AS WELL. 559 00:17:30,416 --> 00:17:32,151 THIS IS MOVE TOWARD 560 00:17:32,217 --> 00:17:33,886 INCREASINGLY LARGE MODEL 561 00:17:33,952 --> 00:17:37,923 SIZES AND THEN ALSO LARGE 562 00:17:37,990 --> 00:17:39,391 AMOUNT OF DATA 563 00:17:39,458 --> 00:17:40,225 CONTRIBUTING TO EACH OF 564 00:17:40,292 --> 00:17:42,127 THE MODELS, AS WELL. 565 00:17:42,194 --> 00:17:44,830 AND ONE OF THE THINGS WE 566 00:17:44,897 --> 00:17:46,899 SORT OF SEE THIS SORT OF 567 00:17:46,965 --> 00:17:49,368 PIECE WHERE AS WE SCALE 568 00:17:49,435 --> 00:17:51,370 LLMS TO CERTAIN SIZES, WE 569 00:17:51,437 --> 00:17:54,606 CAN START TO SEE LIKE 570 00:17:54,673 --> 00:17:55,040 EME 571 00:17:55,107 --> 00:17:56,375 EMERGENT ABILITIES, THIS 572 00:17:56,442 --> 00:17:59,011 NOTION THERE IS 573 00:17:59,077 --> 00:18:00,245 PERFORMANCE JUMP IN DOWN 574 00:18:00,312 --> 00:18:02,214 STREAM TASKS WE WERE NOT 575 00:18:02,281 --> 00:18:03,615 EXPECTING, RIGHT. 576 00:18:03,682 --> 00:18:04,950 ASIDE FROM THAT, YOU OFTEN 577 00:18:05,017 --> 00:18:08,253 SEE SORT OF FEW SHOT 578 00:18:08,320 --> 00:18:09,121 ABILITY WHERE YOU PROVIDE 579 00:18:09,188 --> 00:18:11,023 A FEW EXAMPLES OR CHAIN OF 580 00:18:11,089 --> 00:18:13,025 THOUGHT ABILITIES WHERE IF 581 00:18:13,091 --> 00:18:13,959 YOU INCLUDE SOME OF THOSE 582 00:18:14,026 --> 00:18:17,196 IN THE CONTEXT, PRIOR 583 00:18:17,262 --> 00:18:18,697 PROMPTS, FOR EXAMPLE, YOU 584 00:18:18,764 --> 00:18:20,265 GET BETTER AND BETTER 585 00:18:20,332 --> 00:18:21,600 RESULTS, THIS IS LIKE 586 00:18:21,667 --> 00:18:22,501 UNEXPECTED MAYBE AN 587 00:18:22,568 --> 00:18:23,669 INTERESTING THING. 588 00:18:23,735 --> 00:18:25,571 THE REAL WAY MAYBE TO SEE 589 00:18:25,637 --> 00:18:26,872 THIS, THOUGH, I OFTEN 590 00:18:26,939 --> 00:18:28,207 THINK ABOUT LANGUAGE 591 00:18:28,273 --> 00:18:29,475 EXAMPLES FROM MY OWN WORK. 592 00:18:29,541 --> 00:18:31,477 TO SEE THIS, YOU CAN SEE 593 00:18:31,543 --> 00:18:32,311 THE EFFECTIVE SCALE AND 594 00:18:32,377 --> 00:18:34,413 VISION A LITTLE BETTER. 595 00:18:34,480 --> 00:18:36,482 THIS IS JUST SINGLE PROMPT 596 00:18:36,548 --> 00:18:38,517 TRYING TO PRODUCE SPECIFIC 597 00:18:38,584 --> 00:18:41,253 IMAGE AND YOU CAN SEE LIKE 598 00:18:41,320 --> 00:18:43,088 350 MILLION PARAMETER 599 00:18:43,155 --> 00:18:45,257 MODEL, MAYBE IT IS 600 00:18:45,324 --> 00:18:47,960 KANGAROO, MAYBE NOT QUITE 601 00:18:48,026 --> 00:18:48,160 THERE. 602 00:18:48,227 --> 00:18:48,994 DEFINITELY LOOKING A 603 00:18:49,061 --> 00:18:50,295 LITTLE BETTER, THERE IS 604 00:18:50,362 --> 00:18:52,397 THIS OTHER KANGAROO IN THE 605 00:18:52,464 --> 00:18:54,132 FOREGROUND AND YOU CAN SEE 606 00:18:54,199 --> 00:18:55,601 HOW THIS IS INCREASING 607 00:18:55,667 --> 00:18:57,436 BOTH SORT OF WITH ADVANCE 608 00:18:57,503 --> 00:18:59,471 IN DATA SCALE AND MODEL 609 00:18:59,538 --> 00:19:00,239 SIZE. 610 00:19:00,305 --> 00:19:04,243 SO I'LL TRANSITION FROM 611 00:19:04,309 --> 00:19:05,944 NOW TO THAT LEVEL SET TO 612 00:19:06,011 --> 00:19:07,212 MAYBE WHAT THIS LOOK LIKE 613 00:19:07,279 --> 00:19:09,481 IN THE BIOMEDICAL SPACE, 614 00:19:09,548 --> 00:19:10,816 THE SPACE I CARE ABOUT AND 615 00:19:10,883 --> 00:19:14,286 I THINK THAT ONE OF THE 616 00:19:14,353 --> 00:19:15,721 THINGS THAT HAS BEEN 617 00:19:15,787 --> 00:19:16,989 INTERESTING IS WE'VE OFTEN 618 00:19:17,055 --> 00:19:19,625 HAD THIS CHALLENGE MAYBE 619 00:19:19,691 --> 00:19:21,059 TWO CHALLENGES IN 620 00:19:21,126 --> 00:19:22,594 BIOMEDICINE OR CLINICAL 621 00:19:22,661 --> 00:19:25,430 DOMAINS AT LARGE, THAT OUR 622 00:19:25,497 --> 00:19:28,100 DATA IS NOT GENERAL DOMAIN 623 00:19:28,166 --> 00:19:28,300 DATA. 624 00:19:28,367 --> 00:19:28,934 WHEN WE THINK ABOUT 625 00:19:29,001 --> 00:19:31,336 ESPECIALLY LIKE THE 626 00:19:31,403 --> 00:19:35,107 COUCHES WE ALL DO TO HELP 627 00:19:35,173 --> 00:19:35,641 GENERATE SOME LABELS FOR 628 00:19:35,707 --> 00:19:37,676 GENERAL DOMAIN, IF WE 629 00:19:37,743 --> 00:19:42,347 SHOWED LAY USERS THIS 630 00:19:42,414 --> 00:19:44,116 CAPTURE OVER HERE WITH 631 00:19:44,182 --> 00:19:46,151 AMMONIA, THEY WOULD BE 632 00:19:46,218 --> 00:19:46,952 HARD-PRESSED TO DO IT. 633 00:19:47,019 --> 00:19:50,455 MAYBE SOME IN THE OAUDIENCE 634 00:19:50,522 --> 00:19:51,223 COULD DO IT. 635 00:19:51,290 --> 00:19:52,824 I WOULD FAIL A NUMBER OF 636 00:19:52,891 --> 00:19:54,326 THEM, IF YOU CHANGE DOMAIN 637 00:19:54,393 --> 00:19:56,128 I WOULD FAIL ALL OF THEM. 638 00:19:56,194 --> 00:19:57,596 THAT IS ONE LARGE 639 00:19:57,663 --> 00:19:58,330 DIFFERENCE. 640 00:19:58,397 --> 00:19:59,331 THE OTHER ONE DIFFERENT IN 641 00:19:59,398 --> 00:20:01,099 MY MIND, WAY WE USE 642 00:20:01,166 --> 00:20:02,334 LANGUAGE IS OFTEN 643 00:20:02,401 --> 00:20:03,235 DIFFERENT, AS WELL. 644 00:20:03,302 --> 00:20:04,536 IT IS VERY FREQUENT THAT 645 00:20:04,603 --> 00:20:07,205 WE USE NEGATION IN THIS 646 00:20:07,272 --> 00:20:08,607 DOMAIN DIFFERENTLY THAN 647 00:20:08,674 --> 00:20:09,474 GENERAL DOMAIN. 648 00:20:09,541 --> 00:20:12,210 WE OFTEN LOOK AT ONE OF 649 00:20:12,277 --> 00:20:13,745 THESE CHEST X-RAYS, FOR 650 00:20:13,812 --> 00:20:15,681 EXAMPLE, LOOKING FOR 651 00:20:15,747 --> 00:20:18,951 SOMETHING SPECIFIC, 652 00:20:19,017 --> 00:20:21,386 DESCRIPTION, IT EXISTS OR 653 00:20:21,453 --> 00:20:23,822 DOESN'T EXIST AND THIS TO 654 00:20:23,889 --> 00:20:24,923 DRIVE THAT EXAMPLE HOME, 655 00:20:24,990 --> 00:20:26,558 IF YOU LOOK AT THIS IMAGE 656 00:20:26,625 --> 00:20:30,896 OF TWO COWS GRAZING IN A 657 00:20:30,963 --> 00:20:32,030 FIELD, CLASSIC IMAGE 658 00:20:32,097 --> 00:20:33,265 EXAMPLE, YOU WOULD NEVER 659 00:20:33,332 --> 00:20:36,234 SAY NO PRESENCE OF 660 00:20:36,301 --> 00:20:38,303 DINOSAUR IN THIS LANGUAGE. 661 00:20:38,370 --> 00:20:40,105 THAT APPEARS IN MEDICINE, 662 00:20:40,172 --> 00:20:41,907 MAYBE WE NEED DIFFERENT 663 00:20:41,974 --> 00:20:43,442 TOOL FOR BIOMEDICAL 664 00:20:43,508 --> 00:20:45,677 RELATIVE TO GENERAL DOMAIN 665 00:20:45,744 --> 00:20:48,246 AND MOTIVATES IT, 666 00:20:48,313 --> 00:20:48,880 COMPARABLE DATASET I LIKE 667 00:20:48,947 --> 00:20:51,617 THIS EXAMPLE, GIVEN A 668 00:20:51,683 --> 00:20:54,252 NUMBER OF YEARS AGO, 669 00:20:54,319 --> 00:20:57,089 OBSERVING A LOT OF 670 00:20:57,155 --> 00:20:58,991 DATASETS HAVE ARRIVED IN 671 00:20:59,057 --> 00:21:01,059 CLINICAL AND BIOMEDICAL 672 00:21:01,126 --> 00:21:02,661 SETTINGS OVER A DECADE 673 00:21:02,728 --> 00:21:02,861 LATER. 674 00:21:02,928 --> 00:21:04,663 NIH HAS DONE INCREDIBLE 675 00:21:04,730 --> 00:21:06,531 AMOUNT TO MAKE DATA 676 00:21:06,598 --> 00:21:08,834 AVAILABLE IN THESE 677 00:21:08,900 --> 00:21:09,067 DOMAINS. 678 00:21:09,134 --> 00:21:10,569 BUT BY AND LARGE, THE 679 00:21:10,636 --> 00:21:12,471 ORDER OF MAGNITUDE AND 680 00:21:12,537 --> 00:21:14,673 SCALE THAT WE SEE ARRIVES 681 00:21:14,740 --> 00:21:16,441 ABOUT A DECADE LATER, 682 00:21:16,508 --> 00:21:19,878 THINGS LIKE I2B2 ARRIVED 683 00:21:19,945 --> 00:21:23,782 OVER A DECADE AFTER PEN 684 00:21:23,849 --> 00:21:25,517 DATASET SIZE. 685 00:21:25,584 --> 00:21:26,551 SO I THINK THIS IS MAYBE 686 00:21:26,618 --> 00:21:28,320 TWO OF THE CHALLENGES AND 687 00:21:28,387 --> 00:21:29,688 ONE THING I WANT TO RUN 688 00:21:29,755 --> 00:21:31,723 THROUGH IS WHAT ARE SOME 689 00:21:31,790 --> 00:21:33,425 PARADIGM SHIFTS WE'VE HAD 690 00:21:33,492 --> 00:21:36,028 IN BIOMEDICINE? 691 00:21:36,094 --> 00:21:37,095 WE'VE LANDED IN A SPACE 692 00:21:37,162 --> 00:21:38,530 NOW FOR GENERAL DOMAIN AND 693 00:21:38,597 --> 00:21:42,267 WE LOOK TO HAVING LAYER 694 00:21:42,334 --> 00:21:45,237 WHERE LLM IS ABLE TO 695 00:21:45,303 --> 00:21:46,638 PROVIDE BOTH LANGUAGE 696 00:21:46,705 --> 00:21:47,339 UNDERSTANDING AND 697 00:21:47,406 --> 00:21:47,673 GENERATION. 698 00:21:47,739 --> 00:21:50,575 SO IF I SORT OF BREAK DOWN 699 00:21:50,642 --> 00:21:52,244 WHAT SOME PARADIGM SHIFTS 700 00:21:52,310 --> 00:21:54,746 WITH LLMS ARE, THERE ARE 701 00:21:54,813 --> 00:21:55,414 MAYBE THREE. 702 00:21:55,480 --> 00:21:57,349 I'LL DIVE INTO THESE 703 00:21:57,416 --> 00:21:58,617 BRIEFLY, IT IS MAYBE THIS 704 00:21:58,684 --> 00:22:00,485 SHIFT FROM SPECIALIST TO 705 00:22:00,552 --> 00:22:01,820 GENERALIST MODEL, SHIFT 706 00:22:01,887 --> 00:22:04,723 FROM CLOSED SET TO 707 00:22:04,790 --> 00:22:05,290 OPEN-ENDED GENERATION AND 708 00:22:05,357 --> 00:22:07,526 SHIFT FROM REPRESENTATION 709 00:22:07,592 --> 00:22:09,227 LEARNING TO PROMPTABLE 710 00:22:09,294 --> 00:22:11,563 MALLEABLE INTERFACES. 711 00:22:11,630 --> 00:22:13,465 IN THE FIRST ONE, REALLY 712 00:22:13,532 --> 00:22:14,566 QUICKLY, GO BACK AND THINK 713 00:22:14,633 --> 00:22:16,268 ABOUT WHAT WERE WE DOING 714 00:22:16,334 --> 00:22:18,937 IN LLP FOR A LONG TIME, WE 715 00:22:19,004 --> 00:22:22,240 MIGHT HAVE FOCUSED ON 716 00:22:22,307 --> 00:22:22,908 SUMMARIZATION, 717 00:22:22,974 --> 00:22:24,242 SUMMARIZATION MODELS OR 718 00:22:24,309 --> 00:22:27,446 MAYBE FOCUSED ON ENTITTY 719 00:22:27,512 --> 00:22:28,480 RECOGNITION AND I THINK A 720 00:22:28,547 --> 00:22:30,716 LOT OF COMMUNITIES ARE 721 00:22:30,782 --> 00:22:31,683 DIVIDED AROUND SPECIFIC 722 00:22:31,750 --> 00:22:32,918 AREAS AND HOW TO DO THEM 723 00:22:32,984 --> 00:22:35,187 AND A LOT OF OVERLAP AND 724 00:22:35,253 --> 00:22:36,221 WE WORKED MANY OF THEM 725 00:22:36,288 --> 00:22:37,422 MUCH OF THE TIME. 726 00:22:37,489 --> 00:22:38,590 IT WAS REALLY THIS IDEA 727 00:22:38,657 --> 00:22:42,227 YOU WOULD HAVE THESE SORT 728 00:22:42,294 --> 00:22:43,795 OF SPECIALISTS MODELS AND 729 00:22:43,862 --> 00:22:46,331 I THINK WHAT WE STARTED TO 730 00:22:46,398 --> 00:22:47,332 SEE WITH LARGE LANGUAGE 731 00:22:47,399 --> 00:22:49,267 MODEL WORK, THINGS LIKE 732 00:22:49,334 --> 00:22:51,036 BIRD MAYBE NO LONGER 733 00:22:51,103 --> 00:22:52,637 LARGE-LANGUAGE MODEL, AT 734 00:22:52,704 --> 00:22:53,872 THE TIME, THEY WERE 735 00:22:53,939 --> 00:22:56,775 CERTAINLY LARGE, SHIFT TO 736 00:22:56,842 --> 00:22:59,678 A SET OF HEADERS THAT WERE 737 00:22:59,745 --> 00:23:00,345 SPECIALIST IN NATURE. 738 00:23:00,412 --> 00:23:02,647 YOU WOULD HAVE TASK 739 00:23:02,714 --> 00:23:05,050 AGNOFTIC BASE AND SHIFT TO 740 00:23:05,117 --> 00:23:07,085 TASK SPECIFIC HEADER FOR A 741 00:23:07,152 --> 00:23:09,187 SPECIFIC SET OF TASKS. 742 00:23:09,254 --> 00:23:11,623 WHAT WAS NICE, BOTH WAS 743 00:23:11,690 --> 00:23:15,160 REUSABLE AND ACCELERATED 744 00:23:15,227 --> 00:23:16,261 RESEARCH AND SHARED SOME 745 00:23:16,328 --> 00:23:19,030 OF THE ADVANCES ACROSS 746 00:23:19,097 --> 00:23:20,298 ONES THAT WERE NOT SEEING 747 00:23:20,365 --> 00:23:22,768 AS MANY ADVANCES, AS WELL. 748 00:23:22,834 --> 00:23:24,469 ONE THING THAT HAS BEEN 749 00:23:24,536 --> 00:23:26,638 PRETTY MAJOR SHIFT SINCE 750 00:23:26,705 --> 00:23:29,474 IS LIKE I SAID, MAYBE GOES 751 00:23:29,541 --> 00:23:31,510 INTO LLM AND COMES OUT THE 752 00:23:31,576 --> 00:23:34,312 OTHER SIDE FOR EACH OF THE 753 00:23:34,379 --> 00:23:34,513 TASKS. 754 00:23:34,579 --> 00:23:36,281 SECOND ONE IS DIFFERENCE 755 00:23:36,348 --> 00:23:39,518 BETWEEN CLOSED SET 756 00:23:39,584 --> 00:23:40,652 CLASSIFICATION AND OPEN 757 00:23:40,719 --> 00:23:42,521 ENDED, GOING BACK TO 758 00:23:42,587 --> 00:23:43,755 PREDEFINED SUMMARIZATION 759 00:23:43,822 --> 00:23:46,491 YOU HAD PREDEFINED SET OF 760 00:23:46,558 --> 00:23:48,527 TEMPLATES OR LIMITED LABEL 761 00:23:48,593 --> 00:23:50,162 OR SPACE IS CONSTRAINED IN 762 00:23:50,228 --> 00:23:52,998 SOME WAY OR FOR ENTITY 763 00:23:53,064 --> 00:23:55,133 RECOGNITION, PREDEFINED 764 00:23:55,200 --> 00:23:57,335 SET OF LABELS, THIS 765 00:23:57,402 --> 00:24:00,172 SHIFTED OVER TIME WITH 766 00:24:00,238 --> 00:24:02,374 LLMS TO BE YES, YOU CAN 767 00:24:02,440 --> 00:24:03,842 PROVIDE CLASSIFICATION, 768 00:24:03,909 --> 00:24:06,678 YES, YOU CAN DO VERY 769 00:24:06,745 --> 00:24:07,212 SPECIFIC CLASSIFICATIONS 770 00:24:07,279 --> 00:24:08,713 FOR SPECIFIC DATASET AND 771 00:24:08,780 --> 00:24:10,916 START TO DO SUMMARIZATION, 772 00:24:10,982 --> 00:24:11,716 OPEN-ENDED GENERATION 773 00:24:11,783 --> 00:24:13,919 , 774 00:24:13,985 --> 00:24:15,954 YOU CAN DO STRUCTURED 775 00:24:16,021 --> 00:24:17,322 FORMS OF VARIOUS THINGS 776 00:24:17,389 --> 00:24:19,658 AND DO CLASSIFICATION AND 777 00:24:19,724 --> 00:24:20,458 GENERATION SIMULTANEOUSLY 778 00:24:20,525 --> 00:24:21,993 RATHER THAN DIVIDING THEM 779 00:24:22,060 --> 00:24:23,995 OUT IN SEPARATE MODELS. 780 00:24:24,062 --> 00:24:26,398 LAST ONE WAS REALLY SHIFT 781 00:24:26,464 --> 00:24:27,899 FROM REPRESENTATION 782 00:24:27,966 --> 00:24:29,201 LEARNING WE WERE DOING 783 00:24:29,267 --> 00:24:30,502 PREVIOUSLY TO MORE 784 00:24:30,569 --> 00:24:32,170 PROMPTABLE INTERFACE AND 785 00:24:32,237 --> 00:24:33,839 SPECIFICALLY LIKE WHEN I 786 00:24:33,905 --> 00:24:36,608 DESCRIBED IT BUYEFORE, WE 787 00:24:36,675 --> 00:24:38,476 HAD ARCHITECTURE THAT WAS 788 00:24:38,543 --> 00:24:39,711 REUSABLE AND LOOKING AT 789 00:24:39,778 --> 00:24:41,646 HOW CAN WE BUILD OUT TASK 790 00:24:41,713 --> 00:24:44,249 SPECIFIC HEADERS TO MOVING 791 00:24:44,316 --> 00:24:44,983 TOWARD SOMETHING THAT IS 792 00:24:45,050 --> 00:24:48,486 MUCH MORE ORIENTED AROUND 793 00:24:48,553 --> 00:24:50,755 BUILDING OUT PROMPTABLE 794 00:24:50,822 --> 00:24:53,458 INTERFACE UNIVERSAL IN 795 00:24:53,525 --> 00:24:55,360 SOME SENSE, AND DOES TASKS 796 00:24:55,427 --> 00:24:57,028 WE WOULD LIKE. 797 00:24:57,095 --> 00:24:58,129 UNSURPRISINGLY BECAUSE OF 798 00:24:58,196 --> 00:24:59,197 SUCCESS AND EASE OF USING 799 00:24:59,264 --> 00:25:01,132 SOME TOOLS THERE HAS BEEN 800 00:25:01,199 --> 00:25:03,268 EME 801 00:25:03,335 --> 00:25:03,969 EMERGENCE OF HUGE NUMBER 802 00:25:04,035 --> 00:25:05,537 OF DIFFERENT PIECES IN 803 00:25:05,604 --> 00:25:06,304 THIS SPACE. 804 00:25:06,371 --> 00:25:07,672 WE HAD ACTUALLY LOOKED AT 805 00:25:07,739 --> 00:25:09,741 THIS A LONG TIME AGO WITH 806 00:25:09,808 --> 00:25:11,476 PROJECT RENAMED BUT ONE OF 807 00:25:11,543 --> 00:25:13,144 THE THINGS WE WERE LOOKING 808 00:25:13,211 --> 00:25:15,413 AT WAS NOTION OF HOW 809 00:25:15,480 --> 00:25:17,749 ACCOUNT WE DO DOMAIN 810 00:25:17,816 --> 00:25:20,252 SPECIFIC, LIKE PUB MED, 811 00:25:20,318 --> 00:25:22,621 INCREDIBLE RESOURCE AND 812 00:25:22,687 --> 00:25:24,022 MAYBE SUFFICIENT AMOUNT OF 813 00:25:24,089 --> 00:25:26,491 TEXT WE CAN GO FROM THERE 814 00:25:26,558 --> 00:25:29,327 AND ACTUALLY TRAIN OUT 815 00:25:29,394 --> 00:25:30,028 DOMAIN-SPECIFIC MODEL AND 816 00:25:30,095 --> 00:25:32,230 WE LOOKED AT THIS AND WE 817 00:25:32,297 --> 00:25:33,531 THOUGHT, YOU KNOW, WHY 818 00:25:33,598 --> 00:25:35,166 WOULD WE WANT TO DO THIS? 819 00:25:35,233 --> 00:25:37,302 MAYBE WE COULD DO THIS, 820 00:25:37,369 --> 00:25:38,670 WHY WOULD WE WANT TO DO 821 00:25:38,737 --> 00:25:39,037 THIS? 822 00:25:39,104 --> 00:25:42,874 THE REALIZATION IN BOUNDED 823 00:25:42,941 --> 00:25:44,142 RESOURCE, MAYBE WE WANT TO 824 00:25:44,209 --> 00:25:46,411 HAVE MORE EFFICIENT 825 00:25:46,478 --> 00:25:48,747 LEARNING BY FOCUSING ON 826 00:25:48,813 --> 00:25:52,751 IN-DOMAIN DATA CO 827 00:25:52,817 --> 00:25:53,652 COMPREHENSIVE APPROACH OF 828 00:25:53,718 --> 00:25:54,386 TRYING TO UNDERSTAND 829 00:25:54,452 --> 00:25:54,986 EVERYTHING. 830 00:25:55,053 --> 00:25:56,221 INITIALLY LOOKING AT 831 00:25:56,288 --> 00:25:58,056 THINGS SPECIFICALLY AROUND 832 00:25:58,123 --> 00:25:59,491 VOCABULARY THAT SHOWS UP, 833 00:25:59,557 --> 00:26:03,161 YOU KNOW, THIS IS EXCERPT 834 00:26:03,228 --> 00:26:05,263 FROM THE PAPER RENAME YET, 835 00:26:05,330 --> 00:26:06,364 WE WERE SPECIFICALLY 836 00:26:06,431 --> 00:26:08,266 LOOKING AT MAYBE THE WORD 837 00:26:08,333 --> 00:26:09,734 BREAKS AND TOKENIZATION 838 00:26:09,801 --> 00:26:13,004 SCHEME DON'T MAKE SENSE 839 00:26:13,071 --> 00:26:14,639 FOR OTHER DOMAIN MODEL, IF 840 00:26:14,706 --> 00:26:15,674 WE SPECIFICALLY TRAIN ON 841 00:26:15,740 --> 00:26:18,043 THIS DOMAIN, PRESERVE 842 00:26:18,109 --> 00:26:19,044 INTEGRITY OF THINGS WE 843 00:26:19,110 --> 00:26:20,645 CARE ABOUT FOR DOWN-STREAM 844 00:26:20,712 --> 00:26:22,380 TASK IN BIOMEDICINE, YOU 845 00:26:22,447 --> 00:26:24,382 KNOW COMPARED TO WHAT 846 00:26:24,449 --> 00:26:26,418 OTHER MODELS WERE DOING 847 00:26:26,484 --> 00:26:28,320 SHATTERING TOKENS AND 848 00:26:28,386 --> 00:26:30,922 WORDS INTO MANY TOKENS. 849 00:26:30,989 --> 00:26:32,857 WE MADE THIS AVAILABLE SO 850 00:26:32,924 --> 00:26:35,026 EVERYONE COULD USE IT, 851 00:26:35,093 --> 00:26:36,661 FANTASTIC RESOURCE FOR 852 00:26:36,728 --> 00:26:37,462 EVERYONE. 853 00:26:37,529 --> 00:26:38,697 AGAIN, WHILE THERE IS A 854 00:26:38,763 --> 00:26:41,032 LOT OF UTILITY IN HAVING 855 00:26:41,099 --> 00:26:43,268 SMALLER MODELS ACROSS THE 856 00:26:43,335 --> 00:26:46,871 BOARD WE START TO SEE 857 00:26:46,938 --> 00:26:47,272 SHIFT TOWARD MORE 858 00:26:47,339 --> 00:26:49,641 GENERALIST MODELS TO ALLOW 859 00:26:49,708 --> 00:26:51,743 US TO WORK IN BIOMEDICINE, 860 00:26:51,810 --> 00:26:52,477 AS WELL. 861 00:26:52,544 --> 00:26:53,878 ONE THING INTERESTING, I 862 00:26:53,945 --> 00:26:56,247 LOVE SEEING EVOLUTION 863 00:26:56,314 --> 00:26:57,415 TREES LIKE THIS. 864 00:26:57,482 --> 00:26:59,351 I ENJOY THEM IN PART SO 865 00:26:59,417 --> 00:27:01,519 MUCH BECAUSE THEY JUST 866 00:27:01,586 --> 00:27:02,821 ADVANCED SO QUICKLY AND 867 00:27:02,887 --> 00:27:06,124 THIS IS NOT EVEN CO 868 00:27:06,191 --> 00:27:08,259 COMPREHENSIVE ANYMORE LIKE 869 00:27:08,326 --> 00:27:10,195 COMMAND R AND OTHERS NOT 870 00:27:10,261 --> 00:27:10,829 THERE, AS WELL. 871 00:27:10,895 --> 00:27:12,030 MAYBE I'LL QUICKLY TALK 872 00:27:12,097 --> 00:27:15,533 ABOUT SPECIFICALLY ONE OF 873 00:27:15,600 --> 00:27:17,869 THE THINGS ON ENCODER 874 00:27:17,936 --> 00:27:20,038 STYLE ONLY ARCHITECTURE 875 00:27:20,105 --> 00:27:20,805 REFLECTED IN SEVERAL OF 876 00:27:20,872 --> 00:27:23,908 THESE MODELS AND MAYBE THE 877 00:27:23,975 --> 00:27:25,410 ENCODER DECODER STYLE 878 00:27:25,477 --> 00:27:27,479 ARCHITECTURE, MAYBE THE 879 00:27:27,545 --> 00:27:28,847 MOST IMPORTANT THING TO 880 00:27:28,913 --> 00:27:30,882 SAY HERE, THERE IS CLASS 881 00:27:30,949 --> 00:27:34,052 OF MODELS LIKE T-5 MODELS, 882 00:27:34,119 --> 00:27:35,720 MOSTLY TRYING TO PROVIDE 883 00:27:35,787 --> 00:27:37,155 KEYWORD SO PEOPLE 884 00:27:37,222 --> 00:27:37,989 UNDERSTAND THESE ARE TYPE 885 00:27:38,056 --> 00:27:39,657 OF ARCHITECTURE WE SEE 886 00:27:39,724 --> 00:27:41,326 WHEN TALKING ABOUT THESE 887 00:27:41,393 --> 00:27:41,526 TERMS. 888 00:27:41,593 --> 00:27:43,461 MAYBE THE THIRD TYPE THAT 889 00:27:43,528 --> 00:27:45,764 IS MORE COMMON TO EVERYONE 890 00:27:45,830 --> 00:27:47,465 NOW IS THIS BROADER NOTION 891 00:27:47,532 --> 00:27:52,037 OF LIKE DECORD ONLY STYLE 892 00:27:52,103 --> 00:27:53,171 ARCHITECTURE, YOU ARE 893 00:27:53,238 --> 00:27:54,172 SPECIFICALLY TRAINING FOR 894 00:27:54,239 --> 00:27:55,673 NEXT TOKEN PREDICTION FROM 895 00:27:55,740 --> 00:27:57,575 THE PRIOR TOKENS. 896 00:27:57,642 --> 00:28:00,178 AND AGAIN, THESE HAVE BEEN 897 00:28:00,245 --> 00:28:01,746 SHOWN TO WORK REALLY, 898 00:28:01,813 --> 00:28:03,481 REALLY WELL SPECIFIC WHEN 899 00:28:03,548 --> 00:28:07,352 TRAINED ON SPECIFICALLY 900 00:28:07,419 --> 00:28:08,720 BIOMEDICAL AND DOMAIN SORT 901 00:28:08,787 --> 00:28:11,322 OF DATA, IN I THINK THINGS 902 00:28:11,389 --> 00:28:15,093 LIKE BIO-GPT RELATIVE TO 903 00:28:15,160 --> 00:28:15,627 HUM 904 00:28:15,693 --> 00:28:16,061 HUMANANOTATION AND 905 00:28:16,127 --> 00:28:17,395 RELATIVE TO OTHER 906 00:28:17,462 --> 00:28:18,730 OFFERINGS IN THIS SPACE, 907 00:28:18,797 --> 00:28:20,365 AS WELL. 908 00:28:20,432 --> 00:28:22,200 MAYBE WHAT I'LL DO IS I 909 00:28:22,267 --> 00:28:25,270 THINK MANY OF US HAVE SEEN 910 00:28:25,336 --> 00:28:27,038 SORT OF SOME RESULTS, I 911 00:28:27,105 --> 00:28:28,273 AUN TO ACKNOWLEDGE THEM 912 00:28:28,339 --> 00:28:29,407 AND ACKNOWLEDGE THAT IN 913 00:28:29,474 --> 00:28:31,843 SOME CASES WE ASKED THIS 914 00:28:31,910 --> 00:28:33,278 QUESTION OF HOW WELL DOES 915 00:28:33,344 --> 00:28:34,746 AI PERFORM CLINICALLY AND 916 00:28:34,813 --> 00:28:37,449 LOOK AT OUT OF BOX 917 00:28:37,515 --> 00:28:40,251 EXPERT-LEVEL COMPETENCY 918 00:28:40,318 --> 00:28:43,588 ON -- DEBATE AROUND LIKE 919 00:28:43,655 --> 00:28:45,290 SEEING THIS TEST IS NOT 920 00:28:45,356 --> 00:28:48,259 SUFFICIENT FOR CLINICAL 921 00:28:48,326 --> 00:28:51,496 CLAIMING VICTORY BUT IT IS 922 00:28:51,563 --> 00:28:52,030 REALLY INTERESTING 923 00:28:52,097 --> 00:28:53,865 QUESTION IN MANY WAYS 924 00:28:53,932 --> 00:28:55,066 BECAUSE I THINK AGAIN, IF 925 00:28:55,133 --> 00:28:57,135 I GO BACK MAYBE A COUPLE 926 00:28:57,202 --> 00:29:00,004 YEARS AND YOU ASK ME, IS 927 00:29:00,071 --> 00:29:03,508 SOMEBODY AN MLP, WE WANT 928 00:29:03,575 --> 00:29:05,577 TO DO THESE TYPES OF 929 00:29:05,643 --> 00:29:07,145 QUESTIONS LIKE ALLEN 930 00:29:07,212 --> 00:29:08,646 INSTITUTE ENDEAVOR AND 931 00:29:08,713 --> 00:29:10,782 DOING PROGRESSIVELY HARDER 932 00:29:10,849 --> 00:29:13,351 AND HARDER MAC QUESTIONS 933 00:29:13,418 --> 00:29:14,853 OR WRITTEN EXAM QUESTIONS 934 00:29:14,919 --> 00:29:17,255 IN OTHER DOMAINS. 935 00:29:17,322 --> 00:29:18,490 COULD YOU BUILD A TOOL TO 936 00:29:18,556 --> 00:29:19,124 DO THIS? 937 00:29:19,190 --> 00:29:21,126 I WOULD HAVE GIVEN YOU A 938 00:29:21,192 --> 00:29:25,196 LOT OF REALLY SORT OF 939 00:29:25,263 --> 00:29:26,364 ANSWERS THAT WERE HEDGED 940 00:29:26,431 --> 00:29:28,766 IN DOUBT AND MAYBE WE 941 00:29:28,833 --> 00:29:30,468 COULD DO IT, IT WILL TAKE 942 00:29:30,535 --> 00:29:31,269 QUITE A BIT. 943 00:29:31,336 --> 00:29:32,370 I THINK WHAT IS 944 00:29:32,437 --> 00:29:34,072 INTERESTING, INDEPENDENT 945 00:29:34,139 --> 00:29:36,040 IF WE THINK THIS IS USEFUL 946 00:29:36,107 --> 00:29:37,742 FOR LIKE CLINICAL BASIS IN 947 00:29:37,809 --> 00:29:38,776 MEDICINE, IS REALLY THIS 948 00:29:38,843 --> 00:29:39,644 NOTION THAT THIS IS 949 00:29:39,711 --> 00:29:42,547 SOMETHING THAT 950 00:29:42,614 --> 00:29:43,181 FUNDAMENTALLY COUPLE YEARS 951 00:29:43,248 --> 00:29:44,983 AGO MAYBE WE COULD HAVE 952 00:29:45,049 --> 00:29:47,018 DONE IN WELL-SCOPED 953 00:29:47,085 --> 00:29:48,186 SCENARIOS AND SEEMS TO BE 954 00:29:48,253 --> 00:29:49,821 SOMETHING THAT 955 00:29:49,888 --> 00:29:50,655 INCREASINGLY LARGE 956 00:29:50,722 --> 00:29:52,056 LANGUAGE MODELS ARE ABLE 957 00:29:52,123 --> 00:29:54,058 TO DO OUT OF THE BOX TODAY 958 00:29:54,125 --> 00:29:55,894 AND THAT IS A REALLY 959 00:29:55,960 --> 00:29:58,596 FUNDAMENTAL SHIFT IN THIS 960 00:29:58,663 --> 00:30:00,965 UNDERLYING NLP SPACE. 961 00:30:01,032 --> 00:30:02,667 SO I THINK ONE OF THE 962 00:30:02,734 --> 00:30:04,002 MAYBE ONE OF THE QUESTIONS 963 00:30:04,068 --> 00:30:05,236 THAT COMES UP THIS 964 00:30:05,303 --> 00:30:07,739 QUESTION AROUND GPT FOR 965 00:30:07,805 --> 00:30:10,041 SHARING GENERAL DOMAIN SO 966 00:30:10,108 --> 00:30:11,176 LIKE WHY WOULD WE EXPECT 967 00:30:11,242 --> 00:30:12,844 IT TO WORK FOR 968 00:30:12,911 --> 00:30:13,144 BIOMEDICINE? 969 00:30:13,211 --> 00:30:14,779 MAYBE WE EXPECT IT TO WORK 970 00:30:14,846 --> 00:30:16,147 WELL FOR BIOMEDICINE 971 00:30:16,214 --> 00:30:17,649 BECAUSE IT CONTAINS 972 00:30:17,715 --> 00:30:18,983 BIOMEDICAL TEXT. 973 00:30:19,050 --> 00:30:22,120 LOOK AT SOME COMPONENTS IN 974 00:30:22,187 --> 00:30:25,590 LARGE RESOURCES LIKE FILE, 975 00:30:25,657 --> 00:30:27,592 PUB MED CENTRAL IS 976 00:30:27,659 --> 00:30:29,127 SIGNIFICANT PORTION OF 977 00:30:29,194 --> 00:30:31,462 THAT 825 GIGA BYTES 978 00:30:31,529 --> 00:30:33,031 TOGETHER WITH ABSTRACT 979 00:30:33,097 --> 00:30:35,833 OVER 100 GIGS, AN EIGHTH 980 00:30:35,900 --> 00:30:37,402 OF THE DATA IN MANY LARGE 981 00:30:37,468 --> 00:30:38,937 LANGUAGE MODELS. 982 00:30:39,003 --> 00:30:40,805 THIS IS REALLY INTERESTING 983 00:30:40,872 --> 00:30:42,740 IDEA OF, YOU KNOW, MAYBE 984 00:30:42,807 --> 00:30:45,276 SOME GENERALIST MODELS 985 00:30:45,343 --> 00:30:47,078 BEING TRAINED, NONE OF US 986 00:30:47,145 --> 00:30:49,747 KNOW WHAT ARE IN SOME OF 987 00:30:49,814 --> 00:30:52,217 THESE, BUT BASED ON 988 00:30:52,283 --> 00:30:54,118 OPEN-SOURCE MODELS TRAINED 989 00:30:54,185 --> 00:30:55,520 ON SOME OF THESE DATASET 990 00:30:55,587 --> 00:30:57,522 MAYBE GOOD IMPRESSION WHAT 991 00:30:57,589 --> 00:30:59,057 TYPES OF THINGS ARE IN 992 00:30:59,123 --> 00:30:59,257 THERE. 993 00:30:59,324 --> 00:31:00,325 MAYBE WAY TO THINK ABOUT 994 00:31:00,391 --> 00:31:02,460 SOME OF THIS IS IN 995 00:31:02,527 --> 00:31:03,261 BIOMEDICINE, WHAT WE'LL 996 00:31:03,328 --> 00:31:04,896 HAVE THIS SET OF 997 00:31:04,963 --> 00:31:07,332 GENERALIST MODELS, FOR 998 00:31:07,398 --> 00:31:08,666 EXAMPLE, THINGS LIKE 999 00:31:08,733 --> 00:31:10,435 GPT-4, THEN WE'LL ALSO 1000 00:31:10,501 --> 00:31:12,170 HAVE SET OF LIKE TASK 1001 00:31:12,237 --> 00:31:13,871 SPECIFIC FINE TUNING 1002 00:31:13,938 --> 00:31:15,306 MODELS LIKE MED PALM, IF 1003 00:31:15,373 --> 00:31:17,308 WE'RE LOOKING AT MULTI 1004 00:31:17,375 --> 00:31:19,777 MODAL THINGS OR LAVA OR 1005 00:31:19,844 --> 00:31:21,546 MED PUB-M. 1006 00:31:21,613 --> 00:31:24,015 WE'RE GETTING MORE 1007 00:31:24,082 --> 00:31:25,083 SPECIALIZED AND IT IS 1008 00:31:25,149 --> 00:31:26,484 HARDER TO STEER DOWN TO 1009 00:31:26,551 --> 00:31:27,585 DOMAIN SPECIFIC 1010 00:31:27,652 --> 00:31:28,753 PRETRAINING OF SPECIFIC 1011 00:31:28,820 --> 00:31:30,355 TYPES OF MODELS. 1012 00:31:30,421 --> 00:31:33,191 THERE ARE MORE POWERFUL 1013 00:31:33,258 --> 00:31:34,192 MODELS, EASIER TO STEER 1014 00:31:34,259 --> 00:31:35,560 AND MAYBE THE TRADEOFF TO 1015 00:31:35,627 --> 00:31:38,630 THINK ABOUT. 1016 00:31:38,696 --> 00:31:40,898 WITH THAT IN MIND AND SORT 1017 00:31:40,965 --> 00:31:42,734 OF AN IDEA WHERE WE ARE IN 1018 00:31:42,800 --> 00:31:47,171 THE BIOMEDICAL LLM SPACE I 1019 00:31:47,238 --> 00:31:48,406 WANT TO HIGHLIGHT WORK OUR 1020 00:31:48,473 --> 00:31:49,841 GROUP HAS DONE AND REALLY 1021 00:31:49,907 --> 00:31:51,376 TALK THROUGH SOME OF MAYBE 1022 00:31:51,442 --> 00:31:54,579 LIKE FRONTIERS WE'VE BEEN 1023 00:31:54,646 --> 00:31:55,847 TRYING TO EXPLORE. 1024 00:31:55,913 --> 00:31:57,415 QUICKLY, ONE OF THE FIRST 1025 00:31:57,482 --> 00:32:01,185 IS IN BROAD SPACE OF 1026 00:32:01,252 --> 00:32:01,786 PROMPT 1027 00:32:01,853 --> 00:32:02,053 PROMPTING. 1028 00:32:02,120 --> 00:32:05,290 AS I MENTIONED BEFORE, A 1029 00:32:05,356 --> 00:32:07,025 LOT OF MODELS HAVE SHIFTED 1030 00:32:07,091 --> 00:32:08,259 US TO THINKING MORE AND 1031 00:32:08,326 --> 00:32:09,794 MORE ABOUT HOW DO WE 1032 00:32:09,861 --> 00:32:11,296 PROMPT THE MODEL TO DO THE 1033 00:32:11,362 --> 00:32:12,697 THING WE'D LIKE TO DO OR 1034 00:32:12,764 --> 00:32:14,899 IN TURN, HOW DO WE HAVE A 1035 00:32:14,966 --> 00:32:15,333 MOD 1036 00:32:15,400 --> 00:32:18,036 MODEL START TO ACT ON THE 1037 00:32:18,102 --> 00:32:19,404 SET OF THINGS WE WOULD 1038 00:32:19,470 --> 00:32:20,905 LIKE IT TO DO? 1039 00:32:20,972 --> 00:32:23,941 I'LL SORT OF TAKE INITIAL 1040 00:32:24,008 --> 00:32:25,710 STEP IN THIS DIRECTION. 1041 00:32:25,777 --> 00:32:27,812 IF WE THINK ABOUT SORT OF 1042 00:32:27,879 --> 00:32:29,914 MAYBE LIKE COMPARISON WITH 1043 00:32:29,981 --> 00:32:33,151 THINGS LIKE MED PALM M, 1044 00:32:33,217 --> 00:32:35,353 THEY WERE ABLE TO 1045 00:32:35,420 --> 00:32:37,922 DEMONSTRATE COMP TENSZY ON 1046 00:32:37,989 --> 00:32:39,424 U.S. MEDICAL EXAM, ONE 1047 00:32:39,490 --> 00:32:40,325 THING INTERESTING ABOUT 1048 00:32:40,391 --> 00:32:44,228 THE WORK NOT ONLY WAS MED 1049 00:32:44,295 --> 00:32:46,030 PALM 2 ABLE TO IMPROVE 1050 00:32:46,097 --> 00:32:52,270 OVER MED PALM AND BROAD ER 1051 00:32:52,337 --> 00:32:53,738 RESULTS, BUT IF YOU THINK 1052 00:32:53,805 --> 00:32:55,340 ABOUT LIKE HOW THEY WERE 1053 00:32:55,406 --> 00:32:57,175 ABLE TO DRIVE SOME OF THAT 1054 00:32:57,241 --> 00:32:59,110 IMIMPROVEMENT, ONE THING 1055 00:32:59,177 --> 00:33:01,346 THEY WERE LOOKING AT WAS 1056 00:33:01,412 --> 00:33:09,520 HOW CAN WE COME UP WITH 1057 00:33:09,587 --> 00:33:10,822 APPROPRIATE PROMPTS AND 1058 00:33:10,888 --> 00:33:12,023 RESTYLE STRATEGY. 1059 00:33:12,090 --> 00:33:14,892 LOOK AT RESULT ON MED QA, 1060 00:33:14,959 --> 00:33:16,828 SOME OF THE PROMPTS WERE 1061 00:33:16,894 --> 00:33:21,332 LOOKING AT UP TO 40-45-ISH 1062 00:33:21,399 --> 00:33:23,334 API CALLS PER SPECIFIC 1063 00:33:23,401 --> 00:33:23,701 PROBLEM. 1064 00:33:23,768 --> 00:33:25,136 THIS IS INTERESTING FOR 1065 00:33:25,203 --> 00:33:26,704 VARIETY OF REASONS AROUND 1066 00:33:26,771 --> 00:33:28,206 HOW TO BREAK TASKS DOWN 1067 00:33:28,272 --> 00:33:30,308 INTO THINGS LLMS CAN 1068 00:33:30,375 --> 00:33:32,110 ANSWER MORE EASILY AND 1069 00:33:32,176 --> 00:33:33,578 ALSO INTERESTING IN TERMS 1070 00:33:33,644 --> 00:33:35,279 OF WHAT SHOULD WE EXPECT 1071 00:33:35,346 --> 00:33:37,882 OUT OF BOX WITH ONE PROMPT 1072 00:33:37,949 --> 00:33:39,917 VERSUS MULTIPLE PROMPTS, 1073 00:33:39,984 --> 00:33:40,385 AS WELL. 1074 00:33:40,451 --> 00:33:41,419 WHAT WAS INTERESTING IN 1075 00:33:41,486 --> 00:33:42,987 THIS CASE, AS WELL, WHEN 1076 00:33:43,054 --> 00:33:45,757 WE START LOOKING AT OTHER 1077 00:33:45,823 --> 00:33:48,192 RESULTS IN THE SPACE, 1078 00:33:48,259 --> 00:33:49,527 MIGHT BE THINKING WHAT DO 1079 00:33:49,594 --> 00:33:51,562 WE GET OUT OF THE BOX? 1080 00:33:51,629 --> 00:33:54,465 WE STARTED TO SEE OTHER 1081 00:33:54,532 --> 00:33:57,435 RESULTS THAT STARTED TO 1082 00:33:57,502 --> 00:34:00,171 SUGGEST ACTUALLY WE COULD 1083 00:34:00,238 --> 00:34:02,874 GET OUT OF THE BOX 1084 00:34:02,940 --> 00:34:06,043 PERFORMANCE WITHOUT MORE 1085 00:34:06,110 --> 00:34:07,912 EXPENSIVE FINE-TUNING 1086 00:34:07,979 --> 00:34:09,914 MECHANISM WITH 1087 00:34:09,981 --> 00:34:10,214 PERFORMANCE. 1088 00:34:10,281 --> 00:34:12,517 QUESTION WAS COULD WE 1089 00:34:12,583 --> 00:34:14,419 START, IF WE COULD GET 1090 00:34:14,485 --> 00:34:15,686 THAT WITH NO FINE TUNING 1091 00:34:15,753 --> 00:34:17,588 WITH CALL WITH FEW SHOT 1092 00:34:17,655 --> 00:34:18,956 COULD WE PUSH THAT EVEN 1093 00:34:19,023 --> 00:34:20,458 FURTHER WITH MORE 1094 00:34:20,525 --> 00:34:23,428 ELABORATE PROMPTING 1095 00:34:23,494 --> 00:34:24,095 STRATEGY? 1096 00:34:24,162 --> 00:34:25,396 LET'S SEE WHAT WE CAN DO. 1097 00:34:25,463 --> 00:34:27,632 WE SORT OF CAME UP WITH 1098 00:34:27,698 --> 00:34:28,399 THIS COLLECTIVELY OR 1099 00:34:28,466 --> 00:34:29,667 OTHERS IN THE GROUP COME 1100 00:34:29,734 --> 00:34:31,102 UP WITH COLLECTIVELY HOW 1101 00:34:31,169 --> 00:34:32,904 CAN WE PUSH THIS FORWARD 1102 00:34:32,970 --> 00:34:34,705 EVEN FURTHER? 1103 00:34:34,772 --> 00:34:38,676 AND DO SORT OF SET OF 1104 00:34:38,743 --> 00:34:39,710 ABLATION, WHAT IS ZERO 1105 00:34:39,777 --> 00:34:42,447 SHOT PERFORMANCE, DID 1106 00:34:42,513 --> 00:34:43,147 RANDOM FEW SHOT IDEA AND 1107 00:34:43,214 --> 00:34:46,617 INTRODUCE ED CHAIN OF 1108 00:34:46,684 --> 00:34:48,352 THOUGHT AND INTRODUCED 1109 00:34:48,419 --> 00:34:50,555 NOTION WE WANT TO FIND 1110 00:34:50,621 --> 00:34:52,290 EXAMPLES SORT OF SPECIFIC 1111 00:34:52,356 --> 00:34:53,858 TO THE PROMPT THAT YOU -- 1112 00:34:53,925 --> 00:34:54,892 THE QUESTION YOU ARE 1113 00:34:54,959 --> 00:34:56,861 TRYING TO ANSWER IN THIS 1114 00:34:56,928 --> 00:34:57,094 PROMPT? 1115 00:34:57,161 --> 00:34:58,296 COULD WE COME UP WITH 1116 00:34:58,362 --> 00:35:00,331 SPECIFIC FEW SHOT EXAMPLES 1117 00:35:00,398 --> 00:35:03,634 USING KLM AND TRAIN OF 1118 00:35:03,701 --> 00:35:05,102 THOUGHT TO MOVE INTO 1119 00:35:05,169 --> 00:35:06,671 USEFUL CONTEXT FOR THE 1120 00:35:06,737 --> 00:35:07,538 LLM? 1121 00:35:07,605 --> 00:35:09,707 COULD WE ASSEMBLE ALL 1122 00:35:09,774 --> 00:35:11,209 THESE CHOICES TOGETHER TO 1123 00:35:11,275 --> 00:35:12,310 COME UP WITH A SOLUTION 1124 00:35:12,376 --> 00:35:14,312 THAT ACTUALLY PUSHES THIS 1125 00:35:14,378 --> 00:35:15,546 FURTHER AND I THINK WHAT 1126 00:35:15,613 --> 00:35:17,248 WAS NICE ABOUT THIS WORK 1127 00:35:17,315 --> 00:35:19,016 WAS REALLY COMBINATION OF 1128 00:35:19,083 --> 00:35:21,819 THE ABLATION AND 1129 00:35:21,886 --> 00:35:23,921 UNDERSTANDING PROMPTING 1130 00:35:23,988 --> 00:35:25,022 STRATEGIES MOVING US 1131 00:35:25,089 --> 00:35:27,992 FORWARD AND IT GIVES US 1132 00:35:28,059 --> 00:35:29,227 SIGNIFICANT LIFT ACROSS 1133 00:35:29,293 --> 00:35:30,461 THE BOARD WITH RESPECT TO 1134 00:35:30,528 --> 00:35:33,030 SOME OF THE RESULTS AND 1135 00:35:33,097 --> 00:35:34,198 AGAIN, I WANT TO 1136 00:35:34,265 --> 00:35:39,003 ACKNOWLEDGE THAT U.S. MLE, 1137 00:35:39,070 --> 00:35:40,738 NOT ONLY BENCHMARK FOR 1138 00:35:40,805 --> 00:35:42,473 DOING THIS, LOOK AT BROAD 1139 00:35:42,540 --> 00:35:44,242 SET OF BENCHMARKS TO 1140 00:35:44,308 --> 00:35:45,943 UNDERSTAND MULTIPLE POINTS 1141 00:35:46,010 --> 00:35:48,246 OF EVIDENCE AROUND WHERE 1142 00:35:48,312 --> 00:35:49,914 MIGHT NEEDLE MIGHT BE 1143 00:35:49,981 --> 00:35:51,449 GETTING MOVED EFFECTIVELY. 1144 00:35:51,516 --> 00:35:53,351 I ALSO WANT TO HIGHLIGHT 1145 00:35:53,417 --> 00:35:55,086 WORK OTHERS IN THE GROUP 1146 00:35:55,152 --> 00:35:59,857 ARE DOING AROUND SELF- 1147 00:35:59,924 --> 00:36:00,258 SELF-VERIFICATION. 1148 00:36:00,324 --> 00:36:02,059 I THINK FOR MANY OF US, 1149 00:36:02,126 --> 00:36:02,894 THERE ARE NUMBER OF THINGS 1150 00:36:02,960 --> 00:36:04,662 WE MIGHT CARE ABOUT, WE 1151 00:36:04,729 --> 00:36:06,397 MIGHT CARE ABOUT CLOSED 1152 00:36:06,464 --> 00:36:09,634 NATION AND DATA AND 1153 00:36:09,700 --> 00:36:10,368 INCLUESEFITY AND THINGS 1154 00:36:10,434 --> 00:36:11,068 LIKE THAT. 1155 00:36:11,135 --> 00:36:13,604 MANY SERIOUS ISSUES AND 1156 00:36:13,671 --> 00:36:14,005 RISKS. 1157 00:36:14,071 --> 00:36:15,339 I WANT TO TALK ABOUT 1158 00:36:15,406 --> 00:36:18,776 SPECIFICALLY LIKE 1159 00:36:18,843 --> 00:36:19,810 HALLUCINATIONS AND MAYBE 1160 00:36:19,877 --> 00:36:21,479 TO A LESSER DEGREE MATH 1161 00:36:21,546 --> 00:36:23,414 AND LOGIC ERRORS AND 1162 00:36:23,481 --> 00:36:25,283 REASONING ABILITY. 1163 00:36:25,349 --> 00:36:27,018 BECAUSE I THINK WHAT WE 1164 00:36:27,084 --> 00:36:28,953 ARE LOOKING AT WAS 1165 00:36:29,020 --> 00:36:30,054 INITIALLY THIS SORT OF 1166 00:36:30,121 --> 00:36:32,290 CONCERN WE HAD HAD AROUND 1167 00:36:32,356 --> 00:36:33,591 IF YOU HAD A PROMPT LIKE 1168 00:36:33,658 --> 00:36:35,293 THIS, YOU KNOW, CAN YOU 1169 00:36:35,359 --> 00:36:36,794 TELL ME ABOUT A SPECIFIC 1170 00:36:36,861 --> 00:36:39,230 USE OF A DRUG AND CITE 1171 00:36:39,297 --> 00:36:41,265 YOUR SOURCES. 1172 00:36:41,332 --> 00:36:43,668 YOU WOULD GET LIKE A QUICK 1173 00:36:43,734 --> 00:36:44,936 SORT OF ANSWER AND SOME 1174 00:36:45,002 --> 00:36:47,171 SET OF SOURCES, RIGHT? 1175 00:36:47,238 --> 00:36:52,743 IT IS VERY EASY FOR GPT TO 1176 00:36:52,810 --> 00:36:53,311 HALLUCINATE. 1177 00:36:53,377 --> 00:36:54,779 THIS SPECIFIC SOURCE IS 1178 00:36:54,845 --> 00:36:56,981 NOT WHAT IT SEEMS TO BE 1179 00:36:57,048 --> 00:36:58,215 BASED ON THE SOURCE THAT 1180 00:36:58,282 --> 00:36:59,350 IS THERE. 1181 00:36:59,417 --> 00:37:00,818 IF I CLICK THIS LINK, I GO 1182 00:37:00,885 --> 00:37:02,687 TO SOURCE AND IT IS NOT 1183 00:37:02,753 --> 00:37:05,623 THE CITATION WE THOUGHT IT 1184 00:37:05,690 --> 00:37:06,424 WAS. 1185 00:37:06,490 --> 00:37:07,858 OBVIOUSLY A NUMBER OF 1186 00:37:07,925 --> 00:37:10,094 TOOLS -- GENERATION BEING 1187 00:37:10,161 --> 00:37:13,264 MAYBE KEY TO SORT OF LIKE 1188 00:37:13,331 --> 00:37:15,833 TRYING TO REDUCE THIS. 1189 00:37:15,900 --> 00:37:17,435 ONE THING WE WANTED TO 1190 00:37:17,501 --> 00:37:19,103 COVER IN SOME OF THIS WORK 1191 00:37:19,170 --> 00:37:20,705 WAS REALLY THIS IDEA HOW 1192 00:37:20,771 --> 00:37:22,073 COULD WE TRY TO REDUCE 1193 00:37:22,139 --> 00:37:23,708 SOME OF THESE AND I WANT 1194 00:37:23,774 --> 00:37:26,310 TO ACKNOWLEDGE THAT THERE 1195 00:37:26,377 --> 00:37:28,212 IS ACTUALLY SOME 1196 00:37:28,279 --> 00:37:29,280 SIMILARITY HERE IN THE WAY 1197 00:37:29,347 --> 00:37:31,449 THAT MAYBE LIKE WE RECALL 1198 00:37:31,515 --> 00:37:32,717 FACTS, WHICH IS 1199 00:37:32,783 --> 00:37:33,217 INTERESTING. 1200 00:37:33,284 --> 00:37:35,286 IF I, I ASKED FOR EXAMPLE, 1201 00:37:35,353 --> 00:37:36,687 DID YOU READ THIS REALLY 1202 00:37:36,754 --> 00:37:40,157 LONG BOOK, YOU KNOW, 1203 00:37:40,224 --> 00:37:42,693 PERHAPS LIKE LIMITS OR 1204 00:37:42,760 --> 00:37:44,061 LIKE A PLAY OR SOMETHING. 1205 00:37:44,128 --> 00:37:45,596 YOU MIGHT SAY YES. 1206 00:37:45,663 --> 00:37:50,034 IF I ASKED YOU TO RECITE 1207 00:37:50,101 --> 00:37:51,902 VERBATIM THE FIRST PAGE OR 1208 00:37:51,969 --> 00:37:53,871 WHATEVER MEANINGFUL UNIT 1209 00:37:53,938 --> 00:37:55,806 PROBABLY NONE OF US COULD 1210 00:37:55,873 --> 00:37:56,307 DO IT. 1211 00:37:56,374 --> 00:37:58,009 WE REMEMBER BROADER 1212 00:37:58,075 --> 00:37:58,376 THEMES. 1213 00:37:58,442 --> 00:38:00,277 MAYBE THERE IS SOMETHING 1214 00:38:00,344 --> 00:38:01,112 INTERESTING HAPPENING 1215 00:38:01,178 --> 00:38:03,547 HERE, BUT PRACTICALLY WHAT 1216 00:38:03,614 --> 00:38:05,850 WE WOULD LIKE IS LIKE A 1217 00:38:05,916 --> 00:38:07,218 SOURCE THAT ACTUALLY IS 1218 00:38:07,284 --> 00:38:09,186 THE CORRECT FACTUAL 1219 00:38:09,253 --> 00:38:09,420 SOURCE. 1220 00:38:09,487 --> 00:38:11,188 ONE THING WE THOUGHT ABOUT 1221 00:38:11,255 --> 00:38:14,258 FROM PURE SCIENCE STAND 1222 00:38:14,325 --> 00:38:18,262 POINT ANALOGY TO VERIFKSZ 1223 00:38:18,329 --> 00:38:20,264 BEING EASIER THAN -- WE 1224 00:38:20,331 --> 00:38:22,033 LOOKED AT COUPLE DIFFERENT 1225 00:38:22,099 --> 00:38:25,136 METHOD AROUND HOW CAN WE 1226 00:38:25,202 --> 00:38:27,738 START USING PROMPTS TO 1227 00:38:27,805 --> 00:38:28,739 SELF-FACT CHECK. 1228 00:38:28,806 --> 00:38:30,174 SO WE MIGHT WANT TO GO 1229 00:38:30,241 --> 00:38:32,076 FROM A PATIENT CHART LIKE 1230 00:38:32,143 --> 00:38:33,477 I'M SHOWING ON LEFT TO 1231 00:38:33,544 --> 00:38:34,679 SOMETHING LIKE THE 1232 00:38:34,745 --> 00:38:36,280 SPECIFIC PROBLEM LIST THAT 1233 00:38:36,347 --> 00:38:36,681 IS THERE. 1234 00:38:36,747 --> 00:38:38,282 AND SO WE CAME UP WITH 1235 00:38:38,349 --> 00:38:40,117 THIS PIPELINE OF CAN YOU 1236 00:38:40,184 --> 00:38:44,422 HAVE SORT OF PROMPT THAT 1237 00:38:44,488 --> 00:38:46,323 REPROMPTS TO SAY FIND ALL 1238 00:38:46,390 --> 00:38:48,392 THINGS YOU HAVE MISSED AND 1239 00:38:48,459 --> 00:38:50,728 BUILD EVIDENCE FOR THINGS 1240 00:38:50,795 --> 00:38:52,396 YOU MISSED AND RESULTS, 1241 00:38:52,463 --> 00:38:53,898 MAYBE COMING FROM CLASSIC 1242 00:38:53,964 --> 00:38:55,800 INFORMATION RETRIEVAL 1243 00:38:55,866 --> 00:38:56,834 PIPELINE, WHERE YOU WANT 1244 00:38:56,901 --> 00:38:58,369 TO BOOST RECALL AND 1245 00:38:58,436 --> 00:38:59,904 RESTORE PRECISION 1246 00:38:59,970 --> 00:39:00,271 AFTERWARD. 1247 00:39:00,337 --> 00:39:01,305 EXAMPLE OF WHAT THIS MIGHT 1248 00:39:01,372 --> 00:39:02,573 LOOK LIKE, YOU HAVE A 1249 00:39:02,640 --> 00:39:04,909 PROMPT TEMPLATE AND YOU 1250 00:39:04,975 --> 00:39:05,443 EFFECTIVELY GO THROUGH AND 1251 00:39:05,509 --> 00:39:07,144 SAY EXTRACT DISEASE FROM 1252 00:39:07,211 --> 00:39:09,146 THE TEXT AND IT DOES IT. 1253 00:39:09,213 --> 00:39:12,083 FINE ALL EMISSIONS AND YOU 1254 00:39:12,149 --> 00:39:13,918 SAY GENERATE EVIDENCE FOR 1255 00:39:13,984 --> 00:39:15,720 THIS OMISSION AND ONE OF 1256 00:39:15,786 --> 00:39:18,055 THE -- MAYBE TWO POINTS 1257 00:39:18,122 --> 00:39:18,856 USEFUL HERE. 1258 00:39:18,923 --> 00:39:20,257 ONE, IF YOU GET SOMETHING 1259 00:39:20,324 --> 00:39:23,260 BACK FOR AN LLM THAT IS 1260 00:39:23,327 --> 00:39:24,995 EQUALLY HAND WAVING AND 1261 00:39:25,062 --> 00:39:26,897 GENERATING EVIDENCE, IT IS 1262 00:39:26,964 --> 00:39:28,265 LESS LIKELY IT EXISTS, 1263 00:39:28,332 --> 00:39:30,034 ESPECIALLY IF YOU ASK 1264 00:39:30,101 --> 00:39:31,535 SEVERAL LLMS, IT IS IDEA 1265 00:39:31,602 --> 00:39:32,636 IF YOU WALK DOWN THE 1266 00:39:32,703 --> 00:39:34,972 STREET AND ASK FOR 1267 00:39:35,039 --> 00:39:35,539 DIRECTIONS, THREE PEOPLE 1268 00:39:35,606 --> 00:39:37,742 GIVE SAME SET OF 1269 00:39:37,808 --> 00:39:38,909 DIRECTIONS, PROBABLY 1270 00:39:38,976 --> 00:39:39,643 HEADING RIGHT. 1271 00:39:39,710 --> 00:39:41,979 IF THEY ALL GIVE DIFFERENT 1272 00:39:42,046 --> 00:39:43,914 DIRECTIONS, PAUSE AND 1273 00:39:43,981 --> 00:39:44,815 FIGURE OUT. 1274 00:39:44,882 --> 00:39:46,484 WE SEE SAME THING HERE. 1275 00:39:46,550 --> 00:39:48,285 SECOND POINT, BY 1276 00:39:48,352 --> 00:39:49,620 GENERATING EVIDENCE, MUCH 1277 00:39:49,687 --> 00:39:51,155 EASIER FOR HUMAN TO CHECK 1278 00:39:51,222 --> 00:39:53,023 IS THIS THING THERE OR 1279 00:39:53,090 --> 00:39:53,190 NOT. 1280 00:39:53,257 --> 00:39:56,260 IN A LOT OF HUMAN IN THE 1281 00:39:56,327 --> 00:39:59,230 LOOP USE CASES WE IMAGINE, 1282 00:39:59,296 --> 00:40:00,264 FOR EXAMPLE, THIS IS 1283 00:40:00,331 --> 00:40:02,299 REALLY USEFUL IN PROVIDING 1284 00:40:02,366 --> 00:40:05,035 EVIDENCE HUMANS CAN 1285 00:40:05,102 --> 00:40:05,503 VERIFY. 1286 00:40:05,569 --> 00:40:07,438 WE DO THIS PROCESS AND 1287 00:40:07,505 --> 00:40:08,372 CONTINUE THIS PROCESS 1288 00:40:08,439 --> 00:40:09,540 UNTIL WE LAND ON SOMETHING 1289 00:40:09,607 --> 00:40:15,913 THAT IS A USE FUL PROBLEM 1290 00:40:15,980 --> 00:40:16,313 LIST. 1291 00:40:16,380 --> 00:40:18,983 AND WHAT WE FOUND WHICH IS 1292 00:40:19,049 --> 00:40:22,086 INTERESTING, JUST ADDING 1293 00:40:22,153 --> 00:40:23,320 SIMPLE PROMPTING EXERCISE 1294 00:40:23,387 --> 00:40:25,322 AND FORMULATING INTO 1295 00:40:25,389 --> 00:40:27,391 PIPELINE WE GET COMPARABLE 1296 00:40:27,458 --> 00:40:29,627 TO STATE OF THE ART, 1297 00:40:29,693 --> 00:40:31,729 PRETTY INTERESTING RESULT 1298 00:40:31,796 --> 00:40:33,798 FOR SOMETHING IN ADVERSE 1299 00:40:33,864 --> 00:40:35,499 DRUG EVENT DETECTION, WE 1300 00:40:35,566 --> 00:40:37,568 START TO SEE NOT ONLY CAN 1301 00:40:37,635 --> 00:40:39,436 WE DO THIS, THROUGH 1302 00:40:39,503 --> 00:40:40,171 DISTILLATION, WE SEE 1303 00:40:40,237 --> 00:40:41,872 THINGS SMALLER IN THIS 1304 00:40:41,939 --> 00:40:42,673 CONTEXT, AS WELL. 1305 00:40:42,740 --> 00:40:45,109 MAYBE BROADLY, WAY I WOULD 1306 00:40:45,176 --> 00:40:46,377 THINK ABOUT LIKE SOME OF 1307 00:40:46,443 --> 00:40:48,279 THE EMERGING BEHAVIOR IN 1308 00:40:48,345 --> 00:40:49,814 THIS SPACE, YOU CAN USE 1309 00:40:49,880 --> 00:40:51,982 LARGE LANGUAGE MODEL LIKE 1310 00:40:52,049 --> 00:40:53,384 GPT-4 TO PROVIDE THIS 1311 00:40:53,450 --> 00:40:55,452 MECHANISM FOR UNIVERSAL 1312 00:40:55,519 --> 00:40:56,854 STRUCTURING, YOU CAN DO 1313 00:40:56,921 --> 00:40:58,189 UNIVERSAL SUMMARIZATION, 1314 00:40:58,255 --> 00:40:59,723 IF YOU START THINKING 1315 00:40:59,790 --> 00:41:01,592 ABOUT OTHER THINGS WE CARE 1316 00:41:01,659 --> 00:41:04,128 ABOUT IN PRACTICE, COST 1317 00:41:04,195 --> 00:41:05,930 AND EFFICIENCY, WHITE BOX 1318 00:41:05,996 --> 00:41:08,999 AND BLACK BOX ACCESS, 1319 00:41:09,066 --> 00:41:09,700 CUSTOMIZATION AND CONCERNS 1320 00:41:09,767 --> 00:41:11,435 WE MIGHT HAVE, YOU MIGHT 1321 00:41:11,502 --> 00:41:13,270 WANT TO LOOK AT ASPECTS OF 1322 00:41:13,337 --> 00:41:17,341 KNOWLEDGE DISTILLATION TO 1323 00:41:17,408 --> 00:41:18,642 MODELS YOU CREATE 1324 00:41:18,709 --> 00:41:18,909 YOURSELF. 1325 00:41:18,976 --> 00:41:20,344 I'LL PAUSE THERE AND SHIFT 1326 00:41:20,411 --> 00:41:22,947 AND MOVE TO MULTI MODAL 1327 00:41:23,013 --> 00:41:23,180 MODELS. 1328 00:41:23,247 --> 00:41:24,548 AS I MENTIONED BEFORE, 1329 00:41:24,615 --> 00:41:26,050 THERE IS A LOT OF THINGS 1330 00:41:26,116 --> 00:41:28,285 WE CARE ABOUT AT LARGE IN 1331 00:41:28,352 --> 00:41:29,887 THE LANGUAGE SPACE AND I 1332 00:41:29,954 --> 00:41:31,422 THINK LANGUAGE HAS BEEN 1333 00:41:31,488 --> 00:41:32,523 REALLY DRIVING FORWARD A 1334 00:41:32,590 --> 00:41:34,091 LOT OF SPACE. 1335 00:41:34,158 --> 00:41:35,993 PATIENTS GENERATE MANY, 1336 00:41:36,060 --> 00:41:37,928 MANY DIFFERENT MODALITIES 1337 00:41:37,995 --> 00:41:40,064 OF DATA, RIGHT? 1338 00:41:40,130 --> 00:41:42,466 OUR HOPE WOULD BE MULTI 1339 00:41:42,533 --> 00:41:43,934 MODAL MODELS ARE ABLE TO 1340 00:41:44,001 --> 00:41:44,468 UNDERSTAND AND REASON 1341 00:41:44,535 --> 00:41:46,670 ABOUT MANY OF THESE 1342 00:41:46,737 --> 00:41:47,838 MODALITIES, RIGHT? 1343 00:41:47,905 --> 00:41:49,006 AND I THINK THIS IS REALLY 1344 00:41:49,073 --> 00:41:51,609 INTEREST 1345 00:41:51,675 --> 00:41:53,510 INTERESTING EMERGING SPACE 1346 00:41:53,577 --> 00:41:56,313 IN PART BECAUSE SO MUCH 1347 00:41:56,380 --> 00:41:58,015 WORK IN THIS SPACE AND SO 1348 00:41:58,082 --> 00:42:03,120 MUCH POSSIBILITY, A LOT OF 1349 00:42:03,187 --> 00:42:05,589 WORK GOING ON WITH 1350 00:42:05,656 --> 00:42:07,024 MODALITIES DOING WELL, 1351 00:42:07,091 --> 00:42:08,659 SPECIFICALLY MULTI MODAL, 1352 00:42:08,726 --> 00:42:10,361 WE OFTEN MEAN SPECIFICALLY 1353 00:42:10,427 --> 00:42:12,463 LIKE IMAGING AND TEXT 1354 00:42:12,529 --> 00:42:15,065 RATHER THAN OTHERS WHERE 1355 00:42:15,132 --> 00:42:16,800 THERE IS OPEN NEED AND 1356 00:42:16,867 --> 00:42:19,670 AGAIN, JUST TO PROVIDE 1357 00:42:19,737 --> 00:42:21,405 CONCRETE USE AND MOTIVATE 1358 00:42:21,472 --> 00:42:23,607 SOME OF THE REASONS WHY WE 1359 00:42:23,674 --> 00:42:25,442 MIGHT CARE ABOUT MULTI 1360 00:42:25,509 --> 00:42:27,411 MODAL, WE HAVE CONCRETE 1361 00:42:27,478 --> 00:42:28,746 USE CASES. 1362 00:42:28,812 --> 00:42:30,180 WE LOOK AT FOR EXAMPLE 1363 00:42:30,247 --> 00:42:32,917 LIKE CASE STUDY IN 1364 00:42:32,983 --> 00:42:34,652 IMMUNOTHERAPY, WE MIGHT BE 1365 00:42:34,718 --> 00:42:37,755 INTERESTED IN FINDING 1366 00:42:37,821 --> 00:42:38,055 RESPONDERS. 1367 00:42:38,122 --> 00:42:40,524 WE NEED TO KNOW ABOUT THE 1368 00:42:40,591 --> 00:42:41,091 ENVIRONMENT, WE CAN'T LOOK 1369 00:42:41,158 --> 00:42:42,693 TO TEXT, WE PROBABLY NEED 1370 00:42:42,760 --> 00:42:44,461 TO LOOK TO SOME FORM OF 1371 00:42:44,528 --> 00:42:46,330 PATHOLOGY AND SOME FORM OF 1372 00:42:46,397 --> 00:42:49,566 TEXT OR PERHAPS ENTIRE 1373 00:42:49,633 --> 00:42:50,501 PATIENT RECORD. 1374 00:42:50,567 --> 00:42:53,203 AND SO WHEN WE START 1375 00:42:53,270 --> 00:42:54,438 THINKING ABOUT THE 1376 00:42:54,505 --> 00:42:56,273 CHALLENGES OF BRINGING 1377 00:42:56,340 --> 00:42:58,242 MULTI MODAL TO DOMAIN 1378 00:42:58,309 --> 00:42:59,410 MAYBE FOREMOST IS GOING 1379 00:42:59,476 --> 00:43:01,545 BACK TO DATA PROBLEM I 1380 00:43:01,612 --> 00:43:03,480 MENTIONED ABOUT 1381 00:43:03,547 --> 00:43:06,617 DIFFERENCES IN -- AND 1382 00:43:06,684 --> 00:43:06,917 BIOMEDICAL. 1383 00:43:06,984 --> 00:43:09,219 IF I ASK FOR A PHOTO OF 1384 00:43:09,286 --> 00:43:11,088 LUNG CT SCAN, I MIGHT GET 1385 00:43:11,155 --> 00:43:12,289 THINGS LIKE THIS FROM A 1386 00:43:12,356 --> 00:43:16,760 LOT OF IMAGE GENERATION 1387 00:43:16,827 --> 00:43:17,728 TOOLS AND THESE ARE 1388 00:43:17,795 --> 00:43:18,862 HELPFUL AND SUPER COOL, 1389 00:43:18,929 --> 00:43:21,598 THEY LOOK GREAT IN 1390 00:43:21,665 --> 00:43:22,132 PRESENTATIONS. 1391 00:43:22,199 --> 00:43:25,636 BUT FOR ANYONE WHO IS RA 1392 00:43:25,703 --> 00:43:27,438 RADIOLO 1393 00:43:27,504 --> 00:43:28,172 RADIOLOGIST, NONE OF THESE 1394 00:43:28,238 --> 00:43:29,673 LOOK REAL, ESPECIALLY ON 1395 00:43:29,740 --> 00:43:32,076 LOWER RIGHT, WE EXPECT 1396 00:43:32,142 --> 00:43:33,577 SOMETHING CLOSER TO THIS. 1397 00:43:33,644 --> 00:43:36,613 WE LOOKED AT MULTI MODAL 1398 00:43:36,680 --> 00:43:38,282 MODELS WITHIN OUR GROUP 1399 00:43:38,349 --> 00:43:40,284 HOW CAN WE DO THESE FORMS 1400 00:43:40,351 --> 00:43:43,420 OF GENERATION, FACTUAL 1401 00:43:43,487 --> 00:43:45,289 GENERATION BASED ON RERNS 1402 00:43:45,356 --> 00:43:45,856 EXAMPLES THAT ARE MORE 1403 00:43:45,923 --> 00:43:47,658 REAL IN NATURE THAN THINGS 1404 00:43:47,725 --> 00:43:49,059 WE SEE TODAY AND WHEN WE 1405 00:43:49,126 --> 00:43:51,829 THINK ABOUT THIS BROAD 1406 00:43:51,895 --> 00:43:53,464 LANDSCAPE OF BIOMEDICAL 1407 00:43:53,530 --> 00:43:56,333 LARGE MULTI MODAL MODELS, 1408 00:43:56,400 --> 00:43:58,268 THERE IS A LOT OF EFFORT 1409 00:43:58,335 --> 00:44:02,272 IN THIS SPACE, VISION TR 1410 00:44:02,339 --> 00:44:02,806 TRANSFORMERS AND THINGS 1411 00:44:02,873 --> 00:44:04,408 LOOK AT HOW WE FORM BASIS 1412 00:44:04,475 --> 00:44:06,744 OF GLOBAL AND MODAL 1413 00:44:06,810 --> 00:44:08,579 ALIGNMENT LIKE GLORIA AND 1414 00:44:08,645 --> 00:44:09,747 PROJECTS LIKE WE HAD 1415 00:44:09,813 --> 00:44:10,948 WORKED ON REALLY AROUND 1416 00:44:11,015 --> 00:44:12,850 NOT ONLY CAN WE USE THAT 1417 00:44:12,916 --> 00:44:14,718 GLOBAL AND LOCAL 1418 00:44:14,785 --> 00:44:16,353 ALIGNMENT, CAN WE DO LIKE 1419 00:44:16,420 --> 00:44:17,821 RADIOLOGY SPECIFIC 1420 00:44:17,888 --> 00:44:18,856 LANGUAGE MODELING TO TRY 1421 00:44:18,922 --> 00:44:22,126 TO ADVANCE STATE OF THE 1422 00:44:22,192 --> 00:44:23,260 ART THERE AND OTHER THING 1423 00:44:23,327 --> 00:44:24,962 WE CAN ALSO IMAGINE DOING 1424 00:44:25,029 --> 00:44:27,398 IS LOOKING TO VAST AMOUNT, 1425 00:44:27,464 --> 00:44:28,832 AGAIN, LOOKING TO THINGS 1426 00:44:28,899 --> 00:44:33,670 LIKE PALM MED CENTRAL 1427 00:44:33,737 --> 00:44:35,506 BASIS OF THIS BASIS FOR 1428 00:44:35,572 --> 00:44:36,940 GENERATING DATA COMPARABLE 1429 00:44:37,007 --> 00:44:38,976 TO GENERAL DOMAIN AND SO 1430 00:44:39,043 --> 00:44:40,878 ONE THING WE MIGHT LOOK TO 1431 00:44:40,944 --> 00:44:42,913 FIRST IS IF WE HAVE 1432 00:44:42,980 --> 00:44:44,548 ADVA 1433 00:44:44,615 --> 00:44:46,383 ADVANCES IN CLIP DOING 1434 00:44:46,450 --> 00:44:47,251 CONTRASTING LANGUAGE AND 1435 00:44:47,317 --> 00:44:52,022 YOU NEED IMAGE AND 1436 00:44:52,089 --> 00:44:53,724 CORRESPONDING TEXT COULD 1437 00:44:53,791 --> 00:44:55,159 WE GATHER FROM 1438 00:44:55,225 --> 00:44:56,026 BIOMEDICINE? 1439 00:44:56,093 --> 00:44:57,161 IT LOOKS LIKE WE CAN. 1440 00:44:57,227 --> 00:44:59,963 THERE ARE 30 MILLION PLUS 1441 00:45:00,030 --> 00:45:03,167 FINGER CAPTION PAIRS IN 1442 00:45:03,233 --> 00:45:05,235 PALM MED AND EXAMPLE BEING 1443 00:45:05,302 --> 00:45:05,569 RIGHT HERE. 1444 00:45:05,636 --> 00:45:08,806 IF WE HAD THAT, COULD WE 1445 00:45:08,872 --> 00:45:11,241 IMAGINE TAKING VARIETY OF 1446 00:45:11,308 --> 00:45:12,476 DIFFERENT FIGURES AND 1447 00:45:12,543 --> 00:45:15,546 ADMITTEDLY SOME WILL BE 1448 00:45:15,612 --> 00:45:16,680 PATHOLOGY, CHARTS, 1449 00:45:16,747 --> 00:45:18,115 MOLECULAR DESIGN, 1450 00:45:18,182 --> 00:45:20,217 RADIOLOGY, RADIOGRAPHY, 1451 00:45:20,284 --> 00:45:23,187 BROAD ARRAY OF DIFFERENT 1452 00:45:23,253 --> 00:45:26,123 DATA, CAN WE USE AS BASIS 1453 00:45:26,190 --> 00:45:27,157 FOR CREATING FOUNDATION 1454 00:45:27,224 --> 00:45:28,992 MODEL IN CLIP STYLE 1455 00:45:29,059 --> 00:45:30,527 FOUNDATION MODEL IN SPACE? 1456 00:45:30,594 --> 00:45:33,997 WE WENT AND DID THAT AND 1457 00:45:34,064 --> 00:45:34,998 LOOKS LIKE WE CAN AND WHAT 1458 00:45:35,065 --> 00:45:36,433 IS INTERESTING ABOUT IT, 1459 00:45:36,500 --> 00:45:38,268 RELATIVE TO SOME OF THE 1460 00:45:38,335 --> 00:45:39,303 OTHER ENTRIES PREVIOUSLY 1461 00:45:39,369 --> 00:45:42,106 IN THIS SPACE, WE'RE ABLE 1462 00:45:42,172 --> 00:45:44,374 TO DO QUITE WELL USING 1463 00:45:44,441 --> 00:45:47,044 SOME OF THIS DATA AS BASIS 1464 00:45:47,111 --> 00:45:49,513 FOR THE FOUNDATION MODEL. 1465 00:45:49,580 --> 00:45:51,181 AND I THINK WHAT IS REALLY 1466 00:45:51,248 --> 00:45:52,249 SPECIFICALLY INTERESTING 1467 00:45:52,316 --> 00:45:54,651 TO SEE, NOT JUST ACROSS 1468 00:45:54,718 --> 00:45:57,221 COMMON LIKE TEXT IMAGE 1469 00:45:57,287 --> 00:45:59,590 RETRIEVAL, CLASSIFICATION 1470 00:45:59,656 --> 00:46:01,725 AND BQA BENCHMARK IN THIS 1471 00:46:01,792 --> 00:46:04,128 SPACE, WE START TO SEE 1472 00:46:04,194 --> 00:46:05,662 REALLY WE'RE STARTING TO 1473 00:46:05,729 --> 00:46:09,566 APPROACH SORT OF REALLY 1474 00:46:09,633 --> 00:46:11,468 HIGH ZERO SHOT IMAGE 1475 00:46:11,535 --> 00:46:13,036 CLASSIFICATION PERFORMANCE 1476 00:46:13,103 --> 00:46:14,771 USING SOME OF THESE TOOLS, 1477 00:46:14,838 --> 00:46:15,139 AS WELL. 1478 00:46:15,205 --> 00:46:16,740 IF WE GO BACK TO THINKING 1479 00:46:16,807 --> 00:46:18,709 ABOUT MODEL SIZE 1480 00:46:18,775 --> 00:46:21,311 INCREASES, DATA SIZE 1481 00:46:21,378 --> 00:46:22,079 INCREASES, THINGS LIKE, 1482 00:46:22,146 --> 00:46:23,947 FOR EXAMPLE, PALM MED 1483 00:46:24,014 --> 00:46:25,282 CENTRAL AND THIS 1484 00:46:25,349 --> 00:46:27,851 CONSTRUCTION OF FIGURE 1485 00:46:27,918 --> 00:46:31,855 CAPTION FROM PALM MED AND 1486 00:46:31,922 --> 00:46:32,689 SIGNIFICANTLY LARGER, 1487 00:46:32,756 --> 00:46:34,591 ORDER OF MAGNITUDE LARGER 1488 00:46:34,658 --> 00:46:35,959 THAN SOME OTHER DATA 1489 00:46:36,026 --> 00:46:36,960 AVAILABLE THAT IS CLINICAL 1490 00:46:37,027 --> 00:46:38,962 IN NATURE IN THIS SPACE. 1491 00:46:39,029 --> 00:46:40,464 EVEN THOUGH THEY ARE NOT 1492 00:46:40,531 --> 00:46:42,132 CLINICAL, WE SEE INCREASES 1493 00:46:42,199 --> 00:46:43,934 IN ZERO SHOT PERFORMANCE 1494 00:46:44,001 --> 00:46:46,470 ON CLINICAL TASKS, AS 1495 00:46:46,537 --> 00:46:47,070 WELL. 1496 00:46:47,137 --> 00:46:50,841 SO THIS MOTIVATED US TO GO 1497 00:46:50,908 --> 00:46:53,844 A STEP FURTHER, IMAGE, 1498 00:46:53,911 --> 00:46:56,246 TEXT, PAIR, COULD WE BUILD 1499 00:46:56,313 --> 00:46:57,648 OUT SOMETHING THAT LOOKED 1500 00:46:57,714 --> 00:47:00,284 MORE SPECIFICALLY AT SOME 1501 00:47:00,350 --> 00:47:02,019 INTERACTIVE MULTI MODAL 1502 00:47:02,085 --> 00:47:02,319 MODELS? 1503 00:47:02,386 --> 00:47:03,987 WE WANTED TO START WITH 1504 00:47:04,054 --> 00:47:06,523 THINGS LIKE BIOMED CLIP, 1505 00:47:06,590 --> 00:47:07,891 START WITH THAT DATA 1506 00:47:07,958 --> 00:47:09,660 SOURCE AND THEN USE 1507 00:47:09,726 --> 00:47:11,728 ANOTHER TOOL, GPT-4 OR 1508 00:47:11,795 --> 00:47:13,063 LARGE LANGUAGE MODEL OF 1509 00:47:13,130 --> 00:47:16,266 YOUR CHOICE TO SORT OF DO 1510 00:47:16,333 --> 00:47:19,269 UNIVERSAL ANNOTATION AND 1511 00:47:19,336 --> 00:47:21,705 BUILD OUT A DATES. 1512 00:47:21,772 --> 00:47:23,574 IDEA YOU CAN SAY QUITE 1513 00:47:23,640 --> 00:47:25,309 QUICKLY, BUILD OUT A 1514 00:47:25,375 --> 00:47:27,511 CONVERSATION AROUND THIS 1515 00:47:27,578 --> 00:47:28,278 CAPTION WITHOUT EVER 1516 00:47:28,345 --> 00:47:31,148 HAVING SEEN THE C 1517 00:47:31,215 --> 00:47:32,316 CORRESPONDING IMAN, IT IS 1518 00:47:32,382 --> 00:47:35,552 NOT MULTI MODAL IMAGE OF 1519 00:47:35,619 --> 00:47:38,488 GPT-4, BUILD AROUND THIS 1520 00:47:38,555 --> 00:47:38,922 KAP 1521 00:47:38,989 --> 00:47:41,291 CAPTION AND USE FOR IMAGE. 1522 00:47:41,358 --> 00:47:42,159 IF WE THINK ABOUT 1523 00:47:42,226 --> 00:47:44,161 DIFFERENT TYPES OF DATA 1524 00:47:44,228 --> 00:47:46,296 THAT EXIST IN SOME OF THE 1525 00:47:46,363 --> 00:47:47,531 DOMAINS, YOU HAVE A 1526 00:47:47,598 --> 00:47:49,066 BREADTH OF DATA AND MIGHT 1527 00:47:49,132 --> 00:47:50,133 EXPECT TO BE ABLE TO DO 1528 00:47:50,200 --> 00:47:52,236 THIS QUITE WELL FOR SOME 1529 00:47:52,302 --> 00:47:54,171 INSTRUCTIONS AND C 1530 00:47:54,238 --> 00:47:56,273 CORRESPONDING R 1531 00:47:56,340 --> 00:47:58,542 CORRESPON 1532 00:47:58,609 --> 00:48:00,277 CORRESPONDING RESPONSES. 1533 00:48:00,344 --> 00:48:01,545 CALL OUT FRAMING THIS 1534 00:48:01,612 --> 00:48:03,447 ENTIRE SPACE, IN THIS 1535 00:48:03,513 --> 00:48:05,215 MULTI MODAL SPACE, IF YOU 1536 00:48:05,282 --> 00:48:06,950 SORT OF BREAK DOWN OR 1537 00:48:07,017 --> 00:48:09,253 MODULARIZE THIS OUT, ONE 1538 00:48:09,319 --> 00:48:10,721 INSIGHT, MANY OF THESE 1539 00:48:10,787 --> 00:48:12,556 MODELS ARE FORMED FROM 1540 00:48:12,623 --> 00:48:13,957 INTRODUCING PROJENTHION 1541 00:48:14,024 --> 00:48:16,226 LAYER TO CONVERT IMAGE TO 1542 00:48:16,293 --> 00:48:17,894 TEXT EMBEDDING SPACE. 1543 00:48:17,961 --> 00:48:19,463 THERE ARE A LOT OF 1544 00:48:19,529 --> 00:48:20,197 QUESTIONS, I SHOULD SAY 1545 00:48:20,264 --> 00:48:20,964 ABOUT WHETHER OR NOT 1546 00:48:21,031 --> 00:48:22,899 LANGUAGE IS REALLY THE 1547 00:48:22,966 --> 00:48:24,501 BASE THAT WE SHOULD BE 1548 00:48:24,568 --> 00:48:25,102 WORKING IN. 1549 00:48:25,168 --> 00:48:26,536 I OF COURSE AM VERY HAPPY 1550 00:48:26,603 --> 00:48:28,138 WITH IT, THAT IS SORT OF 1551 00:48:28,205 --> 00:48:30,874 MY BACKGROUND, FOR IMAGES, 1552 00:48:30,941 --> 00:48:32,709 MAYBE NOT THE RIGHT BASIS 1553 00:48:32,776 --> 00:48:34,611 OR LAAT ANY TIME SPACE TO 1554 00:48:34,678 --> 00:48:35,812 BE WORKING IN. 1555 00:48:35,879 --> 00:48:37,781 IT IS FOR LLMS. 1556 00:48:37,848 --> 00:48:39,416 IF WE THINK ABOUT SORT OF 1557 00:48:39,483 --> 00:48:41,818 WHAT THIS FAMILY OF MODELS 1558 00:48:41,885 --> 00:48:43,253 MIGHT LOOK LIKE, WE CAN 1559 00:48:43,320 --> 00:48:44,788 ACTUALLY LOOK AT DIFFERENT 1560 00:48:44,855 --> 00:48:45,956 ENTRIES IN THIS SPACE AND 1561 00:48:46,023 --> 00:48:47,824 LOOK AT THINGS LIKE, YOU 1562 00:48:47,891 --> 00:48:49,893 KNOW, LAVA, WHICH IS A 1563 00:48:49,960 --> 00:48:52,029 MULTI MODAL MODEL THAT 1564 00:48:52,095 --> 00:48:53,664 USES INSTRUCTION TUNING. 1565 00:48:53,730 --> 00:48:55,432 THEY BASICALLY TAKE A 1566 00:48:55,499 --> 00:48:56,733 LANGUAGE MODEL, WHICH IN 1567 00:48:56,800 --> 00:48:59,002 THEIR CASE LLAMA, VISION 1568 00:48:59,069 --> 00:49:00,003 ENCODER, SOMETHING TO 1569 00:49:00,070 --> 00:49:01,672 ENCODE IMAGES LIKE CLIP 1570 00:49:01,738 --> 00:49:03,874 AND PROJECT THAT VISION 1571 00:49:03,940 --> 00:49:05,742 SPACE INTO LANGUAGE SPACE 1572 00:49:05,809 --> 00:49:09,446 IN ORDER TO DO THE BUILD 1573 00:49:09,513 --> 00:49:10,213 THE FINAL MODEL. 1574 00:49:10,280 --> 00:49:11,715 IF WE THINK ABOUT THAT FOR 1575 00:49:11,782 --> 00:49:14,851 THAT SAME SORT OF 1576 00:49:14,918 --> 00:49:15,385 CONSTRUCTION MECHANISM FOR 1577 00:49:15,452 --> 00:49:16,887 MEDICAL SPACE ONE WORK WE 1578 00:49:16,953 --> 00:49:19,189 LOOKED AT WAS LOOKING AT 1579 00:49:19,256 --> 00:49:20,457 ONE I DESCRIBE IS LOOKING 1580 00:49:20,524 --> 00:49:22,526 AT CAN WE TAKE THE 1581 00:49:22,592 --> 00:49:24,094 UNDERLYING LANGUAGE MODEL 1582 00:49:24,161 --> 00:49:28,932 FROM LLAVA AND USE BIOMED 1583 00:49:28,999 --> 00:49:31,201 CLIP AND THEN AGAIN DO 1584 00:49:31,268 --> 00:49:34,304 LINEAR PROJECTION IN. 1585 00:49:34,371 --> 00:49:35,939 THERE ARE NUMBER OF OTHER 1586 00:49:36,006 --> 00:49:37,908 WORKS MAYBE NOTABLY MED 1587 00:49:37,974 --> 00:49:42,112 PALM M THAT FOLLOW SIMILAR 1588 00:49:42,179 --> 00:49:43,046 CONSTRUCTION MECHANISM, 1589 00:49:43,113 --> 00:49:45,315 THEY START WITH PALM AS 1590 00:49:45,382 --> 00:49:46,116 LANGUAGE MODEL AND THEY 1591 00:49:46,183 --> 00:49:48,518 USE BIT FOR VISION ENCODER 1592 00:49:48,585 --> 00:49:50,620 AND DO LINEAR PROJECTION 1593 00:49:50,687 --> 00:49:51,588 AND THIS IS INTERESTING 1594 00:49:51,655 --> 00:49:53,323 BECAUSE IT GIVES A SIMILAR 1595 00:49:53,390 --> 00:49:55,592 SET OF TASKS THEY ARE 1596 00:49:55,659 --> 00:49:56,226 REPORTING ON AND ABLE TO 1597 00:49:56,293 --> 00:49:57,594 DO WITH SLIGHTLY DIFFERENT 1598 00:49:57,661 --> 00:50:02,199 VAR IANT OF WHAT EACH 1599 00:50:02,265 --> 00:50:02,699 COMP 1600 00:50:02,766 --> 00:50:06,703 COMPONENT LOOKS LIKE AND 1601 00:50:06,770 --> 00:50:07,437 ELIXIE 1602 00:50:07,504 --> 00:50:08,972 ELIXIR SIMILAR TO ENCODER 1603 00:50:09,039 --> 00:50:12,042 AND CHOOSE ARGUABLY MORE 1604 00:50:12,109 --> 00:50:13,777 POWERFUL PROJECTION 1605 00:50:13,844 --> 00:50:14,778 MECHANISM MORE SPECIFIC IN 1606 00:50:14,845 --> 00:50:19,950 SOME SENSE, THERE IS MED 1607 00:50:20,016 --> 00:50:21,585 FLAMINGO, COVER SIMILAR 1608 00:50:21,651 --> 00:50:24,421 SET OF IDEA AND IT IS MORE 1609 00:50:24,488 --> 00:50:25,856 POWERFUL PROJECTION 1610 00:50:25,922 --> 00:50:27,524 MECHANISM AND LOOK AT 1611 00:50:27,591 --> 00:50:29,459 SPECIFIC FORM OF DATA TO 1612 00:50:29,526 --> 00:50:32,829 PUSH THE PERFORMANCE 1613 00:50:32,896 --> 00:50:36,700 UPWARD AND MAYBE SORT OF 1614 00:50:36,767 --> 00:50:38,235 CLOSING UP HERE, I AM 1615 00:50:38,301 --> 00:50:40,270 AWARE OF TIME, WE MIGHT 1616 00:50:40,337 --> 00:50:41,605 START THINKING IN SAME WAY 1617 00:50:41,671 --> 00:50:44,441 WE THOUGHT ABOUT LLM, 1618 00:50:44,508 --> 00:50:46,476 MECHANISM FOR UNIVERSAL 1619 00:50:46,543 --> 00:50:47,444 STRUCTURING, HOW WE MIGHT 1620 00:50:47,511 --> 00:50:50,247 LOOK TO SORT OF TAKE MULTI 1621 00:50:50,313 --> 00:50:53,984 MODAL DATA AND ACROSS MANY 1622 00:50:54,050 --> 00:50:55,352 DOMAINS, MAP IT TO LIKE 1623 00:50:55,419 --> 00:50:57,320 THIS NATURAL LANGUAGE 1624 00:50:57,387 --> 00:50:58,355 EMBEDDING SPACE BECAUSE 1625 00:50:58,422 --> 00:50:59,856 THIS IS WHERE OCCURRENCELY 1626 00:50:59,923 --> 00:51:04,194 WE HAVE MOST SOPHISTICATED 1627 00:51:04,261 --> 00:51:04,528 CAPABILITIES. 1628 00:51:04,594 --> 00:51:09,766 BUT, YOU KNOW, IDEALLY THE 1629 00:51:09,833 --> 00:51:10,600 SENSIBLE GOAL OF DOING 1630 00:51:10,667 --> 00:51:12,102 SOMETHING LIKE THIS IS TO 1631 00:51:12,169 --> 00:51:14,704 EMPOWER A LOT OF THE OTHER 1632 00:51:14,771 --> 00:51:16,807 IN THIS SPACE TO DO MORE 1633 00:51:16,873 --> 00:51:18,942 THAN THEY HAVE BEEN ABLE 1634 00:51:19,009 --> 00:51:20,677 TO DO CURRENTLY WITH SOME 1635 00:51:20,744 --> 00:51:22,045 DATA THEY HAVE ACCESS TO. 1636 00:51:22,112 --> 00:51:26,583 AND SO MAYBE WE EMERGED 1637 00:51:26,650 --> 00:51:28,118 AND DONE A GUILTY WORK ON 1638 00:51:28,185 --> 00:51:31,555 SOME OF THESE OTHER 1639 00:51:31,621 --> 00:51:33,123 SPACES, AS WELL AND A LOT 1640 00:51:33,190 --> 00:51:34,825 OF EXPLORATION IS OPEN. 1641 00:51:34,891 --> 00:51:37,227 ONE THING I'VE HEARTENED 1642 00:51:37,294 --> 00:51:38,929 BY AND THINK ABOUT OR 1643 00:51:38,995 --> 00:51:40,096 THOUGHT ABOUT QUITE A BIT 1644 00:51:40,163 --> 00:51:42,666 IS WHAT ARE SOME OF THE 1645 00:51:42,732 --> 00:51:44,334 THINGS WE MIGHT WANT OUT 1646 00:51:44,401 --> 00:51:46,503 OF POPULATION MODEL, 1647 00:51:46,570 --> 00:51:47,871 HEALTH OR BIOMEDICAL 1648 00:51:47,938 --> 00:51:50,974 KNOWLEDGE LARGE-LANGUAGE 1649 00:51:51,041 --> 00:51:52,209 MODEL, TONS AND TONS OF 1650 00:51:52,275 --> 00:51:54,077 OPPORTUNITY WITHIN CONTEXT 1651 00:51:54,144 --> 00:51:55,712 OF PATIENT AND PATIENTSES 1652 00:51:55,779 --> 00:51:58,548 GENERATE DATA AND WHAT 1653 00:51:58,615 --> 00:52:01,985 WOULD SERIALIZED TOKEN 1654 00:52:02,052 --> 00:52:03,019 SEQUENCE LOOK LIKE? 1655 00:52:03,086 --> 00:52:04,721 WHAT DO THINGS LIKE 1656 00:52:04,788 --> 00:52:05,989 CONTINUING TO TRAIN FROM 1657 00:52:06,056 --> 00:52:08,258 GENERAL DOMAIN ON SOME OF 1658 00:52:08,325 --> 00:52:10,560 THESE HEALTH SPECIFIC DATA 1659 00:52:10,627 --> 00:52:11,328 LOOK LIKE. 1660 00:52:11,394 --> 00:52:12,729 TWO REAL QUESTIONS BEHIND 1661 00:52:12,796 --> 00:52:14,831 THIS, IF WE THINK ABOUT 1662 00:52:14,898 --> 00:52:16,266 SCALING FROM BEFORE, LIKE 1663 00:52:16,333 --> 00:52:19,169 WHAT IS MULTI MODAL HEALTH 1664 00:52:19,236 --> 00:52:21,304 SCALING LAW WE SEE AND 1665 00:52:21,371 --> 00:52:22,639 WEREN'T EXPECTING, THAT 1666 00:52:22,706 --> 00:52:24,107 WOULD BE QUITE EXCITING 1667 00:52:24,174 --> 00:52:26,476 AND IN SAME VEIN, WILL 1668 00:52:26,543 --> 00:52:30,313 THERE BE SORT OF EMERGENT 1669 00:52:30,380 --> 00:52:32,082 HEALTH CAPABILITY. 1670 00:52:32,148 --> 00:52:34,384 WITH THAT, I'LL AGAIN 1671 00:52:34,451 --> 00:52:35,252 THANK EVERYONE WHO HAS 1672 00:52:35,318 --> 00:52:37,420 BEEN INVOLVED OVER MANY 1673 00:52:37,487 --> 00:52:38,955 YEARS, THIS IS HUGE EFFORT 1674 00:52:39,022 --> 00:52:40,824 THAT CROSSED MANY 1675 00:52:40,891 --> 00:52:42,726 INSTITUTIONS AND MANY 1676 00:52:42,792 --> 00:52:43,560 PEOPLE, AS WELL. 1677 00:52:43,627 --> 00:52:45,829 SO I'LL STOP THERE AND 1678 00:52:45,896 --> 00:52:46,196 THANK YOU ALL. 1679 00:52:50,400 --> 00:52:51,535 >> THANK YOU SO MUCH FOR 1680 00:52:51,601 --> 00:52:54,738 THIS UNBELIEVABLY INTER 1681 00:52:54,804 --> 00:52:56,273 INTERESTING PRESENTATION. 1682 00:52:56,339 --> 00:52:57,841 INDEED WE ARE ALL LIVING 1683 00:52:57,908 --> 00:53:00,110 THROUGH THIS LLM 1684 00:53:00,176 --> 00:53:02,879 REVOLUTION AND IT IS VERY 1685 00:53:02,946 --> 00:53:04,581 EXCITING TO SEE THAT ALL 1686 00:53:04,648 --> 00:53:06,583 THIS APPLICATIONS YOU ARE 1687 00:53:06,650 --> 00:53:08,485 WORKING ON AND IN OUR 1688 00:53:08,552 --> 00:53:10,353 GROUP, WE HAVE ALSO WORKED 1689 00:53:10,420 --> 00:53:12,622 IN THE SIMILAR DIRECTION, 1690 00:53:12,689 --> 00:53:13,857 FOR EXAMPLE, WORKED ON 1691 00:53:13,924 --> 00:53:15,392 PROBLEM OF MATCHING 1692 00:53:15,458 --> 00:53:16,793 PATIENTS TO CLINICAL 1693 00:53:16,860 --> 00:53:18,428 TRIALS AND THE TOOL THAT 1694 00:53:18,495 --> 00:53:20,830 CAME OUT OF THE WORK IS 1695 00:53:20,897 --> 00:53:21,164 CA 1696 00:53:21,231 --> 00:53:23,600 CALLED TRIAL GPT, WE HAVE 1697 00:53:23,667 --> 00:53:25,969 ALSO WORKED ON MULTI MODAL 1698 00:53:26,036 --> 00:53:28,271 DATA, IN PARTICULAR 1699 00:53:28,338 --> 00:53:30,240 CONSIDERING OPHTHALMOLOGY 1700 00:53:30,307 --> 00:53:34,144 IMAGES WITHANNOTATION AND 1701 00:53:34,210 --> 00:53:36,346 HAD INCREDIBLE ADVANCES 1702 00:53:36,413 --> 00:53:38,214 AND AMAZING HOW MUCH YOU 1703 00:53:38,281 --> 00:53:39,583 CAN BUILD WITH THIS. 1704 00:53:39,649 --> 00:53:42,185 THANK YOU FOR SHARING YOUR 1705 00:53:42,252 --> 00:53:43,720 SIDE OF THE RESEARCH. 1706 00:53:43,787 --> 00:53:46,122 I HAVE A QUESTION FROM AN 1707 00:53:46,189 --> 00:53:48,191 O 1708 00:53:48,258 --> 00:53:49,025 AUDIENCE AND THE QUESTION 1709 00:53:49,092 --> 00:53:52,462 IS FROM JANE, IN THE BIRTH 1710 00:53:52,529 --> 00:53:56,266 ERA DOMAIN SPECIFIC 1711 00:53:56,333 --> 00:53:58,468 BIOBIRD, PALM MED BIRD ARE 1712 00:53:58,535 --> 00:54:00,270 IMPORTANT MODELS, THEY 1713 00:54:00,337 --> 00:54:01,571 PERFORM BETTER THAN 1714 00:54:01,638 --> 00:54:04,541 GENERAL DOMAIN 1715 00:54:04,608 --> 00:54:06,910 COUNTERPARTS IN THE ERA OF 1716 00:54:06,977 --> 00:54:08,812 LLMS WILL THERE BE DOMAIN 1717 00:54:08,878 --> 00:54:11,247 SPECIFIC MODEL SUCH AS 1718 00:54:11,314 --> 00:54:13,917 WIRE GPT 4 THAT CAN BE 1719 00:54:13,984 --> 00:54:17,787 BETTER IN BIOMEDICAL 1720 00:54:17,854 --> 00:54:18,121 TASKS? 1721 00:54:18,188 --> 00:54:19,356 WHAT IS YOUR TAKE ON IT? 1722 00:54:19,422 --> 00:54:20,690 >> I THINK THIS IS REALLY 1723 00:54:20,757 --> 00:54:22,325 OPEN QUESTION AND REALLY 1724 00:54:22,392 --> 00:54:23,927 INTERESTING ONE AULSZ. 1725 00:54:23,994 --> 00:54:25,428 I THINK THERE IS A COUPLE 1726 00:54:25,495 --> 00:54:26,796 OF DIFFERENT WAYS TO THINK 1727 00:54:26,863 --> 00:54:28,632 ABOUT THIS IN GENERAL, I 1728 00:54:28,698 --> 00:54:32,268 THINK ONE OF THEM IS MAYBE 1729 00:54:32,335 --> 00:54:33,436 QUALIFICATION OF LIKE WHAT 1730 00:54:33,503 --> 00:54:36,306 DO YOU THINK OF BIOGPT 4, 1731 00:54:36,373 --> 00:54:38,675 WOULD IT MEAN GPT-4 ONLY 1732 00:54:38,742 --> 00:54:40,276 TRAINED FROM HEALTH DATA 1733 00:54:40,343 --> 00:54:43,079 OR MEAN SOME SORT OF FORK 1734 00:54:43,146 --> 00:54:45,815 THAT BUILDS OFF THE BASIS 1735 00:54:45,882 --> 00:54:49,452 OF GPT-4 SPECIFICALLY FOR 1736 00:54:49,519 --> 00:54:50,954 HEALTHCARE OR MEAN 1737 00:54:51,021 --> 00:54:52,722 EXPOSING SOMETHING LIKE 1738 00:54:52,789 --> 00:54:54,591 GPT-4 TO HEALTHCARE DATA 1739 00:54:54,658 --> 00:54:58,294 IN WAY THAT SORT OF 1740 00:54:58,361 --> 00:54:59,996 PROVIDE FINETUNING 1741 00:55:00,063 --> 00:55:02,666 MECHANISM OVER TIME AND 1742 00:55:02,732 --> 00:55:04,334 DIVERGES FROM LIKE BASE 1743 00:55:04,401 --> 00:55:06,403 MODEL IN THAT SORT OF 1744 00:55:06,469 --> 00:55:07,370 CREATION. 1745 00:55:07,437 --> 00:55:08,905 SO THERE ARE COUPLE 1746 00:55:08,972 --> 00:55:10,540 DIFFERENT WAYS TO THINK 1747 00:55:10,607 --> 00:55:12,776 ABOUT OR ALTERNATIVELY 1748 00:55:12,842 --> 00:55:15,345 JUST USING GPT-4 ENTIRELY 1749 00:55:15,412 --> 00:55:18,281 ALONE AS IT IS IN SOME 1750 00:55:18,348 --> 00:55:19,015 SETTINGS. 1751 00:55:19,082 --> 00:55:21,418 I THINK THERE IS IT 1752 00:55:21,484 --> 00:55:23,853 POINTED TO OR ELUDED TO, 1753 00:55:23,920 --> 00:55:24,954 THERE IS TENSION BETWEEN 1754 00:55:25,021 --> 00:55:27,624 SOME OF THE LIKE SMALLER 1755 00:55:27,691 --> 00:55:28,191 DOMAIN-SPECIFIC MODELS WE 1756 00:55:28,258 --> 00:55:29,125 HAVE CONTROL OVER AND ARE 1757 00:55:29,192 --> 00:55:32,429 MORE LIKE MAYBE A LITTLE 1758 00:55:32,495 --> 00:55:34,964 BIT HARDER TO WORK WITH AT 1759 00:55:35,031 --> 00:55:35,999 TIMES, MAYBE MORE TASK 1760 00:55:36,066 --> 00:55:38,301 SPECIFIC AND SORT OF LIKE 1761 00:55:38,368 --> 00:55:40,036 THIS GENERAL CAPABILITY 1762 00:55:40,103 --> 00:55:42,739 AND SO THERE ARE A LOT OF 1763 00:55:42,806 --> 00:55:43,940 SYSTEMS YOU CAN IMAGINE 1764 00:55:44,007 --> 00:55:46,876 AROUND BASICALLY TRYING TO 1765 00:55:46,943 --> 00:55:49,012 FIRST WORK WITH ONE OF THE 1766 00:55:49,079 --> 00:55:51,181 SYSTEMS THAT IS MORE 1767 00:55:51,247 --> 00:55:52,148 SPECIFIC TO YOUR TASK AND 1768 00:55:52,215 --> 00:55:54,084 THEN BACKING OFF TO SORT 1769 00:55:54,150 --> 00:55:56,653 OF MORE GENERAL SYSTEMS. 1770 00:55:56,720 --> 00:55:58,054 YOU CAN MATCH IT THE OTHER 1771 00:55:58,121 --> 00:56:00,256 WAY OF MAYBE THE INTERFACE 1772 00:56:00,323 --> 00:56:01,958 IS ACTUALLY MORE GENERAL 1773 00:56:02,025 --> 00:56:03,326 SYSTEM AND THEN MORE 1774 00:56:03,393 --> 00:56:04,561 GENERAL SYSTEM IS WORKING 1775 00:56:04,627 --> 00:56:06,629 WITH AND IN COORDINATION 1776 00:56:06,696 --> 00:56:07,697 WITH NUMBER OF TOOLS THAT 1777 00:56:07,764 --> 00:56:09,032 ARE SPECIFIC TO THAT 1778 00:56:09,099 --> 00:56:09,566 SPACE. 1779 00:56:09,632 --> 00:56:10,567 THERE ARE A COUPLE 1780 00:56:10,633 --> 00:56:12,502 DIFFERENT FORMATIONS I'VE 1781 00:56:12,569 --> 00:56:14,771 HEARD ALL OF THEM HAVE AND 1782 00:56:14,838 --> 00:56:16,439 ARE COMPELLING AND HAVE 1783 00:56:16,506 --> 00:56:17,807 PROS AND CONS. 1784 00:56:17,874 --> 00:56:19,943 I THINK ONE OF THE MORE 1785 00:56:20,009 --> 00:56:22,245 INTERESTING THINGS IS THIS 1786 00:56:22,312 --> 00:56:24,481 NOTION AROUND MAYBE THE 1787 00:56:24,547 --> 00:56:25,982 THING THAT WE ALL ACTUALLY 1788 00:56:26,049 --> 00:56:28,518 ENJOY ABOUT USING GPT OR 1789 00:56:28,585 --> 00:56:31,321 FAVORITE LLM IS LIKE THIS 1790 00:56:31,387 --> 00:56:33,456 DISCOURSE IT PROVIDES AND 1791 00:56:33,523 --> 00:56:35,925 ENABLING IT WITH SOME C 1792 00:56:35,992 --> 00:56:36,726 CORRESPONDING TOOLS WE 1793 00:56:36,793 --> 00:56:37,761 WANT IN OUR SPECIFIC 1794 00:56:37,827 --> 00:56:39,629 DOMAIN MAYBE GENERAL PIECE 1795 00:56:39,696 --> 00:56:40,997 IS ACTUALLY THIS INTERFACE 1796 00:56:41,064 --> 00:56:44,100 THAT ALLOW US TO INTERACT 1797 00:56:44,167 --> 00:56:45,001 DIRECTORY WITH SPECIFIC 1798 00:56:45,068 --> 00:56:47,003 TOOLS RATHER THAN JUST 1799 00:56:47,070 --> 00:56:48,271 ALL-OUT REPLACEMENT. 1800 00:56:48,338 --> 00:56:50,840 THAT SAID, I ALSO, IT IS 1801 00:56:50,907 --> 00:56:52,642 HARD TO PREDICT THE FUTURE 1802 00:56:52,709 --> 00:56:54,744 IN ANY MEANINGFUL WAY, I 1803 00:56:54,811 --> 00:56:58,148 THINK MAYBE JUST CALLING 1804 00:56:58,214 --> 00:56:59,082 OUT WHERE I THINK I'VE 1805 00:56:59,149 --> 00:57:00,950 BEEN WRONG, I THINK FOR A 1806 00:57:01,017 --> 00:57:03,753 LONG TIME I REALLY HAD 1807 00:57:03,820 --> 00:57:06,689 REALLY RESISTED THIS IDEA 1808 00:57:06,756 --> 00:57:08,525 FOR BIOMEDICAL MODELS THEY 1809 00:57:08,591 --> 00:57:09,859 WOULD DO AS WELL AS THEY 1810 00:57:09,926 --> 00:57:11,594 DID FOR CLINICAL TASKS 1811 00:57:11,661 --> 00:57:14,197 RATHER BIOMEDICAL TASKS 1812 00:57:14,264 --> 00:57:15,598 AND WAS REALLY SURPRISED 1813 00:57:15,665 --> 00:57:19,202 TO FIND OUT THAT IN MANY 1814 00:57:19,269 --> 00:57:20,603 CASES JUST HAVING ACCESS 1815 00:57:20,670 --> 00:57:22,305 TO MORE DATA ALLOWED THEM 1816 00:57:22,372 --> 00:57:24,374 TO OUT PERFORM ON CLINICAL 1817 00:57:24,440 --> 00:57:25,842 TASKS SOME CLINICAL 1818 00:57:25,909 --> 00:57:26,776 SPECIFIC TOOLS THAT WE HAD 1819 00:57:26,843 --> 00:57:28,278 SEEN AND SO MAYBE WE'LL 1820 00:57:28,344 --> 00:57:30,180 SEE REPEAT OF THAT HERE, 1821 00:57:30,246 --> 00:57:30,947 AS WELL. 1822 00:57:31,014 --> 00:57:33,349 AND I CALL THAT ONE OUT 1823 00:57:33,416 --> 00:57:35,051 BECAUSE I HELPED ORGANIZE 1824 00:57:35,118 --> 00:57:37,287 CLINICAL NATURAL LANGUAGE 1825 00:57:37,353 --> 00:57:40,056 WORKSHOP AT ACL VENUE, 1826 00:57:40,123 --> 00:57:41,224 WHICH IS DIFFERENT FROM 1827 00:57:41,291 --> 00:57:43,159 B 1828 00:57:43,226 --> 00:57:47,030 BIONLP WORKSHOP ALONG WHAT 1829 00:57:47,096 --> 00:57:49,232 TEXT DATA ARE YOU LOOKING 1830 00:57:49,299 --> 00:57:51,067 AT AND SOME DAY MAYBE 1831 00:57:51,134 --> 00:57:53,970 THEY'LL BE THE SAME 1832 00:57:54,037 --> 00:57:55,305 BECAUSE HOW MUCH DATA YOU 1833 00:57:55,371 --> 00:57:56,873 HAVE RATHER THAN SPECIFIC 1834 00:57:56,940 --> 00:57:58,541 TYPES OF DATA. 1835 00:57:58,608 --> 00:58:01,344 >> SO IT IS A LOT UP TO 1836 00:58:01,411 --> 00:58:03,079 HUMAN IMAGINATION HOW WE 1837 00:58:03,146 --> 00:58:05,048 CAN USE THESE MODELS AND 1838 00:58:05,114 --> 00:58:05,582 STACK THEM. 1839 00:58:05,648 --> 00:58:07,684 IT IS TRULY INCREDIBLE 1840 00:58:07,750 --> 00:58:09,719 SOMEBODY CAN GET ALL 1841 00:58:09,786 --> 00:58:12,255 FIGURES AND FROM THE PALM 1842 00:58:12,322 --> 00:58:14,591 MED CENTRAL PAPERS AND USE 1843 00:58:14,657 --> 00:58:16,426 IN SUCH WAY TO ELEVATE THE 1844 00:58:16,492 --> 00:58:18,528 MODEL TO LEVEL THAT WE SEE 1845 00:58:18,595 --> 00:58:18,795 TODAY. 1846 00:58:18,862 --> 00:58:21,965 WELL, WE HAVE COUPLE MORE 1847 00:58:22,031 --> 00:58:23,666 MINUTES, IF I COULD ASK MY 1848 00:58:23,733 --> 00:58:24,200 QUESTION. 1849 00:58:24,267 --> 00:58:25,535 I HAVE QUESTION ABOUT THE 1850 00:58:25,602 --> 00:58:25,768 PROMPTS. 1851 00:58:25,835 --> 00:58:27,670 IT IS INTERESTING, I GOT 1852 00:58:27,737 --> 00:58:28,671 INTERESTING PERSPECTIVE 1853 00:58:28,738 --> 00:58:30,874 FROM YOUR TALK THAT 1854 00:58:30,940 --> 00:58:33,009 REPROMPTING IS A GOOD 1855 00:58:33,076 --> 00:58:33,276 TECHNIQUE. 1856 00:58:33,343 --> 00:58:36,246 SO IF DO YOU HAVE ANY 1857 00:58:36,312 --> 00:58:36,946 PRACTICAL SUGGESTIONS OF 1858 00:58:37,013 --> 00:58:40,683 HOW TO PROMPT OR WHAT IS 1859 00:58:40,750 --> 00:58:42,118 THE BEST WAY OF PROMPTING 1860 00:58:42,185 --> 00:58:43,586 THAT HAS WORKED FOR YOU? 1861 00:58:43,653 --> 00:58:47,090 >> YEAH, SO THIS IS ALSO 1862 00:58:47,156 --> 00:58:47,590 PRETTY OPEN. 1863 00:58:47,657 --> 00:58:49,993 I THINK MED PROMPT WORK 1864 00:58:50,059 --> 00:58:53,263 AND THERE IS CORRESPONDING 1865 00:58:53,329 --> 00:58:55,265 RESOURCE FOR MED PROMPT 1866 00:58:55,331 --> 00:58:57,000 WORK CALLED PROMPT BASE ON 1867 00:58:57,066 --> 00:58:58,935 GITHUB THAT PROVIDES CODE 1868 00:58:59,002 --> 00:59:01,170 TO RUN SOME SAME 1869 00:59:01,237 --> 00:59:02,639 REPROMPTING EXERCISES. 1870 00:59:02,705 --> 00:59:05,642 I THINK IN SOME WAYS, SOME 1871 00:59:05,708 --> 00:59:09,145 OF THE IT DEPENDING ON 1872 00:59:09,212 --> 00:59:10,313 YOUR DOWN-STREAM TASK, 1873 00:59:10,380 --> 00:59:12,415 WHAT ARE YOU ASKING IT TO 1874 00:59:12,482 --> 00:59:13,750 DO, FOR CLASSIFICATION 1875 00:59:13,816 --> 00:59:16,252 TYPE PROBLEMS OR SOME LIKE 1876 00:59:16,319 --> 00:59:19,222 CLASSIC SORT OF QA STYLE 1877 00:59:19,289 --> 00:59:20,323 PROBLEMS, ONE THING 1878 00:59:20,390 --> 00:59:24,494 HELPFUL WAS THIS LIKE KNN 1879 00:59:24,560 --> 00:59:26,329 FEW SHOT PROMPTING AND BY 1880 00:59:26,396 --> 00:59:27,630 THAT I MEAN, IF YOU KNOW 1881 00:59:27,697 --> 00:59:28,731 WHAT THE UNDERLYING 1882 00:59:28,798 --> 00:59:30,400 QUESTION IS GOING TO BE 1883 00:59:30,466 --> 00:59:31,401 AND YOU'RE ABLE TO PROVIDE 1884 00:59:31,467 --> 00:59:33,102 SOME NOTION OF SIMILARITY 1885 00:59:33,169 --> 00:59:35,705 OF OTHER LIKE TWO OTHER 1886 00:59:35,772 --> 00:59:38,875 QUESTIONS, CAN YOU SORT OF 1887 00:59:38,942 --> 00:59:40,610 INTELLIGENTLY SELECT WHAT 1888 00:59:40,677 --> 00:59:42,178 QUESTIONS YOU PROVIDE IN 1889 00:59:42,245 --> 00:59:43,680 CONTEXT IN POSITIVE 1890 00:59:43,746 --> 00:59:44,814 EXAMPLE AND NEGATIVE 1891 00:59:44,881 --> 00:59:45,615 EXAMPLE IF THAT IS THE 1892 00:59:45,682 --> 00:59:47,216 TYPE OF CLASSIFICATION YOU 1893 00:59:47,283 --> 00:59:48,151 ARE DOING AND PROVIDE 1894 00:59:48,217 --> 00:59:49,686 THOSE AS CONTEXT SO YOU 1895 00:59:49,752 --> 00:59:55,658 CAN HELP ELIS EIGHT THE 1896 00:59:55,725 --> 00:59:56,526 POSITIVE AND NEGATIVE 1897 00:59:56,592 --> 00:59:58,594 EXAMPLE AND THAT CONTEXT 1898 00:59:58,661 --> 00:59:59,462 AND GUIDE PROMPT TO 1899 00:59:59,529 --> 01:00:01,030 PROVIDE RIGHT ANSWER FOR 1900 01:00:01,097 --> 01:00:01,764 SPECIFIC QUESTION YOU 1901 01:00:01,831 --> 01:00:01,965 HAVE. 1902 01:00:02,031 --> 01:00:03,199 THERE ARE THINGS LIKE THAT 1903 01:00:03,266 --> 01:00:04,934 THAT ARE INTERESTING AND 1904 01:00:05,001 --> 01:00:08,271 ALSO LIKE FOR SELF- 1905 01:00:08,338 --> 01:00:10,807 SELF-VERIFICATION PIECE, 1906 01:00:10,873 --> 01:00:11,641 NOEGSZ NOTION I DISCUSS 1907 01:00:11,708 --> 01:00:13,710 AROUND DOING MANY TIME 1908 01:00:13,776 --> 01:00:14,277 REPROMPTING AND 1909 01:00:14,344 --> 01:00:17,480 POTENTIALLY WITH DIFFERENT 1910 01:00:17,547 --> 01:00:19,515 AGENTS THAT HAVE DIFFERENT 1911 01:00:19,582 --> 01:00:21,184 CONTEXTS CAN BE 1912 01:00:21,250 --> 01:00:22,518 INTERESTING IN IDENTIFYING 1913 01:00:22,585 --> 01:00:23,519 IS THIS THING ACTUALLY 1914 01:00:23,586 --> 01:00:25,822 REAL OR IS THIS 1915 01:00:25,888 --> 01:00:26,589 HALLUCINATION BECAUSE 1916 01:00:26,656 --> 01:00:28,691 SEVERAL OF THE METHODS 1917 01:00:28,758 --> 01:00:31,794 SEEM TO BE VERY GOOD AT 1918 01:00:31,861 --> 01:00:32,829 SELF-DETECTING 1919 01:00:32,895 --> 01:00:33,162 HALLUCINATION. 1920 01:00:33,229 --> 01:00:34,964 >> I PERSONALLY FIND 1921 01:00:35,031 --> 01:00:36,466 PROMPTING INTRIGUING, IT 1922 01:00:36,532 --> 01:00:38,001 MAKES YOU ACTUALLY THINK 1923 01:00:38,067 --> 01:00:40,036 ABOUT HOW HUMANS THINK. 1924 01:00:40,103 --> 01:00:42,605 IF YOU CAN EXPLAIN TO 1925 01:00:42,672 --> 01:00:44,307 YOURSELF WHAT IS THE 1926 01:00:44,374 --> 01:00:45,441 THOUGHT PROCESS YOU GO 1927 01:00:45,508 --> 01:00:46,542 THROUGH TO MAKE A 1928 01:00:46,609 --> 01:00:48,344 DECISION, WHICH VERY 1929 01:00:48,411 --> 01:00:49,779 FREQUENTLY WE DO 1930 01:00:49,846 --> 01:00:52,248 INTUITIVELY, ARE TWO 1931 01:00:52,315 --> 01:00:53,783 SENTENCES RELATED? 1932 01:00:53,850 --> 01:00:55,351 DECIDE WHETHER THEY ARE 1933 01:00:55,418 --> 01:00:57,353 RELATED OR NOT, THIS WILL 1934 01:00:57,420 --> 01:01:01,724 NOT, WORK WITH CHAT GPT, 1935 01:01:01,791 --> 01:01:02,892 GIVE EXAMPLE HOW TO JUDGE 1936 01:01:02,959 --> 01:01:06,295 TWO SENTENCES AND HELPFUL 1937 01:01:06,362 --> 01:01:07,363 TO UNDERSTAND HOW WE THINK 1938 01:01:07,430 --> 01:01:09,065 ABOUT PROBLEMS AND 1939 01:01:09,132 --> 01:01:09,999 NORMALIZE HUMAN THINKING 1940 01:01:10,066 --> 01:01:10,833 PROCESS AND PROVIDE THIS 1941 01:01:10,900 --> 01:01:13,336 AS DIRECTIONS TO LLMS. 1942 01:01:13,403 --> 01:01:14,470 >> YEAH. 1943 01:01:14,537 --> 01:01:15,772 TALK ABOUT THIS THIS TIME, 1944 01:01:15,838 --> 01:01:17,373 THERE IS WHOLE VARIETY OF 1945 01:01:17,440 --> 01:01:20,443 WORK IN RETRIEVAL 1946 01:01:20,510 --> 01:01:22,412 AUGMENTED GENERATION, 1947 01:01:22,478 --> 01:01:23,746 CONSTRAINED DECODING COME 1948 01:01:23,813 --> 01:01:25,948 FROM LIKE CODE CREATION 1949 01:01:26,015 --> 01:01:26,582 LITERATURE, BOTH OF WHICH 1950 01:01:26,649 --> 01:01:28,384 ARE FASCINATING FOR HOW 1951 01:01:28,451 --> 01:01:30,787 CAN WE ALSO NOT JUST TUNE 1952 01:01:30,853 --> 01:01:33,189 PROMPT SIDE ALTER OUTPUT 1953 01:01:33,256 --> 01:01:34,190 SIDE, AS WELL AND COME UP 1954 01:01:34,257 --> 01:01:37,693 WITH USEFUL RESPONSES. 1955 01:01:37,760 --> 01:01:39,328 SGLL YEAH, UNFORTUNATELY 1956 01:01:39,395 --> 01:01:41,798 IT IS 3 O'CLOCK AND I HAVE 1957 01:01:41,864 --> 01:01:42,632 TO LET YOU GO. 1958 01:01:42,698 --> 01:01:44,100 THANK YOU SO VERY MUCH, 1959 01:01:44,167 --> 01:01:45,735 THIS IS VERY INTERESTING, 1960 01:01:45,802 --> 01:01:48,071 THIS TALK IS RECORDED AND 1961 01:01:48,137 --> 01:01:51,007 WHOEVER DIDN'T HAVE A 1962 01:01:51,074 --> 01:01:52,008 CHANCE WE'LL BE SHARING 1963 01:01:52,075 --> 01:01:53,342 THIS LINK AND I'M SURE 1964 01:01:53,409 --> 01:01:54,644 I'LL COME BACK TO THIS 1965 01:01:54,710 --> 01:01:59,015 LINK FOR THIS PRESENTA 1966 01:01:59,082 --> 01:01:59,348 PRESENTATION. 1967 01:01:59,415 --> 01:02:00,183 THANK YOU SO MUCH, THIS 1968 01:02:00,249 --> 01:02:01,150 WAS WONDERFUL, THANK YOU 1969 01:02:01,217 --> 01:02:02,018 FOR YOUR TIME. 1970 01:02:02,085 --> 01:02:03,519 >> THANK YOU AGAIN FOR 1971 01:02:03,586 --> 01:02:04,253 HAVING ME. 1972 01:02:04,320 --> 01:02:05,154 >> OKAY, BYE.