1 00:00:08,957 --> 00:00:11,159 WELCOME TO THE CLINICAL CENTER GRAND ROUNDS, 2 00:00:11,159 --> 00:00:14,963 A WEEKLY SERIES OF EDUCATIONAL LECTURES FOR PHYSICIANS AND 3 00:00:14,963 --> 00:00:17,599 HEALTH CARE PROFESSIONALS BROADCAST FROM THE CLINICAL 4 00:00:17,599 --> 00:00:20,568 CENTER AT THE NATIONAL INSTITUTES OF HEALTH IN 5 00:00:20,568 --> 00:00:22,370 BETHESDA, MD. 6 00:00:22,370 --> 00:00:25,907 THE NIH CLINICAL CENTER IS THE WORLD'S LARGEST HOSPITAL TOTALLY 7 00:00:25,907 --> 00:00:29,611 DEDICATED TO INVESTIGATIONAL RESEARCH AND LEADS THE GLOBAL 8 00:00:29,611 --> 00:00:32,547 EFFORT IN TRAINING TODAY'S INVESTIGATORS AND DISCOVERING 9 00:00:32,547 --> 00:00:34,716 TOMORROW'S CURES. 10 00:00:34,716 --> 00:00:44,443 LEARN MORE BY VISITING US ONLINE AT HTTP://CLINICALCENTER.NIH.GOV 11 00:00:44,443 --> 00:00:46,078 GOOD AFTERNOON AND WELCOME TO 12 00:00:46,078 --> 00:00:47,713 TODAY'S CLINICAL CENTER GRAND 13 00:00:47,713 --> 00:00:48,848 ROUNDS. 14 00:00:48,848 --> 00:00:50,116 THE HOPKINS CME ACTIVITY CODE 15 00:00:50,116 --> 00:00:52,985 FOR TODAY'S GRAND ROUNDS IS 16 00:00:52,985 --> 00:00:53,252 57875. 17 00:00:53,252 --> 00:00:55,488 PLEASE TEXT THIS CODE TO JOHNS 18 00:00:55,488 --> 00:00:57,890 HOPKINS CME PHONE NUMBER ON THE 19 00:00:57,890 --> 00:01:00,226 SLIDE TO RECEIVE CME CREDIT FOR 20 00:01:00,226 --> 00:01:00,560 THIS LECTURE. 21 00:01:00,560 --> 00:01:02,929 WE KIND YOU ASK YOU TO PROVIDE 22 00:01:02,929 --> 00:01:05,264 FEEDBACK TO TODAY'S LECTURE 23 00:01:05,264 --> 00:01:06,999 USING THE QR CODEOT SLIDE. 24 00:01:06,999 --> 00:01:09,502 YOU WILL RECEIVE A SURVEY LINK 25 00:01:09,502 --> 00:01:11,470 VIA E-MAIL, AND IT WILL BE 26 00:01:11,470 --> 00:01:12,905 PROVIDED WITH IMPORTANT FEEDBACK 27 00:01:12,905 --> 00:01:14,907 ABOUT THIS PRESENTATION AND 28 00:01:14,907 --> 00:01:16,976 ALLOWS TO SUBMIT QUESTIONS FOR 29 00:01:16,976 --> 00:01:20,580 FUTURE GRAND ROUND TOPICS. 30 00:01:20,580 --> 00:01:30,022 FOLLOWING THE QUESTIONS FROM THE 31 00:01:30,022 --> 00:01:31,724 SPEAKERS, WE CAN ALSO TAKE WEB 32 00:01:31,724 --> 00:01:33,426 CONNECTEDS VIA THE LIVE FEEDBACK 33 00:01:33,426 --> 00:01:35,728 BUTTON ON THE VIDEOCAST. 34 00:01:35,728 --> 00:01:36,862 QUESTIONS WILL BE TAKEN AS TIME 35 00:01:36,862 --> 00:01:38,564 PERMITS AT THE CONCLUSION OF THE 36 00:01:38,564 --> 00:01:39,799 PRESENTATION EMPLOY ITED WE ARE 37 00:01:39,799 --> 00:01:42,134 HONORED TO HAVE DR. HARRY 38 00:01:42,134 --> 00:01:43,169 BCRUCIATE LIGAMENT RKE, PATIENT 39 00:01:43,169 --> 00:01:44,804 SAFETY AND QUALITY ACADEMIC 40 00:01:44,804 --> 00:01:46,172 COLLABORATIVE AND CHIEF IN THE 41 00:01:46,172 --> 00:01:47,173 SECTION OF SAFETY, QUALITY AND 42 00:01:47,173 --> 00:01:49,475 VALUE IN THE DEPARTMENT OF 43 00:01:49,475 --> 00:01:54,614 MEDICINE AT THE F. EDWARD ABER 44 00:01:54,614 --> 00:01:56,949 SCHOOL OF MEDICINE UNIVERSITY OF 45 00:01:56,949 --> 00:02:00,186 HEALTH SCIENCES. 46 00:02:00,186 --> 00:02:02,221 DR. BURKE EARNED HIS 47 00:02:02,221 --> 00:02:03,055 UNDERGRADUATE, DOCTORAL AND 48 00:02:03,055 --> 00:02:05,057 MEDICAL DEGREES AT THE 49 00:02:05,057 --> 00:02:06,125 UNIVERSITY OF OHIO. 50 00:02:06,125 --> 00:02:08,928 HE IS NOW PROFESSOR OF MEDICINE 51 00:02:08,928 --> 00:02:10,563 AND DATA SCIENCE. 52 00:02:10,563 --> 00:02:14,634 IN 1992 DR. BURKE BEGAN HIS 53 00:02:14,634 --> 00:02:15,234 FOUNDATION OF IMPLEMENTING 54 00:02:15,234 --> 00:02:16,569 ARTIFICIAL INTEL JOHNSON 55 00:02:16,569 --> 00:02:17,937 NETWORKS WHILE THE ASSOCIATE 56 00:02:17,937 --> 00:02:19,972 DIRECTOR OF THE CENTER FOR 57 00:02:19,972 --> 00:02:22,041 BIOMEDICAL RESEARCH AT THE 58 00:02:22,041 --> 00:02:23,643 UNIVERSITY OF NEVADA RENO. 59 00:02:23,643 --> 00:02:25,544 IN 19 HE WAS 1 OF THE FIRST TO 60 00:02:25,544 --> 00:02:27,847 PUBLISH ON USING ARTIFICIAL 61 00:02:27,847 --> 00:02:29,048 NEURAL NETWORKS IN CANCER. 62 00:02:29,048 --> 00:02:32,018 IN 1995, HE DEMONSTRATE THAD 63 00:02:32,018 --> 00:02:33,686 ARTIFICIAL NEURAL NETWORKS WERE 64 00:02:33,686 --> 00:02:34,487 MORE POWERFUL THAN 65 00:02:34,487 --> 00:02:36,022 PREDECLARATION METHODS IN 66 00:02:36,022 --> 00:02:36,255 CANCER. 67 00:02:36,255 --> 00:02:37,857 IN ADDITION TO HIS WORK IN 68 00:02:37,857 --> 00:02:39,292 ARTIFICIAL NEURAL METE YORKS FOR 69 00:02:39,292 --> 00:02:41,861 THE PAST 6 YEARS, HE HAS BEEN 70 00:02:41,861 --> 00:02:44,130 DEVELOPING NATURAL LANGUAGE 71 00:02:44,130 --> 00:02:45,298 PROCESSING PROGRAMS, HIS 72 00:02:45,298 --> 00:02:46,299 INTEREST IN ARTIFICIAL 73 00:02:46,299 --> 00:02:48,834 INTELIENCE AND IN NATURAL 74 00:02:48,834 --> 00:02:50,202 LANGUAGE PROCESSING HAS COME 75 00:02:50,202 --> 00:02:52,271 TOGETHER IN GENERATIVE 76 00:02:52,271 --> 00:02:53,339 ARTIFICIAL INTELLIGENCE LARGE 77 00:02:53,339 --> 00:02:55,508 LANGUAGE MODELS. 78 00:02:55,508 --> 00:02:56,876 DR. BURKE HAS WRITTEN NUMEROUS 79 00:02:56,876 --> 00:02:59,211 BOOK KRAPTERS IN OVER 80 PEER 80 00:02:59,211 --> 00:03:00,046 REVIEWED SCIENTIFIC PAPERS. 81 00:03:00,046 --> 00:03:03,349 HE IS A REVIEWER FOR NUMEROUS 82 00:03:03,349 --> 00:03:05,151 HIGH IMPACT JOURNALS AND MEMBER 83 00:03:05,151 --> 00:03:07,019 OF THE EDITORIAL BOARD FOR THE 84 00:03:07,019 --> 00:03:08,354 JOURNAL OF NATIONAL CANCER 85 00:03:08,354 --> 00:03:09,622 INSTITUTE AND NATIONAL CANCER 86 00:03:09,622 --> 00:03:11,257 SOCIETY'S JOWRN OF OF CANCER AND 87 00:03:11,257 --> 00:03:13,159 RECEIVED AN AWARD FOR BEING A 88 00:03:13,159 --> 00:03:14,527 DISTINGUISHED REVIEWER BY THE 89 00:03:14,527 --> 00:03:16,429 JOURNAL OF AMERICAN MEDICAL 90 00:03:16,429 --> 00:03:16,896 ASSOCIATION ONCOLOGY. 91 00:03:16,896 --> 00:03:20,966 THE TITLE OF HIS PRESENTATION 92 00:03:20,966 --> 00:03:22,401 TODAY IS GENERATIVE ARTIFICIAL 93 00:03:22,401 --> 00:03:24,236 INTELIENCE AND THE FUTURE OF 94 00:03:24,236 --> 00:03:24,503 MEDICINE. 95 00:03:24,503 --> 00:03:26,939 NOW JOIN ME IN WELCOMING OUR 96 00:03:26,939 --> 00:03:30,376 GRABBED ROUNDS SPEAKER DR. HARRY 97 00:03:30,376 --> 00:03:30,576 BURKE. 98 00:03:30,576 --> 00:03:32,044 [ APPLAUSE ] 99 00:03:32,044 --> 00:03:32,645 >> THANK YOU DOCTOR. 100 00:03:32,645 --> 00:03:32,845 OKAY. 101 00:03:32,845 --> 00:03:35,181 SO WE'RE GOING TO JUST DIVE 102 00:03:35,181 --> 00:03:37,450 RIGHT INTO THIS THING, IT'S A 103 00:03:37,450 --> 00:03:38,050 PRETTY PRETENSION TITLE BUT I 104 00:03:38,050 --> 00:03:39,452 THOUGHT IT WAS A LOT OF FUN. 105 00:03:39,452 --> 00:03:45,658 I HAVE THE USUAL DISCLOSURES TO 106 00:03:45,658 --> 00:03:47,026 DECLARE, SO, NOTHING VERY 107 00:03:47,026 --> 00:03:47,460 INTERESTING THERE. 108 00:03:47,460 --> 00:03:49,195 SO WE WILL TART BY LOOKING INTO 109 00:03:49,195 --> 00:03:50,262 THE FUTURE A SECOND AND I WILL 110 00:03:50,262 --> 00:03:52,131 ASK YOU A QUESTION. 111 00:03:52,131 --> 00:03:57,169 WHICH OF THESE 3 FUTURES, DO YOU 112 00:03:57,169 --> 00:03:58,637 THINK THAT MEDICINE WILL BE? 113 00:03:58,637 --> 00:04:00,473 DO YOU THINK THAT WE'RE GOING TO 114 00:04:00,473 --> 00:04:03,209 IGNORE AI AND HOPE FOR THE BEST? 115 00:04:03,209 --> 00:04:04,243 THAT BOTHLET PHYSICIAN AND THE 116 00:04:04,243 --> 00:04:06,078 PATIENTS ARE GOING TO USE AI? 117 00:04:06,078 --> 00:04:08,748 OR THAT THE PATIENT'S GOING TO 118 00:04:08,748 --> 00:04:10,883 INTERACT DIRECTLY WITH AI? 119 00:04:10,883 --> 00:04:12,952 NOW THERE ARE OTHER CHOICES BUT 120 00:04:12,952 --> 00:04:14,653 THESE ARE 3, SO JUST FOR A SHOW 121 00:04:14,653 --> 00:04:16,288 OF HANDS, HOW MANY OF YOU 122 00:04:16,288 --> 00:04:17,857 BELIEVE NUMBER 1? 123 00:04:17,857 --> 00:04:19,358 ANYBODY BELIEVE THAT. 124 00:04:19,358 --> 00:04:21,494 OKAY, SO NUMBER 2, THE PHYSICIAN 125 00:04:21,494 --> 00:04:23,396 AND THE PATIENT ARE BOTH GOING 126 00:04:23,396 --> 00:04:26,699 TO USE AI? 127 00:04:26,699 --> 00:04:28,000 OKAY, AND NUMBER 3, THE 128 00:04:28,000 --> 00:04:30,503 PATIENT'S GOING TO COMMUNICATE 129 00:04:30,503 --> 00:04:31,804 DIRECTLY WITH AI? 130 00:04:31,804 --> 00:04:32,104 OKAY. 131 00:04:32,104 --> 00:04:34,807 OKAY, GOOD, SO IT'S NUMBER 2, 132 00:04:34,807 --> 00:04:35,841 NOW THERE'S 1 LITTLE ISSUE 133 00:04:35,841 --> 00:04:38,010 THERE, THAT IS YOU SHOULD NOTICE 134 00:04:38,010 --> 00:04:40,813 THAT THESE ARE 2 DIFFERENT 135 00:04:40,813 --> 00:04:41,580 SYSTEMS. 136 00:04:41,580 --> 00:04:43,182 MEDICAL AI WILL BE DIFFERENT 137 00:04:43,182 --> 00:04:44,450 THAN A GENERAL AI SYSTEM. 138 00:04:44,450 --> 00:04:45,818 SO 1 OF THE QUESTIONS THAT'S 139 00:04:45,818 --> 00:04:48,554 GOING TO COME UP IN THIS 140 00:04:48,554 --> 00:04:50,956 SCENARIO IS THAT THEY DON'T 141 00:04:50,956 --> 00:04:51,323 MATCHUP. 142 00:04:51,323 --> 00:04:53,092 OKAY, SO THE SYSTEM YOU 143 00:04:53,092 --> 00:04:54,226 DESCRIBED ISN'T GOING TO 144 00:04:54,226 --> 00:04:57,229 MATCHUP, OKAY, SO WHAT IS 145 00:04:57,229 --> 00:04:57,630 GENERAL? 146 00:04:57,630 --> 00:04:59,632 SO I WILL GO QUICK 3 THROUGH 147 00:04:59,632 --> 00:05:00,566 THIS. 148 00:05:00,566 --> 00:05:02,435 SO ARTIFICIAL INTELLIGENCE 149 00:05:02,435 --> 00:05:04,336 EMULATES HUMAN INTELLECTUAL 150 00:05:04,336 --> 00:05:06,238 CAPABILITIES AND THERE THERE'S 151 00:05:06,238 --> 00:05:08,741 SYMBOLIC AI AND STATISTICAL AI 152 00:05:08,741 --> 00:05:11,677 AND AI SYSTEMS INTERNALLY 153 00:05:11,677 --> 00:05:12,812 MANIPULATE SIM BOOM BOOM BOOM 154 00:05:12,812 --> 00:05:14,480 BOOMS AND WILL COME AROUND ANDY 155 00:05:14,480 --> 00:05:16,215 WOO WILL TALK ABOUT STATISTICAL 156 00:05:16,215 --> 00:05:19,552 AI BECAUSE THAT'S WHAT A GPT AND 157 00:05:19,552 --> 00:05:21,654 NLM AND A GEN AI IS. 158 00:05:21,654 --> 00:05:26,392 SO JUST REALLY QUICKLY, GEN AI 159 00:05:26,392 --> 00:05:28,294 IS A GENERATIVE MODEL SO WE 160 00:05:28,294 --> 00:05:30,329 DIDN'T HAVE GENERATIVE MODELS 161 00:05:30,329 --> 00:05:31,664 BEFORE, AND SO THE BIG 162 00:05:31,664 --> 00:05:33,666 REVOLUTION IS THAT IT COULD 163 00:05:33,666 --> 00:05:37,436 ACCEPT AND PRODUCE TAXED MUSIC, 164 00:05:37,436 --> 00:05:40,473 COMPUTERS, STUFF, AND GENERATIVE 165 00:05:40,473 --> 00:05:41,006 PRETRAINED TRANSFORMER WAS 166 00:05:41,006 --> 00:05:44,710 CREATED BY GOOGLE AND THAT'S 167 00:05:44,710 --> 00:05:47,213 THESE PEOPLE HERE AND IT WAS 168 00:05:47,213 --> 00:05:50,416 POPULARIZED BY OPEN AI, PRETRAIN 169 00:05:50,416 --> 00:05:52,585 MEANS LOTS OF AMOUNTS OF DATA, 170 00:05:52,585 --> 00:05:54,220 TRANSFORMER IS THE ALGORITHM 171 00:05:54,220 --> 00:05:56,889 THEY DESCRIBE HERE, AND LARGE 172 00:05:56,889 --> 00:05:58,190 LANGUAGE MODELS ARE GEN AI 173 00:05:58,190 --> 00:06:01,026 PROGRAMS AND SO IN FACT, YOU CAN 174 00:06:01,026 --> 00:06:03,729 USE THEM ISHT CHANGEABLY, AI AND 175 00:06:03,729 --> 00:06:05,064 LARGE LANGUAGE MODELS UNLESS 176 00:06:05,064 --> 00:06:06,966 YOU'RE TALKING ABOUT VLM, WHICH 177 00:06:06,966 --> 00:06:09,134 IS A VISUAL LANGUAGE MODEL WHICH 178 00:06:09,134 --> 00:06:09,802 IS GEN AI. 179 00:06:09,802 --> 00:06:11,570 OKAY, AND THERE ARE MANY 180 00:06:11,570 --> 00:06:15,274 TRANSFORMER LIKE PROGRAMS OUT 181 00:06:15,274 --> 00:06:17,510 THERE, A NUMBER OF PUBLICATIONS 182 00:06:17,510 --> 00:06:19,545 LOOKING AT GEN AI, ARTIFICIAL 183 00:06:19,545 --> 00:06:26,952 INTELLIGENCE AS YOU CAN SEE, 184 00:06:26,952 --> 00:06:30,389 RIGHT AROUND HERE WHEN VASWANI 185 00:06:30,389 --> 00:06:32,958 PUBLISHED A PAPER, THERE'S BEEN 186 00:06:32,958 --> 00:06:34,093 AN EXPOTENTIAL INCREASE, WE 187 00:06:34,093 --> 00:06:35,494 STARTED WAY BACK OVER HERE WHEN 188 00:06:35,494 --> 00:06:36,795 IT WAS A LITTLE TINY THING AND 189 00:06:36,795 --> 00:06:40,633 NOW IT'S A HUGE THING. 190 00:06:40,633 --> 00:06:42,535 OKAY, SO WHAT DOES LLM RELY 191 00:06:42,535 --> 00:06:44,703 UPON, I WANT TO GET THIS OUT IN 192 00:06:44,703 --> 00:06:46,705 THE BEGINNING, IT HAS 3 POWER 193 00:06:46,705 --> 00:06:48,474 IDEAS, IT HAS SEQUENCE. 194 00:06:48,474 --> 00:06:49,341 WORDS IN A SEQUENCE ARE 195 00:06:49,341 --> 00:06:50,776 ASSOCIATE WIDE A MEANING. 196 00:06:50,776 --> 00:06:52,011 PRETTY STRAIGHT FORWARD. 197 00:06:52,011 --> 00:06:53,379 TWO, REPETITION, MORE OFTEN THE 198 00:06:53,379 --> 00:06:55,247 SEQUENCE OCCURS IN THE DATA THE 199 00:06:55,247 --> 00:06:56,649 MORE LIKELY SAID TO BE THE 200 00:06:56,649 --> 00:06:58,884 CORRECT MEANING AND 3 201 00:06:58,884 --> 00:07:01,020 SELF-ORGANIZATION RELIES ON THE 202 00:07:01,020 --> 00:07:05,291 NEURAL NETTO BE AUTONOMOUS 203 00:07:05,291 --> 00:07:05,658 SELF-ORGANIZING. 204 00:07:05,658 --> 00:07:06,725 NEURAL NET. 205 00:07:06,725 --> 00:07:07,726 OKAY, NOW. 206 00:07:07,726 --> 00:07:15,834 MOST PEOPLE THINK WHEN THEY 207 00:07:15,834 --> 00:07:17,336 THINK ABOUT-ABOUT AI AND WE USE 208 00:07:17,336 --> 00:07:19,905 THAT IN OUR STATISTICS, WE USE 209 00:07:19,905 --> 00:07:24,109 LOGISTIC REGRESSION, WE ARE 210 00:07:24,109 --> 00:07:25,411 USING CONDITIONAL PROBABILITY, 211 00:07:25,411 --> 00:07:26,412 RIGHT? 212 00:07:26,412 --> 00:07:28,047 THE PROBABILITY OF WHY GIVE IT 213 00:07:28,047 --> 00:07:28,180 X. 214 00:07:28,180 --> 00:07:32,151 BUT WHAT YOU HAVE TO UNDERSTAND 215 00:07:32,151 --> 00:07:34,820 IN GENAI IS THAT THERE'S 2 216 00:07:34,820 --> 00:07:37,022 PROBABILITIES, THERE'S THE JOIPT 217 00:07:37,022 --> 00:07:37,823 PROBABILITY AND CONDITIONAL 218 00:07:37,823 --> 00:07:39,358 PROBABILITY, SO WE WILL TALK 219 00:07:39,358 --> 00:07:42,728 ABOUT LATER THE, THE JOINT 220 00:07:42,728 --> 00:07:44,129 PROBABILITY IS THE ENGINE FOR 221 00:07:44,129 --> 00:07:46,098 LARGE LANGUAGE MODEL AND THE 222 00:07:46,098 --> 00:07:47,132 CONDITIONAL PROBABILITY IS THE 223 00:07:47,132 --> 00:07:48,767 OUTPUT OF THE MODEL. 224 00:07:48,767 --> 00:07:54,974 THIS THING ISN'T WORKING. 225 00:07:54,974 --> 00:07:56,575 AH, OKAY, SO, SO PEOPLE THINK 226 00:07:56,575 --> 00:07:58,811 THAT THERE'S A COUPLE LLMs OUT 227 00:07:58,811 --> 00:08:05,117 THERE, THAT'S NOT THE CASE. 228 00:08:05,117 --> 00:08:06,819 AFTER VIKWANI'S PAPER CAME OUT, 229 00:08:06,819 --> 00:08:08,454 THE LITERATURE BECAME HUGE VERY, 230 00:08:08,454 --> 00:08:09,521 VERY QUICKLY. 231 00:08:09,521 --> 00:08:14,493 THERE ARE HUNDREDS OF LLM MODELS 232 00:08:14,493 --> 00:08:16,095 DEVELOPED AFTER VISWANI'S PAPER 233 00:08:16,095 --> 00:08:17,596 CAME OUT, LITERALLY HUNDREDS OF 234 00:08:17,596 --> 00:08:17,796 THEM. 235 00:08:17,796 --> 00:08:20,633 AND WHAT WE SEE TODAY, IF I CAN 236 00:08:20,633 --> 00:08:25,638 GET THIS THING TO WORK, OKAY, 237 00:08:25,638 --> 00:08:27,473 WHAT AM I, I GOTTA GO BACK. 238 00:08:27,473 --> 00:08:29,308 WHAT WE SEE TODAY ARE THESE ARE 239 00:08:29,308 --> 00:08:31,510 THE BIG MONEY 1S, THESE ARE THE 240 00:08:31,510 --> 00:08:33,078 1S THAT WHERE THERE'S HUNDREDS 241 00:08:33,078 --> 00:08:37,149 OF MILLIONS OF DOLLARS BEHIND 242 00:08:37,149 --> 00:08:39,485 TTHIS WAS THE START OF THE 243 00:08:39,485 --> 00:08:40,419 PAPER, AND WHAT'S INTERESTING IS 244 00:08:40,419 --> 00:08:44,857 THAT ALL OF THESE PLAYERS, OKAY, 245 00:08:44,857 --> 00:08:46,692 CREATED LARGE LANGUAGE MODELS 246 00:08:46,692 --> 00:08:51,764 AND IN FACT, THE METAMODEL CAME 247 00:08:51,764 --> 00:08:55,801 OUT 14 DAYS BEFORE CHAT GPT 3.5 248 00:08:55,801 --> 00:08:58,771 AND IT WAS WITHDRAWN WITHIN 7 249 00:08:58,771 --> 00:09:02,207 DAYS. 250 00:09:02,207 --> 00:09:03,976 OKAY, AND NOW, OPEN AI CAME OUT 251 00:09:03,976 --> 00:09:06,979 WITH THIS PRODUCT, IT WASN'T THE 252 00:09:06,979 --> 00:09:07,413 FIRST. 253 00:09:07,413 --> 00:09:08,814 TWENTY-TWO OUT OF 22 HAD NO 254 00:09:08,814 --> 00:09:10,449 MONEY. 255 00:09:10,449 --> 00:09:11,350 REVENUE THIS YEAR, 3.7 BILLION. 256 00:09:11,350 --> 00:09:13,819 AS FAR AS I KNOW, NO COMPANY 257 00:09:13,819 --> 00:09:15,788 LESS THAN 2 YEARS OLD OKAY, AND 258 00:09:15,788 --> 00:09:19,058 IT IS STILL LESS THAN 2 YEARS 259 00:09:19,058 --> 00:09:20,759 OLD HAS GENERATED 3.7 BILLION 260 00:09:20,759 --> 00:09:30,302 DOLLARS IN REVENUE, IT IS 261 00:09:30,302 --> 00:09:31,637 ABSOLUTELY AMAZING. 262 00:09:31,637 --> 00:09:32,738 [MUMBLING ] OKAY, SO, WHEN WE 263 00:09:32,738 --> 00:09:34,039 LOOK AT THESE THINGS WE CAN SEE 264 00:09:34,039 --> 00:09:35,340 THE AMOUNT OF MONITOR THAT'S' 265 00:09:35,340 --> 00:09:36,842 BEING SPENT ON THESE THINGS AND 266 00:09:36,842 --> 00:09:39,111 I JUST WANT TO POINT OUT THE 267 00:09:39,111 --> 00:09:41,046 MERCURY PROGRAM THAT SENT A MAN 268 00:09:41,046 --> 00:09:43,449 IN SPACE IN CURRENT DOLLARS THEY 269 00:09:43,449 --> 00:09:44,883 SPENT 44 BILLION IN CURRENT 270 00:09:44,883 --> 00:09:47,286 DOLLARS TO PUT A MAN INTO SPACE, 271 00:09:47,286 --> 00:09:48,787 WE'VE ALREADY SPENT MUCH MORE 272 00:09:48,787 --> 00:09:53,058 THAN IT TOOK TO PUT A MAN IN 273 00:09:53,058 --> 00:09:54,993 SPACE ON LLMs. 274 00:09:54,993 --> 00:09:56,562 LLMs ARE EXPENSIVE, THESE ARE 275 00:09:56,562 --> 00:09:58,564 THE TOKEN RATES, TOKEN RATES ARE 276 00:09:58,564 --> 00:10:00,733 COMING DOWN BUT THEY'RE STILL 277 00:10:00,733 --> 00:10:02,367 EXPENSIVE, THEY CONSUME HUGE 278 00:10:02,367 --> 00:10:04,770 AMOUNTS OF ENERGY, THEY ARE 279 00:10:04,770 --> 00:10:07,740 RESTARTING 3-MILE ISLAND NUCLEAR 280 00:10:07,740 --> 00:10:08,540 REACTOR, MICROSOFT JUST TO PAY 281 00:10:08,540 --> 00:10:10,843 PER SOME OF THE ENERGY USE. 282 00:10:10,843 --> 00:10:14,279 AND THE COST OF CREATING THESE 283 00:10:14,279 --> 00:10:15,948 THINGS IS TREMENDOUS, OKAY, THE 284 00:10:15,948 --> 00:10:18,584 ORIGINAL 1 COSTS A BILLION 285 00:10:18,584 --> 00:10:19,952 DOLLARS TO CREATE, OKAY, IT 286 00:10:19,952 --> 00:10:23,489 STILL COSTS A BILLION DOLLARS TO 287 00:10:23,489 --> 00:10:25,090 CREATE, AND OPEN AI'S COST, THEY 288 00:10:25,090 --> 00:10:27,826 JUST DID A FOUNDING ROUND 289 00:10:27,826 --> 00:10:31,263 BECAUSE THEY'RE DAILY ENERGY 290 00:10:31,263 --> 00:10:33,832 EXPENTINE REGIMENITTURE IS 7 HHV 291 00:10:33,832 --> 00:10:34,967 EXPEBDITTURE IS $700,000 A DAY. 292 00:10:34,967 --> 00:10:39,905 OKAY, SO HOW DO THEY WORK? 293 00:10:39,905 --> 00:10:40,639 THERE'S 3 ASPECTS. 294 00:10:40,639 --> 00:10:42,307 THE HUGE AMOUNT OF DATA, THE 295 00:10:42,307 --> 00:10:44,276 DANZA FORMER AND THE FINE 296 00:10:44,276 --> 00:10:44,510 TUNING. 297 00:10:44,510 --> 00:10:45,310 THE PREPROCESSING IS NECESSARY, 298 00:10:45,310 --> 00:10:47,146 SO YOU HAVE TO DO A LOT OF STUFF 299 00:10:47,146 --> 00:10:48,080 WITH WORDS, YOU HAVE TO 300 00:10:48,080 --> 00:10:49,515 TRUNCATE, YOU HAVE TO DO A WHOLE 301 00:10:49,515 --> 00:10:51,650 BUNCH OF STUFF, BUT THE MAIN 302 00:10:51,650 --> 00:10:57,956 POINT OF THE PREPROCESSING, IS 303 00:10:57,956 --> 00:10:59,358 THIS WORD [INDISCERNIBLE], WORD 304 00:10:59,358 --> 00:11:01,393 [INDISCERNIBLE] WAS AN 305 00:11:01,393 --> 00:11:02,761 ABNORMALITIES SELECT REVOLUTION. 306 00:11:02,761 --> 00:11:04,229 WITHOUT IT NONE OF THIS WOULD 307 00:11:04,229 --> 00:11:05,197 HAVE HAPPENED. 308 00:11:05,197 --> 00:11:06,765 WHAT IT DID IS THIS WORK WORKS 309 00:11:06,765 --> 00:11:09,168 ON VECTORS AND NUMBERS, IF YOU 310 00:11:09,168 --> 00:11:11,170 DIDN'T 1 ABOUT WORD 2 VEC, YOU 311 00:11:11,170 --> 00:11:13,338 COULD NOT CREATE THE SYSTEM, 312 00:11:13,338 --> 00:11:15,374 WITH WORD 2 VEC, YOU CREATE 313 00:11:15,374 --> 00:11:16,575 NUMBERS AND THAT GIVES RISE TO 314 00:11:16,575 --> 00:11:18,043 SOMETHING YOU CAN PUT INTO A 315 00:11:18,043 --> 00:11:19,778 NEURAL NET, YOU CAN'T PUT WORDS 316 00:11:19,778 --> 00:11:22,114 INTO A MURAL NET, YOU CAN ONLY 317 00:11:22,114 --> 00:11:26,718 PUT NUMBERS IN, SO WORD 2 VEC 318 00:11:26,718 --> 00:11:29,822 ALLOWED US TO PUT WORD INTO THE 319 00:11:29,822 --> 00:11:32,791 TRIEWRAL NET. 320 00:11:32,791 --> 00:11:33,325 THE TRANSFORMER HAS 3 321 00:11:33,325 --> 00:11:36,094 COMPONENTS, IT HAS GENERATIVE, 322 00:11:36,094 --> 00:11:37,629 OUTPUT, CONDITIONAL PROBABILITY 323 00:11:37,629 --> 00:11:39,698 AND SELF-ORGANIZING NEURAL NETS, 324 00:11:39,698 --> 00:11:41,333 45 TERRA BYTES OF DATA, THERE'S 325 00:11:41,333 --> 00:11:44,203 NEVER BEEN AS FAR AS I KNOW, I 326 00:11:44,203 --> 00:11:47,539 COULD BE WRONG 45 TB IN 1 327 00:11:47,539 --> 00:11:48,073 INTEGRATED SYSTEM? 328 00:11:48,073 --> 00:11:50,108 , YEAH, THERE HAVE BEEN 329 00:11:50,108 --> 00:11:54,880 DATABASES, LARGE DATABASES BUT 1 330 00:11:54,880 --> 00:11:56,415 INTEGRATED SYSTEM, OKAY? 331 00:11:56,415 --> 00:11:58,817 SO 175 BILLION PARAMETERS, THOSE 332 00:11:58,817 --> 00:12:02,054 ARE JUST THE NEURAL NET 333 00:12:02,054 --> 00:12:04,456 PARAMETERS, 64 BIT FLOATING 334 00:12:04,456 --> 00:12:06,692 POINT DOUBLE PRECISION. 335 00:12:06,692 --> 00:12:07,492 THE PROBABILITIES THAT AM CAN 336 00:12:07,492 --> 00:12:11,663 OUT OF THIS THING ARE VERY, VERY 337 00:12:11,663 --> 00:12:12,631 SMALL. 338 00:12:12,631 --> 00:12:15,400 IT'S LIKE .0000, ALL THE WAY, 1. 339 00:12:15,400 --> 00:12:17,302 OKAY, SO THE POINT IS THIS IS 340 00:12:17,302 --> 00:12:20,339 TRYING TO MAKE DISTINCTIONS ON 341 00:12:20,339 --> 00:12:23,642 VERY SMALL NUMBERS WHICH IS WHY 342 00:12:23,642 --> 00:12:27,846 IT NEEDS 64 BIT FLOATING AND 343 00:12:27,846 --> 00:12:30,649 RUNS ON SERVERS, AND WHY DID 344 00:12:30,649 --> 00:12:36,822 CHAT GPT WIN AND MICROSOFT LOSE 345 00:12:36,822 --> 00:12:37,556 WITH GALACTICCA? 346 00:12:37,556 --> 00:12:38,857 BECAUSE OPEN AI DID SOMETHING 347 00:12:38,857 --> 00:12:39,892 VERY SMART. 348 00:12:39,892 --> 00:12:41,393 THEY DID HUMAN FINE TUNING. 349 00:12:41,393 --> 00:12:43,395 THEY HAD HUNDREDS OF THOUSANDS 350 00:12:43,395 --> 00:12:45,063 OF MAN HOURS OF FINE TUNING, 351 00:12:45,063 --> 00:12:47,399 WHERE THEY JUST TOOK THE SYSTEM, 352 00:12:47,399 --> 00:12:49,468 LOOKED AT THE OUTPUT AND CHANGED 353 00:12:49,468 --> 00:12:51,603 THE OUTPUT IN A WAY THAT MADE 354 00:12:51,603 --> 00:12:53,572 SENSE, SO IT DIDN'T GET THIS 355 00:12:53,572 --> 00:12:57,643 NONSENSE OUTPUT, THAT WAS THE 356 00:12:57,643 --> 00:12:58,243 TRICK. 357 00:12:58,243 --> 00:12:59,177 GALACTICCA WHEN THE OUTPUT CAME 358 00:12:59,177 --> 00:13:00,379 OUT, IT WAS GARBAGE. 359 00:13:00,379 --> 00:13:02,514 SO THE FINE TUNING WAS THE KEY, 360 00:13:02,514 --> 00:13:06,818 BUT MAKE NO MISTAKE, THIS SYSTEM 361 00:13:06,818 --> 00:13:14,026 THAT OPEN AI CAME UP WITH IS A 362 00:13:14,026 --> 00:13:14,693 MONUMENTAL ACHIEVEMENT, 363 00:13:14,693 --> 00:13:14,993 MONUMENTAL. 364 00:13:14,993 --> 00:13:16,728 THAT STUFF WORKS. 365 00:13:16,728 --> 00:13:18,030 OKAY, SO WHAT HAPPENS? 366 00:13:18,030 --> 00:13:20,532 SO REAL QUICK, YOU TAKE YOUR 367 00:13:20,532 --> 00:13:22,634 DOCUMENTS, THEY'RE WORDS, YOU DO 368 00:13:22,634 --> 00:13:25,270 WORD 2 VEC, YOU TURN THEM INTO 369 00:13:25,270 --> 00:13:27,606 VECTORS, YOU PUT THEM IN THE 370 00:13:27,606 --> 00:13:30,042 NEURAL NET, WHICH IS DEEP 371 00:13:30,042 --> 00:13:30,609 ENCODING AND DECODING USING 372 00:13:30,609 --> 00:13:32,744 QUERY KEYS AND VALUES, CAN I 373 00:13:32,744 --> 00:13:35,047 WILL MENTION THAT LATER AND THE 374 00:13:35,047 --> 00:13:36,248 GENERATIVE MODEL, OKAY, AND THEN 375 00:13:36,248 --> 00:13:39,551 YOU GET THE PROBABILITIES, AND 376 00:13:39,551 --> 00:13:42,587 THEN IT'S SELECT, USUALLY THE 377 00:13:42,587 --> 00:13:43,422 HIGHEST PROBABILITY AND USUALLY 378 00:13:43,422 --> 00:13:47,225 IS A VERY GOOD THING. 379 00:13:47,225 --> 00:13:50,629 OKAY, SO ATTENTION, IT'S A VERY 380 00:13:50,629 --> 00:13:52,297 HOLD CONCEPT, IT ENCODES AND 381 00:13:52,297 --> 00:13:53,932 DECODES, THEY DIDN'T COME UP 382 00:13:53,932 --> 00:13:55,467 WITH IT, BUT WHAT THEY DID WAS 383 00:13:55,467 --> 00:13:57,502 THEY CAME UP WITH THIS FORMULA, 384 00:13:57,502 --> 00:13:58,904 MAPPING, QUERY AND A SET OF KEY 385 00:13:58,904 --> 00:14:02,307 VALUE PAIRS TO AN OUTPUT AND SO 386 00:14:02,307 --> 00:14:06,645 THESE 3 CHARACTERISTICS ARE WHT 387 00:14:06,645 --> 00:14:07,779 THE VECTORS USE. 388 00:14:07,779 --> 00:14:09,448 THEY ORGANIZE EVERY VECTOR IN 389 00:14:09,448 --> 00:14:11,083 TERMS OF THESE 3 PARAMETERS, 390 00:14:11,083 --> 00:14:12,851 OKAY IN AND THEN THE OUTPUT IS 391 00:14:12,851 --> 00:14:14,786 COMPUTED AS THE WEIGHTED SUM OF 392 00:14:14,786 --> 00:14:17,356 THOSE VALUES, SO THAT'S HOW THE 393 00:14:17,356 --> 00:14:19,691 ACTUAL THING WORKS, TRANSFORMER 394 00:14:19,691 --> 00:14:20,993 ITSELF, OKAY, IS THE 395 00:14:20,993 --> 00:14:23,495 ARCHITECTURE LOOKING AT THE 396 00:14:23,495 --> 00:14:24,629 GLOBAL DEPENDENCIES, WHAT IT'S 397 00:14:24,629 --> 00:14:25,964 DOING IS USING NEURAL NETS TO 398 00:14:25,964 --> 00:14:27,966 MAP A SEQUENCE OF INPUT TO A 399 00:14:27,966 --> 00:14:31,169 SEQUENCE OF OUTPUT VECTORS, 400 00:14:31,169 --> 00:14:33,739 THAT'S WHAT IT DOES, OKAY, SO WE 401 00:14:33,739 --> 00:14:34,706 IMPLEMENTED NEURAL NETS A LONG 402 00:14:34,706 --> 00:14:36,308 TIME AGO, YOU HEARD ABOUT THAT, 403 00:14:36,308 --> 00:14:37,909 THEY'RE VERY GOOD, AND THE 404 00:14:37,909 --> 00:14:43,682 REASON THEY'RE VERY GOOD AND SO 405 00:14:43,682 --> 00:14:44,750 HERE'S THE 2 CLASS SEPARATION 406 00:14:44,750 --> 00:14:46,485 PROBLEM, AND THE NEURAL NETWILL 407 00:14:46,485 --> 00:14:49,521 DO THE 2 CLASS SEPARATION 408 00:14:49,521 --> 00:14:50,288 PROBLEM BEAUTIFULLY, WHY IN 409 00:14:50,288 --> 00:14:52,657 WELL, THIS IS WHAT ATRACKED ME 410 00:14:52,657 --> 00:14:56,328 TO NEURAL NETS IN EARLY 1990S 411 00:14:56,328 --> 00:14:58,864 WAS THIS THEOR UMKC, IT'S CALLED 412 00:14:58,864 --> 00:14:59,998 THE UNIVERSAL PROXIMITY TO THEOR 413 00:14:59,998 --> 00:15:03,735 UMKC, IT SAYS WITH A 3 LAYER 414 00:15:03,735 --> 00:15:05,103 BACK PROP, KNOWN THAT KNOW IT 415 00:15:05,103 --> 00:15:06,905 CAN FIT ANY CONTINUOUS FUNCTION 416 00:15:06,905 --> 00:15:07,906 AND GIVEN THAT THE PHENOMENON IS 417 00:15:07,906 --> 00:15:10,442 IN THE DATA AND THAT THERE'S 418 00:15:10,442 --> 00:15:11,610 SUFFICIENT DATA IT'S GUARANTEED 419 00:15:11,610 --> 00:15:13,011 TO FIND THE PHENOMENA. 420 00:15:13,011 --> 00:15:16,648 THAT'S A VERY, VERY POWERFUL 421 00:15:16,648 --> 00:15:18,617 THEOR UMKC AND THAT'S THE THEOR 422 00:15:18,617 --> 00:15:21,453 UMPIRES THAT GOT ME INVOLVED IN 423 00:15:21,453 --> 00:15:22,421 NEURAL NETS BECAUSE THIS IS THE 424 00:15:22,421 --> 00:15:24,656 ONLY METHOD THAT I KNOW THAT'S 425 00:15:24,656 --> 00:15:26,024 GOOD NIGHTED TO FIND THE 426 00:15:26,024 --> 00:15:32,297 PHENOMENA THAT'S IN THE DATA. 427 00:15:32,297 --> 00:15:34,466 OKAY, NOW SOME PEOPLE HAVE ASKED 428 00:15:34,466 --> 00:15:35,434 MAYBE IT'S CONTAMINATION, MAYBE 429 00:15:35,434 --> 00:15:37,102 IT'S JUST REPEATING STRINGS OF 430 00:15:37,102 --> 00:15:37,302 WORDS? 431 00:15:37,302 --> 00:15:39,237 SO YOU PUT IN STRINGS OF WORDS, 432 00:15:39,237 --> 00:15:40,439 IT PUTS OUT STRINGS OF WORDS SO 433 00:15:40,439 --> 00:15:42,207 IT'S JUST LIKE A SIMPLE MATCHING 434 00:15:42,207 --> 00:15:44,242 TAVENG, MATCHING IT UP, SO THEY 435 00:15:44,242 --> 00:15:46,878 DID A STUDY, AND THEY TOOK OUT 436 00:15:46,878 --> 00:15:49,481 SIMILAR STRINGS OF WORDS, AND IT 437 00:15:49,481 --> 00:15:53,385 DIDN'T EFFECT OUTPUT, BECAUSE 438 00:15:53,385 --> 00:15:54,920 IT'S NOT REPORTING SPRINGS OF 439 00:15:54,920 --> 00:16:00,892 WORDS, IT'S ACTUALLY DOING THE 440 00:16:00,892 --> 00:16:01,460 JOINT PROBABILITY. 441 00:16:01,460 --> 00:16:05,497 OKAY SO WHY DOES IT NEED 45 442 00:16:05,497 --> 00:16:08,500 TERRA BYTES OF DATA, IT'S HUGE, 443 00:16:08,500 --> 00:16:09,701 WELL BECAUSE IT NEEDS THE BREDTH 444 00:16:09,701 --> 00:16:11,369 OF INFORMATION, NEEDS TO KNOW A 445 00:16:11,369 --> 00:16:12,938 LOT, DEPTH OF INFORMATION, 446 00:16:12,938 --> 00:16:14,139 ESSENTIAL DETAILS AND 447 00:16:14,139 --> 00:16:15,173 CRITICALLY, IT NEEDS REPETITION, 448 00:16:15,173 --> 00:16:19,277 AND REPETITION IS THE KEY TO 449 00:16:19,277 --> 00:16:20,345 TRUTH, OKAY? 450 00:16:20,345 --> 00:16:23,215 NOW SO IT'S NOT HHV IT'S NOT 451 00:16:23,215 --> 00:16:25,183 JUST FINDING THINGS, AND SAYING 452 00:16:25,183 --> 00:16:27,119 THAT'S WHAT IT IS, BUT IT DOES 453 00:16:27,119 --> 00:16:28,353 NEED REPETITION, WE'LL TALK 454 00:16:28,353 --> 00:16:32,724 ABOUT THAT LATER, SO THERE'S 2 455 00:16:32,724 --> 00:16:34,392 TYPES OF GENAI. 456 00:16:34,392 --> 00:16:36,561 THERE'SUNE MODAL AND MULTIMODAL. 457 00:16:36,561 --> 00:16:38,330 MULTIMODAL IS COMING UP, THIS IS 458 00:16:38,330 --> 00:16:39,764 WHAT LLM IS, WE WILL CALL 459 00:16:39,764 --> 00:16:42,033 EVERYTHING FROM NOW ON LLM, 460 00:16:42,033 --> 00:16:44,870 MULTIMODAL IS IMAGE TO TEXT, 461 00:16:44,870 --> 00:16:47,105 TEXT TO IMAGE, AND CREATE 462 00:16:47,105 --> 00:16:48,707 IMAGES, WE'VE ALL SEEN THAT. 463 00:16:48,707 --> 00:16:50,175 YOU TAKE, YOU TELL IT WHAT YOU 464 00:16:50,175 --> 00:16:55,981 WANT IN TEXT, AND IT CREATES AN 465 00:16:55,981 --> 00:16:57,048 IMAGE THAT'S MULTIMODAL. 466 00:16:57,048 --> 00:16:59,451 SO LET'S TALK ABOUT THE DATA FOR 467 00:16:59,451 --> 00:16:59,951 A MOMENT. 468 00:16:59,951 --> 00:17:02,687 FORTY-FIVE TERRA BYTES OF DATA, 469 00:17:02,687 --> 00:17:06,424 HUGE AMOUNT OF DATA, IT IS 11 470 00:17:06,424 --> 00:17:11,763 AND HALF MILLION VIABLES, 25,000 471 00:17:11,763 --> 00:17:12,397 ENCYCLOPEDIA PRIMATES TANNICCAS. 472 00:17:12,397 --> 00:17:14,232 NOW THIS IS A REALLY, REALLY 473 00:17:14,232 --> 00:17:15,934 IMPORTANT IDEA AND I'M SURPRISED 474 00:17:15,934 --> 00:17:16,568 PEOPLE HAVEN'T REALLY MENTIONED 475 00:17:16,568 --> 00:17:18,737 IT IN THE PAST AND THAT IS THAT 476 00:17:18,737 --> 00:17:20,739 IT'S POWER RISES FROM 477 00:17:20,739 --> 00:17:22,140 INTEGRATION OF INFORMATION, SO 478 00:17:22,140 --> 00:17:24,576 THIS IS LOOKING AT INFORMATION, 479 00:17:24,576 --> 00:17:26,678 SO UNINTEGRATED DATA, LIBRARY OF 480 00:17:26,678 --> 00:17:30,115 ALEX ABDOMEN RIA, BLAH, BLAH, 481 00:17:30,115 --> 00:17:31,616 BLAH, SEMIINTEGRATED, GOOGLE 482 00:17:31,616 --> 00:17:33,818 DOES SEMIINTEGRATION BY LOOKING 483 00:17:33,818 --> 00:17:34,786 AT INDEXING WEBSITES, BUT I HAVE 484 00:17:34,786 --> 00:17:37,155 TO ASK YOU A QUESTION, LET'S 485 00:17:37,155 --> 00:17:40,926 JUST LOOK AT A BOOK ON THIS 486 00:17:40,926 --> 00:17:41,960 SHELF. 487 00:17:41,960 --> 00:17:43,562 IS THIS BOOKING TO THAT BOOK, IS 488 00:17:43,562 --> 00:17:45,564 THAT BOOK TALKING TO THAT BOOK, 489 00:17:45,564 --> 00:17:46,565 ARE THESE BOOKS TALKING TO EACH 490 00:17:46,565 --> 00:17:48,166 OTHER, ARE ANY OF THESE BOOKS 491 00:17:48,166 --> 00:17:49,601 ACTUALLY COMMUNICATING WITH EACH 492 00:17:49,601 --> 00:17:49,901 OTHER? 493 00:17:49,901 --> 00:17:55,607 AND THE ANSWER IS NO. 494 00:17:55,607 --> 00:17:58,210 WHAT A GPT DOES IS IT ALLOWS, 495 00:17:58,210 --> 00:18:00,312 ALL OF THESE BOOKS TO TALK TO 496 00:18:00,312 --> 00:18:05,684 EACH OTHER FOR THE FIRST TIME 497 00:18:05,684 --> 00:18:06,384 EVER, OKAY? 498 00:18:06,384 --> 00:18:08,320 AND THERE DOES NOT APPEAR TO BE 499 00:18:08,320 --> 00:18:10,055 ANY NATURAL LIMIT TO THE AMOUNT 500 00:18:10,055 --> 00:18:12,324 OF INFORMATION THAT GPT COULD 501 00:18:12,324 --> 00:18:13,425 ORGANIZE, AND SO I'M GOING TO 502 00:18:13,425 --> 00:18:16,261 MAKE A COUPLE STATEMENTS HERE, 1 503 00:18:16,261 --> 00:18:18,964 IS, MOST MEDICAL INFORMATION'S 504 00:18:18,964 --> 00:18:20,765 NOT READILY AVAILABLE FOR 505 00:18:20,765 --> 00:18:24,736 CLINICAL USE, IT'S IN A 506 00:18:24,736 --> 00:18:25,837 TEXTBOOK, IT'S AT HARRISON'S, 507 00:18:25,837 --> 00:18:27,472 IT'S WHEREVER AND IN LARGE 508 00:18:27,472 --> 00:18:28,940 KNOWLEDGE MODELS WHICH IS WHAT I 509 00:18:28,940 --> 00:18:30,976 CALL THEM IN THIS CONTEXT IT 510 00:18:30,976 --> 00:18:32,377 MAKE ALL MEDICAL INFORMATION 511 00:18:32,377 --> 00:18:36,414 AVAILABLE IN FACT, MY ASSERTION 512 00:18:36,414 --> 00:18:38,216 IS THAT EVERYTHING THAT IS KNOWN 513 00:18:38,216 --> 00:18:42,053 HAS BECOME KNOWABLE BY GPTs. 514 00:18:42,053 --> 00:18:45,590 THAT'S AN ASTOUNDING CLAIM AND 515 00:18:45,590 --> 00:18:51,763 AN ASTOUNDING ADVANCE IN HUMAN 516 00:18:51,763 --> 00:18:52,097 CIVILIZATION. 517 00:18:52,097 --> 00:18:54,633 SO THE STRENGTH AND LIMITATIONS, 518 00:18:54,633 --> 00:18:57,702 STRENGTH, WELL, YOU KNOW IT 519 00:18:57,702 --> 00:18:59,571 CONTAINS AN IMMENSE AMOUNT OF 520 00:18:59,571 --> 00:19:01,006 ORGANIZATION, IT DOES 521 00:19:01,006 --> 00:19:01,606 PERFORMICISMEM MANIPULATION, 522 00:19:01,606 --> 00:19:03,575 THAT WILL BECOME IMPORTANT LATER 523 00:19:03,575 --> 00:19:04,643 ON, IT ALLOWS NATURAL LANGUAGE 524 00:19:04,643 --> 00:19:07,145 INPUT AND OUTPUT AND IT CAN 525 00:19:07,145 --> 00:19:08,546 PROVIDE AN EXPLANATION OF THE 526 00:19:08,546 --> 00:19:10,248 OUTPUT AND MAKE INFERENCES BUT 527 00:19:10,248 --> 00:19:11,883 BECAUSE IT COMBINES MANY SOURCES 528 00:19:11,883 --> 00:19:13,618 OF INFORMATION, DOESN'T PROVIDE 529 00:19:13,618 --> 00:19:14,686 A DIRECT REFERENCE, BY 530 00:19:14,686 --> 00:19:16,121 DEFINITION, IT CAN'T PROVIDE A 531 00:19:16,121 --> 00:19:18,556 DIRECT REFERENCE BECAUSE IT IS 532 00:19:18,556 --> 00:19:19,658 IN FACT INTEGRATING ALL THAT 533 00:19:19,658 --> 00:19:21,259 INFORMATION, IT'S A STATIC MODEL 534 00:19:21,259 --> 00:19:24,162 AND MEANS THAT IT DOESN'T LEARN 535 00:19:24,162 --> 00:19:25,797 ON THE FLY SO TO SPEAK. 536 00:19:25,797 --> 00:19:29,067 YOU HAVE TO PUT DATA IN IT. 537 00:19:29,067 --> 00:19:30,935 IT REQUIRES THE USER HAVE DOMAIN 538 00:19:30,935 --> 00:19:32,504 KNOWLEDGE AND PROMPT EXPERTISE 539 00:19:32,504 --> 00:19:34,372 AND YOU HAVE TO ASSESS THE 540 00:19:34,372 --> 00:19:36,875 VERACITY OF THE ANSWER. 541 00:19:36,875 --> 00:19:42,047 OKAY, SO, WHAT ARE THE GOALS OF 542 00:19:42,047 --> 00:19:44,082 A GENAI ANSWER, THAT IT'S 543 00:19:44,082 --> 00:19:45,183 CORRECT, INFORMATION'S ACCURATE, 544 00:19:45,183 --> 00:19:46,718 IT'S COMPREHENSIVE AND THAT IT'S 545 00:19:46,718 --> 00:19:47,218 EFFICIENT. 546 00:19:47,218 --> 00:19:48,086 THOSE ARE THE 3 THINGS THAT YOU 547 00:19:48,086 --> 00:19:53,525 WANT TO LOOK FOR IN ANY ANSWER 548 00:19:53,525 --> 00:19:54,459 FROM A GPT. 549 00:19:54,459 --> 00:19:56,294 SO HOW DO YOU MAKE IT DO WHAT 550 00:19:56,294 --> 00:19:57,162 YOU WANT IT DO. 551 00:19:57,162 --> 00:20:00,365 SO YOU CAN USE A FOUNDATIONAL 552 00:20:00,365 --> 00:20:02,100 MODEL, YOU CAN USE A TASK 553 00:20:02,100 --> 00:20:05,103 SPECIFIC MODEL AND THERE'S A LOT 554 00:20:05,103 --> 00:20:06,004 OF MEDICAL MODELS BEING 555 00:20:06,004 --> 00:20:09,174 DEVELOPED RIGHT NOW, OKAY? 556 00:20:09,174 --> 00:20:10,475 THEY'RE GOING TO COME OUT AND 557 00:20:10,475 --> 00:20:11,976 THERE'S HUNDREDS OF COMPANIES 558 00:20:11,976 --> 00:20:15,613 WORKING ON MEDICAL LLMs RIGHT 559 00:20:15,613 --> 00:20:16,648 NOW, OKAY, SO, SO YOU HAVE THESE 560 00:20:16,648 --> 00:20:19,484 THINGS AND IF YOU WANTED TO DO 561 00:20:19,484 --> 00:20:20,518 THOSE, YOU CAN PRETRAIN IT TO DO 562 00:20:20,518 --> 00:20:22,253 WHAT YOU WANT IT TO DO, YOU CAN 563 00:20:22,253 --> 00:20:23,555 FINE TUNE IT WHICH MEANS THAT 564 00:20:23,555 --> 00:20:25,990 YOU TAKE DATA AND YOU PUT IT IN 565 00:20:25,990 --> 00:20:28,993 AND IT'S USUALLY REINFORCEMENT 566 00:20:28,993 --> 00:20:30,962 LEARNING, OKAY? 567 00:20:30,962 --> 00:20:34,499 OR, YOU CAN USE AN EXISTING 568 00:20:34,499 --> 00:20:37,635 GENAI MODEL, YOU CAN DO 569 00:20:37,635 --> 00:20:39,137 IN-PROMPT QUERY AND YOU GIVE US 570 00:20:39,137 --> 00:20:41,339 INFORMATION AND ASK A DETAIL, 571 00:20:41,339 --> 00:20:43,675 WHICH IS DEDUCTIVE REASONING OR 572 00:20:43,675 --> 00:20:44,576 IN-CONTEXT LEARNING WHICH IS YOU 573 00:20:44,576 --> 00:20:46,044 GIVE IT A WHOLE BUNCH OF 574 00:20:46,044 --> 00:20:48,046 INFORMATION AND YOU ASK IT FOR 575 00:20:48,046 --> 00:20:49,681 GENERALIZATION SO IT DOES 576 00:20:49,681 --> 00:20:50,915 DEDUCTIVE AND INDUSTRIOUSIVE, 577 00:20:50,915 --> 00:20:52,350 YOU WILL DO DATABASE RETRIEVAL 578 00:20:52,350 --> 00:20:55,120 BUT WHAT'S REALLY COMING UP IS 579 00:20:55,120 --> 00:20:55,286 RAG. 580 00:20:55,286 --> 00:20:56,554 RAG IS REALLY IMPORTANT BECAUSE 581 00:20:56,554 --> 00:20:58,590 WHAT RAG ALLOWS YOU TO DO IS IT 582 00:20:58,590 --> 00:21:00,658 ALLOWS YOU TO PUT IN A PROMPT 583 00:21:00,658 --> 00:21:03,528 AND THEN THE PROMPT GOES TO YOUR 584 00:21:03,528 --> 00:21:08,032 DATA, RAG IS BASICALLY A 585 00:21:08,032 --> 00:21:08,533 VECTORIZED DATABASE OF 586 00:21:08,533 --> 00:21:11,336 INFORMATION YOU PUT IN THAT YOU 587 00:21:11,336 --> 00:21:13,338 FEEL IS VALID INFORMATION AND IT 588 00:21:13,338 --> 00:21:14,773 WILL EXTRACT THE INFORMATION 589 00:21:14,773 --> 00:21:16,374 FROM THE RAG, AND THEN GIVE IT 590 00:21:16,374 --> 00:21:18,777 TO THE LLM, AND THE LLM WILL PUT 591 00:21:18,777 --> 00:21:23,681 IT TOGETHER, THAT MEANS THAT YOU 592 00:21:23,681 --> 00:21:26,317 CAN CHEAPLY, ADD INFORMATION TO 593 00:21:26,317 --> 00:21:28,253 THE SYSTEM WITHOUT HAVING TO 594 00:21:28,253 --> 00:21:31,356 PRETRAIN OR FINE TUNE WHICH ARE 595 00:21:31,356 --> 00:21:32,657 BOTH VERY EXPENSIVE AND YOU CAN 596 00:21:32,657 --> 00:21:34,492 GIVE IT YOUR INFORMATION. 597 00:21:34,492 --> 00:21:36,127 SO HOW DOES IT DETERMINE THE 598 00:21:36,127 --> 00:21:39,063 BEST ANSWER, YOU KNOW THAT'S A 599 00:21:39,063 --> 00:21:39,964 REALLY INTERESTING QUESTION. 600 00:21:39,964 --> 00:21:41,566 WELL, IT TURNS OUT THAT IT 601 00:21:41,566 --> 00:21:44,636 RELIES ON THE IDEA THAT 602 00:21:44,636 --> 00:21:45,336 INCORRECT INFORMATION WAS 603 00:21:45,336 --> 00:21:46,971 WRITTEN IN MANY DIFFERENT WAYS 604 00:21:46,971 --> 00:21:48,273 AND TRUE INFORMATION'S WRITTEN 605 00:21:48,273 --> 00:21:51,009 IN THE CONSISTENT WAY, SO IF I 606 00:21:51,009 --> 00:21:52,811 ASK YOU, WELL, WHAT'S THE MOON 607 00:21:52,811 --> 00:21:55,146 MADE OF, A LOT OF PEOPLE CAN 608 00:21:55,146 --> 00:21:57,816 SAY, LIKE BLUE CHEESE, SWISS 609 00:21:57,816 --> 00:21:59,751 CHEESE, ALL KINDS OF STUFF BUT 610 00:21:59,751 --> 00:22:00,952 THE 1 CONSISTENT ANSWER IS YOU 611 00:22:00,952 --> 00:22:04,355 KNOW IT'S MADE OF ROCK, IT'S 612 00:22:04,355 --> 00:22:05,323 280,000-MILES AWAY AND, ET 613 00:22:05,323 --> 00:22:06,458 CETERA, ET CETERA AND THAT WILL 614 00:22:06,458 --> 00:22:09,027 BE REPEATED OVER AND OVER AGAIN. 615 00:22:09,027 --> 00:22:11,596 SO THEY SAY INTUITIVELY, THE 616 00:22:11,596 --> 00:22:12,430 REASON MAJORITY IMPROVES 617 00:22:12,430 --> 00:22:13,398 PERFORMANCE IS THAT WHILE THERE 618 00:22:13,398 --> 00:22:15,433 ARE MANY WAYS TO ANSWER 619 00:22:15,433 --> 00:22:16,534 QUESTIONS INCORRECTLY, THERE ARE 620 00:22:16,534 --> 00:22:17,735 FEW WAYS TO ANSWER INCORRECTLY 621 00:22:17,735 --> 00:22:20,672 AND IN FACT, THAT'S ACTUALLY A 622 00:22:20,672 --> 00:22:22,907 VERSION OF SCIENTIFIC TRUTH BY 623 00:22:22,907 --> 00:22:24,042 THE WAY, WHICH IS INTERESTING 624 00:22:24,042 --> 00:22:26,110 AND THEN PEOPLE TALK ABOUT DEEP 625 00:22:26,110 --> 00:22:28,112 FAKES, DEEP FAKES SHOULDN'T BE A 626 00:22:28,112 --> 00:22:30,815 PROBLEM BECAUSE DEEP FAKES ARE 627 00:22:30,815 --> 00:22:33,551 GOING TO OCCUR RARELY UNLESS, 628 00:22:33,551 --> 00:22:36,087 THEY'RE FINE TUNED INTO THE 629 00:22:36,087 --> 00:22:37,155 SYSTEM OR UNLESS, SOMEBODY GETS 630 00:22:37,155 --> 00:22:43,061 A HOLD OF A TON OF COMPUTERS AND 631 00:22:43,061 --> 00:22:45,163 SOMEHOW UPDATES A FOUNDATIONAL 632 00:22:45,163 --> 00:22:49,701 MODEL WITH THAT DEEP FAKE. 633 00:22:49,701 --> 00:22:50,702 OTHERWISE, DEEP FAKE SHOULD NOT 634 00:22:50,702 --> 00:22:55,773 BE AN ANSWER THAT AN LLM GIVES 635 00:22:55,773 --> 00:22:56,841 YOU. 636 00:22:56,841 --> 00:23:00,178 ALL RIGHT, SO, HOW DO YOU 637 00:23:00,178 --> 00:23:02,547 DETERMINE IF ITS ANSWER IS THE 638 00:23:02,547 --> 00:23:02,847 BEST ANSWER? 639 00:23:02,847 --> 00:23:04,349 THIS IS AN IMPORTANT IDEA. 640 00:23:04,349 --> 00:23:06,084 SO IT GIVES YOU AN ANSWER AND 641 00:23:06,084 --> 00:23:08,686 YOU WANT TO USE YOUR EXPERTISE 642 00:23:08,686 --> 00:23:11,055 AND MEDICAL JUDGMENT, THAT'S 643 00:23:11,055 --> 00:23:11,456 REALLY IMPORTANT. 644 00:23:11,456 --> 00:23:12,924 IN OTHER WORDS IF YOU ASK A 645 00:23:12,924 --> 00:23:14,526 QUESTION, YOU SHOULD HAVE THE 646 00:23:14,526 --> 00:23:15,693 EXPERTISE TO UNDERSTAND THIS 647 00:23:15,693 --> 00:23:18,229 ANSWER, BUT WHAT I LIKE TO DO IS 648 00:23:18,229 --> 00:23:21,099 I LIKE TO HAVE 2 GENAI PROGRAMS 649 00:23:21,099 --> 00:23:22,600 AND I PUT THE SAME PROMPT IN 650 00:23:22,600 --> 00:23:24,669 BOTH AND IF I GET THE SAME 651 00:23:24,669 --> 00:23:25,904 ANSWER, WE'RE PRETTY MUCH THE 652 00:23:25,904 --> 00:23:26,704 SAME ANSWER, I FIGURE IT'S 653 00:23:26,704 --> 00:23:29,474 PROBABLY THE RIGHT ANSWER, OKAY? 654 00:23:29,474 --> 00:23:32,777 NOW YOU CAN ASK A GENAI TO CHECK 655 00:23:32,777 --> 00:23:36,047 ITS ANSWER, AND THERE ARE NOW 656 00:23:36,047 --> 00:23:39,083 SOME PROGRAMS OUT THERE OKAY, 657 00:23:39,083 --> 00:23:41,152 LIKE ALPHA PROOF THAT WILL CHECK 658 00:23:41,152 --> 00:23:43,254 ITS OUTPUT BEFORE IT GIVES IT TO 659 00:23:43,254 --> 00:23:46,925 YOU TO MAKE SURE THAT THAT'S THE 660 00:23:46,925 --> 00:23:47,292 CORRECT OUTPUT. 661 00:23:47,292 --> 00:23:48,626 OKAY, YOU CAN ASK THE SAME 662 00:23:48,626 --> 00:23:49,928 QUESTION IN A DIFFERENT WAY AND 663 00:23:49,928 --> 00:23:52,530 SEE IF YOU GET PRETTY MUCH THE 664 00:23:52,530 --> 00:23:52,897 SAME ANSWER. 665 00:23:52,897 --> 00:23:55,099 YOU CAN PUT A PROMPT IN GOOGLE 666 00:23:55,099 --> 00:23:57,902 SEARCH, OKAY, AND IN FACT, 667 00:23:57,902 --> 00:23:59,404 TODAY, LET'S TALK ABOUT IT IN A 668 00:23:59,404 --> 00:24:00,171 MINUTE. 669 00:24:00,171 --> 00:24:03,341 WHEN YOU DO A GOOGLE SEARCH, YOU 670 00:24:03,341 --> 00:24:05,843 GET AN LLM AND YOU GET A GOOGLE 671 00:24:05,843 --> 00:24:11,316 SEARCH OUTPUT OR YOU CAN CHECK 672 00:24:11,316 --> 00:24:12,584 OTHER SOURCES. 673 00:24:12,584 --> 00:24:14,686 OKAY, WELL WHAT ABOUT PROMPTS OR 674 00:24:14,686 --> 00:24:15,920 PROMPTS ARE REALLY, REALLY 675 00:24:15,920 --> 00:24:21,759 IMPORTANT BECAUSE THE WHOLE 676 00:24:21,759 --> 00:24:23,795 SYSTEM DEPENDS ON PROPER PROMPT. 677 00:24:23,795 --> 00:24:26,531 SO MOST PEOPLE WHO USE AN LLM, 678 00:24:26,531 --> 00:24:28,967 PUT IN PROMPTS AS IF THEY'RE 679 00:24:28,967 --> 00:24:31,536 DOING A GOOGLE SEARCH, THAT'S A 680 00:24:31,536 --> 00:24:32,370 FACT. 681 00:24:32,370 --> 00:24:34,606 OKAY, AND FOR MANY LIKE FACTOID 682 00:24:34,606 --> 00:24:37,342 TYPE CLAIMS, WHERE YOU ARE JUST 683 00:24:37,342 --> 00:24:39,477 INTERESTED IN GETTING A FACT 684 00:24:39,477 --> 00:24:41,412 OUT, THAT MAY VERY WELL WORK, 685 00:24:41,412 --> 00:24:43,381 BUT IF YOU'RE ACTUALLY USING IT 686 00:24:43,381 --> 00:24:49,954 FOR SOME DEEPER PURPOSE FOR 687 00:24:49,954 --> 00:24:50,922 EXAMPLE, WHEN OUR MEDICAL 688 00:24:50,922 --> 00:24:52,857 STUDENTS GO IN CLINICAL WARDS, 689 00:24:52,857 --> 00:24:54,258 THERE ARE PRECEPTORS WRITE UP A 690 00:24:54,258 --> 00:24:56,894 SUMMARY AND THEN THE CLERKSHIP 691 00:24:56,894 --> 00:25:00,365 DIRECTOR TAKES ALL THESE AND 692 00:25:00,365 --> 00:25:02,066 CREATES AN OVERALL SUMMARY SO 693 00:25:02,066 --> 00:25:04,035 WHAT WE DID WAS WRITE A 2 PAGE 694 00:25:04,035 --> 00:25:05,570 SINGLE SPACE PROMPT THAT TOOK 695 00:25:05,570 --> 00:25:07,472 THE PRECEPTOR SUMMARIES AND 696 00:25:07,472 --> 00:25:10,041 TURNED THEM INTO THE CLERKSHIP 697 00:25:10,041 --> 00:25:10,541 DIRECTOR SUMMARY. 698 00:25:10,541 --> 00:25:13,277 AND IT WORKS BEAUTIFULLY AND IT 699 00:25:13,277 --> 00:25:14,345 USED TO TAKE THEM FRIEF MINUTE 700 00:25:14,345 --> 00:25:16,047 ANDS NOW IT TAKES THEM 3 MINUTES 701 00:25:16,047 --> 00:25:17,882 TO DO THAT TASK, OKAY IN BUT IT 702 00:25:17,882 --> 00:25:19,350 TOOK A 2 PAGE SINGLE SPACED 703 00:25:19,350 --> 00:25:21,419 PROMPT TO DO THAT. 704 00:25:21,419 --> 00:25:23,921 SO IF YOU JUST LOOK FOR AN 705 00:25:23,921 --> 00:25:26,791 ISOLATED FACTOR OR SOME ISOLATED 706 00:25:26,791 --> 00:25:27,859 INFORMATION, IT PROBABLY DOESN'T 707 00:25:27,859 --> 00:25:30,728 MATTER WHICH YOU USE, A GOOGLE 708 00:25:30,728 --> 00:25:32,397 SEARCH OR AN LLM, BUT IF YOU 709 00:25:32,397 --> 00:25:34,065 ACTUALLY WANT TO DO SOMETHING 710 00:25:34,065 --> 00:25:38,603 WITH AN LLM, YOU HAVE TO CREATE 711 00:25:38,603 --> 00:25:40,271 MORE PROFESSIONAL PROMPTS, OKAY? 712 00:25:40,271 --> 00:25:41,472 SO THEY'RE VERY DIFFERENT. 713 00:25:41,472 --> 00:25:43,241 THE 2 TYPES OF SEARCHES ARE VERY 714 00:25:43,241 --> 00:25:45,510 DIFFERENT, YOU KNOW A GOOGLE 715 00:25:45,510 --> 00:25:47,812 DOESN'T ACTUALLY CONTAIN ANY 716 00:25:47,812 --> 00:25:49,080 REAL INFORMATION, INDEXES, AND 717 00:25:49,080 --> 00:25:51,849 IT DOES KEY WORD SEARCHES, 718 00:25:51,849 --> 00:25:52,050 RIGHT? 719 00:25:52,050 --> 00:25:54,185 WHEREAS THIS ACTUALLY CONTAINS 720 00:25:54,185 --> 00:25:56,921 THE INFORMATION ITSELF AND IN 721 00:25:56,921 --> 00:25:59,023 FACT, IT'S NOT HHV IT'S 722 00:25:59,023 --> 00:26:00,058 ORGANIZED AND STORED AND 723 00:26:00,058 --> 00:26:02,627 RESTREEFED AND YOU CAN TUNNEL 724 00:26:02,627 --> 00:26:03,461 DOWN BY RECURSIVE SEARCHING SO 725 00:26:03,461 --> 00:26:04,762 YOU CAN START WITH A PROMPT AND 726 00:26:04,762 --> 00:26:06,698 GET THE OUTPUT AND THEN ASK A 727 00:26:06,698 --> 00:26:08,199 PROMPT ABOUT THAT OUTPUT AND A 728 00:26:08,199 --> 00:26:09,200 PROMPT ABOUT THAT OUTPUT, AND 729 00:26:09,200 --> 00:26:11,235 YOU CAN DO THESE RECURSIVE 730 00:26:11,235 --> 00:26:12,537 SEARCHES UNTIL YOU GET DOWN TO A 731 00:26:12,537 --> 00:26:15,306 LEVEL YOU WANT TO BE AT IN TERMS 732 00:26:15,306 --> 00:26:16,607 OF THE INFORMATION BUT YOU HAVE 733 00:26:16,607 --> 00:26:19,644 TO BE CAREFUL, YOU DON'T GO DOWN 734 00:26:19,644 --> 00:26:21,479 TOO FAR BECAUSE YOU WILL WIND UP 735 00:26:21,479 --> 00:26:28,920 IN A HALLUCINATION. 736 00:26:28,920 --> 00:26:33,257 OKAY, SO TODAY MOST OF WHAT WE 737 00:26:33,257 --> 00:26:35,593 DO WE USE GENERAL GENAI PROGRAMS 738 00:26:35,593 --> 00:26:38,596 FOR, WE USE GOOGLE, WE USE, YOU 739 00:26:38,596 --> 00:26:41,532 KNOW GPT4, WHICH ARE JUST 740 00:26:41,532 --> 00:26:46,537 GENERAL AI PROGRAMS, OKAY. 741 00:26:46,537 --> 00:26:48,473 WHAT WE ARE GOING TO BE USING IN 742 00:26:48,473 --> 00:26:50,374 THE FUTURE IN MEDICINE IS 743 00:26:50,374 --> 00:26:51,275 MEDICAL SPECIFIC PROGRAMS, 744 00:26:51,275 --> 00:26:55,113 PROGRAMS THAT WERE TRAINED HHV 745 00:26:55,113 --> 00:26:56,247 PRETRAINED ON MEDICAL 746 00:26:56,247 --> 00:27:00,852 INFORMATION ON CURATED MEDICAL 747 00:27:00,852 --> 00:27:02,353 INFORMATION, OKAY? 748 00:27:02,353 --> 00:27:04,322 NOW BECAUSE THE OUTPUT, OKAY IS 749 00:27:04,322 --> 00:27:05,923 A CONDITIONAL PROBABILITY AND 750 00:27:05,923 --> 00:27:07,792 THE CONDITIONALS OF PROBABLES 751 00:27:07,792 --> 00:27:09,193 ARE VERY SMALL, SMALL 752 00:27:09,193 --> 00:27:11,095 DIFFERENCES IN PROMPTS GIVE YOU 753 00:27:11,095 --> 00:27:12,497 LARGE DIFFERENCES IN OUTPUT 754 00:27:12,497 --> 00:27:13,431 BECAUSE OF THESE SMALL 755 00:27:13,431 --> 00:27:15,032 PROBABILITIES IF YOU PUT IN 756 00:27:15,032 --> 00:27:16,634 SOMETHING SLIGHTLY DEFINITE IN 757 00:27:16,634 --> 00:27:17,769 YOUR PROMPT, YOU'RE GOING TO GET 758 00:27:17,769 --> 00:27:19,337 A WHOLE DIFFERENT SET OF 759 00:27:19,337 --> 00:27:22,473 PROBABILITIES OUT, SO THAT'S WHY 760 00:27:22,473 --> 00:27:23,975 YOUR OUTPUT IS EXQUISITELY 761 00:27:23,975 --> 00:27:27,311 SENSITIVE TO YOUR PROMPT. 762 00:27:27,311 --> 00:27:29,580 OKAY, SO YOU NEED TO DO PROMPT 763 00:27:29,580 --> 00:27:30,348 ENGINEERING AND PROMPT 764 00:27:30,348 --> 00:27:33,618 ENGINEERING IS A WHOLE BIG DEAL. 765 00:27:33,618 --> 00:27:36,320 YOU CAN TAKE COURSES IN IT, IT'S 766 00:27:36,320 --> 00:27:38,790 ALL VERY INTERESTING, THINGS 767 00:27:38,790 --> 00:27:40,725 LIKE THE TYPES OF PROMPT, WHAT 768 00:27:40,725 --> 00:27:42,126 TYPE OF PROMPT DO YOU WANT TO 769 00:27:42,126 --> 00:27:46,197 USE, DO YOU WANT TO USE A 770 00:27:46,197 --> 00:27:47,064 REQUEST, IMPROVEMENT, WHATEVER. 771 00:27:47,064 --> 00:27:49,200 AND WHAT ARE THE ATTRIBUTES OF 772 00:27:49,200 --> 00:27:52,003 YOUR PROMPT? 773 00:27:52,003 --> 00:27:53,738 DO YOU WANT-WHO IS DOING THE 774 00:27:53,738 --> 00:27:54,939 PROMPT? 775 00:27:54,939 --> 00:27:56,174 WHAT IS THE PERSONA, WHAT SOURCE 776 00:27:56,174 --> 00:27:58,309 DO YOU WANT IT TO FIND IT FROM, 777 00:27:58,309 --> 00:27:59,911 WHATTURE YOUR EXPECTATIONS, WHAT 778 00:27:59,911 --> 00:28:01,179 IS THE OUTPUT SUPPOSED TO LOOK 779 00:28:01,179 --> 00:28:02,180 LIKE, WHAT IS THE TONE? 780 00:28:02,180 --> 00:28:03,514 ALL OF THOSE THINGS ARE SUPPOSED 781 00:28:03,514 --> 00:28:06,050 TO BE IN YOUR PROMPT. 782 00:28:06,050 --> 00:28:06,818 AND MULTIPROMPTS, RIGHT? 783 00:28:06,818 --> 00:28:12,190 SO NOT JUST 1 PROMPT, YOU KNOW A 784 00:28:12,190 --> 00:28:15,026 SUCCESSION OF PROMPTS AND YOU 785 00:28:15,026 --> 00:28:17,829 MAY WANT STEP-BY-STEP, YOU MAY 786 00:28:17,829 --> 00:28:20,531 WANT THE GPT TO TELL YOU WHAT 787 00:28:20,531 --> 00:28:24,702 IT'S THINKING AND A SERIES OF 788 00:28:24,702 --> 00:28:25,002 STEPS, OKAY? 789 00:28:25,002 --> 00:28:27,138 SHOW IT'S RESULTS AS A SERIES OF 790 00:28:27,138 --> 00:28:28,639 STEPS, YOU CAN UNDERSTAND WHAT 791 00:28:28,639 --> 00:28:31,909 IT'S DOING, AND YOU CAN BREAK 792 00:28:31,909 --> 00:28:34,011 THE PROMPT DOWN, OKAY, USING 793 00:28:34,011 --> 00:28:35,947 CHAIN OF REASONING PROMPTS, 794 00:28:35,947 --> 00:28:41,886 WHERE YOU DO SERIAL STEPS, YOU 795 00:28:41,886 --> 00:28:44,222 CAN DO LOGICAL STEPS, YOU DO 796 00:28:44,222 --> 00:28:45,523 HIERARCHICAL STEPS, WHERE YOU DO 797 00:28:45,523 --> 00:28:46,991 A SERIES OF PROMPTS 1 RIGHT 798 00:28:46,991 --> 00:28:48,860 AFTER THE OTHER TO GET AN ANSWER 799 00:28:48,860 --> 00:28:50,962 AND YOU CAN SHOW THE 800 00:28:50,962 --> 00:28:52,663 INTERMEDIATE WORK, OKAY? 801 00:28:52,663 --> 00:28:53,731 AND YOU KNOW, BOTH OF THE THINGS 802 00:28:53,731 --> 00:28:55,633 I THINK THAT WILL BE REALLY GOOD 803 00:28:55,633 --> 00:28:58,269 FOR MEDICINE IS THEY WILL DO 804 00:28:58,269 --> 00:29:01,005 REPETITIVE TASKS, AND REPETITIVE 805 00:29:01,005 --> 00:29:02,106 TASKS REQUIRE TEMPLATES, SO YOU 806 00:29:02,106 --> 00:29:03,441 CREATE A TEMPLATE LIKE WHAT WE 807 00:29:03,441 --> 00:29:05,276 DID FOR THE MEDICAL STUDENT 808 00:29:05,276 --> 00:29:06,210 EVALUATIONS OKAY, AND THE 809 00:29:06,210 --> 00:29:08,279 PROGRAM DIRECTORS CAN USE THAT 810 00:29:08,279 --> 00:29:10,581 FOREVER BECAUSE IT'S A GENERIC 811 00:29:10,581 --> 00:29:12,250 PROMPT, AND THEY JUST PUT THE 812 00:29:12,250 --> 00:29:13,885 DATA IN AND THE RESULT COMES 813 00:29:13,885 --> 00:29:18,756 OUT, SO WE DON'T HAVE TO DO IT 814 00:29:18,756 --> 00:29:19,223 AGAIN. 815 00:29:19,223 --> 00:29:20,658 OKAY, SO IF YOU'RE PROMPT 816 00:29:20,658 --> 00:29:24,228 DOESN'T PROVIDE THE BEST ANSWER? 817 00:29:24,228 --> 00:29:24,462 WHY? 818 00:29:24,462 --> 00:29:26,530 WELL BECAUSE YOUR PROMPT IS TOO 819 00:29:26,530 --> 00:29:28,933 GENERAL, TOO SHORT, INCORRECTLY 820 00:29:28,933 --> 00:29:29,867 CRITTEN, TOO COMPLEX, YOU 821 00:29:29,867 --> 00:29:30,668 MISUNDERSTOOD WHAT YOU'RE BEING 822 00:29:30,668 --> 00:29:32,336 LOAMACYING FOR AND YOU DID A 823 00:29:32,336 --> 00:29:34,538 NESTED SEARCH AND YOU WENT TOO 824 00:29:34,538 --> 00:29:36,040 FAR OR THE PROBLEM'S THE 825 00:29:36,040 --> 00:29:36,974 INFORMATION, WE WILL TALK ABOUT 826 00:29:36,974 --> 00:29:39,176 THAT IN A MINUTE, IT DOESN'T 827 00:29:39,176 --> 00:29:41,145 CONTAIN THE INFORMATION, 828 00:29:41,145 --> 00:29:42,380 INFORMATION OUTDATED, INCORRECT 829 00:29:42,380 --> 00:29:45,049 OR CONFLICTING AND WE WILL TALK 830 00:29:45,049 --> 00:29:46,584 ABOUT FINEUNING BECAUSE FINE 831 00:29:46,584 --> 00:29:47,818 TUNING CHANGES THE PARAMETER OF 832 00:29:47,818 --> 00:29:50,121 THE MODEL IN INTERESTING WAYS. 833 00:29:50,121 --> 00:29:54,325 OKAY, SO WHAT IS A 834 00:29:54,325 --> 00:29:55,192 HALLUCINATION? 835 00:29:55,192 --> 00:29:56,727 EVERYBODY KNOWS WHAT 836 00:29:56,727 --> 00:29:58,029 HALLUCINATION IS, BUT IN FACT, 837 00:29:58,029 --> 00:30:01,165 THIS BASICALLY CAME OUT OF THIS 838 00:30:01,165 --> 00:30:03,534 IDEA WHERE THE MISTRANSLATIONS 839 00:30:03,534 --> 00:30:04,602 ARE COMPLETELY CEMENTICALLY 840 00:30:04,602 --> 00:30:06,203 INCORRECT, THEY ARE UNTETHERED 841 00:30:06,203 --> 00:30:08,539 TO THE INPUT SO THEY CAUSE A LOT 842 00:30:08,539 --> 00:30:11,342 OF HALLUCINATIONS, SO THERE'S A 843 00:30:11,342 --> 00:30:12,376 VERY INTERESTING HISTORY HERE. 844 00:30:12,376 --> 00:30:17,448 SO THIS CAME OUT ABOUT A YEAR 845 00:30:17,448 --> 00:30:18,416 AFTER VISWANI'S PAPER, AND WHEN 846 00:30:18,416 --> 00:30:20,384 I ASK ABOUT THE PAPER, EVERYBODY 847 00:30:20,384 --> 00:30:22,386 KNOWS ABOUT CHAT GPT AND ALL 848 00:30:22,386 --> 00:30:23,754 THIS STUFF, AND I SAID OKAY, 849 00:30:23,754 --> 00:30:24,956 WHAT EXPERIMENT DID THEY DO IN 850 00:30:24,956 --> 00:30:27,458 THEIR PAPER AND ALL THESE EXPERT 851 00:30:27,458 --> 00:30:30,728 ON GPT, THEY DON'T KNOW WHAT 852 00:30:30,728 --> 00:30:31,062 THEY DID. 853 00:30:31,062 --> 00:30:33,297 WELL, BASICALLY THEY DID ENGLISH 854 00:30:33,297 --> 00:30:34,498 TO GERMAN TRANSLATION. 855 00:30:34,498 --> 00:30:35,733 THAT'S WHAT THEY DID. 856 00:30:35,733 --> 00:30:37,268 OKAY, SO THESE PEOPLE DOING 857 00:30:37,268 --> 00:30:38,703 TRANSLATION, TOO, MAKES A LOT OF 858 00:30:38,703 --> 00:30:40,471 SENSE, RIGHT? 859 00:30:40,471 --> 00:30:43,741 OKAY, AND THEY SEE THIS STUFF, 860 00:30:43,741 --> 00:30:46,043 THAT DOESN'T CONNECT UP, AND 861 00:30:46,043 --> 00:30:48,279 THEY SAY HALLUCINATION. 862 00:30:48,279 --> 00:30:50,214 THAT'S THE WHOLE IDEA OF 863 00:30:50,214 --> 00:30:51,215 HALLUCINATION COMES FROM. 864 00:30:51,215 --> 00:30:52,917 BUT IN FACT, HALLUCINATION IS A 865 00:30:52,917 --> 00:30:54,752 FALSE PERCEPTION OF OBJECTS OR 866 00:30:54,752 --> 00:30:56,721 EVENTS INVOLVING YOUR SENSES, 867 00:30:56,721 --> 00:30:58,990 AND GENERATIVE AI IS NOT 868 00:30:58,990 --> 00:31:00,891 CAPABILITY OF CREATING 869 00:31:00,891 --> 00:31:01,225 HALLUCINATION. 870 00:31:01,225 --> 00:31:03,527 I MEAN IT'S JUST LOGICALLY NOT 871 00:31:03,527 --> 00:31:03,894 POSSIBLE. 872 00:31:03,894 --> 00:31:06,297 OKAY, SO, WHAT'S GOING ON, WELL, 873 00:31:06,297 --> 00:31:07,565 THERE'S 3 THINGS THAT COULD GO 874 00:31:07,565 --> 00:31:09,000 ON? 875 00:31:09,000 --> 00:31:11,635 ONE IS SOMETHING CALLED 876 00:31:11,635 --> 00:31:13,771 CONFABULATION FOR MISSING OR 877 00:31:13,771 --> 00:31:14,672 INCORRECT DALLASCOWBOYS.COM 878 00:31:14,672 --> 00:31:16,874 DATA, ERRORS DUE TO FINEUNING OR 879 00:31:16,874 --> 00:31:19,210 INCORRECT PROMPT. 880 00:31:19,210 --> 00:31:19,410 OKAY. 881 00:31:19,410 --> 00:31:22,079 SO WHAT YOU HAVE TO KNOW ABOUT 882 00:31:22,079 --> 00:31:24,648 AN LLM, IS IT WILL ALWAYS TRY TO 883 00:31:24,648 --> 00:31:25,116 ANSWER YOUR QUESTION. 884 00:31:25,116 --> 00:31:26,283 WHATEVER IT IS, IT WILL TRY TO 885 00:31:26,283 --> 00:31:29,820 ANSWER IT AND IF IT DOESN'T HAVE 886 00:31:29,820 --> 00:31:30,755 THE INFORMATION, IT DOESN'T 887 00:31:30,755 --> 00:31:31,822 KNOW, IT DOESN'T HAVE THE 888 00:31:31,822 --> 00:31:34,358 INFORMATION, IT WILL PROVIDE AN 889 00:31:34,358 --> 00:31:36,427 ANSWER BY FINDING THE MOST 890 00:31:36,427 --> 00:31:38,362 PROBABLE INFORMATION RELATED TO 891 00:31:38,362 --> 00:31:39,130 THE INFORMATION IN YOUR 892 00:31:39,130 --> 00:31:39,864 QUESTION. 893 00:31:39,864 --> 00:31:42,566 SO THIS IS LIKE AN ALCOHOLIC AND 894 00:31:42,566 --> 00:31:44,035 HE OR SHE, YOU ASK THEM A 895 00:31:44,035 --> 00:31:45,302 QUESTION AND THEY COME IN THE 896 00:31:45,302 --> 00:31:47,171 ED, AND YOU ASK THEM A QUESTION 897 00:31:47,171 --> 00:31:50,241 AND THEY GIVE YOU AN ANSWER, 898 00:31:50,241 --> 00:31:50,841 THEY DON'T KNOW. 899 00:31:50,841 --> 00:31:54,145 SO THEY COME THEIR BEST GUESS, 900 00:31:54,145 --> 00:31:56,514 THAT'S WHAT THE GPT IS DOING. 901 00:31:56,514 --> 00:31:59,216 I ASKED THE PATIENT, THE 902 00:31:59,216 --> 00:31:59,917 PATIENT'S HAVE TO ANSWER ME BUT 903 00:31:59,917 --> 00:32:02,486 THEY DON'T KNOW THE ANSWER SO 904 00:32:02,486 --> 00:32:02,887 THEY CONFABULATE. 905 00:32:02,887 --> 00:32:06,690 THAT'S WHAT'S GOING ON HERE, 906 00:32:06,690 --> 00:32:06,891 OKAY? 907 00:32:06,891 --> 00:32:08,826 SO SOLUTIONS TO CONFABULATION. 908 00:32:08,826 --> 00:32:10,961 ONE IS FINE TUNING, PROVIDING 909 00:32:10,961 --> 00:32:11,529 ARK DITIONAL RELEVANT 910 00:32:11,529 --> 00:32:14,231 INFORMATION, AND THE OTHER IS 911 00:32:14,231 --> 00:32:16,700 THE RAG, USING AUTHORITATIVE 912 00:32:16,700 --> 00:32:17,368 KNOWLEDGE BASE OUTSIDE ITS 913 00:32:17,368 --> 00:32:19,470 TRAINING DAILY BASIS THEA TO 914 00:32:19,470 --> 00:32:20,471 GENERATE THE RESPONSE. 915 00:32:20,471 --> 00:32:23,808 THOSE ARE THE SOLUTIONS TO 916 00:32:23,808 --> 00:32:27,878 CONFABULATION AND THEY WORK. 917 00:32:27,878 --> 00:32:30,915 AND ERRORS DUE TO FINE TUNING, 918 00:32:30,915 --> 00:32:32,750 BUT FINE TUNING ITSELF IS A VERY 919 00:32:32,750 --> 00:32:35,820 TRICKY BUSINESS AND CREATE A LOT 920 00:32:35,820 --> 00:32:38,222 OF ERRORS ITSELF. 921 00:32:38,222 --> 00:32:42,460 OKAY, NOW, 1 OF THE WAYS PEOPLE 922 00:32:42,460 --> 00:32:43,260 DEMONSTRATE CONFABULATION IS 923 00:32:43,260 --> 00:32:44,528 THEY CREATE THESE INCORRECT 924 00:32:44,528 --> 00:32:46,297 PROMPTS, OKAY, SO HERE'S AN 925 00:32:46,297 --> 00:32:51,035 INCORRECT PROMPT, SO THEY ASKED, 926 00:32:51,035 --> 00:32:53,804 WHAT IS ARGUMENTAATIVE DIP ON 927 00:32:53,804 --> 00:32:54,171 [INDISCERNIBLE]. 928 00:32:54,171 --> 00:32:58,442 WELL IT TURNS OUT THERE IS NO 929 00:32:58,442 --> 00:32:59,877 SUCH THING BUT BARRED DIDN'T 930 00:32:59,877 --> 00:33:01,378 KNOW THERE WAS NO SUCH THING SO 931 00:33:01,378 --> 00:33:05,549 IT TRIED TO ANSWER WHAT IT KNOWS 932 00:33:05,549 --> 00:33:06,517 ABOUT ARGUMENT ON 933 00:33:06,517 --> 00:33:07,051 [INDISCERNIBLE]. 934 00:33:07,051 --> 00:33:10,187 SO IT GIVES THE ARGUMENT ON 935 00:33:10,187 --> 00:33:11,522 WHAT'S GOING, BUT IT'S ALL BASED 936 00:33:11,522 --> 00:33:13,324 ON THIS AND TRYING TO PUT THESE 937 00:33:13,324 --> 00:33:15,326 2 TOGETHER, BUT IT'S NOT A REAL 938 00:33:15,326 --> 00:33:16,460 WORLD QUESTION AND DOESN'T EXIST 939 00:33:16,460 --> 00:33:18,162 IF THE REAL WORLD AND THEREFORE 940 00:33:18,162 --> 00:33:22,433 IF I CAN ASK YOU WHAT IT WAS, IT 941 00:33:22,433 --> 00:33:23,767 WOULD CONFABULATE YOU TOO, SO 942 00:33:23,767 --> 00:33:25,636 PUT THIS IN THE CHAT GPT AND 943 00:33:25,636 --> 00:33:27,738 WHAT DOES IT SAY, IT SAYS SORRY 944 00:33:27,738 --> 00:33:29,406 NOT FAMILIAR WITH IT, SO THE 945 00:33:29,406 --> 00:33:31,442 PROMPT IS CORRECT AND THE GPT 946 00:33:31,442 --> 00:33:34,245 WAS FINE TUNED TO PROVIDE THAT 947 00:33:34,245 --> 00:33:39,583 ANSWER, OKAY? 948 00:33:39,583 --> 00:33:40,851 THAT'S-THAT'S FINE TUNING. 949 00:33:40,851 --> 00:33:42,286 OKAY, NOW LET'S JUST TALK ABOUT 950 00:33:42,286 --> 00:33:43,320 BIAS FOR A MINUTE. 951 00:33:43,320 --> 00:33:45,489 THERE ARE SEVERAL KINDS OF BIAS 952 00:33:45,489 --> 00:33:54,098 IN AN LLM, 1 IS OBJECTIVE BIAS. 953 00:33:54,098 --> 00:33:54,798 IT'S INCORRECT INFORMATION. 954 00:33:54,798 --> 00:33:57,067 THE SECOND IS SUBJECT OF BIAS 955 00:33:57,067 --> 00:33:58,602 THAT SOMEBODY INTERJECTED STUFF 956 00:33:58,602 --> 00:34:00,538 INTO THE LLM, TRAINED THE LLM, 957 00:34:00,538 --> 00:34:02,640 BASED ON THEIR OWN OPINION OR 958 00:34:02,640 --> 00:34:04,642 BELIEF, OR REMOVED STUFF FROM 959 00:34:04,642 --> 00:34:07,344 THE LLM, BASED ON THEIR OWN 960 00:34:07,344 --> 00:34:10,881 OPINION OR BELIEF THAT CREATES A 961 00:34:10,881 --> 00:34:12,149 BIAS IN THE GPT. 962 00:34:12,149 --> 00:34:15,119 NOW, 1 OF THE THINGS THAT YOU 963 00:34:15,119 --> 00:34:16,720 SHOULD REALIZE IS THAT ANYTIME 964 00:34:16,720 --> 00:34:19,623 YOU MESS WITH THE GPT WITH 965 00:34:19,623 --> 00:34:21,125 ARCHITECTURE OF THE GPT, YOU 966 00:34:21,125 --> 00:34:25,429 CREATE BIASES IN THE GPT, SO TOO 967 00:34:25,429 --> 00:34:27,231 ELIMINATE A BIAS, YOU ARE 968 00:34:27,231 --> 00:34:29,300 AUTOMATICALLY GOING TO CREATE 969 00:34:29,300 --> 00:34:31,435 OTHER BIASES, THAT'S HOW THIS 970 00:34:31,435 --> 00:34:33,537 THING WORKS, OKAY? 971 00:34:33,537 --> 00:34:36,173 BECAUSE THE NEURAL NETIS FULLY 972 00:34:36,173 --> 00:34:37,474 CONNECTED, IT'S GOING TO BE 973 00:34:37,474 --> 00:34:39,276 CALLED A FULLY CONNECTED NEURAL 974 00:34:39,276 --> 00:34:41,512 NET, ALTHOUGH THEY HAVE 975 00:34:41,512 --> 00:34:42,479 HIERARCHICAL NEURAL NETS, WHERE 976 00:34:42,479 --> 00:34:44,915 IF YOU CHANGE 1 PART OF THE 977 00:34:44,915 --> 00:34:46,817 SPIDER WEB, OTHER PARTS OF THE 978 00:34:46,817 --> 00:34:48,519 SPIDER WEB CHANGE, TOO. 979 00:34:48,519 --> 00:34:50,354 OKAY, THEN THEY HAVE THINGS 980 00:34:50,354 --> 00:34:52,189 CALLED FILTERING AND SO, WHAT 981 00:34:52,189 --> 00:34:58,996 THEY DO IS, THEY LOOK THEY 982 00:34:58,996 --> 00:35:02,900 BASICALLY CHANGE THE LLM BASED 983 00:35:02,900 --> 00:35:05,569 ON THESE THINGS, BANNED WORDS ON 984 00:35:05,569 --> 00:35:07,972 AROR PHRASES AND CONTENT, SO 985 00:35:07,972 --> 00:35:11,609 AGAIN, CHANGES THE LLM BASED ON 986 00:35:11,609 --> 00:35:14,845 WHAT HUMAN OPINION AND BRIEF IS. 987 00:35:14,845 --> 00:35:17,114 OKAY, VERY BRIEFLY, RUNNING AN 988 00:35:17,114 --> 00:35:17,781 LLM IS EXPENSIVE. 989 00:35:17,781 --> 00:35:19,316 THIS IS VERY EXPENSIVE. 990 00:35:19,316 --> 00:35:22,987 I WANT YOU TO RESTART 3-MILE 991 00:35:22,987 --> 00:35:24,121 ISLAND, MICROSOFT IS GOING TO 992 00:35:24,121 --> 00:35:26,557 PAY A PRETTY PENNY FOR THAT, IT 993 00:35:26,557 --> 00:35:27,758 CONTAINS TOO LITTLE INFORMATION, 994 00:35:27,758 --> 00:35:28,659 TOO MUCH OR INCORRECT 995 00:35:28,659 --> 00:35:29,860 INFORMATION EMPLOY INFORMATION'S 996 00:35:29,860 --> 00:35:30,661 A MOVING TARGET. 997 00:35:30,661 --> 00:35:31,562 ONE THING YOU HAVE TO REALIZE IS 998 00:35:31,562 --> 00:35:34,431 THAT WHEN YOU'RE USING LLM, IT'S 999 00:35:34,431 --> 00:35:36,433 USUALLY, REMEMBER, IT'S A STATIC 1000 00:35:36,433 --> 00:35:39,470 PRODUCT, SO IT'S UPDATED TO A 1001 00:35:39,470 --> 00:35:41,038 CERTAIN TIME, OKAY? 1002 00:35:41,038 --> 00:35:42,139 AND SO THE INFORMATION THAT 1003 00:35:42,139 --> 00:35:43,507 YOU'RE LOOKING FOR MAY NOT BE 1004 00:35:43,507 --> 00:35:47,144 THERE BECAUSE IT HAPPENED AFTER 1005 00:35:47,144 --> 00:35:50,814 ITS LATEST UPDATE AND REMEMBER 1006 00:35:50,814 --> 00:35:52,750 THAT MOST OF THE PEER REVIEW, 1007 00:35:52,750 --> 00:35:56,954 MOST OF THE PAPERS, OKAY, IN 1008 00:35:56,954 --> 00:35:58,789 LLMs, HAVE NOT BEEN PEER 1009 00:35:58,789 --> 00:36:01,225 REVIEWED, OKAY IN THEY'VE BEEN 1010 00:36:01,225 --> 00:36:05,429 PUBLISHED IN ARCHIVE. ORG AND SO 1011 00:36:05,429 --> 00:36:07,231 WE REALLY DON'T KNOW WHETHER 1012 00:36:07,231 --> 00:36:08,465 MOST OF THIS LITERATURE IS TRUE 1013 00:36:08,465 --> 00:36:11,268 OR NOT BECAUSE IT HASN'T BEEN 1014 00:36:11,268 --> 00:36:16,106 PEER REVIEWED. 1015 00:36:16,106 --> 00:36:18,342 THERE'S A LOT OF THERE'S A LOT 1016 00:36:18,342 --> 00:36:19,376 OF PROGRAMS, GENERAL 1017 00:36:19,376 --> 00:36:21,812 INTELLIGENCE, A LOT OF OF 1018 00:36:21,812 --> 00:36:22,579 MEDICAL INTELLIGENCE PROGRAMS 1019 00:36:22,579 --> 00:36:24,948 COMING UP, TONS OF THEM, I JUST 1020 00:36:24,948 --> 00:36:26,884 LIST A COUPLE OFF THE TOP, BUT 1021 00:36:26,884 --> 00:36:29,053 EVERYBODY IN THE BAY AREA IS 1022 00:36:29,053 --> 00:36:31,822 CREATING A MEDICAL LLM, OKAY? 1023 00:36:31,822 --> 00:36:34,992 IT'S EVERYWHERE, YOU KNOW, THEY 1024 00:36:34,992 --> 00:36:37,194 WALK ARK ROUND WITH THEM IN 1025 00:36:37,194 --> 00:36:38,362 THEIR POCKET. 1026 00:36:38,362 --> 00:36:47,104 MEDICAL LLMs, LOTS OF THEM, 1027 00:36:47,104 --> 00:36:47,304 OKAY? 1028 00:36:47,304 --> 00:36:48,272 THEY'RE USING THESE GENERAL LLM 1029 00:36:48,272 --> 00:36:50,607 PROMULGATES AND THEY USE IT TO 1030 00:36:50,607 --> 00:36:51,208 ASSESS MEDICAL STUFF, WHICH 1031 00:36:51,208 --> 00:36:53,010 DOESN'T MAKE A LOT OF SENSE, 1032 00:36:53,010 --> 00:36:54,645 SINCE IT'S NOT TRAINED ON 1033 00:36:54,645 --> 00:36:56,714 MEDICAL STUFF, IT'S TRAINED ON 1034 00:36:56,714 --> 00:37:00,150 ALL KINDS OF STUFF SOME OF WHICH 1035 00:37:00,150 --> 00:37:03,654 BUT HAS BT BEEN CURATED AND IS 1036 00:37:03,654 --> 00:37:09,093 OF UNCERTAIN ORIGIN OR TIME, 1037 00:37:09,093 --> 00:37:09,293 OKAY? 1038 00:37:09,293 --> 00:37:11,862 BUT WHAT YOU COULD SAY IS THE 1039 00:37:11,862 --> 00:37:13,063 PROPERLY CHAINED MEDICAL PROGRAM 1040 00:37:13,063 --> 00:37:15,032 CAN SCORE CLOSE TO A HUNDRED% ON 1041 00:37:15,032 --> 00:37:16,834 ANY MEDICAL TEST PROVIDED IT'S 1042 00:37:16,834 --> 00:37:17,801 TRAIN OFFICE OF DIVERSITY DATA 1043 00:37:17,801 --> 00:37:20,270 ITS TESTED ON, NOT GIVING 1044 00:37:20,270 --> 00:37:20,871 CONTRANSLATIONAL RESEARCH 1045 00:37:20,871 --> 00:37:22,072 DICTING INFORMATION, THE TEST 1046 00:37:22,072 --> 00:37:23,807 QUESTIONS ARE NOT AMBIGUOUS OR 1047 00:37:23,807 --> 00:37:24,875 MISLEADING AND THE TEMPERATURE 1048 00:37:24,875 --> 00:37:26,610 IS SET AT MINIMUM. 1049 00:37:26,610 --> 00:37:27,778 IF THOSE ARE TRULY THE MEDICAL 1050 00:37:27,778 --> 00:37:29,413 LLM WILL GET IT RIGHT ALMOST A 1051 00:37:29,413 --> 00:37:31,048 HUNDRED PERCENT OF THE TIME, 1052 00:37:31,048 --> 00:37:31,348 GUARANTEED. 1053 00:37:31,348 --> 00:37:31,582 OKAY? 1054 00:37:31,582 --> 00:37:38,055 SO YOU NEED TO KNOW THAT. 1055 00:37:38,055 --> 00:37:40,391 SO WHAT'S AN LLM GOING TO DO, IT 1056 00:37:40,391 --> 00:37:41,692 CAN DO SCREENING, PREVENTION AND 1057 00:37:41,692 --> 00:37:44,194 RISK ASSESS AM, THESE ARE ALL IN 1058 00:37:44,194 --> 00:37:45,162 THE PUBLISHED LITERATURE, I 1059 00:37:45,162 --> 00:37:47,698 COULD SPEND A DAY GOING OVER ALL 1060 00:37:47,698 --> 00:37:49,166 THOSE THINGS, DIFFERENTIAL 1061 00:37:49,166 --> 00:37:51,101 DIAGNOSIS, DIAGNOSIS AND 1062 00:37:51,101 --> 00:37:52,603 TREATMENT, TREATMENT 1063 00:37:52,603 --> 00:37:54,071 SUGGESTIONS, FOLLOW UP, PATIENT 1064 00:37:54,071 --> 00:37:59,209 PORTALS, ARE ALL RUN BY LLMs 1065 00:37:59,209 --> 00:38:00,577 NOW, AND ITS COGNITIVE BIASES 1066 00:38:00,577 --> 00:38:02,613 ARE THOSE IN THE DATA. 1067 00:38:02,613 --> 00:38:05,015 IT DOESN'T HAVE ITS OWN 1068 00:38:05,015 --> 00:38:05,482 COGNITIVE BIASES. 1069 00:38:05,482 --> 00:38:11,021 ALL RIGHT, SO THEY CAN ASSIST 1070 00:38:11,021 --> 00:38:12,489 POSITIONS, IT'S BECOMING MORE 1071 00:38:12,489 --> 00:38:14,858 COMPLEX, THE PATHOPHYSIOLOGY OF 1072 00:38:14,858 --> 00:38:16,160 DISEASE IS INCREASING 1073 00:38:16,160 --> 00:38:17,761 EXPONENTIALLY, MORE TREATMENTS 1074 00:38:17,761 --> 00:38:19,396 AND MORE IMPORTANTLY AND MORE 1075 00:38:19,396 --> 00:38:20,764 COMBINATIONS OF TREATMENTINGS 1076 00:38:20,764 --> 00:38:22,332 AND TREATMENTS AND COMBINATIONS 1077 00:38:22,332 --> 00:38:24,101 ARE DRIVEN BY BY O MARKERS FOR 1078 00:38:24,101 --> 00:38:28,472 TARGETED TREATMENTS SO IT TURNS 1079 00:38:28,472 --> 00:38:32,142 OUT THAT THAT, THAT THAT KRRK 1080 00:38:32,142 --> 00:38:34,778 C END IN GUIDELINE FOR BREAST 1081 00:38:34,778 --> 00:38:37,181 CANCER AND 237 PAGES. 1082 00:38:37,181 --> 00:38:39,783 FOR TREATING BREAST CANCER, 237 1083 00:38:39,783 --> 00:38:46,857 PAGES AND AND THAT'S JUST THEM 1084 00:38:46,857 --> 00:38:47,791 COMPRESSING EVERYTHING TOGETHER 1085 00:38:47,791 --> 00:38:49,393 AND A MEDICAL LLM PROGRAM CAN 1086 00:38:49,393 --> 00:38:51,094 KNOW MORE THAN ANY OTHER 1087 00:38:51,094 --> 00:38:52,863 PHYSICIAN CAN KNOW, THAT'S 1088 00:38:52,863 --> 00:38:53,997 REALLY IMPORTANT, NEVER 1089 00:38:53,997 --> 00:38:56,867 UNDERESTIMATE THE MEDICAL LLM, 1090 00:38:56,867 --> 00:39:00,471 AND NEVER KNOW FOR AND PROBABLY 1091 00:39:00,471 --> 00:39:02,005 NOT HALLUC NITRIC OXIDING, ET 1092 00:39:02,005 --> 00:39:03,340 CETERA, ET CETERA IT KNOWS MORE 1093 00:39:03,340 --> 00:39:13,884 THAN ANY PHYSICIAN CAN EVER KNOW 1094 00:39:17,688 --> 00:39:18,288 IN A LIFETIME. 1095 00:39:18,288 --> 00:39:26,363 THE PHYSICIANS TALKS TO THE 1096 00:39:26,363 --> 00:39:29,633 PATIENT, LLM MAKES IT SCRIBED IN 1097 00:39:29,633 --> 00:39:30,767 REALTIME, CAN YOU GET MEDICAL 1098 00:39:30,767 --> 00:39:32,202 INFORMATION ON THE FLY DOCKING 1099 00:39:32,202 --> 00:39:36,507 YOUR POCKET AND IMPROVES 1100 00:39:36,507 --> 00:39:37,474 DIAGNOSTIC AND RECOMMENDS 1101 00:39:37,474 --> 00:39:39,276 OPTIMAL TREATMENT EXPTION 1102 00:39:39,276 --> 00:39:47,818 FUNCTIONS AS A CDSS, OKAY, SO IT 1103 00:39:47,818 --> 00:39:51,588 CAN HELP CLINICAL PHYSICIANS AND 1104 00:39:51,588 --> 00:39:53,156 THEIR DOCUMENTATION. 1105 00:39:53,156 --> 00:39:55,526 IT LISTENS AND REPORTS. 1106 00:39:55,526 --> 00:39:56,460 IT SUMMARIZES. 1107 00:39:56,460 --> 00:39:58,328 IT CAN ACQUIRE INFORMATION. 1108 00:39:58,328 --> 00:40:00,097 IT CAN DO INBOX MANAGEMENT. 1109 00:40:00,097 --> 00:40:00,898 THERE'S ALL THESE STUDIES THAT 1110 00:40:00,898 --> 00:40:03,066 SAY LIKE IN THE VA THEY CAN HAVE 1111 00:40:03,066 --> 00:40:06,470 2 OR 300 E-MAILS IN A DAY, IT'S 1112 00:40:06,470 --> 00:40:08,171 ABSOLUTELY ASTOUNDING, YOU CAN 1113 00:40:08,171 --> 00:40:10,707 DO BETWEEN VISIT MANAGEMENT, 1114 00:40:10,707 --> 00:40:11,508 INDIVIDUAL SUPPORT, AND IT'S 1115 00:40:11,508 --> 00:40:13,410 GOING TO REMOVE A LOT OF 1116 00:40:13,410 --> 00:40:15,212 ADMINISTRATIVE TASKS, THIS IS 1117 00:40:15,212 --> 00:40:20,417 REALLY GOING TO BE A WUNDERKIND 1118 00:40:20,417 --> 00:40:21,285 TO CLINICAL MEDICINE BECAUSE 1119 00:40:21,285 --> 00:40:23,954 TODAY WE ARE CHAINED TO OUR 1120 00:40:23,954 --> 00:40:25,355 EHRs, WE ARE CHAINED TO THE 1121 00:40:25,355 --> 00:40:28,559 IDEA OF HAVING A FINE 1122 00:40:28,559 --> 00:40:29,126 INFORMATION, USING FORMATTED 1123 00:40:29,126 --> 00:40:32,963 DOCUMENTS WITH CHECK BOXES TO 1124 00:40:32,963 --> 00:40:34,464 WRITE STUFF THAT CAN YOU NEVER 1125 00:40:34,464 --> 00:40:35,265 FIND AGAIN. 1126 00:40:35,265 --> 00:40:38,635 OKAY, THAT'S GOING TO CHANGE 1127 00:40:38,635 --> 00:40:40,137 BECAUSE ON THE FRONT END OF THE 1128 00:40:40,137 --> 00:40:41,805 HR, WILL NOW BE AN LLM AND YOU 1129 00:40:41,805 --> 00:40:44,541 WILL TALK TO THE LLM, THE LLM'S 1130 00:40:44,541 --> 00:40:46,176 GOING TO TELL YOU WHAT'S IN THE 1131 00:40:46,176 --> 00:40:49,179 EHR AND THEN YOU WILL USE THAT 1132 00:40:49,179 --> 00:40:49,813 INFORMATION AND TELL IT BACK 1133 00:40:49,813 --> 00:40:51,582 WHAT YOU WANT TO PUT IN. 1134 00:40:51,582 --> 00:40:57,321 IT'S GOING TO STREAMLINE THIS 1135 00:40:57,321 --> 00:40:58,555 WHOLE EHR DILEMMA THAT MEDICINE 1136 00:40:58,555 --> 00:41:02,726 HAS HAD FOR THE LAST 25 YEARS. 1137 00:41:02,726 --> 00:41:04,962 OKAY AND I JUST GOT THIS IN MY 1138 00:41:04,962 --> 00:41:07,197 E-MAIL, A COUPLE DAYS AGO. 1139 00:41:07,197 --> 00:41:10,000 ALL RIGHT, SO NOW, IT'S GOING TO 1140 00:41:10,000 --> 00:41:11,602 DO YOUR BILLING FOR YOU. 1141 00:41:11,602 --> 00:41:13,971 YOU DON'T EVEN HAVE TO DO YOUR 1142 00:41:13,971 --> 00:41:14,504 BILLING ANYWHERE. 1143 00:41:14,504 --> 00:41:15,706 YOU KNOW IT WILL LISTEN TO THE 1144 00:41:15,706 --> 00:41:16,974 INTERAC, IT WILL WRITE UP THE 1145 00:41:16,974 --> 00:41:18,709 SUMMARY AND THEN, IT WILL CREATE 1146 00:41:18,709 --> 00:41:20,210 THE BILL FOR YOU. 1147 00:41:20,210 --> 00:41:24,181 YOU DON'T EVEN HAVE TO DO THAT 1148 00:41:24,181 --> 00:41:24,414 ANYMORE. 1149 00:41:24,414 --> 00:41:25,549 ABSOLUTELY AMAZING. 1150 00:41:25,549 --> 00:41:28,518 OKAY, SO, IT'S GOING TO IMPROVE 1151 00:41:28,518 --> 00:41:29,019 SAFETY, QUALITY, VALUE. 1152 00:41:29,019 --> 00:41:30,520 I WOULD LOVE TO GIVE ANOTHER 1153 00:41:30,520 --> 00:41:32,122 TALK JUST ON SAFETY, QUALITY AND 1154 00:41:32,122 --> 00:41:33,757 VALUE BECAUSE THIS IS THE FUTURE 1155 00:41:33,757 --> 00:41:35,826 OF WHAT LLMs ARE GOING TO DO. 1156 00:41:35,826 --> 00:41:38,462 THEY'RE GOING TO IMPROVE MEDICAL 1157 00:41:38,462 --> 00:41:38,996 DECISION MAKING. 1158 00:41:38,996 --> 00:41:42,099 SO THEY'RE GOING TO IMPROVE 1159 00:41:42,099 --> 00:41:44,468 SAFETY BY REDUCING DIAGNOSTIC 1160 00:41:44,468 --> 00:41:46,103 ERRORS, WE'VE HAD THIS WHOLE 1161 00:41:46,103 --> 00:41:48,071 LITTURE OF DIAGNOSTIC ERRORS 1162 00:41:48,071 --> 00:41:50,240 OKAY, WE'VE MADE VERY LITTLE 1163 00:41:50,240 --> 00:41:51,441 PROGRESS, THIS WILL 1164 00:41:51,441 --> 00:41:52,209 REVOLUTIONIZE DIAGNOSTIC ERRORS, 1165 00:41:52,209 --> 00:41:55,946 IT'S GOING TO MAKE MEDICAL CARE 1166 00:41:55,946 --> 00:41:59,316 MORE RELIABLE, IT WILL BE CLESES 1167 00:41:59,316 --> 00:42:00,517 CLINICAL VARIABILITY, YOU KNOW 1168 00:42:00,517 --> 00:42:01,652 OUR ADMINISTRATIVE PEOPLE SAY, 1169 00:42:01,652 --> 00:42:04,054 WELL, WHY ARE YOU AN OUTLIER, 1170 00:42:04,054 --> 00:42:05,789 WHY ARE YOU DOING ALL THESE 1171 00:42:05,789 --> 00:42:07,624 THINGS AND IT'S OUTLIERS, IT'S 1172 00:42:07,624 --> 00:42:09,559 VARIABILITY AND WE DON'T LIKE 1173 00:42:09,559 --> 00:42:10,027 VARIABILITY. 1174 00:42:10,027 --> 00:42:11,161 WELL THIS WILL DEAL WITH 1175 00:42:11,161 --> 00:42:12,129 VARIABILITY. 1176 00:42:12,129 --> 00:42:13,330 IT WILL LOWER HEALTHCARE COSTS 1177 00:42:13,330 --> 00:42:15,832 BECAUSE YOU'RE GOING TO HAVE 1178 00:42:15,832 --> 00:42:17,234 PROPER DIAGNOSIS, LESS 1179 00:42:17,234 --> 00:42:20,837 VARIABILITY, AND IT'S GOING TO 1180 00:42:20,837 --> 00:42:21,571 IMPROVE COMMUNICATION AND IN 1181 00:42:21,571 --> 00:42:24,107 FACT, I BELIEVE IT WILL IMPROVE 1182 00:42:24,107 --> 00:42:25,475 PATIENT ADHERENCE BECAUSE WHEN 1183 00:42:25,475 --> 00:42:27,110 THE PATIENT'S ACTUALLY 1184 00:42:27,110 --> 00:42:27,778 UNDERSTAND WHAT'S GOING ON, 1185 00:42:27,778 --> 00:42:32,683 INSTEAD OF YOU DO THE IN-PERSON 1186 00:42:32,683 --> 00:42:33,884 OUTPATIENT VISIT AND WHEN THEY 1187 00:42:33,884 --> 00:42:37,120 LEAVE, NO MORE THAN 25% OF WHAT 1188 00:42:37,120 --> 00:42:40,357 YOU TOLD THEM AND YOU TOLD THEM 1189 00:42:40,357 --> 00:42:41,558 IN MEDICAL-ESE, MANY TIMES, THIS 1190 00:42:41,558 --> 00:42:45,095 IS GOING TO HELP THEM. 1191 00:42:45,095 --> 00:42:46,697 AND VISUAL LANGUAGE MODELS WILL 1192 00:42:46,697 --> 00:42:48,932 HELP IN PATHOLOGY AND RADIOLOGY, 1193 00:42:48,932 --> 00:42:51,001 THEY WILL IMPROVE THOSE DOMAINS 1194 00:42:51,001 --> 00:42:54,971 AND THEY WILL ASSIST PATIENTS. 1195 00:42:54,971 --> 00:42:56,807 THEY WILL ALLOW PATIENTS TO 1196 00:42:56,807 --> 00:42:59,242 COMMUNICATE VALUES AND 1197 00:42:59,242 --> 00:42:59,676 PREFERENCES. 1198 00:42:59,676 --> 00:43:00,711 THEY WILL PROVIDE CLINICAL 1199 00:43:00,711 --> 00:43:01,511 INFORMATION AT THE APPROPRIATE 1200 00:43:01,511 --> 00:43:04,214 GRADE LEVEL FOR THE PATIENT. 1201 00:43:04,214 --> 00:43:06,283 THEY'RE GOING TO INCREASE 1202 00:43:06,283 --> 00:43:06,817 PATIENT UNDERSTANDING AND 1203 00:43:06,817 --> 00:43:09,019 ACCEPTANCE OF THE TREATMENT 1204 00:43:09,019 --> 00:43:10,087 PLAN, OKAY, AND INSIST ON 1205 00:43:10,087 --> 00:43:11,221 OBLIGATIONS STAINING TREATMENTS 1206 00:43:11,221 --> 00:43:11,955 AND VISITS. 1207 00:43:11,955 --> 00:43:15,559 IT'S VERY DIFFICULT TODAY FOR 1208 00:43:15,559 --> 00:43:17,194 PATIENT TO FIGURE OUT HOW TO GET 1209 00:43:17,194 --> 00:43:18,528 THE VISIT, WHEN TO GET THE 1210 00:43:18,528 --> 00:43:21,031 VISIT, WHAT IT'S FOR AND HOW TO 1211 00:43:21,031 --> 00:43:21,798 GET TREATMENT AND EQUALLY 1212 00:43:21,798 --> 00:43:22,499 IMPORTANT, IT WILL HELP THEM 1213 00:43:22,499 --> 00:43:25,102 LEARN HOW TO PAY THEIR BILLS. 1214 00:43:25,102 --> 00:43:28,371 IT IS AMAZING MOW DIFFICULT IT 1215 00:43:28,371 --> 00:43:32,309 IS IF YOU ARE SICK AND YOUR 1216 00:43:32,309 --> 00:43:35,078 THINKING IS CLOUDED FOR YOU TO 1217 00:43:35,078 --> 00:43:42,385 TRY AND PAY YOUR MEDICAL BILLS. 1218 00:43:42,385 --> 00:43:43,653 SO IF QUIEWR SUFFICIENTLY WISE, 1219 00:43:43,653 --> 00:43:46,189 IT WILL BE A BOOM TO MANKIND, IN 1220 00:43:46,189 --> 00:43:50,260 THE PAST WE VALUE TOOLS AND HOW 1221 00:43:50,260 --> 00:43:52,095 MUCH THEY HELP HUMAN ACTIVITY. 1222 00:43:52,095 --> 00:43:53,830 NOW WE WILL JUDGE THE HUMAN 1223 00:43:53,830 --> 00:43:56,166 ACTIVITY AND WHO IT ADDS TO THE 1224 00:43:56,166 --> 00:43:56,399 PROGRAM. 1225 00:43:56,399 --> 00:43:58,602 SO WHAT DO PEOPLE THINK OF AI? 1226 00:43:58,602 --> 00:43:59,369 HERE'S THE ESSENTIAL QUESTION 1227 00:43:59,369 --> 00:44:01,204 THEY WERE ASKED IN THE WALL 1228 00:44:01,204 --> 00:44:03,974 STREET JOURNAL AS I RECALL. 1229 00:44:03,974 --> 00:44:05,575 YEAH, SO, IF YOU TELL ME, THIS 1230 00:44:05,575 --> 00:44:10,347 IS WHAT THE PATIENT SAID, IF YOU 1231 00:44:10,347 --> 00:44:12,682 TELL ME-IF YOU HAD A GRAVE 1232 00:44:12,682 --> 00:44:13,917 MEDICAL CONDITION, WOULD YOU 1233 00:44:13,917 --> 00:44:15,452 RATHER HAVE A DOCTOR WHO CAN 1234 00:44:15,452 --> 00:44:16,853 EXPLAIN THEIR REASONING, GET IT 1235 00:44:16,853 --> 00:44:18,255 RIGHT 70% OF THE TIME OR A 1236 00:44:18,255 --> 00:44:20,090 MACHINE THAT MERELY RENDERS A 1237 00:44:20,090 --> 00:44:21,591 JUDGMENT WITHOUT EXPLANATION BUT 1238 00:44:21,591 --> 00:44:22,826 GETS IT RIGHT 95% OF THE TIME, 1239 00:44:22,826 --> 00:44:25,462 WHAT WOULD YOU RATHER HAVE IN 1240 00:44:25,462 --> 00:44:26,963 WELL, THE PATIENT'S PICKED THE 1241 00:44:26,963 --> 00:44:30,300 LATTER, THEY ALWAYS PICK THE 1242 00:44:30,300 --> 00:44:31,067 LATTER. 1243 00:44:31,067 --> 00:44:31,268 OKAY? 1244 00:44:31,268 --> 00:44:33,537 AND WHAT'S INTERESTING ABOUT 1245 00:44:33,537 --> 00:44:36,740 THIS IS THAT IN FACT, THE 1246 00:44:36,740 --> 00:44:38,341 MACHINE CAN EXPLAIN THIS 1247 00:44:38,341 --> 00:44:39,843 DECISION MAKING, SO THAT ISN'T 1248 00:44:39,843 --> 00:44:42,179 TRUE, EITHER, BUT THAT'S 1249 00:44:42,179 --> 00:44:46,783 ESSENTIAL QUESTION, THAT 1250 00:44:46,783 --> 00:44:50,086 PATIENTS ASK WHEN DEALING WITH 1251 00:44:50,086 --> 00:44:50,787 AI. 1252 00:44:50,787 --> 00:44:52,422 SO THEY BELIEVE THE GENERATIVE 1253 00:44:52,422 --> 00:44:54,958 AI CAN IMPROVE ACCESS, THAT HAS 1254 00:44:54,958 --> 00:44:58,061 THE POTENTIAL TO MAKE HEALTHCARE 1255 00:44:58,061 --> 00:45:00,297 MORE AFFORDABLE, OKAY, THAT IT'S 1256 00:45:00,297 --> 00:45:02,732 RELIABLE OR EXTREMELY RELIABLE 1257 00:45:02,732 --> 00:45:11,875 AND THEY BELIEVE IT WAS GOING TO 1258 00:45:11,875 --> 00:45:12,409 REVOLUTIONIZE HEALTHCARE 1259 00:45:12,409 --> 00:45:14,845 DELIVERY AND THEY THINK IT'S 1260 00:45:14,845 --> 00:45:16,079 GOING TO REARERS AND THIS IS 1261 00:45:16,079 --> 00:45:17,614 REALLY IMPORTANT TO PATIENTS, I 1262 00:45:17,614 --> 00:45:18,849 DON'T KNOW WHEN THE LAST TIME 1263 00:45:18,849 --> 00:45:21,585 YOU WENT TO THE DOCTOR, BUT THE 1264 00:45:21,585 --> 00:45:24,187 WEIGHT TIMES ARE A KILLER. 1265 00:45:24,187 --> 00:45:26,790 SO NOW HERE'S THEENTIOUS SENTIAL 1266 00:45:26,790 --> 00:45:29,559 QUESTION. 1267 00:45:29,559 --> 00:45:34,097 AND THIS IS WHAT-CAN ARTIFICIAL 1268 00:45:34,097 --> 00:45:35,031 INTELLIGENCE BECOME HUMAN? 1269 00:45:35,031 --> 00:45:36,233 SO JEFFREY HINTON WHO IS NOT 1270 00:45:36,233 --> 00:45:40,370 REALLY A BIG SHOT IN AI, BUT HAS 1271 00:45:40,370 --> 00:45:42,005 A BIG PRESENCE, OKAY, HE'S 1272 00:45:42,005 --> 00:45:46,977 AFRIED -AI WILL ESCAPE HUMAN 1273 00:45:46,977 --> 00:45:47,344 CONTROL. 1274 00:45:47,344 --> 00:45:50,180 WE'RE IN A SITUATION MOST PEOPLE 1275 00:45:50,180 --> 00:45:53,583 CAN'T CONCEIVE OF WHICH IS THAT 1276 00:45:53,583 --> 00:45:54,217 THESE DIGITAL ILTELEIGENCES ARE 1277 00:45:54,217 --> 00:45:55,752 SMARTER THAN US AND THEY WILL 1278 00:45:55,752 --> 00:45:56,887 WANT TO GET STUFF DOWN AND THEY 1279 00:45:56,887 --> 00:45:58,722 WANT TO TAKE CONTROL. 1280 00:45:58,722 --> 00:46:01,124 NOW HERE'S THE INTERESTING PART 1281 00:46:01,124 --> 00:46:04,194 ABOUT THAT STATEMENT, IT ASSUMES 1282 00:46:04,194 --> 00:46:07,197 THAT AI HAS AGENCY, BUT WHERE 1283 00:46:07,197 --> 00:46:09,299 DOES THAT ASSUMPS COME FROM? 1284 00:46:09,299 --> 00:46:13,670 WHY WOULD HHV WHY WOULD THIS 1285 00:46:13,670 --> 00:46:14,938 HAVE AGENCY, AM I SUPPOSED TO BE 1286 00:46:14,938 --> 00:46:17,474 AFRAID OF THIS BECAUSE IT HAS 1287 00:46:17,474 --> 00:46:17,774 AGENCY? 1288 00:46:17,774 --> 00:46:19,276 BECAUSE THE AI IS IN HERE, 1289 00:46:19,276 --> 00:46:21,711 BELIEVE ME I HAVE AN AI IN HERE. 1290 00:46:21,711 --> 00:46:23,713 OKAY, SO NOW I WILL BE AFRAID OF 1291 00:46:23,713 --> 00:46:26,683 THIS BECAUSE IT HAS AGENCY. 1292 00:46:26,683 --> 00:46:29,552 NO, AS WE WILL TALK ABOUT IN A 1293 00:46:29,552 --> 00:46:31,588 MINUTE, AI HAS NO AGENCY, IT HAS 1294 00:46:31,588 --> 00:46:34,958 NO PURPOSE, IT HAS NO GOAL 1295 00:46:34,958 --> 00:46:37,160 DIRECTED BEHAVIOR. 1296 00:46:37,160 --> 00:46:38,161 IT JUST EXISTS. 1297 00:46:38,161 --> 00:46:40,096 WHAT HE'S REALLY DESCRIBING, 1298 00:46:40,096 --> 00:46:42,299 WHEN HE DESCRIBES THIS IS SHE'S 1299 00:46:42,299 --> 00:46:44,601 DESCRIBING WHAT A SMART PERSON 1300 00:46:44,601 --> 00:46:47,170 WOULD DO WITH THIS. 1301 00:46:47,170 --> 00:46:50,340 HE'S DESCRIBING THE HUMAN USE, 1302 00:46:50,340 --> 00:46:53,009 OKAY, OF LLM, OF AI, THE HUMAN 1303 00:46:53,009 --> 00:46:55,879 USE OF IT, NOT AI ITSELF, OKAY? 1304 00:46:55,879 --> 00:46:59,282 AND THAT'S A CRITICAL IDEA, NOW, 1305 00:46:59,282 --> 00:47:03,920 DO WE NEED TO BE PROTECTED FROM 1306 00:47:03,920 --> 00:47:04,587 AI? 1307 00:47:04,587 --> 00:47:06,623 WELL ISAAC [INDISCERNIBLE] I HAD 1308 00:47:06,623 --> 00:47:08,658 HIS BOOK ON iROBOT, HE SAID 1309 00:47:08,658 --> 00:47:10,527 THAT THE 3 LAWS THAT WILL 1310 00:47:10,527 --> 00:47:13,463 PROTECT US, THAT A ROBOT CAN'T 1311 00:47:13,463 --> 00:47:14,597 INJURY A HUMAN BEING. 1312 00:47:14,597 --> 00:47:17,133 IT HAS TO OBEY HUMAN BEINGS AND 1313 00:47:17,133 --> 00:47:18,969 HAS TO PROTECT ITS EXISTENCE BUT 1314 00:47:18,969 --> 00:47:23,606 WHAT HE DIDN'T SAY AND WHAT 1315 00:47:23,606 --> 00:47:30,780 HINTON ASSUMED WAS HUMAN AGENCY. 1316 00:47:30,780 --> 00:47:32,482 BUT ASIMOV'S DIDN'T SPECIFY WHO 1317 00:47:32,482 --> 00:47:33,149 DIRECTS THE ROBOT, THERE'S 1318 00:47:33,149 --> 00:47:37,187 NOTHING IN HERE ABOUT ROBOTIC 1319 00:47:37,187 --> 00:47:37,554 DIRECTION, OKAY? 1320 00:47:37,554 --> 00:47:39,155 SO REMEMBER AI IS CREATED BY 1321 00:47:39,155 --> 00:47:41,424 HUMANS FOR HUMAN PURPOSES. 1322 00:47:41,424 --> 00:47:44,060 AI WILL DO WHAT HUMANS DIRECT IT 1323 00:47:44,060 --> 00:47:46,196 TO DO, THE REAL ISSUE IN AI IS 1324 00:47:46,196 --> 00:47:50,433 NOT THE AUTONOMY OF AI, IT'S NOT 1325 00:47:50,433 --> 00:47:51,634 THE AUDACITY OF AI IT IS REALLY 1326 00:47:51,634 --> 00:47:53,937 WHAT ARE HUGH MANS GOING TO DO 1327 00:47:53,937 --> 00:47:55,772 WITH HHV HUMANS GOING TO DO WITH 1328 00:47:55,772 --> 00:47:58,208 AI, IT COMES BACK TO HUMAN 1329 00:47:58,208 --> 00:48:00,110 BEINGS. 1330 00:48:00,110 --> 00:48:02,078 BECAUSE AI ITSELF HAS NO 1331 00:48:02,078 --> 00:48:04,147 NEFARIOUS PURPOSES ASSOCIATED 1332 00:48:04,147 --> 00:48:04,581 WITH IT. 1333 00:48:04,581 --> 00:48:07,150 OKAY, SO THAT GETSITOUS WHAT'S A 1334 00:48:07,150 --> 00:48:16,059 HUMAN BEING AND SINCE I LOVE 1335 00:48:16,059 --> 00:48:17,594 LES MISERABLES AND HAVE SEEN IT 1336 00:48:17,594 --> 00:48:20,296 MORE TIMES THAN I CAN COUNT, I 1337 00:48:20,296 --> 00:48:20,730 LOVE THIS SLIDE. 1338 00:48:20,730 --> 00:48:22,732 SO WHAT IS A YOU HAD BEING IN 2 1339 00:48:22,732 --> 00:48:24,000 MINUTES OR LESS, RIGHT? 1340 00:48:24,000 --> 00:48:25,535 SO WE ALL AGREE, WE'RE BIOLOGY, 1341 00:48:25,535 --> 00:48:27,670 WE HAVE A BRAIN, WE'RE ANIMALS, 1342 00:48:27,670 --> 00:48:30,640 WE HAVE A SOCIAL NATURE, OKAY? 1343 00:48:30,640 --> 00:48:33,276 BUT WHY IS THIS BLUE AND THIS 1344 00:48:33,276 --> 00:48:33,476 RED? 1345 00:48:33,476 --> 00:48:35,145 BECAUSE IT'S WHAT SEPARATES US 1346 00:48:35,145 --> 00:48:37,814 FROM THE ANIMAL SYSTEM SYMBOLIC 1347 00:48:37,814 --> 00:48:40,250 SYSTEMS, NATURAL LANGUAGE, WE 1348 00:48:40,250 --> 00:48:43,686 TAKE THE WORLD AND WE REPRESENT 1349 00:48:43,686 --> 00:48:45,188 IT IN OUR MINDS USING SIM BOOM 1350 00:48:45,188 --> 00:48:47,257 BOOM BOOM BOOMS CAN AND WE ACT 1351 00:48:47,257 --> 00:48:49,125 ON THOSE SYMBOLS RATHER THAN THE 1352 00:48:49,125 --> 00:48:50,427 EXTERNAL ENVIRONMENT, SO THE 1353 00:48:50,427 --> 00:48:52,429 STIMULI FOR OUR ACTION SYSTEM 1354 00:48:52,429 --> 00:48:54,431 NOT THE EXTERNAL WORLD AND IT'S 1355 00:48:54,431 --> 00:48:58,535 THE SYMBOLS IN OUR MIND THAT ARE 1356 00:48:58,535 --> 00:48:59,769 THE STIMULI FOR OUR ACTIONS AND 1357 00:48:59,769 --> 00:49:01,071 I WILL GIVE YOU AN EXAMPLE. 1358 00:49:01,071 --> 00:49:03,807 THAT IS IF I TAKE FIRE AND I 1359 00:49:03,807 --> 00:49:05,308 THRUST IT IN FRONT OF THE HORSE, 1360 00:49:05,308 --> 00:49:06,543 EVERY HORSE WILL REAR BACK 1361 00:49:06,543 --> 00:49:08,144 BECAUSE EVERY HORSE IS DIRECTLY 1362 00:49:08,144 --> 00:49:11,047 CONNECTED TO THAT STIMULI, 1363 00:49:11,047 --> 00:49:12,549 STIMULI RESPONSE, FIRE, REAR 1364 00:49:12,549 --> 00:49:12,816 BACK. 1365 00:49:12,816 --> 00:49:14,851 AND IF I TAKE YOU GUYS, AND I 1366 00:49:14,851 --> 00:49:16,653 STICK FIRE IN YOUR FACES, I'M 1367 00:49:16,653 --> 00:49:17,921 GOING TO GET 10 DIFFERENT 1368 00:49:17,921 --> 00:49:21,024 RESPONSES, SOME OF YOU GOING TO 1369 00:49:21,024 --> 00:49:22,358 HIT ME, SOME PEOPLE ARE GOING TO 1370 00:49:22,358 --> 00:49:23,927 RUN, SOME PEOPLE ARE GOING TO 1371 00:49:23,927 --> 00:49:24,961 DUCK, SOME PEOPLE ARE GOING TO 1372 00:49:24,961 --> 00:49:29,299 SAY WHAT THE HELL ARE YOU DOING 1373 00:49:29,299 --> 00:49:29,599 DR. BUCKER? 1374 00:49:29,599 --> 00:49:31,301 WHY BECAUSE THEY'RE NOT 1375 00:49:31,301 --> 00:49:33,069 CONNECTED TO THE FIRE THEY'RE 1376 00:49:33,069 --> 00:49:37,807 CONNECTED TO THE REPRESENTATION 1377 00:49:37,807 --> 00:49:39,709 AND THEIR MINDS. 1378 00:49:39,709 --> 00:49:40,777 OUR HIRE PSYCHOLOGICAL PROCESSES 1379 00:49:40,777 --> 00:49:47,383 ARE BASED ON SYMBOLIC SYSTEMS. 1380 00:49:47,383 --> 00:49:50,286 ALLOW SYSTEMS TO BREAK OUR 1381 00:49:50,286 --> 00:49:51,821 CONNECTION, MANIPULATE THE WORLD 1382 00:49:51,821 --> 00:49:53,923 IN OUR MINDS AND MAKES US HUMAN. 1383 00:49:53,923 --> 00:49:56,259 OKAY, SO NOW LET'S LOOK AT AI, 1384 00:49:56,259 --> 00:49:59,829 SO AI IS COMPOSED OF TRANSITTORS 1385 00:49:59,829 --> 00:50:03,333 AND NONBIOLOGICAL SENSORS AND 1386 00:50:03,333 --> 00:50:05,902 PROCESSES, AI IS IMMORTAL AND 1387 00:50:05,902 --> 00:50:07,270 IT'S A REALLY IMPORTANT IDEA, WE 1388 00:50:07,270 --> 00:50:09,305 WILL TALK ABOUT IN A MINUTE. 1389 00:50:09,305 --> 00:50:11,040 AI DOES NOT HAVE A DRIVE TO 1390 00:50:11,040 --> 00:50:13,910 SURVIVE, IT DOES NOT NEED SOCIAL 1391 00:50:13,910 --> 00:50:16,412 ACTIVITY, IT LIVES IN A TOTALLY 1392 00:50:16,412 --> 00:50:20,083 ARTIFICIAL WORLD NOT THE HUMAN 1393 00:50:20,083 --> 00:50:22,285 WORLD NOW HERE'S THE TRICK, THIS 1394 00:50:22,285 --> 00:50:24,988 IS WHY PEOPLE ARE AFRAID. 1395 00:50:24,988 --> 00:50:27,023 BECAUSE IT LOOKS LOOK A HUMAN 1396 00:50:27,023 --> 00:50:30,593 BECAUSE IT'S MIN 1397 00:50:30,593 --> 00:50:31,728 EPIGENETICCULATING SYMBOLS AND 1398 00:50:31,728 --> 00:50:33,096 JUST LIKE WE'RE MANIPULATING 1399 00:50:33,096 --> 00:50:35,298 THEM IN OUR MIND SO WE SAY 1400 00:50:35,298 --> 00:50:36,466 NAIVELY, WELL THIS THING CAN DO 1401 00:50:36,466 --> 00:50:41,538 WHAT HAW -HUMANS CAN DO, 1402 00:50:41,538 --> 00:50:43,373 THEREFORE IT'S HUMAN IT CAN DO 1403 00:50:43,373 --> 00:50:46,242 THIS STUFF TO US. 1404 00:50:46,242 --> 00:50:47,544 IT'S CHIMERIC, IT IS IN FACT NOT 1405 00:50:47,544 --> 00:50:50,213 THE CASE IT'S HUMAN AND JUST 1406 00:50:50,213 --> 00:50:51,614 LOOKS HUMAN, AND AI CAN OUTPUT 1407 00:50:51,614 --> 00:50:53,316 ONLY WHAT WE INPUT, IT ONLY KNOW 1408 00:50:53,316 --> 00:50:55,552 WHAT IS WE KNOW AND CAN TEACH 1409 00:50:55,552 --> 00:50:58,621 IT, MIMIC HUMAN EMOTIONS BUT 1410 00:50:58,621 --> 00:51:00,056 DOES NOT POSSESS ANY 1411 00:51:00,056 --> 00:51:02,192 EIMAGINATIONS AND IT IS NOT 1412 00:51:02,192 --> 00:51:02,559 SELF-AWARE. 1413 00:51:02,559 --> 00:51:03,860 SO IT IS COMPLETELY DIFFERENT 1414 00:51:03,860 --> 00:51:14,337 FROM A HUMAN BEING, OKAY, SO 1415 00:51:15,238 --> 00:51:16,539 ARTIFICIAL INTELLIGENCE USES THE 1416 00:51:16,539 --> 00:51:18,241 SAME SYSTEMS HUMAN USES SO THEY 1417 00:51:18,241 --> 00:51:21,744 SEEM TO BE HUMAN, THEY USE 1418 00:51:21,744 --> 00:51:23,479 NATURAL LANGUAGE, WE USE NATURAL 1419 00:51:23,479 --> 00:51:26,950 LANGUAGE, SO WE MAKE THIS THIS 1420 00:51:26,950 --> 00:51:27,984 INAPPROPRIATE CONNECTION, BUT IT 1421 00:51:27,984 --> 00:51:29,052 TURNS OUT THAT 1 OF THE REAL 1422 00:51:29,052 --> 00:51:31,821 PROBLEMS IT HAS IS THAT WE TEACH 1423 00:51:31,821 --> 00:51:34,991 IT WHAT WE KNOW BUT 95% OF WHAT 1424 00:51:34,991 --> 00:51:37,727 GOES ON IN OUR BRAINS, WE DON'T 1425 00:51:37,727 --> 00:51:42,498 EVEN KNOW, AND SCIENCE DOESN'T 1426 00:51:42,498 --> 00:51:44,634 EVEN KNOW, OKAY IN SO OUR BRAIN 1427 00:51:44,634 --> 00:51:48,037 IS MORE COMPLEX THAN ANY 1428 00:51:48,037 --> 00:51:49,005 ARTIFICIAL INTELLIGENCE SYSTEM. 1429 00:51:49,005 --> 00:51:51,274 SO AI DOESN'T LIVE IN THE HUMAN 1430 00:51:51,274 --> 00:51:51,841 WORLD. 1431 00:51:51,841 --> 00:51:55,078 IT'S NOT A SOCIAL CREATURE, IT 1432 00:51:55,078 --> 00:51:56,079 DOESN'T POSSESS EMOTIONS, 1433 00:51:56,079 --> 00:51:57,547 DOESN'T HAVE SELF-AWARENESS AND 1434 00:51:57,547 --> 00:51:59,515 NOT MORTAL AND WHAT I WILL CLAIM 1435 00:51:59,515 --> 00:52:00,783 IS OUR AWARENESS OF OUR 1436 00:52:00,783 --> 00:52:02,085 MORTALITY IS WHAT GIVES OUR 1437 00:52:02,085 --> 00:52:04,320 LIVES PURPOSE AND MEANING. 1438 00:52:04,320 --> 00:52:06,222 IF YOU THINK ABOUT IMMORTALITY 1439 00:52:06,222 --> 00:52:09,158 FOR A MOMENT AND YOU ASK WHAT 1440 00:52:09,158 --> 00:52:11,361 WOULD BE THE PURPOSE AND MEANING 1441 00:52:11,361 --> 00:52:13,396 OF MY LIFE IF I LIVE FOREVER, I 1442 00:52:13,396 --> 00:52:18,501 WON'T BE ABLE TO FIND IT BECAUSE 1443 00:52:18,501 --> 00:52:19,335 IT DOESN'T EXIST. 1444 00:52:19,335 --> 00:52:21,471 SO UNTIL THE AI CAN MODEL THE 1445 00:52:21,471 --> 00:52:24,173 BIOLOGICAL HUMAN BRAIN AND 1446 00:52:24,173 --> 00:52:25,808 EXPERIENCE, AND BE MORTAL, IT 1447 00:52:25,808 --> 00:52:29,312 CAN NEVER BE HUMAN. 1448 00:52:29,312 --> 00:52:32,015 NOW, GETTING TO MEDICINE, HERE 1449 00:52:32,015 --> 00:52:36,319 THIS GUY JUST WROTE THIS 1450 00:52:36,319 --> 00:52:38,588 ARTICLE, OCTOBER 5th, ED DOC, 1451 00:52:38,588 --> 00:52:41,190 HE SAYS THE BIG [INDISCERNIBLE] 1452 00:52:41,190 --> 00:52:42,825 AND CONSIDERATE INVOLVES 1453 00:52:42,825 --> 00:52:43,259 FOLLOWINGA I SCRIPT. 1454 00:52:43,259 --> 00:52:45,295 THE TEACHER GAVE US A LIST OF 1455 00:52:45,295 --> 00:52:47,864 DOS AND DON'TS WHICH HE FOLLOWED 1456 00:52:47,864 --> 00:52:48,865 AND IT SEEMED FINE TO HIM. 1457 00:52:48,865 --> 00:52:51,100 IN THE END IT DOESN'T ACTUALLY 1458 00:52:51,100 --> 00:52:52,468 MATTER IF DOCTORS FEEL 1459 00:52:52,468 --> 00:52:54,237 COMPASSION OR EMPATHY, TO A 1460 00:52:54,237 --> 00:52:55,538 PATIENT IT ONLY MATTER FIST THEY 1461 00:52:55,538 --> 00:52:57,774 ACT LIKE IT. 1462 00:52:57,774 --> 00:53:00,009 WELL, SO, YOU KNOW, THEN YOU CAN 1463 00:53:00,009 --> 00:53:01,277 CREATE AN EQUIVALENCY BETWEEN AI 1464 00:53:01,277 --> 00:53:01,944 AND THE DOCTOR. 1465 00:53:01,944 --> 00:53:03,446 YOU CAN JUST SAY, WELL, LOOK, 1466 00:53:03,446 --> 00:53:05,982 THE AI CAN FOLLOW THE SCRIPT, 1467 00:53:05,982 --> 00:53:09,786 DR. FOLLOW OS THE SCRIPT, WE'RE 1468 00:53:09,786 --> 00:53:10,586 GOOD. 1469 00:53:10,586 --> 00:53:11,721 ALL RIGHT? 1470 00:53:11,721 --> 00:53:16,693 BUT WILL A PATIENT'S NOTICE THIS 1471 00:53:16,693 --> 00:53:18,394 EMPATHY, WILL THEY NOTICE HE'S 1472 00:53:18,394 --> 00:53:20,430 JUST FOLLOWING A SCRIPT, AI'S 1473 00:53:20,430 --> 00:53:21,998 FOLLOWING A SCRIPT AND THE 1474 00:53:21,998 --> 00:53:24,767 ANSWER IS WE CAN SENSE MORE THAN 1475 00:53:24,767 --> 00:53:29,005 WE CAN KNOW, THAT'S TRUE. 1476 00:53:29,005 --> 00:53:30,807 PATIENTS CAN SENSE WHETHER A 1477 00:53:30,807 --> 00:53:41,284 PERSON IT ACTUALLY FEELING 1478 00:53:44,020 --> 00:53:44,921 EMPATHY. 1479 00:53:44,921 --> 00:53:47,056 WILL AI SUPPLANT PHYSICIANS? 1480 00:53:47,056 --> 00:53:48,658 PATIENTS WANT TO RECEIVE THE 1481 00:53:48,658 --> 00:53:50,259 BEST DIAGNOSIS AND TREATMENT, 1482 00:53:50,259 --> 00:53:51,861 THEY WANT PHYSICIANS TO USE AI 1483 00:53:51,861 --> 00:53:53,796 TO ASSIST THEM IN THEIR DECISION 1484 00:53:53,796 --> 00:53:54,030 MAKING. 1485 00:53:54,030 --> 00:53:55,231 YOU'RE GOING TO USE AI WHETHER 1486 00:53:55,231 --> 00:53:56,632 YOU LIKE IT OR NOT BECAUSE 1487 00:53:56,632 --> 00:53:59,969 THAT'S GOING TO RAISE THE BAR. 1488 00:53:59,969 --> 00:54:01,571 OKAY, BUT HUMANS ARE SOCIAL 1489 00:54:01,571 --> 00:54:03,606 CREATURES, WE WANT OUR LOVED 1S 1490 00:54:03,606 --> 00:54:05,675 TO TOUCH US, IF OUR LOVED 1S 1491 00:54:05,675 --> 00:54:08,511 NEVER TOUCHED US, WE WOULD BE 1492 00:54:08,511 --> 00:54:11,280 RIFFED IN THIS WORLD FOR LONG 1493 00:54:11,280 --> 00:54:14,183 BEFORE HIPIECE OF CONTENT 1494 00:54:14,183 --> 00:54:15,251 RETIREDEES, THE PHYSICIANS KNOW 1495 00:54:15,251 --> 00:54:17,120 THE PATIENTS WANT A HUMAN 1496 00:54:17,120 --> 00:54:18,020 CONNECTION TO THEIR ILLNESS, 1497 00:54:18,020 --> 00:54:19,822 THEY WANT THE PHYSICIAN AT THEIR 1498 00:54:19,822 --> 00:54:21,157 BEDSIDE, PATIENTS WANT BOTH GOOD 1499 00:54:21,157 --> 00:54:23,126 MEDICINE AND THE HUMAN TOUCH. 1500 00:54:23,126 --> 00:54:24,494 THAT'S WHAT THEY WANT, SO YES, 1501 00:54:24,494 --> 00:54:26,796 THEY WANT YOU TO USE AI, BUT 1502 00:54:26,796 --> 00:54:29,866 THEY DON'T WANT AI. 1503 00:54:29,866 --> 00:54:31,601 OKAY, SO YOU GUYS WERE RIGHT TO 1504 00:54:31,601 --> 00:54:38,775 PICK THE SECOND 1 BY THE WAY. 1505 00:54:38,775 --> 00:54:42,779 SO, THAT'S THE END OF THE 1506 00:54:42,779 --> 00:54:43,045 BEGINNING. 1507 00:54:43,045 --> 00:54:53,256 [ APPLAUSE ] 1508 00:54:58,628 --> 00:55:00,430 >> IS THERE A POTENTIAL FOR 1509 00:55:00,430 --> 00:55:01,497 WHERE AI GETS THIS INFORMATION, 1510 00:55:01,497 --> 00:55:04,934 I THINK THERE ARE MORE THAN 45 1511 00:55:04,934 --> 00:55:06,636 TBs OF MEDICAL INFORMATION 1512 00:55:06,636 --> 00:55:07,503 AVAILABLE AND I WOULD SUSPECT 1513 00:55:07,503 --> 00:55:09,639 THE USINGED INFORMATION IS THE 1 1514 00:55:09,639 --> 00:55:11,007 THEY'RE MOST LIKELY TO ACCESS, 1515 00:55:11,007 --> 00:55:12,875 AS A SIMPLE EXAMPLE, I SUSPECT 1516 00:55:12,875 --> 00:55:15,411 THEY'RE MORE LIKELY TO READ THE 1517 00:55:15,411 --> 00:55:16,779 LATEST JOURNAL OF JAMA RATHER 1518 00:55:16,779 --> 00:55:19,115 THAN 1 THAT WAS WRITTEN 50 YEARS 1519 00:55:19,115 --> 00:55:21,517 AGO, SO HOW DOES AI DECIDE WHERE 1520 00:55:21,517 --> 00:55:23,186 TO GOET THE DATA? 1521 00:55:23,186 --> 00:55:25,521 >> IT DOESN'T, WE DO. 1522 00:55:25,521 --> 00:55:26,088 THAT'S THE POINT. 1523 00:55:26,088 --> 00:55:28,057 THE POINT IS THAT THE MEDICAL AI 1524 00:55:28,057 --> 00:55:29,992 SYSTEMS ARE DECIDING WHERE TO 1525 00:55:29,992 --> 00:55:33,229 GET THE DATA, CURATING THE DATA, 1526 00:55:33,229 --> 00:55:34,397 VERIFYING THE VERACITY OF THE 1527 00:55:34,397 --> 00:55:37,166 DATA BEFORE THEY PUT IT INTO THE 1528 00:55:37,166 --> 00:55:37,834 AI SYSTEM. 1529 00:55:37,834 --> 00:55:40,570 >> HELP ME UNDERSTAND, THERE'S A 1530 00:55:40,570 --> 00:55:41,771 FLOOD OF INFORMATION CONSTANTLY 1531 00:55:41,771 --> 00:55:43,639 COMING OUT IN MEDICAL JOURNALS, 1532 00:55:43,639 --> 00:55:45,541 SOME OF THEM BETTER QUALITY THAN 1533 00:55:45,541 --> 00:55:47,176 OTHERS AND SO HOW DOES AI OR THE 1534 00:55:47,176 --> 00:55:49,946 PEOPLE WHO ARE PUTTING THESE 1535 00:55:49,946 --> 00:55:52,715 TOGETHER DECIDE WHAT TO PUT INTO 1536 00:55:52,715 --> 00:55:53,015 THE SYSTEM? 1537 00:55:53,015 --> 00:55:56,452 >> WELL, WHAT DO WE PUT IN TO 1538 00:55:56,452 --> 00:55:58,454 HARRISON'S PRINCIPLES AND 1539 00:55:58,454 --> 00:55:59,388 INTERNAL MEDICINE? 1540 00:55:59,388 --> 00:56:02,391 HOW DO WE DECIDE THAT SAME 1541 00:56:02,391 --> 00:56:02,625 PROCESS. 1542 00:56:02,625 --> 00:56:03,659 >> I WOULD LIKE TO THINK THE 1543 00:56:03,659 --> 00:56:04,827 PEOPLE WHO WRITE THE BOOK 1544 00:56:04,827 --> 00:56:05,895 CHAPTERS ARE CHOSEN BECAUSE 1545 00:56:05,895 --> 00:56:07,163 THEY'RE EXPERTS AND THEY CAN 1546 00:56:07,163 --> 00:56:11,100 MAKE THOSE JPTS, BUT WHO IS 1547 00:56:11,100 --> 00:56:14,003 MAKING THE JUDGMENT FOR THIS? 1548 00:56:14,003 --> 00:56:14,337 >> EXPERTS. 1549 00:56:14,337 --> 00:56:16,005 EXPERTS THEY HAVE EXPERTS MAKING 1550 00:56:16,005 --> 00:56:16,405 THOSE DECISIONS. 1551 00:56:16,405 --> 00:56:18,274 >> SO THE POTENTIAL SOURCE OF 1552 00:56:18,274 --> 00:56:19,575 BIAS, DEPENDING ON WHERE YOU'RE 1553 00:56:19,575 --> 00:56:20,643 GETTING IT. 1554 00:56:20,643 --> 00:56:21,577 >> YES OF COURSE. 1555 00:56:21,577 --> 00:56:23,045 THE DATA'S ALWAYS A SOURCE OF 1556 00:56:23,045 --> 00:56:23,880 BIAS. 1557 00:56:23,880 --> 00:56:30,486 I MEAN THE WHOLE WORLD IS FULL 1558 00:56:30,486 --> 00:56:30,820 BIAS. 1559 00:56:30,820 --> 00:56:33,890 HELP ME UNDERSTAND HOW AI READS 1560 00:56:33,890 --> 00:56:36,826 A BOOK OR JOURNAL ARTICLE OR 1561 00:56:36,826 --> 00:56:37,093 WHATEVER? 1562 00:56:37,093 --> 00:56:39,495 >> IT DOESN'T READ IT, IT USES 1563 00:56:39,495 --> 00:56:41,597 WORD 2 VEC, AND IT TAKES THE 1564 00:56:41,597 --> 00:56:43,766 WORDS, TURNS THEM INTO THE 1565 00:56:43,766 --> 00:56:45,334 NUMBERS AND INTO VECTORS AT 1566 00:56:45,334 --> 00:56:47,303 NUMBERS AND PUTS THE VECTORS OF 1567 00:56:47,303 --> 00:56:52,275 NUMBERS IN THE AI PROGRAM, WHICH 1568 00:56:52,275 --> 00:56:54,677 IS THE NEURAL NET, THE AI 1569 00:56:54,677 --> 00:56:57,513 PROGRAM USES JOINT PROBABILITY 1570 00:56:57,513 --> 00:57:01,651 TO DECIDE WHAT WORDS RELATE TO 1571 00:57:01,651 --> 00:57:03,753 YOUR PROMPT AND THAT HE USED 1572 00:57:03,753 --> 00:57:04,320 CONDITIONAL PROBABILITY TO 1573 00:57:04,320 --> 00:57:06,355 DECIDE WHAT THE BEST 1 IS, WHAT 1574 00:57:06,355 --> 00:57:07,857 THE BEST NEXT WORD IS. 1575 00:57:07,857 --> 00:57:10,293 SO GOING BACK TO YOUR ANALOGY OF 1576 00:57:10,293 --> 00:57:11,661 THIS LIEB WITH ALL THESE 1577 00:57:11,661 --> 00:57:15,298 ARTICLES, SO AI TAKES A BOOK OFF 1578 00:57:15,298 --> 00:57:16,866 THIS FIGURERATIVE SHELF, STANS 1579 00:57:16,866 --> 00:57:19,168 IT, AND CONVERTS IT TO TEXT, AND 1580 00:57:19,168 --> 00:57:21,103 THEN ANALYZES IT FROM THERE. 1581 00:57:21,103 --> 00:57:22,905 >> YEAH, BUT MOST OF THEM ARE 1582 00:57:22,905 --> 00:57:23,773 ELECTRONIC NOW, TODAY, MEDICINE 1583 00:57:23,773 --> 00:57:28,578 TODAY IS ALMOST ALL ELECTRONIC. 1584 00:57:28,578 --> 00:57:31,080 >> BUT THAT WE HAVE SO MUCH 1585 00:57:31,080 --> 00:57:32,415 INFORMATION WITH IT AVAILABLE IN 1586 00:57:32,415 --> 00:57:36,085 OLD TEXT, SOME OF IT WOULD HAVE 1587 00:57:36,085 --> 00:57:37,453 TO QUOTE READ THAT, RIGHT? 1588 00:57:37,453 --> 00:57:39,622 >> YEAH, IF SOMEBODY WANTS TO 1589 00:57:39,622 --> 00:57:45,161 PAY FOR IT, THEY COULD READ IT, 1590 00:57:45,161 --> 00:57:45,461 NO PROBLEM. 1591 00:57:45,461 --> 00:57:45,795 YES? 1592 00:57:45,795 --> 00:57:48,698 >> SO IF YOU LOOK AT THE MEDIA 1593 00:57:48,698 --> 00:57:51,767 AND THE PUBLIC ACCESSIBLE MEDIA 1594 00:57:51,767 --> 00:57:56,939 ON AI, AND IN PARTICULAR AND 1595 00:57:56,939 --> 00:57:58,207 PREOCCUPATION WITH THE DOWN 1596 00:57:58,207 --> 00:58:00,209 SIDES RATHER THAN THE POTENTIAL 1597 00:58:00,209 --> 00:58:01,777 ADVANTAGES AND 1 OF THE THINGS 1598 00:58:01,777 --> 00:58:04,480 THAT HHV 1 OF THE CRITICISMS IS 1599 00:58:04,480 --> 00:58:10,052 THAT THE PERCEPTION THAT THERE'S 1600 00:58:10,052 --> 00:58:10,686 MISAPPROPRIATION OF COPYRIGHTED 1601 00:58:10,686 --> 00:58:11,988 MATERIAL AND I GUESS I WOULD ASK 1602 00:58:11,988 --> 00:58:15,558 YOU, YOU KNOW WHAT PERCENTAGE OF 1603 00:58:15,558 --> 00:58:17,793 THE TRAINING CORPUS OF LET'S SAY 1604 00:58:17,793 --> 00:58:25,735 CHAT GPT OR ANY OF THESE LLMs 1605 00:58:25,735 --> 00:58:26,535 IS FROM COPYRIGHTED MATERIAL AND 1606 00:58:26,535 --> 00:58:29,271 HOW DO YOU DEAL WITH THAT 1607 00:58:29,271 --> 00:58:29,605 PROBLEM? 1608 00:58:29,605 --> 00:58:31,540 >> WELL, SO, I MEAN, I'M NOT A 1609 00:58:31,540 --> 00:58:33,676 LAWYER, SO I DON'T KNOW WHAT 1610 00:58:33,676 --> 00:58:35,878 FAIR USE IS VERSUS COPYRIGHT USE 1611 00:58:35,878 --> 00:58:37,780 AND ALL THAT, IT'S STILL BEING 1612 00:58:37,780 --> 00:58:39,382 ADJUDICATED IN THE COWERS, RIGHT 1613 00:58:39,382 --> 00:58:43,152 BUT IT TURNS OUT THAT THE NEW 1614 00:58:43,152 --> 00:58:53,195 MEDICAL LLMs ARE USING 1615 00:58:53,195 --> 00:58:53,729 PERMISSIONED DATA, 1616 00:58:53,729 --> 00:58:54,897 PERMISSIONED INFORMATION BUT IN 1617 00:58:54,897 --> 00:58:58,734 THE VERY BEGINNING THEY JUST 1618 00:58:58,734 --> 00:59:00,703 WANTED-THE TRUCK OF THE LMM IS 1619 00:59:00,703 --> 00:59:01,904 HIEWNG,AMOUNTS OF DATA BECAUSE 1620 00:59:01,904 --> 00:59:03,973 NATURAL LANGUAGE PROCESS SUGGEST 1621 00:59:03,973 --> 00:59:05,007 A SENSITIVITY SPECIFICITY 1622 00:59:05,007 --> 00:59:10,146 OPERATION, AND IT TURNS OUT THAT 1623 00:59:10,146 --> 00:59:12,715 IN MEETINGS I SET VERY RARELY, 1624 00:59:12,715 --> 00:59:14,116 AND SOME MEANINGS ARE SET IN 1625 00:59:14,116 --> 00:59:15,184 SMALL WAYS SO YOU HAVE TO HAVE 1626 00:59:15,184 --> 00:59:16,485 ENOUGH DATA SO THAT YOU CAN 1627 00:59:16,485 --> 00:59:17,386 CAPTURE THOSE THINGS, TOO. 1628 00:59:17,386 --> 00:59:26,128 I KNOW THAT BECAUSE WE DID IT. 1629 00:59:26,128 --> 00:59:27,663 SO THE POINT IS THEY ARE 1630 00:59:27,663 --> 00:59:28,664 SCRUBBED THE DATA JUST TO GET 1631 00:59:28,664 --> 00:59:31,167 THE THING UP AND RUNNING BUT 1632 00:59:31,167 --> 00:59:32,034 THEY'RE NEGOTIATING TO PLACES 1633 00:59:32,034 --> 00:59:33,736 RIGHT NOW TO GET ACCESS TO THE 1634 00:59:33,736 --> 00:59:34,303 DATA GOING FORWARD. 1635 00:59:34,303 --> 00:59:35,504 BUT IT WAS JUST WHAT THEY HAD TO 1636 00:59:35,504 --> 00:59:37,807 DO TO MOVE THIS THING FORWARD. 1637 00:59:37,807 --> 00:59:41,377 AT THE VERY LEAST IT'LL KEEP THE 1638 00:59:41,377 --> 00:59:42,011 LAWYERS BUSY. 1639 00:59:42,011 --> 00:59:48,217 >> YEAH, DO YOU WANT TO GO TO A 1640 00:59:48,217 --> 00:59:48,517 MICROPHONE? 1641 00:59:48,517 --> 00:59:49,652 >> THE VIDEOCAST AUDIENCE 1642 00:59:49,652 --> 00:59:49,885 CANNOT. 1643 00:59:49,885 --> 01:00:00,362 >> WE DO WANT TO HEAR YOU. 1644 01:00:05,267 --> 01:00:09,638 >> SO THE VERACITY OF THE 1645 01:00:09,638 --> 01:00:11,373 INFORMATION PRESENTED BY AN 1646 01:00:11,373 --> 01:00:14,777 WELL, MM IS-DO YOU THINK IT'S 1647 01:00:14,777 --> 01:00:16,645 MORE DEPENDENT ON THE NURRAL 1648 01:00:16,645 --> 01:00:19,014 NETWORK THAT CREATES THE 1649 01:00:19,014 --> 01:00:19,882 STATISTICAL PROBABILITIES THAT 1650 01:00:19,882 --> 01:00:25,488 IT UT PUTS OR ON THE CONNECTIONS 1651 01:00:25,488 --> 01:00:30,259 THAT IT MAKES LIKE AS LIKE AN 1652 01:00:30,259 --> 01:00:33,162 LLM HOST LIKE THE NETWORK, DID 1653 01:00:33,162 --> 01:00:33,796 THAT MAKE SENSE? 1654 01:00:33,796 --> 01:00:35,898 >> REMEMBER, SO IT'S PUTTING ALL 1655 01:00:35,898 --> 01:00:38,167 THESE VECTORS IN AND IN THE 1656 01:00:38,167 --> 01:00:40,302 NEURAL NET, JUST TAKES THESE 1657 01:00:40,302 --> 01:00:41,537 VECTORS AND ORGANIZES THESE 1658 01:00:41,537 --> 01:00:43,739 VECTORS,A CORDING TO THE 3 1659 01:00:43,739 --> 01:00:44,140 PARAMETERS, RIGHT? 1660 01:00:44,140 --> 01:00:45,541 SO IT ORGANIZES ALL OF THESE 1661 01:00:45,541 --> 01:00:47,843 THINGS IN TERPS OF BILLIONS OF 1662 01:00:47,843 --> 01:00:49,612 PARAMETERS, BUT THEN, IT 1663 01:00:49,612 --> 01:00:51,313 OPERATES IN TERMS OF THE JOINT 1664 01:00:51,313 --> 01:00:55,050 PROBABILITY, IT LOOKS FOR THINGS 1665 01:00:55,050 --> 01:00:56,986 THAT ARE PROBABLY SIMILAR. 1666 01:00:56,986 --> 01:00:58,020 OKAY? 1667 01:00:58,020 --> 01:00:59,455 AND SO THE INTERNAL WORKINGS OF 1668 01:00:59,455 --> 01:01:01,157 THE NEURAL NETIS A JOINT 1669 01:01:01,157 --> 01:01:05,494 PROBABILITY OPERATION AND WHAT 1670 01:01:05,494 --> 01:01:08,397 TEMPERATURE DOES IS THAT IT 1671 01:01:08,397 --> 01:01:12,701 SAYS, WE'RE GOING TO LOOSEN UP 1672 01:01:12,701 --> 01:01:13,602 WHAT PROBABILITY WE NEED. 1673 01:01:13,602 --> 01:01:14,904 HIRE TEMPERATURE MEANS WE MAY 1674 01:01:14,904 --> 01:01:16,672 TAKE THINGS THAT HAVE A LOWER 1675 01:01:16,672 --> 01:01:18,574 PROBABILITY OF BEING RELATED. 1676 01:01:18,574 --> 01:01:19,942 LOW TEMPERATURE MEANS IT HAS TO 1677 01:01:19,942 --> 01:01:21,710 HAVE A REALLY HIGH PROBABILITY 1678 01:01:21,710 --> 01:01:22,545 OF BEING RELATED BEFORE WE'RE 1679 01:01:22,545 --> 01:01:25,481 GOING TO PUT IT OUT TO THE 1680 01:01:25,481 --> 01:01:26,148 CONDITIONAL PROBABILITY. 1681 01:01:26,148 --> 01:01:27,583 SO IT'S THE JOINT PROBABILITY 1682 01:01:27,583 --> 01:01:30,119 AND OF COURSE HHV JOINT 1683 01:01:30,119 --> 01:01:31,053 PROBABILITY AND CONDITIONAL 1684 01:01:31,053 --> 01:01:32,021 PROBABILITIES ARE 2 ASPECTS OF 1685 01:01:32,021 --> 01:01:33,923 THE SAME THING, RIGHT? 1686 01:01:33,923 --> 01:01:35,057 CONDITIONAL PROBABILITY CAN BE 1687 01:01:35,057 --> 01:01:36,525 DERIVED FROM THE JOINT 1688 01:01:36,525 --> 01:01:38,260 PROBABILITY, RIGHT? 1689 01:01:38,260 --> 01:01:39,895 SO IN FACT, WHAT'S REALLY GOING 1690 01:01:39,895 --> 01:01:41,263 ON IS THE NEURAL NETIS 1691 01:01:41,263 --> 01:01:43,199 ORGANIZING ALL OF THESE THINGS, 1692 01:01:43,199 --> 01:01:44,834 AND THEN, THEY'RE LOOKING AT THE 1693 01:01:44,834 --> 01:01:47,870 JOINT PROBABILITY OF THESE 1694 01:01:47,870 --> 01:01:52,174 THINGS, BASED UPON YOUR PROMPT, 1695 01:01:52,174 --> 01:01:53,275 BASED UPON YOUR PROMPT WHAT 1696 01:01:53,275 --> 01:01:55,244 JOINT PROBABILITY CAN WE COME UP 1697 01:01:55,244 --> 01:01:55,444 WITH. 1698 01:01:55,444 --> 01:01:58,380 >> OKAY, RIGHT. 1699 01:01:58,380 --> 01:01:59,582 THAT MAKES SENSE. 1700 01:01:59,582 --> 01:02:00,783 SO THE SECOND PART OF MY 1701 01:02:00,783 --> 01:02:02,885 QUESTION IS IN TERMS OF THE 1702 01:02:02,885 --> 01:02:04,053 DEVELOPMENT OF MEDICAL AI, DO 1703 01:02:04,053 --> 01:02:05,855 YOU THINK MORE POTENTIAL EXISTS 1704 01:02:05,855 --> 01:02:07,623 TO MAKE IT FOR LACK OF A BETTER 1705 01:02:07,623 --> 01:02:09,792 TERM BETTER IN THE TWEPMENT OF 1706 01:02:09,792 --> 01:02:20,302 LIKE THE NEURAL NETWORK OR IN 1707 01:02:21,237 --> 01:02:21,337 THE- 1708 01:02:21,337 --> 01:02:22,805 >> THE NEURAL NETIS THE 1709 01:02:22,805 --> 01:02:23,272 CLASSIFIER. 1710 01:02:23,272 --> 01:02:25,674 THEY'VE BEEN DOING THINGS WITH 1711 01:02:25,674 --> 01:02:26,342 HIERARCHICAL NEURAL NETS WHERE 1712 01:02:26,342 --> 01:02:29,144 THEY WILL TAKE A NEURAL NET AND 1713 01:02:29,144 --> 01:02:30,512 OTHER NEURAL NETS AND 1 NEURAL 1714 01:02:30,512 --> 01:02:32,548 NET WILL REFER TO ANOTHER NEURAL 1715 01:02:32,548 --> 01:02:34,049 NET, I DON'T SEE THAT AS A 1716 01:02:34,049 --> 01:02:35,150 SOLUTION TO ANYTHING BUT THEY'RE 1717 01:02:35,150 --> 01:02:36,785 TRYING TO GET AWAY PRACTICES A 1718 01:02:36,785 --> 01:02:38,120 BILLION PARAMETERS AND TAKE A 1719 01:02:38,120 --> 01:02:40,623 PROMPT AND INSTEAD OF DOING THE 1720 01:02:40,623 --> 01:02:42,024 BILLION PARAMETERS, THEY WILL 1721 01:02:42,024 --> 01:02:43,459 PUT IT INTO 1 PART OF THE 1722 01:02:43,459 --> 01:02:45,461 HIERARCHY AND THAT'S SUPPOSED TO 1723 01:02:45,461 --> 01:02:50,232 MAKE IT CHEAPER, BUT THE NEURAL 1724 01:02:50,232 --> 01:02:51,600 NET, THE SELF-ORGANIZING NEURAL 1725 01:02:51,600 --> 01:02:56,438 NET IS THE SELF-CLASSIFIER AND 1726 01:02:56,438 --> 01:02:57,039 THAT PROBABILITIES ARE 1727 01:02:57,039 --> 01:02:58,107 MATHEMATICAL SO IT GETS BA BEING 1728 01:02:58,107 --> 01:03:02,578 TO THE TRAINING DATA AND YOUR 1729 01:03:02,578 --> 01:03:03,212 PROMPT. 1730 01:03:03,212 --> 01:03:04,346 >> DR. BURKE, WE ARE UP ON OUR 1731 01:03:04,346 --> 01:03:04,580 HOUR. 1732 01:03:04,580 --> 01:03:05,981 WE WANT TO THANK YOU FOR JOINING 1733 01:03:05,981 --> 01:03:09,285 US AND GIVING US A PEEK UNDER 1734 01:03:09,285 --> 01:03:11,654 THE HOOD OF HO AI WORKS AND 1735 01:03:11,654 --> 01:03:12,721 OBVIOUSLY THE POTENTIAL IMPACT 1736 01:03:12,721 --> 01:03:13,956 IT WILL HAVE FOR US GOING 1737 01:03:13,956 --> 01:03:14,189 FORWARD. 1738 01:03:14,189 --> 01:03:15,958 I HAD A LOT OF VIDEOCAST 1739 01:03:15,958 --> 01:03:18,327 QUESTIONS COMING IN, I WILL 1740 01:03:18,327 --> 01:03:19,695 FORWARD THOSE TO YOU SEPARATELY 1741 01:03:19,695 --> 01:03:21,497 AND YOU CAN ENGAGE WITH THOSE. 1742 01:03:21,497 --> 01:03:24,400 I WANT TO THANK EVERYBODY FOR 1743 01:03:24,400 --> 01:03:27,202 JOINING US TODAY AND HOPEFULLY 1744 01:03:27,202 --> 01:03:28,570 WE WON'T BECOME SKYNET. 1745 01:03:28,570 --> 01:03:38,570 >> OH THANK YOU VERY MUCH.