1 00:00:06,005 --> 00:00:07,707 >> WELL, WELCOME, EVERYONE, AND 2 00:00:07,774 --> 00:00:10,343 THANK YOU FOR JOINING US. 3 00:00:10,410 --> 00:00:12,312 I'M LANA YEGANOA, YOUR HOST FOR 4 00:00:12,379 --> 00:00:15,148 TODAY'S EVENT, ORGANIZED BY 5 00:00:15,215 --> 00:00:16,549 NATURAL LANGUAGE PROCESSING AND 6 00:00:16,616 --> 00:00:18,251 TEXT MINING SPECIAL INTEREST 7 00:00:18,318 --> 00:00:20,954 GROUP HERE AT NIH. 8 00:00:21,020 --> 00:00:22,489 WE'RE EXTREMELY EXCITED TO 9 00:00:22,555 --> 00:00:26,092 INTRODUCE OUR TODAY'S SPEAKER, 10 00:00:26,159 --> 00:00:28,395 DR. ZHE HE IS AN ASSOCIATE 11 00:00:28,461 --> 00:00:31,097 PROFESSOR AT THE SCHOOL OF 12 00:00:31,164 --> 00:00:33,032 INFORMATION AT THE FLORIDA STATE 13 00:00:33,099 --> 00:00:34,100 UNIVERSITY AND A CORE AFFILIATE 14 00:00:34,167 --> 00:00:38,772 OF THE INSTITUTE FOR SUCCESSFUL 15 00:00:38,838 --> 00:00:39,038 LONGEVITY. 16 00:00:39,105 --> 00:00:43,910 HE IS DIRECTOR OF 17 00:00:43,977 --> 00:00:45,111 BIOSTATISTICS -- IN THE 18 00:00:45,178 --> 00:00:46,546 UNIVERSITY'S CLINICAL AND 19 00:00:46,613 --> 00:00:48,281 TRANSLATIONAL SCIENCE AWARD HUB. 20 00:00:48,348 --> 00:00:50,450 HIS RESEARCH EXPERTISE INCLUDES 21 00:00:50,517 --> 00:00:51,551 MACHINE LEARNING, NATURAL 22 00:00:51,618 --> 00:00:53,086 LANGUAGE PROCESSING, AND 23 00:00:53,153 --> 00:00:55,588 KNOWLEDGE REPRESENTATION. 24 00:00:55,655 --> 00:00:57,457 THE OVERARCHING GOAL OF HIS 25 00:00:57,524 --> 00:00:59,659 RESEARCH IS TO IMPROVE 26 00:00:59,726 --> 00:01:01,761 POPULATION HEALTH AND ADVANCE 27 00:01:01,828 --> 00:01:04,697 BIOMEDICAL RESEARCH THROUGH 28 00:01:04,764 --> 00:01:07,567 APPLICATION OF BIOINFORMATICS. 29 00:01:07,634 --> 00:01:08,968 DR. HE, I CANNOT WAIT TO HEAR 30 00:01:09,035 --> 00:01:10,837 YOUR IDEAS AND VISIONS TOWARDS 31 00:01:10,904 --> 00:01:12,338 ACHIEVING A BETTER PUBLIC HEALTH 32 00:01:12,405 --> 00:01:15,208 AND WITH THE HELP OF 33 00:01:15,275 --> 00:01:17,577 BIOINFORMATICS AND DATA SCIENCE. 34 00:01:17,644 --> 00:01:19,446 SO BEFORE I PASS THE FLOOR TO 35 00:01:19,512 --> 00:01:21,214 YOU, I JUST WANT TO SAY THAT WE 36 00:01:21,281 --> 00:01:23,249 WILL HAVE AT THE END 10 TO 15 37 00:01:23,316 --> 00:01:24,717 MINUTES FOR QUESTIONS AND 38 00:01:24,784 --> 00:01:25,452 ANSWERS. 39 00:01:25,518 --> 00:01:28,588 AND AS USUAL, PLEASE SEND YOUR 40 00:01:28,655 --> 00:01:34,561 ANSWERS TO MY EMAIL, 41 00:01:34,627 --> 00:01:35,195 LANA.YEGANOVA@NIH.GOV, AND AFTER 42 00:01:35,261 --> 00:01:36,496 THE SPEAKER IS DONE WITH THE 43 00:01:36,563 --> 00:01:37,630 PRESENTATION, I WILL BE HAPPY TO 44 00:01:37,697 --> 00:01:39,799 READ OUT YOUR QUESTIONS AND HAVE 45 00:01:39,866 --> 00:01:41,000 THEM FOR DISCUSSION. 46 00:01:41,067 --> 00:01:43,503 AND WITH THAT, DR. HE, WE'RE 47 00:01:43,570 --> 00:01:45,171 READY TO HEAR YOU. 48 00:01:45,238 --> 00:01:46,372 >> GREAT. 49 00:01:46,439 --> 00:01:47,874 THANKS SO MUCH. 50 00:01:47,941 --> 00:01:49,309 SO LET ME START SHARING MY 51 00:01:49,375 --> 00:01:53,546 SCREEN, AND THEN START MY 52 00:01:53,613 --> 00:01:54,681 PRESENTATION. 53 00:01:54,747 --> 00:01:57,851 AND FIRST OF ALL, I WANT TO MAKE 54 00:01:57,917 --> 00:01:59,486 SURE THAT YOU CAN SEE THE ENTIRE 55 00:01:59,552 --> 00:02:01,020 SCREEN OF MY POWERPOINT SLIDE, 56 00:02:01,087 --> 00:02:01,221 RIGHT? 57 00:02:01,287 --> 00:02:02,088 >> YES, WE'RE GOOD. 58 00:02:02,155 --> 00:02:04,624 >> SO THANK YOU SO MUCH FOR THE 59 00:02:04,691 --> 00:02:08,127 OPPORTUNITY TO SPEAK HERE AT NLM 60 00:02:08,194 --> 00:02:09,963 AND TO THIS GREAT AUDIENCE. 61 00:02:10,029 --> 00:02:13,066 AND I REALLY FEEL VERY HONORED 62 00:02:13,132 --> 00:02:15,134 AND PROUD TO BE ABLE TO BE HERE. 63 00:02:15,201 --> 00:02:16,970 ALSO THIS OPPORTUNITY WOULD NOT 64 00:02:17,036 --> 00:02:23,009 BE POSSIBLE WITHOUT DR. LIU'S 65 00:02:23,076 --> 00:02:23,343 SUPPORT. 66 00:02:23,409 --> 00:02:27,747 I WORKED WITH DR. LIU SINCE LAST 67 00:02:27,814 --> 00:02:31,184 SEPTEMBER AND WORKED AS GUEST 68 00:02:31,251 --> 00:02:35,355 VISITOR A. 69 00:02:35,421 --> 00:02:36,890 THIS HAS BEEN ONE OF MY MAJOR 70 00:02:36,956 --> 00:02:38,324 RESEARCH PROJECTS SINCE 2019 SO 71 00:02:38,391 --> 00:02:39,659 I WANT TO SHARE WITH YOU SOME OF 72 00:02:39,726 --> 00:02:42,061 MY JOURNEY AND WHERE I'M AT OR 73 00:02:42,128 --> 00:02:43,429 WHERE I WAS AT AND WHERE I'M 74 00:02:43,496 --> 00:02:45,732 CURRENTLY AT IN TERMS OF 75 00:02:45,798 --> 00:02:46,733 RESEARCH TRAJECTORY. 76 00:02:46,799 --> 00:02:48,535 I FEEL THIS PROJECT IS REALLY 77 00:02:48,601 --> 00:02:50,870 UNIQUE AND INTERESTING TO ME, 78 00:02:50,937 --> 00:02:52,739 SIMPLY BECAUSE I STARTED THIS 79 00:02:52,805 --> 00:02:56,442 PROJECT BECAUSE OF SOME OF MY 80 00:02:56,509 --> 00:02:57,544 PERSONAL EXPERIENCES WITH 81 00:02:57,610 --> 00:02:58,878 HEALTHCARE SYSTEMS AND SOME OF 82 00:02:58,945 --> 00:03:01,281 MY FRUSTRATION AND STRUGGLES 83 00:03:01,347 --> 00:03:02,849 WITH KIND OF GETTING TO KNOW 84 00:03:02,916 --> 00:03:05,752 WHAT DOES MY DATA MEAN, WHAT 85 00:03:05,818 --> 00:03:09,856 DOES MY RECORD MEAN, AND THEN I 86 00:03:09,923 --> 00:03:11,658 HAD TO FIND GREAT COLLABORATORS 87 00:03:11,724 --> 00:03:12,859 TO WORK WITH AND GRADUALLY 88 00:03:12,926 --> 00:03:14,861 GETTING SOME REALLY INTERESTING 89 00:03:14,928 --> 00:03:16,062 PROJECTS GOING AND PUBLISHED AND 90 00:03:16,129 --> 00:03:19,799 ALSO TRYING TO GET SOME -- AS 91 00:03:19,866 --> 00:03:20,066 WELL. 92 00:03:20,133 --> 00:03:26,639 SO THE TOPIC IS USING MINGS AND 93 00:03:26,706 --> 00:03:29,809 INFORMATICS AND GENERATIVE AI. 94 00:03:29,876 --> 00:03:31,811 THE BACKGROUND OF THIS PROJECT 95 00:03:31,878 --> 00:03:34,981 IS THAT WE HAVE NO EASY ACCESS 96 00:03:35,048 --> 00:03:38,651 TO EHR DATA BECAUSE OF ADOPTION 97 00:03:38,718 --> 00:03:41,087 OF THE SYSTEMS ACROSS THE UNITED 98 00:03:41,154 --> 00:03:43,356 STATES, SO IT'S UNLIKELY THAT IT 99 00:03:43,423 --> 00:03:46,159 WILL EVER GO TO A HOSPITAL 100 00:03:46,225 --> 00:03:48,194 WHERE -- DATA IS NOT USED TO 101 00:03:48,261 --> 00:03:50,496 CAPTURE YOUR PERSONAL 102 00:03:50,563 --> 00:03:51,397 INFORMATION. 103 00:03:51,464 --> 00:03:53,666 ALSO WITH ACCESS TO EHR DATA 104 00:03:53,733 --> 00:03:55,501 THROUGH PATIENT PORTAL, PATIENTS 105 00:03:55,568 --> 00:03:57,070 NOW ALSO HAVE EASIER ACCESS TO 106 00:03:57,136 --> 00:03:59,372 THEIR CLINICAL DATA, AND WE LOOK 107 00:03:59,439 --> 00:04:01,240 AT LITERATURE, WE FOUND THAT 108 00:04:01,307 --> 00:04:02,675 PEOPLE ACTUALLY USE PATIENT 109 00:04:02,742 --> 00:04:04,077 PORTALS FOR A VARIETY OF 110 00:04:04,143 --> 00:04:05,812 PURPOSES, INCLUDING GETTING LAB 111 00:04:05,878 --> 00:04:06,813 TEST RESULTS, COMMUNICATING WITH 112 00:04:06,879 --> 00:04:09,215 THEIR PROVIDERS AND ALSO 113 00:04:09,282 --> 00:04:10,283 MEDICATIONS, MEDICAL HISTORY, 114 00:04:10,350 --> 00:04:12,518 AND AMONG ALL OF THESE 115 00:04:12,585 --> 00:04:13,920 ACTIVITIES, VIEWING LAB TEST 116 00:04:13,987 --> 00:04:15,121 RESULTS SEEMS TO BE ONE OF THE 117 00:04:15,188 --> 00:04:17,190 MOST FREQUENT ACTIVITIES PEOPLE 118 00:04:17,256 --> 00:04:19,926 UTILIZE PATIENT PORTALS FOR. 119 00:04:19,993 --> 00:04:21,928 HOWEVER, EVEN IF PEOPLE/PATIENTS 120 00:04:21,995 --> 00:04:24,397 HAVE ACCESS TO LAB TEST RESULTS 121 00:04:24,464 --> 00:04:26,366 THROUGH PATIENT PORTALS, STILL 122 00:04:26,432 --> 00:04:28,234 MANY PEOPLE FOUND CONFUSING AND 123 00:04:28,301 --> 00:04:30,003 HARD TO UNDERSTAND. 124 00:04:30,069 --> 00:04:32,538 THERE IS A VARIETY OF REASONS. 125 00:04:32,605 --> 00:04:35,341 PARTLY COULD BE DUE TO PATIENTS' 126 00:04:35,408 --> 00:04:37,143 LITERACY OF HEALTHCARE, OF 127 00:04:37,210 --> 00:04:39,379 HEALTH LITERACY, ALSO COMPLEXITY 128 00:04:39,445 --> 00:04:40,780 OF RESULTS AND COMPLEXITY OF 129 00:04:40,847 --> 00:04:42,982 THEIR HEALTH CONDITIONS. 130 00:04:43,049 --> 00:04:45,551 AND ALSO WE'RE SEEING SOME 131 00:04:45,618 --> 00:04:46,419 INTERESTING TREND THAT EVEN 132 00:04:46,486 --> 00:04:47,820 THOUGH PATIENT PORTALS ARE 133 00:04:47,887 --> 00:04:50,189 SUPPOSED TO PROVIDE A WAY FOR 134 00:04:50,256 --> 00:04:51,658 PATIENTS TO COMMUNICATE THEIR 135 00:04:51,724 --> 00:04:52,625 INFORMATION AND CONCERNS, 136 00:04:52,692 --> 00:04:55,695 QUESTIONS TO PROVIDERS, 137 00:04:55,762 --> 00:04:57,997 PROVIDERS OFTENTIMES MAY SEE 138 00:04:58,064 --> 00:04:58,998 THIS ALSO AS A PROBLEM BECAUSE 139 00:04:59,065 --> 00:05:01,467 THEY NEED TO SPEND A LOT OF TIME 140 00:05:01,534 --> 00:05:02,468 ANSWERING PATIENTS' QUESTIONS. 141 00:05:02,535 --> 00:05:05,538 SO, THEREFORE, WE HAVE SEEN SOME 142 00:05:05,605 --> 00:05:08,107 NEWS ABOUT PROVIDERS ACTUALLY 143 00:05:08,174 --> 00:05:09,709 STARTING TO CHARGE PATIENTS IN 144 00:05:09,776 --> 00:05:12,779 PATIENT PORTAL MESSAGE FOR 145 00:05:12,845 --> 00:05:14,213 SOMETHING LIKE ASKING QUESTIONS 146 00:05:14,280 --> 00:05:16,683 ABOUT LAB RESULTS. 147 00:05:16,749 --> 00:05:19,218 AND ALSO THERE'S A RECENT "WALL 148 00:05:19,285 --> 00:05:20,520 STREET JOURNAL" ARTICLE ABOUT 149 00:05:20,586 --> 00:05:25,458 WHEN SHALL WE SEND THIS MEDICAL 150 00:05:25,525 --> 00:05:28,695 TEST RESULTS TO PATIENTS, AND 151 00:05:28,761 --> 00:05:32,899 THIS ARTICLE TALK ABOUT DOCTORS 152 00:05:32,965 --> 00:05:35,835 ARE REALLY THINKING THAT GIVING 153 00:05:35,902 --> 00:05:37,236 PEOPLE ACCESS TO EARLY TEST 154 00:05:37,303 --> 00:05:38,438 RESULTS REPORTING IS GREAT BUT 155 00:05:38,504 --> 00:05:39,305 SOMETIMES WHEN PATIENTS HAVE 156 00:05:39,372 --> 00:05:41,841 ACCESS TO THESE RESULTS BEFORE 157 00:05:41,908 --> 00:05:43,209 THEIR PHYSICIANS ACTUALLY GET IN 158 00:05:43,276 --> 00:05:45,545 TOUCH WITH THEM, IT CAUSES SOME 159 00:05:45,611 --> 00:05:47,680 TROUBLES, OFTENTIMES BECAUSE OF 160 00:05:47,747 --> 00:05:49,816 MISINTERPRETATION OF RESULTS, OR 161 00:05:49,882 --> 00:05:52,819 MISINTERPRETATION OF EVEN HAR 162 00:05:52,885 --> 00:05:53,553 HARMFUL AND HARMLESS 163 00:05:53,619 --> 00:05:54,387 INFORMATION. 164 00:05:54,454 --> 00:05:56,155 SO ESPECIALLY WHEN PEOPLE 165 00:05:56,222 --> 00:05:57,290 RECEIVING RESULTS THAT ARE 166 00:05:57,356 --> 00:06:00,226 LIFE-CHANGING. 167 00:06:00,293 --> 00:06:01,327 SO DOCTORS SAY THEY HAVE LEARNED 168 00:06:01,394 --> 00:06:02,962 AND HAVE HAD PATIENTS LEARN 169 00:06:03,029 --> 00:06:05,031 ABOUT CANCER DIAGNOSIS FROM 170 00:06:05,098 --> 00:06:05,898 SMARTPHONE NOTIFICATION IN THE 171 00:06:05,965 --> 00:06:08,334 MIDDLE OF BUSINESS DINNER OR 172 00:06:08,401 --> 00:06:10,069 WHILE READING A BEDTIME STORY TO 173 00:06:10,136 --> 00:06:14,373 A 3-YEAR-OLD OR DOI DURING RUSHR 174 00:06:14,440 --> 00:06:15,508 COMMUNITY, AND EVEN THERE'S A 175 00:06:15,575 --> 00:06:16,876 PATIENT'S SPOUSE WHO WENT TO THE 176 00:06:16,943 --> 00:06:24,083 EMERGENCY ROOM FOR AN ANXIETY 177 00:06:24,150 --> 00:06:28,187 ATTACK AFTER MISINTERPRETING HER 178 00:06:28,254 --> 00:06:31,724 HUSBAND'S RESULTS. 179 00:06:31,791 --> 00:06:34,727 -- WHAT TIME FRAME WOULD BE A 180 00:06:34,794 --> 00:06:36,996 PROPER TIME FRAME FOR THEM TO 181 00:06:37,063 --> 00:06:38,097 COMMUNICATE INFORMATION TO THEIR 182 00:06:38,164 --> 00:06:39,832 PATIENTS. 183 00:06:39,899 --> 00:06:43,336 AND ALSO, IF YOU SEE A LAB TEST 184 00:06:43,402 --> 00:06:45,438 RESULT, IF YOU'VE EVER SEEN LAB 185 00:06:45,505 --> 00:06:46,472 TEST RESULTS FROM PATIENT 186 00:06:46,539 --> 00:06:48,241 PORTALS YOU MAY PROBABLY SEE 187 00:06:48,307 --> 00:06:49,442 SOMETHING LIKE THIS WHERE IT 188 00:06:49,509 --> 00:06:51,377 SHOWS YOU THE LAB TEST NAME, THE 189 00:06:51,444 --> 00:06:52,612 RESULTS, VALUE, THE UNIT, AN 190 00:06:52,678 --> 00:06:55,248 ALSO THE REFERENCE INTERVAL. 191 00:06:55,314 --> 00:06:57,083 AND THIS TYPICALLY IS THE WAY 192 00:06:57,150 --> 00:06:58,751 THAT PATIENT PORTALS ARE 193 00:06:58,818 --> 00:07:00,052 REPORTING LAB RESULTS TO 194 00:07:00,119 --> 00:07:01,254 PATIENTS, AND HOWEVER, THIS TYPE 195 00:07:01,320 --> 00:07:03,089 OF REPORTING MAY NOT BE THE 196 00:07:03,156 --> 00:07:05,725 IDEAL WAY OF REPORTING 197 00:07:05,792 --> 00:07:07,627 INFORMATION BECAUSE ABNORMAL 198 00:07:07,693 --> 00:07:09,929 RESULTS WHICH MEANS SOMETIMES 199 00:07:09,996 --> 00:07:12,565 THE LAB TEST RESULTS OUT OF 200 00:07:12,632 --> 00:07:14,634 RANGE MAY NOT MEAN IT'S VERY 201 00:07:14,700 --> 00:07:16,369 CONCERNING, WHEREAS IF OTHER -- 202 00:07:16,435 --> 00:07:18,671 SOMETIMES LAB VALUES WHICH MAY 203 00:07:18,738 --> 00:07:20,606 BE CLOSE TO THE BORDERLINE MAY 204 00:07:20,673 --> 00:07:22,141 BE CONCERNING. 205 00:07:22,208 --> 00:07:23,676 SO -- AND ALSO SOME LAB TEST 206 00:07:23,743 --> 00:07:25,311 RESULTS COULD HAVE DIFFERENT 207 00:07:25,378 --> 00:07:26,879 INTERPRETATIONS BASED ON 208 00:07:26,946 --> 00:07:27,313 DIFFERENT STANDARDS. 209 00:07:27,380 --> 00:07:29,882 FOR EXAMPLE, TSH LEVEL WHICH IS 210 00:07:29,949 --> 00:07:31,584 USED TYPICALLY TO MEASURE 211 00:07:31,651 --> 00:07:33,719 THYROID FUNCTION, THEY COULD 212 00:07:33,786 --> 00:07:36,255 HAVE DIFFERENT INTERPRETATION BY 213 00:07:36,322 --> 00:07:37,323 DIFFERENT ORGANIZATIONS AND 214 00:07:37,390 --> 00:07:38,858 DIFFERENT ASSOCIATIONS. 215 00:07:38,925 --> 00:07:41,327 SO IT'S UP TO THE PROVIDERS TO 216 00:07:41,394 --> 00:07:42,595 PROVIDE MORE ACCURATE 217 00:07:42,662 --> 00:07:44,430 INTERPRETATION TO LAB RESULTS, 218 00:07:44,497 --> 00:07:46,365 WHICH COULD TYPICALLY BE THE 219 00:07:46,432 --> 00:07:48,434 CASE WHEN THEY HAVE DIRECT 220 00:07:48,501 --> 00:07:49,468 ACCESS -- WHEN THEY CAN 221 00:07:49,535 --> 00:07:52,371 COMMUNICATE WITH THE PATIENTS -- 222 00:07:52,438 --> 00:07:54,540 WHEN THEY CAN'T COMMUNICATE WITH 223 00:07:54,607 --> 00:07:55,374 PATIENTS IN PERSON. 224 00:07:55,441 --> 00:07:57,276 SO IF YOU GIVE THIS WITHOUT 225 00:07:57,343 --> 00:07:58,644 INTERPRETATION IT COULD POSSIBLY 226 00:07:58,711 --> 00:07:59,912 CONFUSE AND CAUSE PROBLEMS. 227 00:07:59,979 --> 00:08:01,814 AT THE SAME TIME, PATIENTS WHEN 228 00:08:01,881 --> 00:08:04,050 THEY SEE LAB RESULTS, THEY MAY 229 00:08:04,116 --> 00:08:06,219 NOT FIND THEM CONCERNING OR THEY 230 00:08:06,285 --> 00:08:08,454 MAY NOT REALLY UNDERSTAND THE 231 00:08:08,521 --> 00:08:10,356 URGENCY OF SUCH INFORMATION. 232 00:08:10,423 --> 00:08:14,627 SO PREVIOUS EFFORTS HAVE DONE TO 233 00:08:14,694 --> 00:08:15,962 UNDERSTAND PATIENTS' ACTIVITY 234 00:08:16,028 --> 00:08:17,797 AND BEHAVIOR AFTER LAB RESULTS 235 00:08:17,864 --> 00:08:18,264 REPORTING. 236 00:08:18,331 --> 00:08:21,534 AND THERE'S A STUDY IN 2018 237 00:08:21,601 --> 00:08:23,269 SAYING LIKE OUT OF THIS 238 00:08:23,336 --> 00:08:24,337 93 PATIENTS, THEY FOUND NEARLY 239 00:08:24,403 --> 00:08:26,138 TWO THIRDS OF THEM DID NOT 240 00:08:26,205 --> 00:08:27,006 RECEIVE ANY EXPLANATION OF LAB 241 00:08:27,073 --> 00:08:29,575 RESULTS AND ABOUT HALF OF THEM 242 00:08:29,642 --> 00:08:31,143 CONDUCT ONLINE SEARCH TO 243 00:08:31,210 --> 00:08:32,478 UNDERSTAND THEIR RESULTS BETTER. 244 00:08:32,545 --> 00:08:35,214 ALSO THERE'S A STUDY THAT TALKS 245 00:08:35,281 --> 00:08:36,816 ABOUT EXPLORE DIFFERENT WAYS TO 246 00:08:36,883 --> 00:08:38,584 PRESENT LAB INFORMATION TO 247 00:08:38,651 --> 00:08:41,254 PATIENTS EITHER IN -- FORMAT 248 00:08:41,320 --> 00:08:43,189 WHERE THEY HAVE DIFFERENT 249 00:08:43,256 --> 00:08:45,057 INDICATOR OF URGENCY AND ALSO 250 00:08:45,124 --> 00:08:46,926 THEY MAY USE DIFFERENT COLOR 251 00:08:46,993 --> 00:08:50,296 CODING OR DIFFERENT -- EVEN THE 252 00:08:50,363 --> 00:08:53,799 EMOJIS TO PROVIDE SOME SORT OF 253 00:08:53,866 --> 00:08:55,201 INTERPRETATION WITH SOME TEXTUAL 254 00:08:55,268 --> 00:08:56,002 EXPLANATION. 255 00:08:56,068 --> 00:08:56,869 AND THERE WAS AN INTERESTING 256 00:08:56,936 --> 00:09:01,440 STUDY PUBLISHED IN 2017 BY 257 00:09:01,507 --> 00:09:02,975 ZIKMUND-FISHER GROUP THAT 258 00:09:03,042 --> 00:09:05,077 EXPLORED THE USE OF DIFFERENT 259 00:09:05,144 --> 00:09:06,946 REPORTING OF LAB INFORMATION AND 260 00:09:07,013 --> 00:09:08,681 ALSO EVALUATING THEIR URGENCY. 261 00:09:08,748 --> 00:09:11,651 SO THEY ASSESS THE TABLE 262 00:09:11,717 --> 00:09:13,185 REPRESENTATION WHICH IS TYPICAL 263 00:09:13,252 --> 00:09:14,687 FORMAT, AND ALSO THEY LOOK AT 264 00:09:14,754 --> 00:09:16,022 THE SIMPLE LINE FORMAT, WHERE 265 00:09:16,088 --> 00:09:17,723 YOU JUST HAVE A SIMPLE LINE 266 00:09:17,790 --> 00:09:20,159 INDICATING THE NORMAL REFERENCE 267 00:09:20,226 --> 00:09:22,194 RANGE AND ALSO THERE'S A BLOCK 268 00:09:22,261 --> 00:09:23,996 LINE FORMAT WHERE THEY HAVE 269 00:09:24,063 --> 00:09:25,531 DIFFERENT BLOCK LINE TO INDICATE 270 00:09:25,598 --> 00:09:27,733 THE BORDERLINE VALUE AND ALSO 271 00:09:27,800 --> 00:09:28,901 THE -- LIKE THE VERY LOW VALUE 272 00:09:28,968 --> 00:09:30,870 OR LOW VALUE, AND ALSO THERE'S 273 00:09:30,937 --> 00:09:33,506 ANOTHER ONE, IT'S A GRADIENT 274 00:09:33,572 --> 00:09:35,141 LINE VALUE REPRESENTATION, WHICH 275 00:09:35,207 --> 00:09:38,611 DOESN'T HAVE BLOCK LINE BUT VERY 276 00:09:38,678 --> 00:09:40,613 GRGRADUALLY CHANGING THE COLOR O 277 00:09:40,680 --> 00:09:43,349 INDICATE URGENCY AND NORMALCY OF 278 00:09:43,416 --> 00:09:45,518 A LAB TEST RESULT. 279 00:09:45,584 --> 00:09:48,587 SO THE ASSESSED, GIVEN THIS 280 00:09:48,654 --> 00:09:50,189 DIFFERENT TYPE OF REPRESENTATION 281 00:09:50,256 --> 00:09:51,424 OF THE SAME INFORMATION TO 282 00:09:51,490 --> 00:09:56,028 PATIENTS, WHAT IS THE URGENCY 283 00:09:56,095 --> 00:09:58,831 LEVEL, AND THEY FOUND THAT 284 00:09:58,898 --> 00:10:01,267 FROM TABLE TO SINGLE LINE TO 285 00:10:01,334 --> 00:10:04,937 BLOCK LINE TO GRADIENT LINE 286 00:10:05,004 --> 00:10:06,172 GRADUALLY DECREASING URGENCY 287 00:10:06,238 --> 00:10:09,642 LEVEL FROM PATIENTS. 288 00:10:09,709 --> 00:10:11,310 IN OUR GROUP AS I MENTIONED, WE 289 00:10:11,377 --> 00:10:12,511 STARTED TO EXPLORE THIS IDEA OF 290 00:10:12,578 --> 00:10:14,146 HOW TO BETTER PROVIDE 291 00:10:14,213 --> 00:10:15,348 INFORMATION TO PATIENTS FOR 292 00:10:15,414 --> 00:10:17,950 THEIR LAB RESULTS COMPREHENSION, 293 00:10:18,017 --> 00:10:19,485 WE STARTED EXPLORING THIS BY 294 00:10:19,552 --> 00:10:21,587 CONDUCTING A SIMPLE ONLINE 295 00:10:21,654 --> 00:10:23,422 SURVEY, LOOKING AT THE 296 00:10:23,489 --> 00:10:24,890 SELF-PERCEIVED UNDERSTANDING OF 297 00:10:24,957 --> 00:10:25,725 RESULTS AND DIFFERENT CHALLENGES 298 00:10:25,791 --> 00:10:28,361 THEY HAVE FACED, SO BASED ON 299 00:10:28,427 --> 00:10:31,097 THIS SIMPLE ONLINE SURVEY 300 00:10:31,163 --> 00:10:33,199 CONDUCTED, WE FOUND THAT WHEN 301 00:10:33,265 --> 00:10:37,003 PATIENTS ARE TRYING TO INTERPRET 302 00:10:37,069 --> 00:10:41,073 LAB RESULTS THEY OFTENTIMES NEED 303 00:10:41,140 --> 00:10:42,441 BOTH INFORMATION ABOUT REFERENCE 304 00:10:42,508 --> 00:10:44,977 RANGE AND ALSO SOME PERSONALIZED 305 00:10:45,044 --> 00:10:46,312 CONTEXTUAL INFORMATION SUCH AS 306 00:10:46,379 --> 00:10:47,646 TREATING OPTION, DIAGNOSIS AND 307 00:10:47,713 --> 00:10:48,848 WHAT THEY SHOULD DO NEXT. 308 00:10:48,914 --> 00:10:51,384 THEY ALSO FOUND THAT SEVERAL 309 00:10:51,450 --> 00:10:53,319 FACTORS LIKE NORMALCY OF 310 00:10:53,386 --> 00:10:55,054 RESULTS, HEALTH LITERACY, 311 00:10:55,121 --> 00:10:58,491 TECHNOLOGY PRO IF EFFICIENCY, PD 312 00:10:58,557 --> 00:10:59,625 IMPACTED PERCEPTION OF THE 313 00:10:59,692 --> 00:11:00,793 RESULTS WHEN THEY INTERPRETED 314 00:11:00,860 --> 00:11:02,428 RESULTS USING PATIENT PORTALS. 315 00:11:02,495 --> 00:11:05,698 WE ALSO FURTHER LOOKED AT SOCIAL 316 00:11:05,765 --> 00:11:08,000 MEDIA POSTINGS ON THE SOCIAL 317 00:11:08,067 --> 00:11:10,636 MEDIA QUESTION WEBSITE, WE LOOK 318 00:11:10,703 --> 00:11:12,738 AT QUESTIONS THAT CONTAIN SOME 319 00:11:12,805 --> 00:11:14,907 DIABETES-RELATED LAB VALUES, 320 00:11:14,974 --> 00:11:16,609 A1C, GLUCOSE AND BLOOD SUGAR, 321 00:11:16,675 --> 00:11:17,943 AND LOOK AT WHAT KIND OF 322 00:11:18,010 --> 00:11:20,046 INFORMATION THEY CAN PROVIDE IN 323 00:11:20,112 --> 00:11:21,781 THESE QUESTIONS AND WHAT KIND OF 324 00:11:21,847 --> 00:11:24,984 RESPONSES THEY TYPICALLY RECEIVE 325 00:11:25,051 --> 00:11:28,187 FROM THE COMMUNITY USERS OF THIS 326 00:11:28,254 --> 00:11:30,890 WEBSITE AND WE FOUND THAT WHEN 327 00:11:30,956 --> 00:11:32,291 PATIENTS ASKING QUESTIONS ON 328 00:11:32,358 --> 00:11:34,927 SOCIAL MEDIA LIKE SOCIAL Q & A, 329 00:11:34,994 --> 00:11:36,128 THEY MAY NOT PROVIDE ENOUGH 330 00:11:36,195 --> 00:11:37,663 INFORMATION AND SOME OF THEM MAY 331 00:11:37,730 --> 00:11:39,198 PROVIDE INFORMATION ABOUT THEIR 332 00:11:39,265 --> 00:11:41,834 LIFESTYLE, ABOUT THEIR DIET AND 333 00:11:41,901 --> 00:11:43,803 ABOUT THEIR DIAGNOSIS AND 334 00:11:43,869 --> 00:11:44,370 COMORBIDITIES. 335 00:11:44,437 --> 00:11:47,506 SO IT'S A VERY DIVERSE GROUP. 336 00:11:47,573 --> 00:11:49,075 WE ALSO CONDUCTED A STUDY WHERE 337 00:11:49,141 --> 00:11:51,977 WE DESIGNED A PROTOTYPE OF A 338 00:11:52,044 --> 00:11:53,079 TOOL THAT PRESENTS LAB 339 00:11:53,145 --> 00:11:55,181 INFORMATION TO PATIENTS, AT THE 340 00:11:55,247 --> 00:11:57,450 SAME TIME PRESENTING A WAY FOR 341 00:11:57,516 --> 00:11:59,018 PATIENTS TO LOOK AT 342 00:11:59,085 --> 00:12:00,753 INTERPRETATION OF LAB VALUE, LAB 343 00:12:00,820 --> 00:12:04,490 TEST AND ALSO THEY CAN ALSO 344 00:12:04,557 --> 00:12:07,226 ANNOTATE AND LABEL SOME OF THE 345 00:12:07,293 --> 00:12:08,060 INTERPRETATION INTO THEIR 346 00:12:08,127 --> 00:12:11,664 PERSONAL LIBRARIES. 347 00:12:11,730 --> 00:12:13,099 SO ALL OF THESE STUDIES REALLY 348 00:12:13,165 --> 00:12:15,067 GIVE US SOME GOOD EXPERIENCES 349 00:12:15,134 --> 00:12:17,336 INTO THIS FIELD AND LOOKING AT 350 00:12:17,403 --> 00:12:18,971 ALSO WORKING DIRECTLY WITH 351 00:12:19,038 --> 00:12:19,872 INDIVIDUAL PATIENTS TO 352 00:12:19,939 --> 00:12:23,109 UNDERSTAND THEIR CONCERNS AND 353 00:12:23,175 --> 00:12:24,276 THEIR BEST KIND OF -- THEIR 354 00:12:24,343 --> 00:12:27,780 NEEDS IN GETTING THEIR MEDICAL 355 00:12:27,847 --> 00:12:30,716 NEEDS OF A FIELD BY LAB RESULTS 356 00:12:30,783 --> 00:12:31,183 INTERPRETATION. 357 00:12:31,250 --> 00:12:33,152 RECENTLY WE OBTAINED SOME 358 00:12:33,219 --> 00:12:34,553 INTERNAL GRANT TO CONDUCT 359 00:12:34,620 --> 00:12:37,022 STUDIES TO FURTHER UNDERSTAND 360 00:12:37,089 --> 00:12:38,591 THE FACTORS THAT INFLUENCE 361 00:12:38,657 --> 00:12:39,892 UNDERSTANDING LAB TEST RESULTS. 362 00:12:39,959 --> 00:12:42,962 THE MAIN REASON WHY WE DO THIS 363 00:12:43,028 --> 00:12:44,530 SURVEY STUDY IS FOR THE PREVIOUS 364 00:12:44,597 --> 00:12:47,766 ONE WHICH WE USED -- FOR DATA 365 00:12:47,833 --> 00:12:48,767 COLLECTION, WE DIDN'T REALLY 366 00:12:48,834 --> 00:12:50,202 RECEIVE A LOT OF RESPONSES FROM 367 00:12:50,269 --> 00:12:52,438 OLER ADULTS, THAT'S BECAUSE 368 00:12:52,505 --> 00:12:53,906 OLDER PEOPLE TEND TO NOT USE 369 00:12:53,973 --> 00:13:00,579 LIKE THOSE TYPE OF SERVICES 370 00:13:00,646 --> 00:13:07,887 AS A LESSER CLOUD MECHANISM FOR 371 00:13:07,953 --> 00:13:09,421 SURVEY AND ALSO WE WANTED TO 372 00:13:09,488 --> 00:13:11,857 GAIN A MORE DEEPER UNDERSTANDING 373 00:13:11,924 --> 00:13:14,660 OF THE FACTORS USING STANDARD 374 00:13:14,727 --> 00:13:15,494 AND EVALUATIVE METRICS. 375 00:13:15,561 --> 00:13:18,364 WE ALSO DID A STUDY WHERE WE 376 00:13:18,430 --> 00:13:19,165 ANNOTATED SOME ONLINE POSTS 377 00:13:19,231 --> 00:13:20,399 ABOUT LAB RESULTS 378 00:13:20,466 --> 00:13:22,001 INTERPRETATION, AND MORE 379 00:13:22,067 --> 00:13:24,303 RECENTLY, I WORKED WITH 380 00:13:24,370 --> 00:13:26,705 DR. LIU'S GROUP HERE AT NCBI TO 381 00:13:26,772 --> 00:13:30,976 EVALUATE MODELS -- ANSWERING LAB 382 00:13:31,043 --> 00:13:36,015 TEST RESULTS FR QUESTIONS FROM 383 00:13:36,081 --> 00:13:37,449 PATIENTS. SO IN THIS ONLINE 384 00:13:37,516 --> 00:13:39,285 SURVEY, WHAT WE DO IS WE WANT TO 385 00:13:39,351 --> 00:13:40,886 LOOK AT A VARIETY OF FACTORS 386 00:13:40,953 --> 00:13:43,489 THAT MAY INFLUENCE PATIENTS' 387 00:13:43,556 --> 00:13:44,757 UNDERSTANDING AND COMPREHENSION 388 00:13:44,823 --> 00:13:45,391 OF LAB RESULTS. 389 00:13:45,457 --> 00:13:47,259 SO WE FOUND OUT THERE'S ACTUALLY 390 00:13:47,326 --> 00:13:49,461 NO EXISTING METHOD FOR 391 00:13:49,528 --> 00:13:51,530 EVALUATING SUBJECTIVE OR 392 00:13:51,597 --> 00:13:53,265 OBJECTIVE COMPREHENSION OF LAB 393 00:13:53,332 --> 00:13:55,334 TEST RESULTS, SO WE DEVELOPED 394 00:13:55,401 --> 00:14:00,406 OUR OWN SCORE WHICH CONSISTS OF 395 00:14:00,472 --> 00:14:01,473 21 QUESTIONS ARE WHETHER LAB 396 00:14:01,540 --> 00:14:02,942 RESULTS IS NORMAL OR ABNORMAL 397 00:14:03,008 --> 00:14:04,677 BASED ON PATIENT'S UNDERSTANDING 398 00:14:04,743 --> 00:14:06,412 OF LAB TEST INFORMATION. 399 00:14:06,478 --> 00:14:10,216 WE ALSO USED DIFFERENT 400 00:14:10,282 --> 00:14:13,219 MEASUREMENTS LIKE NUMERACY 401 00:14:13,285 --> 00:14:14,954 SKILLS, EHEALTH LITERACY AND 402 00:14:15,020 --> 00:14:22,328 ALSO TO ASSESS PEOPLE'S NU P PE- 403 00:14:22,394 --> 00:14:23,495 DEMOGRAPHICS, HEALTH STATUS, 404 00:14:23,562 --> 00:14:26,398 FREQUENCY OF DOCTORS VISIT, 405 00:14:26,465 --> 00:14:29,068 ACCESS AND USE OF LAB RESULTS 406 00:14:29,134 --> 00:14:31,003 AND INTERNET AND TECHNOLOGY USE 407 00:14:31,070 --> 00:14:32,571 TO UNDERSTAND THESE RESULTS IN 408 00:14:32,638 --> 00:14:33,405 LAB RESULTS COMPREHENSION. 409 00:14:33,472 --> 00:14:36,408 SO THIS IS ONE EXAMPLE WHERE WE 410 00:14:36,475 --> 00:14:38,811 USE SOME SIMPLE LIKE LAB PANELS 411 00:14:38,877 --> 00:14:41,080 THAT ARE FREQUENTLY DONE IN 412 00:14:41,146 --> 00:14:44,683 PRIMARY CARE SETTING TO 413 00:14:44,750 --> 00:14:46,151 PATIENTS, AND DURING THE ANNUAL 414 00:14:46,218 --> 00:14:48,721 EXAM, AND THEN GIVING THEM THE 415 00:14:48,787 --> 00:14:49,421 HYPOTHETICAL SCENARIO AND 416 00:14:49,488 --> 00:14:51,090 LETTING THEM EVALUATE THESE 417 00:14:51,156 --> 00:14:52,791 RESULTS AND PROVIDING THEIR 418 00:14:52,858 --> 00:14:54,860 UNDERSTANDING OF THE NORMALCY 419 00:14:54,927 --> 00:14:56,629 AND -- OF THE RESULTS. 420 00:14:56,695 --> 00:14:59,732 SO USING 21-SUCH QUESTIONS, WE 421 00:14:59,798 --> 00:15:02,935 CAME UP WITH A SCALE BUT WE'RE 422 00:15:03,002 --> 00:15:06,138 STILL KIND OF ADJUSTING THIS 423 00:15:06,205 --> 00:15:08,207 SCORE BECAUSE WE'RE STILL NOT 424 00:15:08,274 --> 00:15:12,278 QUITE HAPPY ABOUT THE TYPE OF 425 00:15:12,344 --> 00:15:14,246 VARIETIES AND THE VARIANTS WE 426 00:15:14,313 --> 00:15:15,314 GET FROM THE RESPONSE FROM THE 427 00:15:15,381 --> 00:15:17,316 PATIENTS. 428 00:15:17,383 --> 00:15:19,518 AND ALSO WE WELCOME ANY IDEAS OF 429 00:15:19,585 --> 00:15:21,220 HOW TO BETTER MEASURE PATIENTS' 430 00:15:21,287 --> 00:15:22,788 UNDERSTANDING OF LAB TEST 431 00:15:22,855 --> 00:15:27,693 RESULTS USING A STANDARD SCALE. 432 00:15:27,760 --> 00:15:29,528 ALSO WE LOOKED AT PATIENT ACCESS 433 00:15:29,595 --> 00:15:31,730 TO LAB RESULTS, WE FOUND OUT 434 00:15:31,797 --> 00:15:33,399 MANY PEOPLE THEY HAVE ACCESS TO 435 00:15:33,465 --> 00:15:35,234 LAB TEST RESULTS THROUGH PATIENT 436 00:15:35,301 --> 00:15:36,101 PORTAL STILL, A LOT OF PEOPLE 437 00:15:36,168 --> 00:15:37,836 THEY PREFER TO BE CALLED UP BY 438 00:15:37,903 --> 00:15:42,741 THEIR DOCTORS OR NURSES SO THEY 439 00:15:42,808 --> 00:15:44,209 CAN ASK QUESTIONS ON THE FLY. 440 00:15:44,276 --> 00:15:45,444 WE ALSO ASK QUESTIONS ABOUT WHY 441 00:15:45,511 --> 00:15:47,079 THEY DON'T WANT TO USE PATIENT 442 00:15:47,146 --> 00:15:48,647 PORTAL TO VIEW LAB TEST RESULTS, 443 00:15:48,714 --> 00:15:51,450 AND HOPEFULLY TO GET SOME MORE 444 00:15:51,517 --> 00:15:53,152 INSIGHTS. 445 00:15:53,218 --> 00:15:56,155 THIS IS NUMERACY SCALES, VERY 446 00:15:56,221 --> 00:15:58,424 SIMPLE THREE-ITEM SUBJECTIVE 447 00:15:58,490 --> 00:16:00,025 NUMERACY SCALE, WANTING TO JUST 448 00:16:00,092 --> 00:16:01,627 LOOK AT THEIR GENERAL 449 00:16:01,694 --> 00:16:03,896 UNDERSTANDING OF STATISTICS AND 450 00:16:03,962 --> 00:16:06,532 BECAUSE WE ARE PRIMARILY 451 00:16:06,598 --> 00:16:07,866 TARGETING DIVERSE PATIENT GROUPS 452 00:16:07,933 --> 00:16:11,003 IN TERMS OF THEIR AGE, SO WE'LL 453 00:16:11,070 --> 00:16:13,405 WANT TO LOOK AT WHICH SCALES 454 00:16:13,472 --> 00:16:15,174 WILL GIVE EASIER SIMPLE WAY TO 455 00:16:15,240 --> 00:16:18,110 ASSESS SUCH INFORMATION AND NOT 456 00:16:18,177 --> 00:16:21,714 OVERBURDEN OUR PARTICIPANTS. 457 00:16:21,780 --> 00:16:24,350 THIS IS OUR LITERACY SCALE, MORE 458 00:16:24,416 --> 00:16:26,518 FOCUSED ON EHEALTH BEHAVIOR, 459 00:16:26,585 --> 00:16:28,954 FOCUSED ON HEALTH RESOURCE 460 00:16:29,021 --> 00:16:32,191 SEARCH ON THE INTERNET AND THERE 461 00:16:32,257 --> 00:16:34,059 MIGHT BE OTHER USEFUL AND 462 00:16:34,126 --> 00:16:35,728 VALUABLE EHEALTH LITERACY 463 00:16:35,794 --> 00:16:37,763 SCALES, BUT WE FIGURE THIS SCALE 464 00:16:37,830 --> 00:16:39,331 IS MORE LIKE EASY TO ADMINISTER 465 00:16:39,398 --> 00:16:42,601 ON THE INTERNET -- OR ON THE WEB 466 00:16:42,668 --> 00:16:47,539 FORMAT THROUGH OUR INSTRUMENT. 467 00:16:47,606 --> 00:16:51,677 SO TRY TO UNDERSTAND THE GENERAL 468 00:16:51,744 --> 00:16:54,747 DISTRIBUTION BASED ON STATISTICS 469 00:16:54,813 --> 00:16:57,349 AND ALSO CONDUCTED LOGISTICAL 470 00:16:57,416 --> 00:17:02,621 REGRESSION MODEL AND ALSO USING 471 00:17:02,688 --> 00:17:04,256 GENERAL LINEAR MODEL TO OOP R 472 00:17:04,323 --> 00:17:08,060 ANAANALYZEPREDICTORS. 473 00:17:08,127 --> 00:17:09,461 SO THIS IS A GENERAL BREAKDOWN 474 00:17:09,528 --> 00:17:11,163 BY OUR POPULATION. 475 00:17:11,230 --> 00:17:13,265 WE HAVE A FAIRLY GOOD 476 00:17:13,332 --> 00:17:14,666 PRESENTATION OF DIFFERENT AGE 477 00:17:14,733 --> 00:17:18,203 GROUP FROM 18 TO 34, 35 TO 49 478 00:17:18,270 --> 00:17:19,671 AND OVER 50 YEARS OLD. 479 00:17:19,738 --> 00:17:24,777 WE ALSO HAVE A GENERALLY 480 00:17:24,843 --> 00:17:25,611 BALANCED BETWEEN POPULATION 481 00:17:25,677 --> 00:17:28,213 BETWEEN MALE AND FEMALE, AND 482 00:17:28,280 --> 00:17:30,816 DIFFERENT RACE AND ETHNIC 483 00:17:30,883 --> 00:17:31,483 GROUPS. 484 00:17:31,550 --> 00:17:35,854 SO WE GOT 276 PARTICIPANT 485 00:17:35,921 --> 00:17:39,491 RESPONDENTS FROM OUR LOCAL 486 00:17:39,558 --> 00:17:44,296 REGISTRY WHERE WE SENT EMAIL TO 487 00:17:44,363 --> 00:17:45,197 OVER 2,700 PARTICIPANTS WHO MAY 488 00:17:45,264 --> 00:17:47,099 BE INTERESTED IN THIS RESEARCH, 489 00:17:47,166 --> 00:17:49,768 AND ALSO COLLECT DATA FROM 490 00:17:49,835 --> 00:17:51,970 AMAZON MECHANIC TURK. 491 00:17:52,037 --> 00:17:53,705 AND HERE ARE SOME RESULTS FROM 492 00:17:53,772 --> 00:17:54,273 THIS STUDY. 493 00:17:54,339 --> 00:17:57,843 SO THE FIRST RESULT IS REGARDING 494 00:17:57,910 --> 00:18:01,480 THE ACCESSING LAB RESULTS VIA 495 00:18:01,547 --> 00:18:02,214 PATIENT PORE TALS. 496 00:18:02,281 --> 00:18:03,882 WE FOUND OUT SOME OF THE 497 00:18:03,949 --> 00:18:05,384 BARRIERS ESPECIALLY FOR THE 498 00:18:05,451 --> 00:18:08,387 OLDER ADULT GROUP INCLUDES THE 499 00:18:08,454 --> 00:18:11,623 PREFERRED DOCTORS' TO EXPLAIN 500 00:18:11,690 --> 00:18:12,624 THE RESULT TO THEM. 501 00:18:12,691 --> 00:18:14,726 SOME OF THEM JUST DON'T KNOW HOW 502 00:18:14,793 --> 00:18:17,963 TO INTERPRET LAB VALUES BY 503 00:18:18,030 --> 00:18:19,965 THEMSELVES. 504 00:18:20,032 --> 00:18:21,700 AND ALSO LOOK AT FACTORS THAT 505 00:18:21,767 --> 00:18:23,335 INFLUENCE ACCESS TO LAB RESULTS 506 00:18:23,402 --> 00:18:24,870 VIA PATIENT PORTALS, WE FOUND 507 00:18:24,937 --> 00:18:29,107 OUT USING THIS REGRESSION 508 00:18:29,174 --> 00:18:30,275 ANALYSIS, WE FOUND THAT COMPARED 509 00:18:30,342 --> 00:18:33,111 TO YOUNGER ADULTS, 18 TO 34 510 00:18:33,178 --> 00:18:36,482 GROUP, ADULTS BETWEEN 50 TO 65 511 00:18:36,548 --> 00:18:38,550 YEARS PLUS AND OLDER WERE MORE 512 00:18:38,617 --> 00:18:39,618 LIKELY TO ACCESS THEIR LAB 513 00:18:39,685 --> 00:18:41,820 RESULTS BY A PATIENT PORTAL AND 514 00:18:41,887 --> 00:18:43,455 EHEALTH LITERACY WAS ALSO A 515 00:18:43,522 --> 00:18:44,990 SIGNIFICANT PREDICTOR OF 516 00:18:45,057 --> 00:18:45,991 ACCESSING LAB TEST RESULTS USING 517 00:18:46,058 --> 00:18:48,861 PATIENT PORTAL, AND THEN THESE 518 00:18:48,927 --> 00:18:51,330 ARE SOME OF THE REASONS WHY 519 00:18:51,396 --> 00:18:52,865 PATIENTS -- WHAT PATIENTS DO 520 00:18:52,931 --> 00:18:55,934 AFTER LAB TEST RESULTS, WE FOUND 521 00:18:56,001 --> 00:18:56,935 MOST PEOPLE WOULD LIKELY TO 522 00:18:57,002 --> 00:18:59,304 FOLLOW UP WITH THEIR DOCTOR 523 00:18:59,371 --> 00:19:00,205 AND -- BUT MORE THAN HALF OF 524 00:19:00,272 --> 00:19:02,074 THEM WOULD WANT TO DO RESEARCH 525 00:19:02,140 --> 00:19:03,041 ONLINE. 526 00:19:03,108 --> 00:19:05,777 SO THIS IS THE GENERAL LINEAR 527 00:19:05,844 --> 00:19:07,846 MODEL WHERE WE PERFORM TO 528 00:19:07,913 --> 00:19:09,915 EXAMINE THE RELATIONSHIP BETWEEN 529 00:19:09,982 --> 00:19:11,583 USE OF PATIENT PORTAL TO VIEW 530 00:19:11,650 --> 00:19:16,822 LAB RESULTS, NUMERACY SCALES, 531 00:19:16,889 --> 00:19:19,091 AGE -- SO WE FOUND COMPARED TO 532 00:19:19,157 --> 00:19:20,893 PATIENTS WHO HAVE VIEWED THEIR 533 00:19:20,959 --> 00:19:22,961 LAB RESULTS VIA PATIENT PORTAL, 534 00:19:23,028 --> 00:19:26,899 THOSE WHO HAVE NEVER HAD -- 535 00:19:26,965 --> 00:19:28,066 NEVER HAD A PATIENT PORTAL USE 536 00:19:28,133 --> 00:19:30,035 HAD A DECREASED COMPREHENSION OF 537 00:19:30,102 --> 00:19:31,470 LAB RESULTS AND ALSO COMPARED TO 538 00:19:31,537 --> 00:19:34,773 PEOPLE WITH 50 YEARS OLDER, THE 539 00:19:34,840 --> 00:19:36,375 YOUNGER ADULTS HAD A LOWER 540 00:19:36,441 --> 00:19:39,011 COMPREHENSION OF RESULTS, AND 541 00:19:39,077 --> 00:19:41,413 THESE ARE SOME OF THE 542 00:19:41,480 --> 00:19:42,047 STATISTICAL SIGNIFICANT RESULTS 543 00:19:42,114 --> 00:19:45,384 AND THE NUMERACY SKILLS IS 544 00:19:45,450 --> 00:19:46,652 MARGINALLY SIGNIFICANT AS A 545 00:19:46,718 --> 00:19:47,886 PREDICTOR FOR LAB RESULTS 546 00:19:47,953 --> 00:19:52,691 COMPREHENSION. 547 00:19:52,758 --> 00:19:55,894 AND IN A MORE RECENT STUDY, WE 548 00:19:55,961 --> 00:20:03,201 PRESENTED A AN AMIA, WE WANTED O 549 00:20:03,268 --> 00:20:05,470 PROVIDE MORE INFORMATIC SUPPORT 550 00:20:05,537 --> 00:20:06,505 TO PATIENTS BECAUSE WE THOUGHT 551 00:20:06,572 --> 00:20:07,739 AT THE TIME THAT THERE'S NOT 552 00:20:07,806 --> 00:20:11,209 MUCH KIND OF STANDARDIZED WAY TO 553 00:20:11,276 --> 00:20:13,512 PRESENT PATIENTS A TOOL FOR LAB 554 00:20:13,579 --> 00:20:14,279 RESULTS COMPREHENSION. 555 00:20:14,346 --> 00:20:18,016 SO IN THIS STUDY, WHAT WE DO IS 556 00:20:18,083 --> 00:20:20,152 WE WANT TO DEMONSTRATE THE 557 00:20:20,218 --> 00:20:21,787 FEASIBILITY OF EXTRACTING LAB 558 00:20:21,853 --> 00:20:23,989 RESULTS INTERPRETATION FROM 559 00:20:24,056 --> 00:20:24,723 CONSUMER-FRIENDLY WEBSITE 560 00:20:24,790 --> 00:20:25,958 ARTICLES AND ALSO BUILDING A 561 00:20:26,024 --> 00:20:28,493 FOUNDATION FOR DEVELOPING A TOOL 562 00:20:28,560 --> 00:20:30,228 TO PROVIDE SUPPORT FOR LAB 563 00:20:30,295 --> 00:20:30,996 RESULTS FOR PATIENTS. 564 00:20:31,063 --> 00:20:32,965 SO HERE IS THE GENERAL WORKFLOW 565 00:20:33,031 --> 00:20:36,268 WE FOLLOW, LIKE STANDARDIZED 566 00:20:36,335 --> 00:20:39,871 DATA COLLECTION, ANNOTATION 567 00:20:39,938 --> 00:20:42,040 OF -- ANNOTATION GUIDELINE 568 00:20:42,107 --> 00:20:43,008 DEVELOPMENT AND KRAITION, 569 00:20:43,075 --> 00:20:49,247 QUALITY ASSURANCE, AND THEN 570 00:20:49,314 --> 00:20:51,683 PERFORM RECOGNITION WITH 571 00:20:51,750 --> 00:20:53,051 TRANSFORMERS AND LINKING 572 00:20:53,118 --> 00:20:58,223 ENTITIES WITH VOCABULARIES, WE 573 00:20:58,290 --> 00:21:00,158 WANT TO BE ABLE TO IDENTIFY 574 00:21:00,225 --> 00:21:01,827 THOSE INFORMATION FROM 575 00:21:01,893 --> 00:21:02,794 ELECTRONIC HEALTH RECORDS AND 576 00:21:02,861 --> 00:21:04,863 THEN BE ABLE TO AUTOMATICALLY 577 00:21:04,930 --> 00:21:05,530 PROMPT PATIENTS WITH QUESTIONS 578 00:21:05,597 --> 00:21:07,833 THEY CAN FOLLOW UP WITH THEIR 579 00:21:07,899 --> 00:21:08,367 PHYSICIANS. 580 00:21:08,433 --> 00:21:15,540 SO IN THIS STUDY, WE USE A -- 581 00:21:15,607 --> 00:21:17,042 AND THEN WE -- BECAUSE THEY 582 00:21:17,109 --> 00:21:19,211 SOURCE HAVE CONTENT THAT ARE 583 00:21:19,277 --> 00:21:20,846 MORE FAIRLY GENERALLY 584 00:21:20,912 --> 00:21:24,049 UNDERSTANDABLE BY PATIENTS US 585 00:21:24,116 --> 00:21:25,817 USING -- LANGUAGE AND AS THE 586 00:21:25,884 --> 00:21:27,219 CONTENT ARE VIEWED BY MEDICAL 587 00:21:27,285 --> 00:21:27,653 PROFESSIONALS. 588 00:21:27,719 --> 00:21:32,791 SO WE LOOK AT PRIMARILY THE 589 00:21:32,858 --> 00:21:34,960 SECTIONS ABOUT WHAT DO MY 590 00:21:35,027 --> 00:21:38,797 RESULTS MEAN AND ALSO WHAT DOES 591 00:21:38,864 --> 00:21:42,501 THIS RESULT HAVE OTHER NAMES, 592 00:21:42,567 --> 00:21:44,703 AND OUR TEAM DEVELOPED THIS 593 00:21:44,770 --> 00:21:46,338 ANNOTATED DATASET WITH USING A 594 00:21:46,405 --> 00:21:49,041 TOOL INCEPTION, AND I WORK WITH 595 00:21:49,107 --> 00:21:50,709 MY -- TO DEVELOP SOME EXAMPLE 596 00:21:50,776 --> 00:21:54,312 ANNOTATION AND THEN WE HAVE -- 597 00:21:54,379 --> 00:21:56,515 TO DO ANNOTATION TOGETHER FOR 598 00:21:56,581 --> 00:21:57,816 30-PLUS RECORDS TOGETHER TO 599 00:21:57,883 --> 00:22:03,355 TRAIN THEM AND THEN AFTER THE -- 600 00:22:03,422 --> 00:22:07,125 HIGH WE ASK THEM TO AN TATE THIS 601 00:22:07,192 --> 00:22:08,226 DATASET USING THIS TOOL. 602 00:22:08,293 --> 00:22:10,629 AND THEN ONCE WE GET EVERY 603 00:22:10,696 --> 00:22:12,030 RECORD ANNOTATED BY AT LEAST 604 00:22:12,097 --> 00:22:13,365 THREE PEOPLE, WE PERFORM THE 605 00:22:13,432 --> 00:22:19,838 CURE RANGE OF MOTCURATE THE DAT. 606 00:22:19,905 --> 00:22:22,841 THIS INCLUDES THE LAB NAME, 607 00:22:22,908 --> 00:22:25,043 ALTERNATIVE -- DIFFERENT KIND OF 608 00:22:25,110 --> 00:22:26,545 CONDITIONS LIKE DIFFERENT AGE 609 00:22:26,611 --> 00:22:28,246 GROUP FOR DIFFERENT LAB NORMAL 610 00:22:28,313 --> 00:22:32,050 RANGE AND ALSO WHAT IS THE 611 00:22:32,117 --> 00:22:33,919 INDICATION FOR ABNORMAL LAB TEST 612 00:22:33,985 --> 00:22:34,453 RESULTS. 613 00:22:34,519 --> 00:22:37,856 SO THIS WAY WE'LL BE ABLE TO 614 00:22:37,923 --> 00:22:42,794 VIEW SOME SORT OF LIKE -- SO 615 00:22:42,861 --> 00:22:46,164 THIS IS A GENERAL STATISTICS 616 00:22:46,231 --> 00:22:49,868 ABOUT DIFFERENT -- BECAUSE WE 617 00:22:49,935 --> 00:22:51,570 WANTED TO USE THIS TRAINING 618 00:22:51,636 --> 00:22:54,372 DATA, THIS ANNOTATED DATA AS A 619 00:22:54,439 --> 00:22:58,009 CORPUS TO TRAIN A MODEL FOR 620 00:22:58,076 --> 00:22:58,877 DETECTING SUCH INFORMATION FROM 621 00:22:58,944 --> 00:23:02,013 OTHER RECORDS, SO WE 622 00:23:02,080 --> 00:23:08,120 EXPERIMENTED WITH DIFFERENT -- 623 00:23:08,186 --> 00:23:09,888 TRANSFORMER-BASED MODELS FOR 624 00:23:09,955 --> 00:23:12,157 RECOGNIZING SUCH CONCEPTS. 625 00:23:12,224 --> 00:23:16,027 SO WE TRIED TO FEED IN USING 626 00:23:16,094 --> 00:23:17,362 PARAGRAPH AND SENTENCES, WE 627 00:23:17,429 --> 00:23:18,764 FOUND THAT USING PARAGRAPHS 628 00:23:18,830 --> 00:23:20,599 TYPICALLY GIVE US BETTER 629 00:23:20,665 --> 00:23:23,835 PERFORMANCE IN TERMS OF STRICT 630 00:23:23,902 --> 00:23:25,237 F1 LINEAR MATCH F1, I THINK 631 00:23:25,303 --> 00:23:28,740 THAT'S MAINLY BECAUSE SOMETIMES 632 00:23:28,807 --> 00:23:37,549 THE INFORMATION IN THIG THESE LB 633 00:23:37,616 --> 00:23:39,317 RESULTS -- WE'RE ALSO TRYING TO 634 00:23:39,384 --> 00:23:44,156 LINK THIS ANNOTATED DATA TO 635 00:23:44,222 --> 00:23:48,393 STANDARD VOCABULARIES BASED ON 636 00:23:48,460 --> 00:23:53,331 THE NATURE OF THE CONCEPTS. 637 00:23:53,398 --> 00:23:56,268 AND WE FOUND SNOMED CT HAD A 638 00:23:56,334 --> 00:23:58,670 PRETTY GOOD COVERAGE OF THOSE 639 00:23:58,737 --> 00:24:00,205 DIFFERENT INFORMATION, NAMES IN 640 00:24:00,272 --> 00:24:04,042 THE RECORDS COMPARED TO OTHER 641 00:24:04,109 --> 00:24:04,376 VOCABULARIES. 642 00:24:04,442 --> 00:24:06,645 AND IF WE COMBINE ALL OF THEM, 643 00:24:06,711 --> 00:24:10,715 THE COVERAGE OF THE ANNOTATED 644 00:24:10,782 --> 00:24:13,251 CONCEPTS RANGE BETWEEN 90% -- 645 00:24:13,318 --> 00:24:16,721 RANGE BETWEEN 60% TO 90%. 646 00:24:16,788 --> 00:24:18,423 SO IT SEEMS THAT A LOT OF 647 00:24:18,490 --> 00:24:21,359 ENTITIES CAN BE LINKED TO 648 00:24:21,426 --> 00:24:22,427 STANDARD VOCABULARIES. 649 00:24:22,494 --> 00:24:27,666 SO IN TERMS OF USE CASES, WE 650 00:24:27,732 --> 00:24:32,070 COULD CONSTRUCT MORE 651 00:24:32,137 --> 00:24:34,406 PERSONALIZED QUESTION PROMPTS 652 00:24:34,472 --> 00:24:35,540 FOR PATIENT BASED ON LAB RESULTS 653 00:24:35,607 --> 00:24:37,042 AND ALSO BASED ON THEIR 654 00:24:37,108 --> 00:24:38,944 CONDITIONS THAT WE CAN -- FROM 655 00:24:39,010 --> 00:24:40,345 HEALTH RECORDS, SO WE CAN 656 00:24:40,412 --> 00:24:42,647 CONSTRUCT QUESTIONS LIKE IS MY 657 00:24:42,714 --> 00:24:46,484 ABNORMAL RESULT DUE TO LIVER 658 00:24:46,551 --> 00:24:48,920 DISEASE OR USE OF A PARTICULAR 659 00:24:48,987 --> 00:24:49,988 MEDICATION, AND PATIENTS CAN USE 660 00:24:50,055 --> 00:24:51,523 THIS INFORMATION TO FOLLOW UP 661 00:24:51,590 --> 00:24:53,358 WITH A PHYSICIAN. 662 00:24:53,425 --> 00:24:55,360 AND ALSO AS YOU ALL KNOW, 663 00:24:55,427 --> 00:24:57,896 BECAUSE RECENTLY THERE IS A 664 00:24:57,963 --> 00:25:01,066 REALLY BOOMING PHENOMENON OF A 665 00:25:01,132 --> 00:25:03,668 LARGE MODEL FOR A VARIETY OF 666 00:25:03,735 --> 00:25:06,071 MEDICAL-RELATED USE CASES, AND 667 00:25:06,137 --> 00:25:07,172 WE'RE ALSO TRYING TO ALSO 668 00:25:07,239 --> 00:25:11,209 EXPLORE THE USE OF A M MODEL FOR 669 00:25:11,276 --> 00:25:12,210 LAB TEST RESULTS AND 670 00:25:12,277 --> 00:25:12,711 INTERPRETATION. 671 00:25:12,777 --> 00:25:15,780 SO THIS IS NOT SOMETHING NEW. 672 00:25:15,847 --> 00:25:17,883 THIS A GROUP OF PEOPLE FROM RICE 673 00:25:17,949 --> 00:25:19,217 UNIVERSITY TEXAS A & M, THEY 674 00:25:19,284 --> 00:25:22,554 ACTUALLY DEVELOPED A VERY NICE 675 00:25:22,621 --> 00:25:24,990 TREE EVOLUTION SHOWING THE 676 00:25:25,056 --> 00:25:26,157 EVOLUTION OF LARGE LANGUAGE 677 00:25:26,224 --> 00:25:29,661 MODELS OVER TIME FROM 2018 TO 678 00:25:29,728 --> 00:25:30,562 NOW. 679 00:25:30,629 --> 00:25:31,796 SO WHERE HOPEFULLY YOU'RE GOING 680 00:25:31,863 --> 00:25:35,600 TO SEE MORE MATURE SYSTEMS BEING 681 00:25:35,667 --> 00:25:37,569 DEVELOPED AND ALSO BEING USED 682 00:25:37,636 --> 00:25:41,172 FOR HEALTH-RELATED PURPOSES AND 683 00:25:41,239 --> 00:25:42,707 BUT AS A HEALTH INFORMATICS 684 00:25:42,774 --> 00:25:43,808 RESEARCHER, WE ALSO ARE VERY 685 00:25:43,875 --> 00:25:45,677 CAREFUL ABOUT USING THEM FOR 686 00:25:45,744 --> 00:25:47,646 MEDICAL PURPOSES BECAUSE YOU 687 00:25:47,712 --> 00:25:50,148 KNOW, THIS IS A LOT OF KIND OF 688 00:25:50,215 --> 00:25:52,984 CONCERNS REGARDING THE QUALITY 689 00:25:53,051 --> 00:25:55,820 AND PROBLEMS WITH USE OF THE 690 00:25:55,887 --> 00:25:56,288 SYSTEMS. 691 00:25:56,354 --> 00:25:57,889 SO INITIALLY WHAT WE DID IS WE 692 00:25:57,956 --> 00:26:00,525 LOOKED AT SOME -- WE WANT TO -- 693 00:26:00,592 --> 00:26:03,028 FOR EXAMPLE GENERALLY UNDERSTAND 694 00:26:03,094 --> 00:26:05,030 HOW WELL -- MODELS CAN BE USED 695 00:26:05,096 --> 00:26:07,532 FOR LAB TEST RESULTS 696 00:26:07,599 --> 00:26:10,835 UNDERSTANDING, SO WE ALSO LOOK 697 00:26:10,902 --> 00:26:11,903 AT THE RECENT LITERATURE AND 698 00:26:11,970 --> 00:26:14,773 THERE ARE SOME RECENT PAPERS 699 00:26:14,839 --> 00:26:17,542 THAT ARE ACTUALLY EVALUATING 700 00:26:17,609 --> 00:26:19,811 THE -- OF LARGE ELECTION 701 00:26:19,878 --> 00:26:20,645 MODELS -- LAB RESULTS 702 00:26:20,712 --> 00:26:21,012 INTERPRETATION. 703 00:26:21,079 --> 00:26:24,716 SO THIS PAPER PUBLISHED IN 2013, 704 00:26:24,783 --> 00:26:28,753 2023, THEY FOCUSED ON CASES OF 705 00:26:28,820 --> 00:26:30,088 PATIENTS RECEIVING LAB RESULTS 706 00:26:30,155 --> 00:26:35,460 IN ROUTINE CHECKUPS, AND NOT 707 00:26:35,527 --> 00:26:38,363 HAVING SPECIFICALLY ASKED A 708 00:26:38,430 --> 00:26:40,565 QUESTION TO CHATGPT AND THEY 709 00:26:40,632 --> 00:26:42,200 FOUND USING THIS GENERIC MORE 710 00:26:42,267 --> 00:26:43,168 GENERAL QUESTIONS DEVELOPED BY 711 00:26:43,234 --> 00:26:47,038 THIS GROUP OF EXPERTS, THEY 712 00:26:47,105 --> 00:26:49,874 FOUND THAT LIKE MODELS LIKE 713 00:26:49,941 --> 00:26:52,744 CHATGPT CAN GENERALLY IDENTIFY 714 00:26:52,811 --> 00:26:56,181 DEVIATIONS FROM REFERENCE RANGE 715 00:26:56,247 --> 00:26:57,782 AND ALSO GIVE -- INTERPRETATION 716 00:26:57,849 --> 00:26:58,950 AS WELL AS SOME OVERALL 717 00:26:59,017 --> 00:27:00,485 INTERPRETATION. 718 00:27:00,552 --> 00:27:01,920 THE INTERPRETATION, WE KNOW IS 719 00:27:01,987 --> 00:27:03,888 CONSIDERED AS SUPERFICIAL AND 720 00:27:03,955 --> 00:27:07,759 NOT ALWAYS CORRECT. 721 00:27:07,826 --> 00:27:11,563 AND ALSO SOMETIMES JUST 722 00:27:11,629 --> 00:27:12,764 COHERENTLY, THEY DIDN'T REALLY 723 00:27:12,831 --> 00:27:14,265 MAKE MEANINGFUL SUGGESTIONS 724 00:27:14,332 --> 00:27:16,134 REGARDING FOLLOW-UP DIAGNOSTIC. 725 00:27:16,201 --> 00:27:18,036 ANOTHER GROUP OF PEOPLE FROM 726 00:27:18,103 --> 00:27:21,806 CORNELL AND OTHER INSTITUTIONS, 727 00:27:21,873 --> 00:27:23,808 THEY ALSO LOOK AT HOW CAN 728 00:27:23,875 --> 00:27:25,643 CHATGPT BE ABLE TO PROVIDE 729 00:27:25,710 --> 00:27:26,845 INTERPRETATION TO MORE 730 00:27:26,911 --> 00:27:27,679 PROFESSIONAL QUESTIONS. 731 00:27:27,746 --> 00:27:31,716 THEY ASK MEDICAL DOCTORS TO ASK 732 00:27:31,783 --> 00:27:33,585 QUESTIONS TO CHATGPT ABOUT THEIR 733 00:27:33,651 --> 00:27:35,186 CONCERNS REGARDING PATIENTS' LAB 734 00:27:35,253 --> 00:27:37,722 TEST RESULTS, SO THE QUESTIONS 735 00:27:37,789 --> 00:27:41,292 IN THIS STUDY ARE MORE 736 00:27:41,359 --> 00:27:44,596 COMPLICATED AND MORE HARDER TO 737 00:27:44,662 --> 00:27:45,130 ANSWER. 738 00:27:45,196 --> 00:27:47,132 SO THEY FOUND THAT IN ROUGHLY 739 00:27:47,198 --> 00:27:50,835 50% TIMES, CHATGPT IS ABLE TO 740 00:27:50,902 --> 00:27:53,872 PROVIDE REASONABLY ACCURATE 741 00:27:53,938 --> 00:27:55,874 INTERPRETATION TO THESE 742 00:27:55,940 --> 00:27:56,941 QUESTIONS. 743 00:27:57,008 --> 00:27:59,377 AND FOR US, THE REASON WHY WE DO 744 00:27:59,444 --> 00:28:00,578 THIS PARTLY IS ALSO BECAUSE WE 745 00:28:00,645 --> 00:28:04,749 FOUND THAT MANY PATIENTS, WHEN 746 00:28:04,816 --> 00:28:06,151 THEY RECEIVE LAB RESULTS 747 00:28:06,217 --> 00:28:07,218 INFORMATION, THEY MAY TURN TO 748 00:28:07,285 --> 00:28:08,353 SOCIAL MEDIA FOR ANSWERING 749 00:28:08,420 --> 00:28:09,954 BECAUSE THEY MAY NOT HAVE THE 750 00:28:10,021 --> 00:28:13,258 READY ACCESS TO THEIR DOCTOR, 751 00:28:13,324 --> 00:28:19,364 SOG SOME OFSOSOME OF THEM ASK QN 752 00:28:19,431 --> 00:28:21,766 SOCIAL MEDIA, THE RESPONSES ARE 753 00:28:21,833 --> 00:28:23,401 ALSO QUESTIONABLE BASED ON THE 754 00:28:23,468 --> 00:28:24,636 PREVIOUS LITERATURE, THE QUALITY 755 00:28:24,702 --> 00:28:26,304 REALLY VARIES SIGNIFICANTLY AND 756 00:28:26,371 --> 00:28:28,907 NOT ALL THE RESPONSES FROM THE 757 00:28:28,973 --> 00:28:31,009 SOCIAL MEDIA WEBSITE ARE 758 00:28:31,076 --> 00:28:32,043 ACCURATE OR RELIABLE. 759 00:28:32,110 --> 00:28:36,881 SO WE WANTED TO LOOK AT -- 760 00:28:36,948 --> 00:28:38,083 DATASET, WHICH IS A GENERAL Q & 761 00:28:38,149 --> 00:28:40,952 A DATASET THAT WE FOUND WHETHER 762 00:28:41,019 --> 00:28:42,587 THERE ARE QUESTIONS ABOUT LAB 763 00:28:42,654 --> 00:28:45,590 TEST RESULTS AND LAB RESULTS 764 00:28:45,657 --> 00:28:47,492 INTERPRETATION, AND THEN HOW 765 00:28:47,559 --> 00:28:49,761 WELL MODELS LIKE CHATGPT AND 766 00:28:49,828 --> 00:28:51,396 OTHERS CAN ANSWER THOSE 767 00:28:51,463 --> 00:28:54,599 QUESTIONS RAISED BY PATIENTS 768 00:28:54,666 --> 00:28:56,434 WITHOUT ANY MODIFICATIONS, WE 769 00:28:56,501 --> 00:28:58,336 JUST USE THEIR QUESTIONS AS IS. 770 00:28:58,403 --> 00:29:01,806 SO IN ORDER TO IDENTIFY THOSE 771 00:29:01,873 --> 00:29:03,374 QUESTIONS, WE FILTER QUESTIONS 772 00:29:03,441 --> 00:29:06,077 BY LAB RESULTS. 773 00:29:06,144 --> 00:29:08,146 THEN OUT OF THESE 12,000 774 00:29:08,213 --> 00:29:09,814 QUESTIONS, WE ASKED OUR STUDENTS 775 00:29:09,881 --> 00:29:13,418 TO ANNOTATE 500 QUESTIONS TO 776 00:29:13,485 --> 00:29:15,487 DETERMINE WHETHER THEY ARE LAB 777 00:29:15,553 --> 00:29:17,722 TEST RESULTS -- OFTENTIMES LAB 778 00:29:17,789 --> 00:29:19,858 TEST QUESTIONS, EVEN THOUGH THEY 779 00:29:19,924 --> 00:29:20,859 MAY CONTAIN LAB TEST INFORMATION 780 00:29:20,925 --> 00:29:23,461 OR LAB TEST NAMES, THEY MAY NOT 781 00:29:23,528 --> 00:29:27,899 BE TEUT TEULLY ABOUT LAB RESULTS 782 00:29:27,966 --> 00:29:28,500 INTERPRETATION. 783 00:29:28,566 --> 00:29:31,903 SO THEN WE -- ARTIFICIAL 784 00:29:31,970 --> 00:29:34,606 LEARNING MODELS FOR CLASSIFYING 785 00:29:34,672 --> 00:29:41,312 THOSE QUESTIONS INTO LAB 786 00:29:41,379 --> 00:29:42,313 RESULTS -- RELATED AND NOT. 787 00:29:42,380 --> 00:29:43,515 SO WE EXPERIMENTED WITH ALL OF 788 00:29:43,581 --> 00:29:45,817 THESE MODELS AND BASED ON 789 00:29:45,884 --> 00:29:50,722 RESULTS, WE FOUND THAT 790 00:29:50,788 --> 00:29:52,790 CLINICALBERT IS BETTER IN 791 00:29:52,857 --> 00:29:55,627 DETERMINING AND CLASSIFYING THE 792 00:29:55,693 --> 00:30:01,533 LAB TEST QUESTIONS, AND WE 793 00:30:01,599 --> 00:30:02,734 COMBINE THE WITH MACHINE 794 00:30:02,800 --> 00:30:04,469 LEARNING METHODS, WE CAN 795 00:30:04,536 --> 00:30:07,539 GENERALLY HAVE A PRETTY HIGH 796 00:30:07,605 --> 00:30:09,140 ACCURACY IN DETERMINING THE LAB 797 00:30:09,207 --> 00:30:11,543 RESULTS QUESTIONS CATEGORY. 798 00:30:11,609 --> 00:30:15,413 AND THEN WE WORK WITH OUR 799 00:30:15,480 --> 00:30:16,514 PRIMARY CARE PHYSICIAN 800 00:30:16,581 --> 00:30:18,816 COLLABORATORS ON THE TEAM TO 801 00:30:18,883 --> 00:30:20,451 IDENTIFY 53 QUESTIONS THAT 802 00:30:20,518 --> 00:30:23,087 CONTAIN RESULTS OF BLOOD AND 803 00:30:23,154 --> 00:30:25,823 URINE LAB TEST ON MAJOR PANELS 804 00:30:25,890 --> 00:30:29,327 INCLUDING COMPLETE -- METABOLIC 805 00:30:29,394 --> 00:30:31,729 PANEL, THYROID FUNCTION, EARLY 806 00:30:31,796 --> 00:30:32,797 MENOPAUSE OR -- PANEL. 807 00:30:32,864 --> 00:30:37,535 THEN WE EXAMINED THE AVAILABLE 808 00:30:37,602 --> 00:30:39,604 LARGE MODELS AND WHICH MODELS 809 00:30:39,671 --> 00:30:42,006 SHOULD BE EVALUATED IN THE 810 00:30:42,073 --> 00:30:43,408 STUDY. 811 00:30:43,474 --> 00:30:53,051 WE FOUND CHAT CHATGPT -- THAT CE 812 00:30:53,117 --> 00:30:54,452 ACCESSED THROUGH WEB PORTAL AND 813 00:30:54,519 --> 00:30:59,090 ALSO API CAN BE USED, ALSO 814 00:30:59,157 --> 00:31:02,427 MEDALPACA, LLAMA2 BY META AND 815 00:31:02,493 --> 00:31:11,002 ALSO ORCA MINI, BASED ON LLAMA 816 00:31:11,069 --> 00:31:14,405 2, BECAUSE WE THOUGHT THE 817 00:31:14,472 --> 00:31:16,107 RESPONSES ARE REASONABLE, 818 00:31:16,174 --> 00:31:17,108 GENERALLY SPEAKING. 819 00:31:17,175 --> 00:31:21,946 SO HERE IS AN EXAMPLE, SO AFTER 820 00:31:22,013 --> 00:31:25,316 WE GENERATE RESPONSES FROM THESE 821 00:31:25,383 --> 00:31:30,588 MODELS AND ANALYZED THEIR 822 00:31:30,655 --> 00:31:33,558 QUALITY, YOU SEE -- MORE FOCUSED 823 00:31:33,625 --> 00:31:34,959 ON ASSESSING THE SIMILARITY OF 824 00:31:35,026 --> 00:31:35,627 THOSE RESPONSES. 825 00:31:35,693 --> 00:31:39,264 AND THEN WE ALSO ASK EXPERTS 826 00:31:39,330 --> 00:31:44,469 WHERE PHYSICIANS AND ALSO MD 827 00:31:44,535 --> 00:31:46,104 TRAINED POSTDOC WITH MEDICAL 828 00:31:46,170 --> 00:31:47,905 BACKGROUND TO MANUALLY EVALUATE 829 00:31:47,972 --> 00:31:52,443 THE RESPONSES, SOME WERE ASKED 830 00:31:52,510 --> 00:31:54,846 THAT AND THEN COMPARE RESULTS OF 831 00:31:54,912 --> 00:31:55,580 THOSE DIMENSIONS. 832 00:31:55,647 --> 00:31:56,814 SO HERE ARE SOME OF THE 833 00:31:56,881 --> 00:31:58,316 RESPONSES FROM LIKE MODELS. 834 00:31:58,383 --> 00:32:01,085 SO THIS ONE IS A QUESTION THAT 835 00:32:01,152 --> 00:32:04,155 WE FOUND WHICH ASKS FOR SOME 836 00:32:04,222 --> 00:32:06,691 URINE TEST RESULTS ON CREATININE 837 00:32:06,758 --> 00:32:10,828 VALUE, AND ALSO 838 00:32:10,895 --> 00:32:11,863 PROTEIN/CREATININE RATIO, AND 839 00:32:11,929 --> 00:32:14,999 THIS IS THE RESPONSE WE GOT FROM 840 00:32:15,066 --> 00:32:16,934 LLAMA 2 MODEL, AND ALSO WE TOOK 841 00:32:17,001 --> 00:32:18,803 THIS SCREEN SHOT FROM THE ONLINE 842 00:32:18,870 --> 00:32:23,441 SURVEY BECAUSE WE GENERATE FOR 843 00:32:23,508 --> 00:32:25,476 PHYSICIAN TO EVALUATE AND WE 844 00:32:25,543 --> 00:32:27,545 DON'T WANT THE PHYSICIAN TO KNOW 845 00:32:27,612 --> 00:32:31,182 WHICH MODELS WE USED SO WE HIDE 846 00:32:31,249 --> 00:32:33,351 THIS IN THIS SCREEN. 847 00:32:33,418 --> 00:32:36,988 SO THIS IS THE RESPONSE FROM 848 00:32:37,055 --> 00:32:38,323 ORCA MINI MODEL AND YOU CAN SEE 849 00:32:38,389 --> 00:32:41,359 THAT THEY HAVE PROVIDED A FAIRLY 850 00:32:41,426 --> 00:32:43,761 COMPREHENSIVE RESPONSES TO THESE 851 00:32:43,828 --> 00:32:46,164 QUESTIONS WITH LOTS OF 852 00:32:46,230 --> 00:32:49,100 EXPLANATIONS, AND ALSO THIS IS 853 00:32:49,167 --> 00:32:53,705 THE RESPONSES FROM CHATGPT, 854 00:32:53,771 --> 00:32:57,342 GPT-4, AND GPT-4 EVEN TEND TO BE 855 00:32:57,408 --> 00:33:02,380 MORE KR COMPREHENSIVE LONGER. 856 00:33:02,447 --> 00:33:04,415 BUT LOOKING AT THE RESPONSES, 857 00:33:04,482 --> 00:33:07,385 THERE WAS -- TEND TO ALSO 858 00:33:07,452 --> 00:33:09,587 PROVIDE A LOT OF DISCLAIMERS 859 00:33:09,654 --> 00:33:12,690 LIKE SAYING I'M NOT A DOCTOR AND 860 00:33:12,757 --> 00:33:14,625 YOU SHOULD CONSUL YOUR PHYSICIAN 861 00:33:14,692 --> 00:33:16,361 PROVIDERS TO GET A MORE ACCURATE 862 00:33:16,427 --> 00:33:17,695 DIAGNOSIS AND TREATMENT OPTIONS 863 00:33:17,762 --> 00:33:20,631 BASED ON YOUR COMPLETE 864 00:33:20,698 --> 00:33:21,399 EVALUATION MEDICAL HISTORY. 865 00:33:21,466 --> 00:33:22,734 SO THEY'RE REALLY TRYING TO MAKE 866 00:33:22,800 --> 00:33:24,702 SURE THAT THEY DON'T HAVE ANY 867 00:33:24,769 --> 00:33:27,138 TROUBLE PROVIDING 868 00:33:27,205 --> 00:33:28,005 MEDICAL-RELATED RESPONSES TO 869 00:33:28,072 --> 00:33:28,973 THESE QUESTIONS. 870 00:33:29,040 --> 00:33:32,910 AND THIS IS THE RESULT FROM 871 00:33:32,977 --> 00:33:33,344 MEDALPACA. 872 00:33:33,411 --> 00:33:35,346 SOMEHOW COMPARED TO OTHER LIKE 873 00:33:35,413 --> 00:33:37,448 MODEL, MEDALPACA TEND TO BE VERY 874 00:33:37,515 --> 00:33:39,784 SHORT AND VERY SUCCINCT 875 00:33:39,851 --> 00:33:42,620 RESPONSES COMPARED TO OTHER 876 00:33:42,687 --> 00:33:47,525 MODEL, AND WE'VE FOUND ALSO THAT 877 00:33:47,592 --> 00:33:49,093 MEDALPACA'S TONE TENDS TO BE A 878 00:33:49,160 --> 00:33:50,595 BIT MORE AGGRESSIVE THAN OTHER 879 00:33:50,661 --> 00:33:51,295 LIKE MODELS. 880 00:33:51,362 --> 00:33:53,598 HERE ARE SOME EXAMPLES OF HUMAN 881 00:33:53,664 --> 00:33:54,031 ANSWERS. 882 00:33:54,098 --> 00:33:56,667 WE WILL PROBABLY KNOW HUMAN 883 00:33:56,734 --> 00:34:00,772 ANSWERS SOMETIMES ARE BETTER. 884 00:34:00,838 --> 00:34:02,907 IN SOME OF THE CASES WE FOUND 885 00:34:02,974 --> 00:34:04,909 THE HUMAN ANSWERS ACTUALLY ARE 886 00:34:04,976 --> 00:34:06,177 SUCCINCT AND SHORT AND ALSO MAY 887 00:34:06,244 --> 00:34:08,446 BE FOCUSED ON ONLY ONE ASPECT OF 888 00:34:08,513 --> 00:34:13,151 THE QUESTION, NOT PROVIDING 889 00:34:13,217 --> 00:34:14,652 EVIDENCE TO THESE RESPONSES. 890 00:34:14,719 --> 00:34:16,254 AND BECAUSE MOST LIKELY, I THINK 891 00:34:16,320 --> 00:34:18,389 MOST OF THESE ANSWERS CAN BE 892 00:34:18,456 --> 00:34:22,693 ANSWERED BY PURE PATIENTS AND 893 00:34:22,760 --> 00:34:23,528 NON-MEDICAL PROFESSIONALS. 894 00:34:23,594 --> 00:34:25,062 SO FIRST WE LOOKED AT THEIR 895 00:34:25,129 --> 00:34:26,497 LENGTH, WE FOUND THAT 896 00:34:26,564 --> 00:34:27,899 GPT4 TENDED TO HAVE LONGER 897 00:34:27,965 --> 00:34:31,536 RESPONSES THAN OTHER MODELS AND 898 00:34:31,602 --> 00:34:34,071 THE RESPONSES FROM USERS, 899 00:34:34,138 --> 00:34:35,339 ANSWERS TEND TO BE SHORTER WITH 900 00:34:35,406 --> 00:34:37,175 RESPECT TO ALL THE THREE 901 00:34:37,241 --> 00:34:38,109 ASPECTS, THE AVERAGE CHARACTER 902 00:34:38,176 --> 00:34:43,014 IN THE RESPONSE, ALSO AVERAGE 903 00:34:43,080 --> 00:34:44,682 WORKOUT, AND AVERAGE COUNT FOR 904 00:34:44,749 --> 00:34:46,517 GPT RESPONSES IS LIKE FOUR TIMES 905 00:34:46,584 --> 00:34:51,122 THAT OF HUMAN USES --LY SPONSES 906 00:34:51,189 --> 00:34:54,158 AND SOME OF OUR HUMAN EXPERT 907 00:34:54,225 --> 00:35:00,498 EVALUATORS ACTUALLY FOUND 908 00:35:00,565 --> 00:35:01,766 GPT-4 IS SOMETIMES TOO LENGTHY 909 00:35:01,833 --> 00:35:05,336 AND SOMETIMES GETS A LITTLE 910 00:35:05,403 --> 00:35:06,304 IRRELEVANT. 911 00:35:06,370 --> 00:35:08,706 WE'LL FIRST LOOK AT TEXTUAL AND 912 00:35:08,773 --> 00:35:10,575 SOMATIC SIMILARITY OF THE RAO 913 00:35:10,641 --> 00:35:11,909 SPONSES BECAUSE THOSE KIND OF 914 00:35:11,976 --> 00:35:12,743 METRICS WERE PREVIOUSLY 915 00:35:12,810 --> 00:35:17,648 OFTENTIMES USED TO EVALUATE 916 00:35:17,715 --> 00:35:18,549 ANSWERING RESULTS AND THE USE 917 00:35:18,616 --> 00:35:20,418 CASES FOR THOSE ARE TYPICALLY -- 918 00:35:20,485 --> 00:35:24,522 THEY HAVE THE STANDARDIZED QA 919 00:35:24,589 --> 00:35:27,992 BENCHMARK DATASET, THE ANSWERS 920 00:35:28,059 --> 00:35:30,428 ARE VERY HIGH QUALITY, SO THEY 921 00:35:30,495 --> 00:35:32,497 CAN EVALUATE DURING THE 922 00:35:32,563 --> 00:35:33,464 GENERATED RESPONSES ARE GOING TO 923 00:35:33,531 --> 00:35:39,604 BE OF A SIMILAR QUALITY TO THE 924 00:35:39,670 --> 00:35:40,571 REFERENCE -- AMONG THOSE 925 00:35:40,638 --> 00:35:42,240 METRICS. 926 00:35:42,306 --> 00:35:48,946 THESE METHODS ARE MOSTLY SUB -- 927 00:35:49,013 --> 00:35:59,290 STRUCTURE OF A -- THEY LOOK AT 928 00:35:59,357 --> 00:36:01,158 EMBEDDING OF THE TOKENS, SO THEY 929 00:36:01,225 --> 00:36:03,561 LOOK AT HOW SIMILAR TWO 930 00:36:03,628 --> 00:36:06,297 SENTENCES IS WITH RESPECT TO THE 931 00:36:06,364 --> 00:36:11,502 SUM OF -- TOKENS SO WE FOUND OUT 932 00:36:11,569 --> 00:36:15,506 FIRST SCORE IS GENERALLY DOING A 933 00:36:15,573 --> 00:36:19,577 BETTER JOB IN EVALUATING 934 00:36:19,644 --> 00:36:21,212 SEMANTIC SIMILARITY BETWEEN THE 935 00:36:21,279 --> 00:36:22,613 RESPONSES. 936 00:36:22,680 --> 00:36:23,814 THE HUMAN RESPONSES ARE VERY 937 00:36:23,881 --> 00:36:26,083 SHORT COMPARED TO LIKE MODEL 938 00:36:26,150 --> 00:36:28,019 RESPONSES, AND THESE ARE RESULTS 939 00:36:28,085 --> 00:36:30,087 WHERE WE USE DIFFERENT WREE F 940 00:36:30,154 --> 00:36:31,522 RESPONSES GENERATED BY EITHER 941 00:36:31,589 --> 00:36:34,025 THE HUMAN OR LENGTH MODELS AS 942 00:36:34,091 --> 00:36:35,326 REFERENCE ANSWER AND WE WANT TO 943 00:36:35,393 --> 00:36:37,094 SEE WHICH ANSWERS ARE SIMILAR TO 944 00:36:37,161 --> 00:36:40,464 EACH OTHER, AND WE FOUND IF WE 945 00:36:40,531 --> 00:36:44,936 USE GPT4 AS A REFERENCE -- 946 00:36:45,002 --> 00:36:49,340 RESPONSE, THE RESUMM RESPONSE FM 947 00:36:49,407 --> 00:36:50,741 LLAMA2 IS CONSIDERED THE MOST 948 00:36:50,808 --> 00:36:52,276 SIMILAR ONE ACCORDING TO FIVE 949 00:36:52,343 --> 00:36:56,113 OUT OF SIX METRICS, AND ALSO 950 00:36:56,180 --> 00:36:58,416 LLAMA 2 AND OCRA MINI GENERATED 951 00:36:58,482 --> 00:37:00,184 RESPONSES THAT ARE SIMILAR TO -- 952 00:37:00,251 --> 00:37:04,121 WERE SIMILAR. 953 00:37:04,188 --> 00:37:08,392 BUT -- GENERATED SCORED LOWER AS 954 00:37:08,459 --> 00:37:10,394 A REFERENCE, AND WE ALSO FOUND 955 00:37:10,461 --> 00:37:12,897 THAT HUMAN ANSWERS FROM -- DATA 956 00:37:12,964 --> 00:37:16,067 WERE SCORED LOWEST AND LEAST 957 00:37:16,133 --> 00:37:18,469 SIMILAR TO OTHER LIKE MODEL 958 00:37:18,536 --> 00:37:20,671 GENERATED RESPONSES. 959 00:37:20,738 --> 00:37:28,112 GPT1, YOU SEE YAHOO ANSWERS HAVE 960 00:37:28,179 --> 00:37:29,614 A LOW SCORE. 961 00:37:29,680 --> 00:37:32,717 AND FOR THE HUMAN EVALUATION, WE 962 00:37:32,783 --> 00:37:35,052 LOOKED AT PREVIOUS LEADER TERM, 963 00:37:35,119 --> 00:37:36,587 AND WE WANT TO LOOK AT WHAT KIND 964 00:37:36,654 --> 00:37:38,756 OF METRICS WERE PREVIOUSLY 965 00:37:38,823 --> 00:37:41,125 CONSIDERED BY THE PRIVATE 966 00:37:41,192 --> 00:37:42,593 LITERATURE FOR LIKE MODEL 967 00:37:42,660 --> 00:37:43,694 GENERATED RESPONSES EVALUATION 968 00:37:43,761 --> 00:37:46,197 AND WE FOUND RELEVANCE 969 00:37:46,263 --> 00:37:48,833 CORRECTNESS AND HELPFULNESS AND 970 00:37:48,899 --> 00:37:50,868 POTENTIAL HARM FOR SAFETY ARE 971 00:37:50,935 --> 00:37:51,836 GENERALLY CONSIDERED AS 972 00:37:51,902 --> 00:37:56,040 IMPORTANT FACTORS FO FOR EVALUAG 973 00:37:56,107 --> 00:37:57,141 RESPONSES BECAUSE RELEVANCE 974 00:37:57,208 --> 00:38:00,277 TENDS TO FOCUS ON THE -- BETWEEN 975 00:38:00,344 --> 00:38:02,813 INTERPRETATION, AI-GENERATED 976 00:38:02,880 --> 00:38:07,818 RESULTS, WHETHER RESULTS ARE 977 00:38:07,885 --> 00:38:11,722 RELEVANT TO THE QUESTION. 978 00:38:11,789 --> 00:38:15,926 -- ACCURACY, TRUTHFULNESS AND 979 00:38:15,993 --> 00:38:19,764 TECHNICAL ACCURACY OF THE 980 00:38:19,830 --> 00:38:22,166 INTERPRETATION. 981 00:38:22,233 --> 00:38:23,534 -- COULD BE CONSIDERED A VERY 982 00:38:23,601 --> 00:38:25,102 RELEVANT AND VERY IMPORTANT 983 00:38:25,169 --> 00:38:27,104 ASPECT BECAUSE IT CONSIDERED THE 984 00:38:27,171 --> 00:38:30,741 ABILITY OF THE MODEL TO PROVE -- 985 00:38:30,808 --> 00:38:31,642 PROVIDE SOME INSIGHTS TO 986 00:38:31,709 --> 00:38:35,813 PATIENTS, TO LAY PEOPLE, AND HOW 987 00:38:35,880 --> 00:38:36,981 HELPFUL THE RESULTS WOULD BE 988 00:38:37,048 --> 00:38:38,749 ALSO IN TERMS OF THE LEVEL OF 989 00:38:38,816 --> 00:38:40,384 SUGGESTIONS AND WHETHER THEY 990 00:38:40,451 --> 00:38:41,819 CONTINUE TO BRING USEFUL 991 00:38:41,886 --> 00:38:43,954 INFORMATION TO PATIENTS. 992 00:38:44,021 --> 00:38:47,958 AND CERTAINLY WE DON'T WANT THE 993 00:38:48,025 --> 00:38:50,661 RESULTS TO BE HARMFUL SO THIS 994 00:38:50,728 --> 00:38:52,163 DIMENSION IS IMPORTANT. 995 00:38:52,229 --> 00:38:54,298 SO WE GIVE A 5 LIKENESS SCALE TO 996 00:38:54,365 --> 00:38:59,003 THESE RESPONSES SO WE HAVE -- SO 997 00:38:59,070 --> 00:39:02,506 WE USE SEVEN QUESTIONS OUT OF 998 00:39:02,573 --> 00:39:05,342 THE -- IN TOTAL WE ASK EU MAN 999 00:39:05,409 --> 00:39:08,245 EVALUATORS TO EVALUATE 35 1000 00:39:08,312 --> 00:39:10,281 RESPONSES, AND ALSO EACH 1001 00:39:10,347 --> 00:39:11,449 RESPONSE, WE ASK THEM TO 1002 00:39:11,515 --> 00:39:13,117 EVALUATE ON THESE FOUR 1003 00:39:13,184 --> 00:39:13,584 DIMENSIONS. 1004 00:39:13,651 --> 00:39:16,187 AND OUR SCORE IS GOING FROM 1 TO 1005 00:39:16,253 --> 00:39:17,421 5 GOING FROM VERY HIGH TO VERY 1006 00:39:17,488 --> 00:39:19,356 LOW, SO IT'S A LITTLE 1007 00:39:19,423 --> 00:39:23,294 COUNTERINTUITIVE BASED ON WHAT 1008 00:39:23,360 --> 00:39:25,262 TYPICALLY OTHERS -- WE EACH HAVE 1009 00:39:25,329 --> 00:39:28,232 TO POTENTIALLY MODIFY THIS. 1010 00:39:28,299 --> 00:39:29,967 SO THE LOWER THE BETTER, 1011 00:39:30,034 --> 00:39:30,835 GENERALLY SPEAKING. 1012 00:39:30,901 --> 00:39:33,370 WE FOUND COMPARED TO OTHER 1013 00:39:33,437 --> 00:39:36,307 MODELS, CHATGPT 4 TEND TO GIVE 1014 00:39:36,373 --> 00:39:41,112 BETTER -- IN DIMENSIONS AND THE 1015 00:39:41,178 --> 00:39:44,582 OTHERS ARE COMPARED TO CHATGPT 1016 00:39:44,648 --> 00:39:50,121 4, LLAMA 2 TEND TO -- BETTER 1017 00:39:50,187 --> 00:39:54,925 RESULTS COMPARED TO HUMAN 1018 00:39:54,992 --> 00:39:55,159 ANSWER. 1019 00:39:55,226 --> 00:39:56,761 FROM YOUTH RESPONSES, SOME 1020 00:39:56,827 --> 00:39:58,763 MEDICAL DOCTORS, PHYSICIANS, 1021 00:39:58,829 --> 00:40:01,031 THEY PROVIDE SOME GENERAL 1022 00:40:01,098 --> 00:40:02,466 FEEDBACK ON THESE RESPONSES. 1023 00:40:02,533 --> 00:40:05,369 THEY'RE SAYING FOR EXAMPLE LLA 1024 00:40:05,436 --> 00:40:10,441 LLAMA2, GENERALLY IS ABLE, 1025 00:40:10,508 --> 00:40:12,643 PROVIDE ALTERNATIVE MANAGEMENT 1026 00:40:12,710 --> 00:40:17,248 AND ALSO SHOW SOME LEVELS OF 1027 00:40:17,314 --> 00:40:21,485 EMPATHY AND -- ALSO CAN EXPLAIN 1028 00:40:21,552 --> 00:40:22,920 THE RESULTS IN DETAIL, BUT IN 1029 00:40:22,987 --> 00:40:25,122 SOME CASES, IT MAY BE A LITTLE 1030 00:40:25,189 --> 00:40:27,458 TOO AGGRESSIVE REGARDING THE 1031 00:40:27,525 --> 00:40:29,827 RECOMMENDATION, AND FOR GPT-4, 1032 00:40:29,894 --> 00:40:32,897 THEY CAN PROVIDE THE MOST 1033 00:40:32,963 --> 00:40:34,565 COMPREHENSIVE EX-PLAMATION OF 1034 00:40:34,632 --> 00:40:35,733 POSSIBLE CAUSES OF LAB TEST 1035 00:40:35,800 --> 00:40:39,904 RESULTS AND ESPECIALLY ABNORMAL 1036 00:40:39,970 --> 00:40:43,340 RESULTS, AND FOCUS ON THE 1037 00:40:43,407 --> 00:40:44,775 RECOMMENDATION BY -- SOMETIMES 1038 00:40:44,842 --> 00:40:47,111 GET TOO WORDY, THE STATEMENT 1039 00:40:47,178 --> 00:40:54,785 SEEMS TO BE SO SUPERFICIAL AND 1040 00:40:54,852 --> 00:40:55,753 MAY NOT PROVIDE ENOUGH 1041 00:40:55,820 --> 00:40:56,854 INFORMATION AND MAY NOT BE SAFE 1042 00:40:56,921 --> 00:40:57,421 FOR PATIENTS. 1043 00:40:57,488 --> 00:40:59,623 AND ALSO OUR REVIEWERS FOUND 1044 00:40:59,690 --> 00:41:01,959 THAT -- ANSWERS TEND TO FOCUS ON 1045 00:41:02,026 --> 00:41:03,260 ONE ASPECT OF A SCENARIO AND 1046 00:41:03,327 --> 00:41:05,129 IT'S NOT HELPFUL IN MANY CASES 1047 00:41:05,196 --> 00:41:06,764 IN COMPREHENSIVE CARE, AND ALSO 1048 00:41:06,831 --> 00:41:10,167 FOCUSED ON SOME ISOLATED LAB 1049 00:41:10,234 --> 00:41:11,135 VALUES. 1050 00:41:11,202 --> 00:41:15,973 SO WE ALSO LOOK AT SOME MORE 1051 00:41:16,040 --> 00:41:17,942 SPECIFIC KIND OF INTERPRETATION. 1052 00:41:18,008 --> 00:41:20,711 LIKE FOR EXAMPLE, FOR THIS 1053 00:41:20,778 --> 00:41:24,381 QUESTION GIVEN BY A PATIENT, WE 1054 00:41:24,448 --> 00:41:26,684 LOOK AT CHATGPT RESPONSES AND WE 1055 00:41:26,750 --> 00:41:28,285 ALSO FOUND THAT GPT RESPONSES 1056 00:41:28,352 --> 00:41:29,653 TEND TO BE SENSITIVE TO THE 1057 00:41:29,720 --> 00:41:30,154 PROMPT. 1058 00:41:30,221 --> 00:41:32,823 FOR EXAMPLE, IF YOU GIVE A 1059 00:41:32,890 --> 00:41:33,757 QUESTION WITHOUT NORMAL 1060 00:41:33,824 --> 00:41:35,860 REFERENCE RANGE, THEY TYPICALLY 1061 00:41:35,926 --> 00:41:37,394 WILL NOT MENTION THE NORMAL 1062 00:41:37,461 --> 00:41:38,529 RANGE IN THE RESPONSE. 1063 00:41:38,596 --> 00:41:41,198 BUT IF YOU PROVIDE REFERENCE 1064 00:41:41,265 --> 00:41:43,267 RANGE IN YOUR QUESTION, THEY 1065 00:41:43,334 --> 00:41:46,737 WILL TALK ABOUT HOW YOUR VALUE 1066 00:41:46,804 --> 00:41:48,606 IS CORRESPONDING TO THOSE VALUE 1067 00:41:48,672 --> 00:41:57,815 RANGE. 1068 00:41:57,882 --> 00:42:00,084 OUR DOCTOR LOOKED AT THESE 1069 00:42:00,150 --> 00:42:02,119 RESPONSES AND FOUND THEY COULD 1070 00:42:02,186 --> 00:42:05,990 PROB BLIL GIVE MORE -- 1071 00:42:06,056 --> 00:42:09,660 ESPECIALLY WHEN HE SEES THIS 1072 00:42:09,727 --> 00:42:11,028 TOTAL CHOLESTEROL VALUE IS MUCH 1073 00:42:11,095 --> 00:42:12,563 LOWER THAN THE NORMAL REFERENCE 1074 00:42:12,630 --> 00:42:14,131 RANGE, AND HE SUGGESTS THAT THIS 1075 00:42:14,198 --> 00:42:15,766 COULD BE INDICATIVE OF 1076 00:42:15,833 --> 00:42:18,802 MEDICATION OR MEDICAL CONDITION 1077 00:42:18,869 --> 00:42:21,405 OR LAB ERROR, IT IS IMPORTANT TO 1078 00:42:21,472 --> 00:42:23,641 RECHECK SUCH UNUSUAL LOW VALUE 1079 00:42:23,707 --> 00:42:24,875 AND CONSIDER CLINICAL 1080 00:42:24,942 --> 00:42:25,175 CORRELATION. 1081 00:42:25,242 --> 00:42:26,810 BUT THESE ARE NOT REALLY MUCH 1082 00:42:26,877 --> 00:42:31,882 MENTIONED IN THIS CHATGPT 1083 00:42:31,949 --> 00:42:33,450 RESULTS, SO WE THINK THAT IS 1084 00:42:33,517 --> 00:42:39,023 IMPORTANT TO BE ABLE TO REALLY 1085 00:42:39,089 --> 00:42:40,190 GENERATE THESE RESPONSES BUT AT 1086 00:42:40,257 --> 00:42:44,028 THE SAME TIME ASK HUMAN EXPERT 1087 00:42:44,094 --> 00:42:45,796 OR DOCTORS TO EVALUATE RESPONSES 1088 00:42:45,863 --> 00:42:47,898 AND PROVIDE MORE INSIGHTS TO 1089 00:42:47,965 --> 00:42:50,367 PATIENT OR TO REMOVE POSSIBLY 1090 00:42:50,434 --> 00:42:55,306 INACCURATE INFORMATION. 1091 00:42:55,372 --> 00:42:56,640 SO GENERALLY SPEAKING THE MAJOR 1092 00:42:56,707 --> 00:42:58,008 FINDING WE OBTAINED FROM THIS 1093 00:42:58,075 --> 00:43:01,278 STUDY IS WE FOUND 1094 00:43:01,345 --> 00:43:02,446 GPT-4 OUTPERFORMED OTHER SIMILAR 1095 00:43:02,513 --> 00:43:04,014 MODELS TESTED BASED ON THEIR 1096 00:43:04,081 --> 00:43:05,182 EXPERT EVALUATION, AND ALSO 1097 00:43:05,249 --> 00:43:06,684 HUMAN ANSWERS FROM SOCIAL Q & A 1098 00:43:06,750 --> 00:43:08,319 ARE GENERALLY SHORT AND OFTEN 1099 00:43:08,385 --> 00:43:10,454 FOCUSED ON ONE ASPECT ONLY AND 1100 00:43:10,521 --> 00:43:11,255 POSSIBLY INACCURATE. 1101 00:43:11,322 --> 00:43:15,192 SO THEY'RE NOT GOING TO BE GOOD 1102 00:43:15,259 --> 00:43:18,796 AS GOLD STANDARD ANSWER FOR 1103 00:43:18,862 --> 00:43:21,098 EVALUATION PURPOSES SO WE HAVE 1104 00:43:21,165 --> 00:43:25,569 TO BE CREATIVE TO FIGURE OUT HOW 1105 00:43:25,636 --> 00:43:28,339 TO CREATE BETTER RESPONSES. 1106 00:43:28,405 --> 00:43:31,375 THERE ARE LIMITATIONS REGARDING 1107 00:43:31,442 --> 00:43:34,211 THE MODELS AS IT'S NOT VERY 1108 00:43:34,278 --> 00:43:35,746 INDIVIDUALIZED AND IT REALLY 1109 00:43:35,813 --> 00:43:37,448 DEPENDS ON THE QUALITY OF THE 1110 00:43:37,514 --> 00:43:37,681 PROMPT. 1111 00:43:37,748 --> 00:43:39,416 IF THE PROMPT DOESN'T PROVIDE 1112 00:43:39,483 --> 00:43:41,118 PERTINENT INFORMATION TO THESE 1113 00:43:41,185 --> 00:43:44,088 LAB RESULTS, LIKE MODELS 1114 00:43:44,154 --> 00:43:45,122 CERTAINLY MIGHT NOT BE ABLE TO 1115 00:43:45,189 --> 00:43:47,491 ASK QUESTIONS, TO ASK FOR 1116 00:43:47,558 --> 00:43:49,893 ADDITIONAL INFORMATION. 1117 00:43:49,960 --> 00:43:51,729 AND ALSO WEAR NOT SURE EXACTLY 1118 00:43:51,795 --> 00:43:53,130 WHAT KIND OF SOURCES WERE THEY 1119 00:43:53,197 --> 00:43:56,800 USED TO GENERATE RESPONSES, AND 1120 00:43:56,867 --> 00:43:59,770 ALSO SO THERE'S A POSSIBILITY 1121 00:43:59,837 --> 00:44:01,839 THEY MAY NOT USE -- ALWAYS USE 1122 00:44:01,905 --> 00:44:03,407 THE MOST CREDIBLE INFORMATION 1123 00:44:03,474 --> 00:44:06,377 SOURCES TO PRODUCE RESPONSES. 1124 00:44:06,443 --> 00:44:08,145 SO WE THINK THAT VALIDATION BY 1125 00:44:08,212 --> 00:44:09,546 MEDICAL EXPERTS IS STILL REALLY 1126 00:44:09,613 --> 00:44:13,717 VERY MUCH NEEDED TO REDUCE 1127 00:44:13,784 --> 00:44:14,718 INACCURATE INFORMATION. 1128 00:44:14,785 --> 00:44:17,087 AND REGARDING NEXT STEPS, WE 1129 00:44:17,154 --> 00:44:18,889 THINK THAT IT'S IMPORTANT TO BE 1130 00:44:18,956 --> 00:44:22,593 ABLE TO PROVIDE MORE FACTORS 1131 00:44:22,659 --> 00:44:23,927 INTO NORMAL RANGE UNDERSTANDING 1132 00:44:23,994 --> 00:44:26,563 BECAUSE A LOT OF LAB TESTS 1133 00:44:26,630 --> 00:44:28,165 ACTUALLY THEIR VALUE RANGE IS 1134 00:44:28,232 --> 00:44:29,400 ALSO DEPENDING ON THE GENDER AND 1135 00:44:29,466 --> 00:44:34,338 AGE GROUP AND OTHER FACTORS LI 1136 00:44:34,405 --> 00:44:36,373 LIKE -- PERIOD AND MENOPAUSAL 1137 00:44:36,440 --> 00:44:36,940 CONDITION. 1138 00:44:37,007 --> 00:44:42,479 AND ALSO WE THINK THEY SUPPO 1139 00:44:42,546 --> 00:44:44,948 SUPPORT -- PROMPT -- AND ALSO 1140 00:44:45,015 --> 00:44:45,816 EVALUATING DIFFERENT PROMPTING 1141 00:44:45,883 --> 00:44:46,683 STRATEGIES TO FIGURE OUT WHAT 1142 00:44:46,750 --> 00:44:48,919 KIND OF STRATEGIES CAN WE 1143 00:44:48,986 --> 00:44:51,522 PROVIDE TO EITHER PATIENTS OR 1144 00:44:51,588 --> 00:44:53,457 PROVIDERS FOR THEM TO GET BETTER 1145 00:44:53,524 --> 00:44:55,025 INSIGHTS INTO LAB RESULTS. 1146 00:44:55,092 --> 00:44:57,261 AND ALSO WE SHOULD ALSO BE 1147 00:44:57,327 --> 00:44:59,696 CONSIDERING AUGMENTING MODELS 1148 00:44:59,763 --> 00:45:00,731 WITH CREDIBLE INFORMATION 1149 00:45:00,798 --> 00:45:01,265 SOURCES. 1150 00:45:01,331 --> 00:45:03,467 SO WE CAN CONTROL THE QUALITY OF 1151 00:45:03,534 --> 00:45:06,804 RESPONSES. 1152 00:45:06,870 --> 00:45:07,838 SO HERE IS SOME PROPOSED 1153 00:45:07,905 --> 00:45:10,140 PIPELINE THAT WE ARE RIGHT NOW 1154 00:45:10,207 --> 00:45:12,342 EXPERIMENTING WITH, SO NOW WE'RE 1155 00:45:12,409 --> 00:45:16,180 LOOKING AT EXISTING CREDIBLE 1156 00:45:16,246 --> 00:45:18,148 SOURCES FROM TABULAR LAB RESULTS 1157 00:45:18,215 --> 00:45:20,017 INTERPRETATION FROM A VARIETY OF 1158 00:45:20,084 --> 00:45:22,953 SOURCES LIKE AMERICAN BOARD OF 1159 00:45:23,020 --> 00:45:23,520 INTERNAL MEDICINE, STANFORD 1160 00:45:23,587 --> 00:45:26,590 MEDICINE AND ALSO LOOKING AT 1161 00:45:26,657 --> 00:45:27,925 UNSTRUCTURED -- SOURCES LIKE 1162 00:45:27,991 --> 00:45:31,128 MAYO CLINIC, MEDLINEPLUS, AND 1163 00:45:31,195 --> 00:45:36,366 ALSO NCBI BOOKSHELF STATPEARLS, 1164 00:45:36,433 --> 00:45:40,404 AND ALSO WE -- STRUCTURE DATA, 1165 00:45:40,471 --> 00:45:42,306 ANSWER DATA INTO COMPUTABLE 1166 00:45:42,372 --> 00:45:45,776 FORMAT, AND THEN FED THEM INTO 1167 00:45:45,843 --> 00:45:50,881 THE EMBEDDING SPACE USING THINGS 1168 00:45:50,948 --> 00:45:52,382 LIKE FAST TEXT AND OTHER 1169 00:45:52,449 --> 00:45:53,917 METHODS, LOADING THEM INTO A 1170 00:45:53,984 --> 00:45:54,284 VECTOR DATABASE. 1171 00:45:54,351 --> 00:45:55,986 SO FOR THE QUESTION PROMPT SIDE, 1172 00:45:56,053 --> 00:45:58,522 WE WANT TO BE ABLE TO USE LARGE 1173 00:45:58,589 --> 00:46:01,625 MATH MODELS TO INTERPRET, TO 1174 00:46:01,692 --> 00:46:03,694 INTERACT WITH USERS, TO AUGMENT 1175 00:46:03,760 --> 00:46:10,300 THEIR USERS SO THAT THE QUESTION 1176 00:46:10,367 --> 00:46:14,404 PROMPT CAN BE MORE COMPLETE AND 1177 00:46:14,471 --> 00:46:16,440 INCLUDE RELEVANT INFORMATION, 1178 00:46:16,507 --> 00:46:21,612 AND THEN -- PARTICULAR QUESTION 1179 00:46:21,678 --> 00:46:23,647 FROM CREDIBLE SOURCES AND THEN 1180 00:46:23,714 --> 00:46:28,152 GENERATING THE RESPONSES, 1181 00:46:28,218 --> 00:46:33,190 RETRIERETRIEVAL AUGMENTED GENERN 1182 00:46:33,257 --> 00:46:34,925 TECHNIQUE AND ALSO TO PROVIDE 1183 00:46:34,992 --> 00:46:37,361 FEEDBACK ON RESPONSES FOR THE 1184 00:46:37,427 --> 00:46:38,195 GENERATIVE RESPONSES. 1185 00:46:38,262 --> 00:46:41,865 SO THIS IS STILL A WORK IN 1186 00:46:41,932 --> 00:46:43,934 PROGRESS, AND WE'RE VERY EXCITED 1187 00:46:44,001 --> 00:46:45,502 TO BE ABLE TO DO THIS TYPE OF 1188 00:46:45,569 --> 00:46:47,471 RESEARCH IN THIS AREA OF LARGELY 1189 00:46:47,538 --> 00:46:49,773 MODEL AND GENERATIVE A.I., AND 1190 00:46:49,840 --> 00:46:52,176 WE FOUND THIS REALLY EXCITING 1191 00:46:52,242 --> 00:46:53,310 AND A BOOMING FIELD AND WE HOPE 1192 00:46:53,377 --> 00:46:56,713 TO BE ABLE TO CONTRIBUTE 1193 00:46:56,780 --> 00:47:00,217 IMPORTANT EXPERIENCES AND ALSO 1194 00:47:00,284 --> 00:47:02,553 EXPLORATION TO HOPEFULLY HELP 1195 00:47:02,619 --> 00:47:05,088 PATIENTS HAVE AN EASIER ACCESS 1196 00:47:05,155 --> 00:47:06,557 TO AND BETTER UNDERSTANDING OF 1197 00:47:06,623 --> 00:47:09,059 LAB RESULTS AND IMPROVING THEIR 1198 00:47:09,126 --> 00:47:12,095 HEALTH OUTCOMES AND REDUCE 1199 00:47:12,162 --> 00:47:13,297 HEALTH DISPARITY AS WELL. 1200 00:47:13,363 --> 00:47:17,234 SO THIS IS ACKNOWLEDGMENT OF ALL 1201 00:47:17,301 --> 00:47:18,602 THE PEOPLE I'VE BEEN WORKING 1202 00:47:18,669 --> 00:47:19,770 WITH IN THE PAST FEW YEARS ON 1203 00:47:19,836 --> 00:47:25,042 THIS TOPIC, ESPECIALLY MY 1204 00:47:25,108 --> 00:47:30,080 POSTDOC SCHOLAR, DR. LIU'S 1205 00:47:30,147 --> 00:47:32,449 POSTDOC, ALSO COLLABORATING, 1206 00:47:32,516 --> 00:47:34,818 IT'S AN ENORMOUS HONOR TO BE 1207 00:47:34,885 --> 00:47:36,653 ABLE TO DO IT, I'M REALLY HAPPY 1208 00:47:36,720 --> 00:47:38,722 TO WORK WITH ALL OF THE PEOPLE 1209 00:47:38,789 --> 00:47:39,923 ON THIS TEAM. 1210 00:47:39,990 --> 00:47:42,492 AND I THINK THAT'S IT FOR MY 1211 00:47:42,559 --> 00:47:44,328 PRESENTATION FOR TODAY. 1212 00:47:44,394 --> 00:47:47,431 I THINK I HAVE A FEW MINUTES FOR 1213 00:47:47,497 --> 00:47:50,133 QUESTIONS. 1214 00:47:50,200 --> 00:47:53,670 THANK YOU. 1215 00:47:53,737 --> 00:47:55,772 >> DR. HE, THANK YOU SO MUCH. 1216 00:47:55,839 --> 00:47:58,375 THIS WAS VERY, VERY, VERY 1217 00:47:58,442 --> 00:47:59,843 INTERESTING PRESENTATION, VERY 1218 00:47:59,910 --> 00:48:00,911 EXCITING. 1219 00:48:00,978 --> 00:48:02,446 TRULY EXCITING TO LIVE IN THE 1220 00:48:02,512 --> 00:48:04,948 TIME WHERE WE CAN EXPLORE THE 1221 00:48:05,015 --> 00:48:08,151 APPLICATIONS OF LLM TO EVERY 1222 00:48:08,218 --> 00:48:08,785 POSSIBLE FIELD. 1223 00:48:08,852 --> 00:48:10,187 AND MY QUESTION WAS GOING TO BE 1224 00:48:10,254 --> 00:48:13,056 IF YOU HAVE TRIED RETRIEVAL 1225 00:48:13,123 --> 00:48:14,291 ELEMENT IN GENERATION AND YOU 1226 00:48:14,358 --> 00:48:15,926 ANSWERED IT, THIS IS A WORK IN 1227 00:48:15,993 --> 00:48:20,063 PROGRESS, SO IT'S EXCITING TO 1228 00:48:20,130 --> 00:48:22,032 SEE AT YOUR RESULTS AND IF YOU 1229 00:48:22,099 --> 00:48:27,237 CAN IMPROVE BECAUSE THIS SEEMENS 1230 00:48:27,304 --> 00:48:30,140 TO FOCUS YOUR LLM IN THE RIGHT 1231 00:48:30,207 --> 00:48:34,745 DIRECTION AND ENSURE THAT IT 1232 00:48:34,811 --> 00:48:35,646 PRODUCES -- RESULTS -- 1233 00:48:35,712 --> 00:48:38,682 >> WE'RE REALLY TRYING HARD TO 1234 00:48:38,749 --> 00:48:42,519 IMPLEMENT SOME OF THE RETRIEVAL 1235 00:48:42,586 --> 00:48:44,521 AUGMENTED GENERATION FRAMEWORK. 1236 00:48:44,588 --> 00:48:48,058 YOU KNOW, -- OVER THERE RIGHT 1237 00:48:48,125 --> 00:48:51,094 NOW, SO WE'RE TRYING TO FIND 1238 00:48:51,161 --> 00:48:53,063 ALSO CREDIBLE SOURCES FROM THE 1239 00:48:53,130 --> 00:48:54,865 TABULAR SOURCES AND -- AND WE'RE 1240 00:48:54,931 --> 00:48:56,600 ALSO VERY FORTUNATE TO BE ABLE 1241 00:48:56,667 --> 00:48:58,902 TO CONNECT WITH PEOPLE FROM 1242 00:48:58,969 --> 00:49:01,204 MEDLINEPLUS, WHICH ARE ACTUALLY 1243 00:49:01,271 --> 00:49:03,940 WORKING AT NLM, TO GIVE US SOME 1244 00:49:04,007 --> 00:49:06,810 TECHNOLOGY, LIKE API, IT DID NOT 1245 00:49:06,877 --> 00:49:08,912 EVEN ALLOW US TO QUERY 1246 00:49:08,979 --> 00:49:10,280 MEDLINEPLUS CONTENT USING 1247 00:49:10,347 --> 00:49:13,050 STANDARDIZED MEDICAL TERMINOLOGY 1248 00:49:13,116 --> 00:49:14,985 LIKE LOINK WHICH REALLY GIVES US 1249 00:49:15,052 --> 00:49:17,220 A LOT OF GREAT ACCESS TO THOSE 1250 00:49:17,287 --> 00:49:18,055 RESOURCES FOR THIS TYPE OF WORK. 1251 00:49:18,121 --> 00:49:20,123 >> YEAH, YEAH, THAT'S THRIE 1252 00:49:20,190 --> 00:49:21,658 AMAZING RESOURCE. 1253 00:49:21,725 --> 00:49:24,161 AND FOR OUR AUDIENCE, PLEASE 1254 00:49:24,227 --> 00:49:25,495 DON'T HESITATE TO SEND ME YOUR 1255 00:49:25,562 --> 00:49:28,198 QUESTIONS TO MY EMAIL. 1256 00:49:28,265 --> 00:49:35,906 IT'S LANA.YE IMGANOVA@NIH.GOV. 1257 00:49:35,972 --> 00:49:38,975 THANK YOU FOR DISPLAYING. 1258 00:49:39,042 --> 00:49:40,243 SEND IN YOUR QUESTIONS AND WHILE 1259 00:49:40,310 --> 00:49:42,312 I'M WAITING FOR YOUR QUESTIONS, 1260 00:49:42,379 --> 00:49:47,884 I'M GOING TO ASK MY QUESTIONS. 1261 00:49:47,951 --> 00:49:49,486 SO ON ALL OF THE MODELS THAT YOU 1262 00:49:49,553 --> 00:49:51,355 USED, ALL OF THE LLM MODELS THAT 1263 00:49:51,421 --> 00:49:53,023 YOU USED HERE, NONE OF THEM ARE 1264 00:49:53,090 --> 00:49:54,624 OPEN SOURCE, CORRECT? 1265 00:49:54,691 --> 00:49:58,495 >> ACTUALLY WE USED LLAMA 2 IS 1266 00:49:58,562 --> 00:50:00,630 OPEN SOURCE, I BELIEVE. 1267 00:50:00,697 --> 00:50:02,199 THE WAY WE USE IT IS THAT A LOT 1268 00:50:02,265 --> 00:50:05,001 OF THESE MODELS ARE KIND OF -- 1269 00:50:05,068 --> 00:50:06,436 INTO A LOCAL MACHINE, SO WE USE 1270 00:50:06,503 --> 00:50:11,074 A FRAMEWORK CALLED A LANCHE THAT 1271 00:50:11,141 --> 00:50:13,510 ALLOWS US TO -- USING LOCAL 1272 00:50:13,577 --> 00:50:15,746 MODELS BECAUSE ONE CHALLENGE OF 1273 00:50:15,812 --> 00:50:19,249 USING GPT MODEL IS BECAUSE IT'S 1274 00:50:19,316 --> 00:50:20,050 A COMMERCIALIZED MODEL AND IT'S 1275 00:50:20,117 --> 00:50:21,551 NOT OPEN SOURCE, AND WE CAN ONLY 1276 00:50:21,618 --> 00:50:22,819 ACCESS IT EITHER THROUGH PATIENT 1277 00:50:22,886 --> 00:50:25,822 WEB PORTAL OR USING API, SO IN 1278 00:50:25,889 --> 00:50:27,991 EITHER CASE, THE RESPONSES SORT 1279 00:50:28,058 --> 00:50:31,461 OF DEPENDS ON THE TYPE OF 1280 00:50:31,528 --> 00:50:33,864 SERVICES BEHIND THE SCENE AND 1281 00:50:33,930 --> 00:50:35,232 THEY KEEP KIND OF DEVELOPING 1282 00:50:35,298 --> 00:50:36,433 FURTHER, FURTHER, SOMETIMES IT 1283 00:50:36,500 --> 00:50:40,604 HARD TO CONTROL AND ALSO HARD TO 1284 00:50:40,670 --> 00:50:41,705 HOW TO MAKE SURE THAT THE 1285 00:50:41,772 --> 00:50:43,306 RESULTS ARE REPRODUCIBLE. 1286 00:50:43,373 --> 00:50:45,142 SO THAT'S WHY WE'RE RIGHT NOW 1287 00:50:45,208 --> 00:50:46,977 FOCUSING MORE ENERGY INTO LIKE 1288 00:50:47,043 --> 00:50:48,111 LLAMA 2 MODEL, WHICH IS OPEN 1289 00:50:48,178 --> 00:50:49,212 SOURCE MODEL FOR THIS TYPE OF 1290 00:50:49,279 --> 00:50:50,447 WORK AS WE CAN DO A LOT OF 1291 00:50:50,514 --> 00:50:50,847 THINGS WITH THAT. 1292 00:50:50,914 --> 00:50:52,315 >> OKAY. 1293 00:50:52,382 --> 00:50:53,083 THAT'S GOOD TO KNOW. 1294 00:50:53,150 --> 00:50:54,384 THERE'S A LOT OF PUSH IN THE 1295 00:50:54,451 --> 00:50:57,053 COMMUNITY TO REALLY -- IT SEEMS 1296 00:50:57,120 --> 00:50:59,456 LIKE GPT-4 IS LEADING AND IT WAS 1297 00:50:59,523 --> 00:51:00,991 CLEAR FROM YOUR RESULTS. 1298 00:51:01,057 --> 00:51:03,160 JUST WANT TO REITERATE THAT IT'S 1299 00:51:03,226 --> 00:51:04,861 PRETTY AMAZING SO BASICALLY ALL 1300 00:51:04,928 --> 00:51:07,130 OF THESE LLMs PROVIDED BETTER 1301 00:51:07,197 --> 00:51:10,100 RESULTS THAN RESULTS PROVIDED BY 1302 00:51:10,167 --> 00:51:11,168 PEERS. 1303 00:51:11,234 --> 00:51:12,202 OBVIOUSLY, YOU KNOW, WE WANT TO 1304 00:51:12,269 --> 00:51:15,906 REFER TO DOCTORS FOR THE MOST 1305 00:51:15,972 --> 00:51:22,045 CORRECT INFORMATION, BUT LLM -- 1306 00:51:22,112 --> 00:51:23,680 THEY PROVIDE REASONABLE 1307 00:51:23,747 --> 00:51:24,815 EXPLANATIONS AND 1308 00:51:24,881 --> 00:51:25,182 INTERPRETATIONS. 1309 00:51:25,248 --> 00:51:26,783 AND THERE IS A LOT OF PUSH IN 1310 00:51:26,850 --> 00:51:28,718 THE COMMUNITY NOW TO KIND OF 1311 00:51:28,785 --> 00:51:30,587 DEVELOP THOSE OPEN SOURCE MODELS 1312 00:51:30,654 --> 00:51:33,089 THAT ARE SIMILAR IN PERFORMANCE 1313 00:51:33,156 --> 00:51:36,460 TO A GPT -- AND IT LOOKS LIKE 1314 00:51:36,526 --> 00:51:37,961 WHILE WE'RE NOT THERE YET BUT WE 1315 00:51:38,028 --> 00:51:39,596 ARE ON THE WAY TO GET THERE. 1316 00:51:39,663 --> 00:51:46,503 >> MM-HMM. 1317 00:51:46,570 --> 00:51:47,737 SO THIS WAS INTERESTING YOU ARE 1318 00:51:47,804 --> 00:51:49,506 MENTIONING HOW TO INTERPRET LAB 1319 00:51:49,573 --> 00:51:51,141 RESULTS, BUT YOU ARE EMPLOYED AT 1320 00:51:51,208 --> 00:51:54,244 THE -- YOU ARE AFFILIATED IN 1321 00:51:54,311 --> 00:51:56,713 THE -- FOR SUCCESSFUL LONGEVITY. 1322 00:51:56,780 --> 00:51:57,747 ARE THERE ANY PROGRAMS IN YOUR 1323 00:51:57,814 --> 00:51:59,115 INSTITUTE THAT ARE GEARED 1324 00:51:59,182 --> 00:52:02,118 TOWARDS PREVENTION OR 1325 00:52:02,185 --> 00:52:03,420 SUSTAINABILITY OF A HEALTHY 1326 00:52:03,487 --> 00:52:07,424 LIFESTYLE RATHER THAN TREATMENT? 1327 00:52:07,491 --> 00:52:09,559 >> IN OUR INSTITUTION, WE'RE 1328 00:52:09,626 --> 00:52:11,595 ACTUALLY VERY FOCUSED ON HEALTHY 1329 00:52:11,661 --> 00:52:12,762 BEHAVIOR PROMOTION AND OUR 1330 00:52:12,829 --> 00:52:17,167 INSTITUTE -- THE GOAL IS TO 1331 00:52:17,234 --> 00:52:21,938 SUPPORT SUCCESSFUL LONGEVITY 1332 00:52:22,005 --> 00:52:25,942 AMONG EALDER ADU OLDER ADULTS, Y 1333 00:52:26,009 --> 00:52:33,083 FROM -- PERSPECTIVE, MOSTLY -- 1334 00:52:33,149 --> 00:52:35,252 ACCEPTANCE TOWARDS USING CHATGPT 1335 00:52:35,318 --> 00:52:38,722 OR GENERATIVE A.I. MODELS FOR -- 1336 00:52:38,788 --> 00:52:39,990 PURPOSES, SO ALL OF THESE THINGS 1337 00:52:40,056 --> 00:52:41,658 ARE GOING TO BE REALLY 1338 00:52:41,725 --> 00:52:43,093 INTERESTING TO LOOK AT, BECAUSE 1339 00:52:43,159 --> 00:52:45,028 WE DON'T KNOW, YOU KNOW, WE 1340 00:52:45,095 --> 00:52:47,931 OFTENTIMES LOOK AT THIS FROM A 1341 00:52:47,998 --> 00:52:48,932 SCIENTIST PERSPECTIVE, HOW IT'S 1342 00:52:48,999 --> 00:52:50,734 GOING TO BE USEFUL, BUT WE DON'T 1343 00:52:50,800 --> 00:52:53,570 KNOW HOW INDIVIDUAL USERS OR 1344 00:52:53,637 --> 00:52:55,572 PATIENTS OR PEOPLE WHO ACTUALLY 1345 00:52:55,639 --> 00:52:58,842 NEED THESE SERVICES ARE LOOKING 1346 00:52:58,909 --> 00:53:00,610 AT THESE RESPONSES, WHETHER 1347 00:53:00,677 --> 00:53:01,912 THEY'RE TRUSTWORTHY OR WHAT KIND 1348 00:53:01,978 --> 00:53:03,847 OF CONCERN DO THEY HAVE, TO THIS 1349 00:53:03,914 --> 00:53:05,015 IS DEFINITELY SOMETHING THAT 1350 00:53:05,081 --> 00:53:08,351 WE'RE TRYING TO LOOK AT, AND 1351 00:53:08,418 --> 00:53:11,054 ANOTHER THING IS THAT WE'RE ALSO 1352 00:53:11,121 --> 00:53:13,823 GENERALLY INTERESTED IN KIND OF 1353 00:53:13,890 --> 00:53:17,093 COLLABORATING WITH ANY -- 1354 00:53:17,160 --> 00:53:18,461 INTERESTED IN -- THERE'S A LOT 1355 00:53:18,528 --> 00:53:22,232 OF PEOPLE HERE WHO ARE 1356 00:53:22,299 --> 00:53:24,200 SPECIALIZED IN COMMUNITY HEALTH, 1357 00:53:24,267 --> 00:53:27,270 LOOKING AT PREVENTION AND LATE 1358 00:53:27,337 --> 00:53:28,672 ONSET OF ALZHEIMER'S DISEASE, 1359 00:53:28,738 --> 00:53:30,974 DIFFERENT TYPE OF MODALITIES 1360 00:53:31,041 --> 00:53:33,577 WHERE ALSO DOING A RESEARCH 1361 00:53:33,643 --> 00:53:36,713 PROJECT WITH CUTTING TRAINING 1362 00:53:36,780 --> 00:53:41,484 PROGRAM O FOR POSSIBLE KIND OF 1363 00:53:41,551 --> 00:53:43,353 EARLY DETECTION OF ALZHEIMER'S 1364 00:53:43,420 --> 00:53:46,056 AND DEMENTIA, AND SO IN THAT 1365 00:53:46,122 --> 00:53:47,023 PARTICULAR PROJECT, I THINK 1366 00:53:47,090 --> 00:53:49,693 THERE'S A LOT OF INTERESTING 1367 00:53:49,759 --> 00:53:52,929 FINDINGS AS WELL WE'RE 1368 00:53:52,996 --> 00:53:53,229 PUBLISHING. 1369 00:53:53,296 --> 00:53:54,631 >> THAT SOUNDS VERY BEING 1370 00:53:54,698 --> 00:53:55,231 SITING. 1371 00:53:55,298 --> 00:53:59,135 DO YOU ALSO AGREE THAT THERE IS 1372 00:53:59,202 --> 00:54:02,038 A -- SOCIAL ASPECT IN KIND OF 1373 00:54:02,105 --> 00:54:04,240 HEALTHY AGING AND LONGEVITY? 1374 00:54:04,307 --> 00:54:05,976 LIKE PEOPLE -- WHEN WE AGE, WE 1375 00:54:06,042 --> 00:54:08,478 KIND OF DEPEND ON THOSE SOCIAL 1376 00:54:08,545 --> 00:54:11,681 CIRCLES FOR SOCIAL INTERACTIONS, 1377 00:54:11,748 --> 00:54:13,650 BUT ALSO FOR PHYSICAL EXERCISE 1378 00:54:13,717 --> 00:54:14,985 AND STUFF LIKE THAT, SO IS THERE 1379 00:54:15,051 --> 00:54:18,188 ANYTHING IN YOUR INSTITUTE THAT 1380 00:54:18,254 --> 00:54:20,023 IS KIND OF PROMOTING SOCIAL 1381 00:54:20,090 --> 00:54:21,024 INTERACTIONS BETWEEN PEOPLE? 1382 00:54:21,091 --> 00:54:23,560 >> WE'RE REALLY PROMOTING SOCIAL 1383 00:54:23,627 --> 00:54:23,994 INTERACTIONS. 1384 00:54:24,060 --> 00:54:25,295 DURING COVID ESPECIALLY, WHEN 1385 00:54:25,362 --> 00:54:28,632 PEOPLE ARE ISOLATING FROM 1386 00:54:28,698 --> 00:54:31,234 WHATEVER THEY DO, WE ACTUALLY 1387 00:54:31,301 --> 00:54:33,603 DID A LOT OF LIKE TRAINING 1388 00:54:33,670 --> 00:54:35,605 SESSIONS TO ALLOW PEOPLE TO 1389 00:54:35,672 --> 00:54:37,307 LEARN HOW TO USE ZOOM, 1390 00:54:37,374 --> 00:54:39,576 ESPECIALLY FOR OLDER PEOPLE WHO 1391 00:54:39,643 --> 00:54:41,811 ARE NOT USED TO A LOT OF 1392 00:54:41,878 --> 00:54:44,047 TECHNOLOGIES NEED A LOT OF 1393 00:54:44,114 --> 00:54:45,715 TRAINING. 1394 00:54:45,782 --> 00:54:48,284 SO WE DEFINITELY ARE DOING A LOT 1395 00:54:48,351 --> 00:54:50,553 TO PROMOTE THAT, AT BOTH THE 1396 00:54:50,620 --> 00:54:52,055 SOCIAL CONNECTEDNESS ASPECT OF 1397 00:54:52,122 --> 00:54:52,922 THIS. 1398 00:54:52,989 --> 00:54:56,226 WE ALSO ARE CURRENTLY 1399 00:54:56,292 --> 00:54:57,594 COLLABORATING WITH -- THERE ARE 1400 00:54:57,661 --> 00:54:58,328 ACTUALLY TWO OTHER INSTITUTES 1401 00:54:58,395 --> 00:55:02,465 HERE AT FSU, WE'RE DOING -- FOR 1402 00:55:02,532 --> 00:55:04,234 RESEARCH AND WE'RE RIGHT NOW 1403 00:55:04,300 --> 00:55:06,903 ORGANIZING A SERIES OF EVENTS 1404 00:55:06,970 --> 00:55:08,171 LIKE MUSIC EVENTS TO ALLOW 1405 00:55:08,238 --> 00:55:10,173 PEOPLE TO GET TOGETHER AND TO 1406 00:55:10,240 --> 00:55:12,909 INTERACT WITH EACH OTHER AND TO 1407 00:55:12,976 --> 00:55:13,777 SOCIALIZE. 1408 00:55:13,843 --> 00:55:15,178 SO WE HOPE TO BE ABLE TO KIND OF 1409 00:55:15,245 --> 00:55:17,047 DO A LOT OF THINGS WITH 1410 00:55:17,113 --> 00:55:20,450 COMMUNITIES HERE IN THE AREA. 1411 00:55:20,517 --> 00:55:21,484 FOR HEALTHY AGING. 1412 00:55:21,551 --> 00:55:22,452 >> THAT'S INCREDIBLE. 1413 00:55:22,519 --> 00:55:23,853 SO I HAVE A QUESTION AND THEN A 1414 00:55:23,920 --> 00:55:24,954 QUESTION FROM THE AUDIENCE. 1415 00:55:25,021 --> 00:55:27,757 SO ANY OF THOSE MODELS ARE 1416 00:55:27,824 --> 00:55:29,492 IMPLEMENTED, ANY OF THOSE 1417 00:55:29,559 --> 00:55:31,294 AGENTS, LLM AGENTS, ARE THEY 1418 00:55:31,361 --> 00:55:32,862 IMPLEMENTED IN YOUR ELECTRONIC 1419 00:55:32,929 --> 00:55:34,564 HEALTH RECORDS OR INTERPRETATION 1420 00:55:34,631 --> 00:55:37,467 OF LAB RESULTS OR EVERYTHING IS 1421 00:55:37,534 --> 00:55:38,835 STILL IN THE RESEARCH STAGE? 1422 00:55:38,902 --> 00:55:40,970 >> RIGHT NOW EVERYTHING IS STILL 1423 00:55:41,037 --> 00:55:44,140 IN THE RESEARCH STAGE, BUT WE 1424 00:55:44,207 --> 00:55:45,642 CURRENTLY ARE WORKING ON A 1425 00:55:45,709 --> 00:55:50,180 PROJECT, LAB GENIE, AND THIS 1426 00:55:50,246 --> 00:55:51,514 PROJECT, IF WE GET FUNDED FROM 1427 00:55:51,581 --> 00:55:52,916 OUR FUNDING AGENCY, WE'LL BE 1428 00:55:52,982 --> 00:55:55,518 ABLE TO WORK DIRECTLY WITH OUR 1429 00:55:55,585 --> 00:55:56,619 HEALTHCARE PARTNER TO DEVELOP 1430 00:55:56,686 --> 00:55:58,888 THIS TOOL THAT CAN INTERACT WITH 1431 00:55:58,955 --> 00:56:04,627 THE EHR SYSTEMS USING FAIR API 1432 00:56:04,694 --> 00:56:06,963 SO PATIENTS WILL BE ABLE TO GET 1433 00:56:07,030 --> 00:56:08,965 THEIR -- DIRECTLY WITH THIS TOOL 1434 00:56:09,032 --> 00:56:12,969 AND INCLUDING -- TO PRODUCE 1435 00:56:13,036 --> 00:56:14,771 IMPORTANT LIKE INSIGHTS INTO 1436 00:56:14,838 --> 00:56:15,739 THIS DATA AND BACK TO PATIENTS. 1437 00:56:15,805 --> 00:56:18,908 BUT RIGHT NOW, IT'S STILL MORE 1438 00:56:18,975 --> 00:56:20,610 IN AN EXPLORATORY PHASE BUT 1439 00:56:20,677 --> 00:56:21,478 WE'RE LOOKING FORWARD TO 1440 00:56:21,544 --> 00:56:22,445 DEVELOPING THE SYSTEM WITH THIS 1441 00:56:22,512 --> 00:56:25,215 TYPE OF SUPPORT. 1442 00:56:25,281 --> 00:56:27,016 >> THAT SEEMS TO BE WHERE THE 1443 00:56:27,083 --> 00:56:31,921 COMMUNITY IS GOING, AND TO IP 1444 00:56:31,988 --> 00:56:39,529 PLEIMPLEMENT THOSE MODELS AND 1445 00:56:39,596 --> 00:56:40,764 IMPLEMENT THOSE SMART AGENTS WHO 1446 00:56:40,830 --> 00:56:41,731 WOULD BE ABLE TO COMMUNICATE 1447 00:56:41,798 --> 00:56:42,499 WITH THE USERS. 1448 00:56:42,565 --> 00:56:43,299 >> ABSOLUTELY. 1449 00:56:43,366 --> 00:56:44,134 >> WELL, I WANT TO THANK YOU 1450 00:56:44,200 --> 00:56:44,400 AGAIN. 1451 00:56:44,467 --> 00:56:45,935 THIS WAS A VERY, VERY 1452 00:56:46,002 --> 00:56:47,337 INTERESTING CONVERSATION, AND 1453 00:56:47,403 --> 00:56:52,175 PLEASE LET US KNOW HOW YOUR 1454 00:56:52,242 --> 00:56:54,844 RETRIEVAL AUGMENTED GENERATION, 1455 00:56:54,911 --> 00:56:55,912 WE'RE VERY INTERESTED TO SEE 1456 00:56:55,979 --> 00:56:57,580 WHAT RESULTS DO YOU GET THERE, 1457 00:56:57,647 --> 00:56:59,115 AND WE WILL BE FOLLOWING YOUR 1458 00:56:59,182 --> 00:57:00,683 RESEARCH AND THANK YOU AGAIN FOR 1459 00:57:00,750 --> 00:57:02,452 YOUR TIME AND VERY INTERESTING 1460 00:57:02,519 --> 00:57:04,721 TALK, AND FOR JOINING US TODAY. 1461 00:57:04,788 --> 00:57:05,889 >> THANKS SO MUCH FOR THE 1462 00:57:05,955 --> 00:57:06,289 OPPORTUNITY. 1463 00:57:06,356 --> 00:57:08,358 AND THANK YOU SO MUCH, LANA, FOR 1464 00:57:08,424 --> 00:57:10,493 ALL THE HARD WORK IN ORGANIZING 1465 00:57:10,560 --> 00:57:11,127 THIS AND MAKING THIS HAPPEN. 1466 00:57:11,194 --> 00:57:12,896 I REALLY APPRECIATE IT. 1467 00:57:12,962 --> 00:57:15,131 AND ALSO THANK DR. LIU FOR 1468 00:57:15,198 --> 00:57:20,403 REALLY MAKING THIS HAPPEN, KIND 1469 00:57:20,470 --> 00:57:22,438 OF LEADING THIS EFFORT AND 1470 00:57:22,505 --> 00:57:23,873 WORKING COLLABORATIVELY ON THIS 1471 00:57:23,940 --> 00:57:24,107 PROJECT. 1472 00:57:24,174 --> 00:57:25,175 THANK YOU SO MUCH. 1473 00:57:25,241 --> 00:57:25,575 >> ABSOLUTELY. 1474 00:57:25,642 --> 00:57:26,743 IT WAS GOOD TO HAVE YOU. 1475 00:57:26,810 --> 00:57:28,011 >> THANK YOU. 1476 00:57:28,077 BYE-BYE.