>> WELCOME BACK EVERYONE TO THE
OPEN PORTION OF THE LSTRC
MEETING. WE NEXT HAVE A
DIRECTOR'S UPDATE BY DR. MICHAEL
HUERTA, ACTING DEPUTY DIRECTOR
FOR OPERATIONS AND INNOVATION.
NATIONAL LIBRARY OF MEDICINE. WE
WILL TURN THE TIME OVER TO DR.
HUERTA.
>> GOOD AFTERNOON, EVERYBODY, I
WILL BE SHARING MY SCREEN. THERE
WE GO. OOPS. THERE WE GO. MOVE
SOME OF THESE WINDOWS OFF OF MY
SLIDES. SO I CAN SEE THE SCREEN
ALL RIGHT. THANKS AGAIN, SORRY
FOR THOSE GLITCHES GETTING
STARTED. PATTY IS NOT ABLE TO
MAKE IT TODAY SHE IS AT OUR
ANNUAL TRAINING CONFERENCE. WE
HAVE A NUMBER OF EXTRAMURAL
INSTITUTIONAL TRAINING GRANTS
THAT WE SUPPORT AROUND THE
COUNTRY. AND EVERY YEAR THEY ALL
GET TOGETHER, THIS YEAR THEY ARE
AT STANFORD UNIVERSITY. SO SHE
SENDS REGRETS BUT I WILL BRING
YOU UP TO DATE ON THE LATEST AND
GREATEST AT NLM SINCE OUR LAST
MEETING WHICH I BELIEVE WAS IN
OCTOBER. THE BIGGEST NEWS YOU
PROBABLY HEARD ALREADY IF ONLY
BY MY SHOWING YOU THE SLIDE
ALREADY, JERRY SHEEHAN DEPUTY
DIRECTOR FOR EXTERNAL POLICY AND
AFFAIRS HAS LEFT THE NLM GOING
TO THE ORGANIZATION FOR ECONOMIC
COOPERATIVE DEVELOPMENT IN PARIS
TO LEAD THE SCIENCE TECHNOLOGY
AND INNOVATION DIRECTORATE.
JERRY HAD WORKED AT OECD IN
PARIS A WHILE BACK, A LONG TIME
BEFORE HE GOT TO NLM BUT NOW
HE'S GOING BACK AS THE BIG BOSS.
SO WE WISH HIM WELL. HE
CERTAINLY HAS CONTRIBUTED AN
AWFUL LOT TO THE NLM 16 YEARS HE
SPENT WITH US. HE MADE THE
PLACE BETTER EVERY DAY WE ALL
WOULD AGREE. REGARDING
LEADERSHIP, THIS LAYS OUT
MEMBERS OF LEADERSHIP AT NLM.
YOU CAN SEE JERRY'S POSITION IS
VACANT. BUT I WANT TO BRING YOUR
ATTENTION TO THE FACT THAT MORE
THAN THAT WE WILL BE CHANGING IN
LEADERSHIP IN THE RELATIVELY
NEAR FUTURE. BECAUSE NOT ONLY IS
JERRY GONE BUT SIX OF THE OTHER
11 MEMBER LEADERSHIP POSITIONS
ARE IN ACTING CAPACITIES
INCLUDING MYSELF OF COURSE. WE
ARE ACTIVELY WORKING ON FILLING
THESE AND I WILL SAY I THINK WE
ARE GETTING CLOSE TO BRINGING ON
A SCIENTIFIC DIRECTOR. EVERYBODY
KEEP FINGERS CROSSED FOR US IF
THAT HAPPENS. THIS IS ONE END OF
THE PIPELINE, THE NLM
LEADERSHIP. LET'S LOOK AT THE
OTHER END OF THE PIPELINE IF YOU
WILL. OUR TRAINEES. THIS SUMMER
WE WELCOME SOME 55 INTERIMS IN
VARIOUS PROGRAMS. ONE OF THOSE
PROGRAMS IS THE NLM DATA
SCHOLARS -- DATA SCIENCE AND
INFORMATICS SCHOLARS PROGRAM. WE
HAVE GOT NINE THERE. TOGETHER
THESE VARIOUS INTERNSHIPS
INCLUDE EVERYBODY FROM HIGH
SCHOOL STUDENTS ALL THE WAY UP
TO PROFESSIONAL, M.D.s AND
DENTISTS AND SO FORTH. ALONG
WITH OTHER PARTS OF NIH, WE HAVE
RAISED OUR TRAINEE SALARIES AT
NLM, PROBABLY HAVE SEEN THAT IN
THE NEWS. AT THE BOTTOM YOU SEE
SOME OF OUR TRAINEES AT A
RESEARCH FESTIVAL AT NIH,
PRECEPTORS AND POSTS, ANOTHER
COMING UP AUGUST 3 SUMMER
INTERNS WILL BE PARTICIPATING. I
THIS THEY ARE PLANNING NOW TO
HAVE NINE PRESENTATIONS IN THAT
RESEARCH FESTIVAL. SINCE OUR
LAST MEETING WE HAVE HAD SOME
GOOD DEVELOPMENTS AND CAPACITY
DEVELOPMENT, AS I MENTIONED WE
HAVE TRAINING CONFERENCE GOING
ON RIGHT NOW AT THE TOP. AND
THAT IS ALWAYS BEEN A VERY
IMPORTANT PART OF NLM, OUR
EXTRAMURAL SUPPORT OF TRAINEES
PRE-DOCTORAL AND POST-DOCTORAL
LEVELS. THERE ARE SOME 18 SITES
AROUND THE COUNTRY WITH A TOTAL
178 TRAINING SLOTS, 119 ARE
PRE-DOC AND 59 POST DOCS AS OF
TODAY. I SHOULD SAY WE HAVE HAD
THIS EXTRAMURAL TRAINING PROGRAM
IN PLACE FOR DECADES AND LONG
BEFORE MANY OTHER PEOPLE AT NIH
KNEW WHAT INFORMATICS WAS, NLM
WAS TRAINING. SO MANY OF THE
LEADERS IN BIOMEDICAL
INFORMATICS, DEPARTMENT CHAIRS,
PROFESSORS, THE PROMINENT PEOPLE
IN INFORMATICS DATA SCIENCE AS
RELATES TO BIOMEDICINE HAVE BEEN
TRAINED BY NLM. DIANNE BABSKI,
ANOTHER ITEM COMPLETED THE AI
FEDERAL LEADERSHIP PROGRAM
THROUGH THE PARTNERSHIP FOR
PUBLIC SERVICE AND HAS GIVEN A
TALK ON GENERATIVE AI, ITS
CHALLENGES, PROMISES, HOW NLM IS
USING IT, WHAT THE BIOMEDICAL
COMMUNITY MIGHT BE USING IT FOR,
SO FORTH. SHE'S BEEN GIVING
TALKS LIKE THIS ON PROBABLY
EVERY OTHER DAY BASIS, ACROSS
NIH AND ACROSS FEDERAL
GOVERNMENT. SHE JUST EARLIER
THIS WEEK GAVE A TALK TO THE NIH
INSTITUTE AND CENTER DEPUTY
DIRECTORS, SHE DID A GREAT JOB.
LOTS OF INTEREST, LOTS OF
ATTENTION BEING PAID AT NLM. TWO
PAPERS CAME OUT REGARDING AI,
IMAGE INTERPRETATION, ONE HAS TO
DO WITH A FIELD STUDY THAT
SHOWED THE ALGORITHMS DEVELOPED
IN THE NLM INTRAMURAL RESEARCH
PROGRAM ARE ACTUALLY ABLE TO
PROVIDE EXPERT LEVEL GUIDANCE
WHEN DIAGNOSED WITH DELIRIUM,
THIS USES IMAGING DATA THROUGH
iPHONE ATTACHED TO A
MICROSCOPE. THAT WORK WAS LED BY
STEPHAN YAGER AND A PAPER THAT
ESTABLISHED PRINCIPLE THAT WAS
DEVELOPED BY KIM TO DETECT
VASCULAR LEAKAGE IN THE OKAY
LAS.--OCULAS. SINCE THE LAST TIE
MET IN FEBRUARY THE EXTRAMURAL
RESEARCH PROGRAMS HOSTED A
WORKSHOP CALLED MEETING THE
NEEDS OF DIVERSION POPULATION
THROUGH TAILORED PERSONAL HEALTH
INFORMATICS. PERSONAL HEALTH
INFORMATICS IS AN IMPORTANT PART
OF STRATEGIC PLAN THAT WAS 17
AND THIS EXTRAMURAL INVESTMENT
IN PERSONAL HEALTH INFORMATICS
STARTED BACK AROUND THAT TIME
WHEN THE STRATEGIC PLAN WAS
ISSUE. THAT PLAN, I'M SORRY,
THAT PROGRAM WENT ON FOR NUMBER
OF YEARS. AND WE ENDED UP
FUNDING 12 APPLICATIONS. THESE
APPLICATIONS ARE AIMED FOR USE
CASES LIKE POPULATIONS OF PEOPLE
PREVIOUSLY INCARCERATED. T
PEOPLE WITH SPECIFIC CHRONIC
DISEASE LIKE MULTIPLE SCLEROSIS,
ENDOMETRIOSIS OR EPILEPSY. YOUTH
THAT HAVE AGED OUT OF FOSTER
CARE, INDIVIDUALS WITH OTHER
MULTIPLE CHRONIC CONDITIONS, AND
GROUPS NEGATIVELY AFFECTED BY
HEALTH DISPARITIES INCLUDING
AFRICAN AMERICAN AND HISPANICS.
THE KINDS OF PRODUCTS DEVELOPED
FROM THESE GRANTS INCLUDED
LIBRARIES AND ACCESS TO
INFORMATION THROUGH MOBILE APPS
AND PORTALS. NLP SYSTEMS THAT
ALLOW MEDICAL JARGON IN
ELECTRONIC HEALTH RECORDS TO BE
TRANSLATED INTO ACTUAL
DEFINITIONS SO THAT PEOPLE
LOOKING AT THEIR PATIENT RECORDS
WILL UNDERSTAND WHAT THOSE
RECORDS ARE SAYING. A NUMBER OF
OTHER, I WON'T READ THEM ALL
HERE BUT THIS HAS BEEN A VERY
SUCCESSFUL PROGRAM. AND AS
MENTIONED IN FEBRUARY BROUGHT
TOGETHER PRINCIPLE INVESTIGATORS
FUNDED UNDER THIS PROGRAM AS
WELL AS OTHER EXPERT MS. THE
FIELD AND MEMBERS FROM THE
COMMUNITY, TO TALK ABOUT HOW TO
DESIGN PERSONAL HEALTH
INFORMATICS IN WAYS THAT MEET
THE NEEDS OF VERY SPECIFIC
POPULATIONS. THEY LOOKED AT THE
CURRENT STATE OF THE SCIENCE,
RESEARCH OPPORTUNITIES, AND
RESEARCH CHALLENGES. AND THEY
CAME TOGETHER AND CAME UP WITH A
NUMBER OF RECOMMENDATIONS AND
BEST PRACTICES. THOSE ARE SHOWN
HERE. ONE TO EMPLOY MULTIPLE
METHODS ALONG THE DESIGN
SPECTRUM FROM USABILITY TESTING
TO CO-DESIGN. ANOTHER IS TO
THINK ABOUT SUSTAINABILITY AND
SCALABILITY FROM THE VERY
OUTSET. NEXT IS TO CONSIDER
PARTNERSHIPS WITH COMMUNITY
BASED ORGANIZATIONS AND PRIVATE
SECTOR COLLABORATIONS IN
ADDITION TO YOUR ACADEMIC
COLLABORATORS. FINALLY, THE
USEFULNESS OF HAVING A MODULAR
APPROACH WHERE COMPONENTS ARE
REUSED REASSEMBLED WAS FOUND TO
BE ONE OF THE BEST PRACTICES
THIS GROUP IDENTIFIED. WE HAVE
ALSO IMPROVED LOTS IN OUR WORLD
OF PROVIDING SERVICES TO THE
WORLD. I WON'T GO THROUGH ALL
THE THINGS BUT A FEW THINGS. THE
SEQUENCE READ ARCHIVE HAS BEEN
MADE MORE ROBUST IN A NUMBER OF
WAYS WE HAVE ISSUED A DISCLAIMER
ON PUBMED CENTRAL. TO MAKE CLEAR
TO PEOPLE THAT THE PAPERS THAT
THEY ARE READING FROM PUBMED
CENTRAL ARE NOT THE SCIENCE OF
NLM BUT NHL WILL SIMPLY SERVE
THAT SCIENCE UP. IT IS NOT
UNCOMMON TO READ AN ARTICLE IN
THE NEWSPAPER THAT SAYS
ACCORDING TO NLM X Y AND Z AND
THE X Y AND Z IS A PAPER IN
PUBMED CENTRAL SO THAT IS AN
IMPORTANT DISCLAIMER THAT WE PUT
UP. AS YOU PROBABLY KNOW WE
STARTED HUGE SPANISH JOURNALS
FOR PMC AND AUTOMATED DOCUMENT.
AND A NUMBER OF OTHER THINGS.
NEXT I WANT TO TALK ABOUT
LEADERSHIP NLM IS PROVIDING TO
NIH AND WE DO THIS IN A NUMBER
OF WAYS, THIS IS JUST ONE
EXAMPLE. THERE IS AN INITIATIVE
CALLED DIGITAL NIH, CO--LED BY
PATTY BRENNAN AND DENNIS CAPULA
AN NIH WIDE INITIATIVE TO
TRANSFORM THE WAY NIH PLAYERS,
MANAGERS AND GOVERNS ITS
ENTERPRISE DIGITAL SYSTEMS. NIH
DOESN'T JUST GIVE GRANTS TO
WORLD AND THAT IS IT, WE ARE A
BIG ENTERPRISE WITH TENS OF
THOUSANDS OF EMPLOYEES AND WE
HAVE A LOT OF DIGITAL SYSTEMS
UNDER THE HOOD. VERY OFTEN THE
SYSTEMS ARE DONE IN ONE OFF
NON-COORDINATED WAY BY
INDIVIDUAL INSTITUTE OR CENTER,
THE IDEA IS TO BRING LOTS OF
COORDINATION TO THE WHOLE LIFE
CYCLE, IF YOU WILL, OF THIS
INFRASTRUCTURE. PATTY AND DENNIS
LED THE ENGAGEMENT WITH OVER 400
NIH STAFF, HAD CONVERSATIONS
WITH 20 OUTSIDE ORGANIZATIONS,
MET WITH A NUMBER OF NIH
GOVERNANCE COMMITTEES TO FIGURE
OUT HOW TO PUT IN PLACE THIS
TRANSFORMATIVE INITIATIVE. THE
ENTERPRISE SYSTEMS THAT WERE
SPECIFICALLY TALKING ABOUT ARE
THOSE THAT ARE USED BY OUR
EXTRAMURAL RESEARCH MANAGEMENT
LIKE ERA FOR EXAMPLE, YOU ARE
PROBABLY FAMILIAR WITH THAT.
WITH OUR ADMINISTRATION AND
MANAGEMENT IN OUR HR SYSTEMS AND
PAYROLL AND THINGS LIKE THAT. AS
WELL AS INTRAMURAL BASIC AND
INTRAMURAL CLINICAL RESEARCH
ACTIVITIES, THOSE WILL BE
SCIENTIFIC DATABASES, AS WELL AS
OTHER KINDS OF DIGITAL SYSTEMS.
THEY ALSO LOOK AT CROSS CUTTING
CAPABILITIES THAT ALL THESE
SYSTEMS WILL LEAD. I PUT THIS UP
TO MAKE THE POINT EACH OF THESE
FOUR FOCI IF YOU WILL, IS BEING
LED BY A -- KIND OF THE TOP
SUBSTANTIVE LEADER AT NIH FOR
EACH OF YOU FOR EXAMPLE, NINA
SHORE IS DIRECTOR OF THE
INTRAMURAL RESEARCH PROGRAM FOR
ALL OF NIH. MIKE LAWYER OF LAUYS
DIRECTOR OF OFFICE OF RESEARCH.
THEY HAVE GOOD APPROPRIATE
LEADERSHIP AND WE ARE VERY
EXCITED ABOUT MOVING THIS
FORWARD TO GET MORE EFFICIENCY
AND A LOT OF MORE COMMONS SENSE
BUILT INTO THE WAY WE DEPLOY
SYSTEMS. FINAL THING I THINK I
WILL TALK ABOUT HERE IS THE
BUDGET. THE GOOD NEWS IS THAT IN
FISCAL YEAR 23 WE HAVE $18
MILLION WE INVESTED IN NEW
INITIATIVES. AGAIN I WON'T GO
THROUGH ALL THESE. ONE WAS TO
ADVANCE DIVERSITY INCLUSION
EQUITY AND ACCESSIBILITY THROUGH
A NUMBER OF ACTIVITI ACTIVITIESE
ALSO STRENGTHENED OUR SECURITY
POSTURE WITH SOME OF OUR MAJOR
RESOURCES. WE HAVE IN THE PAST
POLICY LEGISLATIVE ANALYSIS
CAPACITY OF NLM BY ADDING STAFF
AND THINKING ABOUT DOING
CONSOLIDATION AND REORGANIZATION
AROUND THAT. THAT IS AN
IMPORTANT POINT BECAUSE AT NIH
NLM HAS AN OUTSIDE OUTSIZED
POSTURE WHEN IT COMES TO POLICY
AND LEGISLATION, VERY OFTEN YOU
INVOLVED WITH THE NIH OFFICE OF
SCIENCE POLICY. IN OTHER KINDS
OF OFFICES AT THE NIH OD BECAUSE
OF THE NATURE OF THE WORK THAT
WE DO. BUILDING UP POLICY IS A
WELCOME THING I THINK. SO HERE
IS OUR ONE OVER THE LAST FEW
YEARS. YOU CAN SEE FY 23
ENCIRCLED IN RED. AS YOU
PROBABLY KNOW THERE WAS THE
BUDGET DEAL THAT WILL LEAVE
SPENDING CAPS AT FY 123
REFERRALS FOR NON-DISCRETIONARY
FUNDING. THERE IS INTEREST IN
THE HOUSE TO HAVE LOWER
APPROPRIATION THAN FY 23 THAT
MAY NOT GET THROUGH THE SENATE
SO I THINK THIS IS PROBABLY
PRETTY CLOSE, I SHOULD MAKE THE
POINT HOWEVER THAT THAT STAYING
AT THAT LEVEL IS THE OVERALL
BUDGET SO DOESN'T MEAN EVERY
FEDERAL AGENCY GETS THE SAME FY
23, THAT WOULD BE TWEAKED AROUND
THEN. BUT ANY CASE I THINK NEXT
YEAR IS PROBABLY NOT GOING TO BE
AS GOOD AS THIS YEAR IN TERMS OF
GETTING AN INCREASE. THAT WILL
PROBABLY (INAUDIBLE). AS ALWAYS
IF YOU WANT TO CONNECT WITH
PATTY, WITH LEADERSHIP OF NLM,
NUMBER OF WAYS TO DO THAT. YOU
CAN SEE THE EMAIL ADDRESSES
HERE, TWITTER AND I WILL REMIND
EVERYBODY FROM PATTY FROM THE
MEZZANINE BLOG. SHE POSTS
INTERESTING KIND AND THOUGHTFUL
MESSAGES ON HER BLOG. SOMETHING
YOU CAN FOLLOW IF YOU ARE
INTERESTED IN KEEPING UP WITH
NLM ON A MORE REGULAR BASIS THAN
JUST THESE MEETINGS. WITH THAT
THANK YOU FOR YOUR ATTENTION.
AND HAPPY TO ANSWER ANY
QUESTIONS YOU MAY HAVE.
>> THANK YOU VERY MUCH, DR.
HUERTA FOR GIVING THAT
DIRECTOR'S UPDATE. WHILE I GIVE
SOME MEMBERS OF THE COMMITTEE
CHANCE A CHANCE TO COLLECT
THOUGHTS. ONE QUESTION. YOU
TALKED ABOUT THE INITIATIVE
AROUND DIGITAL NIH. CURIOUS WHAT
IMEXACT IF ANY THAT WOULD HAVE
TO NLM, FOR EXAMPLE WOULD IT
AFFECT DOMAIN NAMES, AFFECT
ARCHITECTURE, SPECIFIC SYSTEMS,
THOSE KINDS OF THINGS?
>> I THINK AT THIS POINT CERTAIN
ASPECTS OF WHAT NLM USES WILL BE
EFFECTIVE, THOSE ARE THE ASPECTS
RELATED TO ENTERPRISE. SO ALL OF
NIH SYSTEMS. THE INTRAMURAL
RESEARCH PROGRAM FOR EXAMPLE,
MIGHT FIND CERTAIN RESOURCES
AVAILABLE TO THEM THAT MAYBE
DON'T EXIST NOW IF SOMETHING
LIKE THAT IS STARTED. I DON'T
THINK IT WILL HAVE A BIG IMPACT
ON OUR RESOURCES LIKE SRA SO
FORTH EARLY ON. I CAN IMAGINE
ONCE THIS MATURES IT WILL MAKE
SENSE FOR OUR SYSTEMS, FOR THOSE
SYSTEMS TO START ALIGNING WITH
THESE OTHER ENTERPRISE SYSTEMS.
CERTAINLY NOT AN OVERNIGHT THING
THOUGH.
>> THANK YOU. ANY COMMITTEE
MEMBERS HAVE QUESTIONS? THANKS
SO MUCH FOR THAT. I HOPE YOU
HAVE A GOOD REST OF THE MEETING
>> THANK YOU VERY MUCH, DR.
HUERTA.
>> YOU BET.
>> FYI DR. PRESENTNAN IS OPEN TO
ANYONE WHO WANTS TO DO A GUEST
POST ON HER BLOG. IF THAT IS ON
INTEREST TO SOMEONE ESPECIALLY
SOMETHING AROUND SOME EVENT OR
ANNIVERSARY OR MILESTONE OR
SOMETHING LIKE THAT, SHE IS
ALWAYS OPEN TO THAT IF YOU WOULD
LIKE TO PITCH A SUGGESTION.
>> GREAT TO KNOW. THANK YOU VERY
MUCH, DIANNE. NEXT UP WE HAVE
DR. LAWRENCE HOPPER. LAUREN WE
KNOW ON THE COMMITTEE FAIRLY
WELL, HEAD OF THE DEVELOPMENT
AND ACQUISITIONS SECTION OF THE
NIH. YOU WILL GIVE THE TIME TO
HER.
>> (INAUDIBLE) GAVE ME A QUICK
PROMOTION THER THERE. I I'M HEAF
UNIT ONE BUT MAYBE SOME DAY. SO
I WAS HERE TO GIVE YOU GUYS SOME
QUICK LITTLE ECLECTIC BUNCH OF
UPDATES FOR YOU THAT ARE
RELEVANT TO OUR TWO JOURNAL
LITERATURE DATABASES THAT ARE
INCLUDED IN PUBMED WHICH ARE MED
LINE AND PUBMED CENTRAL. THE
FIRST UPDATES I WANT TO GIVE ARE
OVERDUE HOUSEKEEPING STATISTICS
FROM FISCAL YEAR 2022. I'M A
LITTLE DELINQUENT SENDING YOUR
GUYS WAY. AS AN FYI IN FISCAL
YEAR 2022 WE DID REVIEW 238
APPLICATIONS FOR MED LINE. OUR
ACCEPTANCE RATE ENDED UP BEING
14%. WE DON'T HAVE A SET OR
GOAL ACCEPTANCE RATE FOR MED
LINE BUT DOES SEEM TO SHAKE OUT
EVERY YEAR TO BE CONSISTENT AND
IT HAS BEEN BETWEEN 11 TO 14 OR
15% PRETTY MUCH AS LONG AS I CAN
GO BACK WITH STATISTICS. BUT
LOOKING AT THESE MORE
GRANULARLY, IF WE BREAK DOWN THE
JOURNALS NOT ACCEPTED TO MED
LINE, THEY RECOLLECT ARE
JOURNALS REJECTED. WE LOOK AT
THE PROPORTION OF PERCENTAGE
DEPENDING WHERE THEY WERE
REJECTED IN REVIEW STAGE. WE ARE
ACTUALLY SEEING THAT OVER THE
LAST FEW YEARS, QUITE A FEW ARE
REJECTED DURING OUR INITIAL
APPLICATION STAGE. THESE ARE
JOURNALS THAT DON'T MEET MINIMUM
REQUIREMENTS TO BE ELIGIBLE FOR
MED LINE. THINGS LIKE HAVING
PUBLISHED AT LEAST 40 ARTICLES
PUBLISHED FOR AT LEAST A YEAR,
BEING BIOMEDICAL IN SCOPE. OR
MEETING OUR MINIMUM QUALITY
STANDARD TO EVEN BE ELIGIBLE FOR
THE COLLECTION. SO WE ARE
READING OUT ABOUT 35 TO 40% OF
JOURNALS THAT END UP BEING
REJECTED PRIOR TO LSTRC REVIEW.
SO THE SCIENTIFIC QUALITY REVIEW
STAGE WHICH DID INVOLVE LSTRC IS
RESPONSIBLE FOR 60 TO 65% OF THE
REJECTIONS SO IT IS IMPORTANT TO
KEEP IN MIND WHEN LOOKING AT
ACCEPTANCE RATE THAT THAT DOES
INCLUDE JOURNALS THAT AREN'T
EVEN ELIGIBLE TO BE CONSIDERED
FOR MED LINE. THE NEXT UPDATE TO
GIVE YOU GUYS IS AN UPDATE TO
NLM POLICY OR CHANGES TO
JOURNALS THAT ARE CURRENTLY
INCLUDED IN PUBMED CENTRAL OR
MED LINE SO THIS IS A LONG
STANDING POLICY BUT WE UPDATED
OUR GUIDANCE ON THE POLICY TO
CLARIFY A FEW THINGS AND FINE
TUNE THE POLICY. THE FIRST PART
OF THE POLICY CLARIFIES WHAT NLM
EXPECTS, JOURNALS OR PUBLISHERS
WITH JOURNALS THAT ARE IN PMC
MED LINE TO REPORT BACK TO NLM
ABOUT ANY CHANGES WITH THEIR
JOURNAL SO THE KINDS OF THINGS
WE EXPECT THEM TO REPORT TO US
ARE CHANGES IN JOURNAL TITLE,
CHANGE IN JOURNAL PUBLISHER,
OWNERSHIP, SCOPE, EDITORIAL
BOARD OR EDITORIAL POLICIES.
CONVERSELY, NLM MONITORS
JOURNALS FOR CHANGE IN
PUBLICATION PRACTICE. SO THIS
INCLUDES THINGS SUCH AS CHANGES
IN PUBLICATION VOLUME AND
CHANGES CONTENT TYPES. THE BIG
ONE HERE IS PUBLICATION VOLUME
SO THIS WOULD BE SOMETHING LIKE
IF A JOURNAL IS IN MED LINE AND
PUBLISHES 50 ARTICLES IN ONE
YEAR AND THEN WE SEE TWO YEARS
LATER THEY ARE PUBLISHING 5,000
ARTICLES IN A YEAR. THAT
DEFINITELY HAPPENS. THAT IS THE
KIND OF THING THAT WILL TRIGGER
US TO LOOK MORE CLOSELY INTO THE
JOURNAL. LIKE I SAID, WE DID
RECENTLY RELEASE GUIDANCE ON
THIS. OUR MAIN MECHANISM FOR
COMMUNICATING POLICY UPDATES
THINGS LIKE THIS TO PUBLIC IS
THROUGH THE NLM TECHNICAL
BULLETIN, OUR NEWS LETTER, SURE
YOU SUBSCRIBE TO THIS AND READ
DAILY BUT JUST IN CASE YOU
DON'T, THIS IS WHAT THIS L
BULLETIN IS MEANT TO DO. AND
THIS ARTICLE IN THE BULLETIN
WENT OUT IN MARCH APRIL ISSUE
EARLIER THIS YEAR. AS FAR AS HOW
NLM PROCESSES A CHANGE WHEN
REPORTED TO US, OUR POLICY ALSO
CLARIFYING THAT THE FIRST THING
NLM DOES IS REACH OUT TO THE
JOURNAL FOR MORE INFORMATION. SO
IF A JOURNAL REPORTS TO US A
CHANGE OR IF WE FIND THE CHANGE
IN PUBLICATION VOLUME, FIRST
THING 2002 IS WE SEND THEM A
QUESTIONNAIRE, THAT WILL BE
ASKING EXTENT OF CHANGE AND IF
THERE ARE ACCOMPANYING CHANGES.
FOR EXAMPLE, IF JOURNAL HAD A
CHANGE IN TITLE, IT IS
COMPLETELY LOGICAL TOLL THINK
THERE'S AIMS IN SCOPE OF JOURNAL
OR EDITORIAL THAT REFLECT THAT
CHANGE IN TITLE SO WE ARE ASKING
SO WE CAN SEE WHAT HAS HAPPENED
AND WHAT ARE THE EXTENT OF THE
CHANGES. ONCE WE GET THAT
INFORMATION WE REVIEW IT
CAREFULLY AND I WOULD SAY LIKE
90% OF THE TIME, ANECDOTALLY
SPEAKING, THERE IS NO ISSUE. WE
FIND THAT THE CHANGE IS NOT THAT
BIG A DEAL, NOT SIGNIFICANT AND
WE SIGN OFF IN THE JOURNAL IS
ALLOWED TO CONTINUE ON IN ITS
RESPECTIVE DATABASE. HOWEVER,
SOMETIMES WE FIND THESE CHANGES
ARE SIGNIFICANT AND WHAT WE ARE
LOOKING FOR IS TO SEE IF THIS
JOURNAL IS ESSENTIALLY A
COMPLETELY DIFFERENT JOURNAL
THAN JOURNAL WE ONLY NAYLY
SIGNED OFF ON WHEN IT WAS
INCLUDED MANY THE DATABASE IF WE
FIND THE JOURNAL SEEMS TO BE
DIFFERENT FROM PREVIOUSLY, IT'S
YOU COULD GONE APPROXIMATE
SUBSTANTIVE CHANGE WE WILL KEY
FOR RE-EVALUATION. FOR OUR
RE-EVALUATION PROCESS, THIS
FOLLOWS OUR STANDARD APPLICATION
REVIEW PROCESS. WE WILL REACH
OUT TO THE JOURNAL AND NOTIFY
THEY ARE RE-EVALUATE AND WHY AND
WE WILL ASK THEM TO SUBMIT
APPLICATION SO THAT WE CAN HAVE
UP DO DATE INFORMATION ON THE
JOURNAL. FROM THERE IT PROCEEDS
LIKE IF IT WAS A NEW JOURNAL
APPLYING TO THE DATABASE, GO
THROUGH STANDARD REVIEW PROCESS
AND IF MED LINE IT WILL REVIEW
BY LSTRC. SO SOMETHING TO BE
AWARE OF IF YOU ARE ASSIGNED THE
REVIEW A JOURNAL THAT IS A
RE-EVALUATION IT WILL BE NOTED
IN YOUR SCREENING NOTES AND YOU
SHOULD CONSIDER AND REVIEW THE
WAY YOU WOULD ANY OTHER BRAN NEW
APPLICATION TO MED LINE. BUT
JUST SOMETHING SO YOU GUYS HAVE
MORE CONTEXT ON. THE LAST UPDATE
TO GIVE YOU GUYS, DR. HUERTA
ALLUDED TO IN HIS PRESENTATION,
THIS THIS UPDATE TO PUBMED
CENTRAL LANGUAGE POLICY. SO
PUBMED CENTRAL OR PMC HAS
HISTORICALLY BEEN RESTRICTD TO
REVIEWING ONLY JOURNALS
PUBLISHED IN PRIMARILY ENGLISH.
JUST RECENTLY NLM MADE THE
DECISION TO EXPAND THE LANGUAGE
POLICY TO INCLUDE JOURNALS
PUBLISHED PRIMARILY IN SPANISH.
THERE IS A LOT OF DIFFERENT
REASONS WHY WE DECIDED TO DO
THIS, COUPLE ARE LISTED HERE.
FOR ONE, WHEN LOOKING AT DATA
FROM PUBMED CENTRAL WE NOTICED
STRIKING GEOGRAPHIC COVERAGE
GAPS FOR WHERE WE HAVE JOURNALS
COMING FROM. WE DIDN'T HAVE A
LOT OF JOURNAL LITERATURE IN PMC
FROM SOUTH AMERICA SO WE WERE
HOPING THAT INCLUDING SPANISH AS
PART OF OUR LANGUAGE POLICY,
WOULD HELP TO ADDRESS THIS.
ANOTHER IMPORTANT TOPIC IS THAT
SPANISH SPEAKING USERS, THERE
ARE INCREDIBLE NUMBER OF SPANISH
SPEAKING INDIVIDUALS IN H THE
UNITED STATES. IT IS THE SECOND
MOST SPOKEN LANGUAGE AND WE ARE
NATIONAL LIBRARY OF MEDICINE. WE
HAVE ALSO SEEN SPANISH PRODUCTS
NLM PUTS OUT WE HAVE OTHER
SPANISH PRODUCTS GET A LOT OF
USE. IN FACT SOME GET MORE USE
THAN THE ENGLISH LANGUAGE
COUNTERPARTS. SO WE KNOW THERE
IS A STRONG AUDIENCE FOR THIS
TYPE OF CONTENT. INCLUDING THESE
ARGUES WILL ALSO BE SERVICE O TO
O ARTICLES WILL BE SERVICE TO
OUR USERS. FINAL POINT TO STRESS
IS WE HAVE DONE A LOT OF WORK IN
THE LAST FEW YEARS TO ALIGN PMC
AND MED LINE. THEY HAVE A LOT OF
INTERPLAY WITH EACH OTHER AND A
LOT OF JOURNALS PARTICIPATE IN
BOTH, A LOT OF PUBLISHERS APPLY
TO BOTH. SO WE HAVE REALLY TRIED
TO ALIGN POLICIES SO THEY MAKE
SENSE WITH ONE ANOTHER. AS YOU
GUYS ARE WELL AWARE, MED LINE
REVIEWS JOURNALS IN ANY LANGUAGE
AS LONG AS THEY HAVE ENGLISH
TITLES AND ABSTRACTS. SO
BROADENING THE LANGUAGE POLICY
FOR PUBMED CENTRAL IS ONE STOP
TOWARDS GETTING THESE POLICIES
INTO CLOSER ALIGNMENT AS WELL.
SO JUST LIKE WITH THE PREVIOUS
POLICY UPDATE WE PUT THIS OUT IN
NLM TECHNICAL BULLETIN. THIS
WENT OUT IN FEBRUARY OF THIS
YEAR. AND I'M HAPPY TO REPORT WE
HAVE REVIEWED ALMOST 20
APPLICATIONS. AFTER THIS
ANNOUNCEMENT WENT OUT. IT IS TOO
SOON TO COMMENT HOW THIS IS
GOING BUT WE ARE CLOSELY
MONITORING SEVERAL OUTCOMES TO
HELP INFORM OUR NEXT STEPS. SO
SOME OF THE THINGS WE ARE
MONITORING ARE THE SUCCESS RATES
OF THESE JOURNALS. AS WE
DISCUSSED EARLIER, PUBMED
CENTRAL HAS THE OWN JOURNAL
REVIEW PROCESS WHICH IS QUITE
RIGOROUS SO WE ARE LOOKING TO
SEE HOW THESE JOURNALS ARE
FAIRING COMPARED TO OTHERS. WE
ARE ALSO GOING TO LOOK AT EFFECT
ON GEOGRAPHIC COVERAGE SO I
MENTIONED THERE IS THE COVERAGE
GAP IN SPECIFIC AREAS OF THE
WORLD AND WE ARE HOPING THIS
WILL ADDRESS THAT. WE ARE ALSO
LOOKING AT THE FEASIBILITY OF
REVIEW. SO OUR REVIEWERS FOR
PUBMED CENTRAL WE ARE CHECKING
WITH THEM TO SEE HOW THEY ARE
DOING BECAUSE THIS IS FIRST TIME
FOR MANY OF THEM REVIEWING
NON-ENGLISH CONTENT AND WE WILL
SEE IF IT MAKES SENSE TO PERHAPS
OPEN THE LANGUAGE GUIDELINES
MORE BROADLY OR IF WE SHOULD
TAKE A DIFFERENT DIRECTION.
USING THE OUTCOMES HERE. SO I
WILL EPICOO IT SHORT AND SWEET.
THAT WAS MY UPDATES AND I'M
HAPPY TO TALK ANY QUESTIONS. I
THINK I HAVE TWO MINUTES.
>> THANK YOU VERY MUCH. ANY
COMMUNITY MEMBERS HAVE
QUESTIONS? IF NOT I WILL ASK YOU
TO FEEL FREE TO ANSWER YOUR
FAVORITE. ONE IS REGARDING THAT
LANGUAGE EXPANSION WHAT STEPS
ARE BEING TAKEN TO ENSURE
SPANISH LANGUAGE JOURNALS ARE
BEING EVALUATED CONSISTENT WITH
ENGLISH LANGUAGE JOURNALS,OR?
SECOND PART, IT MAKES SENSE WE
ARE LOOKING AT SPANISH FIRST.
ARE THERE PLANS TO LOOK AT OTHER
LANGUAGES AFTER SPANISH?
>> THOSE ARE GREAT QUESTIONS. T
SO WE ARE HOLDING THESE JOURNALS
TO THE EXACT STANDARD WE DO TO
THE ENGLISH LANGUAGE JOURNALS.
AS AN INITIAL STEP JUST TO MAKE
SURE TO BE CAREFUL WE DID ON
BOARD SPANISH FLUENT REVIEWERS
SPECIFICALLY TO HELP OUT WITH
THIS REVIEW AND WE HAVE BEEN
CO-PAIRING THEM WITH A
NON-FLUENT REVIEWER NOW AND WE
ARE CHECKING IN WITH THE
NON-FLUENT REVIEWER TO SEE HOW
THEY ARE DOING AND COMPARING THE
TWO REVIEWS. ONCE UPON A TIME,
IT WOULD HAVE BEEN MUCH MORE
DIFFICULT TO DO THIS BUT IN THE
SAME AGE AS YOU GUYS KNOW WE
HAVE MANY TRANSLATION TOOLS, SO
WE ARE LOOKING HOW THOSE CAN BE
USED TO FAIRLY EVALUATE THE
SCIENTIFIC QUALITY OF THESE
ARTICLES. ONE LIMITATION I THINK
WE MIGHT RUN INTO, THIS IS
SOMETHING WE ARE AGAIN CHECKING
WITH OUR REVIEWERS ON IS THAT
TRANSLATION TOOLS WORK WELL I
THINK WITH LIKE SPANISH, FRENCH,
LATIN BASED LANGUAGES. WE TEND
TO HAVE MORE ISSUE WHEN TRYING
TO TRANSLATE OTHER LANGUAGES
PARTICULARLY ONES WRITTEN IN THE
OPPOSITE DIRECTION OR HAVE
CHARACTERISTICS LIKE SOMETHING
LIKE CHINESE AND MANDARIN. THEY
DON'T TRANSLATE AS WELL SO THAT
MIGHT BE A LIMITATION FOR US BUT
WE ARE THINKING THAT IT MIGHT
MAKE SENSE TO START OPENING UP
TO OTHER LANGUAGES THAT ARE
FEASIBLE FOR US TO REVIEW.
>> I WOULD LIKE TO ADD, THE
OTHER THING WE ARE THINKING
ABOUT AS WELL IS HOW IS THIS
LANGUAGE SEARCHING IN OUR
CURRENT SYSTEM? SO IF WE DO ADD
-- IF YOU SEARCH IN PMC ON CASA
IS IT MAPPING TO HOUSE OR
SOMETHING TO THAT EFFECT. ARE WE
COVERING BOTH ASPECTS OF IT AND
HOW DO WE INTRODUCE OTHER
LANGUAGES AND SEARCHING IN OUR
CUBS SYSTEM SO A LITTLE MORE
THAN THE REVIEW PROCESS, WE A
ALSO THINKING THINKING HOW THEY
ARE FINDSABLE BY USERS PROPERLY
>> THIS SOUNDS LIKE A VERY
COMPLEX TASK AT HAND OF WHAT IS
BEING DONE. SO THIS SOUNDS
WONDERFUL AND GREAT STEPS
FORWARD, EXCITED TO SEE WHAT THE
FUTURE HOLDS IN THAT AREA.
LAUREN, THANK YOU SO MUCH FOR
THAT UPDATE. WE NOW HAVE TIME
FOR DIANNE BABSKI WHO IS THE
ASSOCIATE DIRECTOR FOR LIBRARY
OPERATIONS, THE LSTRC SCIENTIFIC
REVIEW ADMINISTRATOR NATIONAL
LIBRARY OF MEDICINE. SHE IS
GOING TO BE TALKING WITH US
ABOUT AI, SPECIFICALLY ASKED HER
A QUESTION ABOUT CHATGPT SO I'M
VERY EXCITED TO HEAR WHAT SHE
HAS FOR US THIS AFTERNOON.
DIANNE, TAKE IT AWAY.
>> DIANNE, I'M SORRY TO
INTERRUPT, APPEARS YOU ARE ON
MUTE.
>> HOW IS THAT?
>> THAT IS GREAT.
>> OKAY. DO YOU SEE MY SLIDES?
>> YES.
>> FANTASTIC. LET ME MOVE THIS
OVER HERE. SO THANK YOU,
EVERYBODY. AS KELLY STATED, I'M
DIANNE BABSKI, EXECUTIVE
DIRECTOR FOR OPERATION, AS MIKE
MENTIONED EARLIER I HAD BEEN ON
THE TOP CIRCUIT WITH MY GEN AI
SLIDES, THE BEST BANG FOR MY
BUCK THAT I INVESTED IN A SLIDE
DECK EVER IN MY CAREER I THINK.
BUT IT IS VERY (INAUDIBLE) SLIDE
DECK BECAUSE THE AI LANDSCAPE IS
SHIFTING SO RAPIDLY AND EVERY
DAY THERE SEEMS TO BE ARTICLES
EITHER TOUTING THE POSSIBILITIES
AND -- OF THIS TECHNOLOGY OR THE
END OF CIVILIZATION AS WE KNOW
IT AND THEN EVERYTHING IN
BETWEEN THERE. THIS TECHNOLOGY
GEN AI HAS GAINED POPULARITY
SINCE DECEMBER OF LAST YEARS
BECAUSE OF ITS HUMAN LIKE
CONVERSATIONAL LANGUAGE
ABILITIES, ITS EASE OF ACCESS
YOU DONE HAVE TO KNOW SYNTAX OR
LANGUAGE. IT IS BROAD
VERSATILITY. IT CAN RECALL AND
BUILD ON VAST AMOUNTS OF TEXT,
IT CAN SYNTHESIZE LARGE DATA
SETS, EVEN GENERATE IMAGERY OF
ARTS AND MUSIC, THESE ARE BASED
ON CONVERSATIONAL PROMPTS THAT
ARE ITERATIVE AS WEL WELL. SO IS
AN INTERESTING SET OF
TECHNOLOGIES TO TRY OUT IF YOU
HAVEN'T DONE THAT ALREADY. MANY
OF US AT NLM ARE EXPERIMENTING
AND FINDING THESE VERY
FASCINATING -- SOME OF THE
RESULTS, VERY HELPFUL. I USE IT
QUITE OFTEN FOR ACTUALLY LOOKING
AT SOME OF THE RETURN ON WHAT IT
BRINGS IN FOR QUESTIONS FOR NLM
SO LET'S START WITH SOME BASIC
DEFINITIONS HERE. A CHAT BOT IS
A TOOL DRIVEN BY ARTIFICIAL
INTELLIGENCE AUTOMATED RULES,
NATURAL LANGUAGE PROCESSING AND
MACHINE LEARNING TO PROCESS DATA
TO DELIVER RESPONSES TO REQUESTS
OF ALL KIND KINDS. ARTIFICIAL
INTELLIGENCE BROADLY REFERS TO
THE IDEA OF COMPUTERS THAT CAN
LEARN AND MAKE DECISIONS IN A
HUMAN LIKE WAY. A COMMON TYPE OF
GENERATIVE AI GEN AI AS I REFER
OUT IS THE LARGE LANGUAGE MODELS
OR LLMs AND THOSE CAN
RECOGNIZE SUMMARIZE TRANSLATE
PREDICT AND GENERATE TEXT AND
OTHER CONTENT BASED ON KNOWLEDGE
GAINS FROM THESE MASSIVE DATA
SETS AND SCRAPING ACROSS
MILLIONS OF WEB PAGES AND WEB
URLs. MOST PEOPLE ARE TALKING
ABOUT CHATGPT BUT THERE IS A
LOTS OF DIFFERENT BOTS OUT
THERE. THIS IS JUST A HANDFUL.
THIS IS NOT GOING AWAY. THIS IS
NOT A PASSING FAD. THIS
TECHNOLOGY IS HERE AND SO WHAT I
WILL FINISH WITH IS YOU -- WHAT
ARE WE DOING IN THIS ARENA, THIS
RESEARCH LANDSCAPE. VERY
QUICKLY, THIS DIAGRAM CONVEY
HOBBIES MODELS WORK AT SCALE. ON
THE LEFT SIDE YOU WILL SEE
DIFFERENT TYPES OF FORMATS OF
INFORMATION THAT ARE ENTERED.
BEYOND LANGUAGE TO INCLUDE
AUDIO-VISUAL AND OTHER FORMS OF
SIGNALS LIKE 3-D SIGNALS. THESE
MODELS IN THE MIDDLE COMPLETE A
VARIETY OF TASKS THAT YOU SEE ON
THE RIGHT SUCH AS GENERAL
GENERATING A COMPUTER CODE WHICH
IS ONE OUR USE CASES, GENERATING
SAs, FAIR WELL HOMES O OTHER
THINGS, APPLY FOR INFORMATION
THAT HAD BEEN LEARNED IN
PREVIOUS SITUATIONS. HOWEVER, I
DO WANT TO POINT HERE DOWN TO
THE BOTTOM. THIS IS SOMETHING
THAT I CONSTANTLY POINT OUT TO
PEOPLE, THAT YOU SHOULD KNOW
BEFORE YOU GO. TWO THINGS. FIRST
ANYTHING THAT YOU PUT INTO THESE
MODELS, IF YOU ARE NOT IN A
PRIVATE VERSION OR IF YOU
HAVEN'T OPTED OUT, THEY BECOME
PART OF THE LEARNING ALGORITHM,
PART OF THE PROPERTY OF THE AI
SYSTEM IF YOU WILL. SO REALLY
HAVE TO BE CAREFUL ABOUT
PRIVACY, ENTERING ANY PII,
PATIENT DATA, I SHOULDN'T HAVE
TO SAY THAT OUTLOUD BUT I DO
ANYWAY BECAUSE YOU NEVER KNOW.
SO LET'S MOVE ON HERE. SO THESE
ARE JUST A FEW WAYS THAT WE ARE
THINKING ABOUT GENERATIVE AI IN
THE SCIENTIFIC AND THE RESEARCH
LANDSCAPE. FOR ONE, ITS ABILITY
TO ANALYZE LARGE AND COMPLEX
DATA SETS IN ARPA-H IS REALLY
THINKING HOW THEY CAN HARNESS
THIS POWER AS WELL. ANOTHER GOOD
THING I HAVE HEARD ABOUT FROM
SOME OF THE DRUG DEVELOPERS IS
THAT ITS ABILITY TO RECOGNIZE
NEW CHEMICAL SYNTHESIS ROUTES
THAT ARE EXPONENTIALLY FASTER
THAN HUMANS, SO THEY ARE REALLY
THINKING HOW THESE MODELS CAN
HELP DETECT NEW OPPORTUNITIES
FOR DRUGS AND OTHER MATERIALS.
YOU CAN SEE SOME OF THESE OTHERS
IDEAS, I DO WANT TO DRAW YOUR
ATTENTION DOWN TO THE BOTTOM
BULLET BECAUSE THIS IS ONE WE
HAVE BEEN TALKING ABOUT WITH
SOME PUBLISHERS AND OTHER
COMMUNICATIONS GROUPS, IT IS A
TOOL THAT YOU CAN USE TO SEE THE
PEER REVIEW PROCESS OR CREATE
EASIER TO READS SUMMARIES OR
LANGUAGE APPROPRIATE SUMMARIES
OF SOME OF THE INFORMATION YOU
HAVE OUT THERE. SO VERY
INTERESTING THOUGHTS GOING ON
WITH THIS. I WILL GO THROUGH A
COUPLE OF USE CASES BECAUSE I
THIS I THIS HELPS TO WRAP YOUR
BRAIN AROUND DIFFERENT USES.
STANFORD UNIVERSITY CREATED NLM
AND IT IS A DOMAIN SPECIFIC
LARGE LANGUAGE MODEL THAT USES
THE DATA BEHIND PMC AND PUBMED.
AND THEY USED IT TO ASSIST
RESEARCHERS AND HEALTHCARE
PROFESSIONALS IN CREATING
SOLUTIONS AND SOLVING COMPLEX
MEDICAL QUESTIONS FROM VAST
AMOUNTS OF BIOMEDICAL ARTICLES
AS I STATED IN OUR RESOURCES SO
THEY PUT IT TO THE TEST BY
ANSWERING COMPLEX CASE REPORT
TYPE OF MEDICAL SCHOOL BOARD
QUESTIONS AT AN IMPRESSIVE RATE
OF SUCCESS THAT YOU CAN SEE HERE
BY 82% ACCURACY. SO IT DOES MAKE
IT A COMPELLING CASE, COULD IT
BE USED AS A ENCATIONAL TOOL
BECAUSE OF THE THAT ITERATIVE
PROMPTING CAPABILITY. ANOTHER
USE CASE, SCIENTIFIC WRITING. SO
I KNOW MY CONVERSATIONS WITH
PEOPLE RESEARCHERS ARE ALREADY
USING THIS TECHNOLOGY TO DRAFT
ARTICLES AND ABOUT ADVANTAGES
SUMMARIZE DATA AND GET
SUGGESTIONS FOR REFERENCES WHICH
IF YOU HAVE SEEN SOME OF THE
ARTICLES OUT THERE, THERE'S
THESE THINGS CALLED
HALLUCINATIONS WHERE THEY JUST
MAKE STUFF UP SO I ALWAYS TELL
PEOPLE USE BY VERIFY IF YOU ARE
GOING TO WORK IN THIS
TECHNOLOGY. SO A RECENT STUDY
SHOWN HERE IS THEY HAD SOME
REVIEWERS THAT WERE LOOKING AT
TWO SETS OF ABSTRACTS. ONE
CREATED BY THE AUTHOR, THE
ORIGINAL. AND ANOTHER ONE WAS
CREATED BY AI GENERATED
ABSTRACT. THEY FOUND THIS GROUP
OF REVIEWERS FOUND IT VERY
DIFFICULT TO TELL THE DIFFERENCE
BETWEEN THE ORIGINAL AND
GENERATED VERSIONS OF THE
ABSTRACTS AND SOME COMMENTED
THAT THEY ACTUALLY PREFERRED THE
GENERATED ONE BETTER. IS IT A
WAY WE CAN ACTUALLY HARNESS THIS
TECHNOLOGY TO MAKE TIME FOR
SOMETHING ELSE IF YOU DON'T HAVE
TO WRITE YOUR OWN ABSTRACT.
LET'S MOVE TO ANOTHER ONE THAT
WILL MAKE YOU RAISE SOME EYE
BROWS IN A FEW MINUTES. SO
ANOTHER USE CASE IS RADIOLOGY.
IN THIS PAPER IT STATES
RADIOLOGISTS ARE USING GPT FOR
GENERATION, EDUCATIONAL SUPPORT,
CLINICAL DECISION SUPPORT,
PATIENT COMMUNICATIONS AND OTHER
DATA ANALYSIS. AND THEY SAY THAT
AI HAS POTENTIAL TO IMPROVE THE
ACCURACY AND EFFICIENCY OF --
EXCUSE ME MY PHONE SEEMS TO BE
RINGING. HAVEN'T SEEN THAT IN A
WHILE. IT HAS THE POTENTIAL TO
IMPROVE THE ACCURACY AND
EFFICIENCY OF MEDICAL IMAGING
ANALYSIS AS WELL AS ASSIST
RADIOLOGISTS IN THE DIAGNOSTIC
PROCESS. SO MOVING ON. IS MY
BIAS AND INTERPRETABILITY IN
RADIOLOGY. SO LET'S TAKE A LOOK
AT THIS RADIOLOGY USE CASE. SO
DEEP IN THEIR OWN NETWORKS WE
ARE TRAINED USING IMAGES OF SKIN
LESIONS. AND THEY CLASSIFIED
MALIGNANT VERSUS BENIGN
MELANOMAS WITH A HIGH DEGREE OF
ACCURACY. HOWEVER, IT WAS LATER
FOUND THAT THESE IMAGES OF THE
LESIONS THAT INCLUDED RULERS IN
THE WERE CLASSIFIED ASTHMA
ANYTHING LANT BECAUSE THE MODEL
LEARNED THAT WHEN A RULER
INCLUDED MISDEMEANOR THE IMAGE
THE LESION WAS MORE LIKELY TO BE
MALIGNANT. SO THIS BIAS WAS
INTRODUCED BECAUSE RULERS WERE
INCLUDED IN THE IMAGES WHERE THE
CLINICIAN ALREADY THOUGHT THE
LESION WAS MORE LIKELY TO BE
MALIGNANT. IF RULERS WERE NOT
DETECTED THE MODEL WOULD HAVE
HAD A HIGH FALSE NEGATIVE RATE
FOR THE NEW IMAGES. SO THIS IS A
REALLY CLEAR STAND OUT EXAMPLE
HOW HUMAN EXPERTISE MUST
INTERVENE BEFORE ANY DIAGNOSIS
MADE OR TREATMENTS PRESCRIBED
AGAIN USE BY VERIFY AND ALWAYS
HUMAN INTERVENTION IN THERE.
WHICH LEADS TO MY NEXT TOPIC
WHICH IS SOME OF THE
LIMITATIONS. MY APOLOGIES. WENT
TOO QUICKLY. SOME OF THE
LIMITATIONS IN USING GENERATIVE
AI TECHNOLOGY. I ACTUALLY ASKED
I THINK IT WAS CHATGPT FOR THIS
LIST. WHAT ARE SOME OF THE
LIMITATIONS, IT CREATED A VERY
GOOD LIST WE ARE THINKING HERE.
ACT SCRAY SIRE LIABILITY,
REPRODUCIBILITY. I WANT YOU TO
FOCUS ON NUMBER 3, IMPORTANT FOR
THIS GROUP, BECAUSE THE
INTERNATIONAL COMMITTEE OF
MEDICAL JOURNAL EDITORS, SO
ICMJE, YOU HER OF THEM BEFORE,
THEY RELEASED UPDATE TO THEIR
RECOMMENDATIONS CONCERNING AII.
THAT SPECIFIES THAT AUTHORS
SHOULD NOT LIST AI AND AI
ASSISTED TECHNOLOGIES PASS
AUTHOR OR CO-AUTHOR NOR A SITE
AI AS AN AUTHOR. SO I THINK IN
OUR LAST MEETING IN FEBRUARY I
MENTIONED THERE WERE SEVERAL
INSTANCES OF AUTHORS LISTED IN
PUBMED AND I'M VERY HAPPY TO
REPORT THAT THOSE WERE CORRECTED
AND THOSE WERE RELIVED AS OFFERS
IN PUBMED SO PUBLISHING INDUSTRY
IS SELF-CORRECTING. LET'S PUT IT
THAT WAY. SO IN MY TALKS ABOUT
TOWN Z I DID TALK TO -- I PUT A
CALL OUT TO MY COLLEAGUES ACROSS
NLM AND ASKED IF YOU HAD ACCESS
TO THIS TECHNOLOGY, OR IF YOU
WANTED TO EMBRACE IT WHAT WOULD
YOU DO WITH IT? AND SO I WAS HIT
WITH A LOT OF USE CASES, I WILL
GIVE YOU A FEW THAT I THOUGHT
WERE SOMETHING WE CAN MEASURE
BECAUSE WE WANT TO LOOK AT THIS
MEASURE AND THINK ABOUT WHAT WE
ARE GOING TO DO WITH THIS
TECHNOLOGY GOING FORWARDS.
COUPLE TO WRITE CODE AUTOMATION
SCRIPS FASTER, CONCERT FILES TO
DIFFERENT FORMATS, TEST AND
REPRODUCE FUNCTIONALITY. WE
THOUGHT ABOUT USING IT FOR AUTO
RESPONSES TO TRADITIONAL
CUSTOMER SERVI SERVICE. WE ARE A
LIBRARY AND WE GET A LOT OF
REFERENCE QUESTIONS, HOW CAN WE
RESPOND TO THOSE QUICKER OR
AUTOMATE THAT PROCESS. FEEDING
LARGE AMOUNTS OF DATA TO GET
MEANINGFUL SUMMARIZATIONS USING
THIS TECHNOLOGY IS LARGE USE
CASE FOR US AND EVEN CREATING
USER DOCUMENTAT DOCUMENTATION. D
ME FILES. SO WHEN WE UPDATE
INFORMATION ON OUR SYSTEMS, YOU
CAN INPUT THAT AND GET A WELL
DRAFTED TEXT DOCUMENT OUT OF IT
SO WITH THAT, THERE'S MANY OTHER
USE CASES THAT WE ARE THINKING
ABOUT AS WELL. SO SHORT TERM
OPTIONS FOR FEDERAL AGENCIES.
HOW CAN WE THINK ABOUT NEW WAYS
TO HARNESS THE TECHNOLOGY AND
CONDUCT RESEARCH? IF YOU PUT IN
GO TO REPORTER AND CUT AND PASTE
SOME DIFFERENT RESEARCH TOPICS
AND YOU CAN ASK IT O GENERATE
NEW IDEAS, IT IS THINKING BEYOND
OUR OWN LIMITATIONS AND OUR OWN
BLINDERS SOMETIMES. CONSIDER
INFORMATION MANAGEMENT AND
PRACTICES AND TOOLS FOR MAKING
DATA MORE FAIR. AS I MENTIONED
EARLIER, MUCH OF WHAT WE DO IN
LIBRARY OPERATIONS, IN FACT
ACROSS THE ENTIRE LIBRARY IS
CURATION OF DATA. MAKING IT MORE
INTEROPERABLE AND FRIENDLY
ACROSS THE SYSTEMS.
REPRESENTATION IN YOUR RESEARCH
AND OF COURSE INVESTING IN
DEVELOPING NEW EXPERTISE IN
AREAS LIKE AI LITERACY, PROMPT
ENGINEERING, PROMPT ENGINEERING
IS A BIG THING WE ARE LEARNING
ABOUT. YOU DON'T ENTER IT IN
THESE CHAT BOTS, YOU DON'T TREAT
THEM LIKE GOOGLE WHERE YOU THROW
COUPLE OF WORDS IN AND PRESS THE
ENTER KEY AND GO. YOU CAN CREATE
SOME. YOU CAN CREATE VERY
SPECIFIC PROMPT AND QUESTIONS TO
GET MUCH BETTER QUALITY RETURNS.
AND OF COURSE FOUNDATION MODEL
CUSTOMIZATION. THE BIGGEST THING
HERE THAT I HAVE BEEN
EXPERIENCING IS PEOPLE WANT SOME
GRINDS. THEY ARE LIKE CAN I USE
THIS AND PLAY WITH THIS? AS YOU
CAN IMAGINE, IN THE GOVERNMENT
WE HAVE A LOT OF DIFFERENT INPUT
REGARDING USING AI. THERE'S A
LOT OUT THERE. I HAVE INCLUDED
IN THESE AND I'M SURE LAUREN
WILL SHARE THESE WITH EVERYONE.
I DID ADD AN APPENDIX THAT HAS A
LOT OF THESE EXECUTIVE ORDERS
AND SOME OTHER CORE NIH BILL OF
RIGHTS AND THE AUTOMATED
INTELLIGENCE STRATEGY POSITION
AND THE NEST AI MANAGEMENT
FRAMEWORK, SOMETHING THAT IS ONE
OF THE BEGINNING BASIS FOR
EVERYTHING WE ARE DOING IN AI IS
USING THIS NIST FRAMEWORK. IF
YOU ARE NOT FAMILIAR WITH IT I
ENCOURAGE YOU TO TAKE A PEAK AT
THAT. SO WITH THAT, I'M GOING TO
END HERE AND OPEN IT UP TO ANY
QUESTIONS. OR COMMENTS,
THOUGHTS.
>> THANK YOU VERY MUCH DIANNE. I
APPRECIATE YOU TAKING THIS TOPIC
AND HELPING US UNDERSTAND MORE
HOW THIS INTERFACES AND YOU CAN
JUST -- THINGS TO THINK ABOUT AS
WE REACH OUR INDIVIDUAL AREAS OF
INFLUENCE AND WORK. THIS IS IN
MY MIND A NEW CHAPTER IN A
BRAVER NEW WORLD. WHAT -- SO I
ASSUME THERE ARE RESEARCHERS IN
THIS AREA. WHAT IS DOING OUR
PARTNERING WITH OTHERS TO TAKE
ADVANTAGE OF GENERATIVE AI THAT
IS GOING ON OR CONVERSELY
MONITORING FRAUDULENT USE?
>> THERE WAS A ROUND TABLE
THROUGH THE NIH LIBRARY LAST
WEEK. AND WE HAVE FROM THE
ETHICS OFFICE TALKING ABOUT
FRAUD AND DOING DETECTION
SERVICES AND THINGS LIKE THAT.
WE DON'T HAVE ACROSS NLM OR VERY
SPECIFIC GUIDANCE. WHAT I HAVE
NOTICED IS PEOPLE ARE IN IT AND
THEY ARE PLAYING, THE HORSES ARE
OUTSIDE OF THAT GATE. AND SO
WHAT I HAVE FOUND IS PEOPLE ARE
ASKING ME WHAT IS THE GUIDANCE.
I TURN TO THESE -- THE DIFFERENT
APPENDICES THAT -- THE I ITEMS
ON THE APPENDIX THAT I HAVE IN
MY SLIDES Z AS YOUR GO TO
REFERENCES. I REALLY THINK A BIG
THING WILL BE TRANSPARENCY. I
THINK GOING FORWARD THERE WILL
BE AN ACCEPTANCE OF THESE TOOLS
THAT ARE OUT THERE. YOU CAN USE
THEM AS TIME SAVERS. I THINK THE
BIGGEST THING IS GOING TO BE
ATTRIBUTIONS, TRANSPARENCY, WHEN
DID YOU USE IT, WHICH VERSION
DID YOU USE. AND REALLY
PROVIDING THOSE -- AND HOW YOU
CAN USE THEM. SO AGAIN, THIS CAN
LEAD TO MORE ON THE
REPRODUCIBILITY GOING FORD FORW.
THE SCHOLARLY COMMUNICATIONS AND
PUBLISHING COMMUNITY IS REALLY
SCRAMBLING TO GET AHEAD OF THIS
AS WELL IN THINKING ABOUT WHAT
ARE SOME OF THE DETECTION
SERVICES THEY CAN DO. IT IS
GOING TO BE ABOUT EDUCATING
RESEARCHERS. ON USE BY VERIFY
SO CHECK THE INFORMATION YOU ARE
GETTING. THEN BE TRANSPARENT AND
DISCLOSE IT.
>> THANK YOU. LOOKS LIKE PAMELA
HAS A QUESTION.
>> I'M WONDERING, WE HAD TALKED
ABOUT TRANSLATING ARTICLES USING
GOOGLE TRANSLATE. I USE CHATGPT
TO TRANSLATE ARTICLES FROM ON
THE ONE I TRIED WAS WRITTEN IN
KOREAN. AND I COMPARED CHATGPT I
DON'T SPEAK KOREAN, TO PREFACE.
SO I LOOK AT CHATGPT AND LOOKED
AT HOW GOOGLE TRANSLATED IT.
AND GOOGLE TRANSLATE I GUESS WAS
BEING EXTREMELY LITERAL TO A
POINT OF SOMETIMES BEING GOOFY.
CHATGPT GAVE A BEAUTIFUL FLOWING
ARTICLE BUT I QUESTION HOW MUCH
OF THAT WAS ACCURATE. OR IF IT
WAS JUST TRYING TO MAKE -- IN
THE BALANCE OF THOSE TRYING TO
MAKE IT SOUND GOOD. VIEW LOOKED
AT -- LOOKS LIKE YOU ARE NODDING
ABOUT THIS. SO I'M CURIOUS WITH
TRANSLATING, WHAT YOU THINK
ABOUT CHATGPT?
>> I HAVEN'T USED IT FOR -- I
HAVE USED IT FOR TRANSLATIONS
BUT WHAT I FOUND MORE
INTERESTING IT HAS A SENTIMENT
FILTER ON IT SO YOU CAN SAY TURN
THIS DOWN, MAKE THIS MUCH MORE
PLAIN LANGUA LANGUAGE. MAKE THIE
FORMAL LADDER. SO WILL IS A
SENTIMENT ANALYSIS THAT YOU CAN
ASK FOR IN P YOUR RESPONSES AS
WELL. I THINK THE DEFAULT IS THE
CONVERSATIONAL RESPONSES. IN
YOUR TRANSLATION, IT PROBABLY
GAVE YOU THE CONVERSATIONAL
TRANSLATION VERSION AS OPPOSED
TO GAGGLE LIKE YOU SAID, WHICH
IS -- GOOGLE LIKE YOU SAID WHICH
IS WORD FOR WORD.
>> IT ACTUALLY SOUNDED LIKE
PAPER.
>> THE GOOGLE TRANSLATE I
REMEMBER LIKE THE WORD CASTLE
SHOWING UP AND I THINK THERE WAS
SOME LIT RATION LIKE THIS IS A
GARDEN, BUT SO THE GOOGLE
TRANSLATE YOU CAN GET THE GIST
OF THE ARTICLE BUT USING GOOGLE
TRANSLATE IN THE CHATGPT USING
THEM SIDE BY SIDE IT LIKE I HAD
BETTER UNDERSTANDING OF THE
EARLY A. I WAS CONCERNED THAT
CHATGPT WAS PERHAPS FILLING IN
SOME BLANK.
>> WHAT I HAVE DONE IS I HAVE
HAD SEVERAL DIFFERENT DO MYSELF
HERE WITH AN L. BUT I USE
CHATGPT, I USE THE BING CHAT BOX
AND BARD SO I LIKE TO PUT MY
QUESTIONS IN TO EACH OF THEM SO
SEE HOW IT PULLS THEM OUT. FROM
MY OWN EXPERIENCE I HAVE FOUND
THE OPEN AI CHATGPT VERSION, IT
IS MUCH MORE IN MY OPINION A
CONVERSATIONAL RETURN. OF THE
INFORMATION. I DON'T KNOW IF IT
WAS JUST TRAINED THAT WAY, MAYBE
THAT IS BECAUSE MORE PEOPLE HAVE
GIVEN THE THUMBS UP IN THE CHAT
BOX YEAH THAT WAS A GOOD
RESPONSE SO IT KEEPS LEARNING
AND PROVIDES THE FEEDBACK THAT
WAY AS WELL.
>> THANK YOU.
>> IF YOU HAVE TIME I CAN THROW
TO OTHERS AND SEE IF IT IS A --
IF YOU CAN SEE THE DIFFERENCES
IN IT.
>> I HAVE ASKED SOMEONE WHO
SPEAKS KOREAN IF SHE CAN JUST
LOOK AT IT BUT IT TAKES A LOT OF
TIME FOR AN INDIVIDUAL TO
TRANSLATE AN ARTICLE EVEN IF
THEY SPEAK THE LANGUAGE
FLUENTLY. SO IT IS A BIG ASK.
>> IT IS.
>> LOOKS LIKE WE HAVE ANOTHER
QUESTION IN THE CHAT FROM CHRIS
ANDERSON. HOW DOES THIS GUIDANCE
NOT TO GIVE ATTRIBUTION TO AI
FIT WITHIN FRAMEWORK OF
TRANSPARENCY?
>> I THINK IT IS TO GIVE
ATTRIBUTIONS. TO BE TRANSPARENT
ABOUT HOW YOU ARE USING IT. SO
I THINK AS WE MOVE FORWARD IT IS
GOING TO BE AS YOUR DISCLOSING
YOUR USE OF PARTICULAR CHAT BOT
OR GENERATIVE AI SERVICE YOU
WOULD SAY NOT CITE THEM AS AN
AUTHOR. WE JUST SAW THAT THAT IS
NOT A LEGITIMATE THING TO DO
BASED ON THE AUTHORSHIP
STATEMENT FROM IGME AND OTHER
PUBLISHER STATEMENTS OUT THERE.
BUT I THINK THAT WE WOULD -- WE
WILL BE SEEING RESEARCHERS
ACTUALLY HAVING SOME DISCLOSURE
ON THEIR -- IN THEIR RESEARCH
PROBABLY IN THE PAPERS AND
THINGS LIKE THAT. WHAT I'M
CURIOUS ABOUT, AND WASHINGTON
POST A COUPLE, MAYBE LAST MONTH
TIMES SEEMS TO BE IN A VACUUM
LATELY, THERE WAS A DISCLOSURE
ON ALL OF THE DIFFERENT WEB
SITES THAT OPEN AI WAS ACCESSING
AND AS YOU CAN IMAGINE NIH NLM,
MANY OF OUR SYSTEMS AND OUR
URLs WERE POSTED ON THERE. I'M
CURIOUS AS TO EVERY DAY WE ARE
ADDING MORE INFORMATION, THEY
ARE NOT USING API TO KEEP
CURRENT, THEY ARE SCRAPING SO IT
IS GOT A TIME STAMP. THE OTHER
THING TO THINK ABOUT IS WHEN WAS
THIS DATA SCRAPED. IT IS
TYPICALLY NOT CURRENT. ARE YOU
GETTING THE MOST RECENT
INFORMATION SO IN MY HEAD I'M
THINKING COVIDND EVERY DAY A NEW
CURE OR NEW THING NOT TO DO OR
SOME OTHER SOCIAL MEDIA THING
GOING ON WITH IT. THAT WOULD --
I'M THINKING HOW THIS COULD
REALLY BE A PATHWAY TO SOME
INTERESTING DISINFORMATION
ESPECIALLY WITH TIME STAMPS ON
IT. NOT GETTING THE MOST
RECENT. SO IN MY HEAD THAT IS
WHERE I THINK AT LEAST
DISCLOSING WHEN YOU USED IT AND
WHICH SERVICE YOU USE BECAUSE
THEY ARE NOT DISCLOSING WHICH
SITES THEY ARE SCRAPING AND THE
DATES THERE ARE SCRAPES ON THERE
EITHER. LET ALONE THE ALGORITHM
AND HOW ALL OF THAT HAVE IS
GOING THROUGH ALL OF THE
DIFFERENT INDICES IN THOSE AND
GENERATING A SOLUTION IN ANSWER
TO YOUR TOP QUESTION. SO THERE
IS A LOT THAT REMAINS UNKNOWN IN
THESE THINGS. BUT IT DOES LOOK
EXCITING. WE ARE PUTTING
TOGETHER A PROPOSAL TO INTRODUCE
IT TO NLM STAFF. WE ARE GOING TO
HAVE A SECURE ENVIRONMENT, WE
ARE CALLING IT A PLAYGROUND FOR
STAFF TO USE. WE ARE ASKING IT
TO HAVE SOME KIND OF METRIC,
SOME KIND OF MEASURES TO SHOW
HERE IS WHAT I'M TRYING TO DO,
IS THIS AN EFFECTIVE TOOL. AND
FROM THERE OUR PROPOSAL WILL
TAKE US MUCH FURTHER TO THE
FUTURE. IT IS CREATING A DOMAIN,
A LARGE LANGUAGE -- A LARGE LOM
DOMAIN SPECIFIC FOR ALL OF NLM
OR NIH INFORMATION RESOURCES. IS
THAT SOMETHING THAT WE SHOULD
CONSIDER BUILDING? AND TRAINING
THAT UP SO THAT YOU JUST GET
ACCESS AND PROMPT JUST THAT
CORPUS OF DATA. THAT WOULD BE
MILLIONS OF DOLLARS TO TRAIN
THAT UP INTO ALL THE DATA
TOGETHER BUT YOU SEE THE DIGITAL
NIH, THAT IS A BEGINNING STEP TO
HARMONIZING PLATFORM STANDARDS
AND DATA AND HOW WE CAN MAKE
THINGS MUCH MORE INTERACTIVE.
>> THIS HAS BEEN A FASCINATING
DISCUSSION, DIANNE. VERY MUCH
APPRECIATE YOUR EXPERTISE HERE
AND YOUR LEADING US THROUGH
THIS. APPRECIATE THE QUESTIONS
FROM THE COMMITTEE MEMBERS AS
WELL. IT IS ABOUT 2:30. THIS I
BELIEVE CONCLUDES THE OPEN
SESSION OF THE JUNE 2023 LSTRC
MEETING. LAUREN OR DIANNE,
ANYTHING BEFORE WE GO INTO A TEN
MINUTE BREAK HERE?
>> NONE FROM ME.
>> ALL RIGHT. SEE EVERYONE AT 2:40