WELCOME EVERYONE TO OUR FEBRUARY MEETING. NICE AND COLD IN THE NEW ENGLAND AREA. THIS IS BEING VIDEOCAST TO THE PUBLIC AND STARTING OFF THE FIRST ORDER IS THE OFFICE REPORT FROM THE OFFICE OF SURGEON GENERAL. SHE SPOKE AT OUR LAST MEETING AND GAVE HER AN INTRODUCTION. BEING BRIEF, SHE CURRENTLY SERVES HEALTH AND HUMAN SERVICES OFFICE WITH US SURGEON GENERAL AND WELCOME MRS. JEANE GARCIA-DAVIS. WE WOULD LOVE TO HEAR AN UPDATE. THANK YOU. >> THANK YOU SO MUCH. LOVELY SEEING YOU ALL AND GETTING A THUMBS UP FROM EVERYONE YOU CAN SEE MY SCREEN. >> WE CAN HEAR YOU AND SEE YOUR SCREEN. >> WONDERFUL. THANK YOU DR. REHM. IT IS AN HONOR AND THRILL TO JOIN YOU DR. BRENNAN AND REHM AND TO ALL BOARD MEMBERS HERE TODAY IT IS WONDERFUL TO SEE YOU ALL AND WE ARE GRATEFUL FOR THIS OPPORTUNITY TO SHARE AND ON OUR CONTRIBUTIONS AND DR. MARTHI WAS SORRY HE COULDN'T JIEN THIS WEEK BECAUSE OF HIS PERSONAL LEAF AND HE SENDS GRATITUDE YOUR WAY TO ALL YOUR EFFORTS TO DATE AND PANDEMIC OPENING AND SHARING I'M ONE VOICE AMONG MANY IN THE PANDEMIC THAT FEEL IT'S IS ONGOING AS IT IS COLLECTIVE EXPERIENCE REALLY AND UNDERSCORES IMPORTANCE OF OUR COLLABORATION TOGETHER AND THE WORK THAT YOU ALL BRING AND IMPORTANCE OF THE RESEARCH AND EVIDENCE. >> DIANNE BABSKI: BASE. THEY HAVE BEEN ESSENTIAL FOR OUR OFFICE AND MISSION TOGETHER TO ADVANCE AND PROTECT THE NATION'S HEALTH. THANK YOU FOR HAVING US. WITH THAT I WANT TO SHARE OUR JOURNEY. WITH VERY 18 MINUTES UNTIL ABOUT 20 OR 25 AFTER AND WELCOME COMMENTS AND QUESTIONS FOR MYSELF AND I WANT TO SHARE WITH YOU AS WE LAST SPOKE WE HAVE HAD A NUMBER OF DEVELOPMENTS AND PRODUCTS THAT OUR OFFICE HAS WORK THE ON TOGETHER WITH THE INTERAGENCY AND EXTERNAL PARTNERS IN LAST 3 TO 4 MONTHS AND OTHER PRIORITIES YOU ARE WELL AWARE OF AS WELL AS BROADER EFFORTS AND ANY COMMENTS OR QUESTIONS AND HAPPY TO TAKE THEM AT THE END. WITH THAT, I WANTED TO SHARE TOO HERE THIS THAT YOU HAVE SEEN BEFORE. LOTS OF PRODUCTS WE HAVE SPOKEN ABOUT THAT DR. MURTHI CAME TO SPEAK ABOUT AND I SPOKE ABOUT LAST SEPTEMBER REGARDING A NUMBER OF DIFFERENT SCIENTIFIC PRODUCTS WE HAVE HELPED TO DEVELOP AND TRANSLATE TO THE PUBLIC. I WANTED TO SHARE, AGAIN, OUR MISSION HERE AND WHAT WE HAVE BEEN ABLE TO ACHIEVE TOGETHER INCLUDING THE DISCUSSION WITH PRODUCTS AND WITH THE AMERICAN PUBLIC AND 2021 AND TO THIS DATE THREE YEARS LATER INTO THE PANDEMIC THAT DISSEM NIGS CONTINUES AND ADVANCING THIS MISSION AND WORKING TOGETHER REALLY TO HELP TOGETHER TO GET THE NATION TO REALLY UNDERSTAND, YOU KNOW, THESE IMPORTANT PUBLIC HEALTH ISSUES. WE REALLY WORK ON TRANSLATING THIS EVIDENCE IN WAYS THAT ARE DIEGESTIBLE AND PEOPLE CAN TAKE ACTION AND FROM WHERE THEY PARTICULARLY ARE AND CAN YOU SEE FROM PUBLIC HEALTH CRISIS IT IS HEALTH MISINFORMATION THAT IS TRYING TO ELEVATE THE STRUCTURAL AND CULTURAL AND DIFFERENT POLICIES AS WELL AS CHANGES AT THE INDIVIDUAL LEVEL THAT EVERYONE CAN TAKE INCLUDING THROUGH TOOLKITS AND MESSAGING CAMPAIGNS AND DIDN'T OPPORTUNITIES THAT DR. MIRTHI HAS XOEK ABOUT ONL THE ROAD AND THEY FROM ROLE OF GOVERNMENT OR WHOLE OF SOCIETY CAN TAKE ACTION. WE ARE TRYING TO MEET THEM WHERE THEY ARE AND WHERE THEY CAN TAKE ACTION FROM WHERE THEY ARE. SO, MORE RECENT ONE I WOULD LIKE TO TURN OUR ATTENTION TO IS WHAT THE OFFICE HAS BEEN HELPING TO ADVANCE IN PARTICULAR WITHIN THE INTERAGENCY AS WELL AS EXTERNAL STAKEHOLDERS IN THE LAST YEAR AND IN THE LAST FEW MONTHS SINCE ITS LAUNCH AND SURGEON GENERAL'S FRAMEWORK LAUNCHED IN OCTOBER OF LAST YEAR DESCRIBED HERE TO MAINLY DISTINGUISH IT AS OTHERS KNOW FROM YEARS OF GOVERNMENT SERVICE AND PUBLIC HEALTH WORK THIS IS A LITTLE DIFFERENT. WE HAVE SEEN A NUMBER OF SURGEON GENERAL'S REPORT AND SURGEON GENERAL ADVISORIES AND BULLETINS ACROSS SURGEON GENERALS IN PARTICULAR AND MEANT TO CALL ATTENTION TO A PUBLIC HEALTH ISSUE AND LATEST ONE IS ONWARD PLACE MENTAL HEALTH AND WELL BEING. HERE, THE THINK THE MAIN TOPIC AND THEME WE WANT TO HELP TO UNDERSCORE IS THAT WORKPLACE MENTAL HEALTH AND WELLBEING REALLY EFFECTS US ALL. THE PANDEMIC HAS BROUGHT US TO A SHARE THE AND OPPORTUNITY TO REALLY RETHINK HOW WE WORK AND I WOULD REALLY LIKE TO RECOGNIZE YOUR EFFORTS HERE IN TERMS OF HIGHLIGHTING INNOVATIONS IN FUTURE OF WORK AND HELPING US AS A NATION TO REALLY LOOK AT THE POWER WE HAVE WITH VARIOUS OPPORTUNITIES TO HELP MAKE WORKPLACES ENGINES OF WELLBEING. AS WE RECOVER FROM THE PANDEMIC AND THROUGH THE PANDEMIC THAT CONTINUES TOGETHER, IT IS BUILDING ON A LOT OF WORK THAT SURGEON GENERAL'S ADVISORY ON HEALTH WORKER BURNOUT AND WELLBEING FROM LAST SUMMER THAT IS BUILT ON SHOULDERS OF MANY EXPERTS INCLUDING SO MANY IN THE ROOM AND ACROSS THE COUNTRY THAT HAVE BEEN FOCUSED ON THIS AND NOW MORE FOCUSED ON THIS AND THESE ISSUES AND MANY WERE LONGSTANDING AND I WILL GO INTO SOME OF THEM RIGHT NOW. WE HAVE A Q.R. CODE. IF YOU HAVE A PHONE YOU ARE WELCOME TO SCAN THIS USING YOUR PHONE AND WILL SHOW YOU THROUGH AN INTERACTIVE PORTION A WEBSITE AND RESOURCES AVAILABLE. SO, AS WE WERE DOING THIS WORK AND WITH ALL OF THE WORK THAT WE DO, I WANT TO DEFINITELY RECOGNIZE AND EXPRESS GRATITUDE TO SO MANY OF YOU AND TO ALL OF THE WORKERS OUT THERE ACROSS INDUSTRIES AND ORGANIZATIONS LARGE AND SMALL AND LEADERS AND [INDISCERNIBLE] IN ALL LEVELS BUILDING ON DECADES OF LITERATURE AND LITERATURE REVIEW AND LISTENING SESSIONS AND ENGAGE WITH VARIOUS STAKEHOLDERS OVER THE YEAR ACROSS THE COUNTRY IN THE LAST TWO YEARS AND FRAIM -- PUBLIC HEALTH AND HEALTH WORKERS HERE AND MANY OF US SHARE AND IN THE TRAUMA AND HAVING LOST LOVED WUNLS IN THE PANDEMIC ARE EFFECTED AND HOPING AND AIM IS THAT THIS SERVICE AS A CALL TO ACTION WILL LIFT UP THEIR VOICES AND ESPECIALLY MOST VULNERABLE TO CONTINUE PROTECTING HEALTH AND WELLBEING. WE RECOGNIZE TOO, YOU KNOW, IN THE DEVELOPMENT OF THIS THAT THIS IS PERSONAL TIME FOR ALL OF US AND SOME ASPECT WHAT I WILL SHOW YOU IN THE FRAMEWORK WHERE THEY CAN RESONATE AND LINK BETWEEN OUR WORK HAS BEEN HESITANT AND DEMONSTRATING NUMBER OF WORKERS ENGAGED TODAY IN THE US WORKFORCE AND LEVELS OF ENGAGEMENT OR DISENGAGEMENT WE ARE SEEING ACROSS VARIOUS NATIONAL SURVEYS FROM GALLOP OR OTHERS AND PANDEMIC AS WE KNOW HAS REALLY INTENSIFIED AND SPOTLIGHTED SO MANY ISSUES WE SEE IN THE PARTICULARLY HEALTH AND SERVICE INDUSTRY AND TODAY WE SEE MORE AND MORE WORKERS WHO ARE WORRIED ABOUT MAKING ENDS MEET AND WHO ARE DEALING WITH CHRONIC STRESS AND ARE STRUGGLING TO BALANCE DEMANDS IN WORK AND NON-WORK ROLES THAT INCLUDES, YOU KNOW, CAREGIVERS AND INCREASING ROLES AND RESPONSIBILITIES THAT THEY HAVE WHILE THEY ARE ALSO EITHER ENGAGED IN WORK OR UNPAID WORK. I WANTED TO HELP ALSO HIGHLIGHT THIS LAST POINT TOO I THINK THAT THERE IS AN OPPORTUNITY THAT THE AMERICAN PSYCHOLOGICAL ASSOCIATION AMONG OTHERS HAVE JUST REALLY HELPED TO SHOW US, AGAIN, THIS IS AN OPPORTUNITY. THERE IS HOPE, YOU KNOW, THAT THIS -- THAT WORKERS TODAY ACROSS ALL INDUSTRIES AND THESE SURVEYS LOOKING ACROSS ALL SECTORS AND ACROSS AGE GROUPS REALLY SPEAKS TO THE NEEDS AND WHAT WORKERS ARE LOOKING FOR AND ACROSS TODAY'S INCREDIBLY MULTIGENERAL WORKFORCE THAT IS A SHARED EXPERIENCE AND THERE IS HOPE AND OPPORTUNITY. HERE IS SHARING THE MAIN -- TALKING ABOUT THE FRAMEWORK HERE IS HOW IT LOOKS VISUALLY AND VISUAL AND CONCEPTUAL REPRESENTATION THAT WAS DEVELOPED TOGETHER WITH A NUMBER OF STAKEHOLDERS THAT IS BASED YOU CAN SEE WE ARE CALLING WITH DESCRIBING THEM AS FIVE ESSENTIALS. A NUMBER OF DIFFERENT, YOU KNOW, ENGAGEMENTS AND LITERATURE THAT ALL WENT INTO DESCRIBING THIS INCLUDING ABOUT YOU NOT LIMITED TO. YOU WILL SEE THERE IS BUILT EACH OF THESE FIVE ESSENTIALS ON TWO HUMAN NEEDS ON THE LEFT. WHETHER OR NOT MANY OF YOU ARE HEAL NL CARE PRACTITIONERS AND WHETHER SOME OF THEM YOU MIGHT REMEMBER FROM MASLOW'S HIERARCHY OF NEEDS AND ALSO IS BUILT ON FOUNDATIONAL UNIVERSAL NEEDS THAT ARE NEEDED BY ALL HUMANS IN TERMS OF GROWTH AND DEVELOPMENT AND SAFETY AND SECURITY. IT IS REALLY CENTERED AT THE CORE. YOU WILL SEE AT THE CENTER PIECE OF THIS LEAVING IS WORKER VOICE AND EQUITY. SO, EVERYTHING REALLY BEGINS THERE. IT IS THE FOUNDATION. BECAUSE THERE IS NO ONE THAT IS PRIORITIZED OVER ANY OTHER IN TERMS OF BEING AN ESSENTIAL WORKPLACES THAT HAVE AN OPPORTUNITY TO REALLY LOOK AT THIS AND ENGAGE AND BEGIN WHEREVER THEY ARE. YOU KNOW, RECOGNIZING THAT EVERY WORKPLACE WILL HAVE DIFFERENT RESOURCES FOR IMPROVING AND DIFFERENT STAGES ADDRESSING VARIOUS ESSENTIALS. YOU WILL SEE FIVE ESSENTIALS ON THE LEFT. ON THE RIGHT I WANT TO SHARE WHAT IS DESCRIBED AS COMPONENTS THAT BEGINS TO AS WORKPLACES THEY BEGIN TO OPERATIONALIZE AND CREATE ACTION PLANS AS PART OF THEIR STRATEGIC WORKPLANS ALL TOGETHER. THIS IS NOT LIMITED TO WELLNESS AND ALSO SPEAKS TO ISSUES RELATED TO WELLBEING OVERALL AND LOOKING AT WHOLE PERSON HEALTH. SO, IN THE FIRST AREA, YOU WILL SEE DIFFERENT COMPONENTS SPEAKING TO OPPORTUNITY AREAS IN TERMS OF POLICY PROGRAM AND PRACTICE CHANGES WHERE WE CAN HELP REIMAGINE WORKPLACES AS ENGINES OF WELLBEING AND FIVE AREAS, PROTECTION FROM HARM AND CONNECTION AND COMMUNITY AND WORK/LIFE HARMONY MATTERING AT WORK AND OPPORTUNITY FOR GROENL, AGAIN, ARE BUILT ON CONVERSATIONS AND ON LITERATURE AND BUILT ON DIFFERENT OPPORTUNITY WHERE'S LEADERS AND MANAGERS AGAIN AT VARIOUS LEVELS CAN HARNESS WHETHER EXISTED POLL SURVEYS EXISTING EMPLOYEE ENGAGEMENT SURVEYS AND LOOKING AT EXISTING POLICIES RELATED TO FOR EXAMPLE UNDER THE FIRST COMPONENT DEIA THAT DIVERSITY EQUITY AND INCLUSION AND ACCESSIBILITY WHERE WE ARE DOING WELL AS AN ORGANIZATION AND WHERE DO WE HAVE OPPORTUNITY AREAS FOR GROWTH? HERE, I WANTED TO ALSO SHARE TOO SOME OF THE IMPACT, YOU KNOW, SO FAR. WE HAVE HEARD FROM VARIOUS STAKEHOLDERS IT IS RESONATING. WE HAVE HAD A NUMBER OF E-MAILS AND OPPORTUNITIES FOR SPEAKING AND ENGAGING WITH DIFFERENT -- WHETHER IT IS DIFFERENT MAYORS AND AT A CITY LEVEL AND AT INDIVIDUAL VARIOUS ACADEMIC INSTITUTIONS AND VERY SMALL BUSINESS OWNERS AND EVEN LARGE ORGANIZATIONS AND CORPORATIONS WHO HAVE REACHED OUT TO US TO SAY THIS IS DEFINITELY SOMETHING THEY ARE INTERESTED IN EITHER PRIORITIZING OR PARTNERING WITH US OR OTHERS WITHIN THE INTERAGENCY ON. SO, HERE I WANTED TO JUST STOP FOR A MOMENT AND JUST SHOW YOU. THANK YOU FOR BEARING WITH ME. THERE IS NO SOUND I SHOULD HAVE HAD FUN AND MUSIC ALONG THE WAY AND THIS IS THE WEBSITE. IF ABLE TO DOWNLOAD THE Q.R. CODE, WE WELCOME YOU TO INTERACT WITH THE WEBSITE AND GO THROUGH WITH YOU OR YOUR TEAM OR AGENCY WHEN YOU HAVE TIME. YOU CAN DOWNLOAD STUFF AND DOWNLOAD ANY GRAPHICS FOR USE WITHIN YOUR PEER PRESENTATIONS AND YOUR WORK AND THERE IS AN OPPORTUNITY TO EXPLORE FRAMEWORKING BY FIVE ESSENTIALS AND I WANTED TO REALLY POINT OUT THAT THE INFORMATION AND RESOURCES PAGE THAT HELPS WITH A DEEPER DIVE OF VARIOUS RESOURCES WITHIN THE INTERAGENCY THAT ARE REFERENCED IN THE FRAMEWORK HERE TWL IS A VIDEO -- WORKPLACE AND ENGAGEMENT SURVEYS AND THERE IS A DECK HERE THAT YOU CAN DOWNLOAD AS WELL. THERE WE GO. SO, HERE IS A FEW MORE SLIDES THAT I WANTED TO JUST SHARE THE DISSEMINATION THAT WAS LAUNCHED IN OCTOBER OF LAST YEAR IN PHILADELPHIA BY ENGAGEMENT AND COMMUNICATIONS TEAM AND IS A TOOLKIT IF ANYONE IS INTERESTED IN DISSEMINATING IT FURTHER. THAT IS, I THINK, A MAIN ASK THAT WE WOULD LOVE OF YOUR TEAM AND OF ALL OF YOU HERE ON THE BOARD AS WELL AS YOUR TEAMS THAT IS RELATED TO HELP DISSEMINATE AND PROVIDE FEEDBACK AND HOW IT IS RESONATING AND USING IT AND HOW ELSE CAN WE HELP YOU WITH ANY OPPORTUNITIES FOR COLLABORA COLLABORATION. DR. MIR THINK AND TEAM PARTNERED WITH THE WHITE HOUSE AND WORKING WITH CALIFORNIA ENDOWMENT AND NUMBER OF LOCAL AND NATIONAL ORGANIZATIONS TO GARNER COMMITMENT AND ORGANIZATIONS TO HELP MOVE AND PROTECT YOUTH MENTAL HEALTH MOVING FORWARD PARTNERED WITH SECRETARY BACERA ONGOING MANY OCCASIONS AND SPECIFIC TO MENTAL HEALTH ROUNDTABLE HELD VIRTUALLY AND INCLUDING CAST OF WEST WING AND ONGOING FINANCE AND COMMUNITY ENGAGEMENT MENTAL HEALTH CRISIS AND OPPORTUNITIES AND FINALLY ADDRESSING HEALTH WORKER BURNOUT THAT REMAINS TOP OF MIND FOR US BECAUSE OF THE ONGOING PANDEMIC AND IMPACT ON BURNOUT AS WELL AS RISING HEALTH PROFESSIONAL STUDENTS AND TRAINEES THAT WE CONTINUE TO WORK ON THIS ISSUE. MANY ISSUES WITHIN AND IT IS A COMPLEX AND BEST TOPIC. MANY OF YOU KNOW WITH THE NATIONAL ACADEMY OF MEDICINE AND ACTION COLLABORATIVE HEALTH AND WELLBEING RESILIENCE. FINALLY, I WANTED TO SHARE WITH YOU HERE ONGOING EFFORTS AND ONGOING PUBLIC SERVICE HEALTH CORE AND SAW THERE WAS A CHANGE WITH MEDICAL SECTION STANDARDS THAT CAME OUT IN EARLY DECEMBER. ALSO ONGOING WORK WITH NATIONAL BIODEFENSE STRATEGY AND I WANTED TO HIGHLIGHT THERE WILL BE A FORNL COMING PRODUCT WITH ADDITION TO PUBLIC HEALTH REPORTS AND JOURNAL ARTICLES THERE AND DR. MIR THY HAS WRITTEN PIECES IN JOURNAL COMING OUT IN NEXT ISSUES I CAN SHARE WHEN THEY ARE READY FOR PUBLICATION AND ONGOING ACCESS TO INFORMATION AND TRANSLATING SO MUCH OF THE WORK AROUND COVID-19 AND THROUGH THE TRIPLEDEMIC AND THROUGH A VERY CHALLENGING WINTER IN THE WHITE HOUSE AND IN INTERAGENCY IN TERMS OF RESPONSE AND THERE WILL BE FORTHCOMING ENGAGEMENTS HERE ON LONELINESS AND ISOLATION LINK THE TO EFFORTS RELATED TO WORKPLACE WELLBEING AND YOUTH MENTAL HEALTH AND WE HAVE LAUNCHED HOUSE CALLS PODCAST AND WELCOME A LINK THERE SHARED IN THE SLIDES AND WELCOME YOU TO LISTEN AND SHARE A THUMBS UP IF YOU HAVE NOT ALREADY. THERE IS A NUMBER OF -- THEY COME OUT BIWEEKLY AND LASTLY, I WANTED TO HIGHLIGHT OTHER URGENT PUBLIC HEALTH ISSUES THAT I WOULD CONTINUE TO EMERGE AND REMAIN THAT WE WILL CONTINUE OUR WORK ON WHETHER IT IS THE TOBACCO AND OTHER SUBSTANCE USE AND GUN VIOLENCE AND THANK YOU ALL. I REALLY WANTED TO THANK YOU JUST FOR HAVING US OR ME BACK. I WELCOME ANY REACTIONS TO ANYTHING THAT I HAVE SHARED AND ALSO QUESTIONS EITHER FOR MYSELF OR THAT I'M HAPPY TO TAKE BACK TO MY TEAM OR DR. MIR THY AND I'M ONE VOICE AMONG SO MANY AND APPRECIATE THE OPPORTUNITY AND IT IS AN HONOR AND THANKS FOR YOUR WORK. >> THANK YOU. QUESTIONS? DO I SEE TERESA? THAT IS MY MOUSE OVER HERE. ANYONE HAVE ANY QUESTIONS. CLIFTI? >> YES. HI. FIRST OF ALL, THANKS SO MUCH FOR ALL THESE WONDERFUL RESOURCES THAT IS MORE OF A COMMENT THAN A QUESTION. I WANTED TO JUST THANK YOU FOR CAREFUL ATTENTION THAT YOU PLACED ESPECIALLY IN THE WORKPLACE WELLNESS FRAMEWORK AND ALL THOSE GREAT SUPPORT MATERIALS ASSOCIATED WITH IT. I HAD A CHANCE TO LOOK AT A FEW OF THOSE WHILE YOU WERE SPEAKING. IT IS FANTASTIC AND A TOOLKIT ANYONE CAN TAKE AND USE LOCALLY. SO, IT IS REALLY WONDERFUL. THANK YOU SO MUCH. >> THANK YOU FOR THAT COMMENT THAT MEANS A LOT AND IS GREAT TO HEAR IT IS USEFUL AND PRACTICAL AND I APPRECIATE THAT. THANK YOU. >> I AGREE. PATTY? >> THANK YOU VERY MUCH. THANK YOU SO MUCH, JEANE FOR THE PRESENTATION. I WANTED TO ASK YOU IF THE SURGEON GENERAL'S OFFICE HAS ADVICE FOR THOSE THAT HAVE LARGE WORKFORCES AND WE HAVE 170 PEOPLE HERE TO LOOK AT HOW TO EVALUATE THE IMPACT OF THIS ENGAGING WITH THE WORKPLACE WELLNESS ACTIVITIES AND IN PARTICULAR IF THERE WILL BE OPPORTUNITIES TO UNDERSTAND HOW WE TRACED THE ACCESS TO RESOURCES TO REDUCE TURNOVER AND IMPROVED PRESENTISM AND MORALE AND SUCH. THANK YOU. >> THANK YOU DR. BRENNAN, EXCELLENT AND WONDERFUL QUESTION AND SOMETHING THAT WE ARE STILL THINKING ABOUT. MAYBE I CAN JUST SHARE TWO OPPORTUNITIES. ONE, WE RECENTLY HAVE BEEN ENGAGING AS PART OF THE PRESIDENTIAL MANAGEMENT AGENDA NUMBER STRATEGIC PRIORITY NO. 3 IS ON WORKPLACE AND WORKER HEALTH. AND IT IS IMPROVING WORKPLACE WELLBEING. THAT GROUP THAT JUST LAUNCHED. IT IS A COMMUNITY OF PRACTICE GROUP, WORKING GROUP THAT I WILL SAY IS NOT A TASKFORCE BUT THEY ARE ACTUALLY LOOKING AT VARIOUS -- YOU KNOW, IT IS INTERESTING. IT IS A NUMBER OF -- IT IS -- IT IS NOT ONLY SENIOR LEADERSHIP BUT IT IS FROM US AGENCIES FROM NOA TO OPM TO VA. SO, ALSO LARGE AGENCIES AND NASA TO CDC AND SO THIS IS DEFINITELY, I THINK, IT COULD BE AN INTERESTING OPPORTUNITY TO LOOK AT SOME OF THE METRICS THAT YOU -- THAT YOU MENTIONED AND IN PARTICULAR TURNOVER ENGAGEMENT BECAUSE THE -- BECAUSE OF THE LOW ENGAGEMENT ACROSS SO MANY OF THE AGENCIES THAT WE ARE EVEN SEEING PERSIST INCLUDING AMONG YOUNG WORKERS AND AMONG WOMEN AND WHAT IS HAPPENING THERE AND SHARE IT JUST AS AN OPPORTUNITY. I THINK IT WILL BE A REALLY PIVOTAL WORKING GROUP. GROUP ITSELF IS NOT ONLY SENIOR LEADERSHIP. THERE IS WORKPLACE, WELLNESS AND BENEFITS COORDINATORS THAT HAVE JOINED AND ALSO, YOU KNOW, ARE OFFICERS FROM HUMAN RESOURCES AND ARE ALSO SPECIALISTS FROM -- AND OFFICERS FROM DEIA. IT IS A VERY WONDERFUL AND DIVERSE GROUP THAT HAS POWER THAT CAN BE HARNESSED FOR LOOKING AT SOME OF THE METRICS AND FIGURING OUT WHAT IS AND WHAT IS NOT WORKING OUTSIDE OF EVEN SOME OF THE METRICS LIKE THE FEDS EVERY YEAR AND ARE THERE OTHER DIFFERENT TYPES OF SURVEYS AND OPPORTUNITIES? YOU KNOW, IN ADDITION TO POLL SERVICE THAT COULD BE HARNESSED IS ONE THAT COMES TO MIND. THE OTHER IS JUST MAYBE IT IS AN INTERESTING QUESTION THAT WE HAVE -- WE ARE ALWAYS TRYING TO TACKLE TO INTERNALLY IS TRYING TO FIGURE OUT HOW DO WE BEST EVALUATE THE IMPACT OF OUR PRODUCTS. SO, I WELCOME OTHER IDEAS TO AND MAYBE FUTURE CONVERSATIONS THAT WE CAN HAVE ON THIS. I THINK THAT SOMETHING WE ARE CONSTANTLY TRYING TO DO IS, YOU KNOW, ENGAGE THE PUBLIC AND ALSO THE INTERAGENCY ON WHAT THIS EVALUATION LOOKS LIKE. I WILL SHARE THESE ARE TWO OPPORTUNITIES. ONE IS COMMUNITY OF PRACTICE AND THERE IS A FUTURE OPPORTUNITY FOR US TO CONSIDER ON HOW WE CAN BETTER SIFT MATIZE THIS. >> THANKS. WE HAVE AN EVALUATION SPECIALIST ON OUR TEAM NOW THAT WILL BE HELPFUL FOR THIS ALSO. >> WE ARE ABOUT OUT OF TIME. IF JERRY AND LORDISE TWO QUICK QUESTIONS. >> I WILL TURN MINE INTO A BRIEF QUICK COMMENT AND THANK YOU FOR THE GREAT PRESENTATION AND ON ISSUES OF HEALTH INFORMATION ACCESS WE AS A NATIONAL LIBRARY WOULD LIKE TO FIND WAYS TO WORK MORE COLLABORATIVELY WITH YOU AND FIND WAYS TO DISSEMINATE UNDERSTANDING YOU HAVE AND ALSO AS YOU PUT TOGETHER TOOLKITS ON CONFRONTING HEALTH AND MISINFORMATION AND LOOK FORWARD TO OPPORTUNITY TO ENGAGE WITH YOU MORE ON THOSE ISSUES AND YIELDING REST OF THE TIME TO LORDIS. PLEASE. >> THANK YOU. GOOD MORNING AND THANKS FOR YOUR PRESENTATION. IT WAS VERY COMPREHENSIVE. KEPT TRYING TO FIND HOLES IN IT AND COULDN'T. I HAVE A QUESTION. DO YOU HAVE SPECIFIC MATERIALS IN THERE AND REFERRALS TO CULTURALLY AND LANGUAGE-BASED MENTAL HEALTH SERVICES ACROSS THE COUNTRY AND HOW ARE THEY DISTRIBUTED ACROSS THE COUNTRY AND NE IN EQUITABLE WAYS. >> THANK YOU. >> GARBANAT. BY : >> THANK YOU. IT IS ON MY SCREEN. I APPRECIATE YOUR COMMENTS AND SUPPORT AND COLLABORATION TO YOUR QUESTION. I DEFINITELY -- YOU KNOW, THERE ARE SPECIFIC RESOURCES OR DATA AND THAT DISAGGREGATES LOTS OF RESOURCES IN TERMS OF CULTURAL COMPETENCY THAT IS TAILORED OR AVAILABLE. WE DEFINITELY, YOU KNOW, TO YOUR QUESTION, I THINK IT IS AN INCREDIBLE OPPORTUNITY AND IF THERE IS A WAY THAT WE CAN HELP TO SUPPORT SOME EFFORT OR INITIATIVE THERE. WE DID TALK ABOUT THIS WITH OFFICE OF HEALTH AND THIS CAME UP A NUMBER OF TIMES TOO WITH INDIAN HEALTH SERVICES EVEN. I WELCOME THIS OPPORTUNITY TO LOOK AT THAT. MAYBE WHAT I CAN DO IS SHARE THIS COMMUNITY OF PRACTICE WORKING GROUP WITH YOU. IF ANYONE IS INTERESTED IN JOINING THAT, IT WOULD BE AN INCREDIBLE INITIATIVE OR OPPORTUNITY TO TAKE FORWARD OR LOOK AT THAT DATA FURTHER AND TO ALSO SHARE WHETHER IT IS A TOOLKIT THAT IS NEEDED THERE. TO -- TO HELP AS A RESOURCE IN THE FUTURE. >> THANK YOU. >> GREAT. THANK YOU SO MUCH. IT WAS A WONDERFUL PRESENTATION AND WONDERFUL SET OF RESOURCES. THANK YOU FOR JOINING US. I WILL MOVE US TO OUR NEXT ITEM WHICH IS APPROVAL OF THE SEPTEMBER 2022 MINUTES. MOTION, CAN I HAVE A MOTION TO APPROVE THE MINUTES? BOARD? >> SO MOVED. >> MOVED. >> GREAT. SECOND? >> SECOND. >> SECOND IF I WAS MOVED. >> YUP. TERRIFIC. THANK 4 THAT. NEXT ON AGENDA IS FUTURE MEETING DATES. MOST OF WHICH ARE CONFIRMED WITH A PROPOSED DATE FOR THE 2025 FEBRUARY MEETING. I'LL NOTE THAT MAY 9TH MEETING HAS BEEN CHANGED TO A ONE-DAY VIRTUAL MEETING AND THINK THAT THE PLAN IS NOW SEPTEMBER WILL BE THE TIME WE TRY TO ALL GATHER TOGETHER AGAIN. IT IS AFTER MY TERM IS UP. MIGHT ASK YOU IN PERSON AND EVERYONE MAKE NOTE OF FUTURE MEETING DATES AS WELL AS OTHER RELEVANT DATES ON THE AGENDA. WITH THAT, WE WILL MOVE TO OUR REPORT FROM LMN DIRECTOR, PATTI. GLAD YOU ARE RECEIVING INFORMATION AND TRAVEL LOGISTICS AND OTHERS AND PLANNING THAT FOR A DAY AND A HALF MEETING SPREAD IN THE TWO DAYS HERE IN LOVELY BETHESDA, MARYLAND AND ASKING YOU TO KEEP SCREEN UP WITH THUMBNAILS ON IT AND HAPPEN TO HAVE BOARD MEMBERS HERE TODAY AND TAKING A MINUTE TO READ NAMES OUT LOUD TO HEAR OUR PRESENTATION AND GET TO BE AND SEE EACH OTHER BRIEFLY. PHILIP WALKER ANDY CLARK. JENNY LUCA. NANCY SMIDER LORDES -- JOE FRANCIS TOM KAN TALENA. PAUL WEBSTER, MICHELLE AND LAURA MAGIO. FRIENDS OF NATIONAL LIBRARY OF MEDICINE IS WITH US AND NLM STAFF AND WELCOME ALL OF YOU AND WE CAN GO TO THE PRESENTATION AND GIVE REMARKS TO DIRECTOR OF NATIO NATIONAL LIBRARY OF MEDICINE. GOOD MORNING. I'M DELIGHTED TO BE GREETING BOARD OF REGENTS. YOU ARE AN INCREDIBLE RESOURCE TO NATIONAL LIBRARY OF MEDICINE AND GUIDANCE AND FEEDBACK AND CONNECTIONS TO OUTSIDE WORLD ARE CRITICAL TO OUR FUNCTION AND ABILITY TO LEAD THE FUTURE AND GRATEFUL TO LEADERSHIP TEAM AND VIRTUALLY AND YOU SEE PICTURES ON THE SCREEN AND WE LAST MET AND HAVE A COUPLE APPOINTMENTS GOING INTO DEPTH IN THEM IN A FEW MINUTES LISA FEDERROR OFFICE OF STRATEGIC INITIATIVES AND POINTING ACTIVE DEPUTY DIRECTOR FOR OPERATIONS AND INNOVATION AND ACTING DIRECTOR OF OSIS AND GRATEFUL FOR ACTING ROLES AND PAST COUPLE MONTHS STEVE SHERRI PERMANENT DIRECTOR AFTER EXHAUST AND EXTENSIVE NATIONAL SEARCH AND DELIGHTED ABOUT THAT. NATIONAL LIBRARY OF MEDICINE IS GUIDED BY THREE PILLARS IDENTIFIED IN THE STRATEGIC PLAN GENERATED IN 27 TAENADVANCING DISCOVERY AND HEALTH THROUGH DATA DRIVEN RESEARCH REACHING PEOPLE TO ENHANCE DISSEMINATING ENGAGEMENT AND BUILDING WORKFORCE FOR DATA DRIVEN RESEARCH AND HEALTH AND TEAMS CONTINUE TO ADVANCE IN THREE AREAS SOMETIMES DISCOVERING NEW OPPORTUNITIES WE DIDN'T ENVISION IN 2017 AND WORKING HARD ON NAUGHTY PROBLEMS WE REALIZED OVER TIME. IN LAST FEW YEARS WE USE POWER OF VIDEO TO INTRODUCE TO RESEARCHERS AND SERVICES AND WORK FORCE HERE AND DELIGHTED TOED TO BRING FORWARD A VIDEO THAT IS PRODUCED BY OUR OFFICE OF COMMUNICATIONS AND PUBLIC LIAISON AND PROGRAM HERE AT NLM AND COULD WE SHOW THE VIDEO? IT IS ABOUT FOUR MINUTES. >> PAPER IS ABOUT MESS OF DISCOVERING NOVEL RNA VIRUSES IN TRANSCRIPTOMIC DATA AND METATRANSCRIPTOMIC DATA IS SIMPLY SEQUENCES OF RNA MOLECULES COULD BE ISOLATED FROM A PARTICULAR ENVIRONMENT AND IN THIS WAY YOU GET AN UNBIASED SAMPLE OF DIVERSITY OF VIRUSES AND THIS REALLY OPENS UP A PERSPECTIVE COMPLETELY UNAVAILABLE BEFORE. YOU DISCOVER MANY NOVEL GROUPS OF VIRUSES THAT HAVE NOT BEEN SEEN BY ANYONE BEFORE. THE. SIMPLY BECAUSE HOSTS ARE UNKNOWN AND [INDISCERNIBLE] AND ACCORDINGLY BEFORE THE DEVELOPMENT OF THE NEW TYPE OF [INDISCERNIBLE] SIMPLY [INDISCERNIBLE]. >> WE ARE VERY CONCERNED HERE ABOUT -- AT NLM ABOUT PURITY OF SEQUENCE SAMPLES WE GET IN. WONDER IF YOU ARE ABLE TO FIND LEVEL OF PURITY YOU NEED TO UNDERSTAND DIFFERENT PATHWAYS? >> PURITY IS NOT A SIMPLE CONCEPT PURITY IN RESPECT TO WHAT? WAY WE WORK HERE IS IN A SENSE CONSERVED. BECAUSE WE HAVE CERTAIN SIGNATURES AND CERTAIN TAGS. >> OKAY. >> THAT WE KNOW ARE ASSOCIATED WITH THESE TYPES OF VIRUSES AND DETECT THOSE AS BEST AS WE CAN. SO, WE TRY TO PROVIDE HIGH SPECIFICITY. WITH HIGH SENSITIVITY. IF SOMETHING IS ABSOLUTELY NOVEL, UNRELATED TO ANYTHING THAT HAS BEEN KNOWN UNTIL NOW, WE WOULD THEN MISS IT. OTHER APPROACHES ARE NECESSARY AND IN THIS CASE THE ISSUE OF PURITY AND CONVERSELY CONTAMINATION AND IT IS VERY LARGE. AT LEAST IN THIS PARTICULAR PROJECT I DON'T THINK SO. I THINK THAT RATE OF FALSE POSITIVE CLOSE TO ZERO. >> THIS PARTICULAR ARTICLE IN CELL INTRODUCED IDEA THAT THERE IS MORE DIVERSITY AND MORE SIMILARITY. HOW DO YOU BALANCE DIVERSITY AND SIMILARITY? MORE THAN WHAT WE HAD EXPECTED AND SAME SIMILAR PATTERNS WE HADN'T EXPECTED. >> CONSIDERED MUCH MORE DIVERSITY THAN WE EXPECTED. AT THE SAME TIME, IN PREVIOUS ANALYSIS, ON A CONSIDERABLY SMALLER SCALE, WE HAVE, I BELIEVE, ESTABLISHED OVERALL STRUCTURE OF THIS PART OF THE WORLD OF VIRUSES AND THIS PART OF THE BIOSPHERE, IF WILL YOU. I HAVE NOT IDENTIFIED LARGE DIVISION SO CALLED [INDISCERN [INDISCERNIBLE] [VIDEO CAPTIONED]. >> LET'S TALK ABOUT RESOURCES AND THINGS MADE AVAILABLE TO THE WORLD THROUGH THE NATIONAL LIBRARY OF MEDICINE. WE ARE AT THIS TIME PREPARING FOR THE FUTURE THAT IS COMING UPON US. 200 NL ANNIVERSARY IS 2036, 13 YEARS AWAY AND LIBRARY OPERATIONS TEAM WORKED TOGETHER TO BUILD A LONG-RANGE PLAN FOR NEXT 15 YEARS ENGAGING OVER 10 PEOPLE ACROSS LIBRARY OF OPERATIONS GROUP. YOU SEE FRAMEWORK ON LEFT-HAND SIDE TO COLLECT CONNECT AND CURE RATE RESOURCES. 2023 IS YEAR OF OPEN SCIENCE CELEBRATED ACROSS US AND NLM IS GLOBAL LEADER IN OPEN SCIENCE THANKS TO VISION OF MANY STAFF HERE AND LONGSTANDING SERVICES PUB MED AND OTHER REPOSITORIES AND HARD WORK SUPPORTING LAUNCH OF NEW NATIONAL INSTITUTES OF HEALTH DATA MANAGEMENT AND INSURANCE POLICY THAT IS ACTIVE ON JANUARY 2 FIF NL AND IF YOU HAVE QUESTIONS JERRY IS HAPPY TO ANSWER THEM. WORKING TO PROMOTE NATIONAL LIBRARY OF MEDICINE. WE WHETHER BE GUIDED BY EXTERNAL GROUP PROVIDING OWN EXPERTISE AND RECEIVE RESPONSES IN PROCESS OF COMPETITIVE PROCUREMENT TO MAKE THIS AVAILABLE IF IN NEXT COUPLE MONTHS AND WEEKS AVAILABLE BY FIRST OF MARCH TO LEADERSHIP AND THROUGHOUT ENTIRE OPERATION AND HOPE TO HAVE WHOLE TEAM JOIN WITH US IN VIEW OF INNOVATION AS MECHANISM TO ENSURE NIMBLE SECURE HIGH INTEGRITY IN NLM RESOURCES AND. OUR OSIS TEAM DEVELOPED VULNERABILITY DISCLOESHURE POLICY FOR MONITORING AND REPORTING DASHBOARD FOR NIH FACING WEBSITES EMBRACED AT HHS LEVEL AND CONGRATULATIONS TO THE TEAM FOR THAT LEADERSHIP. INVESTING IN NEW APPROACHES TO SCIENTIFIC COMMUNICATION LAUNCHING SECOND PHASE OF PREPRINT PILOT MAKING PREPRINTS OF NIH FUNDED RESEARCH AVAILABLE IN PUB MED CENTRAL PARTNERING WITH OFFICE OF ENGAGEMENT AND TRAINING TO SPONSOR TUTORIALS ACROSS NIH AND COUNTRY TO UNDERSTAND WHAT PREPRINTS CAN DO FOR SCIENTIFIC COMMUNICATION AND PUB MED IMPROVED PROXIMITY RESEARCH FINDING RELATED TERMS AND ARTICLES RELATED TO INDIVIDUAL RESEARCHER AND PUB MED UTILITIES API MAKING IT EASIER FOR SERVICES TO BE IMBEDDED IN RESOURCES AND DEVELOPING MANY UPDATES WITH MESH AND DEVELOPED COLLABORATIVE RELATIONSHIP WITH LIBRARY SCIENCE COMMUNITY MAKING SURE THEY ARE ABLE TO UNDERSTAND WHAT DEVELOPMENTS AND CHANGES IN VOCABULARY SCIENCE WE NEED TO INCLUDE IN MESH AND SUPPORTED NIH DETAILING CIO AND FORMAL DIRECTOR OF OFFICE OF COMPUTER AND COMMUNICATION SYSTEMS SERVING NOW AS ACTING DIRECTOR FOR CENTER OF INFORMATION AND TECHNOLOGY ONE OF 27 CENTERS AND INSTITUTES OF NIH COMPUTING SYSTEM AND STRIDES INITIATIVES AND MANY DESKTOP SERVICES AND ALONG THE WAY WE MADE ADVANCES IN RESEARCH WITHIN INTRAMURAL PROGRAM DEMONSTRATING ALPHA BAYED PROTEIN STRUCTURED PREDICTION PROCESS ALPHA FOLD 2 WOULD BE APPROVED ATTENDING PROTEIN BIOPHYSICS PROTEIN STRUCTURE AS A SINGLE POINT AND ENSEMBLE AND ANTANI AND TEAM AT NCI DEVELOPED EVALUATION OF SURGICICAL IMAGES AND HIS TECHNIQUE WILL IMPROVE ACCURACY OF THAT APPROACH. [INDISCERNIBLE] AND COLLEAGUES REPORTED NATURE SCIENTIFIC DATA DEMONSTRATION SINGLE CORPUS REPRESENTING AMALGAM OF POSSIBLE COVID-19 DATA SOURCES CAN PERFORM WELL RESPONDING TO QUESTIONS FROM HIGHLY DISPARATE SERVICES AND SHOWING EXTENSION TO MAKE IT ROBUST AND VISIBLE WITHIN EXTRAMURAL RESEARCH PROGRAM WE RELEASED COUPLE NEW FUNDING OPPORTUNITIES UNDER PARENT PROJECT GRANTING GENERAL GRANTS RESEARCH IN BIOINFORMATICS AND DATA SCIENCE AND SPECIAL INTEREST TO STIMULATE COMPUTATIONAL AND STATISTICAL METHODS TO ENHANCE DISCOVERY OF DATA AND ALIGNED IN NIH SUPPORTING EARLY STAGE INVESTIGATORS AND CONCERNED THAT TRAJECTORY TOWARDS INDEPENDENT RESEARCH IS TAKING LONG AND MAKING COMMITMENT TO REACH OUT TO EARLY STAGE INVESTIGATORS 20% OF AWARDEES IN PAST TWO YEARS ARE EARLY STAGE INVESTIGATORS AND WE ARE SUPPORTING NIH RESEARCH PROGRAM SCIENCE TV WIDELY USE TODAY HELP PEOPLE CREATE EFFECTIVE BIBLIOGRAPHIES FOR RESEARCH PROJECTS HAS NEW INTERFACE AND EASIER TO USE FOR GRANT SUBMISSION. FOR ALL OF US TEAM TO REACH OUT TO 5,000 INDIVIDUALS ANDN PARTNERS AND LIBRARIES AND PARTNERSHIP WITH ALL OF US PROGRAM WORKING WITH NATIONAL LIBRARY OF MEDICINE IS PRODUCTIVE AND HELPFUL IN THIS AREA. WITH RESPECT TO RESOURCES DIGITAL REPOSITORY NOW INCLUDES OVER 300ROL HISTORIES AND VIVIAN THOMPSON AND RECORDING OF FLORENCE NIGHTINGALE EXCITING FOR ME TO HEAR. COLLABORATION OF CMS UPDATED VALUE AUTHORITY TO IMPROVE ACCESS TO FIRE MEASURES AND QUALITY BASED MEASURES IMPROVING USEFULNESS OF RESOURCES FOR CARE IN COUNTRY AND IMPORTANT TO SERVICE AND CLINICAL PRACTICE AND WITHIN ACTIVE INITIATIVE THAT IS PUBLIC PRIVATE PARTNERSHIP LAUNCHING THROUGH COVID PANDEMIC TO IDENTIFY QUICKLY STRATEGIES FOR BOTH DIAGNOSING AND TREATING VARIANT AND RELEASING VARIANT CALLING PIPELINE PROVIDING ACCESS TO FILES IN CLOUD MAKING IT EASIER FOR THEM TO BE USED MORE WIDELY AND SEQUENCE ARCHIVE LARGEST PUBLICLY AVAILABLE BIOLOGICAL DATA SET IN THE WORLD HAS IMPROVED PUBLIC ENGAGEMENT POSTING INFORMATION FOR STREAMLINING DISSEMINATION TRANSITIONING TO SRA LIGHT FORMAT HELPING SAVE STORAGE SPACE FOR EVERY GROWING RAPIDLY GROWING IMPORTANT RESOURCE FINDING WAYS TO MAKE IT EFFICIENT AND CLIFTI REPORTING ON THIS LATER FOR THE COLLECTIONS GROUP AND NIH COMPARATIVE GENOME RESOURCES OR CGR IS INITIATIVE TO BRING TOGETHER NLMS HOUSED AND STEWARDED GENOMIC RESOURCES WITH COMMUNITY TOOLS SUCH AS ANNOTATION AND AUGMENTATION AND EXPANDED TOOLS TO USE IN CGR AND GENOME VIEWER AS WELL AS EXPANDING BLAST PROGRAM ALLOWING FOR BETTER ACCESS AND EVALUATION. WE CONTINUE TO WORK HARD ON MAKING NIH A GREAT PLACE TO WORK IN NLM APPROACH TO RACIAL AND ETHNIC EQUITY PLAN IS PRESENTED YESTERDAY ACROSS THE CAMPUS. NLM IS UNUSUAL USING BOTTOM'S UP GRASSROOT APPROACH TO ENGAGE STAFF IN CREATING THIS PLAN. PLAN CONSTRUCTED IN PAST APRIL TO AUGUST AND WE HAVE NOW THREE IMPLEMENTATION TEAMS IN PLACE TO BEGIN TO REALLY TRANSFORM THE NLM TO BE A GREAT PLACE TO WORK TO PREPARE OUR STAFF TO BE ABLE TO PARTICIPATE IN THIS. WE HAVE PROVIDED ACCESS TO SPECIALIZED TRAINING INCLUDING GROUNDWATER TRAINING AS WELL AS RACE AHEAD. THESE ARE TWO BROAD NATIONAL PROGRAMS THAT STAFF IS NOW ACCESSING TRAINING FOR IT TO BE ABLE TO HELP GUIDE US INTO OUR FUTURE. IMPLEMENTATION PLANNING COMMITTEE MEMBERS ARE IN FRONT OF YOU AND TARGETING THREE ACTIVITIES THIS YEAR. ESTABLISHING MANAGEMENT COMMITMENT PLAN DEMONSTRATING MANAGEMENT'S COMMITMENT TO RACIAL/ETHNIC EQUITY ACROSS NLM AND RETENTION AND PROMOTIONS TO ENSURE NO STRUCTURAL BARRIERS THAT EVERY PERSON AT NLM REACHES TOP PEEK OF PERFORMANCE AND INCREASING PARTICIPATION IN WORKPLACE CULTURAL ASSESSMENTS WE WANT TO UNDERSTAND AND LISTEN AND HEAR HOW STAFF ARE EXPERIENCING THEIR EFFORTS HERE. NATIONAL INSTITUTES OF HEALTH PLAYED A KEY ROLE IN THE COVID EXPERIENCE PAUSING TO TAKE A MOMENT TO CALL ATTENTION TO REPORT JUST RELEASED IN SCIENCE DEMONSTRATING KEYMILESTONES THAT NIH'S CONTRIBUTIONS TO VACCINES THERAPEUTIC DIAGNOSTICS AND PUBLIC OUTREACH AND WHAT YOU SEE ON THE SCREEN IN FRONT OF YOU COVERS FIRST THREE YEAR OF THE PANDEMIC AND DEMONSTRATES HOW QUICKLY WE WERE ABLE TO MAKE VIRUS SEQUENCE AVAILABLE ON JANUARY 13TH THAT IS NLM CONTRIBUTION MAKING SEQUENCE PUBLICLY AVAILABLE. N WILL. M PARTICIPATED IN NUMBER OF STEPS ALONG WAY INCLUDING ESTABLISHING AND SUPPORTING COMMUNICATIONS THROUGH SEAL INITIATIVE ASSISTING WITH TECHNICAL STAFF AND SUPPORTING SOME DIAGNOSTIC AND THERAPEUTIC RESEARCH AND USING MANY GENETIC DATA BANKS TO HELP ADVANCE UNDERSTANDING AND DETECTION AND VARIANTS OF THE TERRIBLE VIRUS. THIS ARTICLE IS AVAILABLE ONLINE AND PUBLICLY AVAILABLE. I ENCOURAGE YOU TO TAKE A LOOK AT IT. WE ARE PROUD ACROSS NIH AND ROLE THAT NLM HAS PLAYED IN ADVANCING THIS. I HAVE GREAT NEWS THAT IS ABOUT THE BUDGET. THIS IS ONE OF THE FIRST TIMES I'M ABLE TO COME TO YOU SAYING WE HAVE A BUDGET IN OUR FEBRUARY MEETING AND HAVEN'T HAD APPROPRIATIONS THIS EARLY IN A WHILE. WE ARE VERY PLEASE THE. ON SCREEN IN FRONT OF YOU NIH AND NLM'S COMPARISON. IN FISCAL 20, BASE BUDGET IS 456 MILLION AND 10 MILLION SUPPLEMENT THAT IS BROUGHT ON THAT YEAR FROM THE CARES ACT TO BE ABLE TO MAKE RESOURCES BETTER AVAILABLE. IN FISCAL 21, BUDGET RENTED 462 MILLION. FISCAL 22 TO 479 MILLION AND THIS YEAR WE ARE AT $497 MILLION BUDGET. NIH BUDGET HAS BEEN INCREASING ALSO. ONE IMPORTANT THING ABOUT OUR BUDGET THIS YEAR IS STAFF WILL GET 4.41% RAISE INCREASE FOR FEDERAL STAFF AND WE HAVE COME TO END OF 3-YEAR PROCESS OF BUILDING AND SUPPORTING RENOVATIONS AT NLM. OUR BUDGET IS A BIT MORE FLEXIBLE THIS YEAR. I HAVE BEEN WORKING WITH LEADERSHIP OVER THE LAST FEW WEEKS TO HAVE THEM IDENTIFY BOLD NEW ACTIVITIES THEY WANT TO UNDERTAKE IN THE NEXT YEAR. I HAVE TO SAY THAT WE ARE VERY PLEASED AND MIKE AND I HAVE REVIEWED THESE IN DEPTH SEEING 25 PROPOSALS VERY EXCITING LOOKING AT 35 MILLION WE HAVE 20 MILLION OR SO TO PROVIDE. THAT IS A LITTLE JUDGING HERE AND VERY EXCITING TO HAVE THIS KIND OF FLEXIBILITY. BUDGET AUTHORITY HAS INCREASED OVER THE YEARS AND I WOULD LIKE TO RECOGNIZE THE WORK THAT BUDGET OFFICE DOES MAKING IT POSSIBLE FOR US TO EFFECTIVELY SPEND THESE FUNDS AND WOULD LIKE TO THANK PARTICULARLY NCBI AND OSIS. MANAGING A BUDGET OF THIS SIZE IN THE FEDERAL CONTEXT REQUIRING US TO EXPEND FUNDS IN GIVEN YEAR REQUIRES BUDGET FLEX IBLTH AND GROUPS HAVE BEEN HELPFUL AND NIMBLE SHOWING US TO EXPEND FUNDS GIVEN AND DON'T RETURN ANYTHING TO THE TREASURY TOO FAST. I HAVE PERSONAL UPDATES TO DISCUSS. I'M VERY EXCITED ABOUT THEM ALSO. AS I INDICATED IN OUR OPENING TIME THAT NOEFT HAS TAKEN ON ROLE OF ACTING DEPUTY DIRECTOR FOR OPERATIONS AND INNOVATION AND LAST COUPLE YEARS REALIZE WE NEEDED TO HAVE SPECIALIZED ATTENTION AT DEPUTY DIRECTOR LEVEL FOR TWO KEY AIR YAZ OPERATIONS AND INNOVATIONS PORTFOLIO AND POLICY AND EXTERNAL AFFAIRS AND TWO LEADERS IN THESE POSITIONS HAS EVEN IN THE LAST THREE MONTHS REALLY MADE A MAJOR STEP FORWARD FOR NLM TO BE ABLE TO MORE QUICKLY RESPOND TO MORE CAREFULLY SHEPHERDING AND STEWARDING FEDERAL DOLLAR MAKING SURE INNOVATION BECOMES A PART OF OUR EVERY-DAY WORK AND IN ADDITION WE HAVE GREATER STRENGTH AT LEADERSHIP LEVEL TO SUPPORT TRANSITIONS HAPPENING ACROSS NLM AND STEVE SHERRI WELL KNOWN AT NLM MANY YEARS PERMANENT DIRECTOR OF NCBI NATIONAL SEARCH LEADING US TO CONCLUDE DR. SHERRI WAS BEST CANDIDATE WAS VERY EXTENSIVE AND BROUGHT REALLY INTERESTING THOUGHTS TO US ABOUT HOW NCBI'S INITIATIVES CAN BE BETTER FOCUSED AND CAN APPROACH NEW OPPORTUNITIES WHERE WE ARE PARTICULARLY PLEASE THE WITH DR. SHERRI COMING ON BOARD BRINGING TRUST OF STAFF AND VISION OF FUTURE COMING TOGETHER. I'M PARTICULARLY PLEASED WITH ONE OF KEY NEW INITIATIVES TO POSITION THE NCBI AND NLM AND NIH OVERALL IN INTERNATIONAL CONVERSATION ABOUT GENOMIC PUBLIC HEALTH SEQUENCE -- BASED SURVEILLANCE AND ANLT TO SERVE NEEDS OF COUNTRIES AROUND THE WORLD DIFFERING IN ABILITY TO ACCESS AND SHARE DATA. LOOEZA WHO HAS BEEN WITH US FOR FIVE YEARS IS ACTING DIRECTOR OF OSI AND SHE IS A LIBRARY OF SCIENTIST WITH EXPERTISE AND DATA SCIENCE AND VISUALIZATION AND HAS ESTABLISHED HERSELF ACROSS NIH AS A LEADER AND THINKER AND TRAINING IN THESE AREAS AND HER ROLE IN OSI AND INTERIM PERIOD IS BEING CHARACTERIZE THE ALREADY BY STRONG COMMITMENTS TO EVALUATION AND ENGAGEMENT IN OUR LEADERSHIP IN EVIDENCE-BASED DECISION MAKING. WEI MA YOU ARE HERE 15 YEARS NOW? >> 25. >> OKAY. 25 YEARS AND IN OSIS. SHE HAS BEEN RESPONSIBLE FOR MANY DIVISIONS AND WHEN SUSAN OR I MOVE TO THE RESPONSIBILITY OF ACTING DIRECTOR OF CIT, WEI MA BECAME DIRECTOR OF OCCS AND UNINTERRUPTED POWER SOURCES MAKING SURE CLOUD AND SECURITY STRATEGY IS WELL IN PLACE AND IMPORTANTLY IS OVERSEEING WHAT WE HOPE WILL BE FINISHING AND STRENGTHENING OF OUR DATA CENTER. EXCUSE ME. IN TERMS OF NEW HIRES AND PROMOTION. I'M HAPPY TO INTRODUCE YOU TO THREE PEOPLE. CHRISTINA HAS JOINED OFFICE OF THE DIRECTOR AS A SENIOR MANAGEMENT ANALYST. -- WORKFORCE DEVELOPMENT SPECIALIST IN OFFICE OF THE DIRECTOR AND IN SHORT TENURE HERE SHE REALLY ACCELERATED BRINGING CONFIDENCE TO HIRING PROCESS VERY MUCH NEEDED IN FEDERAL SECTOR AND HIRING HAS BEEN CHALLENGING AND IS HELPFUL IN BETTER POSITIONS AND PACKAGES AND MAKING SURE GOING THROUGH HR IN TIMELY FASHION AND MAKING SURE HR IS RESPONSIVE TO US AND PATRICK HAS BEEN PROMOTE TODAY SECTION HEAD. WE ARE HAPPY TO SEE HIM IN PROGRESSIVELY INCREASING RESPONSIBLE ROLE. SAYING GOODBYE TO DENNIS BENSON RETIRING DECEMBER 30 YENL AFTER 43 YEARS AT WORK OF NLM AND NIH. DENNIS WAS INSTRUMENTAL AS PART OF A GARAGE BIN THAT LAUNCHED NCBI BACK IN THE 80S. HAS BEEN THE KNOWLEDGE BASE AND VOICE OF REASON AND VOICE OF VISION IN MAKING SURE WE TAKE THIS VERY COMPLEX $150 MILLION OPERATION THROUGH PACES EVERY YEAR KEEPING CONTRACTS SAFE AND HUMAN RESOURCES ACTIVITIES ON TARGET. DENNIS WAS A GREAT AND GOOD COLLEAGUE TO HAVE. HIS PRESENCE IN THE HALLWAY IS MISSED AS MUCH AS HIS MIND AND NONL AND UNDERSTAND THEY HAVE HIM ON SPEED DIAL FOR A COUPLE CRITICAL PIECES. TURNING TO TERESA ESCOBAN WITH A POLICY AND LEGISLATION UPDATE. >> GOOD MORNING, EVERYONE. I WANT TO SHARE POLICY AND LEGISLATIVE UPDATES AS ALWAYS AND ENCOURAGE YOU TO READ UPDATE WE PROVIDE IN DETAIL AND IN LENGTH FOR MORE INFORMATION. NEXT SLIDE, PLEASE. THERE HAS BEEN FEDERAL POLICY ACTIVITIES OF INTEREST TO LNM IN LAST FEW MONTHS AND DATA MANAGEMENT AND SHARING POLICY WENT INTO EFFECT ON JANUARY 25TH. AS PATTI ELUDED TO REFLECTS CONSIDERABLE INPUT FROM NLM INCLUDING ON DESIRABLE CHARACTERISTICS OF DATA REPOSITORY THAT WHITE HOUSE OFFICE OF SCIENCE AND TECHNOLOGY POLICY OR OSTP ISSUED LAST SPRING AND MENTIONED IN SEPTEMBER 2022 UPDATE TO YOU. SECOND, NIH IS IN THE PROCESS OF DEVELOPING A PLAN FOR ENHANCING EQUITABLE ACCESS TO RESULTS OF FEDERALLY FUNDED RESEARCH IN RESPONSE TO 2022OSTP POLICY GUIDANCE THAT I MENTIONED ALSO IN SEPTEMBER OF LAST YEAR AND NLM IS VERY ENGAGED IN PARTICULAR SECTIONS RELATED TO PUBLICATION ACCESS AND NIH EXPECTS TO DELIVER A DRAFT PLAN TO OSTP AND WHITE HOUSE OFFICE OF MANAGEMENT AND BUDGET BY END OF FEBRUARY AND DRAFT PLAN MADE AVAILABLE FOR PUBLIC COMMENT AND APPRECIATE BOARD'S ASSISTANCE AND DISSEMINATING IT TO STAKEHOLDERS REMINDING YOU THAT DRAFT PLAN IS A PLAN AND ONCE FINALIZED WILL BE FOLLOWED BY MORE FORMAL POLICY DEVELOPMENT THAT WILL ALSO INCLUDE OPPORTUNITIES FOR PUBLIC AND STAKEHOLDER INPUT AND IN ADDITION I WANTED TO LET YOU KNOW THAT ON JANUARY 12 NL, OFFICE OF NATIONAL COORDINATOR FOR HEALTH INFORMATION TECHNOLOGY RELEASED THE DRAFT OF THE 4 NL VERSION OF US CORDATA FOR INTEROPERABILITY FOR PUBLIC COMMENTS THAT ARE DUE BY APRIL 17TH. WE PLAN TO RELEASE THE FINAL VERSION IN JULY AND DRAFT VERSION 4 INCLUDES NEW DATA CLASS AND A NUMBER OF NEW DATA ELEMENTS AND ONC IS SOLICITING COMMENTS ON ALL OF THEM. REMINDER THAT REQUIREMENT THAT CERTIFIED HEALTH IT DEVELOPERS UPDATE HEALTH IT MODULES TO VERSION ONE OF CDI THAT WENT INTO EFFECT AT END OF LAST YEAR. FINALLY, ON THE POLICY FRONT, NOVEMBER OF LAST YEAR WORLD HEALTH ORGANIZATION RELEASED SET OF PRINCIPLES SHARING PATHOGEN GENOMIC SEQUENCE DATA YOU TO DO PUBLIC HEALTH RESPONSE AND CONTRIBUTE TO DEVELOPMENT DR. SHERI PARTICIPATE IN FORMATIVE WORKSHOP AND SUBMIT COMMENTS WITH MULTIPLE NIH CENTERS AND INSTITUTES AND OFFICES AND CDC COLLABORATORS AND PLEASED TO SEE PRINCIPLES DEMONSTRATE PROGRESS ON INTERNATIONAL CONSENSUS OF VALUE AND PANELO GENERAL SEQUENCE DATA INCLUDING THROUGH SOME REPOSITORIES AT NLM AS WELL AS THOSE THAT ARE OF INTERNATIONAL SEQUENCE DATABASE COLLABORATION PARTNERS. ON LEGISLATIVE FRONT AS PATTI MENTIONED PRESIDENT SIGNED CONSOLIDATED APROPRIATION FOR 2023 FUNDING GOVERNMENT IN SEPTEMBER OF THIS YEAR AND IN ADDITION TO BUDGET INCREASES FOR NIH AND NLM AND ACT ACCOMPANYING EXPLANATORY LANGUAGE CONTAINS SEVERAL ITEMS OF INTEREST TO NLM. IN PARTICULAR I WANT TO HIGHLIGHT THAT LANGUAGE RECOGNIZES IMPORTANCE OF COMMON DATA ELEMENTS AND REPOSITORY THAT NLM DEVELOPS AND MAINTAINS AND ADDITION AS PART OF BROADER LANGUAGE REGARDING NIH FEDERAL ADVISORY COMMITTEES HOUSE EXPRESSES APPRECIATIONS FOR NLM'S EFFORTS TO HAVE INFORMATION ABOUT PROCESS OF BED LINE REVIEWS AND HIGHLIGHTING WITHIN THE CONSOLIDATED APPROPRIATIONS ACT THERE IS PREVENT ENDEMICS ACT THAT YOU MAY RECALL I MENTIONED A DISCUSSION DRAFT OF VERSION OF WHAT IS THEN A BILL THAT WAS RELEASED IN FEBRUARY OF LAST YEAR AND FINAL TEXT OF THE ACT JUST PASSING INCLUDES ACTIVITIES TO IMPROVE PUBLIC HEALTH DATA INCLUDING GENOMIC SEQUENCE DATA USED FOR PUBLIC HEALTH SURVEILLANCE AND USE OF DATA STANDARDS NLM IS NOT SPECIFICALLY CALLED OUT IN ACT WE PLAN TO WORK WITH PARTNERS ACROSS NIH AND NHHS TO WORK ISSUES OF GREAT RELEVANCE TO US ISSUES EXPANDING ACTIVITIES RELATED TO GENOMICS SEQUENCING PATHOGENS AND DEVELOPMENT OF HEALTH DATA STANDARDS AND LAST HIGHLIGHTING APPROPRIATIONS ACT INCLUDES ALSO LANGUAGE WITH REGARDS TO ARPAH FORMALLY ESTABLISHING IT WITHIN NIH AND AUTHORIZING FUTURE BUDGET ALLOCATIONS AND AUTHORIZING OPERATIONS AND FUNCTIONS. PATTY, TURNING IT OVER TO YOU. >> THANK YOU VERY MUCH, TERESA WHO WILL BE HERE FOR QUESTIONS IN THE NEXT PERIOD. I WANT TO CLOSE BY POINTING OUT YOU HAVEN'T SEEN MY BLOG LATELY YOU WILL SEE A NEW PICTURE ON BLOG. CAN'T SAY IT IS THE BEST BUT IS A GOOD PICTURE AND IMPORTANT THIS BLOG SERVED TO BE GREAT COMMUNICATIONS TOOL USED BY NIH AND NLM AND HAVE OVER HALF A MILLION VIEWS OF IT AND CONTINUE TO USE THIS NOW ACROSS THE NIH WHEN SOME IN-DEPTH COMMENTARY NEEDS TO BE PROVIDED AND GETTING CALL FROM NIC DIRECTOR COLLEAGUE TO SAY CAN WE USE YOUR BLOG TO PUT THIS OUT? PLEASED TO GIVE SHOUTOUT TO OCPLT WHO MANAGES A BLOG AND DOES A GREAT JOB WITH IT. AS USUAL, YOU CAN REACH ME BY E-MAIL AND ACTIVE ON TWITTER STAYING WITH TWITTER QUIETLY RIGHT NOW AND STUDYING WAY THAT TWITTER CAN BEST BE USED FOR COMMUNICATING PROPER INFORMATION AND CAREFUL ACKNOWLEDGMENT OF THE CORRECT SOURCES OF HEALTH INFORMATION. THANK YOU. I THINK WE HAVE TIME NOW FOR 15 MINUTES OF QUESTIONS. HEIDI, TURNING IT BACK TO YOU. >> GREAT. THANK YOU FOR THAT UPDATE. MAYBE I WILL START MYSELF AND TURN IT OVER TO THE REST OF THE BOARD AND OTHERS FOR QUESTIONS. YOU MENTIONED PHASE 2 FOR INCORPORATING PREPRINTS INTO THE SERVER AND I MEAN INTO THE PUB MED. WHAT IS PHASE 3? COULD YOU -- YOU KNOW, ELABORATE A LITTLE BIT ON NEXT STAGES AFTER THIS? >> I WILL LET JERRY HANDLE THIS AND STEVE IF HE WANTS TO STEP IN. >> SO, PHASE 2 JUST LAUNCHED; RIGHT? STARTING IN JANUARY. IN FACT, I THINK IT WAS JUST LAST WEEK FIRST BOLUS OF PREPRINTS WAS UPLOADED INTO PUB MED CENTRAL FOR PARTICIPATING SERVERS AND PHASE 3 IS DECIDED BY SUCCESS OF PHASE 2; RIGHT? WE ARE RUNNING THIS PHASE FOR A YEAR. WE WILL BE LOOKING AT A WHOLE RANGE OF FACTORS AND IN TERMS OF SOME OF THE TECHNICAL ABILITIES TO PRESENT AND COLLECT THESE PREPRINTS AND USAGE OF PREPRINTS. WE HAVE BEEN DOING WORK IN PHASE 1 TRYING TO MEASURE HOW THEY EFFECT TRUST IN NLM RESOURCES WHICH IN THE LAST PHASE SEEM TO BOOST TRUST IN THE RESOURCES THAT WE HAD. AGAIN, THAT MIGHT REFLECT FACT THAT FOLKS ARE DOING NIH FUNDED RESEARCH. PHASE 3, IT IS YET TO BE DEVELOPED. WE REALLY WANT TO LET PHASE 2 RUN ITS COURSE AND SEE HOW WELL IT WORKS. THERE IS -- I DON'T KNOW IF WE PUT IN THE BACKGROUND MATERIAL THERE IS A PREPRINT CURRENTLY AVAILABLE SUMMARIZING RESULTS OF PHASE 1 AND I'M HAPPY TO PROVIDE THAT TO THE BOARD. WE HAVEN'T DONE THAT ALREADY. YEAH. I THINK IT IS A VERY GOOD SUMMARY. >> THAT WOULD BE. >> HOW WELL IT WORKED. >> STEVE, I WILL ASK YOU TO SPEAK A LITTLE HOW NCBI IS PREPARING OR MANAGING FOR THIS AND PUB MED CENTRAL IS AN OPERATION THAT IS UNDER NCBI'S PUBLIC FACING SERVICES. STEVE? >> THANK YOU, PATTY. WE ARE STRATEGICALLY SEEING PREPRINTS AS A VALUABLE ASSET AND EXTENSION OF LITERATURE CONNECTION TO DATA AND PART OF THE PITCH AND INTEREST IN THIS IS NOT ONLY JUST RAPID COMMUNICATION OF FINDINGS THAT WE FOUND THAT IN ITSELF IS VERY VALUABLE DURING COVID TRYING TO DO RAPID DISSEMINATION OF NEW INSIGHTS AND PANDEMIC RESPONSE AND FURTHERMORE PROVIDES PLACE TO IMMEDIATELY CONNECT CONTEXT TO DATA FROM DATA SIDE HAVING A PREPRINT OUT THERE CONNECTED TO SUBMISSIONS GIVES IMMEDIATE CONTEXT OF METHODOLOGY AND SAMPLING FRAMES AND CREDIBILITY AND SUITABILITY FOR INQUIRY AND BIG GOAL WE LOOK AT INTERNALLY IN PHASE 2 IS TO UNDERSTAND THAT CONNECTION. CAN WE ACTUALLY SEE USERS FROM PREPRINTS TOUCHING DATA OR FROM DATA GOING TO PREPRINTS? THAT IS HYPOTHESIS WE WANT TO REALLY DRIVE. THIS IS VALUABLE AS A NAVIGATIONAL AND SCIENTIFIC TOOL. THERE WILL BE FINDINGS ABOUT THAT IN THE NEXT ASSESSMENT THAT WE PUBLISH A YEAR FROM NOW THAT JERRY MENTIONED WE WILL DO A CHECK-IN, IN A YEAR AND MIGHT GO TWO YEARS TO SEE WHAT DATA LOOKS LIKE. NOT COMMITTING TO THAT NOW. THERE IS PROBABLY A SOCIOLOGICAL COMPONENT OF THIS AND GETTING WORD OUT TO SUBMIT WILL TAKE SEVERAL MONTHS AND HAVING A MEANINGFUL CORPUS OF DATA TO ANALYZE MIGHT BE 18 MONTHS OF COLLECTION. >> THANK YOU, JERRY. LORDES. >> I RECOMMEND GOOD READING FOR ALL ON THE BOARD. >> YEAH. THIS IS A PREPRINT ABOUT PREPRINT. >> [LAUGHING]. >> LORDES. >> THANK YOU, PATTY, FOR THE GREAT UPDATE. CONGRATULATIONS ALL OF YOU AND YOUR ACCOMPLISHMENTS. THIS IS THE WAY TO GO. I'M WONDERING HOW YOU ARE REACHING OUT TO NEW AND YOUNGER AUDIENCES ON SOCIAL MEDIA? >> LORDES, I HAD DIFFICULTY HEARING YOU. WOULD YOU MIND REPEATING YOUR QUESTION AGAIN? >> SURE. I LOVE THE STORY PART OF IT AND WONDERING HOW YOU ARE REACHING AUDIENCES AND YOUNGER FOLKS TIKTOK AND OTHER PLATFORMS AND SOCIAL MEDIA TO TELL NLM STORY? >> WE ARE NOT USING TIKTOK, I CAN TELL YOU THAT. WE HAVE AN INSTAGRAM POST AND FACEBOOK AND LINKEDIN AND. >> AND TWITTER. >> AND TWITTER. EXCUSE ME. THERE IS -- NIH IS WHAT IS CAREFULLY WATCHING WHAT IS HAPPENING WITH TWITTER TO BE SURE THAT SHOULD DISINFORMATION OR DISPARAGING INFORMATION CONTINUE WE MIGHT NEED TO DIMINISH ENGAGEMENT WITH TWITTER. HAVEN'T MADE PUBLIC DECISIONS ABOUT THAT. WE ARE MINDFUL THERE IS CONCERNS ACROSS THE NATION. WE ARE INCREASING OUR UPTAKE ON USE OF OTHER RESOURCES AND FIRST ONE THAT WAS IDENTIFIED WAS LINKED IN. IT WAS QUICKLY POINTED OUT THERE IS NO ONE UNDER 50 THAT ROUTINELY LOOKS AT LINKED IN UNLESS LOOKING FOR A JOB AND NEED TO FIND OTHER WAYS TO ENGAGE AND WHERE THE INSTAGRAM AND FACEBOOK COME IN. WE ARE OPEN AND HAVE LOOKED AT NIH AS A WHOLE HAS LOOKED AT OTHER PLATFORMS AND DECIDED THEY ARE NOT NECESSARILY RIGHT NOW. ONE THING I THINK THAT YOU NEED TO REALLY SEE IS THAT WE HAVE MOVED FROM A PRIMARILY TEXT-BASED CONTENT TO BETTER GRAPHICS AND VISUALS. JODY I THINK IS ON. CAN YOU COME ON FOR A MOMENT AND TALK ABOUT ORCPL STRATEGY FOR REACHING OUT IN NEW WAYS? >> >> SURE. YEAH. THANK YOU. VIDEOS ARE ACTUALLY ON NLM YOUTUBE CHANNEL. I WILL PUT LINK IN CHAT BOX FOR YOU. WE RESIGNED YOUTUBE CHANNEL THAT IS BETTER CATEGORIZED AND NOW WE HAVE 25 + RESEARCHER VIDEOS AND RIGHT NOW ARE LOOKING AT PRIMARILY USING TWITTER. WE PROMOTE THE VIDEOS VIA TWITTER AND WE ARE USING VIDEOS MUCH MORE IN OUR PRESENTATIONS AND WE HAVE GONE AWAY QUITE A BIT FROM OUR PRINT MESSAGING AND ARE DOING LOTS MORE WITH DIGITAL I WILL CALL YOUR ATTENTION. WE RECENTLY UPDATED NLM NEWS AND EVENTS PAGE THAT NOW IS MUCH MORE INTERACTIVE. PREVIOUSLY, IT WAS JUST LINKS TO DIFFERENT ANNOUNCEMENTS. I WILL PUT THE LINK IN THE CHATBOX AS WELL TO THAT TO GIVE YOU ANOTHER IDEA HOW WE ARE DOG THINGS MORE DIGITALLY AND LOOKING AT HOW WE WANT TO EXPAND LINKED IN PRESENCE AS ANOTHER DISSEMINATION CHANNEL THAT IS ANOTHER STRATEGY WE ARE EXPLORING AT THE MOMENT. >> GREAT. I HAVE A TINY FOLLOWUP QUESTION. DO YOU HAVE CONCERNS ABOUT WHAT GPT CHAT MIGHT DO TO SCIENTIFIC SUMMARIES? >> I WAS GOING TO ASK DIANNE BABSKI DIRECTOR FOR LIBRARY OPERATIONS TO SPEAK TO SPANISH LANGUAGE MEDLINE PLUS AND WILL ASK YOU TO SPEAK TO CHAT GPT. WE HAVE INFORMATION THERE TOO. DIANNE? >> SURE. THANK YOU. I WILL FIRST TOUCH ON SOME WORK WITH MEDLINE PLUS. MEDLINE PLUS WE CREATE IS A CONSUMER-BASED INFORMATION RESOURCE. WE PROVIDE THAT IN BOTH ENGLISH AND SPANISH. IT IS A VERY WELL-USED RESOURCE THAT REALLY HELPS TO POINT PEOPLE TO -- IT ADDRESSES MISINFORMATION POINTING USERS TO REVIEW AND QUALIFY LINKS AND INFORMATION INSTEAD OF JUST, YOU KNOW, ENDING UP ON SOMETHING FROM A WEB SEARCH ENGINE HOPING IT IS GOOD. WE HAVE CONVERTED TO -- WE HAVE CREATED A SPANISH VERSION MEDLINE PLUS IN ESPAN YOL. IT IS VERY WELL RECEIVED AND HAS MORE USERS THAN OUR ENGLISH VERSION. THERE WAS DEFINITELY A NEED TO ADDRESS THAT POPULATION AND HEALTH INFORMATION NEEDS AS WELL. WE ARE VERY EXCITED ABOUT THAT. LATER ON TODAY AFTER COLLECTIONS WORKING GROUP YOU WILL HEAR A SUMMARY AND OTHER THINGS WE ARE DOING TO ADDRESS THE SPANISH SPEAKING POPULATION AS WELL. WE ARE ON TO CHAT GPT AN AI ALGORITHM THAT USES LARGE DATA SETS OR SEARCHES THE WEB TO PROVIDE RESPONSES TO QUESTIONS AND OUR WONDERFUL RESEARCH COMMUNITY HAS FOUND A WAY TO TAP INTO IT TO HELP THEM WITH WRITING SOME RESEARCH PAPERS THAT I THINK YOU HAVE SEEN SOME OF THAT. WE ACTUALLY HAVE TWO CITATIONS IN PUB MED WITH CHAT GPT BOT LISTED AS AN AUTHOR. WE ARE FOLLOWING THIS. I KNOW. YEAH. SO, WE ARE FOLLOWING THIS. WE ARE WATCHING HOW ALL OF THIS UNFERALS WITH AI AND WITH -- YOU KNOW, WHAT CONSTITUTES AN AUTHOR THINKING ABOUT PLAGIARISM AND SO MANY THINGS THAT COULD, YOU KNOW, FALL UNDER SOME OF THE CONCERNS THAT WE HAVE FOR THIS. IT IS EXCITING TECHNOLOGY. WE ARE WORKING ACROSS WITH OTHER PUBLISHING STANDARDS ORGANIZATIONS LIKE WAMMY AND ICMGE TO SEE WHAT THEIR PROCESSES WILL BE AND RECOMMENDATIONS THAT ARE GOING FORWARD. DID I HIT ON ALL THOSE POINTS? >> THANK YOU. WOW. >> SURE. >> PHILIP? >> YEAH. GOOD MORNING, EVERYONE. THANK YOU FOR THE ALWAYS WONDERFUL NLM UPDATE, DR. BRENNAN. I AM WONDERING FOR LATER PHASES OF PREPRINTS, ARE -- IS THERE ANY CONSIDERATION TO INCLUDE OTHER FEDERAL AGENCY FUNDING? I CAN IMAGINE VARIABLE AND OF COURSE FITTING SCOPE OF PUB MED AS WELL. I IMAGINE VALUABLE PREPRINTS THAT COMIE ARE COMING OUT EPA AND DOD AND. >> WE HAVE MANY AGENCIES ALREADY AND HOST PUBLICATION AND THINK IT WOULD BE A NATURAL EXTENSION. AGAIN, IF THE FRAMEWORK AND LOGIC FOR DOING THIS IS SUCCESSFUL, WHAT WE WILL PROBABLY DO IS ENSURE THAT ARTICLES THAT WE ARE DISTRIBUTING BY PREPRINT WOULD HAVE A NATURAL RELATIONSHIP TO PUB MED CENTRAL IN SCOPE OF HOLDINGS SHOULD BE BIOMEDICALLY RELATED AND I THY SAME COST CONTROLS AND LOOKING AT FOUR PRESERVER PREPRINTS WE ARE LOOKING AT AND ANALYZING OVERLAP HOLDINGS AND PUBLISHING AND GET IT. NOTHING IS OFF THE TABLE. WHAT WE WANT TO DO FIRST IS BEYOND COVID IS THERE SUCCESS MOVING THIS TO ENDURING ENTERPRISE? PREPRINTS ARE INITIALLY CHARTERED AS A PANDEMIC RESPONSE STRATEGY AND TURNING IT INTO SOMETHING ONGOING THIS HAS RIGHT FEEL OF PROPER HOLDING AND OFFERING FOR LIBRARY. >> YEAH. I COULD ADD THERE IS A BET OF INTERAGENCY DISCUSSION GOING ON NOW ABOUT PREPRINTS AND A NUMBER OF AGENCIES HAVE LOOKED AT EXPERIENCE SEEING HOW IT FITS IN WITH RESEARCH AND RESEARCH COMMUNITIES. THERE IS THEM LEARNING LESSONS FROM PIONEER WORK IN THIS AREA LEADING TO FUTURE ENGAGEMENTS AS STEVE SAID AND TAKES WILLINGNESS ON THEIR SIDE AND NOT PUTTING PAUL FROM NATIONAL AGRICULTURAL LIBRARY ON SPOT BUT FEW CONVERSATIONS WITH HIM NOT SETTING ANYTHING UP YET BUT SHARING EXPERIENCE WITH THE AGRICULTURAL LIBRARY. >> THANK YOU FOR AN ENGAGING PRESENTATION. IT IS WONDERFUL TO HEAR WHAT IS GOING ON WITH THE NLM. I HAVE A TWITTER-RELATED QUESTION AND WHETHER NLM CONSIDERED MASTODON THAT IS A DECENTRALIZED MICROBLOGING SITE AND NLM OR NIH COULD SPIN UP AND MANAGE OWN SERVER DETOURING SOME MISINFORMATION ISSUE AND IT DOESN'T HAVE THE REACH. IT IS RELATIVELY NEWER BUT IT WOULD ALLOW -- ALLOWS AGENCIES TO CONTROL THEIR OWN INFORMATION AND WONDER IF THAT IS ON THE RADAR. >> ASKING JUDY TO COME OFF MUTE IF SHE WANTS TO SPEAK AND NIH CONSIDERED A NUMBER OF PLATFORMS. >> MASTODON IS BEING DISCUSSED RIGHT NOW AT AN OVERALL NIH COMMUNICATIONS DIRECTOR'S GROUP. I HAVE NOTHING TO REPORT ON THAT OTHER THAN IT IS SOMETHING THAT IS BEING DISCUSSED. >> >> RECORDING IN PROGRESS. >> A LITTLE OECHO. WE ARE GETTING FIXED. CAN YOU PUT YOUR SPEAKER OFF OR ON SILENT? >> GREAT. THAT BRINGS US TO THE END OF THIS CLOSING SECTION. OUR NEXT AGENDA ITEM IS BOR WORKING GROUP BREAKOUTS AND I ASSUME THAT WE WILL BE ASSIGNED TO THOSE ROOMS AUTOMATICALLY? I CAN SEE I'M BEING PROMPTED TO JOIN. SO, WE WILL SEE YOU ALL AFTER THE BREAKOUTS. I'M LOURDES AND I'M HAPPY TO REPORT ON PUBLIC SERVICE WORKING GROUP AND SOME WORK WE HAVE BEEN DOING TO DATE. NEXT SLIDE. ARE YOU ABLE TO PROJECT THE SLIDE SO IT IS SLIGHTLY BIGGER FOR EVERYONE TO PRESENTATION MODE? THERE WE GO. GREAT. THANK YOU. >> THANK YOU, HEIDI. THANKING COMMITTEE THAT HAS BEEN WORKING TIRELESSLY NOW FOR FOUR YEARS GETTING WORK DONE AND RECOGNIZING COLENELWHO WILL BE STEPPING DOWN AND HAS BEEN JUST A TREMENDOUS SUPPORTER OF THE WORK WE HAVE BEEN DOING AND TERRIFIC CONTRIBUTIONS AND WOULD LIKE TO GIVE A SPECIAL THANK YOU TO HIM AS WELL AND LITTLE ROUND OF APPLAUSE. THERE WE GO. NEXT SLIDE IS . TO LET YOU KNOW, WE HAVE BEEN -- WE HAVE HAD 16 DIFFERENT MEETINGS ALREADY TO GET MODERNIZATION OFF THE GROUND. WE HAVE BEEN MEETING SINCE 2019 AND HAVE HAD WEBINARS AND PUBLIC MEETINGS IN ADDITION TO OUR OWN MEETINGS. WITH VERY BEEN ABLE TO RELEASE A FEW PRODUCTS AS A RESULT OF THIS. NEXT? MOST OF THIS EFFORTS CAN BE SEEN AND ARE IN PROGRESS REPORTS THAT REALLY THIS ONE IN PARTICULAR INCLUDES ALL OF THE WORK IN THE LAST THREE YEARS AND INCLUDES ALIGNMENT OF CLINICAL [INDISCERNIBLE].GOV AND PRS DATA AND WITH MODERNIZATION PRIORITIES AND WORKING GROUP WORK AND STAKEHOLDER'S INPUT THAT HAS BEEN REALLY CRITICAL FOR US AND PUBLIC COMMUNICATIONS RELATED TO MODERNIZATION AND FUTURE MODERNIZATION ACTIVITIES THAT IS AVAILABLE ALREADY AND PUBLICLY AVAILABLE AS WELL. NEXT? >> WORKING GROUP ACTIVITIES HAVE BEEN RATHER INTENSE AND WE ARE PLANNING FOR 2023 IS VERY EXCITING RELEASING FINAL VERSIONS AND HAVE A PUBLIC MEETING COMING UP IN APRIL 25 NL WITH AN OPPORTUNITY TO HAVE A BROAD DISCUSSION WITH PUBLIC AND SOME MEETINGS HAVE BEEN WEBINARS. THERE HAS NOT BEEN OPPORTUNITIES FOR ONE-ON-ONE DIALOGUE. WILL PROVIDE US WITH AN OPPORTUNITY TO HEAR VOICES OF COMMUNITY AND THEY HAVE BEEN HEARD IN MULTIPLE WAYS THROUGHOUT AND HAVE BEEN CONTRIBUTING TO DIFFERENT MODERNIZATION EFFORTS. NEXT? SO, SOME WORK THAT HAS BEEN DONE ESPECIALLY MORE RECENTLY HAS BEEN FEEDBACK WE HAVE OBTAINS FROM THE COMMITTEE ON DATA MONITORING COMMITTEE CHARTERS. THERE WAS. NEXT SLIDE. WHAT HAPPENS IS A PROPOSAL CAME IN OR REQUEST SO THAT WE COULD RELEASE MORE INFORMATION ON DMC CHARTERS PUBLICLY. THERE IS NO CURRENT REQUIREMENTS POSTING DMC CHARTERS AND INFORMATION THEY PROV PROVIDE. WE ARE CONSIDERING OPTIONS FOR DEDICATED ACCESS TO CHARTERS AND NATIONAL LIBRARY OF MEDICINE DECIDED TO OFFER AN OPTIONAL ACCESS TO CHARTERS AND HAS BEEN DISCUSSION ABOUT THAT. WE ARE TRYING TO UNDERSTAND BETTER WHAT IT WOULD MEAN AND HOW TO GO ABOUT DOING THAT. NEXT? I WILL TALK ABOUT [INDISCERNIBLE]. NEXT SLIDE. AS YOU CAN SEE, WE HAVE BEEN REALLY RELEASING AND DOG BETA LAUNCHES THROUGHOUT SINCE DECEMBER OF 2021 AND IN JANUARY YOU CAN SEE IN THE LAST FEW MONTHS WE ARE RELEASING THINGS ALMOST ON A REGULAR OR MONTHLY BASIS GETTING FEEDBACK AND SOME RELEASES HAVE BEEN RATHER THAN BUNDLING UP AND IT HAS BEEN YOU KNOW IN PIECES TO GET REALLY FEEDBACK TO BE ABLE TO INTRODUCE FEEDBACK FROM THE COMMUNITY AND USERS ESPECIALLY BACK INTO THE MODERNIZATION AREAS. NEXT? SO, THIS IS OUR PRS MODERNIZATION PLAN. IT IS A PROTOCOL REGISTRATION FORM THAT HAS BEEN TESTED AND ARE EXPECTED TO BE RELEASED IN APRIL OF 2023. SO, THAT IS ANOTHER WORK THAT IS IN PROGRESS WHERE STAFF HAS BEEN WORKING QUITE DILIGENTLY ON THAT. NEXT? CURREN CURRENTLY AVAILABLE WE HAVE CLASSIC SITES WE HAVE USED AND NEW BETA SITE. BY JUNE, NEW SITE WILL BE PRIMARY WEBSITE THAT WE HAVE GIVEN AMPLE OPPORTUNITIES FOR COMMUNITY AND USERS TO SORT OF USE BOTH IN PARALLEL AND HEAR FROM OWN NANCY ON COMMITTEE THAT SOME PREFER EACH HERSELF TO USE BETA AND SEE ALL IMPROVEMENTS ON IT AND HOW IT WILL BE A GREAT IMPROVEMENT AND REALLY MODERN IIZE. FOR REST OF 2023 FOR JUNE CLASSIC WILL REMAIN AVAILABLE THROUGHOUT 2023 AND WE WILL TRANSFER INTO THE NEW MODERNIZED SITE. NEXT? SAVE THE DATE PUBLIC MEETING DATE APRIL 2 FIF NL 2023 TO HAVE AN OPPORTUNITY TO LAUNCH EVERYTHING BE IN CONTACT WITH USERS MORE DIRECTLY THAT WILL BE A VIRTUAL PUBLIC MEETING THAT IS OPEN TO ALL OF YOU AND THE PUBLIC. NEXT? THAT IS PRETTY MUCH IT. LAST ONE? YEAH. I THINK IT IS THE LAST ONE OF OUR SLIDES. I ALSO WANT TO DO A SPECIAL THANK YOU TO ALL OF THE STAFF THAT HAS WORKED ON THIS WITH ANNA'S LEADERSHIP AND PEOPLE THAT CAME BEFORE HER AND THROUGHOUT THIS AND REALIZING WE HAVE WORKED ON THIS SINCE 2019 GETTING THROUGH ENTIRE PANDEMIC WORKING ON THIS. WE MADE TREMENDOUS PROGRESS IN THE LAST FOUR YEARS AND THANKING EACH AND EVERY ONE OF THEM AS WELL THAT IS INSTRUMENTAL IN MAKING THIS POSSIBLE. THANK YOU. >> PERFECT. THANK YOU. I WOULD ECHO THAT. AMAZING EFFORT LOOKING SO GOOD. SUCH A CRITICAL RESOURCE. ANY QUESTIONS FROM THE BOARD OR ANYONE IN THE MEETING, LORDES? >> ALL RIGHT. IF NO QUESTIONS I WILL MOVE TO THE COLLECTIONS WORKING GROUP WHO WILL GIVE A REPORT OUT FOR THAT? >> THIS IS PAUL. WE HAVE A VERY FULL MEETING AND SOME TOPICS RECOVERED IN THE EARLIER PLENARY SESSION OF THE BOARD MEETING AS WELL AND FOUR TOPICS COVERED ARE PMC LANGUAGE EXPANSION AND SPANISH LANGUAGE JOURNALS AND LAUREN GAVE A GREAT PRESENTATION ON THAT AND SECOND MONITORING PUBLICATION CHANGES MEDLINE AND PNC JOURNALS AND JENNIFER GAVE A GREAT PRESENTATION THERE AND CLIFTI HOL MS GAVE UPDATE WITH GENOMICS RESOURCE AND CGR AND WORK GOING ON IN THAT SPACE AND SOME UPCOMING MEETINGS THAT ARE BECOMING WITH THAT. KATE GAVE UPDATE ON PREPRINT PILOT SITUATION. WE COVERED WHERE TO HAVE WRITE UP FORFIRST PHASE OF THE PILOT AND LAUNCHING PHASE 2 THAT IS LAUNCHED EARLY NEXT WEEK AND COMMENTARY I HAVE FROM EACH OF THE FOUR TOPICS WE WENT THROUGH AND THERE IS REALLY AN EMPHASIS THINKING THROUGH DIFFERENT KINDS OF PRIORITIES AND ACTIVITIES THAT NLM IS TRYING TO PURSUE IN THE AREA OF DEIA AS RELATES TO HOW WE APPLY THOSE PRINCIPLES TO WHAT WE ARE DOING IN EACH OF THESE AND DEFINITELY IN THREE OF THE FOUR AND WOULD SAY ACROSS EACH OF THE FOUR AREAS AND PMC LANGUAGE EXPANSION AND SPANISH LANGUAGE JOURNALS TOPIC WAS A RICH DISCUSSION OF THE DIFFERENT KIND OF STANDARDS FOR GETTING INTO TWO DIFFERENT AREAS AND AS TWO DIFFERENT PRODUCTS RELATE TO LARGER NLM COLLECTION. I WAS REALLY STRUCK BY FOCUS ON TRANSPARENCY, TRUST, AND SUSTAINABILITY THAT IS REQUIRED WITHIN EACH OF THOSE FOUR AREAS. KIND OF THE DATA DRIVEN DISCUSSIONS WE HAD AROUND WHAT IS GOING ON IN THOSE DIFFERENT SPACES. FOR INSTANCE, WITH PMC LANGUAGE EXPANSION PIECE, THERE IS LOTS OF THOUGHTFUL ANALYSIS DONE ABOUT WHERE PROSPECTIVE JOURNALS ARE LIKELY TO COME FROM AND UNDER-REPRESENTED AREAS OF BIOMEDICAL LITERATURE NEEDING TO BE BROUGHT IN AND TYPES OF PROCESSES HOW TO EVALUATE JOURNALS AND TRANSLATING THEM TO SPANISH LANGUAGE JOURNALS SCALING THEM UP TO BE ABLE TO WORK WITHIN THE PMC IS PROCESS. GOOD CONVERSATION. MONITORING PUBLICATION CHANGES PORE JOURNALS AND JENNIFER LED IS INTERESTING. WHOLE DATA-DRIVEN APPROACH TO UNDERSTANDING RELATIONSHIP BETWEEN BRINGING JOURNALS INTO PRODUCTS AND UNDERSTANDING HOW THOSE JOURNALS CHANGE AND HAVING CLARITY AROUND POLICIES AND WHAT IS REQUIRED IN JOURNALS AND ANALYSIS TO DO AND AROUND INCREASED VOLUME OF PUBLICATIONS AND HAD GOOD CONVERSATION AROUND THAT RESULTING IN WORKING THAT NLM IS DOING IN THAT SPACE. I SAID CLIFTI GAVE A GREAT UPDATE WHAT IS GOING ON IN COMPARATIVE GENOMICS RESOURCE WORKING GROUP AND HUGE SHOUTOUT TO DIFFERENT PARTICIPANTS FROM THERE GROUP ON THAT WORKING GROUP. WE HAD A DISCUSSION OR WALK-THROUGH OF HOW TRADEOFF EXERCISES HAVE BEEN DONE TO UNDERSTAND WHAT FUNCTIONALITY NEEDS TO BE BROUGHT INTO THAT TOOL GOING FORWARD. DISCUSSION OF OUTREACH APPROACHES UNDERTAKEN IN VOLUME OF THEM AND IN QUALITATIVE ANALYSIS HOW WELL THEY ARE WORKING TO BUILD CAPACITY BUILD INTEREST AND BUILD ENGAGEMENT. GREAT PRESENTATION FROM CLIFTI AND PRESENTATION FROM KATIE ON PREPRINT PILOT AND COVERING GROUND WE TALKED ABOUT IN THE PLENARY AS I DISCUSSED EARLIER AND ONE THING WE WENT BACK TO I MENTIONED DATA-DRIVEN APPROACH TO ANALYSIS UNDERGERDING THESE ACTIVITIES QUANTITATIVE AND QUALITATIVE ANALYSIS BEING DONE AND NOTION THAT STEVE SHERRI BROUGHT UP EARLIER NOTION OF FIRST CLASS RESEARCH OBJECTS AND BEING ABLE TO UNDERSTAND WHAT THOSE ARE AND HOW THEY ARE IT CHANGING AND UNDERSTANDING OF THEM AND HOW TO UNDERSTAND RELATIONSHIPS BETWEEN DIFFERENT FIRST CLASS RESEARCH OBJECTSES AND WHAT THEY MEAN FOR DIFFERENT PRODUCTION AND BUSINESS PROCESSES IN NLM AND WHAT LARGER IMPLICATIONS OF THAT IS FROM A RESEARCH PERSPECTIVE. I WOULD SAY I HAVE PROBABLY 50 SLIDES FROM DIFFERENT PRESENTATIONS AND NOTES FROM CONVERSATIONS AND SEPARATE BOUT THAT WAS COVERED AND HUGE SHOUTOUT TO COLLEAGUES AT NLM GIVING PRESENTATIONS TODAY AND CLIFTI WHO GAVE A PRESENTATION ONL COMPARATIVE GENOMICS RESOURCE GROUP AND THEIR ACTIVITIES AND ANOTHER MEETING WORKING GROUP MEETING BETWEEN NOW AND NEXT MEETING OF BOARDS AND MORE TO REPORT GETTING BACK TOGETHER IN MAY. IT WAS A GREAT SESSION. WANTED TO SEE IF COLLEAGUES WANTED TO OFFER COMMENTS ABOUT WHAT WE COVERED OR TALKED ABOUT IN OUR GROUP. >> ANY COMMENTS FROM WORKGROUP PARTICIPANTS? >> NO. THANK YOU, PAUL. >> I WAS GOING TO CHIME IN. PAUL, THANKS FOR THE UPDATE. GREAT JOB. TO BUILD ON WHAT WE TALKED ABOUT IN THE MAIN SESSION EARLIER IS THAT THIS IS INTRODUCING SPANISH LANGUAGE INTO PMC WILL BUILD ON OTHER TRENDS WE HAVE SEEN WITH SOME OTHER RESOURCES WE CREATE AND SPANISH MEDLINE PLUS NAMELY AND ESPA„OL. WE WILL FOLLOW THIS AND LOOK AT IMPACT AND WE WILL BE EXCITED TO REPORT OUT ON WHAT WE LEARNED FROM THIS AS WE GO FORWARD. THANK YOU FOR THAT. >> GREAT. QUESTIONS FOR THE WORKING GROUP? >> IF OKAY. GIVING A SHOUTOUT TO CHRISTY WHO HAS BEEN REPORTER FOR OUR GROUP FOR LAST THREE OR FOUR MEETINGS AND SAYING HOW HARD IT IS TO PUT TOGETHER CONTENT FROM DIFFERENT PRESENTATIONS AND I HAVE LARGE SHOES TO FILL AND DON'T KNOW I DID IT. >> IT WAS SUCH A GREAT MEETING. YOU DID A GREAT JOB SUMMARIZING IT. SO MUCH CONTENT AND THANKS SO MUCH. >> YOU HAVE TO BE CAREFUL WHEN YOU DO IT REALLY WELL. YOU MIGHT HAVE TO DO IT AGAIN. >> [LAUGHING]. >> JERRY? >> YEAH. THANK YOU. >> A LITTLE ECHO IN OUR ROOM. SORRY. >> SO, I WILL ASK DIANNE IF -- I WILL TRY TO TURN IT OFF. IS THAT BETTER? >> CAN SOMEONE STEP IN, PLEASE? >> STILL GETTING IT. >> YOU ARE. >> STILL GETTING ECHO? >> YEAH. >> THE PARTICIPANTS LIST I WILL CHECK SFWL I WAS GOING TO ASK DIANNE HEARING ME NOW FOUR DIFFERENT TOPICS IN GROUP PREPRINT PILOT PHASE 2 LAUNCHED. IF YOU CAN SAY SOMETHING ABOUT TIMELINES FOR SPANISH LANGUAGE AND NEW NOT NEW BUT NEW INFORMATION ABOUT JOURNAL MONITORING THAT WILL GO INTO EFFECT AND BOTH BEFORE THE NEXT BOARD MEETING. >> YEAH. WE ARE LOOKING AT PROMOTING SPANISH LANGUAGE AND JENNIFER JUMP IN HERE. I LOOKED AT SLIDES SO MUCH YOU THINK I WOULD HAVE IT MEMORIZED. WE PREDICT DOING IT THAT IN -- MARCH 1ST. WE WILL LAUNCH TECHNICAL BULLETIN ARTICLES AND LIKE TO KEEP USERS INFORMED WHAT WE ARE DOING. BOTH WILL BE SPANISH PMC AND CHANGES TO THE REVIEW PROCESSES FOR JOURNALS DOING SYSTEMATIC APPROACH FOR REVIEWING THEM. THAT WILL BE PROMOTED IN EARLY MARCH. SEE KATIE AND LAUREN CHIMING IN. CORRECT ME IF I NEED CORRECTING. >> YOU PRETTY MUCH HAVE IT, DIANNE. PLAN WITH SPANISH IS POSTING TECHNICAL BULLETIN IN THE SECOND HALF OF THE MONTH AND PUBLICATION CHANGES UPDATE IS MARCH FIRST. THAT GENERAL BLOCK OF TIME. >> THANK YOU. >> GREAT. OTHER QUESTIONS FOR COLLECTIONS GROUP? >> I WILL TURN TO JIM. HOPEFULLY YOU FINISHED YOUR LUNCH. >> [INDISCERNIBLE] NOW. >> YEAH. >> ALL RIGHT. I -- I -- SO WE ARE A RESEARCH FRONTIERS GROUP AND EXCELLENT NOTES FROM VALERIE FLORENCE AND THREE TOPICS TO DISCUSS AND TALKING ABOUT GENERATIVE AI WHICH IS ARTIFICIAL INTELLIGENCE THAT PRODUCES NEW CONTENT OPPOSED TO SEARCHING FOR CONTENT OR INFORMATION AND DOING PATTERN MATCHING AND IT THAT SORT OF THING AND ACTUALLY GENERATES NEW THINGS AND CHAT GPT IS EXAMPLE IN THE NEWS A LOT AND THERE ARE OTHERS AND HE RAISED A NUMBER OF ISSUES THAT NLM COULD THINK ABOUT INCLUDING ISSUES AROUND PLAGIARISM AND MISINFORMATION AND SHOULD BE BROADER FRAMEWORK FOR THINKING ABOUT PROGRAMS AND HOW THEY -- YOU KNOW BIOMEDICAL COMMUNITY WOULD INTERACT WITH TH THEM. IMPACTING ON LITERATURE AND PEOPLE USING TO DEVELOP WRITING LITERATURE AND WHEN IT IS APPROPRIATE. SOMEBODY POINTED OUT TWO PAPERS IN MEDLINE ONE AUTHOR IN CHAT IS GPT. IT IS ALREADY HERE. WHETHER WE LIKE IT OR NOT. DISCUSSION TALKED ABOUT NATURAL LANGUAGE PROCESSING AND OVERLAP BETWEEN THOSE AND WHETHER LP COULD DO A BETTER JOB SUMMARIZING AND THERE IS ISSUES AROUND EJICS AND STANDARDS AROUND ARTIFICIAL INTELLIGENCE HAS BEEN DISCUSSED NOW AND HOW DOES THIS NEW TOOL FIT INTO THAT ISSUE? WE COULD TALK ABOUT YOU KNOW SORT OF PASSIVE AND IMPACT ON LITERATURE AND ACT OF USING GENERATIVE AI FOR INSTANCE SHOULD NLM LOOK AT GENERATIVE AI TO GENERATE PUB MED RESPONSE. INSTEAD OF LIST OF CITATIONS SUMMARY OF PAPERS COMING BACK FROM QUERY AND ISSUE RAISE THE IS WHETHER NLM REPOSITORY OF ARTIFACTS MODELS DATA SETS EVALUATION TESTS FOR EVALUATION FOR AI SYSTEMS. ADDRESSING PROPER AND NEEDED INTERVENTIONS FOR HUMANS AND DOMAIN EXPERTS USING AI TECHNOLOGIES. OKAY. THAT WAS GENERATIVE AI AND ANDREW CLARK TALKED ABOUT CAUSAL INFERENCE. YOU KNOW, WE SEE TWO THINGS THAT CO-OCCUR AND CORRELATION DOESN'T IMPLY CAUSATION, NECESSARILY. QUESTION IS HOW TO TELL WHEN YOU SEE ONE THING HAVING OR SEEMING TO BE CO-OCCURRING WITH ANOTHER IS ONE AND CAUSING THE OTHER AND ADVISE VERSA. YOU KNOW, THE -- THERE IS A TECHNIQUE CALLED MEN DEALIAN RANDOMIZATION. I'M NOT SURE WHETHER MEN DEALAN PART COMES IN EXCEPT TO USE X AND YS AND DON'T KNOW WHERE RANDOMIZATION COMES IN. IF YOU SEE X CORRELATES WITH Y AND DO SOMETHING TO ALTER X, WILL Y BE IMPACTED? IF SO OR IF NOT YOU WOULD SAY MAYBE REALLY X DOESN'T HAVE ANY ACTUAL EFFECT ON Y. IT IS SIMPLY CO-OCCURRING WITH IT. IDEA IS DOG TESTING WITH EXISTING DATA RATHER THAN COLLECTING NEW DATA. MIGHT NOT BE ABLE TO CONFIRM HYPOTHESIS OR NOT AND HELPFUL PROJECTING HYPOTHESES OR SCREENING OUT LESS LIKELY ONES. ANDY SHARED A PAPER ON CAUSAL DISCOVERY VERSUS CAUSAL INFERENCE THAT WE WILL GET OUT TO FOLKS WHO CAN LOOK AT THAT AS AN EXAMPLE AND GAVE AN EXAMPLE OF PFIZER DOING A -- WAS IT A BILLION DOLLAR TEST OR STUDY AND STOCK WENT DOWN 27% AFTER STUDY FAILED WHEN THEY COULD HAVE USED A CAUSAL INFERENCE TECHNIQUE TO DETERMINE IT WASN'T WORTH PURSUING. I TALKED ABOUT INCORPORATING BIG NONL INTO CLINICAL DECISIONS AND DEFINING BIG NONL AS NONL WE GET FROM BIG DATA AND THERE IS A LOT GOING ON IN THE AREA FOR SAKE OF TIME I TALKED ABOUT ONE PARTICULAR AREA THAT IS KNOWLEDGE THAT WE GENERATE FROM PATIENT'S GENOMES GENOMES ARE BECOMING PART OF RECORDS TO CORRELATE GENETICS AND PHENOTYPE AND MACHINE LEARNING STARTING TO GENERATE RECOMMENDATIONS BASED ON CORRELATIONS AND DON'T GIVE THIS PERSON THIS DRUG BECAUSE THEY HAVE A VARIANT. WE DON'T KNOW WHY. DON'T DO IT. GENERATING IN WAY THAT NOT ONLY WON'T BE EXPLAINABLE OR MEMORABLIZABLE AS WELL AND DECISION SUPPORT AND HOW IT IS ENHANCED TO HANDLE IT AND GIVING VARIOUS EXAMPLES OF DIFFERENT METHODS WE USE NOW PUBLISHING A PAPER VERSUS PUTTING OUT ALERTS AND REMINDERS THAT TECHNICALLY CAN'T KEEP UP WITH RATE OF NONL THAT WE WILL GENERATE. SO, RECOMMENDATION WAS TO -- FOR NLM TO START TO LOOK AT HOW WE REPRESENT THIS KIND OF NONL IN A WAY TO BE MADE ACTIONABLE AND HOW WE LOOK AT ELECTRONIC HEALTH FLOW WORKFLOWS TO INTEGRATE IN KNOWLEDGE INTO THE WORKFLOW IN A WAY THAT IS FEASIBLE THAT IS RIGHT NOW WE DO ALERTS ONE AT A TIME AND MONTHS TO DO 1 ALERT OR THOUSANDS OF ALERTS OR MILLIONS OF ALERTS AND HOW TO MAINTAIN THEM AND KEEP FROM OVERWHELMING CLINICIANS AND YOU SEE IN MY SLIDES A LINE WITH LOTS OF LINES ON IT SHOWING HOW WHEN KNOWLEDGE IS GENERATED, HOW LONG IT TAKES TO START TO GET IMPLEMENTED OR ADOPTED AND ONCE IT IS ADOPTED HOW QUICK -- HOW LONG DOES IT TAKE FOR IT TO REALLY, YOU KNOW, FOR THE UPTAKE. SOME THINGS TAKE TIME LIKE A PAPER AND SOME THINGS ARE INSTANTANEOUS. LIKE SOON AS YOU INSTITUTE AN ALERT IT IS THERE AND THINKING ABOUT HOW TO REMOVE KNOWLEDGE FROM A SYSTEM. IF A PAPER GETS REFUTED IT IS PRETTY HARD TO CONVINCE PEOPLE. THERE ARE STILL DOCTORS OUT THERE THINKING VACCINES CAUSE AUTISM. HOW DO YOU REMOVE THAT KNOWLEDGE WITH AN ALERT OR REMINDER IT IS FAST. THERE ARE IMPLICATIONS THERE AS WELL. I THINK THAT WAS IT. >> THANK YOU, JIM. QUESTION FOR YOU AROUND EHR DATA SIDE OF THINGS AND KNOW A NUMBER OF GROUPS WORKING IN THIS SPACE IS CRITICALLY IMPORTANT AND AMP ASSOCIATED FROM A TECHNOLOGY CONFERENCE OR SUMMIT OR GLOBAL ALLIANCE IS V DEVELOPING STANDARDS AND WORKING GROUP AND KNOW ETHICS ARE ENGAGED IN THESE EFFORTS AND HOW NLM IS THINKING ABOUT ITS ROLE VERSUS ACTIVITIES OTHERS ARE WORKING ON AND HOW TO MAKE SURE THAT EVERYBODY IS ALIGNED. >> QUESTION FOR ME? >> TO THE WORK GROUP, I GUESS. I MEAN, TO -- WHOEVER. >> I WANT TO ASK PATTY THAT QUESTION. >> YES. GOOD QUESTION. GOOD POINT, JIM. >> AS ANY GOOD DIRECTOR, I WILL DEFER TO STAFF. I NEED TO UNDERSTAND THE QUESTION A LITTLE BETTER. I THINK WHAT IT IS, IS WHAT IS ROLE IN NLM ADJUDICATING TRUTH? >> IT IS NOT JUST TRUTH. IN THE SENSE IT IS JUST HOW TO ORGANIZE ACTIVITIES IN THE SPACE OR MAYBE I MISUNDERSTOOD WHAT NLM MY OLD CLINICAL APP. NLM IS TRYING TO DO IN THIS SPACE SPECIFIC SPECIFICALLY. >> WHEN YOU SAY IN SPACE WE HEARD A COUPLE THINGS I THINK ARE PRETTY IMPORTANT AND ONE IS TO BUILD TOOLS DRAWING KNOWLEDGE FROM CLINICAL RECORDS SYSTEMS INCREASING CLINICAL RECORDS SYSTEM INCLUDING GENOMIC INFORMATION AND WE HAVE A SITUATION WITH THOUSANDS OF DATA POINTS FOR A SINGLE PERSON AND HOW TO PUT THEM TOGETHER INTO A STORY NOT ONLY ABOUT THAT PERSON BUT MIGHT LEAD TO STORIES ABOUT OTHER PEOPLE AND HAD A LITTLE WORK COMING OR GOING IN THAT AREA OF EXTERNAL PROGRAM AND FUNDING TO BRING TOGETHER MULTIPLE STREAMS OF DATA. WE ARE CERTAINLY LOOKING FORWARD TO A VISION OLIVIA WILL SPEAK ABOUT WITH DEALING WITH OBSERVATIONAL DATA AND OPPORTUNITIES WE ARE JUST STARTING AT. BIGGER QUESTION HAS TO BE IN USING WHAT VEHICLE WE USE TO DO THIS AND HEARD A REQUEST THAT WILL MAKE STEVE CRAZY BUT HEARD REQUEST OF RESTORING ARTIFACTS AND MODELS AND DIFFERENT TOOLS INFORMATIVE TO RESEARCH OPPOSE TOTD PRODUCTS OF RESEARCH AND HAVE TO THINK ABOUT THAT AND TALKED ABOUT IT OFF AND ON OVER THE YEARS BUT HAVEN'T SETTLED ON IT. BIGGEST OPPORTUNITY IF YOU ASK ME FRANKLY IS SEATING RESEARCH ENTERPRISE INTRA-AND EXTRAMURAL PROGRAMS AND HOW TO BUILD QUESTIONS IN AND RICHARD IS NIH RESEARCH BASE TODAY AND UNABLE TO BE HERE AND WE WILL TAKE IT BACK AND MIGHT BE SOMETHING WORTH HAVING HIM REPORT ON IT IN THE MAIN MEETING ABOUT HOW EXTRAMURAL COMMUNITY IS LOOKING TO BRING IN GREATER AWARENESS FOR THE BROADER COMMUNITY ABOUT RESEARCH OPPORTUNITIES NOT WRITING SOLICITATIONS BUT GIVING IDEAS WHERE COMMUNITY WANTS TO GO AND OLIVIA TALKING ABOUT LEARNING ABOUT OBSERVATIONAL DATA. >> WE KNOW OBSERVATIONAL DATA SETS FROM EHRS AND CLAIMS INCREASINGLY BECOMING AVAILABLE. WE KNOW THEY ARE DIFFICULT FOR RESEARCHERS TO GET FAMILIAR WITH LATER ON IN THE PROCESS AND HAVE THOUGHT ABOUT A ROLE THAT NLM COULD PLAY IN ANALYZING CAPITAL GAIN IF YOU WISH THESE DATA SETS AND VARIABLES AVAILABLE FOR THEM TO HAVE RESEARCHERS FIGURE OUT HOW TO USE THEM MORE EASILY. LY LEAVE IT AT THIS POINT NOW. >> IS THAT DONE IN CONJUNCTION WITH ALL OF US RESEARCH PROGRAM THAT IS DRAWING LOTS OF STRUCTURED DATA FROM ELECTRONIC HEALTH RECORDS FROM RESEARCH? >> I WOULD CONSIDER ALL OF US ONE OF THE DATA SETS THAT ARE AVAILA AVAILABLE. WE CAN THINK OF OTHERS SUCH AS MEDICARE DATA SETS FROM CMS AND UK BIOBANK DATA SETS AND ARE MANY THAT HAVE BECOME PUBLICLY AVAILABLE IN THE PAST FEW YEARS. >> COULD I ADD ONE THING. >> YOUR MIC. >> OKAY. ONE ADDITIONAL FACT, I THINK, ALL OF YOU KNOW IF YOU ARE EXTRAMURAL OR INTRAMURAL REALLY THAT ALL GRANTEES AND ALL RESEARCH ORGANIZATIONS WITH NIH ARE NOW REQUIRED TO SHARE RESEARCH DATA AND THEY HAVE GENERATED USING FEDERAL FUNDS. THERE IS. I FEEL AN OCEAN AND HOW THAT IS -- TOOLS FOR WORKING WITH IT AND IDENTIFYING IT AND FIGURING OUT EVERYTHING THAT FINDS OF THINGS YOU ARE THINKING ABOUT WILL BE. THEY ARE NOT THERE YET. RIGHT? YOU WILL SEE IT ALL SOON. WILL BE PUBLISHED AND MOST OF I THINK 93% OF RESEARCH GROUPS AT NIH HAVE ESTABLISHED A MODEL FOR DATA ACCESS TO RESEARCH DATA. WE ARE ONE OF THEM. THE FUTURE. >> LOT TO DO. THANK YOU. >> YES. WANTED TO ASK JIM LIVELY DISCUSSION FOR USE OF CHAT GBT FOR GENERATING CONTENT FOR NLM AND AS YOU KNOW HAS BEEN DISCUSSION ABOUT FACT IT GENERATES INACCURATE INFORMATION WITH CONFIDENCE AND PRESENTS IT CONFIDENTLY AND CREATES REFERENCES THAT DON'T EXIST. I WAS TRYING TO GET A SENSE WHAT THE DISCUSSION WAS AND NOW THAT YOU KNOW ICMJ IS BEING ENCOURAGED TO STATE CHAT GPT CAN'T BE A CO-AUTHOR AND DOESN'T MEET ACCOUNTABILITY CITER YA BEING A COAUTHOR AND WOULD LOVE TO HEAR WHAT DISCUSSION WAS ABOUT THAT. >> FIRST OF ALL CHAT GPT IS A PLACEHOLDER FOR GENERAL GROUP OF THIS TECHNOLOGY. IF YOU HAVE A -- IF YOU HAVE A CHAT BOT WITH DATABASE ENTIRE BODY OF MEDICAL LITERATURE FROM NATIONAL LIBRARY OF MEDICINE IT WOULD BEHAVE DIFFERENTLY FIRST OF ALL AND HAVE MORE INFORMATION TO WORK WITH AND WOULD EXPECT YOU COULD -- THAT IT WOULD -- THERE IS A WAY TO, YOU KNOW, MAKE SURE IT IS USING REAL CITATIONS WHEN STATING CITATIONS ESPECIALLY WHEN IN A VERY SPECIFIC FUNCTION AND CHAT GPT IS GENERAL PURPOSE. SOMETHING WITH TELL ME WHAT IS IN MEDICAL LITERATURE OR INSTEAD OF TELL ME ANSWER TELL ME WHAT IS IN MEDICAL LITERATURE ABOUT THIS SO I CAN READ FOR MYSELF. YOU KNOW, WE DON'T TRY TO SIFT THROUGH AND FIGURE OUT WHICH ARTICLE IS RIGHT ARTICLE AND WHICH IS WRONG ARTICLE IN PUB MED. WE GIVE YOU ARTICLES. THIS IS WAY TO SYNTHESIZE AND SUMMARIZE WHAT ARTICLE IS SAYING AND POINT OUT MAYBE SDREPENCIES OR CONTROVERSIES. IT IS MORE ABOUT WE RECOGNIZE CHAT GPT HAS LIMITATIONS AND OTHER THINGS THAT MIGHT BE MORE APPROPRIATE. >> I MIGHT IF ALL RIGHT JUMP IN ON THAT. A COUPLE HERE MIGHT BE ALSO USEFUL. ASKING CHAT GPT TO CONSTRUCT TEXT ON A PARTICULAR TOPIC WITH CITATIONS WILL GIVE YOU CITATIONS LOOKING LIKE WRITE CITATIONS FROM JOURNALS THAT DON'T EXIST AND AUTHORS THAT DON'T EXIST. THEY ARE FORMATTED TO LOOK LIKE THEY SHOULD BE TRUSTABLE AND YOU SHOULD BE ABLE TO TRUST THE INFORMATION BUT THANKFULLY. >> WHAT IT IS DOING PEOPLE ARE THINKING IT IS GIVING THEM AN ANSWER AND IS A LANGUAGE MODEL AND THIS IS WHAT ANSWERS SHOULD LOOK LIKE. FACTS ARE IRRELEVANT TO IT. YOU SHOULD HAVE A CITATION HERE THAT LOOKS LIKE THIS CITATION IF YOU WANT TO MAKE THIS ARGUMENT. NEXT THING TO PUT IN THE ARGUMENT IS THIS THING AND HOW YOU CONCLUDE YOUR ARGUMENT THAT IS A LANGUAGE MODEL NOT FACT-BASED SYSTEM. >> YEAH. OTHER THING I WANTED TO QUICKLY MENTION IS THAT WHEN YOU QUERY OR WHEN YOU INPUT QUERIES, IF YOU PUT SAME QUERY IN, IT RETURNS DIFFERENT RESULTS. THERE IS ISSUES WITH REPRODUCIBILITY TO PUT THAT CAVEAT FRONT AND CENTER HAS BEEN VERY INTERESTING AND SOMETIMES THEY ARE VERY PARTICULAR BUT -- >> DO YOU WANT TO TALK ABOUT THE MISSED ENGAGEMENT NOW OR NO? >> NOT YET. >> THAT IS FINE. >> NEXT TIME. >> WE ARE LOOKING INTO DIFFERENT ASPECTS ABOUT HOW TO PROMOTE TRUST WHEN WE HAVE -- IF YOU ASK SAME QUESTION TWO DAYS IN A ROW IN PUB MED SYSTEM YOU MIGHT GET DIFFERENT LIST OF RESULTS RELEVANCE RANKING SYSTEM THAT LEARNS FROM ITSELF OR NO THE THAT FAR OFF. REPRODUCIBILITY WILL NEED TO BECOME SOMETHING DIFFERENT THAN RETURNS EXACT SAME WORDS AND NUMBERS IN SAME ORDER. WE ARE INTERESTED IN SHOWING INTEGRITY OF RESOURCES WITH RESPECT TO CHAT GPT AT THIS TIME. DIANNE, YOU CAN CORRECT ME IF I'M WRONG. WE ACCEPT WHAT JOURNALS TELL US IS ACCEPTABLE AUTHOR LIST THAT WE DON'T GO AGAINST THAT. MIGHT CHANGE IN THE FUTURE. RIGHT NOW FACT THAT CHAT GPT HAS NURSING OR GNOMICS PAPER WONDERING CHAT GPT ENTERPRISE DID THAT AND NOW WE ARE WORKING WITH JOURNAL AND ICMJ. WANT TO COMMENT, DIANNE? >> YEAH. SO, THERE IS ALREADY A LOT OF DISCUSSION ABOUT THIS WHETHER YOU CAN ACTUALLY HAVE A BOT BE AN AUTHOR. YOU KNOW, TYPICALLY, AN AUTHOR IS EXPECTED TO BE RESPONSIBLE FOR A PIECE OF THAT INTELLECTUAL WORK. SO, THAT IN AND OF ITSELF IS JUST ONE OF THE POINTS OF DISCUSSION IN THIS. AS I MENTIONED EARLIER, THERE IS PUBLISHING STANDARDS AND ORGANIZATIONS THAT ARE REVIEWING THIS AND CONSIDERING IT AND THERE SAY LOT OF LITERATURE OUT THERE COVERING IT. PIECE IN NATURE AND WHAMMY PUT OUT RECOMMENDATIONS IN SOME OF THE OTHER LIST SERVES THAT BLOGS ARE POSTING HEAVILY ABOUT THIS AS WELL. IT IS IN ITS INFANCY. WE ARE GOING TO SEE ADDITIONAL, YOU KNOW, KIND OF GUIDE RAILS COME IN ABOUT THIS AND HOW TO CITE IT IF YOU ARE USING IT. COMMENTS HERE ARE ON, YOU KNOW, IS IT VIABLE? WHAT SOURCES IS IT PULLING FROM? ALL THOSE THINGS ARE THINGS TO TAKE INTO ACCOUNT AS WE GO FORWARD. >> HEIDI, STEVE WANTS TO OFFER SOMETHING. >> THANKS. >> SURE. >> PATTI MENTIONED A MINUTE AGO WE ARE LOOKING AT A COLLABORATION WITHNIST AROUND ARTIFICIAL INTELLIGENCE IN RELEVANCE BASED RACHGING WE DO IN PUB MED AND PRESENTATION ALGORITHMS AROUND PRINCIPLE OF FAIRNESS AND ACCURACY AND GETTING OUT IN SORT ORDER AND WHAT IS MOST RELEVANT AND THINGS EQUALLY RELEVANT ARE YOU BEING DISPERSIVE AND CENTRAL AND PRIORITY BASED RANKING FOR EXAMPLE LEARNS FROM ITSELF AND RECOMMENDS THINGS PEOPLE CLIPPED ON AND BECOMES POSITIVE RE-ENFORCED CYCLE AND THINGS IN GRAPH ARE HARD TO PENETRATE EVEN IF EQUALLY RELEVANT. WE ARE TRYING TO MEASURE AND FORMALIZE METRICS OF PERFORMANCE THAT IS SOMETHING I CAN BRING BACK TO BOARD IN COMING MEETINGS. >> YEAH. >> IT IS INTERESTING IF YOU LEARN. NIST RELEASED A SCAFFOLD PLAYBOOK AND TOOLKIT ON AI AND THERE IS CREATIVE WORK WE WILL DO HERE AT CBI AND NLM TRYING TO MAKE SURE WE ARE THOUGHT LEADERS IN THIS AREA IN TERMS OF PERFORMANCE. >> RIGHT. THANKS, STEVE. ONE MORE QUESTION FROM PHILIP? >> YEAH. HATE TO CHANGE SUBJECT BUT PATTI FOUND YOUR INITIAL QUESTION VERY INTRIGUING AND TESTING PERSONAL NOTE LOOKING AT EHR DATA AND POPULATION DATA AND THINKING ABOUT HOW TO ADDRESS COMORBIDITIES AND SYNDROMES. IT -- I'M A CAREGIVER FOR MY WIFE. I HAVE SPENT LOTS OF TIME IN CASE REPORTS AND FINDING -- BECAUSE THEY COVER MULTIPLE CONDITIONS AS OPPOSED TO SIMPLE GUIDELINES. AND FORTUNATELY WE HAVE FOUND A FEW PHYSICIANS WHO HAVE BEENLE TO LISTEN AND OBVIOUSLY STILL READ LITERATURE POST MEDICAL SCHOOL. WE HAVE BEEN SUCCESSFUL TRYING TO HAVE A DECENT QUALITY OF LIFE. NOT A CURE BUT DECENT QUALITY OF LIFE. I WONDER IF THINGS LIKE ALL OF US AND ALSO SOME OF THIS HAS TO BE MASSIVE AMOUNTS OF SYNDROMIC DATA AND QUESTIONABLE AND UNDIAGNOSED CONDITIONS FLOATING AROUND OUT THERE IN THE POPULATION AND HOW SOME OF THAT CAN BECOME BIG DATA TO BECOME AS JIM SAID BIG KNOWLEDGE 10 WHAT WE KNOW OF A CASE REPORT THAT MIGHT BE A FINAL TRIAL AND SYNDROMIC DATA AND CASE REPORTS GIVES US REAL LIFE EVIDENCE THAT SEEMS TO BE MISSING IN MORE TRADITIONAL LITERATURE THAT MIGHT BE AN OPPORTUNITY, DR. BRENNAN AND TEAM THAT IS SOMETHING IF WHERE YOU ARE GOING HEIDI AND WHAT TICKED IN MY BRAIN. >> OKAY. ONE OF THE THINGS NLM DOES IS SHARES EXPERTISE ACROSS CAMPUS FROM AROUND DIFFERENT ISSUES AND DATA MANAGEMENT SHARING ARCHIVING GENOMIC DATA AND CATALOGING LANGUAGE. I HAVE BEEN WORKING WITH ALL OF US TEAM THAT IS ABOUT THEIR CATEGORIZATION OF THEIR POPULATION. THEY HAVE IT -- THEY STILL HAVE A TENDENCY TO CATEGORIZE ON A DISEASE STATE AND HAVE A REMARKABLE OPPORTUNITY TO DESCRIBE PEOPLE FROM A DEMOGRAPHIC PERSPECTIVE OR IF YOU WILL FROM A HEALTH ASSETS AND DEFICITS PERSPECTIVE OPPOSED TO DISEASE AND IS A BIT OF AN UPHILL BATTLE FRANKLY. LITERATURE DEVELOPING PAPER IN NEW ENGLAND JOURNAL SUMMARIZING POPULATION AND TOP 10 CASES 60% OF PEOPLE REPORTING ANY DISEASE REPORT MORE THAN ONE AND ALREADY WE HAVE DISTORTION AND THINK AS WE MOVE FORWARD GETTING BETTER MODELS CHARACTERIZING CONSTELLATION OF THE EXPERIENCE OPPOSED TO OR IN ADDITION TO I GUESS WHERE THERE ARE DEFINITIVE INDICATORS OF DISEASE WILL BE REALLY HELPFUL. >> AGREED. >> WE WILL MOVE ON TO THE LAST WORK GROUP REPORT OUT. I THINK THAT FOR THE STRATEGIC PLANNING I THINK JOE DREW THE SHORT ST STRAW. >> OR LUCKY STRAW DEPENDING ON PERSPECTIVE. >> THERE YOU GO. >> YES. WE HAD AN INCREDIBLY RICH DISCUSSION THAT COULD HAVE GONE WELL BEYOND ALLOTTED TIME AND WE WILL ONLY COVER I WILL SAY A VERY SUPERFICIAL SUMMARY OF THIS. KIMBERLY KINDLY SUMMARIZED ALL OUR POINTS IN A SINGLE SLIDE THAT IS AN AMAZING ACHIEVEMENT AND SHE PRODUCED A LONGER WORD DOCUMENT FOR THE RECORD. IN LOOKING AT THE STRATEGIC PLAN FRAMEWORK, FOCUS HERE WAS REALLY AROUND PROGRAM EVALUATION AND DRIVEN BY TWO MAIN ISSUES. NO. 1, JUST THE OVERALL NEED FOR ACCOUNTABILITY HOLDERS TO SEE THAT PROGRAMS ARE HAVING THEIR STATED INTENT AND THAT INVESTMENTS ARE PAYING OFF IN SOME WAY FOR BENEFIT OF GENERAL PUBLIC. AND SPECIFIC STAKEHOLDERS AND IF YOU WANTED TO GET DOWN TO THE LOWEST COMMON DENOMINATOR IS BEING DRIVEN FOR ALL FEDERAL AGENCIES BY EVIDENCE-BASED POLICY MAKING ACTIVE IN 2018 WHICH IS NOT A BAD DOCUMENT AND WHITE HOUSE RELEASE IN 2021 WAS REALLY FRAMED AROUND TRUST AND GOVERNMENT TO EVIDENCE THE GOVERNMENT PROGRAMS ARE HAVING STATED AND INTENDED IMPACT. SO, WHAT WE TALKED ABOUT IN TERMS OF THAT FRAMEWORK WAS THREE KEY QUESTIONS. HOW DO WE LOOK AT EVALUATION FRAMEWORK? HEIDI PARSED IT OUT INTO THREE BUCKETS. EVALUATING THINGS NLM DOES THAT ARE REALLY CORE PRODUCTS OR PLATFORMS AND THINGS LIKE PUB MED OR CLINICAL TRIALS.GOV AND EVALUATING ACTIVITIES THAT NLM IS AN IMPORTANT CONTRIBUTOR AND NOT MAIN DRIVER AND THIRD AREA THAT I THINK IS REALLY KEY IS WHAT IS AREAS THAT NLM IS NOT CURRENTLY INVOLVED WITH BUT SHOULD BE. WE HAD REFLECTIVE DISCUSSION AROUND IMPORTANCE OF INNOVATION AND OF RISK TAKING AND BIGGEST RISK IN ANY ORGANIZATION IS DOING THINGS AS THEY ALWAYS HAVE DONE AND IS KEY AND THINK WHEN WE TALK ABOUT EVALUATION IT IS NOT JUST BACKWARD LOOKING BUT FORWARD LOOKING LOOKING THROUGH MORE OPPORTUNITIES. MOST EVALUATION FRAMEWORKS WILL YOU SEE IN LOGIC MODEL THAT WHITE HOUSE PUT OUT FOR EVIDENCE ACT AND SLIDES PUT TOGETHER FOR THIS DISCUSSION. THEY ARE FAIRLY LINEAR. I THINK WHAT THEY DON'T REPRESENT IS ALSO AN INTENT OF THE ORGANIZATION TO HAVE FEEDBACK LOOPS ALONG THE WAY AND CONCEPT OF A FORMATIVE EVALUATION. THIS IS NOT LIKE A CLINICAL TRIAL WITH A FIXED PROTOCOL. YOU ARE ACTUALLY USING EVALUATION AT EVERY STAGE OF A PROCESS AS FEEDBACK THAT COULD CHANGE HOW YOU ARE DOING YOUR WORK. WE HAVE FURTHER DISCUSSION, BY THE WAY. LOOKING AT TRANS NIH ISSUES. I WAS REFLECTING ON OPPORTUNITIES WITHIN THE NIH COVID GUIDELINE PROCESS TO BRING IN EXPERTISE. OF NLM INCLUDING ITS EXPERIENCE PUTTING OUT INFORMATION PRODUCTS THAT ARE WIDELY DISSEMINATED AND VIEWED WITH TRUST AND ALSO OF HAVING SPECIFIC ACTIVITIES TO REACH OUT TO HISTORICALLY DISADVANTAGED POPULATION. BECAUSE GETTING INFORMATION OUT WAS A KEY CHALLENGE WITH COVID AND ARE LESSONS IMPORTANT TO INCORPORATE INCLUDING LESSONS FROM EDUCATIONAL DOMAIN AND, YOU KNOW, EDUCATORS THAT TEACH THE WAY THALWAYS HAVE TAUGHT AND THE IS NOT EVIDENCE WHAT IS MORE OR LESS EFFECTIVE AS A TEACHING METHOD THAT COULD BE ADAPTED INTO STRATEGIC PLANNING AND EVALUATION FRAMEWORK AND DISCUSSION WHAT METRICS ARE APPROPRIATE. YOU HAVE TO SOMETIMES LOOK AT THIS IN A MEDI CONTEXT IF MEASURING EFFECTIVENESS OF PLATFORM LIKE CLINICAL TRIALS.GOV AND COUNTING TRIALS IN CLINICAL TRIALS.GOV IS NOT BEST WAY TO EVALUATE IT. IN ESSENCE YOU HAVE TO REGISTER THERE AND NEED MEASURES OTHER THAN THINGS BEING COUNTED EASILY WITHIN PROGRAMS YOU RUN. TRANSITIONING NOW TO SECOND QUESTION. WHERE SHOULD PRIORITY QUESTIONS COME FROM AND WHAT SHOULD THEY BE? I THINK THAT IN A GENERAL AGREEMENT THAT INFORMATION SERVICES AND RESOURCES VIEWED BROADLY AND IN PRIORITY AND INCLUSIVENESS IS A CHALLENGE NOT JUST A SCIENTIFIC OR MEDICAL COMMUNITY THAT NLM IS REACHING OUT TO. WE TALKED A LITTLE MORE ABOUT TRANS-NIH CAPABILITIES AND DISCUSSIONS HAPPENING AT LEVEL OF DIRECTOR'S OFFICE AND CHALLENGE IN LARGE ORGANIZATION TO BREAK DOWN SILOS SEEING OPPORTUNITIES CROSS DIVISIONS AND OTHER POINT WHEN DOG EVALUATION YOU HAVE TO REALISTICALLY CONSIDER CONSTRAINTS THAT MAY BE CONSTRAINTS OF AUTHORITY AND CULTURE AND TRADITION AND ALSO A BUDGET AND FRANKLY OF POLITICS AND NOT ALL QUESTIONS ARE NECESSARILY AS WELCOME IN A GIVEN PERIOD OF TIME AS OTHERS AND LASTLY WHAT STRATEGIC EVALUATION QUESTIONS FOR NLM TO CONSIDER AND HOW TO BE MORE INNOVATIVE AND HOW TO BE MORE EFFECTIVE IN EDUCATIONAL OUTREACH FOR PRODUCTS AND HOW TO INCORPORATE INNOVATION ITSELF AS A MANAGEMENT STRATEGY AND WHAT EVALUATION PROCESS FOR INNOVATION THAT RECOGNIZES INHERENT RISK INNOVATION OR RISKS THAT MUST BE TAKEN. NOT EVERYTHING WILL SUCCEED. SOMETIMES FAILURES HELP TO CONTRIBUTE TO FUTURE SUCCESS. WHAT MAKES INFORMATION SOURCE TRUSTED? AND YOU KNOW WHAT IS IT IN THE SOCIAL NETWORKS AND IN ENGAGEMENT WITH PUBLIC THAT MAKES NLM PRODUCTS TRUSTWORTHY? LASTLY, HOW DO YOU MAKE SURE THAT INFORMATION THAT GETS PUT OUT IS NOT JUST BEING SHELLED BUT ACTIVELY BEING USED? THAT IS A VERY HIGH-LEVEL FRAMEWORK THAT IS LONG AND ENERGETIC DISCUSSION TO HELP TURN IT TO HEIDI AND KIMBERLY TO POINT OUT WHAT AREAS I MISSED AND WE WILL TAKE QUESTIONS. >> VERY COMPREHENSIVE. THANK YOU, JOE. QUESTIONS FOR STRATEGIC PLANNING WORKGROUP? >> ALL RIGHT. I DON'T SEE ANY. WE CAN MOVE TO THE NEXT TOPIC. NEXT TOPIC IS THE REPORT FROM CENTER FOR CYANTIVIC REVIEW AND HEARING FROM DR. NONI BURNS. >> NONI BURNS. >> THANK YOU. >> DR. BURNS ON? >> DR. BURNS WITH US? >> SHE IS PLANNING TO JOIN AT 1:45. MIGHT NOT HAVE LOGGED IN YET. IF NO OTHER DISCUSSION ON STRATEGIC PLAN WORKING GROUP WE CAN MAYBE TAKE A QUICK PAUSE LOOKING -- CAN TAKING A REAL PAUSE OR TAKE -- I MEAN, JUST GENERAL QUESTION YOU HEARD FROM THE FOUR WORKING GROUPS. ARE THERE -- I MEAN, THESE ARE GROUPS WE ARE TRYING TO ENGAGE IN SOLICITING YOUR INPUT ACROSS A WIDE RANGE OF NLM SERVICES PROGRAMS AND RESEARCH AREAS AND SO FORTH. THINGS YOU THINK WE ARE MISSING OR OPERATIONALLY THINGS TO DO BETTER IN WORKING GROUP PREP AND PLANNING AND WE SURVEYED YOU ABOUT 8 MONTHS AGO OR 9 MONTHS AGO AND TRIED TO INCORPORATE CHANGES THERE AND IF YOU HAVE FEEDBACK GENERALLY ABOUT SUBSTANCE OR PROCESS OF WORKING GROUPS WE CAN TAKE A COUPLE MINUTES FOR THAT NOW WHILE WAITING FOR NEXT SPEAKER. >> [INDISCERNIBLE]. >> [INDISCERNIBLE]. >> I TALKED TO OTHER IC DIRECTORS ABOUT HOW THEY RUN COUNCILS AND WHAT THEY DO WITH THEIR TIME. ABOUT HALF OF THEM DO SOME KIND OF BREAKOUT WORKING GROUP MODEL AS WE USE. AND WHAT THEY HAVE FOUND IS THAT FOR PROPERLY PREPARED THEY BECOME A GOOD CONVERSATION POINT FOR THE BOARD BRINGING NEW ENGAGE MENTD IN WAYS AND FINDING THAT ALSO YOU FEEL YOU HAVE ENOUGH PREPARATION OR UNDERSTANDING WHAT YOU ARE TRYING TO GET OUT OF VARIOUS SESSIONS? >> CLIFTI? >> YEAH. QUICK COMMENT. I -- THE BREAKOUTS ARE ALWAYS REALLY WONDERFUL. I ALWAYS LEARN A LOT AND LOVE THE OPPORTUNITY TO CONNECT AND DISCUSS WITH COLLEAGUES WITH MORE FOCUSED TOPICS. I HAVE NEVER BEEN IN THIS CAPACITY IN ONE PANDEMIC TIMES. I HAVE ONLY ATTENDED VIRTUAL WORKINGS. I AM JUST, I GUESS, CURIOUS WHEN WE DO MOVE TO IN-PERSON MEETINGS WILL WE HAVE MORE TIME? HOW DO THEY WORK? >> IN FIRST FOUR WORKING GROUPS THAT LASTED A YEAR BEFORE THE PANDEMIC ACTUALLY HIT US, WE MET FOR A DAY AND A HALF THAT I EXPECT IS WHAT WE WILL DO STARTING IN SEPTEMBER MEANING WE WILL MEET 9 TO 4:30 OR SO ON TUESDAY. THEN 9 TO 12 ON WEDNESDAY. WE H- TWO SESSIONS OF BREAKOUT SESSIONS AND REPORT OUT AND WE ACTUALLY BUILT IN MORE TIME FOR BREAKOUTS. WHAT I -- WHAT I FIND -- I GET TO HEAR WHAT YOU ARE THINKING ABOUT WHAT WE ARE THINKING ABOUT. THAT IS REALLY IMPORTANT. THERE IS PART OF RESPONSIBILITIES AS A BOARD MEMBER READING PAGES OF BOARD ORIENTATION BOOKLET ON PAGE 4:12. YOU SERVE AS FIDUCIARY OVERSIGHT IN FEDERAL INVESTMENT OF NATIONAL LIBRARY OF MEDICINE AND COULD GIVE YOU MORE INFORMATION ABOUT HOW WE ARE DOING WITH WHAT WE ARE DOG AND VERY HAPPY TO DO THAT. SEEMS WORKING GROUPS FOCUS ON SUBSTANCE AND SEMANTICS OF WORK AND OPERATIONS AND LONLISTICS OF IT. CHANGING THAT WE COULD CERTAINLY PROVIDE MORE INFORMATION. >> THANK YOU. >> OTHER PART OF WORKING GROUPS AND WHERE THEY ARE REALLY HELPFUL FRANKLY IS DOCUMENTING AND DEMONSTRATING TO COLLEAGUES AND PUBLIC THAT WE HAVE A WAY GETTING SYSTEMATIC INFORMATION FROM THE PUBLIC AND HAVING WORKING GROUPS WITH REPRESENTATION ACROSS COUNTRY AND HAVING WORKING GROUPS WITH REPRESENTATION NOT HANDPICKED BY BOARD OR SECRETARY AND DIFFERENT PATH GETTING CITIZEN INPUT AND PUBLIC MEETINGS THAT ARE VALUABLE FOR US AND HAVE A WAY FOR PEOPLE TO EXPRESS CONCERNS ABOUT WHAT THEY NEED TO BRING FORWARD TO US. HEIDI, I THINK NONI IS HERE. IF YOU WANT TO PROCEED, WE CAN. >> SOUNDS GREAT. >> HI, NONI. WELCOME. YES. WE SHOULD GET OR MOVE TO THAT PRESENTATION FROM CENTER FOR SCIENTIFIC REVIEW. THANK YOU, NONI. >> GREAT. THANK YOU. I WILL SHARE MY SCREEN. >> LET ME GIVE BACKGROUND ABOUT YOU FOR A MINUTE. >> OKAY. >> NONI IS A COLLEAGUE ANOTHER INSTITUTE AND CENTER DIRECTOR WHO HAS BEEN WITH NIH FOR A WHILE AND WON'T GIVE NUMBER, NONI. SHE HAS BEEN DIRECTOR OF CENTER FOR RESEARCH -- CENTER FOR RESEARCH REVIEW. >> SCIENTIFIC REVIEW. >> SCIENTIFIC REVIEW. THANK YOU. 3 YEARS NONI OR FOUR? >> FOUR NEXT WEEK. >> OKAY. SHE IS A GOOD PARTNER TO NLM IN HELPING US THINK HOW TO EFFECTIVELY SET UP BOTH EXTRAMURAL REVIEW. WE ARE ONE OF THE FEW INSTITUTES RUNNING OWN REVIEW GROUP AND RELIES ON NIH'S REVIEW SERVICES AND HAS BROUGHT INCREDIBLE LEVEL OF INTEGRITY AND PART OF MOTIVATION FOR TODAY AND TRYING TO MAKE SURE WE RELY ON REVIEW GROUPS FOR SCIENCE TO BE GREAT AND REVIEWERS ALSO BRING WITH THEM KINDS OF THINKING AND UNBIASED APPROACHES OR MINIMAL BIASED APPROACHES AND DONE AN AWFUL LOT IN LAST FEW YEARS TO MAKE SURE REVIEW HES COULD BE GOOD ROOEB VIEWERS AND THANK YOU FOR BEING WITH US NONI APPRECIATE YOUR TIME. >> GREAT TO BE HERE AND THANK RS TO YOU AND JERRY FOR THE INVITE WITH ME. LET ME SHARE MY SCREEN. I'M ASSUMING SOMEONE CAN GIVE A THUMBS UP IF YOU CAN SEE IT. >> YES. STILL IN ADDED VIEW, NONI. YOU ARE NOW CLEAR. WHAT? YUP. >> OKAY. EXCELLENT. THANKS VERY MUCH FOR HAVING ME AND LOOKING FORWARD TO TELLING YOU ABOUT CSR'S INITIATIVES TO STRENGTHEN PEER REVIEW AND BROAD VARIETY I WILL RUN THROUGH. I'M OKAY WITH QUESTIONS ON A SLIDE OR AT END. I THINK WE HAVE 15 MINUTES BUILT IN. STARTING WITH MISSION AND SCOPE OF CENTER FOR SCIENTIFIC REVIEW THAT PATTY MENTIONED AS ONE OF 27 INSTITUTES AND CENTERS OF NIH. OUR MISSION IS SINGULAR AND TO ENSURE NIH GRANT APPLICATIONS GET FAIR INDEPENDENT EXPERT AND TIMELY SCIENTIFIC REVIEW FREE FROM INAPPROPRIATE INFLUENCES TO ALLOW NIH TO IDENTIFY AND FUND MOST PROMISING RESEARCH. TO GIVE IDEA OF SCOPE OF OPERATIONS IN FISCAL YEAR 2022, NIH HAD OVER 80,000 APPLICATIONS SUBMITTED AND CSR REVIEWED 61,000 OF THOSE THAT IS MORE THAN 3 QUARTERS AND WE REVIEW BULK OF NIH'S R01S AND BULK OF SMALL BUSINESS APPLICATIONS AND VAST MAJORITY OF FELLOWSHIP APPLICATIONS AND WE COULDN'T DO THIS WITHOUT ASSISTANCE AND VOLUNTEER TIME OF OVER 19,000 REVIEWERS AND WE INTERACT WITH THEM EACH YEAR IN OVER 1200 MEETINGS REVIEW MEETINGS AND STUDY SECTIONS. IN ADDITION TO R01S AND SBR FELLOWSHIPS CSR HANDLES TRANS NIH AND HIGH PROFILE INITIATIVES AND ANYTHING HAPPENING FROM NIH OFFICE OF DIRECTOR OR THAT INCLUDES MULTIPLE INSTITUTES AND I'M SURE YOU WILL SEE SOMETHING THAW RECOGNIZE AT LEAST THAT IS A SMATTERING WHAT WE REVIEWED IN THE LAST FISCAL YEAR. I WILL GIVE A SNAPSHOT INTERACTING WITH NATIONAL LIBRARY OF MEDICINE THAT IS SMALLER LAST YEAR 310,000 APPLICATIONS AND WE REVIEWED 40% OF THOSE AND STANDING SECTION IN NLM THAT IS MAJORITY OF R01S AND R01S GOING TO OUR STANDING COMMITTEES AS WELL AND DO FELLOWSHIPS AND MSBRS ASSIGNED TO NLM AND QUICK SNAPSHOT OF STANDING STUDY SECTIONS RECEIVING NLM R01S THAT IS ORDER CDMA STUDY SECTIONS GET MOST OF THEM AND SMATTERING OF THE REST THAT GOES INTO OTHER PANELS YOU CAN SEE IS BROAD AND COVERS THE SCOPE OF NLM SCIENCE. SO, TALKING ABOUT STRATEGIC PLAN, WE RELEASED THEM. VERY EXCITING AND RELEASED 5-YEAR STRATEGIC PLAN LAST YEAR AND 5 OVERARCHING GOALS ALL OF WHICH SPEAK TO QUALITY OF REVIEW. VERY QUICKLY, GOALS ARE RELATED TO SCIENTIFIC REVIEW GROUPS AND SCIENCE OF STUDY SECTIONS AND BOUNDARIES. THE SECOND GOAL IS RELATED TO REVIEWERS AND DEVELOPING REVIEWERS AND THIRD GOAL OF COURSE RELATED TO STAFF AND 4 NL ONE IS ABOUT PEER REVIEW PROCESS AND FIF NL IS REALLY MORE OF OPERATIONAL PRINCIPLES AND HOW WE OPERATE TO KIND OF ACHIEVE ALL THESE OTHER GOALS THAT IS TO BE TRANSPARENT AND I WILL TALK AND THINK I HEARD THE TAIL END OF CONVERSATION ABOUT ENGAGEMENT WITH COUNCILS AND WITH THIS BOARD. I THINK THAT WE HAVE A LOT OF ENGAGEMENT WITH ADVISORY COUNCIL IN THE SCIENTIFIC COMMUNITY AS A WHOLE. SO, THIS PLAN WAS RELEASED LAST YEAR IN 2022. IT IS VERY MUCH BUILT ON EXISTING STRATEGIC FRAMEWORK IN PLACE SINCE 2019. IF YOU THINK ABOUT IT IN THREE DOMAINS IT THAT MAP ON TO THE STRATEGIC PLAN. ALL THESE EFFECT THE OUTCOME OF PEER REVIEW IT THAT IS WHAT WE ARE ABOUT AND QUALITY OF THE OUTCOMES. STUDY SECTIONS AND STUDY SECTION SCOPE TO ENSURE THAT STUDY SECTIONS ARE RELEVANT AND THEY MAP ON TO THE CURRENT STATE OF THE SCIENCE AND ARE NOT PERPETUATING AREAS OF SCIENCE THAT ARE DYING BY EXISTING AS A STUDY SECTION AND ENSURING THAT OUTPUT IS REASONABLE. AND THAT THE SIZE IS APPROPRIATE. WE DON'T WANT REALLY SMALL BOUTIQUE STUDY SECTIONS THAT ARE OR DON'T ALLOW ENOUGH COMPETITION OF LIKE SCIENCE ACROSS THE BOARD. SO, THAT MAPS ON TO OUR FIRST GOAL OF THE STRATEGIC PLAN AND IF THE STUDY SECTIONS ARE PERFECT, QUALITY OF THE REVIEWERS AND TRAINING OF REVIEWERS AND DIVERSITY OF REVIEW POOL CAN EFFECT OUTCOMES AND WE SPENT TIME. I WILL MENTION SOME OF THIS AS I GO ALONG. WE SPENT A BUNCH OF TIME LOOKING AT AND ADDRESSING ISSUES RELATED TO REVIEWERS. YOU CAN HAVE FANTASTIC WELL-TRAINED REVIEWERS IN A DIVERSE POOL AND HAVE GREAT STUDY SECTIONS WITH PERFECT BOUNDARIES. BUT, THERE MIGHT BE FLAWS IN THE PROCESS. THEY CONTRIBUTE TO LOWERING THE QUALITY OF THE OUTCOMES AND WE HAVE LOOKED CAREFULLY AT PROCESSES AS WELL THAT MAPS ON TO STRATEGIC GOAL 4. KATHY OR PATTY MENTIONED INTEGRITY OF THE PROCESS, FAIRNESS AND BIAS AND NUMBER OF OTHER ITEMS CRITERIA AND REVIEW PROCESS THAT WE ARE CHANGING AND SINCE 2019 HAVE BEEN OPERATIONAL PRINCIPLES MAPPING TO GOALS 3 AND 5 OF THE STRATEGIC PLAN. HOPEFULLY AS I TALK THROUGH SOME OF THIS, YOU WILL SEE SOME OF THIS OCCURRING. YOU WILL SEE WE ARE MAKING DECISIONS BASED ON DATA AND THROUGH STAKEHOLDER ENGAGEMENT. SO, LET'S START WITH THE SCIENTIFIC SCOPE OF THE STUDY SECTIONS. SO, IN 2019 WE LAUNCHED A SYSTEMATIC DATA DRIVEN PROS TOES EVALUATE GROUPS OF STUDY SECTIONS AND GIVEN SCIENTIFIC AREAS AND GOAL WAS TO RESTRUCTURE STUDY SESSIONS AND AROUND 180 STANDING STUDY SECTIONS IN AREAS RANGING FROM BASIC SCIENCE AND BEHAVIORAL AND COMMUNITY HEALTH AND EVERYTHING IN-BETWEEN AND PLAN AT THAT TIME AND MET THIS GOAL WAS TO EVALUATE 20% OF STUDY SECTIONS EACH YEAR WITH EACH STUDY SECTION GETTING A LOOK EVERY FIVE YEARS IS WHAT IT BREAKS DOWN TO. IT IS A TWO-STAGE PROCESS AND FIRST STAGE IS SCIENTIFIC EVALUATION. WE BRING TOGETHER EXTERNAL SCIENTIFIC EVALUATION PANEL AND ASK THEM TO LOOK AT STUDY SECTIONS AND CLUSTER OF SCIENCE AND IDENTIFY EMERG EMERGING AREAS AND DECLINING AREAS AND FAIR AMOUNT OF FREEDOM TO RECOMMEND CHANGES AND SUNSETTING STUDY SECTION AND MERGING CREATION OF STUDY SECTION AND CHANGING GUIDELINES AND GOAL IS TO OPTIMIZE IDENTIFICATION OF HIGH-IMPACT RESEARCH. AFTER THIS STAGE IS DONE, IT GOES TO AN INTERNAL NIH PROCESS EVALUATION STAGE. INTERNAL NIH NOT CSR AND PARTNER WITH INSTITUTE AND CENTER COLLEAGUES AND LEADERSHIP NICS STAKEHOLDERS IN STUDY SESSIONS AND LOOK AT PROCESSES WE CAN EVALUATE INTERNALLY AND WHETHER SIZE IS ADEQUATE AND HOW EARLY STAGE INVESTIGATORS GIVE IN STUDY SECTION AND IS STUDY SECTION EATING YOUNG AND WE HAVE THESE ISSUES COME UP AND LOOKING AT MEETING DYNAMICS WE HAVE SITE VISITS PEOPLE GO TO STUDY SECTIONS THAT ARE BEING EVALUATED THINGS HAPPEN AT MEETING AND MEETING DYNAMICS CONTRIBUTE TO OUTCOMES AND ALL THIS GOES TO PROCESS EVALUATION PANEL THAT RECOMMENDS CHANGES AND GOES TO CSR ADVISORY COUNCIL AS A THIRD STEP. AND MOVES FORWARD IF THE CSR ADVISORY COUNCIL APPROVES IT. ENTIRE PROCESS IS OVERSEEN BY ONE OF THE SCIENTIFIC DIVISION DIRECTORS WHO MOVES OR LEADS THE PROCESS ALL THE WAY THROUGH THE ADVISORY COUNCIL. NOT DONE AFTER THAT. TAKES MULTIPLE STEPS AFTER ADVISORY COUNCIL APPROVAL WITH DEVELOPING GUIDELINES AND MAKING SURE THEY ARE PUBLISHED ON THE WEB AND INFORM COMMUNITY AND MOK REFERRAL OF PAST APPLICATIONS IF THESE ARE VIABLE WITH ENOUGH APPLICATIONS IN THEM AND MOVE STUDY SECTION MEMBERS FROM WHERE EXPERTISE IS NEEDED AND IF SCIENCE IS MOVING AND NEW STUDY SECTIONS ARE HELD AND THIS TAKES 12 TO 18 MONTHS FROM TIME WE INITIATED TO WHEN WE HAVE NEW STUDY SECTIONS THAT IS A THOROUGH AND CAREFULLY DESIGNED PROCESS AND THUS FAR HAVE HAD 12 STUDY SECTION CLUSTERS COMPLETED SINCE 2019 THAT ARE COMPLETED OR IN PROGRESS AND A COUPLE ARE IN PROGRESS AND COUPLE MORE IN A START-UP THIS YEAR. LET'S TALK ABOUT REVIEWER TRAINING RELATING TO CONFIDENTIALITY INTEGRITY AND FAIRNESS AND BIAS MITIGATION AND LAST YEAR WE STOOD UP A NEW OFFICE WITHIN MY OFFICE OFFICE OF THE DIRECTOR OF CSR THAT INVESTS SIGNIFICANT AMOUNT OF RESOURCES IN OUR CENTRALIZED TRAINING PROGRESS AND OFFICE OF TRAINING DEVELOPMENT DIRECTOR IS MARIANNE--ER AND CONTINUING EDUCATION SYSTEMS TOOLS AND SUPPORT STAFF TRAINING IT ALSO COVERS BROAD REVIEWER TRAINING STRATEGIES AND WANT TO TALK A LITTLE ABOUT THAT AND WHAT WE DO THERE AND SINCE LATE 2021CSR HAD BIAS AWARENESS TRAINING MODULE ROLLED OUT FOR REVIEWERS VERY MUCH NOT IMPLICIT BIAS TRAINING MODULE. WE ARE ASSUMING WE CAN ONLY GET REVIEWERS FOR A VERY SMALL SNAPSHOT OF TIME AND WE ARE HOPING THAT THE INSTITUTIONS ARE DOING IMPLICIT BIAS TRAINING AND UNIVERSITIES AS WE DO AT NIH. GOAL HERE IS TARGETING TOWARDS BIASES OCCURRING DURING PEER REVIEW PROCESS. IT HAS A NUMBER OF MODES OF DELIVERING INFORMATION AND INTERACTIVE EXERCISES AND TESTIMONIALS, ET CETERA. 30 MINUTES LONG AND HOW LONG REVIEWERS CAN PAY ATTENTION IN ADDITION OF DOING REVIEWS OF 8 TO 10 APPLICATIONS THEY GET TO DO. WE DELIVER BEFORE REVIEW MEETINGS AND THUS FAR MORE THAN 16,000 REVIEWERS HAVE TAKEN TRAINING REMARKABLY WELRECEIVED FOR SCIENTIFIC COMMUNITY UNUSUAL FOR TRAINING AND PEOPLE DON'T LIKE TRAINING NORMALLY. WE HAVE HAD TREMENDOUS POSITIVE FEEDBACK ON IT. IT INDICATES A REPORT YOU CAN LOOK AT AND SURVEY REPORT THAT WAS CAREFULLY DONE BY CSR. LY SHARE THIS. COMFORT INTERVENING ON REVIEW BIAS AFTER TAKING TRAINING AND PERCENT OF REVIEWERS THOUGHT IT IMPROVED TO A LARGE EXTENT AND MODERATE EXTENT WAS PRETTY HIGH. I WOULD URGE YOU TO LOOK AT REPORT THAT ACTUALLY HAS A LOT MORE DATA AND ANALYTICS IN THERE. AGAIN, ACKNOWLEDGING AN ADVISORY COUNCIL WORKING GROUP DIDN'T DEVELOP IN A VACUUM PEOPLE SERVE AS REVIEWERS AND ARE APPLICANTS AND THROUGH ADVISORY COUNCIL WORKING GROUP DEVELOP SCENARIOS TACKLING AND DEVELOPING CONTENT FOR THIS AND THANKING WORKING GROUP AND NIH STAFF WHO CONTRIB CONTRIBUTED. QUICK SNAPSHOT EVERY SUMMER DO SUMMER INCOMING CHAIR ORIENTAT ORIENTATIONS. CHAIRPERSON SERVE 2 YEAR TERMS AND EVERY TWO YEARS AND EVERY YEAR WE HAVE 90 PEOPLE WHO CYCLE OFF AND ARE CYCLING IN AND HOLD TWO-HOUR FACILITATED SESSIONS THAT ARE VIRTUAL NOW AND COVER BREAD NL OF CHAIRING A STUDY SECTION. MOST IS FOCUSED ON INTERACTIVE VIGNETTE DISCUSSION FACILITATED BY STAFF AND HAVE ON WEBSITE AN EXAMPLE RECORDING OF ONE SUCH THING AVAILABLE TO THE PUBLIC TO SEE HOW WE TRAIN OUR CHAIRS. REVIEW INTEGRITY MODULE TRAINING REVIEWERS ON INTEGRITY MAINTAINING CONFIDENTIALITY OF PROCESS BEFORE AND DURING AND AFTER MEETING AND HAD MODULE DELIVERED TO REVIEWER WHO'S TOOK THIS TRAINING A COUPLE YEARS AING AND WE UPDATED THAT WITH A VERSION 2. THAT IS MUCH MORE INTERACTIVE. WE LEARNED SOME FROM BIAS AWARENESS TRAINING AND HOW WELL IT WAS RECEIVED AND INCORPORATED LESSONS HERE. CONTENT IS BASED FROM ORIGINAL WE WORKED WITH CSR ADVISORY COUNCIL AND WORKING GROUP OF ADVISORY COUNCIL TO DEVELOP REAL LIFE SCENARIOS WHAT YOU SCIENTISTS EXPERIENCE WHEN OUT THERE AFTER HAVING SERVED ON A STUDY SECTION. WE ROLLED THIS OUT LAST FALL IN AUGUST. SINCE THEN, 8700 REVIEWERS HAVE COMPLETED TRAINING AND WE EXPECT THAT NUMBER TO ALMOST DOUBLE AFTER FEBRUARY/MARCH MEETINGS THIS YEAR. I WANT TO TALK A LITTLE ABOUT, WELL, PROCESS FAIRNESS AND BIAS MITIGATION WHAT WE ARE DOING IN ADDITION TO TRAINING REVIEWERS AND CHANGES TO REVIEW CRITERIA AND SCORING SYSTEM I BELIEVE WILL EFFECT FAIRNESS OF POSITIVE REVIEW IN A FAIR WAY. 2021 ESTABLISHED DIRECT BIAS REPORTING MECHANISMS. ANYONE CAN REPORT DIRECTLY TO GABRIEL WHO IS ASSOCIATE DIRECTOR FOR WORKFORCE DEVELOPMENT AND REPORTS TO ME AND THIS INFORMATION IS INCLUDED IN SIGNATURE OF EVERY CSR STAFF PERSON AND TRYING TO GET MESSAGE OUT IF YOU FEEL AS AN APPLICANT OR REVIEWER OR AS A PROGRAM OFFICER YOU OBSERVE DISRESPECT OR OBSERVE A BIASED REVIEW OR FEEL AS AN APPLICANT THERE WAS A BIASED REVIEW YOU CAN SEND A DIRECT REPORT TO US AND WE INVESTIGATE EVERY ALLEGATION THAT IS NOT DONE BY PERSON WHO HANDLED REVIEW BUT BY SENIOR MANAGEMENT AND BY DR. [INDISCERNIBLE]. IF WE AGREE REVIEW IS FLAWED OR BIASED WE WILL REVIEW APPLICATION. THIS HAPPENS AND HAPPENED JUST THIS LAST MONTH WE REREVIEWED AN APPLICATION BECAUSE OF SOMETHING COMING THROUGH THIS BOX AND AGREED WITH APPLICANT. THERE ARE TIMES WE DON'T AGREE SCIENTIFIC DIFFERENCE OF OPINION AND IN THAT CASE WE ASK OR GUIDE THEM THROUGH IC COUNCIL PROCESS AND APPEALS PROCESS IT THAT REMAINS AVAILABLE TO THEM. WHEN FLAWED REVIEW TO LOOK BACK AND REVIEWER LET'S THEM KNOW AND PART IS TELLING REVIEW THER STATEMENT YOU WROTE WAS BIASED THAT REQUIRED US TO PULL REVIEW AND REVIEW APPLICATION AND MOST PEOPLE ARE RECEPTIVE AND APPRECIATIVE THEY WERE TOLD AND IN SOME WAYS WILL FOSTER A CULTURE CHANGES IN REVIEW COMMUNITY AS WE PROVIDE THEM WITH FEEDBACK. I WILL TAKE YOU QUICKLY THROUGH PROPOSED CHANGES FRAMEWORK FOR RESEARCH PROJECT GRANTS AND MAIN GOAL LOOKING AT CURRENT REVIEW CRITERIA FOR RESEARCH PROJECT GRANTS ENSURING MAIN GOAL IS FACILITATING MISSION OF PEER REVIEW IDENTIFYING STRONGEST AND HIGHEST IMPACT SCIENTIFIC RESEARCH. IN LOOKING AT REVIEW CRITERIA WE HAD TWO APPROACHES TO ACHIEVE THIS. ONE, THERE IS LOTS OF ADMINISTRATIVE QUESTIONS ASKING SCIENTIFIC REVIEWERS TO ASSESS THAT WE HEARD FROM REVIEWERS WHY DON'T YOU NEED ME OR CAN'T AGENCY DO THIS AND YOU DON'T NEED MY EXPERTISE OR NEED PHD TO DO WHAT WE ASK REVIEWERS TO DO AND GOAL TO REDUCE DISTRACTION OF ADMINISTRATIVE QUESTIONS THAT ALLOWS PEER REVIEWERS TO FOCUS ON SCIENCE AND DEVOTE RIGOR AND TIME THEY NEED FOR AGENCY FEEDBACK ABOUT NARRATIVE OF THE SCIENCE. OTHER WAY TO IDENTIFY STRONGEST AND HIGHEST IMPACT RESEARCH IS MITIGATING ROLE OF REPUTATIONAL BIAS IN PEER REVIEW PROCESS. REPUTATIONAL BIAS IS ISSUE IN BIOMEDICAL COMMUNITY AND PEER REVIEW IS NOT IMMUNE TO IT OWE BUT MICROKOCHL OF BROADER BIOMEDICAL COMMUNITY WHAT CAN WE DO TO CHANGE CRITERIA IN WAY WHERE THAT DOESN'T COME IN TO PLAY AND FOCUS IS ON SCIENCE AND SEEING EVALUATION OF THE INVESTIGATOR AND ENVIRONMENT TO BE IN CONTEXT OF SCIENCE PROPOSED CAN THEY DO SCIENCE AND ARE RESOURCES PRESENT TO DO SCIENCE? AGAIN, THIS REQUIRED WORK OR TREMENDOUS WORK DONE BY TWO SEPARATE WORKING GROUPS BREAKING OUT OF LARGER GROUP THAT WAS FOCUSED ON CLINICAL AND NONCLINICAL TRIALS. CLINICAL TRIALS ARE 10% OF NIH APPLICATIONS FOCUSED ON DEVELOPING NEW REVIEW CRITERIAS AND MAKING RECOMMENDATIONS AND WORKING GROUP RECOMMENDATIONS APPROVED BY CSR AND ENDORSED WITH MODIFICATIONS BY NIH LEADERSHIP AND WALKING YOU THROUGH QUICKLY AND CURRENT REVIEW OF COURSE MAIN RESEARCH PROJECT GRANT REVIEW CRITERIA WITH OVERALL IMPACTS FROM 1 WHICH IS BEST SCORE TO 9 WHICH IS WORST SCORE ARE FIVE CURRENT REVIEW CRITERIA SIGNIFICANCE INVESTIGATOR APPROACH ENVIRONMENT LEGALLY WE ARE NOT ABLE TO CHANGE THESE. WE HAVE TO CHANGE THEM AND GROUP THEM HOWEVER WE WANT AND SCORE THEM OR NOT SCORE THEM. UP TO THE AGENCY. FINAL PROPOSAL BEING PUT FORWARD IS THESE CRITERIA BE GROUPED INTO THREE FACTORS. FIRST, IMPORTANCE OF RESEARCH. SHOULD BE SCORED AND LIES SIGNIFICANCE AND IMPORTANCE AND IMPORTANT TO THINK ABOUT INNOVATION THAT WAS A CRITERIA PEOPLE STRUGGLED WITH CONCEPTUAL AND TECHNICAL INNOVATION DOES INNOVATION MATTER IF ADDRESSING AN IMPORTANT PROBLEM THAT MIGHT BE A LITTLE BORING? FOR THIS AUDIENCE I WON'T SAY THAT. DEVELOPING A DATABASE IS NOT AS EXCITING PERHAPS TECHNICALLY BUT IS WRONG AUDIENCE BUT IT MAY BE REALLY, REALLY IMPORTANT FOR COMMUNITY AND MIGHT REALLY EFFECT A POPULATION. SIGNIFICANCE AND INNOVATION TOGETHER ALLOWS REVIEWER TO WEIGHT THEM AS NEEDED FOR IMPORTANCE OF RESOURCE. RIGOR AND FEASIBILITY 1 TO 9 SCORE INCLUDES APPROACH THAT IS HERE NOW. BIGGEST CHANGE MADE WAS TO FACTOR 3 THAT IS CALLED EXPERTISE AND RESOURCES INCLUDING INVESTIGATOR AND ENVIRONMENT INSTEAD OF NOW IS SCORED 1 TO 9 EACH CRITERIA SCORED BEING PROPOSED THIS FACTOR NOT BE SCORED BUT BE RATED EITHER APPROPRIATE OR GAPS IDENTIFIED. IF GAPS ARE IDENTIFIED WE WOULD LIKE REVIEWER TO EXPOUND UPON THAT TO EXPLAIN WHY GAP IS. REASON IS ONE THING WE HEAR IN STUDY SECTIONS AND THOSE BEEN ON IT MIGHT HAVE HEARD IT IS SOMEONE IS A LEADER IN THE FIELD AND INVEST GATOR'S SCORE MIGHT BE A 1. FOR SOMEONE THEY DON'T RECOGNIZE WHO IS QUALIFIED TO UNDERTAKE WORK SCORE MIGHT BE 3. WE HEAR COMMENTS AND RESOURCES PRESENT IN BOTH PLACES TO SUPPORT PROJECT AND NOT ASKING PEOPLE TO ASSESS INSTITUTION BUT IT IS THE ENVIRONMENT AND CONTEXT OF THE PROJECT. ADDITIONAL REVIEW CRITERIA REMAIN UNCHANGED AND ADDITIONAL REVIEW CONSIDERATION CURRENTLY EXISTING AND DON'T HAVE BEARING ON OVERALL IMPACT SCORE IS SOME ADMINISTRATIVE ITEMS HAVE BEEN REMOVED FROM FIRST LEVEL PEER REVIEW OR THE PROPOSAL. CURRENT STATUS IS RFI REQUEST FOR INFORMATION MARCH 10 NL, 2023 ENCOURAGE YOU TO LOOK AT IT. BACKGROUND AND DETAILS WE HAVE WEBPAGE AND MAILBOX AND COUPLE BLOGS FROM CSR AND OER THAT IS SAME BLOG CROSSPOSTED AND LOOK AT THAT AND PASS OUT THIS REQUEST TO YOUR NETWORKS THAT IS OPEN TO MARCH 10 NL. WE WOULD LOVE TO HEAR FROM EVERYONE. WE ARE CURRENTLY INCORPORATING RIGOROUS CLINICAL TRIAL PROCESS WITHIN NEW FRAMEWORK AND PLAN TO ENHANCE DIVERSE PROSPECTIVES INITIATED BY BRAIN INITIATIVE AND TRYING TO SEE HOW TO PUT IN FRAMEWORK IN A MEANINGFUL WAY. OKAY. NEXT ITEM IS CHANGES TO REVIEW CRITERIA AND PROCESS THIS TIME NOT FOR RESEARCH PROJECT GRANTS BUT NRSA FELLOWSHIP. TRAINING GRANTED FELLOWSHIPS FOR POST DOCTORAL AND PREDOCTORAL STUDENTS AND CONCERN HERE IS THAT WE HEARD THIS CONCERN FROM THE COMMUNITY AND WE -- IT IS WARN OUT BY DATA THAT WE GOT. NIH AGAIN BASED ON REPUTATIONAL BIAS IS LEAVING OUT PROMISING YOUNG SCIENTISTS THAT THERE PROMISING YOUNG SCIENTISTS PROCESS FAVORING INSTITUTIONS OR SENIORS OR WELL-KNOWN SPONSORS WITH NARROWER EMPHASIS ON WHAT I CONSIDER TRADITIONAL MARKERS OF EARLY ACADEMIC SUCCESS AND SEE INSTITUTIONS MOVING IN THDIRECTN AS WELL AND M CAT HAS BEEN PASS/FAIL. DATA SHOWS FELLOWSHIP APPLICANTS ARE CONCENTRATED WE GET FELLOWSHIP APPLICATIONS FROM SMALL NUMBER OF INSTITUTIONS THIS IS SAYING. MAJORITY OF THEM COME FROM SMALL NUMBER OF INSTITUTIONS. APPLICATIONS FROM THOSE HIGHLY RESOURCED INSTITUTIONS DO BETTER AND THOSE WITH SPONSOR AND IS A SENIOR SCIENTIST DOES BETTER WHERE SPONSOR IS AN ASSISTANT PROFESSOR AND IN SEPTEMBER OF 2021 WE CHARGED A WORKING GROUP OF ADVISORY COUNCIL, AGAIN. ENGAGING WITH COMMUNITY CHARGING THEM TO EVALUATE PROCESS AND MAKE RECOMMENDATIONS STRENGTHENING FAIRNESS AND EFFECTIVENESS OF REVIEW PROCESS AND ACKNOWLEDGING THIS IS A VERY DYNAMIC EXCELLENT WORKING GROUP LED BY TWO MEMBERS AND CO-CHAIR WAS ELIZABETH FROM OUR COUNCIL AND INCLUDING COUNCIL MEMBERS AND NUMBER OF AD HOCS EITHER REVIEWERS OR SPONSORS OR RECIPIENTS OF FELLOWSHIP GRANTS WITH NIH STAFF AND EXPERTISE IN FELLOWSHIPS AND THEY MADE TWO MAJOR RECOMMENDATIONS. FIRST ONE WAS CHANGING FELLOWSHIP APPLICATION INFORMATION THAT IS PROVIDED TO REVIEWERS AND WALKING THROUGH THIS VERY QUICKLY BECAUSE OF TIME. ONE, ELIMINATE GRADES MOST POPULAR RECOMMENDATION SOLICITING FEEDBACK PRIOR TO DEVELOPING RECOMMENDATIONS REVISE APPLICANT SECTION TO BET EAR LINE IT AND ALLOW STUDENT HIM OR HERSELF TO PRESENT IDEAS AND THINKING IN A COHESIVE WAY AND DON'T CHANGE CURRENT RESEARCH TRAINING PLAN THAT EXISTS AND RECOMMEND REVISING SPONSORS AND COLLABORATOR'S CONSULTANT SECTION TO REVIEW WITH CRITERIA AND IMPORTANT THING INSTEAD OF SPONSOR'S TRACK RECORD THEY WANT TO LEARN ABOUT SPONSOR TRAINING MENTORSHIP APPROACH AND PLAN FOR A PARTICULAR TARGETED STUDENT. REVISION OF LETTERS OF SUPPORT CURRENTLY LETTERS OF SUPPORT TEND TO BE MORE BOILER PLATE AND CUT AND PASTE AND REQUIRES TRAINEE SPECIFIC QUESTIONS TO ACTUALLY MAKE IT EASIER FOR REVIEWERS TO DISTINGUISH DIFFERENTIATE AND EVALUATE APPLICANTS. THEY PROPOSE OPTIONAL STATEMENT OF SPECIAL CIRCUMSTANCES AND IMPORTANT IT REMAIN OPTIONAL AND ALLOWS THOSE WHO ARE IN LABS WHERE PROGRESS MIGHT BE HINDERED FOR ANYTHING ILLNESS OR PANDEMIC OR HARASSMENT THEY MIGHT HAVE HAD TO MOVE LABS AND HAVE A CHANCE FOR A SHORT STATEMENT EXPLAINING SPECIAL CIRCUMSTANCES THAT IS THE RECOMMENDATIONS CHANGES TO CURRENT FELLOWSHIP APPLICATION WE ARE RECOMMENDING AND SECOND RECOMMENDATION IS CHANGING CURRENT FELLOWSHIP REVIEW CRITERIA AND FOCUS HERE IS GETTING AWAY FROM TRACK RECORD AND REPUTATION THAT IS HEAVY INFLUENCE OF INSTITUTION AND SPONSOR WHO ARE VERY IMPORTANT BUT NOT JUST BECAUSE OF THEIR TRACK RECORD; RIGHT? OR HOW MUCH FUNDING INSTITUTION HAS. HERE, FIVE HAVE BEEN COLLAPSED IN THREE AND INSTITUTION GOES INTO A TRAINING RESOURCES AND SCIENTIFIC RESOURCES SPONSOR QUESTIONS GO UNDER SCIENCE AND SCIENTIFIC RESOURCES AND FELLOWSHIP APPLICATION REMAINS. FOCUS IS PE 10 SHAL OF APPLICANT AND SCIENCE MAKES SENSE? STRONG AND WHETHER A STRONG TARGETED TRAINING PLAN. WORKING GROUP RECOMMENDATIONS WENT TO FULL CSR ADVISORY COUNCIL AND DID TREMENDOUS WORK IN QUICK TIMEFRAME AND HAVE HAD 12 MEETINGS TO DEVELOP GUIDELINES AND THERE IS MORE DETAIL THAT WAS ENDORSED BY NIH LEADERSHIP WITH MINIMAL CHANGE R THAT IS CURRENTLY UNDERGOING FURTHER DEVELOPMENT AND READING FULL REPORT AND DATA ON FELLOWSHIP REVIEWS THIS IS AVAILABLE. WE WILL -- I THINK THE POWERPOINT IS AVAILABLE TO YOU IF NOT ALREADY. OKAY. FINALLY, I WANT TO TALK ABOUT THE REVIEW COMMITTEE OR SORT OF BROADENING AND DIVERSIFYING POOL OF REVIEWERS. STARTING WITH OUR EARLY CAREER REVIEWER PROGRAM THAT IS REVAMPED IN 2020 AND IT IS BASED ON RECOMMENDATIONS AGAIN. THE FIRST CSR ADVISORY COUNCIL WORKING GROUP I CONVENED, THAT PROGRAM REVAMPED UPDATING ELIGIBILITY MAKING IT CONSISTENT AND THESE THAT ARE ECRS IT THAT WE USE EACH MEETING PRACHLT YOU CAN GET INFORMATION HERE AND JUNIOR FACULTY THAT ARE IN YOUR DEPARTMENTS I ENCOURAGE THEM TO APPLY FOR THIS PROGRAM AND IF DON'T HAVE R01 AND ARE JUNIOR THEY ARE ELIGIBLE AND CERTAIN ELIGIBLE CRITERIA YOU CAN LOOK AT. WE DEVELOPED ALSO A TOOL A REVIEWER FINDER TOOL FOR SCIENTIFIC REVIEW OFFICERS TO FIND WHAT WE CONSIDER LESSER KNOWN REVIEWERS. NIH HAS OUR DATABASES THAT IS POPULATED WITH THOSE WHO ARE FUNDED BY NIH. IT IS THE POOL THAT OUR STAFF TENDS TO DIP INTO BECAUSE IT IS NATURAL TO DO THAT. WE HAVE BEEN DOING THAT. THIS IS AN EFFORT TO BROADEN THAT POOL BY COMBINING MULTIPLE DATA SOURCES LIKE OTHER AGENCY FUNDED THAT I BELIEVE WE HAVE DOD AND NSF AND ASKED SCIENTIFIC SOCIETY FOR BETTER RECOMMENDATIONS GOING THROUGH HERE AND GROUP CALLED FUNDED UNDERUSED PIS PEOPLE FUNDED BY NIH. NOBODY ASKED THEM TO REVIEW AND HAD LISTENING SESSIONS IN 2020 I HEARD FROM PEOPLE. I HAVE TWO GRANTS NOBODY HAS CALLED ME TO REVIEW AND SHOULD NEVER BE HAPPENING. WE HAVE HAD A NUMBER OF DIFFERENT RECOMMENDATIONS FROM ICS, ET CETERA, THAT HAVE BEEN PUT IN HERE THAT ALLOW AND WITH EXPERTISE THAT ARE LISTED THAT ALLOWS US TO DO QUICK SEARCHES ON EXPERTISE AND IDENTIFY NEW REVIEWERS. WE WANT NEW REVIEWERS AROUND THE TABLE IN ADDITION TO THOSE WHO ARE EXPERIENCED AND FOR TOOLS, AGAIN, THEY ARE NOT GOING TO BE ONLY THING THAT WORKS. TOOLS ARE VERY NICE TO HAVE. OUR STRATEGIES HAVE FOCUSED ON A CULTURE SHIFT INTERNALLY. SO, SOME OF OUR STRATEGIES BEGAN WITH MAKING SURE THAT WE EMPHASIZE WHY IT IS CRITICALLY IMPORTANT FOR NIH TO HEAR DIVERSE PERSPECTIVE IN ORDER FOR BEST AND MOST DISRUPTIVE AND NOVEL SCIENCE TO BE IDENTIFIED AND PEER REVIEWED AND IS OTHERWISE DERIVATIVE AND INSULAR AND HEAR SAME MODELS EVERYBODY IS TRAINED BY SAME PERSON AND IMPORTANT FOR US TO REALLY GET FRESH PERSPECTIVES INTO THE REVIEW PROCESS. AND AGAIN LICHGING TO QUALITY OF REVIEW COMMITTEE IS IMPORTANT AND HIGHEST QUALITY REVIEW COMMITTEES ARE NOT THOSE WITH MOST SENIOR AND MOST HIGHLY FUNDED REVIEWERS BUT THOSE WHO HAVE REFRESHED BROADER PANEL BROADLY DIVERSE AND INCLUDES DIVERSITY CAN BE NOT JUST DEMOGRAPHIC BUT IS DEMOGRAPHIC AND COULD BE SCIENTIFIC PERSPECTIVE AND CAREER STAGE AND WHAT EXPERIENCE WE WANT PEOPLE EXPERIENCED IN PEER REVIEW AND NEW REVIEWERS ON THERE TOO WE WANT AND IS IMPORTANT TO HAVE ORTHOGONAL PERSPECTIVES ON THESE COMMITTEES. FOR 180 STANDING STUDY SECTION MEMBERS AND THOSE WHO SIGN ON FOR FOUR 6-YEAR TERMS IT THAT PROCESS IS THOROUGH AND VETTED AT DIFFERENT LEVELS A PROVE EVERY NOMINATION SLATE AND GOES TO 1 BUILDING NIH DIRECTOR AND WITH THAT LEVEL OF OVERSIGHT WE ARE DOG OKAY IN DIVERSITY. I WILL SHOW YOU NUMBERS IN A BIT. OUR CONCERN WAS THAT THE SPECIAL EMPHASIS PANELS THAT ARE COMMITTEES GETTING FORMED WITH NOT MUCH NOTICE AND ARE NOT MEMBERS VETTED THROUGH MULTIPLE LEVELS ABOUT YOU STAFF CALLS AND INVITES PEOPLE OVER AND WHEN YOU DO THAT WE ALL TEND TO RELY ON MENTAL ROEL ADECHL WHO I KNOW OR DOES MASS SPECK AND THINK OF THREE OR FOUR PEOPLE YOU KNOW AND GIVE CALL AND GO THROUGH LIST GOAL LOOK AT SPECIAL EMPHASIS COMMITTEE AND RAISING COLLECTIVE AWARENESS WITH STAFF AND SETTING EXPECTATIONS AND DATA SHARING IS IMPORTANT TO SHARE DATA ON DIVERSITY WITH STA STAFF. TOOLS THAT I MENTIONED AND PEER-TO-PEER TRAINING IN BROADER RECRUITMENT STRATEGIES HAS BEEN VERY EFFECTIVE. TO GIVE DATA IN YELLOW BENCHMARKED APPLICANTS IN CSR IS 60,000 APPLICANTS WE GET A YEAR PER ROUND DATA AND WOMEN ARE AROUND 34% HASN'T CHANGED A LOT AND RED LINE IS MEMBERS OF STANDING STUDY SECTIONS ENRICHED WOMEN MORE THAN APPLICANT POOL THAT IS GOING UP FROM LOW 40 OR 41 OR SO TO 44%. SPECIAL EMPHASIS PANEL REVIEWERS IS WHERE WE SEE A REAL CHANGE IN BEHAVIOR AND RECRUITMENT THAT HAS BEEN GOOD. SOME STRATEGIES AND AWARENESS ON PART OF STAFF HAS WORKED AND UNDER-REPRESENTED MINORITIES YELLOW LINE IS TOTAL PERCENT OF UNDER-REPRESENTED MINORITIES AROUND CSR APPLICANTS 8.5% AND STANDING STUDY SECTION MEMBERS HAS BEEN ENRICH AND THAT EVEN WENT UP IN A BUMP. WE NOW IN LAST ROUND ARE 15.6% PANELS OF MEMBERSHIP THAT IS UNDER-REPRESENTED MINORITIES. WITH SPECIAL EMPHASIS PANELS ARE ONES THAT ARE A COMPLETE CREDIT TO STAFF AND REAL EFFORT MADE TO -- IT WAS LOWER THAN APPLICANT BENCHMARK. IT HAS GONE UP. OUR PLAN IS TO CONTINUE AND, YOU KNOW, PAY SUSTAINED ATTENTION TO GENDER DIVERSITY AND DEMOGRAPHIC DIVERSITY ON ALL OUR PANEL AND IF YOU WANT TO KNOW MORE, THERE IS A LOT MORE DATA THAT WE HAVE PUT OUT THERE. ANALYTICS SORT OF DIVERSITY DATA FOR YEARS AND IN FACT WE HAVE AN IF NEW WEB PAGE AND I INVITE YOU TO VISIT THAT. THAT IS IT. I CAN STOP SHARING. HAPPY TO TAKE QUESTIONS. >> THANK YOU. THAT WAS JUST AMAZING TO SEE PROGRESS. I'M SO DELIGHTED FOR ALL OF THE CHANGES AND EFFORTS. I WILL OPEN IT UP FOR TWO OF OUR DISCUSSANTS TO ASK. >> EXCUSE ME, HEIDI. >> SORRY. >> I NEED TO LET YOU KNOW I HAVE TO LEAVE TO GO TO RONLER'S EVENT. COLLEAGUE'S FAREWELL EVENT I WILL BE BACK AND BE IN CHARGE OF THE ROOM HERE SO BEHAVE. >> THANK YOU. THANK YOU, PATTY. >> SO, I WANTED TO JUST CALL ON OMALA AND MICHELLE IF EITHER WANT TO KISS US OFF WITH THOUGHTS OR COMMENTS OR QUESTIONS FOR NONI? SFWL THANK YOU. >> THANKS FOR A VERY INFORMATIVE AND THOUGHTFUL PRESENTATION. I WAS REALLY HEARTENED BY ATTEMPT TO EXPAND RENEWAL ESPECIALLY AND DON'T KNOW IF ANYBODY -- OKAY. THERE IS BACKGROUND NOISE. WAYS HEARTENED BY ATTEMPTS TO EXPAND REVIEWER POOL TO CHANGE EMPHASIS ON REVIEW CRITERIA LIKE YOU SAID LEGALLY YOU CAN'T TAKE AWAY SOME CRITERIA I WONDER RESEARCH FOCUSED ON HEALTH DISPARITY AND WILL COME FROM A PERSPECTIVE THAT I DIDN'T SEE EXPLICITLY MENTIONED BUT PROBABLY FEEDS INTO YOUR APPROACH TO DIVERSITY THAT -- DIVERSIFYING REVIEWER POOL THAT IS SOMETIMES TRYING TO ENSURE KWAL REVIEWERS FAMILIAR WITH THE HEAL NL CARE SAFETY NET AND ISSUES WITH DISPARITIES ON SETTING IN REVIEW PANELS AND ARE PEOPLE WHO MIGHT NOT HAVE. THEY DO RESEARCH BUT MIGHT NOT HAVE NIH OR DOD OR NSF GRANTS ABOUT YOU STATE OR FOUNDATION GRANTS. I DON'T KNOW IF THERE IS A WAY TO PULL PEOPLE IN SO THERE IS A BROADER RANGE OF PERSPECTIVES AND PEOPLE ARE NOT -- YOU KNOW, REVIEWS WHERE SOME REVIEWERS ARE JUST NOT FAMILIAR WITH ISSUES IN SAFETY NET. IT IS NOT THEIR EXPERIENCE OR CAREER EXPERIENCES. >> THANK YOU FOR THAT QUESTION THAT IS VERY RELEVANT. ONE OF OUR DIVISIONS, DIVISIONS OF BEHAVIORAL POPULATION SCIENCE RESEARCH INCLUDES NUMBER OF STUDY SESSIONS COVERING THIS AREA AND THROUGH OUR INQUIRE PROCESS, RESTRUCTURING STUDY SECTIONS I THINK THAT THEY HAVE HEARD BOTH FROM THE COMMUNITY AS PART OF THE INQUIRE RECOMMENDATIONS AND THEY HAVE TAKEN ACTION IN RESTRUCTURING STUDY SECTIONS AND INVITE YOU TO LOOK AT THOSE SO I CAN PUT NEW TOUCH WITH DIVISION DIRECTOR HAPPENEDLING THAT AREA AND YOU CAN HAVE A CONVERSATION AND KNOW THAT THIS IS NOT FIRST TIME I'M HEARING IT AND HEARING FROM OWN LEADERSHIP IN THAT AREA. THEY ARE WORKING WITH NIMHD AS WELL WITH HEALTH DISPARITIES AND IS ENGAGING WITH THEM. >> GO AHEAD. NSF WE ARE INTERESTED IN ALL WAYS THAT YOU ARE WORKING ON PEER REVIEW AND WE ARE WORKING ALSO ON OURS. ONE OF THE THINGS THAT HAS COME UP A LOT I WAS INTERESTED IN DISCUSSION WITH PANELS AND INTERNALLY AND I WAS INTERESTED TO SEE IT IN CHANGES TO FELLOWSHIP APPLICATION. IT IS THE OPTIONAL STATEMENT OF SPECIAL CIRCUMSTANCES RELATIVE TO COVID. RIGHT? WE ALL THINK THIS WILL HAVE A LONG TAIL OF IMPACTS. RIGHT? AND IT IS INTERESTING IN THE CONVERSATIONS THAT WE HAVE HAD, SOME PEOPLE REALLY WANTED A CHANCE TO TALK ABOUT THAT. OTHER PEOPLE ARE CONCERNED IT COULD BE HELD AGAINST THEM. I DON'T KNOW HOW -- WHAT YOU HAVE GOT -- LIKE WHAT THE BIGGER CONTEXT YOU ARE WORKING WITH IS. OR HOW YOU ARE GOING TO HANDLE THAT. BECAUSE I THINK IT IS A REALLY HUGE GENUINE CONCERN FOR FOLKS THAT, YES. PERHAPS IT SHOULD BE CONSIDERED. NO WE DON'T NECESSARILY WANT TO DISCLOSE AND WE HAVE HAD ALL KINDS OF SUGGESTIONS LIKE A CHECK BOX AND DON'T HAVE TO TALK ABOUT DETAILS AND SAYS YOU ARE IMPACTED BUT THAT HAS ITS OWN ISSUES. SO, I'M WONDERING HOW YOU WILL HANDLE THOSE THINGS. >> THAT IS A GOOD QUESTION. WE HAVE HAD DISCUSSIONS ABOUT THAT INTERNALLY AND I WANT TO SEPARATE OUT COVID IMPACT STATEMENTS AND THINK RIGHT NOW AS WE STILL ARE UNDER PUBLIC HEALTH EMERGENCY AND COVID FLEXIBLES ARE APPLYING FOR BROADER RANGE OF GRANTS AND APPLICANTS ARE ALLOWED TO EXPLAIN THROUGH BIOSKETCH AND ALLOWED TO EXPLAIN THAT, THAT COULD BE CONSIDERED IN REVIEW WHAT IS PROPOSED HERE IS SPECIFIC TO FELLOWSHIP GRANT APPLICATIONS AND WITH FELLOWS AND TRAINEES IS BEYOND JUST COVID. IT IS ALSO OPTIONAL. I SAID OPTIONAL BECAUSE OF THAT REASON AND BECAUSE OF THE REASON THAT WE DON'T WANT IT TO BECOME A KIND OF REQUIREMENT LIKE EVERYONE HAS TO PUT A STORY IN THERE OR YOU DON'T GET A -- YOU KNOW, SO, THAT WILL HAVE TO BE MANAGED CAREFULLY THAT IS DISCUSSED WITHIN NIH AND THINKING THROUGH IMPLEMENTATION OF THIS, WE HAVE TO CONSIDER HOW WE WILL APPROACH IT AND IS A GOOD QUESTION AND STUFF THAT WE HAVE BEEN DISCUSSING AS WELL. >> YEAH. >> NOT SURE THERE IS A GREAT ANSWER TO HOW TO MAKE IT HAPPEN. >> WILL REQUIRE TRAINING OF REVIEWER. >> REVIEWER OR STAFF TRAINING. >> YEAH. >> IT IS NUANCED. YEAH. >> YEAH. >> THANK YOU FOR QUESTIONS. I WILL ASK ONE. I WILL OPEN IT UP TO EVERYONE ELSE. ONE FRUSTRATION I HAVE HAD OVER THE YEARS IS NUMBER OF INVESTIGATORS WHO DON'T SHARE DATA ADEQUATELY. NO GRATE WAY TO MAKE THAT TRANSPARENT IN REVIEW PROCESS SORT OF BAD DEMONSTRATIONS OF DATA SHARING FROM PAST FUNDED GRANTS AND WONDERING IF WE HAVE TO PUT A PLAN FOR DATA SHARING INTO OUR GRANTS THAT IS PART OF THE PROCESS AND I THINK IS TERRIFIC. NO ASPECT OF THAT IS ANY SUMMARY OF YOUR PAST DATA SHARING EFFORTS AS A WINDOW INTO WHETHER THEY HAVE DONE IT WELL. I WONDER IF IT IS BEING CONSIDERED IN ASPECTS OF SCIENTIFIC REVIEW ESPECIALLY WITH POLICIES AROUND DATA SHARING REQUIREMENTS AND OPEN SCIENCE AND THINGS LIKE THAT. >> I WILL TALK A LITTLE ABOUT THIS. I'M SURE YOU KNOW THERE IS NEW DATA SHARING AND MANAGEMENT POLICY EFFECTIVE FOR ANYTHING THAT IS ARRIVING OR ARRIVES STARTING IF JANUARY; RIGHT? IN ESSENCE, ASSESSMENT OF THAT IS TAKEN OUT OF PEER REVIEW AND DOESN'T INTUITIVELY MAKE SENSE IF YOU WANT SOMETHING IMPORTANT AND CONSIDERED YOU WANT IT IN PEER REVIEW THAT CAN'T ANSWER OR YOU CAN'T RELY ON 19,000 PEER REVIEWERS TO HELP AN AGENCY COMPLY WITH POLICIES;ING RIGHT? DOESN'T DO A VERY GOOD JOB. IF THEY LOVE SCIENCE THEY WILL DISMISS ANYTHING ELSE WE ASK THEM TO DO THAT IS WHAT HAPPENS AND THERE IS ADMINISTRATIVE STUFF. IT GETS SORT OF DISMISSED AND THERE IS NOT ENOUGH BAND WIDTH OR TIME. SO, THE CHOICE WAS MADE TO HAVE THAT DONE BY THE AGENCY. I THINK THAT NIH WILL DO IT BETTER. THEY WILL DO IT MORE CONSISTENTLY AND HOPE IN THE UPCOMING NEW POLICY YOU WILL SEE A CHANGE WHERE MORE ACCOUNTABILITY IS BUILT IN FOR INVESTIGATORS WHO DON'T SHARE DATA. HARD TO DO EVERYTHING IN PEER REVIEW THAT CAN IDENTIFY STRONG SCIENCE. >> I COMPLETELY AGREE WITH THAT, THAT IT IS BETTER TO HAVE THEIR BE OBJECTIVE INTERNAL SORT OF APPROACH THAT QUESTION IS CAN OUTCOME OF THAT BE CONSIDERED DURING REFUNDING INVESTIGATOR. WHAT IS LOOP TO GET INFORMATION BACK? >> RIGHT. NOT FULLY FAMILIAR WITH NEW -- IT DOESN'T OVERLAP WITH REVIEW VERY MUCH AND NOT FULLY FAMILIAR WITH POLICY AND MY UNDERSTANDING IS IN RPPR AND IN PROGRESS REPORTS, AGENCY IS GOING TO LOOK AT HOW INVESTIGATOR IS DOING SHARING DATA. YOU KNOW? THAT SHOULD BE CONSIDERED IN FUNDING DECISIONS AND REFUNDING DECISIONS AND NEGOTIATIONS AND IS OUT OF THE HAND OF PEER REVIEW COMMITTEE AND OUT OF THE HANDS OF INSTITUTE AND PROGRAM STAFF AND JERRY CAN SPEAK TO THAT TOO POSSIBLY. >> I CAN SPEAK BASICALLY BY CONFIRMING THAT AND THINK THE PROCESS WILL CURRENTLY BE MONITORED BY PROGRAM STAFF REVIEWING PROGRAM REPORTS AND AFTER GO THROUGH CSR REVIEW AND DON'T EXPECT INFORMATION WILL FEED INTO FUTURE REVIEWS THAT WILL BE PROBABLY WHEN WE SAY IT IS POSSIBLE THERE COULD BE A PROCESS THAT EXISTS WITH NIH PUBLIC ACCESS POLICY. IF YOU DON'T HAVE IDENTIFIERS FOR YOUR PUBLICATIONS INDICATING THAT IT IS IN PUB MED CENTRAL WE CAN SUSPEND PROCESSING ADMINISTRATIVELY OF APPLICATION OR RENEWLEAL AND THAT IS NOT QUITE REVIEW PROCESS ITSELF BUT ADMINISTRATIVE PROCESS TO HELP ENSURE COMPLIANCE WITH OTHER POLIC POLICIES. SFWL THANK YOU, JERRY. >> IMPACTING THIS YOU HAVE MORE EARLY STAGE INVESTIGATORS ALREADY FUNDED OR LOOKING AT DEMOGRAPHICS OF REVIEWERS OR GEOGRAPHIC VARIATION OF REVIEWING INSTITUTIONS? HOW DO YOU BRAN PLAN TO ASSESS THAT CHANGES MADE ARE WORKING THE WAY THAT YOU INTEND IT? >> THANK YOU FAIRLY STANDARD QUESTION HARD TO LEARN BUT I WILL ANSWER AND ANSWER IS ALL OF THE ABOVE AND EXPECT IF WE HAVE NEW PEER REVIEW CRITERIA CHANGES THAT DISTRIBUTION OF INSTITUTIONS IN TOP SCORING GRANTS WOULD BE DIFFERENT FROM WHAT IT WOULD BE NOW; RIGHT? GLOBALLY WITH LARGE N WOULD TAKE SOME TIME TO GET THAT EVALUATION DONE. THAT SAID THERE IS EFFORTS THAT ARE UNDERWAY; RIGHT? ENGAGEMENT ON PROGRAM SIDE AND NIH UNITE ENGAGING WITH INSTITUTIONS TO ENCOURAGE TO SUPMITT MORE GRANTS GRANT SUBMISSION RATE IS LOW FOR MINORITIES. EFFORTS ARE UNDER WAY THAT ARE SIMULTANEOUS AND REVIEW CHANGES WE MADE BUT WILL ONLY BE PARTIALLY EFFECTIVE THERE. ALL THOSE ARE LOOKING AT BREAD NL WHAT COMES OUT OF PEER REV REVIEW. >> WE ARE RIGHT AT THE HALF HOUR. THAT MEANS IT IS TIME FOR OUR BREAK. SO, WE ARE GOING TO BREAK FOR 15 MINUTES AND REJOIN PROMPTLY AT 2:45. WE WILL TALK ABOUT AUTOMATED INDEXING OF MEDLINE CITATIONS AND EVERYONE PLEASE REJOIN IN 15 MINUTES. THANK YOU. >> VERY GOOD. THANK YOU, NONI, FOR JOINING TODAY. >> YES. THAT WAS TERRIFIC. THANK YOU. >> I'M ASSOCIATE DIRECTOR OF LIBRARY OPERATIONS AND EXCITED TO DO AN UPDATE THAT IS PART OF BROADER MEDLINE 2022 INITIATIVE AND THIS EFFORT HAS BEEN YEARS IN THE MAKING AND SUPPORTED BY HARD WORK AND DEDICATIONS ON DOZENS OF NLM STAFF MEMBERS AND DELIGHTED TO JOIN VIRTUAL PODIUM TODAY JIM IS ACTING CHIEF OF APPLIED CLINICAL AND FORMATICS BRAFRNL PART OF NATIONAL CENTER FOR BIOMEDICAL COMMUNICATIONS. NEXT SLIDE, PLEASE. BEFORE JIM AND I GET STARTED BRIEF AGENDA OUTLINE FOR SESSION TODAY COVERING MEDLINE AND HISTORY BEHIND IT AND MEDLINE 2022 INITIATIVE INCLUDING AUTOMATED INDEXING AS A MAJOR OBJECTIVE AND HANDING TO JIM TO DISCUSS HISTORY OF NLM MEDICAL TEXT INDEXER, MTI AND CLOSE OUT SESSION PROVIDING UPDATES OF IMPLEMENTATION OF AUTOMATED INDEXING AND GLIMPSE INTO WHAT WE ARE WORKING ON NEXT. NEXT SLIDE. JUMP IN WITH AN OVERVIEW OF MEDLINE NEXT LINE. STARTING WITH FUNDAMENTAL QUESTION WHAT IS MEDLINE FOR THOSE NOT FULLY AWARE MEDLINE IS NLM'S PREMIERE DATABASE THAT CONTAINS 30 MILLION CITATIONS TO SELECTED JOURNALS AND ARTICLES AND LIFE SKRIENSS CAN CONCENTRATION OF BIOMEDICINE. OF 35 MILLION ARTICLES CURRENTLY IN PUB MED. 86% ARE FROM WHAT WE CALL MEDLINE SUBSET THAT IS COMPRISED OVER 5,000 TO BE EXACT 5,280 IS IT JOURNALS FOR INDEXING. ONE OF NLM'S FEDERAL ADVISORY XHU COMMITTEE AND REVIEW FOCUSES ON CRITICAL ELEMENTS OF JOURNAL SCOPING COVERAGE AND EDITORIAL PROCESSES AND SKENETIVIC MENL LONL CAD RIGOR AND ADMINISTRATION IMPACT. NEXT SWL SLIDE. ONE OF THE MOST IMPORTANT ASPECTS OF MEDLINE JOURNAL CITATIONS IS PROVIDING ADDITIONAL METADATA BEYOND WHAT IS SUPPLIED BY PUBLISHER AND SIGNIFICANT METADATA INDEXING MEDICAL SUBJECT HEADINGS OR MESH AND MESH TERMS DESCRIBE CONCEPTS OF ARTICLES AND CONCEPTS COULD BE FURTHER DESCRIBED USING SUBHEADINGS. PREVENTION OF DISEASE OR ADVERSE EFFECTS OF A DRUG. MEASURES ARE USE THE TO DESCRIBE PUBLICATION TYPE OF ARTICLE RANDOMIZE THE CONTROLLED TRIAL FOR SYSTEMATIC REVIEW. MESH ALSO SUPPORTS RETRIEVAL OF RELEVANT ARTICLES IN PUB MED EXPLICITLY USING TAG SEARCHES AND IMPLICITLY IN QUERY EXPANSION OF UNTAGGED SEARCHES USING AUTOMATED MAPPING IN PUB MED AND ADDITION TO ETREEFL SUPPORT IN PUB MED USED BY MANY WHO DOWNLOAD MEDLINE SUBSETS AND DATA SETS WE PROVIDE AND VENDORS INCLUDING MESH CONTENT IN PRODUCTS AND RESEARCHERS DOING DATA MINING BIBLIOMETRIC AND SEMANTIC ANALYSIS ARE USE CASES OUT THERE FOR FREE WELL STRUCTURED LARGE DATA SETS LIKE MEDLINE. NEXT SLIDE. MEDLINE CELEBRATED RECENTLY 50 YENL ANNIVERSARY TWO YEARS AGO AND THIS SLIDE ILLUSTRATES MEDLINE 2022 INITIATIVE AUTOMATED INDEXING IS A KEY COMPONENT IT IS DERIVED FROM -- GROUNDBREAKING PUBLICATION INDEXING MEDICAL JOURNAL ARTICLES AND MEDICAL HISTORIANS DESCRIBED INDEX AS QUOTE AMERICA'S GREATEST CONTRIBUTION TO MEDICAL KNOWLEDGE AND SINCE CREATION OF INDEX MEDI CUSS WE HAVE INDEXES CATALOGING AND CURATING AND MESH VOCABULARY IN THE 60S AND MEDLINE IN 1971 AND PUB MED IN 1997 MAJOR ADVANCEMENT IS TEXT INDEXER MTI WE CALL IT 2002 WE WILL GO INTO LATER AND JIM WILL GIVE HISTORY ON THAT. OVERALL MILESTONES REPRESENTS A PORTION OF RICH HISTORY OF INDEXING LEADING TO CURRENT INITIATIVE AUTOMATING PROCESSES INVOLVING META DATA AND MEDLINE CITATIONS INCLUDING MESH AND MILESTONES ELATING TO TRANSITION OF AUTOMATED INDEXING SUCH AS MEDLINE AND PROVIDING DEEPER DIVE OF DEVELOPMENT OF MEDICAL TEXT INDEXER AND HOW IT TIES INTO THE EVOLUTION OF MEDLINE. NEXT SLIDE. BEFORE WE GO FORWARD, I WANT TO PAUSE AND CONSIDER THE SCALABILITY SINCE MEDLINE STARTED BACK IN 1971. FOR DECADES MESH INDEXING WAS DONE MANUALLY BY HUMAN INDEXERS. I WAS AN INDEXER YEARS AGO AND WE CALL WRITING HEADINGS AND CHECKED TAGS ON PAPER THAT WAS KEYED INTO A SYSTEM EVENTUALLY. YEARS LATER ENTERING MESH INTO DATA KEYATION AND MAINTENANCE SYSTEM I ACCESS BY DIALING IN ON A MODEM. WE HAVE COME TO REALIZE ADVANCES WITH PAPER AND ENTERING DATING INTO A SYSTEM MANUAL PROCESS DOESN'T SCALE WITH INCREASING VOLUME OF PUBLISHED BIOMEDICAL LITERATURE IN MEDLINE. YOU CAN SEE FROM GRASS CONTINUOUS GROENL OF MEDLINE CITATIONS SINCE 1971 AND WE FULLY EXPECT GROENL TREND TO CONTINUE AND PAST SEVERAL YEARS FOUND OURSELVES IN SIGNIFICANT BACKLOG CITATIONS OF WAITING INDEXING AND USERS CONFUSED BY TIME TO INDEX LAG AND HOW TO EFFECTIVELY SEARCH FOR MEDLINE CITATIONS AS WELL THE COST OF MANUAL INDEXING CONTINUE TO INCREASE EVERY YEAR AND GROWING BACKLOG POSSIBILITY OF MANUAL INDEXING AT SCALE WAS JUST OUT OF REACH AND NEED TODAY FIND A MORE SUSTAINABLE SOLUTION. NEXT SLIDE. IN EFFORT TO FIND SOLUTIONS WE LOOKED AT INTERNAL REVIEWS WE CONDUCT TODAY DETERMINE CONTINUED VALUE OF MESH INDEXING AND ASSESSMENT CONDUCTED IN TWENLT 16 THAT DOCUMENTED DETAILED USE OF MEDLINE INDEXING NOT ONLY IN PUB MED BUT IN VARIETY OF WAYS AND USE CASES VIA DATA DOWNLOADED TO INTERNAL USERS USING IT IN OTHER WAYS AND OUTSIDE EXTERNALLY WITH VENDORS AND RESEARCHERS AND LOTS OF PEOPLE USE MEDLINE DATA FOR LOTS OF DIFFERENT REASONS AND LATER IN 2019 WE LOOKED AT MESH INDEXING AND PUB MED RETRIEVAL AND SAW VALUE OF MESH AND SEARCH RETRIEVAL IN THAT IMPLICIT SEARCHING BEHIND SCENES AND AUTOMATIC TERM MAPPING AND RESULTS OF ASSESSMENTS CONFIRMED VALUE OF MESH INDEXING KNOWING WE NEEDED TO KEEP INDEXING MEDLINE CITATIONS BUT NEEDED TO MOVE TO SCALEABLE COST EFFECTIVE WAY MAKING DATA ACCESSIBLE FASTER AND MAINTAINING QUALITIES THAT USERS EXPECT FROM US. WITH THIS RATIONALE IN MIND WE CREATED 5-YEAR DEVELOPMENT PLAN WE CALLED MEDLINE 2022 INITIATIVE AND THIS INITIATIVE WAS DESIGNED TO ADDRESS CHALLENGES INVOLVED IN MAINTAINING USEFULNESS OF MEDLINE AS A TOOL FOR DISCOVERING AND ANALYZING BIOMEDICAL RESEARCH LITERATURE AND OVERARCHING GOAL OF MEDLINE 2022 INITIATIVE WAS SCALING MEDLINE INDEXING TO VOLUME OF BIOMEDICAL LITERATURE LEVERAGING NLM'S RESEARCH EXPERIENCE AND MACHINE LEARNING COMBINED WITH HUMAN EXPERTISE. NEXT SLIDE. FIRST GOAL OF NLM'S STRATEGIC PLAN IS ACCELERATING DATA DRIVEN DISCOVERY AND ADVANCED HUMAN HEALTH AS NOTED IN THIS GOAL INL CREASING PACE OF DATA GENERATION AND DEMAPPED RENDERED MANUAL OCCURATION UNSUSTAINABLE FUTURE LIES AND CURERATION AT SCALE AND BACKLOG AND TRANSITION OF FULLY INDEXING MEDLINE AND SEEING HERE MEDLINE 2022 INITIATIVE IS AUTOMATED INDEXING PROVIDES BETTER SUPPORT FOR RESEARCHERS CLINICIANS AND LIBRARIANS BY PROVIDING TIMELY ACCESS AND IMPROVED DISCOVERABILITY TO PUB MED CITATIONS WITH MESH AND OTHER METADATA. NEXT SLIDE. ONCE WE HAD SOLID VISION TO DRIVE WORK AND AFTER THOUGHTFUL CONSIDERATION WE DEVELOPED FOUR GOALS TO STRIVE FOR THROUGH MEDLINE 2022 INITIATIVE AND GOALS WERE, ONE, TO FULLY AUTOMATE MEDLINE INDEXING AND SECOND APPLY CURERATION TO CHEMICAL AND GENETIC INFORMATION SUPPLIED BY MACHINE LEARNING APPROACHES AND THIRD, CONSOLIDATE ALL PROCESSES RELATED TO SUPPORT OF MEDLINE INTO A SINGLE SYSTEM AND 4 TO RESEARCH POTENTIAL AUTOMATION OF THE DATA REVIEW PROCESS THAT CAPTURES ADDITIONAL META DATA LIKE ADDING COMMENTS AND FUNDING SUPPORT AND WITH GOALS IN MIND, I WILL TURN THE MIC OVER TO JIM WHO WILL DISCUSS THE ROAD TO AUTOMATION. NEXT SLIDE. >> THANK YOU, DIANNE. I WILL TALK ABOUT WHAT MEDICAL TEXT INDEXER OR MTI IS AND GIVE YOU A REALLY, REALLY QUICK OVERVIEW OF HISTORY OF HOW WE WENT FROM A BRAND NEW SYSTEM BUILT TO ASSIST THE INDEXERS TO FULLY AUTOMATICALLY INDEXING LITERATURE. NEXT SLIDE, PLEASE. NLM AND MTI I WILL CALL IT MTI BY MISTAKE THROUGHOUT THE WHOLE PRESENTATION WAS ORIGINALLY DESIGNED TO BE INDEXING ASSISTSIVE SOFTWARE DESIGNED TO HELP INDEXERS BE MORE CONSISTENT AND BE QUICKER IN DOG THEIR MESH INDEXING THEY WERE DOING MTI ONLY USES TITLE AND ABSTRACT MAJOR DIFFERENCE BETWEEN WHAT HUMAN INDEXERS USE WHICH IS FULL TEXT AND INCORPORATED KNOWLEDGE OF INDEXING PROCEDURES AND RULES AND CAN'T EMPHASIZE IT ENOUGH. WE WORKED IN VERY TIGHT COLLABORATION WITH INDEXERS WHO WERE VERY GENEROUS WITH TIME AND PATIENCE IN HELPING US TO UNDERSTAND WHAT IS INDEXING PROCESS AND HOW IT WORKS AND WHAT ARE RULES. WHY DOES RULE NOT APPLY IN THIS CASE FWHU CASE? THERE IS LOTS OF BACK AND FORTH IN THIS OVER THE YEAR AND NEXT SLIDE, PLEASE. DIANNE TOUCHED ON THIS A LITTLE EARLIER. THIS IS A MEDLINE CITATION WOW SEE IN PUB MED. I WANTED TO BRING IT BACK HERE TO EMPHASIZE THAT METADATA ADDED BY INDEXERS ARE PUBLICATION TYPES MESH TERMS AND SUBSTANCES AND MTI PROVIDES METADATA TO CITATIONS IN AUTOMATIC INDEXING PROCESS. NEXT SLIDE, PLEASE. BEFORE WE GO DEEP INTO HISTORY OF MTI, I THOUGHT IT WOULD BE GOOD TO SEE HOW PERFORMANCE EVOLVED OVER TIME. THIS IS PERFORMANCE WE TRACKED FROM 2007 TO 2022. AS CAN YOU TELL ON LEFT BOTTOM STARTING MTI, THE PERFORMANCE WAS NOT STELLAR IS THE MOST DIPLOMATIC WAY OF SAYING IT. NUMBERS ARE SLIGHTLY DECEPTIVE. STARTING TO DO MTI GOAL WAS TO PROVIDE A LONG LIST OF TERMS WITH IDEA THAT MOST AN INDEXER WOULD WANT WOULD BE ONL THAT LIST CALLING IT A PICK LIST AND FOCUSED ON RECALL MORE THAN PERFORMANCE IN THE BEGINNING. MOVING THROUGH TIME YOU SEE DIFFERENT FLAVORS OF MTI. I WILL TALK ABOUT THEM DEEPER LATER. FOR NOW, I WANT YOU TO FOCUS ON YOU CAN SEE HOW MUCH BETTER ALL THESE DIFFERENT FLAVORS ARE IN PERFORMANCE OVER WHAT THE AVERAGE PERFORM ANXIETY IS EVEN OVER TIME AS WE GET INTO 2022. THERE IS A REASON FOR THAT. NEXT SLIDE, PLEASE. THIS IS A LITTLE OUTDATED BUT HIGHLIGHTS CRITICAL POINT OF M TICHLT. EVOLUTION. 2000 AND 2012 TIMEFRAME PERFORMANCE SHOT UP AND NOT TOO SURPRISINGLY AVERAGE DAILY REQUESTS AND USAGE OF MTI ALSO SKYROCKETED AT THAT TIME. WE SAT DOWN WITH INDEXERS AND TRIED TO UNDERSTAND WHERE WE DID WELL AND POORLY WITH MTI. BIG TAKEAWAY WE GOT IS IT IS NICE HAVING A LONG LIST. I'M TAKING LONGER TO INDEX THAN I WOULD WITHOUT MTI. REASON IS YOU ARE MAKING ME SECOND GUESS. DID I MISS SOMETHING IN THE ARTICLE WHEN IT IS A CLEARLY WRONG TERM FROM MTI. IT STILL MAKES ME SECOND GUESS MY IDEAS. I HAVE TO ACTUALLY GO BACK TO THE ARTICLE AND LOOK AT IT AGAIN. IT IS TAKING LONGER. AT THIS POINT WE RETOOLED AND FOCUSED MTI MORE ON PRECISION AND BALANCING WITH RECALL THAT YOU WILL SEE IN THE GRAPH. NEXT SLIDE, PLEASE. HOW DID THIS START? BACK IN EARLY 90S YOU REMEMBER THE GRAPH DIANNE SHOWED YOU EARLIER ON EXPONENTIAL GROWTH OF MEDLINE ARTICLES AND KNEW AT THAT TIME SOME POINT IN THE FUTURE WHERE IT WOULD BE OVERWHELMING FOR INDEXERS AND INDEXING INITIATIVE WAS ESTABLISHED AS A PROJECT AT NLM THAT INVOLVED EVERY ASPECT OF NLM TO COME TOGETHER AND BRAINSTORM IDEAS ON HOW WE CAN HELP INDEXERS BE CONSISTENT, QUICKER AND BE ABLE TO MAINTAIN INDEXING FOR AS MUCH ARTICLES WE HAVE COMING IN AND CHARTER WE HAVE TO MAINTAIN QUALITY OF INDEXING THAT IS BEING DONE ALREADY. NEXT SLIDE, PLEASE. OUTCOME OF THE INDEXING INITIATIVE WAS MEDICAL TEXT INDEXER MTI WE STARTED TO PROVIDE SUGGESTIONS IN 2002. THAT IS ALL THEY WERE, MERE WILL I A LIST OF TERMS INDEXER COULD CHOOSE TO LOOK AT OR NOT LOOK AT AND IF LOOKING AT IT COULD DO A PICK LIST AND DECIDE THIS WILL HELP ME AND IT WILL BE QUICKER TO INDEX. NEXT SLIDE, PLEASE. SO, IF YOU REMEMBER ON ONE SLIDE WHERE I WAS TALKING ABOUT INDEXER ACCEPTANCE OF MTI IN 2011 AND 2012 BEING REALLY IMPORTANT. IN 2011 INDEXERS CAME TO US AND SAID WE NOTICED THAT THERE IS A SMALL NUMBER OF JOURNALS WHERE MTI IS DOING REALLY WELL AS AN INDEXER WE WANT TO MAKE IT A FIRST-LINE INDEXER. THAT MEANS FOR FIRST TIME PROGRAM WAS CONSIDERED A JUNIOR INDEXER. SO, THIS IS JUST LIKE IF THEY HIRED A BRAND NEW INDEXER AT NLM, THEY WOULD TAKE MTI RESULTS AND GO THROUGH REVISER AND GO INTO PRODUCTION. THIS WAS A TURNING POINT IN MTI. WE HAVE GONE NOW FROM MERELY SUGGESTING TERMS TO INDEXING AND BEING REVISED AND IS A CRITICAL TRAINING POINT FOR US. NEXT SLIDE, PLEASE. IN 2015, MTI WAS DOING WELL ENOUGH WE WANTED TO GO BACK AND LOOK AT THERE WAS ABOUT 2 MILLION ARTICLES INDEXED BETWEEN 1946 AND 1965. IT WAS BEFORE MEDLINE CRITERIA FOR INDEXING WAS IMPLEMENTED AND WAS VERY EARLY ON IN THE MESH CONTROLLED VOCABULARY ERA. THERE WAS NOT A LOT OF RIGIDITY, I GUESS, IN WHAT INDEXERS WERE INDEXING FOR ARTICLES AND THEY WERE NOT CONSIDERED VALID MESH TERMS TO BE ADDED TO THE INDEXING IN PUB MED. THEY ARE ONLY IN THERE AS KEY WORDS AND WHAT WE DECIDED TO DO WAS RUN THEM ALL THROUGH MTI, AUTOMATICALLY INDEX THEM AND TAKE THE MEDLINE RULES IMBEDDED TO MTI TAKING CONTROL OF VOCABULARY INTO MESH AND PLY TO ARTICLES AND IN LOTS OF CASES WE ALSO -- IF THERE IS AN ARCHAIC TERM AND GOOD EXAMPLE OF THAT IS THEY MIGHT HAVE USED CONSUNLS AS A KEY WORD. WE WOULD HAVE CONVERTED IT TO TUBERCULOSIS AND WOULD HAVE CONFORMED TO MESH VOCABULARY. NEXT SLIDE, PLEASE. 207 TAENDID MTI FOR COMMENTS AND JUST BEFORE THIS COMMENTS AND COMMENT IS IF I WROTE AN ARTICLE AND YOU WROTE AN ARTICLE SAYING I AGREE OR DISAGREE WITH YOU WOULD BE A COMMENT ON. THAT IS WHAT WE LOOK AT IN AUTOMATING IN THIS STAGE. ARTICLES COMMENTING ON ORIGINAL ARTICLE AT THAT POINT WERE AUTOMATICALLY GIVEN I INDEXING FROM REVERENT OR ORIGINAL ARTICLE AND INDEXERS THOUGHT IT WOULD BE A GOOD IDEA TO EXPAND FROM INFORMATION ON THIS COMMENT ON BY HAVING THEM AUTOMATICALLY INDEXED BY MTI. WE WOULD TAKE NEW COMMENT ON TITLE AND WOULD FAKE ORIGINAL TITLES ARTICLE AND INDEX FROM THAT AND IN CASES APPLYING NEW TERMS THAT WOULDN'T HAVE BEEN ADDED TO NEW ARTICLES. NEXT SLIDE, PLEASE. IN 2018 OR BEFORE THIS, WE WERE HAVING DISCUSSIONS. WE NEEDED TO BE ABLE TO MARK XML RECORDS IN PUB MED AND MEDLINE SO RESEARCHERS WOULD KNOW IF IT WAS DONE OR IF THE INDEXING WAS DONE BY AUTOMATIC PROGRAMS OR IF IT WAS LOOKED AT BY HUMAN OR WAS FULLY INDEXED BY HUMANS. THAT WAS SO RESEARCHERS COULD MAKE THE DECISION ON WHAT THEY WANTED TO USE AND EXCLUDE THE REST AND CAUP WITH XML TAX WHER INDEXER AT LEAST LOOKED AT IT AND AUTOMATED MEANS INDEXER DIDN'T NECESSARILY LOOK AT IT BUT MEANS IT IS FULLY AUTOMATED AND NEXT SLIDE, PLEASE. IN 2019 OR 2020, WE STARTED MTI REVIEW. THIS WAS OUR PROOF OF CONCEPT FLAVOR, I GUESS OF MTI. WE KNEW WE WERE MOVING TOWARDS FULLY AUTOMATIC INDEXING AND NEEDED A PLATFORM WHERE WE COULD TEST OUT THE ALGORITHM AGAINST WHAT HUMAN INDEXERS LOOKING AT AND INDEXERS LOOKING AT TITLE AND ABSTRACT THAT MTI HAD ACCESS TO AND DEVELOPED MTI REVIEW TO LET US KNOW HOW DIFFERENT JOURNALS WERE DOING LAND OR NOT THEY COULD BE PROMOTED TO AUTOMATIC INDEXING THAT YOU CAN SEE IN 2020 WE HAVE A TOTAL OF 8 JOURNALS OUT OF 5,000 IN AUTOMA AUTOMATION. IT WAS A PROOF OF CONCEPT MORE THAN ANYTHING ELSE. NEXT SLIDE, PLEASE. IN 2021, WE INCREASED IT TO 1/3 OF THE JOURNALS AND ABOUT 7 TAEN-00 AND IN APRIL 20202 WE WERE FULLY AUTOMATIC. WHAT I WANT YOU TO SEE, I HOPE, AT LEAST IN THIS DIAGRAM IS AUTOMATIC INDEXING IS NOT SOMETHING THAT NLM DECIDED OVER NIGHT THAT IS SOMETHING THAT HAS EVOLVED FROM A TOOL THAT WAS DEVELOPED TO ASSIST INDEXERS TO FULLY AUTOMATIC INDEXING AND ONLY DOING SO WHEN TECHNOLOGY WAS WHERE IT NEED TODAY BE AND PERFORMANCE OF MTI IS WHERE IT NEED TODAY BE TO MAINTAIN QUALITY AND CONFIDENCE IN THE INDEXING. NEXT SLIDE, PLEASE. WHERE ARE WE GOING NEXT? AS YOU SAW IN THE PREVIOUS SLIDE, MTI IS OVER 20 YEARS OLD AT THIS POINT. WE DEVELOPED A NEW MTI OR MTI NEXT GENERATION OR MTIX AND DEVELOPED FROM VERY BEGINNING USING LATEST TECHNOLOGY AND USING MACHINE AND DEEP LEARNING AND TRAINING IT ON OVER 10 MILLION HUMANLY MEDLINE INDEXED ARTICLES INDEXED ON FULL TEXT TO GIVE IT A BETTER PERSPECTIVE AND GOAL OF THIS IS TO INTEGRATE THE NEW VERSION OF MTI INTO THE PUB MED DATA MANAGEMENT SYSTEM STREAM SUCH THAT WHEN A PUBLISHER UPLOADS NEW ARTICLES TO PUB MED THEY SHOULD BE INDEXED EVEN BEFORE THEY GO INTO PUB MED AND BE VIEWED BY PUBLIC AND NEXT SLIDE, PLE PLEASE. >> THANK YOU JIM FOR JOURNEY OF MTI. I WILL PROVIDE AN UPDATE ON AUTOMATED INDEXING WORK AND WHAT WE HAVE GOTTEN DONE THUS FAR. N SLIDE, PLEASE. THESE ARELET GOALS OF MEDLINE 2022 INITIATIVE AND I'M GOING TO EXPAND ON TWO THAT ARE CHECKED OWE OF HERE GOALS 1 AND 3 WE COMPLETED INDEXING AND DID IT IN APRIL OF LAST YEAR AND JIM STATED THAT AS WELL. THIS IS MAJOR GOAL OF MEDLINE 20 WENT TO INITIATIVE THAT WE REFERRED TO THIS IN MULTIPLE NLM COMMUNICATIONS AND INCLUDED LINKS HERE FOR YOU TO SEE AND HAD THEM USING BLOG POSTS ABOUT THIS. WE COMPLETED GOAL 3 ALSO. THIS WAS ACCOMPLISHED IN JANUARY OF 2022 COMPLETING CONSOLIDATION OF FUNCTIONALITY INTO A SINGLE SYSTEM AND PUB MED DATA MANAGEMENT SYSTEM OR WE CALL IT PMDM WHICH IS NOW A ONE-STOP SHOP FOR MAINTENANCE OF MESH INDEXING AND DATA POINTS LIKE GENELING OFS CREATED BETWEEN PUB MED AND GENE INCREASING DEFICIENCIES AND DECREASING POSSIBILITY OF DATA DEGRADATION TRANSFERRING DATA ACROSS SYSTEMS ACHIEVING THESE TWO GOALS WE HAVE SEEN ENORMOUS IMPACT FROM MEDLINE DECREASING TIME AND COST OF INDEXING TO RESOLVE SIGNIFICANT BACKLOG. NEXT FEW SLIDES WILL SHOW YOU A LITTLE IMPACT AND SOME WAYS OF ACHIEVING AUTOMATED INDEXING BENEFITED INTERNAL PROCESSES SUPPORTING USERS. NEXT SLIDE, PLE PLEASE. ONE OF THE BIGGEST IMPACTS IS THAT RESOLUTION OF GROWING INDEX BACKLOG. BACKLOG OF 6 HUND00,0 CITATIONS AND CLEARING BACKLOG TIME FOR CITATION INDEX DROPPED FOR MANY MONTHS TO A SINGLE DAY AND SHOUTOUT IN GRAPHIC BLUE HILL I CALL IT IS BACKLOG THAT OVER TIME IS RELIEVED FROM AUTOMATED INDEXING AND BLACK LINE FOLLOWING SAME TRAJECTORY INDICATES REDUCTION IN TIME TO HAVE MESH ADDED TO CITATIONS FROM 100 OR MORE DAYS TO JUST HOURS AND ARE VERY EXCITED ABOUT THAT IMPACT. NEXT SLIDE. ANOTHER METRIC FOR SUCCESS OF EFFORTS IS CONTINUOUS ACHIEVEMENT OF CURERATION AT SCALE. YOU CAN SEE IN GRAPHIC PROVIDED THAT WE CONTINUE TO SEE INCREASE IN GROSS CITATIONS ADDED TO MEDLINE. WE ACHIEVE AND CONTINUE TO PROVIDE CURATION OF SCALE MEDLINE INDEXING BENEFITING USERS AND SEARCHERS IN PUB MED AND NO NEED TO END COMPLEX SEARCHES TO GET FULL RETRIEVAL AND AUTHORS ARE VERY PLEASED TO SEE CITATIONS ARE ANNOTATED WITH MESH QUICKER. NEXT SLIDE. NOTHING SHOWS OUTSIDE SHOWING PRECISION AND RECALL AND SHOWING IN CHART AUTOMATED INDEX PERFORMANCE IS VERY GOOD JIM SHOWED THIS WELL. PRECISION IS MEASURE OF ACCURACY OVER 90% FOR ALL CATEGORIES OF TURN INCLUDING MESH CHECK TAGS SUPPLEMENTARY CONCEPT RECORDS AND PUBLICATION TYPES AND RECALL A MEASURE OF COMPLETENESS IS QUITE GOOD AND LOWER FOR CERTAIN TYPES OF TERMS AND SOME COMPLEX CHEMICALS AND PROTEINS THAT CAN CAUSE AMBIGUITY AND RESULTS AND WE ARE WORKING ON THAT AND DATA PRESENTED HERE IS DERIVED FROM SAMPLE OF PROBABLY 30 THOUSAND INUM NUMB CITATIONS ACROSS VARIETY OF SUBJECT AREAS IN MEDLINE. NEXT LINE. GOALS ARE WE CHECKED OFF 1 AND 3 AND ARE WORKING ON GOALS 2 AND 4. NEXT SLIDE. GOAL 2 IS APPLYING CURATION TO CHEMICAL AND GENE INFORMATION AND WORKING TO IMPROVE RECOGNITION OF CHEMICALS INDEXING INCORPORATION OF NLM CHEM AND ALGORITHM IN GENE RECOGNITION VIA INCORPORATION OF VERSION OF NLM GENE ALGORITHMS THAT THESE ALGORITHMS HAVE BEEN DEVELOPED BY [INDISCERNIBLE] GROUP IN NCBI'S COMPUTATIONAL NCBI BRANCH. I LISTED THINGS HERE IF INTERESTED IN THIS WORK AS WELL. NEXT SLIDE. PART OF 4TH GOAL BEGINNING TO INVESTIGATE POSSIBILITY OF EXTRACTING DATA POINTS SUCH AS COMMENT LINKSAGES RESEARCH SUPPORT TAGS AND DATA BANK INFORMATION FROM FULL TEXT AND RECOGNIZE IMPORTANCE OF DATA ACCESS POINT AND LINK ANLS PROVIDING DISCOVERABILITY ACROSS NLM PRODUCTS AND EXTERNALLY TO OTHER USERS. NEXT SLIDE. AS JIM MENTIONED WE ARE WORKING ON INCORPORATION OF MTIX AND NEXT GEN ALGORITHM AND PROCESSING WORKFLOW IN PMDM. NEXT SLIDE. THROUGHOUT ALL THIS YOU MIGHT BE WONDERING WHAT ARE INDEXING STAFF'S RESPONSIBILITIES WITH TRANSITION TO AUTOMATED INDEXING AND DOMAIN EXPERT'S ROLES IS CURERATION OF AUTOMATED INDEXING AND NOTED EARLIER STAFF REVIEW PROBABLY 20% OF AUTOMATED INDEX CITATIONS FROM SPECIAL CATEGORIES AND THO THOSE WITH GENE AND PROTEIN TYPES THAT GIVE US HICCUPS AND PERFORM RANDOM QUALITY ASSURANCE CHECKS ON DIFFERENT CITATIONS AS WELL IMPORTANTLY EXPERTS PROVIDING FEEDBACK FOR ALGORITHM AND INCLUDING MANUAL INDEXING TRAINING SETS FOR NEW DATA MTIX TO EVOLVE AND NEW TERMS ADDED TO MESH ANNUALLY AND SCRS DAILY AND NEW CONCEPTS POPPING UP IN LITERATURE AND RETEACHING ALGORITHM HOW TO APPLY MESH EFFECTIVELY AND ACCURATELY. NEXT SLIDE TO SUMMARIZE YOU SEE IMPACT OF AUTOMATIC INDEXING IN INITIATIVE SO FAR AND PROCESS TO SWITCHING TO FULLY AUTOMATED INDEXING WE ELIMINATED HUGE BACKLOG PROVIDING FASTER ACCESS TO EVER GROWING NUMBER OF CITATIONS DOING IT EFFICIENTLY AT REDUCED COST AND WITH HIGH PRECISION AND HOPE YOU HAVE SEEN TODAY THAT PAST AND FUTURE OF MEDLINE IS ROOTED IN RESEARCH AND COLLABORATIVE WORK THAT SPANS ACROSS MANY PARTS OF NLM WE ARE IMPROVING ALGORITHM AND MAKING QUALITY MEDLINE DATA AVAILABLE TO USERS AT SCALE. NEXT SLIDE. WITH THAT, THANKS FOR LISTENING. OPENING IT TO HEIDI FOR QUESTIONS. >> THANK YOU, DIANNE AND JAMES. TERRIFIC AND AMAZING PROGRESS ON INDEXING AND PROTECTION OF RESOURCES TO SUPPORT IT AND TURNING TO ASSIGNED SDUSANTS CLIFTI AND PAUL FOR QUESTIONS FIRST. >> KICKING OFF CONVERSATION JIM AND DIANNE, THIS IS REMARKABLE WORK AND ENDING PRESENTATION WITH IMPACT OF MEDLINE 20202 INITIATIVE WAS A REALLY GREAT WAY TO DRIVE HOME THAT STORY OF THE IMPACT THAT THIS MAKES FOR ALL OF THE WORK THAT WE DO. STARTING CONVERSATION, I WONDER BASED ON EFFICIENCIES AND SUCCESSES WE HAVE SEEN SO FAR ON WORK ON AUTOMATED INDEXING ADDITIONAL APPLICATION AND PRODUCTS IN NLM IN THE PIPELINE THAT WILL USE THIS TECHNOLOGY? OERNL QUESTION ELATED TO THAT THAT PEOPLE WOULD FIND USEFUL RESEARCHERS AND THOSE OWNING SIMILAR TOOLS MAKING USE OF INDEXING HOW ARE YOU ENGAGING WITH RESEARCH COMMUNITY? IS THERE OPPORTUNITY TO USE TOOLS MORE OPENLY? >> JIM,LY TAKE FIRST ON EFFICIENCIES AND WHERE WE ARE USING IT AND TAKING RESEARCHER QUESTION I WILL PUNT IT OVER TO YOU. SO, THANK YOU, KRISTI. WE ARE LOOKING -- YEAH. EFFICIENCIES AND SUCCESSES HAVE BEEN MASSIVE AND PIVOTING IT REALLY IN REALLY A YEAR'S TIME WAS JUST A REMARKABLE FETE BY ALL OUR STAFF INVOLVED IN THIS. WE ARE LOOKING AT MTI AND CATALOGING AS WELL. RIGHT NOW IS IN THE PHASE OF SUGGESTING TERMS TO CATALOGERS TO APPLY INDEXING MESH TERMS TO THAT. WE ARE LOOKING AT CREATING EFFICIENCIES THAT I MENTIONED BEYOND USING MTI TRYING TO SCALE WITH OTHER ALGORITHMS NLM KEM AND NLM GENE TO HELP PROVIDE APPLYING SCRS AND GENE LINKS TO MEDLINE CITATIONS AS WELL. WE ARE OPEN TO IDEAS AND SHOOTING TO JIM TO TALK ABOUT RESEARCH EFFORTS SHARING TECHNOLOGY AND WHERE WE WILL GO NEXT. >> THANK YOU, DIANNE. MTI GARNERED WHAT I WOULD CONSIDER INTERESTING INTEREST FROM THE COMMUNITY AND HAVE HAD PEOPLE WANTING TO USE MTI INDEX EVERYTHING FROM PATENT REGISTRATION NARRATIVES TO CLASS NARRATIVES. THIS IS A PARAGRAPH WHAT MY CLASS IS. WE HAD A CERTIFYING AGENCY THAT WANTED TO USE FOR CE CLASSES SO PEOPLE COULD FIND THEM EASIER. WE HAVE RESEARCHERS FROM AROUND THE WORLD USING MTI EVERY DAY. I WILL PUT LIFSHGS IN MESSAGE THING HERE IN A SECOND. WE HAV THING HERE IN A SECOND. WE HAVE LINKS AND API AXIS. WE CAN'T RELEASE MTI SOFTWARE FOR PEOPLE TO USE ON THEIR OWN RIGHT NOW. WE ARE USING SOME THIRD PARTY APPLICATION WE DON'T HAVE THE RIGHT TO GIVE OUT TO PEOPLE. ON YEARLY BASIS WE ARE SEEING ANYWHERE FROM 30 TO 50 MILLION PIECES OF TEXT GOING THROUGH MTI ON OUR SYSTEMS EVERY YEAR. THAT IS NOT INCLUDING PROCESSING WE DO OVERNIGHT. >> I FORGET WE USE IT BEHIND MESH ON DEMAND. IF YOU HAVE TAKEN TEXT. PEOPLE USE THIS FOR GENERATING KEYWORDS FOR PAPERS AND IS A GREAT TOOL TO CUT AND PASTE ABSTRACT AND GET SOME MESH HEADINGS FOR KEY WORDS TO RECOMMEND WHEN PUBLISHING AND THERE IS AN OPPORTUNITY FOR YOU. >> THAT IS GREAT AND TERRIFIC TOOL BY THE WAY. >> IT IS. >> THANK YOU. >> ANY COMMENTS? >> YEAH. WUNL COMMENT I APPRECIATE FORM OF PRESENTATION WHEN WE HAVE AT BOARD OF RENALENTS MEETING AND THANKS FOR THAT AND ENCOURAGE YOU TO WRITE IT UP IN THE LIBRARY INFORMATION SCIENCES LITERATURE AND THINK WORK YOU ARE DOING IS PATH-BREAKING AND NEEDS TO BE MORE WIDELY UNDERSTOOD IN THAT PARTICULAR DOMAIN AND AS ANOTHER NATIONAL LIBRARY BEING ON THIS JOURNEY AS WELL I WAS WONDERING IF YOU CAN TALK ABOUT YOU COVERED A LITTLE IN THE LAST SLIDE BUT TALK A LITTLE ABOUT THE TRANSITION STAFF MEMBERS HAD TO MAKE FROM SKILLS PERSPECTIVE OR WORK THEY HAVE TO DO THAT IS DIFFERENT AND PROTEIN AND GENOMIC PART OF THIS IS LIKE THE FULL EMPLOYMENT ACT OF INDEXING EVEN IF AUTOMATED AND IF YOU CAN TALK ABOUT HOW STAFF TRANSITIONED FROM SKILL PERSPECTIVE OR SOMETHING DIFFERENT WHAT YOU ARE ABLE TO DO WITH AUTOMATED INDEXING WOULD BE GREAT. >> THANK YOU, PAUL. JUST FOR THOSE WHO DIDN'T KNOW WE REACHED OUT TO [INDISCERNIBLE] WHEN BEGINNING AUTOMATION EFFORTS GOING TO FULLY INDEXING QUITE A WHILE AGO. YOU WERE PEER GROUP THAT WE LOOKED TO FOR WHAT WORKED IN LESSONS LEARNED AND THAT AS WELL AND STAFF HAVE REALLY TRANSITIONED HELPING TO RETRAIN ALGORITHM. WE USE THEIR DOMAIN EXPERTISE VERY MUCH. THERE IS LOTS OF RULES AND INDEXING AND HAS TO HELP WITH TRAINING OUT AMBIGUITY AND COLD TEMPERATURE VERSUS COLD VIRUS EARTH MATERIAL OR EARTH PLANET AVEND ARE THINGS WE NEED DOMAIN EXPERTISE TO HELP WITH TRAINING ALGORITHMS AND IS NOT JUST THAT. WE ARE ALSO, AGAIN, USING THE CHEM AND GENE TOOLS AND WE ARE HELPING TO TRAIN THAT AS WELL. THAT REQUIRES EYES ON IT TO LOOK AT HOW IT IS GENERATING THE CHEMICALS AND GENES AND CHEMICALS ARE REALLY KIND OF MESSY BECAUSE OF VARIANCE AND SCIENTIFIC NOTATIONS HOW PEOPLE WRITE THEM OUT MAPPING THOSE APPROPRIATELY. WE HAVE DONE IT OVER 20 YEARS EXPERTS DOING MANUAL INDEXING HELPING TO BUILD THIS AUTOMATED PROCESS FOR MANY YEARS AND WE CONTINUE TO DO SOME SUBSETS OF INDEXING TO MAKE SURE WE HAVE A COMPARATIVE SUBSET TO ENSURE QUALITY STEP AS WELL. >> ARE THERE QUESTIONS IN THE GROUP? I HAVE ONE MORE AND WANT TO OPEN THE CONVERSATION TO THE REFT OF THE MEETING. >> QUESTION MYSELF. YOU REFERENCED NLM GENE AS A RESOURCE FOR DOG INDEXING WHICH IS GREAT. JIONG DOES WORK WITH INDEXING THRU GENES AND THROUGH VARIANCE AND WONDER IF INTENT TO MOVE GENE ON TO VARIANCE AS INDEXING APPROACH USING SAME WORK JIONG HAS DONE ON THAT FRONT AND LIP BAR. >> YOU WILL ASK YOU TO TRY TO ANSWER THAT ONE. >> WILL GIVE IT A TRY. ONE DIFFERENCE BETWEEN JIONG'S WORK AND INDEXING IS JIONG IS CONCERNED WITH FINDING EVERY GENE OR CHEMICAL IN AN ARTICLE AND INDEXING PROCESS IS DESIGNED TO SEE THAT AND IDENTIFY WHAT IS IMPORTANT IN THE ARTICLE AND ABLE TO USE JIONG'S AND HIS GROUPS TELL US EVERYTHING ABOUT THIS ARTICLE AND IMPORTANTLY THEY HELP US IDENTIFY WHAT SPECIES THE ARTICLE IS ABOUT. LIMITING THAT HELPS MTI IDENTIFY WHAT THE RIGHT PROTEIN WOULD BE AND COULD BE ANYTHING AND MTI DOESN'T ALWAYS KNOW FROM TITLE AND ABSTRACT. LOTS OF TIMES SPECIES IS VARIED IN FULL TEXT AND HAS TO GUESS FROM CONTEXT AND JIONG SOFTWARE WILL TELL US THIS IS A WRAP. OKAY. NOW WE KNOW THIS IS A RAT SPECIES AND CAN SAY IT IS THIS VERSION OF PROTEIN. SO, I THINK IT IS A TOOL TO HELP INDEXING AND DON'T SEE IT EXPANDING MANDATE FOR INDEXING THOUGH. >> GOT IT. MAKES SENSE. THANK YOU, JIM. KRISTI, BACK TO YOU. >> OKAY. QUESTION IS FOR DIANNE AND I WOULD LIKE TO ASK ABOUT THE 2023 RELEASE OF MESH. THAT CAME OUT. IT REFLECTS A HUGE AMOUNT OF WORK AND SIGNIFICANCE EFFORTS TO IMPROVE AND CORRECT POPULATION RACIAL TERMS AND WONDER IF YOU CAN SHARE WITH US A LITTLE BIT ABOUT PARTNERSHIPS AND HELPING TO FACILITATE THIS WORK AT NIH AND BEYOND AND WAYS YOU FACILITATED FEEDBACK WITH COMMUNI COMMUNITIES. >> YEAH. THANK YOU FOR. THAT GIVING BACKGROUND WE RELEASE MESH ON ANNUAL BASIS AND RELEASED 2023 MESH. AND IN THAT WE UPDATED MANY OF THE POPULATION TERMS AND RACIAL TERMS AND WE UPDATED SOME OF THE HISTORY AND SCOPE NOTES IN OUR VOCABULARY AS WELL WE OFTEN REACH OUT TO NETWORK AND COMMUNITY MEMBERS THAT ARE INTEGRATED IN IT AND ESPECIALLY WITH TERMS YOU KNOW? WITH A LOT OF THINGS GOING ON IN CURRENT AWARENESS AND PEOPLE ARE VERY ATTUNED TO HOW WE ARE CREATING OUR VOCABULARY. SO, WE KROEATED A MESH LISTENING SESSION. WE HELD IT AND INVITED PEOPLE TO COME TALK ABOUT WHAT DO YOU THINK IS WORKING WELL? WHAT DO YOU THINK WE COULD PIVOT AND CHANGE AND WHO DO YOU THINK WE SHOULD PARTNER WITH OUTSIDE OF WHO WE ARE PARTNERING WITH? IT WAS EFFECTIVE. WE TOOK FEEDBACK AND HEARD FROM USERS AND CREATED AND CHANGED OUR MESH VOCABULARY. AND WE ADDED POPULATION TERMS CREATING ADDITIONAL TREES AND CONTINUED TO WORK WITH PARTNERS INF IN -- ACROSS FEDERAL GOVERNMENT AND IN NIH. WE WORK WITH OBSSR. SOMEBODY WILL ASK ME TO DEFINE THAT. I'M HOPING A COLLEAGUE COULD HELP ME WITH THE OFFICE OF BEHAVIORAL AND SOCIAL SCIENCES RESEARCH. IS THAT IT? GOOD. I SPEAK MOST WILL I IN ACRONYMS AND MESH VOCABULARY AND HOW TO APPROPRIATELY CREATE TERMS AND DEFINITIONS WITH THEM AND FROM INITIAL LISTENING SESSION WE HAVE MADE CHANGES AND ARE NOW PUTTING OUR LIST OF MESH WE ARE CONSIDERING IF FOR UPDATING OVER THE NEXT YEAR AND CREATED PAGES FOR THAT WITH REPORTS AND PEOPLE CAN SEE WHERE TREES HAVE MOVED AND HELPED TO FIND HOW THE VOCABULARY CHANGED FROM ONE YEAR TO THE NEXT. WE ARE LISTENING AND IMPLEMENTING CHANGES THAT ARE REALLY IMPORTANT FOR USERS. >> THAT IS GREAT. THANK YOU FOR SHARING THAT. >> SURE. THANK YOU. >> HEIDI, HOW ARE WE DOING ON TIME? >> WE ARE ACTUALLY TIME TO MOVE ON IF THAT IS OKAY. >> THANK YOU VERY MUCH. >> THANK YOU. >> THANKS FOR THE UPDATE AND VERY EXCITING TO SEE THE PROGRESS. NEXT SET OF TOPICS ARE AROUND CONCEPT CLEARANCES AND WE HAVE TWO THAT WILL BE PRESENTED AND WILL HAVE DISCUSSION AND VOTING ON EACH ONE. FIRST ONE WILL BE PRESENTED BY LINDA HARDY. I WILL TURN IT WE HAVE LINDA ON? >> I AM HERE. >> GREAT. >> THIS PRESENTATION ADDRESSES NCHL LMGOA RESOURCE GRANT SUPPORTING HEALTH EBBINIQUITY I HEALTH DISPARITIES. GOAL NO. 2 REACHING INDIVIDUALS THROUGH ENHANCED DISSEMINATION AND ENGAGEMENT INCLUDING ACCURATE UNDERSTANDABLE HEALTH INFORMATION RESULTING IN EMPOWERING HEALTH DISPARITY POPULATIONS TO MAKE HEALTH CARE DECISIONS AND PROGRAM PROJECTS UNDER FUNDING OPPORTUNITY DEVELOP AND DEPLOY NEW INFORMATION RESOURCES ORE SERVICES WITH FOCUS ON PUTTING RESEARCH KNOWLEDGE INTO PRACTICE THROUGH TAILORED RESOURCES FOR WELL DEFINED HEALTH DISPARITY POPULATION AND ALSO APPLICANTS ARE ENCOURAGED TO USE INFORMATION TECHNOLOGY TO ENSURE ACCURACY AND MITIGATING POTENTIAL BIAS OF INFORMATION PROVIDED TO INDIVIDUALS AND HEALTH CARE PROVIDERS. NEXT? OSHTTIVES OF GO8 FUNDING OPPORTUNITY ARE COMPLYING WITH FEDERAL HEALTH PUBLIC LEGISLATION PROVIDING ACCURATE, USEFUL, USABLE AND UNDERSTANDABLE HEALTH INFORMATION TO HEALTH DISPARITIES POPULATIONS AND THEIR HEALTH CARE PROVIDERS TO DEVELOP AND DEPLOY NEW INFORMATION RESOURCES OR SERVICES TO MEET HEALTH CARE NEEDS OF HEALTH DISPARITY POPULATIONS ENCOURAGING USE OF TECHNOLOGY TO DISSEMINATE INFORMATION, RESOURCES OR SERVICES. NEXT? G08 FUNDING OPPORTUNITY RECOGNIZING HEALTHY PEOPLE OBJECTIVES REDUCES HEALTH DISPARITIES ACHIEVING EQUITY AND HIFLT TORE CAL AND CONTEMPORARY INJUSTICES SOCIAL DETERMINANTS OF HEALTH AND SEMINAR PUBLICATION CROSSING QUALITY CHASM SUGGESTS QUALITY CARE IS ABSENCE OF DISPARITIES AND AGENCY FOR HEALTH CARE RESEARCH AND QUALITY SUBT SUGGESTS THAT THESE PERSIST AND G08 RESOURCE GRANT MINE MIEZ DIGITAL DIVIDE IN HEALTH CARE MEETING GOALS OF HEALTHY PEOPLE IN 20203. FOCUSING ON NEED TO BRING USEFUL UNDERSTANDABLE HEALTH INFORMATION TO HEALTH DISPARITY POPULATIONS TO FACILITATE HEALTH CARE DECISIONS COMPUTER USE AT INFORMATION TECHNOLOGY PROVIDE COMMUNICATION METHODS HELPING ADDRESS DIGITAL DIVIDE AND FURTHER HEALTHY RECOGNITION FOR PEOPLE IN 23RD. THIS PROVIDES A USER CENTERED INFORMED TOOLKIT USING AMERICAN SIGN LANGUAGE FOR DEAF AND HARD OF HEARING PEOPLE HIGHLY UNREPRESENTED AS RESEARCH PARTICIPANTS AND GOAL DETERMINING KNOWLEDGE ABOUT RESEARCH AND OPPORTUNITY AND PROVIDE CONSENT TOOL FOR RESEARCH INFORMATION AND MEDICINE FOCUSES ON PROVIDING IMPORTANT INFORMATION RELATED TO DIABETES AND HEART DISEASE TO AMERICAN INDIAN AND ALASKAN NATDTIVE INDIGENOUS COMMUNITIES AT HIGH RISK FOR THESE ILLNE ILLNESSES. WE RECEIVED 206 GRANT APPLICATIONS AND G08 APPLICATIONS SUBMITTED ACROSS 5 MAIN INSTITUTIONAL CATEGORIES AND 82% OF APPLICATIONS WERE FROM ACADEMIC INSTITUTIONS PREDOMINANTLY FROM SCHOOLS OF MEDICINE AND SCHOOLS OF NURSING AND REMAINING 18% WERE FROM OTHER RESEARCH ORGANIZATIONS AND G08 SUBMISSIONS WERE PERMITTED EVERY OTHER YEAR THAT MIGHT HAVE LIMITED APPLICATIONS THAT NLM RECEI RECEIVED. WE BELIEVE PREMISE SHOULD REMAIN SAME AND COMPONENTS ADDED INCLUDING PROGRAM EVALUATION PLAN AND NEED FOR EVIDENCE OF OR AT PLAN 4 RESOURCE SUSTAINABILITY AND FACILITATING ADDITIONS WE RECOMMEND FUNDING LEVEL FOR GO 08 APPLICATIONS BE INCREASED FROM $150,000 PER HERE TO 2,000 PER YEAR. COSTS ARE NOT INCLUDED IN FUNDING OPPORTUNITY. NEXT. PRAV THAFRNG YOU FOR LISTENING AND I WILL ANSWER ANY QUESTIONS. >> GREAT. LYNNE. ANY QUESTIONS ON THAT CONCEPT? TERRIFIC. WE WILL TAKE A VOTE OF THE BOARD. ALL OF THOSE IN FAVOR OF THIS CONCEPT, RAISE YOUR HANDS. WE DO RAISE HANDS; RIGHT? IN REACTIONS? OKAY. I THINK WE HAVE EVERYONE THERE. THANK YOU. ANYONE IN OBJECTION? MAKE SURE WE DIDN'T MISS ANYONE. PERFECT. STANDS APPROVED. MOVING ON TO NEXT PRESENTATION ON NLM GRANTS FOR SCHOLARLY WORKS AND BIOMEDICINE AND DR. [INDISCERN [INDISCERNIBLE]. WE HAVE -- FL WE ARE RAVI. WE CAN'T HEAR YOU I CAN'T HEAR YOU. >> CAN YOU HEAR ME. >> THERE WE GO, YUP. >> SORRY ABOUT THIS. ONE SECOND. >> SLIDES LOOK GOOD FROM OUR END. WI HEAR SDWLU GOOD AFTERNOON. I'M A PROGRAM OFFICE ERRAND THIS CONCEPT DOCUMENT IS REISSUING NLM GRANTS SCHOLARLY WORKS FOR BIOMEDICINE AND HEALTH IN THE G13 PROGRAM THAT PROVIDES PREPARATION OF BOOK LENGTHED MANUSCRIPTS ACADEMIC AND POLICY VALUE TO HEALTH PROFESSIONALS BIOMED DAL RESEARCHERS AND HISTORIANS OF HEALTH SCIENCES AND MAJOR CRITICAL REVIEWS STATE OF THE ART SUMMARIES HISTORICAL STUDIES AND USEFUL ORGANIZATIONS OF KNOWLEDGE AND CLINICAL MEDICINE PUBLIC HEALTH BIOMEDICAL RESEARCH AND SCIENCES ASSOCIATED WITH THEM AND WORK PREPARE FORD PUBLICATION AND PRINT AND ELECTRONIC MEDIA OR BOTH AND G13 PROGRAM UNIT PROGRAM IN RESPONSE TO LEGISLA LEGISLATION. MEDICINE PUBLIC HEALTH LIFE SCIENCES. BIOETHICS AND STUDIES ON INTERRELATIONSHIP OF MEDICINE AND SOCIETY AND FOCUSING ON HISTORY AND PHILOSOPHY OF BIOMEDICAL ETHICS AND HEALTH INFORMATION SCIENCES HEALTH COMMUNICATIONS OR HEALTH SCIENCES LIMITED LIABILITY AND PERCENTAGE OF RESEARCH AND ADVANCES THAT HAVE BEEN MADE AND EXAMINATION AND [INDISCERNIBLE]. CURRENTLY PROGRAM PROVIDES SUPPORT UP TO 50,000 PER YEAR UP TO 3 YEARS AND THIS CURRENT YEAR [INDISCERNIBLE] EXPIRES FEBRUARY 24TH, 2023 AND CONCEPT DOCUMENT IS REISSUING THIS. TO SEE IMPACT, 2013 AND 2022NLM RECEIVED 365 APPLICATIONS RESPONDING TO G13FOA. NLM RECEIVES 30 TO 40 APPLICATIONS A YEAR AND OUT OF TOTAL 365 RECEIVED APPLICATIONS SO FAR, 39 WERE FUNDED AND THESE 39 FUNDED GRANTS PRODUCED 9 MONOGRAPHS AND TWO SCHOLARLY ARTICLES AND RESOURCES AND ONE TEXT BOOK PRACH THEY PRODUCED SECONDARY OUTCOMES MONOGRAPHS WEBSITE SCHOLARLY ARTICLE AND EBOOKS AND OVERALL PROGRAM ALIGNED WITH GOALS OF G13 AND FUNDED GRANTS COVERED WIDE RANGE OF SCIENTIFIC TOPICS AS OUTLINED HERE. SO, THIS IS THE LIST OF CURRENTLY FUNDED G13 PROGRAMS AND ACTIVE GRANTS OF THEM AND SEEING IN THE TITLE CURRENT PROGRAMS COVER WIDE RANGE OF TOPICS HISTORY OF HEALTH CARE IN US PRISON FLU MEASLES WRAS ETHNICITY AND HEALTHCARE EXCLUSION AND ALCOHOLIC PROBLEMS IN HIGH RISK COMMUNITIES DIVERSITY AND MEDICINE AND FORMATION OF MODERN MEDICINE AND TRADITIONAL MEDICINE. SO, FOR REISSUE OF G13 AND PROCESS FOLLOWING CHANGES TO CURRENT OPERATIONS OUTCOMES OF G13 FUNDED AWARDS SHOWS ARCHIVAL RESEARCH SUPPORTED BY MECHANISM TNT PRODUCE EXPECTED RESOURCE AND G13 ISSUE WILL SUPPORT WRITING PHRASES AND IF NO LONGER SUPPORTING ARCHIVAL RESEARCH AND APPLICATIONS PROVIDING CAREER MILESTONES TO ENSURE COMPLETION IN TIMELY MANNER AND BROADER MORE DIVERSE SET OF INVEST GATORS AND YOU NEED TO OBSERVE MORE EARLY STAGE INVESTIGATORS AND THOSE WITH NO PRIOR PUBLISHED WORKS LIMIT TO 1 PER PERSON AND ONE G13 FOR A LIFETIME AND CURRENT YEAR SUPPORT OF 15,000 PER YEAR MAXIMUM SUPPORT OF 50% OF EFFORT FOR PI'S PER YEAR UP TO 2 YEARS AND ALL I HAVE AND HAPPY TO ANSWER QUESTIONS. >> I DO HAVE A QUESTION. YOU NOTED MAYBE THE LAST SLIDE IS GETTING AT SOME CHANGES BUT WERE 39 FUNDED APPLICATIONS AND ONLY 14 PRODUCTS OF THE APPLICATIONS. BUT I KNOW THAT SOME ARE CURRENTLY IN PROGRESS. HAVE COMMENT ON ONES WHERE FUNDING HAS ENDED? A LITTLE ON THE TIMELINE HOW MANY ARE CURRENTLY FUNDED OR RECENTLY FUNDED AND REALLY SHOULDN'T COUNT AND NOTHING PRODUCED YET VERS VERSUS THOSE OVER FOR A WHILE AND PROBABLY SHOULDN'T AND NEVER WILL. >> SOME COMPLETED FUNDINGS THEY TAKE EXTRA 1 OR 2 YEARS TO SHOW PRODUCT OUTLINE AND THIS IS ACTUALLY VERY DIFFICULT TO MENTION OUTCOME AND COMPLETED PROJECT AND COMPLETING THREE YEARS AGO ACCOMPLISHING PRODUCT OUTLINE WORKING ON IT AND WITH PUBLISHERS TO COMPLETE PROJECT AND IT IS TRICKY ACTUALLY. >> OKAY. >> DO YOU HAVE SENSE HOW MANY YOU THINK WILL ACTUALLY NEVER RESULT IN A PUBLICATION? >> WE DON'T LET THEM GO WITHOUT GOING INTO ANYTHING. WE NAG THEM EVEN IF TAKES 2 OR 3 YEARS. >> OKAY. >> YEAH. >> ALL RIGHT. THANK YOU. QUESTIONS FOR ROBBY? >> COMMENT. RAVI, THAT IS A MOTIVATION BEHIND SHIFTING FROM SUPPORTING ARCHIVAL RESEARCH YEARS LATER RESULT IN PRODUCT TO SUPPORT FOR WRITING PHASE WITH MILESTONES AND EXPECTATIONS ESTABLISHED. >> YEAH. >> THANK YOU. >> ALL RIGHT. >> WILL THIS ALSO SUPPORT INTERNATIONAL RESEARCH AND PROJECT BASED UPON WORK ABROAD OR NOT ME BUT SOMEONE COULD APPLY FOR THAT AS WELL? >> WE DON'T SUPPORT FOLLOW IING APPLICANTS. >> NO. IF I HAD A PROJECT HERE. IF I HAD A PROJECT TO DO AND RESEARCH WOULD BE BASED IN ANOTHER COUNTRY LIKE PUBLIC HEALTH SCHOLARLY WORK OF RESEARCH. >> SHOULD BE I CAN LEAVE JERRY TO ANSWER QUESTION. >> OKAY. >> OKAY. >> DON'T KNOW IF I CAN ANSWER IT IF YOU WANT TO DO WORK. >> I CAN ANSWER. >> I CAN GIVE A MORE DEFINITIVE ANSWER. >> I CAN ANSWER, JERRY. >> THAT IS FINE. >> PLEASE GO RIGHT AHEAD. >> WITH THIS PROGRAM FOREIGN COMPONENTSES ARE NOT ALLOWED SO THAT IF MAKING ONE TRIP TO A CONFERENCE OR SOMETHING LIKE THAT, THAT IS ONE THING. IF YOU ARE DEPENDING ON DOG THE BULK OF RESEARCH IN FOREIGN COUNTRY IS NOT ALLOWED UNDER FUNDING MECHANISM. >> REJSZ I THOUGHT WE WOULD ONLY SUPPORT WRITTEN STAGE NOT SUPPORT RESEARCH AT ALL. >> RIGHT. YES. BUT -- BUT IN ORDER TO PRODUCE WRITING PRESS ES THERE IS RESEARCH GOING INTO IT WITH BACKGROUND INFORMATION. >> OKAY. >> WE HAVE TO GET CLARIFICATION THOUGH. SOUNDS LIKE WE ARE NOT SUPPORTING ANY NEW GENERATIVE RESEARCH AT ALL JUST DOCUMENTATION OF ONE'S RESEARCH. CAN YOU CLARIFY THAT, RICHARD. >> THAT IS THE CHANGE WE ARE PROPOSING FOR THIS VERSION OF G13. WE NO LONGER WILL PERMIT ARCHIVAL OR ANY RESEARCH AND INDIVIDUAL COMING TO TABLE WITH WRITING PLAN PHASE WITH DATA IN HAND AND CRITICAL COMPONENT ASSESSED IN REVIEW PROCESS INITIATIVE AND THIS IS ABOUT PRODUCING SCHOLARLY WORKS AND ENSURING SCHOLARLY WORKS ARE PRODUCED AND RESOURCES AVAILABLE TO SCIENTIFIC COMMUNITY TO FUND ARCHIVAL RESEARCH. THIS WORK WOULD FWIF PEOPLE PROTECTED TIEF TO TAKE RESEARCH THEY CONDUCTED AND FOCUS ON WRITING PUBLICATION WITHIN TWO YEARS. >> THANK 4 CLARIFIC. >> QUESTION FROM KRISTI. >> IS THERE WORKS LISTING THAT HAS BEEN CREATED IN THIS MECHANISM OVER TIME? I WOULD BE INTERESTED IN HOW THOSE TOPICS MAY HAVE CHANGED OR TO BE ABLE TO UNDERSTAND CONTRIBUTIONS THEY MAKE IN SCHOLARLY DISCOURSE. IS THERE A WEBSITE SOMEWHERE? >> WE WILL MAKE ON TO WEBSITE. I WILL MAKE NOTE OF IT. >> OKAY. THANK YOU. >> THANK YOU. >> ANDY? >> I'M CURIOUS ABOUT THE MOST SUCCESSFUL PROJECTS THAT HAVE BEEN FUNDED CAN YOU ASSESS WHAT WOULD IMPACT BE TO THOSE PROJECTS HAD THEY NOT SUPPORTED ARCHIVAL RESEARCH. ARE THEY READY TO GO AND JUST DO WRITING OR. >> CHALLENGE WITH THAT QUESTION IS THIS PROGRAM HAS LONGSTANDING HISTORY HERE THAT CHANGED OVER THE YEARS AND UNFORTUNATELY IN PROGRAM WE ARE FAIRLY ALL NEW AND HAVE ACCESS TO MORE ECREPT INFORMATION AND MORE RECENT APPLICATION AND MANY ARE STILL IN THE WRITING OR IN DATA COLLECTION OR WRITING PHASE AND LIBRARY FELLOWS LAUGH YEAR WE DID EVALUATION AND LOOKED AT INDIVIDUALS WITH VERY DEFINED TOPICS DIDN'T ASSESS IF DEFINED TOPIC MEANT THEY HAD DATA READY BUT APPEARS THOSE WITH DATA PRODUCE WORKS IN A TIMELY WINDOW THAT MAKES LOGICAL SENSE. >> YEAH. THANK YOU. >> GREAT. REMAINING QUESTIONS F FOR RAV. BY : >> DID WE ANSWER DR. WALKER'S QUESTION? I CAN REPHRASE IT. SOUNDS LIKE SOMEBODY LOCATED IN US WRITING ABOUT TOPIC WITH INTERNATIONAL DIMENSION AM I INTERPRETING THAT CORRECT WOULD IT FIT UNDER THIS MECHANISM >> WE WOULD HAVE TO BE CAUTIOUS THERE AND WOULD BE OPEN TO RECEIVING AN APPLICATION BUT GENERAL SPIRIT OF AWARD IS TOPIC HAS TO BE RELEVANT FOR US MARKET. >> YEAH. YEAH. >> NOT PRECLUDED BUT HAVE TO MAKE A STRONG CASE IT IS RELEVANT. >> HELPFUL CLARIFICATION. THANK YOU. >> ALL RIGHT. OTHER QUESTIONS OR CLARIFICATIONS? I ASK WE VOTE. ALL THOSE IN FAVOR OF THE PASSING THE CONCEPT, PLEASE RAISE YOUR HAND. ALL RIGHT. ANYONE OBJECTING TO THE CONCEPT? I SHOULD HAVE ASKED BEFORE ANYONE ABSTAINING? NO? OKAY. CONSIDER THE CONCEPT APPROVED. ALL RIGHT. NEXT TOPIC IS APPOINTING NOVEMBER NATING COMMITTEE FOR N BOR CHAIR TO REPLACE ME. I DON'T THINK WE NEED A VOTE ON N- >> LOOKING FORWARD TO NOMINATION AT NEXT MEETING. >> MEMBERS OF THE COMMITTEE WE LOOK FORWARD TO NOMINATIONS. THAT CONCLUDES THE OPEN PORTION OF THE MEETING. BOARD WILL MOVE ON TO THE CLOSED PORTION OF THE MEETING.