WELCOME, EVERYBODY.
THIS IS MY FINAL MEETING AS
CO-CHAIR -- AS CHAIR OF THE NLM
BOARD OF SCIENTIFIC COUNSELORS.
KING JORDAN IS JOINING ME, HE
WILL BE CHAIRING THE NEXT
MEETING, THE MEETING OF THE
NATIONAL LIBRARY OF MEDICINE
BOARD OF SCIENTIFIC COUNSELORS
IS CALLED TO ORDER.
AT MULTIPLE POINTS THE BOARD
WILL GO INTO CLOSED SESSION.
THE DESIGNATED STAFF WILL REMAIN
IN THE MAIN TEAMS MEETING ROOM.
EVERYBODY NOT DESIGNATED WILL
NEED TO LOG OUT OF TEAMS UNTIL
THE OPEN SESSION RESUMES AND
DURING CLOSED SESSIONS I'LL ASK
FOR VIDEOCAST TO BE DROPPED AND
RECORDING TO BE STOPPED.
SO, AGAIN, WELCOME, EVERYBODY.
I'D LIKE TO WELCOME TOBIN
TOESMICK, AND AD HOC REVIEWER
TODAY.
I WITH THAT I WILL TURN IT OVER
TO -- STEVE SHERRY IS NOT HERE.
DAVID, WERE YOU GOING TO COVER
HIS REMARKS?
>> RICHARD IS GOING TO SPEND
THE NEXT SEVERAL MINUTES
TALKING.
>> GREAT.
THANK YOU.
>> ALL RIGHT.
LET ME SHARE MY SCREEN.
ARE YOU SEEING THE CORRECT
VERSION?
>> I'M SEEING A COVER SLIDE THAT
SAYS NLM DIVISION OF INTRAMURAL
RESEARCH RECENT ACCOMPLISHMENTS.
>> OKAY, GREAT.
ALL RIGHT.
WELL, FIRST OF ALL, I WANT TO
THANK EVERYBODY FOR ATTENDING.
DR. SHERRY, OUR ACTING NLM
DIRECTOR, IS NOT ABLE TO JOIN US
TODAY BUT SENDS HIS REGARDS AND
HE ALSO WANTED TO EXTEND THANKS
TO THE BOARD MEMBERS FOR
PARTICIPATING IN THIS BOARD.
WE RELY ON YOUR CRITICAL
EVALUATION OF THE RESEARCH
PROGRAM THAT WE HAVE DEVELOPED
HERE AT THE NATIONAL LIBRARY OF
MEDICINE.
IT'S IMPORTANT WE'RE ABLE TO
DEMONSTRATE AN EFFECTIVE RETURN
ON INVESTMENT THAT WE'RE MAKING
IN BIOMEDICAL RESEARCH HERE AT
THE NATIONAL LIBRARY, AND YOUR
EVALUATION OF YOUR RESEARCH
PROGRAMS IS REALLY CRITICAL FOR
MAKING THAT DETERMINATION.
SO, WE WANT TO THANK YOU FOR
YOUR EFFORTS.
SO LET'S SEE.
NEXT SLIDE.
SO, I ALSO WANT TO SPECIFICALLY
THANK PETER TARCZY-HORNOCH,
BONNIE BERGER, AND BRASIELA
HERNANDEZ, THEY WILL BE LEAVING
US AFTER OUR MEETING TODAY.
WE APPRECIATE THE EFFORT YOU'VE
PUT INTO PROVIDING THIS BOARD
REVIEW FOR US IN THE PAST.
IN ADDITION I THANK PETER FOR
SERVING AS CHAIR FOR SEVERAL
YEARS, HAS DONE AN OUTSTANDING
JOB MAKING SURE WE'RE PROVIDING
A FAIR AND COMPREHENSIVE
EVALUATION SO WE APPRECIATE THE
EFFORT OF PETER AND OTHER BOARD
MEMBERS.
THERE ARE THREE MEMBERS WHO WERE
NOT -- WILL NOT BE JOINING US
TODAY.
NOEMIE ELHADAD, BEHNAZ GHORAANI
AROUND JOAN-EMMA SHEA, THEY WERE
TERMINATED BY THE NEW
ADMINISTRATION FOR REASONS NOT
CLEAR TO US.
BUT THEY WILL NOTE BE JOINING US
AT THE MEETING TODAY.
I ALSO WANT TO -- AS A RESULT,
SORRY, AS A RESULT OF THAT WE'RE
ACTUALLY GOING TO BE -- WE'RE IN
THE PROCESS OF RECRUITING SIX
NEW BOARD MEMBERS.
DAVID AND I HAVE BEEN WORKING
WITH OUR INVESTIGATORS TO
IDENTIFY APPROPRIATE CANDIDATES
WHO HAVE THE EXPERTISE NECESSARY
FOR EVALUATING OUR RESEARCH, AND
WE'RE IN THE PROCESS OF REACHING
OUT TO THEM AND RECRUITING THEM
AND HOPEFULLY WE'LL HAVE THEM ON
BOARD FOR OUR NEXT MEETING IN
OCTOBER.
AND FINALLY I WANTED TO ALSO
WELCOME TOBIN SOSNICK, SERVING
AS AD HOC MEMBER ON THE BOARD
TODAY, HELPING WITH THE REVIEW
OF ONE OF OUR TENURE TRACK
INVESTIGATORS, LAUREN PORTER.
SO THANKS AGAIN, EVERYONE, FOR
YOUR ASSISTANCE IN THIS REVIEW.
SO, I JUST WANTED TO USE THIS AS
AN OPPORTUNITY TO GIVE YOU KIND
OF A QUICK OVERVIEW OF THE
OVERALL ACCOMPLISHMENTS OF OUR
INTRAMURAL RESEARCH PROGRAM.
THE DIVISION OF INTRAMURAL
RESEARCH AT NLM HAS 17
INVESTIGATORS, 10 SENIOR
INVESTIGATORS, AND 7 TENURE
TRACK INVESTIGATORS.
THEIR PICTURES ARE ON THE SLIDE.
SORRY.
PROBLEMS WITH TEAMS.
JUST FOR YOUR AWARENESSN
FEBRUARY OF THIS YEAR TWO OF OUR
TENURE TRACK INVESTIGATORS WERE
PUT ON ADMINISTRATIVE LEAVE.
FORTUNATELY WE WERE ABLE TO HAVE
THAT -- THAT WAS PART OF THE
PROBATIONARY TERMINATION THAT
YOU MAY HAVE HEARD ABOUT THAT WE
EXPERIENCED IN FEBRUARY AT NIH.
FORTUNATELY WE WERE ABLE TO
ARGUE FOR THE CRITICAL ROLE THAT
OUR INVESTIGATORS ARE PLAYING IN
NIH'S MISSION AND WE WERE ABLE
TO GET BOTH INDIVIDUALS
REINSTATED, AND SO WE HAVE OUR
FULL COMPLEMENT OF INVESTIGATORS
AS OF TODAY.
SOMETHING ELSE THAT WE WANTED TO
MAKE SURE THE BOARD WAS AWARE OF
IS THAT WE HAVE HAD SOME
ADJUSTMENTS TO OUR
COMMUNICATIONS THAT WE'RE ABLE
TO MAKE HERE AT NIH.
IN JANUARY OF 24 A COMMUNICATION
PAUSE WAS INSTITUTED AT NIH,
THIS INCLUDED OUR ABILITY TO
SUBMIT SCIENTIFIC MANUSCRIPTS
FOR PEER-REVIEWED JOURNALS.
THE COMMUNICATIONS HAVE SLOWLY
BEEN OPENING UP SINCE THEN.
FORTUNATELY, THE MORATORIUM ON
MANUSCRIPT SUBMISSIONS WAS VERY
BRIEF, IT WAS REALLY ONLY A
MONTH OR TWO.
AND SO WE HAVE NOW AFTER THAT
BRIEF PAUSE BEEN ABLE TO SUBMIT
MANUSCRIPTS FOR SCIENTIFIC
PUBLICATION AND SO THAT HAS BEEN
A VERY POSITIVE KIND OF RECENT
CHANGE.
KIND OF RELATED TO THE PAUSE IN
COMMUNICATION WE WERE ALSO -- WE
HAD A BAN ON TRAVEL TO
SCIENTIFIC MEETINGS.
AS OF APRIL 10 THAT BAN HAS BEEN
LIFTED, SO NOW WE'RE ALLOWED TO
ONCE AGAIN ENGAGE IN OFFICIAL
TRAVEL, TO PARTICIPATE IN PUBLIC
SCIENTIFIC CONFERENCES FOR
PRESENTATION OF SCIENTIFIC DATA,
OR FOR THE PURPOSES OF LEARNING
ABOUT SCIENTIFIC RESEARCH.
AND SO THOSE ACTIVITIES HAVE
RESUMED.
AND THEN RELATED TO THIS, WE
ALSO HAVE -- WE'RE STILL KIND OF
PAUSED IN OUR ABILITY TO ISSUE
DOCUMENTS TO THE PUBLIC,
ESPECIALLY DOCUMENTS THAT RELATE
TO THINGS LIKE REGULATION,
GUIDANCE, NOTICES, GRANT
ANNOUNCEMENTS AND THINGS LIKE
THAT.
THOSE DOCUMENTS NEED TO BE
SUBMITTED TO THE -- AT THE
DEPARTMENT LEVEL FOR REVIEW AND
APPROVAL BEFORE THEY CAN BE
RELEASED.
AND SO THERE IS KIND OF
ADDITIONAL SCRUTINY ON PUBLIC
COMMUNICATIONS THAT WE'RE
CURRENTLY EXPERIENCING AND THAT
JUST REQUIRES US TO GO THROUGH
KIND OF THESE ADDITIONAL LEVELS
OF REVIEW AND APPROVAL.
THAT IS WORKING RELATIVELY
SMOOTHLY BUT IT'S A SLIGHT
CHANGE IN THE WAY WE'VE OPERATD
IN THE PAST.
NOW, IN SPITE OF THE KIND OF
PAUSE IN COMMUNICATIONS, I THINK
OUR INVESTIGATORS CONTINUE TO BE
VERY PRODUCTIVE IN TERMS OF
SCIENTIFIC PUBLICATIONS.
SO SINCE THE BEGINNING OF 2025,
WE'VE HAD 28 PEER-REVIEWED
JOURNAL ARTICLES AND CONFERENCE
PROCEEDINGS PUBLISHED, IN
JOURNALS, HIGH-IMPACT JOURNALS
LIKE NATURE, NATURE MEDICINE,
CELL, MOLECULAR SCIENCE AND
OTHERS.
WE HAVE 53 SUBMISSIONS UNDER WE
VIEW CURRENTLY, SO OUR
INVESTIGATORS REMAIN HIGHLY
PRODUCTIVE IN TERMS OF
PUBLICATIONS.
I'VE LISTED FOUR NOTEWORTHY
EXAMPLES HERE ON THE SLIDE.
LAUREN PORTER'S GROUP HAD A
NATURE PAPER THIS YEAR WHERE
THEY ARE REPORTING ON SOME
LIMITATIONS TO THIS VERY POPULAR
ALGORITHM CALLED ALPHAFOLD2 FOR
PROTEIN STRUCTURE DETERMINATION,
WHERE THEY FOUND CERTAIN ASPECTS
OF THE ALGORITHM THAT INCLUDE
SEQUENCE CLUSTERING HAVE AN
IMPACT ON THE PERFORMANCE OF THE
ALGORITHM AS YOU MIGHT EXPECT.
WE ALSO HAVE A PUBLICATION BY
PRITHCA, WHAT ROLE BIOLOGICAL
DATABASES HAVE IN THE AGE OF
GENERATIVE A.I. AND HOW
IMPORTANT THESE KIND OF
DATABASES REMAIN, IN THIS
CURRENT YEAR OF ARTIFICIAL
INTELLIGENCE.
THERE WAS A PUBLICATION BY
EUGENE KOONIN AND COLLABORATORS
IN MOLECULAR CELL WHERE THEY ARE
DEFINE A VERY INTERESTING NEW
PHENOMENON, PRESENCE OF MICRO
PROTEIN HIDDEN IN GENOMES, A NEW
DISCOVERY THAT I THINK WILL HAVE
AN IMPORTANT IMPACT ON OUR
UNDERSTANDING OF THE ROLE OF
PROTEOMES IN BACTERIAL
PHYSIOLOGY.
AND THEN FINALLY, FINAL EXAMPLE,
IS A PUBLICATION IN ANNUAL
REVIEWS OF BIOMEDICAL DATA
SCIENCE BY JOHN LU AND TEAM,
DEFINING THE CHALLENGES WITH
USING AN IMPLEMENTING LARGE
LANGUAGE MODELS IN HEALTHCARE
SETTING, ONE OF THE KINDS --
WHAT ARE THE KINDS OF
LIMITATIONS AND GUARD RAILS AS
WE TRY TO ADOPT NEW METHODS.
THESE ARE A COUPLE EXAMPLES OF
PUBLICATIONS FROM THE DIVISION.
MANY OF OUR INVESTIGATORS
RECEIVED RECENT AWARDS AND
HONORS, DAVID LANDSMAN, MY
DEPUTY DIRECTOR, WAS ELECTED AS
A FELLOW OF THE INTERNATIONAL
SOCIETY OF COMPUTATIONAL BIOLOGY
WHICH IS A VERY PRESTIGIOUS
ORGANIZATION.
DINA DEMNEH-FUSHMAN.
LAUREN PORTER APPOINTED TO
DIRECTOR.
THERESA PRZYTYKKA ELECTED TO
COLLEGE OF FELLOWS OF AMERICAN
INSTITUTE FOR MEDICAL AND
BIOLOGICAL ENGINEERING.
EUGENE KOONIN AWARDED HONORARY
DOCTORATES.
AUGUSTINE LUNA SELECTED AS
ORGANIZER FOR ISMB 2025 MEETING
AND COMMUNITY COORDINATOR FOR
THE COMBINED INITIATIVE WHICH IS
COORDINATING DEVELOPMENT OF
COMMUNITY STANDARDS AROUND
COMPUTATIONAL MODELS.
FINALLY I WAS PROMOTED TO SENIOR
INVESTIGATOR WITH TENURE AT NIH
EARLIER THIS YEAR.
OUR TRAININGS HAVE BEEN QUITE
SUCCESSFUL, AND SO THESE ARE NOW
TRAINEES GETTING READY TO EMBARK
ON THE NEXT PHASE OF THEIR
SCIENTIFIC CAREER, ONE
POSTDOCTORAL FELLOW OLIVER BEAR
DON'T WALK OFFERED ASSISTANT
PROFESSOR POSITION IN DEPARTMENT
OF BIOMEDICAL MEDICAL
INFORMATICS UNIVERSITY OF
WASHINGTON, THAT WAS A GREAT
ACCOMPLISHMENT.
AND THEN I'VE LISTED A NUMBER OF
POSTBAC FELLOWS IN A NUMBER OF
UNIVERSITIES, I WON'T GO THROUGH
ALL IN DETAIL BUT JUST TO
MENTION THE FACT THAT OUR
POSTBAC TRAINING PROGRAM HAS
ALSO BEEN VERY SUCCESSFUL.
SO, THE OTHER THING THAT I
WANTED TO COVER BRIEFLY IS OUR
NLM DIR ROADMAP.
I'VE TALKED ABOUT THIS IN PAST
MEETINGS BUT I'M HAPPY TO REPORT
THAT THE ROADMAP HAS ACTUALLY
BEEN COMPLETED TODAY, AS OF
TODAY, APRIL 30.
WE HAVE PRODUCED TWO VERSIONS OF
THE ROADMAP.
ONE THAT INCLUDES THE VISION AND
MISSION OF THE DIR AS WELL AS A
NUMBER OF GOALS AND SPECIFIC
OBJECTIVES.
THIS VERSION OF THAT DOCUMENT
WILL BE MADE AVAILABLE FOR
DISTRIBUTION TO THE PUBLIC.
AND THEN WE HAVE A SECOND
VERSION THAT IN ADDITION TO THE
MISSION GOALS AND OBJECTIVES
INCLUDES SPECIFIC IMPLEMENTATION
PLANS AND ASSESSMENT METRICS
THAT WE WILL BE USING FOR
INTERNAL USE THAT PROVIDES SOME
OF THE DETAILS ABOUT HOW WE'RE
ACTUALLY GOING TO ACCOMPLISH
THESE GOALS AND OBJECTIVES.
SO, I JUST WANTED TO LIST HERE
VISION AND MISSION STATEMENTS,
VISION TO GROW AS A CENTER OF
EXCELLENCE FOR INNOVATION AND
COMPUTATIONAL BIOLOGY AND HELP
INFORMATICS RESEARCH FOR
INVESTIGATORS ENGAGED IN A
VIBRANT, DIVERSE, COLLABORATIVE
COMMUNITY TO ADVANCE BIOMEDICAL
KNOWLEDGE AND IMPROVE HUMAN
HEALTH.
AND WE ARE ACCOMPLISHING THAT
VISION IN A MANNER CONSISTENT
WITH OUR MISSION, AND THAT IS TO
SERVE AS - TO RESEARCH AND
DEVELOP COMPUTATIONAL
DEVELOPMENT TO A BROAD RANGE OF
INFORMATION AND DATA SCIENCE
PROBLEMS IN BIOLOGY,
BIOMEDICINE, HUMAN HEALTH.
AND LEVERAGING UNIQUE REVERSES
AT THE NATIONAL LIBRARY OF
MEDICINE AS A KEY ASPECT OF OUR
RESEARCH PROGRAM.
AS PART OF THAT MISSION WE
PROMOTE THE FIELDS OF
COMPUTATIONAL BIOLOGY AND HEALTH
INFORMATICS THROUGH VARIOUS
OUTREACH AND TRAINING ACTIVITIES
THAT FOSTER AN ENVIRONMENT THAT
ATTRACTS THE DIVERSE TALENT AND
PERSPECTIVE NEEDED TO TACKLE THE
CHALLENGING BIOMEDICAL PROBLEMS.
SO THESE ARE KIND OF OUR VISION
AND MISSION STATEMENTS THAT
SERVE AS KIND OF THE NORTH STAR
FOR OUR ROADMAP.
AND THEN WE HAVE IDENTIFIED AS
PART OF THE ROADMAP FIVE HIGH
LEVEL GOALS IN A SERIES OF
SPECIFIC OBJECTIVES THAT WILL
SERVE AS KIND OF OUR BLUEPRINT.
WE WANT TO GROW THE DIR
COMPUTATIONAL RESEARCH BY
IDENTIFYING AND REALIGNING
RESEARCH PORTFOLIO TO HIGH
PRIORITY RESEARCH AREAS, TO KIND
OF FILL ANY GAPS THAT WE MIGHT
HAVE IN KIND OF EMERGING
COMPUTATIONAL RESEARCH AREAS.
AND ALSO TO EMPHASIZE CROSS-NIH
COLLABORATIONS AS WELL AS HIGHLY
INNOVATIVE HIGH RISK AND HIGH
REWARD TYPE PROJECTS THAT I
THINK WE'RE UNIQUELY POSITIONED
TO TACKLE.
SECOND GOAL IS TO ADVANCE
INNOVATIVE RESEARCH AND
DEVELOPMENT PROJECTS, TO
COLLABORATE WITH ENGINEERING
BRANCHES, BOTH AT THE NCBI AND
LISTER HILL ENGINEERING
DIVISIONS, TO IDENTIFY KIND OF
RESEARCH PRODUCTS THAT HAVE BEEN
VALIDATED TO THE EXTENT THAT
THEY COULD BE USEFUL PRODUCTION
LEVEL RESOURCES THAT WE COULD
RELEASE TO THE PUBLIC THROUGH
OUR ENGINEERING BRANCHES AND
THEIR PRODUCTION SYSTEMS.
AND SO THAT IS REALLY TO TRY TO
EMPHASIZE THE TRANSLATION OF OUR
RESEARCH ACTIVITIES INTO
PRODUCTION DEVELOPMENT.
THE THIRD HIGH LEVEL GOAL IS
TRAINING THE NEXT GENERATION OF
COMPUTATIONAL SCIENTISTS TO NOT
ONLY TRAIN THEM IN THE
COMPUTATIONAL METHODS BUT ALSO
HOW TO CONDUCT REPRODUCIBLE
SCIENCE AND TO HELP THEM DEVELOP
THE SKILLS THAT ARE NECESSARY
FOR SUBSEQUENT CAREER
ADVANCEMENT IN TERMS OF
MANUSCRIPT PREPARATION, GRANT
PROPOSAL DEVELOPMENT, AND THOSE
KINDS OF -- KIND OF RUNNING A
RESEARCH GROUP, THOSE KINDS OF
ANCILLARY SKILLS THAT ARE
NECESSARY TO BE A PRODUCTIVE
SCIENTIST.
FOURTH HIGH LEVEL GOAL TO RAISE
THE VISIBILITY OF DIR RESEARCH
SO THE COMMUNITY IS AWARE OF THE
KINDS OF INNOVATIVE WORK WE DO
HERE AT THE NATIONAL LIBRARY OF
MEDICINE.
FINALLY TO MODERNIZE OUR DIR
ADMINISTRATION, ESPECIALLY IN
THE CURRENT ENVIRONMENT, WHERE
THERE'S PRETTY SIGNIFICANT
REORGANIZATION OF ADMINISTRATION
AT NIH.
AND SO WE'RE GOING TO NEED TO BE
ABLE TO ADAPT TO THOSE CHANGES
THAT WE'RE EXPERIENCING HERE.
SO THOSE ARE KIND OF THE FIVE
HIGH LEVEL GOALS, AND SOME OF
THE SPECIFIC OBJECTIVES THAT ARE
GOING TO SERVE AS, YOU KNOW, OUR
BLUEPRINT FOR MOVING FORWARD AND
I HAVE TO SAY I THINK WE'RE VERY
EXCITED ABOUT KIND OF
REEMBARKING ON THIS ACTIVITY AND
DRIVING OUR DEVELOPMENT MOVING
FORWARD.
SO, JUST A QUICK SUMMARY.
HERE'S THE AGENDA THAT ALL OF
YOU RECEIVED IN ADVANCE.
WE'RE GOING TO BE REVIEWING ONE
OF OUR SENIOR INVESTIGATORS,
CLEM MCDONALD, AND TWO OF OUR
JUNIOR TENURE TRACK
INVESTIGATORS, JEREMY WEISS AND
LAUREN PORTER.
AND YOU CAN SEE THE SCHEDULE ON
THE AGENDA HERE.
THE OTHER THING THAT WE WANTED
TO JUST PUT ON THE TABLE IS THAT
FOR OUR UPCOMING OCTOBER 2025
MEETING WE ARE PLANNING ON
HAVING FOUR OF OUR INVESTIGATORS
REVIEWED AND THE PLAN IS TO HAVE
REVIEW CONDUCTED IN PERSON AS WE
HAVE DONE HISTORICALLY, YOU
KNOW, PRIOR TO COVID.
BUT BECAUSE WE HAVE FOUR
INVESTIGATORS AND WE'RE PLANNING
ON DOING THIS IN PERSON, WE'RE
PLANNING ON HAVING THE MEETING
LAST FOR A DAY AND A HALF.
AND SO THERE ARE TWO OPTIONS
HERE.
ONE IS WEDNESDAY AND THURSDAY
OCTOBER 15 AND 16, OR THURSDAY
AND FRIDAY, OCTOBER 16 AND 17.
AND WE'LL SEND OUT A POLL TO GET
FEEDBACK FROM THE BOARD ABOUT
WHICH WOULD BE YOUR PREFERRED
TIME FRAME.
TWO OF THESE FOUR ARE TENURE
TRACK.
SHANG AND LUNA ARE TWO OF OUR
TENURE TRACK INVESTIGATORS.
YOU'VE RECEIVED INSTRUCTIONS
ABOUT APPROACH TO REVIEWS,
INCLUDING THE SCORING RANGES AND
REVIEW CRITERIA WHICH INCLUDE
EVALUATING THE QUALITY AND
IMPACT OF OUR INVESTIGATOR
SCIENCE, EVALUATING THEIR
INTELLECTUAL LEADERSHIP IN THE
FIELD, PUBLICATION RECORD,
PROFESSIONAL LEADERSHIP, AND
AWARDS AND HONORS THEY HAVE
RECEIVED, AND THEIR TRAINING AND
MENTORING ACTIVITIES.
WE DID WANT TO REMIND THE BOARD
THAT WE WANT EACH OF OUR
INVESTIGATORS TO BE EVALUATED ON
THEIR OWN MERITS.
AND THAT MEANS AVOID COMPARING
ACCOMPLISHMENTS BETWEEN
INVESTIGATORS AT THE DIVISION.
WE WANT EACH INDIVIDUAL TO BE
EVALUATED ON THEIR OWN
ACCOMPLISHMENTS.
AND SO THAT'S AN IMPORTANT
CONSIDERATION.
ESPECIALLY FOR OUR TENURE TRACK
INVESTIGATORS.
FINALLY, I DID SUMMARIZE SOME
RECENT CHALLENGES WE'VE BEEN
NAVIGATING AROUND HIRING
FREEZES, COMMUNICATION FREEZES,
AND OTHER CONSIDERATIONS THAT
COULD POTENTIALLY HAVE AN IMPACT
ON THE ACTIVITIES OF OUR
INVESTIGATORS, ESPECIALLY THE
START-UP OF SOME OF OUR TENURE
TRACK INVESTIGATORS, SO WE
WANTED TO ASK THE BOARD TO, YOU
KNOW, KEEP IN MIND IMPACT THAT
SOME OF THOSE THINGS MAY HAVE
HAD ON OUR RESEARCH
ACCOMPLISHMENTS.
SO, WITH THAT, IF THERE'S SOME
MORE TIME, I'M HAPPY TO ANSWER
ANY QUESTIONS THAT THE BOARD
MIGHT HAVE BUT, AGAIN, I WANT TO
THANK EVERYBODY FOR VOLUNTEERING
THEIR TIME AND EFFORT TO HELPING
US OUT WITH THESE EVALUATIONS.
SO THANK YOU.
>> A HAND IS UP.
>> THANK YOU SO MUCH FOR THE
SUMMARY.
ALWAYS GOOD TO HEAR FROM YOU
GUYS, AMAZING ACTIVITY AND
REMARKABLE --
>> CAN YOU HOLD ON A SECOND.
WE'RE HAVING TROUBLE HEARING
YOU.
I'M GOING TO TURN THE VOLUME UP.
OKAY, TRY AGAIN.
>> THANK YOU FOR THE SUMMARY.
IMPRESSIVE RESEARCH, AS ALWAYS
AT NLM, NOT SURPRISING.
THANK YOU SO MUCH FOR THAT.
I HAVE A QUICK QUESTION AND I
COULD ASK PETER BUT I ALSO THINK
I WANTED TO ASK DAVID THIS
QUESTION, JUST BECAUSE IF I
DON'T KNOW HE'S GOING TO BE
LATER IN THE ROOM.
FOR TWO TENURE TRACK
INVESTIGATORS DO WE HAVE TO USE
SPECIFIC LANGUAGE ABOUT THEIR
PROMOTION TO TENURE OR ANYTHING
LIKE THAT?
ARE WE REQUIRED TO ALSO LOOK AT
THEIR FILES IN THAT CONTEXT?
AND DO YOU WANT US TO HAVE A
SESSION IN OUR -- SPECIFICALLY
IN OUR REPORT ABOUT THAT
PARTICULAR ITEM?
BECAUSE IN THE PAST WE HAVE HAD
THAT ISSUE WHERE WE DIDN'T KNOW
ABOUT IT UNTIL THE END, AND THEN
WE WEREN'T SURE, YOU KNOW, WE
REALLY DIDN'T SEE THE CANDIDATES
IN THAT LIGHT.
I THINK IT'S GOOD TO BRING IT UP
RIGHT NOW.
THANK YOU.
>> OKAY, I CAN ANSWER THAT
QUESTION.
SO, TWO OF OUR INVESTIGATORS
TODAY ARE TENURE TRACK, LAUREN
PORTER AND JEREMY WEISS.
JEREMY IS -- THIS IS HIS FIRST
REVIEW, SO HE WILL BE -- HE'S
ABOUT TWO YEARS INTO THE
PROGRAM, SO THE STATEMENT
PARAGRAPH OR SOMETHING LIKE THAT
SAYING -- MENTIONING HIS
PROGRESS IN THE FIRST TWO YEARS
WHICH IS NORMALLY A LAG TIME AND
THEN MAKE A STATEMENT ON TRACK
OR NOT ON TRACK FOR TENURE
CONSIDERATION.
WHEREAS WITH LAUREN, SHE'S CLOSE
TO SIX YEARS, IF NOT ALREADY SIX
YEARS IN THE PROGRAM.
AND SO WE'D LIKE SOME SHORT
PARAGRAPH SUPPORTING THE BSC
SUPPORT FOR TENURE TO PROCEED ON
TO THE TENURE, TO RECOMMEND HER
FOR TENURE.
THIS IS THE FIRST PART OF THE
TENURE REVIEW PROCESS.
THE BOARD SAYS YES, THIS PERSON
IS READY, THEN WE GO THROUGH
NINE MONTHS OF PAPERWORK TO GO
THROUGH THE TENURE PROCESS.
BUT YES, WE'D APPRECIATE
HAVING -- FOR EACH OF THOSE TWO
PEOPLE, A PARAGRAPH OR
STATEMENTS RECOMMENDING OR NOT
RECOMMENDING WHATEVER.
THAT INCLUDES -- THAT'S BESIDES
THE STATEMENT THAT YOU WOULD
MAKE ABOUT OUTSTANDING
EXCELLENCE OR WHATEVER.
>> YES, SO FOR JEREMY IS HE ON
TRACK, FOR LAUREN IS SHE READY
FOR CONSIDERATION.
>> AND ALSO OBVIOUSLY THE BETTER
SCORE, THE EASIER IT MAKES OUR
LIFE.
THEIR LIFE, ACTUALLY, NOT OURS.
THANKS.
GOOD QUESTION, SO THANK YOU.
BY THE WAY, I WANT TO SAY THAT
EVEN WITH ALL THE TROUBLES WE'VE
HAD I APPRECIATE ALL OF YOUR
ATTENDANCE TODAY VERY, VERY
MUCH.
AND SO WE'LL BE ABLE TO STAY
THIS WAY, FOR THE WHOLE DAY NOW.
IF YOU COME AND GO THROUGH THE
DAY, THAT'S FINE, BUT NOW --
THANK YOU.
SO IT'S 9:29.
I DON'T SEE WHY WE DON'T START
WITH PETER, CLEM IS REMOTE, SO
HE CAN TURN ON HIS MIC AND SHARE
HIS SLIDES.
WHO IS SHARING THE SLIDES?
AM I SHARING THE SLIDES?
OR, SEAN, CAN YOU SHARE HIS
SLIDES?
I SEE YOU'RE ONLINE.
THAT COULD PROBABLY BE EASIER.
SEAN?
>> GIVE ME ONE SECOND.
YEAH, GIVE ME ONE SECOND.
>> OKAY.
>> DAVID, WHILE WE WAIT, I'M
CURIOUS ABOUT OUR BOARD.
HOW MANY MEMBERS ARE SUPPOSED TO
HAVE, ESPECIALLY SINCE THE NEXT
MEETING WE WILL HAVE FOUR PEOPLE
TO REVIEW AND JUST LOSING THREE,
THE PEOPLE THAT IS ACTUALLY
LEAVING.
>> SO, WE ARE GOING TO BE
NOMINATING SIX NEW MEMBERS.
THIS IS JUNE, THERE WILL BE SIX
LEFT.
WE'RE NOMINATING SIX MORE.
WE CURRENTLY HAVE FOUR AND A
HALF THAT HAVE ACCEPTED.
I NEED ANOTHER PHONE CALL WITH
ONE PERSON.
SO THAT WILL BE FIVE AND ONE
MORE PERSON WHO TURNED US DOWN
BUT THAT'S MY -- WE'LL GET TO
THE SIXTH PERSON.
ONE PERSON AT A TIME, THEY ALL
GET SUBMITTED AS A GROUP, THE
CURRENT CHALLENGE RIGHT NOW IS
WE DON'T KNOW HOW THIS HAPPENS,
BECAUSE THE OFFICE WHO HANDLES
THESE COMMITTEE MEETINGS AND
THESE COMMITTEES IS NOT IN NLM
ANYMORE.
IT'S ABOVE NLM, SO THE PACKAGE
WILL LEAVE NLM.
I DON'T KNOW WHAT THE FUTURE --
HOW THAT'S GOING TO BE AND HOW
QUICK THAT WILL BE.
IF I SEE THAT IT'S NOT GOING TO
BE QUICK, I'M GOING TO ADD IN AT
LEAST THREE OR FOUR AD HOCS,
WHICH IS MUCH EASIER TO DO, FOR
SUPPORT.
SO LET'S KEEP TALKING ABOUT IT.
>> ALL RIGHT, CLEM, ARE YOU
READY TO GO?
IF YOU CAN SAY SOMETHING TO
CHECK YOUR MIC.
CLEM?
>> LOOKS LIKE ON MY END HE MUTED
HIS MIC AGAIN.
DR. MCDONALD, IF YOU CAN UNMUTE
YOUR MIC.
AND THAT WOULD BE AT THE VERY
TOP TOOL BAR.
THERE'S A RED BUTTON THAT SAYS
LEAVE, TO THE LEFT.
THERE YOU GO.
>> THERE YOU GO.
PRESS IT ONCE.
THANK YOU.
CLEM?
WE CAN HEAR YOU.
>> YOU CAN HEAR ME?
>> YES.
>> OH, GOOD.
>> PLEASE BEGIN.
>> PARDON?
>> YOU'RE ON.
>> I'M NOT FOLLOWING.
>> IT'S TIME FOR YOUR
PRESENTATION.
SOMEBODY OTHER THAN YOU, I
BELIEVE, IS SHARING THE SLIDES.
SO YOU'LL NEED TO LET THEM KNOW
WHEN TO ADVANCE YOUR SLIDES BUT
GO AHEAD AND START YOUR
PRESENTATION.
>> OKAY, SEAN, ARE YOU DOING THE
SLIDES?
>> YES, HE'S GOING TO --
>> YES, I'M SHARING THE SLIDES.
>> OKAY.
ALL RIGHT.
NEXT SLIDE.
SO THIS IS OUR AGENDA.
WE'LL FIRST TALK ABOUT THE
RESEARCH TEAM, BACKGROUND AND
OBJECTIVES, WORKFLOW, RESOURCES
AND INFRASTRUCTURE, HIGHLIGHTS,
FUTURE DIRECTIONS, SUMMARY OF
DISCUSSION.
NEXT SLIDE.
THE TEAM IS ME, A SCIENTIST, --
>> HE MUTED AGAIN.
>> I'M SORRY?
>> GO AHEAD.
>> CAN YOU HEAR ME?
>> YES.
>> OKAY.
THE LISTER HILL DIRECTOR.
NEXT SLIDE.
NOW, WE HAD A MARVELOUS STUDENT
WITH US DURING THIS LAST TOUR,
WHO IS NOW A RESIDENT PHYSICIAN
IN BRIGHAM AND WOMEN'S HOSPITAL,
HE DID A STUDY OF EFFECTIVENESS
OF DIRECT ORAL ANTI-COAGULATION
WITH RIVAROXABAN VERSUS APAXABAN
AND RISK OF HOSPITALIZATION FOR
BLEEDING.
THE BACKGROUND, OLDER AND
CHRONICALLY ILL PEOPLE ARE OFTEN
EXCLUDED FROM RANDOMIZED TRIAL,
POST MARKING SURVEILLANCE HELPS
UNCOVER REAL WORLD DRUG EFFECT,
AND WE NEED RETROSPECTIVE
REVIEW.
THAT'S WHAT OUR GAME HAS BEEN,
USING MOSTLY MEDICARE AS THE
LARGE DATASET.
NEXT SLIDE.
AND WE USE IT TO EXTEND DRUG
SAFETY FINDING AND TO
INVESTIGATE EMERGING AND
CONTROVERSIAL DRUG EFFECTS, AND
WE DISSEMINATED POLICY AND
CLINICAL GUIDANCE ON MEDICATION
USE.
NEXT SLIDE.
SO, HYPOTHESIS, TEST IF TARGET
MEDICATIONS ARE SAFE, IF THEY
HAVE ANY BENEFITS.
AND PER DATA WE USED
PRESCRIPTION AND MEDICAL RECORDS
AND METHOD WE USED SURVIVAL
ANALYSIS AND WE TEST HYPOTHESIS
USING COX REGRESSION WITH TIME
VARYING COVARIATES AND TIME
VARYING PROPENSITY SCORE.
NEXT SLIDE.
OUR DATA SOURCE IS MEDICARE VRDC
PROVIDING ACCESS TO AVAILABLE
RECORDS OF 100% OF MEDICARE
ENROLLEES, INCLUDES ENROLLMENT,
DEMOGRAPHICS, CONDITIONS,
SPENDING DATA 1999 TO 2022.
PART A HOSPITAL INSURANCE CLAIMS
INCLUDES 1999 TO 2022.
PART B MEDICAL CLAIMS ARE
PROVIDED BY THE SAME TIME
WINDOW, AND PART C MEDICARE
ADVANTAGE CLAIMS DATA INCLUDED
IN 2015 TO 2022.
AND PART D PRESCRIPTION DRUG
INSURANCE, CLAIMS DATA, 2006 TO
2022.
NEXT SLIDE.
A VERY LARGE DATASET FROM CMS
CAN SUPPORT EPIDEMIOLOGIC AND
SAFETY STUDIES, CRITICAL FOR
REAL WORLD EVIDENCE GENERATION.
EXCUSE ME.
NEXT SLIDE.
NOW, OUR STUDY, A NUMBER OF
THEM, HORMONE REPLACEMENT
THERAPY, USE OF TESTOSTERONE,
ESTROGEN AND PROGESTIN,
MEDICATION HYPERTENSION,
DIABETES, ANTI-COAGULATION,
ANTIBIOTICS AND OTHERS AND
COVID-19 TREATMENT, WE ALSO DID
SOME AREAS SOME STUDY OF LOINC
AS A STANDARD CODE FOR THAT
CLINICAL TEST.
WE PUBLISHED IN MENOPAUSE JAMA,
PLOS MEDICINE, PLOS ONE, AND
OTHERS, BRITISH MEDICAL JOURNAL,
THROMBOSIS, REAL WORLD OUTCOMES,
HEALTH DISPARITIES, ET CETERA.
NEXT SLIDE.
THE JOURNALS WE PUBLISHED IN.
NOW, BACKGROUND, MENOPAUSE, END
OF REPRODUCTION YEARS, HORMONES,
SYMPTOMS INCLUDE HOT FLASHES,
WEIGHT GAIN, ET CETERA, AND
LONG-TERM EFFECTS IF UNTREATED,
PROBLEMS WITH BONE HEALTH,
CARDIOVASCULAR HEALTH, ET
CETERA.
THE WHI HORMONE TRIALS, IT
SHOWED ESTROGEN AND PROGESTIN
THERAPY INCREASED BREAST CANCER,
STROKE, CARDIAC, CORONARY HEART
DISEASE, ESTROGEN ALONE THERAPY
INCREASED STROKE AND DEMENTIA,
AND WAS STOPPED PREMATURELY
BETWEEN 2002 AND 2004.
NEXT SLIDE.
SO WHI STUDY STUDIED ONE
PREPARATION, ORAL .625
MILLIGRAMS, AND ORAL -- THE SAME
DRUG PLUS, AND THE TYPE OF
ESTROGEN WAS CONJUGATED, EQUINE
ESTROGEN, TYPE OF PROGESTIN WAS
MEDROXY, ORAL ROUTE, STRENGTH
.625, AND A BLACK BOX WARNING ON
ARE FROM THE FDA ON ALL
ESTROGEN, QUESTION EFFECTS
ACROSS ROUTES OF ADMINISTRATION
AND STRENGTH.
NEXT SLIDE.
EXCUSE ME.
WE USED PRESCRIPTION AND MEDICAL
AND COUNTER RECORDS FROM
TEN-PLUS MILLION POSTMENOPAUSAL
WOMAN AND HORMONE THERAPY
CLASSIFIED AS ESTRADIOL, AND
OTHERS, TYPE OF PROGESTIN,
ADMINISTRATION ROUTE ORAL,
VAGINAL, TRANSDERMAL AND VILE.
IN TOTAL LOOKED AT 40 DISTINCT
HORMONE THERAPY PREPARATIONS.
NEXT SLIDE.
INCLUDING ALL-CAUSE MORTALITY,
DEMENTIA, CANCERS, BREAST, LUNG,
COLORECTAL, OVARIAN.
AND SIX CARDIOVASCULAR EVENTS,
ISCHEMIC HEART DISEASE, CORONARY
HEART -- CONGESTIVE HEART
FAILURE, VTE, STROKE, AMI.
NEXT SLIDE.
WE USED A TIME TO EVENT DATA
ANALYSIS INCLUDING COX
REGRESSION OF TEN-PLUS MILLION
FOR 13 PATIENT OUTCOMES,
ADJUSTING TIME DEPENDENT,
INCLUDING 40 HORMONE THERAPY
PREPARATIONS, DEMOGRAPHICS, TIME
FIXED, SOCIOECONOMICS, INCOME,
REGIONS, NORTH, SOUTH, ET
CETERA, RURAL RESIDENCY, 49
CHRONIC CONDITIONS, USED
PROPENSITY SCORE ADJUSTMENT.
NEXT SLIDE.
RESULTS -- EFFECTS VARIED BY
TYPES, ROUTES, STRENGTH OF THE
HORMONE THERAPY.
ESTROGEN THERAPY ASSOCIATED WITH
SIGNIFICANT RISK OF REDUCTION,
THAT'S IMPORTANT, REDUCTION, IN
ALL-CAUSE MORTALITY, DECLINE OF
19%, HAZARD RATIO 0.81.
BREAST CANCER REDUCTION 16%,
LUNG 13% REDUCTION IN LUNG
CANCER, 12% REDUCTION IN
COLORECTAL CANCER, 5% REDUCTION
IN HEART FAILURE, 3% REDUCTION
IN VTE, 4% IN A-FIB, 11% AMI, 2%
DEMENTIA.
EPT WAS LINKED TO SIGNIFICANT
INCREASED RISK IN BREAST CANCER,
E+ PROGESTIN INCREASED 10%,
INCLEESED E+ PROGESTERONE
INCREASE OF 19%.
BUT MITIGATED BY LOW VAGINAL --
CAN'T READ THAT ONE,
TRANSDERMAL, AND PROGESTIN, E2+
PROGESTIN.
NEXT SLIDE.
SO, THIS PAPER ABOUT THE USE OF
MENOPAUSAL HORMONE THERAPY AGE
5, EFFECT OF OUTCOMES BY TYPES
OF ROUTES HAD A GOOD SPLASH, TWO
MONTHS AFTER PUBLICATION RANKED
IN THE TOP 5% OF ALL 26 MILLION
RESEARCH OUTPUTS, COVERED 22
DIFFERENT NEWS OUTLETS, THAT WAS
FUN.
NEXT SLIDE.
COVID-19 STUDY, SO THIS IS A
SMALL -- WE DEVELOPED A VACCINE
PRIORITIZATION MODEL, PUBLISHED
IN MEDR 2021, AVAILABILITY OF
THE COVID-19 VACCINE, LIMITED
AVAILABILITY, SO WE WANTED TO
FIGURE OUT WHICH POPULATION
GROUP WOULD GAIN THE MOST
BENEFIT FROM VACCINATION.
WE USED MONTHLY DATA FROM THE
NATIONAL CENTER FOR HEALTH
STATISTICS ABOUT COVID-19
DEATHS, USING CD10 CODE UO 7.1.
NEXT SLIDE.
WE USED -- THE METHOD WAS
LOGISTIC REGRESSION ANALYSIS TO
ASSESS VULNERABILITY TO COVID
MORTALITY, COVID-19 MORTALITY,
BY AGE, RACE, SEX.
RESULTS, VACCINATING THE TOP 17
HIGH RISK GROUPS OF POPULATION
WHICH INCLUDED MALES AGED 85,
BLACK AND HISPANIC, COULD
PREVENT 47% OF COVID-19 DEATHS.
SO, THIS INFORMED EARLY VACCINE
DISTRIBUTION STRATEGIES.
NEXT SLIDE.
EXCUSE ME.
SOME BACKGROUND, INCIDENCE OF
LONG COVID IN THE ELDERLY IS
DIFFICULT TO ESTIMATE.
SYMPTOMS MAY BE CONFUSED WITH
OTHER CHRONIC CONDITIONS.
SO PROCESS WAS INCIDENCE
BASED -- SPECIFIC DIAGNOSIS ON
COVID MAYBE UNDERREPORTED, AND
POST-COVID HEALTH CARE
UTILIZATION PATTERNS ARE LIKELY
TO DIFFER FROM THOSE OF
INFLUENZA.
NEXT SLIDE.
WE USED AS OUR COVID-19 COHORT
WE USED CMS MEDICARE ENCOUNTER
RECORDS OF APRIL 2020, JUNE
2021, AND INFLUENZA COHORT USED
CMS MEDICAL ENCOUNTER RECORDS
FROM OCTOBER-MAY IN 2018-2019.
SO, WE DEFINED LONG COVID BASED
ON ICD-10 EM B94.8 SEQUELAE,
OTHER SPECIFIC INFECTIONS AND
PARASITIC DISEASES, SYMPTOM
BASED, W.H.O. CLINICAL
DEFINITIONS, DESCRIBED 11
SYMPTOMS THAT OCCURRED WITHIN 3
MONTHS OF INFECTION, AND LAST
FOR GREATER THAN 2 MONTHS.
THAT WAS OUR DEFINITION OF LONG
COVID.
NEXT SLIDE.
SO RESULTS IMPLICATION, CODE
BASED VERSUS SYMPTOM BASED LONG
COVID WAS REALLY OVERBALANCED
WITH THE SYMPTOM BASE.
2.6 VERSUS 30%, 2.6% VERSUS 30%
AMONG HOSPITALIZED COVID
PATIENTS UNDERREPORTED.
THE HIGHER BURDEN FROM LONG
COVID COMPARED TO POST INFLUENZA
CONDITIONS, SO THE INCIDENCE OF
DYSPNEA, FATIGUE, PALPITATIONS,
LOSS OF TASTE, SMELL, AND
NEUROCOGNITIVE SYMPTOMS,
INCIDENCE OF HOSPITALIZATION AND
OUTPATIENT VISITS INCREASED
ALSO.
HIGHLIGHTS, THESE STUDIES
HIGHLIGHT THE NEED FOR VIGILANT
MONITORING AND TAILORED
HEALTHCARE STRATEGIES FOR
ELDERLY INDIVIDUALS RECOVERING
FROM COVID-19.
NEXT SLIDE.
SO, SOME BACKGROUND, THERE'S NO
FDA-APPROVED MEDICATIONS FOR THE
PREVENTION OF LONG COVID.
THE WE HYPOTHESIZE OR ASKED
WHETHER TWO COVID-19 TREATMENTS,
PAXLOVID AND LAGEVRIO ARE
SOCIALED WITH REDUCED RISK,
LOOKED AT CMS PRESCRIPTION AND
MEDICAL ENCOUNTER RECORDS IN
2021 AND 2022.
NEXT SLIDE.
METHODS, WE ONLY CONSIDERED TWO
COVID TREATMENTS TAKEN WITHIN
FIVE -- PLUS OR MINUS FIVE DAYS
OF COVID-19 DIAGNOSIS WAS MOST
EFFECTIVE IN THAT WINDOW.
WE USED COX REGRESSION ANALYSIS
OF LONG COVID RISK ON -- SORRY.
EFFECTS ON LONG COVID RISK,
ADJUSTING FOR DEMOGRAPHICS,
SOCIOECONOMIC, GEOGRAPHIC
REGIONS, 51 CHRONIC CONDITIONS.
RESULTS, PAXLOVID DECREASED PCC
RISK BY 13%.
AND LAGEVRIO DECREASED BY 8%, SO
THIS ARGUES FOR USING ONE OR
BOTH OF THESE DRUGS TO MITIGATE
THE RISK GIVEN THE LACK OF
PCC-SPECIFIC TREATMENT.
NEXT SLIDE.
THIS IS A DIFFERENT SORT OF
POINT.
WE DID A LITTLE STUDY ON LOINC,
LOGICAL OBSERVATION IDENTIFIER,
CHANGING CODES, USED TO IDENTIFY
TESTS AND MEASUREMENTS IN
HEALTHCARE, THE AIM WAS ASSIST
ACCURACY OF LAB CODE MAPPINGS TO
LOINC, DEVELOPED AN ALGORITHM TO
AUTOMATICALLY CORRECT ANY
ERRORS, MANY -- MOST ERRORS.
NEXT SLIDE.
THAT WAS PUBLISHED IN 2023.
WE USED 183 MILLION TEST
RESULTS, 4,000 -- OVER 4,000
MEASUREMENTS FROM PCORnet,
OVER 60 DATA MARKS, USED
SOFTWARE TELLS AND MANUAL REVIEW
TO ESTIMATE RATE OF LOINC
MAPPING ERRORS.
MAPPING ERROR RATE WAS 4.6% IN
PCORnet, DECREASED TO 0.1%
AFTER THE APPLICATION OF
AUTOMATIC DETECTION AND
CORRECTION ALGORITHM.
AND THIS ALL SUPPORTS DATA
INTEROPERABILITY AND QUALITY
ATURNS. -- ASSURANCE, PUBLISHED
IN JANUARY 2023.
SO GOING FORWARD, WE PLAN TO
CONTINUE THIS LOINC SUBSAFETY
ACROSS POPULATIONS, WE'RE GOING
TO UPDATE THE MENOPAUSAL HORMONE
THERAPY STUDIES AND USING 13
MILLION POST-MENOPAUSAL WOMEN
DATA, WE WANT TO INFORM POLICY
AND CLINICAL GUIDANCE ON
MEDICATION USE IN THE ELDERLY.
THIS IS THE LAST ONE.
NEXT SLIDE.
WHATEVER IT IS.
FUTURE PLANS, WE HAVE SOME IDEAS
FOR METHODOLOGIC INNOVATION.
THE TEAM IS DEFINING A METHOD TO
INTEGRATE DEEP LEARNING WITH COX
REGRESSION MODELS.
ENHANCE ACCURACY OF SURVIVAL
ANALYSIS AND RISK PREDICTION,
SUPPORTS CUTTING-EDGE ANALYTICS
IN THE REAL WORLD DATA CONTEXT.
NEXT SLIDE.
I THINK THIS MIGHT BE IT.
NEXT SLIDE.
OH, OTHER PLANNED PROJECTS.
WE WANT TO LOOK AT BEING FAIR,
BOTH SEXES, LOOK AT TESTOSTERONE
REPLACEMENT THERAPY AND EFFECT
ON PROSTATE DISORDERS.
CURRENTLY THERE'S A PAPER WE
SUBMITTED WHICH IS UNDER
REVISION IN THE JOURNAL OF
CLINICAL ENDOCRINOLOGY AND
METABOLISM.
TESTOSTERONE SLIGHTLY INCREASED
RISK OF BENIGN PROSTATIC
HYPERTROPHY BY 3%, DECREASING
RISK OF PROSTATE CANCER BY 19%.
WE DON'T HAVE A BIOLOGIC
EXPLANATION FOR THAT.
IT INCREASED BENIGN PROSTATE
HYPERTROPHY, MINIMIZED WITH
TOPICAL, LONG-TERM, AVOIDING
AROMATASE INHIBITOR
CO-ADMINISTRATION.
NEXT SLIDE.
SO, ONGOING AND PLANNED
PROJECTS, WE'RE LOOKING -- WE
DRAFTED A PAPER WHICH IS UNDER
REVISION, ABOUT GLP-1 AND OPTIC
NEUROPATHY RISK, NOW UNDER
REVISION AT JAMA OPHTHALMOLOGY
COMPARED TO SECOND-LINE DIABETES
MEDICATIONS, GLP-1 RA, INCREASED
RISK OF OPTIC NEUROPATHY BY 16%.
AMONG FOUR GLP-1 RAs, ONLY
SEMAGLUTIDE AND LIRAGLUTIDE WERE
RINKED TO INCREASED RISK.
WE HAVE A PAPER ABOUT
INTEGRATION FRAMEWORK UNDER
REVIEW, DIABETES MED AND
PANCREATITIS, PRELIMINARY
RESULTS INDICATE GLP-1 RA NOT
ASSOCIATED WITH INCREASED RISK
OF PANCREATITIS IN CONTRAST TO
PRIOR EVIDENCE.
NEXT SLIDE.
WE HAVE ONGOING AND PLANNED
PROJECTS, COMPARISONS, BLEEDING
RISK, IN PREPARATION, A PAPER IN
PREPARATION.
PRELIMINARY FINDINGS INDICATE A
RISK.
THIS WAS DISSEMINATED BY A
STUDENT WHO WORKED WITH US FOR A
PERIOD OF TIME.
WE ALSO ARE WORKING ON DEEP
LEARNING, A NEW METHOD, A METHOD
FOR SURVIVAL ANALYSIS.
THIS IS IN PREPARATION, IT'S
PRETTY COOL.
NEXT SLIDE.
SUMMARY IMPACT WE MADE
SUBSTANTIAL CONTRIBUTION TO DRUG
SAFETY AND COVID-19 RESEARCH,
USED BIG DATA TO SUPPORT
CLINICAL DECISION MAKING.
WE'VE INNOVATED IN METHODOLOGY
AND DATABASE INFRASTRUCTURE.
AND ONGOING WORK POSITIONS TEAM
FOR CONTINUED LEADERSHIP.
NEXT SLIDE.
SO I WANT TO THANK
COLLABORATORS, DATA PARTNERS,
RESEARCH TEAM, ESPECIALLY
APPRECIATION TO NLM REVIEWERS.
SO QUESTIONS AND FEEDBACK ARE
WELCOME NOW.
>> GO AHEAD.
>> HI, CLEM.
THIS IS BILL HOGAN.
CURIOUS HOW YOU DECIDE WHICH
PROBLEMS TO WORK ON, HOW IS THAT
INFORMED?
>> I THINK THERE'S SORT OF
OPPORTUNITY -- OPPORTUNISTIC, SO
SOMETHING GETS A LOT OF
ATTENTION IN THE RESEARCH WORLD,
WE'LL JUMP ON IT.
SO IT'S NOT A VERY SCIENTIFIC
PROCESS.
DO YOU HAVE ANY THOUGHTS ABOUT
HOW WE COULD DO IT?
>> JUST WONDERING IF THERE'S --
SINCE THIS KIND OF IS IN THE
SPACE OF POST-MARKETING
SURVEILLANCE, IS THERE ANY
DISCUSSION WITH FDA AND WHAT
THEY ARE KIND OF WORRIED ABOUT
OR INTERESTED IN PURSUING?
>> WE HAVEN'T, BUT IT WOULD BE
A GOOD IDEA.
>> AND I'M -- IF I MAY GET
ANOTHER QUESTION IN, THE UNIFIED
CODE IN LOINC THAT YOU'RE DOING
IS THAT RELEVANT TO YOUR OWN
DATA ANALYSIS OR, YOU KNOW, I
WAS THINKING IF YOU'RE USING
MEDICARE CLAIMS, MAYBE THE LAB
DATA AREN'T THERE, MAYBE THEY
ARE AND I'M JUST NOT AWARE.
>> WELL, I'M NOT SURE HOW TO
ANSWER THAT.
IT IS -- IT'S USED WIDELY IN THE
HEALTHCARE SYSTEM, LOINC IS.
I DON'T THINK MEDICARE ACTUALLY
ACTIVELY USES IT.
NOT SO SURE THAT ANSWERED YOUR
QUESTION.
>> SO I GUESS I WAS JUST ASKING
IF IT'S -- IF THE LOINC WORK WAS
RELEVANT TO YOUR DATA ANALYTICS
ON DRUGS.
>> OH.
I DON'T KNOW, TO BE HONEST.
>> THIS IS KING JORDAN, GEORGIA
TECH.
THANKS FOR YOUR PRESENTATION.
I'M WONDERING ABOUT THE
IMPLICATIONS OF YOUR WORK.
OBVIOUSLY VERY RELEVANT FOR
POPULATIONS LEFT OUT OF RCTs
SO COULD HAVE AN IMPACT BOTH ON
CLINICAL DECISION MAKES, --
MAKING BUT ALSO AS POLICY.
HOW DOES YOUR GROUP WORK TO
TRANSLATE THESE RESULTS TO THE
REAL WORLD?
DO YOU HAVE CONNECTIONS WITH
POLICYMAKERS?
DO YOU INTERACT WITH CLINICAL
DECISION MAKERS?
I'M CURIOUS ABOUT THAT.
>> I DON'T THINK WE HAVE DIRECT
CONNECTIONS, EXCEPT THROUGH
PUBLIC FACING POLICYMAKERS.
I WISH WE DID.
BUT I DON'T THINK WE DO.
DO YOU HAVE ANY SUGGESTIONS?
>> NO, I DON'T, UNFORTUNATELY.
THAT'S ABOVE MY PAY GRADE.
BUT THAT SEEMS LIKE AN
OPPORTUNITY HERE.
YOU KNOW, THE WORK IS CLEARLY
RELEVANT IN THAT SPACE.
SO -- AND THE ATTENTION THE
MENOPAUSE PAPER GOT SUGGESTS THE
POSSIBILITY TO MAKE A REAL WORLD
IMPACT..
I HAVE MORE QUESTIONS BUT I'LL
HOLD OFF AND PASS IT.
>> I CAN'T SEE IF SOMEONE ELSE
RAISED A HAND.
IMPRESSIVE WORK.
THANK YOU SO MUCH FOR SHARING.
I SEE YOUR OUTSTANDING NUMBER OF
PUBLICATIONS AND SUCH, WONDERING
SOMEONE AT YOUR LEVEL OF
SENIORITY DOING THIS IMPRESSIVE
WORK HOW DO YOU FOLLOW UP WITH
THE PREVIEWS, QUESTIONS, WITH
DIFFERENT ANGLE, COLLABORATIONS,
WHAT YOU FEEL IS YOUR MENTORING
ROLE WITHIN NIH AND OUTSIDE NIH,
YOU KNOW, FOR THE NEW GENERATION
TO PASS ALL THIS?
>> CAN YOU RESTATE THE QUESTION?
>> YES, MY QUESTION WAS ABOUT
MENTORING AND I MEAN I DON'T
KNOW IF YOU HEAR ANY PART OF
WHAT I SAID, I'LL START BY
SAYING THAT YOU HAVE IMPRESSIVE
NUMBER OF PUBLICATIONS AND
DEFINITELY GREAT IMPACT IN OUR
RESEARCH WORLD AND I WAS
WONDERING ABOUT HOW DO YOU
ENVISION YOUR MENTORING, PASSING
THOSE SKILLS, BOTH AS A MENTOR
AND ALSO AS A COLLABORATOR.
>> SO, WE'RE -- I DON'T HAVE A
REGULAR FLOW OF MENTEES, AT
LEAST I HAVEN'T, BECAUSE I'M
REMOTE FROM NIH.
BUT WE HAVE HAD SOME GOOD
MENTEES, BECAUSE THERE'S A GUY
NAMED LUKE BEST WE MIGHT HAVE
MENTIONED, NOW AT BRIGHAM AND
WOMEN'S, WHO WE WERE FORTUNATE
TO SEE SUCH A QUICK STUDY, HE
ACTUALLY DID THE STUDY, SO WE
LOVE HAVING GOOD MENTEES.
AND I GUESS WE'VE BEEN LUCKY TO
HAVE HAD THEM.
DOES THAT ANSWER YOUR QUESTION?
>> YES, THANK YOU.
>> ANY NON-BOARD OF SCIENTIFIC
COUNSELOR QUESTIONS WHILE WE'RE
STILL IN PUBLIC SESSION?
OR MORE BOARD OF SCIENTIFIC
COUNSELOR QUESTIONS?
>> I HAVE ANOTHER QUESTION.
I NOTICED ON YOUR SLIDES THE CMS
DATA THAT YOU'RE USING ENDS IN
2022.
YOU DID -- OR GOES THROUGH 2022.
YOU DID A LOT OF WORK ON COVID,
OBVIOUSLY A MAJOR FOCUS OF YOUR
GROUP DURING THIS PERIOD.
WHAT KIND OF CHANGES DO YOU
ANTICIPATE IN THE RESULTS, THE
DATA YOU'RE SEEING, AS THAT CMS
DATA GO FURTHER BEYOND THE
PANDEMIC?
>> GOOD QUESTION.
WELL, THERE WAS CERTAINLY THE
SPIKE FROM COVID WILL DIMINISH.
BUT OTHERWISE, I MEAN, IT'S A
PRETTY STABLE POPULATION,
MEDICARE POPULATION.
THEY JUST GET OLDER.
AND BUT I DON'T THINK WE SEE
ANYTHING DRAMATIC AFTER THE --
EXCEPT POST-COVID.
I MEAN, FORGETTING ABOUT COVID
FOR A MOMENT.
>> SIMILARLY I NOTICE YOU'RE
WORKING ON GLPs.
THINKING AGAIN IF THE DATA GO
THROUGH 2022, MOST OF THE ACTION
OBVIOUSLY WITH GLPs HAS
HAPPENED VERY, VERY RECENTLY.
WHAT ARE YOU THINKING ABOUT IN
THAT SPACE AS THEY START TO ROLL
OUT MORE UPDATED DATA?
WHEN WERE GLPs FIRST APPROVED
AND PUT ON THE MARKET?
>> I DON'T KNOW EITHER, THE
LAST COUPLE YEARS.
I THINK THAT'S A VERY
INTERESTING THING TO LOOK AT.
I'M SURE IT WILL HAVE SOME
EFFECT.
I WOULD BET IT WOULD HAVE SOME
EFFECT, BECAUSE THEY ARE BEING
USED WIDELY, PRETTY EFFECTIVE
DRUGS, SO WE'LL LOOK AT THAT TO
SEE IF WE CAN SEE A BUMP IN
ANYTHING WITH RELATIONSHIP TO
AVAILABILITY OF THE GLP DRUGS.
>> CLEM, I HAVE A QUESTION WHICH
MERGES TWO QUESTIONS.
ONE OF WHICH WAS THE EARLIER
QUESTION, HOW DO YOU CHOOSE
AREAS TO INVESTIGATE, AS WELL AS
YOU DISCUSSED SOME ONGOING AND
PLANNED PROJECTS BUT I'D BE
CURIOUS WHAT YOU SEE AS
POTENTIALLY EMERGING BEYOND GHB
GLP.
>> IT'S NEW, SO THAT DREW US TO
IT, PRETTY STRONGLY.
I DON'T KNOW IF THERE'S
COMPARABLE -- ANYTHING
COMPARABLE BESIDES THAT RIGHT
NOW IN LABORATORIES, ET CETERA.
IF I HAD AN HOUR I MIGHT BE ABLE
TO THINK OF SOMETHING.
>> FAIR ENOUGH.
>> OTHER QUESTIONS?
>> I HAVE A SIMILAR QUESTION
ABOUT AREAS OF FOCUS.
IT CAUGHT MY ATTENTION THAT YOU
MENTIONED ONE OF YOUR -- ONE OF
THE THINGS THAT INTERESTS YOU IS
CONTROVERSIAL CLAIMS ABOUT THE
EFFICACY OR SAFETY OF DRUGS N
COVID THERE'S OF COURSE ALL
KINDS OF CONTROVERSY SWIRLING
OVER IVERMECTIN, DO YOU HAVE ANY
PLANS TO GO INTO THAT SPACE W
ITH RESPECT TO COVID?
>> MANY THINGS WITH RESPECT TO
COVID.
THE DRUG YOU MENTIONED, I DON'T
THINKS WE HAVE ACCESS TO DATA.
ISN'T THAT AN ANIMAL DRUG?
OR THAT'S --
>> IT IS.
IT'S USED IN HUMANS TOO.
IT'S BEEN PRESCRIBED BUT OFF
LABEL.
IT'S NOT APPROVED AS OF YET SO I
DON'T KNOW IF THAT DATA WOULD
EVEN COME THROUGH IN CMS.
>> WE'LL GET DATA ON IT BUT IT'S
AN INTERESTING -- MY
UNDERSTANDING THAT DOESN'T DO
ANY GOOD ANYWAY.
I MEAN, IT WAS JUST AN IDEA,
SOME PEOPLE ON THE EDGE, BUT I
MAY BE WRONG.
BUT I DON'T THINK IT'S REALLY
USEFUL DRUG FOR THE DISEASE.
MY RECOLLECTION.
SO BUT WE COULD LOOK AT THAT.
BUT I DON'T KNOW WHERE WE'RE
GOING TO GET DRUG DATA ON
IVERMECTIN BECAUSE I THINK IT'S
A VETERINARY DRUG.
WE MIGHT BE ABLE TO GET IT.
I DON'T KNOW.
IS THAT RIGHT?
>> IT'S A HUMAN DRUG TOO.
IT'S BEEN PRESCRIBED, BUT OFF
LABEL FOR COVID.
SO SINCE I DON'T KNOW MUCH ABOUT
THE CMS SYSTEM AND HOW THAT
WORKS, MEDICAID AND MEDICARE,
YOU MIGHT NOT BE ABLE TO GET IN.
IN GENERAL BEYOND IVERMECTIN
SPECIFICALLY IS A FOCUS AREA IS
CONTROVERSIAL DRUGS WHEN COVID
WOULD BE RIPE FOR THE
RETROSPECTIVE STUDIES YOU'RE
DOING.
>> I THINK THAT'S A GREAT IDEA.
WE'LL TAKE A LOOK AT IT.
SEAN, DO YOU HAVE ANY COMMENTS?
>> YOU'RE WELCOME TO BREAK INTO
YOUR CLOSED SESSION EARLY IF YOU
WISH.
>> ALL RIGHT.
WHY DON'T WE GO AHEAD AND DO
THAT.
SO, AT THIS POINT, THE BOARD OF
SCIENTIFIC COUNSELORS WILL
CONVENE A CLOSED SESSION WITH
JUST CLEM FOR FURTHER QUESTIONS
AND ANSWERS.
AFTER WHICH WE WILL -- AFTER 15
MINUTES WE'LL BREAK FOR A CLOSED
SESSION OF ASSESSMENT AND WE'LL
RESUME THE PUBLIC SESSION AT
11:20:00 A.M. EASTERN TIME.
ALL RIGHT.
SO, WE'RE OPENING UP THE
RECORDING, AND THE PUBLIC
SESSION FOR THE NEXT PART OF THE
MEETING OF THE NATIONAL LIBRARY
OF MEDICINE BOARD OF SCIENTIFIC
COUNSELORS, BROADCAST AS WELL,
AND I WILL WELCOME LAUREN PORTER
TO GIVE HER PRESENTATION.
>> GOOD MORNING.
I WANT TO TALK ABOUT
UNDERSTANDING ALTERNATIVE
PROTEIN CONFIRMATION, AND WE'LL
TALK ABOUT EXPERIMENT ALSO.
I'M SURE ALL OF YOU KNOW THAT
ALPHAFOLD HAS MADE A HUGE IMPACT
ON STRUCTURAL BIOLOGY,
PARTICULARLY THE PREDICTION
OFTEN VERY ACCURATE OF THREE
DIMENSIONAL STRUCTURES OF
PROTEIN SEQUENCES.
SO JUST A MAPPING FROM SEQUENCE
TO STRUCTURE THAT HAPPENS VERY
QUICKLY AND HAS MADE SUCH AN
IMPACT, SO ACCURATE, IT'S BEEN
USED IN SO MANY CASES THAT IT
WAS AWARDED THE NOBEL PRIZE A
FEW MONTHS AGO.
WITH THAT COMES A QUESTION,
WHAT'S NEXT FOR PROTEIN
STRUCTURE?
AND AS GREG BOWMAN WROTE IN AN
OPINION PIECE, PREDICTING
ALTERNATIVE CONFIRMATION IS NEW
FRONTIER.
THIS WAS SUPPOSED TO BE A VIDEO
TO SHOW YOU OUT WARD AND INWARD
FACING TRANSLATIONS OF THE MONO
CARBO LATE TRANSPORTER.
THIS IS SOMETHING THAT ALPHAFOLD
CAN PREDICT EASILY BUT
PREDICTING ALTERNATIVE
CONFIRMATION WHICH IS HERE IN
PINK IS A HARDER THING TO DO.
AND SO THIS HAS BECOME A FOCUS
OF PROTEIN STRUCTURE PREDICTION.
YOU'LL SEE A LOT IN MY TALK WHEN
I TALK ABOUT DOMINANT
CONFIRMATIONS THAT ALPHAFOLD CAN
PREDICT EASILY.
ALTERNATIVE IN PINK ARE HARDER
TO PREDICT.
SO THIS IS AN EXAMPLE MONO
CARBOXYLATE TRANSPORTER IS AN
EXAMPLE OF A RIGID BODY MOTION
BETWEEN TWO DOMAINS, THERE ARE
OTHER CHANGES YOU'RE FAMILIAR
WITH SUCH AS SUBDOMAIN
REORIENTATIONS, CRYPTIC POCKETS
IMPORTANT FOR DRUG DESIGN.
MY LAB LOOKS AT AN EXTREME
VERSION OF CONFIRMATIONAL CHANGE
KNOWN AS FOLD SWITCHING.
WHAT IS FOLD SWITCHING?
IT IS A DRAMATIC REMODELING OF
SECONDARY AND TERTIARY STRUCTURE
THAT LEADS TO CHANGES IN
FUNCTION AND REGULATION, THIS IS
RFH, N-TERMINAL DOMAIN MAINTAINS
THE STRUCTURE BUT C-TERMINAL
DOMAIN, NOT A PHYSICAL PATHWAY,
JUST A PATHWAY -- A LINEAR
PATHWAY BETWEEN TWO
CONFIRMATION, TWO VERY DIFFERENT
CONFIRMS, BINDING A SUBUNIT OF
THE RIBOSOME, ALPHA HELICAL FORM
IS IMPORTANT FOR SPECIFIC BIND
TOCK A DNA SITE CALLED OX.
BASED ON NOTATION THE BETA SHEET
CONFIRMATION IS EASY ONE TO
PREDICT, ALPHA HELICAL
ALTERNATIVE IS MORE DIFFICULT.
IT TURNS OUT THIS IS ONE EXAMPLE
OF 100 KNOWN FOLD SWITCHING
PROTEINS, BUT THERE ARE MANY
THAT SEEM TO BE INVOLVED IN
HUMAN DISEASE, SO HERE ARE A FEW
EXAMPLES, TUBERCULOSIS, CANCER,
ALZHEIMER'S, GASTROENTERITIS,
EVEN A COVID-19 PROTEIN THAT'S
BEEN FOUND TO SWITCH FOLDS.
NOT ONLY THAT, BUT MINOR
SEQUENCE MODIFICATIONS CAN ALSO
CAUSE MAJOR STRUCTURAL CHANGES.
SO, THIS IS A HUMAN CANCER
PROTEIN BCCIP, TWO ISOFORMS,
ALPHA AND BETA, SO THESE TWO
LINES DOWN HERE IN NAVY BLUE
REPRESENT 80% OF SEQUENCE,
COMPLETELY IDENTICAL.
SO BBPIP ALPHA AND BETA
SEQUENCES ARE IDENTICAL EXCEPT
FOR EXON SUBSTITUTION AND THIS
ONE SUBSTITUTION COMPLETELY
CHANGES THE STRUCTURES OF THE
PROTEINS.
RELATIVE TO EACH OTHER, IF YOU
ALIGN THEM THEY ARE 13 ANGSTROMS
DIFFERENT.
THEY HAVE DIFFERENT FUNCTIONS
THEMSELVES, SO BECAUSE THERE ARE
DIFFERENT STRUCTURES ALLOW THEM
TO HAVE THESE FOLDS.
OR THESE DIFFERENT FUNCTIONS.
THE QUESTIONS MY LAB HAS BEEN
INTERESTED IN ASKING ARE HOW DO
YOU MINOR CHANGES IN SEQUENCE OR
ENVIRONMENT CAUSE PROTEINS TO
SWITCH FOLDS AND FUNCTION?
THERE'S MANY REASONS WHY THIS
QUESTION IS SIGNIFICANT, FIRST
OF ALL THIS IS A REALLY AS I
SAID EXTREME EXAMPLE OF PROTEIN
DYNAMICS, CONFIRMATIONAL CHANGE,
COULD BE AS WE UNDERSTAND IT
BETTER WE CAN UNDERSTAND OTHER
SORTS OF CON FIRMATIONAL CHANGES
AS I'LL GIVE AN EXAMPLE LATER IN
THE TALK.
IT COULD LEAD TO DISCOVERY OF
BIOLOGICAL PATHWAYS, BECAUSE
THERE ARE FOLD SWITCHING
PROTEINS KNOWN TO BE INTEGRAL
INTO BIOLOGICAL PATH WAYS,
KILLING THE CLOCK.
MUTATIONS CAN LEAD TO
ALTERNATIVE FOLDED STATES.
I JUST SHOWED THAT A PRETTY
SMALL AXON SUBSTITUTION CHANGES
A PROTEIN'S FOLD AND FUNCTION.
ALSO SINGLE NUCLEOTIDE
POLYMORPHISMS CAN CHANGE PROTEIN
FOLD AND FUNCTION.
THERE'S A PROTEIN INVOLVED IN
NON-HODGKIN LYMPHOMA MEF 2B, SNP
CAUSES A MUTATION THAT SWITCHING
C-TERMINAL HELIX TO BETA SHEET,
BELIEVED TO INACTIVATE THE
PROTEIN.
THESE ARE THE QUESTIONS MY LAB
IS INTERESTED IN ASKING BUT
THERE ARE BARRIERS TO PROGRESS.
AS I MENTIONED, THERE ARE ONLY A
FEW EXAMPLES OF KNOWN FOLD
SWITCHING PROTEINS, SO THERE ARE
100 TO DATE.
SECOND, IN SPITE OF THE AMAZING
TECHNICAL ADVANCE THE ALPHAFOLD
TEAM MADE FOLD SWITCHING IS
DIFFICULT TO PREDICT, UNTIL
REASONLY NO KNOWN MUTATIONAL
PATHWAYS WERE KNOWN TO SWITCH
FOLDS, AND THE FIRST WAS
DISCOVERED, I CAN'T DISCUSS IN
THIS TALK BUT CAN SHOW YOU LATER
IF YOU HAVE QUESTIONS.
THERE AREN'T RELIABLE METHODS TO
PREDICT FOLD SWITCHING.
OUR LAB AND OTHERS MADE SOME
PROGRESS WHICH I'LL SHOW YOU BUT
THERE'S STILL A LOT OF ROOM TO
GROW.
I THINK PART OF THE REASON FOLD
SWITCHING IS UNDERREPRESENTED IS
THAT FOLD SWITCHING PROTEINS ARE
DIFFICULT TO HANDLE
EXPERIMENTALLY.
THIS IS FROM A DAY DOING
EXPERIMENTS, THIS WHITE STUFF
WHICH OUR SYSTEM IS PRONE TO
DOING, WHICH MEANS WE DIDN'T
COLLECT DATA THAT DAY BECAUSE
THE SAMPLE WASN'T STABLE ENOUGH
TO COLLECT IT.
THIS IS A PERENNIAL CHALLENGE.
WE'VE BEEN ABLE TO MAKE A BIG
ADVANCE ON THAT RECENTLY.
I T TURNS ON YOU FOLD SWITCHING
IS DIFFICULT TO PREDICT.
WE HAVE AN X-AXIS PREDICTION
ACCURACY OF DOMINANT, Y-AXIS
ALTERNATIVES.
HERE WE'RE QUANTIFYING ACCURACY
BY TM SCORE, SAME IF WE WERE
USING RMSD.
THIS DOTTED LINE IS IDENTITY
LINE.
I WANT YOU TO SEE THAT 94% OF
PREDICTIONS FALL BELOW THE
IDENTITY LINE, THAT IS THEY ARE
SKEWED TOWARD PREDICTING THE
DOMINANT CONFIRMATION.
ALTERNATIVE USING VANILLA MODEL
IS HARDLY PREDICTED AT ALL.
SO NOW A LOT OF METHODS FOR
IMPROVING ALPHAFOLD HAVE BEEN
DEVELOPED.
THIS CAME FROM TESTING.
WE GOT A LITTLE BIT OF
IMPROVEMENT, THESE ARE VANILLA
IMPLEMENTATIONS.
FOLD SWITCHING CAN BE TRIGGERED
BY BINDING ANOTHER PROTEIN OR
BINDING A DIFFERENT BIOMOLECULE.
WE TESTED IT ALSO ON ALPHAFOLD2
MULTIMER, AND 3, AND YOU CAN SEE
NEITHER DID VERY WELL EITHER
EVEN WHEN MODELING EVERY
INTERACTION WE COULD.
FURTHERMORE THERE WERE ENHANCED
SAMPLING METHODS THAT WERE MEANT
TO TIP ALPHAFOLD TO PREDICT ONE
CONFIRMATION OR ANOTHER AND YOU
CAN SEE THOSE DID NOT FARE VERY
WELL EITHER.
300,000, ALPHAFOLD HAS 35%
SUCCESS RATE WHICH IS MODEST,
AND ESPECIALLY CONCERNING
BECAUSE ALL OF THE STRUCTURES
FOR BOTH CONFIRMATIONS OF FOLD
SWICHRS WOULD HAVE BEEN IN
TRAINING SETS.
WHERE DO WE GO?
IS IT POSSIBLE TO PREDICT FOLD
SWITCHING FROM SEQUENCE?
BEFORE ALPHAFOLD WAS DEVELOPED
MY LAB DEVELOPED A METHOD TO
PREDICT FOLD SWITCHING FOR THE
FIRST TIME IN A FAMILY OF
PROTEINS AND THIS IS AGAIN THE
RFAH TRANSCRIPTIONAL REGULATOR.
AGAIN N-TERMINAL DOMAIN IS THE
SAME, C SWITCHES.
WHAT'S INTERESTING IT HAS
NUMEROUS HOMOLOGS THAT ONLY
ASSUME ALPHA HELICAL
CONFIRMATION, THEY ARE SINGLE
FOLDING PROTEINOME BETA SHEET.
COULD WE DISTINGUISH BETWEEN
SEQUENCES THAT SWITCH, BETWEEN
HELIX AND BETA SHEET AND ONESOE
BETA SHEET.
I CAN'T GET INTO DETAILS USING
CONSERVATION PATTERNS WITH A
SECONDARY PREDICTOR AND WERE
ABLE TO BE SUCCESSFUL.
SO THIS IS NETWORK DIAGRAM OF
ALL 15,000 SEQUENCES FROM THE
RFA IN THE SHEET FAMILY, NODES
REPRESENT CLUSTERS OF SEQUENCES,
THE SIZE OF THE NODES HAVE TO DO
WITH HOW MANY SEQUENCES ARE IN
EACH CLUSTER, SO THE SMALLEST
HAVE 75% OR FEWER, LARGEST HAS A
LITTLE OVER 1100, CONNECTED BY
EDGES HAVE AN AVERAGE SEQUENCE
IDENTITY OF AT LEAST 24%, TAKEN
FROM SOMETHING.
WE PICKED SEQUENCES FROM MANY
CLUSTERS WITH MANY PREDICTIONS,
AVERAGE PAIR-WIDE SEQUENCE
IDENTITY BETWEEN THE TWO
SEQUENCES WAS 35% BECAUSE WE
WANTED TO TEST FOR ROBUSTNESS.
WE DID IN THE LAB CIRCULAR
MEASUREMENTS, YOU CAN SEE
PROTEINS PREDICTED TO SWITCH
FOLDS DIFFERENT PROFILE THAN
THOSE SINGLE FOLDERS AND ALSO
ALPHA HELIX TO BETA STRAND
RATIOS SHOWED SEPARATION.
WE DID FOLLOW-UP NMR
EXPERIMENTS, ASSIGNED A COUPLE
AND LOOKED AT SECONDARY
STRUCTURES, NOT SHOWING THAT ON
THE SLIDE BUT THOSE WERE
CONSISTENT WITH WHAT WE WERE
FINDING.
NOW, THE INTERESTING THING FOR
FUTURE PREDICTIONS AND MORE
GENERALIZABLE PREDICTIONS IS
WHEN WE LOOKED AT OUR SIX
PREDICTED TO SWITCH FOLDS WHICH
MEANS THEY WOULD HAVE A GROUND
STATE OF HELICAL CONFIRMATION
AND WE TESTED THEM AGAINST THE
MOST ADVANCED METHODS AT THE
TIME, WE FOUND THAT ALL OF THE
DIFFERENT METHODS THAT WERE
PREDICTING THIS GROUND STATE OF
THESE PROTEINS ONLY PREDICTED
BETA SHEET.
THE ONLY EXCEPTION WAS E. COLI
RFAH FOR ALPHAFOLD2, THAT
STRUCTURE WAS PRESUMABLY IN ITS
TRAINING SET.
SO THEN THE QUESTION IS, SO CAN
WE MAKE ALPHAFOLD2 WORK BETTER
SO IT CAN MAKE MORE ROBUST
PREDICTIONS OF THE SORTS OF
PROTEINS THAT WE'RE INTERESTED
IN?
TO BE ABLE TO ANSWER THAT
QUESTION WE NEED TO UNDERSTAND
HOW ALPHAFOLD2 WORKS.
IF WE HAVE INPUT SEQUENCE IN
GENETIC DATABASE, WE MAKE A
MULTIPLE SEQUENCE ALIGNMENT OR
MSA, WHICH I'LL TALK ABOUT IN
THIS TALK.
ALPHAFOLD WILL MAKE A
REPRESENTATION, ALSO A PAIR
REPRESENTATION, THAT'S
INITIALIZED AS BASICALLY NOISE,
THAT'S PUT INTO A BLOCK THAT
THEN TAKES MULTIPLE SEQUENCE
ALIGNMENT AND PAIR
REPRESENTATION AND INFERS
PAIRWISE DISTANCE RESTRAINTS
THAT ARE USED PASSED INTO THE
STRUCTURAL MODULE WITH A SINGLE
SEQUENCE REPRESENTATION TO
OUTPUT 3D MODEL.
THIS IS THE INITIAL PASS THROUGH
ALPHAFOLD NETWORK, PASSED BACK
ALONG WITH THE UPDATED PAIR
REPRESENTATION, AND RECYCLED
NUMEROUS TIMES TO IMPROVE
INITIAL PREDICTION.
WHAT I WANT TO EMPHASIZE HERE IS
THAT EVEN THOUGH THIS IS KIND OF
AN EXPLANATION AT HIGH LEVEL HOW
ALPHAFOLD IS WORKING, IT'S STILL
BECAUSE IT'S A DEEP LEARNING
NETWORK A BLACK BOX.
WE DON'T REALLY KNOW WHAT'S
HAPPENING UNDER THE HOOD AND
WHAT ALPHAFOLD IS SEEING THAT'S
ALLOWING IS TO MAKE INFERENCES.
THIS IS AN IMPORTANT QUESTION
FOR MY LAB BECAUSE IF WE'RE
GOING TO BUILD A MODEL OR
IMPROVE UPON AN EXISTING MODEL
TO REALLY BE ABLE TO PREDICT
FOLD SWITCHING ROBUSTLY WE NEED
TO KNOW HOW IT WORKS.
NOW, IF YOU READ THE NOBEL PRIZE
ANNOUNCEMENT FROM 2024, THE
CLAIM IS THAT CO-EVOLUTION IS A
REALLY IMPORTANT PART IN DRIVING
ALPHAFOLD BASE PREDICTIONS,
BASICALLY WHEN YOU HAVE PAIRS OF
AMINO ACIDS THAT VARY AT A
STATISTICALLY SIGNIFICANT LEVEL
AND WHEN THAT HAPPENS THOSE
AMINO ACIDS HAVE BEEN FOUND TO
BE OFTEN VERY CLOSE IN SPACE.
SO THESE CAN BE -- CO-EVOLVING
RESIDUES CAN BE USED TO INFORM
DISTANCE MAP OR THE RESTRAINTS
THAT ALPHAFOLD IS USING TO BE
ABLE TO PREDICT THREE
DIMENSIONAL STRUCTURED
APPROACHES.
INDEED I'M SURE THIS IS HOW
ALPHAFOLD WORKS MUCH OF THE
TIME.
BUT WHAT I'M GOING TO TELL YOU
NOW IS THAT IS NOT HOW IT WORKS
FOR FOLD SWITCHING PROTEINS.
WHY IS CO-EVOLUTION NOT DRIVING
PREDICTIONS OF ALTERNATIVE
PROTEINS OF FOLD SWITCHING?
THIS COMES FROM STUDIES WE HAVE
DONE PREVIOUSLY WHERE WE LOOKED
AT ALPHAFOLD PREDICTIONS OF
FAVORITE PROTEIN RFAH, THIS
BLACK PAIRWISE DISTANCES ARE
THINGS WITHIN ANGSTROMS OF ONE
ANOTHER.
IF WE COMPARE TO CONTACT UNIQUE
TO THE BETA SHEET FOLD ON THE
LOWER DIAGONAL, THOSE ARE BLUE,
AND UNIQUE TO ALPHA HELICAL
CONFIRMATION WHICH UPPER
DIAGONAL ARE YELLOW, YOU CAN SEE
THERE'S GOOD CORRESPONDENCE
BETWEEN WHAT IS SEEN,
CO-EVOLUTIONARY INFORMATION, AND
BETA SHEET CONFIRMATION, NOT
GOOD CORRESPONDENCE BETWEEN WHAT
IT'S SEEING WITH THE ALPHA
HELICAL CONFIRMATION.
THAT'S TRUE FOR THIS MULTIPLE
SEQUENCE ALIGNMENT WHICH WE
EXTRACTED FROM THE ALPHAFOLD
MODEL SO WE KNOW THAT'S WHAT IT
USED TO MAKE THESE PREDICTIONS,
BUT IT IS TRUE, SORRY, IT IS
TRUE FOR ALL THE MULTIPLE
SEQUENCE ALIGNMENTS THAT IT'S
USING TO BE ABLE TO MAKE ITS
PREDICTION.
THEY ALL MATCH UP BETTER WITH OF
THE BETA SHEET THAN WITH THE
ALPHA HELIX WHEN WE ONLY DO ONE
PASS THROUGH THE MODEL.
INTERESTINGLY, IT'S THE
RECYCLING PROCESS THAT LEADS TO
THE CHANGE, SO INITIALLY IF YOU
PASS AN MSA INTO ALPHAFOLD IT
WILL PREDICT THIS BETA SHEET BUT
AS YOU RECYCLE WITH THE EXACTLY
NSA WE SHOWED WITH THE SAME
INFORMATION FOR BETA SHEET,
EVENTUALLY BECOMES HELICAL.
WE THOUGHT THAT'S STRANGE, HOW
IS IT INFERRING HELIX FROM
SOMETHING THAT HAS A VERY STRONG
BETA SHEET CONFIRMATION?
AND SO WE WERE ABLE TO -- OUR
THOUGHT WAS THIS HAS MORE TO DO
WITH LEARNING OF PROTEIN
STRUCTURE AND CO-EVOLUTIONARY
INFERENCE.
WE TESTED THIS DIRECTLY BECAUSE
ANOTHER ALPHAFOLD MODEL, MODEL
BASED ON ARCHITECTURE USING SAME
PROTOCOL, WAS DEVELOPED CALLED
C-FOLD.
THE ONLY DIFFERENCE IS THE WAY
IT WAS TRAINED.
SO IT WAS TRAINED ON A
CONFIRMATIONAL SPLIT OF THE PDP
SO NOT ALL ALTERNATIVE
CONFIRMATIONS WERE THERE, THE
DEVELOPER PATRICK BRYANT SENT ME
THE LIST, NO RFA HELICAL
STRUCTURES IN THERE.
THEN WE CAN PASS IN THE SAME
MULTIPLE SEQUENCE ALIGNMENT
THAT'S USED BY ALPHAFOLD TO GET
HELICAL CON FIRMATION, PASSION
INTO C-FOLD, END UP GETTING THE
BETA SHEET CONFIRMATION.
THIS STRONGLY SUGGESTS THIS
HELICAL CONFIRMATION IS INFORMED
BY ALPHAFOLD TRAINING RATHER
THAN CO-EVOLUTIONARY INFERENCE.
THIS MAY EXPLAIN WHY IT
STRUGGLED TO PREDICT NEW PROTEIN
FOLDS CORRECTLY.
WHEN YOU PASS THIS INTO
ALPHAFOLD, IT PREDICTS THE
STRUCTURE OF BCCIP BETA, WITHIN
2 ANGSTROMS, BUT 13 DIFFERENT
FROM THE CORRECT STRUCTURE.
HUMAN CO-INTERLEUKIN-2, AN
IMPORTANT PROTEIN, SOLVED BY
LOUIS CAGE THIS SUMMER.
THREE DIMENSIONAL ALPHAFOLD
PREDICTS MATCHES SOMETHING IN
THE TRAINING SAID BUT DOES NOT
MATCH WITH ANGSTROMS OFF.
FURTHERMORE, A DNA BINDING
PROTEIN AN SAYS ANCESTOR MATCHE
TRAINING SET BUT NOT
EXPERIMENTS.
CAN WE LEVERAGE THIS KNOWLEDGE
THAT WE HAVE, IF ALPHAFOLD IS
FOR ALTERNATIVE CONFIRMATIONS
MAKING ASSOCIATIONS IN ITS TRAIN
SET CAN WE LEVERAGE TO DO THAT
BETTER.
WE DEVELOPED A METHOD CALLED CF
RANDOM, WE HAVE INPUT SEQUENCE
AND FULL LENGTH MULTIPLE
SEQUENCE ALIGNMENT.
WE FROM THIS EXPECT WE'LL GET
THE DOMINANT CONFIRMATION
WITHOUT A PROBLEM.
THEN THE QUESTION IS HOW ARE WE
GOING TO GET ALTERNATIVE
CONFIRMATION.
TO BE ABLE TO GET THAT WE
GENERATE EXTREMELY SHALLOW
MULTIPLE SEQUENCE ALIGNMENTS TOO
SHALLOW FOR CO-EVOLUTIONARY
INFERENCE, THEY LIKE THE
RESEQUENCES.
WHAT WE THINK WE'RE DOING IS
JOGGING ALPHAFOLD MEMORY TO MAKE
ASSOCIATIONS WITH WHAT'S LEARNED
DURING TRAINING SO WE CAN RUN IT
ON A BUNCH OF SHALLOW RANDOM
SEQUENCE ALIGNMENTS AND IN AN
UNBIASED WAY SELECT THE MOST
LIKELY PROBABLE ALTERNATIVE
CONFIRMATION.
NOW WE CAN COMPARE CF RANDOM TO
ANOTHER METHOD FOR PREDICTING
ALTERNATIVE CONFIRMATIONS CALLED
AF CLUSTER WHICH CLAIMS TO WORK
BY CO-EVOLUTIONARY INFERENCE,
AND THE WAY THEY SAY IT WORKS IS
THAT YOU HAVE THESE SEQUENCED
CLUSTERS THAT CONTAIN ROBUST
CO-EVOLUTIONARY INFORMATION THAT
ENCODE ALTERNATIVE PROTEIN
CO-FOLDS.
YOU EXPECT IF WE RUN ALPHAFOLD
AND GIVING IT CONCERTED
INFORMATION FOR THE ALTERNATIVE
CONFIRMATION WHICH IS WHAT AF
CLUSTER CLAIMS THAT WOULD DO
BETTER THAN GIVING IT NOISY
RANDOM SHALLOW INFORMATION.
BUT ACTUALLY THE OPPOSITE IS
TRUE.
CF RANDOM OUTPERFORMS.
ALL THE EXAMPLES OF THE
STRUCTURES PREDICTED BY
AF-CLUSTER AND HAVE BEEN
EXPERIMENTALLY CONFIRMED, SO IN
THESE CASES YOU CAN SEE THAT
PRETTY MUCH ALLS CF RANDOM IS
MORE -- PRODUCES MORE ACCURATE
PREDICTS THAN AF CLUSTER OFTEN
WITH HIGHER CONFIDENCE,
ESPECIALLY FOR ALTERNATIVE
CONFIRMATIONS.
NOT ONLY IS AF CLUSTER MORE
ACCURATE BUT OVERALL WHEN WE
LOOK AT THE ENSEMBLE WE SEE IT
PRODUCES A BETTER COEFFICIENT IN
ALL CASES OF THE PROTEIN
FAMILIES TESTED, MUCH HIGHER
SUCCESS RATE, NOT ONLY MORE
ACCURATE AND SUCCESSFUL BUT ALSO
MUCH MORE EFFICIENT.
WE ONLY NEED TO DO A FEW RUNS ON
CHI MAG 2 AND IFH COMPARED TO
HUNDREDS FOR AF CLUSTER AND
NUMBER OF PREDICTIONS NEEDED IS
SUBSTANTIALLY FEWER TO REALIZE
ALTERNATIVE CONFIRMATION.
NOW WE COME BACK TO THE QUESTION
OF CO-EVOLUTION AGAIN AND ASK IS
THERE SOMETHING WE MISSING, IS
THERE CO-EVOLUTIONARY
INFORMATION USED BY CF RANDOM?
WE EXTRACTED THE EXACT MSAs
THAT CF RANDOM WAS USING TO
PREDICT, AND HERE IS WHAT WE
GOT.
IN PINK THIS IS THE DEPTH WE
SAMPLED THAT GAVE US ALPHA
HELICAL CONFIRMATION, THAT'S
LIKE IN A PINK BOX.
AND YOU CAN SEE WE'VE GOT OUR
COMMON CONTACTS, THE ONES UNIQUE
TO THE SHARED N-TERMINAL DOMAIN.
WE'VE GOT DOMINANT CONTACTS,
UNIQUE TO BETA SHEET,
ALTERNATIVE CONTACTS UNIQUE TO
ALPHA HELIX.
WHAT I WANT TO DRAW YOUR
ATTENTION TO HERE IS THAT EVEN
THOUGH THE ALPHA HELICAL
CONFIRMATION IS BEING PREDICTED,
THERE'S ESSENTIALLY NO
CO-EVOLUTIONARY INFORMATION
UNIQUE TO IT.
AT ANY SAMPLING.
THIS AGAIN SUGGESTS THAT
CO-EVOLUTION IS NOT DRIVING THIS
PREDICTION OF ALTERNATIVE
CONFIRMATION, TRUE NOT ONLY FOR
RFAH BUT ALSO TRUE FOR MAG 2 AND
CHI B.
THERE'S NO PINK INFORMATION.
THERE'S ACTUALLY MORE BLUE
INFORMATION THAN PINK.
SO, THAT'S INTERESTED THAT THIS
IS THE WAY IT SEEMS TO BE
WORKING BUT I'VE ONLY SHOWN
THREE EXAMPLES.
NEXT NATURAL QUESTION IS HOW
WELL DOES CF RANDOM GENERALIZE
ON FOLD SWICHRS?
AND WE DID THAT TEST, YOU CAN
SEE THAT CF RANDOM NOT ONLY
PREDICTS MORE SUCCESSFUL
PREDICTIONS, IT REQUIRES
SUBSTANTIALLY LESS SAMPLING, ALL
THE METHODS WITHOUT TEMPLATES,
WE NEEDED TO SAMPLE 30,000
STRUCTURES AS OPPOSED TO OVER
200,000.
AND IT TURNS OUT THAT CF RANDOM
GENERALIZES BEYOND FOLD
SWITCHING.
THIS IS USING THE HARDEST
EXAMPLE, WE'RE ABLE TO DO OTHERS
ALSO IMPORTANT BUT MAYBE NOT AS
TECHNICALLY CHALLENGING.
WE WERE ABLE TO BENCHMARK CF
RANDOM AGAINST TWO METHODS,
SHOWING YOU ONE HERE, AGAINST AF
SAMPLE 2 WHERE WE'RE USING OPEN
AND CLOSED CONFIRMATIONS OF
TRANSPORTERS OR PARAPLASMIC
BINDING PROTEINS.
THIS IS MORE ACCURATE IN
PREDICTING BOTH OPEN AND CLOSED
CONFIRMATIONS IN AF SAMPLE 2,
AND IT IS MUCH MORE EFFICIENT
MOST OF THE TIME.
AF SAMPLE 2 REQUIRES A THOUSAND
PREDICTIONS PER MODEL, WE
USUALLY ONLY NEED 200 TO REALIZE
MORE ACCURATE STRUCTURES.
THIS REALLY ATTESTS TO THE POWER
OF CF RANDOM AND TO APPROACH
WE'RE USING RIGHT NOW.
SO IF ALPHAFOLD ISN'T PREDICTING
THESE ALTERNATIVE CONFIRMATIONS
BY CO- EVOLUTIONARY INFERENCE
HOW IS IT WORKING?
WE LOOKED AT FOLD SWICHRS NOT
PREDICTED WELL.
THIS IS A PROTEIN, AN ALPHA BETA
CONFIRMATION.
WHEN WE RUN AF CLUSTER WE GET
THIS CONFIRMATION BACK MANY
TIMES, DON'T GET THREE HELICALL
CONFIRMATION AT ALL.
WHAT'S INTERESTING IF WE DO PDB
SEARCH OF THE SEQUENCE
REPRESENTED IN BLACK WE FIND
THREE HOMOLOGS THAT ASSUME THREE
ALPHA HELICAL BUNDLE STRUCTURE.
SO THE TARGET SEQUENCE, ONE
THREE ALPHA HELICAL BUNDLE
SEQUENCE, PASS THOSE INTO
ALPHAFOLD WE GET THREE ALPHA
HELIX BUNDLE STRUCTURES EVERY
TIME.
THIS IS WITH HIGH CONFIDENCE
BASED ON ALPHAFOLD'S SCORE.
SO THIS SUGGESTS ALPHAFOLD MAY
BE USING SOME SORT OF SEQUENCE
ASSOCIATION TO BE ABLE TO
PREDICT THESE ALTERNATIVE CON
FIRMATIONINGS OF FOLD SWITCHING
PROTEINS.
WE THOUGHT THIS WAS AN
INTERESTING OBSERVATION, COULD
WE TAKE IT BEYOND FOLD SWICHRS
AND USE IT FOR OTHER PROTEIN
FOLDS AS WELL?
MADELINE IN MY LAB HAS BEEN
HELPING TO INVESTIGATE THIS
QUESTION, SO WE HAVE AN UNBIASED
SET OF PROTEIN STRUCTURES WITH
50 TO 3 300 AMINO ACIDS, HAVE N
ABLE TO FIND SEQUENCES ACROSS
FAMILIES AND SIZES THAT ALLOW
ALPHAFOLD TO PREDICT
CONSISTENTLY JUST ONE PROTEIN
FOLD.
AGAIN, THIS ISN'T FOLD SWITCHING
ANYMORE.
WHAT'S GREAT IS IT ALLOWS US TO
BETTER UNDERSTAND WHAT'S
HAPPENING WITH THE MODEL.
SO HOW CAN WE INVESTIGATE HOW
ALPHAFOLD IS PREDICTING?
WE CAN DO A BUNCH OF RANDOM
ALANINE SUBSTITUTIONS, MANY
TIMES, AND THEN WE CAN DO
STRUCTURE PREDICTIONS ON ALL OF
THEM.
SO WE'RE DOING 3,000 ALANINE
KNOCKOUTS FOR PROTEIN AND
SCORING OUR STRUCTURES BY
ACCURACY.
THEN WE CAN USE A LINEAR
CLASSIFIER AND SELECT THE
HIGHEST WEIGHTS TO IDENTIFY
WHICH AMINO ACIDS MAY BE THE
MOST IMPORTANT FOR PREDICTING
THIS CONFIRMATION.
WHAT'S REALLY INTERESTING ABOUT
THIS APPROACH IS 70% OF THE
SECOND SEQUENCE TURNS OUT TO BE
ALANINE.
ONLY 30% OF THE INFORMATION IS
WHAT SEEMS TO BE REQUIRED TO BE
ABLE TO PREDICT THIS
CONFIRMATION THAT WE'RE
INTERESTED IN.
NOW, HERE IS WHERE THE RUBBER
MEETS THE ROAD.
WHAT HAPPENS IF WE COMPARE THE
FULL MSA PREDICTION OF THIS
PROTEIN WITH THE REDUCED TWO
SEQUENCE MSA THAT'S 70% ALANINE?
TURNS OUT THEY ARE COMPARABLE.
SO, YOU CAN SEE HERE THE
STRUCTURES ARE WITHIN ONE
ANGSTROM OF EACH OTHER, PLDTs
ARE PRETTY MUCH THE SAME SO IT
DOES SEEM THIS APPROACH IS
IDENTIFYING AT LEAST IN SOME
CASES INFORMATION THAT ALPHAFOD
IS USING.
WHERE ARE RESIDUES IN THE
PROTEIN?
THEY ARE ALL OVER THE PLACE.
SOME OF THEM ARE POINTING INWARD
AS WE WOULD EXPECT FOR CO-
EVOLUTIONARY INFORMATION, SOME
AREN'T.
I'VE DONE MORE STUDIES AND
ANALYSIS AND I CAN TALK TO YOU
MORE ABOUT THAT LATER.
BUT FOR NOW WHAT I WANT TO SAY
WHAT WE'RE TRYING TO DO IS IN
OUR NEXT STEPS APPLY TO HUNDRES
OF PROTEINS TO GET A SENSE OF
WHAT ALPHAFOLD HAS LEARNED TO
DEVELOP A MORE EFFICIENT MODEL
PREDICTING FOLD SWITCHING
PROTEINS MORE DIRECTLY.
IF WE UNDERSTAND PATTERNS
ALPHAFOLD IS USING TO PREDICT
CONFIRMATIONS WE CAN TEST THOSE
PATTERNS AGAINST GIVEN SEQUENCES
AND MAYBE BE ABLE TO ACHIEVE
ALTERNATIVE CONFIRMATION MORE
EFFICIENTLY THAN WE CAN NOW.
IN THE MEANTIME WE STILL HAVE CF
RANDOM WHICH WE HAVE RUN --
>> QUICK QUESTION.
TIME CHECK HERE.
I SEE YOU'RE ON 50 OF 81 SLIDES,
THE SCHEDULE HAS YOU FINISHING
NOW.
SO --
>> I'M NOT ACTUALLY -- I'M GOING
TO 60.
>> OKAY.
WE'RE GETTING CLOSE.
ALL RIGHT.
SO WE RAN CF RANDOM ON 2,000 E.
COLI PROTEINS AND GOT
PREDICTIONS OF ALTERNATIVE
CONFIRMATIONS FROM THOSE 2,000.
AND WE LOOKED FOR EVIDENCE OF
EVOLUTIONARY SELECTION OF THOSE
PROTEINS.
AND FROM THAT WE PREDICT UP TO
52 PROTEINS FROM THE E. COLI
GENOMES WHICH FOLDS, THEY HAVE
FUNCTION IN LINE WITH WHAT WE
SEE OF FOLD SWITCHING PROTEINS
SO TRANSCRIPTION TRANSLATION
REGULATOR IS SOMETHING WE
ALREADY KNOW IS ENRICHED IN FOLD
SWITCHING PROTEINS, FROM THIS WE
FIND -- WE ESTIMATE UP TO 5% OF
E. COLI PROTEINS SWITCH FOLDS
WHICH WE ARE WORKING WITH BRIAN
AND LOUIS TO TEST ALL OF THESE
PREDICTIONS EXPERIMENTALLY.
IN THE MEANTIME WHAT WE NEED ARE
EXPERIMENTAL METHODS TO BE ABLE
TO TEST FOR FOLD SWITCHING AT
SCALE.
LESLIE IN MY LAB IN
COLLABORATION WITH CAITLIN DAVIS
HAS DEVELOPED A FRET BASED ASSAY
FOR PROTEINS WITH DIFFERENT
PREDICTED END TO END DISTANCES,
WE'VE TESTED ON RFAH PROTEINS
AND PROTEINS WE CHARACTERIZED IN
THE LAB.
IT WORKS 13 OUT OF 15 TIMES AND
THE GREAT THING IS IT'S IN
CELLS, DOES NOT REQUIRE ANY
PROTEIN PURIFICATION.
SO WE WANT TO USE THE CELL AT
LARGE SCALE, LESLIE IS WORKING
ON THAT FOR RFAH AND THEN EXTEND
TO OTHER SYSTEMS.
WHAT WE WANT TO UNDERSTAND HOW
FOLD SWITCHING WORKS.
THAT MAY INFORM US TO BE ABLE TO
MAKE BETTER PREDICTIONS, ALSO
UNDERSTANDING MECHANISM IS
IMPORTANT FOR PROTEINS IN
GENERAL.
SO WHAT WE WANTED TO DO IS WE
HAVE THIS C-TERMINAL DOMAIN OF
RFAH WHICH IS HELICAL, BETA
SHEET IN ISOLATION.
WHAT WE WANT TO DO TO TITRATE
IN, FORCE IT TO ASSUME HELLICAL
CONFIRMATION BUT THE TECHNICAL
CHALLENGE IS THE NATIVE
N-TERMINAL DOMAIN OF RFAH IS
SOLUBLE TO 2 MICROMOLAR AND WE
END UP WITH AGGREGATION
PROGRESS.
WE NEED AT LEAST HUNDREDS OF
MICRO MOLARS TO DO THESE
EXPERIMENTS.
JOELY IN MY LAB WILL TELL YOU
MORE.
HE HAS BEEN ABLE TO DESIGN A
SOLUBLE N-TERMINAL DOMAIN, UP TO
AT LEAST 500 MICROMOLAR,
POSSIBLY MORE.
AND WE CAN SEE WHEN WE LOOK AT
THE ISOLATED C-TERMINAL DOMAIN
IT HAS ONE CHEMICAL SHIFT
PROFILE BY NMR.
IF WE ADD IN UNLABELED
N-TERMINAL DOMAIN WE SEE A
DRAMATIC SHIFT IN THE CHEMICAL
SHIFT PROFILE AND NATHAN IN THE
LAB IS WORKING ON ASSIGNING THIS
PROTEIN.
YOU CAN SEE WE'RE SEEING HELICAL
CHARACTER.
DOES SEEM THIS PROTEIN IS
WORKING.
THIS SYSTEM IS WORKING AS
EXPECTED AND COLLABORATING TO
TAKE SNAPSHOTS OF THE FOLD
SWITCHING PROCESS AS MUCH AS WE
CAN.
SO WITH THAT I'D LIKE TO THANK
MY LAB MEMBERS AND MY
COLLABORATORS, AND I'M HAPPY TO
TAKE QUESTIONS.
I APOLOGIZE FOR GOING OVER.
IT WAS HARD WITH THE TEAMS TO BE
ABLE TO ADVANCE MY SLIDES.
THANK YOU FOR YOUR PATIENCE WITH
THAT.
>> OKAY.
I THINK WE CAN RESET THE SYSTEM
HERE SO THAT WE DON'T HAVE -- WE
CAN DISPLAY PROPERLY ON THE
MONITORS IN THE ROOM HERE.
AUDIO/VISUAL GUY CAN BE HERE
MOMENTARILY, WE MIGHT DROP YOU
FOR A MOMENT.
.
>> DO YOU WANT TO STOP SHARING?
>> OKAY.
I'LL BE BACK.
>> STOP SHARING.
>> OH, STOP SHARING.
>> WE HAVE SOME QUESTIONS,
LAUREN, FROM THE BOARD.
BONNIE AND ANDRE A, GO AHEAD.
>> WE HAD TO RESET THE KODECK IN
THE ROOM SO IT'S GOING TO BE
JUST A MOMENT BEFORE THEY ARE
GOING TO BE ABLE TO HEAR YOU.
IF YOU COULD GIVE US A VERY
BRIEF PAUSE, WE'LL BE BACK
ONLINE IN JUST A MOMENT.
THANKS.
LOOKS LIKE THEY ARE BACK.
>> LOOKS LIKE LAUREN WAS STILL
THERE IN THE ROOM.
NOW SHE'S GONE.
>> I'M GONE NOW?
>> NO, SHE'S THERE.
>> WHERE?
NOW I CAN SEE HER.
I CAN SEE YOU, LAUREN.
THEN THERE ARE QUESTIONS FROM
BONNIE AND ANDREA.
GO AHEAD.
>> I WAS WONDERING HOW CF RANDOM
WORKED.
>> WHY DO I THINK IT WORKS?
>> NO, I DON'T EVEN KNOW WHAT
THE ALGORITHM IS.
WHAT YOU'RE USING FOR CF RANDOM.
>> SURE.
LET ME TRY TO SHARE MY SLIDES
AGAIN.
YEAH, I HAD TO BREEZE PAST THAT.
SO, LET ME GO BACK AND SHOW YOU.
THIS IS NOT THE MOST UPDATED BUT
IT DOES WELL ENOUGH.
SO WHAT WE'RE DOING --
>> BEAUTIFUL WORK, I SHOULD HAVE
SAID.
>> THANK YOU.
SO, WE HAVE -- WE'RE USING --
WHAT WE HAVE, WE'RE INPUTTING
JUST OUR INPUT SEQUENCE, AND A
FULL LENGTH MULTIPLE SEQUENCE
ALIGNMENT INTO THAT.
THAT GIVES DOMINANT CON
FIRMATION.
THE PROBLEM IS GETTING
ALTERNATIVE CONFIRMATION THAT'S
HARD.
WHAT WE'RE DOING IS SAMPLING
RANDOMLY GIVING OTHER PASSES
EXTREMELY SHALLOW MSAs, Seq
1 AND SEQ2, AND WE THINK WE'RE
DOING IS JOGGING ALPHAFOLD'S
MEMORY TO SEE HOW IT'S MAKING
ASSOCIATIONS WITH RANDOMLY
SHALLOW ALIGNMENTS.
LET ME SHOW WHAT YOU WE USED ON
THE E. COLI GENOME.
THIS IS WHY I HAVE THESE
BACKUPS.
SHOOT, DO I NOT HAVE IT?
HERE IT IS.
OKAY.
HERE WE GO.
WE'VE GOT OUR DOMINANT STRUCTURE
SEARCH, FULL MSA, AND OUR
SHALLOW MSA.
WE'RE CLEAR ON THAT.
NOW WE NEED AN UNBIASED WAY TO
PREDICT ALTERNATIVE
CONFIRMATIONS BECAUSE HOW ALSO
ARE WE GOING TO RUN IT?
>> PEOPLE ARE HAVING TROUBLE
SEEING YOUR SLIDES.
MONA, I WOULD GO THROUGH LIKE A
DIFFERENT BROWSER.
LIKE SAFARI, PASTE THE LINK INTO
GOOGLE CHROME.
OR SOMETHING.
I DON'T THINK IT WORKS THROUGH
TEAMS DIRECTLY.
>> I'M ACTUALLY GETTING
EVERYTHING WORKING NOW THROUGH
TEAMS DIRECTLY BUT THE AV FOLKS
SAID GO TO VIEW AND GALLERY VIEW
AT TOP, YOU SEE PEOPLE UP AT THE
TOP, THE SLIDE DOWN AT THE
BOTTOM.
RICHARD'S SEEING IT.
>> AND I'M SEEING IT ON TEAMS.
SO A VIEW WITH NO SLIDES FOR
SOME PEOPLE.
MONA IF YOU --
>> YOU'RE CHANGING THE RECYCLING
PROCESS BY INPUTTING SMALLER
MSAs, AND THEREFORE IT'S
FASTER, DOESN'T HAVE TO DO AS
MUCH WORK, AND YOU'RE
POTENTIALLY CAPTURING THE
SEQUENCES THAT LEAD TO THESE
OTHER STRUCTURES BETTER.
SO WE'RE NOT -- I DON'T THINK
WE'RE NECESSARILY -- DO YOU SEE
MY SLIDE NOW?
>> I'M SEEING YOUR SLIDE.
I TRYING TO HELP OTHERS SEE YOUR
SLIDE.
>> SORRY.
SO WE'RE USING ALPHAFOLD'S
RECYCLING AS THEY USE IT, A
DIFFERENT TINY SHALLOW RANDOM
RSA, AND RANDOMLY SELECTED.
>> HAVE YOU TRIED MODELS THAT
ARE JUST PROTEIN LANGUAGE MODELS
THAT DON'T NEED ALIGNMENT.
>> YES.
WE HAVE TRIED THOSE MODELS AND
WHAT WE'VE FOUND IS THAT
ESPECIALLY ON SWITCHING PROTEINS
THEY TENDING TO OVERTRAINED.
EVEN MORE SO THAN ALPHAFOLD,
ALTERNATIVE CON CONFIRMATION.
IF WE MASK OUT C-TERMINAL
DOMAINS IT DOESN'T SEE
SEQUENCES, GIVES HELICAL
CONFIRMATION.
WE DID NOT WANT TO GO THAT WAY
WHEN DOING THESE PREDICTIONS.
>> TRAINED IN A SENSE THAT GAVE
THE ALTERNATIVE CONFIRMATIONS,
YOU THOUGHT THEY WERE
OVERTRAINED ON THOSE?
>> OVER TRAINED WHEN WE GAVE IT
NO INFORMATION IT WAS GIVING US
ALTERNATIVE CONFIRMATION.
>> WAIT A MINUTE.
BUT THAT'S A GOOD THING.
YOU DON'T KNOW THAT THEY ARE
OVERTRAINED.
>> NO, I DO THINK IT'S
OVERTRAINED BECAUSE IF WE MASK
THE WHOLE SEQUENCE OF THE
C-TERMINAL DOMAIN, ALMOST ALL OF
IT, WE'RE GETTING THE RIGHT
ANSWER.
I DON'T THINK THAT'S A GOOD
THING.
THAT'S NOT ROBUST.
I HAVE OTHER WORK SHOWING
ALPHAFOLD ALSO IS WAY
OVERPREDICTING ALTERNATIVE
CONFIRMS, ONLY REQUIRING TWO
AMINO ACIDS TO MAKE PREDICTIONS,
WAY OVERPREDICTING ALTERNATIVE
CON FIRMATION.
HERE IS WHAT I'M TALKING ABOUT.
THIS IS A PROTEIN XCL 1, TO GET
IT TO PREDICT ALTERNATIVE CON
FIRMATION FROM SINGLE SEQUENCE,
EXPERIMENTAL TEST THE VARIANTS.
IF WE ONLY CHANGE TWO AMINO
ACIDS, WE GET IT TO ALWAYS
PREDICT THAT ALTERNATIVE
CONFIRMATION BUT THAT'S NOT NOW
IT'S WORKING.
IT'S NOT ROBUST FOR THIS
PROBLEM.
>> I'LL LET OTHERS ASK THEIR
QUESTIONS
>> IT'S MY TURN, RIGHT?
FANTASTIC TALK.
REALLY ILLUMINATING.
BUT FIRST, WHY YOU NEED TWO
DIFFERENT ESSENTIALLY FUNCTIONS
OF THE SAME PROTEIN, RIGHT?
WHAT'S BIOLOGIC NECESSITY OF
HAVING TWO DIFFERENT STRUCTURES,
FIRST.
SECOND, DID I UNDERSTAND
CORRECTLY YOU CAN DETECT
CO-EVOLUTIONARY CHANGES IN BOTH
STRUCTURES, RIGHT?
>> OKAY.
YES, SO TO ANSWER YOUR FIRST
QUESTION THERE ARE A FEW REASONS
FOLD SWITCHING EXISTS.
ONE IS BECAUSE IT CAN BE A
REGULATORY MECHANISM.
SO THAT YOU CAN IMAGINE CASES
LIKE FOR CHI B IT'S IN ONE
STATE, MOST OF THE TIME BECAUSE
IT'S SWITCHING VERY SLOWLY THAT
ALLOWS THE CLOCK TO HAVE THE
RIGHT TIMING.
OR YOU CAN IMAGINE A PROTEIN
BEING ABLE TO RESPOND QUICKLY IN
RESPONSE TO CELLULAR STIMULI.
SO USUALLY WHEN PEOPLE ARE
THINKING ABOUT THAT, THEY THINK
ABOUT SUBFUNCTIONALIZATION,
ANOTHER PROTEIN COPIES AND THEN
IT RESPONDS TO CELLULAR CHANGES
BY PRODUCING ANOTHER PROTEIN.
BUT IT'S MORE COST EFFECTIVE TO
JUST HAVE A PROTEIN SWITCH BE
ABLE TO DO SOMETHING.
I THINK THERE ARE JUST CASES
WHERE IT MAKES SENSE TO HAVE
ENVIRONMENTALLY RESPONSIVE
PROTEINS IN THE CELL THAT HAVE
COUPLED FUNCTIONS.
TO ANSWER THE QUESTION OF
EVOLUTIONARY SELECTION WE DID A
STUDY IN 2023, IT WAS PUBLISHED,
WE WERE ABLE TO SEE SIGNS OF
DUAL FOLD CO-EVOLUTION, IN 58 I
THINK FOLD SWITCHING PROTEINS.
AND WE CAN ONLY DO IT BECAUSE WE
NEEDED DEEP MSAs TO DO IT BUT
IN THOSE CASES WE DID SEE IT.
I DO THINK THAT AT LEAST IN SOME
CASES FOLD SWITCHING IS SELECTED
BY EVOLUTION.
>> THANK YOU SO MUCH.
>> YEAH.
>> AM I NEXT?
THERE WE GO.
ANYWAY, FANTASTIC STUFF, REALLY
ILLUMINATING.
UNFORTUNATELY I HAD TO RELOG IN
WHEN BONNIE WAS ASKING THE
QUESTION ABOUT CF RANDOM, HOW
DID IT WORK.
LET ME PARAPHRASE IN TERMS OF
WHEN DO YOU THINK YOU'RE
LEARNING ADDITIONAL PHYSICS
THROUGH YOUR MODIFICATION OF THE
ALPHAFOLD ALGORITHM?
AND THAT'S WHAT'S HELPING YOU
FIND THE SECOND CONFIRMATION?
>> I DO NOT THINK WE'RE
LEARNING PHYSICS.
AND ACTUALLY DUBLINA WILL SHOW
YOU WHY IN HER TALK.
I THINK ALPHAFOLD IS REALLY GOOD
AT KNOWING WHAT A PLAUSIBLE
PROTEIN STRUCTURE LOOKS LIKE,
BUT I DON'T THINK THAT'S
PHYSICS.
IF IT'S OKAY WITH YOU I'LL LET
YOU SEE BECAUSE DUBLINA HAS HER
SLIDES AND WILL SHOW YOU WHY.
IS THAT OKAY OR DO YOU WANT A
MORE --
>> JUST AS A TIME CHECK, WHY
DON'T YOU HAVE THE LAST QUESTION
SO WE DON'T CUT MUCH INTO THE
POSTDOCS' TIME.
>> LAST QUESTION FIRST --
>> I CAN'T HEAR YOU.
SORRY, COULD YOU SAY THAT AGAIN?
>> YES, REALLY REMARKABLE WORK
THAT YOU'RE DOING.
PARTICULARLY COMBINING --
(INAUDIBLE).
VERY SHORT QUESTIONS, SEEMS THE
FOLD SWITCHING COULD COME FROM
DIFFERENT REASONS, AND SOME
COULD BE A TYPE OF (INAUDIBLE) I
DON'T KNOW FOR THE ONE EXAMPLE
YOU'RE SHOWING US, MUTATION OR
ENVIRONMENTAL.
HOW DOES THE ORIGIN OF THE FOLD
SWITCHING CHANGE AS YOUR
ALGORITHMS MOVES TO EXPLORE AND
PREDICT?
>> THIS IS A GREAT QUESTION.
SO I THINK WHAT WE HAVE NOW,
LIKE REALLY THINK WHAT CF RANDOM
DOES MOSTLY IS LEVERAGE
STRUCTURES THAT ALPHAFOLD HAS
ALREADY SEEN DURING ITS TRAINING
AND ASSOCIATE THAT WITH AN INPUT
STRUCTURE.
AND I DON'T -- SO THAT'S USEFUL
TO AN EXTENT.
BUT IT'S NOT FULLY USEFUL.
SO ONE EXAMPLE WHERE I THINK
THIS IS WHY WE'RE DOING A
DETAILED INTERPRETATION OF
ALPHAFOLD TRYING TO UNDERSTAND
WHAT IT'S SEEING IS WITH THAT
BCCIP ALPHA PROTEIN, BETA IS
REALLY WELL CONSERVED, ALL THE
WAY DOWN TO YEAST.
AND HIGHER EUKARYOTES.
ALPHA IS CONSERVED IN PRIMATES
AND A FEW OTHER ORGANISMS.
THERE'S NOT THAT MUCH GENETIC
VARIATION AMONG HIGHER
EUKARYOTES.
TO SEE SOME SORT OF
CO-EVOLUTIONARY SIGNAL TO MAKE
PREDICTIONS MAY NOT BE POSSIBLE.
WE NEED OTHER APPROACHES TO
PREDICT ALTERNATIVE CON
FIRMATIONS.
ANOTHER CHALLENGE WITH THIS
RESEARCH YOU MAY BE ALLUDING TO
IS THAT WE DON'T KNOW WHAT
TRIGGERS FOLD SWITCHING.
AND THAT'S A REALLY BIG
CHALLENGE.
WITH EXPERIMENTS WHAT WE'RE
HOPING FOR IS TO RUN NMR
EXPERIMENTS OR FRET EXPERIMENTS,
DO WE SEE TWO OR DO WE SEE
ANYTHING, THIS IS WHY WE'RE
COLLABORATING WITH NUMEROUS NMR
PEOPLE.
WE'VE BEEN WORKING ON A HYDROGEN
EXCHANGE BASED SCREENING METHOD
WITH SUSAN'S LAB, MAYBE ALLOWIG
US TO OVERCOME SOME BARRIERS,
BUT KNOWING THE TRIGGER IS AN
ACHILLES HEEL AND WE'RE HOPING
TO PICK UP SOMETHING GOOD FOR
NOW BUT WE HOPE AS WE GET MORE
EXAMPLES AND UNDERSTAND THE
PRINCIPLES BETTER, WE MAY BE
ABLE TO DO THINGS IN A MORE
INFORMED WAY.
>> GREAT.
WE'LL HAVE -- THE BOARD WILL
HAVE A CHANCE TO ASK QUESTIONS
SO I WILL RECONVENE THE MEETING
OF THE NATIONAL LIBRARY OF
MEDICINE BOARD OF SCIENTIFIC
COUNSELORS.
WE'RE NOW IN PUBLIC SESSION,
RECORDING AND BROADCASTING.
PLEASE CONFIRM, AV, FOLKS, THAT
THAT'S THE CASE.
>> GOOD.
>> GREAT.
SO, JEREMY, GO AHEAD AND START.
THANK YOU FOR BEING HERE AND
PRESENTING TO US, JEREMY WEISS.
YOU HAVE ABOUT 30 MINUTES HERE.
>> ALL RIGHT.
CAN EVERYBODY SEE THE SCREEN
OKAY?
>> YES.
>> OKAY, PERFECT.
SO, WELCOME, EVERYBODY.
MY NAME IS JEREMY WEISS, AN
INVESTIGATOR COMPUTATIONAL
HEALTH AT THE NLM, JOINED IN
2022, WITH THE BOARD OF
SCIENTIFIC COUNSELORS.
TODAY'S TALK IS ENTITLED
QUANTITATIVE MACHINE LEARNING
FOR RISK CHARACTERIZATION.
I LEAD THE LAB, CARE HEALTH AND
REASONING, AND OUR RESEARCH
GOALS TO ADVANCE CAPABILITIES OF
TEMPORAL MACHINE LEARNING IN
CLINICAL SETTINGS, RISK
CHARACTERIZATION OF DISEASE, AND
MY GROUP IS HERE.
I HAVE THREE FELLOWS, ONE WILL
BE PRESENTING A POSTER, A
Ph.D. STUDENT, AND SHANTON AND
JING WANG WHO JOINED IN 2025 AND
2024 RESPECTIVELY.
IN TODAY'S TALK I'D LIKE TO
FIRST START WITH A MOTIVATION
FROM THE LENS OF CLINICAL
DECISION MAKING AND OVERLAYING
OUR THINKING ABOUT RISK
CHARACTERIZATION ON TOP OF THAT.
AND I'LL GIVE A HIGH LEVEL
OVERVIEW OF OUR GROUP'S WORK AND
LONGITUDINAL RISK PREDICTION
FOCUSING IN PARTICULAR ON
SUBGROUPS.
WHETHER THEY BE CLUSTERS OR RISK
STRATA OR LONGITUDINAL
TRAJECTORIES OR PATHWAYS.
I'LL DO A DEEPER DIVE INTO
RECENT WORK.
I'D LIKE TO INTRODUCE THE
CONCEPTUAL FRAMEWORK WHICH IS
TRYING TO GET AT HOW DO WE
TRANSLATE OUR BODY OF KNOWLEDGE
FROM BODY OF EVIDENCE INTO THE
DECISION-MAKING PROCESS AT THE
POINT OF CARE.
I'D ARGUE THERE'S REALLY TWO
PILLARS OF THIS PROCESS, FIRST
IS USE OF EVIDENCE-BASED
PROTOCOLS, AND THINKING ABOUT
CLINICAL EQUIPOISE AT THE POINT
OF CARE, DESCRIBED IN DETAIL,
LET'S LOOK AT THE DIAGRAM.
ON THE LEFT WE HAVE THE LEVELS
OF EVIDENCE FROM EVIDENCE-BASED
MEDICINE, THE TYPES OF DATA THAT
AND STUDIES WE CONDUCT TO
GENERATE OUR BODY OF KNOWLEDGE,
AND THEN WE HAVE TO THINK HOW DO
WE TRANSLATE FIGURING OUT HOW
THAT APPLIES TO THE PATIENTS AS
PROVIDERS, TO HELP DETERMINE
WHAT THE BEST COURSE OF ACTION
IS.
AND SO THIS KIND OF HAPPENS IN A
COUPLE WAYS.
WE HAVE CLINICAL GUIDELINES FROM
SOCIETIES THAT MANIFEST, SAY AT
A HOSPITAL OR CLINIC LEVEL.
IF THOSE GUIDELINES APPLY TO THE
PATIENT AT HAND WE APPLY THEM.
IN MANY CASES CONTEXT SUGGESTS
MAY NOT DIRECTLY APPLY.
SO THEN DRAW UPON EXPERIENCE AND
SCIENTIFIC LITERATURE TO FIGURE
OUT WHAT THAT COURSE OF ACTION.
THAT'S WHEN YOU TRANSITION FROM
EVIDENCE-BASED PROTOCOLS TO THIS
NOTION OF CLINICAL EQUIPOISE.
IN MANY CASES STRAIGHTFORWARD TO
FIGURE OUT THE RIGHT COURSE OF
ACTION BUT WHEN THERE'S A
BALANCE IN TERMS OF RISK/BENEFIT
PROFILE YOU'RE IN CLINICAL HE
EQUIPOISE, GETTING AS MUCH
INFORMATION ABOUT THE PATIENT AS
POSSIBLE, PERSONALIZATION
COMPONENT, AND OBTAINING
CONSENT, CONVERSATION,
COMMUNICATING ABOUT RISKS AND
BENEFITS IS PART OF THIS SECOND
PART OF CLINICAL DECISION
MAKING.
I'D LIKE TO OVERLAY NOTION OF
RISK ON TOP OF THIS CLINICAL
DECISION-MAKING PROCESS.
AND IT KIND OF COVERS ALL
SORTS -- ALL ASPECTS OF THIS
DIAGRAM.
SO INFORMALLY, WE THINK ABOUT
RISK, WE CAN THINK ABOUT RISK AS
THE PROBABILITY OF OUTCOMES,
SEQUELAE OF DISEASE, GIVEN THE
PATIENT INFORMATION WE HAVE
AVAILABLE.
AND THIS MANIFESTS WITH RESPECT
TO TREATMENT EFFECT, OR RISK
DIFFERENCES, MIGHT BE LOOKING AT
THIS QUANTITY UNDER DIFFERENT
COURSES OF ACTION, POTENTIAL
INTERVENTIONS, WE CAN DECORATE
QUANTITY IN THE LANGUAGE OF
CAUSAL INFERENCE, MIGHT WANT TO
BUILD THE BEST PREDICTIVE MODELS
USING LARGE OBSERVATIONAL
STUDIES OR OTHER INFORMATION TO
BUILD PREDICTIVE MODELS THAT MAY
APPLY TO THE PATIENT THAT WE
HAVE AT HAND AND THEN AGAIN WE
HAVE THIS EXERCISE, WE HAVE A
WHOLE SUITE OF RECOMMENDATIONS
FROM RANDOMIZED TRIALS AS WELL
AS PREDICTIVE MODELS AND HAVE TO
FIGURE OUT DO THEY APPLY,
TRANSPORT TO THE PERSON THAT I'M
TAKING CARE OF.
SO, WHEN MY GROUP THINKS ABOUT
RISK, WE TYPICALLY THINK ABOUT
IT IN THE TEMPORAL SETTING.
YET ANOTHER OVERLAY HERE.
SO TRANSITIONING FROM THE IF
QUESTION TO THE WHEN QUESTION,
INSTEAD OF THE PROBABILITY OF
WHY GIVEN X, WE'RE NOW THINKING
ABOUT SURVIVAL, THINKING ABOUT
WHEN DOES THIS EVENT OCCUR, AND
WHEN WE TRANSITION FROM THE IF
TO THE WHEN QUESTION THIS
POTENTIALLY CHANGES BOTH OUR
FEATURES AND OUR OUTCOMES.
THE OUTCOME, ONE OF THE
MOTIVATIONS FOR MOVING INTO THIS
KIND OF SURVIVAL ANALYSIS SETUP
IS THAT OFTENTIMES IN LAUNCHING
STUDIES WE DON'T GET TO SEE WHAT
HAPPENS TO THE INDIVIDUAL.
SO WE WANT TO MAKE MODELING
ASSUMPTION OR SPECIFY A MODEL
THAT ALLOWS YOU TO CAPTURE THAT.
LIKEWISE, WHEN WE HAVE
INFORMATION COMING ACROSS
TIMING, IN A DYNAMIC WAY, THE
FUTURE REPRESENTATIONS CHANGE,
PERHAPS X TURNS INTO AN X STAR,
X STAR IS SOME OTHER DOMAIN
BECAUSE INFORMATION IS REVEALED
ACROSS TIME THAT CHANGES OUR
BEST REPRESENTATION OF THE
PATIENT INFORMATION THAT INFORMS
DECISION MAKING.
THIS IS CLEAR MODIFICATIONS WHEN
WE TRANSITION INTO TEMPORAL
ANALYSIS.
ALSO IMPLICATIONS WITH RESPECT
TO OPTIMIZATION FRAMEWORK.
SO IF WHEN WE'RE THINKING ABOUT
HAZARD OPTIMIZATION FOR THESE
FUNCTIONS CAN OFTEN LEAD TO
OVEREMPHASIS OR UNDEREMPHASIS.
SO HERE'S AN EXAMPLE OF SOME
LONGITUDINAL CLINICAL DATA.
WHAT I'M SHOWING HERE IS ONE
PATIENT OVER THE COURSE OF
ROUGHLY A WEEK IN A HOSPITAL,
AND DE-IDENTIFY.
EVERY DOT REPRESENTS THE
COLLECTION OF SOME MEASUREMENT,
LIKE A BLOOD PRESSURE, BLOOD
GLUCOSE.
SO THIS IS WHAT WE TYPICALLY SEE
IN ELECTRONIC HEALTH RECORDS
DATA, PRETTY MUCH A PERVASIVE
SOURCE OF LONGITUDINAL DATA
ABOUT PATIENTS AT THIS POINT.
YOU KNOW, WITH INCREASE IN USE
OVER THE LAST 20 YEARS.
AND WE TYPICALLY GET THIS IN A
STRUCTURED FORMAT DRAWING FROM
TABLES SUCH AS ADMISSION,
CHARTING, DIAGNOSES, LABS,
MEDICATIONS, SO FORTH.
THIS NICELY REPRESENTS REAL
WORLD DATA BECAUSE IT'S
COLLECTED AS NEEDED, REFLECTED
AT LARGE SCALE OF REAL WORLD
SITUATIONS AND POPULATIONS, BUT
HAS A NUMBER OF REALLY
STATISTICAL CHALLENGES AND
INCOMPLETE, INCOMPLETE IN A WAY
THAT'S NOT AT RANDOM AS IN THE
MODEL DOESN'T KNOW WHY SOME
THINGS ARE COLLECTED OR NOT, NOT
COLLECTED IN A REGULAR FASHION,
SO WE DON'T HAVE THE INFORMATION
NECESSARY TO UNDERSTAND THOSE
PROCESSES.
WE'RE JUST GIVEN CAUTIONS,
MEASUREMENTS WHEN THEY ACTUALLY
DO HAPPEN.
SO THIS CAN AFFECT THE MODELING
THAT TRANSPORTABILITY OF THE
MODEL THAT WE BUILD ON THESE
DATA.
IF YOU HAVE DIFFERENT
COLLECTIONS AT ANOTHER LOCATION
MODEL THAT YOU PREDICT AT YOUR
ORIGIN SITE MIGHT NOT TRANSFER
TO THE DESTINATION BECAUSE OF
THIS.
IN THE CONTEXT OF ALL OF THIS MY
GROUP FOCUSES ON THREE AREAS.
I'LL HIGHLIGHT THE FIRST TWO AT
A HIGH LEVEL, JUST A TASTE, AND
DO A DEEPER DIVE INTO THE
TEXTUAL COMPONENT.
OKAY.
SO, IN THE FIRST EXAMPLE, WE
WERE INTERESTED, AS PART OF THE
STUDY, INVOLVED IN JUST
IDENTIFYING SUBGROUPS OF SEPSIS,
THIS IS JAMA, 2019.
THE IDEA HERE IS THAT SEPSIS,
WHICH IS A BODY'S RESPONSE TO
INFECTION, THAT CAN LEAD TO
ORGAN FAILURE AND DEATH, IS THAT
SEPSIS IS A NOTORIOUSLY
HETEROGENEOUS DISEASE.
THE SITE OF INFECTION, THE
COMORBIDITY PROFILE, ORGANS AT
RISK MAY ALL DIFFER, AND SO WHAT
WE HAD SEEN IN THE LITERATURE IS
THAT, YOU KNOW, TWO DECADES
WORTH OF RANDOMIZED CONTROLLED
TRIALS THAT FAILED TO FIND A
POSITIVE FINDING FOR NEW
TREATMENT OR THERAPEUTIC OR
FAILED TO REPLICATE IN
SUBSEQUENT TRIALS.
SO THE HYPOTHESIS IS THAT IF YOU
CAN IDENTIFY SUBGROUPS THAT ARE
HOMOGENOUS IN NATURE YOU MIGHT
BE ABLE TO USE TARGETED
THERAPEUTICS THAT WOULD SUCCEED
WITHIN THAT SUBGROUP.
THIS PARTICULAR WORK IDENTIFIED
FOUR SUBGROUPS BASED ON
INFORMATION COLLECTED WITHIN THE
FIRST SIX HOURS OF ADMISSIONS.
NOW, ONE OF THE LIMITATIONS OF
THIS STUDY IS THAT THE ONLY TIME
THIS CLUSTERING HAPPENED, OR
DATA THAT WAS USED, WAS FROM THE
FIRST SIX HOURS ONLY.
AND SO IT'S QUITE POSSIBLE THAT
A PERSON WHO INITIALLY WAS IN
LET'S SAY SUBGROUP ALPHA
ACTUALLY IN A COUPLE HOURS DUE
TO DETERIORATION WOULD THEN BE
PLACED INTO ANOTHER GROUP AT
HIGHER RISK, LET'S SAY GROUP
DELTA.
BECAUSE WE WANT TO BE ABLE TO
HAVE A PERSON, YOU KNOW, NOT DO
THIS HAVE THIS JUMPING AS WE SEE
IT, WE WANTED TO BE ABLE TO
CLUSTER ON SNAPSHOTS.
SO FOLLOW-UP WORK ALLOWED US TO
DO THIS.
WE INTRODUCED A TEMPORAL
SUPERVISED CONTRASTED LEARNING
METHOD INSTEAD OF TAKING THE
INFORMATION JUST AT THE TIME, WE
TOOK FOUR-HOUR SNAPSHOTS AND
USED THAT INFORMATION, PLACED
INTO EMBEDDING SPACE.
THIS SPACE WAS CHARACTERIZED BY
HAVING EACH OF THE SNAPSHOTS OF
THE PATIENT BEING LOCALIZED
CLOSE TO ONE ANOTHER AND THEN
EMBEDDING SHOULD HAVE SIMILAR
REPRESENTATION WITH RESPECT TO
FEATURES, ALSO WITH RESPECT TO
RISK.
AS A RESULT WE WERE ABLE TO
IDENTIFY TWO SUBGROUPS THAT WERE
HOMOGENOUS WITH RESPECT TO RISK,
BUT HAD VARYING PROFILES.
SO WHAT I'M SHOWING IN THE HEAT
MAP IS PROBABILITY OF
REMEMBERING WHAT'S IN THE
SUBGROUP TO HAVE THAT PARTICULAR
LABORATORY CHARACTERISTICS AND
DESPITE PEOPLE HAVING VARIOUS
SIMILAR RISK WITH RESPECT TO
90-DAY MORTALITY, THEY ACTUALLY
LOOKED VERY DIFFERENT WITH
RESPECT TO LABORATORY CHARACTER.
ANOTHER WORK OF OURS HAS FOCUSED
ON CHARACTERIZING THE HAZARD
FUNCTION AND OPTIMIZATIONS
AROUND THE LIKELIHOOD OF THAT
HAZARD, AND MOTIVATION HERE IS
THAT PEOPLE WHO ARE LOW RISK,
OFTEN A PLURALITY OF PATIENTS
ARE NOT WELL REPRESENTED WHEN
USING OBJECTIVE FUNCTIONS, THAT
IS OBJECTIVE FUNCTIONS WILL
FOCUS ON HIGHEST RISK
INDIVIDUALS.
IF YOU THESE TO MAKE
RECOMMENDATIONS FOR PLURALITY,
MAY NOT BE OPTIMIZED FOR MAKING
RECOMMENDATIONS, THE WORK HELPED
HIGHLIGHT THAT ASPECT.
AND THEN IF YOU WANT TO NOT ONLY
PREDICT WELL, BUT YOU ALSO WANT
TO HAVE SOME OTHER
CHARACTERISTIC PROPERTIES, WE'VE
INTRODUCED A SERIES OF WORKS
FOCUSED ON HAVING HIGH CAPACITY
PERFORMANCE, NOT TOO DIFFERENT
ACROSS SUBGROUPS OF INTEREST.
IN PARTICULAR, INSTEAD OF USING
COMMONLY USED NOTIONS, LIKE E.
COLI'S ODDS OF STATISTICAL
PARITY, THESE TIMES OF
DEFINITIONS ACTUALLY DON'T WORK
IN THE RIGHT SENSORSHIP SETTING
WHERE OUTCOME IS UNKNOWN.
WE DON'T KNOW WHAT HAPPENS TO
INDIVIDUALS.
WE'VE IDENTIFIED TERMINOLOGY TO
HELP YOU IDENTIFY, CONCORDANCE,
PARITY, MODELS TO PREDICT BUT
HAVE LOW DIFFERENCES ACROSS
SUBGROUPS.
OKAY.
I'M GOING TO DIVE INTO THE
TEXTUAL LONGITUDINAL DATA
COMPONENT FIRST BUT I WANTED TO
HIGHLIGHT THAT WE JUST RECENTLY
HAD A PAPER ACCEPTED, AND THE MY
Ph.D. STUDENT CO-ADVISED
PRESENTING HIS POSTER ON
ANTIPSYCHOTIC.
SO, NOW WE'RE TALKING ABOUT
TEXTUAL TIME PERIOD, I WANTED TO
GIVE YOU A DESCRIPTION OF THIS
TASK.
SO, WE HAVE MANY CASE REPORTS,
DISCHARGE SUMMARIES REPRESENTED
BY THE BLURB ON THE LEFT.
AND THESE ARE DOCUMENTS THAT ARE
TIME STAMPED AT THE END OF THE
ATTEMPT, AFTER THE FACT.
HOWEVER, THERE ARE MANY MENTIONS
OF -- WITHIN THE CASE REPORT
THAT CAN BE TIME STAMPED WITH
MUCH GREATER GRANULARITY.
SO, THE TASK IS TO TAKE MENTIONS
AND PLACE THEM ON A TIMELINE
BECAUSE THIS INFORMATION WOULD
BE ABLE TO PROVIDERS TRYING TO
DO RISK ASSESSMENTS AT THE POINT
OF CARE.
HOWEVER, THEY DON'T END UP
MAKING IT INTO THE LONGITUDINAL
RISK MODELS BECAUSE THEY HAVE
BEEN TIME STAMPED AFTER THE
FACT.
SO OUR GOAL IS TO BE ABLE TO
EXTRACT THIS INFORMATION,
PROVIDED AS A TIMELINE SO WE CAN
BUILD MODELS ON THE TEXTUAL TIME
PERIOD, ALONGSIDE TABULAR
PRESENTATION.
SO, THIS IS A MOTIVATING SLIDE
ABOUT WHY THIS IS IMPORTANT.
THIS PAPER WHICH WAS PUBLISHED
EARLIER THIS YEAR IN JMIR
BASICALLY MAPS TEXTUAL CASE
REPORTS TO CLINICAL ENTITIES,
FROM UML ONTOLOGY, AND THEN THE
SAME THING FOR TABULATING.
ONLY 13 WERE PRESENT, AND THIS
GIVES A SINCE THE TWO DATA FORMS
ARE COMPLEMENTARY YET RESEARCH
COMMITTEE FOCUSES ON HEAVILY ON
THE TABULAR DATA ALONE BECAUSE
OF ITS AVAILABILITY.
IT'S BEEN OUR GOAL TO HAVE
MODELS BUILT ON THE BEST BUILT
DATA, RICHEST FROM PREVIOUS
SLIDES, WE WANT TO PROVIDE THIS
NEW DATA SOURCE TO IMPROVE THE
PREDICTED MODELS.
HERE IS THE DEFINITION, TEXTUAL
TIME SERIES IS A SEQUENCE OF
EVENTS, HERE WE'RE DEFINING
EVENT AS CONTINUOUS SPAN, TIME
LOCALIZABLE EVENT, AND THE TIME,
EVENT TIME, RELATIVE TO INITIAL
PRESENTATION.
FOR THE SAKE OF TIME, I'M GOING
TO FOCUS ON THE CONTRAST HERE.
EVENTS AND TIMES WE'RE USING ARE
SLIGHTLY DIFFERENT FROM THOSE
THAT ARE USED, AND THE CLINICAL
TEMPORAL COMMUNITY.
THERE'S BEEN A LOT OF WORK IN
ANY COMMUNITY ON TEMPORAL
RELATIONS, CLINICAL CONCEPTS.
AND HERE BECAUSE WE WANT TO DO
THESE DOWNSTREAM TASKS OF
FORECASTING CAUSAL INFERENCE WE
DON'T WANT TO DO NECESSARILY A
MAPPING TO ONTOLOGY.
WE WANT TO RETAIN CONTEXTUAL
INFORMATION LOCALIZED AT THIS
TIME POINT SO OTHER TYPES OF
MODELS CAN PERFORM AS BEST AS
POSSIBLE.
SIMILARLY, THIS NLT COMMUNITYE
TEMPORAL RELATION, ASKS WHETHER
EVENT A PRECEDES EVENT B, YES OR
NO, AND HERE WE WANT TO CONDUCT
SURVIVAL ANALYSIS, LONGITUDINAL
ANALYSIS, WE WANT THE TIMING
RELATIVE TO THIS.
SO, WE'RE GOING TO USE LARGE
LANGUAGE MODELS TO GET
ANNOTATIONS AT SCALE BUT NEED
SOME GROUND TRUTH.
FIRST TO DO THIS WE CONDUCTED
SEVERAL PILOTS TO COLLECT
INFORMATION FROM CLINICAL
ANNOTATORS, MYSELF AND OTHER
CLINICAL PROVIDERS.
AND OUR JOB ESSENTIALLY WAS TO
IDENTIFY EVENTS FROM THE TEXT
AND THEN SELECT TIME INTERVALS
BASED ON TABULAR REPRESENTATIONS
OR JUST TIMELINES AND THEN
REPEATEDLY SELECT ANNOTATES TO
GENERATE THIS CORPUS OF DATA.
THEN WE'RE ABLE TO COMPARE THESE
MANUAL ANNOTATIONS WHICH ARE
GOING TO BE SMALL SCALE
EXPENSIVE TO COLLECT, WITH THE
ANNOTATIONS WE CAN GET FROM
LARGE LANGUAGE MODELS.
SO THIS IS ONE TYPE PLAN THAT
WE'VE DEVELOPED WHICH HIGH LEVEL
INVOLVES EXTRACTION, ANNOTATION,
AND THIS IS FOR THE PubMed
OPEN ACCESS CORPUS.
THIS CORPUS HAS 1.5 MILLION
PUBLISHED MANUSCRIPTS, OF THESE
WE CAN IDENTIFY APPROXIMATELY
125,000 CASE REPORTS OF WHICH
WE'VE ANNOTATED, YOU KNOW, ALL
OF THEM WITH THE LARGE LANGUAGE
MODELS TO CONSTRUCT OUR TEXTUAL
TIME SERIES.
WE'RE ABLE TO COMPARE AGAINST
OUR MANUAL SUBSET AND THEN ALSO
PROVIDE THE ENTIRE CORPUS TO THE
COMMUNITY, TO APPEND ADDITIONAL
INFORMATION, OUTCOME
INFORMATION, ALONGSIDE
CHARACTERIZATIONS THAT ALLOW
PEOPLE TO DRILL IN TO IDENTIFY
COHORTS.
AND THEN BECAUSE WE'RE DOING
THIS NEW PROCESS OF, YOU KNOW,
EXTRACTING EVENTS IN THE WAY WE
THINK IS RIGHT TO SPECIFY
ALONGSIDE THE TIMING WE HAVE TO
DEVELOP OUR OWN ASSESSMENT TOOLS
TO ASSESS THE QUALITY OF THE
ANNOTATIONS.
SO I'LL GIVE THE HIGH LEVEL
PICTURE OF THIS.
BASICALLY THERE ARE A COUPLE
STEPS THAT WE DO, WE TAKE.
FIRST IS IDENTIFYING TEXT
MATCHING.
SO YOU COULD THINK ABOUT
CHARACTER LEVEL SIMILARITIES BUT
WE FIND THAT USING SENTENCE
TRANSFORMING EMBEDDINGS AND
DISTANCES BETWEEN, SIMILARITY
WORKS BETTER.
WE TRIED A BUNCH OF MODELS, TO
ANNOTATE AND COMPARE THE
SIMILARITIES, OR WHETHER OR NOT
THOSE EVENTS MATCH OUR MANUAL
ANNOTATED EVENTS, AND WHAT WE
FIND IS THAT USING COSINE
DISTANCE, 75% MATCH WITH THE
ONES WE MANUALLY ENTERED.
HOW DO WE CHOOSE THAT DISTANCE
LEVEL?
BASED ON MANUAL REVIEW, WHERE WE
CAN SEE IN FACT IT CORRESPONDS
TO THE SAME NOTION.
ONCE WE'VE LOOKED AT EVENT
MATCHES WE CAN CONSIDER WHETHER
OR NOT THE TIMES PRODUCED BY
THESE LARGE LANGUAGE MODELS
CORRESPOND TO THE TIME WE
IDENTIFY IN OUR REPORTS.
SINCE WE'RE LOOKING AT TIME
DIFFERENCE, MEAN ABSOLUTE AREA,
THESE DON'T WORK VERY WELL WHEN
YOU THINK ABOUT THE DIFFERENT
TIME SCALES THAT WE'RE WORKING
OVER, POTENTIALLY YEARS TO
MINUTES.
IF YOU THINK ABOUT FURTHER
ERRORS AT THE LEVEL OF YEARS,
WE'RE GOING TO DOMINATE THE
ERROR THAT I MADE AT THE LEVEL
OF MINUTES.
WE CAN EITHER DRAW ON, YOU KNOW,
IDEAS FROM ROBUST STATISTICS
LIKE MEDIAN ABSOLUTE ERROR OR
DEVELOP OUR OWN METRIC OF AREA
UNDER THE LOG TIME CURVE, WHAT
I'M SHOWING IN THE GRAPH IS THE
LOG TIME PLUS OF TIME
DISCREPANCIES BETWEEN LARGE
LANGUAGE MODEL TIME AND MANUAL
ANNOTATION OUTCOME.
WE CAN SEE THAT APPROXIMATELY
50% OF THESE EVENTS GETTING THE
TIMES EXACTLY RIGHT AND THEN 70%
OF THE TIME GETTING THE TIME
CORRECT WITHIN ONE DAY AND SO
FORTH.
THE AREA UNDER THE CURVE IS
CAPTURED.
FINALLY WE MAPPED BACK TO
CONCORDANCE BECAUSE THAT'S WHAT
MAJORITY OF THIS COMMUNITY
FOCUSED ON BECAUSE TEMPORAL
RELATIONS, A MEASURE OF THAT
PERFORMANCE, CONCORDANCE,
WHETHER YOU'RE ORDERING EVENTS
THE SAME WAY, ANNOTATION, LARGE
LANGUAGE MODEL ANNOTATION.
SO THIS CAN BE SUMMARIZED IN THE
SUMMARY TABLE ON THE LEFT.
WE TRIED DIFFERENT MODELS.
THIS IS FOCUSING ON SEPSIS
COHORT, TIME SERIES, AND WE CAN
SEE THE LARGE FRONTIER MODELS
PERFORMED THE BEST, THERE ISN'T
A SINGLE BEST PERFORMER BUT
CONSUMER SMALLER ONES, PARAMETER
MODELS, THEY REALLY DON'T
PERFORM WELL.
PRETTY SHARP DEGRADATION TO THE
SMALLER MORE ACCESSIBLE MODELS.
SO BEING ABLE TO HAVE ACCESS TO
THESE FRONTIER MODELS IS REALLY
IMPORTANT.
ON THE RIGHT, WE ALSO PROVIDED
MANUAL ANNOTATIONS TO REVIEW OF
THE ANNOTATIONS WE GOT.
AND THIS ALLOWED US TO IDENTIFY,
YEAH, WE HAVE QUANTITATIVE
MEASURES FROM QUALITATIVE
PERSPECTIVE ARE ANNOTATIONS ALSO
OF HIGH QUALITY WHEN WE'RE
RELEASING CORPUS, DO WE THINK
THE ANNOTATIONS OF THE TIME ARE
STRONG.
SO WE CAN SEE, AGAIN, FRONTIER
MODELS HAVE ACCEPTABLE QUALITY,
HIGH PROPORTION OF THE TIME.
THE SMALLER MODELS REALLY DON'T
PROVIDE WHEN PERFORMING AT ALL.
SO JUST BACK TO HIGH LEVEL
AGAIN, WE'VE ILLUSTRATED
EXTRACTION FOR PubMed OPEN
ACCESS, ALSO IN PARALLEL FOR THE
DISCHARGE SUMMARIES, THE CORPUS
OF DISCHARGE SUMMARIES, 330,000
FROM A CRITICAL CARE DATASET.
AND IN THIS CASE WE HAVE ACCESS
TO BOTH THE TEXTUAL INFORMATION,
ALSO COMPLEMENTARY TABULAR DATA,
PROVIDING OPPORTUNITIES FOR
PATIENTS TO THINK ABOUT
MULTI-MODAL IN THE TABULARY
TEXTURE.
TO HIGHLIGHT A DOWNSTREAM USE,
WE COULD ASK CAN THESE MODELS
FORECAST WELL GIVEN THE TEXTUAL
TIME SERIES.
YOU MIGHT THINK MAYBE WE DON'T
NEED TO DO THIS, MAYBE LARGE
LANGUAGE MODELS OUT OF THE BOX
WILL BE ABLE TO ASSESS RISK FOR
THESE.
ONE WAY OF MEASURING IS ASKING,
OF THE EVENTS THAT DO OCCUR IN
THE
THE TIME SERIES WILL THE NEXT
EVENT OCCUR WITHIN ONE DAY,
PROMPTING THE ENGINEERING
EXERCISE, YOU GET A SCORE OF
.31.
YOU CAN DO A FAIR BIT BETTER, IF
YOU'RE GOING TO, YOU KNOW, BUILD
THIS INTERFACE, YOU'RE GOING TO
TRAIN MULTI-LAYERS ON THE OUTPUT
EMBEDDING OF THE LARGE LANGUAGE
MODEL, YOU CAN GET HIGHER POINT.
BETTER TO DO THIS ADDITIONAL
EXERCISE, NOT USE JUST LARGE
LANGUAGE MODEL STRAIGHT OUT OF
THE BOX.
THERE'S ALSO THIS QUESTION ABOUT
WHETHER OR NOT WE SHOULD BE
USING DECODER TYPE MODELS VERSUS
ENCODER.
MORE LIKE IN THE ChatGPT.
AND TURNS OUT THAT THE MORE
CONTEMPORARY VERSION OF THE BERT
MODELS HAVE HIGHER PERFORMANCE
CHARACTERISTICS, AND IF YOU USE
THE STANDARD MASKING PROCEDURE,
LET'S SAY USED FOR PREDICTION,
YOU GET THIS CHARACTERISTICS .33
BUT THE BEST PERFORMING SET OF
THE FORM OF THE MODEL IS USING
THE BERT-BASED MODELS OR
CONTEMPORARY VERSIONS, APPLYING
THIS INTERFACE TO GET PREDICTIVE
OUTPUT.
OKAY.
SO JUST KIND OF MOVING UP A
LEVEL AGAIN, WHAT ARE SOME
FUTURE DIRECTIONS FOR OUR GROUP?
IN THE TEMPORAL SUBGROUP AREA
WE'RE INTERESTED IN TWO FOCUS
AREAS.
WE'RE INTERESTED IN LOOKING AT
GLP-1 USERS WITH THE MAJOR
INCREASE IN POPULARITY OF THIS
MEDICATION, PARTICULARLY AWAY
FROM THE ORIGINAL INTENDED US,
SECONDARY TREATMENT FOR
DIABETES.
AND LOOKING AT SUBGROUPS TO
IDENTIFY ARE THERE PARTICULAR
INDIVIDUALS WHO RESPOND IN
PARTICULAR WAYS, PARTICULARLY
HELPFUL FROM A CARDIOVASCULAR
ASPECT OR PARTICULARLY, YOU
KNOW, DETRIMENTAL AND SO FORTH.
WE'VE GOT A NUMBER OF, YOU KNOW,
DATA RESOURCES THAT WE CAN APPLY
HERE.
WE WORK WITH "ALL OF US" DATA,
WORK WITH CMS DATA, AND WE ALSO
HAVE OUR TEXTUAL TIME SERIES
COHORTS, A COUPLE VIEWPOINTS TO
TRY TO TACKLE THIS QUESTION.
AND THEN IN COLLABORATION WITH
NIEHS WE WERE AWARDED THE NIH
DIRECTOR'S CHALLENGE AWARD LAST
YEAR, TO FOCUS ON LONGITUDINAL
AND MATERNAL HEALTH RISK AROUND
TIME OF CONCEPTION,
INVESTIGATING THAT WITH SOME OF
OUR TOOLS.
GOING BACK TO THE SERIES, WE'D
LIKE TO CONDUCT THIS PROCESS
BETTER, RECOGNIZING THAT
PATIENTS ARE COSTLY, SO WE'VE
GOT THAT SORT OF ACTIVE
LEARNING, HUMAN IN THE LOOP
PROCEDURE FOR TRYING TO GET THE
BEST ANNOTATIONS WE CAN POSSIBLY
GET WHILE, YOU KNOW, RESTRICTING
THE NUMBER OF ANNOTATIONS.
AND THEN GOING BACK TO THE
COMPLEMENTARITY AND TABULAR
DATA, THINKING ABOUT A
MULTI-MODAL ALIGNMENT OF DATA,
TO IMPROVE TEXTUAL TIME SERIES
ANNOTATIONS, ALSO TO GET BACK AT
THE QUESTION THAT WE HAVE
COMPLEMENTARY DATA, NOT USING
THEM AS MUCH AS WE COULD BE,
WHAT CAN WE SAY AND COMMUNICATE
ABOUT RISK, GIVEN WE HAVE ACCESS
TO BOTH IN A LONGITUDINAL
FASHION NOW.
AND THEN FINALLY GOING BACK TO
THE LEVEL OF EVIDENCE FIGURE
THAT I DESCRIBED AT THE VERY
BEGINNING OF THIS TALK, WE
FOCUSED ON OUR CASE REPORTS FOR
OUR TEXTUAL TIME SERIES
REPRESENTATION, HIGHER LEVELS OF
EVIDENCE ARE TYPICALLY
POPULATION LEVEL KIND OF PIECE
OF KNOWLEDGE, HOW CAN WE OVERLAY
THAT IN A TEMPORAL FASHION WITH
ALL THE CASE REPORTS WE HAVE
ANALYSIS FOR.
WE'D LIKE TO FIGURE OUT HOW TO
COMBINE THE TWO SO WE CAN
PROVIDE THE BEST, YOU KNOW,
EVIDENCE AND RECOMMENDATIONS
GOING FORWARD.
SO IN SUMMARY, TIME, WE BELIEVE
THE CORNERSTONE ELEMENT OF
HEALTH RISK, THEREFORE SOME
DECISION MAKING.
WE'VE EXPANDED OUR WORK INTO
TEXTUAL TIME SERIES, THIS
ENABLES MACHINE LEARNING
FORECASTING WITH CONTENT THAT
PROVIDERS ROUTINELY USE, BUT
OFTEN MODELS DON'T TAKE INTO
ACCOUNT.
AND THIS NEW WORLD OF LLMs WE
BELIEVE CLINICAL LARGE LANGUAGE
MODELS SHOULD BE IN A
QUANTITATIVE SENSE, THEY ARE NOT
THERE YET, WORKING TO CLOSE THAT
GAP.
SO THANK YOU FOR YOUR ATTENTION
THE GROUP WILL PRESENT RECENTLY
ACCEPTED WORK ON MEDICATION
ADHERENCE, AND I'M HAPPY TO TAKE
YOUR QUESTIONS.
THANKS.
>> THANKS A LOT, JEREMY.
NOW WE'RE GOING TO HAVE THE OPEN
QUESTION-AND-ANSWER SESSION
WHETHER WE CAN TAKE QUESTIONS
FROM BOARD MEMBERS OR THE
PUBLIC.
IT WILL BE 15 MINUTES, UNTIL 2:
35 AND THEN MOVED TO THE CLOSED
POSTER SESSION.
I SEE GRACIELA HAS A HAND UP.
GRACIELA?
>> SURE.
EXCELLENT WORK.
VERY, VERY INTERESTING.
I HAVE I GUESS ONE QUESTION
WRAPPED IN TWO, OR TWO QUESTIONS
WRAPPED IN ONE.
HAVE YOU CONSIDERED OR TRIED
YOUR MODELS ON ACTUAL CLINICAL
RECORDS, SOMETHING LIKE MIMIC OR
ANY OTHER CLINICAL RECORDS OTHER
THAN CASE REPORTS?
>> YES.
>> AND SECOND PART, WAS
BENCHMARKING, GOLD STANDARDS
THAT ARE AVAILABLE.
>> SURE.
YES, SO WE HAVE EXTRACTED THE
TEXTUAL TIME SERIES FOR UNIT
DISCHARGE SUMMARIES.
WE'VE DONE ANNOTATIONS OF THE
CASES, 23 CASES FROM MIMIC.
OVERLAP IS PARTIALLY WITH THE
i2b2 CORPUS FOCUSING ON
TEMPORAL RELATIONS, MIMIC
TWO-THREE, AN OLDER VERSION.
THEN ON MIMIC 4, MORE RECENT, A
LITTLE BIT MORE SANITIZED CASE
FOR DISCHARGE SUMMARIES.
>> SO YOU DID ANNOTATE THOSE?
THE MIMIC?
>> WE DID, YES, WE DID ANNOTATE
23 OF THOSE CASES AND WE'VE ALSO
EVALUATED, AND THOSE RESULTS ARE
FROM THE PAPER FROM THIS YEAR.
>> ALL RIGHT.
I WAS EXPECTING YOU WOULD
COMMENT ON HOW WELL IT PERFORMED
BUT LIKE --
>> YEAH.
>> YOU DON'T HAVE THE EXACT
NUMBERS JUST LIKE -- I'M
CURIOUS.
>> SO I DON'T HAVE THE EXACT
NUMBERS IN THIS DECK.
THE PERFORMANCE DOES AGREE TO
SOME DEGREE.
SO THERE ARE -- DAYS REPORTS ARE
FORMATTED QUITE DIFFERENTLY THAN
THE DISCHARGE SUMMARIES.
THERE'S A LOT OF LIKE COPY/PASTE
IN THE DISCHARGE SUMMARIES, LIKE
TABLES WILL AUTOPOPULATE.
AND THOSE EVENTS CAN BE HARDER
TO CAPTURE FOR LARGE LANGUAGE
MODELS, SO WHAT WE SEE IS THE
EVENT MATCH RATE GOES DOWN
CONSIDERABLY, I THINK DOWN TO
LIKE 40 TO 50%.
AND THEN THE TEMPORAL ORDERING
ALSO DEGRADES TO SOME DEGREE.
I DON'T REMEMBER THE NUMBERS OFF
THE TOP OF MY HEAD.
>> NO, IT'S SORT OF WHAT I WAS
EXPECTING TO HEAR, BUT LIKE I'M
SURE THAT WAS MY QUESTION, I'LL
SAVE THIS FOR TIME FOR OTHER
PEOPLE.
>> RICHARD?
>> YEAH, I GUESS I'M ALLOWED TO
ASK QUESTIONS IN THE OPEN
SESSION.
SO, EARLIER WE HAD A
PRESENTATION FROM CLEM, AND AS
YOU KNOW HE WORKS ON POST-MARKET
SURVEILLANCE AND ONE OF HIS
FUTURE DIRECTIONS IS LOOKING AT
GLP-1 INHIBITORS.
I DON'T THINK HE HAS PLANS TO
INCORPORATE TEMPORAL RELATIONS
IN HIS EVALUATION.
HAVE YOU THOUGHT ABOUT MAYBE
PARTNERING UP WITH CLEM TO KIND
OF COMBINE FORCES ON THIS
QUESTION?
>> YEAH, I MEAN, THINK THAT
WOULD BE A REALLY EXCITING
OPPORTUNITY.
I WANT TO BE COGNIZANT OF LIKE
INSTITUTIONAL POLICIES,
INVESTIGATORS ARE OFTEN
ENCOURAGED, AS I UNDERSTAND IT,
TO, YOU KNOW, DEMONSTRATE THEIR
INDEPENDENCE AND SCIENTIFIC
AGENDA.
SO I THINK THERE'S A TON OF
OVERLAP.
I WOULD BE EXCITED TO WORK WITH
THEM.
I WANT TO MAKE SURE THAT'S
SOMETHING WE THINK IS STRATEGIC
FROM THE PERSPECTIVE, A GOOD
IDEA.
>> ONE THING, DO YOU REALLY
ESTABLISH YOURSELF AS KIND OF AN
EXPERT IN THIS TEMPORAL
COMPONENT TO THE ANALYSIS, AND
THAT'S NOT SOMETHING CLEM WAS
KNOWN FOR.
IF YOU WERE TO ADD THAT TO AND
COLLABORATE WITH HIM AND BRING
THAT COMPONENT IN I THINK YOU
WOULD BE RECOGNIZED FOR BRINGING
IT TO THE TABLE.
>> THANK YOU, GREAT IDEA.
>> OTHER QUESTIONS FOR JEREMY AT
THIS TIME?
>> YEAH, JUST WONDERING WITH THE
GLP-1, THINKING ABOUT THAT FROM
A TEMPORAL POINT OF VIEW, WHAT
TIME SCALE ARE YOU LOOKING AT
AND HOW LONG, YOU KNOW, GIVEN
THAT THESE ARE KIND OF COMING TO
THE MARKET, AND HAVE DIFFERENT
PRESCRIPTIONS FOR THEM, YOU
KNOW, WHAT TIME SCALE ARE YOU
LOOKING AT AND DO YOU PLAN ON
LOOKING AT KIND OF WHAT THE
REASON FOR GOING ON THESE IS?
>> YEAH, SO GLP-1s HAVE BEEN
AROUND OVER TWO DECADES.
BUT THE INDICATION FOR USE HAS
KIND OF SHIFTED DRAMATICALLY
OVER THE LAST COUPLE YEARS.
SO SINCE WE'RE INTERESTED IN
KIND OF THAT SHIFT, WE'RE
LOOKING AT RELATIVELY SHORTER
TIME, ON THE ORDER OF, YOU KNOW,
SIX MONTHS TO THREE YEARS,
ROUGHLY SPEAKING.
SUFFICIENT WHERE THERE WILL BE
SOME ADVERSE OUTCOMES, IN LARGE
POPULATIONS.
BUT NOT TO THE POINT WHERE WE
DON'T HAVE TO WAIT, YOU KNOW,
MANY YEARS TO ACTUALLY OBTAIN
THOSE OUTCOMES, ESPECIALLY WITH
THE RISE OF ITS USE FOR WEIGHT
LOSS.
WE ALREADY HAVE A SUBMITTED
MANUSCRIPT WHETHER WE -- GLP-1
FOCUSING ON AGAINST PROFILE,
PARTICULARLY SNP PROFILES.
IN THAT CASE WE'RE LOOKING, YOU
KNOW, AT ROUGHLY THAT TIME POIN.
>> I HAVE A COUPLE BIG PICTURE
QUESTIONS.
YOU MENTIONED A COUPLE TIMES
CAUSAL INFERENCE.
OBVIOUSLY TEMPORAL DIMENSION OF
YOUR WORK SUPPORTS CAUSAL
INFERENCE GIVEN THE TIME
PRECEDENCE OF THE RISK FACTORS
BEFORE THE OUTCOME.
BEYOND THE TEMPORAL DIMENSION,
ARE THERE OTHER METHODS YOU'RE
USING OR THINKING ABOUT USING TO
SUPPORT CAUSAL INFERENCE FROM
OBSERVATIONAL DATA?
>> YES, I THINK YOU'RE GOING TO
HEAR ABOUT THAT SPECIFICALLY
FROM SHARA AT THE POSTER
SESSION.
SO, WE THINK ABOUT PRIOR WORK
THAT FOCUSES ON MACHINE LEARNING
MODELS, OR CAUSAL INFERENCE, AND
CLASSICALLY YOU ADOPT BASICALLY
POTENTIAL OUTCOME FRAMEWORK FOR
OBSERVATIONAL DATA AND YOU'RE
GOING TO TRY TO DETERMINE
WHETHER OR NOT THE ASSUMPTIONS
THAT ARE REQUIRED FOR THAT
FRAMEWORK ARE, YOU KNOW,
JUSTIFIABLE.
AND ONE OF THE THINGS THAT YOU
CAN NEVER KNOW ABOUT IS
UNMEASURED CONFOUNDING, AND SO
ONE ARGUMENT ABOUT OUR WORK IS
THAT BECAUSE WE'RE EXTRACTING
THE BEST AVAILABLE INFORMATION
THAT WE CAN ABOUT THE PATIENTS,
IN A TEMPORAL WAY, THAT THIS IS
ACTUALLY IMPROVING OUR ABILITY
TO CHARACTERIZE THE COVARIATE,
WHICH GIVES YOU A BETTER CHANCE
AT MODELING THE UNMEASURED
CONFOUNDING THAT MAY NOT BE
CAPTURED IN SIMPLER
REPRESENTATIONS OF THE COVARIATE
PROFILE.
SO THAT'S ONE WAY.
AND THEN YOU'LL HEAR MORE FROM
SHARA BUT, YOU KNOW, THERE ARE
SOME OPEN QUESTIONS WITH RESPECT
TO HOW CAUSAL INFERENCE METHODS
CAN BE APPLIED IN SURVIVAL
ANALYSIS SETTINGS.
AND SO WE'VE GOT A LINE OF WORK
THAT TRIES TO DEMONSTRATE WHAT
ARE BEST PRACTICES WHEN WE
COMBINE THESE TWO AREAS OF STUDY
AND YOU'LL SEE THAT IN MORE
DETAIL.
>> I LOOK FORWARD TO THAT.
THANKS.
GRACIELA?
>> JUST A POINT OF
CLARIFICATION ABOUT YOUR WORK ON
SEPSIS.
THE OUTCOME THAT YOU WERE
PREDICTING OR MEASURING IS ONLY
SURVIVAL OR DO YOU DO ANY
OTHERS?
OR PRESENCE OR ABSENCE OF
SEPSIS?
OR IT WASN'T CLEAR TO MY WHEN
YOU PRESENTED, WHAT WAS IT THAT
YOU WERE TRYING TO PREDICT IN
THE EVENT?
>> SURE.
YES, SO MORTALITY IS THE PRIMARY
OUTCOME.
THERE ARE ALSO SECONDARY
OUTCOMES OF MULTI-ORGAN FAILURE
AND MECHANICAL VENTILATION.
DEPENDING WHICH WORK YOU'RE
REFERRAL TO IT WILL BE A
MIXTURE, WE'RE LOOKING
DOWNSTREAM, NOT TRYING TO DETECT
SEPSIS.
>> IS IT WITHIN A CERTAIN PERIOD
OF TIME?
BECAUSE IF YOU WERE USING DATA
IN THE FIRST SIX HOURS I THINK
YOU SAID, WAS THERE A WINDOW IN
YOUR LONGITUDINAL PREDICTION?
>> YES, IT VARIES.
IT COULD BE WITH LIKE INPATIENT
MORTALITY, IT COULD BE 30-DAY
MORTALITY, IT COULD BE 90-DAY
MORTALITY.
>> OKAY.
ALL RIGHT.
AND IF NOBODY ELSE HAS OTHER
QUESTIONS I HAVE ONE MORE, WHICH
IS ABOUT SHARED TASKS.
HAVE YOU ORGANIZED OR
PARTICIPATED, OR ORGANIZED IS MY
QUESTION, ACTUALLY, SHARED TASKS
WITH DATA THAT YOU HAVE
ANNOTATED AND CREATED?
>> NOT YET.
SO WE ARE IN PREPARATION FOR A
SUBMISSION FOR BENCHMARKS VENUE
COMING IN A MONTH, AND SO PART
OF THAT RELEASE WAS GOING TO BE,
YOU KNOW, GATHER PEOPLE TO
ITERATE AND FIGURE OUT HOW WELL
THEY CAN DO.
>> ALL RIGHT.
JUST FOR THE RECORD, BIOCREATIVE
IS ORGANIZED, I THINK THIS WILL
BE PERFECT FOR THE FUTURE
ITERATIONS.
WE'RE RUNNING THE NEXT ONE
ALREADY AND WE HAVE THE SHARED
TASK, BUT IT WILL BE GREAT TO
SEE YOU ON THAT.
OR OTHERS.
SHARED TASKS PLAY A REALLY
PIVOTAL ROLE IN MOTIVATING OR
PEOPLE AND PREPARING OUR WORK
AND SO ON.
>> YEAH, SOUNDS LIKE A GREAT
OPPORTUNITY.
>> JEREMY, YOU GAVE AN
INTERESTING PREVIEW OF YOUR
LARGE LANGUAGE MODEL TOWARDS THE
END, LLMs ARE GOODED A MAKING
PREDICTIONS, USING LATENT SPACE
TO MAKE PREDICTIONS, WONDERING
IF YOU HAVE INTEREST OR PLANS IN
TRYING TO UNPACK OR DECODE THAT
LATENT SPACE AND FIGURE OUT THE
RISK FACTORS OF INTEREST, WHAT
IS THE INFORMATION CLINICS ARE
COLLECTING MOST RELEVANT TO YOUR
OUTCOMES OF INTEREST.
>> THAT'S A VERY INTERESTING
QUESTION, A REALLY IMPORTANT
QUESTION.
A Ph.D. IN INTERPRETIVE
MACHINE LEARNING FOR HEALTH
CARE, SO THAT'S SOMETHING THAT
HE COULD DEFINITELY LOOK TO
TACKLE.
WE HAVE WORKED TRYING TO DETECT
LIKE THE HIGH-LEVEL CONCEPTS
THAT CAN BE USED IN AND
TRANSPORT BETTER WHEN MAKING
PREDICTIONS FROM, YOU KNOW,
ESSENTIALLY BLACK BOX TYPE
MODELS, HAVEN'T APPLIED THEM TO
EMBEDDING SPACINGS OF LARGE
LANGUAGE MODELS, THERE'S WORK IN
FROM INTERROGATING EMBEDDING
SPACE OF THESE MODELS.
SO, YEAH, I THINK IT'S AN
IMPORTANT QUESTION.
WE HAVEN'T JUMPED INTO THAT
SPACE WITH OUR CURRENT PROJECT,
BUT IT'S ON THE -- WE'RE
DEFINITELY THINKING BIT,
THINKING WE HAVE AN ANGLE THAT
WE THINK, YOU KNOW, WILL HELP US
OR LIKE WHERE WE CAN DO IT
BETTER AND SHOW THE COMMUNITY
THAT IT'S ALREADY THINKING ABOUT
THAT, YOU KNOW, WHAT WE CAN
OFFER.
>> GOOD.
THANKS.
I DON'T SEE ANY OTHER HANDS UP.
DO WE HAVE ANY MORE QUESTIONS
FOR JEREMY BEFORE WE MOVE TO THE
CLOSED POSTER SESSION?
OKAY.
JEREMY, THANKS A LOT.