Welcome to the Clinical Center Grand Rounds,
a weekly series of educational lectures for physicians and
health care professionals broadcast from the Clinical
Center at the National Institutes of Health in
Bethesda, MD.
The NIH Clinical Center is the world's largest hospital totally
dedicated to investigational research and leads the global
effort in training today's investigators and discovering
tomorrow's cures.
Learn more by visiting us online at http://clinicalcenter.nih.gov
>>OUR SPEAKER IS GEORGE HRIPCSAK
DIRECTOR OF MEDICAL
INFORMATICS SERVICES FOR
NEW YORK PRESBYTERIAN HOSPITAL
COLUMBIA CAMPUS, HE IS A BOARD
CERTIFIED INTERNIST AND HE LEADS
THE OBSERVATIONAL HEALTH DATA
SCIENCES AND INFORMATICS
COORDINATING CENTER AND
INTERNATIONAL NETWORK WITH
THOUSANDS OF COLLABORATORS.
HE EARNED HIS MEDICAL DEGREE
FROM COLUMBIA MEDICAL SCHOOL AND
MASTERS DEGREE IN BIOSTATISTICS
AND THE SCHOOL OF PUBLIC HEALTH.
HIS RESEARCH FOCUSES ON THE
CLINICAL INFORMATION STORE
INDEED ELECTRONIC HEALTH RECORDS
AND ON THE DEVELOPMENT OF NEXT
GENERATION HEALTH RECORDS
SYSTEMS.
USING NONLINEAR TIME SERIES
ANALYSIS, MACHINE LEARNING,
KNOWLEDGE ENGINEERING AND
NATIONAL LANGUAGE PROCESSING,
DR. HRIPC SAK, THE DOCTOR
CHAIRED THE NATIONAL LIBRARY AND
INFORMATICS REVIEW COMMITTEE, HE
IS AN ELECTED MEMBER OF THE
NATIONAL ACADEMY OF MEDICINE AND
AN ELECTED FELLOW OF THE
AMERICAN COLLEGE OF INFORMATICS
IN THE NEW YORK ACADEMY OF
MEDICINE, HE IS ALSO A MEMBER OF
INTERNATIONAL ACADEMY OF HEALTH
SCIENCES INFORMATICS.
HE HAS PUBLISHED OVER 450
PUBLICATIONS.
THE TITLE OF DR. HRIPSAC'S
PRESENTATION IS GREAT TEACHERS
DRAWING REPRODUCIBLE CONCLUSIONS
IN OBSERVATIONAL CLINICAL DATA,
PLEASE WELCOME ME IN JOINING
TODAY'S GREAT TEACHER DR. HRIPSCAK
THANK YOU SO MUCH, AGAIN IT'S AN
HONOR TO BE ASKED TO DO THIS AND
IT'S SUCH FUN TO TALK ABOUT
THIS, SO AS YOU HEARD ALL I'LL
BE TALKING ABOUT DRAWING
REPRODUCIBLE CONCLUSIONS FROM
CLINICAL DATA, NO CONFLICTS OF
INTEREST OR OFFLABEL USE, THE
OBJECTIVES ARE TO IDENTIFY THE
CHALLENGES TO REPRODUCIBILITY
AND OBSERVATIONAL RESEARCH AND
WHAT APPROACHES WE CAN USE TO
ADDRESS THOSE CHALLENGES.
SO WE ALL KNOW THERE ARE
CHALLENGES, HERE'S AN EXAMPLE OF
2 PAPERS PUBLISHED A MONTH
APART, 2 TOP JOURNALS, SAME
QUESTION, DO FOSTER NURSED
FOCUSED ONINATES CAUSE
ESOPHAGEAL CANCER, 1 SAYS YES, 1
SAYS NO BUT THEY USE THE SAME
DATABASE SO IT'S 2 DIFFERENT
GROUP COMING TO THE OPPOSITE
CONCLUSION AT LEAST TO THE
PUBLIC'S EYE THAT TO THE MORE
SOPHISTICATED READER THEY WOULD
LOOK AND SAY, WELL, THE
CONFIDENCE INTERVAL IS NOT
REALLY THAT DIFFERENT, 1 JUST
MADE AND 1 JUST MISSED THE
THRESHOLD BUT STILL THE PUBLIC
DOESN'T KNOW WHAT TO DO WITH
THIS AND THEY READ THAT COFFEE
CAUSES CANCER 1 YIER AND THE
NEXT YEAR CURES IT AND THEY
DON'T KNOW WHAT TO DO WITH
OBSERVATIONAL RESEARCH.
MORE RECENTLY WE'VE SEEN PAPERS
RETRACTED FROM TOP JOURNALS IN
THE SETTING OF COVID-19, AND A
LOT OF IT ABOUT THE DATA THAT
WAS USED AS A SOURCE.
AND IF WE SEE EDITORIALSOT TOPIC
OF OBSERVATIONAL RESEARCH HERE
BY FRANCIS COLLINS, LOOKING AT
BOTH RANDOMIZED TRIALS ARE KNOWN
TO WORK, AND OBSERVATIONAL
RESEARCH ARE REAL WORLD EVIDENCE
OFTEN LEADS TO DISTRACTING
CONCLUSIONS.
SO THE THESIS OF AND I WILL HAVE
A SLIGHTLY DIFFERENT TOPIC BUT
MOST OF THIS WILL BE ABOUT GRAVE
PROBLEMS AND WHAT I'M CALLING
OBSERVATIONAL RESEARCH, DATABASE
RESEARCH, NONEXPERIMENTAL
RESEARCH, THAT IS GETTING DATA
FROM THESE LARGE DATABASES AND
DRAWING CONCLUSIONS.
BUT I BELIEVE THAT MODERN
METHODS AND DATA HAVE THE
POTENTIAL TO CHANGE THE WAY WE
GENERATE EVIDENCE.
OUR CLINICAL GUIDELINES TODAY WE
TRY TO BASE THEM ON RANDOMIZED
TRIALS BUT WE DON'T HAVE THE
TRIAL SO WE BASE THEM MOSTLY ON
EXPERT OPINION OPINION AND I
WILL GIVE YOU AN EXAMPLE OF THAT
LATER AND WE CAN FILL THOSE GAPS
WITH OBSERVATIONAL RESEARCH SO
IT'S NOT ROLELY A QUESTION OF
WHETHER WE DO RANDOMIZED TRIALS
OR OBSERVATIONAL RESEARCH, IT'S
WHETHER WE DO EXPERT OPINION FOR
OBSERVATIONAL RESEARCH AND HOW
CAN WE FILL IN THE GAP WHERE
RANDOMIZED TRIALS CAN DO THE
TASK THIS, IS A FRAMEWORK BY
SOMEONE NAMED PATRICK RYAN WHO'S
STRONG IN THE EFFORT I WILL BE
TALKING ABOUT AND YOU SEE THE
WORDS ON THE LEFT, DON'T WORRY
TOO MUCH WE HAD TO COME UP WITH
LABELS OF REPEATABLE,
REPRODUCIBLE AND THE THING IS
WHEN YOU DO A STUDY, WHAT DOES
IT MEAN FOR IT TO BE RELIABLE,
LFIRST IF THE SAME RESEARCH
USING THE SAME DATA AND THE SAME
ANALYSIS WILL HOPEFULLY GET THE
SAME ANSWER AND THAT'S THE FIRST
ROW OF THE TABLE, THE SECOND ROW
OF THE TABLE IF A DIFFERENT
RESEARCHER DOES THE SAME STUDY
AND SAME DATA THEY WOULD GET THE
SAME ANSWER.
THE THIRD ROW SAYING IF WE
CHANGE THE DATA TO A DIFFERENT
POPULATION BUT WITH SIMILAR
CHARACTERISTICS WE WOULD HOPE TO
GET A SIMILAR OUTCOME.
AND THEN SLIGHTLY DIFFERENT
TOPIC, IF THE STUDIES
GENERALIZE DIFFERENT POPULATIONS
SO WE'RE TALKING
ABOUT OBSERVATIONAL RESEARCH AND
WE'RE TALKING ABOUT CHALLENGES
TO REPRODUCIBILITY.
SO I'M GOING TO LIST A FEW OF
THOSE CHALLENGES, IT USED TO BE
SAMPLE SIZE WAS 1 OF THEM BUT
REALLY WITH OUR LARGE MODERN
DATABASES, THE STATISTICAL ERROR
SCHUNK SO MUCH, WHAT'S LEFT IS
SYSTEMATIC ERROR AND IN OTHER
WORD FIST I DO A STUDY ON A
MODERN DATABASE WITH A HUNDRED
MILLION DOLLAR PATIENTS IN IT, I
WILL GET SMALL CONFERENCE
INTERVALS, AND RPGHT BIASES THAT
ARE INVOLVED THERE, SO I FALSELY
THINK I'M GOING TO GET AN
ACCURATE ANSWER WHETHER IN FACT,
MY SYSTEMATIC ERROR IS JUST AS
BIG AS IT WAS WHEN I HAD A
SMALLER DAT BASE, SO WHAT'S THE
BIG PROBLEM THERE IS CAUSAL
INFERENCE, WHAT'S THE DIFFERENCE
TBEEN AN ASSOCIATION AND ACTUAL
CAUSE, WITHOUT RANDOMIZATION,
CONFOUNDERS CAN LINK TREATMENTS
TO OUTCOMES THAT DON'T BELONG
THERE.
SO IMAGINE THAT I'M STUDYING
BUTIANYL GAS AND DOES BUTIANYL
GAS CAUSE LUNG CANCER, OKAY?
IF I DO A TODAY WITH A
OBSERVATIONAL DATABASE, I MIGHT
LOOK AND SAY WHO'S CARRYING A
BUTIANYL LIGHTER THAT CONTAINS
BUTIANYL GAS, I FOUND A LOT
GETTING LUNG CANCER, I FOUND A
LINK, THEN I REALIZED A LINK,
SMOKING, SMOKING BITE BE THE
REASON SOMEONE'S CARRYING A
BUTIANYL LIGHTER AND THE SMOKING
MAY CAUSE LUNG CANCER SO THAT'S
A CONFOUNDER AND THAT'S OVER A
LOT OFFER WORK GOES INTO HOW TO
HANDLE THAT WITHOUT
RANDOMIZEITIONA.
ALSO, ANOTHER CHALLENGE IS
MEASURING AN ERROR THAT IS DO
ELECTRONIC HEALTH RECORDS
ACCURATELY REPRESENT THE STATUS
OF THE PATIENT.
THERE'S ALSO DECEPTION, OFTEN
SUBCONSCIOUS P-VALUE HACKING
THAT IS WHEN YOU RUN A STUDY YOU
HAVE AN EXPECTATION OF A RESULT,
YOU DON'T GET THE RESULT, YOU
INVESTIGATE TO SEE WHAT'S WRONG
WITH YOUR STUDY, YOU CHANGE YOUR
STUDY UNTIL YOU GET THE ANSWER
YOU'RE EXPECTING AND THEN YOU
STOP.
IF YOU DO YOUR STUDY AND GET THE
ANSWER YOU WERE EXPECTING THEN
YOU STOP DOING YOUR STUDY AS
SOON AS YOU GET THAT ANSWER, SO
YOU ALWAYS GRAVITATE TOWARDS THE
ANSWER YOU WERE EXPECTING.
AND LAST GENERALIZABILITY DOES
IT APPLY TO MY PATIENTS?
SO, I'M GOING TO TALK ABOUT
SPECIFICALLY ABOUT THE
OBSERVATIONAL HEALTH DATA
SCIENCES AND INFORMATICS OR
ODESSY INITIATIVE, THIS IS AN
INITIATIVE--IT'S A GLOBAL
COMMUNITY, WE HAVE OVER 2000
COLLABORATORS FROM STORY TELLERS
COUNTRYINGS WE HAVE A DATA STATE
LOCAL, BUT IN THAT FEDERATED
DATABASE WE HAVE OVER
800 MILLION UNIQUE PATIENTS AND
300 DATABASES OR 300 SITES SWEE
HAVE OVER 10% OF THE WORLD'S
POPULATION IN THIS DATABASE.
AS SINCE ODESSY WAS CREATED IN
2014, WE PUBLISHED 344 PAPERS,
AND WE'VE HAD SPECIFIC
INFLUENCEOT EUROPEAN MEDICINE'S
AGENCY, AND THE FOOD AND DRUG
ADMINISTRATION FOR EXAMPLE, WITH
THE EVIDENCE THAT WE'VE
PRODUCED.
WHY DO WE NEED SUCH LARGE
DATABASES?
WHEN I DO A STUDY AT COLUMBIA, I
MIGHT HAVE A DATABASE WITH A
MILLION PATIENTS, ISN'T THAT
ENOUGH TO DO MY STUDIES, WELL IF
YOU THINK ABOUT IT, ONCE YOU
PICK A DISEASE, AND YOU PICK A
DRUG AND YOU PICK A SIDE EFFECT,
YOU MAY BE DOWN FROM THAT
1 MILLION TO 10 PATIENTS BUT IF
I HAD STARTED WITH A BILLION
PATIENTS THEN I WOULD BE DOWN TO
10,000, I COULD ACTUALLY CARRY
OUT A STUDY AND DO THE KIND OF
ANALYSIS AND DIAGNOSIS THAT
ODESSY DOES.
SO ODESSY GENERATES 3 KINDS OF
EVIDENCE, WE BREAK IT INTO
CLINICAL CHARACTERIZATION AND
THAT SIMPLE TALLYING COUNTING
HOW OFTEN DO THINGS HAPPEN AND
THAT IS SOMETIMES USEFUL, THE
SECOND 1 IS THE MAIN 1 WE THINK
OF AND THAT'S POPULATION LEVEL
ESTIMATION OR CAUSAL INFERENCE,
SO THIS 1 IS LIKE THIS, DOES
METFORMIN CAUSE LACTIC
ADIDOSEIS, AS OPPOSE TO THE
FIRST 1 WHICH IS HOW MANY PEOPLE
HAVE DIABETES AND THE LAST 1 IS
PATIENT LEVEL PREDICTION, THAT
IS WHAT ARE THE ODDS OF A
PATIENT GETTING A SIDE EFFECT
BASED ON THEIR HISTORY.
ODESSY IS AN OPEN SCIENCE
APPROACH WHICH MEANS EVERYTHING
IS OPEN, OPEN SOURCE SOFTWARE,
EVERYTHING WE DO IS SHARED SO
WHEN WE GENERATE A STUDY, THE
FIRST THING WE DO IS SHARE OUR
PROTOCOL ON THE INTERNET, THEN
WE SHARE OUR SOURCE CODE ON THE
INTERNET, WE NEVER SHARE PATIENT
DATA SO THAT'S THE 1 EXCEPTION,
AS WE GENERATE RESULTS WE PUT
THEM ON THE INTERNET, TOO, WE
FIRST SHOW THE DIAGNOSTICS TO
SEE IF WE PASS OUR OWN
DIAGNOSTICS AND THEN WE SHOW THE
RESULTS, IN THIS WAY, BY FIRST
SHARING THE PROTOCOL SO YOU KNOW
WHAT WE'RE GOING TO DO AND
SECOND BY LETTING EVERYONE ELSE
SEE THE RESULTS AS SOON AS WE
DO, THERE'S NO WAY FOR US TO
P-HACK OR TO CHANGE THE SYSTEM
TO GET THE ANSWERS WE WERE
LOOKING FOR.
>> JUST SO YOU KNOW, THE SLIDES
ARE UP.
>> OH GREAT.
THANK YOU.
NOW HOW DO YOU GET--I'M GOING TO
FLIP BACK BECAUSE I LIKE THIS
GRAPH, SO THAT'S OUR COMMUNITY
RIGHT NOW, I JUST WANTED TO SHOW
IT AS A WORLD GRAPH, STRONG IN
NORTH AMERICA AND IN EUROPE, AND
IN ASIA-PACIFIC AND WORKING ON
SOUTH AMERICA AND AFRICA.
WE HAD 1 PERSON IN ANT ARCTICCA
BRIEFLY AND THE NEXT PERSON DID
NOT RETURN OUR CALLS, IT WASSA
SMALL DATABASE SO WE COULD COVER
THIS.
>> NOW MOVING FORWARD, HOW DOES
ODESSY WORK, IS WELL HOW DO YOU
GET 10% OF THE WORLD'S
POPULATION, THE ANSWER IS YOU
DON'T, YOU LEAVE THE DATA LOCAL,
YOU DO HAVE THEM CONVERTED TO A
COMMON DATA MODEL WHICH IS OMOP
WAS BASICALLY OBSERVATIONAL
WORK, YOU CONVERT YOUR DATA TO
OMOP AND THEN WE SHARE THE
QUESTION WITH YOU, YOU RUN THE
QUESTION LOCALLY, IT GENERATES
RESULTS WHICH YOU SHARE
ESSENTIALLY AND THEN WE
COLLABORATE TOGETHER TO
INTERPRET THOSE RESULTS AND
PUBLISH THEM, AND THAT'S WHY
ODESSY PAPERS TEND TO HAVE A LOT
OF AUTHORS.
HERE'S ANOTHER VIEW OF OW ODESSY
WORKS THAT STRETCHES THE
NETWORK.
EACH NOTE CAN GENERATE QUESTIONS
THAT WILL RUN, NOW WE REALIZE
IT'S A VOLUNTARY NETWORK SO IN
NO STUDIES DO WE ACTUALLY HAVE
800 MILLION PATIENTS, WE WILL
FREQUENTLY GET 300-400 MILLION
POPULATION AS DIFFERENT GROUPS
JOIN IN THE STUDY A LOT.
THEN WE JOIN IN, THAT DON'T
NECESSARILY HAVE COLLABORATION
BUT HELP US IN THE METHOD LOGIC
RESEARCH AND GENERATING TOOLS
AND MOST IMPORTANTLY IN
GENERATING STUDIES FOR US TO
CARRY OUT.
THEN WE RUN THESE NETWORK
STUDIES AS I SAID, COLLECT THE
RESULTS AND DISSEMINATE THE
EVIDENCE.
THIS COMMON DATA MODEL IS VERY
IMPORTANT TO US SO JUST LIKE
WHEN YOU HAVE AN APPLIANCE AND
YOU TRAVEL AROUND THE WORLD AND
TRY TO PLUG IT IN, IT DOESN'T
WORK, YOU COULD FIND WAYS TO PUT
IT IN BUT IF YOU HAVE A COMMON
STANDARD THEN SUDDENLY YOU CAN
USE YOUR APPLIANCE ANYWHERE IN
THE SAME WAY THAT THE MODEL IS
THE STANDARD THAT ALLOWS US TO
SHARE--AS IS FIRST OF ALL A DEEP
INFORMATION MODEL SHOWN HERE SO
THIS IS THE DATA SCHEMA, AS YOU
LOOK AT IT, YOU WILL SEE THINGS
YOU RECOGNIZE, DRUG EXPOSURE,
PROCEDURE, DEVICE, MEASUREMENT
WHICH IS THE LAB TEST, CONDITION
OCCURRENTS DIAGNOSIS AND SO ON,
IT'S INTENDED TO BE ACCESSIBLE
TO NEW RESEARCHERS SO THAT CAN
YOU LOOK AT OUR TABLES AND
FIGURE OUT WHAT IT IS YOU NEED
TO DO, OF COURSE YOU STILL NEED
THE DOCUMENTATION, BUT WE MAKE
IT AVAILABLE TO RESEARCHERS.
THEN WE NEED OF COURSE,
EXTENSIVE VOCABULARY SINCE WE'RE
DOING RESEARCH AROUND THE WORMED
THIS, IS 1 IMAGE OF IT BUT
HERE'S A LISTING OF THE
VOCABULARIES AND YOU NOTE WE
HAVE OVER 9 MILLION CONCEPTS
TAKEN FROM 153 VOCABULARIES
FROM,A ROUND THE WORLD.
THESE THINGS GET CONVERTED TO A
STANDARD WHICH IS PRETTY MUCH
THE FORMER MEANINGFUL USE 3
STANDARD SNOW MED, ARC STORM AND
LOINC, WHICH ARE USED FOR ANY
DATABASE IN ANY COUNTRY.
AND FOR THOSE WHO HAVE DONE DATA
CONVERSION BEFORE KNOW, THE FACT
THAT YOU HAVE A COMMON SCHEMA
AND A COMMON VOCABULARY, YOU
GIVE THE SAME DATA TO 2
DIFFERENT PEOPLE AND THEY WILL
COME UP WITH 2 DIFFERENT WAYS CO
CONVERTING IT TO THAT DATA
MODEL.
SO THE IMPORTANT PART OF THE
ODESSY OF CONVENTIONS THAT
EXPLAIN TO YOU, HOW DO YOU
CONVERT YOUR DATA TO THE MODEL,
FOR EXAMPLE, IF SOMEONE DIES,
WHAT'S THEIR DIAGNOSIS, IS IT
THAT THEIR HEART STOPS, IS IT
THE THING THAT OCCURRED A FEW
MINUTES BEFORE THAT PRECIPITATED
THEIR HEART STOPPING OR THE
CHRONIC DISEASE THAT THEY HAD
AND WHERE DO YOU PUT THOSE
DIFFERENT THINGS?
THOSE ARE ALL IN COWER
CONVENTIONS?
AND THEN WE GIVE YOU AMPLE TOOLS
TO CONVERT YOUR DATA TO THE
COMMON DATA MODEL.
SHOWN HERE.
I WON'T GO THROUGH THEM IN
DETAIL.
DATA QUALITY IS CRITICALLY
IMPORTANT AND WE HAVE A
DASHBOARD WITH THOUSANDS OF
CONCESTENCY CHECKS THAT GO
THROUGH THE TABLES AND MAKE SURE
THAT THE DATA PASSES THOSE
CHECKS, OFTEN PEOPLE TELL US
THAT THE MOST IMPORTANT THING
THAT ODESSY BROUGHT THEM WAS NOT
THE RESEARCH BUT THE FACT THAT
THEY FOUND ALL THOSE PROBLEMS IN
THEIR SOURCE TABLES AND END UP
GOING BACK AND FIXING THEIR
DATABASES AND FRY TO DO THE
CONVERSION TO ODESSY, THEY LEARN
THERE ARE PROBLEMS IN THE DATA
THAT THEY HAVE.
THEN WE GIVE TOOLS TO TRAVERSE
THESE 9 MILLION TERMS IN THE
TERMINOLOGY TO FIND THE
PHENOTYPES THAT IS THE CLINICAL
CONDITIONS YOU'RE LOOKING FOR
AND THEN A TOOL TO BUILD IT
WHICH IS SEMIGRAPHICAL SHOWN
THERE ON THE BOTTOM OF THE
SLIDE.
WE HAVE A WHOLE PROCESS FOR WHAT
WE CALL PHENOTYPING.
NOW WHEN I SAY PHENOTYPING HERE,
I MEAN THE WAY THAT CLINICAL
INFORMATICS HAS STOLEN THE TERM
AND IT MEANS CONVERTING RAW DATA
YOU MIGHT FIND IN THE ELECTRONIC
HELT RECORD OR CLAIMS DAILY
BASIS THEA BASE AND TURN IT INTO
A CLINICAL CONCEPT OF USE FOR
RESEARCH.
AND THERE ARE A NUMBER OF TOOLS
LIKE CD AND ATLAS THAT EXPLOIT
INFORMATION IN OUR NETWORK TO
FIGURE OUT WHAT BEST TO CALL
THINGS AND THEN TOOLS,
DIAGNOSTICS AND FEE EVALUATOR TO
SEMIAUTOMATICALLY REVIEW THOSE
PHENOTYPES AND MEASURE OR AT
LEAST ESTIMATE THEIR SENSITIVITY
AND POSITIVE PREDICTIVE VALUE.
AND THEN WE STORE THOSE
PHENOTYPES FOR FUTURE USE AND
THEN TOOLS FOR VISUALIZING AND
TOOLS TO DESIGN MORE COMPLEX
STUDIES, SOME OF THE EXAMPLES OF
WHICH I'LL SHOW IN A LITTLE BIT.
NOW WHILE IESM TALKING ABOUT THE
COMMON DATA MODEL, I WANT TO
MENTION THE HL7 ODESSY
PARTNERSHIP.
HR7 AND HEALTH LEVEL 7 AND 1 OF
THE MOST IMPORTANT CARE STANDARD
ORGANIZATIONS THAT OVER SEE THE
STANDAS WE USE IN HEALTH RECORDS
AND ON MARCH 1st WE ANNOUNCED
A PARTNERSHIP TO BRING OUR
STANDARDS TOGETHER.
WE'RE STILL IN THE CREATION OF
OUR JOINT WORKING GROUPS AND
SCOPES STARTING WITH EXISTING
WORK WE HAD BEEN DOING WITH
PLACES AT GEORGIA TECH AND IN
EUROPE.
THE RANGE OF WHAT WE MIGHT
ACCOMPLISH IS FIRST MAPPING
BETWEEN THE DAT STANDARDS WHICH
IS A BEAR MINIMUM BECAUSE WE'RE
DOING FURTHER LIKE SHARED
KNOWLEDGE ENGINEERING AND COMMON
STANDARDS.
THE INITIAL WORK--WORKING GROUPS
THAT CAME OUT OF THE FIRST
WORKSHOP ARE SHOWN HERE, DATA
HARMONIZATION BETWEEN HL7'S FIRE
STANDARD AND ODESSY'S OMOP
STANDARD AND THEN DIGITAL
QUALITY MEASURES AND THEN KIND
OF A CATCH ALL FOR THINGS OTHER
PEOPLE WERE INTERESTED IN
REGISTRIES, PREDICTIVE MODELS
AND DECISION SUPPORT.
SO NOW I WANT TO MOVE TO TALKING
ABOUT ODESSY IN ACTION, THAT IS
GENERATING EVIDENCE SINCE THAT
IS OUR PURPOSE.
THIS WAS OUR FIRST LARGE STUDY,
WE DID PUBLISHED IN 2016, ON
TREATMENT PATHWAYS, THAT IS WHAT
DRUGS TO PEOPLE ACTUALLY TAKE?
WE KNOW HOW OFTEN DRUGS ARE SOLD
AND WE KNOW IN SOME ACADEMIC
MEDICAL CENTERS MIGHT REPORT ON
WHAT'S--AND SO FORTH WITH MANY
TAKE STOLEDDERS, THIS STUDY HAD
240 MILLION AND UNDERLYING
POPULATION FROM 5 DIFFERENT
COUNTRIES.
IT'S SHOWN IN THE NATIONAL ACAT
ME OF SCIENCES I'LL SO YOU THE
MAIN EVENT AS A IS CIRCLE, THIS
IS A SUN BURST PLOT, THE INNER
CIRCLE IS THE FIRST DRUG THE
PATIENT TOOK, THE SECOND CIRCLE
IS THE SECOND DRUG AND SO ON AS
YOU MAKE YOUR WAY OUT, THERE
COULD BE AS MANY AS 20 CIRCLES
BUT ONCE IT BECOMES A RARE THING
THEN WE STOP SHOWING IT ON THE
SCREEN BECAUSE THE LINE WOULD BE
TOO SMALL.
WHERE THE CIRCLES STOP GOING,
WHERE I HAVE AN ARROW SAYING
ONLY DRUG, THAT'S SO FOR EXAMPLE
IN THIS SLIDE, 29% OF THE
PATIENTS GOT METFORMIN FOR TYPE
2 DIABETES AS THEIR ONLY DRUG.
SEVENTY-FIVE% OF THE PATIENTS
STARTED ON METFORMIN, 29% DID
ONLY METFORMIN IN A 3 YEAR
PERIOD, AND THEN AND THE SECOND
DRUG, THEY'RE SHOWN HERE WITHOUT
A LOT OF--THERE'S PARTICULARLY
NO PREDOMINANT DRUG AFTER THAT,
LET ME TELL YOU WHAT KNOWLEDGES,
DID THEY SWITCH TO A DRUG?
I'VE MIXED AND SWITCHING DRUGS.
THREE DISEASES TYPE 2 DIABETES,
HYPER TENSION AND DEPRESSION.
THE LEFT MOST COLUMN, YOU SEE
CCACE, THAT'S THE US, THAT'S THE
CPRD, THAT'S THE UK, THE BOTTOM
1 IS JAPAN, IF YOU LOOK AT OUR
DWELVE DATA SETS, METFORMIN IS
USED 75% OF THE TIME, THE BOTTOM
1 JAPAN YOU SEE IT'S USED ONLY
ABOUT 25% OF THE TIME.
WE THOUGHT THAT WAS PROBABLY DUE
TO THE FORMULARY AND HOW THE
GOVERNMENT APPROVED THINGS, IT
TURNS OUT SPEAKING TO A
DIABETOLOGYST IN JAPAN THAT IT'S
BELIEVED THAT JAPAN PATIENTS ARE
SLIGHTLY GENETIC RADIOLY
DIFFERENT AND DON'T GET INSULIN
RESISTANCE AND THEREFORE DON'T
USE METFORMIN DRUGS, THEY STILL
USE IT MORE OFTEN THAN OTHER
DRUGS BUT OTHER DRUGS MORE MORE
PREDOMINANT.
SO IN HYPER TENSION THERE'S
STILL AGREEMENT BUT BUT LESS
AGREEMENT.
BUT THERE'S VERY LITTLE
AGREEMENT WHICH MAKES SENSE, WE
DON'T KNOW WHICH DRUGS SHOULD BE
TARGET TO WHICH PATIENTS AND YOU
SEE EVEN THOSE 3 U.S. DATABASES,
WHICH ARE SHOWN ON THE RIGHT,
THERE'S COMPLETE DISAGREEMENT ON
THE TOP DRUG.
AN INTERESTING FINDING IS THAT
25% OF THE HYPER TENSION
PATIENTS HAVE THE UNIQUE PATH OF
DRUGS DESPITE THIS
250 MILLION-DOLLAR POPULATION,
SO WHAT I'M SAYING IN AN
UNDERLYING POPULATION OF
250 MILLION, YOU TAKE EVERYONE
WITH HYPER TENSION, A COUPLE
MILLION AND 25% OF THEM TOOK A
PATH OF DRUGS, HYPER TENSION
DRUGS THAT MATCHED NO 1 ELSE IN
THE DATABASE AND THAT MEANS THAT
WE'RE DOING VERY PERSONALIZED
TREATMENT, ALTHOUGH NOT VERY
PRECISE.
THIS STUDY SHOWED US JUST ON THE
NETWORK RESEARCH SIDE THAT IT'S
FEASIBLE TO ENCODE THE WORLD
POPULATION OVER 10% NOW, I
SHOULD FIX THAT SLIDE,
GENERATING EVIDENCE IS FEASIBLE,
STACK HOLDERS ARE WILLING TO
SHARE RESULTS AND WE'RE ABLE TO
ACCOMMODATE VAST DIFFERENCES IN
PRIVACY AND REGULATION IN THOSE
COUNTRIES.
SO NOW, THAT'S A
CHARACTERIZATION STUDY, NOW
LET'S DO SOME CAUSAL INFERENCE,
THAT IS LET'S GENERATE EVIDENCE
THAT SAYS, WHAT CAUSES WHAT.
HOW DO WE DO THIS?
HOW DO WE ADDRESS THE CHALLENGES
I SHOWED EARLIER, WELL, THROUGH
OPEN SCIENCE, FIRST ADMIT
THERE'S A PROBLEM IN
RILIABILITY.
STUDY IT SCIENTIFICALLY AND DO
RESEARCH INTO NEW METHODS TO
ADDRESS IT.
ODESSY IN ITS PREDECESSOR OMOP
HAVE TAKEN A SCIENTIFIC APPROACH
TO THIS OBSERVATIONAL RESEARCH,
HERER OLD ARMS THAT PRECEDED IT
BUT I WILL SHOW YOU IN THE NEXT
SLIDES SPECIFICALLY WHAT WE'RE
WORKING ON RECENTLY.
SO THIS IS A KEY--THIS IS A
SLIDE TO SHOW YOU TO DEFINE
WHAT'S GOING TO BE ON THE NEXT
SLIDE, SO ON THE X-AXIS IS THE
HAZARD RATIO, 1 MEANS A DRUG HAS
NO EFFECT, ON THE Y-AXIS IS THE
STANDARD ERROR OF THAT ESTIMATE,
THE IMPORTANT THING HERE IS THAT
WE'RE THE PURPLE TRIANGLE UP
TOP, THAT'S NOT STATISTICALLY
SIGNIFICANT.
THE TRIANGLE ON THE RIGHT, THE
BLUE 1 MEANS IT'S STATISTICALLY
SIGNIFICANT AND HARMFUL FOR THIS
SIDE EFFECT AND ON THE LEFT, I
GUESS IT'S TAN, OR GOLD, THAT'S
STATISTICALLY SIGNIFICANT AND
PROTECTIVE AGAINST THAT SIDE
EFFECT.
AND WHAT WE WILL DO IS PLOT
STUDIES FROM THE LITERATURE ON
THIS AXIS AND SEE HOW MANY ARE
STATISTICALLY SIGNIFICANT OR
NOT.
SO WE PARSE 29,000 STUDIES AND
PLOTTED THEM ON THIS FIGURE AND
WHAT WE SEE--
AND WHAT'S LACKING IS THE
TRIANGLE IN THE MIDDLE WHICH IS
NOT SIGNIFICANT.
NOW WE KNOW ABOUT PUBLICATION
BIAS WHICH IS WHEN AUTHORS MAY
NOT SUBMIT THE WORK THAT'S NOT
STATISTICALLY SIGNIFICANT AND
EDITORS MAY NOT REVIEW IT AND
REVIEWERS MAY NOT ACCEPT IT AS
UNIMPORTANT IF IT SHOWS NO
RESULT, THOSE NEGATIVE RESULTS
ISSUES THE PROBLEM WITH THIS IS
IF WE DON'T KNOW HOW MANY STUDY
VS BEEN DONE BEFORE THIS
POSITIVE 1 WAS FOUND, THEN WE
CAN'T TELL WHETHER THE STUDY IS
ACCURATE OR NOT SO WHAT IT'S
POSITIVE PREDICTIVE VALUE IS, SO
WE DON'T KNOW THE DENOMINATOR OF
NEGATIVE STUDIES AND THEN WE SEE
SOMETHING ELSE ON THIS FIGURE,
YOU'LL NOTICE THAT IT'S A BIT
HIGHLIGHTED WHERE THOSE ARROWS
SHOW ON THE BOTTOM, RIGHT ON THE
OTHER SIDE OF .05.
I MEAN I SUSPECT THAT WHAT WE'RE
SEEING HERE IS THAT WHEN SOMEONE
GETTINGS A STUDY WITH THE
P-VALUE OF SAY .06 CHRKS IS NOT
STATISTICALLY SIGNIFICANT WHICH
MEANS NO EDITOR OR REVIEWER WILL
BE EXCITED ABOUT IT, YOU TELL
THE POST DOC YOU THINK THEY GOT
IT WRONG AND THAT YOU SHOULD
INCLUDE THIS OTHER VARIABLE OR
USE THIS OTHER METHOD AND
SUDDENLY THE P-VALUE BECOMES .04
AND IT'S STATISTICALLY
SIGNIFICANT AND YOU SAY THAT'S
WHAT I INTEND IN THE FIRST PLACE
AND THAT MUST BE RIGHT WHEN YOU
SENT THAT IN TO BE PUBLISHED AND
THAT'S CALLED THE P-VALUE
TACTSIC, NOW THE WAY I EXPLAINED
IT I MADE IT SOUND DECEPTIVE,
BUT IN FACT, IT HAPPENED SUBTLEY
AND SUBCONSCIOUSLY, SO IT'S
SOMETHING TO WATCH OUT FOR.
SO INDIVIDUALS MAY PRODUCE GOOD
RESEARCH STUDIES BUT IF
AGGREGATE THE MEDICAL RESEARCH
SYSTEM, OBSERVATIONAL RESEARCH
SYSTEM IS KIND OF A DATA
DREDGING MACHINE ONLY PUTTING
OUT THE POSITIVE RESULTS, SO NOW
EACH OF THE FOLLOWING SLIDES
WOULD BE AN HOUR LECTURE SO I
WOULD JUST SUMMARIZE ODESSY'S
APPROACH TO ADDRESS
PREPRODUCIBILITY.
FIRST OF ALL I MENTION
CONFOUNDING EARLIER, EXCUSE
ME--WE DON'T HAVE RANDOMIZATION
AS YOU WOULD HAVE IN A
RANDOMIZED CLINICAL TRIAL SO WE
USE SOMETHING THAT HAS BEEN
AROUND SINCE THE LATE 1970S
PROPENSITY SCORE ADJUSTMENT AND
IT ATTEMPTS TO MAKE THE 2 GROUPS
YOU'RE COMPARING SIMILAR TO EACH
OTHER, AND WHAT WE DO THAT'S A
LITTLE BIT DIFFERENT IS THAT
INSTEAD OF SELECTING WHICH
VARIABLES MIGHT BE DOING THE
CONFOUNDING, IN OTHER WORDS IN
THAT STUDY OF BUTIANYL GAS
VERSUS LUNG CANCER, I CAN THINK
WELL I BETTER CHECK IF THERE'S
SMOKING AND CORRECT FOR THAT IN
MY STUDY, WE DO SOMETHING
SLIGHTLY DIFFERENT WHICH IS WE
ACTUALLY TAKE EVERY VARIABLE
WHICH IS GENERALLY 10S OF
THOUSANDS OF VAB VARIABLES IN
THE ELECTRONIC HEALTH RECORD AND
ADJUST FOR ALL OF THEM.
THERE'S A WHOLE PAPER ON WHY
THAT DOESN'T HURT US IN OTHER
WAYS I DON'T WANT TO GET INTO
THE STATISTICAL INFERENCE, THE
DIAGNOSTICS ARE SHOWN THERE, I
WILL GO INTO DIAGNOSTICS LATER
ON, BUT LET ME JUST SAY, THAT
THE GOOD THING HERE IS WE DON'T
HAVE A HUMAN BEING DECIDING
WHICH 20 SHOULD I CONFOUND FOR
SO WHEN HUMAN BEINGS PREDICT
VARIABLES THEY MISS 1 AND
INCLUDE 1 THAT MAYBE SHOULDN'T
BE THERE WHERE IF YOU TAKE A
SYSTEMATIC APPROACH, THEN WE
TAKE OUT THAT VARIABLE.
BUT WE DON'T TRUST THAT SO THIS
NEXT STEP IS TO USE NEGATIVE
CONTROLS AND YOU HEARD OF
NEGATIVE CONTROLS IN STUDIES
WHERE SOMEONE DOES A STUDY AND
THEN PICK SOMETHING THEY KNOW
AND APPROXIMATE THEY BELIEVE NOT
TO BE TRUE AND MAKE SURE IT CAME
OUT NEGATIVE F. YOU DO 1 OR 2 OF
THOSE, THAT'S ANECDOTAL EVIDENCE
BUT WE TAKE 50-100 NEGATIVE
CONTROLS AND SEE IF NOW UNDER
STATISTICS IF YOU HAVE A--IF
YOU'RE LOOKING FOR A P-VALUE OF
.05, THEN ON AVERAGE 95% OF
THEM, OF YOUR NEGATIVE CONTROL
SHOULD BE NEGATIVE, 5% SHOULD BE
MISTAKENLY POSITIVE.
WHAT WE FIND IN TYPICAL STUDIES
THAT LIKE, 40% OF THEM MAY BE
POSITIVE DUE TO RESIDUAL ERROR
IF WE DON'T DO OUR PROPENSITY
SCORES SO WE DOUBLE CHECK FOR
RESIDUAL CONFOUNDING USING
NEGATIVE CONTROLS.
THEN WE ALWAYS USE MULTIPLE
DATABASES, MULTIPLE LOCATIONS,
MULTIPLE PRACTICE TAPES TO LOOK
FOR CONSISTENCY AND ALSO IF
THOSE DATABASES ARE DIFFERENT
FROM EACH OTHER FOR EXAMPLE IF
OLDER GROUP THAT ADDS TO
GENERALLABILITY.
I ALREADY TALKED ABOUT OPEN
LIST, OPEN LIST ARE A
HYPOTHESIS, CODE, PARAMETERS,
RUNS WE PRESPECIFY THE KNOWN
CODES, WE PUBLISH ALL CODES SO
OTHERS CAN RUN IT, WE PUBLISHED
MASKS RESULTS CHECKED
DIAGNOSTICS AND REVEAL OUR
RESULTS.
AND FIEPALLY WE --FINALLY WE
CARRY OUT A LINE TYPE HYPOTHESIS
AT SCALE THAT IS WE GENERALLY
TEST OUT NOT 1 HYPOTHESIS BUT
SEVERAL AT ONCE, NOT NEGTIARAS
CONTROLS HYPOTH WHERE THERE'S A
POSITIVE ANSWER.
WE DO THIS BY CHECKING
CHARACTERISTICS OF OUR ANALYSIS.
--IF I FIND THAT 98% OF THEM ARE
ALSO POSITIVE, THAT WOULD MAKE
ME WORRY.
AND THEN, AGAIN TO EMPHASIZE
THESE DIAGNOSTICS, THAT IS EVERY
TIME WE MAKE AN ASSERTION THAT
WE THINK THE THING IS BALANCED
WE WANT TO SHOW EVIDENCE THAT
IT'S BALANCED, HERE IS A PLOT OF
BALANCE BETWEEN THE 2 DPROWPS,
SAY, THIS MIGHT BE 60,000 CO
VARIANTS AND WE LOOK TO SEE ON
THE X-AXIS WE LOOK FOR THE
PROENSEL PENSITY ADJUSTMENT, AS
CAN YOU SEE THEY DIFFER BY .02,
THAT'S A HIGH DIFFERENCE IN THE
STANDARDIZED DIFFERENCE OF THE
MEAN.
ON THE Y-AXIS AFTER WE DO OUR
ADJUSTMENT, AFTER WE DO OUR
ADJUSTMENT EVERYTHING'S LESS
THAN .02, .025, AND EVERYTHING'S
UNDER .05 WELL UNDER OUR
THRESHOLD OF .1 WHICH WAS NOT
SET BY US BUT BY THE LITERATURE
SO WE DO OUR PROPENSITY
ADJUSTMENT AND WE DO DIAGNOSTIC
TO DOUBLE CHECK IT DID WHAT WE
EXPECT IT TO DO THAT IS BALANCE
THE 2 GROUPS.
AND THIS IS THE NEG--MAYBE I
SHOULD WIDE JUMPSEN THE
CONFERENCE SO THE RATE OF
FALSE-POSITIVES IS CORRECT.
IF I DO MY--THE ANALYSIS I
SHOWED YOU BEFORE BUT IF I DO IT
NOW USING THE ODESSY METHODS WE
GET THE DIAGRAM SHOWN HERE IN
THE MIDDLE FOR MOST OF THE
RESULTS ARE NEGATIVE WHICH IS
WHAT WE WOULD EXPECT AND
COMPARES ARE POSITIVE, WE EXPECT
5% OF THEM ARE THERE BY CHANCE
ALONE WHICH MEANS 6% ARE REAL
SIDE CENTER FOR EXCELLENCE ON
AGINGS IN THIS CASE THIS WAS A
STUDY OF DEPRESSION DRUGS AND
WHAT ARE THEIR SIDE EFFECTS.
WHAT THAT SAYS IS IN OUR IDEAL
STUDY, 55% OF OUR POSITIVE
RESULTS ARE ACTUALLY POSITIVE,
SO NOW LOOKOT LOWER LEFT WHICH
IS THE LOWER LITERATURE, WHEN
YOU HAVE A NEW OBSERVATIONAL
RESEARCH STUDY THAT SHOWS THE
RESULTS, THE ODDS IT'S CORRECT
ARE PROBABLY WAY LESS THAN 55%.
NOW WHEN WE DO THESE STUDIES,
YOU WILL ASK WELL, WE HAVE SO
MANY HYPOTHESIS, HOW DO YOU
CORRECT FOR THAT, ISN'T THAT
DATA DREDGING.
DATA DREDGE SUGGEST NOT ABOUT
WHAT YOU DO BUT WHAT YOU THROW
AWAY.
THE PROBLEM WITH THE LITERATURE
IS THAT WE'VE THROWN AWAY MOST
OF OUR NEGATIVE STUDIES, WE KEEP
OUR NIGGATIVE STUDIES, WHEN YOU
COME TO US AND TEST A
HYPOTHESIS, WE ASK YOU WHAT ARE
YOUR HYPOTHESIS, WHAT DO YOU
WANT TO TEST AND YOU CORRECT FOR
THOSE HYPOTHESIS, WE NEVER GO IN
OUR DAT BASE AND SAY GIVE ME THE
SMALLEST P-VALUE JUST AS I WOULD
NEVER ASK A STUDENT IF I WERE
NEVER INTERESTED IN A CLINICAL
AREA, I WOULD SAY SCAN
LITERATURE FOR THE SMALLEST
P-VALUES I WOULD SAY HERE'S MY
INFORMATION, WHAT'S YOUR BEST
ESTIMATE, SO WE DO THIS WITH
HYPOTHESIS AND WE DON'T SCOUR
THE RESULTS FOR THINGS THAT ARE
PUBLISHABLE.
WE'VE SUMMARIZED THIS WORK IN
10--WHAT WE CALL LEGEND
PRINCIPLES, LEDGE SEND SHORT FOR
LARGE SCALE EVIDENCE GENERATION
AND EVALUATION ACROSS A NETWORK
OF DATABASES, THIS WILL NOT
DEPEND ON ESTIMATED EFFECTS THAT
MEANS WE WILL NONAPOPTOTIC THE
HAVE PUBLICATION BIAS, WE
PUBLISH ALL OUR STUDIES ON THE
INTERNET EVEN IF THE JOURNALS
DON'T WANT THEM.
WE USE A PRESPECIFIED ANALYTIC
DESIGN.
WE DO CONFOUND AND CONTROL
THROUGH A SYSTEMATIC PROCESS, WE
USE BEST PRACTICES, WE USE
NEGATIVE CONTROL AND POSITIVE
CONTROL QUESTIONS, EVERYTHING IS
OPEN SOURCE WE DON'T EVALUATE
NEW METHODS WHILE TESTING
HYPOTHESIS, WE COME UP WITH
METHODS BUT NOT FOR THE SAKE OF
GENERATING CLINICAL EVIDENCE
THAT WE USE FOR PRACTICE, WE GO
ACROSS MULTIPLE DATABASES AND WE
PROTECT CONFIDENTIALITY.
SO HERE'S AN EXAMPLE OF THE
ODESSY LEDGE ENDED HYPER TENSION
STUDY, HERE'S THE 2017
GUIDELINE, A LITTLE LESS THAN 60
DRUGS, HALF ARE FIRST LINE, HALF
ARE SECOND LINE, NOT MUCH
DIFFERENTIATING AMONG THE FIRST
LINE DRUGS THOUGH, AMONG THE
EVIDENCE THAT'S AVAILABLE.
HERE'S THE EVIDENCE THAT'S
AVAILABLE, SO I TAKE THOSE
61st AND SECOND LINE DRUGS AND
PUT THEM IN A CIRCLE, THOSE ARE
THE DOTS AROUND THE CIRCLE, AND
I LINK THEM WITH ARCs WHETHER
WE HAVE RANDOMIZED TRIAL
EVIDENCE, WHAT YOU CAN SEE A
MOSTLY EMPTY INERN CIRCLE AND
RANDOMIZED TRIAL EVIDENCE IS
JUST A SUBSET WHAT HAVE IS IN
THE GUIDELINE, THE REST IS
EXPERT OPINION, IT'S REASONABLE
OPINION BECAUSE IT SAYS THAT
DRUGS ARE IN A CLASS WILL ACT
SIMILARLY, ALTHOUGH IN SOME
CASES WE DON'T HAVE COMPARISONS
BETWEEN 2 CLASSES NEVER MIND
BETWEEN 2 INDIVIDUALS.
--OUTCOMES WE LOOKEDDA THE BOTH
EFFECTIVENESS AND SAFETY, AND IF
I LOOK AT PAIRS OF DRUGS BECAUSE
REMEMBER A LOT OF TIMES PEOPLE
ARE TAKING DUAL THERAPY SO EVERY
PAIR OF DRUGS AGAINST EVERY PAIR
OF DRUGS AGAINST ALL THESE
OUTCOMES, YOU GET MANY BILLIONS
OF STUDIES, BUT MOST PEOPLE
DON'T TAKE MOST COMBINATIONS SO
IT WAS ABOUT HALF A MILLION
HYPOTHESIS WE TESTED.
AND HERE'S SOME OF THE RESULTS
THAT I WILL SUMMARIZE.
ON THE LEFT IS RANDOMIZED
CLINICAL TRIALS AND IT DIDN'T
SHOW DIFFERENCES AMONG THESE
CLASSES ON THE RIGHT IS OUR
LEGEND, DIRECT EVIDENCE.
ON THE RIGHT IS OUR LEGEND
STUDY, NOW THE KEY HERE IS THAT
IF IT'S GREEN, IT MEANS THE
THING ON THE LEFT IS BETTER AND
IF IT'S PURPLE, THE THING ON THE
BOTTOM IS BETTER, BUT WE SEE IS
THAT EVEN CARDIO SELECTIVE BATTA
BLOCKERS ARE NOT AS GOOD AS THE
OTHER CLASSINGS AND OUR U.S.
GUIDELINE SAYS THAT BETA
BLOCKERS ARE A SECOND LINE
TREATMENT SO OUR EVIDENCE GOES
ALONG WITH THE DECISIONS MADE
FOR THE U.S. GUIDELINE ALTHOUGH
I POINTED OUT THE EUROPEAN
GUIDELINE HAS BETA BLOCKERS, I
THINK HAS BETA BLOCKERS AS FIRST
LEVEL DRUGS.
WE ALSO HAVE THIS EFFECT WHERE
THIACIDES ARE BETTER THAN MY O
CARDIAL INFARCTION AND WE'RE
LOOKING INTO THAT FURTHER.
THOSE ARE CLASSIC BUT WE LOOK AT
INDIVIDUAL DRUGS, THEY'RE SHOWN
HERE, THE FIRST LINE DRUGS ARE
IN THE TOP HALF AND THEN COME
THE SECOND LINE DRUGS.
THE GREEN SQUARE ON THE RIGHT
SAYS THE FIRST LINE DRUG SEEMS
TO BE ON AVERAGE BETTER THAN THE
SECOND LINE DRUG SO IT AGREEN
CELLSS WITH THE GUIDELINE AND
THE LOWER LEFT, THAT'S THE UPPER
RIGHT, THAT'S PURPLE AND HE WILL
THAT SAYS THE SAME THING BECAUSE
THE QUAR IS SYMMETRIC.
ON THE UPPER LEFT YOU SEE THE
GRAY SQUARE AND THAT'S MOSTLY
NOT MUCH THERE, THERE'S SOME
PURPLE AND GREEN THAT'S MOSTLY
DUE TO CAPTOPRIL, AND THEN ON
THE BOTTOM RIGHT, WE SEE A BUNCH
OF PURPLE AND GREEN SAYING AMONG
THE THIS LARGELY AGREES WITH THE
EVIDENCE AND THE DECISIONS THAT
WERE MADE IN THE U.S. GUIDELINE.
ONE SPECIFIC EXAMPLE TO POINT
OUT IS DIURETICS HIDE O CHLOR O
THIOSWRIED IS BEING USED AND
IT'S A DRUG THAT THE GUIDELINE
FOR EXAMPLE, ALTHOUGH IT GIVES
BOTH THE FIRST LINE IT DOES THEY
THAT CHLORTHALIDONE MAY BE
PREFERRED BUT WE'RE FINDING IS
THAT THE 2 DRUGS ARE EQUALLY
EFFECTIVE, LET ME NOT SAY
EQUALLY EFFECTIVE, THERE'S NO
DETECTABLE EFFECTIVENESS AND BIG
DIFFERENCE IN SAFETY, 3 TIMES
FOR HYPOHYPOCANNED WHAT AND
INCLUDING KIDNEY, ELECTROLYTES,
ET CETERA, WE'VE LOOKED INTO
THIS DEEPLY, WE'VE SO CTD IS A
STRONGER DIURETIC AND A LONGER
LASTING 1 SO IT SHOULD BE
WORKING BETTER AND IT'S MORE
EFFECT AND I HAVE THAT'S WHY IT
HAS MORE SIDE EFFECTS BUT WE'RE
NOT SEEING THE UPSIDE ON THE
EFFECTIVENESS.
THAT'S WHY IT'S--I'M SORRY
THAT'S WHY IT HAS A MOST SAFETY
PATTERN BECAUSE IT'S MORE
EFFECTIVE AS A DIARETINAL
LOCATIONIC THEREFORE IT CAUSES
ELECTROLIGHTS TO BE PROBLEMS BUT
WE'RE NOT SEEING A DIFFERENCE IN
THE ACUTE MY O CARDIAL
INFARCTIONS AND THINGS LIKE
THAT, SO THERE'S A COUPLE
REASONS, 1 IS THAT, LET ME NOT
GO INTO TOO MUCH DETAIL BUT IT
SAYS THAT HALF THE PATIENTS STOP
WITHIN THE FIRST 2 MONTHS
PERHAPS BECAUSE IT'S NOT
COMPLETE CONTROL AND A LITTLE
LATER THAT KNOW THAT, PEOPLE
START HIDE HCT SO WE'RE LOOKING
MORE DEEPLY INTO THIS EFFECT
ALSO, NOW IF I TAKE THE LEGEND
STUDY I SHOWED YOU AND I PICK
OUT THE RCTs THAT DO EXIST
WHICH IS ABOUT 40 OF THEM AND 30
OF THEM MATCH OUR STUDIES 10 OF
THEM LIKE WE COULDN'T MATCH BUT
OF THOSE 30, OUR CIs OVERLAP
ON THEM, SO 95% OF THE TIME OUR
LEGEND STUDY MATCHES OUR CTs
IN 5 OR 6% OF THE TIME WE DON'T
BUT OF COURSE BY CHANCE ALONE WE
WOULD EXPECT NOT TO MATCH ON 5
OR 6% OF THE TIME.
SO WE HAVE GOOD CONCORDANCE FOR
RCTs WITH THE THINGS WE'RE
OVERLAPPING.
AND THEN IF YOU LOOK AT THIS
DIAGRAM, THE ODESSY EVIDENCE
THAT WAS PRODUCED HELPS FILL IN
THAT CIRCLE ON THE LEFT AND THE
RESULTS OF THIS STUDY WERE
PUBLISHED IN JOURNALS, LANCET,
JOURNAL OF INTERNAL MEDICINE AND
JAMIA, JUST CHANGING TOPIC FOR A
MOMENT TO POINT OUT THAT ODESSY
WORKS THAT THIRD TYPE OF
EVIDENCE THAT IS PREDICTIVE
ALGORITHMS THAT PATIENT LEVEL
PREDICTION, THE MOST IMPORTANT
THING THERE FOR ODESSY IS I JUST
WANT TO HIGHLIGHT A FEW THINGS
WE DID IN COVID.
THIS IS THE TIMELINE.
WE HAVE MANY MILLIONS OF COVID
RECORDS WE PUBLISHED 41 STUDIES
IN THE PASTIER ON COVID BUT
PUBLISHING STUDIES IS 1 THING,
THE BIGGER QUESTION IS WHAT
IMPACT.
HERE'S A STUDY ON THE SAFETY OF
HYDROXYCHLOROQUINE.
BASICALLY WHEN
HYDROXYCHLOROQUINE WAS BEING
CONSIDERED, USED BY BOTH THE
EUROPEAN MEDICINE'S AGENCY OR
EMA WHICH IS SHOWN HERE AND THE
FDA, THEY'RE LOOKING AT BOTH
SAFETY AND EFFECTIVENESS AND
ODESSY STUDY IS 1 OF 2 STUDIES
THAT THE EUROPE MEDICINE'S
AGENCY HIGHLIGHTED AS THEIR
EVIDENCE FOR WHY THEY DECIDED TO
RETRACT SUPPORT FOR USE OF
HYDROXYCHLOROQUINE IN THE
SETTING OF COVID.
SO THAT WAS A STUDY, THE
EUROPEAN MEDICINE AGENCY, TENDS
TO TELL YOU WHAT EVIDENCE THEY
USE TO MAKE THEIR DECISION, SO I
CAN POINT TO THINGS WHERE THEY
HIGHLIGHTED ODESSY EVIDENCE.
A SECOND 1 IS ACE INHIBITORS AND
ARMS AND SUSCEPTIBILITY, WE WERE
WORRIED BECAUSE OF THE
SIMILARITY OR USE OF THE
CHANNELS THAT THE ACE--TAKING AN
ACE INHIBITOR OR AN R, ANTIHYPER
TENSION DRUG, MIGHT MAKE YOU
MORE SUSCEPTIBLE TO COVID-19
INFECTION OR COMPLICATIONS, LIKE
OTHERS WE DID NOT FIND THAT
EFFECT.
BUT WHAT I HIGHLIGHT HERE IN THE
BOTTOM IS THE EUROPEAN
MEDICINE'S AGENCY AGAIN,
POINTING OUT THE MORALES STUDY
WHICH WAS OURS AND USE IT AS AN
EXEMPLAR OF REPRODUCIBLE
RESEARCH SO THAT THE POLICY
MAKERS ARE LOOKING AT THE WAY
EVIDENCE IS GENERATED AND SEEING
THIS AS A TEMPLATE FOR HOW
RESEARCH SHOULD BE CONDUCTED.
[INDISCERNIBLE] ARE LARGE
STUDIES ON DRUG SAFETY AND
PATIENT CHARACTERISTICS.
SOMETIME AGO WE SWUNG FULLY INTO
VACCINE SAFETY RESEARCH, IF YOU
REMEMBER THIS PAST YEAR IN
MARCH, ASTRAZENECA, WAS HELD IN
13 COUNTRIES BECAUSE OF FEAR OF
BLOOD CLOTS THEN ON
MARCH 18th, THE EMA DETERMINED
THAT THE BENEFITS OUTWEIGH THE
RISK THAT YES THERE'S SOME RISK
OF CLOT BUT LIFE ON EARTH LOWER
THAT THE RISK OF COVID.
WELL THAT DETERMINATION WAS
BASED ON THE ODESSY DATA FOR THE
BASE LINE RATES SO THIS WAS
PUBLISHED LATER IN BMJ, AND WE
GOT A LETTER FROM THE EMA TO
PUBLISH THE STUDY SO THAT IT
WOULD INTEREST INTO THE
LITERATURE.
WE ALSO WORK STRONGLY WITH THE
U.S. FDA, THE CENTER FOR
BIOLOGICS EVALUATION ON RESEARCH
OR CBER, AND THEN FUN STUFF
LOOKING IN CHANGES IN POLICY IN
HOW IT CORRELATES WITH EVIDENCE,
THAT IS WHAT WAS PUBLISHED AND
HOW DID USAGE CHANGE AND HOW DID
POLICY DECISIONS FROM THE
AGENCIES IN U.S. AND EUROPE
CHANGE?
AND THEN SOME WORK ON LONG COVID
SHOWING HERE OF COURSE THERE'S
ALSO SAYING, WELL, IT'S HARD TO
SAY, WHICH SYMPTOMS OCCUR LONG
AFTER COVID WERE NOT JUST
BECAUSE THE PATIENT WAS GOING TO
GET THOSE THINGS BECAUSE OF
THEIR HIGHER RISK OF GETTING
COVID OR HAVING SEVERE COVID
BECAUSE THEY'RE OLDER ANDIC
ISER, MAYBE THOSE ARE REALLY
RELATED TO THEIR OTHER DISEASE,
SO WHAT WE DID IS WE SAID IS
WHAT'S LONG INFLUENZA, WHAT'S
LONG COVID, AND WHAT POPS THEN
AND THEN YOU GET ALTERED TASTE
AND SMELL, MY O CARDITIS ACUTE
KIDNEY INJURY, ALOPECIA AND
THINGS THAT IT BEINGLY SHOW UP
IN POST COVID RATHER THAN POST
OTHER VIRALS.
THIS WAS PUBLISHED IN TOP
JOURNALS AROUND THE WORLD.
WE DID WORK WITH THE NATIONAL
CANCER INSTITUTE UNDERSTANDING
THE SEQUENCE, I WILL GO THROUGH
THIS QUICKLY, IT'S BASICALLY A
MEASUREMENT STUDY TO SAY COULD
ODESSY WORK DO WORK IN CANCER MY
POST DOC WAS WORKING ON THIS, HE
FIRST REPEAT THAD TREATMENT
PATHWAYS PAPER I SHOW INDEED
CANCER AND SHOWED THAT PATIENTS
WITH CANCER GET A SIMILAR
SEQUENCE OF DRUGS TO PATIENTS
WITHOUT CANCER.
WE SHOWED HIGH SENSITIVITY AND
SPECIFICITY IN SAYING DO YOU
HAVE A CANCER DIAGNOSIS AND DID
YOU GET A PARTICULAR CANCER
TREATMENT ALSO HIGH SENSITIVITY
AND SPECIFICITY.
WE LOOKED AT THINGS LIKE CANCER
TREATMENTS OVERTIME, THIS IS FOR
PROSTATE CANCER SHOWING, FOR GET
ABOUT THE LEFT HAND OF THE
FIGURE, LOOK THE LEFT HAND, BUT
ON THE RIGHT HAND PROSECTOMYOSIN
ME, DECREASING OVER TIME,
CREATION THERAPY, STAYING ABOUT
THE SAME OR COMING UP AT THE
END, HORMONE THERAPY RISING AND
OTHER FORMS OF CHEMO THERAPY
FALLING OVER TIME SO OUR ODESSY
ONCOLOGY WORKING GROUP FORMED IN
2017 IS EXTENDING--BECAUSE THE
COMPLEXITY OF THE PHENOTYPE AND
THE TREATMENT PATHWAYS SO YOU
NEED A DATA MODEL AND YOU NEED
ANALYTIC TOOLS THAT CAN HANDLE
THAT AND THAT'S WHAT THAT
WORKING GROUP IS WORKING ON.
AND JUST TO POINT OUT THIS WORK
ON CANCER IN THE SETTING OF
COVID-19, WHAT YOU SEE HERE,
THESE GRAY FIGURURES, I WILL
JUST SAY QUICKLY THAT THEY LOOK
PRETTY MUCH SYMMETRIC WHICH
MEANS IN THE SETTING OF COVID
AND THE SETTING OF INFLUENZA,
CANCER PATIENTS ARE NO DIFFERENT
THAN OTHER PEOPLE IN OTHER WORDS
IT DOESN'T SEEM THAT COVID IS
PARTICULARLY HURTING CANCER
ANYMORE THAN IT HURTS--ANYMORE
THAN INFLUENZA DOES.
AND 1 EFFORT THAT IS GETTING
CLOSER TO CLINICAL CARE IS THE
DAILY BASIS THEA CONSULT
SERVICE, THANK YOU TO STANFORD'S
GREEN BUTTON TEAM FOR INSIGHTS
WHO PRODUCED A SIMILAR SERVICE
EARLIER [INDISCERNIBLE] IS THE
PERSON AT COLUMBIA WHO DID THIS.
BASICALLY YOU GO ON A TEAM, YOU
LOOK IF THE LITERATURE, IF IT'S
NOT THERE, YOU GENERATE IT IN A
DAY ON THE ODESSY BASIS, SOMEONE
LIKE A PATIENT WITH PENICILLIN
HAD ANGIO DEMA AND THE QUESTION
WAS DOES PENICILLIN TYPICALLY
CAUSE ANGIO DEMA, THE ANSWER WAS
NO, BUT WAS IT A SIDE EFFECT
THEY SHOULD HAVE KNOWN ABOUT?
TALKING ABOUT ODESSY
COLLABORATIONS AS WE WORK WITH
THE FOOD AND DRUG ADMINISTRATION
CBER AND IT SERVES AS A CONVENER
FOR ACTIVITIES THERE IN
BIOLOGICS, AND THE ALL OF US
RESEARCH PROGRAM AND A MODEL AS
THE BASIS OF THAT AND WE WORK ON
PROCESSING THAT DATABASE,
COLLECTING THE DATA AND MAKING
IT RELIABLE DATABASE.
THE NIH'S EMERGE INITIATIVE,
OMOP BEING A COMMON DATA MODEL
FOR THAT 1 TOO AND COLUMBIA
PARTICIPATE NOTHING THAT, AND N3
C, THE NATIONAL COHORT
COLLABORATIVE, BEING THE BASIS
OF THAT, ALSO IN EUROPE, THE
EATON INITIATIVE, SO THE LARGE
EUROPEAN NETWORK USING OMOP AND
FOR EXAMPLE, COUNTRIES LIKE
KOREA, PICKING OMOP AS THE BASIS
OF ITS WORK AND DOG A LOT OF
STUDIES USING IT.
SO TO SUMMARIZE I WILL SAY THE
CURRENT OBSERVATIONAL RESEARCH
IS SUSPECT, POLICIES TO BE SET,
LARGE SCALE OBSERVATION APPEARS
TO BE POSSIBLE AND MORE RELIABLE
THAN CURRENT APPROACHES ISSUES
USING THOSE THINGS I ALREADY
OUTLINES WE'RE GENERATING
EVIDENCE IS PUBLISH NOTHING TOP
JOURNALS, DATA MODELS BEING USED
IN U.S. AND DPLOABALLY, WE'RE
SUCCESSFULLY, COLLABORATING ON
GENERATING EVIDENCE AND JUST GO
TO ODESSY.ORGFOR MORE
INFORMATION.
I'M GOING TO SKIP THE NEXT
COUPLE OF SLIDES.
THAT WAS THERE IN CASE I WANT TO
SAY THANK YOU A LOT WAS STARTED
BY RO1 LMOO 16-10 WHICH I HAD
WELL FOR 2121 YEARS IT WAS
FUNDED FOR A BIT LONGER AND OUR
TRAINING GRANT WHICH HAS BEEN
AROUND FOR 30 YEARS.
>> IDDK BOTH IN FUNDING AND
COLLABORATION ON SOME OF THE
WORK THAT I WASN'T ABLE TO SHOW
AT THE END, NCI CONTRACTS AND
NHGRI, AND FDA CBER BEST, IT'S
BEEN AN HONOR TO DO THIS WORK
AND IT'S BEEN FUN DOING THIS,
OUR MOTTO AT COLUMBIA
BIO-INFORMATICS MODEL IS
DISCOVER AND IMPACT AND THAT'S
WHAT WE STRIVE TO ACHIEVE.
LET ME STOP THERE.
>> THANK YOU SO MUCH
DR. HRIPSCAK, IT'S AN INCREDIBLE
HONOR TO HAVE YOU HERE AND SPEAK
TO US AND I CERTAINLY LEARNED A
LOT, I'M SURE EVERYONE IN THE
AUDIENCE DID AS WELL, I DO WANT
TO SHARE THAT SINCERE APOLOGIES
TO YOU AND THE AUDIENCE FOR OUR
PROBLEM EARLIER.
IT WAS ACTUALLY A HARDWARE
GLITCH AND I APPRECIATE IT'S
PROMPT DETECTION AND TROUBLE
SHOOTING OF THE ISSUE.
WE HAVE ABOUT 3 OR 4 MINUTES AND
I WANT TO LEVERAGE THIS TIME TO
MAYBE ASK A FEW QUESTIONS.
FIRST THERE IS THIS PROBLEM OF
AND OVER AND ABOVE, THE FACT
THAT PEOPLE LOOK AT
OBSERVATIONAL STUDIES WITH A
GRAIN OF SALT BECAUSE OF THE
LACK OF RANDOMIZATION, ON TOP OF
THAT, WHEN THERE'S THIS ISSUE
OF, YOU KNOW WHAT WAS A PRIORITY
AND WHAT WAS PRESPECIFIED, THERE
HAS BEEN A PARADIGM SHIFT
TOWARDS ENCOURAGING
INVESTIGATORS TO REGISTER THEIR
OBSERVATIONAL STUDIES.
WHAT ARE YOUR VIEWS ON THAT
BECAUSE I THINK THAT THERE'S NO
HARD AND FAST POLICY, AND DO YOU
FEEL LIKE THAT HAS HELPFUL AND
WILL OFFER--
>> AND SEE WHAT WE CHANGE IN
ADDITION WE TRY TO PUT IT IN
NATIONAL AND INTERNATIONAL REG
STEES IN THIS EDITION BUT WE
ALWAYS PUT IT ON GITHUB BUT IT
ALWAYS MAKES A DIFFERENCE
BECAUSE IT'S SO HARD NOT TO
ACCIDENTALLY STEAL RESULTS, LOOK
IF SOMEONE'S GOING TO BE
DECEPTIVE, THEY WILL DO
ANYTHING, WE ARE TRYING TO
IMPROVE THE QUALITY AND HONESTY
OF EVIDENCE PRESENTED BY THEM
AND PRESPECIFYING YOUR PROTOCOL
IS 1 OF THE SINGLE BIGGEST
THINGS WE CAN DO EASILY BECAUSE
WE'RE USED TOKING IT FOR
RANDOMIZED TRIALS AND WHERE ARE
WE NOT DOING IT FOR
OBSERVATIONAL RESEARCH.
I AGREE IN MY OWN EXPERIENCE I
FIND IT HELPFUL NOW TO REGISTER
EVEN OUR OBSERVATIONAL STUDIES,
AND INITIALLY THERE WAS A
CONCERN THAT WE MAY NOT BE ABLE
TO CHANGE THINGS LATER ON, BUT I
THINK THAT BY DEFAULT THIS IS A
TYPE OF WORK THAT DOES REQUIRE
CHANGES AND JUST REQUIRES A
RATIONAL EXPLANATION AS TO WHY
THE CHANGE HAD TO BE DONE.
>> EXACTLY SO WE DO WANT TO
SUPPORT THOSE CHANGES BUT THEN
PEOPLE CAN SEE WHAT YOU DID AND
YOU CAN ASSESS WHY IS WHETHER
THAT WILL HAVE AN EFFECT ON THE
TRUSTWORTHINESS.
>> THAT'S CORRECT.
>> YOU KNOW THE OTHER QUESTION
IS YOU ARE ON THE INTRAMURAL
SIDE BUT YOU'RE AWARE OF THE
ACTIVITIES THAT OCCUR THAT HERE
ON THE CAMPUSOT INTRAMURAL SIDE.
DO YOU HAVE A SENSE OF WHAT THE
FUTURE SCOPE MIGHT BE FOR THIS
TYPE OF RESEARCH WITHIN THE
INTRAMURAL SPACE AND HOW
INTRAMURAL INVESTIGATORS CAN
LEVERAGE THIS NOT ONLY FOR THEIR
OWN STUDIES BUT ALSO
INDEPENDENTLY DURING THIS WORK?
>> SURE, THING.
WELL A COUPLE OF THINGS, SO
NUMBER 1, YOU KNOW, YOU CAN LOOK
AT THIS EXTERNAL BODY OF
OBSERVATIONAL RESEARCH AS BEING
HYPOTHESIS GENERATE FIGURE THE
STUDIES YOU DO INTRA MIEWRALLY,
AS WELL AS HYPOTHESIS TESTING
THAT ARE GENERATED INTRA
MIEWRALLY, SO THAT'S GOOD.
IT MIGHT BE WORTH WHILE FOR THE
CLINICAL CENTER TO HAVE SOMEONE
BECOME PART OF ODESSY, THAT IS,
YOU KNOW TO HAVE A
COLLABORATION, YOU REALLY NEED
SOMEONE WHO STANDS BETWEEN AND
INTERPRETS BETWEEN THE 2 SO
HAVING SOMEONE AT A POST DOC
LEVEL, GOES ALL IN AND LEARNS
ABOUT IT AND DECIDES STUDIES AND
PUTS THEM FORWARD TO THE NETWORK
COULD BEING A PATHWAY TO GET
THINGS DONE.
NOW THE CLINICAL CENTER COULD
PUT ITS DATA INTO THE OMOP DATA
MODAND HE WILL RUN STUDIES AND
BECOME A NODEOT ODESSY NETWORK,
REMEMBER WE NEVER SHARE DATA SO
IT'S NOT A PRIVACY CONCERN.
BUT YEAH AND THEN, I DIDN'T GET
A CHANCE TO SHOW TODAY BUT WE
ARE ALSO BRANCHING INTO THINGS
THAT INVOLVE PHYSIOLOGY AND SAY
WORKING WITHIN NIDDK ON THE
METABOLISM, THAT'S WHERE WE'RE
BOTH USING OBSERVATIONAL
RESEARCH AND LOOKING AT
DEEP--EXCUSE ME PHYSIOLOGIC
MODELS, AND JOINING THOSE 2 AND
THAT'S BEEN QUITE FUN.
I THINK THERE'S A LOT OF
POTENTIAL THERE IN INTRAMURAL
RESEARCH.
>> WONDERFUL.
WE'VE UNFORTUNATELY RUN OUT OF
TIME.
WE HAVE A LOT MORE QUESTIONS
THAT CAME THROUGH AND GIVEN THE
FACT THAT WE LOST SOME TIME
WE'RE UNABLE TO TOUCH ON ALL OF
THEM, BUT AGAIN, ONCE AGAIN
THANK YOU SO MUCH FOR YOUR TIME
AND THANK YOU EVERYONE FOR
LISTENING.
>> THANK YOU SAMEER.
>> HAVE A GOOD DAY, EVERYONE.